How to join to table with Sequelize without relations - sql

I made an SQL command line to select what the data I want but I can't find a way to do it with Sequelize.
I have two tables. One with some id linked to names and the other one with long texts.
I want to select all the text containing names from the id of the name.
This SQL command is working :
SELECT
texts.id,
texts.content
FROM
text
INNER JOIN users AS u ON u.id=38
WHERE
LOWER(texts.content) like concat('%',u.name,'%')
When I try to do it with Sequelize, I have something like this :
this.modelTexts.findAll({
include: [
{
model: this.modelUsers,
where: { id: 38 },
}
],
where: ...
})
When I try this, I have this error message :
SequelizeEagerLoadingError: users is not associated to texts!
I know I have to make something like :
Users.hasMany(Texts)
But I can't find a way to make it.
Thanks for taking a bit of your time for this request !

You can use Sequelize Raw for this, so your query can be something like this
sequelize.query(`
SELECT
texts.id,
texts.content
FROM
text
INNER JOIN users AS u ON u.id=38
WHERE
LOWER(texts.content) like concat('%',u.name,'%')`
{ type: sequelize.QueryTypes.SELECT});
Or you can add associations. So wherever in your code you have define models, add the following code
User.hasMany('Text', {foreignKey: 'user_id'});

NB: this solution was associated but if what you mean is no foreign keys on both tables
To solve this, here is what i would do
I Added this to my association first
models.User.hasMany(models.Text)
then also added this in my find query
Text.findAll({
attributes: ['id', 'content'],
include: [
{
model: users,
on: Sequelize.literal('users.id = 13')
},
],
});

While the above answer is great, I think it may defeat the whole purpose of using sequelize. Have you tried the following:
In your Text model definition:
Text.associate = models => {
Text.belongsTo(models.User)
}
In your User model definition:
User.associate = models => {
User.hasMany(models.Text)
}
Let me know if you're still having trouble! Understanding how sequelize works will save you many many hours in the long run! :)

Related

How to achieve generic Audit.NET json data processing?

I am using Audit.Net library to log EntityFramework actions into a database (currently everything into one AuditEventLogs table, where the JsonData column stores the data in the following Json format:
{
"EventType":"MyDbContext:test_database",
"StartDate":"2021-06-24T12:11:59.4578873Z",
"EndDate":"2021-06-24T12:11:59.4862278Z",
"Duration":28,
"EntityFrameworkEvent":{
"Database":"test_database",
"Entries":[
{
"Table":"Offices",
"Name":"Office",
"Action":"Update",
"PrimaryKey":{
"Id":"40b5egc7-46ca-429b-86cb-3b0781d360c8"
},
"Changes":[
{
"ColumnName":"Address",
"OriginalValue":"test_address",
"NewValue":"test_address"
},
{
"ColumnName":"Contact",
"OriginalValue":"test_contact",
"NewValue":"test_contact"
},
{
"ColumnName":"Email",
"OriginalValue":"test_email",
"NewValue":"test_email2"
},
{
"ColumnName":"Name",
"OriginalValue":"test_name",
"NewValue":"test_name"
},
{
"ColumnName":"OfficeSector",
"OriginalValue":1,
"NewValue":1
},
{
"ColumnName":"PhoneNumber",
"OriginalValue":"test_phoneNumber",
"NewValue":"test_phoneNumber"
}
],
"ColumnValues":{
"Id":"40b5egc7-46ca-429b-86cb-3b0781d360c8",
"Address":"test_address",
"Contact":"test_contact",
"Email":"test_email2",
"Name":"test_name",
"OfficeSector":1,
"PhoneNumber":"test_phoneNumber"
},
"Valid":true
}
],
"Result":1,
"Success":true
}
}
Me and my team has a main aspect to achieve:
Being able to create a search page where administrators are able to tell
who changed
what did they change
when did the change happen
They can give a time period, to reduce the number of audit records, and the interesting part comes here:
There should be an input text field which should let them search in the values of the "ColumnValues" section.
The problems I encountered:
Even if I map the Json structure into relational rows, I am unable to search in every column, with keeping the genericity.
If I don't map, I could search in the Json string with LIKE mssql function but on the order of a few 100,000 records it takes an eternity for the query to finish so it is probably not the way.
Keeping the genericity would be important, so we don't need to modify the audit search page every time when we create or modify a new entity.
I only know MSSQL, but is it possible that storing the audit logs in a document oriented database like cosmosDB (or anything else, it was just an example) would solve my problem? Or can I reach the desired behaviour using relational database like MSSQL?
Looks like you're asking for an opinion, in that case I would strongly recommend a document oriented DB.
CosmosDB could be a great option since it supports SQL queries.
There is an extension to log to CosmosDB from Audit.NET: Audit.AzureCosmos
A sample query:
SELECT c.EventType, e.Table, e.Action, ch.ColumnName, ch.OriginalValue, ch.NewValue
FROM c
JOIN e IN c.EntityFrameworkEvent.Entries
JOIN ch IN e.Changes
WHERE ch.ColumnName = "Address" AND ch.OriginalValue = "test_address"
Here is a nice post with lot of examples of complex SQL queries on CosmosDB

How do I filter fields with Directus

Let's say I make an api call like this
const { data } = await client.getItems(`module/${module.id}`, {
fields: [
'questions.module_question_id.question_text',
'questions.module_question_id.slug',
'questions.module_question_id.type',
'questions.module_question_id.answer_options.*',
],
});
I am grabbing the fields, but I also want to filter out a certain question ala its slug, is there a way to do this at the api level? I know filters exist as a global query api, but have not found examples of them being used in conjunction with fields.
Perhaps you are looking for deep? This should allow you to filter on a deeply nested relational field.
https://docs.directus.io/reference/api/query/#deep

Search related data in different data base

I use yii2 to build one app which need to connect some tables. I can join them simply and search for data in related fields. I am doing it by adding the connection like this>
public function getNextTab()
{
return $this->hasOne(NextTab::className(),['id' =>'id_nexttab']);
}
and ask for the data in search model using like this ->
->where ('id'='ok') ->
->joinWith('nextTab')
->joinWith('nextTab.nextTab1')
->joinWith('nextTab.nextTab1.nextTab2');
My problem is when I try to do this with tables from different database. The query is give me error like
SQLSTATE[42S02]: Base table or view not found:
any tips how to pass it? or how to do other way of connection to have the data.
Joining tables from different databases may not be supported by your RDBMS (PostgreSQL for example). But if supported (MSSQL, MySQL) then table names should be prefixed with database name (and schema if needed). You can achieve this in Yii2 using {{%TableName}} syntax in tableName() function.
public static function tableName()
{
return '{{%table_name}}';
}
But be careful with joining tables from different databases if they are located on different servers -- this can be very slow.
If you just want to get related data (joined tables are not used in WHERE) then use with() instead of joinWith(). This will be executed as separate query with IN statement. In most cases this way has a better performance and no problems with different sources (and even different DBMS).
->with('nextTab', 'nextTab.nextTab1', 'nextTab.nextTab1.nextTab2')
Configure your second database component in the application's config.
Override the getDB() function in your ActiveRecord Model to return the second DB component.
This will attach your Model to the secondary DB and allow you to query from the table in secondary DB.
Config sample:
'components' => [
'db2' => [
'class' => 'yii\db\Connection',
'dsn' => 'mysql:host=localhost;dbname=db2',
'username' => 'hello',
'password' => 'world',
],
],
getDB() function override sample:
class Tab extends ActiveRecord
{
public static function getDb()
{
// use the "db2" component
return \Yii::$app->db2;
}
}
Good Luck!

Delete record field via deepstream Record

I'm looking for a way to clean document nested field, for example, consider I have a JSON object:
{
fieldToClean: {
fieldA: '..',
fieldB: '..',
fieldC: '..'
}
}
I know that I don't need fieldB anymore. I found one solution that looks like:
var record = deepstream.record.getRecord('<proper path>')
record.whenReady(function(){
var fieldToClean = record.get('fieldToClean')
delete fieldToClean.fieldB
record.set('fieldToClean', fieldToClean)
})
I wonder if deepstream provides something like:
record.delete('fieldToClean.fieldB')
or
record.set('fieldToClean.fieldB', undefined)
I wasn't able to find something like this in documentation.
Thank you for your time!
There's actually an issue for this open, our main design question is around deleting an index in array. Is that a null or splice? Be great to have your feedback!
https://github.com/deepstreamIO/deepstream.io/issues/29

MongoDB Update / Upsert Question - Schema Related

I have an problem representing data in MongoDB. I was using this schema design, where a combination of date and word is unique.
{'date':2-1-2011,
'word':word1'
users = [user1, user2, user3, user4]}
{'date':1-1-2011,
'word':word2'
users = [user1, user2]}
There are a fixed number of dates, approximately 200; potentially 100k+ words for each date; and 100k+ users.
I inserted records with an algorithm like so:
while records exist:
message, user, date = pop a record off a list
words = set(tokenise(message))
for word in words:
collection1.insert({'date':date, 'word':word}, {'user':user})
collection2.insert('something similar')
collection3.insert('something similar again')
collection4.insert('something similar again')
However, this schema resulted in extremely large collections and terrible performance was terrible. I am inserting different information into each of the four collections, so it is an extremely large number of operations on the database.
I'm considering representing the data in a format like so, where the words and users arrays are sets.
{'date':'26-6-2011',
'words': [
'word1': ['user1', 'user2'],
'word2': ['user1']
'word1': ['user1', 'user2', 'user3']]}
The idea behind this was to cut down on the number of database operations. So that for each loop of the algorithm, I perform just one update for each collection. However, I am unsure how to perform an update / upsert on this because with each loop of the algorithm, I may need to insert a new word, user, or both.
Could anyone recommend either a way to update this document, or could anyone suggest an alternative schema?
Thanks
Upsert is well suited for dynamically extending documents. Unfortunately I only found it working properly if you have an atomic modifier operation in your update object. like the $addToSet here (mongo shell code):
db.words is empty. add first document for a given date with an upsert.
var query = { 'date' : 'date1' }
var update = { $addToSet: { 'words.word1' : 'user1' } }
db.words.update(query,update,true,false)
check object.
db.words.find();
{ "_id" : ObjectId("4e3bd4eccf7604a2180c4905"), "date" : "date1", "words" : { "word1" : [ "user1" ] } }
now add some more users to first word and another word in one update.
var update = { $addToSet: { 'words.word1' : { $each : ['user2', 'user4', 'user5'] }, 'words.word2': 'user3' } }
db.words.update(query,update,true,false)
again, check object.
db.words.find()
{ "_id" : ObjectId("4e3bd7e9cf7604a2180c4907"), "date" : "date1", "words" : { "word1" : [ "user1", "user2", "user4", "user5" ], "word2" : [ "user3" ] } }
I'm using MongoDB to insert 105mil records with ~10 attributes each. Instead of updating this dataset with changes, I just delete and re insert everything. I found this method to be faster than individually touching each row to see if it was one that I needed to update. You will have better insert speeds if you create JSON formatted text files and use MongoDB's mongoimport tool.
format your data into JSON txt files (one file per collection)
mongoimport each file and specify the collection you want it inserted into