In my Nodered Bluemix application, I'm trying to make a SqlDB query, but I can't find sufficient documentation or examples on how to use the parameter markings in the query. Are there any examples and further insight into what I am doing wrong? Here is the flow I am having trouble with:
[
{
"id":"7924a83a.03355",
"type":"websocket-listener",
"path":"/ws/dbdata",
"wholemsg":"false"
},
{
"id":"b84efad2.9a2a58",
"type":"function",
"name":"Parse JSON",
"func":"msg.payload = JSON.parse(msg.payload);\nvar begin = msg.payload[0].split(\" \");\nbegin[1] = begin[1]+\":00\";\nvar date1 = begin[0].split(\"-\");\nvar processStart = date1[2]+\"-\"+date1[0]+\"-\"+date1[1]+\" \"+begin[1];\n\nvar end = msg.payload[0].split(\" \");\nend[1] = end[1]+\":00\";\nvar date2 = end[0].split(\"-\");\nvar processEnd = date2[2]+\"-\"+date2[0]+\"-\"+date2[1]+\" \"+end[1];\n\nmsg.payload[0] = processStart;\nmsg.payload[1] = processEnd;\nreturn msg;",
"outputs":1,"noerr":0,"x":381.79998779296875,"y":164.8000030517578,"z":"3f9da5d2.b3f0aa",
"wires":[["4f92b16a.cf981"]]
},
{
"id":"3e20f8a4.06451",
"type":"websocket in",
"name":"dbInput",
"server":"7924a83a.03355",
"client":"",
"x":159.8000030517578,"y":164.8000030517578,"z":"3f9da5d2.b3f0aa",
"wires":[["b84efad2.9a2a58"]]
},
{
"id":"68a4a35.5983f5c",
"type":"debug",
"name":"",
"active":true,"console":"false",
"complete":"true",
"x":970.7999877929688,"y":162.8000030517578,"z":"3f9da5d2.b3f0aa",
"wires":[]
},
{
"id":"5a0aed1c.34279c",
"type":"sqldb in",
"service":"LabSensors-sqldb",
"query":"",
"params":"{msg.begin},{msg.end}",
"name":"db Request",
"x":787.7999877929688,"y":163.8000030517578,"z":"3f9da5d2.b3f0aa",
"wires":[["68a4a35.5983f5c"]]
},
{
"id":"e08c4a85.e95e68",
"type":"debug",
"name":"",
"active":true,"console":"false",
"complete":"true",
"x":791.7999877929688,"y":233.8000030517578,"z":"3f9da5d2.b3f0aa",
"wires":[]
},
{
"id":"4f92b16a.cf981",
"type":"function",
"name":"Construct Query",
"func":"msg.begin = msg.payload[0];\nmsg.end = msg.payload[1];\nmsg.payload = \"SELECT * FROM IOT WHERE TIME >= '?' AND TIME < '?'\";\nreturn msg;",
"outputs":1,"noerr":0,"x":583.7999877929688,"y":163.8000030517578,"z":"3f9da5d2.b3f0aa",
"wires":[["5a0aed1c.34279c",
"e08c4a85.e95e68"]]
}
]
In the node-red documentation for the SQLDB query node it says:
"Parameter Markers is a comma delimited set of json paths. These will replace any question marks that you place in your query, in the order that they appear."
Have you tried removing the curly braces, i.e. to set the "params" field in the node to just "msg.begin,msg.end"?
You just need remove single quotes this is a correct sentence:
msg.payload = "SELECT * FROM IOT WHERE TIME >= ? AND TIME < ?";
Related
I have been struggling with this for a couple of days now and I felt like I should reach out. This might be very simple but I am not from a programming background and I haven't found any resources to solve this so far.
Basically, I want to parameterize a SQL query that is running for BigQuery within Google APp Script, it takes a variable from a user from a Google From they have submitted and I wanted to ensure that this won't be injectable by parameterizing the query, however, I got the following error that I could not fix:
GoogleJsonResponseException: API call to bigquery.jobs.query failed with error: Query parameter 'account_name' not found at [1:90]
Here is how I run the query:
//Query
const sqlQuery = 'SELECT district FROM `table` WHERE account_name = #account_name AND ent_theatre=("X") LIMIT 1;'
const request = {
query: sqlQuery,
params: { account_name: queryvar },
useLegacySql: false,
};
// Run Query
var queryResult = BigQuery.Jobs.query(request,projectID);
I have created the query based on Google's documentation
Your syntax for request object is not correct. The right syntax for the BigQuery.Jobs.query Request is like below:
const request = {
query: sqlQuery,
queryParameters: [
{
name: "account_name",
parameterType: { type: "STRING" },
parameterValue: { value: queryvar }
}
],
useLegacySql: false,
};
For more detail about QueryRequest Object refer to this link.
I am having multiple nested where conditions and want to generate them without too much code duplication with typeORM.
The SQL where condition should be something like this:
WHERE "Table"."id" = $1
AND
"Table"."notAvailable" IS NULL
AND
(
"Table"."date" > $2
OR
(
"Table"."date" = $2
AND
"Table"."myId" > $3
)
)
AND
(
"Table"."created" = $2
OR
"Table"."updated" = $4
)
AND
(
"Table"."text" ilike '%search%'
OR
"Table"."name" ilike '%search%'
)
But with the FindConditions it seems not to be possible to make them nested and so I have to use all possible combinations of AND in an FindConditions array. And it isn't possible to split it to .where() and .andWhere() cause andWhere can't use an Object Literal.
Is there another possibility to achieve this query with typeORM without using Raw SQL?
When using the queryBuilder I would recommend using Brackets
as stated in the Typeorm doc: https://typeorm.io/#/select-query-builder/adding-where-expression
You could do something like:
createQueryBuilder("user")
.where("user.registered = :registered", { registered: true })
.andWhere(new Brackets(qb => {
qb.where("user.firstName = :firstName", { firstName: "Timber" })
.orWhere("user.lastName = :lastName", { lastName: "Saw" })
}))
that will result with:
SELECT ...
FROM users user
WHERE user.registered = true
AND (user.firstName = 'Timber' OR user.lastName = 'Saw')
I think you are mixing 2 ways of retrieving entities from TypeORM, find from the repository and the query builder. The FindConditions are used in the find function. The andWhere function is use by the query builder. When building more complex queries it is generally better/easier to use the query builder.
Query builder
When using the query build you got much more freedom to make sure the query is what you need it to be. With the where you are free to add any SQL as you please:
const desiredEntity = await connection
.getRepository(User)
.createQueryBuilder("user")
.where("user.id = :id", { id: 1 })
.andWhere("user.date > :date OR (user.date = :date AND user.myId = :myId)",
{
date: specificCreatedAtDate,
myId: mysteryId,
})
.getOne();
Note that depending on your used database the actual SQL that you use here needs to be compatible. With that could also come a possible draw back of using this method. You will tie your project to a specific database. Make sure to read up about the aliases for tables you can set if you are using relations this would be handy.
Repository
You already saw that this is much less comfortable. This is because the find function or more specific the findOptions are using objects to build the where clause. This makes is harder to implement a proper interface to implement nested AND and OR clauses side by side. There for (I assume) they have chosen to split AND and OR clauses. This makes the interface much more declarative and means the you have to pull your OR clauses to the top:
const desiredEntity = await repository.find({
where: [{
id: id,
notAvailable: Not(IsNull()),
date: MoreThan(date)
},{
id: id,
notAvailable: Not(IsNull()),
date: date
myId: myId
}]
})
I cannot imagin looking a the size of the desired query that this code would be very performant.
Alternatively you could use the Raw find helper. This would require you to rewrite your clause per field, since you will only get access to the one alias at a time. You could guess the column names or aliases but this would be very poor practice and very unstable since you cannot directly control this easily.
if you want to nest andWhere statements if a condition is meet here is an example:
async getTasks(filterDto: GetTasksFilterDto, user: User): Promise<Task[]> {
const { status, search } = filterDto;
/* create a query using the query builder */
// task is what refer to the Task entity
const query = this.createQueryBuilder('task');
// only get the tasks that belong to the user
query.where('task.userId = :userId', { userId: user.id });
/* if status is defined then add a where clause to the query */
if (status) {
// :<variable-name> is a placeholder for the second object key value pair
query.andWhere('task.status = :status', { status });
}
/* if search is defined then add a where clause to the query */
if (search) {
query.andWhere(
/*
LIKE: find a similar match (doesn't have to be exact)
- https://www.w3schools.com/sql/sql_like.asp
Lower is a sql method
- https://www.w3schools.com/sql/func_sqlserver_lower.asp
* bug: search by pass where userId; fix: () whole addWhere statement
because andWhere stiches the where class together, add () to make andWhere with or and like into a single where statement
*/
'(LOWER(task.title) LIKE LOWER(:search) OR LOWER(task.description) LIKE LOWER(:search))',
// :search is like a param variable, and the search object is the key value pair. Both have to match
{ search: `%${search}%` },
);
}
/* execute the query
- getMany means that you are expecting an array of results
*/
let tasks;
try {
tasks = await query.getMany();
} catch (error) {
this.logger.error(
`Failed to get tasks for user "${
user.username
}", Filters: ${JSON.stringify(filterDto)}`,
error.stack,
);
throw new InternalServerErrorException();
}
return tasks;
}
I have a list of
{
date: specificCreatedAtDate,
userId: mysteryId
}
My solution is
.andWhere(
new Brackets((qb) => {
qb.where(
'userTable.date = :date0 AND userTable.type = :userId0',
{
date0: dates[0].date,
userId0: dates[0].type,
}
);
for (let i = 1; i < dates.length; i++) {
qb.orWhere(
`userTable.date = :date${i} AND userTable.userId = :userId${i}`,
{
[`date${i}`]: dates[i].date,
[`userId${i}`]: dates[i].userId,
}
);
}
})
)
That will produce something similar
const userEntity = await repository.find({
where: [{
userId: id0,
date: date0
},{
id: id1,
userId: date1
}
....
]
})
I am using s3 select query along with where clause to retrieve data from s3.
The query is working fine and returning the expected result when there's no where clause. Although when I am using where clause, the filtered data is correct, but the key in the object is the first row after the header and not the header.
Example : csv file
A B C
1 2 3
1 5 6
Query : select * from s3object s where s._1 = '1' limit 100
Expected Output : [{A : 1, B:2, C:3}, {A:1, B:5, C:6}]
Actual Output : [{1:1, 2:5, 3:6}]
This is the params object I am using to query :
let params = {
Bucket: S3_BUCKET,
Key: S3_PATH,
Expression: "select * from s3object s where s._1 = '1' limit 100"
ExpressionType: "SQL",
InputSerialization: {
CSV: {
FileHeaderInfo: "NONE",
RecordDelimiter: "\n",
FieldDelimiter: ","
}
},
OutputSerialization: {
CSV: {}
}
};
I get the same output even when I use FileHeaderInfo : "USE", and change the query to select * from s3object s where id = '22' and s.date > '2020-05-01' limit 100
AWS Doc : https://docs.aws.amazon.com/AmazonS3/latest/API/API_SelectObjectContent.html
So it seems, while fetching the query results from s3, it is impossible to get the headers as well. We can query with headerNames, or with columnNumber, but if we use the where clause, then we should use headerNames, and in that case, the header row doesn't come in the results.
So, I have now hardcoded the headers in my api call from where I am calling s3 select query, and appending those in the results.
Change the params to the following should work.
let params = {
Bucket: S3_BUCKET,
Key: S3_PATH,
ExpressionType: "SQL",
Expression: "select * from s3object s where s.A = '1' limit 100"
InputSerialization: {
CSV: {
FileHeaderInfo: "USE",
RecordDelimiter: "\n",
FieldDelimiter: ","
}
},
OutputSerialization: {
JSON: {}
}
};
I'm trying to get full-text search working with modeshape. I'm particularly interested in ranked results based on lucene index. Here is my repository configuration
"indexProviders": {
"lucene": {
"classname": "lucene",
"directory": "${user.home}/repository/indexes"
}
},
"indexes": {
"textFromFiles": {
"kind": "text",
"provider": "lucene",
"nodeType": "nt:resource",
"columns": "jcr:data(BINARY)"
}
},
I noticed a lucene index created at the specified location. I added 10-15 filesc with varied number of occurrence of search term into repository, and tried searching using some words. I am printing the score as shown below
QueryManager querymgr = session.getWorkspace().getQueryManager();
String query = "SELECT file.* FROM [nt:hierarchyNode] as file LEFT JOIN [nt:resource] as data ON ISCHILDNODE(data , file) WHERE "
+ "contains(data.*, '" + searchText + "')";
Query createQuery = querymgr.createQuery(query, Query.JCR_SQL2);
QueryResult result = createQuery.execute();
RowIterator rows = result.getRows();
while(rows.hasNext()){
Row nextRow = rows.nextRow();
LOGGER.info("score : {}", nextRow.getScore());
}
But, here score is always 1.0 for all results.
Also tried a simpler query without join...
SELECT data.* FROM [nt:resource] as data WHERE contains(data.*, 'searchterm')
but no luck
I am having some issues retrieving the correct date within a particular radius from MongoDB. I have a json example shown at the bottom. My goal is to search for all items close to a defined geolocation as well as filtering on a starttime. This also includes creating the correct index. There is currently not a lot guidance out and about. Thanks in advance for any help.
In summary, what I'm trying to do is:
retrieve all items from MongoDB that are within 3 miles of a provided geolocation
filter results for items with a starttime after x but before y.
Create an index that works for my query.
Regards,
Lance
If I just wanted to query on dates I could do the following:
DateTime maxdateTime = new DateTime(); //Joda Time
//Adjust my datetime, simply add 2 hours to the time
DateTime mindateTime = new DateTime(); Joda Time
//Adjust my datetime, subtract 1 day
Date maxDate = new Date(maxdateTime.getMillis());
Date minDate = new Date(mindateTime.getMillis());
DBCollection coll = db.getCollection("BikeRides");
coll.ensureIndex( { startTime:1, } )
BasicDBObject query = new BasicDBObject();
query.put("startTime", BasicDBObjectBuilder.start("$gte", minDate).add("$lte", maxDate).get());
DBCursor cursor = coll.find(query);
try {
while(cursor.hasNext()) {
System.out.println(cursor.next());
}
} finally {
cursor.close();
}
If I just wanted to query on GeoLocation I could do the following:
//A GeoLoc is passed into this method
int distance = 3;
int earthRadius = 3963.192;
DBCollection coll = db.getCollection("BikeRides");
coll.ensureIndex( { Location.GeoLoc : "2d" } )
db.runCommand( { geoNear: "BikeRides",
nearSphere: [ "Location.GeoLoc.latitude": geoLoc.longitude, "Location.GeoLoc.longitude": geoLoc.latitude ]
maxDistance: distance / earthRadius
} )
I attempted this with Jongo as well but without any success:
DateTime nowDateTime = new DateTime(DateTimeZone.UTC); // Joda time
DateTime maxStartTime = nowDateTime.plusMinutes(TIME_IN_MINUTES);
DateTime minStartTime = nowDateTime.minusDays(1); //This will cut out most old bike rides
Long now = nowDateTime.toInstant().getMillis();
Long max = maxStartTime.toInstant().getMillis();
Long min = minStartTime.toInstant().getMillis();
//Get the objects using Jongo
MongoCollection bikeRidesCollection = MongoDatabase.Get_DB_Collection(MONGO_COLLECTIONS.BIKERIDES, "Location.GeoLoc");
//Currently not placing a limit on this result. If we find this is an issue we can add later.
bikeRidesCollection.ensureIndex("{Location.GeoLoc: '2d', RideStartTime: 1}");
Iterable<BikeRide> all = bikeRidesCollection
.find("{Location.GeoLoc: {$near: [#, #], $maxDistance: #}, RideStartTime: {$lte: #, $gte: #}}", //, $maxDistance: #
location.getGeoLoc().longitude,
location.getGeoLoc().latitude,
RADIUS_IN_MILES/EarthRadiusInMILES,
max ,
min )
.as(BikeRide.class);
List<BikeRide> closeBikeRides = Lists.newArrayList(all);
My sample Json:
{
"BikeRides": [
{
"StartTime": "2013-03-08T00:01:00.000 UTC",
"bikeRideName": "Strawberry Ride",
"id": "513afc2d0364b81b8abfa86e",
"location": {
"city": "Portland",
"geoLoc": {
"longitude": "-122.71446990966797",
"latitude": "45.49216842651367"
},
"state": "OR",
"streetAddress": "4214 SE 36th"
},
"startTime": "2013-03-07T16:01:00-08:00"
}
]
}
Solved. To get Jongo to work I simply needed to use the correct maxDistance. Instead of EarthRadiusInMILES, I should have defined this; Double ONE_DEGREE_IN_MILES = 69.11;
Note: 1° of latitude = about 69.11 miles