I'm trying to use IF THEN style AQL, but the only relevant operator I could find in the AQL documentation was the ternary operator. I tried to add IF THEN syntax to my already working AQL but it gives syntax errors no matter what I try.
LET doc = DOCUMENT('xp/a-b')
LET now = DATE_NOW()
doc == null || now - doc.last >= 45e3 ?
LET mult = (doc == null || now - doc.last >= 6e5 ? 1 : doc.multiplier)
LET gained = FLOOR((RAND() * 3 + 3) * mult)
UPSERT {_key: 'a-b'}
INSERT {
amount: gained,
total: gained,
multiplier: 1.1,
last: now
}
UPDATE {
amount: doc.amount + gained,
total: doc.total + gained,
multiplier: (mult < 4 ? FLOOR((mult + 0.1) * 10) / 10 : 4),
last: now
}
IN xp
RETURN NEW
:
RETURN null
Gives the following error message:
stacktrace: ArangoError: AQL: syntax error, unexpected identifier near 'doc == null || now - doc.last >=...' at position 1:51 (while parsing)
The ternary operator can not be used like an if/else construct in the way to tried. It is for conditional (sub-)expressions like you use to calculate mult. It can not stand by itself, there is nothing it can be returned or assigned to if you write it like an if-expression.
Moreover, it would require braces, but the actual problem is that the body contains operations like LET, UPSERT and RETURN. These are language constructs which can not be used inside of expressions.
If I understand correctly, you want to:
insert a new document if no document with key a-b exists yet in collection xb
if it does exist, then update it, but only if the last update was 45 seconds or longer ago
Does the following query work for you?
FOR id IN [ 'xp/a-b' ]
LET doc = DOCUMENT(id)
LET key = PARSE_IDENTIFIER(id).key
LET now = DATE_NOW()
FILTER doc == null || now - doc.last >= 45e3
LET mult = (doc == null || now - doc.last >= 6e5 ? 1 : doc.multiplier)
LET gained = FLOOR((RAND() * 3 + 3) * mult)
UPSERT { _key: key }
INSERT {
_key: key,
amount: gained,
total: gained,
multiplier: 1.1,
last: now
}
UPDATE {
amount: doc.amount + gained,
total: doc.total + gained,
multiplier: (mult < 4 ? FLOOR((mult + 0.1) * 10) / 10 : 4),
last: now
}
IN xp
RETURN NEW
I added _key to INSERT, otherwise the document will get an auto-generated key, which does not seem intended. Using a FOR loop and a FILTER acts like an IF construct (without ELSE). Because this is a data modification query, it is not necessary to explicitly RETURN anything and in your original query you RETURN null for the ELSE case anyway. While yours would result in [ null ], mine produces [ ] (truly empty result) if you try execute the query in quick succession and nothing gets updated or inserted.
Note that it is necessary to use PARSE_IDENTIFIER() to get the key from the document ID string and do INSERT { _key: key }. With INSERT { _key: doc._key } you would run into an invalid document key error in the insert case, because if there is no document xp/a-b, DOCUMENT() returns null and doc._key is therefore also null, leading to _key: null - which is invalid.
Related
I'm trying to conditionally add a part to my SQL query using Exposed's DAO API. My goal is to have:
SELECT * FROM table
WHERE column1 = 1
AND column2 = $value
AND column3 = 3
where the existence of the AND column2 = $value part depends on a filter.
I've tried with:
TableDAO.find {
Table.column1 eq 1 and (
when (filter.value) {
null -> null // Type mismatch. Required: Expression<Boolean>. Found: Op<Boolean>?
else -> Table.column2 eq filter.value
}) and (
Table.column3 = 3
)
}.map { it.toModel() }
but I can't find a way to return an empty expression or somehow exclude that part from the query. The only solution I can make work is something like
null -> Table.column2 neq -1
but I feel like there should be a better way.
You'll have to assign your expressions into a local variable:
var expr = Table.column1 eq 1
if(filter.value) {
expr = expr and (Table.column2 eq filter.value)
}
expr = expr and (
Table.column3 = 3
)
I don't have my IDE in front of me, but this is the general idea. You can try to figure out something clever, but it would make your code unnecessarily complex.
I have about 20 columns in one row and not all columns are required to be filled in when row created also i dont want to cardcode name of every column in SQL query and on http.post request on frontend. All values are from form. My code:
var colNames, values []string
for k, v := range formData {
colNames = append(colNames, k)
values = append(values, v)
}
Now i have 2 arrays: one with column names and second with values to be inserted. I want to do something like this:
db.Query("insert into views (?,?,?,?,?,?) values (?,?,?,?,?,?)", colNames..., values...)
or like this:
db.Query("insert into views " + colNames + " values" + values)
Any suggestions?
Thanks!
I assume your code examples are just pseudo code but I'll state the obvious just in case.
db.Query("insert into views (?,?,?,?,?,?) values (?,?,?,?,?,?)", colNames..., values...)
This is invalid Go since you can only "unpack" the last argument to a function, and also invalid MySQL since you cannot use placeholders (?) for column names.
db.Query("insert into views " + colNames + " values" + values)
This is also invalid Go since you cannot concatenate strings with slices.
You could fromat the slices into strings that look like this:
colNamesString := "(col1, col2, col3)"
valuesString := "(val1, val2, val3)"
and now your second code example becomes valid Go and would compile but don't do this. If you do this your app becomes vulnerable to SQL injection and that's something you definitely don't want.
Instead do something like this:
// this can be a package level global and you'll need
// one for each table. Keep in mind that Go maps that
// are only read from are safe for concurrent use.
var validColNames = map[string]bool{
"col1": true,
"col2": true,
"col3": true,
// ...
}
// ...
var colNames, values []string
var phs string // placeholders for values
for k, v := range formData {
// check that column is valid
if !validColNames[k] {
return ErrBadColName
}
colNames = append(colNames, k)
values = append(values, v)
phs += "?,"
}
if len(phs) > 0 {
phs = phs[:len(phs)-1] // drop the last comma
}
phs = "(" + phs + ")"
colNamesString := "(" + strings.Join(colNames, ",") + ")"
query := "insert into views " + colNamesString + phs
db.Query(query, values...)
I'm writing a UDF to process Google Analytics data, and getting the "UDF out of memory" error message when I try to process multiple rows. I downloaded the raw data and found the largest record and tried running my UDF query on that, with success. Some of the rows have up to 500 nested hits, and the size of the hit record (by far the largest component of each row of the raw GA data) does seem to have an effect on how many rows I can process before getting the error.
For example, the query
select
user.ga_user_id,
ga_session_id,
...
from
temp_ga_processing(
select
fullVisitorId,
visitNumber,
...
from [79689075.ga_sessions_20160201] limit 100)
returns the error, but
from [79689075.ga_sessions_20160201] where totals.hits = 500 limit 1)
does not.
I was under the impression that any memory limitations were per-row? I've tried several techniques, such as setting row = null; before emit(return_dict); (where return_dict is the processed data) but to no avail.
The UDF itself doesn't do anything fancy; I'd paste it here but it's ~45 kB in length. It essentially does a bunch of things along the lines of:
function temp_ga_processing(row, emit) {
topic_id = -1;
hit_numbers = [];
first_page_load_hits = [];
return_dict = {};
return_dict["user"] = {};
return_dict["user"]["ga_user_id"] = row.fullVisitorId;
return_dict["ga_session_id"] = row.fullVisitorId.concat("-".concat(row.visitNumber));
for(i=0;i<row.hits.length;i++) {
hit_dict = {};
hit_dict["page"] = {};
hit_dict["time"] = row.hits[i].time;
hit_dict["type"] = row.hits[i].type;
hit_dict["page"]["engaged_10s"] = false;
hit_dict["page"]["engaged_30s"] = false;
hit_dict["page"]["engaged_60s"] = false;
add_hit = true;
for(j=0;j<row.hits[i].customMetrics.length;j++) {
if(row.hits[i].customDimensions[j] != null) {
if(row.hits[i].customMetrics[j]["index"] == 3) {
metrics = {"video_play_time": row.hits[i].customMetrics[j]["value"]};
hit_dict["metrics"] = metrics;
metrics = null;
row.hits[i].customDimensions[j] = null;
}
}
}
hit_dict["topic"] = {};
hit_dict["doctor"] = {};
hit_dict["doctor_location"] = {};
hit_dict["content"] = {};
if(row.hits[i].customDimensions != null) {
for(j=0;j<row.hits[i].customDimensions.length;j++) {
if(row.hits[i].customDimensions[j] != null) {
if(row.hits[i].customDimensions[j]["index"] == 1) {
hit_dict["topic"] = {"name": row.hits[i].customDimensions[j]["value"]};
row.hits[i].customDimensions[j] = null;
continue;
}
if(row.hits[i].customDimensions[j]["index"] == 3) {
if(row.hits[i].customDimensions[j]["value"].search("doctor") > -1) {
return_dict["logged_in_as_doctor"] = true;
}
}
// and so on...
}
}
}
if(row.hits[i]["eventInfo"]["eventCategory"] == "page load time" && row.hits[i]["eventInfo"]["eventLabel"].search("OUTLIER") == -1) {
elre = /(?:onLoad|pl|page):(\d+)/.exec(row.hits[i]["eventInfo"]["eventLabel"]);
if(elre != null) {
if(parseInt(elre[0].split(":")[1]) <= 60000) {
first_page_load_hits.push(parseFloat(row.hits[i].hitNumber));
if(hit_dict["page"]["page_load"] == null) {
hit_dict["page"]["page_load"] = {};
}
hit_dict["page"]["page_load"]["sample"] = 1;
page_load_time_re = /(?:onLoad|pl|page):(\d+)/.exec(row.hits[i]["eventInfo"]["eventLabel"]);
if(page_load_time_re != null) {
hit_dict["page"]["page_load"]["page_load_time"] = parseFloat(page_load_time_re[0].split(':')[1])/1000;
}
}
// and so on...
}
}
row = null;
emit return_dict;
}
The job ID is realself-main:bquijob_4c30bd3d_152fbfcd7fd
Update Aug 2016 : We have pushed out an update that will allow the JavaScript worker to use twice as much RAM. We will continue to monitor jobs that have failed with JS OOM to see if more increases are necessary; in the meantime, please let us know if you have further jobs failing with OOM. Thanks!
Update : this issue was related to limits we had on the size of the UDF code. It looks like V8's optimize+recompile pass of the UDF code generates a data segment that was bigger than our limits, but this was only happening when when the UDF runs over a "sufficient" number of rows. I'm meeting with the V8 team this week to dig into the details further.
#Grayson - I was able to run your job over the entire 20160201 table successfully; the query takes 1-2 minutes to execute. Could you please verify that this works on your side?
We've gotten a few reports of similar issues that seem related to # rows processed. I'm sorry for the trouble; I'll be doing some profiling on our JavaScript runtime to try to find if and where memory is being leaked. Stay tuned for the analysis.
In the meantime, if you're able to isolate any specific rows that cause the error, that would also be very helpful.
A UDF will fail on anything but very small datasets if it has a lot of if/then levels, such as:
if () {
.... if() {
.........if () {
etc
We had to track down and remove the deepest if/then statement.
But, that is not enough. In addition, when you pass the data into the UDF run a "GROUP EACH BY" on all the variables. This will force BQ to send the output to multiple "workers". Otherwise it will also fail.
I've wasted 3 days of my life on this annoying bug. Argh.
I love the concept of parsing my logs in BigQuery, but I've got the same problem, I get
Error: Resources exceeded during query execution.
The Job Id is bigquery-looker:bquijob_260be029_153dd96cfdb, if that at all helps.
I wrote a very basic parser does a simple match and returns rows. Works just fine on a 10K row data set, but I get out of resources when trying to run against a 3M row logfile.
Any suggestions for a work around?
Here is the javascript code.
function parseLogRow(row, emit) {
r = (row.logrow ? row.logrow : "") + (typeof row.l2 !== "undefined" ? " " + row.l2 : "") + (row.l3 ? " " + row.l3 : "")
ts = null
category = null
user = null
message = null
db = null
found = false
if (r) {
m = r.match(/^(\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d\.\d\d\d (\+|\-)\d\d\d\d) \[([^|]*)\|([^|]*)\|([^\]]*)\] :: (.*)/ )
if( m){
ts = new Date(m[1])/1000
category = m[3] || null
user = m[4] || null
db = m[5] || null
message = m[6] || null
found = true
}
else {
message = r
found = false
}
}
emit({
ts: ts,
category: category,
user: user,
db: db,
message: message,
found: found
});
}
bigquery.defineFunction(
'parseLogRow', // Name of the function exported to SQL
['logrow',"l2","l3"], // Names of input columns
[
{'name': 'ts', 'type': 'timestamp'}, // Output schema
{'name': 'category', 'type': 'string'},
{'name': 'user', 'type': 'string'},
{'name': 'db', 'type': 'string'},
{'name': 'message', 'type': 'string'},
{'name': 'found', 'type': 'boolean'},
],
parseLogRow // Reference to JavaScript UDF
);
Scratching head here. I've got a pulldown and if I query it in SQL Server Manager Query Window I get 5 different values (these are sample points for a water system).
However, when the pulldown loads, there are 5 options of the first value. Can someone see something I can't?
I narrowed it down to the code below because I held my cursor over "results" which was the final step in my Controller's code, and it showed 5 items all of the same value:
else if ((sampletype == "P") || (sampletype == "T") || (sampletype == "C") || (sampletype == "A"))
{
var SamplePoints = (from c in _db.tblPWS_WSF_SPID_ISN_Lookup
where c.PWS == id && c.WSFStateCode.Substring(0, 1) == "S"
select c).ToList();
if (SamplePoints.Any())
{
var listItemsBig = SamplePoints.Select(p => new SelectListItem
{
Selected = false,
Text = p.WSFStateCode.ToString() + ":::" + p.SamplePointID.ToString(),
Value = p.WSFStateCode.ToString()
}).ToList();
results = new JsonResult { Data = listItemsBig };
}
}
return results ;
}
I have had a similar problem in nHibernate, it was caused by how I defined my primary keys/foreign keys in the ORM, leading to a bad join and duplicate values.
in websql we can request a certain row like this:
tx.executeSql('SELECT * FROM tblSettings where id = ?', [id], function(tx, rs){
// do stuff with the resultset.
},
function errorHandler(tx, e){
// do something upon error.
console.warn('SQL Error: ', e);
});
however, I know regular SQL and figured i should be able to request
var arr = [1, 2, 3];
tx.executeSql('SELECT * FROM tblSettings where id in (?)', [arr], function(tx, rs){
// do stuff with the resultset.
},
function errorHandler(tx, e){
// do something upon error.
console.warn('SQL Error: ', e);
});
but that gives us no results, the result is always empty. if i would remove the [arr] into arr, then the sql would get a variable amount of parameters, so i figured it should be [arr]. otherwise it would require us to add a dynamic amount of question marks (as many as there are id's in the array).
so can anyone see what i'm doing wrong?
aparently, there is no other solution, than to manually add a question mark for every item in your array.
this is actually in the specs on w3.org
var q = "";
for each (var i in labels)
q += (q == "" ? "" : ", ") + "?";
// later to be used as such:
t.executeSql('SELECT id FROM docs WHERE label IN (' + q + ')', labels, function (t, d) {
// do stuff with result...
});
more info here: http://www.w3.org/TR/webdatabase/#introduction (at the end of the introduction)
however, at the moment i created a helper function that creates such a string for me
might be better than the above, might not, i haven't done any performance testing.
this is what i use now
var createParamString = function(arr){
return _(arr).map(function(){ return "?"; }).join(',');
}
// when called like this:
createparamString([1,2,3,4,5]); // >> returns ?,?,?,?,?
this however makes use of the underscore.js library we have in our project.
Good answer. It was interesting to read an explanation in the official documentation.
I see this question was answered in 2012. I tried it in Google 37 exactly as it is recommened and this is what I got.
Data on input: (I outlined them with the black pencil)
Chrome complains:
So it accepts as many question signs as many input parameters are given. (Let us pay attention that although array is passed it's treated as one parameter)
Eventually I came up to this solution:
var activeItemIds = [1,2,3];
var q = "";
for (var i=0; i< activeItemIds.length; i++) {
q += '"' + activeItemIds[i] + '", ';
}
q= q.substring(0, q.length - 2);
var query = 'SELECT "id" FROM "products" WHERE "id" IN (' + q + ')';
_db.transaction(function (tx) {
tx.executeSql(query, [], function (tx, results1) {
console.log(results1);
debugger;
}, function (a, b) {
console.warn(a);
console.warn(b);
})
})