Fusion table SQL query fails with numbers - sql

I'm making a tiny webapp to see what are the pub's specials around you and I have a very silly problem with my SQL query to fusion table.
Here is my table and her is my query:
ST_INTERSECTS(address, CIRCLE(LATLNG(-33.898672999999995, 151.2063809), 300))
AND type IN ('food','drinks')
AND days CONTAINS 'tuesday'
AND from <= 2000
AND to >= 2000
My problem is with the from and to, if I remove them my query is fine and if I simplify them (remove from and put just to > 0) my query is still wrong.
As you can see in my fusion table, from and to are both numbers so I really don't get what's wrong.
EDIT:
So I'm using https://developers.google.com/apis-explorer/#p/fusiontables/v2/fusiontables.query.sql to test my queries:
Good query (200 OK):
SELECT *
FROM 1BHnaan3YfSDq9_LzjthDXjj5dzJZANjLSb8JHPl5
WHERE
ST_INTERSECTS(address, CIRCLE(LATLNG(-33.898672999999995, 151.2063809), 300))
AND type IN ('food','drinks')
AND days CONTAINS 'tuesday'
Bad query:
SELECT *
FROM 1BHnaan3YfSDq9_LzjthDXjj5dzJZANjLSb8JHPl5
WHERE
ST_INTERSECTS(address, CIRCLE(LATLNG(-33.898672999999995, 151.2063809), 300))
AND type IN ('food','drinks')
AND days CONTAINS 'tuesday'
AND from <= 1619
AND to >= 1619
I get this error but I don't see what's wrong because <= is in the docs:
{
"error": {
"errors": [
{
"domain": "fusiontables",
"reason": "badQueryCouldNotParse",
"message": "Invalid query: Parse error near 'from' (line 1, position 218).",
"locationType": "parameter",
"location": "q"
}
],
"code": 400,
"message": "Invalid query: Parse error near 'from' (line 1, position 218)."
}
}

from & to are reserved words in fusion table, here is the list of reserved words:
AND
ASC
AS
BY
CASE
CIRCLE
CONTAINS
CONTAIN
CREATE
DELETE
DESCRIBE
DESC
DOES
DROP
ENDS
EQUAL
FROM
GROUP
IGNORING
IN
INSERT
INTO
LATLNG
LIKE
LIMIT
MATCHES
NEAR
NOT
OFFSET
ORDER
POLYGON
RECTANGLE
ROWID
SELECT
SET
SHOW
SKIP
ST_DISTANCE
ST_INTERSECTS
STARTS
TABLES
TABLE
TO
UPDATE
VALUES
VIEW
WHERE
WITH
ID
NUMBER
DOCID
STRING
I figured it out because they were blue in the syntax highlighting in my question.

If you try that query from the Fusion Tables UI, you also see no results, because no values match those criteria. The days are "monday" and "tuesday", and the times are 1130-1400 and 1800-2359.

Related

Postgres jsonb_array_elements() returns "cannot extract elements from a scalar", even when given a valid JSON array?

I have a table in postgres called day, which contains a jsonb column called plan_activities.
I have a day record with day.id = 18 and plan_activities contains the following JSON:
[
{
"activity": "Gym",
"cardio": false,
"strength": true,
"quantity": 20,
"units": "mins",
"timeOfDay": "Evening",
"summary": "Gym - 20 mins - Evening",
"timeOfDayOrder": 4
},
{
"activity": "Walk",
"cardio": true,
"strength": false,
"quantity": 15,
"units": "minutes",
"timeOfDay": "morning",
"summary": "Walk - 15 minutes - Lunchtime",
"timeOfDayOrder": 1
}
]
When I execute the following query:
select jsonb_array_elements(day.plan_activities) as activities
from day
where day.id = 18;
I get the following error:
Failed to run sql query: cannot extract elements from a scalar
The JSON contains a valid JSON array as far as I can tell. What am I doing wrong?
My eventual goal if I can extract this list is to create separate records elsewhere, each of which contains all the fields plus a reference back to the day record.
This error happens when you try to treat a JSON scalar, like a single string or number, as an array.
-- ERROR: cannot extract elements from a scalar
select jsonb_array_elements('23'::jsonb);
One of the rows of your query does not contain a JSON array.
Check with select plan_activities from day where id = 18. Although id is normally a unique primary key and it should be impossible to have more than one row returned.
Another way this could happen is if the JSON structure was accidentally added as a single JSON string.
-- 1, 2, 3
select jsonb_array_elements('[1, 2, 3]'::jsonb);
-- Note the extra quotes.
-- ERROR: cannot extract elements from a scalar
select jsonb_array_elements('"[1, 2, 3]"'::jsonb);

How to remove array wrapper from mariadb server

I want to remove the array wrapper surrounding a query result as I'm running a for loop to push the object into an array. This is my query
"SELECT * FROM jobs WHERE id = ? FOR JSON PATH, WITHOUT_ARRAY_WRAPPER"
but I'm getting this result in postman
{
"status": "Failed",
"message": "You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'JSON PATH, WITHOUT_ARRAY_WRAPPER' at line 1"
}
for json path is a feature of Microsoft SQL Server. There is a standard for JSON in SQL, but don't expect most SQL servers to follow it.
You can get a single JSON object for each row with json_object.
-- {"id": 2, "name": "Bar"}
select
json_object('id', id, 'name', name)
from jobs
where id = 2
Rather than query each job individually and then appending to an array, you can do this in single query using the in operator to query all desired rows at once, and then json_arrayagg to aggregate them into a single array.
-- [{"id": 1, "name": "Foo"},{"id": 3, "name": "Baz"}]
select
json_arrayagg( json_object('id', id, 'name', name) )
from jobs
where id in (1, 3)
This is much more efficient. In general, if you're querying SQL in loops there's a better way.
Demonstration.

Using Athena to get terminatingrule from rulegrouplist in AWS WAF logs

I followed these instructions to get my AWS WAF data into an Athena table.
I would like to query the data to find the latest requests with an action of BLOCK. This query works:
SELECT
from_unixtime(timestamp / 1000e0) AS date,
action,
httprequest.clientip AS ip,
httprequest.uri AS request,
httprequest.country as country,
terminatingruleid,
rulegrouplist
FROM waf_logs
WHERE action='BLOCK'
ORDER BY date DESC
LIMIT 100;
My issue is cleanly identifying the "terminatingrule" - the reason the request was blocked. As an example, a result has
terminatingrule = AWS-AWSManagedRulesCommonRuleSet
And
rulegrouplist = [
{
"nonterminatingmatchingrules": [],
"rulegroupid": "AWS#AWSManagedRulesAmazonIpReputationList",
"terminatingrule": "null",
"excludedrules": "null"
},
{
"nonterminatingmatchingrules": [],
"rulegroupid": "AWS#AWSManagedRulesKnownBadInputsRuleSet",
"terminatingrule": "null",
"excludedrules": "null"
},
{
"nonterminatingmatchingrules": [],
"rulegroupid": "AWS#AWSManagedRulesLinuxRuleSet",
"terminatingrule": "null",
"excludedrules": "null"
},
{
"nonterminatingmatchingrules": [],
"rulegroupid": "AWS#AWSManagedRulesCommonRuleSet",
"terminatingrule": {
"rulematchdetails": "null",
"action": "BLOCK",
"ruleid": "NoUserAgent_HEADER"
},
"excludedrules":"null"
}
]
The piece of data I would like separated into a column is rulegrouplist[terminatingrule].ruleid which has a value of NoUserAgent_HEADER
AWS provide useful information on querying nested Athena arrays, but I have been unable to get the result I want.
I have framed this as an AWS question but since Athena uses SQL queries, it's likely that anyone with good SQL skills could work this out.
It's not entirely clear to me exactly what you want, but I'm going to assume you are after the array element where terminatingrule is not "null" (I will also assume that if there are multiple you want the first).
The documentation you link to say that the type of the rulegrouplist column is array<string>. The reason why it is string and not a complex type is because there seems to be multiple different schemas for this column, one example being that the terminatingrule property is either the string "null", or a struct/object – something that can't be described using Athena's type system.
This is not a problem, however. When dealing with JSON there's a whole set of JSON functions that can be used. Here's one way to use json_extract combined with filter and element_at to remove array elements where the terminatingrule property is the string "null" and then pick the first of the remaining elements:
SELECT
element_at(
filter(
rulegrouplist,
rulegroup -> json_extract(rulegroup, '$.terminatingrule') <> CAST('null' AS JSON)
),
1
) AS first_non_null_terminatingrule
FROM waf_logs
WHERE action = 'BLOCK'
ORDER BY date DESC
You say you want the "latest", which to me is ambiguous and could mean both first non-null and last non-null element. The query above will return the first non-null element, and if you want the last you can change the second argument to element_at to -1 (Athena's array indexing starts from 1, and -1 is counting from the end).
To return the individual ruleid element of the json:
SELECT from_unixtime(timestamp / 1000e0) AS date, action, httprequest.clientip AS ip, httprequest.uri AS request, httprequest.country as country, terminatingruleid, json_extract(element_at(filter(rulegrouplist,rulegroup -> json_extract(rulegroup, '$.terminatingrule') <> CAST('null' AS JSON) ),1), '$.terminatingrule.ruleid') AS ruleid
FROM waf_logs
WHERE action='BLOCK'
ORDER BY date DESC
I had the same issue but the solution posted by Theo didn't work for me, even though the table was created according to the instructions linked to in the original post.
Here is what worked for me, which is basically the same as Theo's solution, but without the json conversion:
SELECT
from_unixtime(timestamp / 1000e0) AS date,
action,
httprequest.clientip AS ip,
httprequest.uri AS request,
httprequest.country as country,
terminatingruleid,
rulegrouplist,
element_at(filter(ruleGroupList, ruleGroup -> ruleGroup.terminatingRule IS NOT NULL),1).terminatingRule.ruleId AS ruleId
FROM waf_logs
WHERE action='BLOCK'
ORDER BY date DESC
LIMIT 100;

SQL exceeds the maximum allowed error

Hi I'm pulling data within Data Manager and which is coming via Salesforce so the data type and limits are set in Salesforce which I have no control over. I would like to know how I can limit the the amount of data coming in and inserted into the MS SQL. This is the error:
[42000]Microsoft SQL Server Native Client 10.0
The size (32000) given to the column 'Description' exceeds the maximum allowed for any data type (8000).
And this is my query which pulls the data:
SELECT "Id",
"IsDeleted",
"Name",
"ParentId",
"Type",
"Status",
"StartDate",
"EndDate",
"CurrencyIsoCode",
"ExpectedRevenue",
"BudgetedCost",
"ActualCost",
"Description",
"ExpectedResponse",
"NumberSent",
"IsActive",
"NumberOfLeads",
"NumberOfConvertedLeads",
"NumberOfContacts",
"NumberOfResponses",
"NumberOfOpportunities",
"NumberOfWonOpportunities",
"AmountAllOpportunities",
"AmountWonOpportunities",
"OwnerId",
"CreatedDate",
"CreatedById",
"LastModifiedDate",
"LastModifiedById",
"SystemModstamp",
"LastActivityDate",
"CampaignMemberRecordTypeId",
"Date_of_Event__c",
"Venue__c"
FROM "SF Schema"."Campaign"
Is there anyway to set limit on the "Description" the max is 8000.
Thanks!
You can get first 8000 characters using LEFT()
SELECT LEFT(Description,8000) AS Description FROM YourTable
When is this eror happening?
Your error is not comming from your data, but from the DDL of the "SF Schema"."Campaign" table.
How is it possible for that table to have a column of size 32000? Where was it defined?
This little sample code shows how to reproduce your error message.
DECLARE #table TABLE
(
dummy nvarchar(32000)
)
This produces this output:
Msg 131, Level 15, State 2, Line 3
The size (32000) given to the column 'dummy' exceeds the maximum allowed for any data type (8000).
Maybe LEFT, as stated in another answer helps you, but I guess that you are going to get the same error, because of the problem being in the definition of the source table.
LEFT on the MSDN: http://msdn.microsoft.com/es-es/library/ms177601.aspx

AVG is not taking null values into consideration

I have loaded the following test data:
name, age,gender
"John", 33,m
"Sam", 33,m
"Julie",33,f
"Jimbo",, m
with schema: name:STRING,age:INTEGER,gender:STRING and I have confirmed that the Jimbo row shows a null for column "age" in the BigQuery Browser Tool > mydataset > Details > Preview section.
When I run this query :
SELECT AVG(age) FROM [peterprivatedata.testpeople]
I get 24.75 which is incorrect. I expected 33 because the documentation for AVG says "Rows with a NULL value are not included in the calculation."
Am I doing something wrong or is this a known bug? (I don't know if there's a public issues list to check). What's the simplest workaround to this?
This is a known bug where we coerce null numeric values to to 0 on import. We're currently working on a fix. These values do however, show up as not not defined (which for various reasons is different from null), so you can check for IS_EXPLICITLY_DEFINED. For example:
SELECT sum(if(is_explicitly_defined(numeric_field), numeric_field, 0)) /
sum(if(is_explicitly_defined(numeric_field), 1, 0))
AS my_avg FROM your_table
Alternately, you could use another column to represent is_null. Then the query would look like:
SELECT sum(if(numeric_field_is_null, 0, numeric_field)) /
sum(if(numeric_field_is_null, 0, 1))
AS my_avg FROM your_table