I have a Log Group with this kind of messages.
{
"m": [
{
"id": "5b6973c7c86e8689368b4569",
"ts": 1634112000.062
},
{
"id": "6116d21e02e38f5045079c42",
"ts": 1634120807.402
},
{
"id": "60c368ff1085fc0d546fad52",
"ts": 1634120807.512
},
{
"id": "6053536817a46610797ed472",
"ts": 1634120809.249
}
]
}
I want to run a query over the field m.*.ts (It's an array). Something like this...
fields #message
| filter (m.*.ts > 1634112000.062 and m.*.ts < 1634120807.000 )
It's posible?
fields #message
| parse #message "[*] *" as id, ts
| filter (ts > 1634112000.062 and ts < 1634120807.000)
Hi I don't know what format you want, so try this and you can adapt it, many more samples here on AWS
Option 1: helps you break it down in steps to debug
fields #message
|"[*] *" as id, ts
| filter ts > 1634112000.062
| filter ts < 1634120807.000
Option 2:
fields #message
| parse #message '[] * {"*"}' as id, ts
| filter (ts > 1634112000.062 and ts < 1634120807.000)
Related
I have a products table which contains a JSON column product_logs. Inside of this, it contains something similar to:
{
"c8eebc99-d936-3245-bc8d-17694f4ecb58": {
"created_at": "2022-05-08T15:33:33.591166Z",
"event": "product-created",
"user": null
},
"ce7b171b-b479-332f-bf9e-54b948581179": {
"created_at": "2022-05-08T15:33:33.591174Z",
"event": "near-sell-by",
"user": null
}
}
I only want to return rows of products that have a near-sell-by event in the product_logs so I try to do this:
SELECT
products.*
FROM products,
JSON_TABLE(product_logs, '$[*]', COLUMNS (
created_at DATETIME PATH '$.created_at',
event VARCHAR(MAX) PATH '$.event'
) logs
WHERE
logs.event = 'near-sell-by'
However, I seem to be getting the following error:
#1064 - You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '(product_logs, '$[*]', COLUMNS (
created_at DATETIME PATH '$.cr...' at line 4
Any help to where I'm going wrong would be greatly appreciated
You seem to have copied, from another database, there is no varchar8max) in mysql, to syntax is a bit complicated, and you need to undestand json pretty well.
a gui like workbench, at least can help you identify the error, but it will not help you
CREATE TABLE products (product_logs varchar(1209))
INSERT INTO products VALUES ('{
"c8eebc99-d936-3245-bc8d-17694f4ecb58": {
"created_at": "2022-05-08T15:33:33.591166Z",
"event": "product-created",
"user": null
},
"ce7b171b-b479-332f-bf9e-54b948581179": {
"created_at": "2022-05-08T15:33:33.591174Z",
"event": "near-sell-by",
"user": null
}
}
')
SELECT
products.*,logs.created_at,logs.event
FROM products,
JSON_TABLE(products.product_logs, '$.*'
COLUMNS (
created_at DATETIME PATH '$.created_at',
event Text PATH '$.event'
)) logs
WHERE
logs.event = 'near-sell-by'
product_logs | created_at | event
:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------ | :-----------
{<br> "c8eebc99-d936-3245-bc8d-17694f4ecb58": {<br> "created_at": "2022-05-08T15:33:33.591166Z",<br> "event": "product-created",<br> "user": null<br> },<br> "ce7b171b-b479-332f-bf9e-54b948581179": {<br> "created_at": "2022-05-08T15:33:33.591174Z",<br> "event": "near-sell-by",<br> "user": null<br> }<br>}<br> | 2022-05-08 15:33:34 | near-sell-by
db<>fiddle here
I have a table with a json field with the following json:
[
{
productId: '1',
other : [
otherId: '2'
]
},
{
productId: '3',
other : [
otherId: '4'
]
}
]
I am trying to select the productId and otherId for every array element like this:
select JSON_EXTRACT(items, $.items[].productId) from order;
But this is completely wrong since it takes only the first element in the array
Do I need to write a loop or something?
First of all, the data you show is not valid JSON. It has multiple mistakes that make it invalid.
Here's a demo using valid JSON:
mysql> create table orders ( items json );
mysql> insert into orders set items = '[ { "productId": "1", "other": { "otherId": "2" } }, { "productId": "3", "other" : { "otherId": "4" } } ]'
mysql> SELECT JSON_EXTRACT(items, '$[*].productId') AS productIds FROM orders;
+------------+
| productIds |
+------------+
| ["1", "3"] |
+------------+
If you want each productId on a row by itself as a scalar value instead of a JSON array, you'd have to use JSON_TABLE() in MySQL 8.0:
mysql> SELECT j.* FROM orders CROSS JOIN JSON_TABLE(items, '$[*]' COLUMNS(productId INT PATH '$.productId')) AS j;
+-----------+
| productId |
+-----------+
| 1 |
| 3 |
+-----------+
This is tested in MySQL 8.0.23.
You also tagged your question MariaDB. I don't use MariaDB, and MariaDB has its own incompatible implementation of JSON support, so I can't predict how it will work.
I receive JSON from API in the following format:
[
{
"scId": "000DD2",
"sensorId": 2,
"metrics": [
{
"s": 5414,
"dateTime": "2018-02-02T13:03:30+01:00"
},
{
"s": 5526,
"dateTime": "2018-02-02T13:04:56+01:00"
},
{
"s": 5631,
"dateTime": "2018-02-02T13:06:22+01:00"
}
}, .... ]
Currently trying to display these metrics on the linear chart with dateTime for the X-axis and "s" for Y.
I use the following search query:
index="main" source="rest://test3" | spath input=metrics{}.s| mvexpand metrics{}.s
| mvexpand metrics{}.dateTime | rename metrics{}.s as s
| rename metrics{}.dateTime as dateTime| table s,dateTime
And I receive the data in the following format which is not applicable for linear chart. The point is - how to correctly parse the JSON to apply date-time from dateTime field in JSON to _time in Splunk.
Query results
#Max Zhylochkin,
Can you please try following search?
index="main" source="rest://test3"
| spath input=metrics{}.s
| mvexpand metrics{}.s
| mvexpand metrics{}.dateTime
| rename metrics{}.s as s
| rename metrics{}.dateTime as dateTime
| table s,dateTime
| eval _time = strptime(dateTime,"%Y-%m-%dT%H:%M:%S.%3N")
Thanks
My JSON data:
{
"date": 1484219926,
"uid": "1234567",
"interest": [
"2000001",
"2000002",
"....",
"2000xxx"
],
"other": "xxxx"
}
The search result as the following SQL:
select count(*)
from xxxxx
where date > time1 and
date < time2 and
interest="20000xxx"
The filed interest may have 500 itmes. I want to quickly get the search result, what shou I do? The total data may be 2 billion.
UPDATE:
I've turned my xml into a query table in coldfusion, so this may help to solve this.
So my data is:
[id] | [code] | [desc] | [supplier] | [name] | [price]
------------------------------------------------------
1 | ABCDEF | "Tst0" | "XYZ" | "Test" | 123.00
2 | ABCDXY | "Tst1" | "XYZ" | "Test" | 130.00
3 | DCBAZY | "Tst2" | "XYZ" | "Tst2" | 150.00
Now what I need is what the linq to xml query outputs below. Output should be something like (i'll write it in JSON so it's easier for me to type) this:
[{
"code": "ABCD",
"name": "Test",
"products":
{
"id": 1,
"code": "ABCDEF",
"desc": "Tst0",
"price": 123.00
},
{
"id": 2,
"code": "ABCDXY",
"desc": "Tst1",
"price": 130.00
}
},
{
"code": "DCBA",
"name": "Tst2",
"products":
{
"id": 3,
"code": "DCBAZY",
"desc": "Tst2",
"price": 150.00
}
}]
As you can see, Group by the first 4 characters of 'CODE' and 'Supplier' code.
Thanks
How would i convert the following LINQ to XML query to SQL?
from q in query
group q by new { Code = q.code.Substring(0, 4), Supplier = q.supplier } into g
select new
{
code = g.Key.Code,
fullcode = g.FirstOrDefault().code,
supplier = g.Key.Supplier,
name = g.FirstOrDefault().name,
products = g.Select(x => new Product { id = x.id, c = x.code, desc = string.IsNullOrEmpty(x.desc) ? "Description" : x.desc, price = x.price })
}
Best i could come up with:
SELECT c, supplier, n
FROM products
GROUP BY C, supplier, n
Not sure how to get the subquery in there or get the substring of code.
ps: this is for coldfusion, so I guess their version of sql might be different to ms sql..
The easiest way is to attache a profiler to you database and see what query is generate by the linq-to-SQL engine.