It seems like sql group by is more of aggregate functions (COUNT, MAX, MIN, SUM, AVG).
select count(Id), Country
from Customer
where Country <> 'CountryX'
group by Country
But do we have a linq-like query where we want to return all results grouped by a certain column, E.g. in linq I would do
id | title | category | email
------------------------------------------
1 | tname-1 | cat1 | test#example.com
2 | tname-2 | cat1 | test1#example.com
3 | tname-3 | cat2 | TEst#example.com
linq group-by:
var groupedBy = list.GroupBy(item => item.Email);
or even throw in some comparison
var groupedBy = list.GroupBy(item => item.Email, StringComparer.OrdinalIgnoreCase);
and a result will be something like:
key | items
----------------------------------------------------------------------------------------------
test#example.com | [{Id :1, Title : "tname-1", category: "cat1", email: "test#example.com" },{Id :3, Title : "tname-3", category: "cat2", email: "TEst#example.com" } ]
test1#example.com| [{Id :2, Title : "tname-2", category: "cat1", email: "test1#example.com" }]
but with sql I would definitely want to return only the subset of the columns, say id, title and email.
Related
There are two tables, one is called "user_preference" that contains all users:
id | firstname | lastname | email |
And "match" which combines users with meetups they joined:
id | matcher | partner | meetup |
Both matcher and partner are foreign keys that represent user_preference.id, meaning that same user can be both matcher and a partner in the same meetup.
What I need to know is what percentage of total unique users joined what number of meetings.
For example:
17% of users joined 5 meetups
20% of users joined 3 meetups
40% of users joined 1 meetup
23% of users joined 0 meetups
The number of meetups should not be hardcoded but dynamic.
But I want to avoid duplication of users for a single meetup and count them only once. For example this:
id | matcher | partner | meetup |
1 | user1 | user2 | meetup1 |
2 | user1 | user3 | meetup1 |
3 | user5 | user1 | meetup1 |
4 | user6 | user1 | meetup2 |
Should count that user1 visited only 2 meetups.
What I managed to do so far is to display the count of meetups each user visited but that is not what I need:
SELECT distinct up.email users, COUNT(m.user) meetups
FROM user_preference up
LEFT JOIN
(
SELECT matcher AS user FROM match
UNION ALL
SELECT partner AS user FROM match
) m ON m.user = up.id
GROUP BY up.email
ORDER BY meetups desc;
In the end I did this by making simple queries and looping through them in the code, its far from elegant solution but it should work.
If someone posts SQL solution I will accept and upvote it...
export const getDevStats = async () => {
const users = await getRepository(UserPreference).query(
`SELECT * FROM user_preference;`
);
const meetups = await getRepository(Meetup).query(
`SELECT * FROM meetup;`
);
const matches = await getRepository(Match).query(
`SELECT * FROM match;`
);
let userMatches: any = {};
users.forEach((user: any) => {
userMatches[user.id] = []
matches.forEach((match: any) => {
if(user.id == match.matcher || user.id == match.partner) {
if(userMatches[user.id].indexOf(match.meetup) === -1) {
userMatches[user.id].push(match.meetup);
}
}
});
});
let matchStats: any = {};
for (var userId of Object.keys(userMatches)) {
if (typeof matchStats[userMatches[userId].length] === 'undefined') {
matchStats[userMatches[userId].length] = 0;
}
matchStats[userMatches[userId].length]++;
}
return {
users: users,
meetups: meetups,
matches: matches,
userMatches: userMatches,
matchStats: matchStats
};
};
This is as example taken from another thread, but essentially I would like to achieve this:
Sample data
ID Name Value
1 TV1 {"URL": "www.url.com", "Icon": "some_icon"}
2 TV2 {"URL": "www.url.com", "Icon": "some_icon", "Facebook": "Facebook_URL"}
3 TV3 {"URL": "www.url.com", "Icon": "some_icon", "Twitter": "Twitter_URL"}
..........
Expected output
ID Name URL Icon Facebook Twitter
1 TV1 www.url.com some_icon NULL NULL
2 TV2 www.url.com some_icon Facebook_URL NULL
3 TV3 www.url.com some_icon NULL Twitter_URL
I'm totally new to Snowflake so I'm shaking my head on how to do this easily (and hopefully automatically, in the case where some rows might have more elements in the json than other rows, which would be tedious to assign manually). Some lines might have sub-categories too.
I found the parse_json function for Snowflake, but it's only giving me the same json column in a new column, still in json format.
TIA!
You can create a view over your table with the following SELECT:
SELECT ID,
Name,
Value:URL::varchar as URL,
Value:Icon::varchar as Icon,
Value:Facebook::varchar as Facebook,
Value:Twitter::varchar as Twitter
FROM tablename;
Additional attributes will be ignored unless you add them to the view. There is no way to "automatically" include them into the view, but you could create a stored procedure that dynamically generates the view based on all the attributes that are in the full variant content of a table.
You can create a SP to automatically build the CREATE VIEW for you based on the JSON data in the VARIANT.
I have some simple example below:
-- prepare the table and data
create or replace table test (
col1 int, col2 string,
data1 variant, data2 variant
);
insert into test select 1,2, parse_json(
'{"URL": "test", "Icon": "test1", "Facebook": "http://www.facebook.com"}'
), parse_json(
'{"k1": "test", "k2": "test1", "k3": "http://www.facebook.com"}'
);
insert into test select 3,4,parse_json(
'{"URL": "test", "Icon": "test1", "Twitter": "http://www.twitter.com"}'
), parse_json(
'{"k4": "v4", "k3": "http://www.ericlin.me"}'
);
-- create the SP, we need to know which table and
-- column has the variant data
create or replace procedure create_view(
table_name varchar
)
returns string
language javascript
as
$$
var final_columns = [];
// first, find out the columns
var query = `SHOW COLUMNS IN TABLE ${TABLE_NAME}`;
var stmt = snowflake.createStatement({sqlText: query});
var result = stmt.execute();
var variant_columns = [];
while (result.next()) {
var col_name = result.getColumnValue(3);
var data_type = JSON.parse(result.getColumnValue(4));
// just use it if it is not a VARIANT type
// if it is variant type, we need to remember this column
// and then run query against it later
if (data_type["type"] != "VARIANT") {
final_columns.push(col_name);
} else {
variant_columns.push(col_name);
}
}
var columns = {};
query = `SELECT ` + variant_columns.join(', ') + ` FROM ${TABLE_NAME}`;
stmt = snowflake.createStatement({sqlText: query});
result = stmt.execute();
while (result.next()) {
for(i=1; i<=variant_columns.length; i++) {
var sub_result = result.getColumnValue(i);
if(!sub_result) {
continue;
}
var keys = Object.keys(sub_result);
for(j=0; j<keys.length; j++) {
columns[variant_columns[i-1] + ":" + keys[j]] = keys[j];
}
}
}
for(path in columns) {
final_columns.push(path + "::STRING AS " + columns[path]);
}
var create_view_sql = "CREATE OR REPLACE VIEW " +
TABLE_NAME + "_VIEW\n" +
"AS SELECT " + "\n" +
" " + final_columns.join(",\n ") + "\n" +
"FROM " + TABLE_NAME + ";";
snowflake.execute({sqlText: create_view_sql});
return create_view_sql + "\n\nVIEW created successfully.";
$$;
Execute the SP will return below string:
call create_view('TEST');
+---------------------------------------+
| CREATE_VIEW |
|---------------------------------------|
| CREATE OR REPLACE VIEW TEST_VIEW |
| AS SELECT |
| COL1, |
| COL2, |
| DATA1:Facebook::STRING AS Facebook, |
| DATA1:Icon::STRING AS Icon, |
| DATA1:URL::STRING AS URL, |
| DATA2:k1::STRING AS k1, |
| DATA2:k2::STRING AS k2, |
| DATA2:k3::STRING AS k3, |
| DATA1:Twitter::STRING AS Twitter, |
| DATA2:k4::STRING AS k4 |
| FROM TEST; |
| |
| VIEW created successfully. |
+---------------------------------------+
Then query the VIEW:
SELECT * FROM TEST_VIEW;
+------+------+-------------------------+-------+------+------+-------+-------------------------+------------------------+------+
| COL1 | COL2 | FACEBOOK | ICON | URL | K1 | K2 | K3 | TWITTER | K4 |
|------+------+-------------------------+-------+------+------+-------+-------------------------+------------------------+------|
| 1 | 2 | http://www.facebook.com | test1 | test | test | test1 | http://www.facebook.com | NULL | NULL |
| 3 | 4 | NULL | test1 | test | NULL | NULL | http://www.ericlin.me | http://www.twitter.com | v4 |
+------+------+-------------------------+-------+------+------+-------+-------------------------+------------------------+------+
Query the source table:
SELECT * FROM TEST;
+------+------+------------------------------------------+-----------------------------------+
| COL1 | COL2 | DATA1 | DATA2 |
|------+------+------------------------------------------+-----------------------------------|
| 1 | 2 | { | { |
| | | "Facebook": "http://www.facebook.com", | "k1": "test", |
| | | "Icon": "test1", | "k2": "test1", |
| | | "URL": "test" | "k3": "http://www.facebook.com" |
| | | } | } |
| 3 | 4 | { | { |
| | | "Icon": "test1", | "k3": "http://www.ericlin.me", |
| | | "Twitter": "http://www.twitter.com", | "k4": "v4" |
| | | "URL": "test" | } |
| | | } | |
+------+------+------------------------------------------+-----------------------------------+
You can refine this SP to detect nested data and have them added to the columns list as well.
The below code gives the error: A recognition error occurred
let vips = datatable (name: string)
['xxxx',
'yyyy',
'zzzz',
'gggg'];
DeviceLogonEvents
| where AccountName in~ (vips)
| summarize by DeviceName
| summarize vippc = make_list(DeviceName)
DeviceAlertEvents
| where DeviceName in (vippc)
Any suggestions how I can search for the items in the list vippc in the DeviceAlertEvents in the column DeviceName?
you could try this:
let vips = datatable(name: string)
[
'xxxx',
'yyyy',
'zzzz',
'gggg'
]
;
let vippc =
DeviceLogonEvents
| where AccountName in~ (vips)
| distinct DeviceName
;
DeviceAlertEvents
| where DeviceName in (vippc)
There are three tables.
Tables :
Trip
id | start_destination_id | end_destination_id | arrive_time |
-------------------------------------------------------------------
1 | S | E | 09:00 |
Destination
id | name
---------
S | Start
E | End
Schedule
id | start_destination_id | end_destination_id | should_arrive |
-------------------------------------------------------------------
1 | S | E | 08:00 |
2 | A | E | 10:00 |
Query
SELECT
Trip.*,
Schedule.should_arrive
FROM
Trip
LEFT JOIN
Schedule
ON
Trip.start_destination_id = Schedule.start_destination_id
AND
Trip.end_destination_id = Schedule.end_destination_id
I am trying to include Schedule in Trip.findAll but receive error
Exception: SequelizeEagerLoadingError: Schedule is not associated to Trip!
Is there a way that I can join them together without using foreign keys and raw queries?
Many thanks.
Finally I found a solution (not sure if it is a hack).
Schedule.ts
// add these lines
...
#ForeignKey(() => Trip)
#Column({ type: DataType.VIRTUAL })
private _dummyForTrip: undefined;
...
Then create an association between Schedule and Trip.
Trip.ts
#HasMany(() => Schedule)
public schedules: Schedule[] | null
Then you can include Schedule inside Trip by using include.on
const trips = await Trip.findAll({
include: [{
model: Schedule,
on: {
'$schedules.start$': { [Op.col]: "Trip.start_destination" },
'$schedules.end$': { [Op.col]: "Trip.end_destination" },
}
}],
where: {
id: { [Op.in]: payload.inputTripIdArr }
}
});
I have incoming json structure like
{
"type":1,
"location":[
{"lattitude":"0", "longitude":"0"},
{"lattitude":"0", "longitude":"0"},
{"lattitude":"0", "longitude":"0"}]
}
I need to insert this into a database like
|------|-----------|-----------|
| type | lattitude | longitude |
|------|-----------|-----------|
| 1 | 0 | 0 |
|------|-----------|-----------|
| 1 | 0 | 0 |
|------|-----------|-----------|
| 1 | 0 | 0 |
|------|-----------|-----------|
how do I parse the json and build an sql query?
If you want to use Postgres solution, you may do
--INSERT INTO yourtab( type,lattitude,longitude)
select jsoncol->>'type' , j->>'lattitude'
j->>'longitude'
from
( values (:yourjsonstr :: jsonb ) ) as t(jsoncol) cross join
lateral jsonb_array_elements(jsoncol->'location')
as j;
DEMO
You can use json-sql
Example:
var sql = jsonSql.build({
type: 'insert',
table: 'users',
values: {
name: 'John',
lastname: 'Snow',
age: 24,
gender: 'male'
}
});
sql.query
// insert into users (name, lastname, age, gender) values ($p1, $p2, 24, $p3);
sql.values
// { p1: 'John', p2: 'Snow', p3: 'male' }
See the documentation: https://www.npmjs.com/package/json-sql