BigQuery Asynchronus Query Job garbles the Query String - google-bigquery

i'm using the Ruby Google API Client to call BigQuery. The Query works just fine if I call it synchronously via bq.jobs.query.
Here's the job: job_sQUillv1JOSPg7XdJa7k8JTexPc
Here's the actual query:
select uid,group_concat(event) as path from
(select uid, event,t from
(select properties.distinct_id as uid, event , properties.time as t, t1.ev as ev, t1.time as originalTime from
[ziptrips.ziptrips_events ]
join
( select properties.distinct_id as user, properties.time as time, event as ev from
[ziptrips.ziptrips_events ] where properties.time > 1385856000 and properties.time < 1388534399 and event = 'Home Page Loaded'
group by user,time,ev) as t1
on t1.user = properties.distinct_id
where properties.time < t1.time)
group by uid,event ,t
order by t desc)
group by uid
However when I do the same call asynchronously via bq.jobs.insert, the query string gets garbled (It has a whole bunch of \n and \u003e etc). And the job fails.
select uid,group_concat(event) as path from\n (select uid, event,t from \n (select properties.distinct_id as uid, event , properties.time as t, t1.ev as ev, t1.time as originalTime from \n [ziptrips.ziptrips_events ] \n join \n ( select properties.distinct_id as user, properties.time as time, event as ev from \n [ziptrips.ziptrips_events ] where properties.time \u003e 1385856000 and properties.time \u003c 1388534399 and event = 'Home Page Loaded' \n group by user,time,ev) as t1 \n on t1.user = properties.distinct_id \n where properties.time \u003c t1.time) \n group by uid,event ,t \n order by t desc) \n group by uid
Here's the code:
client,bq = initialize()
config = {
'query' => {
'query' => query
}
}
result = client.execute(:api_method => bq.jobs.insert,
:parameters => {'projectId' => 'onefold-1'},
:body_object => {
"configuration" => config,
"kid" => "bigquery#queryRequest",
"query" => query,
"maxResults" => 1000,
"defaultDataset" => {
"datasetId" => "#{dataset_id}",
"projectId" => 'onefold-1'
},
"timeoMs" => 3000,
"dryRun" => false,
"preserveNulls" => false,
"useQueryCache" => true,
"miCompletionRatio" => 1.0
}
)
I believe it has something to do with encoding etc. Can't put my finger on whether and where to specify the encoding. Appreciate any pointers or suggestions.
Thanks,
Navneet

I found out that this was a non-issue. There was a different error happening at the same time and I linked that error to this.
It works!

Related

TypeORM distinctOn with orderBy

I'm trying to get the latest message for a user, either if he is the sender or the receiver.
But I can't use distinctOn with orderBy
const query = this.createQueryBuilder('message')
.orderBy('message.date', 'DESC')
.where('message.senderId = :id OR message.receiverId = :id',{id:user.id})
.distinctOn(['message.senderId','message.receiverId'])
const messages = await query.getRawMany();
SELECT DISTINCT ON ("message"."senderId", "message"."receiverId") "message"."id" AS "message_id", "message"."message" AS "message_message", "message"."date" AS "message_date", "message"."status" AS "message_status", "message"."senderId" AS "message_senderId", "message"."receiverId" AS "message_receiverId"
FROM "message" "message"
WHERE "message"."senderId" = $1
OR "message"."receiverId" = $2
ORDER BY "message"."date" DESC
SELECT DISTINCT ON expressions must match initial ORDER BY expressions
You can also try this way
const messages = this.createQueryBuilder('message')
.distinctOn('DISTINCT ON (message.senderId) as senderId, message.receiverId' as "receiverId")
.where('message.senderId = :id OR message.receiverId = :id',{id:user.id})
.orderBy('message.date', 'DESC').getRawMany()

postgresql - Aggregate function calls cannot be nested. How can I format this query in a nice json format?

I'm attempting to create a psql query that returns data in a nice json format. Originally for the project we used sequelize and i'm trying to convert it to pure psql. The format i'm looking for is as follows -
...
vehicles:
[
{
services:
[
{
name: "example"
duration: "example"
}
]
id: 1
registration: "example"
}
]
...
Below is what I tried first but got the error. ERROR: aggregate function calls cannot be nested
SELECT
"Bookings"."id",
"Bookings"."startTime",
"Bookings"."code",
"Bookings"."statusId",
"Bookings"."address",
"Bookings"."latitude",
"Bookings"."longitude",
"Bookings"."createdAt",
"Bookings"."overridePrice",
json_build_object(
'fullName', "customer"."fullName",
'id', "customer"."id"
) AS "customer",
json_agg(json_build_object(
'fullName', "serviceProvider"."fullName",
'id', "serviceProvider"."id"
)) AS "serviceProvider",
-- Array of vehicles, this is where the issue lies
json_agg(
json_build_object(
'services', json_agg(
json_build_object(
'name', "ServiceHistories"."name",
'duration', "ServiceHistories"."duration"
)
),
'id', "Vehicles"."id",
'registration', "Vehicles"."registration"
)) AS "vehicles",
-- Everything else works fine
json_agg(
json_build_object(
'amount', "payments"."amount",
'amountFromGiftCard', "payments"."amountFromGiftCard"
)
) AS "payments"
FROM
"Bookings",
"Users" AS "customer",
"Users" AS "serviceProvider",
"BookingServiceProviders",
"BookingVehicleServices",
"BookingVehicles",
"Vehicles",
"Payments" AS "payments",
"ServiceHistories"
WHERE
"Bookings"."isDeleted" IS NOT true AND
"Bookings"."statusId" != 4 AND
"Bookings"."customerId" = "customer"."id" AND "customer"."userTypeId" = 1 AND
"Bookings"."id" = "BookingServiceProviders"."BookingId" AND
"BookingServiceProviders"."UserId" = "serviceProvider"."id" AND
"Bookings"."id" = "BookingVehicles"."BookingId" AND
"Vehicles"."id" = "BookingVehicles"."VehicleId" AND
"Bookings"."id" = "payments"."bookingId" AND
"BookingVehicles"."id" = "BookingVehicleServices"."BookingVehicleId" AND
"ServiceHistories"."bookingId" = "Bookings"."id"
GROUP BY
"Bookings"."id",
"customer"."id",
"customer"."fullName"
Order by "startTime" ASC
After doing some googling I found that i'll probably have to do a sub query for the services array. Below is what I attempted but the query runs indefinitely.
SELECT
"Bookings"."id",
"Bookings"."startTime",
"Bookings"."code",
"Bookings"."statusId",
"Bookings"."address",
"Bookings"."latitude",
"Bookings"."longitude",
"Bookings"."createdAt",
"Bookings"."overridePrice",
json_build_object(
'fullName', "customer"."fullName",
'id', "customer"."id"
) AS "customer",
json_agg(json_build_object(
'fullName', "serviceProvider"."fullName",
'id', "serviceProvider"."id"
)) AS "serviceProvider",
-- Issue lies here
json_agg(
json_build_object(
'services', (
SELECT
json_agg(json_build_object(
'name', "ServiceHistories"."name",
'duration', "ServiceHistories"."duration"
))
FROM
"ServiceHistories"
WHERE
"ServiceHistories"."bookingId" = "Bookings"."id"
),
'id', "Vehicles"."id",
'registration', "Vehicles"."registration"
)) AS "vehicles",
-- End of issue
json_agg(
json_build_object(
'amount', "payments"."amount",
'amountFromGiftCard', "payments"."amountFromGiftCard"
)
) AS "payments"
FROM
"Bookings",
"Users" AS "customer",
"Users" AS "serviceProvider",
"BookingServiceProviders",
"BookingVehicleServices",
"BookingVehicles",
"Vehicles",
"Payments" AS "payments"
WHERE
"Bookings"."isDeleted" IS NOT true AND
"Bookings"."statusId" != 4 AND
"Bookings"."customerId" = "customer"."id" AND "customer"."userTypeId" = 1 AND
"Bookings"."id" = "BookingServiceProviders"."BookingId" AND
"BookingServiceProviders"."UserId" = "serviceProvider"."id" AND
"Bookings"."id" = "BookingVehicles"."BookingId" AND
"Vehicles"."id" = "BookingVehicles"."VehicleId" AND
"Bookings"."id" = "payments"."bookingId" AND
"BookingVehicles"."id" = "BookingVehicleServices"."BookingVehicleId"
GROUP BY
"Bookings"."id",
"customer"."id",
"customer"."fullName"
Order by "startTime" ASC
If anyone could help or point me in the right direction I'd appreciate it.
** EDIT **
I kind of have it working by adding more where clauses on how the tables relate, now the services are returning correctly within the vehicles array. Sadly its super slow. I had it working faster but there were duplicate services when it should have only been ones related to the specific booking.
SELECT
bookings."id",
bookings."startTime",
bookings."code",
bookings."statusId",
bookings."address",
bookings."latitude",
bookings."longitude",
bookings."createdAt",
bookings."overridePrice",
json_build_object(
'fullName', "customer"."fullName",
'id', "customer"."id"
) AS "customer",
json_agg(json_build_object(
'fullName', "serviceProvider"."fullName",
'id', "serviceProvider"."id"
)) AS "serviceProvider",
json_agg(
json_build_object(
'amount', "payments"."amount",
'amountFromGiftCard', "payments"."amountFromGiftCard"
)
) AS "payments",
(
SELECT
(nested_vehicles)
FROM (
SELECT
"Vehicles".id as vehicle_id,
"Vehicles".registration,
(
SELECT
json_agg(nested_services)
FROM (
SELECT
"ServiceHistories"."id",
"ServiceHistories"."name",
"ServiceHistories"."duration"
FROM
"ServiceHistories", "BookingVehicleServices", "BookingVehicles"
WHERE
"ServiceHistories"."vehicleId" = "Vehicles"."id" AND
"ServiceHistories"."id" = "BookingVehicleServices"."ServiceHistoryId" AND
bookings.id = "BookingVehicles"."BookingId" AND
"Vehicles"."id" = "BookingVehicles"."VehicleId" AND
"ServiceHistories"."bookingId" = bookings.id
) AS nested_services
) AS services
FROM
"Vehicles", "BookingVehicles"
WHERE "Vehicles"."id" = "BookingVehicles"."VehicleId" AND bookings.id = "BookingVehicles"."BookingId"
GROUP BY "Vehicles"."id"
) AS nested_vehicles
) AS vehicles
FROM
"Bookings" as bookings,
"Users" AS "customer",
"Users" AS "serviceProvider",
"BookingServiceProviders",
"Payments" AS "payments"
WHERE
bookings."isDeleted" IS NOT true AND
bookings."statusId" != 4 AND
bookings."customerId" = "customer"."id" AND "customer"."userTypeId" = 1 AND
bookings."id" = "BookingServiceProviders"."BookingId" AND
"BookingServiceProviders"."UserId" = "serviceProvider"."id" AND
bookings."id" = "payments"."bookingId"
GROUP BY
bookings."id",
"customer"."id",
"customer"."fullName"
Order by "startTime" ASC
Not really sure what to do here. I might just return the services separate and then format them externally :/
ERROR: aggregate function calls cannot be nested
Not sure for best practice but I use subqueries or Common Table Expressions (CTE).
First aggregate function call makes from subquery/CTE, second aggregate function call makes from query.
-- Common Table Expressions
WITH regional_sales AS (
SELECT region, SUM(amount) AS total_sales
FROM orders
GROUP BY region
), top_regions AS (
SELECT region
FROM regional_sales
WHERE total_sales > (SELECT SUM(total_sales)/10 FROM regional_sales)
)
SELECT region,
product,
SUM(quantity) AS product_units,
SUM(amount) AS product_sales
FROM orders
WHERE region IN (SELECT region FROM top_regions)
GROUP BY region, product;

Snowflake task gives me no error but table is empty

I have the following TASK in Snowflake:
CREATE OR REPLACE TASK TSK_MASTER
WAREHOUSE = MYWH
SCHEDULE = '1 MINUTE'
WHEN
SYSTEM$STREAM_HAS_DATA('MYSTREAM')
AS
INSERT INTO USER_DATA(ID,CREATED_AT,DEACTIVATED,EMAIL_ADDRESS,NAME,ROLE)
SELECT
vm.VALUE:id::string AS "ID",
vm.VALUE:created_at::timestamp AS "CREATED_AT",
vm.VALUE['profile']:deactivated::boolean AS "DEACTIVATED",
vm.VALUE['profile']:email_address::string AS "EMAIL_ADDRESS",
vm.VALUE['profile']:name::string AS "NAME",
vm.VALUE:role::string AS "ROLE"
FROM
MYSTREAM
, lateral flatten(input => RAW_JSON) vm
WHERE
vm.VALUE:entity_type::string = 'member'
AND
METADATA$ACTION = 'INSERT';
SELECT SYSTEM$STREAM_HAS_DATA('MYSTREAM') is TRUE but my task doesn't seem to run.
I then troubleshoot the following and get the IDs as intended.
SELECT
vm.VALUE:id::string AS "ID"
FROM
MYSTREAM
, lateral flatten(input => RAW_JSON) vm
WHERE
vm.VALUE:entity_type::string = 'member'
AND
SYSTEM$STREAM_HAS_DATA('MYSTREAM')
AND
METADATA$ACTION = 'INSERT';
All you need to do is ALTER TASK <TASK_NAME> RESUME

Laravel 5: Join On with IN query

I'm trying to do something like:
$results = $query->leftJoin('checklist_items', function($join) use ($days) {
$join->on('users.id', '=', 'checklist_items.user_id')
->on('checklist_items.due_date', 'IN', $days);
})
->where('checklist_items.user_id', null)
->limit(10)
->get();
This is an example of the query I'm attempting to execute:
SELECT *
FROM users
LEFT JOIN checklist_items
ON users.id = checklist_items.user_id
AND checklist_items.due_date IN ('2015-07-09', '2015-07-10')
WHERE checklist_items.user_id IS NULL
ORDER BY users.id
So this is a left outer join. In query builder, most of this is no problem. The problem is the fact that my AND line uses an IN query. If it were part of a WHERE clause I would use ->whereIn but since I need it in the Join clause, whereIn won't work and there is no orIn or some such.
Suggestions?
You can use ->whereIn() within the ->leftJoin() closure (Tested in Laravel 5.7.16):
$days = ['2015-07-09', '2015-07-10'];
$results = \DB::table('users')
->leftJoin('checklist_items', function($join) use ($days) {
$join->on('users.id', '=', 'checklist_items.user_id')
->whereIn('checklist_items.due_date', $days);
})
->where('checklist_items.user_id', null)
->orderby('users.id')
->get();
Output from dd(\DB::getQueryLog(); produces your example query:
array:1 [▼
0 => array:3 [▼
"query" => "select * from `users` left join `checklist_items` on `users`.`id` = `checklist_items`.`user_id` and `checklist_items`.`due_date` in (?, ?) where `checklist_items`.`user_id` is null order by `users`.`id` asc ◀"
"bindings" => array:2 [▼
0 => "2015-07-09"
1 => "2015-07-10"
]
"time" => 6.97
]
]
I think you would need to use DB::raw() so it doesn't try to quote your days and wrap your days in parenthesis as well. This should do the trick.
$days = '(\'2015-07-09\', \'2015-07-10\')';
$results = DB::table('users')->leftJoin('checklist_items', function($join) use ($days) {
$join->on('users.id', '=', 'checklist_items.user_id')
->on('checklist_items.due_date', 'IN', DB::raw($days));
})
->where('checklist_items.user_id', null)
->limit(10)
->toSql();
echo $results;
This Query will Work
$results = DB::table('users')
->join('checklist_items','checklist_items.user_id','=','users.id')
->whereIn('checklist_items.due_date',['2015-07-09', '2015-07-10'])
->whereNull('checklist_items.user_id')
->orderBy('users.id','asc')
For Laravel 7 and above use whereIn instead of on :
$join->on('users.id', '=', 'checklist_items.user_id')
->whereIn('checklist_items.due_date',$days);

SQL query to cakePHP format (invalid json result)

Hello I have a PostgreSQL query that I would like to write using cakePHP format
SELECT
id,
title,
author,
postdate,
postcontent,
userID
FROM posts
WHERE
userID = 12
AND id IN (SELECT articleID FROM favourites WHERE articlesUserID = 12)
ORDER BY postdate DESC;
this is the format my query has right now in cakePHP :
$favouritearticles = $this->Favourite->query('SELECT id, title, author, postdate, postdatecreation, posteditdate, postcontent, "userID" FROM posts WHERE "userID" = '.$userID.'AND id IN (SELECT lngblogarticleid FROM favourites WHERE lngloginuserid = '.$userID.') ORDER BY postdate DESC');
It's working but if echo json_encode the result like this :
echo json_encode($favouritearticles);
I get an invalid json format like the following :(checked with JSONLint)
[
[
{
"id": 2,
"title": "Prison Or Treatment For the Mentally ill ",
"author": "mike123",
"postdate": "March 12, 2013 at 6:46 pm",
"postdatecreation": "2013-03-12",
"posteditdate": null,
"postcontent": "<p><span>The public revulsion over repeated mass shootings has placed mental health in the spotlight. This is both good and bad.<\/span><\/p>",
"userID": 34
}
]
][
]
So I thought that maybe I should rewrite my query using cakePHP format "using find method" something like :
$favouritearticles = $this->Favourite->find('all',array('conditions'=>array(".........
however the query is quite complex and I don't see how to do so.
Thank you for any help.
Format of JSON is fine except for extra [ ] at the end.
If you still want to rewrite the query in CakePHP format, use following:
private function getFavouritePostsByUserId($userId) {
$db = $this->Post->getDataSource();
$subQuery = $db->buildStatement(
array(
'fields' => array('Favourite.articleID'),
'table' => $db->fullTableName($this->Favourite),
'alias' => 'Favourite',
'conditions' => array(
'Favourite.articlesUserID' => $userId
),
),
$this->Favourite
);
$subQuery = 'Post.id IN (' . $subQuery . ') ';
$subQueryExpression = $db->expression($subQuery);
$conditions = array($subQueryExpression, 'Post.userID' => $userId);
$fields = array('Post.*');
$order = 'Post.postdate DESC';
$this->Post->find('all', compact('conditions', 'fields', 'order'));
}