I am trying to translate this sql into a slick 3.1 style collection query (single call). This sql (postgres) returns what I am looking for:
select
minDate.min as lastModified,
(select count("id") from "Items" where "orderId" = 1) as totalItemCount,
(select count("id") from "Items" where "orderId" = 1 and "dateModified" >= minDate.min) as addedCount
from
(select min("dateModified") as "min" from "Items" where "orderId" = 1 and "state" = 'new') as minDate
Returns: for a specified set of Items (from orderId), returns:
date of item last modified
total number of items
number of items added since the lastModified
But after many attempts, I can't figure out how to translate this to a single slick-style query
This codes
import scala.slick.driver.PostgresDriver
case class Item(id: Int, orderId: Int, state: String, dateModified: Int)
object SlickComplexQuery {
def main(args: Array[String]) = {
val driver = PostgresDriver
import driver.simple._
class ItemsTable(tag: Tag) extends Table[Item](tag, "Items"){
def id = column[Int]("id")
def orderId = column[Int]("orderId")
def state = column[String]("state")
def dateModified = column[Int]("dateModified")
def * = (id, orderId, state, dateModified) <> (Item.tupled, Item.unapply)
}
val items = TableQuery[ItemsTable]
val query1 = items
.filter(i => i.orderId === 1 && i.state === "new")
.map(_.dateModified)
.min
val query2 = items
.filter(_.orderId === 1)
.map(_.id)
.length
val query3 = items
.filter(i => i.orderId === 1 && i.dateModified >= query1)
.map(_.id)
.length
val query = Query(query1, query2, query3)
results in such query:
select x2.x3, x4.x5, x6.x7
from (select min(x8.x9) as x3
from (select x10."dateModified" as x9
from "Items" x10
where (x10."orderId" = 1) and (x10."state" = 'new')) x8) x2,
(select count(1) as x5
from (select x11."id" as x12
from "Items" x11
where x11."orderId" = 1) x13) x4,
(select count(1) as x7
from (select x14."id" as x15
from "Items" x14, (select min(x16.x17) as x18
from (select x19."dateModified" as x17
from "Items" x19
where (x19."orderId" = 1) and (x19."state" = 'new')) x16) x20
where (x14."orderId" = 1) and (x14."dateModified" >= x20.x18)) x21) x6
This query is much alike yours, slick 2.0 was used.
Related
I have a table like the following in Google BigQuery. I am trying to get all possible unique combination(all subsets except the null subset) of the Item column partitioned on Group.
Group Item
1 A
1 B
1 C
2 X
2 Y
2 Z
I am looking for an output like the following:
Group Item
1 A
1 B
1 C
1 A,B
1 B,C
1 A,C
1 A,B,C
2 X
2 Y
2 Z
2 X,Y
2 Y,Z
2 X,Z
2 X,Y,Z
I have tried to use this accepted answer to incorporate Group to no avail:
How to get combination of value from single column?
Consider below approach
CREATE TEMP FUNCTION generate_combinations(a ARRAY<STRING>)
RETURNS ARRAY<STRING>
LANGUAGE js AS '''
var combine = function(a) {
var fn = function(n, src, got, all) {
if (n == 0) {
if (got.length > 0) {
all[all.length] = got;
} return;
}
for (var j = 0; j < src.length; j++) {
fn(n - 1, src.slice(j + 1), got.concat([src[j]]), all);
} return;
}
var all = []; for (var i = 1; i < a.length; i++) {
fn(i, a, [], all);
}
all.push(a);
return all;
}
return combine(a)
''';
with your_table as (
select 1 as _Group,'A' as Item union all
select 1, 'B' union all
select 1, 'C' union all
select 2, 'X' union all
select 2, 'Y' union all
select 2, 'Z'
)
select _group, item
from (
select _group, generate_combinations(array_agg(item)) items
from your_table
group by _group
), unnest(items) item
with output
Try this
with _data as
(
select 1 as _Group,'A' as Item union all
select 1 as _Group,'B' as Item union all
select 1 as _Group,'C' as Item union all
select 2 as _Group,'X' as Item union all
select 2 as _Group,'Y' as Item union all
select 2 as _Group,'Z' as Item
)
select distinct _Group ,Item from
(
select _Group,
Item
from _data
union all
select _Group,
string_agg(Item ,',') over(partition by _Group order by Item ) as item
from _data
union all
select a._Group ,
concat(a.item,',',b.item)
from _data a left join _data b on a._group = b._group and a.Item < b.Item
)
where item is not null
order by _group
I'd like to use the following raw query in a ListAPIView in the Django REST Framework.
I'm not sure how to use a raw SQL query instead of a queryset to support pagination.
I've looked into it and didn't find much that made sense to me.
If I want to use a query set, it would look like this
How do I do this if I want to use raw SQL queries?
class VideoListView(generics.ListAPIView):
queryset = Video.objects.raw(...)
serializer_class = VideoSerializer
SELECT DISTINCT "t"."id",
"t"."title",
"t"."thumbnail_url",
"t"."preview_url",
"t"."embed_url",
"t"."duration",
"t"."views",
"t"."is_public",
"t"."published_at",
"t"."created_at",
"t"."updated_at",
EXISTS
(SELECT (1) AS "a"
FROM "videos_history" U0
WHERE (U0."user_id" IS NULL
AND U0."video_id" = "t"."id")
LIMIT 1) AS "is_viewed",
EXISTS
(SELECT (1) AS "a"
FROM "videos_favorite" U0
WHERE (U0."user_id" IS NULL
AND U0."video_id" = "t"."id")
LIMIT 1) AS "is_favorited",
EXISTS
(SELECT (1) AS "a"
FROM "videos_track" U0
INNER JOIN "videos_playlist" U1 ON (U0."playlist_id" = U1."id")
WHERE (U1."is_wl"
AND U1."user_id" IS NULL
AND U0."video_id" = "t"."id")
LIMIT 1) AS "is_wl"
FROM (
(SELECT "videos_video".*
FROM "videos_video"
WHERE ("videos_video"."is_public"
AND "videos_video"."published_at" <= '2022-01-03 05:20:16.725884+00:00'
AND "videos_video"."title" LIKE '%word')
ORDER BY "videos_video"."published_at" DESC
LIMIT 20)
UNION
(SELECT "videos_video".*
FROM "videos_video"
LEFT OUTER JOIN "videos_video_tags" ON ("videos_video"."id" = "videos_video_tags"."video_id")
LEFT OUTER JOIN "videos_tag" ON ("videos_video_tags"."tag_id" = "videos_tag"."id")
WHERE ("videos_video"."is_public"
AND "videos_video"."published_at" <= '2022-01-03 05:20:16.725884+00:00'
AND "videos_tag"."name" LIKE '%word')
ORDER BY "videos_video"."published_at" DESC
LIMIT 20)
) AS t
ORDER BY "t"."published_at" DESC
LIMIT 20;
Do I need to create a custom paginator or something? I'm even more confused because I'm using LIMIT and other methods for subqueries.
--- Add models.py, etc... ---
# models.py
class Tag(models.Model):
name = models.CharField(unique=True, max_length=30)
created_at = models.DateTimeField(default=timezone.now)
...
class Video(models.Model):
title = models.CharField(max_length=300)
tags = models.ManyToManyField(Tag, blank=True)
updated_at = models.DateTimeField(auto_now=True)
...
class History(models.Model):
user = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE)
video = models.ForeignKey(Video, on_delete=models.CASCADE)
...
class Favorite(models.Model):
user = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE
video = models.ForeignKey(Video, on_delete=models.CASCADE)
...
class Playlist(models.Model):
user = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE)
is_wl = models.BooleanField(default=False, editable=False)
...
class Track(models.Model):
playlist = models.ForeignKey(Playlist, on_delete=models.CASCADE, null=True)
video = models.ForeignKey(Video, on_delete=models.CASCADE)
...
The slow queryset and SQL queries that were originally used.
Video.objects.annotate(
is_viewed=Exists(History.objects.filter(user=user, video=OuterRef("pk"))),
is_favorited=Exists(
Favorite.objects.filter(user=user, video=OuterRef("pk"))
),
is_wl=Exists(
Track.objects.filter(
playlist__user=user, playlist__is_wl=True, video=OuterRef("pk")
)
),
).filter(
Q(title__contains=value)
| Q(tags__name__contains=value)),
is_public=True,
published_at__lte=timezone.now(),
).order_by("-published_at").distinct()
SELECT DISTINCT "videos_video"."id",
"videos_video"."published_at",
EXISTS
(SELECT (1) AS "a"
FROM "videos_history" U0
WHERE (U0."user_id" IS NULL
AND U0."video_id" = "videos_video"."id")
LIMIT 1) AS "is_viewed",
EXISTS
(SELECT (1) AS "a"
FROM "videos_favorite" U0
WHERE (U0."user_id" IS NULL
AND U0."video_id" = "videos_video"."id")
LIMIT 1) AS "is_favorited",
EXISTS
(SELECT (1) AS "a"
FROM "videos_track" U0
INNER JOIN "videos_playlist" U1 ON (U0."playlist_id" = U1."id")
WHERE (U1."is_wl"
AND U1."user_id" IS NULL
AND U0."video_id" = "videos_video"."id")
LIMIT 1) AS "is_wl"
FROM "videos_video"
LEFT OUTER JOIN "videos_video_tags" ON ("videos_video"."id" = "videos_video_tags"."video_id")
WHERE ("videos_video"."is_public"
AND "videos_video"."published_at" <= '2021-12-27 13:34:29.103369+00:00'
AND ("videos_video"."title" &#~ 'word'
OR "videos_video_tags"."tag_id" IN
(SELECT U0."id"
FROM "videos_tag" U0
WHERE U0."name" &#~ 'word')))
ORDER BY "videos_video"."published_at" DESC
LIMIT 20;
If you are asking about generic PageNumberPagination from rest_framework.pagination, usually it's done this way. You make your own pagination class with default values:
class VideoPagination(PageNumberPagination):
max_page_size = 100
page_size_query_param = 'page_size'
page_size = 25
Then you just add it to your view:
class VideoListView(generics.ListAPIView):
queryset = Video.objects.all()
serializer_class = VideoSerializer
pagination_class = VideoPagination
That's all, you can use parameters page and page_size.
I have a scenario where the SQL query with a where condition will result in 2 Rows. How can assert if it is resulting in 2 rows? At present, the karate is throwing an error org.springframework.dao.IncorrectResultSizeDataAccessException: Incorrect result size: expected 1, actual 2
* def response = db.readRow( 'SELECT * from database_name.table_name where id = \"'+ id + '\";')
I believe this should help you : https://github.com/intuit/karate#schema-validation
* def foo = ['bar', 'baz']
# should be an array of size 2
* match foo == '#[2]'
Also, you should use db.readRows instead of db.readRow.
* def dogs = db.readRows('SELECT * FROM DOGS')
* match dogs contains { ID: '#(id)', NAME: 'Scooby' }
* def dog = db.readRow('SELECT * FROM DOGS D WHERE D.ID = ' + id)
* match dog.NAME == 'Scooby'
I have two queries which links to different databases
query = "select name ,ctry from xxxx where xxxx"
cursor.execute(query)
results1 = list(cursor.fetchall())
for row in results1:
query1 = "SELECT sessionname, country FROM xxx where and sessions.sessionname = '"+row[0] +"'"
cur.execute(query1)
results2.append(cur.fetchall())
How to connect them if they have common value(sessionname and name) and save it's output to file. Both queries are located in different dbo (oracle, postgresql)
My code is here :
try:
query = """select smat.s_name "SQLITE name" ,smed.m_ctry as "Country", smed.m_name "HDD Label" from smart.smed2smat ss, smart.smed smed, smart.smat smat where ss.M2S_SMAT=smat.s_id and ss.m2s_smed=smed.m_id and smed.m_name like '{0}%' order by smat.s_name""" .format(line_name)
cursor.execute(query)
columns = [i[0] for i in cursor.description]
results1 = cursor.fetchall()
for row in results1:
query1 = "SELECT sessions.sessionname, projects.country , projects.projectname FROM momatracks.sessions, momatracks.projects, momatracks.sessionsgeo where sessions.projectid = projects.id and sessionsgeo.sessionname = sessions.sessionname and sessions.sessionname = '"+row[0] +"' order by sessions.sessionname"
cur.execute(query1)
results2 =cur.fetchall()
print "results1 -----> \n:", row
tmp=[]
output_items = []
for tmp in results2:
print "---> \n", tmp
try:
stations_dict = dict([(item[0], item[1:]) for item in tmp])
for item in row:
output_item = list(item) + stations_dict.get(item[0], [])
output_items.append(output_item)
except Exception, f:
print str (f)
cursor.close()
cur.close()
except Exception, g:
print str ( g )
except Exception, e:
print str ( e )
My results from row and tmp are :
row - WE246JP_2015_10_11__14_53_33', 'NLD', '031_025_SQLITE_NLD1510_03INDIA
and
tmp - WE246JP_2015_10_11__14_53_33', 'NLD', 'NLD15_N2C1-4_NL'
How to properly connect them? I want output look like this :
output_items - WE246JP_2015_10_11__14_53_33', 'NLD', '031_025_SQLITE_NLD1510_03INDIA', 'NLD15_N2C1-4_NL'
At the moment i get this error :
can only concatenate list (not "str") to list
Also value station_dict looks like this :( And this is not what i intended to do
'W': 'E246JP_2015_10_11__15_23_33', 'N': 'LD15_N2C1-4_NL3'
I know there is something wrong with my code which is simmilar to join. Can anyone explain this to me ? I used method below :
http://forums.devshed.com/python-programming-11/join-arrays-based-common-value-sql-left-join-943177.html
If the sessions are exactly the same in both databases then just zip the results:
query = """
select
smat.s_name "SQLITE name",
smed.m_ctry as "Country",
smed.m_name "HDD Label"
from
smart.smed2smat ss
inner join
smart.smed smed on ss.M2S_SMAT = smat.s_id
inner join
smart.smat smat on ss.m2s_smed = smed.m_id
where smed.m_name like '{0}%'
order by smat.s_name
""".format(line_name)
cursor.execute(query)
results1 = cursor.fetchall()
query1 = """
select
sessions.sessionname,
projects.country,
projects.projectname
from
momatracks.sessions,
inner join
momatracks.projects on sessions.projectid = projects.id
inner join
momatracks.sessionsgeo on sessionsgeo.sessionname = sessions.sessionname
where sessions.sessionname in {}
order by sessions.sessionname
""".format(tuple([row[0] for row in results1]))
cur.execute(query1)
results2 = cur.fetchall()
zipped = zip(results1, results2)
output_list = [(m[0][0], m[0][1], m[0][2], m[1][2]) for m in zipped]
If the sessions are different then make each result a dictionary to join.
I think you can use a subquery here. There's no way for me to test it, but I think it should look like this:
SELECT *
FROM (SELECT smat.s_name "SQLITE name" ,
smed.m_ctry as "Country",
smed.m_name "HDD Label"
FROM smart.smed2smat ss,
smart.smed smed,
smart.smat smat
WHERE ss.M2S_SMAT=smat.s_id
AND ss.m2s_smed=smed.m_id
AND smed.m_name like '{0}%'
ORDER BY smat.s_name) t1,
(SELECT sessions.sessionname,
projects.country ,
projects.projectname
FROM momatracks.sessions,
momatracks.projects,
momatracks.sessionsgeo
WHERE sessions.projectid = projects.id
AND sessionsgeo.sessionname = sessions.sessionname
AND sessions.sessionname = '"+row[0] +"'
ORDER BY sessions.sessionname) t2
WHERE t1."SQLITE name" = t2.sessionname ;
I am trying to construct a double count left join query in Slick 3.1 similar to this:
SELECT
COUNT(p.id) AS replies,
COUNT(f.id) AS images
FROM posts AS p
LEFT JOIN files AS f
ON p.id = f.post_id
WHERE thread = :thread_id
Join part of the query is quite simple and looks like this:
val joinQ = (threadId: Rep[Long]) =>
(postDAO.posts joinLeft fileRecordDAO.fileRecords on (_.id === _.postId))
.filter(_._1.thread === threadId)
Using joinQ(1L).map(_._1.id).length generates COUNT(1) which counts all rows - not the result I want to obtain. Using joinQ(1L).map(_._1.id).countDistinct generates COUNT(DISTINCT p.id) which is somewhat what I'm looking for, but trying to do two of these generates this monstrosity:
select x2.x3, x4.x5
from (select count(distinct x6.`id`) as x3
from `posts` x6
left outer join `files` x7 on x6.`id` = x7.`post_id`
where x6.`thread` = 1) x2,
(select
count(distinct (case when (x8.`id` is null) then null else 1 end)) as x5
from `posts` x9
left outer join `files` x8 on x9.`id` = x8.`post_id`
where x9.`thread` = 1) x4
Here's the double countDistinct code:
val q = (threadId: Rep[Long]) => {
val join = joinQ(threadId)
val q1 = join.map(_._1.id).countDistinct // map arg type is
val q2 = join.map(_._2).countDistinct // (Post, Rep[Option[FileRecord]])
(q1, q2)
}
Any ideas? :)
Count is aggregation function, so you also need grouping by some field (e.x. id). Try next query:
def joinQ(threadId: Long) = (postDAO.posts joinLeft fileRecordDAO.fileRecords on (_.id === _.postId))
.filter(_._1.thread === threadId)
val q = (threadId: Long) => {
joinQ(threadId).groupBy(_._1.id).map {
case (id, qry) => (id, qry.map(_._1.id).countDistinct, qry.map(_._2.map(_.id)).countDistinct)
}
}
It generates next sql:
SELECT x2.`id`,
Count(DISTINCT x2.`id`),
Count(DISTINCT x3.`id`)
FROM `posts` x2
LEFT OUTER JOIN `files` x3
ON x2.`id` = x3.`post_id`
WHERE x2.`thread` = 10
GROUP BY x2.`id`