Translate nested join and groupby query to Slick 3.0 - sql

I'm implementing a todo list. A user can have multiple lists and a list can have multiple users. I want to be able to retrieve all the lists for a user, where each of these lists contain a list of the users for which it's shared (including the owner). Not succeeding implementing this query.
The table definitions:
case class DBList(id: Int, uuid: String, name: String)
class Lists(tag: Tag) extends Table[DBList](tag, "list") {
def id = column[Int]("id", O.PrimaryKey, O.AutoInc) // This is the primary key column
def uuid = column[String]("uuid")
def name = column[String]("name")
// Every table needs a * projection with the same type as the table's type parameter
def * = (id, uuid, name) <> (DBList.tupled, DBList.unapply)
}
val lists = TableQuery[Lists]
case class DBUser(id: Int, uuid: String, email: String, password: String, firstName: String, lastName: String)
// Shared user projection, this is the data of other users which a user who shared an item can see
case class DBSharedUser(id: Int, uuid: String, email: String, firstName: String, lastName: String, provider: String)
class Users(tag: Tag) extends Table[DBUser](tag, "user") {
def id = column[Int]("id", O.PrimaryKey, O.AutoInc) // This is the primary key column
def uuid = column[String]("uuid")
def email = column[String]("email")
def password = column[String]("password")
def firstName = column[String]("first_name")
def lastName = column[String]("last_name")
def * = (id, uuid, email, password, firstName, lastName) <> (DBUser.tupled, DBUser.unapply)
def sharedUser = (id, uuid, email, firstName, lastName) <> (DBSharedUser.tupled, DBSharedUser.unapply)
}
val users = TableQuery[Users]
// relation n:n user-list
case class DBListToUser(listUuid: String, userUuid: String)
class ListToUsers(tag: Tag) extends Table[DBListToUser](tag, "list_user") {
def listUuid = column[String]("list_uuid")
def userUuid = column[String]("user_uuid")
def * = (listUuid, userUuid) <> (DBListToUser.tupled, DBListToUser.unapply)
def pk = primaryKey("list_user_unique", (listUuid, userUuid))
}
val listToUsers = TableQuery[ListToUsers]
I created an additional class to hold the database list object + the users, my goal is to map the query result somehow to instances of this class.
case class DBListWithSharedUsers(list: DBList, sharedUsers: Seq[DBSharedUser])
This is the SQL query for most of it, it gets first all the lists for the user (inner query) then it does a join of lists with list_user with user in order to get the rest of the data and the users for each list, then it filters with the inner query. It doesn't contain the group by part
select * from list inner join list_user on list.uuid=list_user.list_uuid inner join user on user.uuid=list_user.user_uuid where list.uuid in (
select (list_uuid) from list_user where user_uuid=<myUserUuuid>
);
I tested it and it works. I'm trying to implement it in Slick but I'm getting a compiler error. I also don't know if the structure in that part is correct, but haven't been able to come up with a better one.
def findLists(user: User) = {
val listsUsersJoin = listToUsers join lists join users on {
case ((listToUser, list), user) =>
listToUser.listUuid === list.uuid &&
listToUser.userUuid === user.uuid
}
// get all the lists for the user (corresponds to inner query in above SQL)
val queryToGetListsForUser = listToUsers.filter(_.userUuid===user.uuid)
// map to uuids
val queryToGetListsUuidsForUser: Query[Rep[String], String, Seq] = queryToGetListsForUser.map { ltu => ltu.listUuid }
// create query that mirrors SQL above (problems):
val queryToGetListsWithSharedUsers = (for {
listsUuids <- queryToGetListsUuidsForUser
((listToUsers, lists), users) <- listsUsersJoin
if lists.uuid.inSet(listsUuids) // error because inSet requires a traversable and passing a ListToUsers
} yield (lists))
// group - doesn't compile because above doesn't compile:
queryToGetListsWithSharedUsers.groupBy {case (list, user) =>
list.uuid
}
...
}
How can I fix this?
Thanks in advance
Edit:
I put together this emergency solution (at least it compiles), where I execute the query using raw SQL and then do the grouping programmatically, it looks like this:
case class ResultTmp(listId: Int, listUuid: String, listName: String, userId:Int, userUuid: String, userEmail: String, userFirstName: String, userLastName: String, provider: String)
implicit val getListResult = GetResult(r => ResultTmp(r.nextInt, r.nextString, r.nextString, r.nextInt, r.nextString, r.nextString, r.nextString, r.nextString, r.nextString))
val a = sql"""select (list.id, list.uuid, list.name, user.id, user.uuid, user.email, user.first_name, user.last_name, user.provider) from list inner join list_user on list.uuid=list_user.list_uuid inner join user on user.uuid=list_user.user_uuid where list.uuid in (
select (list_uuid) from list_user where user_uuid=${user.uuid}
);""".as[ResultTmp]
val result: Future[Vector[ResultTmp]] = db.run(a)
val res: Future[Seq[DBListWithSharedUsers]] = result.map {resultsTmp =>
val myMap: Map[String, Vector[ResultTmp]] = resultsTmp.groupBy { resultTmp => resultTmp.listUuid }
val r: Iterable[DBListWithSharedUsers] = myMap.map {case (listUuid, resultsTmp) =>
val first = resultsTmp(0)
val list = DBList(first.listId, listUuid, first.listName)
val users: Seq[DBSharedUser] = resultsTmp.map { resultTmp =>
DBSharedUser(resultTmp.userId, resultTmp.userUuid, resultTmp.userEmail, resultTmp.userFirstName, resultTmp.userLastName, resultTmp.provider)
}
DBListWithSharedUsers(list, users)
}
r.toSeq
}
But that's just horrible, how do I get it working the normal way?
Edit 2:
I'm experimenting with monadic joins but also stuck here. For example something like this would get all the lists for a given user:
val listsUsersJoin = for {
list <- lists
listToUser <- listToUsers
user_ <- users if user_.uuid === user.uuid
} yield (list.uuid, list.name, user.uuid, user.firstName ...)
but this is not enough because I need the get also all the users for those lists, so I need 2 queries. So I need to get first the lists for the user and then find all the users for those lists, something like:
val queryToGetListsForUser = listToUsers.filter(_.userUuid===user.uuid)
val listsUsersJoin = for {
list <- lists
listToUser <- listToUsers
user_ <- users /* if list.uuid is in queryToGetListsForUser result */
} yield (list.uuid, list.name, user.uuid, user.firstName ... )
But I don't know how to pass that to the join. I'm not even sure if groupBy, at least at database level is correct, so far I see this used only to aggregate the results to a single value, like count or avg. I need them in a collection.
Edit 3:
I don't know yet if this is right but the monadic join may be the path to the solution. This compiles:
val listsUsersJoin = for {
listToUser <- listToUsers if listToUser.userUuid === user.uuid // get the lists for the user
list <- lists if list.uuid === listToUser.listUuid // join with list
listToUser2 <- listToUsers if list.uuid === listToUser.listUuid // get all the users for the lists
user_ <- users if user_.uuid === listToUser2.userUuid // join with user
} yield (list.uuid, list.name, user.uuid, user.email, user.firstName, user.lastName)

Ah, look at that, I came up with a solution. I still have to test if works but at least the compiler stopped shouting at it. I’ll edit this later if necessary.
val listsUsersJoin = for {
listToUser <- listToUsers if listToUser.userUuid === user.uuid
list <- lists if list.uuid === listToUser.listUuid
listToUser2 <- listToUsers if list.uuid === listToUser.listUuid
user_ <- users if user_.uuid === listToUser2.userUuid
} yield (list.id, list.uuid, list.name, user_.id, user_.uuid, user_.email, user_.firstName, user_.lastName, user_.provider)
val grouped = listsUsersJoin.groupBy(_._2)
val resultFuture = db.run(grouped.result).flatMap {groupedResults =>
val futures: Seq[Future[DBListWithSharedUsers]] = groupedResults.map {groupedResult =>
val listUuid = groupedResult._1
val valueQuery = groupedResult._2
db.run(valueQuery.result).map {valueResult =>
val first = valueResult(0) // if there's a grouped result this should never be empty
val list = DBList(first._1, listUuid, first._3)
val users = valueResult.map {value =>
DBSharedUser(value._4, value._5, value._6, value._7, value._8, value._9)
}
DBListWithSharedUsers(list, users)
}
}
Future.sequence(futures)
}

Related

How to join two tables on two fields using Exposed?

I have the following database Tables:
// CurrenciesTable
object CurrenciesTable : Table("currencies") {
val symbol = varchar("symbol", 48)
val name = varchar("name", 48)
override val primaryKey = PrimaryKey(symbol)
}
// OrdersTable
object OrdersTable : IntIdTable("orders") {
val baseCurrency = varchar("base_currency", 48)
val counterCurrency = varchar("counter_currency", 48)
val price = decimal("price", DECIMAL_PRECISION, DECIMAL_SCALE)
val createdAtEpochSecond = long("created_at_epoch_second")
}
In the OrdersTable, I have to fields which reference CurrenciesTable:
baseCurrency
counterCurrency
I want to select records from OrdersTable and join them with CurrenciesTable on two fields. So I get the symbol and name for each currency.
Here is my DSL query to join on baseCurrency field only.
// Exposed DSL
OrdersTable.join(CurrenciesTable, JoinType.INNER, OrdersTable.baseCurrency, CurrenciesTable.symbol)
.selectAll()
.forEach {
// Getting OrdersTable record data
it[OrdersTable.id].value
it[OrdersTable.price]
it[OrdersTable.createdAtEpochSecond]
// Getting CurrenciesTable record data (for baseCurrency only)
it[CurrenciesTable.symbol]
it[CurrenciesTable.name]
}
I tried to do a second join as follows:
// Exposed DSL
OrdersTable.join(CurrenciesTable, JoinType.INNER, OrdersTable.baseCurrency, CurrenciesTable.symbol)
.join(CurrenciesTable, JoinType.INNER, OrdersTable.counterCurrency, CurrenciesTable.symbol)
However, I get the following exception.
Caused by: java.sql.SQLSyntaxErrorException: Not unique table/alias: 'currencies'
Try to add alias for CurrenciesTable:
OrdersTable.innerJoin(CurrenciesTable.alias("baseCurrency"), { OrdersTable.baseCurrency }, { CurrenciesTable.symbol })
.innerJoin(CurrenciesTable, { OrdersTable.counterCurrency }, { CurrenciesTable.symbol })

Compare multiple fields of Object to those in an ArrayList of Objects

I have created a 'SiteObject' which includes the following fields:
data class SiteObject(
//Site entry fields (10 fields)
var siteReference: String = "",
var siteAddress: String = "",
var sitePhoneNumber: String = "",
var siteEmail: String = "",
var invoiceAddress: String = "",
var invoicePhoneNumber: String = "",
var invoiceEmail: String = "",
var website: String = "",
var companyNumber: String = "",
var vatNumber: String = "",
)
I want to filter an ArrayList<SiteObject> (call it allSites) by checking if any of the fields of the objects within the list match those in a specific <SiteObject> (call it currentSite).
So for example, I know how to filter looking at one field:
fun checkIfExistingSite(currentSite: SiteObject) : ArrayList<SiteObject> {
var matchingSites = ArrayList<SiteObject>()
allSites.value?.filter { site ->
site.siteReference.contains(currentSite.siteReference)}?.let { matchingSites.addAll(it)
}
return matchingSites
}
But I am looking for an elegant way to create a list where I compare the matching fields in each of the objects in allSites with the corresponding fields in currentSite..
This will give me a list of sites that may be the same (allowing for differences in the way user inputs data) which I can present to the user to check.
Use equals property of Data Class:
val matchingSites: List<SiteObject> = allSites
.filterNotNull()
.filter { it.equals(currentSite) }
If you are looking for a more loose equlity criteria than the full match of all fields values, I would suggest usage of reflection (note that this approach could have performance penalties):
val memberProperties = SiteObject::class.memberProperties
val minMatchingProperties = 9 //or whatever number that makes sense in you case
val matchingItems = allSites.filter {
memberProperties.atLeast(minMatchingProperties) { property -> property.get(it) == property.get(currentSite) }
}
fun <E> Iterable<E>.atLeast(n: Int, predicate: (E) -> Boolean): Boolean {
val size = count()
return when {
n == 1 -> this.any(predicate)
n == size -> this.all(predicate)
n > size - n + 1 -> this.atLeast(size - n + 1) { !predicate.invoke(it) }
else -> {
var count = 0
for (element in this) {
if (predicate.invoke(element)) count++
if (count >= n) return true
}
return false
}
}
}
you could specify all the fields by which you want to match the currentSite inside the filter predicate:
fun checkIfExistingSite(currentSite: SiteObject) =
allSites.filter {
it.siteAddress == currentSite.siteAddress
|| it.sitePhoneNumber == currentSite.sitePhoneNumber
|| it.siteReference == currentSite.siteReference
}
Long but fast solution because of short circuiting.
If the list is nullable you can transform it to a non nullable list like:
allSites?filter{...}.orEmpty()
// or imho better
allSites.orEmpty().filter{...}

Kotlin Exposed - selecting based on sub-query count

In my data model I have a very simple one-to-many relationship between challenges and it's whitelist items.
I am trying to select a challenge filtered by whitelist. Basically the challenge selection criteria is when the challenge is either does not have any entries in whitelist for itself or the whitelist matches by name.
This can be achieved with quite simple SQL query:
select c.* from challenge c, challenge_whitelist w where (c.id = w."challengeId" and w."userName" = 'testuser') or ((select count(*) where c.id = w."challengeId") = 0);
I am unable to translate it to Exposed though:
// will not compile
fun listAll(userName: String) {
ExposedChallenge.wrapRows(
ChallengeTable.innerJoin(ChallengeWhitelistTable)
.slice(ChallengeTable.columns)
.select((ChallengeWhitelistTable.userName eq userName) or (ChallengeTable.innerJoin(ChallengeWhitelistTable).selectAll().count() eq 0))
).toList()
}
The userName check works correctly but ChallengeTable.innerJoin(ChallengeWhitelistTable).selectAll().count() eq 0) is not qualified as the valid expression (will not compile).
Note that the mappings are super-simple:
object ChallengeTable : IntIdTable() {
val createdAt = datetime("createdAt")
}
class ExposedChallenge(id: EntityID<Int>) : IntEntity(id) {
companion object : IntEntityClass<ExposedChallenge>(ChallengeTable)
var createdAt by ChallengeTable.createdAt
val whitelist by ExposedChallengeWhitelist referrersOn ChallengeWhitelistTable.challenge
}
object ChallengeWhitelistTable : IntIdTable(name = "challenge_whitelist") {
var userName = varchar("userName", 50)
var challengeId = integer("challengeId")
val challenge = reference("challengeId", ChallengeTable).uniqueIndex()
}
class ExposedChallengeWhitelist(id: EntityID<Int>) : IntEntity(id) {
companion object : IntEntityClass<ExposedChallengeWhitelist>(ChallengeWhitelistTable)
val challengeId by ChallengeWhitelistTable.challengeId
val challenge by ExposedChallenge referencedOn ChallengeWhitelistTable.challenge
}
Any help would be appreciated.
Your SQL query is invalid as you use select count(*) without from part.
But it can be rewritten with Exposed DSL like:
ChallengeTable.leftJoin(ChallengeWhitelistTable).
slice(ChallengeTable.columns).
selectAll().
groupBy(ChallengeTable.id, ChallengeWhitelistTable.userName).having {
(ChallengeWhitelistTable.userName eq "testUser") or
(ChallengeWhitelistTable.id.count() eq 0)
}
Another way is to use just left join:
ChallengeTable.leftJoin(ChallengeWhitelistTable).
slice(ChallengeTable.columns).
select {
(ChallengeWhitelistTable.userName eq "testUser") or
(ChallengeWhitelistTable.id.isNull())
}

How to convert query result to case class?

I'm using Slick 2.0 and I have the following User case class:
case class User(id: Option[Long], email: String, password: String)
class Users(tag: Tag) extends Table[User](tag, "user") {
def id = column[Long]("id", O.PrimaryKey, O.AutoInc)
def email = column[String]("email")
def password = column[String]("password")
def * = (id.?, email, password) <> ((User.apply _).tupled, User.unapply _)
}
and I have the following function getting a user via their database id:
def findOneById(id: Long) = DB.withSession { implicit session =>
val result = (for {
u <- users if u.id === id
} yield (u.id, u.email, u.password)).first
User(Option(result._1), result._2, result._3)
}
Is there an easier way to convert the Seq response back result into the User case class?
Ok, I found my answer. I got it in part from #benji and in part from this post: Converting scala.slick.lifted.Query to a case class.
Instead of using a for comprehension, the following returns an Option[User], which is exactly what I need:
def findOneById(id: Long):Option[User] = DB.withSession { implicit session =>
users.filter(_.id === id).firstOption
}
For comprehension is easier in use with sql joins. Example:
def findOneById(id: Long) = DB.withSession { implicit session =>
val query = for {
u <- users if u.id === id
t <- token if t.user_id = u.id && token.isActive === `A`
} yield u
query.firstOption
}
To convert the result to list of User case class, you can simply do
result.map(User.tupled)
However, that I will only work, if your data model is consitent. For example: Your user case class has id as optional where as it is a primary key in the DB which is wrong.

Scalding: retaining all fields after groupBy

I'm doing a groupBy for calculating a value, but it seems that when I group by, I lose all the fields that are not in the aggregation keys:
filtered.filterNot('site) {s:String => ...}
.filterNot('date) {s:String => ...}
aggr = filtered.groupBy('id, 'contentHost) { group =>
group.min('timestamp -> 'min)
//how do I keep original fields? (eg: site, date)
}
aggr.store(Tsv(...)) //eg: field "site" won't be here
in pig, it would be something like this:
aggr = group filtered by concat('id, 'contentHost);
result = foreach aggr {
generate flatten(filtered), //how to do this in scalding?
min(filtered.timestamp) as min;
}
I had the same problem with the tuple API and could only solve it by using the typed API.
You can either use Scala tuples or define your own case class outside your job. E.g.:
case class Data(id: String, site: String, date: String, contentHost: String)
Then you'd process it like this:
val filtered: TypedPipe[Data] = TypedPipe.from(Seq(Data("...", "2014-04-14", "...", "...")))
filtered
.filterNot ( data => data.site == "fr" )
.filterNot ( data => data.date == "2014-02-01" )
.groupBy (data => (data.id, data.contentHost)) // (String,String) -> Data
.min // or .minBy { ... }
.toTypedPipe
.write(TypedTsv[((String, String), Data)]("/path/"))