I'm not entirely sure the best way to phrase this problem, but hopefully my description and the code will show what I mean.
I'm building an API that uses a SQL db. One of the record types, Order can contain a Key, but the key ID is null until the Order is finalized. I have a function that queries the DB for all orders, joined with the Key DB so that the key details are populated if the ID is not null.
How do I scan in Orders where some have keyId null and some do not?
Order struct:
type orderDb struct {
id int
certificate certificateDb
location string
status string
knownRevoked bool
err sql.NullString // stored as json object
expires sql.NullInt32
dnsIdentifiers commaJoinedStrings // will be a comma separated list from storage
authorizations commaJoinedStrings // will be a comma separated list from storage
finalize string
finalizedKey *keyDb
certificateUrl sql.NullString
pem sql.NullString
validFrom sql.NullInt32
validTo sql.NullInt32
createdAt int
updatedAt int
}
keyDb struct
type keyDb struct {
id int
name string
description string
algorithmValue string
pem string
apiKey string
apiKeyViaUrl bool
createdAt int
updatedAt int
}
Partial SQL query & scan
query := `
SELECT
ao.id, ao.status, ao.known_revoked, ao.error, ao.dns_identifiers, ao.valid_from,
ao.valid_to, ao.created_at, ao.updated_at,
pk.id, pk.name,
c.id, c.name, c.subject,
aa.id, aa.name, aa.is_staging
FROM
acme_orders ao
LEFT JOIN private_keys pk on (ao.finalized_key_id = pk.id)
LEFT JOIN certificates c on (ao.certificate_id = c.id)
LEFT JOIN acme_accounts aa on (c.acme_account_id = aa.id)
err = rows.Scan(
&oneOrder.id,
&oneOrder.status,
&oneOrder.knownRevoked,
&oneOrder.err,
&oneOrder.dnsIdentifiers,
&oneOrder.validFrom,
&oneOrder.validTo,
&oneOrder.createdAt,
&oneOrder.updatedAt,
&oneOrder.finalizedKey.id,
&oneOrder.finalizedKey.name,
Essentially sometimes key id and name are null because the key is null. How do I set finalizedKey to null when this is the case, but scan in the values when the key isn't null?
I don't really want to do a separate query for keys because the slice of Orders could have 20, 50, or 100+ records and I don't want to do 101 queries to return a slice of 100 Orders.
Related
I have a GORM query with a preload that works just fine because I'm binding it to a struct called "companies" which is also the name of the corresponding database table:
var companies []Company
db.Preload("Subsidiaries").Joins("LEFT JOIN company_prod ON company_products.company_id = companies.id").Where("company_products.product_id = ?", ID).Find(&companies)
Now I want to do something similar, but bind the result to a struct that does not have a name that refers to the "companies" table:
var companiesFull []CompanyFull
db.Preload("Subsidiaries").Joins("LEFT JOIN company_prod ON company_products.company_id = companies.id").Where("company_products.product_id = ?", ID).Find(&companies)
I've simplified the second call for better understanding, the real call has more JOINs and returns more data, so it can't be bound to the "companies" struct.
I'm getting an error though:
column company_subsidiaries.company_full_id does not exist
The corresponding SQL query:
SELECT * FROM "company_subsidiaries" WHERE "company_subsidiaries"."company_full_id" IN (2,1)
There is no "company_subsidiaries.company_full_id", the correct query should be:
SELECT * FROM "company_subsidiaries" WHERE "company_subsidiaries"."company_id" IN (2,1)
The condition obviously gets generated from the name of the struct the result is being bound to. Is there any way to specify a custom name for this case?
I'm aware of the Tabler interface technique, however it doesn't work for Preload I believe (tried it, it changes the table name of the main query, but not the preload).
Updated: More info about the DB schema and structs
DB schema
TABLE companies
ID Primary key
OTHER FIELDS
TABLE products
ID Primary key
OTHER FIELDS
TABLE subsidiaries
ID Primary key
OTHER FIELDS
TABLE company_products
ID Primary key
Company_id Foreign key (companies.id)
Product_id Foreign key (products.id)
TABLE company_subsidiaries
ID Primary key
Company_id Foreign key (companies.id)
Subsidiary_id Foreign key (subsidiaries.id)
Structs
type Company struct {
Products []*Product `json:"products" gorm:"many2many:company_products;"`
ID int `json:"ID,omitempty"`
}
type CompanyFull struct {
Products []*Product `json:"products" gorm:"many2many:company_products;"`
Subsidiaries []*Subsidiary `json:"subsidiaries" gorm:"many2many:company_products;"`
ID int `json:"ID,omitempty"`
}
type Product struct {
Name string `json:"name"`
ID int `json:"ID,omitempty"`
}
type Subsidiary struct {
Name string `json:"name"`
ID int `json:"ID,omitempty"`
}
Generated SQL (by GORM)
SELECT * FROM "company_subsidiaries" WHERE "company_subsidiaries"."company_full_id" IN (2,1)
SELECT * FROM "subsidiaries" WHERE "subsidiaries"."id" IN (NULL)
SELECT companies.*, company_products.*, FROM "companies" LEFT JOIN company_products ON company_products.company_id = companies.id WHERE company_products.product_id = 1
Seems like the way to go in this case may be to customize the relationship in your CompanyFull model. Using joinForeignKey the following code works.
type CompanyFull struct {
Products []*Product `json:"products" gorm:"many2many:company_products;joinForeignKey:ID"`
Subsidiaries []*Subsidiary `json:"subsidiaries" gorm:"many2many:company_subsidiaries;joinForeignKey:ID"`
ID int `json:"ID,omitempty"`
}
func (CompanyFull) TableName() string {
return "companies"
}
func main(){
...
result := db.Preload("Subsidiaries").Joins("LEFT JOIN company_products ON company_products.company_id = companies.id").Where("company_products.product_id = ?", ID).Find(&companies)
if result.Error != nil {
log.Println(result.Error)
} else {
log.Printf("%#v", companies)
}
For more info regarding customizing the foreign keys used in relationships, take a look at the docs https://gorm.io/docs/many_to_many.html#Override-Foreign-Key
I’m trying to return values from a join where the where clause might not have a matches.
Here’s my database schema
strings
-------
id: INT
name: VARCHAR
value: VARCHAR
fileId: INT FOREIGN KEY files(id)
languages
---------
id: INT
code: CHAR
name: VARCHAR
translations
------------
id: INT
string_id: INT, FOREIGN KEY strings(id)
language_id: INT, FOREIGN KEY languages(id)
translation: VARCHAR
I’m trying to select all the strings, and all the translations in a given language. The translations may or may not exist for a given language, but I want to return the strings any way.
I’m using a query similar to:
SELECT s.id, s.name, s.value, t.translation
FROM strings s LEFT OUTER JOIN translations t ON s.id = t.string_id
WHERE s.file_id = $1 AND t.language_id = $2
I want to return strings regardless of whether matches are found in the translations table. If translations don’t exist for a given language, that field would of course be null. I think the problem is with the WHERE clause having the t.language_id = ..., since language_id doesn't exist in this particular case. But not sure the best way to fix this.
Database Postgresql
Conditions on the second table need to go in the ON clause:
SELECT s.id, s.name, s.value, t.translation
FROM strings s LEFT OUTER JOIN
translations t
ON s.id = t.string_id AND t.language_id = $2
WHERE s.file_id = $1;
Otherwise, t.language_id is NULL and that fails the comparison in the WHERE clause.
Question
When dealing with a one-to-many or many-to-many SQL relationship in Golang, what is the best (efficient, recommended, "Go-like") way of mapping the rows to a struct?
Taking the example setup below I have tried to detail some approaches with Pros and Cons of each but was wondering what the community recommends.
Requirements
Works with PostgreSQL (can be generic but not include MySQL/Oracle specific features)
Efficiency - No brute forcing every combination
No ORM - Ideally using only database/sql and jmoiron/sqlx
Example
For sake of clarity I have removed error handling
Models
type Tag struct {
ID int
Name string
}
type Item struct {
ID int
Tags []Tag
}
Database
CREATE TABLE item (
id INT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY
);
CREATE TABLE tag (
id INT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
name VARCHAR(160),
item_id INT REFERENCES item(id)
);
Approach 1 - Select all Items, then select tags per item
var items []Item
sqlxdb.Select(&items, "SELECT * FROM item")
for i, item := range items {
var tags []Tag
sqlxdb.Select(&tags, "SELECT * FROM tag WHERE item_id = $1", item.ID)
items[i].Tags = tags
}
Pros
Simple
Easy to understand
Cons
Inefficient with the number of database queries increasing proportional with number of items
Approach 2 - Construct SQL join and loop through rows manually
var itemTags = make(map[int][]Tag)
var items = []Item{}
rows, _ := sqlxdb.Queryx("SELECT i.id, t.id, t.name FROM item AS i JOIN tag AS t ON t.item_id = i.id")
for rows.Next() {
var (
itemID int
tagID int
tagName string
)
rows.Scan(&itemID, &tagID, &tagName)
if tags, ok := itemTags[itemID]; ok {
itemTags[itemID] = append(tags, Tag{ID: tagID, Name: tagName,})
} else {
itemTags[itemID] = []Tag{Tag{ID: tagID, Name: tagName,}}
}
}
for itemID, tags := range itemTags {
items = append(Item{
ID: itemID,
Tags: tags,
})
}
Pros
A single database call and cursor that can be looped through without eating too much memory
Cons
Complicated and harder to develop with multiple joins and many attributes on the struct
Not too performant; more memory usage and processing time vs. more network calls
Failed approach 3 - sqlx struct scanning
Despite failing I want to include this approach as I find it to be my current aim of efficiency paired with development simplicity. My hope was by explicitly setting the db tag on each struct field sqlx could do some advanced struct scanning
var items []Item
sqlxdb.Select(&items, "SELECT i.id AS item_id, t.id AS tag_id, t.name AS tag_name FROM item AS i JOIN tag AS t ON t.item_id = i.id")
Unfortunately this errors out as missing destination name tag_id in *[]Item leading me to believe the StructScan is not advanced enough to recursively loop through rows (no criticism - it is a complicated scenario)
Possible approach 4 - PostgreSQL array aggregators and GROUP BY
While I am sure this will not work I have included this untested option to see if it could be improved upon so it may work.
var items = []Item{}
sqlxdb.Select(&items, "SELECT i.id as item_id, array_agg(t.*) as tags FROM item AS i JOIN tag AS t ON t.item_id = i.id GROUP BY i.id")
When I have some time I will try and run some experiments here.
the sql in postgres :
create schema temp;
set search_path = temp;
create table item
(
id INT generated by default as identity primary key
);
create table tag
(
id INT generated by default as identity primary key,
name VARCHAR(160),
item_id INT references item (id)
);
create view item_tags as
select id,
(
select
array_to_json(array_agg(row_to_json(taglist.*))) as array_to_json
from (
select tag.name, tag.id
from tag
where item_id = item.id
) taglist ) as tags
from item ;
-- golang query this maybe
select row_to_json(row)
from (
select * from item_tags
) row;
then golang query this sql:
select row_to_json(row)
from (
select * from item_tags
) row;
and unmarshall to go struct:
pro:
postgres manage the relation of data. add / update data with sql functions.
golang manage business model and logic.
it's easy way.
.
I can suggest another approach which I have used before.
You make a json of the tags in this case in the query and return it.
Pros: You have 1 call to the db, which aggregates the data, and all you have to do is parse the json into an array.
Cons: It's a bit ugly. Feel free to bash me for it.
type jointItem struct {
Item
ParsedTags string
Tags []Tag `gorm:"-"`
}
var jointItems []*jointItem
db.Raw(`SELECT
items.*,
(SELECT CONCAT(
'[',
GROUP_CONCAT(
JSON_OBJECT('id', id,
'name', name
)
),
']'
)) as parsed_tags
FROM items`).Scan(&jointItems)
for _, o := range jointItems {
var tempTags []Tag
if err := json.Unmarshall(o.ParsedTags, &tempTags) ; err != nil {
// do something
}
o.Tags = tempTags
}
Edit: code might behave weirdly so I find it better to use a temporary tags array when moving instead of using the same struct.
You can use carta.Map() from https://github.com/jackskj/carta
It tracks has-many relationships automatically.
I have four tables.
DocumentList:
DocumentID int
DocumentDescription varchar(100)
DocumentName varchar(100)
DocumentTypeCode int
Archive ud_DefaultBitFalse:bit
DocumentStepLevel:
DocumentStepID int
DocumentID int
StepLevelCode int
DocumentAttachment:
DocumentAttachmentGenID int
DocumentStepID int
AttachmentGenID int
FacilityGenID int
Submitted ud_DefaultBitFalse:bit
Attachment:
AttachmentGenId int
FileName varchar(255)
FileDescription varchar(255)
UploadDate ud_DefaultDate:datetime
DocumentData varbinary(MAX)
MimeType varchar(30)
Archive ud_DefaultBitFalse:bit
UpdateBy int
UpdateDate ud_DefaultDate:datetime
Documentlist table contains a list of documents.
DocumentStepLevel is a table that associate documents in DocumentList with a step level. We have six steps right now and each step have some documents associated with it.
DocumentAttachment table is junction/relationship table that create relationship between DocumentStepLevel and Attachment table.
Attachment table has the actual files data uploaded to the system
Question:
I need to write a query that will fetch the following columns.
DocumentList.[DocumentDescription]
DocumentList.[DocumentName]
DocumentStepLevel.[DocumentStepID]
DocumentStepLevel.[StepLevelCode]
DocumentAttachment.[DocumentAttachmentGenID]
DocumentAttachment.[FacilityGenID]
DocumentAttachment.[Submitted]
Attachment.[FileName]
Attachment.[FileDescription]
Attachment.[UploadDate]
Query should return data from DocumentList table for specific step level. When DocumentAttachment.[Submitted] column is set to true it should also return the data from DocumentAttachment and Attachment tables as well. Otherwise those columns will return nothing.
I tried using left outer join but problem happen when I add Submitted column to query. When I add that column to query it stop returning any data until that flag is set to true.
SELECT *
FROM documentStepLevel dsl
JOIN documentList dl
ON dl.documentId = dsl.documentId
LEFT JOIN
documentAttachment da
ON da.documentStepID = dsl.documentStepId
AND submitted = 1
LEFT JOIN
attachment a
ON a.attachmentGenId = da.attachmentGenId
WHERE dsl.stepLevelCode = #stepLevelCode
Is it DocumentAttachment you're left outer joining?
Difficult to say for sure without seeing your current query, but I'm guessing you've outer joined to DocumentAttachment and then have something like "where documentattachment.submitted = 1"?
In this case I believe it won't return anything as for rows where documentattachment doesn't exist, submitted is effectively null. So you might need to change your where statement to "where (documentattachment.submitted = 1 or documentattachment.submitted is null)"
This also assumes that when DocumentAttachment is populated, submitted by default has a 0 value rather than a null value (otherwise you'll need a different method of ascertaining the absence of a DocumentAttachment)
I have some tables:
Sessions
SessionID int PK
Created datetime
SiteId int FK
Tracking_Parameters
ParamID int PK
ParamName nvarchar
Session_Custom_Tracking
SessionID int FK
ParamID int FK
ParamValue nvarchar
Site_Custom_Parameters
SiteID int FK
ParamID int FK
ParamKey nvarchar
Sessions: Contains the unique session id for a visitor and the time they entered the site.
Tracking_Parameters: Contains a list of things that you may want to track on a site (i.e. Email Open, Email Click, Article Viewed, etc.)
Site_Custom_Parameters: For a particular site (table not shown), declares the key value for a Tracking_Parameter (i.e. the key to look for in a query string or route)
Session_Custom_Tracking: The link between a session and a tracking parameter and also contains the value for the parameter's key when it was found by my application.
Question:
I want to select session id's where for these particular sessions, there is a record in the Session_Custom_Tracking for two different ParamID's. I want to find sessions where a user both opened an email (paramid 1) and clicked (paramid 3) a link in that email.
You can join to the same table twice:
SELECT S.SessionID
FROM Sessions AS S
JOIN Session_Custom_Tracking AS SCT1
ON SCT1.SessionID = S.SessionID
AND SCT1.ParamID = 1
JOIN Session_Custom_Tracking AS SCT2
ON SCT2.SessionID = S.SessionID
AND SCT2.ParamID = 3
An alteranative that might be easier to read (because it more closely matches the way you describe the problem) is to use WHERE EXISTS:
SELECT S.SessionID
FROM Sessions AS S
WHERE EXISTS
(
SELECT *
FROM Session_Custom_Tracking AS SCT1
WHERE SCT1.SessionID = S.SessionID
AND SCT1.ParamID = 1
)
AND EXISTS
(
SELECT *
FROM Session_Custom_Tracking AS SCT2
WHERE SCT2.SessionID = S.SessionID
AND SCT2.ParamID = 3
)