Question
When dealing with a one-to-many or many-to-many SQL relationship in Golang, what is the best (efficient, recommended, "Go-like") way of mapping the rows to a struct?
Taking the example setup below I have tried to detail some approaches with Pros and Cons of each but was wondering what the community recommends.
Requirements
Works with PostgreSQL (can be generic but not include MySQL/Oracle specific features)
Efficiency - No brute forcing every combination
No ORM - Ideally using only database/sql and jmoiron/sqlx
Example
For sake of clarity I have removed error handling
Models
type Tag struct {
ID int
Name string
}
type Item struct {
ID int
Tags []Tag
}
Database
CREATE TABLE item (
id INT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY
);
CREATE TABLE tag (
id INT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
name VARCHAR(160),
item_id INT REFERENCES item(id)
);
Approach 1 - Select all Items, then select tags per item
var items []Item
sqlxdb.Select(&items, "SELECT * FROM item")
for i, item := range items {
var tags []Tag
sqlxdb.Select(&tags, "SELECT * FROM tag WHERE item_id = $1", item.ID)
items[i].Tags = tags
}
Pros
Simple
Easy to understand
Cons
Inefficient with the number of database queries increasing proportional with number of items
Approach 2 - Construct SQL join and loop through rows manually
var itemTags = make(map[int][]Tag)
var items = []Item{}
rows, _ := sqlxdb.Queryx("SELECT i.id, t.id, t.name FROM item AS i JOIN tag AS t ON t.item_id = i.id")
for rows.Next() {
var (
itemID int
tagID int
tagName string
)
rows.Scan(&itemID, &tagID, &tagName)
if tags, ok := itemTags[itemID]; ok {
itemTags[itemID] = append(tags, Tag{ID: tagID, Name: tagName,})
} else {
itemTags[itemID] = []Tag{Tag{ID: tagID, Name: tagName,}}
}
}
for itemID, tags := range itemTags {
items = append(Item{
ID: itemID,
Tags: tags,
})
}
Pros
A single database call and cursor that can be looped through without eating too much memory
Cons
Complicated and harder to develop with multiple joins and many attributes on the struct
Not too performant; more memory usage and processing time vs. more network calls
Failed approach 3 - sqlx struct scanning
Despite failing I want to include this approach as I find it to be my current aim of efficiency paired with development simplicity. My hope was by explicitly setting the db tag on each struct field sqlx could do some advanced struct scanning
var items []Item
sqlxdb.Select(&items, "SELECT i.id AS item_id, t.id AS tag_id, t.name AS tag_name FROM item AS i JOIN tag AS t ON t.item_id = i.id")
Unfortunately this errors out as missing destination name tag_id in *[]Item leading me to believe the StructScan is not advanced enough to recursively loop through rows (no criticism - it is a complicated scenario)
Possible approach 4 - PostgreSQL array aggregators and GROUP BY
While I am sure this will not work I have included this untested option to see if it could be improved upon so it may work.
var items = []Item{}
sqlxdb.Select(&items, "SELECT i.id as item_id, array_agg(t.*) as tags FROM item AS i JOIN tag AS t ON t.item_id = i.id GROUP BY i.id")
When I have some time I will try and run some experiments here.
the sql in postgres :
create schema temp;
set search_path = temp;
create table item
(
id INT generated by default as identity primary key
);
create table tag
(
id INT generated by default as identity primary key,
name VARCHAR(160),
item_id INT references item (id)
);
create view item_tags as
select id,
(
select
array_to_json(array_agg(row_to_json(taglist.*))) as array_to_json
from (
select tag.name, tag.id
from tag
where item_id = item.id
) taglist ) as tags
from item ;
-- golang query this maybe
select row_to_json(row)
from (
select * from item_tags
) row;
then golang query this sql:
select row_to_json(row)
from (
select * from item_tags
) row;
and unmarshall to go struct:
pro:
postgres manage the relation of data. add / update data with sql functions.
golang manage business model and logic.
it's easy way.
.
I can suggest another approach which I have used before.
You make a json of the tags in this case in the query and return it.
Pros: You have 1 call to the db, which aggregates the data, and all you have to do is parse the json into an array.
Cons: It's a bit ugly. Feel free to bash me for it.
type jointItem struct {
Item
ParsedTags string
Tags []Tag `gorm:"-"`
}
var jointItems []*jointItem
db.Raw(`SELECT
items.*,
(SELECT CONCAT(
'[',
GROUP_CONCAT(
JSON_OBJECT('id', id,
'name', name
)
),
']'
)) as parsed_tags
FROM items`).Scan(&jointItems)
for _, o := range jointItems {
var tempTags []Tag
if err := json.Unmarshall(o.ParsedTags, &tempTags) ; err != nil {
// do something
}
o.Tags = tempTags
}
Edit: code might behave weirdly so I find it better to use a temporary tags array when moving instead of using the same struct.
You can use carta.Map() from https://github.com/jackskj/carta
It tracks has-many relationships automatically.
I have two tables employees and posting.
employees with fields id,name
AND
postings with fields id,employee_id,status
Now to retrieve all the employees(with posting details) whose Posting.status = 1, I wrote in PostingsController.php's view() action:
$cond = array('Posting.status'=>1);
$e = $this->Employee->find('all',array('conditions' => $cond));
$this->set('emp',$e);
In Employee.php model var $hasMany = 'Posting';
In Posting.php model var $belongsTo = 'Employee';
But the above code generates an error like :
Error: SQLSTATE[42S22]: Column not found: 1054 Unknown column
'Posting.status' in 'where clause'
Generated SQL:
SQL Query: SELECT `Employee`.`id`, `Employee`.`name` FROM `cakelab`.`employees` AS `Employee` WHERE `Posting`.`status` = 1
I am new to CakePhp joins, please guide me.
Cake doesn't join the tables when the relation type is hasMany. Rather it performs a query on the first table (employees) and with the id of the employee it performs a second query on postings table to retrieve all the postings related to that employee
you can manually join the tables but I suggest querying on postings table and then group by Employee
$this->Posting ->find(
'all',
array(
'fields' => array('Employee.id', 'Employee.name', 'Posting.status' ),
'conditions' => $cond,
'group' => array('Employee.id')
)
);
I have two tables that represent say a Mixture and the components of the Mixture. The table layouts are like:
Mixture table:
MixtureID uniqueidentifier
MixtureDescription varchar(50)
Components table:
ComponentID uniqueidentifier
MixtureID uniqueidentifier (FK to previous table)
ComponentName varchar(50)
ComponentRatioPercentage int
Now, what I want to do is take a list of component names inputted from the user and find the ID's of any mixes that contain all of those components.
In SQL I could so something like:
select distinct MixtureID
from Mixture Mixture
where exists (select ComponentName
from Components Components1
where Components1.MixtureID = Mixture.MixtureID and
Components1.ComponentDescription = 'SomeComponentName')
and exists (select ComponentName
from Components2
where Components2.MixtureID = Mixture.MixtureID and
Components2.ComponentDescription = 'SomeOtherComponentName')
and exists....
etc, adding a sub-select for each component.
How would I do something like this in linq to sql? The number of components to look for would not be known in advance, until the user is done with input, though there would be a max of 10. Thanks in advance!
var components = new string[] {"SomeComponentName", "SomeOtherComponentName"};
var query = Mixtures.AsQueryable();
foreach (var component in components)
{
var tmpComponent = component;
query = query.Where(m => m.Components
.Any(c => t.ComponentDescription == tmpComponent)
);
}
var mixturesIds = query.Select(m=>m.MixtureId).Distinct();
I have a MySQL table that collects player data from various game servers (Urban Terror). The bot that collects the data runs 24/7, and currently the table is up to about 475,000+ records. Because of this, querying this table from PHP has become quite slow. I wonder what I can do on the database side of things to make it as optomized as possible, then I can focus on the application to query the database. The table is as follows:
CREATE TABLE IF NOT EXISTS `people` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`name` varchar(40) NOT NULL,
`ip` int(4) unsigned NOT NULL,
`guid` varchar(32) NOT NULL,
`server` int(4) unsigned NOT NULL,
`date` int(11) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `Person` (`name`,`ip`,`guid`),
KEY `server` (`server`),
KEY `date` (`date`),
KEY `PlayerName` (`name`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 COMMENT='People that Play on Servers' AUTO_INCREMENT=475843 ;
I'm storying the IPv4 (ip and server) as 4 byte integers, and using the MySQL functions NTOA(), etc to encode and decode, I heard that this way is faster, rather than varchar(15).
The guid is a md5sum, 32 char hex. Date is stored as unix timestamp.
I have a unique key on name, ip and guid, as to avoid duplicates of the same player.
Do I have my keys setup right? Is the way I'm storing data efficient?
Here is the code to query this table. You search for a name, ip, or guid, and it grabs the results of the query and cross references other records that match the name, ip, or guid from the results of the first query, and does it for each field. This is kind of hard to explain. But basically, if I search for one player by name, I'll see every other name he has used, every IP he has used and every GUID he has used.
<form action="<?php echo $_SERVER['PHP_SELF']; ?>" method="post">
Search: <input type="text" name="query" id="query" /><input type="submit" name="btnSubmit" value="Submit" />
</form>
<?php if (!empty($_POST['query'])) { ?>
<table cellspacing="1" id="1up_people" class="tablesorter" width="300">
<thead>
<tr>
<th>ID</th>
<th>Player Name</th>
<th>Player IP</th>
<th>Player GUID</th>
<th>Server</th>
<th>Date</th>
</tr>
</thead>
<tbody>
<?php
function super_unique($array)
{
$result = array_map("unserialize", array_unique(array_map("serialize", $array)));
foreach ($result as $key => $value)
{
if ( is_array($value) )
{
$result[$key] = super_unique($value);
}
}
return $result;
}
if (!empty($_POST['query'])) {
$query = trim($_POST['query']);
$count = 0;
$people = array();
$link = mysql_connect('localhost', 'mysqluser', 'yea right!');
if (!$link) {
die('Could not connect: ' . mysql_error());
}
mysql_select_db("1up");
$sql = "SELECT id, name, INET_NTOA(ip) AS ip, guid, INET_NTOA(server) AS server, date FROM 1up_people WHERE (name LIKE \"%$query%\" OR INET_NTOA(ip) LIKE \"%$query%\" OR guid LIKE \"%$query%\")";
$result = mysql_query($sql, $link);
if (!$result) {
die(mysql_error());
}
// Now take the initial results and parse each column into its own array
while ($row = mysql_fetch_array($result, MYSQL_NUM)) {
$name = htmlspecialchars($row[1]);
$people[] = array(
'id' => $row[0],
'name' => $name,
'ip' => $row[2],
'guid' => $row[3],
'server' => $row[4],
'date' => $row[5]
);
}
// now for each name, ip, guid in results, find additonal records
$people2 = array();
foreach ($people AS $person) {
$ip = $person['ip'];
$sql = "SELECT id, name, INET_NTOA(ip) AS ip, guid, INET_NTOA(server) AS server, date FROM 1up_people WHERE (ip = \"$ip\")";
$result = mysql_query($sql, $link);
while ($row = mysql_fetch_array($result, MYSQL_NUM)) {
$name = htmlspecialchars($row[1]);
$people2[] = array(
'id' => $row[0],
'name' => $name,
'ip' => $row[2],
'guid' => $row[3],
'server' => $row[4],
'date' => $row[5]
);
}
}
$people3 = array();
foreach ($people AS $person) {
$guid = $person['guid'];
$sql = "SELECT id, name, INET_NTOA(ip) AS ip, guid, INET_NTOA(server) AS server, date FROM 1up_people WHERE (guid = \"$guid\")";
$result = mysql_query($sql, $link);
while ($row = mysql_fetch_array($result, MYSQL_NUM)) {
$name = htmlspecialchars($row[1]);
$people3[] = array(
'id' => $row[0],
'name' => $name,
'ip' => $row[2],
'guid' => $row[3],
'server' => $row[4],
'date' => $row[5]
);
}
}
$people4 = array();
foreach ($people AS $person) {
$name = $person['name'];
$sql = "SELECT id, name, INET_NTOA(ip) AS ip, guid, INET_NTOA(server) AS server, date FROM 1up_people WHERE (name = \"$name\")";
$result = mysql_query($sql, $link);
while ($row = mysql_fetch_array($result, MYSQL_NUM)) {
$name = htmlspecialchars($row[1]);
$people4[] = array(
'id' => $row[0],
'name' => $name,
'ip' => $row[2],
'guid' => $row[3],
'server' => $row[4],
'date' => $row[5]
);
}
}
// Combine people and people2 into just people
$people = array_merge($people, $people2);
$people = array_merge($people, $people3);
$people = array_merge($people, $people4);
$people = super_unique($people);
foreach ($people AS $person) {
$date = ($person['date']) ? date("M d, Y", $person['date']) : 'Before 8/1/10';
echo "<tr>\n";
echo "<td>".$person['id']."</td>";
echo "<td>".$person['name']."</td>";
echo "<td>".$person['ip']."</td>";
echo "<td>".$person['guid']."</td>";
echo "<td>".$person['server']."</td>";
echo "<td>".$date."</td>";
echo "</tr>\n";
$count++;
}
// Find Total Records
//$result = mysql_query("SELECT id FROM 1up_people", $link);
//$total = mysql_num_rows($result);
mysql_close($link);
}
?>
</tbody>
</table>
<p>
<?php
echo $count." Records Found for \"".$_POST['query']."\" out of $total";
?>
</p>
<?php
}
$time_stop = microtime(true);
print("Done (ran for ".round($time_stop-$time_start)." seconds).");
?>
Any help at all is appreciated!
Thank you.
SELECT id,
name,
Inet_ntoa(ip) AS ip,
guid,
Inet_ntoa(server) AS server,
DATE
FROM 1up_people
WHERE ( name LIKE "%$query%"
OR Inet_ntoa(ip) LIKE "%$query%"
OR guid LIKE "%$query%" )
Some issues with the above query:
The query uses 3 fields in the where clauses and OR's the condition on each of the field. MySQL can use only one index for a query. So it has to select index on either name or ip or guid for this query. Even if there is a compound index (name,ip,guid) it cannot be used in this scenario as the conditions are OR-ed. A better way to do such queries is to use UNION. Eg.
SELECT <fields> FROM table1 WHERE field1='val1' /*will use index on field1*/
UNION
SELECT <fields> FROM table1 WHERE field2='val2' /*will use index on field2*/
...
SELECT <fields> FROM table1 WHERE fieldn='valn' /*will use index on fieldn*/.
In the above query you do a select on each field separately and then UNION it. This allows the indexes on each of those fields to be used making the query efficient. It has a downside of getting duplicate results if the same row matches on more than one condition. To avoid that you can use UNION DISTINCT instead of UNION, but will be more expensive as mysql has to de-dedupe the output. For this suggestion to work the issues discussed below also needs to be addressed. (There is not index on guid and it needs to be build).
The conditions use LIKE '%query%' for name and guid i.e wildcard(%) at the beginning. This means the index cannot be used even if it exists. Index can be used when you use = or % in the end of the string as "query%". When % is used in the start of the string index will not be used. (Ref: http://dev.mysql.com/doc/refman/5.1/en/mysql-indexes.html). A possible way out is to use only wildcard in the end or use full-text indexing on these fields.
The condition on ip is as INET_NTOA(ip) LIKE "%query%". When a function is used on the field any index on that field cannot be used. MySQL does not support functional index as of now. If such a query needs to be supported you may have to store this field also as a varchar and treat it similar to name and guid.
Because of the above issues the query will always do a full table scan and will not use any index. Using UNION (as suggested in 1) will not provide any improvement 2 and 3 are not fixed, and in fact it may hurt the performance as it may be doing 3 table scans instead of 1. You can try creating a full-text index on (name,guid,ip_string) and do your query as MATCH(name, guid, ip_string) AGAINST ("$query")
From looking at the code I see that after getting the results from the above query, subsequent queries are fired based on the results of this query. I am not sure if that is required as I think it will not find any new records. When you search for f LIKE "%q%" and use the results do searches like f='r1', the LIKE condition should have already captured all occurences of 'r1' and the subsequent queries will only be returning the duplicate results. In my opinion the additional queries can be skipped, but may be I am missing something.
On a side note do not interpolate the query strings in the SQL statement as name LIKE "%$query%". This is not secure and can be used for SQL injection attack. Use prepared statements with binded variables.
Since your table is MyISAM, create FULLTEXT indexes which will perform better then LIKE '%%'
to avoid all the queries in the loop, insert the main query into a temporary table, which you will use later to query related records:
Example
Instead of the primary SELECT, insert the rows first:
CREATE TEMPORARY TABLE IF NOT EXISTS `tmp_people` (
`id` bigint(20) unsigned NOT NULL,
`name` varchar(40) NOT NULL,
`ip` int(4) unsigned NOT NULL,
`guid` varchar(32) NOT NULL,
`server` int(4) unsigned NOT NULL,
`date` int(11) NOT NULL,
PRIMARY KEY (`id`),
KEY `server` (`server`),
KEY `date` (`date`),
KEY `PlayerName` (`name`)
);
TRUNCATE TABLE tmp_people;
INSERT tmp_people
SELECT id, name, ip AS ip, guid, server AS server, date
FROM up_people
WHERE (name LIKE \"%$query%\" OR INET_NTOA(ip) LIKE \"%$query%\" OR guid LIKE \"%$query%\")
Then, query the results:
SELECT id, name, INET_NTOA(ip) AS ip, guid, INET_NTOA(server) AS server, date FROM tmp_people;
Finally, instead of looping over individual records, query all related records in the same select:
To get the related by ip:
SELECT up.id, up.name, INET_NTOA(up.ip) AS ip, up.guid, INET_NTOA(up.server) AS server, up.date FROM up_people up JOIN tmp_people tmp ON up.ip = tmp.ip
to get the related by guid:
SELECT up.id, up.name, INET_NTOA(up.ip) AS ip, up.guid, INET_NTOA(up.server) AS server, up.date FROM up_people up JOIN tmp_people tmp ON up.guid = tmp.guid;
to get the related by name:
SELECT up.id, up.name, INET_NTOA(up.ip) AS ip, up.guid, INET_NTOA(up.server) AS server, up.date FROM up_people up JOIN tmp_people tmp ON up.name = tmp.name
Side notes:
you do not need the PlayerName Index, since the name field is the left most field in the Person Index
There is no index on the guid field, so the query that finds related by guid will be slow.
Going back to the original structure, I would get rid of the composite index on (name, ip, guid) and create a non-unique index on name, and another non-unique index on ip.
I am not sure what to do about the guid. If you want to prevent duplicate player records, and neither the name alone, nor the name-with-ip is sufficient to guarantee uniqueness, perhaps appending an autoincrementing-integer-converted-to-string rather than a guid would be better.
As others have noted, "contains substring" i.e %foo% searches cannot take full advantage of an index; since the substring could occur in any/every indexed value, the entire index would have to be scanned. On the other hand, "starts-with" substring searches i.e. foo% are able to take advantage of an index.