Help Optimizing MySQL Table (~ 500,000 records) and PHP Code - sql

Question

I have a MySQL table that collects player data from various game servers (Urban Terror). The bot that collects the data runs 24/7, and currently the table is up to about 475,000+ records. Because of this, querying this table from PHP has become quite slow. I wonder what I can do on the database side of things to make it as optomized as possible, then I can focus on the application to query the database. The table is as follows:
CREATE TABLE IF NOT EXISTS `people` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`name` varchar(40) NOT NULL,
`ip` int(4) unsigned NOT NULL,
`guid` varchar(32) NOT NULL,
`server` int(4) unsigned NOT NULL,
`date` int(11) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `Person` (`name`,`ip`,`guid`),
KEY `server` (`server`),
KEY `date` (`date`),
KEY `PlayerName` (`name`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 COMMENT='People that Play on Servers' AUTO_INCREMENT=475843 ;
I'm storying the IPv4 (ip and server) as 4 byte integers, and using the MySQL functions NTOA(), etc to encode and decode, I heard that this way is faster, rather than varchar(15).
The guid is a md5sum, 32 char hex. Date is stored as unix timestamp.
I have a unique key on name, ip and guid, as to avoid duplicates of the same player.
Do I have my keys setup right? Is the way I'm storing data efficient?
Here is the code to query this table. You search for a name, ip, or guid, and it grabs the results of the query and cross references other records that match the name, ip, or guid from the results of the first query, and does it for each field. This is kind of hard to explain. But basically, if I search for one player by name, I'll see every other name he has used, every IP he has used and every GUID he has used.
<form action="<?php echo $_SERVER['PHP_SELF']; ?>" method="post">
Search: <input type="text" name="query" id="query" /><input type="submit" name="btnSubmit" value="Submit" />
</form>
<?php if (!empty($_POST['query'])) { ?>
<table cellspacing="1" id="1up_people" class="tablesorter" width="300">
<thead>
<tr>
<th>ID</th>
<th>Player Name</th>
<th>Player IP</th>
<th>Player GUID</th>
<th>Server</th>
<th>Date</th>
</tr>
</thead>
<tbody>
<?php
function super_unique($array)
{
$result = array_map("unserialize", array_unique(array_map("serialize", $array)));
foreach ($result as $key => $value)
{
if ( is_array($value) )
{
$result[$key] = super_unique($value);
}
}
return $result;
}
if (!empty($_POST['query'])) {
$query = trim($_POST['query']);
$count = 0;
$people = array();
$link = mysql_connect('localhost', 'mysqluser', 'yea right!');
if (!$link) {
die('Could not connect: ' . mysql_error());
}
mysql_select_db("1up");
$sql = "SELECT id, name, INET_NTOA(ip) AS ip, guid, INET_NTOA(server) AS server, date FROM 1up_people WHERE (name LIKE \"%$query%\" OR INET_NTOA(ip) LIKE \"%$query%\" OR guid LIKE \"%$query%\")";
$result = mysql_query($sql, $link);
if (!$result) {
die(mysql_error());
}
// Now take the initial results and parse each column into its own array
while ($row = mysql_fetch_array($result, MYSQL_NUM)) {
$name = htmlspecialchars($row[1]);
$people[] = array(
'id' => $row[0],
'name' => $name,
'ip' => $row[2],
'guid' => $row[3],
'server' => $row[4],
'date' => $row[5]
);
}
// now for each name, ip, guid in results, find additonal records
$people2 = array();
foreach ($people AS $person) {
$ip = $person['ip'];
$sql = "SELECT id, name, INET_NTOA(ip) AS ip, guid, INET_NTOA(server) AS server, date FROM 1up_people WHERE (ip = \"$ip\")";
$result = mysql_query($sql, $link);
while ($row = mysql_fetch_array($result, MYSQL_NUM)) {
$name = htmlspecialchars($row[1]);
$people2[] = array(
'id' => $row[0],
'name' => $name,
'ip' => $row[2],
'guid' => $row[3],
'server' => $row[4],
'date' => $row[5]
);
}
}
$people3 = array();
foreach ($people AS $person) {
$guid = $person['guid'];
$sql = "SELECT id, name, INET_NTOA(ip) AS ip, guid, INET_NTOA(server) AS server, date FROM 1up_people WHERE (guid = \"$guid\")";
$result = mysql_query($sql, $link);
while ($row = mysql_fetch_array($result, MYSQL_NUM)) {
$name = htmlspecialchars($row[1]);
$people3[] = array(
'id' => $row[0],
'name' => $name,
'ip' => $row[2],
'guid' => $row[3],
'server' => $row[4],
'date' => $row[5]
);
}
}
$people4 = array();
foreach ($people AS $person) {
$name = $person['name'];
$sql = "SELECT id, name, INET_NTOA(ip) AS ip, guid, INET_NTOA(server) AS server, date FROM 1up_people WHERE (name = \"$name\")";
$result = mysql_query($sql, $link);
while ($row = mysql_fetch_array($result, MYSQL_NUM)) {
$name = htmlspecialchars($row[1]);
$people4[] = array(
'id' => $row[0],
'name' => $name,
'ip' => $row[2],
'guid' => $row[3],
'server' => $row[4],
'date' => $row[5]
);
}
}
// Combine people and people2 into just people
$people = array_merge($people, $people2);
$people = array_merge($people, $people3);
$people = array_merge($people, $people4);
$people = super_unique($people);
foreach ($people AS $person) {
$date = ($person['date']) ? date("M d, Y", $person['date']) : 'Before 8/1/10';
echo "<tr>\n";
echo "<td>".$person['id']."</td>";
echo "<td>".$person['name']."</td>";
echo "<td>".$person['ip']."</td>";
echo "<td>".$person['guid']."</td>";
echo "<td>".$person['server']."</td>";
echo "<td>".$date."</td>";
echo "</tr>\n";
$count++;
}
// Find Total Records
//$result = mysql_query("SELECT id FROM 1up_people", $link);
//$total = mysql_num_rows($result);
mysql_close($link);
}
?>
</tbody>
</table>
<p>
<?php
echo $count." Records Found for \"".$_POST['query']."\" out of $total";
?>
</p>
<?php
}
$time_stop = microtime(true);
print("Done (ran for ".round($time_stop-$time_start)." seconds).");
?>
Any help at all is appreciated!
Thank you.

Answer 1

SELECT id,
name,
Inet_ntoa(ip) AS ip,
guid,
Inet_ntoa(server) AS server,
DATE
FROM 1up_people
WHERE ( name LIKE "%$query%"
OR Inet_ntoa(ip) LIKE "%$query%"
OR guid LIKE "%$query%" )
Some issues with the above query:
The query uses 3 fields in the where clauses and OR's the condition on each of the field. MySQL can use only one index for a query. So it has to select index on either name or ip or guid for this query. Even if there is a compound index (name,ip,guid) it cannot be used in this scenario as the conditions are OR-ed. A better way to do such queries is to use UNION. Eg.
SELECT <fields> FROM table1 WHERE field1='val1' /*will use index on field1*/
UNION
SELECT <fields> FROM table1 WHERE field2='val2' /*will use index on field2*/
...
SELECT <fields> FROM table1 WHERE fieldn='valn' /*will use index on fieldn*/.
In the above query you do a select on each field separately and then UNION it. This allows the indexes on each of those fields to be used making the query efficient. It has a downside of getting duplicate results if the same row matches on more than one condition. To avoid that you can use UNION DISTINCT instead of UNION, but will be more expensive as mysql has to de-dedupe the output. For this suggestion to work the issues discussed below also needs to be addressed. (There is not index on guid and it needs to be build).
The conditions use LIKE '%query%' for name and guid i.e wildcard(%) at the beginning. This means the index cannot be used even if it exists. Index can be used when you use = or % in the end of the string as "query%". When % is used in the start of the string index will not be used. (Ref: http://dev.mysql.com/doc/refman/5.1/en/mysql-indexes.html). A possible way out is to use only wildcard in the end or use full-text indexing on these fields.
The condition on ip is as INET_NTOA(ip) LIKE "%query%". When a function is used on the field any index on that field cannot be used. MySQL does not support functional index as of now. If such a query needs to be supported you may have to store this field also as a varchar and treat it similar to name and guid.
Because of the above issues the query will always do a full table scan and will not use any index. Using UNION (as suggested in 1) will not provide any improvement 2 and 3 are not fixed, and in fact it may hurt the performance as it may be doing 3 table scans instead of 1. You can try creating a full-text index on (name,guid,ip_string) and do your query as MATCH(name, guid, ip_string) AGAINST ("$query")
From looking at the code I see that after getting the results from the above query, subsequent queries are fired based on the results of this query. I am not sure if that is required as I think it will not find any new records. When you search for f LIKE "%q%" and use the results do searches like f='r1', the LIKE condition should have already captured all occurences of 'r1' and the subsequent queries will only be returning the duplicate results. In my opinion the additional queries can be skipped, but may be I am missing something.
On a side note do not interpolate the query strings in the SQL statement as name LIKE "%$query%". This is not secure and can be used for SQL injection attack. Use prepared statements with binded variables.

Answer 2

Since your table is MyISAM, create FULLTEXT indexes which will perform better then LIKE '%%'
to avoid all the queries in the loop, insert the main query into a temporary table, which you will use later to query related records:
Example
Instead of the primary SELECT, insert the rows first:
CREATE TEMPORARY TABLE IF NOT EXISTS `tmp_people` (
`id` bigint(20) unsigned NOT NULL,
`name` varchar(40) NOT NULL,
`ip` int(4) unsigned NOT NULL,
`guid` varchar(32) NOT NULL,
`server` int(4) unsigned NOT NULL,
`date` int(11) NOT NULL,
PRIMARY KEY (`id`),
KEY `server` (`server`),
KEY `date` (`date`),
KEY `PlayerName` (`name`)
);
TRUNCATE TABLE tmp_people;
INSERT tmp_people
SELECT id, name, ip AS ip, guid, server AS server, date
FROM up_people
WHERE (name LIKE \"%$query%\" OR INET_NTOA(ip) LIKE \"%$query%\" OR guid LIKE \"%$query%\")
Then, query the results:
SELECT id, name, INET_NTOA(ip) AS ip, guid, INET_NTOA(server) AS server, date FROM tmp_people;
Finally, instead of looping over individual records, query all related records in the same select:
To get the related by ip:
SELECT up.id, up.name, INET_NTOA(up.ip) AS ip, up.guid, INET_NTOA(up.server) AS server, up.date FROM up_people up JOIN tmp_people tmp ON up.ip = tmp.ip
to get the related by guid:
SELECT up.id, up.name, INET_NTOA(up.ip) AS ip, up.guid, INET_NTOA(up.server) AS server, up.date FROM up_people up JOIN tmp_people tmp ON up.guid = tmp.guid;
to get the related by name:
SELECT up.id, up.name, INET_NTOA(up.ip) AS ip, up.guid, INET_NTOA(up.server) AS server, up.date FROM up_people up JOIN tmp_people tmp ON up.name = tmp.name
Side notes:
you do not need the PlayerName Index, since the name field is the left most field in the Person Index
There is no index on the guid field, so the query that finds related by guid will be slow.

Answer 3

Going back to the original structure, I would get rid of the composite index on (name, ip, guid) and create a non-unique index on name, and another non-unique index on ip.
I am not sure what to do about the guid. If you want to prevent duplicate player records, and neither the name alone, nor the name-with-ip is sufficient to guarantee uniqueness, perhaps appending an autoincrementing-integer-converted-to-string rather than a guid would be better.
As others have noted, "contains substring" i.e %foo% searches cannot take full advantage of an index; since the substring could occur in any/every indexed value, the entire index would have to be scanned. On the other hand, "starts-with" substring searches i.e. foo% are able to take advantage of an index.