I have a table with ~30 million rows ( and growing! ) and currently i have some problems with a simple range select.
The query, looks like this one:
SELECT SUM( CEIL( dlvSize / 100 ) ) as numItems
FROM log
WHERE timeLogged BETWEEN 1000000 AND 2000000
AND user = 'example'</pre>
It takes minutes to finish and i think that the solution would be at the indexes that i'm using. Here is the result of explain:
+----+-------------+-------+-------+---------------------------------+---------+---------+------+----------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+---------------------------------+---------+---------+------+----------+-------------+
| 1 | SIMPLE | log | range | PRIMARY,timeLogged | PRIMARY | 4 | NULL | 11839754 | Using where |
+----+-------------+-------+-------+---------------------------------+---------+---------+------+----------+-------------+
My table structure is this one ( reduced to make it fit better on the problem ):
CREATE TABLE IF NOT EXISTS `log` (
`origDomain` varchar(64) NOT NULL default '0',
`timeLogged` int(11) NOT NULL default '0',
`orig` varchar(128) NOT NULL default '',
`rcpt` varchar(128) NOT NULL default '',
`dlvSize` varchar(255) default NULL,
`user` varchar(255) default NULL,
PRIMARY KEY (`timeLogged`,`orig`,`rcpt`),
KEY `timeLogged` (`timeLogged`),
KEY `orig` (`orig`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
Any ideas of what can I do to optimize this query or indexes on my table?
You may want to try adding a composite index on (user, timeLogged):
CREATE TABLE IF NOT EXISTS `log` (
...
KEY `user_timeLogged` (user, timeLogged),
...
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
Related Stack Overflow post:
Database: When should I use a composite index?
In addition to the suggestions made by the other answers, I note that you have a column user in the table which is a varchar(255). If this refers to a column in a table of users, then 1) it would most likely to far more efficient to add an integer ID column to that table, and use that as the primary key and as a referencing column in other tables; 2) you are using InnoDB, so why not take advantage of the foreign key capabilities it offers?
Consider that if you index by a varchar(n) column, it is treated like a char(n) in the index, so each row of your current primary key takes up 4 + 128 + 128 = 260 bytes in the index.
Add an index on user.
Related
In my postgres db there is a table called testing_thing, which I can see (by running \d testing_thing in my psql prompt) it is defined as
Table "public.testing_thing"
Column | Type | Collation | Nullable | Default
--------------+-------------------+-----------+----------+-----------------------------------------------------
thing_id | integer | | not null | nextval('testing_thing_thing_id_seq'::regclass)
thing_num | smallint | | not null | 0
thing_desc | character varying | | not null |
Indexes:
"testing_thing_pk" PRIMARY KEY, btree (thing_num)
I want to drop it and re-create it exactly as it is, but I don't know how to reproduce the
nextval('testing_thing_thing_id_seq'::regclass)
part for column thing_id.
This is the query I put together to create the table:
CREATE TABLE testing_thing(
thing_id integer NOT NULL, --what else should I put here?
thing_num smallint NOT NULL PRIMARY KEY DEFAULT 0,
thing_desc varchar(100) NOT NULL
);
what is it missing?
Add a DEFAULT to the column you want to increment and call nextval():
CREATE SEQUENCE testing_thing_thing_id_seq START WITH 1;
CREATE TABLE testing_thing(
thing_id integer NOT NULL DEFAULT nextval('testing_thing_thing_id_seq'),
thing_num smallint NOT NULL PRIMARY KEY DEFAULT 0,
thing_desc varchar(100) NOT NULL
);
Side note: Keep in mind that attaching a sequence to a column does not prevent users to manually fill it with random data, which can create really nasty problems with primary keys. If you want to overcome it and do not necessarily need to have a sequence, consider creating an identity column, e.g.
CREATE TABLE testing_thing(
thing_id integer NOT NULL GENERATED ALWAYS AS IDENTITY,
thing_num smallint NOT NULL PRIMARY KEY DEFAULT 0,
thing_desc varchar(100) NOT NULL
);
Demo: db<>fiddle
I am having trouble figuring out how to insert multiple values to a table, which checks if another table has the needed values stored. I am currently doing this in a PostgreSQL server, but will be implementing it in PreparedStatements for my java program.
user_id is a foreign key which references the primary in mock2. I have been trying to check if mock2 has values ('foo1', 'bar1') and ('foo2', 'bar2').
After this I am trying to insert new values into mock1 which would have a date and integer value and reference the primary key of the row in mock2 to the foreign key in mock1.
mock1 table looks like this:
===============================
| date | time | user_id |
| date | integer | integer |
| | | |
And the table mock2 is:
==================================
| Id | name | program |
| integer | text | test |
Id is a primary key for the table and the name is UNIQUE.
I've been playing around with this solution https://dba.stackexchange.com/questions/46410/how-do-i-insert-a-row-which-contains-a-foreign-key
However, I haven't been able to make it work. Could someone please point out what the correct syntax is for this, I would be really appreciative.
EDIT:
The create table statements are:
CREATE TABLE mock2(
id SERIAL PRIMARY KEY UNIQUE,
name text NOT NULL,
program text NOT NULL UNIQUE
);
and
CREATE TABLE mock1(
date date,
time_spent INTEGER,
user_id integer REFERENCES mock2(Id) NOT NULL);
Ok so I found an answer to my own question.
WITH ins (date,time_spent, id) AS
( VALUES
( '22/08/2012', 170, (SELECT id FROM mock3 WHERE program ='bar'))
)
INSERT INTO mock4
(date, time_spent, user_id)
SELECT
ins.date, ins.time_spent, mock3.id
FROM
mock3 JOIN ins
ON ins.id = mock3.id ;
I was trying to take the 2 values from the first table, match these and then insert 2 new values to the next table, but I realised that I should be using the Primary and Foreign keys to my advantage.
I instead now JOIN on the ID and then just select the key I need by searching it from the values with (SELECT id FROM mock3 WHERE program ='bar') in the third row.
I have this table which is automatically created in my DB.
This is the description of the table using the \d command.
Table "public.tableA":
Column | Type | Modifiers
----------------------------+----------+-----------------------------------------------------
var_a | integer | not null
var_b | integer | not null
var_c | bigint | not null default nextval('var_c_sequence'::regclass)
var_d | integer |
var_e | integer |
var_f | smallint | default mysessionid()
var_g | smallint | default (-1)
var_h | boolean | default false
var_g | uuid |
Indexes:
"tableA_pkey" PRIMARY KEY, btree (var_c)
"tableA_edit" btree (var_g) WHERE var_g <> (-1)
"tableA_idx" btree (var_a)
Check constraints:
"constraintC" CHECK (var_f > 0 AND var_d IS NULL AND var_e IS NULL OR (var_f = 0 OR var_f = (-1)) AND var_d IS NOT NULL AND var_e IS NOT NULL)
Triggers:
object_create BEFORE INSERT ON tableA FOR EACH ROW EXECUTE PROCEDURE create_tableA()
object_update BEFORE DELETE OR UPDATE ON tableA FOR EACH ROW EXECUTE PROCEDURE update_tableA()
I'm interested in creating this table myself, and I'm not quite sure on how to define this indices manually, any ideas?
Unless I've totally missed the boat:
alter table public."tableA"
add constraint "tableA_pkey" PRIMARY KEY (var_c);
create index "tableA_edit" on public."tableA" (var_g) WHERE var_g <> (-1);
create index "tableA_idx" on public."tableA" (var_a);
Btree is default, so I don't bother specifying that, but you can if you want.
You didn't ask, but the check constraint syntax is:
alter table public."tableA"
add constraint "constraintC"
CHECK (var_f > 0 AND var_d IS NULL AND var_e IS NULL OR
(var_f = 0 OR var_f = (-1)) AND var_d IS NOT NULL AND var_e IS NOT NULL)
By the way, the cheat would be to just look at the DDL in PgAdmin.
All that said, I generally discourage the use of the "quoteS" around a table to enforce upper/lowercase. There are cases where it makes sense (otherwise, why would the functionality exist), but in many cases it creates so much extra work in the future. In the case of the index names, it doesn't even buy you anything, since you don't really refer to them in any SQL.
Is it possable to optimize this query?
SELECT count(locId) AS antal , locId
FROM `geolitecity_block`
WHERE (1835880985>= startIpNum AND 1835880985 <= endIpNum)
OR (1836875969>= startIpNum AND 1836875969 <= endIpNum)
OR (1836878754>= startIpNum AND 1836878754 <= endIpNum)
...
...
OR (1843488110>= startIpNum AND 1843488110 <= endIpNum)
GROUP BY locId ORDER BY antal DESC LIMIT 100
The table looks like this
CREATE TABLE IF NOT EXISTS `geolitecity_block` (
`startIpNum` int(11) unsigned NOT NULL,
`endIpNum` int(11) unsigned NOT NULL,
`locId` int(11) unsigned NOT NULL,
PRIMARY KEY (`startIpNum`),
KEY `locId` (`locId`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
UPDATE
and the explain-query looks like this
+----+-------------+-------------------+-------+---------------+-------+---------+------+------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------------------+-------+---------------+-------+---------+------+------+----------------------------------------------+
| 1 | SIMPLE | geolitecity_block | index | PRIMARY | locId | 4 | NULL | 108 | Using where; Using temporary; Using filesort |
+----+-------------+-------------------+-------+---------------+-------+---------+------+------+----------------------------------------------+
To optimize performance, create an index on startIpNum and endIpNum.
CREATE INDEX index_startIpNum ON geolitecity_block (startIpNum);
CREATE INDEX index_endIpNum ON geolitecity_block (endIpNum);
Indexing columns that are being grouped or sorted on will almost always improve performance. I would suggest plugging this query into the DTA (Database Tuning Advisor) to see if SQL can make any suggestions, this might include the creation of one or more indexes in addition to statistics.
If it is possible in your use case, create a temporary table TMP_RESULT (remove order) and than submit a second query that orders results by antal. Filesort is extremely slow and -- in your case -- you can not avoid this operation, because you do not sort by any of keys/indices. To perform count operation, you have to scan complete table. A temporary table is a much faster solution.
ps. Adding an index on (startIpNum, endIpNum) definitely will help you to get better performance but -- if you have a lot of rows -- it will not be a huge improvement.
Is there any to get the an AUTO_INCREMENT field of a InnoDB to start counting from 0 not 1
CREATE TABLE `df_mainevent` (
`idDf_MainEvent` int(11) NOT NULL AUTO_INCREMENT,
PRIMARY KEY (`idDf_MainEvent`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
MySQL documentation:
If a user specifies NULL or 0 for the
AUTO_INCREMENT column in an INSERT,
InnoDB treats the row as if the value
had not been specified and generates a
new value for it.
So it means that 0 is a 'special' value which is similar to NULL. Even when you use AUTO_INCREMENT = 0 is will set the initial value to 1.
Beginning with MySQL 5.0.3, InnoDB
supports the AUTO_INCREMENT = N table
option in CREATE TABLE and ALTER TABLE
statements, to set the initial counter
value or alter the current counter
value. The effect of this option is
canceled by a server restart, for
reasons discussed earlier in this
section.
CREATE TABLE `df_mainevent` (
`idDf_MainEvent` int(11) NOT NULL AUTO_INCREMENT,
PRIMARY KEY (`idDf_MainEvent`)
) ENGINE=InnoDB AUTO_INCREMENT=0 DEFAULT CHARSET=latin1;
works with MySQL >= 5.0.3.
EDIT:
Just noticed that MySQL in general does not like auto-increment values equal to 0 - that's independent from the used storage engine. MySQL just uses 1 as the first auto-increment value. So to answer the question: NO that's not possible but it does not depend on the storage engine.
This works in both InnoDB and MyISAM, and the second insert is a 1 not a 2:
CREATE TABLE ex1 (id INT AUTO_INCREMENT PRIMARY KEY) ENGINE=MyISAM;
SET sql_mode='NO_AUTO_VALUE_ON_ZERO';
INSERT INTO ex1 SET id=0;
INSERT INTO ex1 SET id=NULL;
SELECT * FROM ex1;
+----+
| id |
+----+
| 0 |
| 1 |
+----+
2 rows in set (0.00 sec)
CREATE TABLE ex2 (id INT AUTO_INCREMENT PRIMARY KEY) ENGINE=InnoDB;
SET sql_mode='NO_AUTO_VALUE_ON_ZERO';
INSERT INTO ex2 SET id=0;
INSERT INTO ex2 SET id=NULL;
SELECT * FROM ex2;
+----+
| id |
+----+
| 0 |
| 1 |
+----+
2 rows in set (0.00 sec)
Daren Schwenke's technique works. To bad that the next record inserted will be 2.
For example:
CREATE TABLE IF NOT EXISTS `table_name` (
`ID` INT UNSIGNED NOT NULL AUTO_INCREMENT,
`Name` VARCHAR(100) NOT NULL,
PRIMARY KEY( `ID` )
) ENGINE=InnoDB AUTO_INCREMENT=0 DEFAULT CHARSET=latin1;
INSERT INTO `table_name` (`Name`) VALUES ('Record0?');
UPDATE `table_name` SET `ID`=0 WHERE `ID`=1;
INSERT INTO `table_name` (`Name`) VALUES ('Record1?');
SELECT * FROM `table_name`;
ID Name
0 Record0?
2 Record1?
This isn't a big deal its just annoying.
Tim
I have not been able to have autoincrement start at 0, but starting at 1 and then setting it to 0 via an UPDATE works fine.
I commonly use this trick to detect deletes in a table.
On update of any row, I set that row's last update time.
On deletes, I set the last update time of row 0.