Can anyone suggest a way we can store the data in Redis Cache where the keys are not unique.
For an instance, I have a database with two columns A and B. A is always unique and hence we can create a Redis hash using it.
However, B is not unique and for B there can be multiple data of column A.
Can someone suggest a way we can create a Redis Hash that supports non unique Keys.
It is impossible to create non unique keys in redis hash.My suggestion is to append the values of column B in single hash key.
For example,if column B contains the following value.,
B | value
key_1 | value1
key_2 | value2
key_3 | value3
key_1 | value4
key_2 | value5
you can store the value in redis hash by appending the value1 and value4 in key_1,like this.,
key_1 = value1:value4
key_2 = value2:value5
key_3 = value3
P.S: For implementing this method, you need to make two round trips to the redis-server for getting the value already present in the redis and then back setting the new value in redis.
If you want to store the values on B column(which are not unique), you can use Redis list
See Redis data types
Related
I have a table place2022 which has a very long CHAR column
timestamp | user_id | pixel_color | coordinate
-----------------+------------------------------------------------------------------------------------------+-------------+------------
17:38:20.021+00 | p0sXpmkcmg1KLiCdK5e4xKdudb1f8cjscGs35082sKpGBfQIw92nZ7yGvWbQ/ggB1+kkRBaYu1zy6n16yL/yjA== | #FF4500 | 371,488
17:38:20.024+00 | Ctar52ln5JEpXT+tVVc8BtQwm1tPjRwPZmPvuamzsZDlFDkeo3+ItUW89J1rXDDeho6A4zCob1MKmJrzYAjipg== | #51E9F4 | 457,493
17:38:20.025+00 | rNMF5wpFYT2RAItySLf9IcFZwOhczQhkRhmTD4gv0K78DpieXrVUw8T/MBAZjj2BIS8h5exPISQ4vlyzLzad5w== | #000000 | 65,986
17:38:20.025+00 | u0a7l8hHVvncqYmav27EARAE6ciLtpUTPXMI33lDrUmtj5Ei3ixlfRuG28KUvs7r5LpeiE/iOKPALVjkILhrYg== | #3690EA | 73,961
The user_ids are already hashes, so all I really care about here is having some sort of id column which is 1-1 with the user_id.
I've counted the number of unique user_ids, which is 10381163, which fits into 24 bits. Therefore, I can compress the id field down to a 32-bit integer using the obvious scheme of "Assign 1 to the first new user_id you see, 2 to the second new user_id you see", etc. I don't even care that the user_id's are mapped in the order that they're seen: I just need them to be mapped in an invertible manner to 32-bit ints somehow. I'd also like to persist this mapping somewhere so that, if I want to, I can go backwards.
What would be the best way to achieve this? I imagine that we could create a new table (create table place2022_user_ids as select distinct(user_id) from place2022;?) and then reverse-lookup the user_id column in that table, but I don't know quite how to formulate the queries and also make sure that I'm not doing something ridiculously slow.
I am using postgresql, if it matters.
If you have a recent (>8) version of Postgres you can add an auto increment id column to an existing table.
ALTER TABLE place2022
ADD COLUMN id SERIAL PRIMARY KEY;
NB If the existing column is a PRIMARY KEY you will need to drop it first.
See drop primary key constraint in postgresql by knowing schema and table name only
I have three tables that look like
table1
ID Title Description
1 Title1 Desc1
2 Title2 Desc2
table2
ID HistoryID
1 1001
2 1002
2 1003
2 1004
table3
HistoryID Value
1001 val1
1002 val2
1003 val3
1004 val4
Now I am planning to it using "only" two tables
table1
ID Title Description HistoryIDList
1 Title1 Desc1 1001
2 Title2 Desc2 1002,1003,1004
table3
HistoryID Value
1001 val1
1002 val2
1003 val3
1004 val4
I have created a sql table-value function that will return indexed values 1002,1003,1004 that could be joined with HistoryID from table3.
Since I am losing normalization, and do not have FK for HistoryIDList, my questions are
should there by significant performance issue running a query that would join HistoryIDList
would indexing sql function do the trick or not since there is no relation between two columns.
In that case is it possible to add FK on table created in sql function?
Why would you change a good data structure to a bogus data structure? The two table version is a bad choice.
Yes, there is a significant performance difference when joining from a single id to a list, compared to a simple equi-join. And, as bad as that normally is, the situation is even worse here because the type of the id is presumably an int in the original table and a character string in the other table.
There is no way to enforce foreign key relationships with your proposed structure without using triggers.
The only thing that you could possibly do to improve performance would be to have a full text index on the HistoryIdList column. This might speed the processing. Once again, this is complicated by the fact that you are using numeric ids.
My recommendation: just don't do it. Leave the table structure as it is, because that is the best structure for a relational database.
Let us consider the following table structures:
Table1
Table1_ID A
1 A1
2 A1;B1
and
Table2
Table2_ID Table1_ID B C
1 1 foobar barfoo
2 2 foofoo barbar
The view I'm using is defined by the following query:
SELECT Table1.A, B, C
FROM Table2
INNER JOIN Table1 ON Table1.Table1_ID = Table2.Table1_ID;
95% of A's data consists in a 2 characters long string. In this case, it works fine. However, 5% of it is actually a list (using a semicolon as a separator) of possible values for this field.
This means my users would like to choose between these values when it is appropriate, and keep using the single value automatically the rest of the time. Of course, this is not possible with a single INNER JOIN, since there cannot be a constant selected value.
Table2 is very large, while Table1 is quite small. Manually filling a local A field in each row within Table2 would be a huge waste of time.
Is there an efficient way for SQL (or, more specifically, SQL Server 2008) to handle this? Such as a list with a selected item within a field?
I was planning to add a "A_ChosenValue" field that would store the chosen value when there's a list in A, and remain empty when A only stores a single value. It would only require users to fill it 5% of the time, which is okay. But I was thinking there could be a better way than using two columns to store a single value.
Ideally you would just alter your schema and add a new entity to support the many-to-many relationship between Table1 and Table2 such as the following with a compound key of all three columns.
Table3
| Table1_ID | Table2_ID | A |
-----------------------------
| 1 | 1 | A1 |
------------------------------
| 2 | 2 | A1 |
------------------------------
| 2 | 2 | B1 |
------------------------------
You could then do a select and join on this table and due to it being indexed you won't lose any performance.
Without altering the table structure or normalizing data it is possible using a conditional select statement like that shown in this SO post but the query wouldn't perform so well as you would have to use a function to split the values containing a semi-colon.
Answering my own question:
I added a LocalA column in Table1, in order that my view actually selects ISNULL(LocalA, Table1.A). Therefore, the displayed value equals A by default, and users can manually overwrite it to select a specific value when A stores a list.
I am not sure whether this is the most efficient solution or not, but at least it works without requiring two columns in the view.
In a MySQL database I have a table with the following primary key
PRIMARY KEY id (invoice, item)
In my application I will also frequently be selecting on item by itself and less frequently on only invoice. I'm assuming I would benefit from indexes on these columns.
MySQL does not complain when I define the following:
INDEX (invoice),
INDEX (item),
PRIMARY KEY id (invoice, item)
But I don't see any evidence (using DESCRIBE -- the only way I know how to look) that separate indexes have been established for these two columns.
Are the columns that make up a primary key automatically indexed individually? Is there a better way than DESCRIBE to explore the structure of my table?
I'm not intimately familiar with the internals of indices on mySql, but on the two database vendor products that I am familiar with (MsSQL, Oracle) indices are balanced-Tree structures, whose nodes are organized as a sequenced tuple of the columns the index is defined on (In the Sequence Defined)
So, unless mySql does it very differently, (probably not), any composite index (on more than one column) can be useable by any query that needs to filter or sort by a subset of the columns in the index, as long as the list of columns is compatible, i.e., if the columns, when sequenced the same as the sequenced list of columns in the complete index, is an ordered subset of the complete set of index columns, which starts at the beginning of the actual index sequence, with no gaps except at the end...
In other words, this means that if you have an index on (a,b,c,d) a query that filters on (a), (a,b), or (a,b,c) can also use the index, but a query that needs to filter on (b), or (c) or (b,c) will not be able to use the index...
So in your case, if you often need to filter or sort on column item alone, you need to add another index on that column by itself...
I personally use phpMyAdmin to view and edit the structure of MySQL databases. It is a web application but it runs well enough on a local web server (I run an instance of apache on my machine for this and phpPgAdmin).
As for the composite key of (invoice, item), it acts like an index for (invoice, item) and for invoice. If you want to index by just item you have to add that index yourself. Your PK will be sorted by invoice and then by item where invoice is the same in multiple records. While the order in a composite PK does not matter for uniqueness enforcement, it does matter for access.
On your table I would use:
PRIMARY KEY id (invoice, item), INDEX (item)
I'm not that familiar with MySQL, but generally an multiple-column index is equally useful on the first column in the index as an index on that column alone. The multiple-column index becomes less useful for querying against a single column the further the column appears into the index.
This makes some sense if you think of the multi-column index as a hierarchy. The first column in the index is the root of the hierarchy, so searching it is just a matter of scanning that first level. However, in order to scan the second column, the database has to look up the tree for each unique value found in the first column. This can be costly enough that most optimizers won't bother to look deeply into a multi-column index, instead opting to full-table-scan.
For example, if you have a table as follows:
Col1 |Col2 |Col3
----------------
A | 1 | Z
A | 2 | Y
A | 2 | X
B | 1 | Z
B | 2 | X
Assuming you have an index on all three columns, in order, the tree will look something like this:
A
+-1
+-Z
+-2
+-X
+-Y
B
+-1
+-Z
+-2
+-X
Looking for Col1='A' is easy: you only have to look at 2 ordered values. However, to resolve col3='X', you have to look at all of the values in the 4 bigger buckets, each of which is ordered individually.
To return table index information, you can use:
SHOW INDEX FROM <table>;
See: http://dev.mysql.com/doc/refman/5.0/en/show-index.html
To view table information:
SHOW CREATE TABLE <table>;
See: http://dev.mysql.com/doc/refman/5.0/en/show-create-table.html
Primary keys are indexes, so there's no need to create additional indexes. You can find out more information about them under the CREATE TABLE syntax (there's too much to insert here):
http://dev.mysql.com/doc/refman/5.0/en/create-table.html
There is a difference between composite index and composite primary key.
If you have defined a composite index like below
INDEX idx(invoice,item)
the index wont work if you query based on item and you need to add a separate index
INDEX itemidx(item)
But, if you have defined a composite primary key like below
PRIMARY KEY(invoice, item)
the index would work if you query based on item and no separate index is required.
Working example:
mysql>create table test ( col1 int(20), col2 int(20) ) primary key(col1,col2);
mysql>explain select * from test where col2 = 1;
+----+-------------+-------+-------+---------------+---------+---------+------+------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+---------------+---------+---------+------+------+--------------------------+
| 1 | SIMPLE | test | index | NULL | PRIMARY | 8 | NULL | 10 | Using where; Using index |
+----+-------------+-------+-------+---------------+---------+---------+------+------+--------------------------+
Mysql auto create an index for composite keys. Depending on your queries, you may have to create separate index for individual column in the composite key.
If you are using mysql workbench, you can manually right click the schema and click on edit to see everything about the table
If your query is using both columns in where clause then you don't need to create a separate index in a composite primary key.
EXPLAIN SELECT * FROM `table` WHERE invoice = 1 and item = 1
You are also fine if you want to query with first column only
EXPLAIN SELECT * FROM `table` WHERE invoice = 1
But if you want to query with subsequent columns col2, col3 in composite PK then you would need to create separate indexes on those columns. The following explain query shows the second column does not have a possible key detected by MySQL
EXPLAIN SELECT * FROM `table` WHERE item = 1
I need to speed up a query. Is an index table what I'm looking for? If so, how do I make one? Do I have to update it each insert?
Here are the table schemas:
--table1-- | --tableA-- | --table2--
id | id | id
attrib1 | t1id | attrib1
attrib2 | t2id | attrib2
| attrib1 |
And the query:
SELECT
table1.attrib1,
table1.attrib2,
tableA.attrib1
FROM
table1,
tableA
WHERE
table1.id = tableA.t1id
AND (tableA.t2id = x or ... or tableA.t2id = z)
GROUP BY
table1.id
You need to create a composite index on tableA:
CREATE INDEX ix_tablea_t1id_t2id ON table_A (t1id, t2id)
Indexes in MySQL are considered a part of a table: they are updated automatically, and used automatically whenever the optimizer decides it's a good move to use them.
MySQL does not use the term index table.
This term is used by Oracle to refer to what other databases call CLUSTERED INDEX: a kind of table where the records themselves are arranged according to the value of a column (or a set of columns).
In MySQL:
When you use MyISAM storage, an index is created as a separate file that has .MYI extension.
The contents of this file represent a B-Tree, each leaf containing the index key and a pointer to the offset in .MYD file which contains the data.
The size of the pointer is determined by the server setting called myisam_data_pointer_size, which can vary from 2 to 7 bytes, and defaults to 6 since MySQL 5.0.6.
This allows creating MyISAM tables up to 2 ^ (8 * 6) bytes = 256 TB
In InnoDB, all tables are inherently ordered by the PRIMARY KEY, it does not support heap-organized tables.
Each index, therefore, in fact is just a plain InnoDB table consisting of a single PRIMARY KEY of N+M records: N records being an indexed value, and M records being a PRIMARY KEY of the main table record which holds the indexed data.