Querying on key of a Map in Aerospike - aerospike

I'm trying to store a map in aerospike and fetch the data based on the key of the map.
First I created a Index on the bin where i'm storing the map
aql> create mapkeys index status on test.myset (state) String
aql> show indexes
+--------+---------+-----------+---------+-------+-----------+---------+------------+----------+
| ns | bin | indextype | set | state | indexname | path | sync_state | type |
+--------+---------+-----------+---------+-------+-----------+---------+------------+----------+
| "test" | "state" | "MAPKEYS" | "myset" | "RW" | "status" | "state" | "synced" | "STRING" |
+--------+---------+-----------+---------+-------+-----------+---------+------------+----------+
1 row in set (0.000 secs)
OK
Then I used java client to store the map
AerospikeClient client = new AerospikeClient("127.0.0.1",3000);
WritePolicy writePolicy = new WritePolicy();
writePolicy.timeout=500;
for(int i = 1;i<10;i++){
Key key = new Key("test","myset",""+i);
client.delete(writePolicy, key);
HashMap<String,String> map = new HashMap<String,String>();
map.put("key1", "string1");
map.put("key2", "string2");
map.put("key3", "string3");
Bin bin = new Bin("state", map);
client.put(writePolicy, key, bin);
}
I checked the data through apl and the data is clearly present.
aql> select * from test.myset
+--------------------------------------------------------+
| state |
+--------------------------------------------------------+
| {"key1":"string1", "key2":"string2", "key3":"string3"} |
| {"key1":"string1", "key2":"string2", "key3":"string3"} |
| {"key1":"string1", "key2":"string2", "key3":"string3"} |
| {"key1":"string1", "key2":"string2", "key3":"string3"} |
| {"key1":"string1", "key2":"string2", "key3":"string3"} |
| {"key1":"string1", "key2":"string2", "key3":"string3"} |
| {"key1":"string1", "key2":"string2", "key3":"string3"} |
| {"key1":"string1", "key2":"string2", "key3":"string3"} |
| {"key1":"string1", "key2":"string2", "key3":"string3"} |
| {"key1":"string1", "key2":"string2", "key3":"string3"} |
+--------------------------------------------------------+
10 rows in set (0.019 secs)
Now when I try to query based on the index created it gives
aql> select * from test.myset where status = 'key1'
0 rows in set (0.000 secs)
Error: (204) AEROSPIKE_ERR_INDEX
aql> select * from test.myset where state = 'key1'
0 rows in set (0.000 secs)
Error: (201) AEROSPIKE_ERR_INDEX_NOT_FOUND
Can someone help me with this. I searched for that error but found no information. Thank you.

Secondary indexes on MapKeys, MapValues, Lists are supported by Aerospike, apart from Numeric, String and Geo2DSphere types.
For your scenario, you can query on the Mapkey as follows.
select * from test.myset in mapkeys where state='key1'
This should return the results.
In AQL, if you type help, you should get the following for queries
SELECT <bins> FROM <ns>[.<set>]
SELECT <bins> FROM <ns>[.<set>] WHERE <bin> = <value>
SELECT <bins> FROM <ns>[.<set>] WHERE <bin> BETWEEN <lower> AND <upper>
SELECT <bins> FROM <ns>[.<set>] WHERE PK = <key>
SELECT <bins> FROM <ns>[.<set>] IN <indextype> WHERE <bin> = <value>
SELECT <bins> FROM <ns>[.<set>] IN <indextype> WHERE <bin> BETWEEN <lower> AND <upper>
Similarly, you can run a query for the MapValue as well.

Update:
As of Aerospike 3.8.1, Secondary Index on List and Map are officially supported.
Original Response:
Query by secondary index on map keys, map values, or list values are
not officially supported yet.
That said, the functionality and syntax is somewhat available. You need to:
Create a secondary index with type MAPKEYS, MAPVALUES or LIST (you're using type STRING at the moment)
Select as follows (you're missing the IN MAPKEYS part):
SELECT * FROM namespace.setname IN MAPKEYS WHERE bin = 'keyValue'
The query syntax, as well as some other bits, is available if you type help while in the AQL console.

Related

Do UPSERT based on specific value of JSON in Postgres 10

I have a Postgres table messages as follows:
Column | Type | Collation | Nullable |
-----------+--------------------------+-----------+----------
id | integer | | not null |
message | jsonb | | |
date | timestamp with time zone | | not null |
id | message | date
1 | {"name":"alpha", "pos":"x"} | 2020-02-11 12:31:44.658667+00
2 | {"name":"bravo", "pos":"y"} | 2020-02-11 12:32:43.123678+00
3 | {"name":"charlie", "pos":"z"}| 2020-02-11 12:38:37.623535+00
What I would like to do is do an UPSERT based on the value of the name key i.e., if there is an insert with same name value, then the other value pos is updated, otherwise a new entry is created.
I did CREATE UNIQUE INDEX message_name ON messages((message->>'name'));
I found the INSERT ON CONFLICT in Postgres 9.5+ but I can't understand how to use the unique index with this.
I don't know if this is the correct approach to do it in the first place so if there is a better way to do this, I would appreciate the input.
You need to repeat the expression from the index:
insert into messages (message)
values ('{"name":"alpha", "pos":"new pos"}')
on conflict ((message->>'name'))
do update
set message = jsonb_set(messages.message, '{pos}'::text[], excluded.message -> 'pos', true)
;
If you have more keys in the JSON and want to replace (or add) all of them, you can use this:
insert into messages (message)
values ('{"name":"alpha", "pos":"new pos", "some key": 42}')
on conflict ((message->>'name'))
do update
set message = messages.message || (excluded.message - 'name')
;

How to display all columns and its data type in a table via SQL query

I am trying to print the column names from a table called 'meta' and I need also its data types.
I tried this query
SELECT meta FROM INFORMATION_SCHEMA.TABLES;
but it throws an error saying no information schema available. Could you please help me, I am a beginner in SQL.
Edit:
select tables.name from tables join schemas on
tables.schema_id=schemas.id where schemas.name=’sprl_db’ ;
This query gives me all the tables in database 'sprl_db'
You can use the monetdb catalog:
select c.name, c.type, c.type_digits, c.type_scale
from sys.columns c
inner join sys.tables t on t.id = c.table_id and t.name = 'meta';
as you are using monetDB you can get that by using sys.columns
sys.columns
it will return all information related to table columns
you can also check Schema, table and columns documentation for monetDB
in sql server we get that like this exec sp_columns TableName
If I understand correctly you need to see the columns and the types of a table you (or some other user) defined called meta?
There are at least two ways to do this:
First (as #GMB mentioned in their answer) you can query the SQL catalog: https://www.monetdb.org/Documentation/SQLcatalog/TablesColumns
SELECT * FROM sys.tables WHERE NAME='meta';
+------+------+-----------+-------+------+--------+---------------+--------+-----------+
| id | name | schema_id | query | type | system | commit_action | access | temporary |
+======+======+===========+=======+======+========+===============+========+===========+
| 9098 | meta | 2000 | null | 0 | false | 0 | 0 | 0 |
+------+------+-----------+-------+------+--------+---------------+--------+-----------+
1 tuple
So this gets all the relevant information about the table meta. We are mostly interested in the value of the column id because this uniquely identifies the table.
(Please note that this id will probably be different in your system)
After we have this information we can query the columns table with this table id:
SELECT * FROM sys.columns WHERE table_id=9098;
+------+------+------+-------------+------------+----------+---------+-------+--------+---------+
| id | name | type | type_digits | type_scale | table_id | default | null | number | storage |
+======+======+======+=============+============+==========+=========+=======+========+=========+
| 9096 | i | int | 32 | 0 | 9098 | null | true | 0 | null |
| 9097 | j | clob | 0 | 0 | 9098 | null | true | 1 | null |
+------+------+------+-------------+------------+----------+---------+-------+--------+---------+
2 tuples
Since you are only interested in the names and types of the columns, you can modify this query as follows:
SELECT name, type FROM sys.columns WHERE table_id=9098;
+------+------+
| name | type |
+======+======+
| i | int |
| j | clob |
+------+------+
2 tuples
You can combine the two queries above with a join:
SELECT col.name, col.type FROM sys.tables as tab JOIN sys.columns as col ON tab.id=col.table_id WHERE tab.name='meta';
+------+------+
| name | type |
+======+======+
| i | int |
| j | clob |
+------+------+
2 tuples
The second, and preferred way to get this information if you are using the mclient utility of MonetDB, is by using the describe meta-command of mclient. When used without arguments it presents a list of tables that have been defined in the current database and when it is given the name of the table it prints its SQL definition:
sql>\d
TABLE sys.data
TABLE sys.meta
sql>\d sys.meta
CREATE TABLE "sys"."meta" (
"i" INTEGER,
"j" CHARACTER LARGE OBJECT
);
You can use the \? meta-command to see a list of all meta-commands in mclient:
sql>\?
\? - show this message
\<file - read input from file
\>file - save response in file, or stdout if no file is given
\|cmd - pipe result to process, or stop when no command is given
\history - show the readline history
\help - synopsis of the SQL syntax
\D table - dumps the table, or the complete database if none given.
\d[Stvsfn]+ [obj] - list database objects, or describe if obj given
\A - enable auto commit
\a - disable auto commit
\e - echo the query in sql formatting mode
\t - set the timer {none,clock,performance} (none is default)
\f - format using renderer {csv,tab,raw,sql,xml,trash,rowcount,expanded,sam}
\w# - set maximal page width (-1=unlimited, 0=terminal width, >0=limit to num)
\r# - set maximum rows per page (-1=raw)
\L file - save client-server interaction
\X - trace mclient code
\q - terminate session and quit mclient
For MySQL:
SELECT column_name,
data_type
FROM information_schema.columns
WHERE table_schema = ’ yourdatabasename ’
AND table_name = ’ yourtablename ’;
Output:
+-------------+-----------+
| COLUMN_NAME | DATA_TYPE |
+-------------+-----------+
| Id | int |
| Address | varchar |
| Money | decimal |
+-------------+-----------+

Update referencing on subquery (sqlite)

I have a table with md5 sums for files and use the following query to find the files which exist in one hashing-run and not in the other (oldt vs newt):
SELECT *
FROM md5_sums as oldt
WHERE NOT EXISTS (SELECT *
FROM md5_sums as newt
WHERE oldt.file = newt.file
and oldt.relpath = newt.relpath
and newt.starttime = 234)
and oldt.starttime = 123
now I want to put a flag in an extra column with an update clause, like
update md5_sums
set only_in_old = 'X'
where
and there I want a reference to the upper query as subquery, but i cannot find a proper way. Is there a possibility to use the results from the upper query for the where clause from the update-query?
(I added now some Table Screenshots with simple Table Data)
Table Description
Table Data before UPDATE
desired Table Data after UPDATE
SQLite does not support aliasing the updated table.
In your case you don't need that.
You can use the table's name md5_sums inside the subquery since you aliased the table of the SELECT statement as newt.
UPDATE md5_sums
SET only_in_old = 'X'
WHERE NOT EXISTS (
SELECT 1 FROM md5_sums AS newt
WHERE md5_sums.file = newt.file
AND md5_sums.relpath = newt.relpath
AND newt.starttime = 234
)
AND starttime = 123
See the demo.
Results:
| file | relpath | starttime | only_in_old |
| ------- | -------- | --------- | ----------- |
| abc.txt | /var/tmp | 123 | |
| abc.txt | /var/tmp | 234 | |
| def.txt | /tmp | 123 | X |
| xyz.txt | /tmp | 234 | |
I hope this helps you in converting the select statement into an update statement,
UPDATE md5_sums
SET only_in_old = 'X'
WHERE NOT EXISTS (SELECT *
FROM md5_sums newt
WHERE file = newt.file
and relpath = newt.relpath
and newt.starttime = 1551085649.7764235)
and starttime = 1551085580.009046

Filter json values regardless of keys in PostgreSQL

I have a table called diary which includes columns listed below:
| id | user_id | custom_foods |
|----|---------|--------------------|
| 1 | 1 | {"56": 2, "42": 0} |
| 2 | 1 | {"19861": 1} |
| 3 | 2 | {} |
| 4 | 3 | {"331": 0} |
I would like to count how many diaries having custom_foods value(s) larger than 0 each user have. I don't care about the keys, since the keys can be any number in string.
The desired output is:
| user_id | count |
|---------|---------|
| 1 | 2 |
| 2 | 0 |
| 3 | 0 |
I started with:
select *
from diary as d
join json_each_text(d.custom_foods) as e
on d.custom_foods != '{}'
where e.value > 0
I don't even know whether the syntax is correct. Now I am getting the error:
ERROR: function json_each_text(text) does not exist
LINE 3: join json_each_text(d.custom_foods) as e
HINT: No function matches the given name and argument types. You might need to add explicit type casts.
My using version is: psql (10.5 (Ubuntu 10.5-1.pgdg14.04+1), server 9.4.19). According to PostgreSQL 9.4.19 Documentation, that function should exist. I am so confused that I don't know how to proceed now.
Threads that I referred to:
Postgres and jsonb - search value at any key
Query postgres jsonb by value regardless of keys
Your custom_foods column is defined as text, so you should cast it to json before applying json_each_text. As json_each_text by default does not consider empty jsons, you may get the count as 0 for empty jsons from a separate CTE and do a UNION ALL
WITH empty AS
( SELECT DISTINCT user_id,
0 AS COUNT
FROM diary
WHERE custom_foods = '{}' )
SELECT user_id,
count(CASE
WHEN VALUE::int > 0 THEN 1
END)
FROM diary d,
json_each_text(d.custom_foods::JSON)
GROUP BY user_id
UNION ALL
SELECT *
FROM empty
ORDER BY user_id;
Demo

Unable to use stream UDFs on MAPKEYS index

I have a bin with map as datatype and created a secondary on MAPKEYS. Now i want to run a udf with filter on MAPKEYS index. It gives the error AEROSPIKE_ERR_INDEX_NOT_FOUND.
This is my aql query:
aql> aggregate test.check_password('hii') on test.user in MAPKEYS where pids = 'test2'
Error: (201) AEROSPIKE_ERR_INDEX_NOT_FOUND
whereas the normal query works
aql> select * from test.user in MAPKEYS where pids = 'test2'
returns some data
Sample data inserted for testing, in the ideal case it will be a Map of String to Object
aql> INSERT INTO test.user (PK, pids, test2, test1) VALUES ('k1', MAP('{"test1": "t1", "test2": "t2", "test3":"t3", "test4":"t4", "test5":"t5"}'), "t2bin", "t1bin")
aql> INSERT INTO test.user (PK, pids, test2, test1) VALUES ('k2', MAP('{"test1": "t1", "test3":"t3", "test4":"t4", "test5":"t5"}'), "t2b", "t1b")
aql> INSERT INTO test.user (PK, pids, test2, test1) VALUES ('k3', MAP('{"test1": "t1", "test2":"t22", "test4":"t4", "test5":"t5"}'), "t2b", "t1b")
aql> CREATE MAPKEYS INDEX pidIndex ON test.user (pids) STRING
OK, 1 index added.
aql> select * from test.user in MAPKEYS where pids="test2"
+--------------------------------------------------------------------------------+---------+---------+
| pids | test2 | test1 |
+--------------------------------------------------------------------------------+---------+---------+
| MAP('{"test2":"t22", "test4":"t4", "test5":"t5", "test1":"t1"}') | "t2b" | "t1b" |
| MAP('{"test2":"t2", "test3":"t3", "test4":"t4", "test5":"t5", "test1":"t1"}') | "t2bin" | "t1bin" |
+--------------------------------------------------------------------------------+---------+---------+
I inserted three records in your format, one did not have the test2 key in its map (k2). I then created the secondary index on the MAPKEY and ran the query, gave me the desired result.
AGGREGATE is used to run a stream User Defined Function on this result set of records. What is the UDF code that you want to run?
(AGGREGATE test.check_password("hii") ....implies you have a test.lua file which has a check_password() function that takes a string argument. )
You must create the secondary index on the MAP Keys first. Its reporting index not found. To check if you have the index, you can do:
aql> show indexes
+--------+--------+-----------+--------+-------+------------+--------+------------+----------+
| ns | bin | indextype | set | state | indexname | path | sync_state | type |
+--------+--------+-----------+--------+-------+------------+--------+------------+----------+
| "test" | "pids" | "MAPKEYS" | "user" | "RW" | "pidIndex" | "pids" | "synced" | "STRING" |
+--------+--------+-----------+--------+-------+------------+--------+------------+----------+
1 row in set (0.000 secs)
OK