Apache Ignite - querying a cache through an sql driver - ignite

I created a cache in Ignite and now I'm attempting to query it via sqlline provided in the ignite bundle. In the documentation it only talks about creating tables, indexes, querying those tables, etc.. but nothing about querying caches created via ignite.getOrCreateCache() in this manner. Given that ignite is so feature rich I assume this is possible.
What am I missing?
Notes:
Ignite Version = 2.3.0
SqlFieldsQuery via the ignite Sql mechanism is working fine within my code.
This looks very similar to this issue but my distribution only has one h2 jar, h2-1.4.196.jar, although I am using spring-boot
java code to create the cache
cacheConfig.setName("eventCache");
cacheConfig.setTypes(TenantKey.class, EventCachePojo.class);
cacheConfig.setIndexedTypes(TenantKey.class, EventCachePojo.class);
Cache<String, EventCachePojo> cache = ignite.getOrCreateCache(cacheConfig);
Then running sqlline
# i tried jdbcUrl jdbc:ignite:thin://127.0.0.1/ and jdbc:ignite:thin://127.0.0.1/eventCache
$ ./sqlline.sh --color=true --verbose=true -u jdbc:ignite:thin://127.0.0.1/
issuing: !connect jdbc:ignite:thin://127.0.0.1/ '' '' org.apache.ignite.IgniteJdbcThinDriver
Connecting to jdbc:ignite:thin://127.0.0.1/
Connected to: Apache Ignite (version 2.3.0#20171028-sha1:8add7fd5)
Driver: Apache Ignite Thin JDBC Driver (version 2.3.0#20171028-sha1:8add7fd5)
Autocommit status: true
Transaction isolation: TRANSACTION_REPEATABLE_READ
sqlline version 1.3.0
0: jdbc:ignite:thin://127.0.0.1/> !tables
+--------------------------------+--------------------------------+--------------------------------+--------------------------------+-----------------------------+
| TABLE_CAT | TABLE_SCHEM | TABLE_NAME | TABLE_TYPE | REMARKS |
+--------------------------------+--------------------------------+--------------------------------+--------------------------------+-----------------------------+
| | eventCache | EVENTCACHEPOJO | TABLE | |
| | eventCache | EVENTCACHEPOJO | TABLE | |
+--------------------------------+--------------------------------+--------------------------------+--------------------------------+-----------------------------+
0: jdbc:ignite:thin://127.0.0.1/> select count(*) from eventCache.EVENTCACHEPOJO;
Error: Failed to parse query: select count(*) from eventCache.EVENTCACHEPOJO (state=42000,code=0)
java.sql.SQLException: Failed to parse query: select count(*) from eventCache.EVENTCACHEPOJO
at org.apache.ignite.internal.jdbc.thin.JdbcThinConnection.sendRequest(JdbcThinConnection.java:671)
at org.apache.ignite.internal.jdbc.thin.JdbcThinStatement.execute0(JdbcThinStatement.java:130)
at org.apache.ignite.internal.jdbc.thin.JdbcThinStatement.execute(JdbcThinStatement.java:299)
at sqlline.Commands.execute(Commands.java:823)
at sqlline.Commands.sql(Commands.java:733)
at sqlline.SqlLine.dispatch(SqlLine.java:795)
at sqlline.SqlLine.begin(SqlLine.java:668)
at sqlline.SqlLine.start(SqlLine.java:373)
at sqlline.SqlLine.main(SqlLine.java:265)
0: jdbc:ignite:thin://127.0.0.1/> select count(*) from eventCache.EVENTCACHEPOJO;
Error: Failed to parse query: select count(*) from eventCache.EVENTCACHEPOJO (state=42000,code=0)
java.sql.SQLException: Failed to parse query: select count(*) from eventCache.EVENTCACHEPOJO
at org.apache.ignite.internal.jdbc.thin.JdbcThinConnection.sendRequest(JdbcThinConnection.java:671)
at org.apache.ignite.internal.jdbc.thin.JdbcThinStatement.execute0(JdbcThinStatement.java:130)
at org.apache.ignite.internal.jdbc.thin.JdbcThinStatement.execute(JdbcThinStatement.java:299)
at sqlline.Commands.execute(Commands.java:823)
at sqlline.Commands.sql(Commands.java:733)
at sqlline.SqlLine.dispatch(SqlLine.java:795)
at sqlline.SqlLine.begin(SqlLine.java:668)
at sqlline.SqlLine.start(SqlLine.java:373)
at sqlline.SqlLine.main(SqlLine.java:265)
0: jdbc:ignite:thin://127.0.0.1/> select count(*) from EVENTCACHEPOJO;
Error: Failed to parse query: select count(*) from EVENTCACHEPOJO (state=42000,code=0)
java.sql.SQLException: Failed to parse query: select count(*) from EVENTCACHEPOJO
at org.apache.ignite.internal.jdbc.thin.JdbcThinConnection.sendRequest(JdbcThinConnection.java:671)
at org.apache.ignite.internal.jdbc.thin.JdbcThinStatement.execute0(JdbcThinStatement.java:130)
at org.apache.ignite.internal.jdbc.thin.JdbcThinStatement.execute(JdbcThinStatement.java:299)
at sqlline.Commands.execute(Commands.java:823)
at sqlline.Commands.sql(Commands.java:733)
at sqlline.SqlLine.dispatch(SqlLine.java:795)
at sqlline.SqlLine.begin(SqlLine.java:668)
at sqlline.SqlLine.start(SqlLine.java:373)
at sqlline.SqlLine.main(SqlLine.java:265)


i finally figured it out. This Post led me to the answer. The schemaName and table need to be quoted per h2 db.
Here is the syntax:
$ ./sqlline.sh --color=true --verbose=true -u jdbc:ignite:thin://127.0.0.1/
issuing: !connect jdbc:ignite:thin://127.0.0.1/ '' '' org.apache.ignite.IgniteJdbcThinDriver
Connecting to jdbc:ignite:thin://127.0.0.1/
Connected to: Apache Ignite (version 2.3.0#20171028-sha1:8add7fd5)
Driver: Apache Ignite Thin JDBC Driver (version 2.3.0#20171028-sha1:8add7fd5)
Autocommit status: true
Transaction isolation: TRANSACTION_REPEATABLE_READ
sqlline version 1.3.0
0: jdbc:ignite:thin://127.0.0.1/> !tables
+--------------------------------+--------------------------------+--------------------------------+--------------------------------+--------------------------------+---------+
| TABLE_CAT | TABLE_SCHEM | TABLE_NAME | TABLE_TYPE | REMARKS | |
+--------------------------------+--------------------------------+--------------------------------+--------------------------------+--------------------------------+---------+
| | eventCache | EVENTCACHEPOJO | TABLE | | |
| | eventCache | EVENTCACHEPOJO | TABLE | | |
+--------------------------------+--------------------------------+--------------------------------+--------------------------------+--------------------------------+---------+
0: jdbc:ignite:thin://127.0.0.1/> select count(*) from "eventCache"."EVENTCACHEPOJO";
+--------------------------------+
| COUNT(*) |
+--------------------------------+
| 2619705 |
+--------------------------------+


Here are a few options to connect to a specific schema in Ignite:
jdbc:ignite:thin://172.16.0.5/mySchema - will connect to "MYSCHEMA" (case insensitive)
jdbc:ignite:thin://172.16.0.5/"mySchema" - will connect to "mySchema" (case sensitive)

Related

PostgreSQL script - drop table if exist and create table

I have a problem with PostgreSQL script runing with docker-compose. I want to create only ONE table in PostgreSQL.
My script is:
DROP TABLE IF EXISTS person;
CREATE TABLE person
(
id INT NOT NULL,
name VARCHAR(255),
surname VARCHAR(255),
age INT,
email VARCHAR(255) UNIQUE
);
Log from drocker-compose
Error message is:
postgres | Success. You can now start the database server using:
postgres |
postgres | pg_ctl -D /var/lib/postgresql/data -l logfile start
postgres |
postgres | waiting for server to start....2022-07-11 08:39:50.970 UTC [48] LOG: starting PostgreSQL 14.4 (Debian 14.4-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
postgres | 2022-07-11 08:39:50.978 UTC [48] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres | 2022-07-11 08:39:51.000 UTC [49] LOG: database system was shut down at 2022-07-11 08:39:50 UTC
postgres | 2022-07-11 08:39:51.008 UTC [48] LOG: database system is ready to accept connections
postgres | done
postgres | server started
postgres | CREATE DATABASE
postgres |
postgres |
postgres | /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/create-table-person.sql
postgres | 2022-07-11 08:39:51.431 UTC [62] ERROR: syntax error at or near "CREATE" at character 30
postgres | 2022-07-11 08:39:51.431 UTC [62] STATEMENT: DROP TABLE IF EXISTS person
postgres | CREATE TABLE person
postgres | (
postgres | id integer NOT NULL PRIMARY KEY,
postgres | name VARCHAR(255),
postgres | surname VARCHAR(255),
postgres | age integer,
postgres | email VARCHAR(255)
postgres | );
postgres | psql:/docker-entrypoint-initdb.d/create-table-person.sql:9: ERROR: syntax error at or near "CREATE"
postgres | LINE 2: CREATE TABLE person
postgres | ^
postgres exited with code 3
and my Dockerfile for PostgreSQL is:
FROM postgres
ENV POSTGRES_DB testdatabase1
COPY create-table-person.sql /docker-entrypoint-initdb.d/

How to read data from hive table using flink sql client?

I tried to read the data from hive table using the flink sql client as per the flink documentation but it failed. i can read the table meta information,but not the data.
here is my hive data:
0: jdbc:hive2://localhost:10000> create database testdb ;
No rows affected (2.048 seconds)
0: jdbc:hive2://localhost:10000> use testdb ;
No rows affected (0.106 seconds)
0: jdbc:hive2://localhost:10000> create table source (a bigint, b bigint) ;
No rows affected (1.026 seconds)
0: jdbc:hive2://localhost:10000> show tables ;
+-----------+--+
| tab_name |
+-----------+--+
| source |
+-----------+--+
1 row selected (0.877 seconds)
0: jdbc:hive2://localhost:10000> insert into source values (1, 1) ;
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. tez, spark) or using Hive 1.X releases.
No rows affected (7.498 seconds)
0: jdbc:hive2://localhost:10000> select a, b from source ;
+----+----+--+
| a | b |
+----+----+--+
| 1 | 1 |
+----+----+--+
here is my flink sql client:
Flink SQL> show catalogs ;
+-----------------+
| catalog name |
+-----------------+
| default_catalog |
| myhive |
+-----------------+
2 rows in set
Flink SQL> use catalog myhive ;
[INFO] Execute statement succeed.
Flink SQL> show databases ;
+---------------+
| database name |
+---------------+
| default |
| testdb |
+---------------+
2 rows in set
Flink SQL> use testdb ;
[INFO] Execute statement succeed.
Flink SQL> show tables ;
+------------+
| table name |
+------------+
| source |
+------------+
1 row in set
Flink SQL> SET sql-client.execution.result-mode=tableau;
[INFO] Session property has been set.
Flink SQL> select a, b from source ;
Empty set
I have added a new project to quickly reproduce this error: https://github.com/lianxmfor/flink-hive

Table issue in karate v1

After upgrade to karate v1.0.1, table data type has an issue, here is my scenario and error returned:
Scenario: table test
* table table1 =
| column |
| 'row1' |
| 'row2' |
* print table1
* print table1
>>>> js failed:
01: karate.log('[print]',table1)
<<<<
org.graalvm.polyglot.PolyglotException: ReferenceError: "table1" is not defined
- <js>.:program(Unnamed:1)
I downgraded to v0.9.6 to check and this issue did not occur

The = symbol is not required.
* table table1
| column |
| 'row1' |
| 'row2' |
* print table1
You are welcome to update the docs via an open-source PR.

Hive revoke the grant err by hive.security.authorization.createtable.role.grants ( SQL Standard Based Hive Authorization )

How to revoke the grant by hive.security.authorization.createtable.role.grants ?
Env:
host 172.31.10.119
port 50033
version apache-hive-2.3.5-bin
database tools
hive-site.xml
<property>
<name>hive.security.authorization.createtable.role.grants</name>
<value>da:select;</value>
</property>
<property>
<name>hive.users.in.admin.role</name>
<value>root,tools </value>
</property>
Start hiveserver2
/opt/hive/hive-bin/bin/hiveserver2 --hiveconf hive.server2.thrift.port=50033 --hiveconf hive.server2.webui.port=10003
create table
/opt/hive/hive-bin/bin/beeline -u jdbc:hive2://172.31.10.119:50033 -n tools
use tools;
create table test1 as select * from tools.test99 limit 10;
show grant on table tools.test1;
+-----------+--------------+------------+---------+-----------------+-----------------+------------+---------------+----------------+----------+
| database | table | partition | column | principal_name | principal_type | privilege | grant_option | grant_time | grantor |
+-----------+--------------+------------+---------+-----------------+-----------------+------------+---------------+----------------+----------+
| tools | test1 | | | da | ROLE | SELECT | true | 1565061852000 | tools |
+-----------+--------------+------------+---------+-----------------+-----------------+------------+---------------+----------------+----------+
revoke select on role da
set role damin;
revoke select on table tools.test1 from role da;
err log
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Cannot find privilege Privilege [name=SELECT, columns=null] for Principal [name=da, type=ROLE] on Object [type=TABLE_OR_VIEW, name=tools.test1] granted by tools

Beeline query output coming in JSON format instead of csv table

I an using a Beeline Query like below,the underlying data sitting in HDFS comes from a mainframe server. All I want is to execute a query and dump it to a csv (or any tabular format):
beeline -u 'jdbc:hive2://server.com:port/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;transportMode=binary' -–showHeader=false --outputformat=csv2 -e "SELECT * FROM tbl LIMIT 2;"> tables1.csv
My issues are:
The format is not clean, there are extra rows at top and bottom ;
It appears as JSOn and not a table.
Some numbers seem hexadecimal format.
+-----------------------------------------------------------------------------------------------------------------------------+
| col1:{"col1_a":"00000" col1_b:"0" col1_c:{"col11_a":"00000" col11_tb:{"mo_acct_tp":"0" col11_c:"0"}} col1_d:"0"}|
+-----------------------------------------------------------------------------------------------------------------------------+
I want a regular csv with column names on top and no nesting.

Please help us understanding your data in a better way.
Is your table has data like below when you run the select query either in beeline or hive?:
> select * from test;
+------------------------------------------------------------------------------------------------------------------------+--+
| test.col |
+------------------------------------------------------------------------------------------------------------------------+--+
| {"col1_a":"00000","col1_b":"0","col1_c":{"col11_a":"00000","col11_tb":{"mo_acct_tp":"0","col11_c":"0"}},"col1_d":"0"} |
+------------------------------------------------------------------------------------------------------------------------+--+
If yes, you might have to parse out the data from the Json Objects which would as below:
select
get_json_object(tbl.col, '$.col1_a') col1_a
, get_json_object(tbl.col, '$.col1_b') col1_b
, get_json_object(tbl.col, '$.col1_c.col11_a') col1_c_col11_a
, get_json_object(tbl.col, '$.col1_c.col11_tb.col11_c') col1_c_col11_tb_col11_c
, get_json_object(tbl.col, '$.col1_c.col11_tb.mo_acct_tp') col1_c_col11_tb_mo_acct_tp
, get_json_object(tbl.col, '$.col1_d') col1_d
from test tbl
INFO : Completed executing command(queryId=hive_20180918182457_a2d6230d-28bc-4839-a1b5-0ac63c7779a5); Time taken: 1.007 seconds
INFO : OK
+---------+---------+-----------------+--------------------------+-----------------------------+---------+--+
| col1_a | col1_b | col1_c_col11_a | col1_c_col11_tb_col11_c | col1_c_col11_tb_mo_acct_tp | col1_d |
+---------+---------+-----------------+--------------------------+-----------------------------+---------+--+
| 00000 | 0 | 00000 | 0 | 0 | 0 |
+---------+---------+-----------------+--------------------------+-----------------------------+---------+--+
1 row selected (2.058 seconds)
Then you can use this query in your command line to export results into a file.
>beeline -u 'jdbc:hive2://server.com:port/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;transportMode=binary' --showHeader=false --outputformat=csv2 -e "select
get_json_object(tbl.col, '$.col1_a') col1_a
, get_json_object(tbl.col, '$.col1_b') col1_b
, get_json_object(tbl.col, '$.col1_c.col11_a') col1_c_col11_a
, get_json_object(tbl.col, '$.col1_c.col11_tb.col11_c') col1_c_col11_tb_col11_c
, get_json_object(tbl.col, '$.col1_c.col11_tb.mo_acct_tp') col1_c_col11_tb_mo_acct_tp
, get_json_object(tbl.col, '$.col1_d') col1_d
from corpde_commops.test tbl;" > test.csv
If you need the column names in the file then turn the --showHeader=true
Final output would be:
>cat test.csv
col1_a,col1_b,col1_c_col11_a,col1_c_col11_tb_col11_c,col1_c_col11_tb_mo_acct_tp,col1_d
00000,0,00000,0,0,0
I clearly don't see anything wrong in your beeline statement.
If your data is not as above example, the solution might be in a different way..
All the best.

You have to do showHeader=true and you will get the desired result
beeline -u 'jdbc:hive2://server.com:port/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;transportMode=binary' -–showHeader=true --outputformat=csv2 -e "SELECT * FROM tbl LIMIT 2;"> tables1.csv
You can also try the table format, outputformat=table, this will not give csv as output but gives you a clean tabular structure like below:
+-----+---------+-----------------+
| id | value | comment |
+-----+---------+-----------------+
| 1 | Value1 | Test comment 1 |
| 2 | Value2 | Test comment 2 |
| 3 | Value3 | Test comment 3 |
+-----+---------+-----------------+