I want to set a lot of dictionaries in my clickhouse server and some of them aren't just plain MySQL queries to get the existing values, for a few I need to do JOINs and WHERES, and the dictionary configuration in Clickhouse only allows me to tell which MySQL table it will read the data from.
Is it possible to set a custom MySQL query for it?
Other thing that would be helpful is to use ALIASES in the attributes names.. that way I wouldn't be force to use the MySQL column name later.
Thank you.
you can try use external shell script which run
mysql -u<user> -p<password> -h <host> -N -B -e "SELECT field AS field_alias... FROM table1 JOIN table2"
and try read this article
https://www.altinity.com/blog/2017/4/12/dictionaries-explained
Related
I need make an Sql query into Hql query.
select count(column_name) from user_tab_columns where table_name='EMP_TABLE';
i do not know how to make it into an hql query if any one know please assist me.
I'm pretty sure Hive doesn't have this kind of metadata information available in a nice tabular format. You could look into querying the metastore directly, but that's ugly, and not really what you want anyway.
If you just want to know the number of columns in a Hive table, you can do that through a shell script that calls hive, for example:
hive -S -e 'describe my_table' | wc -l
I am connecting to Hive via an ODBC driver from a .NET application. Is there a query to determine if a table already exists?
For example, in MSSQL you can query the INFORMATION_SCHEMA table and in Netezza you can query the _v_table table.
Any assistance would be appreciated.
Execute the following command : show tables in DB like 'TABLENAME'
If the table exists, its name will be returned, otherwise nothing will be returned.
This is done directly from hive. for more options see this.
DB is the database in which you want to see if the table exists.
TABLENAME is the table name you seek,
What actually happens is that Hive queries its metastore (depends on your configuration but it can be in a standard RDBMS like MySQL) so you can optionally connect directly to the same metastore and write your own query to see if the table exists.
There are two approaches by which you can check that:
1.) As #dimamah suggested, just to add one point here, for this approach you need to
1.1) start the **hiveserver** before running the query
1.2) you have to run two queries
1.2.1) USE <database_name>
1.2.2) SHOW TABLES LIKE 'table_name'
1.2.3) Then you check your result using Result set.
2.) Second approach is to use HiveMetastoreClient APIs, where you can directly use the APIs to check whether the table_name exist in a particular database or not.
For further help please go through this Hive 11
When programming on Hive by Spark SQL, you can use following method to check whether Hive table exists.
if (hiveContext.hql("SHOW TABLES LIKE '" + tableName + "'").count() == 1) {
println(tableName + " exists")
}
If someone is using shell script like me then my answer could be useful. Assume that your table is in the default namespace.
table=your_hive_table
validateTable=$(hive --database default -e "SHOW TABLES LIKE '$table'")
if [[ -z $validateTable ]]; then
echo "Error:: $table cannot be found"
exit 1
fi
If you're using SparkSQL you can do the following.
if "table_name" in sqlContext.tableNames("db_name"):
...do something
http://spark.apache.org/docs/2.1.0/api/python/pyspark.sql.html#pyspark.sql.SQLContext.tableNames
Code similar to below one can find in many of my Spark notebooks:
stg_table_exists = sqlCtx.sql("SHOW TABLES IN "+ stg_db)
.filter("tableName='%s'" % stg_tab_name) .collect()
(made two-liner for readability)
I wish Spark would have an API call to check the same.
If you're using a scala spark app and SparkSQL you can do the following
if spark.catalog.tableExists("tablename") {do something}
How can I tranfer data between different DB Servers, this is a daily job,
i.e:
Insert into ServerA..table1
select data from ServerB.table2.
(this is just an example, the real situation is we select data from many servers, and then do some join, then insert into the destination).
We can not use SSIS, we can not use linked server,
How can we do this?
btw, this is a daily job, and the data is huge.
A simple command line BCP script should work for you. For instance:
bcp AdventureWorks2012.Sales.Currency out Currency.dat -T -c -SServer1
bcp AdventureWorks2012.Sales.Currency in Currency.dat -T -c -SServer2
Here's more details
The Sync Framework might be worth a look : http://msdn.microsoft.com/en-us/sync/bb736753.aspx
Look at this question:
SQL backup version is incompatible with this server
The first options from my answer should work for your case
You can use C#.net SqlBulkCopy method.
My answer was converted into comment but I'm adding some more info.
I guess you are looking for this answer on SO:
What is the best way to auto-generate INSERT statements for a SQL Server table?
Once you have the code, just add USE your_databasename_where_to_copy_data at the begining, execute and voila
Edit:
As you want to do it on the fly, using code, try some of the solutions provided on this question on SO. Basically it is similar to your code proposal, with some few differences, as for example:
INSERT INTO bar..tblFoobar( *fieldlist* )
SELECT *fieldlist* FROM foo..tblFoobar
I'm working in Ubuntu with MySql and I also have Query Browser and Administrator installed, I'm not afraid of the command line either if it helps.
I want simply to be able to run a query and see a result set but then convert that result set into a series of commands that could be used to create the same rows in a table of an identical schema.
I hope the question makes sense, it's quite a simple problem and one that must have been solved but I can't for the life of me work out where this kind of conversion is made available.
Thanks in advance,
Gav
I think you need to use a command line utility mysqldump http://dev.mysql.com/doc/refman/5.1/en/mysqldump.html
if you want to dump one or more tables.
If you need to dump a result of an arbitrary query and restore it later, take a look on SELECT ... INTO OUTFILE and LOAD DATA INFILE( http://dev.mysql.com/doc/refman/5.0/en/load-data.html)
I do not know if I understood you at all but you can use a SELECT INTO statement.
SELECT *
INTO new_table_name
FROM old_tablename
WHERE ...
I usually use SQLDeveloper to browse the database, but I couldn't make it work with HSQLDB and I don't know which tables are already created…
I guess it's a vendor-specific question and not plain SQL, but the point is: how can I see the tables so I can drop/alter them?
The ANSI SQL92 standard for querying database metadata is contained within the INFORMATION_SCHEMA data structures.
I have no idea whether your database supports this or not, but try the following:
SELECT *
FROM INFORMATION_SCHEMA.TABLES
On further research, it appears that HSQLDB does support INFORMATION_SCHEMA, but with slightly non-standard naming.
All of the tables have SYSTEM_* prepended to them, so the above example would read
SELECT *
FROM INFORMATION_SCHEMA.SYSTEM_TABLES
I have no means of testing this, and the answer was found on sourceforge.
Awesome, thanks! Been scouring the Web for that info.
This will fetch only your tables' field info:
SELECT TABLE_NAME, COLUMN_NAME, TYPE_NAME, COLUMN_SIZE, DECIMAL_DIGITS, IS_NULLABLE FROM INFORMATION_SCHEMA.SYSTEM_COLUMNS WHERE TABLE_NAME NOT LIKE 'SYSTEM_%'
You can retrieve indexes, primary key info, all kinds of stuff from INFORMATION_SCHEMA.SYSTEM_TABLES.
Gotta love oo documentation :p
If you're on the command line, you may want to try the Hsqldb SqlTool, documented in the SqlTool Manual (hsqldb.org).
Put your database connection information in "~/sqltool.rc" and choose any DBNAME you want, substitute correct username and password, if known.
urlid DBNAME
url jdbc:hsqldb:/path/to/hsql/database
username SA
password
Install tool with: apt-get install hsqldb-utils (on Ubuntu)
Connect with hsqldb-sqltool DBNAME # on Ubuntu
Hint for other systems: java -jar YourHsqlJar.jar DBNAME
Show tables with: \dt
Show columns with: \d TABLENAME
Standard queries like: SELECT * FROM …;
Edit (append) last command with: :a
Quit with: \q
View special commands with: \? OR :?
Good luck!
Use the \dt command when you hit the >sql prompt in the command line for HSQLDB.
Check out DBVisualiser and SQuirreL SQL Client. Both of these have support for HSQLDB, and a GUI for editing/modifying/viewing the tables.
You run querying using hsql database manager, are you?
If you use this, below may give some hints:
Select your connection:
type: HSQL DATABASE ENGINE SERVER
Driver: jdbc.hsqldb.jdbcDriver
URL: jdbc:hsqldb:hsql://localhost/
Then, you will browse the database.