I want to alter 1000s table in hive database, but some of their tables exits some doesn't. As I execute that .sql file, as soon as it found table not present, it exits from hive. so help me out to override or skip those queries whose table is not present in hive
Try this configuration parameter:
set hive.cli.errors.ignore=true;
After setting it to 'true', all commands in the script are executed, no matter how many failed.
See here: https://issues.apache.org/jira/browse/HIVE-1847
Related
I use liquibase tool to manage a postgres database. I work as the following :
I have a solution composed of different folders containing SQL scripts responsible for schema creation, tables creations, types creation, procedures creation, etc... Then, I have a ChangeLog file in xml format, containing the following informations :
includeAll path="#{Server.WorkingDirectory}#/02 - Schema" relativeToChangelogFile="false"
includeAll path="#{Server.WorkingDirectory}#/03 - Types" relativeToChangelogFile="false
includeAll path="#{Server.WorkingDirectory}#/04 - Tables" relativeToChangelogFile="false"
includeAll path="#{Server.WorkingDirectory}#/05 - Fonctions" relativeToChangelogFile="false"
includeAll path="#{Server.WorkingDirectory}#/06 - Stored Procedures" relativeToChangelogFile="false"
I run liquibase via command line :
liquibase --changeLogFile=$(Changelog.File.Name) --driver=$(Driver.Name) --classpath=$(Driver.Classpath) --url=$(BDD.URL) --username=$(BDD.Login) --password=$(BDD.Password) update
This enable Liquibase to take all the SQL scripts in the different folders listed in the changelogFile, compare it with the current database at url $(BDD.URL), and generate a delta script containing all the SQL queries to be executed to have a database corresponding to my solution.
This works well when I add new scripts (new tables or procedures) or modify existing scripts, my database is correctly updated by the command line, as expected. BUT it does not do anything when I delete a script from my solution.
To be more factual, here is what I want :
I have a SQL file containing the query "CREATE TABLE my_table" located in the folder "04 - Tables".
I execute the update command above, and it creates the table "my_table" in my database.
I finally do not want this table in my database any more. Thus I would like to simply remove the corresponding SQL script from my solution, and then run again the "update" command to simply remove my table in my database, generating automatically a "DROP TABLE my_table" by the liquibase "update" command. But this is not working as Liquibase doesn't record any change when I remove a sql file (whereas it does when I add or modify a file).
Does anyone know a solution to this ? Is there a specific command to drop an element when there is no "CREATE" query for this element, in a SQL solution ?
Many thanks in advance for you help :)
You will need to explicitly write a script to drop the table.
Other option is to rollback the change IF YOU HAVE Specified the Rollback SQL as part of your original SQL script.
There is a Pro Version option to rollback a single update , with free / community version, you can rollback last few changes in sequence
ex; I did "liquibase rollbackCount 5" will rollback the last 5 changes that were applied ONLY IF I HAD Coded the rollback sql needed as part of my script.
My Sql script sample that included the code to rollback is
--rollback drop TABLE test.user1 ; drop table test.cd_activity;
CREATE TABLE test.user1 (
user_type_id int NOT NULL
);
CREATE TABLE test.cd_activity (
activity_id Integer NOT NULL
userid int);
I currently have a .sql file with the likes of:
DROP VIEW IF EXISTS vw_example;
CREATE VIEW vw_example as
SELECT a FROM b;
When running this command as part of a flyway migration, if the view already exists, it fails, as if the create command is not waiting for the DROP IF EXISTS to finish.
I know SQL server has a GO type keyword. Is there a way to sort of tell cockroachdb to wait for the first command?
As per the issue mentioned in the link, it is better to have the drop and create scripts in different migration files as flyway runs each migration in a single transaction.
I created a Cloudera cluster for Impala.
Cloudera version: Cloudera Express 5.8.1
Impala version: 2.6.0-cdh5.8.0 RELEASE
If I run the following command via impala-shell:
create table test as select 1;
The following error is returned:
WARNINGS: Failed to open HDFS file for writing: hdfs://[DNhostname]:8020/user/hive/warehouse/test/_impala_insert_staging/[...]/[...].0.
Error(255): Unknown error 255
However, if I run:
create table test (testcol int);
insert into test select 1;
...The table is created without a hitch.
Any ideas on why the first statement might fail while the second set of commands would succeed, and what I could do to fix it? I might have messed something up with directory permissions, either locally or on HDFS, however I've set dfs.permissions to false to turn off HDFS permissions. I don't know what to check on the local folders to ensure the correct user(s) have the right permissions. In either case, I don't know why the permissions would cause the CREATE TABLE AS SELECT statement to fail but not CREATE, then INSERT.
I should also mention that DNhostname is the hostname of the HDFS datanode/impala daemon that I'm SSHed into, not the hostname of the namenode. This worries me because DNhostname was originally where my namenode was located; I moved it to a different host for reasons outside the scope of this question. Is it possible that CREATE TABLE AS SELECT is still expecting the namenode to be DNhostname for some reason?
You are creating a new table with the default DB path since you are not specifying a new path in your create statement. If you try to use another DB for this process, you most likely to be successful.
create database newdb
use newdb
create table test as select 1
This will prove the location of this DB is wrong in the metastore. Go to your metastore.dbs and find the ID of your database there. You need to set the location of your db correctly, like:
update <metastoreDB>.DBS set LOCATION = 'hdfs://NN_URI:8020/user/hive/warehouse' where id = id_of_your_db;'
What is the impact on running queries in Hive if i swap the partition using
ALTER TABLE user_data
PARTITION (name = 'ABC')
SET LOCATION = 'db/partitions/new';
Does this command wait until queries finished executing?
Hive translate your query into temporary Map/Reduce job and that job executed on behalf of your hive query.When you submit hive query it creates a Map/Reduce job based on your query and that job gets executed and you get a result from that job. But if you ALTER your hive query and change partition or anything during the execution of query, command will not wait to finish your running job, it will alter your table and you will get result from your previous query unless or until you kill your previous job.
Best way to understand this is try and run. Just submit your hive query and redirect it to store the result into file and then change the partition and again submit the query and redirect it to store the result into file. Verify the both output.
I needed to load 100,000 rows of data from an excel file into a temporary table that I created using "on commit preserve rows". But somehow the most efficient methods did not seem to populate the temporary table due to session issues?
I used Toad to Import Table Data and it showed that x amount of records are imported. But when I select from the temp table, it was empty. Then I generated a bunch of insert scripts and saved them in a notepad.sql and called it from toad editor using #/script/location/notepad.sql and hit F5. It ran and showed how many records were inserted. Again the temp table was somehow still empty. So, I decided to run a random insert script manually in the editor and it showed up in the temp table. I believe the methods that didn't work are not considered to be the same session?
I haven't try SQLLDR but I am assuming it will not work judging from the methods I tried. Can someone confirm? I can't access SQLLDR so I won't know.
Is there anyway to get this to work? I can't run the insert scripts manually. That will be time consuming and Toad can't take that many scripts at the same time.
Oracle temp tables created with ON COMMIT PRESERVE ROWS are session-specific, so the data put into them is only visible within a single session, and for the duration of that session. Toad may be creating a separate session for each window and thus data which is populated from one window/session isn't visible from another window/session. The fact that you can run an insert script and then select the data back suggests this may be the case if both operations were done from the same window. I expect you'd see the same behavior if you used SQL*Loader to load the tables because the load would run in one session and the data would be discarded when the session terminated. Best of luck.