I am trying to sqoop hive view to SQL server database however i'm getting "object not found error". Does sqoop export works for hive views?
sqoop export --connect 'jdbc:jtds:sqlserver:<Connection String>' --table 'tax_vw' --hcatalog-database default --hcatalog-table tax_vw --connection-manager org.apache.sqoop.manager.SQLServerManager --driver net.sourceforge.jtds.jdbc.Driver --username XXX --password YYY --update-mode allowinsert
INFO hive.metastore: Connected to metastore. ERROR tool.ExportTool:
Encountered IOException running export job: java.io.IOException:
NoSuchObjectException(message:default.tax_vw table not found)
Need help on this.
Unfortunately, this is not possible to do using sqoop export, even if --hcatalog-table specified, it works only with tables and if not in HCatalog mode, it supports only exporting from directories, also no queries are supported in sqoop-export.
You can load your view data into table:
create table tax_table as select * from default.tax_vw;
And use --hcatalog-table tax_table
Related
I am trying to import all tables from mysql schema to hive by using blow sqoop query:-
sqoop import-all-tables --connect jdbc:mysql://ip-172-31-20-247:3306/retail_db --username sqoopuser -P --hive-import --hive-import --create-hive-table -m 3
it is saying ,
18/09/01 09:24:52 ERROR tool.ImportAllTablesTool: Encountered IOException running import job: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory
hdfs://ip-172-31-35-141.ec2.internal:8020/user/kumarrupesh2389619/categories already exists
Run the below command
hdfs dfs -rmr /user/kumarrupesh2389619/categories
Your command is failing since the directory already exists.
Im using the below command for importing data from sqlserver to Azure blob storage
sqoop import -Dorg.apache.sqoop.splitter.allow_text_splitter=true --connect "jdbc:sqlserver://server-IP;database=database_name;username=user;password=password"
--username test --password "password" --query "select top 5 * from employ where \$CONDITIONS" --delete-target-dir --target-dir 'wasb://sample#workingclusterblob.blob.core.windows.net/source/employ'
-m 1
getting below error
18/01/30 03:35:45 INFO tool.ImportTool: Destination directory wasb://sample#workingclusterblob.blob.core.windows.net/source/employ is not present, hence not deleting.
18/01/30 03:35:45 INFO mapreduce.ImportJobBase: Beginning query import.
18/01/30 03:35:46 INFO client.AHSProxy: Connecting to Application History server at headnodehost/10.0.0.19:10200
18/01/30 03:35:46 ERROR tool.ImportTool: Encountered IOException running import job: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory wasb://sample#workingclusterblob.blob.core.windows.net/source/employ already exists
Logs statement are confusing which tells both as not present for deleting and present while writing.
From Apache Sqoop user guide:
By default, imports go to a new target location. If the destination
directory already exists in HDFS, Sqoop will refuse to import and
overwrite that directory’s contents. If you use the --append argument,
Sqoop will import data to a temporary directory and then rename the
files into the normal target directory in a manner that does not
conflict with existing filenames in that directory.
I'm not in the Azure environment to reproduce and validate the solution, But Try adding the --append to your sqoop import and let me know.
sqoop import -Dorg.apache.sqoop.splitter.allow_text_splitter=true --connect "jdbc:sqlserver://server-IP;database=database_name;username=user;password=password"
--username test --password "password" --query "select top 5 * from employ where \$CONDITIONS" --delete-target-dir --append --target-dir 'wasb://sample#workingclusterblob.blob.core.windows.net/source/employ'
-m 1
I am trying to load a table from mysql to hive using --hive-import in parquet format, We want to do incremental update of hive table. When we try below command. its failing with the error mentioned below. Can anybody please help here.
sqoop job --create users_test_hive -- import --connect 'jdbc:mysql://dbhost/dbname?characterEncoding=utf8&dontTrackOpenResources=true&defaultFetchSize=1000&useCursorFetch=true&useUnicode=yes&characterEncoding=utf8' --table users --incremental lastmodified --check-column n_last_updated --username username --password password --merge-key user_id --mysql-delimiters --as-parquetfile --hive-import --warehouse-dir /usr/hive/warehouse/ --hive-table users_test_hive.
Error while running it.
16/02/27 21:33:17 INFO mapreduce.Job: Task Id : attempt_1454936520418_0239_m_000000_1, Status : FAILED
Error: parquet.column.ParquetProperties.newColumnWriteStore(Lparquet/schema/MessageType;Lparquet/column/page/PageWriteStore;I)Lparquet/column/ColumnWriteStore;
I am trying to import table from mysql to hive then i am getting following error.
can you please provide the solution for this.
bin/sqoop import --connect jdbc:mysql://202.63.155.22:3306/demo --username careuser --P --table caremanager --hive-import --verbose -m 1
3/12/30 02:42:05 WARN hive.TableDefWriter:
Column createddate had to be cast to a less precise type in Hive
13/12/30 02:42:05 ERROR tool.ImportTool: Encountered IOException running import job:
java.io.IOException: Cannot run program "hive": error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1041)
Actually sqoop is trying to access hive for creating hive table but cant execute it. Is hive installed on your machine and is Hive home is set in PATH environment variable.Please verify it.Hope these will solve your problem.
I have successfully installed and data imported from MySql to the HDFS but I couldn't get the
data in the Hive. the corresponding data imported from the Mysql is found in Hdfs as comma separated format.
Older versions of sqoop using
sqoop import --connect jdbc:mysql://remote-ip/db --username xxx --password xxx --table tb --hive-import --hive-table foo.tb
In Sqoop2 i couldn't find an option to set the hive table options while creating the job or connectors
Please help...
Sqoop2 currently do not have integration with Hive and can import data only to HDFS. You can load them into Hive manually from there though (using LOAD DATA command). Or you can continue using Sqoop1 for the time being.