In PostgreSQL I previously created a table like so:
CREATE TABLE IF NOT EXISTS stock_data (
code varchar,
date date,
open decimal,
high decimal,
low decimal,
close decimal,
volume decimal,
UNIQUE (code, date)
);
The idea is to import multiple csv files into this table. My approach is to use COPY ... FROM STDIN instead of COPY ... FROM '/path/to/file', as I want to be able to cat from the shell multiple csv files and pipe them to the sql script. The sql script to accomplish this currently looks like this:
CREATE TEMPORARY TABLE IF NOT EXISTS stock_data_tmp (
code varchar,
ddate varchar,
open decimal,
high decimal,
low decimal,
close decimal,
volume decimal,
UNIQUE (code, ddate)
);
\copy stock_data_tmp FROM STDIN WITH CSV;
INSERT INTO stock_data
SELECT code, to_date(date, 'YYYYMMDD'), open, high, low, close, volume
FROM stock_data_tmp;
DROP TABLE stock_data_tmp;
An example csv file looks like this
AAA,20140102,21.195,21.24,1.16,1.215,607639
BBB,20140102,23.29,2.29,2.17,2.26,1863
CCC,20140102,61.34,0.345,0.34,0.34,112700
DDD,20140102,509.1,50.11,50.09,50.11,409863
From the shell I try:
cat /path/to/20140102.txt | psql -d my_db_name -f ~/path/to/script/update_stock_data.sql
But it gives me this error:
psql:/path/to/script/update_stock_data.sql:22: ERROR: missing data for column "date"
CONTEXT: COPY stock_data_tmp, line 1: ""
However, if in my script I change the COPY command to:
\copy stock_data_tmp FROM '/path/to/20140102.txt' WITH csv;
... and simply call
psql -d my_db_name -f ~/path/to/script/update_stock_data.sql
it succeeds.
Why am I getting this error when using cat and STDIN, and not when using the file PATH?
Because if you use -f, COPY will try to read from that file and not from stdin.
Related
I want to get the current date as YYMMDD and then set it to variable in order to use it as table name.
Here is my code:
set dates= date +%Y-%m-%d;
CREATE EXTERNAL TABLE IF NOT EXISTS dates(
id STRING,
region STRING,
city STRING)
But this method doesn't work, because it seems the assignments are wrong. Any idea?
Hive does not calculate variables, it substitutes them as is, in your case it will be exactly this string 'date +%Y-%m-%d'. Also it is not possible to use UDF like current_date() in place of table name in DDL.
The solution is to calculate variable in the shell and pass it to Hive:
In the shell
dates=$(date +%Y_%m_%d);
hive --hivevar date="$dates" -f myscript.hql
In the script:
use mydb; create table if not exists tab_${hivevar:date} (id int);
Or you can execute hive script from command line using hive -e, in this case variable can be substituted using shell:
dates=$(date +%Y_%m_%d);
hive -e "use mydb; create table if not exists tab_${dates} (id int);"
I am going through a tutorial on how to use DB2 which is in a linux environment.
I am supposed to connect to a database, create a table, insert some data under db2 shell:
db2 connect to c3421m
db2
db2 => update command options using z ON Assignment_0.txt
db2 => update command options using v ON
db2 => CREATE TABLE BAND_2015 // gives error I got stuck here
// here is where I get stuck i am supposed to create a table and execute the follwing command under DB2 shell: CREATE TABLE BAND_2015
Code given:
create table band_2015 ( \
band_no integer not null primary key, \
band_name varchar(25) not null, \
band_home varchar(25) not null, \
band_type varchar(10) check (band_type in (‘concert’,’rock’,’jazz’,’military’)), \
b_start_date date not null, \
band_contact varchar(10) not null )
So how do I create this table? I was told to copy it to a text editor(do i save it as band_2015.sql ?). I am completely new to this but i have a lot of experience in other programming languages...
The problem is the terminating character. By default in the carriage return (enter). However, for your tutorial, you should type multi-line commands. For this case, you change the terminator character by defining another
For semi-colon
db2 -t
select *
from table;
For at sign or any other character.
db2 -td#
select *
from table #
For no character:
db2
select * from table
In the DB2 command-line processor by default commands and statements cannot span multiple lines, so it treats CREATE TABLE BAND_2015 as a complete statement, which is of course not the case. In the code given to you those backslashes appear for a reason -- they indicate to the CLP that the statement continues on the next line.
Alternatively, you can start the CLP with the command line option -t, which will designate the semicolon, instead of the new line, as the statement terminator. You can then type the statement as you did, without the backslashes, and terminate it with the ";".
I am using the copy command to dump the data of a table in PostgreSQL to a txt file. I run the following command in PSQL:
\copy (select * from TableName) to 'C:\Database\bb.txt' with delimiter E'\t' null as '';
now in the bb.txt file, I see some special characters which are not there in the table itself. The Database has been configured with UTF8 encoding.
For example: when I run the above mentioned copy query, if the special character shows up in the column with ID=5. If I run the same copy query with (select * from tablename where ID=5), the special char is not there:
\copy (select * from TableName where ID=5) to 'C:\Database\bb.txt' with delimiter E'\t' null as '';
This happens on a Windows machine. Can someone tell me where these special characters are coming from?
I am using SQL LOADER to load multiple csv file in one table.
The process I found is very easy like
LOAD
DATA
INFILE '/path/file1.csv'
INFILE '/path/file2.csv'
INFILE '/path/file3.csv'
INFILE '/path/file4.csv'
APPEND INTO TABLE TBL_DATA_FILE
EVALUATE CHECK_CONSTRAINTS
REENABLE DISABLED_CONSTRAINTS
EXCEPTIONS EXCEPTION_TABLE
FIELDS TERMINATED BY ","
OPTIONALLY ENCLOSED BY '"'
TRAILING NULLCOLS
(
COL0,
COL1,
COL2,
COL3,
COL4
)
But I don't want to use INFILE multiple time cause if I have more than 1000 files then I have to mention 1000 times INFILE in control file script.
So my question is: is there any other way (like any loop / any *.csv) to load multiple files without using multiple infile?
Thanks,
Bithun
Solution 1: Can you concatenate the 1000 files into on big file, which is then loaded by SQL*Loader. On unix, I'd use something like
cd path
cat file*.csv > all_files.csv
Solution 2: Use external tables and load the data using a PL/SQL procedure:
CREATE PROCEDURE myload AS
BEGIN
FOR i IN 1 .. 1000 LOOP
EXECUTE IMMEDIATE 'ALTER TABLE xtable LOCATION ('''||to_char(i,'FM9999')||'.csv'')';
INSERT INTO mytable SELECT * FROM xtable;
END LOOP;
END;
You can use a wildcards (? for a single character, * for any number) like this:
infile 'file?.csv'
;)
Loop over the files from the shell:
#!/bin/bash
for csvFile in `ls file*.csv`
do
ln -s $csvFile tmpFile.csv
sqlldr control=file_pointing_at_tmpFile.ctl
rm tmpFile.csv
done
OPTIONS (skip=1)
LOAD DATA
INFILE /export/home/applmgr1/chalam/Upload/*.csv
REPLACE INTO TABLE XX_TEST_FTP_UP
FIELDS TERMINATED BY ','
TRAILING NULLCOLS
(FULL_NAME,EMPLOYEE_NUMBER)
whether it will check all the CSV and load the data or not
I'm trying to insert binary data into a blob using SQLite3's shell, which means regular SQL statements. Here's my table:
CREATE TABLE MYTABLE
(ID INTEGER,
BINDATA BLOB NOT NULL,
SOMEFK INTEGER REFERENCES OTHERTABLE(ID) NOT NULL,
PRIMARY KEY(ID)
);
And this is the kind of insert statement I'm trying:
INSERT INTO MYTABLE (BINDATA, SOMEFK)
VALUES (__READBINDATA('/tmp/somefile'), 1);
With __READBINDATA(file) being the function I am looking for. Is that possible?
There is no built-in or shell function to read a file into a blob.
However, with the help of the hexdump tool, it's possible to transform a file's contents into a blob literal:
echo "insert into mytable(bindata, somefk) " \
"values(x'"$(hexdump -v -e '1/1 "%02x"' /tmp/somefile)"', 1);"
This command can then be piped into the sqlite3 shell.