I have a current problem loading my .CSV file into Oracles SQL Database
I am using SQLLDR
I have an excel file that has a lot of stock information in it I will give you a sample of what it looks like
Tdate Symbol Open High Low Close Volume
19500103 SPX 16.66 16.66 16.66 16.66 1260000
19500104 SPX 16.85 16.85 16.85 16.85 1890000
19500105 SPX 16.93 16.93 16.93 16.93 2550000
Tdate , symbol , open , high , low , close and volume isnt in the .CSV file I just put it there because my database table will hold those values under those names.
I created my Table in Sql Developer
create table cts ( tdate date, symbol varchar(20), open numeric ( 18,8), high numeric (18,8), low ( numeric 18,8), close numeric (18,8) , volume int ) ;
So then I opened up a notepad file and created this
LOAD Data infile c:\cts.dump.csv
into table CTS
fields terminated by "," optionally enclosed by '"'
( tdate, symbol, open, high, low , close, volume)
I save it as loaderval.ctl in folder c:\data
I then proceed to open up my cmd window and type
sqlldr username/password control=c:\data\loaderval.ctl
I receive back that 64 lines have been committed which is impossible since the file has tons and tons of data. I then check my database and the table is empty.
I also receive a .bad file and the .bad file has the records from the first couple of rows of the excel sheet
( 19500103,SPX,16.66,16.66,16.66,16.66,1260000
19500104,SPX,16.85,16.85,16.85,16.85,1890000
19500105,SPX,16.93,16.93,16.93,16.93,2550000
19500106,SPX,16.98,16.98,16.98,16.98,2010000
19500109,SPX,17.08,17.08,17.08,17.08,2520000
19500110,SPX,17.03,17.03,17.03,17.03,2160000
19500111,SPX,17.09,17.09,17.09,17.09,2630000
19500112,SPX,16.76,16.76,16.76,16.76,2970000
19500113,SPX,16.67,16.67,16.67,16.67,3330000
19500116,SPX,16.72,16.72,16.72,16.72,1460000
19500117,SPX,16.86,16.86,16.86,16.86,1790000
19500118,SPX,16.85,16.85,16.85,16.85,1570000
19500119,SPX,16.87,16.87,16.87,16.87,1170000
19500120,SPX,16.90,16.90,16.90,16.90,1440000
19500123,SPX,16.92,16.92,16.92,16.92,1340000
19500124,SPX,16.86,16.86,16.86,16.86,1250000
19500125,SPX,16.74,16.74,16.74,16.74,1700000
19500126,SPX,16.73,16.73,16.73,16.73,1150000
19500127,SPX,16.82,16.82,16.82,16.82,1250000
19500130,SPX,17.02,17.02,17.02,17.02,1640000
19500131,SPX,17.05,17.05,17.05,17.05,1690000
19500201)
Please help :)
Looking at code it seems that date column may be the culprit here. You can check below link to how to handle dates for sql loader
https://oracle-base.com/articles/12c/sql-loader-enhancements-12cr1
LOAD DATA
INFILE c:\cts.dump.csv
INTO TABLE CTS
FIELDS CSV WITH EMBEDDED
(tdate DATE "YYYYDDMM" ":tdate",
symbol,
open,
high,
low,
close,
volumn)
$ sqlldr userid=userid/passwd#connect_string control=test.ctl
Related
I have 4 csv files each having 500,000 rows. I am trying to import the csv data into my Exasol databse, but there is an error with the date column and I have a problem with the first unwanted column in the files.
Here is an example CSV file:
unnamed:0 , time, lat, lon, nobs_cloud_day
0, 2006-03-30, 24.125, -119.375, 22.0
1, 2006-03-30, 24.125, -119.125, 25.0
The table I created to import csv to is
CREATE TABLE cloud_coverage_CONUS (
index_cloud DECIMAL(10,0)
,"time" DATE -- PRIMARY KEY
,lat DECIMAL(10,6)
,lon DECIMAL(10,6)
,nobs_cloud_day DECIMAL (3,1)
)
The command to import is
IMPORT INTO cloud_coverage_CONUS FROM LOCAL CSV FILE 'D:\uni\BI\project 1\AOL_DB_ANALYSIS_TASK1\datasets\cloud\cfc_us_part0.csv';
But I get this error:
SQL Error [42636]: java.sql.SQLException: ETL-3050: [Column=0 Row=0] [Transformation of value='Unnamed: 0' failed - invalid character value for cast; Value: 'Unnamed: 0'] (Session: 1750854753345597339) while executing '/* add path to the 4 csv files, that are in the cloud database folder*/ IMPORT INTO cloud_coverage_CONUS FROM CSV AT 'https://27.1.0.10:59205' FILE 'e12a96a6-a98f-4c0a-963a-e5dad7319fd5' ;'; 04509 java.sql.SQLException: java.net.SocketException: Connection reset by peer: socket write error
Alternatively I use this table (without the first column):
CREATE TABLE cloud_coverage_CONUS (
"time" DATE -- PRIMARY KEY
,lat DECIMAL(10,6)
,lon DECIMAL(10,6)
,nobs_cloud_day DECIMAL (3,1)
)
And use this import code:
IMPORT INTO cloud_coverage_CONUS FROM LOCAL CSV FILE 'D:\uni\BI\project 1\AOL_DB_ANALYSIS_TASK1\datasets\cloud\cfc_us_part0.csv'(2 FORMAT='YYYY-MM-DD', 3 .. 5);
But I still get this error:
SQL Error [42636]: java.sql.SQLException: ETL-3052: [Column=0 Row=0] [Transformation of value='time' failed - invalid value for YYYY format token; Value: 'time' Format: 'YYYY-MM-DD'] (Session: 1750854753345597339) while executing '/* add path to the 4 csv files, that are in the cloud database folder*/ IMPORT INTO cloud_coverage_CONUS FROM CSV AT 'https://27.1.0.10:60350' FILE '22c64219-cd10-4c35-9e81-018d20146222' (2 FORMAT='YYYY-MM-DD', 3 .. 5);'; 04509 java.sql.SQLException: java.net.SocketException: Connection reset by peer: socket write error
(I actually do want to ignore the first column in the files.)
How can I solve this issue?
Solution:
IMPORT INTO cloud_coverage_CONUS FROM LOCAL CSV FILE 'D:\uni\BI\project 1\AOL_DB_ANALYSIS_TASK1\datasets\cloud\cfc_us_part0.csv' (2 .. 5) ROW SEPARATOR = 'CRLF' COLUMN SEPARATOR = ',' SKIP = 1;
I did not realise that mysql is different from exasol
Looking at the first error message, a few things stand out. First we see this:
[Column=0 Row=0]
This tells us the problem is with the very first value in the file. This brings us to the next thing, where the message even tells us what value was read:
Transformation of value='Unnamed: 0' failed
So it's failing to convert Unnamed: 0. You also provided the table definition, where we see the first column in the table is a decimal type.
This makes sense. Unnamed: 0 is not a decimal. For this to work, the CSV data MUST align with the data types for the columns in the table.
But we also see this looks like a header row. Assuming everything else matches we can fix it by telling the database to skip this first row. I'm not familiar with Exasol, but according to the documentation I believe the correct code will look like this:
IMPORT INTO cloud_coverage_CONUS
FROM LOCAL CSV FILE 'D:\uni\BI\project 1\AOL_DB_ANALYSIS_TASK1\datasets\cloud\cfc_us_part0.csv'
(2 FORMAT='YYYY-MM-DD', 3 .. 5)
ROW SEPARATOR = 'CRLF'
COLUMN SEPARATOR = ','
SKIP = 1;
I have a txt file containing numerous items in the following format
DBSERVER: HKSER
DBREPLICAID: 51376694590
DBPATH: redirect.nsf
DBTITLE: Redirect AP
DATETIME: 09.03.2015 09:44:21 AM
READS: 1
Adds: 0
Updates: 0
Deletes: 0
DBSERVER: HKSER
DBREPLICAID: 21425584590
DBPATH: redirect.nsf
DBTITLE: Redirect AP
DATETIME: 08.03.2015 09:50:20 PM
READS: 2
Adds: 0
Updates: 0
Deletes: 0
.
.
.
.
please see the source capture here
I would like to import the txt file into the following format in SQL
1st column 2nd column 3rd column 4th column 5th column .....
DBSERVER DBREPLICAID DBPATH DBTITLE DATETIME ......
HKSER 51376694590 redirect.nsf Redirect AP 09.03.2015 09:44:21 AM
HKSER 21425584590 redirect.nsf Redirect AP 08.03.2015 01:08:07 AM
please see the output capture here
Thanks a lot!
You can dump that file into a temporary table, with just a single text column. Once imported, you loop through that table using a cursor, storing into variables the content, and every 10 records inserting a new row to the real target table.
Not the most elegant solution, but it's simple and it will do the job.
Using Bulk insert you can insert these headers and data in two different columns and then using dynamic sql query, you can create a table and insert data as required.
For Something like this I'd probably use SSIS.
The idea is to create a Script Component (As a Transformation)
You'll need to manually define your Output cols (Eg DBSERVER String (100))
The Src is your File (read Normally)
The Idea is that you build your rows line by line then add the full row to the Output Buffer.
Eg
Output0Buffer.AddRow();
Then write the rows to your Dest.
If all files have a common format then you can wrap the whole thiing in a for each loop
Unfortunately I have had issues with my storage and was forced to reacquire data. However, this came in many .csv files and don't know how to import all of them without doing it one by one. I would like to have the 10000+ .csv files into one table and would like help with coding all imports one time.
All of the files have the same schema:
'Symbol' (varchar(15))
'Date' (Date)
'Open' (Float)
'High' (Float)
'Low' (Float)
'Close' (Float)
'Volume' (Int)
Also: All files will have the same structure for their naming:
XXXXXX_YYYYMMDD
(XXXXXX is the name of the market; I have 7 unique names)
Create Table [investment data 1].dbo.AA
(
Symbol varchar(15),
[Date] Date,
[Open] Float,
High Float,
Low Float,
[Close] Float,
Volume Int
)
At this point I do not know how to generate a loop that will look at all files in the "Investment Data" folder; the below example is the sample code for one .csv file. If there is a better way than "bulk insert" then I will modify the statement below.
bulk insert [investment data 1].dbo.AA
from 'R:\Investment Data\NASDAQ_20090626.csv'
with
(
firstrow=2
,rowterminator = '\n'
,fieldterminator = ','
)
Any help is appreciated; if I can be more clear please let me know. Thanks for your time.
Does what you wrote (for that one file) work ?
Great.
Open a dos prompt
Navigate to the folder with your 10,000 files
type DIR /b >c:\temp\files.txt
Now install a decent text editor, like Notepad++ (these instructions are for notepad ++)
Open c:\temp\files.txt in that editor
Open the find/replace dialog, place a tick next to "Extended (\n, \r..." - this makes it match newlines, and support newlines in replacements
Put this in Find: \r\n
Put this in Replace: ' with(firstrow=2,rowterminator = '\\n',fieldterminator = ',');\r\nbulk insert [investment data 1].dbo.AA from 'R:\Investment Data\
This will make your list of files that used to look like this:
a.txt
b.txt
c.txt
d.txt
Look like this:
a.txt' with(firstrow=2,rowterminator = '\n',fieldterminator = ',')
bulk insert [investment data 1].dbo.AA from 'R:\Investment Data\b.txt' with(firstrow=2,rowterminator = '\n',fieldterminator = ',');
bulk insert [investment data 1].dbo.AA from 'R:\Investment Data\c.txt' with(firstrow=2,rowterminator = '\n',fieldterminator = ',');
bulk insert [investment data 1].dbo.AA from 'R:\Investment Data\d.txt' with(firstrow=2,rowterminator = '\n',fieldterminator = ',');
bulk insert [investment data 1].dbo.AA from 'R:\Investment Data\
Now just clean up the first and last lines so it's a proper SQL. Paste and run in SSMS
Seeking an solution to the following problem.
In our application we were having an functionality of converting the units as per user selection. e.g . Convert kg to lbs or vice versa based on unit selection by user.
We have a screen where in user can select the unit in which he is providing the value e.g Kg or lbs
We have a table which contains all the formulas for the predefined unit which our system supports e.g 1Kg = 2.20462lbs like so
We fetch the formula from the database and calculate / evaluate the formula with actual values as provided by the user.
Since , user have ability to see any units as per his/her choice. we decided to save the value which user provided on screen in standard unit only in our database for easier maintainablity . in this case let's say we have decided to save in kg only database however , user may have selected 'lbs' on screen , but while saving we do convert the 'lbs' to 'kg' and then we will save to database
while reading from database we do convert to 'lbs' if user intended to see in 'lbs' on screen
Hope till now the implementaton is clear , now the problem to above solution is
Let's assume user have selected value '1' as 'lbs' and saved it
In database we will save in 'kg' so value of '1' is saved as 2.20462 in database
while reading again we will do convert to show in 'lbs' now we get value as 0.999998. this is expected due to precision in formula
But user asking to atleast show the same value what he as entered , in this case '1' but we have to do calculation to convert it back to 'lbs' since we are saving in 'kg' in database
I am seeking an solution to this problem what could be better solution with minimal changes.
Table Definition
Id | UnitConversion
1 | 100
2 | 200
3 | 30
Our application is developed on
Angularjs, webapi and sql server
Just for fun, here is a stripped down version of my conversion utility. Just showing MASS here. By storing the conversion factors as varchar, precision is at the individual record level.
Declare #Map table (MapType varchar(50),MapName varchar(50),MapValue varchar(50))
Insert Into #Map values
('Mass','tonnes (metric)','1000'),
('Mass','tons (US)' ,'907.18474'),
('Mass','tons (UK)' ,'1016.0469088'),
('Mass','stones' ,'6.35029318'),
('Mass','slugs(g-pounds)','14.593903'),
('Mass','Solar masses' ,'1.989e30'),
('Mass','pounds (troy)' ,'0.3732417216'),
('Mass','pounds' ,'0.45359237'),
('Mass','picograms' ,'1e-15'),
('Mass','ounces' ,'0.028349523'),
('Mass','ounces (troy)' ,'0.0311034768'),
('Mass','nanograms' ,'1e-12'),
('Mass','milligrams' ,'1e-6'),
('Mass','micrograms' ,'1e-9'),
('Mass','megatonnes' ,'1e9'),
('Mass','kilotonnes' ,'1e6'),
('Mass','kilograms' ,'1'), --- << Base
('Mass','hundredweights' ,'50.80234544'),
('Mass','hectograms' ,'0.1'),
('Mass','grams' ,'1e-3'),
('Mass','grains' ,'0.00006479891'),
('Mass','femtograms' ,'1e-18'),
('Mass','Earth masses' ,'5.980e24'),
('Mass','decagrams' ,'0.01'),
('Mass','cental' ,'45.359237'),
('Mass','carats (metric)','0.0002')
Declare #Value float = 1
Declare #From varchar(50)= 'kilograms'
Declare #To varchar(50)= 'pounds'
Select #Value * Max(IIF(MapName=#From,cast(MapValue as float),null)) / Max(IIF(MapName=#To,cast(MapValue as float),null))
From #Map
Where MapName in(#From,#To)
Returns
2.20462262184878
At Sql server side choose a finer scale then that you need to return to a client. For example, if you need 5 digits to the right of the decimal point, set scale to 7 for DB column
declare #n decimal(20,5) = 2.20462;
declare #t table (
m decimal(20,7)
);
insert #t(m) values
(1./#n),
(10./#n),
(1000./#n);
select cast(m*#n as decimal(20,5)) as r
from #t;
Note "Decimal and numeric are synonyms and can be used interchangeably." msdn.microsoft.com/en-us/library/ms187746.aspx
I am new to apache hive and was running queries on sample data which is saved in a csv file as below:
0195153448;"Classical Mythology";"Mark P. O. Morford";"2002";"Oxford University Press";"//images.amazon.com/images/P/0195153448.01.THUMBZZZ.jpg";"http://images.amazon.com/images/P/0195153448.01.MZZZZZZZ.jpg";"images.amazon.com/images/P/0195153448.01.LZZZZZZZ.jpg"
and the table which i created is of form
hive> describe book;
OK
isbn bigint
title string
author string
year string
publ string
img1 string
img2 string
img3 string
Time taken: 0.085 seconds, Fetched: 8 row(s)
and the script which I used to create the table is:
create table book(isbn int,title string,author string, year string,publ string,img1 string,img2 string,img3 string) row format delimited fields terminated by '\;' lines terminated by '\n' location 'path';
When I try to retrieve the data from the table by using the following query:
select *from book limit 1;
I get the following result:
NULL "Classical Mythology" "Mark P. O. Morford" "2002" "Oxford University Press" "http://images.amazon.com/images/P/0195153448.01.THUMBZZZ.jpg" "images.amazon.com/images/P/0195153448.01.MZZZZZZZ.jpg" "images.amazon.com/images/P/0195153448.01.LZZZZZZZ.jpg"
Even though I specify the first column type as int or bigint the data into the table is getting loaded as NULL.
I tried searching on the internet and could figure out that I have to specify the row delimiter. I used that too but no change in the data from the table.
Is there anything that I am making a mistake... Please help.