I am trying to update only one column of the 1 million records in a table based on the value in the CSV file.
Sample CSV file:
1,Apple
2,Orange
3,Mango
The first column in the file is the PK I will use to filter the record and the second column is the new value of the column in the table I want to update.The PK in the CSV file may or may not exist in the DB though. I was thinking of creating a script to make a million update statements based on the file. I would like to know if there are any better way on how could I do this?
personally i would
load the CSV file into a new table using sqlldr
make sure the correct indexes are on the new and existing table
write ONE update statement to update the existing table from the new
one
I would:
Create an external table using the csv
Update existing table from the new external table in just one update
Related
I have created a new table in SQLPro for Postgres, and I want to upload multiple CSV into that Table.
Each CSV has about 5K records. Basically, whenever I want to upload another one it deletes/overrides the information from the table.
Can you help? :)
Basically,
Merge all the CSV with the headers insert them into your table.
Delete all the rows that were created by headers.
Might be obvious, but remember it will only work with CSV where data is mapped the same way.
How to load a CSV file into the table using the console? The problem is that I have to somehow omit the headers from the CSV file (I can not delete them manually).
From the sqlite3 doc on CSV Import:
There are two cases to consider: (1) Table "tab1" does not previously
exist and (2) table "tab1" does already exist.
In the first case, when the table does not previously exist, the table
is automatically created and the content of the first row of the input
CSV file is used to determine the name of all the columns in the
table. In other words, if the table does not previously exist, the
first row of the CSV file is interpreted to be column names and the
actual data starts on the second row of the CSV file.
For the second case, when the table already exists, every row of the
CSV file, including the first row, is assumed to be actual content. If
the CSV file contains an initial row of column labels, that row will
be read as data and inserted into the table. To avoid this, make sure
that table does not previously exist.
It is either/or. You will have to outsmart it.
Assuming "I can not delete them manually" means from the csv, not from the table, you could possibly sql delete the header line after the import.
Or: Import into a temp table in the target database, insert into target table from the temp table, drop the temp table.
Or:
connect to an in-memory database
import the CSV into a table
attach the target database
insert into target table from the imported in-memory table
just add option --skip 1, see https://www.sqlite.org/cli.html#importing_csv_files
consider 2000 year data.
test.csv
country_code,product_code,rpt_period
us,crd,2000
us,pcl,2000
us,mtg,2000
in,crd,2000
in,pcl,2000
in,mtg,2000
now i am appending newly generated 2001 records to test.csv. after appending new data to test.csv my data looks like below.
append.csv
country_code,product_code,rpt_period
us,crd,2000
us,pcl,2000
us,mtg,2000
in,crd,2000
in,pcl,2000
in,mtg,2000
us,crd,2001
us,pcl,2001
us,mtg,2001
in,crd,2001
in,pcl,2001
in,mtg,2001
Below scenarios are possible in the hive? If yes, please answer questions.
How to create schema for Partition table Foo using this data?. and also I
want partition columns as country_code and product_code.
For instance, i want to load (from test.csv file records) to table Foo? using hive LOAD DATA comand ?
How to load append.csv (only 2001 records) to table Foo. this also needs to be done using hive LOAD DATA command
Thanks.
Yes, All the scenarios you have mentioned are possible with Hive.
Create temp table and load all the data you have and the you can create partitioned table with 2 columns you have mentioned.
For 2 and 3: Just the load command will work. If your intention is to load into partitioned table you have to go via creating temp table and insert into partitioned table.
Let me know this is what you want else update your question.
I am looking for a good generic SP in SQL server to load CSV files using the header row to create the necessary table columns for the temp table.
For example, I have a table with 20 columns in it, and I have been told to load a csv file into SQL server I have to load all the columns into the table. So, that means I have to go in and create all this by hand and then load the data. Think that is a time waster.
So, I was wondering if there was a way to read the header row use it to create the columns, whether my table has 10 or 20 and the use bulk update to load the data itself.
Suggestions?
I want to import new DUMP file to my old database which has same schema. I want to overwrite if there is anychange in record with dump file record I am importing and want to add new records. I dont want to delete the existing records in the DB I am gonna import.
If you create the dump file using mysqldump, you can give it the option --replace,which generates a dump file containing replace statements instead of insert statements.
When this dump file is loaded into MySQL, records that matches primary or unique keys in the database will replace the old ones, while records not matching existing keys will be inserted.