I wish to use an external Text/CSV file to read data and run an SQL Query. Is this possible without using the External_Table concept? I do not have write permissions in the DB, hence cannot create a temp table in the DB.
Basically, I have a list of employee numbers (around 100) in a text file, using which I wish to run the following query each time:
SELECT emp_record FROM emp_data WHERE emp_no = "#file-containing-number"
I have to run a series of tasks on these and they are in no particular order or sequence, but have been provided in that text file as a list.
I am using the TOAD client and have only read-only permissions on the DB I connect to.
When I do this sort of thing I will open the file in notepad, add a comma to the end of each line and use the following SQL query:
select emp_record FROM emp_data WHERE emp_no IN (
... Paste contents of file here.
)
No - based on the limitations you mention in your question.
Are you saying you cannot even insert these records into a table in the database? Who is imposing these restrictions? You have a job to do. Other support staff should help in providing a means to accomplish the job.
Related
I have a csv file with two columns. The file has over 200.000 rows. Inside database I have the same table with the same values.
How can I write a script so that I can search for the values that are present in file but not in database?
I am using SQL Developer for this
Creating an External table is the best option when you want to read the contents of a flat-file using a select query.
Click here to know more about how to create an external table.
After creating the external table, you can make use of a query similar to below to identify the records which are exclusively available in the external table(i.e. flat file).
select *
from new_external_table et
where not exists (select 1 from source_table st where et.column_name=st.column_name);
I'm trying to insert around 80 million records created using MATLAB into Vertica Database table. I wanted to know if we can call COPY LOCAL statement in MATLAB as a regular sql statement using exec(conn, sql). For test purpose, I tried with a dat file having around 4 million records as following:
sqlstmnt = 'COPY schema.table_name (FK_CUSTOMER_ID,FK_RUN_START_DATE_ID,FK_RUN_END_DATE_ID,FK_TRAVEL_ID,FK_ORIGIN_ID,FK_DEST_ID,FK_SEGMENT_ID,SEGMENT_PERCENTAGE,LAST_UPDATED) FROM LOCAL ''/my/file/full/path/test1.dat''';
results = exec(conn,sqlstmnt);
But it gave an error in results.Message like:
[Vertica]JDBC A ResultSet was expected but not generated from query "COPY schema.table_name(FK_CUSTOMER_ID,FK_RUN_START_DATE_ID,FK_RUN_END_DATE_ID,FK_TRAVEL_ID,FK_ORIGIN_ID,FK_DEST_ID,FK_SEGMENT_ID,SEGMENT_PERCENTAGE,LAST_UPDATED) FROM LOCAL '/my/file/full/path/test1.dat'". Query not executed.
I have the data in the '.dat' file in the order in which the columns are mentioned in COPY LOCAL.
I could not find any helpful resource explaining this error.
I have this test1.dat file which I'm able to insert using COPY from vsql but since I run my codes in MATLAB with many iterations,each iteration producing about a million records, I would want to insert them during each iteration. Any help will be really great.
COPY command return ResultSet that includes the amount of loaded data , i see two main options
1) results =exec(conn,sqlstmnt);
2)results = runsqlscript(conn,'nameOfSQLScriptthatIncludeTheCopyCommand.sql')
I hope you will find it useful
Thanks
I just finish reviewing you’re your input sample data .
i see major problem with the mapping of the input csv to the target table .
Main issues are :
1) Lines are broken into 2 lines ( you should prefer having one sample per line and avoid brock it into 2 lines )
Eg : "1,20150101,0,2,2573,2714,1,8.147237e-01
50,48,49,54,45,48,51,-28 12:11:46"
2) when you define data types on vertica table ,eg: timestamp the data on the csv must reflect to it ( what you have is "-28 12:11:46" , this will not work )
After you fix all this issues , make sure you test it using vsql , then go and try it with matlab
I hope you will find it useful.
So I have a db with a bunch of member info (7 million records) that I can pull info from. Then in a separate system, I have a list of 800K emials. What I want to do is match all those members in the db to this list of 800K emails. I don't have the ability to create a table in the db with these emails - I can only read from the db.
So my question is, what is the best way to do this? Can I write a sql statement that reads these 800k in to memory from a csv file and then does a lookup comparing against this list? What is the approach? I just want to pull all emmebr info for members who's info are in that external list...
Thanks
If you have a separate Oracle database available where you do have DDL access, you could try creating a DB Link between the two databases using your read-only account.
CREATE DATABASE LINK <DB Link Name>
CONNECT TO <Your read-only account>
IDENTIFIED by <password>
USING <ServiceName or TNS Entry of remote database>;
This gives you the ability to build queries in your own database but refer to the 7 million records table in the read-only database:
Select * from
emails e -- your local 800K emails table
Join customers#readonlydatabase c -- read-only table in restricted database
on e.uniqueid = c.uniqueid
Per the documentation, the prerequisites for this setup are:
Prerequisites
To create a private database link, you must have the CREATE DATABASE
LINK system privilege. To create a public database link, you must have
the CREATE PUBLIC DATABASE LINK system privilege. Also, you must have
the CREATE SESSION system privilege on the remote Oracle database.
Oracle Net must be installed on both the local and remote Oracle
databases.
Also note that a dblink access is only as good as the account which it uses to connect. So if your account is read-only then your dblink will retain the same restrictions.
Build a huge SQL statement like this:
select *
from member_data
join
(
--Add all the text information here.
--
--Up to 32767 values can be stored in this collection.
select column_value email_address from table(sys.odcivarchar2list(
'asdf1#asdf.com',
'qwer1#qwer.com'
--...
))
--Another 32767 values
union all
select * from table(sys.odcivarchar2list(
'asdf2#asdf.com',
'qwer2#qwer.com'
--...
))
--...
) other_system
on member_data.email = other_system.email_address;
It's ugly but it's not that hard to build and it doesn't require any additional privileges. With a few text processing tricks, maybe a regular expression in a text editor or using Excel to add single quotes and commas, that statement can probably be built in a few minutes. SQL statements this large are usually a bad idea but it will be fine for a one-time process.
Using a SQL tool like SQL Developer / Toad for Oracle
Is it possible to write a SQL query that will do the following
SELECT * FROM TABLE
WHERE COLUMN1 IN CSV_FILE
The CSV file is just one column of data with no delimiters.
How can I achieve this?
Constraints
I cannot create a temp table to insert CSV file (no create permissions)
The data I am using of this column is the only index in that table so I cannot use other columns to query or else it will be really slow.
Thanks
Creating external table is the best way. If you dont have permission then the other way is to move the file to the path of any oracle directory(Oracle object - Directory). And with help of utl_file read the file, loop through it and do your operation inside a PL/SQL block which is too tedious.
See the eaxmples for using utl_file - http://psoug.org/reference/utl_file.html
But its better if you try and get create access.
Toad for Oracle data import (uses sqlldr internally)
Create a temp table and load the data using this utility and select the values
External tables
Create external table, load the data through the same and select the values.
Using SQL developer you can create a table in your schema and load this table with data from a csv file.
Notes:
You will need to create a void column per each column to import from excel
Excel export csv with ";" delimiter
If SQL developer(4.1.5) doesn't preview the fields in separated columns try moving forward/backwards with Next/back buttons
and a very graphical guide in the following page:
http://www.thatjeffsmith.com/archive/2012/04/how-to-import-from-excel-to-oracle-with-sql-developer/
Hi I often have to insert a lot of data into a table. For example, I would have data from excel or text file in the form of
1,a
3,bsdf
4,sdkfj
5,something
129,else
then I often construct 6 insert statements in this example and run the SQL script. I found this was slow when I have to send thousands of small packages to server, it also causes extra overhead to the network.
What's your best way of doing this?
Update: I'm using ORACLE 10g.
Use Oracle external tables.
See also e.g.
OraFaq about external tables
What Tom thinks about external tables
René Nyffenegger's notes about external tables
A simple example that should get you started
You need a file located in a server directory (get familiar with directory objects):
SQL> select directory_path from all_directories where directory_name = 'JTEST';
DIRECTORY_PATH
--------------------------------------------------------------------------------
c:\data\jtest
SQL> !cat ~/.gvfs/jtest\ on\ 192.168.xxx.xxx/exttable-1.csv
1,a
3,bsdf
4,sdkfj
5,something
129,else
Create an external table:
create table so13t (
id number(4),
data varchar2(20)
)
organization external (
type oracle_loader
default directory jtest /* jtest is an existing directory object */
access parameters (
records delimited by newline
fields terminated by ','
missing field values are null
)
location ('exttable-1.csv') /* the file located in jtest directory */
)
reject limit unlimited;
Now you can use all the powers of SQL to access the data:
SQL> select * from so13t order by data;
ID DATA
---------- ------------------------------------------------------------
1 a
3 bsdf
129 else
4 sdkfj
5 something
Im not sure if this works in Oracle but in SQL Server you can use BULK INSERT sql statement to upload data from a txt or a csv file.
BULK
INSERT [TableName]
FROM 'c:\FileName.txt'
WITH
(
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n'
)
GO
Just make sure that the table columns correctly matches whats in the txt file. With a more complicated solution you may want to use a format file see the following:
http://msdn.microsoft.com/en-us/library/ms178129.aspx
There are alot of ways to speed this up.
1) Do it in a single transaction. This will speed things up by avoiding connection opening / closing.
2) Load directly as a CSV file. If you load data as a CSV file, the "SQL" statements aren't required at all. in MySQL the "LOAD DATA INFILE" operation accomplishes this very intuitively and simply.
3) You can also simply dump the whole file as text into a table called "raw". And then let the database parse the data on its own using triggers. This is a hack, but it will simplify your application code and reduce network usage.