I am creating a table from a CSV file and was wondering where would I find a doc that will tell me what columns names are acceptable? I know you cant have "/" or spaces in the columns names.
What is the fastest way to clean a csv and turn it into a sql table?
There are a couple of recommendations:
Uvoid special characters (except for underscore)
Use CamelCase if you don't want to use unserscore
Set a naming convention for your company that you will enforce
Here are some useful articles:
General Naming Conventions
SQL Server Standards
A fast way would be to create the database and table in sql, or the platform you are using. Then create each column with type/restrictions in the order the CSV file is in. Then upload the CSV file.
Related
Is there a way to copy data from S3 into Snowflake without manually defining the columns beforehand?
I don't want to have to define the schema for the table in Snowflake OR the schema for which columns should be imported from S3. I want it to be schema-on-read, not schema-on-write.
I'm using a storage integration for access to the S3 external stage.
My question is a bit similar to this question, but I don't want to have to define any columns individually. If there's a way to just add on additional columns on the fly, that would solve my issue too.
We do not currently have schema inference for COPY. I am assuming that you already know about the variant column option for JSON but it will not give you full schematization.
https://docs.snowflake.net/manuals/user-guide/semistructured-concepts.html
Dinesh Kulkarni
(PM, Snowflake)
You need to use a third party tool that analyses your whole S3 data file in order to build an SQL schema from the data set in the file. Or maybe the tool is given access to the data source definition (which Snowflake hasn't) to make the job for the tool easier.
You might find snippets of Snowflake Stored Procedure code by searching around here at stackoverflow, that outputs schema definitions by eg. recursively flattening JSON data files.
If you want the import to be flexible, you need to use a flexible data format like JSON and a flexible SQL data type like VARIANT. This will work even if your data structures change.
If you want to use rigid formats like CSV or rigid SQL data types (most are rigid) then things get complicated. Rigid data are not flexible, and eg CSV files do not have any embedded type information, making for massive non-future-proof guesswork.
And maybe you are satisfied having all your columns end up as VARCHAR...
I would like to use csv to describe a database schema. I found examples but for a single table, is there a standardized specification (or ideas, tracks etc.) for describing multiple (and linked) tables ?
CSV file only can have data with delimiter ( one line for one row and field separated with another delimiter)
So if you store data from different table in the same CSV, all data will be added to only one table.
best way is create different csv or choose another format ( why not sql ?)
I want to read text file and store(using clob datatype) its data in table . I want to do string comparison on loaded data.
Loaded text file contains DDL scripts and i want to get segregation of new/modified tables, new/modified indexes and constraints.
This can be done as Tom suggested Ask tom article
Challenge i'm facing here is that, i have to get above details before running those scripts, otherwise i would have used some DDL trigger to audit schema changes.
My qustion is , is it feasible to do string comparison on large text ? or is there any better alternative. please share your views/ideas on this.
Example file
Create table table_one
Alter table table_two
create index index_table_one_idx table_one (column_one)
etc etc... 100s of statements
from above code i want to get table_one , table_two as modified tables, and index_table_one_idx as newly created index.
i want to achieve this by looking for 'create table','alter table' strings in large text file and get the table name using substring.
It is perfectly feasible to do string comparison on large text.
There are a couple of different approaches. One is to read the file line by line using UTL_FILE. The other word be to load it into a temporary CLOB and process it in chunks. Probably the second way is is the better option. Make sure to use the DBMS_LOB functions for string manipulation, because they will perform better. Find out more.
Your main problem is a specification one. You need to isolate all the different starting points for SQL statements. If your script just as CREATE, ALTER and DROP then it's not too difficult, depending on how much subsequent detail you need (extract object type? extract object name? etc) and what additional processing you need to do.
If your scripts contain DML statements the task becomes harder. If the DDL encompasses programmatic objects (TYPE, PACKAGE, etc) then it's an order of magnitude harder.
I need to get unload of group of tables in oracle 9i.
The unload should be a .txt file or .unl file.
I have googled and found that the script specifying the column names of the table.
As I have huge number columns in the tables, I don't want to specify column names while taking unload. Anybody can please give me a simple solution to take unload without using column names.
Why do you need to specify the column names?
./sqlldr_exp user/passdword#instance > emp.ctl
No column names explicitly required.
Do you mean you don't even want the columns mentioned in the loader file? It is my understanding that would not be possible: you will need to allow oracle to place the columns. Similar to views using select * from some_table: the columns actually become materialized so that if the underlying columns change the view has to be recreated.
I have lately learned what is dynamic sql and one of the most interesting features of it to me is that we can use dynamic columns names and tables. But I cannot think about useful real life examples. The only one that came into my mind is statistical table.
Let`s say that we have table with name, type and created_data. Then we want to have a table that in columns are years from created_data column and in row type and number of names created in years. (sorry for my English)
What can be other useful real life examples of using dynamic sql with column and table as parameters? How do you use it?
Thanks for any suggestions and help :)
regards
Gabe
/edit
Thx for replies, I am particulary interested in examples that do not contain administrative things or database convertion or something like that, I am looking for examples where the code in example java is more complicated than using a dynamic sql in for example stored procedure.
An example of dynamic SQL is to fix a broken schema and make it more usable.
For example if you have hundreds of users and someone originally decided to create a new table for each user, you might want to redesign the database to have only one table. Then you'd need to migrate all the existing data to this new system.
You can query the information schema for table names with a certain naming pattern or containing certain columns then use dynamic SQL to select all the data from each of those tables then put it into a single table.
INSERT INTO users (name, col1, col2)
SELECT 'foo', col1, col2 FROM user_foo
UNION ALL
SELECT 'bar', col1, col2 FROM user_bar
UNION ALL
...
Then hopefully after doing this once you will never need to touch dynamic SQL again.
Long-long ago I have worked with appliaction where users uses their own tables in common database.
Imagine, each user can create their own table in database from UI. To get the access to data from these tables, developer needs to use the dynamic SQL.
I once had to write an Excel import where the excel sheet was not like a csv file but layed out like a matrix. So I had to deal with a unknown number of columns for 3 temporary tables (columns, rows, "infield"). The rows were also a short form of tree. Sounds weird, but was a fun to do.
In SQL Server there was no chance to handle this without dynamic SQL.
Another example from a situation I recently came up against. A MySQL database of about 250 tables, all in MyISAM engine and no database design schema, chart or other explanation at all - well, except the not so helpful table and column names.
To plan for conversion to InnoDB and find possible foreign keys, we either had to manually check all queries (and the conditions used in JOIN and WHERE clauses) created from the web frontend code or make a script that uses dynamic SQL and checks all combinations of columns with compatible datatype and compares the data stored in those columns combinations (and then manually accept or reject these possibilities).