A bit of a vague title, I will explain.
I am writing an SQL script to create an insert statement for each row of a table in my database, purely to be able to apply that data back to another database.
Here is what I have at the moment:
SELECT 'INSERT INTO products (id,name,description) VALUES ('||ID||','''||name||''','''||description||''');' FROM products
And it works great, outputting this:
INSERT INTO products (id,name,description) VALUES (1,'Lorem','Ipsum');
INSERT INTO products (id,name,description) VALUES (2,'Lorem','Ipsum');
INSERT INTO products (id,name,description) VALUES (3,'Lorem','Ipsum');
INSERT INTO products (id,name,description) VALUES (4,'Lorem','Ipsum');
The problem is if one of the fields is empty that row will fail to produce an update script, in the output file the line is just blank. Obviously as there are 20+ fields, some optional this means that hardly any of my scripts are generated.
Is there a way to solve this issue?
pg_dump -a -U user1 -t products -f products.copy database1
and then:
psql -U user2 -d database2 -f products.copy
and you're done. It's also safer and faster.
In the case of NULL fields you can do something like
Select COALESCE(Name, '') from...
The coalesce function returns the first nonnull value in the list.
For truly blank fields (empty nvarchar for instance) I believe your script above will work.
Use the quote_nullable() function new in PostgreSQL 8.4. In addition to permitting NULL values, it retains your data types and protects you from Bobby Tables (SQL injections):
SELECT 'INSERT INTO products (id,name,description) VALUES (' ||
quote_nullable(ID) || ',' || quote_nullable(name) || ',' ||
quote_nullable(description) || ');' FROM products;
In older versions, you get the same behavior with coalesce() and quote_literal():
SELECT 'INSERT INTO products (id,name,description) VALUES (' ||
coalesce(quote_literal(ID), 'null') || ',' ||
coalesce(quote_literal(name), 'null') || ',' ||
coalesce(quote_literal(description), 'null') || ',' ||
');' FROM products;
I wrote a python script based on #intgr answer to construct the select statement.
It takes comma separated list of columns from stdin (use -).
I wanted to use sqlparse but I couldn't understand how to use that lib.
import fileinput
names = ' '.join(fileinput.input())
ns = [x.strip() for x in names.split(',')]
quoted = ['quote_nullable(' + x + ')' for x in ns]
insert = "SELECT 'INSERT INTO <TABLE> ( " + (', ').join(ns) + " ) VALUES(' || " + (" || ',' || ").join(quoted) + " || ');' FROM <TABLE>"
print insert
A gist of the script is here: https://gist.github.com/2568047
Related
I am trying to pull rows as columns in bigquery.
This is how my data looks like now:
This is how I want my data to look like:
PS: While I have shown only 3 values in column SUB_CLASS_DESC actual count is in 100s. Hence, I am looking to use Procedural language as per documentation here. I followed the example shared here in towardsdatascience.com and wrote below code, but unfortunately that doesn't work:
DECLARE DEPT_CLASS_SUB_CLASS STRING;
SET DEPT_CLASS_SUB_CLASS = (SELECT CONCAT('("', STRING_AGG(DISTINCT DEPT_CLASS_SUB_CLASS, '", "'), '")')
FROM `analytics-mkt-cleanroom.Workspace.HS_AF_SG_R12_800K_SAMPLE_SALES_11_TEST`
);
EXECUTE IMMEDIATE FORMAT("""
CREATE OR REPLACE TABLE `analytics-mkt-cleanroom.Workspace.HS_AF_SG_R12_800K_SAMPLE_SALES_PIVOTED_12_TEST` AS
SELECT * FROM
(SELECT HH_ID,DEPT_CLASS_SUB_CLASS,SALE_AMT
FROM `analytics-mkt-cleanroom.Workspace.HS_AF_SG_R12_800K_SAMPLE_SALES_11_TEST`
)
PIVOT
(SUM(SALE_AMT)
,FOR DEPT_CLASS_SUB_CLASS IN %s
)""",DEPT_CLASS_SUB_CLASS);
Error I am getting:
Error message suggests to declare before the execute block, and I am doing exactly that, but I don't understand why the error still persists.
I tried declaring variables DEPT_CLASS_SUB_CLASS in different ways but not successful yet. Could anyone please point out where I might be making the mistake.
Much appreciated!
Consider below approach
execute immediate (select '''
select *
from your_table
pivot (any_value(sale_amt) for replace(sub_class_desc, ' ', '_') in (''' || list || '''))
'''
from (
select string_agg(distinct "'" || replace(sub_class_desc, ' ', '_') || "'") list
from your_table
)
)
if applied to dummy data as in your question - output is
How can I save these results into a new pivoted table? Specifically where can I put my CREATE OR REPLACE TABLE?
execute immediate (select '''
create or replace table `your_project.your_dataset.pivot_table` as
select *
from your_table
pivot (any_value(sale_amt) for replace(sub_class_desc, ' ', '_') in (''' || list || '''))
'''
from (
select string_agg(distinct "'" || replace(sub_class_desc, ' ', '_') || "'") list
from your_table
)
);
DEPT_CLASS_SUB_CLASS variable should be placed before any other statement, not just before an execute block being referenced.
From your error message, you seems to declare a variable at [411:1] which means at 411 line. Kindly move it to the top of your script at line 1 and test it again.
you have kind of a PIVOTing problem. I wrote down some test query which do PIVOTing and list columns in an alphabetical order at the same time.
DECLARE sample_data ARRAY<STRUCT<HH_ID STRING, SUB_CLASS_DESC STRING, SALE_AMT FLOAT64>> DEFAULT [
('HHH_001', 'K&B FIXTURE/PLUMBING', 139.),
('HHH_001', 'PULLDOWN KITCHEN FAUCETS', 129.),
('HHH_001', 'TUBULAR REPAIR & REPLACE', 0.)
];
CREATE TEMP TABLE data AS
SELECT r.* REPLACE(TRANSLATE(SUB_CLASS_DESC, ' &/', '___') AS SUB_CLASS_DESC)
FROM UNNEST(sample_data) r
;
EXECUTE IMMEDIATE FORMAT ("""
SELECT *
FROM data
PIVOT (SUM(SALE_AMT) AS sale_amt FOR SUB_CLASS_DESC IN ('%s'));
""", (SELECT STRING_AGG(DISTINCT SUB_CLASS_DESC, "','" ORDER BY SUB_CLASS_DESC ASC) FROM data)
);
Query Result
I need a prepare an SQL script to be given to my production support team to run this in production. We have created a new column in the DB2 table. This column should be filled with the data by concatenating 3 column values of the same table in the same row.
To give a history, all the reason text which are entered in the front end are inserted into the request table by unstringing into 3 columns. Since they had limited length, we created a new column with increased length and going forward all insert will go into the new column. But we need to move all the existing data in the old 3 columns to his new one. S this SQL update is just an one time exercise.
Table: tab_request
We added new column to have a increased character length and to align with other table nomenclature. Now I need the script to update the column reasontext as below
update set field1 || .... should be your dml script. use coalesce() function to convert those null values to ''.
Update table1
set reasontext =
(coalesce(reason_1, '') || ' ' || coalesce(reason_2,'') || ' ' || coalesce(reason_3,''))
update tab_request set reasontext=CONCAT(reason_1,' ',reason_2,' ',reason_3)
If you want to avoid unnecessary spaces -- in the event that some reasons are NULL -- then you can use trim() with coalesce():
update table1
set reasontext = trim(coalesce(' ' || reason_1, '') ||
coalesce(' ' || reason_2, '') ||
coalesce(' ' || reason_3, '')
);
This is equivalent to concat_ws() available in some databases.
I'm trying to create a separate log table, made from insert trigger values using now. The table has a lot of columns, so I was wondering if it is possible to convert all values of the row to one string.
Sample working code with two columns:
CREATE TRIGGER `TEST` AFTER INSERT ON `table`
FOR EACH ROW BEGIN
INSERT INTO change_log (string)
VALUES (CONCAT("User: ", USER(), " Time: ", NOW(), " Insert: " , now.id, now.value));
END
You can get a list of columns from the catalog. You can (group) concatenate them to form a concatenation expression.
SELECT concat('concat(',
group_concat(concat('''',
column_name,
': '', ',
column_name)
SEPARATOR ', '),
')')
FROM information_schema.columns
WHERE table_schema = '<schema name>'
AND table_name = '<table name>';
(Replace <schema name> and <table name> with your table's name and its schema.)
That will give you an expression like concat('a: ', a, 'b: ', b, 'c: ', c) (if the column name were a, b and c).
Amend the result to include your meta data (user, time, etc.) and copy it into your trigger.
Or, if you want, you could also call the query from the catalog in your trigger. Amend it, so that in the trigger a string is built which holds the statement you want to execute and execute it with PREPARE ... EXECUTE ... DEALLOCATE PREPARE ... or EXECUTE IMMEDIATE ... (which is essentially shorthand for the former). More information on dynamic SQL in MariaDB can be found in "Prepared Statements ".
Use your application language to build a JSON string -- which is very similar to what you are doing, except that it also takes care of embedded quotes and backslashes.
I would like to add a keyword before and after each field value in Oracle.
For example if I am getting 123 as my ID, I would like to make it
Test123Test
Here is my query:
SELECT
CAST("ID" as varchar(10))
FROM
TABLENAME;
I have tried add + "Test" but it is giving me error.
Use || instead of + to concatenate strings in Oracle.
SELECT 'test' || CAST(ID as varchar(10)) || 'test'
FROM TABLENAME
Note that I removed the " around ID too, since you most likely won't need them and they can cause problems when it unintended strictly matches column names.
I have tried add + "Test" but it is giving me error.
Perhaps, + is used as concatenation in SQL Server. In Oracle, you could use the CONCAT function or the || operator.
The concat function is restricted to only two strings. You can have a look the concat function in the documentation http://docs.oracle.com/cd/B19306_01/server.102/b14200/functions026.htm.
Let's see an example using the operator -
SELECT 'test' || to_char(id) || 'test' new_id
FROM TABLENAME
I want to join few attributes in select statement as one for example
select id, (name + ' ' + surname + ' ' + age) as info from users
this doesn't work, how to do it?
I'm using postgreSQL.
Postgres uses || to concatenate strings; your query needs to be:
select id, (name || ' ' || surname || ' ' || age) as info from users
It should be the key just above the Enter key on a full keyboard, but on mine it's not a sold line - there's a break in the middle of it. You have to hold the Shift key to get the character (called a pipe, btw).
I believe the ANSI standard concatenation operator is: ||
SELECT id, name ||' ' || surname || ' ' || age AS info FROM users
It perfectly might be dependent of the database I would take a look for a concatenation function for the database you are running the select for.
Example. Mysql: CONCAT(name, surname, age).
You may need to cast the fields to a common type before concatenating. In T-SQL for example this would read.
Select id, Cast(name as varchar(50)) + ' ' + Cast(surname as varchar(50)) + ' ' +
Cast(age as varchar(3)) As info From Users
|| is used for this purpose.
Use it like
select name ||' ' ||surname ||' ' ||age as info from users
If you're using mysql or oracle then try CONCAT function:
SELECT id, CONCAT(name, ' ', surname, ' ', age) as info FROM users
That should work as is, but in general it is better not to do too much concatenation sql side if you can help it; rather return all the columns and concat them on the other end.
What are the data types you are using? You may need to CAST / CONVERT into (n)varchar
Make sure your data types are similar and convert any datatypes to string as necessary:
select id, (name + ' ' + surname + ' ' + convert(varchar(3),age)) as info from users
Works fine in most databases I know, although you probably have to CAST the age field to be TEXT. The exact method for doing this depends on the database you're using, which you did not specify.