I am trying to load all the tables in "ABCD_BKP" which starts with TEST_
The tables in my database are as follows:
ABCD_BKP
TEST_1
TEST_2
TEST_3
And I am trying to load it as per below but it does not seem to work.
SELECT *
FROM "ABCD_BKP".dbo.TEST_*
To load all tables you need to have list with the tables first and loop through this list and load the tables one by one.
For example if you are using MSSQL your script will be:
// Get all tables in "ABCD_BKP"
TableNames:
SQL
SELECT
TABLE_NAME
FROM
"ABCD_BKP".dbo.Tables
;
// Filter only table names that are starting with "TEST_"
Test_TableNames:
LOAD DISTINCT
TABLE_NAME
RESIDENT
TableNames as TestTables
WHERE
LEFT(TABLE_NAME, 5) = 'TEST_'
;
DROP TABLE TableNames; // the table with all table names is no longer needed
FOR i = 1 TO FieldValue('TestTables') // loop through all "TEST_*" tables
LET vTableName = FieldValue( 'TestTables', $(i) ); // current iteration table name
$(vTableName): //give our QV table the same name as the SQL table
SQL
SELECT
*
FROM
"ABCD_BKP".dbo.$(vTableName) // load the sql table in QV
;
NEXT
DROP TABLE Test_TableNames; // drop the QV table that contains the list with the "TEST_" tables
The sql to get the list with the tables in database is different for each database
Related
I have a CSV file in blob storage with the following format:
**Column,DataType**
Acc_ID, int
firstname, nvarchar(500)
lastname, nvarchar(500)
I am trying to read this file in data factory and loop through the column names and check the destination table if these columns already exits, if not I want to create the missing columns in the SQL table.
I know that we can use the following SQL query to create columns that do not exist.
IF NOT EXISTS (
SELECT
*
FROM
INFORMATION_SCHEMA.COLUMNS
WHERE
TABLE_NAME = 'contact_info' AND COLUMN_NAME = 'acc_id')
BEGIN
ALTER TABLE contact_info
ADD acc_id int NULL
END;
But I am not sure if we can read the CSV file and pass the column names from the CSV file to the above SQL query in a data factory pipeline. Any suggestions for this please?
You can create a column if not exist using the Pre-copy script in the Copy data activity.
• Table columns before executing the pipeline.
SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = 'contact_info'
• Source file:
ADF pipeline:
Using the lookup activity, get the list columns and datatypes by connecting the source dataset to the source file.
Output of lookup activity:
Connect the lookup output to the ForEach activity to loop all the values from the lookup.
#activity('Lookup1').output.value
Add Copy data activity inside ForEach activity and connect the source to the SQL table. Select query instead of a table in Use query properties. Write a query that does not result in any result as we are using this copy activity only to add a column to the table if not exist.
select * from dbo.contact_info where 1= 2
In the Copy data activity sink, connect the sink dataset to the SQL table, and in the Pre-copy script write your query to add a new column. Here use the current ForEach loop items (column, datatype) values instead of hardcoding the values as below.
#{concat('IF NOT EXISTS ( SELECT * FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = ','''','contact_info','''',' AND COLUMN_NAME = ','''',item().Column,'''',') ALTER TABLE contact_info ADD ',item().Column,' ', item().DataType,' NULL')}
When the pipeline is executed, the FoEach loop executes till it completes all the values in the lookup output and creates a new column in the table if not exist.
Columns in the table after the pipeline is executed:
I have about 200 tables in a schema.
I need to replicate these tables in a new backup schema with an automatic procedure.
I would like to create a procedure to dynamically recreate all the Tables in a Schema (potentially dynamic number of tables and columns) on a different schema.
I can cycle all the tables and create the SELECT * INTO dbo_b.TABLE FROM dbo.TABLE statement, but I get the error:
Column 'AMBIENTE' has a data type that cannot participate in a columnstore index.
I created a view that simply SELECT * FROM TABLE, and tried to perform the SELECT * INTO dbo_b.TABLE from dbo.VIEW but I got the same issue.
It works only if I create the dbo_b.Table and INSERT INTO it: so I would need to generate a script to automatically cycle all the tables in my schema and generate a script to create the tables in the new schema.
It's not a one time job, it should run every day so I cannot do it manually.
Seams we get the same issue.
You can try to loop on all table and create table in the new schema in this way:
IF EXISTS(SELECT * FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_NAME = 'YYYY' AND TABLE_SCHEMA = 'XXXX')
drop table [ZZZZ].[YYYY]
CREATE TABLE [ZZZZ].[YYYY]
WITH ( DISTRIBUTION = ROUND_ROBIN
, HEAP ) as
( SELECT * FROM XXXX.YYYY )
Let me know. BR
I want to create a table in Hive using a select statement which takes a subset of a data from another table. I used the following query to do so :
create table sample_db.out_table as
select * from sample_db.in_table where country = 'Canada';
When I looked into the HDFS location of this table, there are no field separators.
But I need to create a table with filtered data from another table along with a field separator. For example I am trying to do something like :
create table sample_db.out_table as
select * from sample_db.in_table where country = 'Canada'
ROW FORMAT SERDE
FIELDS TERMINATED BY '|';
This is not working though. I know the alternate way is to create a table structure with field names and the "FIELDS TERMINATED BY '|'" command and then load the data.
But is there any other way to combine the two into a single query that enables me to create a table with filtered data from another table and also with a field separator ?
Put row format delimited .. in front of AS select
do it like this
Change the query to yours
hive> CREATE TABLE ttt row format delimited fields terminated by '|' AS select *,count(1) from t1 group by id ,name ;
Query ID = root_20180702153737_37802c0e-525a-4b00-b8ec-9fac4a6d895b
here is the result
[root#hadoop1 ~]# hadoop fs -cat /user/hive/warehouse/ttt/**
2|\N|1
3|\N|1
4|\N|1
As you can see in the documentation, when using the CTAS (Create Table As Select) statement, the ROW FORMAT statement (in fact, all the settings related to the new table) goes before the SELECT statement.
Bit of a SQL newbie so hopefully explain this ok:
So I have a USCars and a USCars_AUD (for Audit) table. Then I have a Cars and a Cars_AUD table.
In my USCars and USCars_AUD table I had a column Description. I am deleting this column from the table and adding it to the Main Cars and Cars_AUD table.
We have a database project to do this and there is Pre and Post Deployment scripts. The Id column in the USCars and USCars_AUD will be the same Id as the Cars and Cars_AUD table has.
So I have updated my schema objects in the database project to reflect that USCars and a USCars_AUD will have description deleted and that it is added to Cars and Cars_AUD.
I am struggling a bit with the query to run pre-deployment and post-deployment. Ideally I want an IF type condition - to say only run if USCars table contains description column. (not sure if that is possible)
I then need to copy all the data from both tables into temp tables in pre-deploy
-- need an If conditional to only run if USCars table contains Description
-- then begin /end
print 'Moving USCars and Cars and Audit tables and related data to temp tables'
exec sp_executesql N'
select * into Upgrade_USCars from USCars
select * into Upgrade_USCars_AUD from USCars_AUD
'
Now Cars table has some extra info in it that is not in USCars so would I need to select everything from it into a temp table joined by the Id? So the result will be I want my final Cars and Cars_AUD table to contain everything it had in it originally but where it had a Description in the USCars and USCars_AUD table I want to copy across the description to the row with the same Id
In the Post Deploy then would I insert the values from this Joined temp table back into Cars Table and then Drop the temp tables
Syntax to check for column existence is:
IF EXISTS (SELECT * FROM sys.columns
WHERE object_id = OBJECT_ID('YourTableName')
AND name = 'YourColumnName')
BEGIN
-- your logic goes here
END
I have a SQL server 2005. In that server I have 3 databases -> a,b,c.
If I want to delete tables
Tables only from database "c".
The table's name should start with "bkp"
Table should be created one day before.
Try this:
USE C
GO
SELECT
'DROP TABLE ' + name
FROM sys.tables
WHERE create_date >= '20101211' -- substitute your date you're interested in
AND name like 'bkp%'
This will create as output a list of DROP TABLE:.... statement - copy those and paste them into a new SSMS window and execute those - and you're done!