Hello Everyone I wanted to know if it is possible to Insert new values into multiple tables(in my case 22) in a single query. Once they are inserted I want to delete these 22(newly inserted rows) and move the deleted rows into a separate table ?
It depends on your definition of "a single query". You cannot do it in a single SQL statement like SELECT, UPDATE, DELETE, MERGE. But you can create a single query with multiple SQL statements using BigQuery Scripting (currently in Beta).
https://cloud.google.com/bigquery/docs/reference/standard-sql/scripting
Note that these multiple updates are not transactional, BigQuery uses single-statement transactions.
Related
I'm contributing to a Kafka Connector loading data onto BigQuery.
It has a temporary table (my_tmp_tmp) and a destination table (detionation_tbl).
The way the data is loaded into detionation_tbl is through a MERGE
https://github.com/confluentinc/kafka-connect-bigquery/blob/d5f4eaeffa683ad8813a337cfeb66b5344e6dd91/kcbq-connector/src/main/java/com/wepay/kafka/connect/bigquery/MergeQueries.java#L216
The MERGE statement uses:
dedup
both insert and updates
However:
on the first load,
all of the requests will contain only INSERT statement (nothing is in the table),
and if two merges are run at the same time, (many workers with retry),
the same records might be inserted twice (according to this A MERGE DML statement does not conflict with other concurrently running DML statements as long as the statement only inserts rows and does not delete or update any existing rows. This can include MERGE statements with UPDATE or DELETE clauses, as long as those clauses aren't invoked when the query runs., source). I also see this in practice leading to duplicates
Duplicates are not wanted since the whole point of running a MERGE is lost (compared to a solution that run INSERTs and dedup later)
Since it is a live dataset (being queried by users), Duplicates will break integrity of the dataset, hence we need to find a solution at Sink/BigQuery level.
Is it possible to make the merge statement always conflict with others so this doesn't happen? Any other solution?
is it possible to declare one Sql Trigger Insert, Update, Delete for all tables in the database instead of creating a separate trigger for each table? I just want a simple history of what actions have been taken e.g. TABLE A deletes a row ,TABLE B updates a row ,TABLE C Add new row .. and the trigger will insert a new row to another table with that information I want.
No, a trigger can only be defined on a specified table.
You should read up on the auditing features of SQL Server (http://technet.microsoft.com/en-us/library/cc280386.aspx). They are more performant and more flexible in what you want to achieve. Unfortunately they are not available in the Express Edition.
I need to merge two tables in following way:
Target has one extra Column ID. This Id is coming FROM another Single Column Master Table.
While Inserting the Record in Merge Statement I need to INSERT a new row into mater table and use its id to insert into TARGET table.
I have created a Stored Procedure that Inserts and returns newly inserted ID. Now the Problem is inside SQL Merge, we can't call a stored Proc.
What could be the solution of this issue? Cant use Scalar functions as INSERT can't be performed in Functions.
DECLARE #temp INT
MERGE dbo.mytabletarget T
USING dbo.mytableSource S
ON T.refId=S.RefId
WHEN MATCHED THEN
UPDATE
SET T.col1=S.col1,
T.Col2=S.Col2
WHEN NOT MATCHED BY TARGET THEN
INSERT (Id,col1,col2)
VALUES({Here i need value from SP. SP simply Inserts a new Id into master table and Returns it},S.col1,S.col2);
GO
What could be the solution of this issue?
Do not use a stored procedure. Obvious, isn't it?
For a merge statement, you pretty much are stuck with doing the commands right there in the statement. Merge focuses on ETL loads and has advantages as well as limitations.
Basically, put the logic into the merge statement.
While Inserting the Record in Merge Statement I need to INSERT a new row into mater table
and use its id to insert into TARGET table.
Hm, lookup table maintenance?
The regular approach for that is ti make sure the lookup table is filled first (in a separate statement). ETL (and that is where merge comes from) often works along stages for that particular reason.
Sorry, I do not have a better solution either ;(
Just like Multi-table insert(using INSERT ALL) is there multi-table Update facility in Oracle? In which version?
AFAIK there is no such facility in Oracle - you can't UPDATE multiple tables within one statement - you can use mutiple tables within an UPDATE statement but only one table can be updated.
What you could do is either wrap multiple UPDATE statements into one transaction and/or use an INSTEAD OF trigger which in turn contains multiple UPDATE statements...
Will it be possible to insert into two tables with same insert command?
No you cannot perform multiple inserts into two tables in one query.
No you can't.
If you want to ensure the atomicity of an operation that requires data to be inserted into 2 tables, you should protect it in a transaction. You either use the SQL statements BEGIN TRAN and COMMIT TRAN, or you use a transaction boundary in whatever language you're using to develop the db access layer. E.g. something like Connection.StartTransaction and Connection.Commit (or Connection.Rollback on an error).
You can call a stored procedure with inserts into two tables.
Maybe in a future release of MySQL you could create a View containing the 2 tables and insert into that.
But with MySQL 5.1.41 you'll get the error:
"Can not modify more than one base table through a join view"
But inserting into 2 tables with 1 query is a weird thing to do, and I don't recommend it.
For more on updatable views check out the MySQL reference.