I have had a look at similar problems, however none of the answers helped in my case.
Just a little bit of background. I have Two databases, both have the same table with the same fields and structure. Data already exists in both tables. I want to overwrite and add to the data in db1.table from db2.table the primary ID is causing a problem with the update.
When I use the query:
USE db1;
INSERT INTO db2.table(field_id,field1,field2)
SELECT table.field_id,table.field1,table.field2
FROM table;
It works to a blank table, because none of the primary keys exist. As soon as the primary key exists it fails.
Would it be easier for me to overwrite the primary keys? or find the primary key and update the fields related to the field_id? Im really not sure how to go ahead from here. The data needs to be migrated every 5min, so possibly a stored procedure is required?
first you should try to add new records then update all records.you can create a procedure like below code
PROCEDURE sync_Data(a IN NUMBER ) IS
BEGIN
insert into db2.table
select *
from db1.table t
where t.field_id not in (select tt.field_id from db2.table tt);
begin
for t in (select * from db1.table) loop
update db2.table aa
set aa.field1 = t.field1,
aa.field2 = t.field2
where aa.field_id = t.field_id;
end loop;
end;
END sync_Data
Set IsIdentity to No in Identity Specification on the table in which you want to move data, and after executing your script, set it to Yes again
I ended up just removing the data in the new database and sending it again.
DELETE FROM db2.table WHERE db2.table.field_id != 0;
USE db1;
INSERT INTO db2.table(field_id,field1,field2)
SELECT table.field_id,table.field1,table.field2
FROM table;
Its not very efficient, but gets the job done. I couldnt figure out the syntax to correctly do an UPDATE or to change the IsIdentity field within MariaDB, so im not sure if they would work or not.
The overhead of deleting and replacing non-trivial amounts of data for an entire table will be prohibitive. That said I'd prefer to update in place (merge) over delete /replace.
USE db1;
INSERT INTO db2.table(field_id,field1,field2)
SELECT t.field_id,t.field1,t.field2
FROM table t
ON DUPLICATE KEY UPDATE field1 = t.field1, field2 = t.field2
This can be used inside a procedure and called every 5 minutes (not recommended) or you could build a trigger that fires on INSERT and UPDATE to keep the tables in sync.
INSERT INTO database1.tabledata SELECT * FROM database2.tabledata;
But you have to keep length of varchar length larger or equal to database2 and keep the same column name
Related
I have table which contains millions of rows.
I want to delete all the data which is over a week old based on the value of column last_updated.
so here are my two queries,
Approach 1:
Delete from A where to_date(last_updated,''yyyy-mm-dd'')< sysdate-7;
Approach 2:
l_lastupdated varchar2(255) := to_char(sysdate-nvl(p_days,7),'YYYY-MM-DD');
insert into B(ID) select ID from A where LASTUPDATED < l_lastupdated;
delete from A where id in (select id from B);
which one is better considering performance, safety and locking?
Assuming the delete removes a significant fraction of the data & millions of rows, approach three:
create table tmp
Delete from A where to_date(last_updated,''yyyy-mm-dd'')< sysdate-7;
drop table a;
rename tmp to a;
https://asktom.oracle.com/pls/apex/f?p=100:11:0::::P11_QUESTION_ID:2345591157689
Obviously you'll need to copy over all the indexes, grants, etc. But online redefinition can help with this https://oracle-base.com/articles/11g/online-table-redefinition-enhancements-11gr1
When you get to 12.2, there's another simpler option: a filtered move.
This is an alter table move operation, with an extra clause stating which rows you want to keep:
create table t (
c1 int
);
insert into t values ( 1 );
insert into t values ( 2 );
commit;
alter table t
move including rows where c1 > 1;
select * from t;
C1
2
While you're waiting to upgrade to 12.2+ and if you don't want to use the create-as-select method for some reason then approach 1 is superior:
Both methods delete the same rows from A* => it's the same amount of work to do the delete
Option 1 has one statement; Option 2 has two statements; 2 > 1 => option 2 is more work
*Statement level consistency means you might get different results running the processes. Say another session tries to update an old row that your process will remove.
With just the delete, the update will be blocked until the delete finishes. At which point the row's gone, so the update does nothing.
Whereas if you do the insert first, the other session can update & commit the row before the insert completes. So the update "succeeds". But the delete will then remove it! Which can lead to some unhappy customers...
Your stored dateformat seems suitable for proper sorting, so you could go the other way round and convert sysdate to string:
--this is false today
select * from dual where '2019-06-05' < to_char(sysdate-7, 'YYYY-MM-DD');
--this is true today
select * from dual where '2019-05-05' < to_char(sysdate-7, 'YYYY-MM-DD');
So it would be:
Delete from A where last_updated < to_char(sysdate-7, ''yyyy-mm-dd'');
It has the benefit that your default index (if there is any) will be used.
It has the disadvantage on relying on the String/Varchar ordering which might be changed i.e. bei NLS changes (if i remember right), so in any case you should do a little testing before...
In the long term, you should of cource alter the colum to a proper date-datatype, but I guess that doesn't help you right now ;)
If you are trying to delete most of the rows in the table, I would advise you go with a different approach, namely:
create <new table name> as
select *
from <old table name>
where <predicates for the data you want to keep>;
then
drop table <old table name>;
and finally you can rename the new table back to the old table.
You could always partition the new table (i.e. create the new table with a separate statement containing the partitioning clauses, and then have an insert as select into the new table from the old table).
That way, when you need to delete rows, it's a simple matter of dropping the relevant partition(s).
I´m trying to execute stored procedure but I get an issue of an existing temporal table, but I just create one time and use into another part of code
SELECT ...
INTO #tmpUnidadesPresupuestadas
FROM proce.table1
--Insertar in table src..
INSERT INTO table (
....)
SELECT
....
FROM
#tmpUnidadesPresupuestadas
I get this message:
There is already an object named
'#tmpUnidadesPresupuestadas' in the database.
How can I solve it? Regards
A temp table lives for the entirety of the current session. If you run this statement more than once, then the table will already be there. Either detect that and truncate it, or before selecting into it drop it if it exists:
DROP TABLE IF EXISTS #tmpUnidadesPresupuestadas
If prior to SQL Server 2016, then you drop as such:
IF OBJECT_ID('tempdb.dbo.#tmpUnidadesPresupuestadas', 'U') IS NOT NULL
DROP TABLE #tmpUnidadesPresupuestadas;
Without seeing more of the code, it's not possible to know if the following situation is your problem, but it could be.
When you have mutually exclusive branches of code that both do a SELECT...INTO to the same temp table, a flaw causes this error. SELECT...INTO to a temp table creates the table with the structure of the query used to fill it. The parser assumes if that occurs twice, it is a mistake, since you can't recreate the structure of the table once it already has data.
if #Debug=1
select * into #MyTemp from MyTable;
else
select * into #MyTemp from MyTable;
While obviously not terribly meaningful, this alone will show the problem. The two paths are mutually exclusive, but the parser thinks they may both get executed, and issues the fatal error. You extend that, wrapping each branch in a BEGIN...END, and add the drop table (conditional or not) and the parser will still give the error.
To be fair, in fact both paths COULD be executed, if there were a loop or GOTO so that one time around #Debug = 1, and the other time it does not, so it may be asking too much of a parser. Unfortunately, I don't know of a workaround, and using INSERT INTO instead of SELECT INTO is the only way I know to avoid the problem, even though that can be terribly onerous to name all the columns in a particularly column-heavy query.
I am a bit unclear as to what you are attempting. I assume you don't want to drop the table at this point. I believe the syntax you may be looking for is
Insert Into
Insert into #tmpUnidadesPresupuestadas (Col1, col2, ... colN)
Select firstcol, secondcol... nthCol
From Data
If you do indeed wish to drop the table, the previous answers have that covered.
This might be useful for someone else, keep in mind that If more than one temporary table is created inside a single stored procedure or batch, they must have different names. If you use the same name you won't be able to ALTER the PROCEDURE.
https://learn.microsoft.com/en-us/previous-versions/sql/sql-server-2012/ms174979(v=sql.110)#temporary-tables
Make sure the stored procedure and the table doesn't have same name.
Add logic to delete if exists. Most likely you ran it previously. The table remains from the previous running of the stored procedure. If you log out and log in then run it, that would likely clear it. But the cleanest way is to check if it exists and delete it if it does. I assume this is MsSql.
At first you should check if temp table is already exist if yes then delete it then create a empty table then use insert statement. refer below example.
IF OBJECT_ID('tempdb..#TmpTBL') IS NOT NULL
DROP TABLE #TmpTBL;
SELECT TOP(0) Name , Address,PhoneNumber
INTO #TmpTBL
FROM EmpDetail
if #Condition=1
INSERT INTO #TmpTBL (Name , Address,PhoneNumber)
SELECT Name , Address,PhoneNumber FROM EmpDetail;
else
INSERT INTO #TmpTBL (Name , Address,PhoneNumber)
SELECT Name , Address,PhoneNumber FROM EmpDetail;
I want to delete new record if the same record created before.
My columns are date, time and MsgLog. If date and time are same, I want to delete new one.
I need help .
You can check in the table whether that value exists or not in the column using a query. If it exists, you can show message that a record already exists.
To prevent such kind of erroneous additions you can add restriction to your table to ensure unique #Date #Time pairs; if you don't want to change data structure (e.g. you want to add records with such restrictions once or twice) you can exploit insert select counstruction.
-- MS SQL version, check your DBMS
insert into MyTable(
Date,
Time,
MsgLog)
select #Date,
#Time,
#MsgLog
where not exists(
select 1
from MyTable
where (#Date = Date) and
(#Time = Time)
)
P.S. want to delete new one equals to do not insert new one
You should create a unique constraint in the DB level to avoid invalid data no matter who writes to your DB.
It's always important to have your schema well defined. That way you're safe that no matter how many apps are using your DB or even in case someone just writes some inserts manually.
I don't know which DB are you using but in MySQL can use to following DDL
alter table MY_TABLE add unique index(date, time);
And in Oracle you can :
alter table MY_TABLE ADD CONSTRAINT constaint_name UNIQUE (date, time);
That said, you can also (not instead of) do some checks before inserting new values to avoid dealing with exceptions or to improve performance by avoiding making unnecessary access to your DB (length \ nulls for example could easily be dealt with in the application level).
You can avoid deleting by checking for duplicate while inserting.
Just modify your insert procedure like this, so no duplicates will entered.
declare #intCount as int;
select #intCount =count(MsgLog) where (date=#date) and (time =#time )
if #intCount=0
begin
'insert procedure
end
> Edited
since what you wanted is you need to delete the duplicate entries after your bulk insert. Think about this logic,
create a temporary table
Insert LogId,date,time from your table to the temp table order by date,time
now declare four variables, #preTime,#PreDate,#CurrTime,#CurrDate
Loop for each items in temp table, like this
while
begin
#pkLogID= ' Get LogID for the current row
select #currTime=time,#currDate=date from tblTemp where pkLogId=#pkLogID 'Assign Current values
'Delete condition check
if (#currDate=#preDate) and (#currTime=#preTime)
begin
delete from MAINTABLE where pkLogId=#pkLogID
end
select #preDate=#currDate,#preTime=#currTime 'Assign current values as preValues for next entries
end
The above strategy is we sorted all entries according to date and time, so duplicates will come closer, and we started to compare each entry with its previous, when match found we deleting the duplicate entry.
In a database application, I want to insert, update and delete records in a table of database.
Table is as below:
In this table, Ga1_ID is Primary Key.
Suppose, I insert 5 records as show currently.
In second attempt, if I want to insert 5 other records and if any of these new records contains a primary key attribute which is already present in table it show error. Its fine.
But, when I insert new 5 records... how I can verify these new records's primary key value is not present. I mean, how to match or calculate the already present primary key attributes and then insert new records.
What is the best approach to manage this sort of situation ?
use following query in dataadapter:
da=new SqlDataAdapter("select Ga1_ID from table where Ga1_ID=#pkVal",conn);
DataSet=new DataSet();
da.fill(ds);
//pass parameter for #pkVal
da.SelectCommand.Parameters(1).Value = pkValue;
if(ds.Tables[0].Rows.Count>0) //If number of rows >0 then record exists
BEGIN
messagebox.show("Primary key present");
END
Hope its helpful.
Do not check existing records in advance, i.e. do not SELECT and then INSERT. A better (and pretty common) approach is to try to INSERT and handle exceptions, in particular, catch a primary key violation if any and handle it.
Do the insert in a try/catch block, with different handling in case of a primary key violation exception and other sql exception types.
If there was no exception, then job's done, record was inserted.
If you caught a primary key violation exception, then handle it appropriately (your post does not specify what you want to do in this case, and it's completely up to you)
If you want to perform 5 inserts at once and want to make sure they all succeed or else roll back if any of them failed, then do the inserts within a transaction.
you can do a lookup first before inserting.
IF EXISTS (SELECT * FROM tableName WHERE GA1_id=#newId)
BEGIN
UPDATE tableName SET Ga1_docid = #newdocID, GA1_fieldNAme = #newName, Ga1_fieldValue = #newVal where GA1_id=#newId
END
ELSE
BEGIN
INSERT INTO tableName(GA1_ID, Ga1_docid, GA1_fieldNAme Ga1_fieldValue) VALUES (value1,val2,value3,value4)
END
If you're using SQL Server 2012, use a sequence object - CREATE SEQUENCE.
This way you can get the next value using NEXT VALUE FOR.
With an older SQL Server version, you need to create the primary key field as an IDENTITY field and use the SCOPE_IDENTITY function to get the last identity value and then increment it manually.
Normally, you would like to have a surrogate key wich is generally an identity column that will automatically increment when you are inserting rows so that you don't have to care about knowing which id already exists.
However, if you have to manually insert the id there's a few alternatives for that and knowing wich SQL database you are using would help, but in most SQL implementations, you should be able to do something like:
IF NOT EXISTS
IF NOT EXISTS(
SELECT 1
FROM your_table
WHERE Ga1_ID = 1
)
INSERT INTO ...
SELECT WHERE NOT EXISTS
INSERT INTO your_table (col_1, col_2)
SELECT col_1, col_2
FROM (
SELECT 1 AS col_1, 2 AS col_2
UNION ALL
SELECT 3, 4
) q
WHERE NOT EXISTS (
SELECT 1
FROM your_table
WHERE col_1 = q.col_1
)
For MS SQL Server, you can also look at the MERGE statement and for MySQL, you can use the INSERT IGNORE statement.
We have a status table. When the status changes we currently delete the old record and insert a new.
We are wondering if it would be faster to do a select to check if it exists followed by an insert or update.
Although similar to the following question, it is not the same, since we are changing individual records and the other question was doing a total table refresh.
DELETE, INSERT vs UPDATE || INSERT
Since you're talking SQL Server 2008, have you considered MERGE? It's a single statement that allows you to do an update or insert:
create table T1 (
ID int not null,
Val1 varchar(10) not null
)
go
insert into T1 (ID,Val1)
select 1,'abc'
go
merge into T1
using (select 1 as ID,'def' as Val1) upd on T1.ID = upd.ID --<-- These identify the row you want to update/insert and the new value you want to set. They could be #parameters
when matched then update set Val1 = upd.Val1
when not matched then insert (ID,Val1) values (upd.ID,upd.Val1);
What about INSERT ... ON DUPLICATE KEY? First doing a select to check if a record exists and checking in your program the result of that creates a race condition. That might not be important in your case if there is only a single instance of the program however.
INSERT INTO users (username, email) VALUES ('Jo', 'jo#email.com')
ON DUPLICATE KEY UPDATE email = 'jo#email.com'
You can use ##ROWCOUNT and perform UPDATE. If it was 0 rows affected - then perform INSERT after, nothing otherwise.
Your suggestion would mean always two instructions for each status change. The usual way is to do an UPDATE and then check if the operation changed any rows (Most databases have a variable like ROWCOUNT which should be greater than 0 if something changed). If it didn't, do an INSERT.
Search for UPSERT for find patterns for your specific DBMS
Personally, I think the UPDATE method is the best. Instead of doing a SELECT first to check if a record already exists, you can first attempt an UPDATE but if no rows are affected (using ##ROWCOUNT) you can do an INSERT.
The reason for this is that sooner or later you might want to track status changes, and the best way to do this would be to keep an audit trail of all changes using a trigger on the status table.