I have two identical tables in 2 different DBs ( one local and one remote ). Both of them have a Datetime ( or Datetime2 or anything else, I can still change that ) column which specifies when they were added. What I want is to get the records from the remote table to the local table that have not already been brought.
For this a get the max value from the local table, then I tried to put the condition WHERE remoteTable.CreateTime > maxLocalValue, but I don't know why this seems to overwrite some records on the boundry. Any better way to do this ? On a second or minute level ?
since we know which values have been added already to the local table, only add the values from the remote table that are not in the local table. Let local_table repersents the local table and remote_table repersents remote table and that rcd_1 is a column in both tables that repersents the DATE type.
INSERT INTO local_table (col_1,.....(how evermany columns you need), rcd_1)
SELECT rcd_1 FROM remote_table
where remote_table.rcd_1 <> local_table.rcd_1;
the way you explained you question would suggest that only newer dates are added to local from remote so as long as this hold true you should get the desired results.
Related
I have a SQL Server table with just 3 columns, one of which is of type varbinary. The data in this column is actually a Json document which among other properties contains information about when the data was last modified. Unfortunately the SQL table itself does not contain information about when its rows were modified.
Now when doing sorting and filtering of the data I of course don't want fetch all rows in order to find e.g. the latest 100 entries.
So my question is: does SQL Server somehow remember when a row was added/modified? I have tried adding a timestamp and this is applied to all existing rows but this is applied randomly I think, because the sorting doesn't work. I don't need a datetime or anything, I just want to be able sort the records based on when they were last modified.
Thanks
For those looking to insert a tamestamp column of type DateTime into an existing DB table, you can do this like so:
ALTER TABLE TestTable
ADD DateInserted DATETIME NOT NULL DEFAULT (GETDATE());
The existing records will automatically get the value equal to the date/time of the moment when column is added.
New records will get up-to-date value upon insertion.
SQL Server will not track historically when a row was inserted or modified so you need to rely on the JSON data to figure that out yourself. You are going to need a new column to make this efficient to query. Once you have your new column you have some options:
Loop through all your records populating the new column with the relevant value from the JSON data.
If your version of SQL Server is recent enough, you can query the JSON data directly. Populate this column using a query like this:
UPDATE MyTable
SET MyNewColumn = JSON_VALUE(JsonDataColumn, '$.Customer.DateCreated')
The downside of this method is that you need to maintain this
Make SQL Server compute the value from the JSON automatically, for example:
ALTER TABLE MyTable
ADD MyNewColumn AS JSON_VALUE(JsonDataColumn, '$.Customer.DateCreated')
And, create an index to make it efficient:
CREATE INDEX IX_MyTable_MyNewColumn
ON MyTable(MyNewColumn)
Use a new column CreatedDate and store datetime every time you make an Insert.
You could use GetDate() for inserting date in the column.
A UpdatedDate column can be used for updates.
in order to find e.g. the latest 100 entries.
Timestamp is indeed what you need.
It's ever-increasing value, it's updated automatically, so you are always able to find all last modified/inserted rows.
Here is an example:
create table dbo.test1 (id int);
insert into dbo.test1 values(1), (2), (3);
alter table dbo.test1 add ts timestamp;
update dbo.test1
set id = 10
where id = 2
select top 1 *
from dbo.test1
order by ts desc;
--id ts
--10 0x000000001FCFABD2
insert into dbo.test1 (id)
values (100);
select top 1 *
from dbo.test1
order by ts desc;
--id ts
--100 0x000000001FCFABD3
As you see, you always get the last modified/inserted row.
For your purpose just use
select top 100 *
...
order by ts desc;
Thanks. Apparently I didn't look hard enough before I posted this question. The question has been asked a couple of times before and the answer is: Nope! There is no easy solution to this.
SQL Server does not keep track of when a record was created or modified, which was somehow what I was looking for. So I will go for the next best solution, which is probably to create a datetime column, retrieve the modified date from the Json document and then update the record. Or rather, the 1,4 million records:-(
I need to run a query at specific time an save it in other table, that part I've already done it, but later I need to run the same query but need to exclude the data that it was generated in the first run of the query. It is a process that has to be done three times a day. Any ideas? Thanks.
The question is a little confusing, but I think you want to backup several times a day the records that were inserted in the source table after the last backup.
Let's assume that your structure is the following:
**DataTable**
Id (IDENTITY)
Value
**BackupTable**
Id
SourceId -- identifies the source. Can be used when merging data from multiple sources
RefId -- identifier in source's scope (in our case DataTable.Id)
Value -- value copied
In our case, the job should look for those values that are not yet inserted:
DECLARE #SourceId INT = 1 -- this is a constant for DataTable as source
DECLARE #maxIdForSource INT = (SELECT MAX(RefId) FROM BackupTable WHERE SourceId = #SourceId)
INSERT INTO BackupTable
(SourceId, RefId, Value)
SELECT #SourceId, Id, Value
FROM DataTable
WHERE Id > #maxIdForSource
If you are not merging data from multiple sources, you do not define SourceId at all.
I want to copy the content of one column in table A and replace the contents (not insert into it - the number of rows will be the same) of another column in another table.
I can't a where condition, the table has only just been created at this point with one empty timestamp column. it will be populated via pyodbc class after the timestamps have been added - this query will fill the timestamps for me
What is the SQL command for this?
Thanks!
After discussion, this is the query needed : INSERT INTO OCAT_test_table (DateTimeStamp) SELECT DateTimeStamp FROM DunbarGen
im new to Oracle and sql but I was assigned this job and I hope someone can help me out with this one.
Basically I am given a database link to connect to a remote database and I extract some information from a single table in there and a few other tables from a local database, and then process it and insert it into a table in the local database. I`ve managed to do this succesfully but now I need a way to confirm that all of the data from the remote database was actually copied into the local database. How would I go about doing this?
This is the code I have to insert the information to my local db.
INSERT INTO kcrt_requests_int RI
RI.TRANSACTION_ID,
RI.DESCRIPTION,
RI.CREATED_USERNAME,
RI.REQUEST_TYPE_ID,
RI.STATUS_ID,
RI.WORKFLOW_ID,
RI.WORKFLOW_STEP_ID,
RI.RELEASED_FLAG,
RI.USER_DATA1,
RI.USER_DATA2,
RI.USER_DATA3,
RI.USER_DATA4,
RI.USER_DATA7)
SELECT
KCRT_TRANSACTIONS_S.NEXTVAL,
RD.PARAMETER13||' '||R.DESCRIPTION,
'[SYS.USERNAME]',
'0001',
'31876',
'34987', '1234',
'Y',
PP.PROJECT_ID,
VP.REVIEWDATE,
RD.PARAMETER9,
R.REQUEST_ID,
RD.PARAMETER13
FROM
KCRT_REQUEST_TYPES_NLS RT,
KCRT_REQUESTS R,
KCRT_REQUEST_DETAILS RD,
v_projects#XXXXX VP,
PM_PROJECTS PP
WHERE
R.REQUEST_TYPE=RT.REQUEST_TYPE_ID
AND R.REQUEST_ID=RD.REQUEST_ID
AND RD.BATCH_NUMBER=1
AND RT.REQUEST_TYPE_NAME 'AAAAA'
AND R.STATUS_CODE = 'BBBBB'
AND RD.PARAMETER13 = to_char(VP.IDBANK)
AND VP.REVIEWDATE=(SELECT MAX (VP.REVIEWDATE) FROM v_projects#XXXXX VP)
AND R.REQUEST_ID=PP.PFM_REQUEST_ID
AND RD.BATCH_NUMBER=1
So pretty much I will try to compare RI.USER_DATA7 to VP.IDBANK and see if KCRT_REQUESTS_INT has every row that v_projects#XXXXX has.
Thanks for any help!
If there is a unique identifier column which is defined as primary key.if yes,you can join both tables on Primary key and see if the count matches with count without join on source table.
assumes ,Table A is your source and Table B is where you have loaded data. let P_key be primary key column.
You can match:
select count(1)
from Table_A with
select count (1)
from Table_A,Table_B
where Table_A.P_key=Table_B.P_Key
If they match ,you have all the records. Hope this helps.
I have a table in a database that represents dates textually (i.e. "2008-11-09") and I would like to replace them with the UNIX timestamp. However, I don't think that MySQL is capable of doing the conversion on its own, so I'd like to write a little script to do the conversion. The way I can think to do it involves getting all the records in the table, iterating through them, and updating the database records. However, with no primary key, I can't easily get the exact record I need to update.
Is there a way to get MySQL to assign temporary IDs to records during a SELECT so that I refer back to them when doing UPDATEs?
Does this not do it?
UPDATE
MyTable
SET
MyTimeStamp = UNIX_TIMESTAMP(MyDateTime);
If for some reason you do have to iterate (the other answers cover the situation where you don't), I can think of two ways to do it (these aren't MySQL-specific):
Add a column to the table that's an auto-assigned number. Use that as the PK for your updates, then drop the column afterwards (or just keep it around for future use).
In a table with no defined PK, as long as there are no exact duplicate rows, you can use the entire row as a composite PK; just use every column in the row as your distinguishing characteristic. i.e., if the table has 3 columns, "name", "address", and "updated", do the following:
UPDATE mytable SET updated = [timestamp value] WHERE name = [name] AND address = [address] AND timestamp = [old timestamp]
Many data access frameworks use this exact strategy to implement optimistic concurrency.
No, you should be able to do this with a single update statement. If all of the dates are yyyy-mm-dd and they are just stored in some sort of text column instead of DATETIME, you can just move the data over. SQL would be like:
ALTER TABLE t ADD COLUMN dates DATETIME;
UPDATE t set t.dates=t.olddate;
This shouldn't be dependent on a PK because MySQL can scan through each row in the table. The only time PK's become an issue is if you need to update a single row, but the row may not be unique.
You can generate values during a SELECT using the MySQL user variables feature, but these values do not refer to the row; they're temporary parts of the result set only. You can't use them in UPDATE statements.
SET #v := 0;
SELECT #v:=#v+1, * FROM mytable;
Here's how I'd solve the problem. You're going to have to create another column for your UNIX timestamps anyway, so you can add it first. Then convert the values in the old datetime column to the UNIX timestamp and place it in the new column. Then drop the old textual datetime column.
ALTER TABLE mytable ADD COLUMN unix_timestamp INT UNSIGNED NOT NULL DEFAULT 0;
UPDATE mytable
SET unix_timestamp = UNIX_TIMESTAMP( STR_TO_DATE( text_timestamp, '%Y-%m-%d' ) );
ALTER TABLE mytable DROP COLUMN text_timestamp;
Of course you should confirm that the conversion has been done correctly before you drop the old column!
See UNIX_TIMESTAMP() and STR_TO_DATE()