I am calculating Euclidean distance between faces and want to store results in a table.
Current setup :
Each face is stored in Objects table and Distances between faces is stored in Faces_distances table.
The object table has the following columns objects_id, face_encodings, description
The faces_distances table has the following columns face_from, face_to, distance
In my my data set I have around 22 231 face objects which result in 494 217 361 pairs of faces - Although I understand it could be divided by 2 because
distance(face_from, face_to) = distance(face_to, face_from)
The database is Postgres 12.
The request below enables to insert the pairs of faces (without performing the distance calculation) that have not been calculated yet, but the execution time is very very very long (started 4 Days ago and still not done). Is there a way to optimize it ?
'''
-- public.objects definition
-- Drop table
-- DROP TABLE public.objects;
CREATE TABLE public.objects
(
objects_id int4 NOT NULL DEFAULT
nextval('objects_in_image_objects_id_seq'::regclass),
filefullname varchar(2303) NULL,
bbox varchar(255) NULL,
description varchar(255) NULL,
confidence numeric NULL,
analyzer varchar(255) NOT NULL DEFAULT 'object_detector'::character
varying,
analyzer_version int4 NOT NULL DEFAULT 100,
x int4 NULL,
y int4 NULL,
w int4 NULL,
h int4 NULL,
image_id int4 NULL,
derived_from_object int4 NULL,
object_image_filename varchar(2023) NULL,
face_encodings _float8 NULL,
face_id int4 NULL,
face_id_iteration int4 NULL,
text_found varchar NULL COLLATE "C.UTF-8",
CONSTRAINT objects_in_image_pkey PRIMARY KEY (objects_id),
CONSTRAINT objects_in_images FOREIGN KEY (objects_id) REFERENCES
public.objects(objects_id)
);
CREATE TABLE public.face_distances
(
face_from int8 NOT NULL,
face_to int8 NOT NULL,
distance float8 NULL,
CONSTRAINT face_distances_pk PRIMARY KEY (face_from, face_to)
);
-- public.face_distances foreign keys
ALTER TABLE public.face_distances ADD CONSTRAINT face_distances_fk
FOREIGN KEY (face_from) REFERENCES public.objects(objects_id);
ALTER TABLE public.face_distances ADD CONSTRAINT face_distances_fk_1
FOREIGN KEY (face_to) REFERENCES public.objects(objects_id);
Indexes
CREATE UNIQUE INDEX objects_in_image_pkey ON public.objects USING btree (objects_id);
CREATE INDEX objects_description_column ON public.objects USING btree (description);
CREATE UNIQUE INDEX face_distances_pk ON public.face_distances USING btree (face_from, face_to);
Query to add all pair of faces that are not already in the table.
insert into face_distances (face_from,face_to)
select t1.face_from , t1.face_to
from (
select f_from.objects_id face_from,
f_from.face_encodings face_from_encodings,
f_to.objects_id face_to,
f_to.face_encodings face_to_encodings
from objects f_from,
objects f_to
where f_from.description = 'face'
and f_to.description = 'face' ) as t1
left join face_distances on (
t1.face_from= face_distances.face_from
and t1.face_to = face_distances.face_to )
where face_distances.face_from is null;
try this simplified query.
It took only 5 minutes on my apple M1, SQLServer, with 22231 objects 'face', generated 247.097.565 pairs, which is excatly C(22231,2) number. The syntax is compatible with postgressql.
optimizations: join instead of the old jointure way, ranking functions to remove duplicates permutations (A,B)=(B,A),
removed the last [left join face_distance]: an empty table to recompute is a lot faster than checking for existance as an index search key lookup would be initiated for each key pair
insert into face_distances (face_from,face_to)
select f1,f2
from(
select --only needed fields here as this will fill temporary tables
f1.objects_id f1
,f2.objects_id f2
,dense_rank()over(order by f1.objects_id) rank1
,rank()over(partition by f2.objects_id order by f1.objects_id) rank2
from objects f1
-- generates all permutations
join objects f2 on f2.objects_id <> f1.objects_id and f2.description = 'face'
where f1.description = 'face'
)a
where rank2 >= rank1 --removes duplicate permutations
In my game, an archetype is a collection of associated traits, an attack type, a damage type, and a resource type. Each piece of data is unique to each
archetype. For example, the Mage archetype might look like the following:
archetype: Mage
attack: Targeted Area Effect
damage: Shock
resource: Mana
trait_defense: Willpower
trait_offense: Intelligence
This is the archetype table in SQLite syntax:
create table archetype
(
archetype_id varchar(16) not null,
attack_id varchar(16) not null,
damage_id varchar(16) not null,
resource_type_id varchar(16) not null,
trait_defense_id varchar(16) not null,
trait_offense_id varchar(16) not null,
archetype_description varchar(128),
constraint pk_archetype primary key (archetype_id),
constraint uk_archetype unique (attack_id, damage_id,
resource_type_id,
trait_defense_id,
trait_offense_id)
);
The primary key should be the complete composite, but I do not want to pass
all the data around to other tables unless necessary. For example, there are
crafting skills associated with each archetype which do not need to know any
other archetype data.
An effect is a combat outcome that can be applied to a friend or foe. An effect has an application type (instant, overtime), a type (buff, debuff, harm, heal, etc.) and a detail describing to which stat the effect applies. It also has most of the archetype data to make each effect unique. Also included is the associated trait used for progress and skill checks. For example, an effect might look like:
apply: Instant
type: Harm
detail: Health
archetype: Mage
attack_id: Targeted Area Effect
damage_id: Shock
resource: Mana
trait_id: Intelligence
This is the effect table in SQLite syntax:
create table effect
(
effect_apply_id varchar(16) not null,
effect_type_id varchar(16) not null,
effect_detail_id varchar(16) not null,
archetype_id varchar(16) not null,
attack_id varchar(16) not null,
damage_id varchar(16) not null,
resource_type_id varchar(16) not null,
trait_id varchar(16),
constraint pk_effect primary key(archetype_id, effect_type_id,
effect_detail_id, effect_apply_id,
attack_id, damage_id, resource_type_id),
constraint fk_effect_archetype_id foreign key(archetype_id, attack_id,
damage_id, resource_type_id)
references archetype (archetype_id, attack_id,
damage_id, resource_type_id)
);
An ability is a container that can hold multiple effects. There is no limit to
the kinds of effects it can hold, e.g. having both Mage and Warrior effects in
the same ability, or even having two of the same effects, is fine. Each effect
in the ability is going to have the archetype data, and the effect data.
Again.
Ability tables in SQLite syntax:
create table ability
(
ability_id varchar(64),
ability_description varchar(128),
constraint pk_ability primary key (ability_id)
);
create table ability_effect
(
ability_effect_id integer primary key autoincrement,
ability_id varchar(64) not null,
archetype_id varchar(16) not null,
effect_type_id varchar(16) not null,
effect_detail_id varchar(16) not null,
effect_apply_id varchar(16) not null,
attack_id varchar(16) not null,
damage_id varchar(16) not null,
resource_type_id varchar(16) not null,
trait_id varchar(16),
constraint fk_ability_effect_ability_id foreign key (ability_id)
references ability (ability_id),
constraint fk_ability_effect_effect_id foreign key (archetype_id,
effect_type_id,
effect_detail_id,
effect_apply_id)
references effect (archetype_id,
effect_type_id,
effect_detail_id,
effect_apply_id)
);
This is basically a one to many to many relationship, so I needed a technical
key to have duplicate effects in the ability_effect table.
Questions:
1) Is there a better way to design these tables to avoid the duplication of
data over these three tables?
2) Should these tables be broken down further?
3) Is it better to perform multiple table lookups to collect all the data? For example, just passing around the archetype_id and doing lookups for the data when necessary (which will be often).
UPDATE:
I actually do have parent tables for attacks, damage, etc. I removed those
tables and their related indexes from the sample to make the question clean,
concise, and focused on my duplicate data issue.
I was trying to avoid each table having both an id and a name, as both would be candidate keys and so having both would be wasted space. I was trying to keep the SQLite database as small as possible. (Hence, the many "varchar(16)"
declarations, which I now know SQLite ignores.) It seems in SQLite having both
values is unavoidable, unless being twice as slow is somehow ok when using the
WITHOUT ROWID option during table creation. So, I will rewrite my database to
use ids and names via the rowid implementation.
Thanks for your input guys!
1) Is there a better way to design these tables to avoid the
duplication of data over these three tables?
and also
2) Should these tables be broken down further?
It would appear so.
It would appear Mage is a unique archtype, as is Warrior. (based upon For example, the Mage archetype might look like the following:).
As such why not make the archtype_id a primary key and then reference the attack type, damage etc from tables for these. i.e. have an attack table and a damage table.
So you could, for example, have something like (simplified for demonstration) :-
DROP TABLE IF EXISTS archtype;
DROP TABLE IF EXISTS attack;
DROP TABLE IF EXISTS damage;
CREATE TABLE IF NOT EXISTS attack (attack_id INTEGER PRIMARY KEY, attack_name TEXT, a_more_columns TEXT);
INSERT INTO attack (attack_name, a_more_columns) VALUES
('Targetted Affect','ta blah'), -- id 1
('AOE','aoe blah'), -- id 2
('Bounce Effect','bounce blah') -- id 3
;
CREATE TABLE IF NOT EXISTS damage (damage_id INTEGER PRIMARY KEY, damage_name TEXT, d_more_columns TEXT);
INSERT INTO damage (damage_name,d_more_columns) VALUES
('Shock','shock blah'), -- id 1
('Freeze','freeze blah'), -- id 2
('Fire','fire blah'), -- id 3
('Hit','hit blah')
;
CREATE TABLE IF NOT EXISTS archtype (id INTEGER PRIMARY KEY, archtype_name TEXT, attack_id_ref INTEGER, damage_id_ref INTEGER, at_more_columns TEXT);
INSERT INTO archtype (archtype_name,attack_id_ref,damage_id_ref,at_more_columns) VALUES
('Mage',1,1,'Mage blah'),
('Warrior',3,4,'Warrior Blah'),
('Dragon',2,3,'Dragon blah'),
('Iceman',2,2,'Iceman blah')
;
SELECT archtype_name, damage_name, attack_name FROM archtype JOIN damage ON damage_id_ref = damage_id JOIN attack ON attack_id_ref = attack_id;
Note that the aliases of rowid have been used for id's rather than the name as these are generally the most efficient.
The data for rowid tables is stored as a B-Tree structure containing one entry for each table row, using the rowid value as the key. This means that retrieving or sorting records by rowid is fast. Searching for a record with a specific rowid, or for all records with rowids within a specified range is around twice as fast as a similar search made by specifying any other PRIMARY KEY or indexed value. SQL As Understood By SQLite - CREATE TABLE- ROWIDs and the INTEGER PRIMARY KEY
A rowid is generated for all rows (unless WITHOUT ROWID is specified), by specifying ?? INTEGER PRIMARY KEY column ?? is an alias of the rowid.
Beware using AUTOINCREMENT, unlike other RDMS's that use this for automatically generating unique id's for rows. SQLite by default creates a unique id (the rowid). The AUTOINCREMENT keyword adds a constraint that ensures that the generated id is larger than the highest existing. To do this requires an additional table sqlite_sequence that has to be maintained and interrogated and as such has overheads. The AUTOINCREMENT keyword imposes extra CPU, memory, disk space, and disk I/O overhead and should be avoided if not strictly needed. It is usually not needed. SQLite Autoincrement
The query at the end will result in :-
Now say you wanted types to have multiple attacks and damages per type then the above could easily be adapted by using many-many relationships by introducing reference/mapping/link tables (all just different names for the same). Such a table will have two columns (sometime other columns for data specific to the distinct reference/map/link) one for the parent (archtype) reference/map/link and the other for the child (attack/damage) referenced/mapped/linked.
e.g. the following could be added :-
DROP TABLE IF EXISTS archtype_attack_reference;
CREATE TABLE IF NOT EXISTS archtype_attack_reference
(aar_archtype_id INTEGER NOT NULL, aar_attack_id INTEGER NOT NULL,
PRIMARY KEY(aar_archtype_id,aar_attack_id))
WITHOUT ROWID;
DROP TABLE IF EXISTS archtype_damage_reference;
CREATE TABLE IF NOT EXISTS archtype_damage_reference
(adr_archtype_id INTEGER NOT NULL, adr_damage_id INTEGER NOT NULL,
PRIMARY KEY(adr_archtype_id,adr_damage_id))
WITHOUT ROWID
;
INSERT INTO archtype_attack_reference VALUES
(1,1), -- Mage has attack Targetted
(1,3), -- Mage has attack Bounce
(3,2), -- Dragon has attack AOE
(2,1), -- Warrior has attack targetted
(2,2), -- Warrior has attack AOE
(4,2), -- Iceman has attack AOE
(4,3) -- Icemane has attack Bounce
;
INSERT INTO archtype_damage_reference VALUES
(1,1),(1,3), -- Mage can damage with Shock and Freeze
(2,4), -- Warrior can damage with Hit
(3,3),(3,4), -- Dragon can damage with Fire and Hit
(4,2),(4,4) -- Iceman can damage with Freeze and Hit
;
SELECT archtype_name, attack_name,damage_name FROM archtype
JOIN archtype_attack_reference ON archtype_id = aar_archtype_id
JOIN archtype_damage_reference ON archtype_id = adr_archtype_id
JOIN attack ON aar_attack_id = attack_id
JOIN damage ON adr_damage_id = damage_id
;
The query results in :-
With a slight change the above query could even be used to perform a random attack e.g. :-
SELECT archtype_name, attack_name,damage_name FROM archtype
JOIN archtype_attack_reference ON archtype_id = aar_archtype_id
JOIN archtype_damage_reference ON archtype_id = adr_archtype_id
JOIN attack ON aar_attack_id = attack_id
JOIN damage ON adr_damage_id = damage_id
ORDER BY random() LIMIT 1 -- ADDED THIS LINE
;
You could get :-
Another time you might get :-
3) Is it better to perform multiple table lookups to collect all the
data? For example, just passing around the archetype_id and doing
lookups for the data when necessary (which will be often).
That's pretty hard to say. You may initially think gather all the data once and keep it in memory say as an object. However, at times the underlying data may well already be in memory due to it being cached. Perhaps it could be better to utilise part of each. So I believe the answer is, you will need to test various scenarios.
I would probably avoid those composite primary keys.
And use the more commonly used integer with an autoincrement.
Then add the unique or non-unique composite indexes where needed.
Although i.m.h.o it's not always a bad idea to use a short CHAR or VARCHAR as the primary key in some cases. Mostly when easy to understand abbreviations can be used.
An example. Suppose you have a reference table for Countries. With a primary key on the 2 character CountryCode. Then when querying a table with a foreign key on that CountryCode, then for the human mind it's way easier to understand 'US' than some integer. Even without joining to Countries you'll probably know what Country is referenced.
So here are your tables with a slightly different layout.
create table archetype
(
archetype_id integer primary key autoincrement,
attack_id varchar(16) not null,
damage_id varchar(16) not null,
resource_type_id varchar(16) not null,
trait_defense_id varchar(16) not null,
trait_offense_id varchar(16) not null,
archetype_description varchar(128),
constraint uk_archetype unique (attack_id, damage_id,
resource_type_id,
trait_defense_id,
trait_offense_id)
);
create table effect
(
effect_id integer primary key autoincrement,
archetype_id integer not null, -- FK archetype
effect_apply_id varchar(16) not null,
effect_type_id varchar(16) not null,
effect_detail_id varchar(16) not null,
attack_id varchar(16) not null,
damage_id varchar(16) not null,
resource_type_id varchar(16) not null,
trait_id varchar(16),
constraint pk_effect unique(archetype_id, effect_type_id,
effect_detail_id, effect_apply_id,
attack_id, damage_id, resource_type_id),
constraint fk_effect_archetype_id foreign key(archetype_id)
references archetype (archetype_id)
);
create table ability
(
ability_id integer primary key autoincrement,
ability_description varchar(128)
);
create table ability_effect
(
ability_effect_id integer primary key autoincrement,
ability_id integer not null, -- FK ability
effect_id integer not null, -- FK effect
attack_id varchar(16) not null,
damage_id varchar(16) not null,
resource_type_id varchar(16) not null,
trait_id varchar(16),
constraint fk_ability_effect_ability_id foreign key (ability_id)
references ability (ability_id),
constraint fk_ability_effect_effect_id foreign key (effect_id)
references effect (effect_id)
);
I would like to do analysis of what codes appear in multiple tables under certains conditions. However I don't think the database schema suits the task very well but maybe there's something I don't know about that can help me. Here's a simplified schema:
CREATE TABLE "batchDescription" (
id INTEGER NOT NULL,
name TEXT NOT NULL UNIQUE,
PRIMARY KEY (id)
);
CREATE TABLE "simulationDetails" (
id INTEGER NOT NULL,
ko_index_id INTEGER NOT NULL,
batch_description_id INTEGER NOT NULL,
data1 REAL NOT NULL,
data2 INTEGER NOT NULL,
PRIMARY KEY (id)
FOREIGN KEY(ko_index_id) REFERENCES "koIndex" (id)
FOREIGN KEY(batch_description_id) REFERENCES "batchDescription" (id)
);
CREATE TABLE "koIndex" (
id INTEGER NOT NULL,
number_of_kos INTEGER NOT NULL,
PRIMARY KEY (id)
);
CREATE TABLE "1kos" (
ko_index_id INTEGER NOT NULL,
ko1 INTEGER NOT NULL,
PRIMARY KEY (ko_index_id)
FOREIGN KEY(ko_index_id) REFERENCES "koIndex" (id)
);
CREATE TABLE "2kos" (
ko_index_id INTEGER NOT NULL,
ko1 INTEGER NOT NULL,
ko2 INTEGER NOT NULL,
PRIMARY KEY (ko_index_id)
FOREIGN KEY(ko_index_id) REFERENCES "koIndex" (id)
);
CREATE TABLE "3kos" (
ko_index_id INTEGER NOT NULL,
ko1 INTEGER NOT NULL,
ko2 INTEGER NOT NULL,
ko3 INTEGER NOT NULL,
PRIMARY KEY (ko_index_id)
FOREIGN KEY(ko_index_id) REFERENCES "koIndex" (id)
);
This goes up to table "525kos" which has ko1 to ko525 in it - ko1 to ko525 are IDs that are primary keys in a table not shown here. I want to do an analysis of how often certain IDs are present under certain conditions. Here is a simple example to illustrate:
I would like to like to count the amount of times a certain ID (let's say 127) (in any koX column) in the "13kos" table occurs when simulationDetails.data1 not equal to 0. I would do this on a database called ko.db from the bash command line like:
for ko_idx in {1..13}; do sqlite3 ko.db "select count(ko${ko_idx}) from '13kos' where ko${ko_idx} = 127 and ko_index_id in (select ko_index_id from simulationDetails where data1 != 0);"; done
Already this is slow and inefficient but is simple compared to what I would like to do. What if I wanted to do an analysis of all the IDs in all possible columns in all "Xkos" tables and compare them to where data1 is equal and not equal to zero?
Can anybody direct me to a better way of doing this or is the schema design just not very good for this kind of analysis and I'll have to give up?
EDIT: Thought I'd add a bit of extra detailto avoid confusion. I suspect that a good way to achieve want I want would be to somehow combine all the "Xkos" tables into one temporary table and then search for certain IDs from that table. How would I combine all 525 ko tables without writing out each table name?
How would I combine all 525 ko tables without writing out each table
name?
Create a table with the same number of columns as the largest table (the table into which you merge) allowing nulls.
query the sqlite_master table using something like :-
SELECT * from sqlite_master WHERE name LIKE '%kos%' AND type = 'table'
Loop through the extracted table names building an INSERT SELECT for each table that will insert the rows from the tables into the table created in 1.
See 2. INSERT INTO table SELECT ...; especially in regard to handling missing columns.
All done, the table created in 1 will be populated accordingly.
Using mySQLAdmin tool, I try to create a table. The tool generates the SQL statement, and then replorts a "Can't create table" with no other clue on what error it is!
Here it is :
CREATE TABLE `C121535_vubridge`.`Products` (
`pr_ID` INTEGER UNSIGNED NOT NULL AUTO_INCREMENT,
`pr_Name` VARCHAR(45) NOT NULL,
`pr_Type` VARCHAR(2) NOT NULL COMMENT 'H=Hand Series V=VuBridge software E=Event Subs S=Sponsoring',
`pr_AuthorID` INTEGER UNSIGNED COMMENT '= m_ID (for Bridge Hand Series',
`pr_SponsorID` INTEGER UNSIGNED NOT NULL,
`pr_DateCreation` DATETIME NOT NULL,
`pr_Price` FLOAT NOT NULL,
`pr_DescriptionText` TEXT,
`pr_Description` VARCHAR(245),
PRIMARY KEY (`pr_ID`),
CONSTRAINT `FK_prAuthor` FOREIGN KEY `FK_prAuthor` (`pr_AuthorID`)
REFERENCES `Members` (`m_ID`)
ON DELETE SET NULL
ON UPDATE NO ACTION,
CONSTRAINT `FK_Sponsor` FOREIGN KEY `FK_Sponsor` (`pr_SponsorID`)
REFERENCES `Members` (`m_ID`)
ON DELETE SET NULL
ON UPDATE NO ACTION
) ENGINE = InnoDB;
Can someone help?
The CREATE TABLE works for me if I omit the foreign key references:
CREATE TABLE `Products` (
`pr_ID` INTEGER UNSIGNED NOT NULL AUTO_INCREMENT,
`pr_Name` VARCHAR(45) NOT NULL,
`pr_Type` VARCHAR(2) NOT NULL COMMENT 'H=Hand Series V=VuBridge software E=Event Subs S=Sponsoring',
`pr_AuthorID` INTEGER UNSIGNED COMMENT '= m_ID (for Bridge Hand Series',
`pr_SponsorID` INTEGER UNSIGNED NOT NULL,
`pr_DateCreation` DATETIME NOT NULL,
`pr_Price` FLOAT NOT NULL,
`pr_DescriptionText` TEXT,
`pr_Description` VARCHAR(245),
PRIMARY KEY (`pr_ID`)
)
...so I'm inclined to believe that C121535_vubridge.MEMBERS does not already exist. C121535_vubridge.MEMBERS needs to be created before the CREATE TABLE statement for the PRODUCTS table is run.
Just split up the create table and try one part at the time. This way you should be able to identify a single line that it fails on.
I do note in the reference manual that if a symbol subclause is given for the CONSTRAINT clause (in your case, the back-quoted strings before FOREIGN KEY in each clause, FK_prAuthor and FK_Sponsor) have to be unique over the database. Are they? If not, that symbol can be omitted and InnoDB will assign then automatically.
Similarly, the tables your FKs refer to may not have the structure that this create statement expects.
I have two tables. Table A and Table B. They are identical. Ever 10 min i need to check if there any changs happend (New and updated) to Table A and copy into Table B. And also enter in Table C if i see a differance and new.
I also need to log if there any new records in Table A to table B and Table C
Iam planning to do join and compare the records. If i do that i might miss the new records. Is there any better way to do this kind of sync. It has to be done in SQL i can not use any other tools like SSIS.
Here's what I came up with in making some simple tables in SQL:
# create some sample tables and data
DROP TABLE alpha;
DROP TABLE beta;
DROP TABLE charlie;
CREATE TABLE `alpha` (
`id` INT(10) UNSIGNED NOT NULL AUTO_INCREMENT,
`data` VARCHAR(32) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=MYISAM DEFAULT CHARSET=latin1;
CREATE TABLE `beta` (
`id` INT(10) UNSIGNED NOT NULL AUTO_INCREMENT,
`data` VARCHAR(32) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=MYISAM DEFAULT CHARSET=latin1;
CREATE TABLE `charlie` (
`id` INT(10) UNSIGNED NOT NULL AUTO_INCREMENT,
`data` VARCHAR(32) DEFAULT NULL,
`type` VARCHAR(16) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=MYISAM DEFAULT CHARSET=latin1;
INSERT INTO alpha (data) VALUES ("a"), ("b"), ("c"), ("d"), ("e");
INSERT INTO beta (data) VALUES ("a"), ("b"), ("c");
# note new records of A, log in C
INSERT INTO charlie (data, type)
(SELECT data, "NEW"
FROM alpha
WHERE id NOT IN
(SELECT id
FROM beta));
# insert new records of A into B
INSERT INTO beta (data)
(SELECT data
FROM alpha
WHERE id NOT IN
(SELECT id
FROM beta));
# make a change in alpha only
UPDATE alpha
SET data = "x"
WHERE data = "c";
# note changed records of A, log in C
INSERT INTO charlie (data, type)
(SELECT alpha.data, "CHANGE"
FROM alpha, beta
WHERE alpha.data != beta.data
AND alpha.id = beta.id);
# update changed records of A in B
UPDATE beta, alpha
SET beta.data = alpha.data
WHERE alpha.data != beta.data
AND alpha.id = beta.id;
You would of course have to expand this for the type of data, number of fields, etc. but this is a basic concept if it helps.
It's a pity that you can't use SSIS (not allowed?) because it's built for this kind of thing. Anyway, using pure SQL you should be able to something like the following: if your tables have got a created/updated timestamp column, then you could query Table B for the highest one and get all records from table A with timestamps higher than that one.
If there's no timestamp to use, hopefully there's a PK like an int that can be used in the same way.
Hope that helps?
Valentino.
I would try using a trigger or transactional replication.
Hopefully you have a good unique key that is used in the tables. To get new records you can do the following:
SELECT * FROM tableA
WHERE NOT EXISTS( SELECT * FROM tableB WHERE pkey.tableA = pkey.TableB)