PostgreSQL stored function with multiple queries - sql

I'm working on a PostgreSQL function, but having a real struggle with it. It's not my main area so that's probably why, but I wanted to see if this is doable.
I'm trying to create a function to generate records based on year.
In php I'd do something along the lines of:
function recordsByYear($year=''){
$years = array();
if(empty($year)){
$sql = "SELECT year FROM myTable GROUP BY year ORDER BY year;";
$res = $conn->query($sql);
if($res !== false){
$data = $res->fetchAll(PDO::FETCH_COLUMN);
foreach($data as $oneYear){
$years[] = $oneYear;
}
}
}
else{
$years[] = $year;
}
foreach($years as $thisYear){
//do some queries based on $thisYear
}
}
But I'd like to create a function inside of PostgreSQL, and struggling because I'm not familiar enough with how it all works. I'd like to return a table. I can do some basic stuff, but haven't been able to get something like this working where I have one query then loop through those results running an additional query for each year, then combine the results of that second query and spit out the results as a table.

Is hard to guess without more information but you probably need a JOIN
SELECT *
FROM (
SELECT DISTINCT year
FROM myTable
) y
JOIN ( SELECT year, ... <some query>
....
) t
ON y.year = t.year

PostgreSQL function:
CREATE FUNCTION record_years() RETURNS SETOF int AS $$
SELECT DISTINCT year FROM myTable ORDER BY 1
$$ LANGUAGE SQL;
Your PHP code:
$sql = "SELECT record_years()";
or
$sql = "SELECT year FROM record_years() AS year";
or alternatively all values in a single row, as a JSON array:
$sql = "SELECT array_to_json(ARRAY(SELECT record_years())) AS arr";
$res = $conn->query($sql);
if($res !== false){
$row = $res->fetch();
$years[] = json_decode($row['arr']);
}

Related

How to select from subquery using Laravel Query Builder?

I'd like to get value by the following SQL using Eloquent ORM.
- SQL
SELECT COUNT(*) FROM
(SELECT * FROM abc GROUP BY col1) AS a;
Then I considered the following.
- Code
$sql = Abc::from('abc AS a')->groupBy('col1')->toSql();
$num = Abc::from(\DB::raw($sql))->count();
print $num;
I'm looking for a better solution.
Please tell me simplest solution.
In addition to #delmadord's answer and your comments:
Currently there is no method to create subquery in FROM clause, so you need to manually use raw statement, then, if necessary, you will merge all the bindings:
$sub = Abc::where(..)->groupBy(..); // Eloquent Builder instance
$count = DB::table( DB::raw("({$sub->toSql()}) as sub") )
->mergeBindings($sub->getQuery()) // you need to get underlying Query Builder
->count();
Mind that you need to merge bindings in correct order. If you have other bound clauses, you must put them after mergeBindings:
$count = DB::table( DB::raw("({$sub->toSql()}) as sub") )
// ->where(..) wrong
->mergeBindings($sub->getQuery()) // you need to get underlying Query Builder
// ->where(..) correct
->count();
Laravel v5.6.12 (2018-03-14) added fromSub() and fromRaw() methods to query builder (#23476).
The accepted answer is correct but can be simplified into:
DB::query()->fromSub(function ($query) {
$query->from('abc')->groupBy('col1');
}, 'a')->count();
The above snippet produces the following SQL:
select count(*) as aggregate from (select * from `abc` group by `col1`) as `a`
The solution of #JarekTkaczyk it is exactly what I was looking for. The only thing I miss is how to do it when you are using
DB::table() queries. In this case, this is how I do it:
$other = DB::table( DB::raw("({$sub->toSql()}) as sub") )->select(
'something',
DB::raw('sum( qty ) as qty'),
'foo',
'bar'
);
$other->mergeBindings( $sub );
$other->groupBy('something');
$other->groupBy('foo');
$other->groupBy('bar');
print $other->toSql();
$other->get();
Special atention how to make the mergeBindings without using the getQuery() method
From laravel 5.5 there is a dedicated method for subqueries and you can use it like this:
Abc::selectSub(function($q) {
$q->select('*')->groupBy('col1');
}, 'a')->count('a.*');
or
Abc::selectSub(Abc::select('*')->groupBy('col1'), 'a')->count('a.*');
There are many readable ways to do these kinds of queries at the moment (Laravel 8).
// option 1: DB::table(Closure, alias) for subquery
$count = DB::table(function ($sub) {
$sub->from('abc')
->groupBy('col1');
}, 'a')
->count();
// option 2: DB::table(Builder, alias) for subquery
$sub = DB::table('abc')->groupBy('col1');
$count = DB::table($sub, 'a')->count();
// option 3: DB::query()->from(Closure, alias)
$count = DB::query()
->from(function ($sub) {
$sub->from('abc')
->groupBy('col1')
}, 'a')
->count();
// option 4: DB::query()->from(Builder, alias)
$sub = DB::table('abc')->groupBy('col1');
$count = DB::query()->from($sub, 'a')->count();
For such small subqueries, you could even try fitting them in a single line with PHP 7.4's short closures but this approach can be harder to mantain.
$count = DB::table(fn($sub) => $sub->from('abc')->groupBy('col1'), 'a')->count();
Note that I'm using count() instead of explicitly writing the count(*) statement and using get() or first() for the results (which you can easily do by replacing count() with selectRaw(count(*))->first()).
The reason for this is simple: It returns the number instead of an object with an awkwardly named property (count(*) unless you used an alias in the query)
Which looks better?
// using count() in the builder
echo $count;
// using selectRaw('count(*)')->first() in the builder
echo $count->{'count(*)'};
Correct way described in this answer: https://stackoverflow.com/a/52772444/2519714
Most popular answer at current moment is not totally correct.
This way https://stackoverflow.com/a/24838367/2519714 is not correct in some cases like: sub select has where bindings, then joining table to sub select, then other wheres added to all query. For example query:
select * from (select * from t1 where col1 = ?) join t2 on col1 = col2 and col3 = ? where t2.col4 = ?
To make this query you will write code like:
$subQuery = DB::query()->from('t1')->where('t1.col1', 'val1');
$query = DB::query()->from(DB::raw('('. $subQuery->toSql() . ') AS subquery'))
->mergeBindings($subQuery->getBindings());
$query->join('t2', function(JoinClause $join) {
$join->on('subquery.col1', 't2.col2');
$join->where('t2.col3', 'val3');
})->where('t2.col4', 'val4');
During executing this query, his method $query->getBindings() will return bindings in incorrect order like ['val3', 'val1', 'val4'] in this case instead correct ['val1', 'val3', 'val4'] for raw sql described above.
One more time correct way to do this:
$subQuery = DB::query()->from('t1')->where('t1.col1', 'val1');
$query = DB::query()->fromSub($subQuery, 'subquery');
$query->join('t2', function(JoinClause $join) {
$join->on('subquery.col1', 't2.col2');
$join->where('t2.col3', 'val3');
})->where('t2.col4', 'val4');
Also bindings will be automatically and correctly merged to new query.
I like doing something like this:
Message::select('*')
->from(DB::raw("( SELECT * FROM `messages`
WHERE `to_id` = ".Auth::id()." AND `isseen` = 0
GROUP BY `from_id` asc) as `sub`"))
->count();
It's not very elegant, but it's simple.
This works fine
$q1 = DB::table('tableA')->groupBy('col');
$data = DB::table(DB::raw("({$q1->toSql()}) as sub"))->mergeBindings($q1)->get();
I could not made your code to do the desired query, the AS is an alias only for the table abc, not for the derived table.
Laravel Query Builder does not implicitly support derived table aliases, DB::raw is most likely needed for this.
The most straight solution I could came up with is almost identical to yours, however produces the query as you asked for:
$sql = Abc::groupBy('col1')->toSql();
$count = DB::table(DB::raw("($sql) AS a"))->count();
The produced query is
select count(*) as aggregate from (select * from `abc` group by `col1`) AS a;
->selectRaw('your subquery as somefield')
Deriving off mpskovvang's answer, here is what it would look like using eloquent model. (I tried updating mpskovvang answer to include this, but there's too many edit requests for it.)
$qry = Abc::where('col2', 'value')->groupBy('col1')->selectRaw('1');
$num = Abc::from($qry, 'q1')->count();
print $num;
Produces...
SELECT COUNT(*) as aggregate FROM (SELECT 1 FROM Abc WHERE col2='value' GROUP BY col1) as q1

Using more than one queries on perl DBI

I'm sure there is a better way to do this on the same line, but I'm unable to figure out how, since I'm a beginner in Perl. Basically what I need to do is select, delete and count the results.
$sth = $dbh->prepare("SELECT env,server, mwp.is_reference where env='$ARGV[1]';");
$sth->execute();
$sth2 = $dbh->prepare("delete from mwp.is_info_package where env='$ARGV[1]'");
$sth2->execute();
$sth3 = $dbh->prepare("SELECT count(1) from mwp.is_reference where env='$ARGV[1]'");
$sth3->execute()
The objective is how do i use the 3 queries at the same line, instead having 3 executes.
Well you could start out using placeholders ( '?' ).
my #qlist
= ( 'SELECT env,server FROM mwp.is_reference where env=?'
, 'DELETE mwp.is_info_package WHERE env=?'
, 'SELECT count(1) FROM mwp.is_reference where env=?'
);
And then you can iterate through them like this:
my $env = $ARGV[1];
foreach my $query ( #qlist ) {
$dbh->prepare( $query )->execute( $env );
Carp::croak( $dbh->errstr ) if $dbh->err;
}
But of course, you really want to select the two outputs, don't you?
use Carp qw<croak>;
my $select_query = 'SELECT env,server FROM mwp.is_reference where env=?';
my $delete_query = 'DELETE mwp.is_info_package WHERE env=?';
my $count_query = 'SELECT count(1) FROM mwp.is_reference where env=?';
my %empty_atts;
my $rows
= $dbh->selectall_arrayref( $select_query, \%empty_atts, $env )
;
croak( $dbh->errstr ) if $dbh->err;
$dbh->prepare( $delete_query )->execute( $env );
croak( $dbh->errstr ) if $dbh->err;
my ( $count )
= $dbh->selectrow_array( $count_query, \%empty_atts, $env )
;
croak( $dbh->errstr ) if $dbh->err;
I solved the problem using the following query statement:
$sth = $dbh->prepare("select env,iserver, ( select count(1) from is_reference where env='$ARGV[1]' ) as total from is_reference where env='$ARGV[1]'");
not the most elegant way, but solved my problem with the less lines. Regarding the delete query, i moved to another condition to check if the table have data or not.
THanks all.
You could use a stored procedure that performs those functions and returns the results of the select as well as a count, then you only need to do:
my $sth = $dbh->prepare("EXEC procedure_name ?");
$sth->execute( $ARGV[1] );
As an aside, the way you're using prepare and execute is undesirable. You use prepare to avoid having to have Perl variables directly in the query; your Perl variables should be passed to execute() as values, not part of the string given to prepare(). There are a number of good reasons to do this, including protection against SQL Injection attacks.
I also noticed oddness in your last SQL query. I think you probably want
SELECT count(env) FROM mwp.is_reference where env=?
Otherwise it will always return "1" as the count... Likewise, unless there are database triggers doing something interesting, you could combine the first and last query into one this way (I'll leave count(1) for this in case that's really what you want):
my $sth = $dbh->prepare('SELECT env,server FROM mwp.is_reference where env=?');
$sth = $sth->execute( $ARGV[1] );
my $result_set = $sth->fetchall_arrayref();
my $count = scalar #{ $result_set };
The $result_set will be a reference to an ARRAY of ARRAYRefs containing the results; $count will contain the number of rows in that result set.

How to execute query with subqueries on a table and get a Rowset object as a result in Zend?

I'm currently struggling on how to execute my query on a Table object in Zend and get a Rowset in return. Reason I need particularly THIS is because I'm modifying a code for existing project and I don't have much flexibility.
Query:
SELECT *
FROM `tblname` ud
WHERE ud.user_id = some_id
AND
(
(ud.reputation_level > 1)
OR
(
(SELECT COUNT( * )
FROM `tblname` t
WHERE t.user_id = ud.user_id
AND t.category_id <=> ud.category_id
AND t.city_id <=> ud.city_id
) = 1
)
)
Is there a way to describe this query using Select object?
Previous SQL solution was very simple and consisted of one WHERE clause:
$where = $this->getAdapter()->quoteInto("user_id = ?",$user_id);
return $this->fetchAll($where);
I need to produce same type of the result (so that it could be processed by existing code) but for more complicated query.
Things I've tried
$db = Zend_Db_Table::getDefaultAdapter();
return $db->query($sql)->fetchAll();
---------------- OR ----------------------
return $this->fetchAll($select);
---------------- OR ----------------------
return $this->_db->query($sql)->fetchAll();
But they either produce arrays instead of objects or fail with Cardinality violation message.
I would appreciate any help on how to handle SQL text queries in Zend.
$dbAdapter = Zend_Db_Table::getDefaultAdapter();
//change the fetch mode becouse you don't like the array
$dbAdapter->setFetchMode(Zend_Db::FETCH_OBJ);
$sql = "you're long sql here";
$result = $dbAdapter->fetchAll($sql);
Zend_Debug::dump($result);
exit;
For a list of all fetch modes go to Zend_Db_Adapter
To write you're query using Zend_Db_Select instead of manual string , look at Zend_Db_Slect

SELECT MAX query returns only 1 variable + codeigniter

I use codeigniter and have an issue about SELECT MAX ... I couldnot find any solution at google search...
it looks like it returns only id :/ it's giving error for other columns of table :/
Appreciate helps, thanks!
Model:
function get_default()
{
$this->db->select_max('id');
$query = $this->db->getwhere('gallery', array('cat' => "1"));
if($query->num_rows() > 0) {
return $query->row_array(); //return the row as an associative array
}
}
Controller:
$default_img = $this->blabla_model->get_default();
$data['default_id'] = $default_img['id']; // it returns this
$data['default_name'] = $default_img['gname']; // it gives error for gname although it is at table
To achieve your goal, your desire SQL can look something like:
SELECT *
FROM gallery
WHERE cat = '1'
ORDER BY id
LIMIT 1
And to utilise CodeIgniter database class:
$this->db->select('*');
$this->db->where('cat', '1');
$this->db->order_by('id', 'DESC');
$this->db->limit(1);
$query = $this->db->get('gallery');
That is correct: select_max returns only the value, and no other column. From the specs:
$this->db->select_max('age');
$query = $this->db->get('members');
// Produces: SELECT MAX(age) as age FROM members
You may want to read the value first, and run another query.
For an id, you can also use $id = $this->db->insert_id();
See also: http://www.hostfree.com/user_guide/database/active_record.html#select
CodeIgniter will select * if nothing else is selected. By setting select_max() you are populating the select property and therefore saying you ONLY want that value.
To solve this, just combine select_max() and select():
$this->db->select('somefield, another_field');
$this->db->select_max('age');
or even:
$this->db->select('sometable.*', FALSE);
$this->db->select_max('age');
Should do the trick.
It should be noted that you may of course also utilize your own "custom" sql statements in CodeIgniter, you're not limited to the active record sql functions you've outlined thus far. Another active record function that CodeIgniter provides is $this->db->query(); Which allows you to submit your own SQL queries (including variables) like so:
function foo_bar()
{
$cat = 1;
$limit = 1;
$sql = "
SELECT *
FROM gallery
WHERE cat = $cat
ORDER BY id
LIMIT $limit
";
$data['query'] = $this->db->query($sql);
return $data['query'];
}
Recently I have been utilizing this quite a bit as I've been doing some queries that are difficult (if not annoying or impossible) to pull off with CI's explicit active record functions.
I realize you may know this already, just thought it would help to include for posterity.
2 helpful links are:
http://codeigniter.com/user_guide/database/results.html
http://codeigniter.com/user_guide/database/examples.html

MySQL: Compare differences between two tables

Same as oracle diff: how to compare two tables? except in mysql.
Suppose I have two tables, t1 and t2 which are identical in layout but which may contain different data.
What's the best way to diff these two tables?
To be more precise, I'm trying to figure out a simple SQL query that tells me if data from one row in t1 is different from the data from the corresponding row in t2
It appears I cannot use the intersect nor minus. When I try
SELECT * FROM robot intersect SELECT * FROM tbd_robot
I get an error code:
[Error Code: 1064, SQL State: 42000] You have an error in your SQL
syntax; check the manual that corresponds to your MySQL server version
for the right syntax to use near 'SELECT * FROM tbd_robot' at line 1
Am I doing something syntactically wrong? If not, is there another query I can use?
Edit: Also, I'm querying through a free version DbVisualizer. Not sure if that might be a factor.
INTERSECT needs to be emulated in MySQL:
SELECT 'robot' AS `set`, r.*
FROM robot r
WHERE ROW(r.col1, r.col2, …) NOT IN
(
SELECT col1, col2, ...
FROM tbd_robot
)
UNION ALL
SELECT 'tbd_robot' AS `set`, t.*
FROM tbd_robot t
WHERE ROW(t.col1, t.col2, …) NOT IN
(
SELECT col1, col2, ...
FROM robot
)
You can construct the intersection manually using UNION. It's easy if you have some unique field in both tables, e.g. ID:
SELECT * FROM T1
WHERE ID NOT IN (SELECT ID FROM T2)
UNION
SELECT * FROM T2
WHERE ID NOT IN (SELECT ID FROM T1)
If you don't have a unique value, you can still expand the above code to check for all fields instead of just the ID, and use AND to connect them (e.g. ID NOT IN(...) AND OTHER_FIELD NOT IN(...) etc)
I found another solution in this link
SELECT MIN (tbl_name) AS tbl_name, PK, column_list
FROM
(
SELECT ' source_table ' as tbl_name, S.PK, S.column_list
FROM source_table AS S
UNION ALL
SELECT 'destination_table' as tbl_name, D.PK, D.column_list
FROM destination_table AS D
) AS alias_table
GROUP BY PK, column_list
HAVING COUNT(*) = 1
ORDER BY PK
select t1.user_id,t2.user_id
from t1 left join t2 ON t1.user_id = t2.user_id
and t1.username=t2.username
and t1.first_name=t2.first_name
and t1.last_name=t2.last_name
try this. This will compare your table and find all matching pairs, if any mismatch return NULL on left.
Based on Haim's answer I created a PHP code to test and display all the differences between two databases.
This will also display if a table is present in source or test databases.
You have to change with your details the <> variables content.
<?php
$User = "<DatabaseUser>";
$Pass = "<DatabasePassword>";
$SourceDB = "<SourceDatabase>";
$TestDB = "<DatabaseToTest>";
$link = new mysqli( "p:". "localhost", $User, $Pass, "" );
if ( mysqli_connect_error() ) {
die('Connect Error ('. mysqli_connect_errno() .') '. mysqli_connect_error());
}
mysqli_set_charset( $link, "utf8" );
mb_language( "uni" );
mb_internal_encoding( "UTF-8" );
$sQuery = 'SELECT TABLE_NAME FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA="'. $SourceDB .'";';
$SourceDB_Content = query( $link, $sQuery );
if ( !is_array( $SourceDB_Content) ) {
echo "Table $SourceDB cannot be accessed";
exit(0);
}
$sQuery = 'SELECT TABLE_NAME FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA="'. $TestDB .'";';
$TestDB_Content = query( $link, $sQuery );
if ( !is_array( $TestDB_Content) ) {
echo "Table $TestDB cannot be accessed";
exit(0);
}
$SourceDB_Tables = array();
foreach( $SourceDB_Content as $item ) {
$SourceDB_Tables[] = $item["TABLE_NAME"];
}
$TestDB_Tables = array();
foreach( $TestDB_Content as $item ) {
$TestDB_Tables[] = $item["TABLE_NAME"];
}
//var_dump( $SourceDB_Tables, $TestDB_Tables );
$LookupTables = array_merge( $SourceDB_Tables, $TestDB_Tables );
$NoOfDiscrepancies = 0;
echo "
<table border='1' width='100%'>
<tr>
<td>Table</td>
<td>Found in $SourceDB (". count( $SourceDB_Tables ) .")</td>
<td>Found in $TestDB (". count( $TestDB_Tables ) .")</td>
<td>Test result</td>
<tr>
";
foreach( $LookupTables as $table ) {
$FoundInSourceDB = in_array( $table, $SourceDB_Tables ) ? 1 : 0;
$FoundInTestDB = in_array( $table, $TestDB_Tables ) ? 1 : 0;
echo "
<tr>
<td>$table</td>
<td><input type='checkbox' ". ($FoundInSourceDB == 1 ? "checked" : "") ."></td>
<td><input type='checkbox' ". ($FoundInTestDB == 1 ? "checked" : "") ."></td>
<td>". compareTables( $SourceDB, $TestDB, $table ) ."</td>
</tr>
";
}
echo "
</table>
<br><br>
No of discrepancies found: $NoOfDiscrepancies
";
function query( $link, $q ) {
$result = mysqli_query( $link, $q );
$errors = mysqli_error($link);
if ( $errors > "" ) {
echo $errors;
exit(0);
}
if( $result == false ) return false;
else if ( $result === true ) return true;
else {
$rset = array();
while ( $row = mysqli_fetch_assoc( $result ) ) {
$rset[] = $row;
}
return $rset;
}
}
function compareTables( $source, $test, $table ) {
global $link;
global $NoOfDiscrepancies;
$sQuery = "
SELECT column_name,ordinal_position,data_type,column_type FROM
(
SELECT
column_name,ordinal_position,
data_type,column_type,COUNT(1) rowcount
FROM information_schema.columns
WHERE
(
(table_schema='$source' AND table_name='$table') OR
(table_schema='$test' AND table_name='$table')
)
AND table_name IN ('$table')
GROUP BY
column_name,ordinal_position,
data_type,column_type
HAVING COUNT(1)=1
) A;
";
$result = query( $link, $sQuery );
$data = "";
if( is_array( $result ) && count( $result ) > 0 ) {
$NoOfDiscrepancies++;
$data = "<table><tr><td>column_name</td><td>ordinal_position</td><td>data_type</td><td>column_type</td></tr>";
foreach( $result as $item ) {
$data .= "<tr><td>". $item["column_name"] ."</td><td>". $item["ordinal_position"] ."</td><td>". $item["data_type"] ."</td><td>". $item["column_type"] ."</td></tr>";
}
$data .= "</table>";
return $data;
}
else {
return "Checked but no discrepancies found!";
}
}
?>
Problem below, is to compare table before and after i do big update!.
If you use Linux, you can use commands as follow:
In terminal,
mysqldump -hlocalhost -uroot -p schema_name_here table_name_here > /home/ubuntu/database_dumps/dump_table_before_running_update.sql
mysqldump -hlocalhost -uroot -p schema_name_here table_name_here > /home/ubuntu/database_dumps/dump_table_after_running_update.sql
diff -uP /home/ubuntu/database_dumps/dump_some_table_after_running_update.sql /home/ubuntu/database_dumps/dump_table_before_running_update.sql > /home/ubuntu/database_dumps/diff.txt
You will need online tools for
Formatting SQL exported from the dumps,
e.g http://www.dpriver.com/pp/sqlformat.htm [Not the best I've seen]
We have diff.txt, you have to take manually the + - showing inside, which is 1 line of insert statements, that has the values.
Do diff online for the 2 lines - & + in diff.txt, past them in online diff tool
e.g https://www.diffchecker.com [you can save and share it, and has no limit on file size!]
Note: be extra careful if its sensitive/production data!
you can try The big data comparison platform in https://github.com/zhugezifang/dataCompare
this is a introduction of it
Design and practice of open source big data comparison platform
1. Background & current situation
In the process of developing large numbers, it is often encountered that data migration or upgrade, or different business parties have processed data according to their needs, but think that the data on both sides is still the same, so it will be necessary to manually compare the data. So is the data on both sides consistent? If not, what are the differences?
If there is no platform, you need to manually write some SQL scripts for comparison, and there is no evaluation standard. This is inefficient.
"Alibaba's Road to Big Data" actually mentions such a platform, but because it is not used externally, the introduction in the book is relatively simple. Based on previous work experience, a big data comparison platform was developed to assist in verifying data, named dataCompare.
Main solutions:
(1) Verify data and data comparison, which wastes great labor costs
(2) Without a set of standards, the results of verification are difficult to evaluate
(3) Automatic data verification and comparison can be achieved by interface interaction, check or low-code
[enter image description here][1]
2. Purpose
(1) Automatic data verification and comparison can be achieved by interface interaction, check or low-code.
(2) The data team's data comparison efficiency is increased by at least about 50%.
(3) A set of unified data verification scheme to meet the standard specifications of data verification and comparison
3. System architecture design
4. The current version has implemented the following functions
(1) Low-code simple configuration completes the core function of data comparison
(2) Data magnitude comparison and data consistency comparison
5. Follow-up Development Plan
(1) Discrepancy case finding
(2) Data pointer detection---- enumeration value detection, range detection, numerical detection, primary key mode detection
(3) Data comparison task is scheduled and automatically scheduled
(4) Automatically send an email report to the comparison results
6. The core code is opening in githup
https://github.com/zhugezifang/dataCompare
[enter image description here][1]
Based on Haim's answer here's a simplified example if you're looking to compare values that exist in BOTH tables, otherwise if there's a row in one table but not the other it will also return it....
Took me a couple of hours to figure out. Here's a fully tested simply query for comparing "tbl_a" and "tbl_b"
SELECT ID, col
FROM
(
SELECT
tbl_a.ID, tbl_a.col FROM tbl_a
UNION ALL
SELECT
tbl_b.ID, tbl_b.col FROM tbl_b
) t
WHERE ID IN (select ID from tbl_a) AND ID IN (select ID from tbl_b)
GROUP BY
ID, col
HAVING COUNT(*) = 1
ORDER BY ID
So you need to add the extra "where in" clause:
WHERE ID IN (select ID from tbl_a) AND ID IN (select ID from tbl_b)
Also:
For ease of reading if you want to indicate the table names you can use the following:
SELECT tbl, ID, col
FROM
(
SELECT
tbl_a.ID, tbl_a.col, "name_to_display1" as "tbl" FROM tbl_a
UNION ALL
SELECT
tbl_b.ID, tbl_b.col, "name_to_display2" as "tbl" FROM tbl_b
) t
WHERE ID IN (select ID from tbl_a) AND ID IN (select ID from tbl_b)
GROUP BY
ID, col
HAVING COUNT(*) = 1
ORDER BY ID
you can user my own developed tool
https://github.com/hardeepvicky/MySql-Schema-Compare
I tried the above answer but found that if one table has null values and the second table has values in a column then the intersect code above does not report this fact.
select p.pcn,p.period,p.account_no,p.ytd_debit,a.ytd_debit
-- select count(*) -- 157,283
from Plex.account_period_balance p -- 157,283/202207,148,998
join Azure.account_period_balance a -- 157,283/202207,148,998
on p.pcn = a.pcn
and p.period = a.period
and p.account_no = a.account_no -- 157,283
where p.period_display = a.period_display -- 157,283
and p.debit = a.debit -- 157,283
-- and p.ytd_debit = a.ytd_debit -- 148,998
-- and p.ytd_debit != a.ytd_debit -- 0