Migration from Mysql to postgre - pdo

I was working on migrating data from MYSQL to postgre.
How can I insert mysql data in postgre table?
Thanks in advance
The code:
try {
$conexion = new PDO("mysql:host=$direccion;dbname=$base_d", $us, $contra);
$conexion->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
$consulta = $conexion->prepare('SELECT * FROM ' . $tabla_b);
//Ejecuto la consulta
$consulta->execute();
$consulta_array = $consulta->fetchAll();
var_export($consulta_array);
$dbconn = pg_connect("host=$direccion2 dbname=$base_d2 user=$us2 password=$contra2")
or die('No se ha podido conectar: ' . pg_last_error());
$insercion= pg_copy_from($conexion, $tabla_b2, $consulta_array,",");
} catch (PDOException $e) {
echo "Error conectando con la base de datos: " . $e->getMessage();
}
}

Have you tried working with Pgloader?
I've found it very quick and effective way to migrate MySql to Postgres.
Here is a quick example of how to proceed with Pgloader.
Create a .load file with the below content. (you can exclude the unwanted lines with a "--" in front)
load database
from mysql://root#localhost/base_d
into postgresql:///base_d
WITH include drop, create tables, no truncate,
create indexes, reset sequences, foreign keys
-- SET maintenance_work_mem to '128MB', work_mem to '12MB', search_path to 'base_d'
CAST type datetime to timestamptz
drop default drop not null using zero-dates-to-null,
type date drop not null drop default using zero-dates-to-null
-- MATERIALIZE VIEWS film_list, staff_list
-- INCLUDING ONLY TABLE NAMES MATCHING ~/film/, 'actor'
-- EXCLUDING TABLE NAMES MATCHING ~<ory>
BEFORE LOAD DO
$$ create schema if not exists base_d; $$;
Note that here pgloader will benefit from the meta-data information found in the MySQL database to create a PostgreSQL database capable of hosting the data as described, then load the data.
Note that we are using the MATERIALIZE VIEWS clause of pgloader: the selected views here will be migrated over to PostgreSQL with their contents.
Download Pgloader from here.
Now you can use the pgloader command to migrate data.
$ pgloader base_d.load (the file in which you saved the above content)
Reference : http://pgloader.io/howto/mysql.html

Related

Perl DBI modifying Oracle database by creating a VIEW

I wrote a Perl script to check the data in an Oracle database. Because the query process is very complex I chose to create a VIEW in the middle. Using this view the code could be largely simplified.
The Perl code run well when I used it to query the database starting from a file, like Perl mycode.pl file_a. The Perl code reads lines from file_a and creates/updates the view until the end of the input. The results I achieved are completely right.
The problem came when I simultaneously run
perl mycode.pl file_a
and
perl mycode.pl file_b
to access the same database. According to my observation, the VIEW used by the first process will be modified by the second process. These two processes were intertwined on the same view.
Is there any suggestion to make these two processes not conflict with one another?
The Perl code for querying database is normally like this, but the details in each real query is more complex.
my ($gcsta,$gcsto,$cms) = #t; #(details of #t is read from a line in file a or b)
my $VIEWSS = 'CREATE OR REPLACE VIEW VIEWSS AS SELECT ID,GSTA,GSTO,GWTA FROM TABLEA WHERE GSTA='.$gcsta.' AND GSTO='.$gcsto.' AND CMS='.$cms;
my $querying = q{ SELECT COUNT(*) FROM VIEWSS WHERE VIEWSS.ID=1};
my $inner_sth = $dbh->prepare($VIEWSS);
my $inner_rv = $inner_sth->execute();
$inner_sth = $dbh->prepare($querying);
$inner_rv = $inner_sth->execute();
You must
Create the view only once, and use it everywhere
Use placeholders in your SQL statements, and pass the actual parameters with the call to execute
Is this the full extent of your SQL? Probably not, but if so it really is fairly simple.
Take a look at this refactoring for some ideas. Note that is uses a here document to express the SQL. The END_SQL marker for the end of the text must have no whitespace before or after it.
If your requirement is more complex than this then please describe it to us so that we can better help you
my $stmt = $dbh->prepare(<<'END_SQL');
SELECT count(*)
FROM tablea
WHERE gsta = ? AND gsto = ? AND cms= ? AND id = 1
END_SQL
my $rv = $stmt->execute($gcsta, $gcsto, $cms);
If you must use a view then you should use placeholders in the CREATE VIEW as before, and make every set of changes into a transaction so that other processes can't interfere. This involves disabling AutoCommit when you create the database handle $dbh and adding a call to $dbh->commit when all the steps are complete
use strict;
use warnings;
use DBI;
my $dbh = DBI->connect('dbi:Oracle:mydbase', 'user', 'pass',
{ AutoCommit => 0, RaiseError => 1 } );
my $make_view = $dbh->prepare(<<'END_SQL');
CREATE OR REPLACE VIEW viewss AS
SELECT id, gsta, gsto, gwta
FROM tablea
WHERE gsta = ? AND gsto = ? AND cms= ? AND id = 1
END_SQL
my $get_count = $dbh->prepare(<<'END_SQL');
SELECT count(*)
FROM viewss
WHERE id = 1
END_SQL
while (<>) {
my ($gcsta, $gcsto, $cms) = split;
my $rv = $make_view->execute($gcsta, $gcsto, $cms);
$rv = $get_count->execute;
my ($count) = $get_count->fetchrow_array;
$dbh->commit;
}
Is the view going to be the same or different?
If the views are all the same then create it only once, or check if it exists with the all_views table : http://docs.oracle.com/cd/B12037_01/server.101/b10755/statviews_1202.htm#i1593583
You can easily create a view including your pid with the $$ variable to be the pid, but it wont be unique across computers, oracle has also some unique ids, see http://docs.oracle.com/cd/B14117_01/server.101/b10759/functions150.htm, for example, the SESSIONID.
But do you really need to do this? why dont you prepare a statement and then execute it? http://search.cpan.org/dist/DBI/DBI.pm#prepare
thanks,
mike

Perl DBI - transfer data between two sql servers - fetchall_arrayref

I have 2 servers: dbh1 and dbh2 where I query dbh1 and pull data via fetchall_arrayref method. Once I execute the query, I want to insert the output from dbh1 into a temp table on server dbh2.
I am able to establish access to both servers at the same time and am able to pull data from both.
1. I pull data from dbh1:
while($row = shift(#$rowcache) || shift(#{$rowcache=$sth1->fetchall_arrayref(undef, $max_rows)})) {
#call to sub insert2tempData
&insert2tempData(values #{$row});
}
2. Then on dbh2 I have an insert query:
INSERT INTO ##population (someid, Type, anotherid)
VALUES ('123123', 'blah', '634234');
Question:
How can I insert the bulk result of the fetchall_arrayref from dbh1 into the temp table on server dbh2 (without looping through individual records)?
Ok - so i was able to resolve this issue and was able to implement the following code:
my $max_rows = 38;
my $rowcache = [];
my $sum = 0;
if($fldnames eq "ALL"){ $fldnames = join(',', #{ $sth1->{NAME} });}
my $ins = $dbh2->prepare("insert into $database2.dbo.$tblname2 ($fldnames) values $fldvalues");
my $fetch_tuple_sub = sub { shift(#$rowcache) || shift(#{$rowcache=$sth1->fetchall_arrayref(undef, $max_rows)}) };
my #tuple_status;
my $rc;
$rc = $ins->execute_for_fetch($fetch_tuple_sub, \#tuple_status);
my #errors = grep { ref $_ } #tuple_status;
The transfer works but it is still slower than if I were to transfer data manually through SQL Server export/import wizard . The issue that i notice is that the data flows row by row into the destination and I was wondering if it is possible to increase the bulk transfer size. It downloads the data extremely fast, but when i combine download and upload then the speeds decreases dramatically and it takes up to 10 minutes to transfer a 5000 row table between servers.
It would be better if you said what your goal was (speed?) rather than asking a specific question on avoiding looping.
For a Perl/DBI way:
Look at DBI's execute_array and execute_for_fetch however as you've not told us which DBD you are using it is impossible to say more. Not all DBDs support bulk insert and when they don't DBI emulates it. DBD::Oracle does and DBD::ODBC does (in recent versions see odbc_array_operations) but in the latter it is off by default.
You didn't mention which version of SQL Server you are using. First, I would look into the "BULK INSERT" support of that version.
You also didn't mention how many rows are involved. I'll assume that they fit into memory, otherwise a bulk insert won't work.
From there it's up to you to translate the output of fetchall_arrayref into the syntax needed for the "BULK INSERT" operation.

Django: copy data from one database to another

I have two sqlite.db files. I'd like to copy the contents of one column in a table of on db file to another.
for example:
I have the model Information in db file called new.db:
class Information(models.Model):
info_id = models.AutoField(primary_key = True)
info_name = models.CharField( max_length = 50)
and the following information model in db file called old.db:
class Information(models.Model):
info_id = models.AutoField(primary_key = True)
info_type = models.CharField(max_length = 50)
info_name = models.CharField( max_length = 50)
I'd like to copy all the data in column info_id and info_name from old.db to info_id and info_name in new.db.
I was thinking something like:
manage.py dbshell
then
INSERT INTO "new.Information" ("info_id", "info_name")
SELECT "info_id", "info_name"
FROM "old.Information";
This doesn't seem to be working. It says new.Information table does not exist... any ideas?
You'd need to switch your database URL in your settings file to db2 and run syncdb to create the new tables. After that the easiest thing to do imo would be to switch back to db1 and run ./manage.py dumpdata myapp > data.json, followed by another switch to db2 where you can run ./manage.py loaddata data.json.
Afterwards, you can drop the data you don't need from db2.
Edit: Another approach would be to use the ATTACH function from sqlite. First, I recommend you do the first step above (change database settings and use syncdb to create the tables), then you can switch back and do this:
./manage.py dbshell
> ATTACH DATABASE 'new.db' AS newdb;
> INSERT INTO newdb.Information SELECT * FROM Information;
The dumped file from old.db contains info_type field which is not in the new Information model. This will fail the loaddata which checks all field loaded from JSON file. You could comment out info_type line before dump from old model.
The Attach way mentioned by Alex is easier and great, which needs a tiny tweak
INSERT INTO newdb.Information SELECT * FROM Information;
note the missing parentheses around the SELECT, sqlite does not accept them. Refs http://sqlite.org/lang_insert.html
If you are performing migration, have you tried South

Remove html entities from a databases

Due to errors of my predecessors a (MySQL) database I would like to use contains a lot of HTML entities (e.g. € instead of €).
As the database should contain raw data (a database shouldn't have anything to do with HTML) I want to remove them from the DB and store it in proper UTF8, the collocation is already that.
What would be a good way to fix this? The only thing I can think of is to write a PHP script that gets all the data, runs it through html_entity_decode() and writes it back. It's doable since it's a one-time-operation and the DB is only about 100MB large, but it's still less than optimal.
Any ideas?
Since no-one could provide a satisfying SQL-only solution, I solved it with a script similar to this one.
Note that it only works if all the tables you use it on have a primary key, but this will usually be the case
<?php
// Specify which columns need to be de-entitiezed
$affected = array(
'table1' => array('column1', 'column2'),
'table2' => array('column1', 'column2'),
);
// Make database connection
$db = new PDO("mysql:dbname=yourdb;host=yourhost", "user", "pass");
foreach($affected as $table => $columns){
// Start a transaction for each table
$db->beginTransaction();
// Find the table primary key. PHP5.4 syntax!
$pk = $db->query("SHOW INDEX FROM " . $table . " WHERE Key_name = 'PRIMARY'")->fetch()[0];
foreach($columns as $column){
// Construct a prepared statement for this column
$ps = $db->prepare("UPDATE " . $table . " SET " . $column . " . = ? WHERE " . $pk . " = ?");
// Go through all rows
foreach( $db->query("SELECT " . $column . ", " . $pk . " FROM " . $table) as $row){
$row[0] = html_entity_decode($row[0]); // Actual processing
$ps->execute($row);
}
}
// Everything went well for this table, commit
$db->commit();
}
?>
I tnink u need to create a mysql procedure. (with SELECT loop and update replace) REPLACE(TextString, '&apos;','"') ;
Depending on the database (Oracle, MySql, etc) and whether you can take it offline you might be able to export all the DDL and data as a large SQL script (containing INSERTs for all the tables). Then you could do a standard search/replace using sed:
sed -i 's/€/€/g' script.sql
then drop the database or truncate the tables and recreate it using the script.
Ultimately I think you are going to have to resort to PHP at some stage, converting a lot of these entites in SQL is going to invole a huge amount of desicion logic.
However, One approach I can think of if you must use SQL, is to create a user defined function, that esentially has a huge case statement in (Or lots of if/then's) :
http://dev.mysql.com/doc/refman/5.0/en/case-statement.html
Then you should simply be able to do something like:
SELECT col1,col2,col3,mtuserdecodefunction(column-with-entities-in) FROM mytable
Which should in theory return you a cleaned table.

How to use plain SQL in Kohana 3.x

I dont want to rely on all those ORM/Querybuilder etc. tools in Kohana 3.x. I just want do use plain old SQL to Insert a new row in my table of my MySQL Database.
How can I do that?
You can use DB::query($type, $sql) method to create Database_Query object:
$sql = "INSERT INTO table(column1, column2) VALUES(:value1, :value2)";
$result = DB::query(Database::INSERT, $sql)->bind(':value1', $val1)->bind(':value2', $val2)->execute();
echo $result[0]; // last_insert_id
echo $result[1]; // total rows inserted
More info