I am writing a data warehouse, using MySQL as the back-end. I need to partition a table based on two integer IDs and a name string.
A more concrete example would be to assume that I am storing data about a school. I want to partition the school_data table based on COMPOSITE 'Key' based on the following:
school id (integer)
course_id (integer)
student_surname (string)
For the student surname, it is just the first character of the surname that determines which 'partitioned table' the data should go in to.
How may I implement this requirement using MySQL (5.1) with InnoDb tables?
Also, I am doing my development on a Windows box, but I will deploy onto a *nix box for production. I have two further questions:
I am assuming that I will have to dump and restore the data when moving from Windows to Linux. I don't know if this is OK if the database contains partitioned tables (pointer to where it states this in the documentation will put my mind to rest - I have not been able to find any specific mention of dump/restore regarding partitioned tables.
I may also need to change databases (if Oracle pulls a surprise move on MySQL users) in which case I will need to SOMEHOW export the data into another database. In this (hopefully unlikely scenario) - what will be the best way to dump data out of MySQL (maybe to text or something) bearing in mind the partitioned table?
RANGE Partitioning
A table that is partitioned by range is partitioned in such a way that each partition contains rows for which the partitioning expression value lies within a given range.
CREATE TABLE employees (
school id (integer)
course_id (integer)
student_surname (string)
)
PARTITION BY RANGE (student_surname) (
PARTITION p0 VALUES LESS THAN ('ezzzzzzzzzzzzzzzzzzzzzzz'),
PARTITION p1 VALUES LESS THAN ('ozzzzzzzzzzzzzzzzzzzzzzz'),
PARTITION p2 VALUES LESS THAN ('tzzzzzzzzzzzzzzzzzzzzzzz'),
PARTITION p3 VALUES LESS THAN (MAXVALUE)
);
Range partitioning
Data Migration to Another DB
MySQLDUMP will output the table and data to a file. However, Oracle supports connecting to other databases via ODBC, just as SQL Server has it's linked server capability.
Addendum
It looks like you are partitioning by only one of the 3 fields I mentioned (i.e. name). I saw partitioning by a single field in the MySQL docs, but not 3 fields (int, int, string) like I want to do.
Partitioning by three columns is possible, but my example is per your requirements in the OP:
For the student surname, it just the first character of the surname that determines which 'partitioned table' the data should go in to.
How may I implement this requirement using mySQL (5.1) with InnoDb tables?
Have a look at the Chapter 18. Partitioning of MySQL documentation and especially the Partition Types (I'd look at the HASH partitioning). But keep in mind that the partitioning implementation in MySQL 5.1 is still undergoing development and there are some limitations and restrictions.
I am assuming that I will have to dump and restore the data when moving from windows to Linux. I dont know if this is OK if the db contains partitioned tables (pointer to where it states this in the docs will put my mind to rest - I have not been able to find any specific mention of dump/restore regarding partitioned tables.
I didn't find anything in 18.3 Partition Management but, according to this post, backing up and restoring a partitioned table is nothing special. To backup:
mysqldump --opt db_name table_name > file.dump
And to restore:
mysql db_name < file.dump
I would do some testing though.
I may also need to change databases (if Oracle pulls a suprise move on mySQL users) in which case I will need to SOMEHOW export the data into another database. In this (hopefully unlikely scenario) - what will be the best way to dump data out of mySQL (maybe to text or something) bearing in mind the partitioned table?
Oracle SQL Developer incorporates migration support by including redeveloped features and greatly extending the functionality and usability offered by the original Oracle Migration Workbench to migrate Microsoft Access, Microsoft SQL Server, MySQL and Sybase databases to Oracle.
Related
I'm working on developing a fast way to make a clone of a database to test an application. My database has some specif tables that are quite big (+50GB), but the big majority of the tables only have a few MBs. On my current server, the dump + restore takes some hours. These bigs tables have date fields.
With the context in mind, my question is: Is possible to use some type of restrictions on table rows to select the data that is being dumped? e.g. On table X only dump the rows that date is Y.
If this is a possible show can I do it? if it's not possible what would be a good alternative?
You can use COPY SELECT whatever FROM yourtable WHERE ... TO '/some/file' to limit what you export.
COPY command
You could use row level security and create a policy that lets the dumping database user see only those rows that you want to dump (make sure that that user is neither a superuser nor owns the tables, because these users are exempt from row level security).
Then dump the database with that user, using the --enable-row-security option of pg_dump.
Server1: Prod, hosting DB1
Server2: Dev hosting DB2
Is there a way to query databases living on 2 different server with a same select query? I need to bring all the new rows from Prod to dev, using a query
like below. I will be using SQL Server DTS (import export data utility)to do this thing.
Insert into Dev.db1.table1
Select *
from Prod.db1.table1
where table1.PK not in (Select table1.PK from Dev.db1.table1)
Creating a linked server is the only approach that I am aware of for this to occur. If you are simply trying to add all new rows from prod to dev then why not just create a backup of that one particular table and pull it into the dev environment then write the query from the same server and database?
Granted this is a one time use and a pain for re-occuring instances but if it is a one time thing then I would recommend doing that. Otherwise make a linked server between the two.
To backup a single table in SQL use the SQl Server import and export wizard. Select the prod database as your datasource and then select only the prod table as your source table and make a new table in the dev environment for your destination table.
This should get you what you are looking for.
You say you're using DTS; the modern equivalent would be SSIS.
Typically you'd use a data flow task in an SSIS package to pull all the information from the live system into a staging table on the target, then load it from there. This is a pretty standard operation when data warehousing.
There are plenty of different approaches to save you copying all the data across (e.g. use a timestamp, use rowversion, use Change Data Capture, make use of the fact your primary key only ever gets bigger, etc. etc.) Or you could just do what you want with a lookup flow directly in SSIS...
The best approach will depend on many things: how much data you've got, what data transfer speed you have between the servers, your key types, etc.
When your servers are all in one Active Directory, and when you use Windows Authentification, then all you need is an account which has proper rights on all the databases!
You can then simply reference all tables like server.database.schema.table
For example:
insert into server1.db1.dbo.tblData1 (...)
select ... from server2.db2.dbo.tblData2;
I want to Copy a data from One oracle database to another.
I have checked Import/Export Utility but the problem is import utility doesn't support conflicts resolution techniques between rows.
For Example if there's a table in the source database have the same row key in the destination database. if i use 'Ignore' parameter with value = y, the destination table will have a duplicate rows.
I want to ask if there's another way to import data from oracle database to another with some mechanism of detecting the conflicts and resolve them?
You might want to consider using a database link from database A to database B. You can query the data from database B to insert into your database A tables. You are free to query whatever you want using SQL or PL/SQL.
More on database links:
http://docs.oracle.com/cd/B19306_01/server.102/b14200/statements_5005.htm
So I am stuck on this Teradata problem and I am looking to the community for advice as I am new to the TD platform. I am currently working with a Teradata Data Warehouse and have an interesting task to solve. Currently we store our information in a live production database but want to stage tables in another database before using FastExport to export the files. Basically we want to move our tables into a database to take a quick snapshot.
I have been exploring different solutions and am unsure how to proceed. I need to be able to automate a create table process from one DB in Teradata to another. The tricky part is I would like to create many tables off of the source table using a WHERE clause. For example, I have a transaction table and want to take a snapshot of the transaction table for a certain date range month by month. Meaning that the original table Transaction would be split into many tables such as Transaction_May2001, Transaction_June2001, Transaction_July2001 and so on and so forth.
Thanks
This is assuming by two databases you are referring to the same physical installation of Teradata.
You can use the CREATE TABLE AS construct to accomplish this:
CREATE TABLE {MyDB}.Transaction_May2001
AS (
SELECT *
FROM Transaction
WHERE Transaction_Date BETWEEN DATE '2001-05-01' AND '2001-05-31'
)
{UNIQUE} PRIMARY INDEX ({Same PI definition as Transaction Table})
WITH DATA AND STATS;
If you neglect to specify the explicit PI in the CREATE TABLE AS then Teradata will take the first column of the SELECT clause and use it as the PI of the new table.
Otherwise, you would be looking to use a Teradata utility as suggested by ryanbwork in the comment to your question.
I have a 5 database with same schema, i want to copy all data in one database with same schema
or how can i copy data from *.mdf files in database.
i am using sql server 2005
Copy Database with T-SQL:
sqlauthority
http://blog.sqlauthority.com/2009/07/29/sql-server-2008-copy-database-with-data-generate-t-sql-for-inserting-data-from-one-table-to-another-table/
Copy Database with Wizard:
kodyaz
http://www.kodyaz.com/sql-server-tools/sql-server-copy-database-wizard.aspx
I'd suggest taking a look at Red Gate SQL Data Compare. That will enable you to merge the data between the two databases and directly control which one wins in any given situation.
As mentioned above you need to deal with the Primary Keys as well...
One way to deal with this to add a "Database ID" to all the tables in the single central version. The central PKs become the PK from the source table, plus the "Database ID". This way you have unique PKs in the central version AND you can tell which database the row came from. This is what sql-hub does - there is a free licence which will let you do this as a one-off task - or you could do the inserts for each database and table in SQL.