I have a webapp that spans many different users, each with selective permissions about what they are able to see. The app is built on top of a MySQL database.
One feature I am interested in providing to my "power users" is a sql dump of all their data so that they can run off and do their own things with it. Now I can't just use mysqldump because there are things belonging to other users that should not be made available to anybody else on download.
Is there any other easy way to get data in and out of MySQL that allows you to selectively specify what to export, and without having to jump through all kinds of hoops? Note that I need control at the query level - i.e. being able to specify a list of tables is NOT sufficient. In an ideal world, such a tool would automatically find all relationships based on traversal of foreign keys, but if I have to write queries at the table-level I'm willing to, provided it's easy for others to get the data back into mysql without too much trouble.
Anyone know if such a tool exists, or if I am in "roll my own" territory?
Mysqldump does have a "--where" flag that you can use to selectively return rows. I think you should be able to do something like:
mysqldump --where="foreign_key_id=5"
Which should return only those specific rows, more documentation on the MySQL Site
However, I'm not sure you wouldn't be further ahead to do an export as comma separated value files. CSV files can be imported back into MySQL as well as give your users many other options for ways to work with their data (spreadsheets, other RDBMS, text analysis).
Here is a tool that can help you export data into CSV/Excel files, but not to import data. It has a permission management that should provide the necessary access you required.
You can find it here: https://github.com/mpetcu/report-manager.
Related
Is somebody aware of a tool that lets me browse MySQL-Files without having to import them into my database system?
I'm looking for an easy way to inspect MySQL Backups quickly without having to import them - but still being nicely displayed, so viewing the SQL source is not really an option.
Maybe there's a program that takes the SQL dump and automatically imports it into a temporary database, then presenting it in an interface similar to HeidiSQL (or any other SQL-Gui-Tool).
Why are you eliminating the obvious solution? You just need to load the backup into a mysql database. Either load the backup into a separate mysql instance, or if your backup is of just one database (i.e. you didn't pass --databases or --all-databases to mysqldump), load it into a database of a different name.
I came here looking for an answer to the same question, because it can be cumbersome to wait to load a 20 GB sql dump just for inspection and drop again. While I hope to find a standalone shortcut tool, best I can recommend is a cocktail of linux cli text manipulation tools like grep, sed, and cut. Some useful output could be:
What tables are being created/inserted into?
Are the mysqldump INSERTs one line-per-record or all stuffed into one? (Because this can affect other things like)
How many rows are being inserted into table XYZ?
What is some representative data being inserted into table XYZ?
What is the ABC column value for the last row inserted into table XYZ?
Good luck!
I have 3 computers having the same sql server 2005 database, I would like to gather the data from the 3 computers to another computer which has the same database. Please help me.
This is called "data conversion" and a lot of your work will be to determine uniqueness on each one of them and coming up with strategies to prevent collisions, mainly primary keys that likely are the same across these databases. No simple answer here, it can be a project in itself.
It might be difficult without any manual data transformation. It depends on your database and type of the data. For example what do you use as a keys? If you have sequential integers as a primary/foreign keys, then you will have to do some manual data transformation. IF you use GUIDS, it will get slightly easier, but you still have to ensure that for example some lookup tables doesn't have different guid keys for same items etc.. But there is no took for doing this automatically.
Maybe if you have some very simple data without any relations to other tables (like table with one column with text messages etc) you can script the data with SQL Server Database Publishing Wizard, and then execute the scripts against your target database.
You need to backup your databases by right clicking in Enterprise Manager and choosing backup before choosing the location etc.
After backing up you can then restore to your local Sql Server by right clicking and choosing restore.
After you have the data locally you will need to write queries to transfer the data to your local database.
Alternatively you can use something like Red Gates Sql Data Compare to compare and transfer data using a visual interface. Although this costs money.
Redgate SQL Toolbelt may be able to help you. You would first copy database to that another computer and then compare it with Sql Data Compare against 3 databases always copying data only one way (to your new database). However I am not 100% sure if it will work like i think it would. You would have to verify it yourself.
Like other people suggested some things like primary keys etc may be problematic.
I often find myself writing one off queries to either answer someone's question or trouble shoot something and I would like to be able to quickly expose the on demand refreshable results of the query graphically so that I can share these results to others without having to go through the process of creating an SSRS report and publishing it to a reporting services server.
I have thought about using excel to do this or maybe running a local SSRS server but both of these options are still labor intensive and I cannot justify the time it would take to do these since no one has officially requested that I turn this data into a report.
The way I see it the business I work for has invested money in me creating these queries that often return potentially useful data that other people in the organization might want but since it isn't exposed in any way and I don't know that this data is something they want and they may not even realize they want this data, the potential value of the query is not realized. I want to increase the company's return on investment on all these one off queries that I and other developers write by exposing their results graphically so that they can be browsed by others and then potentially turned into more formalized SSRS reports if they provide enough value to justify the development of the report.
What is the fastest way for me to take a query and turn it into a refreshable graph of the results set?
Why dont you simply use what you may already have. Excel...you can import data via an ODBC / Oracle / SQL Connection. Get Data..and bam you can run the query and format it right in the spreadsheet and provide sorting etc. All you need to supply is the database name and user name and password to connect to the db.
JonH is right regarding Excel's built in ODBC support, but I have had tons of trouble with this. In my case, the ODBC connection required the client software to be installed so that it could use the encryption methods, etc. Also, even if that were not the case, the user (I believe) would still have to manually install and set up an ODBC connection.
Now if you just want something on your machine to do the queries and refresh them, JohH's solution is great and my caveats are probably irrelavent. But if you want other users to have access, you should consider having a middle-man app (basically a PHP script, assuming a web server is an option for you), that does a query, transforms the results into XML, and outputs it as "report-xyz.xml". You can then point anybody running a newer version of Excel to that address and they can very easily import the data into Excel with no overhead. (basically a kind of web service).
Keep in mind, I don't think you should have a web script that will allow users to make queries to your Database server! You would have some admin page where you make pass the query in and a new xml file with the results gets made. So my idea is also based on the idea that you want to run the same queries over and over without any specifics passed in. (if that were the case, I'd look into just finding a pre-built web services bridge for your database that already has security features built in. Then you could have users make the limited changes allowed.)
The site I am working on as a student will be redesigned and released in the near future and I have been assigned the task of manually searching through every table in the DB the site uses to find tables we can consider for deletion. I'm doing the search through every HTML files source code in dreamweaver but I was hoping there is an automated way to check my work. Does anyone have any suggestions as to how this is done in the business world?
If you search through the code, you may find SQL that is never used, because the users never choose those options in the application.
Instead, I would suggest that you turn on auditing on the database and log what SQL is actually used. For example in Oracle you would do it like this. Other major database servers have similar capabilities.
From the log data you can identify not only what tables are being used, but their frequency of use. If there are any tables in the schema that do not show up during a week of auditing, or show up only rarely, then you could investigate this in the code using text search tools.
Once you have candidate tables to remove from the database, and approval from your manager, then don't just drop the tables, create them again as an empty table, or put one dummy record in the table with mostly null values (or zero or blank) in the fields, except for name and descriptive fields where you can put something like "DELETED" "Report error DELE to support center", etc. That way, the application won't fail with a hard error, and you have a chance at finding out what users are doing when they end up with these unused tables.
Reverse engineer the DB (Visio, Toad, etc...), document the structure and ask designers of the new site what they need -- then refactor.
I would start by combing through the HTML source for keywords:
SELECT
INSERT
UPDATE
DELETE
...using grep/etc. None of these are HTML entities, and you can't reliably use table names because you could be dealing with views (assuming any exist in the system). Then you have to pour over the statements themselves to determine what is being used.
If [hopefully] functions and/or stored procedures were used in the system, most DBs have a reference feature to check for dependencies.
This would be a good time to create a Design Document on a screen by screen basis, listing the attributes on screen & where the value(s) come from in the database at the table.column level.
Compile your list of tables used, and compare to what's actually in the database.
If the table names are specified in the HTML source (and if that's the only place they are ever specified!), you can do a Search in Files for the name of each table in the DB. If there are a lot of tables, consider using a tool like grep and creating a script that runs grep against the source code base (HTML files plus any others that can reference the table by name) for each table name.
Having said that, I would still follow Damir's advice and take a list of deletion candidates to the data designers for validation.
I'm guessing you don't have any tests in place around the data access or the UI, so there's no way to verify what is and isn't used. Provided that the data access is consistent, scripting will be your best bet. Have it search out the tables/views/stored procedures that are being called and dump those to a file to analyze further. That will at least give you a list of everything that is actually called from some place. As for if those pages are actually used anywhere, that's another story.
Once you have the list of the database elements that are being called, compare that with a list of the user-defined elements that are in the database. That will give you the ones that could potentially be deleted.
All that being said, if the site is being redesigned then a fresh database schema may actually be a better approach. It's usually less intensive to start fresh and import the old data than it is to find dead tables and fields.
Is there an automatic way in SQL Server 2005 to create a database from several tables in another database? I need to work on a project and I only need a few tables to run it locally, and I don't want to make a backup of a 50 gig DB.
UPDATE
I tried the Tasks -> Export Data in Management studio, and while it created a new sub database with the tables I wanted, it did not copy over any table metadata, ie...no PK/FK constraints and no Identity data (Even with Preserve Identity checked).
I obviously need these for it to work, so I'm open to other suggestions. I'll try that database publishing tool.
I don't have Integration Services available, and the two SQL Servers cannot directly connect to each other, so those are out.
Update of the Update
The Database Publishing Tool worked, the SQL it generated was slightly buggy, so a little hand editing was needed (Tried to reference nonexistent triggers), but once I did that I was good to go.
You can use the Database Publishing Wizard for this. It will let you select a set of tables with or without the data and export it into a .sql script file that you can then run against your other db to recreate the tables and/or the data.
Create your new database first. Then right-click on it and go to the Tasks sub-menu in the context menu. You should have some kind of import/export functionality in there. I can't remember exactly since I'm not at work right now! :)
From there, you will get to choose your origin and destination data sources and which tables you want to transfer. When you select your tables, click on the advanced (or options) button and select the check box called "preserve primary keys". Otherwise, new primary key values will be created for you.
I know this method can hardly be called automatic but why don't you use a few simple SELECT INTO statements?
Because I'd have to reconstruct the schema, constraints and indexes first. Thats the part I want to automate...Getting the data is the easy part.
Thanks for your suggestions everyone, looks like this is easy.
Integration Services can help accomplish this task. This tool provids advanced data transformation capabilities so you will be able to get exact subset of data that you need from large database.
Assuming that such data is needed for testing/debugging you may consider applying Row Sampling to reduce amount of data exported.
Create new database
Right click on it,
Tasks -> Import Data
Follow instructions