I have a task which require the data manipulation to transpose the rows to columns. The data is stored in SQL Server, typical relational database model. I know it can be done with TSQL, but it is so complex that there are almost ten different row groups to be transposed to about 200 columns.
Just wondering whether there is other better tools to do that?
You will have to generate dynamic SQL to do this. Someone asked something similar a couple days ago:
sql-query-to-display-db-data
Are you using SQL Server 2005+? I have a stored procedure that makes it pretty easy to pivot data like that.
This post as an example of using it: SQL query result monthly arrangement
For 200 columns, you should probably break it up into logical groups and store the pivoted data in temp tables (the stored proc does that as an option) - then join up the temp tables for your final result. I've had squirrely results with more than 147 columns with PIVOT.
Some example set up source can be found here.
Without a detailed question, it's hard to say which way is the best. However, you can turn rows to columns in TSQL using PIVOT. You may want to check it out and see if it can do what you need.
I would also try to do it from the application as this might work faster than pivoting the data. But it is one of those things you probably need to try both ways to really see waht is best in your own particular case.
Related
I've a challenge, but I even don't know if it will be possible to solve the way I need.
I need to transform the information on the left table into the data of the right table.
Is it possible to archive this results without using procedures (T-SQL, PLSQL)(I can use intermediate tables to store transformations and so one, but I can't use procedures).
Many thanks in advance
I use oracle database. I have many table with data very big (300-500 million record).
I use query statement have join many table together. I set index for table but get report very slow.
Please, help me solution when working with big data.
Thanks.
Do you really need to have all the data at once?
Try creating a table that stores just the information you need for the report, and run a query once a day (or few hours) to update the table. You can also use Sql Server Integration Services, although I have not tried SSIS with Oracle myself.
I agree with the other users, you really need to give more info on the problem.
Say I have a sample for which 5 million data objects are stored as rows in SQL Server. If I need to run some stats on the data, would it be better to have a table for each sample, or one giant table, where I would select by sample id and then run the stats?
There may eventually be hundreds or even thousands of samples- which seems like one massive table.
But I'm not a SQL Server expert so I can't say whether one would be faster than the other...
Or maybe a better way to deal with such a large data set? I was hoping to use SQL CLR with C# to do my heavy lifting...
If you need to deal with such a large dataset, my gut feeling tells me T-SQL and working in sets will be significantly faster than anything you can do in SQL-CLR and a RBAR (row-by-agonizing-row) approach... dealing with large sets of data, summing up and selecting, that's what T-SQL is always been made for and what it's good at.
5 million rows isn't really an awful lot of data - it's a nice size dataset. But if you have the proper indices in place, e.g. on the columns you use in your JOIN conditions, in your WHERE clause and your ORDER BY clause, you should be just fine.
If you need more and more detailed advice - try to post your table structure, explain how you will query that table (what criteria you use for WHERE and ORDER BY) and we should be able to provide some more feedback.
I have inherited a Access database that has a query that SELECTs over 50 columns. Even in the MS Access graphical query design tool, it's too much information to handle IMO.
A quick disclaimer - I have a programming background, but almost no experience with databases.
For this particular problem, I need data from 50+ columns. Is there a better way to get this information than one huge query?
I am a bit at a loss on how to proceed. The consensus on the web seems to be that SQL queries should be kept relatively small and well formated. I just don't see how to apply that principle when you need so many fields. Can I use lots of simple queries with a UNION or INSERT INTO a temporary table? Any suggestion or RTFMs?
EDIT: More info on the application. The data is spread across 14 tables. I'm grabbing the data to write it out to an external file which has 50+ fields per row (think CSV version of a spreadsheet).
EDIT: Managing and debugging SQL queries in MS Access looks like it contains relevant advice.
I think you're worrying yourself over nothing at all. Pulling 50 fields from 14 tables in order to output to a flat file is nothing at all. The point is that you're intentionally denormalizing data because you need a flat-file output. By definition, that will mean lots of columns.
As others have said, 50 columns in one table would be out of the ordinary (though not necessarily a design error), but in the situation you describe, I don't see an issue at all.
In short, it sounds like a perfectly fine setup and I just have to wonder what makes you think it's a problem to begin with.
I would echo most of the comments about needing 50+ columns every time you run your query.
However, if it's the ugliness of the query that gets you the 50+ columns that is causing your grief, you could create a view for that query. It doesn't get rid of the fact that you're dealing with a bunch of data, but it could encapsulate a large, hairy beast of an SQL statement.
If you're retrieving data from all columns, then you can write SELECT * FROM ....
If there are 50 columns in a single table, the database design is questionable at best. That might practically mean that you'll just have to cope with not-so-elegant solutions.
Do you really need all that data at once? If not, it's probably better to only SELECT the data you need at a given time. You should provide more information on the actual task.
If your data is coming from multiple tables, you could use Common Table Expressions to divide the query into groupings.
I am creating a table to summarize data that is gathered from about 8 or so queries that have very light logic/WHERE clauses and all select against different tables.
I was wondering what the best option would be to fetch the summarized data:
One query with multiple JOINS to gather all relevant information
A stored proc that encapsulates the logic and maybe executes the 8 queries and does the "joining" in some other way? This seems more modular and maintainable to me...but I'm not sure.
I am using SQL Server 2008 for this. Any suggestions?
If you can, then use usual SQL methods. Db's are optimized to run them. This "joining in some other way" would probably require the use of cursor which slows down everything. Just let the db do its job. If you need more performance then you should examine execution plan and do what has to be done there(eg. adding indexes).
Databases are pretty good at figuring out the optimal way of executing SQL. It is what they are designed to do. Using stored procedures to load the data in chunks and combining it yourself will be more complex to write, and likely to be less efficient than letting the database just do it for you.
If you are concerned about reusing a complex query in multiple places, consider creating a view of it instead.
Depending on the size of the tables, joining 8 of them could be pretty hairy. I would try it that way first - as others have said, the db is pretty good at figuring this stuff out. If the performance is not as good as you would like, I would try a stored proc which creates a table variable (or a temp table) and inserts the data from each of the 8 tables separately. Then you can return the contents of the table variable to your app.
This method also makes it a little easier to add the 9th, 10th, etc tables in the future. And it gives you an easy way to do any processing you may need on the summarized data before returning it to your app.