My customer has a table with ~150 columns in their DB. I don't have access to the DB, I only have summary stats about each column in the table - distinct values in each column along with their likelihood of occurrence.
I'm trying to create a representative copy of this table on my own DB so that I can run queries against it for testing purposes. The only way I know to do this is to write a huge SELECT statement that uses the random() function to randomly choose between the possible values of each column (and other methods for timestamps and IDs). This SELECT is then used inside an INSERT INTO.
This approach just isn't scalable though. I want to be able to do this for a lot more tables. Is there an easier way to do this? I'd like to avoid paid tools if possible.
Related
Due to the way our database is stored, we have tables for each significant event that occurs within a products life:
Acquired
Sold
Delivered
I need to go through and find the status of a product at any given time. In order to do so I'd need to query all of the tables within the schema and find the record with the most up to date record. I know this is possible by union-ing all tables and then finding the MAX timestamp but I wonder if there's a more elegant solution?
Is it possible to query all tables by just querying the root schema or database? Is there a way to loop through all tables within the schema and substitute that into the FROM clause?
Any help is appreciated.
Thanks
You could write a Stored Procedure but, IMO, that would only be worth the effort (and more elegant) if the list of tables changed regularly.
If the list of tables is relatively fixed then creating a UNION statement is probably the most elegant solution and relatively trivial to create - if you plan to use it regularly then just create it as a View.
The way I always approach this type of problem (creating the same SQL for multiple tables) is to dump the list of tables out into Excel, generate the SQL statement for the first table using functions, copy this function down for all the table names and then concatenate all these statements in a final function. You can then just paste this text back into your SQL editor
I've got a table that can contain a variety of different types and fields, and I've got a table definitions table that tells me which field contains which data. I need to select things from that table, so currently I build up a dynamic select statement based on what's in that table definitions table and select it all into a temp table, then work from that.
The actual amount of data I'm selecting is quite big, over 5 million records. I'm wondering if a temp table is really the best way to go around doing this.
Are there other more efficient options of doing what I need to do?
If your data is static, reports like - cache most popular queries results, preferably on Application Server. Or do multidimensional modeling (cubes). That is the really "more efficient option to do that".
Temp tables, table variables, table data types... In any case you will use your tempdb, and if you want to optimize your queries, try to optimize tempdb storage (after checking IO statistics ). You can aslo create indexes for your temp tables.
You can use Table Variables to achieve the functionality.
If you are using the same structure in multiple queries, you can go for custom defined Table data types as well.
http://technet.microsoft.com/en-us/library/ms188927.aspx
http://technet.microsoft.com/en-us/library/bb522526(v=sql.105).aspx
I have an aggregate data set that spans multiple years. The data for each respective year is stored in a separate table named Data. The data is currently sitting in MS ACCESS tables, and I will be migrating it to SQL Server.
I would prefer that data for each year is kept in separate tables, to be merged and queried at runtime. I do not want to do this at the expense of efficiency, however, as each year is approx. 1.5M records of 40ish fields.
I am trying to avoid having to do an excessive number of UNIONS in the query. I would also like to avoid having to edit the query as each new year is added, leading to an ever-expanding number of UNIONs.
Is there an easy way to do these UNIONs at runtime without an extensive SQL query and high system utility? Or, if all the data should be managed in one large table, is there a quick and easy way to append all the tables together in a single query?
If you really want to store them in separate tables, then I would create a view that does that unioning for you.
create view AllData
as
(
select * from Data2001
union all
select * from Data2002
union all
select * from Data2003
)
But to be honest, if you use this, why not put all the data into 1 table. Then if you wanted you could create the views the other way.
create view Data2001
as
(
select * from AllData
where CreateDate >= '1/1/2001'
and CreateDate < '1/1/2002'
)
A single table is likely the best choice for this type of query. HOwever you have to balance that gainst the other work the db is doing.
One choice you did not mention is creating a view that contains the unions and then querying on theview. That way at least you only have to add the union statement to the view each year and all queries using the view will be correct. Personally if I did that I would write a createion query that creates the table and then adjusts the view to add the union for that table. Once it was tested and I knew it would run, I woudl schedule it as a job to run on the last day of the year.
One way to do this is by using horizontal partitioning.
You basically create a partitioning function that informs the DBMS to create separate tables for each period, each with a constraint informing the DBMS that there will only be data for a specific year in each.
At query execution time, the optimiser can decide whether it is possible to completely ignore one or more partitions to speed up execution time.
The setup overhead of such a schema is non-trivial, and it only really makes sense if you have a lot of data. Although 1.5 million rows per year might seem a lot, depending on your query plans, it shouldn't be any big deal (for a decently specced SQL server). Refer to documentation
I can't add comments due to low rep, but definitely agree with 1 table, and partitioning is helpful for large data sets, and is supported in SQL Server, where the data will be getting migrated to.
If the data is heavily used and frequently updated then monthly partitioning might be useful, but if not, given the size, partitioning probably isn't going to be very helpful.
I am currently writing an application that needs to be able to select a subset of IDs from Millions of users...
I am currently writing software to select a group of 100.000 IDs from a table that contains the whole list of Brazilian population 200.000.000 (200M), I need to be able to do this in a reasonable amount of time... ID on Table = ID on XML
I am thinking of parsing the xml file and starting a thread that performs a SELECT statement on a database, I would need a connection for each thread, still this way seems like a brute force approach, perhaps there is a more elegant way?
1) what is the best database to do this?
2) what is a reasonable limit to the amount of db connections?
Making 100.000 queries would take a long time, and splitting up the work on separate threads won't help you much as you are reading from the same table.
Don't get a single record at a time, rather divide the 100.000 items up in reasonably small batches, for example 1000 items each, which you can send to the database. Create a temporary table in the database with those id values, and make a join against the database table to get those records.
Using MS SQL Server for example, you can send a batch of items as an XML to a stored procedure, which can create the temporary table from that and query the database table.
Any modern DBMS that can handle an existing 200M row table, should have no problem comparing against a 100K row table (assuming your hardware is up to scratch).
Ideal solution: Import your XML (at least the IDs) into to a new table, ensure the columns you're comparing are indexed correctly. And then query.
What language? If your using .NET you could load your XML and SQL as datasources, and then i believe there are some enumerable functions that could be used to compare the data.
Do this:
Parse the XML and store the extracted IDs into a temporary table1.
From the main table, select only the rows whose ID is also present in the temporary table:
SELECT * FROM MAIN_TABLE WHERE ID IN (SELECT ID FROM TEMPORARY_TABLE)
A decent DBMS will typically do the job quicker than you can, even if you employed batching/chunking and parallelization on your end.
1 Temporary tables are typically created using CREATE [GLOBAL|LOCAL] TEMPORARY TABLE ... syntax and you'll probably want it private for the session (check your DBMS's interpretation of GLOBAL vs. LOCAL for this). If your DBMS of choice doesn't support temporary tables, you can use "normal" tables instead, but be careful not to let concurrent sessions mess with that table while you are still using it.
I'm currently working with MS SQL 2005, and have a table that has 17 columns, and the space that data in each row would take is only a bit less than what is allowed(per row/record) in MS SQL 2005. And it is for sure that I cannot break this up into smaller tables as the data stored in this table is input from excel sheets whose contents I'm not in control of.
Now the point is, that for almost everything on the Website that uses this database, that main table is providing the result sets, and these result sets are previously known. So, which would be better of the two:
a) I make use of the big table every time.
b) I create smaller tables, and depopulate/populate them as soon as data is edited in the big table.
For eg: Excel sheets containing details of products arrive(almost weekly) from various manufacturers, and they are stored in the PRODUCTS(big) table. Now there are queries like:
SELECT DISTINCT Brand_name, Model_name FROM PRODUCTS
and
SELECT DISTINCT Brand_name, Model_name FROM PRODUCTS WHERE Price < 10 and about 10-15 like these.
Now my question is: Should I build already aggregated tables for these things which amount to about 5 more other than the PRODUCTS table, and update them whenever a sheet comes in, or should I just execute all my retrieval queries on the PRODUCTS table?
The PRODUCTS table would contain about 500,000 rows at the max at a time.
I would be inclined to stick with your single table. 500k records isn't overly massive. If you make sure its properly index for the common selects you are using on it you will probably find it is fairly quick.
Try and run some controlled and repeatable tests to see what sort of speed gains you can get with the right indexes.