Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 9 years ago.
Improve this question
I have a lot of databases with relatively large amounts of columns ranging from 5 to 300. Each table has at least 50,000 rows in it.
What is the most effective way to store this data? Presently the data has just been dumped into an indexed sql database.
It was suggested to me to create 3 columns as follows.
Column Name, Column category, Row ID, Row Data.
example data would be
Male, 25-40, 145897, 365
Would this be faster? Would this be slower? Is there better ways to store such large and bulky databases?
I will almost never be updating or changing data. It simply be outputted to a 'datatables' dynamic table where it will be sorted, limited and ect. The category column will be used to break up the columns on the table.
Normalize your db!
I have struggled with this "theory" for a long time and experience has proven that if you can normalize data across multiple tables it is better and performance will not suffer.
Do Not try to put all the data in one row with hundreds of columns. Not because of performance but because of development ease.
Learn More here
Related
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 4 years ago.
Improve this question
I have a program where the user fills data into a table. The tables all have different sizes, some are 3x2, others are 30x20.
Its just like an excel sheet and looks like this, where the user can add rows and columns as much as she needs them. These are charts related to products so each table has a unique product number
Whats the best way to organize this data in a SQL database?
Is it one SQL table per user generated table?
This seems excessive as i would end up with a database with 10000s of tables. Are there other, better ways to store the data? Can i combine it into one SQL table?
You can make SQL tables like
table_header
- id
- rows
- cols
table_detail
- header_id
- row
- col
- value
Then to store a table you create a header record and add the contents one entry at a time. To read the table back you would basically just pull all data for the table, maybe with an ORDER BY if you want to make parsing back into the table easier. Something like that.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I have been given an 800 lines SQL Query which is taking around 20 hours to fetch around 400 million records.
There are 13 tables which are partitioned by month.
The tables have records ranging from 10k to 400 million in each partition.
The tables are indexed on primary keys.
The query uses many inline views and outer joins and a few group by functions.
DBAs say we cannot add more indexes as it would slow down the performance since it is an OLTP system.
I have been asked to convert the query logic to pl/sql and then populate a table in chunks.Then do a select * from that table.
My end result should be a query which can be fed to my application.
So even after I use pl/sql to populate a table in chunks,ultimately I need to fetch the data from that table as a query.
My question is, since pl/sql would require select and insert both, are there any chances pl/sql can be faster than sql?
Are there any cases where pl/sql is faster for any result which is achievable by sql?
I will be happy to provide more information if the given info doesn't suffice.
Implementing it as a stored procedure could be faster because the SQL will already be parsed and compiled when the procedure is created. However, given the volume of data you are describing its unclear if this will make a significant difference. All you can do is try it and see.
I think you really need to identify where the performance problem is; where the time is being spent. For example (and I have seen examples of this many times), the majority of the time might be in fetching to 400M rows to whatever the "client" is. In that case, re-writing the query or as PL/SQL will make no difference.
Anyway, once you can enumerate the problem, you have a better chance of getting sound answers, rather than guesses...
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I am making a attendance system , i have a database which is i created and it has a tables from jan-dec (12 tables) and each table has this kind of column (Jan1,jan2//feb1,feb2//mar1,mar2... etc ) i know it is not a good practice tho i'm not familiar with sql, I would like to ask if how would i be able to make a much lesser tables/columns ? and it will based on my datepicker in my vb.net program?
Delve deeper into relational database design (take the link as a first step).
One thing is to create just one table, and have in it a column of type DATE or DATETIME to denote the date. Additional columns would have related data that is linked to the date. That would simplify your table structure greatly. From 12 tables with approx. 30 colums, to just one table with one column + columns with related information.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
Each of the salesmen should make a forecast of his sales. I know how he may input data directly from Excel sheet to SQL table. Do I need to create different tables - one table per salesman? At the end I need to aggregate all the forecasts. Is it possible to make it with just one table?
The condition is that one salesman is not allowed to see the other salesmen forecasts.
It seems to be a common problem of inputting data to database by many different users with restrictions on access.
Update. Each salesman is in different town. Say we have 500 salesmen so it is not the way to gather data from 500 Excel files into one big Excel file and then load it to SQL.
actually you don't need to create different tables for each salesmen. one table is enough to load all your salesman info Excel data. to find each salesmen's forecast sales simple transmission query will help u
You need at least two tables. You need a staging table to receive the excel data and perform the necessary validation, transformation, etc. You need at least one table for data storage. Given that you are talking about people and sales, you probably want a normalized database. If you don't know what that means, I've heard good things about the book, Database Design for Mere Mortals.
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 11 years ago.
Improve this question
I have the option to use a single table that will expand upwards of 1,000,000 records per year.
With that said, I could use a foreign key to break up this table into muitiple smaller tables, which will reduce this expansion to each smaller table of 100,000 records per year.
Lets say 50% of the time, users will query all of the records where the other 50% of the time users will query the segmented smaller table data set. ( think based on all geographic areas vs. specific geographic areas)
Using a database managed by a shared hosting account ( think site5, godaddy, etc... ), is it faster to use a single larger table or to use several smaller segmented tables given this situation?
Where each dataset is accessed 10%/%90, 20%/%80, %30/%70... etc, at what point would using a single table vs muiltiple smaller tables be the most/least efficient?
In general do it so as to reduce the amount of duplicated information. If you are making smaller tables which have many redundant columns, then it seems like it'd be more efficient to have just one table. But otherwise, one table.
It also depends on what percent of the row is being used per query, and how your queries are structured. If you are adding lots of joins or subqueries, then it'll most likely be slower.