Need to exceed the 8k record limit in SQL Server 2008 when using wide columns/sparse tables.
Long story new client old system using survey system, pivoting data so all answers are a column
I have 1500 columns and now I am getting
Cannot create a row that has sparse data of size 9652 which is greater than the allowable maximum sparse data size of 8019.
I need to exceed the 8k record limit if possible
Not possible, because SQL Server stores rows on 8K pages. The only way to do so would be to store some of the data off-row (e.g. using MAX or other LOB types for some of your columns). To your application, this will still look like it's on the same row, even though logically it is on a complete different area of disk.
If your sparse column set alone exceeds the limit, sorry, you'll need to look at a different way to store the data (either not pivoted, EAV, or simply use two tables joined by a key, each containing half of the column set). For the latter you can make this relatively transparent to users by using views and/or enforcing all data access / DML through stored procedures that understand the division.
Related
My table having 80K+ records. I need to calculate each and every row size. This sum of row size must be same as table size
There is very little relationship between the size of table and the size of a row. Tables sizes are really measured in numbers of pages that the table occupies (and perhaps associated indexes as well). Page sizes can be converted to bytes.
Record sizes (and you can look into [datalength()][1]) return sizes in bytes, which is a more typical measure of size on a computer.
Why are these not related very much? Here are three reasons I can readily think of:
Pages have over head needed by the database.
Pages may not be completely filled.
The size of the table should probably include associated index structures, but that is not part of the records.
In particular, you might have one 10-byte record on an 8k page. Or, you might have 800 of them. Both are valid representations of the data inside the database.
So, what you want to do does not make sense given how data is structured in a database.
If I create a SQL Server table that has more than 8060 bytes per page, will the querying be considerably hurt for columns that exceed this limit?
I also don't understand quite right if a row occupies the whole 8060 bytes even if it's empty.. in the case it's true, will the query performance be impacted just for the particular rows that exceed the limit or for all rows?
For the first Question:
Yes, it could affect performance. Having a combination of varchar, nvarchar, sql_variant and varbinary in one table, with total size greater than 8,060 bytes results in reallocation data to another page.
While this affects update, I'm not sure if it's too important for reading a data. Internally, SQL Server puts a pointer to the reallocated portion of data within new page so I guess it's quite fast operation.
It's up to you (DBA/developer) to analyze and predict the percentage of such rows in table. If it occurs too often, consider moving large columns into separate table(s).
Use sys.dm_db_index_physical_stats to find out what's going on with your data.
Second Question:
I guess you asked for situation when some columns (especially varchar) are empty. You can "help" SQL Server to save space using sparse columns
Also, I'd recommend this article.
I have created a table with column types as nvarchar(max), which my understanding is that they can support 2GB. However on inserting, I still receive this error:
Cannot create a row of size 8061 which is greater than the allowable maximum row size of 8060.
Is there a global setting on the database required, or is there another limit I am hitting? Is there a limit to the number of varchar(max) fields per table?
SQL server uses page to store data. Page size is 8kb.
So a record size (row size) in SQL server cannot be greater than 8060 bytes.
If data is not fitted in 8060 bytes then reference pointers are used.
When a combination of varchar, nvarchar, varbinary, sql_variant, or CLR user-defined type columns exceeds this limit, the SQL Server Database Engine moves the record column with the largest width to another page in the ROW_OVERFLOW_DATA allocation unit, while maintaining a 24-byte pointer on the original page.
Moving large records to another page occurs dynamically as records are lengthened based on update operations. Update operations that shorten records may cause records to be moved back to the original page in the IN_ROW_DATA allocation unit.
Also, querying and performing other select operations, such as sorts or joins on large records that contain row-overflow data slows processing time, because these records are processed synchronously instead of asynchronously.
The record-size limit for tables that use sparse columns is 8,018 bytes. When the converted data plus existing record data exceeds 8,018 bytes, MSSQLSERVER ERROR 576 is returned. When columns are converted between sparse and nonsparse types, Database Engine keeps a copy of the current record data. This temporarily doubles the storage that is required for the record. .
To obtain information about tables or indexes that might contain row-overflow data, use the sys.dm_db_index_physical_stats dynamic management function.
From the SQL Server documentation:
The length of individual columns must still fall within the limit of
8,000 bytes for varchar, nvarchar, varbinary, sql_variant, and CLR
user-defined type columns. Only their combined lengths can exceed the
8,060-byte row limit of a table.
The sum of other data type columns, including char and nchar data,
must fall within the 8,060-byte row limit. Large object data is also
exempt from the 8,060-byte row limit.
More info here: https://technet.microsoft.com/en-us/library/ms186981%28v=sql.105%29.aspx
This comes from an earlier thread on StackOverflow that can be found here:
Cannot create a row of size 8937 which is greater than the allowable maximum of 8060
The error is caused because you cannot have a row in SQL server which is larger than 8KB (the size of 1 page) because rows are not allowed to span pages - its a basic limit of SQL Server [...]
Note that SQL server will allow you to create the table, however if you try to actually insert any data which spans multiple pages then it will give the above error.
Of course this doesn't quite add up, because if the above was the whole truth then single VARCHAR(8000) column would fill a row in a table! (This used to be the case). SQL Server 2005 got around this limitation by allowing certain data from a row to be stored in another page, and instead leaving a 24-bit pointer instead.
I would suggest normalizing your table into one or more relalted tables.
I have a large table (~2 million rows), each row of which represents one image. I want to store the EXIF metadata in JSON format for each image. This JSON blob is about 6KB per image.
This EXIF metadata won't be queried/used very often, and I'm wondering whether it'd be significantly more efficient to store it in a separate table with two columns (imageid, exifjson), or whether PostgreSQL will deal with this just fine as a text column on existing table. I wouldn't want adding the column to significantly slow down ordinary queries on the table, or the millions of 6KB text values to bog PostgreSQL down.
I would make that column TOAST-ed.
ALTER TABLE ... ALTER <column> SET STORAGE <EXTERNAL|EXTENDED>;
-- EXTERNAL - out-of-line storage, not compression[1]
-- EXTENDED - both compression and out-of-line storage
PostgreSQL already try to use it for data larger than ~2kB.
[1] "The compression technique used is a fairly simple and very fast member of the LZ family of compression techniques."
Better to use separate table, but you will be ok with existing table. You'll scarcely get a impact unless you retrieve this field using existing "select * from" queries. And you'll never fill postgres up with such data since it has almost infinite thresholds:
Maximum Database Size Unlimited
Maximum Table Size 32 TB
Maximum Row Size 1.6 TB
Maximum Field Size 1 GB
Maximum Rows per Table Unlimited
Maximum Columns per Table 250 - 1600 depending on column types
Maximum Indexes per Table Unlimited
http://www.postgresql.org/about/
About interfering during selection of other column data:
Very long values are also stored in background tables so that they do not interfere with rapid access to shorter column values.
http://www.postgresql.org/docs/current/static/datatype-character.html
For example, a website offers the ability to create mobile surveys. Each survey ID is a FK in the survey response table, which contains ALL of the survey responses.
What is the size limitation of this table in a SQL Server 2008 db, if the table contains, say 20 varchar(255) fields including the bigint PK & FK?
I realize this would depend on the file size limitation as well, but I would like some more of an educated answer rather than my guess on this.
In terms of searchability, some fields that contain geo-related details such as the survey ID, city, state, and two commends fields would have to be searchable, and thus indexed ... index only these fields?
Also, aged responses would expire after a given amount of time - thus deleted from the table. Does the table, at this point being very large, need to be re-indexed/cleaned up, after the deletions (which would be an automated process)?
Thanks.
Maximum Capacity Specifications for SQL Server
Bytes per row: 8,060
Rows per table: Limited by available storage
Note
SQL Server supports row-overflow storage which enables variable length
columns to be pushed off-row. Only a 24-byte root is stored in the
main record for variable length columns pushed out of row; because of
this, the effective row limit is higher than in previous releases of
SQL Server. For more information, see the "Row-Overflow Data Exceeding
8 KB" topic in SQL Server Books Online
You mention 'table size' -- does this mean number of rows?
Maximum Capacity Specifications for SQL Server
Rows per table : Limited by available storage
As per this Reference, the max size of a table is limited by the available storage.
It sounds like you are going to have a high traffic and high content table. You should consider performance and storage enhancements like Table Partitioning. Also, because this table will be the victim of often INSERTS/UPDATES/DELETES, carefully plan out your indexing, as indexes add overhead for DML statements on the table.