How does SQL Server Page Allocation work? - sql

Question

My question follows on from a previous one that I asked here (Using SQLBulkCopy - Significantly larger tables in SQL Server 2016 than in SQL Server 2014)
where my use of SQLBulkcopy with SQL2016 was causing large amounts of memory to be allocated and not used. This was due to an optimisation in SQL2016 (detailed fully here - https://blogs.msdn.microsoft.com/sql_server_team/sql-server-2016-minimal-logging-and-impact-of-the-batchsize-in-bulk-load-operations) that optimised the getting of storage for speed, and did not care to fill in small amounts of allocated storage.
When running the following sql on the database
SELECT
t.NAME AS TableName,
s.Name AS SchemaName,
p.rows AS RowCounts,
SUM(a.total_pages) * 8 AS TotalSpaceKB,
SUM(a.used_pages) * 8 AS UsedSpaceKB,
(SUM(a.total_pages) - SUM(a.used_pages)) * 8 AS UnusedSpaceKB
FROM
sys.tables t
INNER JOIN
sys.indexes i ON t.OBJECT_ID = i.object_id
INNER JOIN
sys.partitions p ON i.object_id = p.OBJECT_ID AND i.index_id = p.index_id
INNER JOIN
sys.allocation_units a ON p.partition_id = a.container_id
LEFT OUTER JOIN
sys.schemas s ON t.schema_id = s.schema_id
WHERE
t.NAME not like 'dt%'
AND t.is_ms_shipped = 0
AND i.OBJECT_ID > 255
GROUP BY
t.Name, s.Name, p.Rows
ORDER BY
UnusedSpaceKB desc
you would see output like the below. (See the amount of UnusedSpace).
There are two ways to fix this.
1. Switch on TraceFlag 692 as detailed in the above article. (Note – This does work, but is not allowable in my current situation for non-technical reasons).
2. Load the data a different way – (i.e. do not use SqlBulkCopy)
To this end I wrote a mechanism that took in a database and created the necessary SQL that I needed, which I then invoked using ExecuteNonQuery. This initially looked very promising, but then I found that some of my tables exhibited the old behaviour (Note – I proved this by rerunning the process with the TraceFlag 692 on (which I can do in test situations in order to investigate but not in live situations) and did not suffer a repeat of the unwanted extra space behaviour).
In my current situation I am preparing some sql and firing it to the database using ExecuteNonQuery. The table that I am inserting to looks like
CREATE TABLE [dbo].[TestTable](
[TagIndex] [smallint] NOT NULL,
[TimeStamp] [datetime] NOT NULL,
[SystemTime] [datetime] NOT NULL,
[ModuleControllerWatchdog] [smallint] NULL
) ON [PRIMARY]
GO
What is strange is that
If i run my process and create Sql that sends 511 rows to the database at a time that the storage looks fine and I get a result that looks like this.
If i run my process and create Sql that sends 512 rows to the database at a time then the storage expands and I get wasted storage (as below)
(Note - i have left out the sql that was generated as it quite mundane I think) and I did not want to further bloat an already long question but if anybody does want extra information of this nature then feel free to ask.
After all that, my question. do anybody know what is happening here?: Does SQL Server have a tipping point or similar whereby it will decide to do a bulk insert or use a similar mechanism one the insert has grown to a certain size etc?
Any help would be gratefully appreciated.
As per Dan Guzman requests I reran the process two times - one with 511 rows and one with 512 rows (on the same data so that I can present the difference here).
511 row chunks
512 row chunks
and then having thought it a little I decided I would include the results of the following query (which shows the distributions and allocation of pages) and shows the gaps that are present when the space is being wasted.
use [Extract_ForGuzman512Rows]
select * from sys.dm_db_database_page_allocations
(DB_id() , object_id('[dbo].[TestTable]') , NULL , NULL , 'DETAILED')
From the 511 rows table
From the 512 rows table
Following up on the idea from Dan that this may be about how much data fits on one page, I decided to run my process into different databases (one with 511 chunks and one with 512 chunks) and to run only until the first chunk had gone in so that I could check in particular to see how the pages were being allocated right from the start.
With 511 Size Chunks I could see that 2 data pages (along with a third for were being used meaning that it seemed like it went over a page's worth of data (as below)
With 512 Size Chunks I could see that again 2 data pages are being used (along with a third for the index page) but these are spaced out whereas they are tightly packed in the 511 example.