I want to implement a simple in memory cache for an sql server 2008 R2 Express Edition.
Requirements:
The cache must be visible for all users
The cache is synchronized with a persistent table. Over this persistent table might occur heavy CRUD operations.
My solution:
Create a bucket of CLR procedures that stands into a .NET assembly. These procedures will be named:
Add (param1, ... paramN);
Remove (param1 ... paramN);
GetAll ();
where param1... paramN is my persistent table data specific that i want to cache.
Behind the scenes I will organize in memory data maybe into a HashSet for O (1) add / remove. Then, I will create a trigger for Insert / Delete operations (only inserts and deletes can occur on the persistent table) that would call these methods (Add / Remove). The big issue is that the transaction can be rolled back. How can I find out that the transaction with a specific ID was rolled back? There is a server level event that I can listen?
Please give me whatever alternatives you know. I know that SQL Server 2014 supports memory optimized OLTP tables but ... this feature is active only for SQL Server enterprise edition :D.
Another alternative would be to create a temporary table with ## (visible on all connections) BUT:
The table will be stored in TEMPDB. If the tempDB is stored in RAM, there might be a performance boost but then it comes the second issue:
Searching in it (for a specific item) in this temporary table will cost me O ( Log ( N )) because behind the scenes if I create an index, there will be a B-tree. My In memory alternative will ensure that lookups will cost O (1).
If you have enough RAM, you're really not reading the data from disc, all your often accessed data will be in memory. I don't really know the limits of express edition in detail, so those could be causing issues you're facing. I think you should at least consider upgrading to standard version, especially if you very big number of records, since you'll most likely run out of space too.
Temp DB is no exception, it will be either in disc or RAM, depending on how much memory you have and what is being accessed the most. Just creating cache 1:1 for data from other tables into tempdb is not useful, you'll just waste your memory for storing the same data twice.
The in memory OLTP of SQL Server 2012 wouldn't probably help you at all. The point of it is not that you have the data in memory, but to reduce the overhead of locking & latching etc.
Related
While i have got some further insight from this post i am still struggling to get a fuller understanding of my recent use of variable tables and the associated tempdb growth.
I have recently been using variable tables within both stored procedures and table-valued functions. My use of variable tables v's local / global temporary tables is one area that might affect the larger challenge i'm experiencing.
Since using this type of temp table the tempdb has grown to around the 50+GB region and when inspecting the table using exec sp_spaceused #updateusage=true i am seeing: database_size: 51935.13MB unallocated_space: 51908.80MBand when checking the contents of the DB no temporary or system tables are present. For ref the tempdb.ldf is very small.
On inspecting my session usage using exec sp_who i am also seeing multiple rows which indicate sleeping which i suspect may be where we are experiencing issues with connections not closing properly.
From reading various posts and SO's the general consensus is not to attempt shrinking of the tempdb and associated files and in truth i'd prefer to resolve the underlying issue than move to more fragmented data storage.
Is there any advice on why my existing approach could be affecting the tempdb growth and if the use of local / global temporary tables would be more appropriate.
Regarding the tempdb it's self, while storage is cheap, i need to ensure this growth is contained so any advice on maintenance (Splitting the DB into multiple files, possible shrinking, moving the DB to a separate drive) ect would be appreciated.
You can inspect objects inside tempdb database to know what is going on
The following code list objects in tempdb and sort asc based on creation date and compute duration in minute
use TempDb
go
SELECT
name ,object_id , SCHEMA_NAME(schema_id) obj_schema,
type , type_desc,
create_date , modify_date,
OBJECT_NAME(parent_object_id) parent_obj ,
DATEDIFF(hour,create_date , GETDATE()) duration_hour
FROM sys.objects
where name not like 'sys%'
order by create_date
The objects in tempDb is classified into three groups:
Internal objects
External objects
Version store
The following code show how much the tempdb disk space is allocated to each category
SELECT
SUM(user_object_reserved_page_count)/128.0 UserObjectsMB,
SUM(user_object_reserved_page_count) UserObjectPages_count ,
SUM(version_store_reserved_page_count)/128.0 VersionStoreMB,
SUM(version_store_reserved_page_count) VersionStorePages_count,
SUM(internal_object_reserved_page_count) InternalObjectPages_count,
SUM(internal_object_reserved_page_count)/128.0 InternalObjectsMB,
SUM(unallocated_extent_page_count)/128.0 FreeSpaceMB,
SUM(unallocated_extent_page_count) FreePages_count
FROM sys.dm_db_file_space_usage;
you can review:
local-and-global-temporary-tables-in-sql-server
The size of tempdb should be big enough for daily and peak workload, to avoid the grow while the WorkLoad is running.
My advice, Don't shrink tempDb during startup or any other time, unless if necessary needed.
Storage is cheap, and you can assign dedicated storage for tempDb (even SSD)
For more details:
Capacity Planning for tempdb
Optimizing tempdb Performance
We have a 60 GB production database in my new organization. We run closely 500 reports overnight from this DB. I notice that all the report scripts create tables in TempDB and then populate the final report. TempDB size is 6 GB. There is no dependency set up for these report scripts, which are called from PowerShell.
Is it a good practice to use TempDB extensively in this manner? Or is it better to create all the staging tables in the production database itself and drop them after the report is generated?
Thanks,
Roopesh
Temporary tables always gets created in TempDb. However, it is not necessary that size of TempDb is only due to temporary tables. TempDb is used in various ways
Internal objects (Sort & spool, CTE, index rebuild, hash join etc)
User objects (Temporary table, table variables)
Version store (AFTER/INSTEAD OF triggers, MARS)
So, as it is clear that it is being use in various SQL operations so size can grow due to other reasons also. However, in your case if your TempDb has sufficient space to operate normally and if your internal process is using TempDb for creating temporary tables and it is not an issue. You can consider TempDb as an toilet for SQL Server.
You can check what is causing TempDb to grow its size with below query
SELECT
SUM (user_object_reserved_page_count)*8 as usr_obj_kb,
SUM (internal_object_reserved_page_count)*8 as internal_obj_kb,
SUM (version_store_reserved_page_count)*8 as version_store_kb,
SUM (unallocated_extent_page_count)*8 as freespace_kb,
SUM (mixed_extent_page_count)*8 as mixedextent_kb
FROM sys.dm_db_file_space_usage
if above query shows,
Higher number of user objects then it means that there is more usage of Temp tables , cursors or temp variables
Higher number of internal objects indicates that Query plan is using a lot of database. Ex: sorting, Group by etc.
Higher number of version stores shows Long running transaction or high transaction throughput
You can monitor TempDb via above script and identify the real cause of its growth first. However, 60 GB is quite a small database with 6GB TempDB size is fairly acceptable.
Part of above answer is copied from my other answer from SO.
I have an INSERT statement that is eating a hell of a lot of log space, so much so that the hard drive is actually filling up before the statement completes.
The thing is, I really don't need this to be logged as it is only an intermediate data upload step.
For argument's sake, let's say I have:
Table A: Initial upload table (populated using bcp, so no logging problems)
Table B: Populated using INSERT INTO B from A
Is there a way that I can copy between A and B without anything being written to the log?
P.S. I'm using SQL Server 2008 with simple recovery model.
From Louis Davidson, Microsoft MVP:
There is no way to insert without
logging at all. SELECT INTO is the
best way to minimize logging in T-SQL,
using SSIS you can do the same sort of
light logging using Bulk Insert.
From your requirements, I would
probably use SSIS, drop all
constraints, especially unique and
primary key ones, load the data in,
add the constraints back. I load
about 100GB in just over an hour like
this, with fairly minimal overhead. I
am using BULK LOGGED recovery model,
which just logs the existence of new
extents during the logging, and then
you can remove them later.
The key is to start with barebones
tables, and it just screams. Building
the index once leaves you will no
indexes to maintain, just the one
index build per index.
If you don't want to use SSIS, the point still applies to drop all of your constraints and use the BULK LOGGED recovery model. This greatly reduces the logging done on INSERT INTO statements and thus should solve your issue.
http://msdn.microsoft.com/en-us/library/ms191244.aspx
Upload the data into tempdb instead of your database, and do all the intermediate transformations in tempdb. Then copy only the final data into the destination database. Use batches to minimize individual transaction size. If you still have problems, look into deploying trace flag 610, see The Data Loading Performance Guide and Prerequisites for Minimal Logging in Bulk Import:
Trace Flag 610
SQL Server 2008 introduces trace flag
610, which controls minimally logged
inserts into indexed tables.
In Sqlsever Enterprise manager, there are some default databases are provided like tempdb and etc. What is significance of those databases?
TempDB is used for temporary work in SQL Server. Anytime you create a temp table that storage is done inside of TempDB. Here is a very good article from MSDN
Here are some points referenced from the MSDN:
The tempdb system database is a global resource that is available to all users connected to the instance of SQL Server and is used to hold the following:
•Temporary user objects that are explicitly created, such as: global or local temporary tables, temporary stored procedures, table variables, or cursors.
•Internal objects that are created by the SQL Server Database Engine, for example, work tables to store intermediate results for spools or sorting.
•Row versions that are generated by data modification transactions in a database that uses read-committed using row versioning isolation or snapshot isolation transactions.
•Row versions that are generated by data modification transactions for features, such as: online index operations, Multiple Active Result Sets (MARS), and AFTER triggers.
Operations within tempdb are minimally logged. This enables transactions to be rolled back. tempdb is re-created every time SQL Server is started so that the system always starts with a clean copy of the database. Temporary tables and stored procedures are dropped automatically on disconnect, and no connections are active when the system is shut down. Therefore, there is never anything in tempdb to be saved from one session of SQL Server to another. Backup and restore operations are not allowed on tempdb.
There is also the master database (holds information about all databases), Model database, MSDB (stores information on the sql agent, dts, jobs, etc).
More info here as well
MASTER - This keeps all server-level information, and meta-data about all databases on the server. Don't lose this :)
MSDB - Holds information about SQL Agent jobs and job run history
TEMPDB - Used as a temporary "work space" for temporary tables and lots of other stuff (like sorting and grouping)
MODEL - When you create a new, blank database, it makes a copy of MODEL as a template
DISTRIBUTION - (You will only see this on servers where you have set up replication) Holds records pending for replication.
SQL Server uses tempdb to store internal objects such as the intermediate results of a query. You can get more details here.
I am trying to understand the tempDB and following are the doubts popping in my mind.
What is the lifetime of data in tempDB? Say a query is doing some Order By and uses tempDB for performing that. After this query finishes, someone else also executes a query which utilizes the tempDB. Will the second query find records written by first query in the tempDB or will they be deleted?
Are there any visible tables created inside the tempDB by the Sql Engine? How can I know which temporary table is created because of this query? Is there any naming convention followed by the Sql engine for naming these temporary tables?
I am new to tempDB so please pardon me for asking such silly (if at all) questions :-)
It will be very nice if someone can point me to a good resource which can help me learn about tempDB.
Temp table is stored in tempdb until the connection is dropped (or in the case of a global temp tables when the last connection using it is dropped). You can also (and it is a good practice to do so) manually drop the table when you are finished using it with a drop table statement.
No, others cannot see your temp tables if they are local temp tables (They can see and use global temp tables) Multiple people can run commands which use the same temp table name but they will not be overlapping in a local temp table and so you can have a table named #test and so can 10,000 other users, but each one has its own structure and data.
You don't want to generally look up temp tables in tempdb. It is possible to check for existence, but that is the only time I have ever referenced tempdb directly. Simply use your temp table name. Example below of checking for existence
IF OBJECT_ID('TempDB.dbo.#DuplicateAssignments') IS NOT NULL
BEGIN
DROP TABLE #DuplicateAssignments
END
You name temp tables by prefacing the name with # (for local tables the ones you would use 999.9% of the time) and ## for global temp tables, then the rest of the name you want.
There's a few MSDN articles that are probably the best source of information on the tempDB database in SQL Server.
tempdb Database
The tempdb system database is a global
resource that is available to all
users connected to the instance of SQL
Server and is used to hold the
following:
Temporary user objects that are explicitly created, such as: global or
local temporary tables, temporary
stored procedures, table variables, or
cursors.
Internal objects that are created by the SQL Server Database Engine, for
example, work tables to store
intermediate results for spools or
sorting.
Row versions that are generated by data modification transactions in a
database that uses read-committed
using row versioning isolation or
snapshot isolation transactions.
Row versions that are generated by data modification transactions for
features, such as: online index
operations, Multiple Active Result
Sets (MARS), and AFTER triggers.
Operations within tempdb are minimally
logged. This enables transactions to
be rolled back. tempdb is re-created
every time SQL Server is started so
that the system always starts with a
clean copy of the database. Temporary
tables and stored procedures are
dropped automatically on disconnect,
and no connections are active when the
system is shut down. Therefore, there
is never anything in tempdb to be
saved from one session of SQL Server
to another. Backup and restore
operations are not allowed on tempdb.
There's also tempdb and Index Creation, this blog post along with Working with tempdb in SQL Server 2005 which states:
The SQL Server system database, tempdb, has undergone a number of changes in SQL Server 2005. There are new tempdb usages and internal optimizations in SQL Server 2005; tempdb architecture is mostly unchanged since SQL Server 2000.
The tempdb system database is very similar to a user database. The main difference is that data in tempdb does not persist after SQL Server shuts down.
The temporary tables created in TempDB are dropped when the query is completed.
I'm not sure on this (I would have to try it), but I think theoretically ALL tables created in TempDB are visible, although only the user that created the table has permission to access it.