I am creating a large intermediate data set (millions of records, half hour runtime) that is used as parts of several other different queries. Ideally this would be stored as a materialized view or cached table but I don't have DDL access on the server.
Subqueries and CTE's are inefficent because they must be re-executed every time and can't be indexed. Local/global temporary tables work well but get lost when I disconnect.
Is there any way to create a more persistent temporary table (or equivalent) without DDL access?
Related
I know basic sql commands, and this is my first time working with stored procedures. In the stored procedure I am looking at, there are several temporary tables.
The procedure, is triggerred every morning, which then pulls a specific ID and then loops through each ID to grab certain parameters.
My question is: are temporary tables used in stored procedures so that when the procedure goes off the variables will be instantly passed into the parameters and loop, and then the temporary tables will be cleared, thus restarting the process for the next loop?
Temporary tables are used because as soon as the session that created them (or stored procedure) is closed the temporary table is gone. A Temp table with a single # in front of the name (also called a local temp table) is only visible in the session it was created in so a temp table with the same name can be created in multiple sessions without bumping into each other (SQL Server adds characters to the name to make it unique). If a temp table with two ## in front of it is created (a global temp table) then it is unique within SQL Server so other sessions can see it. Temp tables are the equivalent of a scratch pad. When SQL Server is restarted all temp tables and their values are gone. Temp tables can have indexes created against them and SQL Server can use statistics on Temp tables to create efficient query plans.
For stored procedures (SPs), they are the least restricted and most capable objects, for example:
Their usual alternatives, views and functions, are not allowed to utilize many things, like DML statements, temp. tables, transactions, etc.
SPs can avoid returning result sets. They also can return more than one result set.
For temp. tables:
Yes, once the SP is done, the table disappears along with its
contents (at least for single-# tables).
They have advantages &
disadvantages compared to their alternatives (actual tables, table
varaibles, various non-table solutions).
Stored Procedures, in my opinion, don't forcibly need Temporary tables. It's up to the SP's scope to decide if using a TempTable is the best approach.
For example, let's suppose that we want to retrieve a List of elements that come out from joining a few tables, then it's the best to have a TempTable to put the joined fields. On the other hand, if we're using a Stored Procedure to retrieve a single or field, I don't see the need for a Temp table.
Temp tables are only available during the usage of the Stored Procedure, once it's finished, the table goes out of scope.
I want to create a temp table so as to be able to join it to a few tables because joining those tables with the content of the proposed temporary table takes a lot of time (fetching the content of the temporary table is time consuming.Repeating it over and over takes more and more time). I am dropping the temporary table when my needs are accomplished.
I want to know if these temporary tables would be visible over other client session(my requirement is to make them visible only for current client session). I am using postgresql. It would be great if you could suggest better alternatives to the solution I am thinking of.
PostgreSQL then is the database for you. Temporary tables done better than the standard. From the docs,
Although the syntax of CREATE TEMPORARY TABLE resembles that of the SQL standard, the effect is not the same. In the standard, temporary tables are defined just once and automatically exist (starting with empty contents) in every session that needs them. PostgreSQL instead requires each session to issue its own CREATE TEMPORARY TABLE command for each temporary table to be used. This allows different sessions to use the same temporary table name for different purposes, whereas the standard's approach constrains all instances of a given temporary table name to have the same table structure.
Pleas read the documentation.
Temporary tables are only visible in the current session and are automatically dropped when the database session ends.
If you specify ON COMMIT, the temporary table will automatically be dropped at the end of the current transaction.
If you need good table statistics on a temporary table, you have to call ANALYZE explicitly, as these statistics are not collected automatically.
By default , temporary tables are visible for current session only and temporary tables are dropped automatically after commit or closing that transaction, so you don't need to drop explicitly.
The auto vacuum daemon cannot access and therefore cannot vacuum or analyze temporary tables. For this reason, appropriate vacuum and analyze operations should be performed via session SQL commands. For example, if a temporary table is going to be used in complex queries, it is wise to run ANALYZE on the temporary table after it is populated.
Optionally, GLOBAL or LOCAL can be written before TEMPORARY or TEMP. This presently makes no difference in PostgreSQL and is deprecated
Does the time it takes to drop a table in SQL reflect the quantity of data within the table?
Lets say for dropping an identical table one with 100000 rows and 1000 rows.
Is this MySQL? The ext3 Linux filesystem is known to be slow to drop large tables. The time to drop will have to do more with the size of the table, more than the rows in the table.
http://www.mysqlperformanceblog.com/2009/06/16/slow-drop-table/
It will definitely depend on the dbserver and on the specific storage params in that db server and upon the existence of LOBs in the table. For MOST scenarios, it's very fast.
In Oracle, dropping a table is very quick and unlikely to matter compared to other operations you would be performing (creating the table and populating it).
It would not be idomatic to be creating and dropping tables enough in Oracle for this to be any sort of a performance factor. You would instead consider using global temporary tables.
From http://download.oracle.com/docs/cd/B28359_01/server.111/b28318/schema.htm
In addition to permanent tables,
Oracle Database can create temporary
tables to hold session-private data
that exists only for the duration of a
transaction or session.
The CREATE GLOBAL TEMPORARY TABLE
statement creates a temporary table
that can be transaction-specific or
session-specific. For
transaction-specific temporary tables,
data exists for the duration of the
transaction. For session-specific
temporary tables, data exists for the
duration of the session. Data in a
temporary table is private to the
session. Each session can only see and
modify its own data. DML locks are not
acquired on the data of the temporary
tables. The LOCK statement has no
effect on a temporary table, because
each session has its own private data.
I read about temporary tables, global temporary tables and table variables. I understood it but could not imagine a condition when I have to use this. Please elaborate on when I should use the temporary table.
Most common scenario for using temporary tables is from within a stored procedure.
If there is logic inside a stored procedure which involves manipulation of data that cannot be done within a single query, then in such cases, the output of one query / intermediate results can be stored in a temporary table which then participates in further manipulation via joins etc to achieve the final result.
One common scenario in using temporary tables is to store the results of a SELECT INTO statement
The table variable is relatively new (introduced in SQL Server 2005 - as far as i can remember ) can be used instead of the temp table in most cases. Some differences between the two are discussed here
In a lot of cases, especially in OLTP applications, usage of temporary tables within your procedures means that you MAY possibly have business processing logic in your database and might be a consideration for you to re-look your design - especially in case of n tier systems having a separate business layer in their application.
The main difference between the three is a matter of lifetime and scope.
By a global table, I am assuming you mean a standard, run of the mill, table. Tables are used for storing persistent data. They are accessible to all logged in users. Any changes you make are visible to other users and vice versa.
A temporary table exist solely for storing data within a session. The best time to use temporary tables are when you need to store information within SQL server for use over a number of SQL transactions. Like a normal table, you'll create it, interact with it (insert/update/delete) and when you are done, you'll drop it. There are two differences between a table and a temporary table.
The temporary table is only visible to you. Even if someone else creates a temporary table with the same name, no one else will be able to see or affect your temporary table.
The temporary table exists for as long as you are logged in, unless you explicitly drop it. If you log out or are disconnected SQL Server will automatically clean it up for you. This also means the data is not persistent. If you create a temporary table in one session and log out, it will not be there when you log back in.
A table variable works like any variable within SQL Server. This is used for storing data for use in a single transaction. This is a relatively new feature of TSQL and is generally used for passing data between procedures - like passing an array. There are three differences between a table and a table variable.
Like a temporary table, it is only visible to you.
Because it is a variable, it can be passed around between stored procedures.
The temporary table only exists within the current transaction. Once SQL Server finishes a transaction (with the GO or END TRANSACTION statements) or it goes out of scope, it will be deallocated.
I personally avoid using temporary tables and table variables, for a few reasons. First, the syntax for them is Microsoft specific. If your program is going to interact with more than one RDBMS, don't use them. Also, temporary tables and table variables have a tendency to increase the complexity of some SQL queries. If your code can be accomplished using a simpler method, I'd recommend going with simple.
I am trying to understand the tempDB and following are the doubts popping in my mind.
What is the lifetime of data in tempDB? Say a query is doing some Order By and uses tempDB for performing that. After this query finishes, someone else also executes a query which utilizes the tempDB. Will the second query find records written by first query in the tempDB or will they be deleted?
Are there any visible tables created inside the tempDB by the Sql Engine? How can I know which temporary table is created because of this query? Is there any naming convention followed by the Sql engine for naming these temporary tables?
I am new to tempDB so please pardon me for asking such silly (if at all) questions :-)
It will be very nice if someone can point me to a good resource which can help me learn about tempDB.
Temp table is stored in tempdb until the connection is dropped (or in the case of a global temp tables when the last connection using it is dropped). You can also (and it is a good practice to do so) manually drop the table when you are finished using it with a drop table statement.
No, others cannot see your temp tables if they are local temp tables (They can see and use global temp tables) Multiple people can run commands which use the same temp table name but they will not be overlapping in a local temp table and so you can have a table named #test and so can 10,000 other users, but each one has its own structure and data.
You don't want to generally look up temp tables in tempdb. It is possible to check for existence, but that is the only time I have ever referenced tempdb directly. Simply use your temp table name. Example below of checking for existence
IF OBJECT_ID('TempDB.dbo.#DuplicateAssignments') IS NOT NULL
BEGIN
DROP TABLE #DuplicateAssignments
END
You name temp tables by prefacing the name with # (for local tables the ones you would use 999.9% of the time) and ## for global temp tables, then the rest of the name you want.
There's a few MSDN articles that are probably the best source of information on the tempDB database in SQL Server.
tempdb Database
The tempdb system database is a global
resource that is available to all
users connected to the instance of SQL
Server and is used to hold the
following:
Temporary user objects that are explicitly created, such as: global or
local temporary tables, temporary
stored procedures, table variables, or
cursors.
Internal objects that are created by the SQL Server Database Engine, for
example, work tables to store
intermediate results for spools or
sorting.
Row versions that are generated by data modification transactions in a
database that uses read-committed
using row versioning isolation or
snapshot isolation transactions.
Row versions that are generated by data modification transactions for
features, such as: online index
operations, Multiple Active Result
Sets (MARS), and AFTER triggers.
Operations within tempdb are minimally
logged. This enables transactions to
be rolled back. tempdb is re-created
every time SQL Server is started so
that the system always starts with a
clean copy of the database. Temporary
tables and stored procedures are
dropped automatically on disconnect,
and no connections are active when the
system is shut down. Therefore, there
is never anything in tempdb to be
saved from one session of SQL Server
to another. Backup and restore
operations are not allowed on tempdb.
There's also tempdb and Index Creation, this blog post along with Working with tempdb in SQL Server 2005 which states:
The SQL Server system database, tempdb, has undergone a number of changes in SQL Server 2005. There are new tempdb usages and internal optimizations in SQL Server 2005; tempdb architecture is mostly unchanged since SQL Server 2000.
The tempdb system database is very similar to a user database. The main difference is that data in tempdb does not persist after SQL Server shuts down.
The temporary tables created in TempDB are dropped when the query is completed.
I'm not sure on this (I would have to try it), but I think theoretically ALL tables created in TempDB are visible, although only the user that created the table has permission to access it.