As I understood reading some articles in internet, SQL Server has a buffer cache where it stores pages, and when insert statement is executed, the modified data is written only to that buffer in memory, not to the disk.
And when system checkpoint comes all dirty pages are flushed to disk.
Does this mean that when we execute insert statement and get a return value that everything was ok, the data might still not be written to disk and in theory if system crash occurs before checkpoint, the dirty pages wont be saved to disk although we received information that everything was ok and the transaction is commited?
No. Because you totally ignore the 2nd part of the mechanism - the LOG FILE. The Log file keeps all changed pages and is flushed to disc. In case of a crash, upon start, the server will replay the changes from the log file.
Related
I am learning some basic knowledge about Oracle Database Architecture, and there are two examples.
The steps involved in executing a data manipulation language (DML) statements.
The steps involved in executing COMMIT command.
Executing DML steps
The steps are as follows:
The server process receives the statement and checks the library cache for any shared SQL area that contains a similar SQL statement.
If a shared SQL area is found, the server process checks the user’s
access privileges for the requested data, and the existing shared SQL
area is used to process the statement. If not, a new shared SQL area
is allocated for the statement, so that it can be parsed and
processed.
If the data and undo segment blocks are not already in the buffer cache, the server process reads them from the data files into the
buffer cache. The server process locks the rows that are to be
modified.
The server process records the changes to be made to the data buffers as well as the undo changes. These changes are written to the
redo log buffer before the in-memory data and undo buffers are
modified. This is called write-ahead logging.
The undo segment buffers contain values of the data before it is modified. The undo buffers are used to store the before image of the
data so that the DML statements can be rolled back, if necessary. The
data buffers record the new values of the data.
The user gets feedback from the DML operation (such as how many rows were affected by the operation).
COMMIT Process steps
The steps are as follows:
The server process places a commit record, along with the system change number (SCN), in the redo log buffer. The SCN is monotonically
incremented and is unique within the database.
The LGWR background process performs a contiguous write of all the redo log buffer entries up to and including the commit record to the
redo log files.
If modified blocks are still in the SGA, and if no other session is modifying them, then the database removes lock-related transaction
information from the blocks.
The server process provides feedback to the user process about the completion of the transaction.
First question: Do server process or background process finally move or migrate Redo log files to Data files? If yes, how to do this process?
Thanks to Nicholas Krasnov & JSapkota comments. There doesn't exist this kind of "Migration" process because they serve different purposes. Data files are the data of the database and Redo log files are used to recover the database. DBWn is responsible for writing data to data files and LGWR write redo log buffer to the active redo log file on disk.
My Second question: When does DBwn(Database writer process) modify buffers in the cache to database disk? Update database disk before COMMIT or after COMMIT.
DBwn will dose not write to database files because of issuing commit statement. Commit just means that this is a end of transaction. Locks on table or rows are released, SCN is incremented, and LGWR writes the SCN and changes to online redolog file.
The database buffer cache has two list
1) Write list
2) least-recently-used (LRU) list
Least-recently-used (LRU) list has dirty buffer. Dirty buffer is those buffers which were modified. Commit will indirectly make a buffer dirty. It will be in buffer as its been recently accessed.
DBwn neither writes in the datafile before the commit nor after. It has its own scenarios like when checkpoint happens or dirty buffers reaches threshold or there is no free buffer etc.
I hope, I answered your question. Thank you.
I am running a MERGE statement on a large table (5M of rows) with a small log file size (2GB). I am getting an error:
Merge for MyTable failed: The transaction log for database 'MyDb' is full due to 'ACTIVE_TRANSACTION'.
Could be this solved by another action than extending the log file? I can’t really afford to extend the log file currently.
If you have a fixed log file size, you have essentially two options:
Temporarily change the recovery mode of your database from FULL to BULK-LOGGED. You'll lose the ability to do point-in-time recovery during this period, but it allows you to quickly do the operation and then go back. There are other caveats, so you need to do some research to make sure this is what you want to do.
Instead of changing the transaction log, you can adopt a batching approach to commit small batches of changes at a time, thus allowing the log to flush as needed.
I am dealing with someone else's backup Maintenance Plan and have an issue with the log file, I have a database that sits on one drive with a size of 31 GB and a log file that sits on another server with a size of 20 GB, the database is in Full Recovery Model. There is a maintenance plan that runs once a day to do a complete backup and a second plan that does a backup of the log file every 15 minutes. I have checked and the drive that the log file gets backed up to and there is still plenty of room but the log file never gets smaller after the backup, is there something missing from the maintenance plan?
Thanks in advance
The situation as you describe it seems fine.
A transaction log backup does not shrink the log file. However, it does truncate the log, file, which means that space can be reused:
From Books Online (Transaction Log Truncation):
Log truncation automatically frees space in the logical log for reuse
by the transaction log.
Also, from Managing the Transaction Log:
Log truncation, which is automatic under the simple recovery model, is
essential to keep the log from filling. The truncation process reduces
the size of the logical log file by marking as inactive the virtual
log files that do not hold any part of the logical log.
This means that each time the transaction log backup occurs in your scenario, it's creating free space in the file which can be used by subsequent transactions.
Leading on from this, should you shrink the file as well? Generally speaking, the answer is no. Assuming your database does not suddenly have massive one-off spikes in usage, the transaction log will have grown to a size to accommodate the typical workload.
This means if you start shrinking the log, SQL Server will just need to grow it again... This is a resource intensive operation, affecting server performance, and no transactions can complete while the log is growing.
The current plan and file sizes all seem reasonable to me.
I don't know if this applies to your situation, but earlier versions of SQL Server 2012 have a bug that crops up when model is set to Simple recovery model. For any database created with model set to Simple, log files will continue to grow in an attempt to reach the 2,097,152 MB limit. This still applies if you alter to Full afterwards. KB article 2830400 states that altering to Full, then altering back to Simple is a workaround -- that was not my experience. Running CU 7 for SP1 was the only trick that worked for me.
The article provides links for the first updates that resolved this bug: "Cumulative Update 4 for SQL Server 2012 SP1", as well as, "Cumulative Update 7 for SQL Server 2012" (if you haven't installed SP1).
If you change the recovery to full and then back to simple, the shrink will work successfully.
I have a machine which creates a new log file at the beginning of the day(12am) and updates the log file whenever there is any changes until the end of the day.
How do I import the data in real time (30 sec, 1min or whenever there is any changes) to my SQL server database?
Will SQL Server 2008 be able to access the active log file? If not will it be easier if i let my machine create a new log file whenever there is any updates? But if it is so, how do i import so many log files with different names in real time. ( I must be able to scale the solution up to multiple machines)
Thx a lot
You can log each new line with a reversed time stamp.
Since you need to log only when the file changes you can implement an in memory queue
which reads from the file and stores the data.
Then implement a producer consumer model wherein one thread reads and loads data from the queue and the consumer logs to the database.
A windows service then can keep reading from the queue and log to the SQL Server.
(Since it's a producer consumer there will not be any busy waiting in case the queue is empty)
Somehow you will also have to notify the producer thread whenever every log is made. This can be done through Sockets/or some other means in case you have access to the code which is doing the logging.
If you have no control over the application producing the file then you have little option but to poll the file. Write an application that regularly polls the file and writes the deltas to the database. The application will need to record a high water mark that it has last read to.
Another wrinkle is that if the application does not close the file between writes then the last accessed time stamp might not be updated, so checking the age of the file may not be reliable. In this case you need to implement something like this process:
Open the log file
Seek to your last recorded EOF position
Try reading
If successful, process the new data until you get to the new EOF.
Update your persistent EOF position
Close the file
You will need to make sure that the number of bytes read aligns with your file seek position. If the log file is unicode then it may not have a 1:1 mapping between bytes and characters. You may need to read chunks of the file in binary mode and do the translation to characters from the buffer.
Once you have the log file entries parsed then you can just insert the data, or use SQLBulkCopy for larger data volumes.
If you can relax your latency constraints and the log file is small enough then you could possibly just implement a process that copies the log file to a staging area and reloads the whole thing periodically.
How about an SSIS package being called by an SQL Server Scheduled Job?
Story: today one of our customers asked us if all the data he deleted in the program was not recoverable.
Aside scheduled backups, we shrink the log file once a day, and we use the DELETE command to remove records inside our tables where needed.
Though, just for the sake of it, I opened the .mdf file with an editor (used PSPad), and searched for a particular unique piece of data -I was sure- was inside one of tables.
Problem: I tracked it in the file, then executed the DELETE command, and it was still there.
Question:
Is there a particular command we are not aware of to delete the records physically form the disk?
Note: we know there are particular techniques to recover lost data from the hard drives, but here I am talking about a notepad-wannabe!
The text may still be there, but SQL Server has no concept of that data having any structure or being available.
The "freed space" is simply deallocated: not removed, compacted or zeroed.
The "Instant File Initialization" feature relies on this too (not zeroing the entire MDF file) and previous disk data is still available eben for a brand new database:
Because the deleted disk content is overwritten only as new data is written to the files, the deleted content might be accessed by an unauthorized principal.
Edit: To reclaim space:
ALTER INDEX...WITH REBUILD is the best way
DBCC SHRINKFILE using NOTRUNCATE can compact pages into gaps caused by deallocated pages, but won't reclaim space in a page for deleted row
SQL Server just marks the space of deleted rows as available, but does not reorganize the database and does not zero out the freed up space. Try to "Shrink" the database, and the deleted rows should no longer be found.
Thanks, gbn, for your correction. A page is the allocation unit of the database, and shrinking a database only eliminates pages, but does not compact them. You'd have to delete all rows in a page in order to see them disappear after shrinking.
If your client is concerned about data security it should use Transparent Database Encryption. Even if you obliterate information from the table, the record is still in the log. Even when log is recycled, the info is still in the backups.
You could update the record with dummy values before issuing the delete, thereby overwriting the data on disk before the database marks it as free. (Whether this also works with LOB fields would warrant investigation, though).
And of course, you'd still have the problem of logs and backups, but I take it you already solved those.