We have a SQL Server 2008 R2 Enterprise a database that is populated with realtime data coming from multiple sources.
We want to have this data available as is in an internal application. However we would like to let this same data be accessed through another application (web) but in a "delayed" way.
Meaning that this application would only "see" the data as it was 5-10 or 15 minutes in the past.
Is there some out of the box way to do that? If not how would you suggest tackling that problem?
I don't think there is anything "out-of-the-box" to solve this.
The only thing I can think of is to add a datetime column to each table that should be accessed by the "other" application. That datetime needs to record the creation timestamp of the row.
Then create a view for each table that restricts the rows being returned based on the current time and the creation timestamp.
The "other" app is then only allowed access through the views.
You can clone the database every 15 minutes and let the other app use the clone. Just backup the original database and restore it with diffrent name. This way you do not need to modify the data model at all. Just make sure you create a new database first and then let the web app use it. Never overwrite the database when its in use.
Related
we need some help, choosing the best database for the following upcoming situation:
Write:
We'll get a huge number of data each minute (round about one million entries) and need to save them in a database. One of the unique identifiers of the entries are timestamps.
Read:
The software should be able to load the stored data for an user-defined time range. The loading process should be as fast as possible.
Currently, we are using the Microsoft SQL Server in our application. I'm not sure, if this is the right technology for our new requirements.
Which database should we use? Do we need to replace MSSQL with something or is another database able to run simultaneously with it?
Thank you!
In my application (C# application, using Entity-Framework and SQL Database), I am needed to create a daily task to update/insert data from a third-party application (both the applications are using SQL server database). For efficiency sake, I am looking for a way to determine what all records from the previous day have been modified and thus import only those records.
I know I can add a modified_on column to the source table and create a trigger to update that column when something is changed on that record, but that will need me to make changes to the third-party application's database schema which I want to avoid.
There's the change tracking feature but it's of limited use to you as you're using EF and that makes the way the data is queried awkward. You may be able to use it somehow, but I doubt it's elegant.
Way easier is to indeed change the schema but add only a single column of type rowversion. That binary datatype (loaded as byte[] in EF) is special and gets larger every time something (such as the third-party application) updates the row. No need for any triggers. You can look what the largest one is you already processed and then query all those that are larger than that.
In addition to change tracking suggested by John, in another answer, you can think of setting up Temporal tables.
You can run queries against the temporal tables to identify the changed records and pull them accordingly from main table.
For reasons I'm not about to explain, We keep a Access database that is to be a copy of a subset of a larger oracle database. It is not feasible to refer to data directly in the Oracle database due to speed issues (don't ask).
Every time a specific application is opened the local Access database is updated from the newest data found to the time of opening the application. First of all this does not capture changes in the existing records. Secondly it does not take into account changes in the source database made after opening the application.
For this reason several checks may be needed when carrying out certain operations in the application. So is it possible to update the local Access database only with changes in the Oracle database in a smarter and faster way than the hard way I am imagining (I'm not a PL/SQL / SQL expert)? Possibly it might be sufficient to look for changes only after a certain date (stored in one of the fields of the recordset retrieved).
Any suggestions?
You might want to look into data replication beethween Oracle en MSAccess databases. For example thru an ODBS drive or sqlserver database. Just google "ms access oracle replication" an see if this solves your problem.
Simple situation. I've created an application which uses SQL Server as database. I could add a table to this database which contains the version number of my application, so my application can check if it's talking to the correct version of the database. But since there are no other settings that I store inside a database, this would mean that I would add a single table with a single field, which contains only one record.
What a waste of a good resource...
Is there another wat that I can tell the SQL Server database about the product version that it's linked to?
I'm not interested in the version of SQL Server itself but of the database that it's using.
(Btw, this applies to both SQL Server 2000 and 2005.)
If you're using SQL 2005 and up, you can store version info as an Extended Property of the database itself and query the sys.extended_properties view to get the info, eg :
sys.sp_addextendedproperty #name=N'CurrentDBVersion', #value=N'1.4.2'
SELECT Value FROM sys.extended_properties WHERE name = 'CurrentDBVersion' AND class_desc = 'DATABASE'
If SQL 2000, I think your only option is your own table with one row. The overhead is almost non-existent.
I'd go with the massive overhead of a varchar(5) field with a tinyint PK. It makes the most sense if you're talking about a product that already uses the SQL Server database.
You're worried about overhead on such a small part of the system, that it becomes negligible.
I would put the connection settings in the application or a config file that the application reads. Have the app check the version number in the connection settings.
Even if there was such a feature in SQL Server, I wouldn't use it. Why?
Adding a new table to store the information is negligible to both the size and speed of the application and database
A new table could store other configuration data related to the application, and you've already got a mechanism in place for it (and if your application is that large, you will have other configuration data)
Coupling the application to a specific database engine (especially this way) is very rarely a good thing
Not standard practice, and not obvious to someone new looking at the system for the first time
I highly recommend writing the data base version into the database.
In an application we maintained over a decade or so we had updates of the database schema every release.
When the user started the application after an update installation it could detect if the database was to old and convert it to the newer schema. We actually did an incremental update: In order to get from 7 to 10 we did 7 -> 8, 8->9, 9->10.
Also imagine the scenario when somebody restores the database to an older state from a backup.
Don't even think about adding a single table, just do it (and think about the use cases).
I am writing code to migrate data from our live Access database to a new Sql Server database which has a different schema with a reorganized structure. This Sql Server database will be used with a new version of our application in development.
I've been writing migrating code in C# that calls Sql Server and Access and transforms the data as required. I migrated for the first time a table which has entries related to new entries of another table that I have not updated recently, and that caused an error because the record in the corresponding table in SQL Server could not be found
So, my SqlServer productions table has data only up to 1/14/09, and I'm continuing to migrate more tables from Access. So I want to write an update method that can figure out what the new stuff is in Access that hasn't been reflected in Sql Server.
My current idea is to write a query on the SQL side which does SELECT Max(RunDate) FROM ProductionRuns, to give me the latest date in that field in the table. On the Access side, I would write a query that does SELECT * FROM ProductionRuns WHERE RunDate > ?, where the parameter is that max date found in SQL Server, and perform my translation step in code, and then insert the new data in Sql Server.
What I'm wondering is, do I have the syntax right for getting the latest date in that Sql Server table? And is there a better way to do this kind of migration of a live database?
Edit: What I've done is make a copy of the current live database. Which I can then migrate without worrying about changes, then use that to test during development, and then I can migrate the latest data whenever the new database and application go live.
I personally would divide the process into two steps.
I would create an exact copy of Access DB in SQLServer and copy all the data
Copy the data from this temporary SQLServer DB to your destination database
In that way you can write set of SQL code to accomplish second step task
Alternatively use SSIS
Generally when you convert data to a new database that will take it's place in porduction, you shut out all users of the database for a period of time, run the migration and turn on the new database. This ensures no changes to the data are made while doing the conversion. Of course I never would have done this using c# either. Data migration is a database task and should have been done in SSIS (or DTS if you have an older version of SQL Server).
If the databse you are converting to is just in development, I would create a backup of the Access database and load the data from there to test the data loading process and to get the data in so you can do the application development. Then when it is time to do the real load, you just close down the real database to users and use it to load from. If you are trying to keep both in synch wile you develop, well I wouldn't do that but if you must, make a nightly backup of the file and load first thing in the morning using your process.
You may want to look at investing in a tool like SQL Data Compare.
I believe it has support for access databases too, and you can download a trial.
I you are happy with you C# code, but it fails because of the constraints in your destination database you temporarily can disable them and then enable after you copy the whole lot.
I am assuming that your destination database is brand new DB with no data, and not used by anyone when the transfer happens
It sounds like you have two problems:
You're migrating data from one database to another.
You're changing your schema.
Doing either of these things is tricky if you are trying to migrate the data while people are using the data.
The simplest approach is to migrate the data based on a static copy of the data, and also to queue updates to that data from the moment you captured the static copy. I don't know how easy this is in Access, but in SQLServer or Oracle you can use the redo logs for this or a manual solution using triggers. The poor-man's way of doing this is to make triggers for all the relevant tables that log the primary key of the records that have changed. Then after the old database is shut off you can iterate over those keys and get those records from the old database and put them into the new database. Just copy the whole record; if the record was deleted then delete it from the new database.
Your problem is compounded by the fact that you can't simply copy the data, you have to transform it. This means you probably have to shut down both databases and re-migrate the records based on the change list. It will take a lot of planning to ensure you get things right and I'd recommend writing a testing script that can validate that the resulting data is correct.
Also I'd ensure that the code for the migration runs inside one of the databases if possible. Otherwise you are copying the data twice and this will significantly harm the performance.