Here is the scenario: I have some database model with about 500K new records everyday. The database is almost never updated (only insert statement and delete).
Many users would like to perform queries against database with tools such as PowerBI or so, but I haven't given any access to anybody to prevent deadlocking (I only allow specific IT managed resource to access the data).
I would like to open up data access, but I must prevent any one from blocking the new records insertions.
Could I create a view with nested no-lock inside it assuming no dirty read are created since no update are performed?
Would that be an acceptable design? I know it's is not a perfect solution and it's not mean for that.
It's a compromise to allow user with no SQL skills to perform ad-hoc queries and lookup.
Anything I might be missing?
I think that you can use 'WITH (NoLock) ' in front of table name in query, such as :
SELECT * FROM [table Name] WITH (NoLock)
Related
Consider the following abstract situation (just as an example):
I have two tables TableA and TableB. They have unique IDs and possibly other columns (which are irrelevant) The relatioship between them is many to many so I have a third table AssociationTable that is used to store the relationships between them. Basically, AssociationTable will have two columns (ID_A and ID_B - foreign keys).
If I delete a row in AssociationTable and the ID_A that was deleted was the last one, I would also like to delete the entry from TableA that corresponds to that ID.
I could do this:
a) From the application that uses the database
b) by using an SQL trigger
My question, basically, is the following:
Is there any good practice that says "if you can do something from both the application and from SQL, always prefer sql." ?
Or does it depend on the case? If so, what should I take into account?
Performance: The query plan for stored procedures is compiled onn DB Server and subsequent requests can run faster.
A stored procedure can execute multiple steps and the intermediate results need not go back to application layer, reducing traffic between an application and the DB server.
Security: Stored procedures are well defined database objects that can be locked down with security measures. Use of typed parameters can help prevent SQL injection attacks.
Code re-use: SQL queries can be written once and re-used across multiple clients without writing the same SQL commands over and over again.
Abstraction: By putting all the SQL code into a stored procedure, the application is completely abstracted from the field names, tables names, etc. So when a SQL query needs to be changed, there is almost zero or NO impact in the application code.
There are more benefits of doing it in the database.
Other client application code need not worry about data integrity.
The data logic should remain as close to data as possible
It could be faster if managed by DB (trigger invocation).
I have a particular SQL file in which i copy all contents from on table in a database to another table in another database.
The traditional INSERT statements are used to perform the same operation. However this table has 8.5 Million records and it fails. The queries succeed with a smaller database.
Also in when i run the select * query for that particular table the SQL query express shows out of memory exception.
In particular there is one table that has some many records. So this table alone i want to copy from the old Db to the new Db.
What are alternate ways to achieve this?
Is there any quick work around by which we can avoid this exception and make the queries succeed?
Let me put it this way. Why would this operation fail when there are a lot of records?
I don't know if this counts as "traditional INSERT", but have you tried "INSERT INTO"?
http://www.w3schools.com/sql/sql_select_into.asp
I have a Rails 3.2 multi-tenant subdomain based app which I'm trying to migrate over to PostgreSQL's schemas (each account getting its own schema -- right now all of the accounts use the same tables).
So, I'm thinking I need to:
Create a new DB
Create a schema for each Account (its id) and the tables under them
Grab all the data that belongs to each account and insert it into the new DB under the schema of said account
Does that sound correct? If so, what's a good way of doing that? Should I write a ruby script that uses ActiveRecord, plucks the data, then inserts it (pretty inefficient, but should get the job done) into the new DB? Or does Postgres provide good tools for doing such a thing?
EDIT:
As Craig recommended, I created schemas in the existing DB. I then looped through all of the Accounts in a Rake task, copying the data over with something like:
Account.all.each do |account|
PgTools.set_search_path account.id, false
sql = %{INSERT INTO tags SELECT DISTINCT "tags".* FROM "tags" INNER JOIN "taggings" ON "tags"."id" = "taggings"."tag_id" WHERE "taggings"."tagger_id" = #{admin.id} AND "taggings"."tagger_type" = 'User'}
ActiveRecord::Base.connection.execute sql
#more such commands
end
I'd do the conversion with SQL personally.
Create the new schemas in the same database as the current one for easy migration, because you can't easily query across databases with PostgreSQL.
Migrate the data using appropriate INSERT INTO ... SELECT queries. To do it without having to disable any foreign keys, you should build a dependency graph of your data. Copy the data into tables that depend on nothing first, then tables that depend on them, and so on.
You'll need to repeat this for each customer schema, so consider creating a PL/PgSQL function that uses EXECUTE ... dynamic SQL to:
Create the schema for a customer
Create the tables within the schema
Copy data in the correct order by looping over a hard-coded array of table names, doing:
EXECUTE `'INSERT INTO '||quote_ident(newschema)||'.'||quote_ident(tablename)||' SELECT * FROM oldschema.'||quote_ident(tablename)||' WHERE customer_id = '||quote_literal(customer_id)'||;'
where newschema, tablename and customer_id are PL/PgSQL variables.
You can then invoke that function from SQL. While you could do just select convert_customer(c.id) FROM customer GROUP BY c.id, I'd probably do it from an external control script just so each customer's work got done and committed individually, avoiding the need to start again from scratch if the second-to-last customer conversion fails.
For bonus crazy points it's even possible to define triggers on the main customer schema's tables that replicate changes to already-migrated customers over to the copy of their data in the new schema, so they can keep using the system during the migration. I'd avoid that unless the migration was just too big to do without downtime, as it'd be a nightmare to test and you'd still need the triggers to throw an error on access by customer id x while the migration of x's data was actually in-progress, so it wouldn't be fully transparent.
If you're using different login users for different customers (strongly recommended) your function can also:
REVOKE rights on the schema from public
GRANT limited rights on the schema to the user(s) or role(s) who'll be using it
REVOKE rights on public from each table created
GRANT the desired limited rights on each table to the user(s) and role(s)
GRANT on any sequences used by those tables. This is required even if the sequence is created by a SERIAL pseudo-column.
That way all your permissions are consistent and you don't need to go and change them later. Remember that your webapp should never log in as a superuser.
One third party app is storing data in a huge database (SQL Server 2000/2005). This database has more than 80 tables. How would I come to know that how many tables are affected when application stores a new record in database? Is there something available I can retrieve the list of tables affected?
You might be able to tell by running a trace in SQL Profiler on the database - the SQL:StmtCompleted event is probably the one to monitor - i.e. if the application does a series of inserts into multiple tables, you should see them go through in Profiler.
You can use SQL Profiler to trace SQL queries. So you will see sequence of calls caused by one button click in your application.
Also use can use metadata or SQL tools to get list of triggers which could make a lot of actions on simple insert.
If you have the SQL script that used to store the new record(Usually, it should be insert statement, or other DML statement such as update, merge and so on). Then you may know how many tables were affected by parsing those SQL script.
Take this SQL for example:
Insert into emp(fname, lname)
Values('john', 'reyes')
You can get result like this:
sstinsert
emp(tetInsert)
Tables:
emp
Fields:
emp.fname
emp.lname
you can add triggers on tables that get fired on update - you could use this to update a log table that would report what was being updated.
see more here: http://www.devarticles.com/c/a/SQL-Server/Using-Triggers-In-MS-SQL-Server/
Profiler is the way to go, as others have said especially with an unfamilar third party database.
I would also spend some time creating diagrams so you can see the foreign key relationships and understand how the database is put together. I usaully know my database structure so well, I can tell from the fields being inserted what tables they affect and I know what triggers are on my tables and what they affect. There is no substitute for taking the time to understand the database you support.
I'm pretty sure there is no way, but i'm putting this out there for those expert beyond my knowledge.
What i am looking to do is to somehow alter SELECT statements before they are executed, at the database level. For a seriously pared-down example, i'd like to do something like the following... when someone executes the following SQL
SELECT * FROM users.MESSAGES
i'd like to catch it, before it executes, and alter the statement to something like
SELECT * FROM users.MESSAGES WHERE RECIPIENT = ORIGINAL_LOGIN()
allowing me to enforce user limitations on the data in a fashion similar to ORACLE's VPDs, without needing to resort to creating views on top of all my tables that might need this.
Look into using a VIEW.
Sadly, this is not possible.
Even the Microsoft SQL Server row-level security features (e.g. in the security catalogs) are implemented using views.
So, if you really need the feature, you're going to have to set up views with SUSER_NAME() or some similar individual- or role-identifier in the WHERE clauses.
Sorry!
Use views (or inline table-valued functions), generate the views automatically and remove rights from the tables.
There used to be a round-about unethical way in SQL 2000. You could create a trigger on master..sysprocesses table for INSERT and do this kind of manipulation. Thankfully, this is not possible, at least AFAIK, in SQL 2005, as master..sysprocesses is a fake table.
For the benefit of some of US still using SQL 2000, here is how to do this in SQL 2000:
In the console right click on
servername
In the properties go to server
settings tab
Then check allow modifications to be
made directly to the system catalogs
checkbox.
Selected the sysprocesses table in
sysobjects and change it's xtype
from S(system) to U(user)
Now go to Master DB, tables - right
click on sysprocesses-All
tasks-Manage Triggers
Then you can write your trigger
After you are done, turn back
everything to it's original state.
Even with all of this, I still doubt if you can change the Select statement.
Raj
Disclaimer: Try this at your own risk. Be warned that you are making changes to system objects which could lead to undesirable results.
First, grant no direct table access to the users so they can't see the data from an ad hoc query. Make them use a stored proc to access the table and one of the parameters of the proc is the user login. Then write the code so that it only selects records for that login.