We have a SQL server that we are using for our data warehouse.
We want to give a department the ability to update the data when they want (and not just on schedule).
Was is the best way to do this? We have a SP that we are thinking of calling from a batch script, but is there a more elegant way?
The data will eventually go into Palo Jedox for BI.
I do this sort of thing by writing a ColdFusion web page that the user can run. It could also be done with .net, php, java, etc.
Do not give users the ability to change the tables directly.
Instead, create one or more stored procedures to do the updates/inserts/deletes that you want to do. If it is one record, you can just pass in the values as arguments. If it is a bunch of records, you need a mechanism to transfer larger data into the database -- either reading from a text file or putting it into a table in the database some way.
Be sure the stored procedure has the same owner as the underlying tables. Using owner chaining, the stored procedure will be able to make changes to the tables. At no time can a user make a change to the data directly, only through the stored procedure.
Then, log, log, log everything that gets done. You want to know every time this stored procedure is called to change the data.
Related
I need to collect data from a SQL Server table, format it, and then put it into a different table.
I have access to SQL Server but cannot setup triggers or scheduled jobs.
I can create tables, stored procedures, views and functions.
What can I setup that will automatically collect the data and insert it into a SQL Server table for me?
I would probably create a stored procedure to do this task.
In the stored procedure you can create a CTE or use temp tables (depending on the task) and do all the data manipulation you require and once done, you can use the SELECT INTO statement to move all the data from the temp table into the SQL Server table you need.
https://www.w3schools.com/sql/sql_select_into.asp
You can then schedule this stored procedure to run at a time desired by you
A database is just a storage container. It doesn't "do" things automatically all by itself. Even if you did have the access to create triggers, something would have to happen to the table to cause the trigger to fire, typically a CRUD operation on the parent table. And something external needs to happen to initiate that CRUD operation.
When you start talking about automating a process, you're talking about the function of a scheduler program. SQL Server has one built in, the SQL Agent, and depending on your needs you may find that it's appropriate to enlist help from whoever in your organization does have access to it. I've worked in a couple of organizations, though, that only used the SQL Agent to schedule maintenance jobs, while data manipulation jobs were scheduled through an outside resource. The most common one I've run across is Control-M, but there are other players in that market. I even ran across one homemade scheduler protocol that was just built in C#.NET that worked great.
Based on the limitations you lay out in your question, and the comments you've made in response to others, it sounds to me like you need to do socialize your challenge within your organization to find out what their routine mechanism is for setting up data transfers. It's unlikely that this is the first time it's come up, unless the company was founded in the last week or two. It will probably require that you set up your code, probably a stored procedure or maybe an SSIS package, and then work with someone else, perhaps a DBA or a Site Operations team or some such, to get that process automated to fire when you need it to, whether through an Agent job or maybe a file listener.
Well you have two major options, SP and SSIS.
Both of them can be scheduled to run at a given time with a simple Job from the SQL Server Agent. Keep in mind that if you are doing this on a separate server you might need to add the source server as a Linked Server so you can access it from the script.
I've done this approach in the past and it has worked great. Note, for security reasons, I am not able to access the remote server's task scheduler, so I go through the SQL Server Agent:
Run a SQL Server Agent on a schedule of your choice
Use the SQL Server Agent to call an SSIS Package
The SSIS Package then calls an executable which can pull the data you want from your original table, evaluate it, and then insert a formatted version of it, one record at a time. Alternatively, you can simply create a C# script within the SSIS package via a Script Task.
I hope this helps. Please let me know if you need more details.
My Asp.Net MVC web-site uses stored procedures based data access to the SQL Server database. Almost every procedure should check permissions, if current user can perform this operation. I've solve it by passing additional parameter UserId to every procedure and checking if user has special permission code in special table.
It causes many copy-pasted script. I wonder, is there any another way? Or may be you have an advices to improve my solution...
Just a thought, but what if you encapsulate the logic to lookup the permission for the given user and item in a user defined function. Then, invoke the function from the necessary stored procedures and check the return value. You still will wind up with some copying and pasting but in theory it should result in a cleaner approach.
I've got a string containing an SQL statement. I want to find out whether the query will modify data or database structure, or if it will only read data. Is there some way to do this?
More info: In our application we need to let the users enter SQL-queries, mainly as part of the applications report system. These SQL queries should be allowed to read whatever they like from the databse, but they shouldn't be allowed to modify anything. No updates, deletes insert, table drops, constraint removals etc.
As of now I only test whether the first word in the string is "select", but this is too constricting and too insecure.
You should grant only select privileges on your tables for the login used by the application to be sure.
Create a new user for that part of the application that only has select privileges. Bear in mind that you'll also need to create synonyms for all the tables/views that that "read-only" user will be able to view.
The "regular" part of your application will still be able to do other operations (insert, update, delete). Just the reporting will use the read-only user.
As Horacio suggests, it is also a good idea/practice to add "wrapper" views that only expose what you want to expose. Some sort of "public API". This can give you flexibility if you need to change the underlying tables and don't want to/can't change the reports to the new definitions of said tables. This might, however, be seen as a lot of "extra work".
I agree with others that the right thing to do is use a separate schema with limited access & privileges for those queries that should be read-only.
Another option, however, is to set the transaction read-only before executing the statement entered by the user (SET TRANSACTION READ ONLY).
Create VIEWS to expose the data to end users, this is worthy because of three things:
The end user doesn't know how really your database look like.
You may can provide a simpler way to extract some pieces of data.
You can create the view with a read-only constraint:
CREATE VIEW items (name, price, tax)
AS SELECT name, price, tax_rate
FROM item
WITH READ ONLY;
Something that has worked well for me in the past, but may not fit your situation:
Use stored procedures to implement an API for the application. All modifications are done via that API. The procedures exposed to the front end are all complete units of work, and those procedures are responsible for rights enforcement.
The users running the front end application are only allowed to call the API stored procedures and read data.
Since the exposed API does complete units of work that correspond to actions the user could take via the GUI, letting them run the procedures directly doesn't get them any additional ability, nor allow them to corrupt the database accidently.
SELECT * FROM table FOR UPDATE works even with only SELECT privilege, and can still cause a lot of damage. If you want to be safe, the read only transactions are better.
I am creating an application in which i am using stored procedure where i m implementing my logic.
Now what i need to know is that- i want my database not to contain any invalid entry, for this purpose should i create triggers, for implementing my data validation logic such that when FailPasswordAttemptCount is changed to some value then should i make changes in corresponding column IsLocked accordingly thru triggers or leave it on dba to manage.
eg
if FailPassowrdAttemptCount > 3
IsCaptchaActivated=True
if FailPasswordAttemptCount>6
IsLocked=true
now if a dba changes the value of FailPasswordAttemptCount to 4 without changing IsCaptchaActivated to true then this will make an invalid entry for my frontend.
SO should i manage it thru triggers or should i left it over dba to make correct entry.
Although this type of entry is not possible thru frontend but in case any1 having privilages to access database, changes directly thru database.
For this should i leave it on user or should i manage thru triggers.
I want to make my database to remain consistent in all circumstances.
I'd make the following:
Put the data validation logic into a stored procedure
Made the stored procedure the only way the application interacts with the table
Put the code you want into the stored procedure.
Trigger-based programming paradigma grows too hard to code and maintain as the business logic complexity of your application increases.
However, if you are absolutely sure you will only have the simple logic like this, it is OK to put it into a trigger since this will require minimal changes in the ways the application interacts with the database.
I wouldn't use a trigger for something like this. Triggers are obscure and can be hard to debug for the developer. Use your tables and stored procedures to deal with the issue. Use triggers for when you don't have an alternative.
I would use a combination of both. I will try to restrict data as far as possible. And will fire trigger, so that no one can insert any invalid entry.
For this type of situation, I would probably not use a trigger, just for the situation you describe. Though I would wonder why you have dba's manually altering data in a field that closely tied to the security of your app.
I would implement this in the application logic. When calling the login sproc you can return both whether it succeeded as well as the number of failed password attempts and if captcha is needed. Regardless of if the DBA changes the 3 to a 4, your code will see the 4, ignore the result of the validation and present the user with a captcha. If you're worried about DBA's modifying the code directly you can also check the APP_NAME() function/variable to see what program is trying to modify the data. Its something to be very careful with but so is DBAs modifying fields directly.
I am thinking about creating stored procedures on the fly.
ie running CREATE PROCEDURE... when the (web) application is running.
What are the risks or problems that it can cause?
I know that the database account needs to have the extra privileges.
It does NOT happen everyday. Only from time to time.
I am using sql server and interested in mysql and postgres as well.
Update1:
Thanks to comments, I am considering creating a new version of stored procedure and switching over instead of ALTERing the sp. example: sp1 -> sp2 -> sp3
Update2:
The reason:
My schema changes because of custom fields (unknown number and type of columns)
I tried dynamic sql and sp_executesql first. Of course it works. Dynamic sql works greate for 1,2,3 simple update,inserts.
But it got too ugly and a lot of work and it does not mix well with stored procedure, problems with sql parameterization because it is used inside a stored procedure and the number and type of params is not known at compile time (long story).
At least the basic scenario for this solution is not that complicated.
The logic of the sp does NOT change. For each custom field I have to add a new parameter to sp and add a column to update, insert, etc.
I also considered making stored procedure parameters dynamic like sp_executesql that accepts any number and type of params but could not find a way.
For a transactional system it's probably quite expensive. If you have a large batch job and want to use a code generator for some reason (quite a common architecture in ETL tools, notably Oracle Warehouse Builder and Wherescape Red), it's not unreasonable to do this.
You mentioned that you would be adding and/or changing the calling profile of the stored procedure when you do this alteration. How are you lock-stepping the new calling profile with the application that makes the call to this? What's your roll-back plan if you ever have to revert a change that was made?
In the past what I've done is just append an incrementing numeric suffix to the stored procedure name with the new calling profile -- then you can modify the old version of the SP to call the new one with a default value for the parameter, and then you can release your software calling the new version.
If something breaks in your new version and you have to rollback, calls to the old stored proc will still work without error, and just populate the custom fields with your default values.
Firstly, the answer to this question really depends on what exactly this stored procedure is intended to do. If it's just reading data or creating a result set for reporting and you don't mind if it's a little inconsistent, then you're probably fine. If it's doing anything remotely interesting with your data then it's a very risky thing to be doing. You should think about whether it's possible (and what would happen) for two users users (or the same user twice) to run multiple versions of the the same stored procedure at the same time. Smells like a train wreck to me. One option is to only allow this procedure alteration to take place when no other users are logged into the system, or forcibly boot them off the database if they are. Another option is to create your new stored procedure with a slightly different name and swap them over when you deem it safe to do so.
Another issue is that one of the major benefits of stored procedures is that the execution plan is cached, meaning it will execute faster. If you are creating them on the fly you lose that advantage.
If you really need to do this then you should randomise the name of the procedure to avoid clashing with other users. Remember always that other users may be doing their own thing at the same time - most database systems won't give transactional isolation for stored procedures (Postgres is the only one I know of that does).
It would be extremely rare that this would be a desirable thing to do - could you elaborate at all on what made you choose this approach?
I would not do that personally.
As you mentioned you will need extra privileges to grant access to create/alter database objects. That can create a serious security risk as nothing would stop your application from creating a malicious stored procedure if someone discovered a security hole in it.
If your schema changes, change the stored procedures with the schema.
You will not be able to alter the procedure if one or more users are running the procedure, or another procedure that references your procedure. You will block until all the dependent procedures and the procedure you want to compile (and I think the procedure s you invoke from your procedure, but I am not certain) are not in use. This may be a long time on a busy production system, and if you are unlucky, you may timeout waiting for all the dependencies to not be in use (5 minutes on Oracle).
You can also get into very ugly situations (I have). Take for example stored procedures B and C, both of which call A, the procedure that you are trying to compile. When no one is running B, the system locks B. Now any user trying to run B will stall. The system then tries to lock C, but C is generating a very lengthy report that will not be done for another 10 minutes. You will timeout waiting for the lock, and some of your users will have an unresponsive system for 5 minutes. My experience is with Oracle, I would make sure your target DBMS does not behave in the same fashion, or has quicker failures or a better lock acquisition strategy.
I guess I am cautioning that what looks like may work on a development server may fail dramatically on a busy production system.
I'm not sure that the locking discussed by Tony BanBrahim is true in SQL Server 2005.
I have some long-running SPs (a 3 hours batch process of about 30 sub-processes), and I have been able to alter the SP while it is still running. (I don't believe the changes take effect until the next run, but it doesn't cause any blocking or any error). Now the outer long-running SP does both call SPs dynamically with EXEC and statically, but I've change both the root and nested SPs while they are running without error messages or blocks.
WRT your original question, I would think that your tactic is fine if used in a controlled way.
I don't know for sure, but it sounds like one or both:
an architectural problem
is existing code locking the schema tables from the application?
I'd take a look to see what code is locking the schema tables and rewrite that code. Do you have a 3rd party something or other that is locking those tables?