FileTable and Foreign Key from another table - sql

I try to use FileTable with Entity Framework (I know it is not supported directly). So I use custom Sql commands to insert and delete (no update) the data. My problem is I have a table which refers to the FileTable with a foreign key to the stream_id of the FileTable. If I insert into the FileTable, how can I get the stream_id back?
I want to use SqlBulkCopy to insert lots of files, I can bulk insert into the FileTable, but SqlBulkCopy won´t tell me the inserted stream_id values.
If I execute single insert statements with select scopeIdentity() or something similar, the performance becomes worse.
I want to insert like 5.000 files (2MB until 20MB) into the FileTable and connect them with my own Table via foreign key. Is this bad practice and I should use a simple path column and store the data directly in the filesystem? I thought FileTable is doing exactly this for me, because I need to secure the database and the files are always in sync even if I go one hour or 4 days back in the past. I cannot backup the database and the filesystem exactly at the same time so they are 100 percent synchronized.

I want to use SqlBulkCopy to insert lots of files, I can bulk insert into the FileTable, but SqlBulkCopy won´t tell me the inserted stream_id values.
SqlBulkCopy doesn't allow to retrieve inserted identity values or any other values.
Solution 1
You can find on the web a lot of code snippets to insert into a temporary table using SqlBulkCopy. Then from the temporary table to the destination table using the OUTPUT clause to get the stream_id values.
It's a few more steps, but the performance is still very great.
Solution 2
Disclaimer: I'm the owner of the project Entity Framework Extensions
Disclaimer: I'm the owner of the project Bulk Operations
Both libraries are not free but allow to overcome SqlBulkCopy limitation more easily.
Both of them support to output identity value.
// Easy to customize
var bulk = new BulkOperation<Customer>(connection);
bulk.BatchSize = 1000;
bulk.ColumnInputExpression = c => new { c.Name, c.FirstName };
bulk.ColumnOutputExpression = c => c.CustomerID;
bulk.ColumnPrimaryKeyExpression = c => c.Code;
bulk.BulkMerge(customers);
// Easy to use
var bulk = new BulkOperation(connection);
bulk.BulkInsert(dt);
bulk.BulkUpdate(dt);
bulk.BulkDelete(dt);
bulk.BulkMerge(dt);

Related

How to make only one user can insert data at a time in SQL Server?

I have a SQL Server database, multi-user can insert to it.
But for many reasons, I want only 1 user can insert at a time.
Example:
User 1 want to insert 100 record, while user 1 is inserting (100 record not saved to table). Other user can not insert to the table.
I have thought to use a flag, but I want to find another way.
Is there any SQL statement that can do that?
Thanks for reading!
It seems that you need to use
INSERT INTO TABLE with (rowlock)
Read the following post to have a better understanding.
Using ROWLOCK in an INSERT statement (SQL Server)
Updated
SQL supports us to handle 1 record at a time e. And your case is to want multiple records to handle serialized format.
I think the best you put into the temp table, there is a window service running real-time (Background service: using quartz job or hangfire): insert and delete then the temporary table with a column named IsInserted.
For that purpose you can used table lock or row loack concept.
ALTER TABLE Table_name SET (LOCK_ESCALATION = < TABLE | AUTO | DISABLE > –One of those options)
For more details you can also visit this link
locking in SQL Server

How to encrypt/decrypt data in some columns of a table and when a new record gets inserted it should also get encrypt

i know like this to insert a new record
INSERT INTO dbo.Customer_data (Customer_id, Customer_Name, Credit_card_number)
VALUES (25665, 'mssqltips4', EncryptByKey( Key_GUID('SymmetricKey1'), CONVERT(varchar,'4545-58478-1245') ) );
but i want to insert a new record with a normal insert statement which should get encrypted.
ex:
INSERT INTO dbo.Customer_data (Customer_id, Customer_Name, Credit_card_number)
VALUES (25665, 'mssqltips4','4545-58478-1245') ) );
Few months ago I had similar situation. A table containing personal data need to have some of the columns encrypted, but the table is used in legacy application and have many references.
So, I you can create a separate table to hold the encrypted data:
CREATE TABLE [dbo].[Customer_data_encrypted]
(
[customer_id] PRIMARY KEY -- you can create foreign key to the original one, too
,[name] VARBANRY(..)
,[cretit_card_numbe] VARBINARY(..)
);
Then create a INSTEAD OF INSERT UPDATE DELETE trigger on the original table.The logic in the trigger is simple:
on delete, delete from both tables
on update/insert - encrypt the data and insert in the new table; use some kind of mask to the original table (for example *** or 43-****-****-****)
Then, perform a initial migration to move the data from the original table to the new one and then mask it.
Performing the steps above are nice because:
every insert/update to the original table continue to works
you can create the trigger with EXECUTE AS OWNER in order to have access to the symmetric keys and perform changes directly in the T-SQL statement without opening the certificates or by users who have not access to them
in all reads references you are going to get mask data, so you are not worried for breaking the application critically
having trigger gives you ability to easy create and changes information
It depends on your environment and business needs because for one of the tables I have stored the encrypted value as new column, not separate table. So, choose what is more appropriate for you.

Batch insert data counting new inserts

Suppose i have a simple schema with a composite pk with columns. e.g
pk1: string
pk2: int
date: Timestamp
I am reading data from somewhere else in batches of about 50 and would like to store this. The data source i am pulling from is a sliding window so I will be receiving data from the data source that i have already inserted so i cant just blindly insert otherwise i get a pk constraint violation.
I would like a reasonable way to insert the new items as a batch but also knowing how many new items i actually inserted for logging purposes.
doing the insert
For postgresql version 9.5+, it is possible to use the following:
insert ... on conflict do nothing
example:
INSERT INTO users (id, user_name, email)
VALUES (1, 'hal', 'hal#hal.hal')
ON CONFLICT DO NOTHING
For recent earlier versions (since 9+, i think), it is possible to create a CTE from raw values & then insert from there:
WITH batch (id, user_name, email) AS (
VALUES
(1, 'hal', 'hal#hal.hal'),
(2, 'sal', 'sal#sal.sal')
)
INSERT INTO users (id, user_name, email) (
SELECT id, user_name, email
FROM batch
WHERE batch.id NOT IN (SELECT id FROM users)
)
or, instead of using a CTE, stage the values in a staging table that is truncated after every batch is processed.
Also, note that it might be necessary to explicitly cast strings to appropriate data types if the CTE method is used.
A third option would be to implement this using a stored procedure & trigger. This is more complicated than the other two, but would work with earlier versions of postgresql.
logging
Both of those methods should report the number of rows inserted, but the logging would have to be performed by the database client.
e.g. in Python, the library psycopg2 is used to interact with postgresql, and psycopg2 cursor objects have a property rowcount. I'm sure other well designed libraries written in other languages / frameworks will have implemented this same functionality somehow. Logging the # of rows inserted will have to be done from the part of the program interacting with the database.
However, if the logs of how many rows are inserted are required in the same database, then both the upsert & the logging may be performed via a single trigger + stored procedure.
Finally, as this is a special case of upsert, more information can be found by searching postgresql upsert on stack overflow or other sites. I found the following from the postgresql wiki very informative:
https://wiki.postgresql.org/wiki/UPSERT#PostgreSQL_.28today.29

SQL Server bulk insert for large data set

I have 1 million rows of data in a file, I want to insert all the records into SQL Server. While inserting I am doing some comparison with existing data on the server, if the comparison satisfied I will update the existing records in the server or else I will insert the record from the file.
I'm currently doing this by looping from C#, which consume more than 3 hours to complete the work. Can anyone suggest idea to improve the performance?
Thanks,
Xavier.
Check if your database in Full or Simple recovery mode:
SELECT recovery_model_desc
FROM sys.databases
WHERE name = 'MyDataBase';
If database is SIMPLE recovery mode you can create a staging table right there. If it is in Full mode then better create Staging table in separate database with Simple model.
Use any BulkInsert operation/tool (for instance BCP, as already suggested)
Insert only those data from your staging table, which do not exist in your target table. (hope you know how to do it)

SQL Server, Using a stored procedure to insert data

I'm trying to insert data into my database by using a stored procedure but 3 of my columns are using the int identity type and I cannot insert. It keeps saying cannot do this whilst identity insert is off
When IDENTITY_INSERT is on, it just means that you can put your own data in IDENTITY column. It doesn't disable the FK constraint you have on the table. You can delete the FK constraint, or disable it, and risk having logically inconsistent data in your DB, or you can fix your SP so you won't insert any duplicate values.
Something is amiss. Three columns in a single table of type Identity? I'm having difficulty imagining what they could represent, and I have to wonder where the natural keys are.
In any case, IDENTITY_INSERT isn't something you want to putz with casually. It's an administrative feature to allow ad hoc changes to the data, for example bulk loading the database.
If you do actually know what the identities are (as input to your stored procedure) then the table is misdefined, because it's supposed to be the identity source. If you don't know, or you're willing to let the table generate identity values, then you simply don't mention those columns in your INSERT statement. At most, the generated values would be OUTPUT parameters to your stored procedure.