I have below text file having different words inside it:
My aim is to insert only 4 character words from the text file into a table variable which is #temp, using bcp command.
So, at the end, the table variable #temp will look like below:
Create a table where you will store the data coming from your file:
create table import(WORDS nvarchar(100))
Import data from file with bcp into the table created in the first step:
bcp [test].[dbo].[import] in d:\test.txt -c -T
Declare #table variable:
declare #table table ([ID] int identity(1,1), WORDS nvarchar(100))
Insert into #table variable only words with length = 4:
insert into #table
select WORDS
from import
where len(WORDS) <= 4
Now #table variable contains this data:
Related
I am working on a project that has a C# front end that will be used to select a file for importing into an MSSQL SQL Database. In the table there will be an additional column called 'recommendedAction' (tinyint - 0-5 only)
I would like to have sql fill in the 'recommendedAction' column based on criteria in a different table.
Is there a way that when SQL is importing (SSIS or pure TSQL) it could read the values of a table and fill in the 'action' based on the criteria? Or is this something that should be done in the C# frontend?
EDIT
SQL table structure for imported data (with additional column)
Create Table ImportedData (
Column1 INT Identity,
Column2 VARCHAR(10) NOT NULL,
Column3 CHAR(6) NOT NULL,
RecommendedAction TINYINT NOT NULL
)
Table structure of recommended action criteria
Create Table RecommendedActions(
ID INT Identity,
ActionID TINYINT NOT NULL, --value to put in the RecommendedAction column if criteria is a match
CriteriaColumn VARCHAR(255) NOT NULL --Criteria to match against the records
)
Example records for RecommendedActions
ID ActionID CriteriaColumn
1 2 'Column2 LIKE ''6%'''
2 3 'Column2 LIKE ''4%'''
Now when a new set of data is imported, if Column2 has a value of '6032' it would fill in a RecommendedAction of 2
Many ways exist. For example you can insert into the tb table a value selected from the ta table according to criteria.
Example setup
create table ta(
Id int,
val int);
insert into ta(ID, val) values
(1, 30)
,(2, 29)
,(3, 28)
,(4, 27)
,(5, 26);
create table tb
(Id int,
ref int);
Example insert
-- parameters
declare #p1 int = 1,
#p2 int = 27;
-- parameterized INSERT
insert tb(Id, ref)
values(#p1, (select ta.id from ta where ta.val=#p2));
Below added Stored procedure will do the job. It gets the Action column value based on the Column2 parameter and insert into the ImportedData table. You can execute this Stored procedure inside the C# code with required parameters. I added sample execute statements for to test the query.
Sample data inserted to the RecommendedActions Table:
INSERT INTO RecommendedActions
VALUES
(2, 'Column2 LIKE ''6%''')
,(3, 'Column2 LIKE ''4%''')
Stored Procedure Implementation :
CREATE PROCEDURE Insert_ImportedData(
#Column2 AS VARCHAR(10)
,#Column3 AS CHAR(3)
)
AS
BEGIN
DECLARE #RecommendedAction AS TINYINT
SELECT #RecommendedAction = ActionID
FROM RecommendedActions
WHERE SUBSTRING(CriteriaColumn, 15, 1) = LEFT(#Column2 , 1)
INSERT INTO ImportedData VALUES (#Column2,#Column3,#RecommendedAction)
END
GO
This is the execute statement for the Above Stored procedure
EXEC Insert_ImportedData '43258' , 'ATT'
EXEC Insert_ImportedData '63258' , 'AOT'
you can use sqlalchemy in python and load your data into a dataframe then append the dataframe to the sql table. You can set the dtype for each of the field datatype in the read_csv using a dictionary. Loading data with Python is super powerful because the bulk load is fast. Use your c# code to build the csv file using stream io and use linq to for your conditions for data fields. Then use python to load your csv.
import pandas as pd
from sqlalchemy import create_engine
engine = create_engine(connectionstring)
df = pd.read_csv("your_data.csv", header=None)
df.columns = ['field1', 'field2', 'field3']
df.to_sql(name="my_sql_table", con=connection, if_exists='append', index=False)
I have a table that looks like this:
memberno(int)|member_mouth (varchar)|Inspected_Date (varchar)
-----------------------------------------------------------------------------
12 |'1;2;3;4;5;6;7' |'12-01-01;12-02-02;12-03-03' [7 members]
So by looking at how this table has been structured (poorly yes)
The values in the member_mouth field is a string that is delimited by a ";"
The values in the Inspected_Date field is a string that is delimited by a ";"
So - for each delimited value in member_mouth there is an equal inspected_date value delimited inside the string
This table has about 4Mil records, we have an application written in C# that normalizes the data and stores it in a separate table. The problem now is because of the size of the table it takes a long time for this to process. (the example above is nothing compared to the actual table, it's much larger and has a couple of those string "array" fields)
My question is this: What would be the best and fastest way to normilize this data in MSSQL proc? let MSSQL do the work and not a C# app?
The best way will be SQL itself. The way followed in the below code is something which worked for me well with 2-3 lakhs of data.
I am not sure about the below code when it comes to 4 Million, but may help.
Declare #table table
(memberno int, member_mouth varchar(100),Inspected_Date varchar(400))
Insert into #table Values
(12,'1;2;3;4;5;6;7','12-01-01;12-02-02;12-03-03;12-04-04;12-05-05;12-07-07;12-08-08'),
(14,'1','12-01-01'),
(19,'1;5;8;9;10;11;19','12-01-01;12-02-02;12-03-03;12-04-04;12-07-07;12-10-10;12-12-12')
Declare #tableDest table
(memberno int, member_mouth varchar(100),Inspected_Date varchar(400))
The table will be like.
Select * from #table
See the code from here.
------------------------------------------
Declare #max_len int,
#count int = 1
Set #max_len = (Select max(Len(member_mouth) - len(Replace(member_mouth,';','')) + 1)
From #table)
While #count <= #max_len
begin
Insert into #tableDest
Select memberno,
SUBSTRING(member_mouth,1,charindex(';',member_mouth)-1),
SUBSTRING(Inspected_Date,1,charindex(';',Inspected_Date)-1)
from #table
Where charindex(';',member_mouth) > 0
union
Select memberno,
member_mouth,
Inspected_Date
from #table
Where charindex(';',member_mouth) = 0
Delete from #table
Where charindex(';',member_mouth) = 0
Update #table
Set member_mouth = SUBSTRING(member_mouth,charindex(';',member_mouth)+1,len(member_mouth)),
Inspected_Date = SUBSTRING(Inspected_Date,charindex(';',Inspected_Date)+1,len(Inspected_Date))
Where charindex(';',member_mouth) > 0
Set #count = #count + 1
End
------------------------------------------
Select *
from #tableDest
Order By memberno
------------------------------------------
Result.
You can take a reference here.
Splitting delimited values in a SQL column into multiple rows
Do it on SQl server side, if possible a SSIS package would be great.
Given a FilePath how to retrieve contents from the file and store all contents into a column in blob data type.
Ex: Input: "F:\Data\sample.txt"
Output Logic: Get Contents of "F:\Data\sample.txt" and store it in Table Column
Thanks in advance
Here is what I have.
create table #ORStable (doclen bigint, doc varbinary(max))
insert into #ORStable
select len(bulkcolumn), *
from
openrowset(bulk 'C:\Users\xxxx\Desktop\Notes\master.txt', SINGLE_BLOB) --filepath here
as r
select *
from #ORStable
All I could find... Closest you are going to get to your answer
Source
Hi Finally i got the answer, this works for me
CREATE TABLE #tempfile (line varchar(8000))
EXEC ('bulk INSERT #tempfile FROM ''D:\New Text Document.txt'' ')
SELECT * FROM #tempfile
DROP TABLE #tempfile
I have a big SQL file (~ 200MB) with lots of INSERT instructions:
insert into `films_genres`
(`id`,`film_id`,`genre_id`,`num`)
values
(1,1,1,1),
(2,1,17,2),
(3,2,1,1),
...
How could I remove or ignore columns id, num in the script?
Easiest way might be to do the full insert into a temporary holding table and then insert the desired columns into the real table from the holding table.
insert into `films_genres_temp`
(`id`,`film_id`,`genre_id`,`num`)
values
(1,1,1,1),
(2,1,17,2),
(3,2,1,1),
...
insert into `films_genres`
(`film_id`,`genre_id`)
select `film_id`,`genre_id`
from `films_genres_temp`
CREATE TABLE #MyTempTable (id int,film_id smallint, genre_id int, num int)
INSERT INTO #MyTempTable (id,film_id,genre_id,num)
[Data goes here]
insert into films_genres (film_id,genre_id) select film_id,genre_id from #MyTempTable
drop table #MyTempTable
This Perl one-liner should do it:
perl -p -i.bak -e 's/\([^,]+,/\(/g; s/,[^,]+\)/\)/g' sqlfile
It edits the file in place, but creates a backup copy with the extension .bak.
Or if you prefer Ruby:
ruby -p -i.bak -e 'gsub(/\([^,]+,/, "("); gsub/, ")");' sqlfile
What's wrong with this T-SQL :
DECLARE #temp TABLE(ID INT IDENTITY,[Value] VARCHAR(100))
SET #temp = dbo.[fnCSVToTable](',2,3')
I don't think you can assign to the table variable like that (unless it is a new thing in SQL 2008).
At least for SQL2005 you would need to do the following.
DECLARE #temp TABLE(ID INT IDENTITY,[Value] VARCHAR(100))
INSERT INTO #temp
SElECT [value]
FROM dbo.[fnCSVToTable](',2,3')
From the docs for SET (SQL 2008; SQL 2005) (my emphasis):
# local_variable
Is the name of a
variable of any type except cursor, text, ntext, image, or table.
To populate a table variable, use
INSERT #table_variable
SELECT columns
FROM dbo.fnTableValuedFunction