I am trying to perform a simply copy activity in Azure Data Factory from CSV to SQL Table, but I'm getting the following error:
{
"errorCode": "2200",
"message": "ErrorCode=DelimitedTextMoreColumnsThanDefined,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Error found when processing 'Csv/Tsv Format Text' source 'organizations.csv' with row number 6696: found more columns than expected column count 41.,Source=Microsoft.DataTransfer.Common,'",
"failureType": "UserError",
"target": "Copy data1",
"details": []
}
The copy activity is as follows
Source
My Sink is as follows:
As preview of the data in source is as follows:
This seems like a very straight forward copy activity. Any thoughts on what might be causing the error?
My row 6696 looks like the following:
3b1a2e5f-d08b-166b-4b91-eb53009b2377 Compassites Software Solutions organization compassites-software https://www.crunchbase.com/organization/compassites-software 318375 17/07/2008 10:46 05/12/2022 12:17 company compassitesinc.com http://www.compassitesinc.com IND Karnataka Bangalore "Pradeep Court", #163/B, 6th Main 3rd Cross, JP Nagar 3rd phase 560078 operating Custom software solution experts Big Data,Cloud Computing,Information Technology,Mobile,Software Data and Analytics,Information Technology,Internet Services,Mobile,Software 01/11/2005 51-100 info#compassitesinc.com 080-42032572 http://www.facebook.com/compassites http://www.linkedin.com/company/compassites-software-solutions http://twitter.com/compassites https://res.cloudinary.com/crunchbase-production/image/upload/v1397190270/c3e5acbde40f36eaf4f8c6f6eda3f803.png company
No commas
As the error message indicates, there is a record at row number 6696 where there is a value containing , as a character in it.
Look at the following demonstration where I have taken a similar case. I have 3 columns in my source. The data looks as shown below:
When I run use similar dataset settings and read these values, the same error would be thrown.
So, the value T1,OG is being considered as if they belong to 2 different columns since they have dataset delimiter within the value.
Such values would throw an error as it is ambiguous to read. One way to avoid this is to enclose such values with quote character (double quote in this case).
Now when I run the copy activity, it would give the desired output.
The table data would look like this:
I have a few large fixed with text files that have multiple specification formats in them. I need to parse out the txt files based on a character with a set location in the file. That character can have a different position in the file.
I have written queries for each of the different specifications (95 of them) with the start position and length hard coded into the query using the mid() function with a WHERE() function to filter the [Record Identifier] from the specification. As you can see below the 2 specifications in the WHERE() function have different placements in the txt file.
\\\
SELECT Mid([AllData],1,5) AS PlanNumber, Mid([AllData],6,4) AS Spaces1, Mid([AllData],10,3) AS Filler1, Mid([AllData],13,11) AS SSN, Mid([AllData],24,1) AS AccountIdentifier, Mid([AllData],25,5) AS Filler2, Mid([AllData],30,2) AS RecordIdentifier, Mid([AllData],32,1) AS FieldType, Mid([AllData],33,4) AS Filler3, Mid([AllData],37,8) AS HireDate, Mid([AllData],45,8) AS ParticipationDate, Mid([AllData],53,8) AS VestinDate, Mid([AllData],61,8) AS DateOfBirth, Mid([AllData],77,1) AS Spaces2, Mid([AllData],78,1) AS Reserved1, Mid([AllData],79,1) AS Reserved2, Mid([AllData],80,1) AS Spaces3
FROM TBL_Company1
WHERE (((Mid([AllData],30,2))="02") AND ((Mid([AllData],32,1))="D"));
\\\
Or
\\\
SELECT Mid([AllData],1,5) AS PlanNumber, Mid([AllData],6,4) AS Spaces1, Mid([AllData],10,3) AS Filler1, Mid([AllData],13,11) AS SSN, Mid([AllData],24,1) AS AccountIdentifier, Mid([AllData],25,7) AS RecordIdentifier, Mid([AllData],32,22) AS StreetAddressForBank, Mid([AllData],54,20) AS CityForBank, Mid([AllData],74,2) AS StateForBank, Mid([AllData],76,5) AS ZipCodeForBank
FROM TBL_Company1
WHERE (((Mid([AllData],25,7))="49EFTAD"));
\\\
Is there a way to Parse out this without having to hard code every position and length into the code?
I was thinking of having a table with all of the specifications in it and have an import function look to the specification table and parse out the data accordingly to a new table or maybe something else.
What I have done is not very scalable and if the format changes a little I would have to go back to each query to change it.
Any Help is greatly appreciated
I think in your situation, I'd want to be able to generate the SQL statement dynamically, as you suggest.
I'd have a table something like:
Format#,Position,OutColName,FromPos,Length,WhereValue
1,1,"PlanNumber",1,5,
1,2,"Spaces1",6,4,
...
1,n,,30,2,"02"
1,n+1,,32,1"D"
and then some VBA to process it and build and execute the SQL string(s). The SELECT clause entries would be recognized by having a value in the OutColName field and WHERE clause entries by values in the the WhereValue column.
Of course this is only more "efficient" in the sense that it's a bit easier to code up new formats or fix/modify existing ones.
I wrote the following UPDATE statement:
UPDATE print_archive
SET pages = pages*2
WHERE printer ILIKE '%plot%' AND paper_size = 'Arch D';
It will look in a table named print_archive for any printers with "plot" in their name and the paper size 'Arch D'.
If it finds any it is suppose to multiply the page count by 2.
Below is a sample of the print_archive table data -
(column names)
"name_id","printer","pages","source","date","file_name","duplex","paper_size"
Sample Data:
jane, \\PRINTSRV\plot9, 1, \\COMP-01, 01/21/2017 14:30:39, hello_world.pdf, No, Arch D,
billy, \\PRINTSRV\Plot13, 1, \\COMP-02, 02/20/2016 10:37:23, bye_world.doc, No, Arch D,
But no matter what I change or how many times I run the UPDATE statement it always returns 0.
What am I doing wrong?
So initially I had added the latin1 encoding when importing the CSV file as the import process kept producing encoding error messages when using the default encode. I switched the import of the CSV file to 'latin2' and that worked better. There does seem to be a leading space as well in some of the fields so adding '%' helped as well (though this was not working in latin1). pgAdmin4 (which is how my users will interact) handles latin2 pretty good. Though from psql cmd I need to set the client encoding to latin2 before it can work (SET client_encoding to 'latin2';)
Thanks guys. All your recommendations and leads helped me.
One result, under one field of data contains the below, it's extremely long. I need to be able to pull out certain substrings into seperate columns.
Desired Result:
1) email addresses that it's being sent to, identified by "TO": gregory.dettorre#cardinalhealth.com; scott.ballard#cardinalhealth.com
2) email addresses that it's being CC'd to, identified by "CC":
GMB-OptiFreight-CCBABR#cardinalhealth.com
3) email addresses that it's being CC'd to, identified by "ReplyTo":
OptiFreightcustomercare#cardinalhealth.com
4) Include report: True
5) Render Format: Excel
6) Subject: 13 Week Volume File - LifePoint Health - Brentwood, TN
Result:
"<ParameterValues><ParameterValue><Name>TO</Name>
<Value>gregory.dettorre#cardinalhealth.com;
scott.ballard#cardinalhealth.com</Value></ParameterValue><ParameterValue>
<Name>CC</Name><Value>GMB-OptiFreight-CCBABR#cardinalhealth.com</Value>
</ParameterValue><ParameterValue><Name>ReplyTo</Name>
<Value>OptiFreightcustomercare#cardinalhealth.com</Value></ParameterValue>
<ParameterValue><Name>IncludeReport</Name><Value>True</Value>
</ParameterValue><ParameterValue><Name>RenderFormat</Name>
<Value>EXCEL</Value></ParameterValue><ParameterValue><Name>Subject</Name>
<Value>13 Week Volume File - LifePoint Health - Brentwood, TN</Value>
</ParameterValue><ParameterValue><Name>Comment</Name><Value>Please see the
attached 13 week volume file and let us know if you have any questions.
OptiFreightcustomercare#cardinalhealth.com</Value></ParameterValue><ParameterValue><Name>IncludeLink</Name><Value>False</Value></ParameterValue><ParameterValue><Name>Priority</Name><Value>NORMAL</Value></ParameterValue></ParameterValues>"
Here there is an answered question on splitting strings, using SUBSTRING and CHARINDEX in SSRS. You Get the indexes of 2 delimiters (e.g. "TO" and "CC"), and by applying SUBSTRING between these 2 delimiters you get the value that you wanted.
Also, the best practice would probably be splitting the data in the dataset (e.g. SQL query) itself, instead of doing so in the report itself.
I have text in columns that I want to parse out to multiple columns. The text is separated by \
Example 1 column:
alarmname
Rainier Beach (Henderson)\Tunnel Facility\C29_Facility\Offline - PLC 70
Rainier Beach (Henderson)\Tunnel Facility\C30_Facility\PLC DC Power 3 - Fault
I want to parse them into separate columns for each record. For example, for the first record above it would look like:
Rainier Beach (Henderson)
Tunnel Facility
C29_Facility
Offline - PLC 70
I can parse the first text value by using:
LEFT(alarmname,charindex('\',alarmname)-1)
which gives me:
Rainier Beach (Henderson)
But I don't know how to parse the second, third, fourth values...
Try this link... it has a great Function you can use. All you need to do is add the function and then you call split('/', fieldname) and it works like a charm.
http://www.webtechminer.com/split-function-in-sql-server-to-break-comma-separated-string/
It returns a table.