sqlcmd - How to get around column length limit without empty spaces? - sql

I'm trying to use sqlcmd on a windows machine running SQL Server 2005 to write a query to a csv file. The command line options we typically use are:
-l 60 -t 300 -r 1 -b -W -h -1
However, the columns are getting truncated at 256 bytes. In an attempt to circumvent this, I tried using this command line option in place of -W:
-y 8000
This captures the entire fields, but the problem with this method is that the file balloons up from just over 1mb to about 200mb due to all the extra space (I realize 8000 is probably overkill, but it will probably have to be at least 4000 and I'm currently only working with a small subset of data). The -W option typically eliminates all this extra space, but when I try to use them together it tells me they're mutually exclusive.
Is there a way to get sqlcmd around this limit, or does anyone know if another program (such as bcp or osql) would make this easier?
Edit:
Here are the code snippets we're using to get the field that's being truncated (similar code is used for a bunch of fields):
SELECT ALIASES.AliasList as complianceAliases,
...
LEFT OUTER JOIN (Select M1.ID, M1.LIST_ID,stuff((SELECT '{|}' + isnull(Content2,'')+' '+isnull(Content3,'')+' '+isnull(Content4,'')+' '+isnull(Content5,'')+' '+isnull(Content6,'')+' '+isnull(Content7,'')
FROM fs_HOST3_TEST_web.ISI_APP_COMP_MULTI M2 with (nolock)
WHERE M1.LIST_ID = M2.LIST_ID and M1.ID = M2.ID and M1.TYPE = M2.TYPE
FOR XML PATH('')
),1,1,'') as AliasList
FROM fs_HOST3_TEST_web.ISI_APP_COMP_MULTI M1 with (nolock)
WHERE M1.LIST_ID = 2001 AND M1.TYPE = 'Aliases'
GROUP BY m1.list_id,m1.ID,m1.Type) as ALIASES
ON ALIASES.LIST_ID = PAIR.COMP_LIST_ID AND ALIASES.ID = PAIR.COMP_ID

I ended up solving this by using the "-y0" argument. It still left a bunch of whitespace but it looks like it only went to the end of the longest piece of data in each field.
I then ran the output through a program that removed repeating spaces and that solved all of the problems.

Related

In MS Access VBA get Number of Processor Cores

I need to get the number of processor cores available on a computer programmatically from within MS Access. As an example, the computer I work from most frequently has one processor with 6 cores. I want to grab the number '6' through VBA.
Thus far, I have found two ways to find this information through CMD. (1) I can execute the line echo %NUMBER OF PROCESSORS% and the result is 6 (simple and clean, I like it). (2) I have also tried wmic cpu get numberorcores, but the result of that prompt is as follows:
NumberOfCores
6
I intend to pipe the output to and read from the clipboard. The reason I use the clipboard is to avoid creating, reading, and deleting little text files of data. Prompt (2) works, I can successfully pipe the output to the clipboard and read it into a variable in VBA, but it's messy and I would have to parse the result to get the information I need. I would much prefer using prompt (1), but it's not working and the problem seems to be echo. I have tried using shell() and CreateObject(WScript.Shell).Run without success. The strings I have used to try to execute the echo prompt are as follows:
str = "echo %NUMBER OF PROCESSORS% | clip"
str = "cmd ""echo %NUMBER OF PROCESSORS% | clip"""
So, is there a way to successfully send an echo prompt to CMD through VBA and get a result?
Alternatively, is there a different way in VBA to get the number of cores?
TIA!
Why not keep it simple like this:
Dim result As Variant
result = Environ("NUMBER_OF_PROCESSORS")
Debug.Print "Number of processors is " & result

How to split a large query in command line?

I installed oracle db version 19c in my docker environment and set up a database filled with dummy data. However, when I try to run a very large query I get the error:
SP2-0341: line overflow during variable substitution (>3000 characters at line 1).
I tried splitting it up with linebreaks but depending on how I split it I get all kinds of errors like:
ERROR at line 2: ORA-00933: SQL command not properly ended
or
ERROR at line 2:
SP2-0341: line overflow during variable substitution (>3000 characters at line 3)
The query is formatted as
SELECT AA.n_name AS AA_n_name, AA.n_nationkey AS ...
FROM nation AS AA FULL OUTER JOIN supplier...
WHERE (AC.p_partkey = ... AND...) OR((AC.p_partkey = ...)); -- The where part is over 5000 characters long--
Is there an alternative or solution to tackling this in the command line? I tried running the query as a sql file as well and hit a 4999 limit. I am on a Ubuntu server if that would help and any assistance would be appreciated.
It depends on the environment that you're working in, but generally you are able to continue a command onto the next line by ending the line with a 'back slash' \.

Troubleshooting BCP and Format File Errors

First off, sorry for the long post. I wanted to be thorough with my examples/data, and the bulk of this post is just that.
I inherited a Bulk Import Process using a format file (.fmt) at my new job. This process was created by the guy that worked here before me, and it is my job to learn this process (and fix it now). I have limited knowledge of this stuff, but I have done some research. After a few weeks, I haven't really gotten anywhere. Here is what I am working with...
--BCP Command to import data from C:\Desktop\20180629_2377167_PR_NP.txt to table LA_Temp.dbo.ProvReg
bcp LA_Temp.dbo.ProvReg IN C:\Desktop\20180629_2377167_PR_NP.txt -f C:\Desktop\PROVREG.FMT -T -S SERVERNAME -k -m 1000000
--Table Structure which format file is created from:
SELECT [NPI]
,[D1]
,[EntityType]
,[D2]
,[ReplaceNPI]
,[D3]
,[ProvName]
,[D4]
,[MailAddr1]
,[D5]
,[MailAddr2]
,[D6]
,[MailCity]
,[D7]
,[MailState]
,[D8]
,[MailZip]
,[D9]
,[MailCountry]
,[D10]
,[MailPhone]
,[D11]
,[MailFax]
,[D12]
,[LocAddr1]
,[D13]
,[LocAddr2]
,[D14]
,[LocCity]
,[D15]
,[LocState]
,[D16]
,[LocZip]
,[D17]
,[LocCountry]
,[D18]
,[LocPhone]
,[D19]
,[LocFax]
,[D20]
,[Taxonomy1]
,[D21]
,[Taxonomy2]
,[D22]
,[Taxonomy3]
,[D23]
,[OtherProvID]
,[D24]
,[OtherProvIDType]
,[D25]
,[ProvEnumDate]
,[D26]
,[LastUpdate]
,[D27]
,[DeactivateRC]
,[D28]
,[DeactivateDate]
,[D29]
,[ReactivateDate]
,[D30]
,[Gender]
,[D31]
,[License]
,[D32]
,[LicenseState]
,[D33]
,[AuthorizedContact]
,[D34]
,[ContactTitle]
,[D35]
,[ContactPhone]
,[D36]
,[PanelOpen]
,[D37]
,[Language1]
,[D38]
,[Language2]
,[D39]
,[Language3]
,[D40]
,[Language4]
,[D41]
,[Language5]
,[D42]
,[AgeRestrict]
,[D43]
,[PCPMax]
,[D44]
,[PCPActual]
,[D45]
,[PCPAll]
,[D46]
,[EnrollInd]
,[D47]
,[EnrollDate]
,[D48]
,[FamilyOnly]
,[D49]
,[SubSpec1]
,[D50]
,[SubSpec2]
,[D51]
,[SubSpec3]
,[D52]
,[ContractName]
,[D53]
,[ContractBegin]
,[D54]
,[ContractEnd]
,[D55]
,[Parish1]
,[D56]
,[Parish2]
,[D57]
,[Parish3]
,[D58]
,[Parish4]
,[D59]
,[Parish5]
,[D60]
,[Parish6]
,[D61]
,[Parish7]
,[D62]
,[Parish8]
,[D63]
,[Parish9]
,[D64]
,[Parish10]
,[D65]
,[Parish11]
,[D66]
,[Parish12]
,[D67]
,[Parish13]
,[D68]
,[Parish14]
,[D69]
,[Parish15]
,[D70]
,[PCPInd]
,[D71]
,[DisplayOnline]
,[D72]
,[ExpAgeRestrict]
,[D73]
,[Suffix]
,[D74]
,[Title]
,[D75]
,[PrescriberInd]
,[Spaces]
,[End]
FROM [LA_Temp].[dbo].[ProvReg]
--Example Text File Data (this is one line)
9999999999 ^0^ ^ ^3800 HMA BLVD STE 305 ^ ^METAIRIE ^LA^70006 ^ ^5048729679^ ^3800 HMA BLVD ^ ^METAIRIE ^LA^70006 ^ ^9999999999^ ^207Q00000X^ ^ ^0000000^2001^ ^00000000^ ^00000000^00000000^F^ ^LA^ ^ ^ ^N^1^0^0^0^0^2^00000^00000^00000^ ^ ^ ^ ^ ^ ^000000000000000000000000000000^00000000^00000000^26^00^00^00^00^00^00^00^00^00^00^00^00^00^00^0^0^Accept patients of age 000-000^ ^MD ^ ^
--Format file
11.0
153
1 SQLCHAR 0 40 "\t" 1 NPI SQL_Latin1_General_Pref_CP1_CI_AS
2 SQLCHAR 0 2 "\t" 2 D1 SQL_Latin1_General_Pref_CP1_CI_AS
3 SQLCHAR 0 2 "\t" 3 EntityType
...all the way to...
153 SQLCHAR 0 2 "\r\n" 153 End
I have changed directories, servername, and some of the text file data to maintain security, however, it is very similar.
Here is the problem I am encountering:
With the "\t" used in the format file I just created from the SQL table, I get the error: [Microsoft][SQL Server Native Client 11.0]Unexpected EOF encountered in BCP data-file.
If I change this to just "" or "^" (as I 'think' it should be since the text file is using carrot delimiter), the rows began to copy with error
[Microsoft][SQL Server Native Client 11.0]String data, right truncation SQLState = 22001, NativeError = 0. BCP copy in failed.
If anyone can please point me in the right direction here for troubleshooting this issue, or if you see anything out of place, please let me know. As I mentioned, I have been at this for some time, and can use any suggestions I can get. Unfortunately, there is no one at my company I can ask about this.
try adding the -e option to your bcp command. this will give you an error file in which BCP will write some samlpe lines from the file that it had problems with. Very helpful with troubleshooting the type of error you are getting now (you are correct to change your delimiter in the format file).
The error you are getting now "string data" and "truncation" is just as it states. However, this truncation can be occurring for a number of reasons. The destination table's columns may not be large enough to hold the data that is contained between the defined field delimiters. There may be delimiters appearing in your data and so this could be tricking the bcp utility into thinking a column has ended before it was intended to end in the file (this is less likely with the delimiter you are using... but ya never know... I always prefer fixed width if possible.). And, of course, the source of the data may very well have written you a file that contradict whatever agreed upon spec led you to define your destination as you have.
The error is accurate, teh trick is finding where. Use the -e option to allow BCP to capture problematic lines:
BCP table_dest IN "C:\FILE.TXT" -S SVR -T -f"C:\FORMAT_FILE.txt" -e"C:\ERROR_FILE.txt"
The "error_file.txt" will include line numbers and will include a sample of lines that it couldn't handle. Just copy and past to find in the file youare trying to load to see for yourself.
Strongly suggest using a more advanced text editing tool. Do not use windows notepad or wordpad. Use something like notepad++ or ultraedit to inspect ascii text files.

How to skip errors when using json_extract_path_text in Redshift?

I have the following query:
SELECT 'curl -s http://www.mde.operator.com/MRE/api?profile=CANCEL_AUTH&mode=assync-oneway&Auth='||json_extract_path_text(external_reference_id,'transactionIdAuth') + '&NUM=' + phone FROM dbo.cancelled WHERE id like '%Auth%';
It will bring more than 60 thousands results, but the json is broken and I cannot manage to delete the broken lines.
Is there any way to skip the rows which shows any kind of errors?
Note: It isn't null rows.
I've already try:
json_extract_path_text(regexp_replace(event_properties,'\\\\.',''),'someValue')
You can make them null rows though, by setting the null_if_invalid argument of the json_extract_path_text function to true.
Source: https://docs.aws.amazon.com/redshift/latest/dg/JSON_EXTRACT_PATH_TEXT.html

Why does database query using sqlcmd produce just empty file while same query works in Management Studio?

I have made recently a huge MSSQL script which was working until the server environment changed and some queries are not anymore allowed. So I had to make a .bat file which executes this query from command line.
I get no error or something else. I just get a file with no entry. But I receive a lot of entries if I use the code of the query in the Management Studio.
Does somebody see where is the mistake in my command line?
I inserted some new lines for reading the code better. Everything except command PAUSE is on one line in the batch file.
EDIT: I figured out that the problem is in the last WHERE clause in the LIKE operater. If I take out the LIKE operater it works. It has nothing todo with the % caracter. it is effective the LIKE operater. Does anybody know how to fix that?
sqlcmd -S connection\string -U user -P password -d dbName -s";" -Q "SET NOCOUNT ON;
SELECT [per_nummer] as EmployeeID,
[per_id] as System_nr,
[per_pid] as PID,
[per_anrede] as Gender,
[per_vname] as Vorname,
[per_name] as Name,
[per_telEx] as Telefon,
[per_email] as Email,
[per_instradierungHauptort] as Instradierung,
[per_gebnr] as Gebaeudenummer,
(SELECT mobileTelephoneNumber FROM [dbName].[dbo].[import_zuko_GLDAP_DUMP] WHERE UserID = per_pid ) as Mobile,
[per_business_area] as Business_Area,
(SELECT csgdivision FROM [dbName].[dbo].[import_zuko_GLDAP_DUMP] WHERE UserID = per_pid ) as Division,
NULL as Bereich,
(SELECT csgCompany FROM [dbName].[dbo].[import_zuko_GLDAP_DUMP] WHERE UserID = per_pid ) as Firma,
[per_sprache] as Korespondenzsprache,
(SELECT roomnumber FROM [dbName].[dbo].[import_zuko_GLDAP_DUMP] WHERE UserID = per_pid ) as Bueronummer,
[per_floor] as Etage,
(SELECT convert(varchar, convert(date,[per_eintrittsdatum]), 104)) as Eintritt_per,
(SELECT convert(varchar, convert(date,[per_austrittsdatum]), 104)) as Austritt_per,
CONVERT(VARCHAR(10), GETDATE(), 104) AS letzte_mutation,
per_lm as Linemanager,
(SELECT TOP 1 per_pid FROM [dbName].[dbo].[person] WHERE per_nummer = master_table.per_lm) AS lm_pid,
(SELECT convert(varchar, convert(date,[per_lm_von]), 104)) as lm_von,
(SELECT convert(varchar, convert(date,[per_lm_bis]), 104)) as lm_bis,
NULL FROM [dbName].[dbo].[person] as master_table
WHERE
[per_nummer] IS NOT NULL AND per_pidStatus = 'A' AND
([per_pid] LIKE('A%')OR [per_pid] LIKE('F%')OR [per_pid] LIKE('W%'))"
-w 1000 -W -o "\\servername\G$\path\to\file.csv"
PAUSE
Different default options
Do you have read carefully sqlcmd Utility documentation by Microsoft?
There is at top the important information:
Because different default options may apply, you might see different behavior when you execute the same query in SQL Server Management Studio in SQLCMD Mode and in the sqlcmd utility.
Therefore the empty file could be caused by different default options.
Space between option and value of option
Further the documentation contains at top:
Currently, sqlcmd does not require a space between the command line option and the value. However, in a future release, a space may be required between the command line option and the value.
Double quotes inside an argument string are often problematic, see answer on Why double quotes should be always only at beginning and end of an argument string?
Therefore I suggest to follow the advice of Microsoft and change -s";" to -s ";" with a space charcter after -s as done for all other options, too.
Escaping percentage sign
Do you have tried to escape each % in the query string with an additional % and using therefore 3 times %% in the query string?
Command line interpreter cmd.exe could interpret the string between two % as a reference to an environment variable and as there is no such environment variable, removes everything between two % from the command line.
Update: It turned out that indeed the not escaped percentage signs resulted in a wrong query string and therefore in an empty results file.
Command with path and extension
General hint:
It is always advisable to specify in batch files commands like sqlcmd with full name which means with path and file extension as this makes the execution of the application independent on the values of the environment variables PATH and PATHEXT.