I have a Powershell script that builds a "az storage entity insert" command dynamically. Basically I have a CSV file that I use to create the content of a table by converting it to a long command it then invokes. It has worked fine until I added a field that contains a Regexp.
I started to get strange "The system cannot find the path specified." errors. Not from accessing the CSV as you would first suspect, but from running the command generated. I found out that some special characters in the field's value breaks the command and it tries to execute what comes after that as some separate command or something.
I made the expression simpler and found that not much characters work. As simple commands as this does not work:
az storage entity insert --table-name table --account-name $StorageAccountName --if-exists replace --connection-string $StorageConnectionString --entity PartitionKey=ABC RowKey=DEF Field="(abc)" Field#odata.type=Edm.String
This causes a different error "Field#odata.type was unexpected at this time."
Also | character causes problems, like:
az storage entity insert --table-name table --account-name $StorageAccountName --if-exists replace --connection-string $StorageConnectionString --entity PartitionKey=ABC RowKey=DEF Field="|abc" Field#odata.type=Edm.String
gives "'abc' is not recognized as an internal or external command, operable program or batch file.
This instead works fine:
az storage entity insert --table-name table --account-name $StorageAccountName --if-exists replace --connection-string $StorageConnectionString --entity PartitionKey=ABC RowKey=DEF Field="abc" Field#odata.type=Edm.String
So why do those special characters break the command and how can fix it? I need both of those characters for the regexp and some others too that won't work.
These errors happen both when I run directly from Powershell as well as in my script that uses Invoke-Expression
I initially thought this had to do with the way that PowerShell handles single quotation marks vs double quotation marks but it turns out that I was only half way there. Octopus Deploy lists several solutions including this with wrapped single quotes:
'"(abc)"'
Here are your original commands and then your commands with the wrapped single quotes around double quotes (where I now instead error out on failing to provide an account name):
Related
I have a dynamic sql file in which name of TBCREATOR changes as given in a parameter.
I use a simple python script to change the TBCREATOR=<variable here> and write the result to an output sql file.
calling this file using db2 -td# -vf <generated sql file>gives
DSNC105I : End of file reached while reading the command
Here is the file i need the TBCREATOR variable replaced:
CONNECT to 204.90.115.200:5040/DALLASC user *** using ****#
select REMARKS from sysibm.SYSCOLUMNS WHERE TBCREATOR='table' AND NAME='LCODE'
#
Here is the python script:
#!/usr/bin/python3
# #------replace table value with schema name
# print(list_of_lines)
fin = open("decrypt.sql", "rt")
#output file to write the result to
fout = open("decryptout.sql", "wt")
for line in fin:
fout.write(line.replace('table', 'ZXP214'))
fin.close()
fout.close()
After decryptout.sql is generated I call it using db2 -td# -vf decryptout.sql
and get the error given above.
Whats irritating is I have another sql file that contains exactly same data as decryptout.sql which runs smoothly with the db2 -td# -vf ... command. I tried to use the unix command cmp to compare the generated file and the one which I wrote, with the variable ZXP214 already replaced but there are no differences. What is causing this error?.
here is the file (that executes without error) I compare generated output with:
CONNECT to 204.90.115.200:5040/DALLASC user *** using ****#
select REMARKS from sysibm.SYSCOLUMNS WHERE TBCREATOR='ZXP214' AND NAME='LCODE'
#
I found that specifically on the https://ibmzxplore.influitive.com/ challenge, if you are using the java db2 command and working in the Zowe USS system (Unix System Services of zOS), there is a conflict of character sets. I believe the system will generally create files in EBCDIC format, whereas if you do
echo "CONNECT ..." > syscat.clp
the resulting file will be tagged as ISO8859-1 and will not be processed properly by db2. Instead, go to the USS interface and choose "create file", give it a folder and a name, and it will create the file untagged. You can use
ls -T
to see the tags. Then edit the file to give it the commands you need, and db2 will interoperate with it properly. Because you are creating the file with python, you may be running into similar issues. When you open the new file, use something like
open(input_file_name, mode=âwâ, encoding=âcp1047â)
This makes sure the file is open as an EBCDIC file.
If you are using the Db2-LUW CLP (command line processor) that is written in c/c++ and runs on windows/linux/unix, then your syntax for CONNECT is not valid.
Unfortunately your question is ambigiously tagged so we cannot tell which Db2-server platform you actually use.
For Db2-LUW with the c/c++ written classic db2 command, the syntax for a type-1 CONNECT statement does not allow a connection-string (or partial connection string) as shown in your question. For Db2-LUW db2 clp, the target database must be externally defined (i.e not inside the script) , either via the legacy actions of both catalog tcpip node... combined with catalog database..., or must be defined in the db2dsdriver.cfg configuration file as plain XML.
If you want to use connection-strings then you can use the clpplus tool which is available for some Db2-LUW client packages, and is present on currently supported Db2-LUW servers. This lets you use Oracle style scripting with Db2. Refer to the online documentation for details.
If you not using the c/c++ classic db2 command, and you are instead using the emulated clp written in java only available with Z/OS-USS, then you must open a ticket with IBM support for that component, as that is not a matter for stackoverflow.
Is there a way to mass update the TableStorage entities?
Say, I want to rename all the Clients having "York" in the City field to "New-York".
Is there some tools to do it directly (without the need to writing code)?
You could try to use Microsoft Azure Storage Explorer to achieve it.
First, you have some entities in table storage with a City field in your Storage Explorer.
Then you could click Export button to export all your entities to a .csv file.
Enter Ctrl + F and choose Replace item.
Fill the find and replace item with what you want then choose Replace All.
Finally, go back to the Storage Explorer and click Import button to choose the .csv file you have edited before.
I wanted to do the trick with export/import but it's a no go when you have millions of records. I exported all the records and ended up with ~5gb file. Azure Storage Explorer couldn't handle it (my pc i7, 32gb ram).
If someone is also struggling with similar issue, you can do as follow:
Export records to csv file
Remove the lines that you don't want to modify (if needed). You can use grep "i_want_this_phrase" myfile > mynewfile or use -v option to find all that doesn't match the given phrase. If file is too large, split it with some command eg. cat bigFile.csv | parallel --header : --pipe -N999 'cat >file_{#}.csv'
Remove everything except the RowKey column.
Prepare az cli command similar to az storage entity merge --connection-string 'XXX' --account-name your_storage -t your_table -e PartitionKey=your_pk MyColumn=false MyColumn#odata.type=Edm.Boolean RowKey=. Remember about odata.type. At first I did an update without this and instead of bools, I switched to strings. Luckily it was easy to fix.
Open the file in VSC, select all with ctrl+a, then shift+alt+i to put a cursor at the end of all lines and then paste previously prepared az cli command. This way you will get a list of az cli updates for each RowKey.
Add #!/bin/bash at the beginning of the file, save as .sh, modify privileges chmod +x yourfile and run.
Of course if you want, you can create some bash script for that and read a file line by line and execute az command. I just did it my way as it was much simpler for me, I'm not so experienced in bash, so it would take me a while to dev&test the script.
I am trying to export tables from BQ to CS using the command-line tool.
i followed the instructions here: https://cloud.google.com/bigquery/docs/exporting-data
I tried to run the example script:
extract --compression=GZIP 'bigquery-public-data:samples.shakespeare' gs://my_bucket/shakespeare.zip
I keep getting:
Error parsing command: Unknown command line flag 'compression'
Any idea anyone?
I just run below and it perfectly worked for me
bq extract --compression=GZIP bigquery-public-data:samples.shakespeare gs://my_bucket/shakespeare.zip
it also worked with double quotes around full qualified table name, but not with single quotes (as it is in link in your question)
I am trying to copy a file, export.txt from one directory to another within a .mshs script. I currently have:
shell copy 'E:\RPTG\Export.txt' 'E:\FCST\';
I'm getting an error that says "end of file breaks the statement."
Is there a command to copy a file with .mshs?
Thanks!
Typically you would perform the copy from a batch file itself. For example, your batch file would do the copy, run a MaxL script, and then do other things. That said, you can run shell commands from within MaxL if you need to (I don't usually recommend it though). In this case, you need to pass the whole statement to the shell command. Your statement should work if you write it like this instead:
shell "copy 'E:\RPTG\Export.txt' 'E:\FCST\'";
Note that I have enclosed your command in double quotes. There are some nuances to using double quotes and single quotes at the same time, but in this case you should be okay.
When developing Pig scripts that use the STORE command I have to delete the output directory for every run or the script stops and offers:
2012-06-19 19:22:49,680 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 6000: Output Location Validation Failed for: 'hdfs://[server]/user/[user]/foo/bar More info to follow:
Output directory hdfs://[server]/user/[user]/foo/bar already exists
So I'm searching for an in-Pig solution to automatically remove the directory, also one that doesn't choke if the directory is non-existent at call time.
In the Pig Latin Reference I found the shell command invoker fs. Unfortunately the Pig script breaks whenever anything produces an error. So I can't use
fs -rmr foo/bar
(i. e. remove recursively) since it breaks if the directory doesn't exist. For a moment I thought I may use
fs -test -e foo/bar
which is a test and shouldn't break or so I thought. However, Pig again interpretes test's return code on a non-existing directory as a failure code and breaks.
There is a JIRA ticket for the Pig project addressing my problem and suggesting an optional parameter OVERWRITE or FORCE_WRITE for the STORE command. Anyway, I'm using Pig 0.8.1 out of necessity and there is no such parameter.
At last I found a solution on grokbase. Since finding the solution took too long I will reproduce it here and add to it.
Suppose you want to store your output using the statement
STORE Relation INTO 'foo/bar';
Then, in order to delete the directory, you can call at the start of the script
rmf foo/bar
No ";" or quotations required since it is a shell command.
I cannot reproduce it now but at some point in time I got an error message (something about missing files) where I can only assume that rmf interfered with map/reduce. So I recommend putting the call before any relation declaration. After SETs, REGISTERs and defaults should be fine.
Example:
SET mapred.fairscheduler.pool 'inhouse';
REGISTER /usr/lib/pig/contrib/piggybank/java/piggybank.jar;
%default name 'foobar'
rmf foo/bar
Rel = LOAD 'something.tsv';
STORE Rel INTO 'foo/bar';
Once you use the fs command, there a lot of ways to do this. For an individual file, I wound up adding this to the beginning of my scripts:
-- Delete file (won't work for output, which will be a directory
-- but will work for a file that gets copied or moved during the
-- the script.)
fs -touchz top_100
rm top_100
For a directory
-- Delete dir
fs -rm -r out