When I run a query in Biquery using the command line bq query and indicating a text_file.sql containing the query, I got this error. The query was working using the bigquery console (https://console.cloud.google.com/bigquery?xxxxxxxx). I chose UTF-8.
How should I handle this syntax error?
Please check file format is correct or not. because this issue is due file type other than UTF-8 LF in my case.
when i was change file format to UTF-8 it was running file for me.
Related
We're seeing BigQuery produce invalid utf-8 errors when the " - " (dash) character is used in pipe delimited csv files. The weird thing is, these characters are in files that are over a year old, have not changed, and BigQuery has been reading the files for many months just fine until a few days ago. Here's an example of one of the errors.
Christus Trinity Clinic \\x96 Rheumatology is not a valid UTF-8 string
The way the string looks in the original file is like this:
Christus Trinity Clinic – Rheumatology
Does anyone know the fix for this or if BigQuery has changed it's functionality in a way that might cause this issue? I know that I can just upload a corrected file, but in this scenario the files are not supposed to change for auditing purposes.
I had the same issue from aug 14.
I am using gsutil to load the csv into bigquery.
I had used the encoding option while loading the csv and it is working for me.
Encoding:
--encoding ISO-8859-1
Command line:
bq --location=US load --skip_leading_rows=1 --encoding ISO-8859-1 --replace --source_format=CSV gcs.dim_employee
We saw the same thing suddenly happen since yesterday.
For me, the solution was to add a encoding type to the loadconfig.
(I'm using the PHP client, but your client probably also has this option)
$loadConfig->encoding('ISO-8859-1');
In a pre-build event, a batch file is executed to combine multiple SQL files into a single one.
It is done using this command :
COPY %#ProjectDir%\Migrations\*.sql %#ProjectDir%ContinuousDeployment\AllFilesMergedTogether.sql
Everything appear to work fine but somehow the result give an incorrect syntaxe error.
After two hours of investigation, it turn out the issue is caused by an invisible character that remain invisible even with notepad++.
Using an online website, the character has been spotted and is U+FEFF has shown in following image.
Here are the two input scripts.
PRINT 'Script1'
PRINT 'Script2'
Here is the output given by the copy command.
PRINT 'Script1'
PRINT 'Script2'
Additional info :
Batch file is encoded with UTF-8
Input files are encoded with UTF-8-BOM
Output file is encoded with UTF-8-BOM.
I'm not sure it is possible to change the encoding output of command copy.
I've tried and failed.
What should be done to eradicate this extremely frustrating parasitic character?
It has turned out that changing encoding of input files to ANSI does fix the issue.
No more pesky character(s).
Also, doing so does change the encoding of the result file to UTF-8 instead of UTF-8-BOM which is great I believe.
Encoding can be changed using Notepad++ as show in following picture.
I'm having an issue matching regular expression in BigQuery. I have the following line of code that tries to identify user agents:
when regexp_contains((cs_user_agent), '^AppleCoreMedia\/1\.(.*)iPod') then "iOS App - iPod"
However, BigQuery doesn't seem to like escape sequences for some reason and I get this error that I can't figure out:
Syntax error: Illegal escape sequence: \/ at [4:63]
This code works fine in a regex validator I use, but BigQuery is unhappy with it and I can't figure out why. Thanks in advance for the help
Use regexp_contains((cs_user_agent), r'^AppleCoreMedia\/1\.(.*)iPod')
Select * from mytable where field=
'ce7bd3d4-dbdd-407e-a3c3-ce093a65abc9;cdb597073;7cf6cda5fc'
Getting Below Error while running above query in Hive
FAILED: ParseException line 1:92 character '' not supported here
<EOF> here means End Of File. When you get an "unexpected End Of File" error it means the parser reached the end of the query unexpectedly. This typically happens when the parser is expecting to find a closing character, such as when you have started a string with ' or " but have not closed the string (with the closing ' or ").
When you come across these types of errors it is good to check that your query can be parsed correctly. In addition, the error gives you the location where the parser failed: line 1:92 in this case. You can usually look at this location (character 92 of the query) and work backwards to find the problem character.
Try adding the database name to the "from" statement as below.
Select * from my_db_name.mytable where field= 'ce7bd3d4-dbdd-407e-a3c3-
ce093a65abc9;cdb597073;7cf6cda5fc';
Hive uses the default database when no database was previously specified.
On command line
convert(varchar,getdate(),120)
gives below error:
Unknown argument '04:59:42.xml'
I am saving data to an xml file.
when I use below command,proc works perfectly.
convert(varchar,getdate(),112)
I need the file to save with time.
You are trying to create a file with a colon in the name, which is not allowed. No way to get around that restriction. However, you can replace the colons with other characters when naming the file, e.g.
replace(convert(varchar,getdate(),120),':','')