I get an "unexpected error" message every time I try to create a table on BigQuery. I've tried inputting and omitting the schema and made sure that the file was compatible with csv format.
I was having the same issue.
I did a couple of things to clean up the csv file and I was then able to upload it successfully
Remove commas from any numerical data
If using auto detect for schema remove any spaces between headings
Remove header data completely and define schema
I HAD THE SAME PROBLEM AND I WRITE DATA TYPES IN CAPITAL.
and my problem of "unexpected error was gone"
FOR EX- TYPE (STRING) NOT (string)
TYPE (INTEGER) NOT (integer) same on.......
Related
I want to create a Table for my dataset in BigQuery. I want to upload CSV file. When I upload it and clicked on "create table" it is saying:
unexpected error. Tracking number c986854671035387
What is this error and How can I solve this? (I also upgraded my BigQuery to 90 days free trial).
You need to check the data inside CSV. If it has column names and no faulty records.
you can download any sample CSV file from here and try
http://www.mytrapture.com/sampledata/
I'm attempting to upload a CSV file (which is an output from a BCP command) to BigQuery using the gcloud CLI BQ Load command. I have already uploaded a custom schema file. (was having major issues with Autodetect).
One resource suggested this could be a datatype mismatch. However, the table from the SQL DB lists the column as a decimal, so in my schema file I have listed it as FLOAT since decimal is not a supported data type.
I couldn't find any documentation for what the error means and what I can do to resolve it.
What does this error mean? It means, in this context, a value is REQUIRED for a given column index and one was not found. (By the way, columns are usually 0 indexed, meaning a fault at column index 8 is most likely referring to column number 9)
This can be caused by myriad of different issues, of which I experienced two.
Incorrectly categorizing NULL columns as NOT NULL. After exporting the schema, in JSON, from SSMS, I needed to clean it
up for BQ and in doing so I assigned IS_NULLABLE:NO to
MODE:NULLABLE and IS_NULLABLE:YES to MODE:REQUIRED. These
values should've been reversed. This caused the error because there
were NULL columns where BQ expected a REQUIRED value.
Using the wrong delimiter The file I was outputting was not only comma-delimited but also tab-delimited. I was only able to validate this by using the Get Data tool in Excel and importing the data that way, after which I saw the error for tabs inside the cells.
After outputting with a pipe ( | ) delimiter, I was finally able to successfully load the file into BigQuery without any errors.
I am trying to import data from a CSV file into a table.
I am presented by this error:
"Error 0xc02020a1: Data Flow Task 1: Data conversion failed. The data
conversion for column "dstSupport" returned status value 2 and status
text "The value could not be converted because of a potential loss of
data."."
However, I do not even need to convert this column, as you see in the image below:
The column is of type bit, and I used DST_BOOL.
This ended up being a parsing error. I had a string with commas within it, and these strings within the commas were being placed in my bit column. I fixed it by changing the delimiter from a comma to a pipe
My requirement is to pull the data from Different sources(Facebook,youtube, double click search etc) and load into BigQuery. When I try to pull the data, in some of the sources I was getting "NULL" when the column is empty.
I tried to load the same data to BigQuery and BigQuery is treating as a string instead of NULL(empty).
Right now replacing ""(empty string) where NULL is there before loading into BigQuery. Instead of doing this is there any way to load the file directly without any manipulations(replacing).
Thanks,
What is the file format of source file e.g. CSV, New Line Delimited JSON, Avro etc?
The reason is CSV treats an empty string as a null and the NULL is a string value. So, if you don't want to manipulate the data before loading you should save the files in NLD Json format.
As you mentioned that you are pulling data from Social Media platforms, I assume you are using their REST API and as a result it will be possible for you to save that data in NLD Json instead of CSV.
Answer to your question is there a way we can load this from web console?:
Yes, Go to your bigquery project console https://bigquery.cloud.google.com/ and create table in a dataset where you can specify the source file and table schema details.
From Comment section (for the convenience of other viewers):
Is there any option in bq commands for this?
Try this:
bq load --format=csv --skip_leading_rows=1 --null_marker="NULL" yourProject:yourDataset.yourTable ~/path/to/file/x.csv Col1:string,Col2:string,Col2:integer,Col3:string
You may consider running a command similar to: bq load --field_delimiter="\t" --null_marker="\N" --quote="" \
PROJECT:DATASET.tableName gs://bucket/data.csv.gz table_schema.json
More details can be gathered from the replies to the "Best Practice to migrate data from MySQL to BigQuery" question.
I have a CSV with 225 rows and see BigQuery expects the schema to be
column1_name:data_type,
I have removed all spaces however BigQuery doesn't like my schema it returns "Parsing Error" and returns the first field name.
my pasted schema looks like this (partial)
transaction_status:STRING(6),dollarsobligated:NUMERIC(10,2),baseandexercisedoptionsvalue:NUMERIC(10,2),baseandalloptionsvalue:NUMERIC(12,2),maj_agency_cat:STRING(35),mod_agency:STRING(37),maj_fund_agency_cat:STRING(35),contractingofficeagencyid:STRING(37),contractingofficeid:STRING(51),
Try removing the dimensioning, not needed. Declaring "String" is optional, as it's the default. Instead of numeric, do "float".
So
transaction_status:STRING(6),dollarsobligated:NUMERIC(10,2),baseandexercisedoptionsvalue:NUMERIC(10,2),baseandalloptionsvalue:NUMERIC(12,2),maj_agency_cat:STRING(35),mod_agency:STRING(37),maj_fund_agency_cat:STRING(35),contractingofficeagencyid:STRING(37),contractingofficeid:STRING(51),
should be
transaction_status,dollarsobligated:float,baseandexercisedoptionsvalue:float,baseandalloptionsvalue:float,maj_agency_cat,mod_agency,maj_fund_agency_cat,contractingofficeagencyid,contractingofficeid