Error loading a local file to a BigQuery table - google-bigquery

Question

I'm trying to load a local file into BigQuery via the API, and it is failing. The file size is 98 MB and as a bit over 5 million rows. Note that I have loaded tables with the same number of rows and slightly bigger file size without problems in the past.
The code I am using is exactly the same as the one in the API documentation, which I have used successfully to upload several other tables. The error I get is the following:
Errors:
Line:2243530, Too few columns: expected 5 column(s) but got 3 column(s)
Too many errors encountered. Limit is: 0.
Job ID: job_6464fc24a4414ae285d1334de924f12d
Start Time: 9:38am, 7 Aug 2012
End Time: 9:38am, 7 Aug 2012
Destination Table: 387047224813:pos_dw_api.test
Source URI: uploaded file
Schema:
tbId: INTEGER
hdId: INTEGER
vtId: STRING
prId: INTEGER
pff: INTEGER
Note that the same file loads just fine from CloudStorage (dw_tests/TestCSV/test.csv), so the problem cannot be the one reported about one line having fewer columns, as it would fail from CloudStorage too, and I have also checked that all the rows have the correct format.
The following jobs have the same problem, the only difference is the table name and the name of the fields in the schema are different (but it is the same data file, fields and types). In those attempts it claimed a different row in trouble:
Line:4288253, Too few columns: expected 5 column(s) but got 4 column(s)
The jobs are the following:
job_cbe54015b5304785b874baafd9c7e82e load FAILURE 07 Aug 08:45:23 0:00:34
job_f634cbb0a26f4404b6d7b442b9fca39c load FAILURE 06 Aug 16:35:28 0:00:30
job_346fdf250ae44b618633ad505d793fd1 load FAILURE 06 Aug 16:30:13 0:00:34
The error that the Python script returns is the following:
{'status': '503', 'content-length': '177', 'expires': 'Fri, 01 Jan 1990 00:00:00 GMT', 'server': 'HTTP Upload Server Built on Jul 27 2012 15:58:36 (1343429916)', 'pragma': 'no-cache', 'cache-control': 'no-cache, no-store, must-revalidate', 'date': 'Tue, 07 Aug 2012 08:36:40 GMT', 'content-type': 'application/json'}
{
"error": {
"errors": [
{
"domain": "global",
"reason": "backendError",
"message": "Backend Error"
}
],
"code": 503,
"message": "Backend Error"
}
}
This looks like there may be an issue at BigQuery. How can I fix this problem?

Answer 1

The temporary files were still around for this import, so I was able to check out the file we tried to import. For job job_6464fc24a4414ae285d1334de924f12d, the last lines were:
222,320828,bot,2,0
222,320829,bot,4,3
222,320829,
It looks like we dropped part of the input file at some point... The input specification says that the MD5 hash should be 58eb7c2954ddfa96d109fa1c60663293 but our hash of the data is 297f958bcf94959eae49bee32cc3acdc, and file size should be 98921024, but we only have 83886080 bytes.
I'll look into why this is occurring. In the meantime, imports though Google Storage use a much simpler path and should be fine.