value = self.bq_client.import_data_from_dataframe(table=table_id, dataframe=dataframe,
source_format=bigquery.SourceFormat.CSV,
autodetect=True,
write_disposition=operation_type,
allow_quoted_newlines=True)
The code I am using right now is the above but it throws an exception as I mentioned when I try to append data into my BQ table. There are no nulls in the data to be appended.
Related
I'm trying to upload a .csv file to Postgres and I'm getting this error:
invalid input syntax for type double precision: " chargebackvalue"
image error
Here it is the structure of the table:
table structure here
The code of the .csv file:
stoneid; mundipaggid; cardnumber; emblem; chargebackvalue; cardmask; chargebackdate; emitter; description; purchasedate; clientName; tacomorderid; useremail
0155477; 'or_3E2W0X5s5jtPjWYO';0670000546857; 'Visa'; 60.6; '498453******3271'; '2019-10-17'; 'Banco do Brasil S.A.'; 'Teste'; '2019-10-10'; 'Silvana Teixeira Da Silva';99854; 'teste#teste.com'
This is too long for a comment.
I would recommend loading the data into a staging table, where all the columns are strings.
Then, select from that table to load the final table. This makes it easier to track down problems in the data that might occur during the load.
Clearly the row you have shown is not the cause of the error. Or, if this is the entire file, then you simply have not skipped the first line because it has header names rather than values.
I'm new to Dataflow. I've got a predefined-schema containing a non-repeated RECORD field called "device":
device.configId: STRING
device.version: STRING
Using a ParDo transform, I tried inserting a TableRow with this kind of field, as follows:
TableRow row = new TableRow();
row.put("field1", "val1");
TableRow device = new TableRow();
device.put("configId", "conf1");
device.put("version", "1.2.3");
row.put("device", device);
out.output(row);
I logged the table row, it looks like this:
{field1=val1, device={configId=conf1, version=1.2.3}}
I output it to a standard transform: BigQueryIO.write()
But the latter issues an error:
java.lang.RuntimeException: java.io.IOException:
Insert failed: [{"errors":[{
"debugInfo":"",
"location":"device.configid",
"message":"This field is not a record.",
"reason":"invalid"
}],"index":0}]
Not sure why, but note the location spells "configid" in lowecase - not in camel case as in the original log.
Any ideas on how to insert such an object to BigQuery?
Found out the problem. Apparently, this error message was caused only when the "configId" field was set to null rather than "conf1". To be exact, it was implicitly set to JSONObject.NULL coming from some input object.
My current table in BigQuery has a column that uses complex types. The "family" column is actually a list ("repeated" feature) of records (with 2 fields: id & name).
When I try to get the 1st "id" value of 1 row with the following syntax:
FieldValueList c = qr.getValues().iterator().next();
c.get("family").getRepeatedValue().get(0).getRecordValue().get("id");
I get the exception:
Method threw 'java.lang.UnsupportedOperationException' exception.
Retrieving field value by name is not supported when there is no fields schema provided
This is a bit annoying because my table has a clearly defined schema. And when I do the "read" query with the same Java call, I can also see that this schema is correctly found:
qr.getSchema().getFields().get("family").getSubFields().toString();
-->
[Field{name=id, type=INTEGER, mode=NULLABLE, description=null}, Field{name=name, type=STRING, mode=NULLABLE, description=null}]
Due to this exception, the workaround that I have found is to pass the "index" of the record field instead of giving it its name
c.get("family").getRepeatedValue().get(0).getRecordValue().get(0).getLongValue();
However, this seeks awkward to pass an index instead of a name.
Is there a better way to get the value of a field in a record inside an array (if my column is only a record, without array, then I don't get the exception) ?
Is this exception normal?
You can wrap the unnamed FieldValueList with a named one using the "of" static method:
FieldList subSchema = qr.getSchema().getFields().get("family").getSubFields();
FieldValueList c = qr.getValues().iterator().next();
FieldValueList.of(
c.get("family").getRepeatedValue().get(0).getRecordValue(),
subSchema).get("id");
The "of" method takes a FieldValueList (returned by getRecordValue() in this case) and a FieldList (subSchema here), and returns the same FieldValueList but with named access.
I am getting this error:
String was not recognized as a valid Boolean.Couldn't store <No> in meetsstd Column. Expected type is Boolean
When I am running this query:
SELECT * FROM work_nylustis_2013_q3.nylustis_details WHERE siteid = 'NYLUSTIS-155718' LIMIT 50
From this code:
Adapter.SelectCommand = New NpgsqlCommand(SQL, MLConnect)
Adapter.Fill(subDT) ' This line throws error
The meetsstd field is a varchar(3) and it does store either a 'Yes' or a 'No' value. How is this getting this confused with a boolean - a varchar should not care whether is holds 'Yes', or 'Si', or 'Oui'? And it only happens on 27 records out of the 28,000 in the table.
I usually blame npgsql for this kind of strangeness, but the last entry in the stack trace is: System.Data.DataColumn.set_Item(Int32 record, Object value)
Any clues?
Thanks!
Brad
To check if it is problem with database or with driver you can reduce problem to one row and column using your current environment:
SELECT meetsstd FROM work_nylustis_2013_q3.nylustis_details WHERE sitenum=1
(of course you must change sitenum into primary key)
Then try such query using psql, pgAdmin or some JDBC/ODBC based general editor.
If psql shows such record which raises error with your Npgsql based application then problem is with Npgsql driver or problem is with displaying query results.
If other tools shows such strange errors then problem is with your data.
Have you changed type of meetsstd field? Mayby you try to show it on some grid and this grid used Boolean field which was converted to Yes/No for displaying?
I get the error : OIP-04908: This operation is not permitted on a Null LOB when
Set MyClOB_0 = lOraDynaset_0.Fields("FILE_BODY").Value
lOraDynaset_0.Edit
amount_written = MyClOB_0.Write(buffer, chunksize, ORALOB_FIRST_PIECE)
I've just had the same problem when trying to insert into an Oracle CLOB field.
I got around it by:
Setting the Oracle field to 'Empty' (the VB6 keyword Empty - e.g. MyClOB_0.Value = Empty)
Performing the insert (e.g. lOraDynaset_0.Update)
Editing the dynaset record (e.g. lOraDynaset_0.Edit)
Setting the Clob field value to what you want it
Performing the update (e.g. lOraDynaset_0.Update)
Hope this helps (sorry if its not clear)