I am writing data into nested array BQ table(array name inside the table is -merchant_array)using my dataflow template.
Sometime its running fine and loading the data but sometime its giving me that error at run time.
java.lang.RuntimeException: org.apache.beam.sdk.util.UserCodeException: com.fasterxml.jackson.databind.JsonMappingException: Null key for a Map not allowed in JSON (use a converting NullKeySerializer?) (through reference chain: com.google.api.services.bigquery.model.TableRow["null"])
"message" : "Error while reading data, error message: JSON parsing error in row starting at position 223615: Only optional fields can be set to NULL. Field: merchant_array; Value: NULL",
Anyone has any idea why I am getting this error.
Thanks in advance.
here I got the issue that was causing error so I am posting my own question's answer,it might be helpful for anyone.
So the error was like-
Only optional fields can be set to NULL. Field: merchant_array; Value: NULL",
And here merchant_array is defined as an array that contains record (repetitive) data.
As per google doc the the array can not be-
ARRAYs cannot be NULL.
NULL ARRAY elements cannot persist to a table.
At the same time I was using arraylist in my code, that allows null values. So before making a record type data in code or setting the data in arraylist, just remove the NULL tablerows if exist.
hope this will helpful.
Related
DATA: t_mara type STANDARD TABLE OF mara WITH EMPTY KEY.
DATA(t_data1) = VALUE ty_data( FOR s_mara IN t_mara ( s_mara–matnr ) ).
I am trying to implement a similar code using FOR statement but I am getting an error that the field is unknown in the work area even though it would be declared inline.
Can you please let me know what went wrong? This is my first time I am facing this error on FOR loop.
Not sure because you are not providing too much detail but try this:
DATA t_mara type STANDARD TABLE OF mara WITH EMPTY KEY.
DATA(t_data1) = VALUE ty_data( FOR s_mara IN t_mara ( matnr = s_mara-matnr ) ).
I'm new to Dataflow. I've got a predefined-schema containing a non-repeated RECORD field called "device":
device.configId: STRING
device.version: STRING
Using a ParDo transform, I tried inserting a TableRow with this kind of field, as follows:
TableRow row = new TableRow();
row.put("field1", "val1");
TableRow device = new TableRow();
device.put("configId", "conf1");
device.put("version", "1.2.3");
row.put("device", device);
out.output(row);
I logged the table row, it looks like this:
{field1=val1, device={configId=conf1, version=1.2.3}}
I output it to a standard transform: BigQueryIO.write()
But the latter issues an error:
java.lang.RuntimeException: java.io.IOException:
Insert failed: [{"errors":[{
"debugInfo":"",
"location":"device.configid",
"message":"This field is not a record.",
"reason":"invalid"
}],"index":0}]
Not sure why, but note the location spells "configid" in lowecase - not in camel case as in the original log.
Any ideas on how to insert such an object to BigQuery?
Found out the problem. Apparently, this error message was caused only when the "configId" field was set to null rather than "conf1". To be exact, it was implicitly set to JSONObject.NULL coming from some input object.
I am using connect-mls RESO API and I am having a problem forming the query to search for via PropertyType.
http://odata.reso.org/RESO/OData/Property?$filter=/PropertyType/Name eq "Residential"
The above query keeps coming up with malformed URI.
I also run into a problem is if try to filter on the PropertyType field directly via $filter=(PropertyType eq 'Residental') or $filter=(PropertyType eq 'DE').
I get the following error message:
"message": "StatusCodeError: 400 - {\"error\":{\"code\":null,\"message\":\"The types 'ODataService.PropertyType' and 'Edm.String' are not compatible.\"}}"
Also looked at values in the data dictionary because it seems property type is a enum but have not had any success in any of the formats.
http://ddwiki.reso.org/display/DDW16/Property+Type+Summary
Appreciate any guidance on this.
I was able to find the answer from another source. For the enums they are in a format of ODataService.PropertyType'DE'. A proper API call example is listed below.
https://connectmls-api.mredllc.com/reso/odata/Property?$filter=PropertyType eq ODataService.PropertyType'DE'
For more detailed information on how to properly construct these types of queries, you can look at http://www.odata.org/documentation/
My current table in BigQuery has a column that uses complex types. The "family" column is actually a list ("repeated" feature) of records (with 2 fields: id & name).
When I try to get the 1st "id" value of 1 row with the following syntax:
FieldValueList c = qr.getValues().iterator().next();
c.get("family").getRepeatedValue().get(0).getRecordValue().get("id");
I get the exception:
Method threw 'java.lang.UnsupportedOperationException' exception.
Retrieving field value by name is not supported when there is no fields schema provided
This is a bit annoying because my table has a clearly defined schema. And when I do the "read" query with the same Java call, I can also see that this schema is correctly found:
qr.getSchema().getFields().get("family").getSubFields().toString();
-->
[Field{name=id, type=INTEGER, mode=NULLABLE, description=null}, Field{name=name, type=STRING, mode=NULLABLE, description=null}]
Due to this exception, the workaround that I have found is to pass the "index" of the record field instead of giving it its name
c.get("family").getRepeatedValue().get(0).getRecordValue().get(0).getLongValue();
However, this seeks awkward to pass an index instead of a name.
Is there a better way to get the value of a field in a record inside an array (if my column is only a record, without array, then I don't get the exception) ?
Is this exception normal?
You can wrap the unnamed FieldValueList with a named one using the "of" static method:
FieldList subSchema = qr.getSchema().getFields().get("family").getSubFields();
FieldValueList c = qr.getValues().iterator().next();
FieldValueList.of(
c.get("family").getRepeatedValue().get(0).getRecordValue(),
subSchema).get("id");
The "of" method takes a FieldValueList (returned by getRecordValue() in this case) and a FieldList (subSchema here), and returns the same FieldValueList but with named access.
I'm trying to get the maximum value of an integer field in a table. Specifically, I'm trying to automatically increment the "InvoiceNumber" field when adding a new invoice. I don't want this to be an autoincrement field in the database, however, since it's controlled by the user -- I'm just trying to take care of the default case. Right now, I'm using
session.CreateCriteria<Invoice>()
.SetProjection(Projections.Max("InvoiceNumber"))
.FutureValue<int>();
to get the biggest invoice number already in the database. This works great, except when there are no invoices already in the database. Then I get a System.ArgumentException: The value "" is not of type "System.Int32" and cannot be used in this generic collection. Changing to FutureValue<int?>() didn't solve the problem. Is there a way to tell NHibernate to map the empty string to null? Or is there a better way to accomplish my goal altogether?
The stack trace of the exception (at least the relevant part) is
NHibernate.HibernateException: Error executing multi criteria : [SELECT max(this_.[InvoiceNumber]) as y0_ FROM dbo.[tblInvoice] this_;
SELECT this_.ID as ID647_0_, this_.[NHVersion] as column2_647_0_, this_.[Description] as column3_647_0_, this_.[DiscountPercent] as column4_647_0_, this_.[DiscountDateDays] as column5_647_0_, this_.[PaymentDueDateDays] as column6_647_0_, this_.[Notes] as column7_647_0_, this_.[DiscountDateMonths] as column8_647_0_, this_.[PaymentDueDateMonths] as column9_647_0_, this_.[DiscountDatePeriod] as column10_647_0_, this_.[DiscountDateMonthlyDay] as column11_647_0_, this_.[DiscountDateMonthlyDayDay] as column12_647_0_, this_.[DiscountDateMonthlyDayMonth] as column13_647_0_, this_.[DiscountDateMonthlyThe] as column14_647_0_, this_.[DiscountDateMonthlyTheDOW] as column15_647_0_, this_.[DiscountDateMonthlyTheMonth] as column16_647_0_, this_.[DiscountDateMonthlyTheWeek] as column17_647_0_, this_.[PaymentDueDatePeriod] as column18_647_0_, this_.[PaymentDueDateMonthlyDay] as column19_647_0_, this_.[PaymentDueDateMonthlyDayDay] as column20_647_0_, this_.[PaymentDueDateMonthlyDayMonth] as column21_647_0_, this_.[PaymentDueDateMonthlyThe] as column22_647_0_, this_.[PaymentDueDateMonthlyTheDOW] as column23_647_0_, this_.[PaymentDueDateMonthlyTheMonth] as column24_647_0_, this_.[PaymentDueDateMonthlyTheWeek] as column25_647_0_ FROM dbo.[tblTermsCode] this_;
] ---> System.ArgumentException: The value "" is not of type "System.Int32" and cannot be used in this generic collection.
Parameter name: value
at System.ThrowHelper.ThrowWrongValueTypeArgumentException(Object value, Type targetType)
at System.Collections.Generic.List`1.VerifyValueType(Object value)
at System.Collections.Generic.List`1.System.Collections.IList.Add(Object item)
at NHibernate.Impl.MultiCriteriaImpl.GetResultsFromDatabase(IList results)
use....UniqueValue<int?>();
NH uses a non-generic IList in their MultiCriteria implementation. Which is used for FutureValue batching. see here for why List<int?> fails to add null through it's IList implementation. I'm surprised I've never run into this before. Avoid using nullable value types with Future or MultiCriteria.
With the QueryOver API:
Session.QueryOver<T>()
.Select(Projections.Max<Statistic>(s => s.PeriodStart))
.SingleOrDefault<object>();
if nothing is returned its null, otherwise cast the result as numeric