I am trying to publish data directly from pubsub to bigquery.
I have created a topic with a schema.
I have created a table.
But when I create the subscription, I get an error request contains an invalid argument
gcloud pubsub subscriptions create check-me.httpobs --topic=check-me.httpobs --bigquery-table=agilicus:checkme.httpobs --write-metadata --use-topic-schema
ERROR: Failed to create subscription [projects/agilicus/subscriptions/check-me.httpobs]: Request contains an invalid argument.
ERROR: (gcloud.pubsub.subscriptions.create) Failed to create the following: [check-me.httpobs].
there's not really a lot of diagnostics i can do here.
Is there any worked out example that shows? What am i doing wrong for this error?
Side note: its really a pain to have to create the BQ schema w/ its native json format, and then create the message schema in avro format. Similar, but different, and no conversion tools that I can find.
If i run with --log-http, it doesn't really enlighten:
{
"error": {
"code": 400,
"message": "Request contains an invalid argument.",
"status": "INVALID_ARGUMENT"
}
}
-- update:
switched to protobuf, same problem.
https://gist.github.com/donbowman/5ea8f8d8017493cbfa3a9e4f6e736bcc has the details.
gcloud version
Google Cloud SDK 404.0.0
alpha 2022.09.23
beta 2022.09.23
bq 2.0.78
bundled-python3-unix 3.9.12
core 2022.09.23
gsutil 5.14
I have confirmed all the fields are present and correct format, as per https://github.com/googleapis/googleapis/blob/master/google/pubsub/v1/pubsub.proto#L639
specifically:
{"ackDeadlineSeconds": 900, "bigqueryConfig": {"dropUnknownFields": true, "table": "agilicus:checkme.httpobs", "useTopicSchema": true, "writeMetadata": true}, "name": "projects/agilicus/subscriptions/check-me.httpobs", "topic": "projects/agilicus/topics/check-me.httpobs"}
I have also tried using the API Explorer to post this, same effect.
I have also tried using the python example:
https://cloud.google.com/pubsub/docs/samples/pubsub-create-bigquery-subscription#pubsub_create_bigquery_subscription-python
to create. Same error w/ a slight bit more info (ip, grpc_status 3)
debug_error_string = "UNKNOWN:Error received from peer ipv6:%5B2607:f8b0:400b:807::200a%5D:443 {created_time:"2022-10-04T20:54:44.600831924-04:00", grpc_status:3, grpc_message:"Request contains an invalid argument."}"
Related
We use Azure Data Factory copy pipeline to transfer data from REST api's to a Azure SQL Database and it is doing some strange things. Because we loop over a set of API's that need to be transferred the mapping is empty from the copy activity.
But for one API the automatic mapping is going wrong, the destination table is created with all the needed columns and correct datatypes based on the received metadata. When we run the pipeline for that specific API, the following message is showed.
{ "errorCode": "2200", "message": "ErrorCode=SchemaMappingFailedInHierarchicalToTabularStage,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed to process hierarchical to tabular stage, error message: Ticks must be between DateTime.MinValue.Ticks and DateTime.MaxValue.Ticks.\r\nParameter name: ticks,Source=Microsoft.DataTransfer.ClientLibrary,'", "failureType": "UserError", "target": "Copy data1", "details": [] }
As a test we did do the mapping for that API manually by using the "Import Schema" option on the Mapping page. there we see that all the fields are correctly mapped. We execute the pipeline again using the mapping and everything is working fine.
But of course, we don't want to use a manually mapping because it is used in a loop for different API's also.
When I upsert a row that mismatches schema I get a PartialFailureError along with a message, e.g.:
[ { errors:
[ { message: 'Repeated record added outside of an array.',
reason: 'invalid' } ],
...
]
However for large rows this isn't sufficient, because I have no idea which field is the one creating the error. The bq command does report the malformed field.
Is there either a way to configure or access name of the offending field, or can this be added to the API endpoint?
Please see this Github Issue: https://github.com/googleapis/nodejs-bigquery/issues/70 . Apparently node.js client library is not getting the location field from the API so it's not able to return it to the caller.
Workaround that worked for me: I copied the JSON payload to my Postman client and manually sent a request to REST API (let me know if you need more details of how to do it).
I want to create new ASA job using rest end point with input output and query for job.
I've followed this link
https://learn.microsoft.com/en-us/rest/api/streamanalytics/stream-analytics-job
and formed below url according to above link
https://management.azure.com/subscriptions/{subscription-id}/
resourcegroups/{resourcegroup-name}/providers/Microsoft.StreamAnalytics/
streamingjobs/dynamicStream?api-version=2015-10-01
Though I've provided api version query parameter, postman is giving me below error
{
"error": {
"code": "MissingApiVersionParameter",
"message": "The api-version query parameter (?api-version=) is required for all requests."
}
}
Any idea what am I doing wrong ?
Api for creating new input for stream analytics job is behaving in same way
Thank you for reading !!!
POSTMAN issue !
Tried using different method and it's working .
We want to programatically order VSI using the flavor (for example. Balanced type), however instead of using the standard os_code, we want the VSI to be created from a public image template (ie. CentOS7-ChangeStable). From the following doc it seems to be possible.
http://softlayer-python.readthedocs.io/en/latest/_modules/SoftLayer/managers/vs.html
However I tried but got the following error:
SoftLayer.exceptions.SoftLayerAPIError: SoftLayerAPIError(SoftLayer_Exception_InvalidValue): Invalid value provided for 'blockDevices'. Block devices may not be provided when using an image template.
Using slcli is failing as well with a different error:
# slcli vs create --hostname testvsi --domain vmonic.local --flavor BL2_4X8X100 --image 1cc8be72-f230-4ab9-b4b2-329c3e747853 --datacenter tok02 --private
This action will incur charges on your account. Continue? [y/N]: y
SoftLayerAPIError(SoftLayer_Exception_Public): Order is missing the following category: Operating System.
Please advice whether using "image_id" with "flavor" is supported in SL API / python API. Thanks!
this is an issue with the API, the python client uses the http://sldn.softlayer.com/reference/services/softlayer_virtual_guest/createObject method to create the VSI using RESTFul the same request would be something like this:
POST: https://$USERNAME:#APIKEY#api.softlayer.com/rest/v3.1/SoftLayer_Virtual_Guest/createObject
Payload:
{
"parameters": [{
"datacenter": {
"name": "tok02"
},
"domain": "softlayer.local",
"hourlyBillingFlag": true,
"blockDeviceTemplateGroup": {
"globalIdentifier": "1cc8be72-f230-4ab9-b4b2-329c3e747853"
},
"hostname": "rcabflav",
"privateNetworkOnlyFlag": true,
"supplementalCreateObjectOptions": {
"flavorKeyName": "BL2_4X8X100"
}
}]
}
and you will get the same error, I reported this error in Softlayer, if you want you can submit a ticket in softlayer and report it as well.
one of my jobs keeps failing and when I looked into why (by requesting job details) I get the following output:
status": {
"errorResult": {
"location": "gs://sf_auto/Datastore Mapper modules.models.userData/15716706166748C8426AD/output-46",
"message": "JSON table encountered too many errors, giving up. Rows: 1; errors: 1.",
"reason": "invalid"
},
"errors": [
{
"location": "gs://sf_auto/Datastore Mapper modules.models.userData/15716706166748C8426AD/output-46",
"message": "JSON table encountered too many errors, giving up. Rows: 1; errors: 1.",
"reason": "invalid"
}
],
"state": "DONE"
Problem is, it doesn't help at all, and I need more details. Is there anyway to understand which column or attribute caused the failings? Is there any way to get more information?
Edit Additional Details
We're running a map reduce job on appengine to transfer our datastore from appengine to BigQuery
The files are stored on Google Cloud Store
It's creating a brand new table instead of adding to an existing one
Update #2
I played around with the query trying lots of things as well as adjusting the scheme and i've narrowed down the problem to the uuid. For some reason this type of data messes everything up:
"uuid": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"
The schema defines it as a String
OK, after loads of debugging I found the error... in the JSON Newline file we had two attributes that were similar:
uuid: "XXX..."
uuId: "XXX..."
This has been there for a while so I think some change within bigquery started to require that keys be unique regardless of capitalization. Will test so more and confirm!
A recent change made loads of JSON data case insensitive in field names similar to be consistent with how SQL queries treat field names. I have opened a work item to track the improvement of error message for this case.