Error loading file stored in Google Cloud Storage to Big Query - google-bigquery

I have been trying to create a job to load a compressed json file from Google Cloud Storage to a Google BigQuery table. I have read/write access in both Google Cloud Storage and Google BigQuery. Also, the uploaded file belongs in the same project as the BigQuery one.
The problem happens when I access to the resource behind this url https://www.googleapis.com/upload/bigquery/v2/projects/NUMERIC_ID/jobs by means of a POST request. The content of the request to the abovementioned resource can be found as follows:
{
"kind" : "bigquery#job",
"projectId" : NUMERIC_ID,
"configuration": {
"load": {
"sourceUris": ["gs://bucket_name/document.json.gz"],
"schema": {
"fields": [
{
"name": "id",
"type": "INTEGER"
},
{
"name": "date",
"type": "TIMESTAMP"
},
{
"name": "user_agent",
"type": "STRING"
},
{
"name": "queried_key",
"type": "STRING"
},
{
"name": "user_country",
"type": "STRING"
},
{
"name": "duration",
"type": "INTEGER"
},
{
"name": "target",
"type": "STRING"
}
]
},
"destinationTable": {
"datasetId": "DATASET_NAME",
"projectId": NUMERIC_ID,
"tableId": "TABLE_ID"
}
}
}
}
However, the error doesn't make any sense and can also be found below:
{
"error": {
"errors": [
{
"domain": "global",
"reason": "invalid",
"message": "Job configuration must contain exactly one job-specific configuration object (e.g., query, load, extract, spreadsheetExtract), but there were 0: "
}
],
"code": 400,
"message": "Job configuration must contain exactly one job-specific configuration object (e.g., query, load, extract, spreadsheetExtract), but there were 0: "
}
}
I know the problem doesn't lie either in the project id or in the access token placed in the authentication header, because I have successfully created an empty table before. Also I specify the content-type header to be application/json which I don't think is the issue here, because the body content should be json encoded.
Thanks in advance

Your HTTP request is malformed -- BigQuery doesn't recognize this as a load job at all.
You need to look into the POST request, and check the body you send.
You need to ensure that all the above (which seams correct) is the body of the POST call. The above Json should be on a single line, and if you manually creating the multipart message, make sure there is an extra newline between the headers and body of each MIME type.
If you are using some sort of library, make sure the body is not expected in some other form, like resource, content, or body. I've seen libraries that use these differently.
Try out the BigQuery API explorer: https://developers.google.com/bigquery/docs/reference/v2/jobs/insert and ensure your request body matches the one made by the API.

Related

Where do I store GraphQL errors

Basically what the title asks. As far as I know, GraphQL responses are structured as follows:
{
"data": { ... },
"errors":
{
"message": "This message describes what happened",
"path": [
"somePath",
"inGraphQLQuery"
],
"locations": [
{
"line": 2,
"column": 3
}
]
}
]
}
And that works for simple errors. But what if my error is more complicated than just a message? For example, I have mutation that changes some fields of user, such as username, email, password, etc. That mutation might result in error if user-provided data for any field is invalid. But what do I do if validation for multiple fields fails? Ideal way in my opinion to show that to user would be to send structure like this
[
{
"field": "username",
"details": "Username must contain only latin letters and digits"
},
{
"field": "email",
"details": "Email address must contain '#'"
}
{
"field": "password",
"details": "Password must contain at least 8 characters"
}
]
But to my knowledge, there's no way to send this in errors section. So I tend to put this into data section of response and specify that user will receive either User or FieldValidationError.
So, to sum it up, there are some structures that I can't fit into errors, so (for consistency) it seems logical to put every error into data. And by doing so, errors section becomes redundant, since it isn't used for errors at all. What am I getting wrong?

Netcore server respond to slack interactive message

I am trying to build Slack application with asp net core server. Currently, I have added Slash command which makes a request to my local server through ngrok. Once my server receives that request, it makes a post request to configured slack webhook to display interactive message which looks like pic from attachment.
I want user to be able to select yes or no and receive the result in my controller, but I can't realize how to tell Slack where should it make a post request. I attach the code of this message which is posted through weebhook into Slack:
{
"blocks": [
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "This is a section block with a button."
}
},
{
"type": "actions",
"block_id": "actionblock789",
"elements": [
{
"type": "button",
"text": {
"type": "plain_text",
"text": "Yes"
},
"style": "primary",
"value": "yes"
},
{
"type": "button",
"text": {
"type": "plain_text",
"text": "No"
},
"value": "no"
}
]
}
]
}
What I have in netcore is TestController with route /api/test which I suppose should receive from Slack a payload where information about selected button is set, but I couldn't find a way where specify a url in this json code.
You can not configure a URL in the JSON code. That is not how it works with the Slack API.
Slack will send all responses to interactive message - like when someone clicks on a button - to the configured request URL of your Slack app. Check out this link how to find that parameter: documentation

JSON Schema Array Validation Woes Using oneOf

Hope I might find some help with this validation issue: I have a JSON array that can have multiple object types (video, image). Within that array, the objects have a rel field value. The case I'm working on is that there can only be one object with "rel": "primaryMedia" allowed — either a video or an image.
Here are the object representations for the image and video case, both showing the "rel": "primaryMedia".
{
"media": [
{
"caption": "Caption goes here",
"id": "ncim87659842",
"rel": "primaryMedia",
"type": "image"
},
{
"description": "Shaima Swileh arrived in San Francisco after fighting for 17 months to get a waiver from the U.S. government to be allowed into the country to visit her son.",
"headline": "Yemeni mother arrives in U.S. to be with dying 2-year-old son",
"id": "mmvo1402810947621",
"rel": "primaryMedia",
"type": "video"
}
]
}
Here's a stripped-down version of the schema I created to validate this using oneOf (assuming this will handle my case). It doesn't however, work as intended.
{
"$id": "http://example.com/schema/rockcms/article.json",
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {},
"properties": {
"media": {
"items": {
"oneOf": [
{
"additionalProperties": false,
"properties": {
"caption": {
"type": "string"
},
"id": {
"type": "string"
},
"rel": {
"enum": [
"primaryMedia"
],
"type": "string"
},
"type": {
"enum": [
"image"
],
"type": "string"
}
},
"required": [
"caption",
"id",
"rel",
"type"
],
"type": "object"
},
{
"additionalProperties": false,
"properties": {
"description": {
"type": "string"
},
"headline": {
"type": "string"
},
"id": {
"type": "string"
},
"rel": {
"enum": [
"primaryMedia"
],
"type": "string"
},
"type": {
"enum": [
"video"
],
"type": "string"
}
},
"required": [
"description",
"headline",
"id",
"rel",
"type"
],
"type": "object"
}
]
},
"type": "array"
}
}
}
Using the JSON Schema validator at https://www.jsonschemavalidator.net, the schema validates when data is correctly presented, but the doesn't work when trying to catch errors.
In the case below, headline is added for a video, and it's missing id. This should fail because headline isn't allowed on video, and id is required.
{
"media": [
{
"headline": "Yemeni mother arrives in U.S. to be with dying 2-year-old son",
"rel": "primaryMedia",
"type": "video"
}
]
}
The results I get from the validator, however, aren't entirely expected. Seems it's conflating the two object schemas in its response.
In addition to this, I've separately found that the schema will allow the population of BOTH a video and image object in media, which isn't expected.
Been trying to figure out what I've done wrong, but am stumped. Would very much appreciate some feedback, if anyone has to offer. Thanks in advance!
Validator Response:
Message: JSON is valid against no schemas from 'oneOf'.
Schema path: #/properties/media/items/oneOf
Message: Property 'headline' has not been defined and the schema does not allow additional properties.
Schema path: #/properties/media/items/oneOf/0/additionalProperties
Message: Value "video" is not defined in enum.
Schema path: #/properties/media/items/oneOf/0/properties/type/enum
Message: Required properties are missing from object: description, id.
Schema path: #/properties/media/items/oneOf/1/required
Message: Required properties are missing from object: caption, id.
Schema path: #/properties/media/items/oneOf/0/required
Your question is formed of three parts, but the first two are linked, so I'll address those, although they don't have a "solution" as such.
How can I validate that only one object in an array has a specific key, and the others do not.
You cannot do this with JSON Schema.
The set of JSON Schema keywords which are applicable to arrays do not have a means for expressing "one of the values must match a schema", but rather are applicable to either ALL of the items in a the array, or A SPECIFIC item in the array (if items is an array as opposed to an object).
https://datatracker.ietf.org/doc/html/draft-handrews-json-schema-validation-01#section-6.4
The validation output is not what I expect. I expect to only see the failing branch of oneOf which relates to my object type, which
is defined by the type key.
The JSON Schema specification (as of draft-7) does not specify any format for returning errors, however the error structure you get is pretty "complete" in terms of what it's telling you (and is similar to how we are specifying errors should be returned for draft-8).
Consider, the validator knows nothing about your schema or your JSON instance in terms of your business logic.
When validating a JSON instance, a validator may step through all values, and test validation rules against all applicable subschemas. Looking at your errors, both of the schemas in oneOf are applicable to all items in your array, and so all are tested for validation. If one does not satisfy the condition, the others will be tested also. The validator cannot know, when using a oneOf, what your intent was.
You MAY be able to get around this issue, by using an if / then combination. If your if schema is simply a const of the type, and your then schema is the full object type, you may get a cleaner error response, but I haven't tested this theory.
I want ALL items in the array to be one of the types. What's going on here?
From the spec...
items:
If "items" is a schema, validation succeeds if all elements in the
array successfully validate against that schema.
oneOf:
An instance validates successfully against this keyword if it
validates successfully against exactly one schema defined by this
keyword's value.
What you have done is say: Each item in the array should be valid according to [schemaA]. SchemA: The object should be valid according to on of these schemas: [the schemas inside the oneOf].
I can see how this is confusing when you think "items must be one of the following", but items applies the value schema to each of the items in the array independantly.
To make it do what you mean, move the oneOf above items, and then refactor one of the media types to another items schema.
Here's a sudo schema.
oneOf: [items: properties: type: const: image], [items: properties: type: image]
Let me know if any of this isn't clear or you have any follow up questions.

How do I create an event using the SocialTables API?

I'm trying to use the /4.0/legacyvm3/teams/{team}/events endpoint to create an event. I'm running into some trouble with spaces.
I used the /4.0/legacyvm3/teams/{team}/venues endpoint to get a list of venues. I chose one to include in the spaces section and posted this:
{
"name": "Event via API Test 04",
"category": "athletic event",
"public": true,
"attendee_management": true,
"start_time": "2017-04-05T16:13:54.217Z",
"end_time": "2017-04-05T16:13:54.217Z",
"uses_metric": false,
"venue_mapper_version": 0,
"spaces": [
{
"venue_id": 128379,
"name": "Snurrrggggg"
}
]
}
The endpoint returns a 400 code and this error:
{
"code": 400,
"message": "Cannot read property 'toLowerCase' of undefined"
}
I tried including the wizard section, but each time it would return this error:
{
"message": "Access Denied to this feature"
}
After some experimentation, this body succeeded:
{
"name": "Event via API Test 03",
"category": "athletic event",
"public": true,
"attendee_management": true,
"start_time": "2017-04-05T16:13:54.217Z",
"end_time": "2017-04-05T16:13:54.217Z",
"uses_metric": false,
"venue_mapper_version": 0,
"spaces": [
{
"name": "Fake News Room"
}
]
}
But the application itself would not display the diagram, and the newly created room did not show up in my list of venues. Perhaps it did not assign permissions to it?
In any case, I don't actually want to create a new venue/space. I want to pass in an existing venue/space. How do I do that?
The short answer is to create a working diagram in 4.0 you will need to POST some data to the /4.0/diagrams endpoint.
The room you create doesn't map to the same concept as venues. When you create an event as you did, it creates a new space entity. The spaces endpoints can return information on those.

JIRA Rest API select minimal resource

I am using JIRA REST API for querying issues with below jql
jql=project =SLUB and "Agile Team" in ("Iris (B2C)")&fieldsByKeys=true&fields=status&maxResults=1
I am getting api response as
{
"expand": "names,schema",
"startAt": 0,
"maxResults": 1,
"total": 1172,
"issues": [
{
"expand": "operations,versionedRepresentations,editmeta,changelog,renderedFields",
"id": "35988",
"self": "https://xyz.atlassian.net/rest/api/2/issue/35988",
"key": "SLUB-7071",
"fields": {
"status": {
"self": "https://xyz.atlassian.net/rest/api/2/status/10200",
"description": "",
"iconUrl": "https://xyz.atlassian.net/",
"name": "To Do",
"id": "10200",
"statusCategory": {
"self": "https://xyz.atlassian.net/rest/api/2/statuscategory/2",
"id": 2,
"key": "new",
"colorName": "blue-gray",
"name": "To Do"
}
}
}
}
]
}
How can I only fetch status name instead of complete status resource. Please suggest.
https://docs.atlassian.com/jira/REST/latest/#d2e3181
Check This .
The fields param (which can be specified multiple times) gives a comma-separated list of fields to include in the response. This can be used to retrieve a subset of fields. A particular field can be excluded by prefixing it with a minus.
By default, only navigable (*navigable) fields are returned in this search resource. Note: the default is different in the get-issue resource -- the default there all fields (*all).
*all - include all fields
*navigable - include just navigable fields summary,comment - include just the summary and comments
-description - include navigable fields except the description (the default is *navigable for search)
*all,-comment - include everything except comments
Copied From Here.