We are getting an error when running a query job on BigQuery :
'message' => 'Request timed out. Please try again.',
'reason' => 'timeout',
'location' => 'script'
We will be running it again as it requested, but since it takes 1/2 hour for it to time out, it would be helpful if we knew what the problem is so we can avoid it.
Specifically:
What does "script" mean in this context? Is it the SQL-query or one
of the UDFs? (the query calls 3 UDFs).
The exact same query failed twice already with "internal error". The "request timed out" was its 3rd attempt at mocking us. Any correlation between the two error types?
It would be really helpful to receive more detailed information on what failed to work, so that we can track it down.
Related
More info from stms :
Short Text
SQL error "SQL code: -10108" occurred while accessing table "CDHDR".
What happened?
Database error text: "SQL message: Session has been reconnected."
Return value of the database layer: "SQL dbsl rc: 99"
The problem is that client is running this code through the job in which I am not available to debug it. All I know is that it is working for other countries selected but for some specific it is not so the question is :
Was it a temporary connection error between two servers or is it data overload issue due to SELECT statement?
At the moment I am not really sure which thing I have to look on the system, the same program once produced the error for exceeding the limit in the db query ( > 10 minutes) so this might be related to system configuration or what?
Thanks in advance
I have an API in WSO2. When I try to test it in the store with valid parameters via GET, it returns the following error message:
<am:fault xmlns:am="http://wso2.org/apimanager">
<am:code>101504</am:code>
<am:type>Status report</am:type>
<am:message>Runtime Error</am:message>
<am:description>Send timeout</am:description>
</am:fault>
I have already searched and tried a lot, but with no success, always returning the same error. Don't know if helps, but the api that I try to access is a PHP file.
Any ideas?
EDIT:
I have identical apis to this one, changing only the response, that are working properly. Even if I erase the php file that the API is pointing, the error keep coming.
EDIT:
I changed the code in Management Console, in Metadata>List>APIs:
{"production_endpoints":
{"url":"http://site/myapi.php","config":
{"format":"leave-as-is","optimize":"leave-as-
is","actionSelect":"fault","actionDuration":30000}},
"sandbox_endpoints":
{"url":"http://site/myapi.php","config":
{"format":"leave-as-is","optimize":"leave-as-
is","actionSelect":"fault","actionDuration":30000}},
"implementation_status":"managed","endpoint_type":"http"}
to this:
{"production_endpoints":
{"url":"http://10.20.40.189/ConsultaAutorizacaoCadPos.php","config":null},
"sandbox_endpoints":{"url":"http://10.20.40.189/ConsultaAutorizacaoCadPos.php","config":null},
"implementation_status":"managed","endpoint_type":"http"}
And this error message vanishes. But this new one appears:
<am:fault xmlns:am="http://wso2.org/apimanager">
<am:code>303001</am:code>
<am:type>Status report</am:type>
<am:message>Runtime Error</am:message>
<am:description>Currently , Address endpoint : [ Name : admin--myapi_APIproductionEndpoint_0 ] [ State : SUSPENDED ]</am:description>
EDIT: this error message is appearing just sometimes. Most of the time the request takes long time to run and then returns nothing( no content )
Any ideas how to solve it?
I am looking to search for error type in my spunk. A typical error log looks like this:
ERROR 2016/03/16 22:13:55 Program exited with error Calling service: Post http://hostname/v1.21/resource/create?name=/60b80cf9-ebc4-11e5-a9cb-3c4a92db9491-2: read unix #->/var/run/program.sock: use of closed network connection (Client.Timeout exceeded while awaiting headers)
Note that common part is "Program exited with error". I am looking to capture the part that follows this common part of the error message. I tried with a couple of rex expressions. Both returned different results. Importantly, neither captured the error type I have shown above. I am giving the one that worked better here.
* | rex "Program exited with error\s+(?<reason>.+)" | top reason
An example of the log it matched-
Unable to get program status, Get http://192.168.0.2:2774/program/v1/status: net/http: timeout awaiting response headers
However, it did not match log of the form-
initial ZK connection failed, stat /var/program/f47aae5c-ea42-11e5-8975-fc15b40f4cc4/srcheck/started: no such file or directory
Calling service: Post http://hostname/v1.21/resource/create?name=/60b80cf9-ebc4-11e5-a9cb-3c4a92db9491-2: read unix #->/var/run/program.sock: use of closed network connection (Client.Timeout exceeded while awaiting headers)
Could someone help me understand what's wrong with my rex expression and what the right one would be so I get all possible error types?
This recipe:
"ERROR.*Program exited with error.*:.*:.*:\s+(?<reason>.+)"
will yield:
use of closed network connection (Client.Timeout exceeded while awaiting headers)
I don't have enough sample data to know if this will hold up or not. For example, I'm counting on exactly 3 colons, to get me to the interesting part. Also I don't know if you care about other things like the hostname, the fact that it's a Post, etc.. But based on your sample of 1, this answer should do the trick.
Streaming data into BigQuery keeps failing due to the following error, which occurs more frequently recently:
com.google.api.client.googleapis.json.GoogleJsonResponseException: 503 Service Unavailable
{
"code" : 503,
"errors" : [ {
"domain" : "global",
"message" : "Connection error. Please try again.",
"reason" : "backendError"
} ],
"message" : "Connection error. Please try again."
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:145)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:312)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1049)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:410)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:343)
Relevant question references:
Getting high rate of 503 errors with BigQuery Streaming API
BigQuery - BackEnd error when loading from JAVA API
We (the BigQuery team) are looking into your report of increased connection errors. From the internal monitoring, there hasn't been global a spike in connection errors in the last several days. However, that doesn't mean that your tables, specifically, weren't affected.
Connection errors can be tricky to chase down, because they can be caused by errors before they get to the BigQuery servers or after they leave. The more information your can provide, the easier it is for us to diagnose the issue.
The best practice for streaming input is to handle temporary errors like this to retry the request. It can be a little tricky, since when you get a connection error you don't actually know whether the insert succeeded. If you include a unique insertId with your data (see the documentation here), you can safely resend the request (within the deduplication window period, which I think is 15 minutes) without worrying that the same row will get added multiple times.
Using the Java SDK I am creating a load job for just a single record with a fairly complicated schema. When monitoring the status of the load job, it takes a surprisingly long time (but perhaps this is due to working out the schema), but then says:
11:21:06.975 [main] INFO xxx.GoogleBigQuery - Job status (21694ms) create_scans_1384744805079_172221126: DONE
11:24:50.618 [main] ERROR xxx.GoogleBigQuery - Job create_scans_1384744805079_172221126 caused error (invalid) with message
Too many errors encountered. Limit is: 0.
11:24:50.810 [main] ERROR xxx.GoogleBigQuery - {
"message" : "Too many errors encountered. Limit is: 0.",
"reason" : "invalid"
?}
BTW - how do I tell the job that it can have more than zero errors using Java?
This load job does not appear in the list of recent jobs in the console, and as far as I can see, none of the Java objects contains any more details about the actual errors encountered. So how can I pro-grammatically find out what is going wrong? All I can find is:
if (err != null) {
log.error("Job {} caused error ({}) with message\n{}", jobID, err.getReason(), err.getMessage());
try {
log.error(err.toPrettyString());
}
...
In general I am having a difficult time finding good documentation for some of these things and am working it out by trial and error and short snippets of code found on here and older groups. If there is a better source of information than the getting started guides, then I would appreciate any pointers to that information. The Javadoc does not really help and I cannot find any complete examples of loading, querying, testing for errors, cataloging errors and so on.
This job is submitted via a NEWLINE_DELIMITIED_JSON record, supplied to the job via:
InputStream dummy = getClass().getResourceAsStream("/googlebigquery/xxx.record");
final InputStreamContent jsonIn = new InputStreamContent("application/octet-stream", dummy);
createTableJob = bigQuery.jobs().insert(projectId, loadJob, jsonIn).execute();
My authentication and so on seems to work correctly as separate Java code to list the projects, and the datasets in the project all works correctly. So I just need help in working what the actual error is - does it not like the schema (I have records nested within records for instance), or does it think that there is an error in the data I am submitting.
Thanks in advance for any help. The job number cited above is an actual failed load job if that helps any Google staffers who might read this.
It sounds like you have a couple of questions, so I'll try to address them all.
First, the way to get the status of the job that failed is to call jobs().get(jobId), which returns a job object that has an errorResult object that has the error that caused the job to fail (e.g. "too many errors"). The errorStream list is a lost of all of the errors on the job, which should tell you which lines hit errors.
Note if you have the job id, it may be easier to use bq to lookup the job -- you can run bq show <job_id> to get the job error information. If you add the --format=prettyjson it will print out all of the information in the job.
A hint you also might want to consider is to supply your own job id when you create the job -- then even if there is an error starting the job (i.e. the insert() call fails, perhaps due to a network error) you can look up the job to see what actually happened.
To tell BigQuery that some errors are allowed during import, you can use the maxBadResults setting in the load job. See https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/java/latest/com/google/api/services/bigquery/model/JobConfigurationLoad.html#getMaxBadRecords().