I'm having a problem with some endpoints within the ADLS Gen 2 API Path operations.
I can create, list, get properties of, and delete file systems just fine.
However, after adding a directory to a file system, certain verbs are failing - HEAD, GET, and DELETE.
For example, I have created a filesystem named c79b0781, with a directory path of abc/def
Call failed with status code 400 (The HTTP verb specified is invalid - it is not recognized by the server.): DELETE https://myadls.dfs.core.windows.net/c79b0781/abc?recursive=true&timeout=30
For headers, I have:
x-ms-version: 2018-11-09
I can delete the filesystem from the Azure Storage Explorer, but the API is refusing my query.
The List action is also failing with a similar error
Call failed with status code 400 (The HTTP verb specified is invalid - it is not recognized by the server.): GET https://myadls.dfs.core.windows.net/c79b0781?resource=filesystem&recursive=false&timeout=30
With headers:
x-ms-version: 2018-11-09
And finally, my Get Properties is also failing
Call failed with status code 400 (The HTTP verb specified is invalid - it is not recognized by the server.): HEAD https://myadls.dfs.core.windows.net/c79b0781?resource=filesystem&timeout=30
It seems to only happen when I add directories to the file system.
A bit more in depth:
This Test works
PUT https://myadls.dfs.core.windows.net/c79b0781?resource=filesystem
GET https://myadls.dfs.core.windows.net/c79b0781?recursive=false&resource=filesystem
DELETE https://myadls.dfs.core.windows.net/c79b0781?resource=filesystem
My second Test with directory creation
PUT https://myadls.dfs.core.windows.net/c79b0781?resource=filesystem
PUT https://myadls.dfs.core.windows.net/c79b0781/abc/123?resource=directory
After this point, the calls begin rejecting HTTP verbs
GET https://myadls.dfs.core.windows.net/c79b0781?recursive=false&resource=filesystem
Examining my directory create request closer, it looks like this:
PUT https://myadls.dfs.core.windows.net/c79b0781/abc/123?resource=directory
With Headers:
Authorization: [omitted]
Content-Length: 0
And I can see the folders in Storage explorer, I just cannot act on them after this point.
Test Case 2
I have started down a path wondering if it is permissions. So, I created a new File System through the Azure Storage Explorer with abc/def folder structure within.
Test 1 (passing)
Get List for directory "abc"
Get List for directory "abc/def"
Test 2 (failing)
Create Directory "uvw/xyz"
Get List for directory "abc" Fails here
Get List for directory "abc/def"
Get List for directory "uvw/xyz"
Once I create a directory through the api, it is as if the entire filesystem begins rejecting all HTTP requests.
This bug ended up leading me down a rabbit hole into my Flurl implementation that I am using for performing rest requests.
The Put method had no body and was calling PutJsonAsync where according to the spec, it expects the content type to be application/octet-stream with content length 0.
I replaced the call to PutJsonAsync to PutAsync and everything magically started working.
So, there seems to be some bug within Flurl itself that caused this issue, due to my misuse in my wrapper code.
Related
Appendix A of The Definitive Guide to Jython describes downloading SetupTools for use with Jython.
https://jython.readthedocs.io/en/latest/appendixA/
This indicates to me that it should be possible to download and use SetupTools from within a Jython automation script in Maximo (v7.6 in my case). The book points us to the following url to copy a Jython script that will do this:
http://peak.telecommunity.com/dist/ez_setup.py
I add one line to the above script to call the function "use_setuptools":
use_setuptools()
Then I create a push button on a Maximo application and associate the aforementioned script with the button press I get the following error:
System Message BMXAA7837E - An error occured that prevented the
EZ_SETUP script for the EZ_SETUP launch point from running.
urllib2.HTTPError: HTTP Error 403: SSL is required in at line
number 280
The stack trace points to the following line in the function "download_setuptools" which is called by "use_setuptools":
src = urllib2.urlopen(url)
This appears to be because the url requested, in my case:
http://pypi.python.org/packages/2.5/s/setuptools/setuptools-0.6c11-py2.5.egg
...is redirected a few times before arriving at the following url:
https://files.pythonhosted.org/packages/98/d3/ed3bc7e3f4b143092862dab99d984261ac4be6a40d4fb1e66d2a10e9ea99/setuptools-0.6c11-py2.5.egg
Note the url uses HTTPS not HTTP. The following indicates why this may be so:
https://sourceforge.net/p/pypi/support-requests/300/
The jython.jar included with Maximo does not include the ssl module so we could either:
Download the ssl module manually and copy it to the correct location on the server.
Download the appropriate egg file manually over HTTPS and copy it to the correct location on the server.
Bypass the problem by creating a mirror for the file we're looking for that is accessible over HTTP and use that url in the code.
Whilst these are feasible I'd prefer to modify the code to ignore the SSL certificate if possible, however all the workarounds on StackOverflow and elsewhere seem to require that you're able to "import ssl" in order to bypass it which rather seems to defeat the purpose.
Ideally I'm looking for a solution that modifies the code from the url provided above to get it to work with Maximo/Jython 2.5.2 and doesn't require downloading and adding new modules or packages and all that this entails with Maximo. Bypassing or temporarily disabling ssl is fine as the code checks the hash of the downloaded .egg file. This would be my preferred solution if possible.
In my experience, automation scripting works best if you can stay "as Java as possible" and "as Maximo as possible". So, I would use the LIB_HTTPCLIENT script from the Scripting 76 Features document (the first example code, whose name is given by inference in the second bit of code) to try to download the SetupTools.
In case that document moves again, here is the LIB_HTTPCLIENT script. Note that the url variable is expected to be passed to this library script by the calling script.
from psdi.iface.router import HTTPHandler
from java.util import HashMap
from java.util import String
handler = HTTPHandler()
map = HashMap()
map.put("URL",url)
map.put("HTTPMETHOD","GET")
responseBytes = handler.invoke(map,None)
response = String(responseBytes,"utf-8")
I am trying to do multiple file upload simultaneously to google big-query using command line tool. I got following error :
BigQuery error in load operation: Could not connect with BigQuery server.
Http response status: 503
Http response content:
Service Unavailable
Any way to workaround this problem ?
How do I upload multiple files simultaneously to google big-query using command line tool.
Multiple file upload should work (and we use it every day). If you're getting a 503, that indicates something is wrong with the service. One thing you might want to make sure of is that if you're using a * in your command line that you have it quoted so that the shell doesn't expand it automatically before it gets passed to bq.
If you're getting a 503 error, can you retry the command the flag --apilog=- (this needs to be one of the first params) which will dump the interaction with the server to stdout. The problem may be obvious from that log, but if it isn't can you update your question with the relevant portions of the log? If you're not comfortable posting that information on a public forum, can you e-mail it to me at tigani at google dot com?
I've seen hundreds of pages explaining how to create custom error pages in Apache 2 server. My question is different. I have a web application running in Apache (it is a ISAPI DLL, but it could also be a CGI executable). My application can handle internal server errors and generate a detailed error message (for instance, include a full stack trace), included in the response together with error code 500. AFAIK, Apache just let me use redirection in order to display custom error messages: http://httpd.apache.org/docs/2.2/custom-error.html
HTTP spec (RFC 2616 - section 10), not only allows but also recommend that detailed error message should be included in the BODY section of the response in case of error code > 500.
Link: http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5
Seems that Apache won't let my custom error message go to the browser, and always replace it with its own internal error message and I believe that it is not the correct behavior, based on RFC 2616.
So my question is: Is there any setting in Apache server that will let my custom message go to the browser? Or, is there anything that can be done in my application that will instruct Apache to send my custom error message (something like some specific header field in the response)?
More on the subject:
When my ISAPI application returns error code 500, with other error information in the response body, Apache replaces it with its standard "500 Internal Server Error" message/HTML content, and inside Error.log file I can see the "useless" "Premature end of script headers" message. I'm deeply sure that my headers are fine, including the Content-Type field.
If I replace the 500 error code with any other server error code (e.g. 501) it works flawlessly and my response goes to the browser as is. The same header is sent to the Apache server, only the error code is different (501, instead of 500). With this test result in mind, one of these two must be true:
1- Apache requires some specific header field when status code is 500
2- Apache won't let custom error messages with status code 500 go to the browser.
I don't see any other alternative.
I think you're conflating two questions. You can generate a 500 response with a CGI script and include your custom body. Or you can override any 500 with any resource you want.
If you're failing to do the former, it's likely because of some subtle thing in the ISAPI interface between Apache and your module. Desk-checking the code says you should be able either set the pseudo
Status: 500
Header, or basically return any ISAPI error and end up with a 500 and your custom body.
Apache has two notions of a status code -- the one in the status line (r->status) and an error code returned separately from the module that handles the request (return HTTP_INTERNAL_SERVER_ERROR, return r->status).
When the former is used as the latter is when the custom error messages get lost. All of that happens in./modules/arch/win32/mod_isapi.c in Apache. Whatever is going on, it is ISAPI unique.
Background
I have an API built with OpenRasta that serves JSON-encoded data to an HTML/AJAX enabled interface.
Problem
When hosted with IIS 6, I'm experiencing intermittent 404 errors when a resource is queried. Everything works well after an application pool restart; after a few successful requests the 404 errors start and persist until the app pool is recycled. I have confirmed ISAPI mapping is set in IIS 6, what am I missing?
Trace output for intermittent failed request
Unregistering host of type OpenRasta.Hosting.AspNet.AspNetHost
Trace output for successful request for comparison (namespace and resource URIs redacted)
Registering host of type OpenRasta.Hosting.AspNet.AspNetHost
Using dependency resolver of type OpenRasta.DI.InternalDependencyResolver.
Registering host's root dependencies..
Using dependency registrar of type OpenRasta.Configuration.DefaultDependencyRegistrar..
Ignoring constructor, following dependencies didn't have a registration:OpenRasta.TypeSystem.Surrogates.ISurrogateBuilder[].
Registering host's leaf dependencies..
Using configuration source <namespace>.api.Configuration.
Ignoring constructor, following dependencies didn't have a registration:System.Func`1[System.Collections.Generic.IEnumerable`1[OpenRasta.Configuration.MetaModel.Handlers.IMetaModelHandler]].
Ignoring constructor, following dependencies didn't have a registration:System.Diagnostics.TraceSource.
Ignoring constructor, following dependencies didn't have a registration:OpenRasta.OperationModel.MethodBased.IMethodFilter[].
Incoming host request for http://<resource URI>.
Adding communication context data.
Entering OpenRastaRewriterHandler: Rewriting to original path.
Entering OpenRastaIntegratedHandler: Request for http://<resource URI>.
Incoming host request for http://<resource URI>.
Adding communication context data.
Found 1 operation(s) with a matching name..
Found 0 operation(s) with matching [HttpOperation] attribute..
Operation <handler>::Get() selected with 0 required members and 0 optional members, without request codec.
Ignoring constructor, following dependencies didn't have a registration:OpenRasta.OperationModel.Interceptors.IOperationInterceptor[].
Executing OperationResult OperationResult: type=OK, statusCode=200..
Selected codec JsonDataContractCodec out of 1 codecs for entity of type List`1 and negotiated media type application/json; q=0.5..
Codec JsonDataContractCodec selected..
Entering ResponseEntityWriterContributor: Generating response entity..
Setting Content-Length to 133.
Exiting ResponseEntityWriterContributor.
Writing http headers..
Exiting OpenRastaIntegratedHandler.
Exiting OpenRastaRewriterHandler.
Request finished..
OpenRasta.Hosting.AspNet.dll built from GitHub master:
I'm using SharpBITS to download file from AmazonS3.
> // Create new download job. BitsJob
> job = this._bitsManager.CreateJob(jobName, JobType.Download);
> // Add file to job.
> job.AddFile(downloadFile.RemoteUrl, downloadFile.LocalDestination);
> // Resume
> job.Resume();
It works for files which do no need authentication. However as soon as I add authentication query string for AmazonS3 file request the response from server is http state 403 -unauthorized. Url works file in browser.
Here is the HTTP request from BIT service:
HEAD /mybucket/6a66aeba-0acf-11df-aff6-7d44dc82f95a-000001/5809b987-0f65-11df-9942-f2c504c2c389/v10/summary.doc?AWSAccessKeyId=AAAAZ5SQ76RPQQAAAAA&Expires=1265489615&Signature=VboaRsOCMWWO7VparK3Z0SWE%2FiQ%3D HTTP/1.1
Accept: */*
Accept-Encoding: identity
User-Agent: Microsoft BITS/7.5
Connection: Keep-Alive
Host: s3.amazonaws.com
The only difference between the one from a web browser is the request type. Firefox makes a GET request and BITS makes a HEAD request. Are there any issues with Amazon S3 HEAD requests and query string authentication?
Regards, Blaz
You are probably right that a proxy is the only way around this. BITS uses the HEAD request to get a content length and decide whether or not it wants to chunk the file download. It then does the GET request to actually retrieve the file - sometimes as a whole if the file is small enough, otherwise with range headers.
If you can use a proxy or some other trick to give it any kind of response to the HEAD request, it should get unstuck. Even if the HEAD request is faked with a fictitious content length, BITS will move on to a GET. You may see duplicate GET requests in a case like this, because if the first GET request returns a content length longer than the original HEAD request, BITS may decide "oh crap, I better chunk this after all."
Given that, I'm kind of surprised it's not smart enough to recover from a 403 error on the HEAD request and still move on to the GET. What is the actual behaviour of the job? Have you tried watching it with bitsadmin /monitor? If the job is sitting in a transient error state, it may do that for around 20 mins and then ultimately recover.
Before beginning a download, BITS sends an HTTP HEAD request to the server in order to figure out the remote file's size, timestamp, etc. This is especially important for BranchCache-based BITS transfers and is the reason why server-side HTTP HEAD support is listed as an HTTP requirement for BITS downloads.
That being said, BITS bypasses the HTTP HEAD request phase, issuing an HTTP GET request right away, if either of the following conditions is true:
The BITS job is configured with the BITS_JOB_PROPERTY_DYNAMIC_CONTENT flag.
BranchCache is disabled AND the BITS job contains a single file.
Workaround (1) is the most appropriate, since it doesn't affect other BITS transfers in the system.
For workaround (2), BranchCache can be disabled through BITS' DisableBranchCache group policy. You'll need to do "gpupdate" from an elevated command prompt after making any Group Policy changes, or it will take ~90 minutes for the changes to take effect.