I want to build a wcf web service so that the client and the server would be able to transfer files between each other. Do you know how I can achieve this? I think I should turn it into a byte array but I have no idea how to do that. The file is also quite big so I must turn on streamed response.
It sounds like you're on the right track. A quick search of the interwebz yielded this link: http://www.codeproject.com/Articles/166763/WCF-Streaming-Upload-Download-Files-Over-HTTP
Your question indicates that you want to send a file from a java client to a WCFd endpoint, but the contents of your question indicate that this should be a bidirectional capability. If this is the case, then you'll need to implement a service endpoint on your client as well. As far as that is concerned, I cannot be of much help, but there are resources out there like this SO question: In-process SOAP service server for Java
As far as practical implementation, I would think that using these two links you should be able to produce some code for your server and client.
As far as reading all bytes of a file, in C# you can use: File.ReadAllBytes It should work as in the following code:
//Read The contents of the file indicated
string fileName = "/path/to/some/file";
//store the binary in a byte array
byte[] buffer = File.ReadAllBytes(fileName);
//do something with those bytes!
Be sure to use the search function in the future:
Related
I need to invoke a remote REST interface handler and submit it a file in request body. Please note that I don't control the server. I cannot change the request to be multipart, the client has to work in accordance to external specification.
So far I managed to make it work like this (omitting headers etc. for brevity):
byte[] data = readFileCompletely ();
client.target (url).request ().post (Entity.entity (data, "file/mimetype"));
This works, but will fail with huge files that don't fit into memory. And since I have no restriction on filesize, this is a concern.
Question: is it somehow possible to use streams or something similar to avoid reading the whole file into memory?
If possible, I'd prefer to avoid implementation-specific extensions. If not, a solution that works with RESTEasy (on Wildfly) is also acceptable.
ReastEasy as well as Jersey support InputStream out of the box so simply use Entity.entity(inputStream, "application/octet-stream"); or whatever Content-Type header you want to set.
You can go low-level and construct the HTTP request using a library such as the plain java.net.URLConnection.
I have not tried it myself but there is example code which reads a local file and writes it to the request stream without loading it into a byte array.
Upload files from Java client to a HTTP server
Of course this solution requires more manual coding but it should work (unless java.net.URLConnection loads the whole file into memory)
I'm developing a simple HTTPS proxy (written in Python) which receives POST/GET requests/responses, applies some transformation and finally forwards the result to the recipient.
I need to handle chunked-encoded requests/responses in a "streaming" fashion, meaning that as soon as a chunk is received the proxy transforms it and forwards it to the recipient.
Before deciding to support chunked-encoded requests, I've been using mitmproxy http://mitmproxy.org/ and it worked perfectly. Unfortunately, I noticed that it waits until the entire body is received before letting me handle the response/request.
How can I implement a proxy supporting chunked-encoded requests/responses? Has anyone of you ever done something like this?
Thanks
EDIT: MORE INFO ON MY USE CASE
I need to handle POST requests and GET responses.
In the POST request I receive a JSON object and I have to encrypt some of its values.
In the GET response I receive a JSON object and I have to decrypt some of its values.
Till now, the following code has worked perfectly:
def handle_request(self, r):
if(r.method=='POST'):
// encryption of r.get_form_urlencoded()
def handle_response(self, r):
if(r.request.method=='GET'):
// decryption of r.content
How can I do the same thing with single chunks?
EDIT: UPDATES
After evaluating different solutions, I decided to go for Squid (proxy) + ICAP (content adaptation).
I've successfully configured Squid and the performance are just great. Unfortunately, I can't find a suitable ICAP server (in Python, if possible) for doing content adaptation (modification). I thought this one https://github.com/netom/pyicap could do the job but looks like it doesn't read the body of myPOST requests.
Do you guys know a Python ICAP server that I can use together with Squid?
Thanks
The answer below is outdated. You can now pass --stream to mitmproxy, whose behaviour is explained in the mitmproxy documentation.
mitmproxy developer here. This is definitely a feature we want for mitmproxy as well, but it's not that trivial and probably not coming very soon. If you really want to implement that yourself, I can recommend two things:
If you have a very specific use case, you can employ libmproxy.protocol.http.HTTPRequest.from_stream for parsing the header and do the body processing yourself.
If you do not want to modify the request/response body, you may find it sufficient to modify mitmproxy itself. In a nutshell, you would need to read the request/response without content (see 1.), modify it to your needs, pass it to the server and then delegate control to the libmproxy.protocol.tcp (see https://github.com/mitmproxy/mitmproxy/blob/master/libmproxy/proxy/server.py#L169)
If you have further questions, don't hesistate to ask here or on mitmproxy's IRC channel.
Re Comment #1:
You can't take too much out of mitmproxy, but at least you get delegate the header parsing & processing.
# ...accept request, socket.makefile() etc...
req = HTTPRequest.from_stream(client_conn.rfile, include_content=False)
# manually forward to the server (req._assemble_head())
# manually receive response body chunk by chunk and forward it to the server, see
# https://github.com/mitmproxy/netlib/blob/master/netlib/http.py#L98
resp = HTTPResponse.from_stream(server_conn.rfile, include_content=False)
# manually forward headers
# manually process body and forward
That being said, this is a fairly complex topic. Eventually, you're better off hacking that directly into libmproxy.protocol.http.HTTPHandler.
Another option, depending on your use case again: Use mitmproxy, set the conntype to tcp and forward traffic as-is and use regex replacements on the content in libmproxy.protocol.tcp . Probably the easiest way, but the most hacky one.
If you can provide some context, I may guide you further in the right direction.
Re Comment #2:
Before we get to the main part: JSON is a really bad choice for streaming/chunking as long as you don't want to encrypt the complete JSON object and treat it as a single string. You should definitely consider something like tnetstrings if you only want to encrypt parts.
Apart from that, hooking into read_chunk works, but first you need to get to the point where you can actually receive chunks over the line. Then, it's as simple as reading the single chunks, encrypting them and forwarding them.
I have an app on the roll that requires me to upload files from it to a webservice. I am basically a noob in file transfer and in app-webservice comunications and would appreciate a direction in which to go.
The first question is how to encode the file? The webservice has to interpret the data sent so in which way do i do it? I don't own the webservice so i don't know how they pick up files, but i assume in a XML/JSON format.
The second question is how am i to send it? Having a XML object, how do i send it to, let's say, "http://www.website.com/path/to/upload". Is Async available?
If i'm missing or am wrong about a step (or 10) please correct me! Your help is much appreciated!
1: If it is a XML you will need to parse it, and create objet from it. If it's a Json, i recommend you https://github.com/johnezang/JSONKit JSONKIt, which is really awesome.
2: You 'll need get that data from webService and ASIHTTP is a really great tool to get fast web service call. The doc is really clear and comprehensible http://allseeing-i.com/ASIHTTPRequest/ is a must have.
3: You send your data either by url (get) or post.
Hope for you it's not into XML format, this is really painful, believe me
I have an application to upload files to a server. I am using nettcpbinding and wshttpbinding. When the files is larger than 200 MB, I get a memory exception. Working around this, I have seen people recommend streaming, and of course it works with nettcpbinding for large files (>1GB), but when using wshttpbinding, what would be the approach?? Should I change to basichttpbinding?? what?? Thanks.
I suggest you expose another end point just to upload such large size data. This can have a binding that supports streaming. In our previous project we needed to do file uploads to server as part of business process. We ended up creating 2 endpoints one just dedicated to file upload, and another for all other business functionality.
The streaming data service can be a generic service to stream any data to the server and maybe return a token for identifying the data on server.For subsequent requests this token can be passed along to manipulate the data on server.
If you don't want to (or cannot because of legit reasons) change the binding nor use streaming, what you can do is have some method with a signature along the lines of the following:
void UploadFile(string fileName, long offset, byte[] data)
Instead of sending the whole file, you send little packets, and tell where the data should be placed. You can add more data, of course, like the whole filesize, CRC of the file to know if the transfer was successful, etc.
I have WCF service that returns an object that contains an array of bytes that can be saved as a PDF (my .NET component of this implementation works like a charm, the bytes are saved to a stream and they write out the PDF without issue). I can modify the WCF to return just the bytes if necessary. My question is: How can I get those bytes to the asp page so I can save them. I've spent several hours searching the web for methods/help/hints/anything, but have been relatively unsuccessful. I have a method for writing files in asp, but it takes an array of bytes, but I dont know how to get the bytes from the WCF. The only way I've been able to communicate with WCF is to use SOAP type calls and parse the XML that is returned (which is fine for the rest of the page, since I just need the text values that are returned), but this one needs the bytes returned to save the file. (And no, the WCF cannot save the file on its own, it lives on a different server and does not have access to share a drive map or something like that).
Any thoughts/hints/tips/etc would be GREATLY appreciated, Im going insane with this project!
Thanks in advance everyone!
It sounds like you are calling the WCF service and getting a string in return, if not see Calling WCF service by VBScript.
Once you have the string, convert it to a byte sub type(using ChrB()) and save it to a file or write it out to the client.
I found this code sample at Create and work with binary data in ASP/VBScript:
Function StringToMultiByte(S)
Dim i, MultiByte
For i=1 To Len(S)
MultiByte = MultiByte & ChrB(Asc(Mid(S,i,1)))
Next
StringToMultiByte = MultiByte
End Function