How to avoid OutOfMemoryError when uploading a large file using Jersey client - file-upload

Question

I am using Jersey client for http-based request. It works well if the file is small but run into error when I post a file with size of 700M:
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
at sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:61)
at com.sun.jersey.api.client.CommittingOutputStream.write(CommittingOutputStream.java:90)
at com.sun.jersey.core.util.ReaderWriter.writeTo(ReaderWriter.java:115)
at com.sun.jersey.core.provider.AbstractMessageReaderWriterProvider.writeTo(AbstractMessageReaderWriterProvider.java:76)
at com.sun.jersey.core.impl.provider.entity.FileProvider.writeTo(FileProvider.java:103)
at com.sun.jersey.core.impl.provider.entity.FileProvider.writeTo(FileProvider.java:64)
at com.sun.jersey.multipart.impl.MultiPartWriter.writeTo(MultiPartWriter.java:224)
at com.sun.jersey.multipart.impl.MultiPartWriter.writeTo(MultiPartWriter.java:71)
at com.sun.jersey.api.client.RequestWriter.writeRequestEntity(RequestWriter.java:300)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:204)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:147)
at com.sun.jersey.api.client.Client.handle(Client.java:648)
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:680)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:568)
at TestHttpRequest.main(TestHttpRequest.java:42)
here is my code:
ClientConfig cc = new DefaultClientConfig();
Client client = Client.create(cc);
WebResource resource = client.resource("http://localhost:8080/JerseyWithServletTest/helloworld");
FormDataMultiPart form = new FormDataMultiPart();
File file = new File("E:/CN_WXPPSP3_v312.ISO");
form.field("username", "ljy");
form.field("password", "password");
form.field("filename", file.getName());
form.bodyPart(new FileDataBodyPart("file", file, MediaType.MULTIPART_FORM_DATA_TYPE));
ClientResponse response = resource.type(MediaType.MULTIPART_FORM_DATA).post(ClientResponse.class, form);

Answer 1

You could use streams.Try something like this on the client:
InputStream fileInStream = new FileInputStream(fileName);
String sContentDisposition = "attachment; filename=\"" + fileName.getName()+"\"";
WebResource fileResource = a_client.resource(a_sUrl);
ClientResponse response = fileResource.type(MediaType.APPLICATION_OCTET_STREAM)
.header("Content-Disposition", sContentDisposition)
.post(ClientResponse.class, fileInStream);
with resource like this on the server:
#PUT
#Consumes("application/octet-stream")
public Response putFile(#Context HttpServletRequest a_request,
#PathParam("fileId") long a_fileId,
InputStream a_fileInputStream) throws Throwable
{
// Do something with a_fileInputStream
// etc

Answer 2

In order for your code not to depend on the size of the uploaded file, you need:
Use streams
Define the chuck size of the jersey client. For example:
client.setChunkedEncodingSize(1024);
Server:
#POST
#Path("/upload/{attachmentName}")
#Consumes(MediaType.APPLICATION_OCTET_STREAM)
public void uploadAttachment(#PathParam("attachmentName") String attachmentName, InputStream attachmentInputStream) {
// do something with the input stream
}
Client:
...
client.setChunkedEncodingSize(1024);
WebResource rootResource = client.resource("your-server-base-url");
File file = new File("your-file-path");
InputStream fileInStream = new FileInputStream(file);
String contentDisposition = "attachment; filename=\"" + file.getName() + "\"";
ClientResponse response = rootResource.path("attachment").path("upload").path("your-file-name")
.type(MediaType.APPLICATION_OCTET_STREAM).header("Content-Disposition", contentDisposition)
.post(ClientResponse.class, fileInStream);

Answer 3

Below is the code for uploading a (potentially large) file with chunked transfer encoding (i.e. streams) using Jersey 2.11.
Maven:
<properties>
<jersey.version>2.11</jersey.version>
</properties>
<dependencies>
<dependency>
<groupId>org.glassfish.jersey.core</groupId>
<artifactId>jersey-client</artifactId>
<version>${jersey.version}</version>
</dependency>
<dependency>
<groupId>org.glassfish.jersey.media</groupId>
<artifactId>jersey-media-multipart</artifactId>
<version>${jersey.version}</version>
</dependency>
<dependencies>
Java:
Client client = ClientBuilder.newClient(clientConfig);
client.property(ClientProperties.REQUEST_ENTITY_PROCESSING, "CHUNKED");
WebTarget target = client.target(SERVICE_URI);
InputStream fileInStream = new FileInputStream(inFile);
String contentDisposition = "attachment; filename=\"" + inFile.getName() + "\"";
System.out.println("sending: " + inFile.length() + " bytes...");
Response response = target
.request(MediaType.APPLICATION_OCTET_STREAM_TYPE)
.header("Content-Disposition", contentDisposition)
.header("Content-Length", (int) inFile.length())
.put(Entity.entity(fileInStream, MediaType.APPLICATION_OCTET_STREAM_TYPE));
System.out.println("Response status: " + response.getStatus());

Answer 4

In my case (Jersey 2.23.2) rschmidt13's solution gave this warning:
WARNING: Attempt to send restricted header(s) while the [sun.net.http.allowRestrictedHeaders] system property not set. Header(s) will possibly be ignored.
This can be solved adding the following line:
System.setProperty("sun.net.http.allowRestrictedHeaders", "true");
However I think a cleaner solution can be obtained using the StreamingOutput interface.
I post a complete example hoping it could be useful.
Client (File upload)
WebTarget target = ClientBuilder.newBuilder().build()
.property(ClientProperties.CHUNKED_ENCODING_SIZE, 1024)
.property(ClientProperties.REQUEST_ENTITY_PROCESSING, "CHUNKED")
.target("<your-url>");
StreamingOutput out = new StreamingOutput() {
#Override
public void write(OutputStream output) throws IOException,
WebApplicationException {
try (FileInputStream is = new FileInputStream(file)) {
int available;
while ((available = is.available()) > 0) {
// or use a buffer
output.write(is.read());
}
}
}
};
Response response = target.request().post(Entity.text(out));
Server
#Path("resourcename")
public class MyResource {
#Context
HttpServletRequest request;
#POST
#Path("thepath")
public Response upload() throws IOException, ServletException {
try (InputStream is = request.getInputStream()) {
// ...
}
}
}

Answer 5

If possible, can you split the file you send into smaller parts? This will reduce memory usage, but you need to change the code on both sides of the uploading/downloading code.
If you can't, then your heap space is too low, try increasing it with this JVM parameter. In your application server add/change the Xmx JVM options. For example
-Xmx1024m
to set Max Heap Space to 1Gb

Answer 6

#Consumes("multipart/form-data")
#Produces(MediaType.TEXT_PLAIN + ";charset=utf-8")
public String upload(MultipartFormDataInput input, #QueryParam("videoId") String videoId,
#Context HttpServletRequest a_request) {
String fileName = "";
for (InputPart inputPart : input.getParts()) {
try {
MultivaluedMap<String, String> header = inputPart.getHeaders();
fileName = getFileName(header);
// convert the uploaded file to inputstream
InputStream inputStream = inputPart.getBody(InputStream.class, null);
// write the inputStream to a FileOutputStream
OutputStream outputStream = new FileOutputStream(new File("/home/mh/Téléchargements/videoUpload.avi"));
int read = 0;
byte[] bytes = new byte[1024];
while ((read = inputStream.read(bytes)) != -1) {
outputStream.write(bytes, 0, read);
}
System.out.println("Done!");
} catch (IOException e) {
e.printStackTrace();
return "ko";
}
}