I made a simple spring-boot application that returns a static json response for all requests.
When the app gets a request with a large payload (~5mb json, 1 TP ), the client receives the following error:
java.net.SocketException: Broken pipe (Write failed)
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
I have tried increasing every limit i could - here are my tomcat settings:
spring.http.multipart.max-file-size=524288000
spring.http.multipart.max-request-size=524288000
spring.http.multipart.enabled=true
server.max-http-post-size=10000000
server.connection-timeout=30000
server.tomcat.max-connections=15000
server.tomcat.max-http-post-size=524288000
server.tomcat.accept-count=10000
server.tomcat.max-threads=200
server.tomcat.min-spare-threads=200
What can I do to make this simple spring boot with just one controller, to handle such payloads successfully?
This springboot application and the client sending the large payload run on an 8-core machine with 16gb ram. So resources shouldn't be a problem.
This was because the controller was returning a response without consuming the request body.
So the server closes the connection as soon as it receives the request, without consuming the full request body. The client still hadn't finished sending the request and the server closed the connection before that.
Solution:
1. Read the full request body in your code
2. Set tomcat's maxSwallowSize to a higher value (default : 2mb)
server.tomcat.max-swallow-size=10MB
Related
I want to run code when a HTTP client has finished downloading a file from Ktor Server. The most simple approach does not work:
routing {
get("/download") {
call.response.header(
ContentDisposition,
Attachment.withParameter(FileName, file.fileName).toString()
)
call.respondFile(file)
// client has not finished downloaded when we reach this point
}
}
I already tried intercepting pipelines, writing out the file in byte arrays myself, but all that detects a completed download too soon. If I for example stop the webserver and related services too soon, the client isn't able to actually complete the download.
So is there a way to reliably detect when the client has ACKed the last bytes of the file?
I use ApiKit to receive queries. Occasionally I get the following line in a log file:
WARN org.mule.module.http.internal.listener.grizzly.ResponseCompletionHandler - HTTP response sending task failed with error: Locally closed
It seems that in this case the integration has not sent a response to the party that called the integration. I thought that there might be a sort of timeout before ApiKit closes the connection to caller but based on timestamps that might not be the case as everything happens within a second.
In this case the payload is sent to Artemis queue before this warning and despite the warning the message is read from Artemis normally and the whole flow works otherwise just fine besides this warning and not sending the response.
So, am I correct when I think that this warning is an indication why the response is not sent? In addition to that what can be done to prevent this situation?
I was making a simple file upload&download service and found out that, as far as I understand, Netty doesn't release direct buffers until request processing is over. As a result, I can't upload bigger files.
I was trying to make sure that the problem is not inside my code, so I created the most simple tiny Ktor application:
routing {
post("upload") {
call.receiveMultipart().forEachPart {}
call.respond(HttpStatusCode.OK)
}
}
The default direct memory size is about 3Gb, to make test simpler I limit it with:
System.setProperty("io.netty.maxDirectMemory", (10 * 1024 * 1024).toString())
before starting the NettyApplicationEngine.
Now if I upload a large file, for example with httpie, I got "Connection reset":
http -v --form POST http://localhost:42195/upload file#/tmp/FileStorageLoadTest-test-data1.tmp
http: error: ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) while doing POST request to URL: http://localhost:42195/upload
On the server side there is no information about the problem except for the "java.io.IOException: Broken delimiter occurred" exception. But if I put the breakpoint in NettyResponsePipeline#processCallFailed, the real exception is:
io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 65536 byte(s) of direct memory (used: 10420231, max: 10485760)
It is a pity that this exception is not logged.
Also, I found out that the same code works without problems if I use Jetty engine instead.
Environment:
Ubuntu Linux
Java 8
Ktor=1.2.5
netty-transport-native-epoll=4.1.43.Final
(but if Netty started without native-epoll support, the problem is the same)
I'm attempting to post some json to an express.js endpoint. If the size of the json is less than 64k, then it succeeds just fine. If it exceeds 64k, the request is never completely received by the server. The problem only occurs when running express directly locally. When running on heroku, the request proceeds without issue.
The problem is seen across MacOS, Linux (ubuntu 19), and Windows. It is present when using Chrome, Firefox, or Safari.
When I make requests using postman, the request fails.
If I make the request using curl, the request succeeds.
If I make the request after artificially throttling chrome to "slow 3G" levels in network settings, the request succeeds.
I've traced through express and discovered that the problem appears when attempting to parse the body. The request gets passed to body-parser.json() which in turns called getRawBody to get the Buffer from the request.
getRawBody is processing the incoming request stream and converting it into a buffer. It receives the first chunk of the request just fine, but never receives the second chunk. Eventually the request continues parsing with an empty buffer.
The size limit on bodyparser is set to 100mb, so it is not the problem. getRawBody never returns, so body-parser never gets a crack at it.
If I'm logging the events from getRawBody I can see the first chunk come in, but no other events are fired.
Watching wireshark logs, all the data is getting sent over the wire. But it looks like for some reason, express is not receiving all the chunks. I think it's got to be due to how express is processing the packets, have no idea how to proceed.
In the off chance anyone in the future is running into the same thing: The root problem in this case was that we were overwriting req.socket with our socket.io client. req.socket is used by node internally to transfer data. We were overwriting such that the first packets would get through, but not subsequent packets. So if the request was processed sufficiently quickly, all was well.
tl;dr: Don't overwrite req.socket.
How do I find the HTTP timeout set on the WebLogic 8.1 application server?
I only have Weblogic 9 and 10 available but on those platforms, you can go to the console, click on the name of your domain, then (in "Configuration" tab) "Web Applications". There you will have 3 parameters:
Post Timeout: The amount of time this server waits between receiving chunks of data in an HTTP POST data before it times out. (This is used to prevent denial-of-service attacks that attempt to overload the server with POST data.)
Maximum Post Time: Max Post Time (in seconds) for reading HTTP POST data in a servlet request. MaxPostTime < 0 means unlimited
Maximum Post Size: The maximum post size this server allows for reading HTTP POST data in a servlet request. A value less than 0 indicates an unlimited size.
However, there might be other parameters involved depending on what your problem exactly is.