Now I have ktor server is based on netty.
When I do to long GET request (about 9300 characters (mostly in query params)), ktor answers Unhandled: GET - /bad-request.
If I reduce length of url, it works fine.
In your embedded server config you can provide a function “httpServerCodec” to create HttpServerCodec (https://netty.io/4.1/api/io/netty/handler/codec/http/HttpServerCodec.html) in which you can set the maxInitialLineLength property.
embeddedServer(Netty, configure = {
// Size of the queue to store [ApplicationCall] instances that cannot be immediately processed
requestQueueLimit = 16
// Do not create separate call event group and reuse worker group for processing calls
shareWorkGroup = false
// User-provided function to configure Netty's [ServerBootstrap]
configureBootstrap = {
// ...
}
httpServerCodec = {
HttpServerCodec(.......)
}
// Timeout in seconds for sending responses to client
responseWriteTimeoutSeconds = 10
}) {
// ...
}.start(true)
Related
We have a kotlin code like the following, I am trying to disable the options method for the API's as follows using Javalin(3.12.0), but it is resulting in blocking all the other methods like get and post as well. What is that I am missing here?
val app = Javalin.create {
it.defaultContentType = "application/json"
it.enableWebjars()
it.addStaticFiles("", Location.CLASSPATH)
it.enableCorsForAllOrigins()
it.dynamicGzip = true
}
app.options("/*") {ctx -> ctx.status(405)}
app.routes {
path("/auth") {
post("/login") {
Auth.doLogin(it)
}
get("/metrics") {
val results = getData()
it.json(results)
}
}
Also there are 2 questions
1.want to implement the ratelimit for the get APi's for 20 request for an hour using the below code
app.get("/") { ctx ->
RateLimit(ctx).requestPerTimeUnit(5, TimeUnit.MINUTES) // throws if rate limit is exceeded
ctx.status("Hello, rate-limited World!")
}
How to achieve it?
How to restrict the jetty server version to display when the API call is made?
For Jetty...
There is only 1 Rate Limit concept in Jetty, and that's the org.eclipse.jetty.server.AcceptRateLimit, added as a Jetty Container LifeCycle bean to the ServerConnector, it cannot adjust rates for specific request endpoints, only for the entire connector.
If you want specific endpoint rates, then the org.eclipse.jetty.servlets.QoSFilter is the way that's done with Jetty.
The org.eclipse.jetty.server.HttpConfiguration for the org.eclipse.jetty.server.ServerConnector contains the controls to enable/disable the server announcement.
See
HttpConfiguration.setSendServerVersion(boolean)
HttpConfiguration.setSendXPoweredBy(boolean)
HttpConfiguration.setSendDateHeader(boolean)
I am creating simple HTTP request with ktor-client (ApacheHttpClient engine)
val client = HttpClient(Apache) {
engine {
followRedirects = false
this#HttpClient.expectSuccess = false
}
}
and using it to submit a form
client.submitForm<HttpResponse>(
url = "https://foo.com/login",
formParameters = Parameters.build {
append("_username", username)
append("_password", password)
})
In logs, I can see a correct response with 302-redirection which I want to get and obtain a cookie from it. But instead, I see that the client moves on and makes several more requests and finally fails with:
io.ktor.client.features.SendCountExceedException: Max send count 20 exceeded
How can I completely disable 302-based redirections in ktor-client?
ktor-client follows redirects by default, to prevent infinite redirects use:
val client = HttpClient(HttpClientEngine) {
followRedirects = false
}
I need to get the items from all pages of a pageable REST API. I also need to start processing items, as soon as they are available, not needing to wait for all the pages to be loaded. In order to do so, I'm using Spring WebFlux and its WebClient, and want to return Flux<Item>.
Also, the REST API I'm using is rate limited, and each response to it contains headers with details on the current limits:
Size of the current window
Remaining time in the current window
Request quota in window
Requests left in current window
The response to a single page request looks like:
{
"data": [],
"meta": {
"pagination": {
"total": 10,
"current": 1
}
}
}
The data array contains the actual items, while the meta object contains pagination info.
My current solution first does a "dummy" request, just to get the total number of pages, and the rate limits.
Mono<T> paginated = client.get()
.uri(uri)
.exchange()
.flatMap(response -> {
HttpHeaders headers = response.headers().asHttpHeaders();
Limits limits = new Limits();
limits.setWindowSize(headers.getFirst("X-Window-Size"));
limits.setWindowRemaining(headers.getFirst("X-Window-Remaining"));
limits.setRequestsQuota(headers.getFirst("X-Requests-Quota");
limits.setRequestsLeft(headers.getFirst("X-Requests-Remaining");
return response.bodyToMono(Paginated.class)
.map(paginated -> {
paginated.setLimits(limits);
return paginated;
});
});
Afterwards, I emit a Flux containing page numbers, and for each page, I do a REST API request, each request being delayed enough so it doesn't get past the limit, and return a Flux of extracted items:
return paginated.flatMapMany(paginated -> {
return Flux.range(1, paginated.getMeta().getPagination().getTotal())
.delayElements(Duration.ofMillis(paginated.getLimits().getWindowRemaining() / paginated.getLimits().getRequestsQuota()))
.flatMap(page -> {
return client.get()
.uri(pageUri)
.retrieve()
.bodyToMono(Item.class)
.flatMapMany(p -> Flux.fromIterable(p.getData()));
});
});
This does work, but I'm not happy with it because:
It does initial "dummy" request to get the number of pages, and then
repeats the same request to get the actual data.
It gets rate limits only with the initial request, and assumes the
limits won't change (eg, that it's the only one using the API) -
which may not be true, in which case it will get an error that it
exceeded the limit.
So my question is how to refactor it so it doesn't need the initial request (but rather get limits, page numbers and data from the first request, and continue through all pages, while updating (and respecting) the limits.
I think this code will do what you want. The idea is to make a flux that make a call to your resource server, but in the process to handle the response, to add a new event on that flux to be able to make the call to next page.
The code is composed of:
A simple wrapper to contains the next page to call and the delay to wait before executing the call
private class WaitAndNext{
private String next;
private long delay;
}
A FluxProcessor that will make HTTP call and process the response:
FluxProcessor<WaitAndNext, WaitAndNext> processor= DirectProcessor.<WaitAndNext>create();
FluxSink<WaitAndNext> sink=processor.sink();
processor
.flatMap(x-> Mono.just(x).delayElement(Duration.ofMillis(x.delay)))
.map(x-> WebClient.builder()
.baseUrl(x.next)
.defaultHeader("Accept","application/json")
.build())
.flatMap(x->x.get()
.exchange()
.flatMapMany(z->manageResponse(sink, z))
)
.subscribe(........);
I split the code with a method that only manage response: It simply unwrap your data AND add a new event to the sink (the event beeing the next page to call after the given delay)
private Flux<Data> manageResponse(FluxSink<WaitAndNext> sink, ClientResponse resp) {
if (resp.statusCode()!= HttpStatus.OK){
sink.error(new IllegalStateException("Status code invalid"));
}
WaitAndNext wn=new WaitAndNext();
HttpHeaders headers=resp.headers().asHttpHeaders();
wn.delay= Integer.parseInt(headers.getFirst("X-Window-Remaining"))/ Integer.parseInt(headers.getFirst("X-Requests-Quota"));
return resp.bodyToMono(Item.class)
.flatMapMany(p -> {
if (p.paginated.current==p.paginated.total){
sink.complete();
}else{
wn.next="https://....?page="+(p.paginated.current+1);
sink.next(wn);
}
return Flux.fromIterable(p.getData());
});
}
Now we just need to initialize the system by calling for the retrieval of the first page with no delay:
WaitAndNext wn=new WaitAndNext();
wn.next="https://....?page=1";
wn.delay=0;
sink.next(wn);
How can the following code be re-written to work with the CF Workers feature?
# Start
if(req.url ~ "^/app" ) {
set req.url = regsub(req.url, "^/app/", "/");
set req.http.X-DR-SUBDIR = "app";
}
#end condition
Cloudflare Workers implements the Service Worker standard, so you will need to reimplement the VCL code snippet you posted in terms of a Service Worker.
Before I show you how to do that, consider what happens when a request for https://example.com/apple arrives at the proxy. I would expect the first regex for ^/app to match, but the second one for ^/app/ not to match -- i.e., the request would be passed through with no change to the URL, but with the addition of an X-DR-SUBDIR: app header.
I suspect that behavior is a bug, so I'll first implement a worker as if the first regex were ^/app/.
addEventListener("fetch", event => {
let request = event.request
// Unlike VCL's req.url, request.url is an absolute URL string,
// so we need to parse it to find the start of the path. We'll
// need it as a separate object in order to mutate it, as well.
let url = new URL(request.url)
if (url.pathname.startsWith("/app/")) {
// Rewrite the URL and set the X-DR-SUBDIR header.
url.pathname = url.pathname.slice("/app".length)
// Copying the request with `new Request()` serves two purposes:
// 1. It is the only way to actually change the request's URL.
// 2. It makes `request.headers` mutable. (The headers property
// on the original `event.request` is always immutable, meaning
// our call to `request.headers.set()` below would throw.)
request = new Request(url, request)
request.headers.set("X-DR-SUBDIR", "app")
}
event.respondWith(fetch(request))
})
To revisit the https://example.com/apple case, if we really wanted a Cloudflare Worker which pedantically reproduces the VCL snippet's behavior, we could change these lines (comments elided):
if (url.pathname.startsWith("/app/")) {
url.pathname = url.pathname.slice("/app".length)
// ...
}
to these:
if (url.pathname.startsWith("/app")) {
if (url.pathname.startsWith("/app/")) {
url.pathname = url.pathname.slice("/app".length)
}
// ...
}
I am a newbie to WebRTC. I am building an application that enables users to view each other's video stream, as well as exchange files. The audio/video part is implemented and working. The problem is I need to add the ability to exchange files now. I am using the below code to initialize the PeerConnection object
var connection = _getConnection(partnerId);
console.log("Initiate offer")
// Add our audio/video stream
connection.addStream(stream);
// Send an offer for a connection
connection.createOffer(function (desc) { _createOfferSuccess(connection, partnerId, desc) }, function (error) { console.log('Error creating session description: ' + error); });
_getConnection creates a new RTCPeerConnection object using
var connection = new RTCPeerConnection(iceconfig);
i.e., with no explicit constraints. It also initializes the different event handlers on it. Right after this, I attach the audio/video stream to this connection. I also cache these connections using the partner id, so I can use it later.
The question is, can I later recall the connection object from the cache, add a data channel to it using something like
connection.createDataChannel("DataChannel", dataChannelOptions);
And use it to share files, or do I have to create a new RTCPeerConnection object and attach the data channel to it?
You certainly do not have to create a another PeerConnection for file transfer alone. Existing PeerConnection can utilize RTCDatachannel with behaves like traditional websocket mechanism ( ie 2 way communication without a central server )
`var PC = new RTCPeerConnection();
//specifying options for my datachannel
var dataChannelOptions = {
ordered: false, // unguaranted sequence
maxRetransmitTime: 2000, // 2000 miliseconds is the maximum time to try and retrsanmit failed messages
maxRetransmits : 5 // 5 is the number of times to try to retransmit failed messages , other options are negotiated , id , protocol
};
// createing data channel using RTC datachannel API name DC1
var dataChannel = PC.createDataChannel("DC1", dataChannelOptions);
dataChannel.onerror = function (error) {
console.log("DC Error:", error);
};
dataChannel.onmessage = function (event) {
console.log("DC Message:", event.data);
};
dataChannel.onopen = function () {
dataChannel.send(" Sending 123 "); // you can add file here in either strings/blob/array bufers almost anyways
};
dataChannel.onclose = function () {
console.log("DC is Closed");
};
`
PS : while sending files over datachannel API , it is advisable to break down the files into small chunks beforehand . I suggest chunk size of almost 10 - 15 KB .