We have a kotlin code like the following, I am trying to disable the options method for the API's as follows using Javalin(3.12.0), but it is resulting in blocking all the other methods like get and post as well. What is that I am missing here?
val app = Javalin.create {
it.defaultContentType = "application/json"
it.enableWebjars()
it.addStaticFiles("", Location.CLASSPATH)
it.enableCorsForAllOrigins()
it.dynamicGzip = true
}
app.options("/*") {ctx -> ctx.status(405)}
app.routes {
path("/auth") {
post("/login") {
Auth.doLogin(it)
}
get("/metrics") {
val results = getData()
it.json(results)
}
}
Also there are 2 questions
1.want to implement the ratelimit for the get APi's for 20 request for an hour using the below code
app.get("/") { ctx ->
RateLimit(ctx).requestPerTimeUnit(5, TimeUnit.MINUTES) // throws if rate limit is exceeded
ctx.status("Hello, rate-limited World!")
}
How to achieve it?
How to restrict the jetty server version to display when the API call is made?
For Jetty...
There is only 1 Rate Limit concept in Jetty, and that's the org.eclipse.jetty.server.AcceptRateLimit, added as a Jetty Container LifeCycle bean to the ServerConnector, it cannot adjust rates for specific request endpoints, only for the entire connector.
If you want specific endpoint rates, then the org.eclipse.jetty.servlets.QoSFilter is the way that's done with Jetty.
The org.eclipse.jetty.server.HttpConfiguration for the org.eclipse.jetty.server.ServerConnector contains the controls to enable/disable the server announcement.
See
HttpConfiguration.setSendServerVersion(boolean)
HttpConfiguration.setSendXPoweredBy(boolean)
HttpConfiguration.setSendDateHeader(boolean)
Related
I am trying to call a twitter endpoint that gives you a constant streams of json results back to the client
https://documenter.getpostman.com/view/9956214/T1LMiT5U#977c147d-0462-4553-adfa-d7a1fe59c3ec
I try to make a call to the endpoint like this
val url = "https://api.twitter.com/2/tweets/search/stream"
_streamChannel = _client.get<ByteReadChannel>(token, url) //Stops here
val byteBufferSize = 1024
val byteBuffer = ByteArray(byteBufferSize)
_streamChannel?.let {
while (_streamChannel!!.availableForRead > 0) {
_streamChannel!!.readAvailable(byteBuffer, 0, byteBufferSize)
val s = String(byteBuffer)
parseStreamResponseString(s).forEach {
emit(Response.Success(it))
}
}
}
my client.get code is this
suspend inline fun <reified T> get(authKey: String, url: String): T? {
val response = _client.get<HttpResponse>(url) {
header("Authorization", "Bearer $authKey")
}
when (response.status.value) {
in 300..399 -> throw RedirectResponseException(response)
in 400..499 -> throw ClientRequestException(response)
in 500..599 -> throw ServerResponseException(response)
}
if (response.status.value >= 600) {
throw ResponseException(response)
}
return response.receive<T>()
}
When I make the request it just sits there in what I am assuming is waiting for the full response to be returned before giving it to me.
Edit
I also tried using scoped streaming but it just sits at the line readAvailable I know there are messages coming through because when I run the request via cURL I am constantly getting data
_client.get<HttpStatement> {
header("Authorization", "Bearer $authKey")
url(urlString)
contentType(ContentType.Application.Json)
method = HttpMethod.Get
}.execute {
val streamChannel = it.receive<ByteReadChannel>()
val byteBufferSize = 1024
val byteBuffer = ByteArray(byteBufferSize)
streamChannel.readAvailable(byteBuffer, 0, byteBufferSize) // Stops here
val s = String(byteBuffer)
}
How do I process a constant stream of json data using Ktor?
As far as I am aware, the Ktor client does note expose access to the IO buffer of the request in the way that twitter's streaming API requires.
From the twitter documentation here:
Some HTTP client libraries only return the response body after the connection has been closed by the server. These clients will not work for accessing the Streaming API. You must use an HTTP client that will return response data incrementally. Most robust HTTP client libraries will provide this functionality. The Apache HttpClient will handle this use case, for example.
What you are doing is telling Ktor that the thing you are getting is a ByteReadChannel, and so, once the request closes (which will never happen with this twitter endpoint) the Ktor client would attempt to use whatever plugin (json for example) you were using to parse that data into a ByteReadChannel. It would also not be able to do that, because the data you are getting from twitter is not a ByteReadChannel, it is a new line seperated list of json objects.
I have a Spring Cloud Gateway service sitting in front of a number of backend services but currently it does not log very much. I wanted to log responses from backend services whenever they return unexpected response status codes but I've hit on the following problem.
I can log the response body for requests using a modifyResponseBody filter and a RewriteFunction like so:
.route("test") { r ->
r.path("/httpbin/**")
.filters { f ->
f.modifyResponseBody(String::class.java, String::class.java){ exchange, string ->
if(e.response.statusCode.is2xxSuccessful)println("Server error")
Mono.just(string)
}
}.uri("http://httpbin.org")
}
My issue with this method is that I'm parsing the response ByteArray to a String on every response, with the overhead that implies, even though I'm just using the body string on a small subset of those responses.
I've tried instead to implement a custom filter like so:
.route("test2") {r ->
r.path("/httpbin2/**")
.filters { f ->
f.filter(LogUnexpectedResponseFilter()
.apply(LogUnexpectedResponseFilter.Config(listOf(HttpStatus.OK))))
}.uri("http://httpbin.org")
}
class LogUnexpectedResponseFilter : AbstractGatewayFilterFactory<LogUnexpectedResponseFilter.Config>() {
val logger = LoggerFactory.getLogger(this::class.java)
override fun apply(config: Config?): GatewayFilter {
return GatewayFilter { exchange, chain ->
logger.info("running custom filter")
if (config?.errors!!.contains(exchange.response.statusCode)){
return#GatewayFilter ModifyResponseBodyGatewayFilterFactory().apply(ModifyResponseBodyGatewayFilterFactory.Config()
.setInClass(String::class.java)
.setOutClass(String::class.java)
.setRewriteFunction(String::class.java, String::class.java) { _, body ->
logger.error(body)
Mono.just(body)
}).filter(exchange, chain)
} else {
chain.filter(exchange)
}
}
}
data class Config(val errors: List<HttpStatus>)
}
What this is supposed to do is simply let the request pass through on most requests but apply the log filter on those that I have configured it to (although in this example I'm having it log on 200 status responses).
What I'm seeing when I debug is that it correctly applies the right filter but the RewriteFunction inside it isn't being run at all. What am I missing?
Now I have ktor server is based on netty.
When I do to long GET request (about 9300 characters (mostly in query params)), ktor answers Unhandled: GET - /bad-request.
If I reduce length of url, it works fine.
In your embedded server config you can provide a function “httpServerCodec” to create HttpServerCodec (https://netty.io/4.1/api/io/netty/handler/codec/http/HttpServerCodec.html) in which you can set the maxInitialLineLength property.
embeddedServer(Netty, configure = {
// Size of the queue to store [ApplicationCall] instances that cannot be immediately processed
requestQueueLimit = 16
// Do not create separate call event group and reuse worker group for processing calls
shareWorkGroup = false
// User-provided function to configure Netty's [ServerBootstrap]
configureBootstrap = {
// ...
}
httpServerCodec = {
HttpServerCodec(.......)
}
// Timeout in seconds for sending responses to client
responseWriteTimeoutSeconds = 10
}) {
// ...
}.start(true)
I need to get the items from all pages of a pageable REST API. I also need to start processing items, as soon as they are available, not needing to wait for all the pages to be loaded. In order to do so, I'm using Spring WebFlux and its WebClient, and want to return Flux<Item>.
Also, the REST API I'm using is rate limited, and each response to it contains headers with details on the current limits:
Size of the current window
Remaining time in the current window
Request quota in window
Requests left in current window
The response to a single page request looks like:
{
"data": [],
"meta": {
"pagination": {
"total": 10,
"current": 1
}
}
}
The data array contains the actual items, while the meta object contains pagination info.
My current solution first does a "dummy" request, just to get the total number of pages, and the rate limits.
Mono<T> paginated = client.get()
.uri(uri)
.exchange()
.flatMap(response -> {
HttpHeaders headers = response.headers().asHttpHeaders();
Limits limits = new Limits();
limits.setWindowSize(headers.getFirst("X-Window-Size"));
limits.setWindowRemaining(headers.getFirst("X-Window-Remaining"));
limits.setRequestsQuota(headers.getFirst("X-Requests-Quota");
limits.setRequestsLeft(headers.getFirst("X-Requests-Remaining");
return response.bodyToMono(Paginated.class)
.map(paginated -> {
paginated.setLimits(limits);
return paginated;
});
});
Afterwards, I emit a Flux containing page numbers, and for each page, I do a REST API request, each request being delayed enough so it doesn't get past the limit, and return a Flux of extracted items:
return paginated.flatMapMany(paginated -> {
return Flux.range(1, paginated.getMeta().getPagination().getTotal())
.delayElements(Duration.ofMillis(paginated.getLimits().getWindowRemaining() / paginated.getLimits().getRequestsQuota()))
.flatMap(page -> {
return client.get()
.uri(pageUri)
.retrieve()
.bodyToMono(Item.class)
.flatMapMany(p -> Flux.fromIterable(p.getData()));
});
});
This does work, but I'm not happy with it because:
It does initial "dummy" request to get the number of pages, and then
repeats the same request to get the actual data.
It gets rate limits only with the initial request, and assumes the
limits won't change (eg, that it's the only one using the API) -
which may not be true, in which case it will get an error that it
exceeded the limit.
So my question is how to refactor it so it doesn't need the initial request (but rather get limits, page numbers and data from the first request, and continue through all pages, while updating (and respecting) the limits.
I think this code will do what you want. The idea is to make a flux that make a call to your resource server, but in the process to handle the response, to add a new event on that flux to be able to make the call to next page.
The code is composed of:
A simple wrapper to contains the next page to call and the delay to wait before executing the call
private class WaitAndNext{
private String next;
private long delay;
}
A FluxProcessor that will make HTTP call and process the response:
FluxProcessor<WaitAndNext, WaitAndNext> processor= DirectProcessor.<WaitAndNext>create();
FluxSink<WaitAndNext> sink=processor.sink();
processor
.flatMap(x-> Mono.just(x).delayElement(Duration.ofMillis(x.delay)))
.map(x-> WebClient.builder()
.baseUrl(x.next)
.defaultHeader("Accept","application/json")
.build())
.flatMap(x->x.get()
.exchange()
.flatMapMany(z->manageResponse(sink, z))
)
.subscribe(........);
I split the code with a method that only manage response: It simply unwrap your data AND add a new event to the sink (the event beeing the next page to call after the given delay)
private Flux<Data> manageResponse(FluxSink<WaitAndNext> sink, ClientResponse resp) {
if (resp.statusCode()!= HttpStatus.OK){
sink.error(new IllegalStateException("Status code invalid"));
}
WaitAndNext wn=new WaitAndNext();
HttpHeaders headers=resp.headers().asHttpHeaders();
wn.delay= Integer.parseInt(headers.getFirst("X-Window-Remaining"))/ Integer.parseInt(headers.getFirst("X-Requests-Quota"));
return resp.bodyToMono(Item.class)
.flatMapMany(p -> {
if (p.paginated.current==p.paginated.total){
sink.complete();
}else{
wn.next="https://....?page="+(p.paginated.current+1);
sink.next(wn);
}
return Flux.fromIterable(p.getData());
});
}
Now we just need to initialize the system by calling for the retrieval of the first page with no delay:
WaitAndNext wn=new WaitAndNext();
wn.next="https://....?page=1";
wn.delay=0;
sink.next(wn);
I want to use Guzzle which will send a few (~20) requests. I use asyncRequest because I want to send it and wait if all is complete and then continue, it will be probably faster than call each request separately so I use Pool.
The problem is, that I don't know if its possible to call each request with different authentication because Pool has only array with options but it's for all requests.
Is there any other faster solution? Because now response is slow: when I call 20 apis it takes about 25 seconds and more.
I have $api which is class in getHeader() i return [ 'auth'=> ['username','password','digist'] ]
code:
$requests = function ($apis) {
foreach ($apis as $api)
{
yield new Request($api->getTYPE(),$api->getURL(),$api->getHeader());
}
};
$myresponses = new Pool($client,$requests($array_of_external_apis),['concurrency'=>5,
'fulfilled'=>function($response,$index)
{
resultArray[] = $response;
},
'rejected'=> function($reason,$index)
{
resultArrayFailed[] = $response;
}]);
$promise= $responses->promise();
$promise->wait();
The code works well for unauthorization pages but if I need authorization for page then probably the authorization values isn't posted.
Thanks for helpful answers.