use HttpObjectAggregator conditionally - file-upload

Netty version: 4.0.37
I have a requirement to have a netty server which handles both simple JSON requests and also large file uploads. HttpObjectAggregator has a limit of 2 GB for a request size, so I would prefer to use the HttpUploadServer example available here.
So, I want the pipeline to conditionally change depending on the type of request coming in. If it's a POST request, and it's a Multipart type of request, I want the request to be handled by the Upload handler and I want to skip all the rest of the handlers. If not, I want it to pass through the HttpObjectAggregator and then be handled by the Default handler.
I thought of creating one single pipeline looking like this:
HttpRequestDecoder
HttpContentDecompressor
FileUploadHandler <--- My handler to handle file uploads
HttpObjectAggregator
DefaultHandler <---- My handler to handle normal requests, without file body
And inside the "FileUploadHandler", I added the if else logic like this:
private boolean uploadURL(HttpObject object) {
HttpRequest request = (HttpRequest) object;
boolean isMultipart = HttpPostRequestDecoder.isMultipart(request);
if (request.getMethod().equals(HttpMethod.POST) && isMultipart) {
// To be handled by file upload handler
return true;
}
return false;
}
public void channelRead0(ChannelHandlerContext channelHandlerContext,
HttpObject object) throws Exception {
if (!uploadURL(object)) {
ReferenceCountUtil.retain(object);
channelHandlerContext.fireChannelRead(object);
} else {
// Handle the File Upload
....
My objective was to make the UploadHandler "pass on" the message to HttpObjectAggregator IF it's anything other than a POST Multipart request with file body. However, this isn't working for a GET request as the request times out after sometime for lack of a response.
I don't entirely understand why this is happening, but my guess is that HttpObjectAggregator is not receiving the initial HttpRequest object from my UploadHandler at all? And that in turn, isn't delivering it to the Default Handler either.
Is my approach wrong? Is there a different way of handling this conditional routing, outside of my Upload Handler?
Can I have any handler before HttpObjectAggregator or should all custom/user handlers come AFTER the HttpObjectAggregator?

I did this by using a Decoder before HttpObjectAggregator. The pipeline looks like:
HttpRequestDecoder
HttpContentDecompressor
RequestURLDecoder <--- New decoder to route requests.
FileUploadHandler <--- My handler to handle file uploads
HttpObjectAggregator
DefaultHandler <---- My handler to handle normal requests, without file body
The new decoder looks at the request and if it's a POST multipart, dynamically modifies the pipeline to remove the Object aggregator and the default handler. If it's not, then it removes the file upload handler.
(list.add(ReferenceCountUtil.retain(object)) is very important!)

Related

Can't perform HTTP Post Action from Logic App to Asp.net Core Web API

I've built many Logic Apps. I've also integrated with the Logic App API. For some reason, a Post request to an Asp.net Core Web API won't work. It works in Postman, but I can't get Logic Apps to complete the request.
The request arrives at my Web API. I can step through it during a remote debug session. I'm using the [FromBody] decorator on the API method. All the string values in the object are null.
Logic App Headers
Accept = "application/json"
ContentType = "application/json"
ContentLength = "35"
Host = "****.centralus.logic.azure.com"
API method
[HttpPost]
[Route("CreateSomething")]
public async Task<IActionResult> CreateSomething([FromBody] MyObject object)
{
//Create something great
}
I think it might have something to do with the Headers. I noticed that the Postman request won't succeed unless I check the Host and Content-Length box in the Headers section. According to this article, Logic Apps ignores those Headers.
https://learn.microsoft.com/en-us/azure/connectors/connectors-native-http
I've built the HTTP Post Action using the API as well as configured it manually using the Logic App UI in Azure.
By the way, does anyone know the Expression that will automatically calculate the ContentLength?
UPDATE:
I finally figured this out. I had to do some Ninja coding crap to make this work. I'll post my solution tomorrow.
Does anyone know how to make this work? Thanks in advance!
When you use the Logic App API to programmatically create Logic Apps, you have to specify the Body class for when you do something like an HTTP Post. When the Body JSON displayed in the designer, it contained a single object with the objects properties. My API method could not handle this. The key was to simply post the properties in the JSON Body. To make matters worse, I'm doing two HTTP Posts in this particular Logic App. When I tried to add my object properties to the existing Body class, it caused my other HTTP Post to stop working. To overcome this, I had to create a Body2 class with the objects properties. I then had to use the following line of code to replace body2 with body before adding the JSON to the Logic App API call.
This did not work.
body = new Body()
{
object = new Object()
{
//Properties
}
}
This worked.
body2 = new Body2()
{
Type = 0,
Description = "#{items('For_each_2')?['day']?['description']}",
Locations = item.Locations,
Cold = "#{items('For_each_2')?['temperature']?['cold']?['value']}",
Hot = "#{items('For_each_2')?['temperature']?['hot']?['value']}",
Hide = 0
}
Notice I used Replace on body2.
var options = new JsonSerializerOptions { WriteIndented = true, IgnoreNullValues = true};
string jsonString = ReplaceFirst(JsonSerializer.Serialize(myApp, options), "schema", "$schema").Replace("_else", "else").Replace("_foreach", "foreach").Replace("body2", "body");

Ending a response inside custom error handler in expressjs the right way

The official documentation of expressjs at https://expressjs.com/en/guide/error-handling.html states the following:
Notice that when not calling “next” in an error-handling function, you are responsible for writing (and ending) the response. Otherwise those requests will “hang” and will not be eligible for garbage collection.
Their example for the last custom handler is as follows:
...
app.use(logErrors)
app.use(clientErrorHandler)
app.use(errorHandler)
function errorHandler (err, req, res, next) {
res.status(500)
res.render('error', { error: err })
}
At this point I am not sure if this example actually ended the response.
Is it already ended when the last custom error handler calls .status() and .render() ?
The documentation further states
res.end([data] [, encoding])
Ends the response process. This method actually comes from Node core, specifically the response.end() method of http.ServerResponse.
Use to quickly end the response without any data. If you need to respond with data, instead use methods such as res.send() and res.json().
This implies to me, that e.g. send() will end the response.
In restify, which is built on express, the documentation states
Unlike other REST frameworks, calling res.send() does not trigger next() automatically. In many applications, work can continue to happen after res.send(), so flushing the response is not synonymous with completion of a request.
This is why unsure, whether just calling send(),json() or status is really enough to release those resources to garbage collection
I would have expected that we either have to
call next() with no parameter that (so that the default handler will eventually end it) or
res.end() to end manually
after we have written a response to the client by
res.status() or
res.status(...).res.send(...) or
res.send(...) or
What is the proper way to end the response in a custom error handler?
res.send internally calls res.end, but res.status only sets the HTTP status and does not call res.end. With res.render, it depends on the callback function. The default callback uses res.send to output the rendering (implying res.end), or next(err) in case of a rendering error.
So res.send, res.end and res.render (with the default callback) will end the response.

Read raw Request.Body in Asp.Net Core MVC Action with Route Parameters

I need to process the raw request body in and MVC core controller that has route parameters
[HttpPut]
[Route("api/foo/{fooId}")]
public async Task Put(string fooId)
{
reader.Read(Request.Body).ToList();
await _store.Add("tm", "test", data);
}
but It seems like the model binder has already consumed the request stream by the time it gets to the controller.
If I remove the route parameters, I can then access the request stream (since the framework will no longer process the stream to look for parameters).
How can I specify both route parameters and be able to access Request Body without having to manually parse request URI etc.?
I have tried decorating my parameters with [FromRoute] but it had no effect.
Please note I cannot bind the request body to an object and have framework handle the binding, as I am expecting an extremely large payload that needs to be processed in chunks in a custom manner.
There are no other controller, no custom middle-ware, filters, serialzier, etc.
I do not need to process the body several times, only once
storing the stream in a temp memory or file stream is not an options, I simply want to process the request body directly.
How can I get the framework to bind paramters from Uri, QueryString, etc. but leave the request body to me?
Define this attribute in your code:
[AttributeUsage(AttributeTargets.Class | AttributeTargets.Method)]
public class DisableFormValueModelBindingAttribute : Attribute, IResourceFilter
{
public void OnResourceExecuting(ResourceExecutingContext context)
{
var factories = context.ValueProviderFactories;
factories.RemoveType<FormValueProviderFactory>();
factories.RemoveType<JQueryFormValueProviderFactory>();
}
public void OnResourceExecuted(ResourceExecutedContext context)
{
}
}
If you're targeting .Net Core 3 you also need to add
factories.RemoveType<FormFileValueProviderFactory>();
Now decorate your action method with it:
[HttpPut]
[Route("api/foo/{fooId}")]
[DisableFormValueModelBinding]
public async Task Put(string fooId)
{
reader.Read(Request.Body).ToList();
await _store.Add("tm", "test", data);
}
The attribute works by removing Value Providers which will attempt to read the request body, leaving just those which supply values from the route or the query string.
HT #Tseng for the link Uploading large files with streaming which defines this attribute
As I suspected the root cause was MVC inspecting the request body in order to try to bind route parameters. This is how model binding works by default for any routes that are not parameter-less, as per documentation.
The framework however does this only when the request content type is not specified, or when it is form data (multipart or url-encoded I assume).
Changing my request content-type to any thing other than form data (e.g. application/json) I can get the framework to ignore the body unless specifically required (e.g. with a [FromBody] route parameter). This is an acceptable solution for my case since I am only interested accepting JSON payloads with content-type application/json.
Implementation of DisableFormValueModelBindingAttribute in Uploading large files with streaming pointed out by #Tseng seems to be a better approach however, so I will look into using that instead, for complete

Prevent getting old updates from Telegram Bot API using a web hook

I'm writing a Telegram bot and I'm using the official bot API. I've got a webhook server that handles requests and sends a 200 OK response for every request.
Before the server stops, the webhook is detached so Telegram does not send updates anymore. However, whenever I turn the bot on and set the webhook URL again, Telegram starts flooding the webhook server with old updates.
Is there any way I can prevent this without requesting /getUpdates repeatedly until I reach the last update?
Here's a heavily simplified version of how my code looks like:
var http = require('http'),
unirest = require('unirest'),
token = '***';
// Attach the webhook
unirest.post('https://api.telegram.org/bot' + token + '/setWebhook')
.field('url', 'https://example.com/api/update')
.end();
process.on('exit', function() {
// Detach the webhook
unirest.post('https://api.telegram.org/bot' + token + '/setWebhook')
.field('url', '')
.end();
});
// Handle requests
var server = http.createServer(function(req, res) {
res.writeHead(200, { 'Content-Type': 'text/plain' })
res.end('Thanks!');
});
server.listen(80);
Thanks in advance.
The best way is to use update_id which is a specific number that increases on every new request (i.e. update). How to implement it?
First off, let's start with the following anonymous class (using PHP7):
$lastUpdateId = new class()
{
const FILE_PATH = "last-update-id.txt";
private $value = 1;
public function __construct()
{
$this->ensureFileExists();
$this->value = filesize(self::FILE_PATH) == 0
? 0 : (int)(file_get_contents(self::FILE_PATH));
}
public function set(int $lastUpdateId)
{
$this->ensureFileExists();
file_put_contents(self::FILE_PATH, $lastUpdateId);
$this->value = $lastUpdateId;
}
public function get(): int
{
return $this->value;
}
public function isNewRequest(int $updateId): bool
{
return $updateId > $this->value;
}
private function ensureFileExists()
{
if (!file_exists(self::FILE_PATH)) {
touch(self::FILE_PATH);
}
}
};
What the class does is clear: Handling the last update_id via a plain file.
Note: The class is tried to be as short as possible. It does not provide error-checking. Use your custom implementation (e.g. use SplFileObject instead of file_{get|put}_contents() functions) instead.
Now, there are two methods of getting updates: Long Polling xor WebHooks (check Telegram bot API for more details on each methods and all JSON properties). The above code (or similar) should be used in both cases.
Note: Currently, it is impossible to use both methods at the same time.
Long Polling Method (default)
This way, you send HTTPS requests to Telegram bot API, and you'd get updates as response in a JSON-formatted object. So, the following work can be done to get new updates (API, why using offset):
$botToken = "<token>";
$updates = json_decode(file_get_contents("https://api.telegram.org/bot{$botToken}/getUpdates?offset={$lastUpdateId->get()}"), true);
// Split updates from each other in $updates
// It is considered that one sample update is stored in $update
// See the section below
parseUpdate($update);
WebHook Method (preferred)
Requiring support for HTTPS POST method from your server, the best way of getting updates at-the-moment.
Initially, you must enable WebHooks for your bot, using the following request (more details):
https://api.telegram.org/bot<token>/setWebhook?url=<file>
Replace <token> with you bot token, and <file> with the address of your file which is going to accept new requests. Again, it must be HTTPS.
OK, the last step is creating your file at the specified URL:
// The update is sent
$update = $_POST;
// See the section below
parseUpdate($update);
From now, all requests and updates your bot will be directly sent to the file.
Implementation of parseUpdate()
Its implementation is totally up to you. However, to show how to use the class above in the implementation, this is a sample and short implementation for it:
function parseUpdate($update)
{
// Validate $update, first
// Actually, you should have a validation class for it
// Here, we suppose that: $update["update_id"] !== null
if ($lastUpdateId->isNewRequest($update["update_id"])) {
$lastUpdateId->set($update["update_id"]);
// New request, go on
} else {
// Old request (or possible file error)
// You may throw exceptions here
}
}
Enjoy!
Edit: Thanks to #Amir for suggesting editions made this answer more complete and useful.
When you server starts up you can record the timestamp and then use this to compare against incoming message date values. If the date is >= the timestamp when you started...the message is ok to be processed.
I am not sure if there is a way you can tell Telegram you are only interested in new updates, their retry mechanism is a feature so that messages aren't missed...even if your bot is offline.
In the webhook mode, Telegram servers send updates every minute until receives an OK response from the webhook program.
so I recommend these steps:
Check your webhook program that you specified its address as url parameter of the setWebhook method. Call its address in a browser. It does not produce an output to view, but clears that probably there is no error in your program.
Include a command that produces a '200 OK Status' header output in your program to assure that the program sends this header to the Telegram server.
I have the same issue, then I tried to reset the default webhook with
https://api.telegram.org/bot[mybotuniqueID]/setWebhook?url=
after that, i verified the current getUpdates query were the same old updates but I sent new requests through the telegram's bot chat
https://api.telegram.org/bot[mybotuniqueID]/getUpdates
when I set up my webhook again the webhook read the same old updates. Maybe the getUpdates method is not refreshing the JSON content.
NOTE:
in my case, it was working fine until I decided to change /set privacy bot settings from botfather

Apache Http Client Put Request Error

I'm trying to upload a file using the Apache Http Client's PUT method. The code is as below;
def putFile(resource: String, file: File): (Int, String) = {
val httpClient = new DefaultHttpClient(connManager)
httpClient.getCredentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(un, pw))
val url = address + "/" + resource
val put = new HttpPut(url)
put.setEntity(new FileEntity(file, "application/xml"))
executeHttp(httpClient, put) match {
case Success(answer) => (answer.getStatusLine.getStatusCode, "Successfully uploaded file")
case Failure(e) => {
e.printStackTrace()
(-1, e.getMessage)
}
}
}
When I tried running the method, I get to see the following error:
org.apache.http.NoHttpResponseException: The target server failed to respond
at org.apache.http.impl.conn.DefaultResponseParser.parseHead(DefaultResponseParser.java:101)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:252)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:281)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:247)
at org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:219)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:298)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:633)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:454)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
I do not know what has gone wrong? I'm able to do GET requests, but PUT seems not to work! Any clues as to where I should look for?
Look on the server. If GET Works, but PUT does not, then you have to figure out the receiving end.
Also, you may want to write a simple HTML File that has a form with PUT Method in it to rule out your Java Part.
As a sidenode: Its technically possible that something in between stops the request from going through or the response reaching you. Best setup a dummy HTTP Server to do the testing against.
Maybe its also a timeout issue, so the server takes to long to process your PUT.
The connection you are trying to use is a stale connection and therefore the request is failing.
But why are you only seeing an error for the PUT request and you are not seeing it for the GET request?
If you check the DefaultHttpRequestRetryHandler class you will see that by default HttpClient attempts to automatically recover from I/O exceptions. The default auto-recovery mechanism is limited to just a few exceptions that are known to be safe.
HttpClient will make no attempt to recover from any logical or HTTP protocol errors (those derived from HttpException class).
HttpClient will automatically retry those methods that are assumed to be idempotent. Your GET request, but not your PUT request!!
HttpClient will automatically retry those methods that fail with a transport exception while the HTTP request is still being transmitted to the target server (i.e. the request has not been fully transmitted to the server).
This is why you don't notice any error with your GET request, because the retry mechanism handles it.
You should define a CustomHttpRequestRetryHandler extending the DefaultHttpRequestRetryHandler. Something like this:
public class CustomHttpRequestRetryHandler extends DefaultHttpRequestRetryHandler {
#Override
public boolean retryRequest(IOException exception, int executionCount, HttpContext context) {
if(exception instanceof NoHttpResponseException) {
return true;
}
return super.retryRequest(exception, executionCount, context);
}
}
Then just assign your CustomHttpRequestRetryHandler
final HttpClientBuilder httpClientBuilder = HttpClients.custom();
httpClientBuilder.setRetryHandler(new CustomHttpRequestRetryHandler());
And that's it, now your PUT request is handled by your new RetryHandler (like the GET was by the default one)