For one call, I am replying with a huge JSON object which sometimes causes the Node event loop to become blocked. As such, I'm using Big Friendly JSON package to stream JSON instead. My issue is I cannot figure out how to actually reply with the stream
My original code was simply
let searchResults = s3Access.getSavedSearch(guid)).Body;
searchResults = JSON.parse(searchResults.toString());
return reply(searchResults);
Works great but bogs down on huge payloads
I've tried things like, using the Big Friendly JSON package https://gitlab.com/philbooth/bfj
const stream = bfj.streamify(searchResults);
return reply(stream); // according to docs it's a readable stream
But then my browser complained about an empty response. I then tried to add the below to the reply, same result.
.header('content-encoding', 'json')
.header('Content-Length', stream.length);
I also tried return reply(null, stream); but that produced a ton of node errors
Is there some other way I need to organize this? My understanding was I could just reply a readable stream and Hapi would take care of it, but the response keeps showing up as empty.
Did you try to use h.response, here h is reply.
Example:
handler: async (request, h) => {
const { limit, sortBy, order } = request.query;
const queryString = {
where: { status: 1 },
limit,
order: [[sortBy, order]],
};
let userList = {};
try {
userList = await _getList(User, queryString);
} catch (e) {
// throw new Boom(e);
Boom.badRequest(i18n.__('controllers.user.fetchUser'), e);
}
return h.response(userList);
}
Related
I have developed an API in express that returns text and images separately in two different calls, but for performance reasons I would like to return the text and image (probably in a blob object) in the same response.
Now I'm doing it with the res.send() and res.sendFile() methods but I haven't been able to read anything in the documentation about how to send both at the same time. How could I do it?
Examples of what I'm doing now:
async getImage(req, res){
//Doing things with req
let mypath = './foo/bar'
let file = new Blob(path.resolve(mypath), {type: 'image/png'});
return res.sendFile(file)
}
async getText(req, res){
// Doing things with req
return res.status(200).send({ message: 'foo' })
}
I need to return an array of labels, but I can only return 1 of the labels so far. The error which I get is "Cannot set headers after they are sent to the client". So I tried res.write and placed res.end after my for loop then I get the obvious error of doing a res.end before a res.write. How do I solve this?
for(let i=0;i<arr.length;i++){
request.get(arr[i], function (error, response, body) {
if (!error && response.statusCode == 200) {
myfunction();
async function myfunction(){
const Labels = await Somefunctioncallwhoseresponseigetlater(body)
res.send(Labels);
}
}
});}
New code-
async function getDataSendResponse(res) {
let allLabels = [];
for (let url of arr) {
let body = await got(url).buffer();
var imgbuff= Buffer.from(body,'base64')
const imageLabels = await rekognition.detectLabels(imgbuff);
allLabels.push(...imageLabels);
}
res.send(allLabels);
}
The error I have with this code is
"Resolver: AsyncResolver
TypeError: Cannot destructure property Resolver of 'undefined' or 'null'."
You are trying to call res.send() inside a for loop. That means you'll be trying to call it more than once. You can't do that. You get to send one response for any given http request and res.send() sends an entire response. So, when you try to call it again inside the loop, you can the warning you see.
If what you're trying to do is to send an array of labels, then you need to accumulate the array of labels first and then make one call to res.send() to send the final array.
You don't show the whole calling context here, but making the following assumptions:
Somefunctioncallwhoseresponseigetlater() returns a promise that resolves when it is done
You want to accumulate all the labels you collected in your loop
Your Labels variable is an array
Your http request returns a text response. If it returns something else like JSON, then .text() would need to be changed to .json().
then you can do it like this:
const got = require('got');
async function getDataSendResponse(res) {
let allLabels = [];
for (let url of arr) {
let body = await got(url).buffer();
const labels = await Somefunctioncallwhoseresponseigetlater(body);
allLabels.push(...labels);
}
res.send(allLabels);
}
Note, I'm using the got() library instead of the deprecated request() library both because request() is not deprecated and because this type of code is way easier when you have an http library that supports promises (like got() does).
As it will become quickly apparent, I have never seriously written a webserver before
Here is the current scenario:
Clients make requests to webserver, asking to save some data
Server looks at payload, and makes 2 checks
a. Is this client banned from saving data?
b. Does the payload of this data pass a language filter?
Server responds with success, or one of those 2 errors
My endpoint is written with Express in TypeScript
class ChatRequest {
public uid: string;
public message: string;
}
export const register = (app: express.Application, deps: dependencies.IDependencies) => {
app.post("/sendChat", (req: express.Request, res: express.Response) => {
transformAndValidate(ChatRequest, req.body)
.then((sendGlobalChatRequest: SendGlobalChatRequest) => {
const payload = {
message: sendGlobalChatRequest.message,
uid: sendGlobalChatRequest.uid
};
//Check if uid is banned here
//Check if payload passes language filter here
//Save Payload here
res.sendStatus(200);
}, (err) => {
deps.logger.error(err);
res.sendStatus(503);
});
});
I have been using this article for reference:
https://hackernoon.com/the-request-sent-bad-data-whats-the-response-94088bd290a
But I think my conclusion is that they are discussing something slightly different.
So from my understanding, I can just make up HTTP codes...
so I could just do res.sendStatus(499); if the uid is banned, and maybe res.sendStatus(498); if the payload doesn't pass language filter
Then my client can just read the Int statusCode and quickly determine the failure.
But even though I think I can do that, and it would work, it doesn't seem right?
Should I instead be using a standard HTTP Response Code? https://developer.mozilla.org/en-US/docs/Web/HTTP/Status
And then add in the body of the response, a String or something that my client can parse to determine the error?
The String parsing seems way harder to maintain, but technically seems more "legal" if that makes sense?
What is the best way for me to have a client determine the type of server-side error?
I decided to return 400 with a JSON mapping errors to bools
if (isProfane(message)) {
res.status(400).json({messageContentBlocked: true});
}
In this way the client can receive multiple errors for the request at once, and it's more explicit
And in case anyone is googling around, I am using RxSwift/RxCocoa
Here is how I handle the error on the client:
extension Error {
var chatMessageBlockedURLError: Bool {
guard let rxCocoaURLError = self as? RxCocoaURLError else {return false}
switch rxCocoaURLError {
case let .httpRequestFailed(response, data):
guard response.statusCode == 400, let data = data else {return false}
let decoder = JSONDecoder()
decoder.dateDecodingStrategy = .millisecondsSince1970
guard let errors = try? decoder.decode([String:Bool].self, from: data) else {return false}
return errors["messageContentBlocked"] == true
default:
return false
}
}
}
I am trying to insert multiple images to couchdb via nano using express 4 and formidable. I can access and insert individual files without difficulty using formidable and nano, however, when I try to insert one file after another, I get conflict errors. I am very new to js and node, and I know my understanding of callbacks and asynchronous functions is limited. This is what I have at the moment. Any help would be greatly appreciated.
function uploadImage(req, res) {
var form = new formidable.IncomingForm(),
files = [],
fields = [];
uploadcount = 1;
form.on('field', function(field, value) {
fields.push([field, value]);
})
form.on('file', function(field, file) {
files.push([field, file]);
var docid = fields[0][1];
getrevision();
function getRevision(){
dbn.get(docid, { revs_info: true }, function(err,body, file){
if (!err) {
exrev = body._rev;
insertImage(exrev);
}else{
console.log(err);
}
});
}
function insertImage(exrevision){
var exrev = exrevision;
fs.readFile(file.path, function (err, data) {
if (err){
console.log(err);}else{
var imagename = docid + "_" + uploadcount + ".png";
dbn.attachment.insert(docid,imagename,data,'image/png',
{ rev: exrev }, function(err,body){
if (!err) {
uploadcount++;
}else{
console.log(err);
}
});
};
});
};
});
form.on('end', function() {
console.log('done');
res.redirect('/public/customise.html');
});
form.parse(req);
};
I found a solution by dumping the files first into a temporary directory, then proceeding to insert the files into couchdb via nano all within a single function. I couldn't find a way to pause the filestream to wait for the couchdb response, so this sequential method seems adequate.
This is a problem handling asynchronous calls. Because each attachment insert requires the doc's current rev number, you can't do the inserts in parallel. You must insert a new attachment only after you get a response from the the previous one.
You may use the promise and deferred mechanism to do this. But, I personally have solved a similar problem using a package called "async". In async, you can use async.eachSeries() to make these async calls in series.
Another point is about the revision number, you may just use the lighter weight db.head() function, instead of db.get(). The rev number is presented under the "etag" header. You can get the rev like this:
// get Rev number
db.head(bookId, function(err, _, headers) {
if (!err) {
var rev = eval(headers.etag);
// do whatever you need to do with the rev number
......
}
});
In addition, after each attachment insert, the response from couchdb will look something like this:
{"ok":true,"id":"b2aba1ed809a4395d850e65c3ff2130c","rev":"4-59d043853f084c18530c2a94b9a2caed"}
The rev property will give the new rev number which you may use for inserting to the next attachment.
I'd like to use custom headers to provide some more information about the response data. Is it possible to get the headers in a response from a dojo datagrid hooked up to a jsonRest object via an object store (dojo 1.7)? I see this is possible when you are making the XHR request, but in this case it is being made by the grid.
The API provides an event for a response error which returns the response object:
on(this.grid, 'FetchError', function (response, req) {
var header = response.xhr.getAllResponseHeaders();
});
using this I am successfully able to access my custom response headers. However, there doesn't appear to be a way to get the response object when the request is successful. I have been using the undocumented private event _onFetchComplete with aspect after, however, this does not allow access to the response object, just the response values
aspect.after(this.grid, '_onFetchComplete', function (response, request)
{
///unable to get headers, response is the returned values
}, true);
Edit:
I managed to get something working, but I suspect it is very over engineered and someone with a better understanding could come up with a simpler solution. I ended up adding aspect around to allow me to get hold of the deferred object in the rest store which is returned to the object store. Here I added a new function to the deffered to return the headers. I then hooked in to the onFetch of the object store using dojo hitch (because I needed the results in the current scope). It seems messy to me
aspect.around(restStore, "query", function (original) {
return function (method, args) {
var def = original.call(this, method, args);
def.headers = deferred1.then(function () {
var hd = def.ioArgs.xhr.getResponseHeader("myHeader");
return hd;
});
return def;
};
});
aspect.after(objectStore, 'onFetch', lang.hitch(this, function (response) {
response.headers.then(lang.hitch(this, function (evt) {
var headerResult = evt;
}));
}), true);
Is there a better way?
I solved this today after reading this post, thought I'd feed back.
dojo/store/JsonRest solves it also but my code ended up slightly different.
var MyStore = declare(JsonRest, {
query: function () {
var results = this.inherited(arguments);
console.log('Results: ', results);
results.response.then(function (res) {
var myheader = res.xhr.getResponseHeader('My-Header');
doSomethingWith(myheader);
});
return results;
}
});
So you override the normal query() function, let it execute and return its promise, and attach your own listener to its 'response' member resolving, in which you can access the xhr object that has the headers. This ought to let you interpret the JsonRest result while fitting nicely into the chain of the query() all invokers.
One word of warning, this code is modified for posting here, and actually inherited from another intermediary class that also overrode query(), but the basics here are pretty sound.
If what you want is to get info from the server, also a custom key-value in the cookie can be a solution, that was my case, first I was looking for a custom response header but I couldn't make it work so I did the cookie way getting the info after the grid data is fetched:
dojo.connect(grid, "_onFetchComplete", function (){
doSomethingWith(dojo.cookie("My-Key"));
});
This is useful for example to present a SUM(field) for all rows in a paginated datagrid, and not only those included in the current page. In the server you can fetch the COUNT and the SUM, the COUNT will be sent in the Content-Range header and the SUM can be sent in the cookie.