When run an application(Nest.js) on Google Cloud Run with GraphQL file upload(Apollo Server), got this error:
BadRequestError: Request disconnected during file upload stream parsing. at IncomingMessage.abort (/app/node_modules/graphql-upload/public/processRequest.js:127:9)
The source on 127 line of processRequest is createError:
/**
* Handles when the request is closed before it properly ended.
* #kind function
* #name processRequest~abort
* #ignore
*/
const abort = () => {
exit(
createError(
499,
'Request disconnected during file upload stream parsing.'
)
);
};
From this issue it seems it's necessary to use await Promise for createReadStream method:
https://github.com/jaydenseric/apollo-upload-client/issues/149
But the file operate method already did it:
private async parseFile(...
...
const parsed: ImportPoint[] = await new Promise((resolve, reject) => {
const results = [];
file.createReadStream().pipe(
parse({ headers: (headers) => renameHeaders(headers, colorTypeMap) })
.validate((data: ImportPoint, cb) =>
validatePointRow(
data,
pointTypeColors.map((color) => color.id),
cb,
),
)
.on('error', (error) => reject(error))
.on('data-invalid', (_row, rowNumber, reason) => {
reason === INVALID_HEADER_MESSAGE && reject(new UserInputError('Bad header'));
invalidRows.push({ row: rowNumber, columns: reason });
})
.on('data', (row: ImportPoint) => results.push(row))
.on('end', () => resolve(results)),
);
});
What may be the problem?
Related
I had a simple endpoint using Express to allow user to download a csv file.
How should I make a test with just Jest for a file download endpoint
I'm not sure which function or mock should I use to test out this scenario, as it returns with Number of calls: 0 for below test
controller.js
const getFile = async (req, res, next) => {
try {
res.setHeader('Content-Type', 'text/csv');
res.setHeader('Content-Disposition', 'attachment; filename=sample.csv');
const csv = 'ID\n1\n2\n3';
res.send(csv);
} catch (e) {
next(
new HttpException(
'internal error',
'file download error',
e.message,
),
);
}
}
controller.test.js
test('should successfully download the csv', async () => {
const mockReq = {};
const mockRes = {
send: jest.fn(),
};
await controller.getFile(mockReq, mockRes, jest.fn());
expect(mockRes.send).toHaveBeenCalledWith({ 'Content-Type': 'text/csv' });
});
In case anyone face similar problem like me, I think the easiest way is using supertest library. This library support HTTP assertions, so I can test at route level:
const request = require('supertest');
...
const response = await request(app).get(
'/api/.../download-file',
);
expect(response.status).toEqual(200);
expect(response.headers['content-type']).toMatch('text/csv; charset=utf-8');
expect(response.headers['content-disposition']).toMatch(
'attachment; filename=' + 'sample.csv',
);
This error is really driving me crazy for the last 2 days. Please help.
So when I try to login with google the 1st time on my website, it doesn't cause any problem but when I try to do it the second time, with any account, it shows this error in the console:
The FetchEvent for "http://localhost:3000/auth/google/callback?code=4%2F0AX4somethingsomethingsomethingsomething&scope=profile+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.profile" resulted in a network error response: an object that was not a Response was passed to respondWith().
and the webpage shows this error:
This site can’t be reached The web page at http://localhost:3000/auth/google/callback?code=4%2F0AX4somethingsomethingsomethingsomething&scope=profile+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.profile might be temporarily down or it may have moved permanently to a new web address.
I am quite new to pwa and don't understand some of the code in the service worker file (I have copy pasted the 'fetch' part of the code from this webiste: blog.bitsrc.io) so that might be the reason I am not able to identify the error in the code. But you might identify it, this is my service worker code:
const staticCacheName = "site-static-v2";
const dynamicCacheName = "site-dynamic-v2";
const assets = ["/", "/stories", "/groups", "offline.html"];
// cache size limit function
const limitCacheSize = (name, size) => {
caches.open(name).then((cache) => {
cache.keys().then((keys) => {
if (keys.length > size) {
cache.delete(keys[0]).then(limitCacheSize(name, size));
}
});
});
};
// install event
self.addEventListener("install", (evt) => {
//console.log('service worker installed');
evt.waitUntil(
caches.open(staticCacheName).then((cache) => {
console.log("caching shell assets");
cache.addAll(assets);
})
);
});
// activate event
self.addEventListener("activate", (evt) => {
//console.log('service worker activated');
evt.waitUntil(
caches.keys().then((keys) => {
//console.log(keys);
return Promise.all(
keys
.filter((key) => key !== staticCacheName && key !== dynamicCacheName)
.map((key) => caches.delete(key))
);
})
);
});
// fetch events
self.addEventListener("fetch", function (event) {
event.respondWith(
fetch(event.request)
.catch(function () {
return caches.match(event.request);
})
.catch("offline.html")
);
});
This is my script in main.hbs (just like index.html).
if('serviceWorker' in navigator) {
window.addEventListener('load', () => {
navigator.serviceWorker.register('/serviceworker.js', { scope: '/' })
.then((reg) => console.log('Success: ', reg.scope))
.catch((err) => console.log('Failure: ', err));
})
}
I am making my website using express by the way.
I have tried pretty much every solution on stackoverflow but none seem to work.
Just for Information, I have also tried this for the 'fetch' part:
self.addEventListener('fetch', evt => {
evt.respondWith(
caches.match(evt.request).then(cacheRes => {
return cacheRes || fetch(evt.request).then(fetchRes => {
return caches.open(dynamicCacheName).then(cache => {
cache.put(evt.request.url, fetchRes.clone());
// check cached items size
limitCacheSize(dynamicCacheName, 15);
return fetchRes;
})
});
}).catch(() => {
return caches.match('offline.html');
})
);
}
);
(The above code also lets me login only once but doesn't let me logout unlike the previous code)
I have copy pasted almost every 'fetch' code on the internet but all of them have a problem with google auth (I am using passport for google auth).
This is my auth.js code:
const express = require("express");
const router = express.Router();
const passport = require("passport");
//Authenticate with google
//GET /auth/google
router.get("/google", passport.authenticate("google", { scope: ["profile"] }));
//Google auth callback
//GET /auth/google/callback
router.get(
"/google/callback",
passport.authenticate("google", { failureRedirect: "/" }),
function (req, res) {
// Successful authentication, redirect home.
res.redirect("/stories");
}
);
router.get("/logout", (req, res) => {
req.logout();
res.redirect("/");
});
module.exports = router;
You can also suggest a workaround with workbox
I'm pretty new to react native and need some help reg. the possiblity to read an image from a file (like an image). I'm using the expo filesystem library with the following code:
const uploadImages = (file) => {
let data = null;
try {
data = await FileSystem.readAsStringAsync(file);
console.log(data)
} catch (err) {
console.log(err)
}
The issue I have is that I get: 'await' is only allowed within async functions
How can I call this function to wait until the data is loaded into the data variable ?
FileSystem.readAsStringAsync(...) returns Promise.
You can use Promise api like .then() and .catch():
const uploadImages = (file) => {
FileSystem.readAsStringAsync(file)
.then(data => {
// Do something with your data in this block
console.log(data);
})
.catch(err => {
console.log(err.message)
})
}
When using supertest like so,
import app from "../../src/app";
import request from "supertest";
describe("GET / - a simple api endpoint", () => {
it("Hello API Request", () => {
const result = request(app)
.get("/api/location/5eda6d195dd81b21a056bedb")
.then((res) => {
console.log(res);
})
// expect(result.text).toEqual("hello");
// expect(result.status).toEqual(200);
});
});
Im getting "Right-hand side of 'instanceof' is not callable".
at Response.toError (node_modules/superagent/lib/node/response.js:94:15)
at ResponseBase._setStatusProperties (node_modules/superagent/lib/response-base.js:123:16)
at new Response (node_modules/superagent/lib/node/response.js:41:8)
at Test.Request._emitResponse (node_modules/superagent/lib/node/index.js:752:20)
at node_modules/superagent/lib/node/index.js:916:38
at IncomingMessage.<anonymous> (node_modules/superagent/lib/node/parsers/json.js:19:7)
at processTicksAndRejections (internal/process/task_queues.js:84:21) {
status: 500,
text: `"Right-hand side of 'instanceof' is not callable"`,
method: 'GET',
path: '/api/location/5eda6d195dd81b21a056bedb'
This is just with supertest, the API works when using Postman.
Rest of the code for this call,
router.get(
"/location/:id",
(req, res) => {
locationController.getLocation(req, res);
}
);
const getLocation = async (req: Request, res: Response): Promise<void> => {
const { id } = req.params;
const location = await data.readRecord(id, Location);
res.status(location.code).json(location.data);
};
const readRecord = async (id: string, model: IModel): Promise<Response> => {
try {
const response = await model.findById(id);
if (response == null) return { code: 404, data: `ID ${id} Not Found` };
return { code: 200, data: response };
} catch (error) {
return errorHandler(error);
}
};
Is there a configuration im missing for supertest and typescript?
This approach worked,
import request = require("supertest");
import app from "../../src/app";
describe("GET/ api/location/id", () => {
it("should connect retrieve record and retrieve a code 200 and json response", async () => {
const res = await request(app)
.get(`/api/location/${id}`)
expect(res.status).toBe(200);
expect(res.body._id).toBe(`${id}`);
});
});
If you don't want to use "await" in your code , you can use "done()" in callback function.
like this.
import app from "../../src/app";
import request from "supertest";
describe("GET / - a simple api endpoint", () => {
it("Hello API Request", (done) => {
const result = request(app)
.get("/api/location/5eda6d195dd81b21a056bedb")
.then((res) => {
console.log(res);
expect(res.text).toEqual("hello");
expect(res.status).toEqual(200);
done();
//done() function means this test is done.
})
});
});
Awaiting the expect call (with Jest) worked for me.
await expect(...).rejects.toThrow()
I'm using express for showing the result from doing some web scraping with puppeteer but I'm having a performance issue.
I call several times the scraper file because I want to get multiple results at once.
For instance:
const express = require('express')
const app = express()
const scraper = require('./scrapers/scraper.js');
app.get('/getResults', function(req, res, next) {
const url = 'http://www.example.com';
const val1 = new Promise((resolve, reject) => {
scraper
.getPrice(results, url, nights)
.then(data => {
resolve(data)
})
.catch(err => reject('Medium scrape failed'))
})
const url = 'http://www.example.com';
const val2 = new Promise((resolve, reject) => {
scraper
.getPrice(results, url, nights)
.then(data => {
resolve(data)
})
.catch(err => reject('Medium scrape failed'))
const url = 'http://www.example.com';
const val3 = new Promise((resolve, reject) => {
scraper
.getPrice(results, url, nights)
.then(data => {
resolve(data)
})
.catch(err => reject('Medium scrape failed'))
const url = 'http://www.example.com';
const val4 = new Promise((resolve, reject) => {
scraper
.getPrice(results, url, nights)
.then(data => {
resolve(data)
})
.catch(err => reject('Medium scrape failed'))
Promise.all([val1, val2, val3, val4])
.then(data => {
console.log(data)
})
.catch(err => res.status(500).send(err))
}
The code above will call the scraper.js file 4 times at once, but what should I do in order to call each one once the previous one is done? I mean, when val1 is completed, it should run val2 and so on.
In fact, my code calls the scraper file 18 times and that's not good for the computer performance since puppeteer is based with Chromium and it literally opens a new Chromium instance 18 times at once.
I even get this error when I run it:
(node:26600) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 exit listeners added. Use emitter.setMaxListeners() to increase limit
async await
You can write your code with async await. The fun thing is, you can handle all errors and the value is returned automagically with promises.
app.get('/getResults', async function(req, res, next) { //<-- notice the async here
try{
const val1 = await scraper.getPrice(results, url, nights)
const val2 = await scraper.getPrice(results, url, nights)
const val3 = await scraper.getPrice(results, url, nights)
const val4 = await scraper.getPrice(results, url, nights)
return res.send([val1, val2, val3, val4])
} catch(err) {
res.status(500).send(err)
}
})
p-limit
You can use a package called p-limit, which run multiple promise-returning & async functions with limited concurrency.
const pLimit = require('p-limit');
const limit = pLimit(1);
const input = [
limit(() => scraper.getPrice(results, url, nights)),
limit(() => scraper.getPrice(results, url, nights)),
limit(() => scraper.getPrice(results, url, nights))
];
(async () => {
// Only one promise is run at once
const result = await Promise.all(input);
console.log(result);
})();
for..of loop
You can optimize these codes and reduce code duplication. With async..await and for..of, you can reduce the code even more,
// assuming you have these urls
const urls = [
'http://example.com', 'http://example.com', 'http://example.com'
];
const results = []
for(let url of urls){
const data = await scraper.getPrice(results, url, nights);
results.push(data)
}
console.log(results)
Do you know that promises can be made sequentially?
val1.then(v1 => return val2).then(v2=> {...})
You should open a new Chromium tab, not an instance. (Did you just confuse concepts?)
And most importantly - you need to better manage download processes. The queue will be best here. It can be a simple: that makes sure that there are no more than n processes running or more advanced: that monitors the server resources.
You may be able to find some package. If nothing fits you, remember to handle the situation when something gets out and Node will not notice the end of the process.
I use methods interchangeably:
flag URL as being downloaded and if it is not retrieved for a given time, it returns to the queue (More specifically: specify when to re-download the URL. At the time of downloading it is +1 minute, after downloading it is eg 1 month)
I save the PID of the download process and check periodically it works
There are also rate-limits that control the number of HTTP calls. On the endpoint, on the number of simultaneous orders with IP.