How to limit the number of calls in Express.js? - express

Question

I'm using express for showing the result from doing some web scraping with puppeteer but I'm having a performance issue.
I call several times the scraper file because I want to get multiple results at once.
For instance:
const express = require('express')
const app = express()
const scraper = require('./scrapers/scraper.js');
app.get('/getResults', function(req, res, next) {
const url = 'http://www.example.com';
const val1 = new Promise((resolve, reject) => {
scraper
.getPrice(results, url, nights)
.then(data => {
resolve(data)
})
.catch(err => reject('Medium scrape failed'))
})
const url = 'http://www.example.com';
const val2 = new Promise((resolve, reject) => {
scraper
.getPrice(results, url, nights)
.then(data => {
resolve(data)
})
.catch(err => reject('Medium scrape failed'))
const url = 'http://www.example.com';
const val3 = new Promise((resolve, reject) => {
scraper
.getPrice(results, url, nights)
.then(data => {
resolve(data)
})
.catch(err => reject('Medium scrape failed'))
const url = 'http://www.example.com';
const val4 = new Promise((resolve, reject) => {
scraper
.getPrice(results, url, nights)
.then(data => {
resolve(data)
})
.catch(err => reject('Medium scrape failed'))
Promise.all([val1, val2, val3, val4])
.then(data => {
console.log(data)
})
.catch(err => res.status(500).send(err))
}
The code above will call the scraper.js file 4 times at once, but what should I do in order to call each one once the previous one is done? I mean, when val1 is completed, it should run val2 and so on.
In fact, my code calls the scraper file 18 times and that's not good for the computer performance since puppeteer is based with Chromium and it literally opens a new Chromium instance 18 times at once.
I even get this error when I run it:
(node:26600) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 exit listeners added. Use emitter.setMaxListeners() to increase limit

Answer 1

async await
You can write your code with async await. The fun thing is, you can handle all errors and the value is returned automagically with promises.
app.get('/getResults', async function(req, res, next) { //<-- notice the async here
try{
const val1 = await scraper.getPrice(results, url, nights)
const val2 = await scraper.getPrice(results, url, nights)
const val3 = await scraper.getPrice(results, url, nights)
const val4 = await scraper.getPrice(results, url, nights)
return res.send([val1, val2, val3, val4])
} catch(err) {
res.status(500).send(err)
}
})
p-limit
You can use a package called p-limit, which run multiple promise-returning & async functions with limited concurrency.
const pLimit = require('p-limit');
const limit = pLimit(1);
const input = [
limit(() => scraper.getPrice(results, url, nights)),
limit(() => scraper.getPrice(results, url, nights)),
limit(() => scraper.getPrice(results, url, nights))
];
(async () => {
// Only one promise is run at once
const result = await Promise.all(input);
console.log(result);
})();
for..of loop
You can optimize these codes and reduce code duplication. With async..await and for..of, you can reduce the code even more,
// assuming you have these urls
const urls = [
'http://example.com', 'http://example.com', 'http://example.com'
];
const results = []
for(let url of urls){
const data = await scraper.getPrice(results, url, nights);
results.push(data)
}
console.log(results)

Answer 2

Do you know that promises can be made sequentially?
val1.then(v1 => return val2).then(v2=> {...})
You should open a new Chromium tab, not an instance. (Did you just confuse concepts?)
And most importantly - you need to better manage download processes. The queue will be best here. It can be a simple: that makes sure that there are no more than n processes running or more advanced: that monitors the server resources.
You may be able to find some package. If nothing fits you, remember to handle the situation when something gets out and Node will not notice the end of the process.
I use methods interchangeably:
flag URL as being downloaded and if it is not retrieved for a given time, it returns to the queue (More specifically: specify when to re-download the URL. At the time of downloading it is +1 minute, after downloading it is eg 1 month)
I save the PID of the download process and check periodically it works
There are also rate-limits that control the number of HTTP calls. On the endpoint, on the number of simultaneous orders with IP.