I've written a very small nodejs script. Nothing fancy, just listen to a request, generate a big random string of 108KB, perform an i/o by writing it to a file, then reading from it.
And then finally, put it back on the response stream.
The idea is to benchmark node.js capabilities against other systems like lamp/asp.net for a web app I'm going to develop:
//server.js
var http = require('http');
var server = http.createServer(handler);
function handler(request, response) {
//console.log('request received!');
response.writeHead(200, {'Content-Type': 'text/plain'});
s=""; //generate a random string of 108KB
for(i=0;i<108000;i++)
{
n=Math.floor(65 + (Math.random()*(122-65)) );
s+=String.fromCharCode(n);
}
//write s to a file
var fs = require('fs');
fs.writeFile("godspeed.txt", s, function(err) {
if (err) throw err;
//console.log("The file was saved!");
//read back from the file
fs.readFile('godspeed.txt', function (err, data) {
if (err) throw err;
result = data;
response.end(result);
});
}
);
}
server.listen(8124);
console.log('Server running at http://127.0.0.1:8124/');
The above works fine and returns the string when I open it from a browser. But when I run the apache benchmarking tool over this (with 2000 total requests and 200 concurrent requests), about 911 requests are shown as failed meaning something is wrong with my code:
Concurrency Level: 200
Time taken for tests: 41.887 seconds
Complete requests: 2000
**Failed requests: 911**
Can you help me figure out what is wrong with my code?
(Apart from these failed requests, node.js is excellent at performance considering the time total taken compared to other systems.)
Related
I am bulding a cloudflare worker and I want to cache a fetch request for at least 24 hours. This mean if I make the same request twice within 24h the fetch() should not be called and used the cached response.
I've written this script, but the remote website (unixtimestamp.com) it's called every time.
addEventListener("fetch", event => {
event.respondWith(handleRequest(event.request))
});
async function handleRequest(request) {
const url = 'https://www.unixtimestamp.com/'
let response = await fetch(url, {
cf: {
cacheTtlByStatus: { "200-299": 60*60*24, 404: -1, "500-599": -1 }
}
})
return response
}
Documentation: https://developers.cloudflare.com/workers/examples/cache-using-fetch
I'm wondering if Page rules is more efficient in this case than workers in your case ? (moreover, workers count calls can be limited depending on your plan)
I have done once in page rules (and is easily configurable) with Cache TTL by status code option.
I want to allow users request audio files. The files are hosted on a separate file server. I don't want users to get these files unless they've gone through my server first.
How do I make a function that basically acts as a middle man between the user and the file server. I currently have something like this:
async (req, res) => {
const mediaLink = `https://www.example.com/audio.mp3`;
const mediaResponse = await fetch(mediaLink, {
headers: {
Range: req.headers.range,
}
});
const blob = await mediaResponse.blob(); // I'm guessing here. Idk.
res.send(blob);
}
I tested this in an <audio> tag but the audio never loaded:
<audio controls src="http://localhost:5001/file-server-middleware" />
The correct way to handle such a request would be to pipe the response body back to the client making sure to copy across any relevant headers that may be in the response from the file server. Read up on the HTTP Functions documentation to see what use cases you should be looking out for (e.g. CORS).
async (req, res) => {
const mediaLink = `https://www.example.com/audio.mp3`;
// You may wish to pass through Cache-related headers, such as
// If-None-Match, If-Match, If-Modified-Since, If-Range, If-Unmodified-Since
const mediaResponse = await fetch(mediaLink, {
headers: {
Range: req.headers.range,
}
});
// note: this currently passes the body & status from the file server
// as-is, you may want to send your own body on failed status codes to
// hide the existence of the external server (i.e. custom 404 pages, no
// Nginx error pages, etc)
// mediaResponse.status may be:
// - 200 (sending full file)
// - 206 (sending portion of file)
// - 304 (not modified, if using cache headers)
// - 404 (file not found)
// - 412 (precondition failed, if using cache headers)
// - 416 (range not satisfiable)
// - 5xx (internal server errors from the file server)
res
.status(mediaResponse.status)
.set({
/* ... other headers (e.g. CORS, Cache-Control) ... */
'Content-Type': mediaResponse.headers.get('content-type'),
'Content-Length': mediaResponse.headers.get('content-length'),
'Content-Encoding': mediaResponse.headers.get('content-encoding'),
'Etag': mediaResponse.headers.get('Etag'), // for caching
'Last-Modified': mediaResponse.headers.get('last-modified') // for caching
});
mediaResponse.body.pipe(res);
}
You may also want to look into the various express-compatible proxy modules that can handle the bodies and headers for you. Note that some of these may not function properly if used in a Firebase Cloud Function as the request bodies are automatically consumed for you.
I have a little problem with my application.
My architecture:
Angular 6 (front)
NodeJS + Express + MongoDB (back)
There is a part, in my NodeJS application, that communicates with a REST API.
When an user clicks on button on the Angular WebSite, I send a request to Express.
In my Express application, I send another request to the API to retrieve information.
But this process takes a lot of time. My request timeout
What is the best solution to keep my process working after the Express reply has been sent?
Should I do otherwise?
Assuming the problem is with timeout, you can increase the default timeout:
You may set the timeout either globally for entire server. Read more here:
var server = app.listen(app.get('port'), function() {
//Server is running
});
server.timeout = 1000; //1000 is one second.
or just for specific route. Read more here:
app.post('/xxx', function (req, res) {
req.setTimeout(500000);
});
You can also change the timeout for the other API request you are making. Read more here
//Assuming you are using request module.
var options = {
url: 'http://url',
timeout: 120000
}
request(options, function(err, resp, body) {});
I'm trying to execute API.AI tutorial for building a weather bot for Google Assistant (the one here: https://dialogflow.com/docs/getting-started/basic-fulfillment-conversation)
I made everything successfully, created the bot within API, created the Fulfillments, installed NodeJS on my pc, connected Google Cloud Platform, etc.
Then I created the index.js file by copying it exactly how it's stated on API.ai tutorial with my API key from World Weather Organisation (see below).
But when I use the bot, it doesn't work. On the Google Cloud Platform the error is always the same:
Error: getaddrinfo ENOTFOUND api.worldweatheronline.com
api.worldweatheronline.com:80
at errnoException (dns.js:28)
at GetAddrInfoReqWrap.onlookup (dns.js:76)
No matter how often I do it I get the same error. So I don't actually reach the API. I tried to see if anything changed from WWO side (URL, etc.) but apparently no. I updated NodeJS and still same issue. I refreshed the Google Cloud platform completely and didn't help.
That one I really can't debug. Could anyone help?
Here's the code from API.ai:
'use strict';
const http = require('http');
const host = 'api.worldweatheronline.com';
const wwoApiKey = '[YOUR_API_KEY]';
exports.weatherWebhook = (req, res) => {
// Get the city and date from the request
let city = req.body.result.parameters['geo-city']; // city is a required param
// Get the date for the weather forecast (if present)
let date = '';
if (req.body.result.parameters['date']) {
date = req.body.result.parameters['date'];
console.log('Date: ' + date);
}
// Call the weather API
callWeatherApi(city, date).then((output) => {
// Return the results of the weather API to Dialogflow
res.setHeader('Content-Type', 'application/json');
res.send(JSON.stringify({ 'speech': output, 'displayText': output }));
}).catch((error) => {
// If there is an error let the user know
res.setHeader('Content-Type', 'application/json');
res.send(JSON.stringify({ 'speech': error, 'displayText': error }));
});
};
function callWeatherApi (city, date) {
return new Promise((resolve, reject) => {
// Create the path for the HTTP request to get the weather
let path = '/premium/v1/weather.ashx?format=json&num_of_days=1' +
'&q=' + encodeURIComponent(city) + '&key=' + wwoApiKey + '&date=' + date;
console.log('API Request: ' + host + path);
// Make the HTTP request to get the weather
http.get({host: host, path: path}, (res) => {
let body = ''; // var to store the response chunks
res.on('data', (d) => { body += d; }); // store each response chunk
res.on('end', () => {
// After all the data has been received parse the JSON for desired data
let response = JSON.parse(body);
let forecast = response['data']['weather'][0];
let location = response['data']['request'][0];
let conditions = response['data']['current_condition'][0];
let currentConditions = conditions['weatherDesc'][0]['value'];
// Create response
let output = `Current conditions in the ${location['type']}
${location['query']} are ${currentConditions} with a projected high of
${forecast['maxtempC']}°C or ${forecast['maxtempF']}°F and a low of
${forecast['mintempC']}°C or ${forecast['mintempF']}°F on
${forecast['date']}.`;
// Resolve the promise with the output text
console.log(output);
resolve(output);
});
res.on('error', (error) => {
reject(error);
});
});
});
}
Oh boy, in fact the reason was most stupid ever. I didn't enable "billing" on Google Cloud Platform and that's why it blocked everything (even though I'm using a free test of the API). They just wanted my credit card number. It works now
I had the same issue trying to hit my db. Billing wasn't the fix as I had billing enabled already.
For me it was knexfile.js setup for MySql - specifically the connection object. In that object, you should replace the host key with socketPath; and prepend /cloudsql/ to the value. Here's an example:
connection: {
// host: process.env.APP_DB_HOST, // The problem
socketPath: `/cloudsql/${process.env.APP_DB_HOST}`, // The fix
database: process.env.APP_DB_NAME,
user: process.env.APP_DB_USR,
password: process.env.APP_DB_PWD
}
Where process.env.APP_DB_HOST is your Instance connection name.
PS: I imagine that even if you're not using Knex, the host or server parameter of a typical DB connectionstring will have to be called socketPath when connecting to Google Cloud SQL.
When using https.request with node.js v04.7, I get the following error:
Error: socket hang up
at CleartextStream.<anonymous> (http.js:1272:45)
at CleartextStream.emit (events.js:61:17)
at Array.<anonymous> (tls.js:617:22)
at EventEmitter._tickCallback (node.js:126:26)
Simplified code that will generate the error:
var https = require('https')
, fs = require('fs')
var options = {
host: 'localhost'
, port: 8000
, key: fs.readFileSync('../../test-key.pem')
, cert: fs.readFileSync('../../test-cert.pem')
}
// Set up server and start listening
https.createServer(function (req, res) {
res.writeHead(200, {'Content-Type': 'text/plain'})
res.end('success')
}).listen(options.port, options.host)
// Wait a second to let the server start up
setTimeout(function() {
var clientRequest = https.request(options, function(res) {
res.on('data', function (chunk) {
console.log('Called')
})
})
clientRequest.write('')
clientRequest.end()
}, 1000)
I get the error even with the server and client running on different node instances and have tested with port 8000, 3000, and 443 and with and without the SSL certificates. I do have libssl and libssl-dev on my Ubuntu machine.
Any ideas on what could be the cause?
In
https.createServer(function (req, res) {
you are missing options when you create the server, should be:
https.createServer(options, function (req, res) {
with your key and cert inside
I had a very similar problem where the response's end event never fired.
Adding this line fixed the problem:
// Hack to emit end on close because of a core bug that never fires end
response.on('close', function () {response.emit('end')});
I found an example of this in the request library mentioned in the previous answer.
Short answer: Use the the latest source code instead of the one you have. Store it where you will and then require it, you are good to go.
In the request 1.2.0 source code, main.js line 76, I see
http.createClient(options.uri.port, options.uri.hostname, options.uri.protocol === 'https:');
Looking at the http.js source code, I see
exports.createClient = function(port, host) {
var c = new Client();
c.port = port;
c.host = host;
return c;
};
It is requesting with 3 params but the actual function only has 2. The functionality is replaced with a separate module for https.
Looking at the latest main.js source code, I see dramatic changes. The most important is the addition of require('https').
It appears that request has been fixed but never re-released. Fortunately, the fix seems to work if you just copy manually from the raw view of the latest main.js source code and use it instead.
I had a similar problem and i think i got a fix. but then I have another socket problem.
See my solution here: http://groups.google.com/group/nodejs/browse_thread/thread/9189df2597aa199e/b83b16c08a051706?lnk=gst&q=hang+up#b83b16c08a051706
key point: use 0.4.8, http.request instead of http.createClient.
However, the new problem is, if I let the program running for long time, (I actually left the program running but no activity during weekend), then I will get socket hang up error when I send a request to http Server. (not even reach the http.request). I don't know if it is because of my code, or it is different problem with http Server