const puppeteer = require('puppeteer');
const browser = await puppeteer.launch();
const page = await browser.newPage();
This one works
await page.goto('https://example.com');
This doesn't work (without the protocol i.e http/https)
await page.goto("www.example.com');
It throws error
Protocol error (Page.navigate): Cannot navigate to invalid URL
Why doesn't it append the protocol like it does when we open in Google Chrome?
The Google Chrome Omnibox (Address Bar) has built in functionality to handle multiple complexities, such as: appending protocols, autocomplete, etc.
Puppeteer provides an API to control Chrome or Chromium over the DevTools Protocol, so much of this functionality is currently out of the scope of Puppeteer.
The Puppeteer documentation for the function page.goto() explicitly states:
The url should include scheme, e.g. https://.
This is because page.goto() utilizes Page.navigate from the Chrome DevTools Protocol.
The Chromium source code shows that navigation via Page.navigate is explicitly checked for validity, and if the URL is not valid, it will return the error, "Cannot navigate to invalid URL."
You can easily create a function in Node.js that will append protocols to URLs, and that could be a workaround for your issue.
I got the same error when I was sending the url as an array of arrays
const urls = [["https://www.example1.com"], ["https://www.example2.com]]
destructuring it solved it for me
urls = [].concat(...urls)
for(let url of urls) {
await page.goto(url)
}
Related
old Rails and php programmer that is new to js, tyring something and feel I am missing something pretty basic. I have a node server running a small webapp, express, ejs, jquery as well as alpinejs. (very exp with jquery, but learning alpinejs). I have a mqtt server running and can manipulate a couple of topics via my phone, and I can even see them in the node console (code below)
what I fail to understand is how to get, or maybe "bind" the subscription to a variable that I can "pass" into the express app so I can just show the mqtt message in the html. Im currently just trying to set things like fridgeFan = true or somehthing. My idea, I think, is to have mqtt update the alpinejs variable, hence updating the page, then viceversa...
Can someone point me in the general good direction of what im attemtping?
options={
clientId:"node-admin",
clean:true,
};
const express = require('express');
const app = express();
const mqtt = require('mqtt')
const client = mqtt.connect('mqtt://192.168.10.66', options)
client.on("connect",function(){
console.log("connected");
});
client.on('message',function(topic, message, packet){
console.log("message is "+ message);
console.log("topic is "+ topic);
// assume id like to bind topic and message to
// some variable that I can pass or have access to
// in the res.render calls below??
// couple of things I was trying
//document.getElementById('mqtt_div').innerHTML += topic;
//document.querySelector('[x-data]').__x.getUnobservedData().fridgeFan = message;
});
client.subscribe("sensors/sensor1",{qos:1});
app.set('view engine', 'ejs');
app.use(express.static(__dirname + '/public'));
app.get('/', (req, res) => {
res.render('index', {
title: 'Homepage'
});
});
with the above code, I can publish to "sensors/sensor1" via a phone app, I can see it in the node console, of course not the browser console.
How can I bind the "message" to a alpinejs variable so it will update?
OR do I go a different direction and put the client.on calls elsewhere, say the ejs file? in my testings, the ejs files dont know about require mqtt
You can't move your on message handler into the EJS files as they are only "executed" at the point you call render()
You can declare a global variable in the ExpressJS app and update fields with in it from the on('message',...) callback, you then pass this variable into the call to render() but this would mean that the page would only show the value at the time the page was loaded.
If you want live updates in the page as messages are published then you have one real option. That is to use a broker that supports MQTT over Websockets (nearly all modern brokers support this). With this you can load an MQTT client (either the Paho JavaScript client or you can use the MQTT.js client) into the page and subscribe to the topics you are interested in directly from the page. This way new messages are delivered directly to the page and you can then use what ever JavaScript framework you want to update the page.
Our app is in React + NextJS. Framework is like (Browser code -> (Making call to)->Next JS Api->(Making call to)->External Rest API). I want to perform E2E Testing using Testcafe where I will mock only the External API call. How can we achieve the same?
Yes you can write E2E tests using testcafe. It simulates actual user experience and interact with your website. It is not dependent underlying tech stack. You just need to host your app and provide the url to testcafe config. Use regular css selectors to pick page elements and make assertions. You can also intercept API requests and use mock data for API. Mocking is not recommended for E2E testing.
Sample code for API response:
import { RequestLogger } from 'testcafe'
const logger= RequestLogger(['Domain URLs Array'], {
logResponseHeader: true,
logResponseBody: true
})
fixture(`description`).page('app url').requestHooks(logger)
test('description', async () => {
const domain1Res = logger.requests[0].response
const domain1ResStr = domain1Res.body.toString()
const domain1ResJson = domain1ResStr.length ? JSON.parse(domain1ResStr) : null
// domain1ResJson is the API response object, use it in assertions
})
On latest react-native 0.60 onwards, the default app will include a list of links at landing page as below
Now if this react-native project running on simulator and we click on one of the link, it will open up on our laptop instead of opening on browser from within the simulator. I'm wondering how it works so I took a look at node_modules/react-native/Libraries/Core/Devtools/openURLInBrowser.js and found below code
'use strict';
const getDevServer = require('./getDevServer');
function openURLInBrowser(url: string) {
// Made a console.log here and getDevServer().url = http://localhost:8081
fetch(getDevServer().url + 'open-url', {
method: 'POST',
body: JSON.stringify({url}),
});
}
module.exports = openURLInBrowser;
Despite having this source code, I still can't understand how can we use a fetch library to launch a url in host's browser? So far I've only use fetch to perform http request from some backend but apparently there's something more that fetch is providing?
The Metro bundler is running on 8081. fetch is performing a HTTP POST request to http://localhost:8081/open-url. Metro is able to handle this request and open up the browser accordingly.
An example Express web server may handle it like so:
app.post('/open-url', req => {
const { url } = req.body;
// Handle code to open URL
});
I'm working on a VueJS app, and I want to use the Yahoo! Shopping API (documentation: https://developer.yahoo.co.jp/webapi/shopping/shopping/v1/itemsearch.html) to fetch products using their barcode. However, I keep getting a CORS error and I'm not sure why, or what I can do to make it work. My code is as follows:
data() {
return {
appId: 'dj00aiZpPUQ4RTBUUTVSNUs3TyZzPWNvbnN1bWVyc2VjcmV0Jng9NTI-',
apiUrl: 'https://shopping.yahooapis.jp/ShoppingWebService/V1/json/itemSearch'
}
}
...
axios.get(this.apiUrl, {
params: {
appid: this.appId,
jan: 4589707054951
}
})
.then((response) => {
console.log(response);
})
.catch((error) => {
alert(error)
})
The specific error I get is this:
Access to XMLHttpRequest at
'https://shopping.yahooapis.jp/ShoppingWebService/V1/json/itemSearch?appid=dj00aiZpPUQ4RTBUUTVSNUs3TyZzPWNvbnN1bWVyc2VjcmV0Jng9NTI-&jan=4589707054951'
from origin 'https://0.0.0.0:8080' has been blocked by CORS policy: No
'Access-Control-Allow-Origin' header is present on the requested
resource.
What can I do to make this work? So far the only way it works is using the CORS Chrome extension, but that's naturally just for testing.
Thanks a lot.
The server/api owner needs to send this in his response header:
Or if you have a server settings panel or something, make sure to add the domain from where you are making the request.
Access-Control-Allow-Origin: 'your-domain-here'.
Please read more at: https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS
I got the same error. I solved it in the following manner.
This error occurs because we are tryign to access the data hosted on a different server. CORS issue is a browser issue where a certain browser does allow you to access the contents hosted on a different or a virtual server. If you observe the app that you are running on is on the local-host (https://0.0.0.0:8080) and trying to access the data hosted on https://shopping.yahooapis.jp/ShoppingWebService/V1/json/itemSearch. You can solve this problem simply by adding in a few lines of code in your vue application.
Step 1:
Create a new file called vue.config.js in your root directory of the vue application, that is beside your package.json file. Add the following code in your vue.config.js file:
module.exports = {
devServer:{
proxy: "https://shopping.yahooapis.jp"
}
}
Note: https://shopping.yahooapis.jp would be your base url.
Step 2:
Now, go back to your code in data(). Replace the domain name/base url of the apiUrl with https://0.0.0.0:8080. that is now your apiUrl would be https://0.0.0.0:8080/ShoppingWebService/V1/json/itemSearch.
data() {
return {
appId: 'dj00aiZpPUQ4RTBUUTVSNUs3TyZzPWNvbnN1bWVyc2VjcmV0Jng9NTI-',
apiUrl: 'https://0.0.0.0:8080/ShoppingWebService/V1/json/itemSearch'
}
}
Step 3:
Restart your application. Eg, npm run serve.
I found another solution which worked without creating a proxy server or a mock server. You can disable the security settings for accessing cross origin apis on your web browser.
You can disable the CHROME security settings for accessing apis out of the origin by typing the below command on the terminal:
/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --user-data-dir="/tmp/chrome_dev_session" --disable-web-security
After running the above command on your terminal, a new chrome window with security settings disabled will open up. Now, run your program (npm run serve / npm run dev) again and this time you will not get any CORS error and would be able to GET request using axios.
Hope this helps!
Question
How do I use Express within Firebase Cloud Functions?
Expectations
Using either of the URLs I've setup, I expect to see, "Hello from Express on Firebase!" in the console logs.
Why? My understanding is, "*" means all routes requested should response.send("Hello from Express on Firebase!");
app.get("*", (_request, response) => {
response.send("Hello from Express on Firebase!");
});
Issue
When I use, https://us-central1-myapp.cloudfunctions.net/helloWorld I get the expected Hello from Firebase! in the logs. Should I also see "Hello from Express on Firebase!"?
When I use, https://us-central1-myapp.cloudfunctions.net/api, I get a 404 error
The URL, https://us-central1-myapp.cloudfunctions.net/api is the issue. See why in the answer below.
Code
// Express
import express = require("express");
const app = express();
const cors = require("cors")({
origin: "*"
});
app.use("*", cors);
// Firebase Functions SDK
import functions = require("firebase-functions");
app.get("*", (_request, response) => {
response.send("Hello from Express on Firebase!");
});
exports.api = functions.https.onRequest(app);
exports.helloWorld = functions.https.onRequest((_request, response) => {
response.send("Hello from Firebase!");
});
tl;dr
An example of what I'm hoping to accomplish is here, but none of the code examples worked for me. I get I get a 404 error with each one.
The Express Documentation here shows a similar HelloWorld example, but I confuse how Firebase takes the place of app.listen(3000, () => console.log('Example app listening on port 3000!'))
Is cors working properly in my example code? Although I get the expected response and log, Chrome console warns: Cross-Origin Read Blocking (CORB) blocked cross-origin response https://appengine.google.com/_ah/lo....
I have a Slack App that is hitting these URLs (I hit them with chrome too). Eventually, I'd like to use Botkit middleware in my Google Cloud Functions. I don't yet grasp proper setup of Express app.use() and app.get()
Answer
I made a simple mistake by treating /api as a function when it's actually a part of the path.
By using this URL with the trailing /
https://us-central1-myapp.cloudfunctions.net/api/
I'm now hitting the Express route and function.