Capturing Websocket Messages From Network Tab - chromium

Using puppeteer I am trying to read websocket messages and put them into a string to output onto the console. To see an example of some websocket messages you can visit the link provided below within the code and then follow the steps within the parenthesis. (CTRL+SHIFT+I > Network > WS (websocket) > Messages)
var puppeteer = require('puppeteer');
async function run() {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('http://powerline.io');
console.log('navigated successfully');
page.on('response', response => {
const isWSS = ['websocket'].includes(response.request().resourceType())
if (isWSS){
console.log(isWSS);
log(response.url());
response.text().then(log)
}
})
}
run();

Here is an example, how it can be captured:
var puppeteer = require('puppeteer');
async function run() {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('http://powerline.io');
const cdp = await page.target().createCDPSession();
await cdp.send('Network.enable');
await cdp.send('Page.enable');
const printResponse = response => console.log('response: ', response);
cdp.on('Network.webSocketFrameReceived', printResponse); // Fired when WebSocket message is received.
cdp.on('Network.webSocketFrameSent', printResponse); // Fired when WebSocket message is sent.
}
run();
Read more about network events.

Related

How to emit an event in express js and handle it in discord.js

I have a simple express.js backend that handles a get request from discord which runs a heavy process in the background using bull and throng.
I got it all working, Discord sends the get request. backend reserves the request and adds it to the job queue, once it's done I receive the returned value in the Queue.on() listener.
My problem is how can I send this result I get in the queue.on() listener back to discord as a message. how can I make discord listen to this event and post the result?
Note: discord and express are 2 separate apps. can't integrate them into one project.
Index.js file:
const express = require('express')
const scrapTikTok = require('./tiktokScraping')
const Queue = require('bull')
const app = express()
// Initialize BULL
const REDIS_URL = process.env.REDIS_URL || 'redis://127.0.0.1:6379'
const workQueue = new Queue('work', REDIS_URL, {
defaultJobOptions: {
removeOnComplete: true,
removeOnFail: true
}
})
app.get('/scrap', async (req, res) => {
res.send('Working on it...')
await scrapTikTok()
})
app.get('/url', async (req, res) => {
const url = 'https://www.tiktok.com/#tristanvincentt'
let job = await workQueue.add({url: url})
res.json({id: job.id})
})
workQueue.on('global:completed', (jobId, result) => {
console.log(`Job completed with result ${result}`)
// SEND RESULT BACK TO DISCORD
})
app.listen(process.env.PORT || 3000, () => {
console.log('App is running...')
})
Worker file:
const throng = require('throng')
const Queue = require("bull")
const REDIS_URL = process.env.REDIS_URL || "redis://127.0.0.1:6379"
const workers = process.env.WEB_CONCURRENCY || 1
const maxJobsPerWorker = 50
const sleep = ms => new Promise(r => setTimeout(r, ms));
function start() {
const workQueue = new Queue('work', REDIS_URL)
workQueue.process(maxJobsPerWorker, async job => {
console.log(job.data.url)
await sleep(10000)
return {value: job.id}
})
}
throng({workers, start})
you can create another express server in your discord app. and add a post endpoint:
const express = require('express')
const app = express()
app.post('/secreturl', async (req, res) => {
res.send('Discordjs Received the data!')
let jobId = req.body.jobId
let result = req.body.result
//now you have the results here!
})
then when the queue is done (based on the code you provided):
workQueue.on('global:completed', (jobId, result) => {
console.log(`Job completed with result ${result}`)
// SEND RESULT BACK TO DISCORD:
axios.post('(your-discordjs-server-ip)/secreturl', {
jobId: jobId,
result: result
})
})
just make sure you add the axios package to the queue app
npm install axios
and require it in the index.js file
const axios = require("axios")
tick this as the answer if it solved your problem!

Puppeteer - Login falied only at automation

I'm trying to do login at a web page but when I try to do it through Puppeteer it fails.
The code below should redirect with success to https://prenotami.esteri.it/UserArea but it fails and redirect to https://prenotami.esteri.it/Home/Login
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch({headless:false, ignoreDefaultArgs: ['--enable-automation']});
const page = await browser.newPage();
await page.goto('https://prenotami.esteri.it/UserArea');
await page.evaluate(() => {
document.querySelector('#login-email').value = "****#gmail.com"
document.querySelector('#login-password').value = "pass***"
document.querySelector('button[type="submit"]').click()
})
})();
As it can be seen, I've tried to disable automation.I also tried to put some delay but I couldn't realize where exactly
SOLVED:
I've changed the code and it works now
const extra = require('puppeteer-extra');
const stealth = require('puppeteer-extra-plugin-stealth')
extra.use(stealth())
extra
.launch({
headless: false,
args: ['--no-sandbox']
})
.then(async browser => {
const page = await browser.newPage();
await page.goto('https://prenotami.esteri.it/');
page.setDefaultNavigationTimeout(30000)
await page.type("#login-email", "*****#gmail.com")
await page.type("#login-password", "*****pass")
await page.click('button[type="submit"]')
await page.waitForNavigation({ waitUntil: 'domcontentloaded' })
await page.goto("https://prenotami.esteri.it/Services/Booking/751")
await page.waitForNavigation({ waitUntil: 'domcontentloaded' })

Slackbot not able to pick up messages

I'm trying to create a simple slackbot and I can get the thing to run on start and almost anything else, but it will not pick up incoming messages from any channel. It's not generating any errors what so ever which makes it very frusterating.
things I've tried so far:
adding the bot to the channel
console logging inside the message block
adding a catch
No matter what I do the bot seems to just outright ignore the prompt "on.message". I don't get it. My code is below.
require("dotenv").config();
const { WebClient } = require("#slack/web-api");
const { createEventAdapter } = require("#slack/events-api");
const slackEvents = createEventAdapter(process.env.SLACK_SIGNING_SECRET);
const { RTMClient } = require("#slack/rtm-api");
const axios = require('axios')
const sgMail = require("#sendgrid/mail");
sgMail.setApiKey(process.env.SENDGRID_API_KEY);
const express = require("express");
const bodyParser = require("body-parser");
const pool = require("./pool");
const app = express();
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));
app.use(express.static("build"));
let config = {
headers: {
"X-Auth-Client": process.env.BG_AUTH_CLIENT,
"X-Auth-Token": process.env.BG_AUTH_TOKEN,
},
};
const PORT = process.env.PORT || 5000;
app.listen(PORT, () => {
console.log("server running on: ", PORT);
});
const token = process.env.SLACK_TOKEN;
const web = new WebClient(token);
const conversationId = "C0139RJPUEM";
// The client is initialized and then started to get an active connection to the platform
const rtm = new RTMClient(token);
slackEvents.on('error', console.error);
slackEvents.start().then(() => {
(async () => {
const res = await web.chat.postMessage({
icon_emoji: ":email:",
channel: conversationId,
text: "Testing connection",
});
message
console.log("Message sent: ", res.ts);
})();
console.log("bot listening on port", PORT);
});
slackEvents.on("message", (event) => {
console.log(
`Received a message event: user ${event.user} in channel ${event.channel} says ${event.text}`
);
(async () => {
// See: https://api.slack.com/methods/chat.postMessage
const res = await web.chat.postMessage({
icon_emoji: ":email:",
channel: conversationId,
text: "Testing message",
});
// `res` contains information about the posted message
console.log("Message sent: ", res.ts);
})();
// }
});
I think I got it, I was using a library that was not compatible, plus I didn't set up API event subscriptions and a few other dumb things. Oh well. Hopefully this will help someone who might casually come across this.

page.goto (puppeteer) does not stop calling itself when opening the same url it is navigating to

I am trying to convert an html web page into a pdf file by using puppeteer. It is converting the html web page into pdf file but the problem is that, page.goto is not stopping after its job is done. When this happens, the terminal I run this localhost server from becomes a little unresponsive. I also deployed it on heroku but when I visit the url, the server responds me with error code 503 and it does not convert to pdf there.
let printPDF = async () => {
try {
const browser = await puppeteer.launch({ headless: true});
const page = await browser.newPage();
await page.goto('http://localhost:3000/preview')
await page.pdf({ format: 'A4' , path: `${__dirname}/contact.pdf`, printBackground: true});
await page.close();
await browser.close();
console.log("repeat") //logging to see its repetition
}
catch (error) {
await page.close();
await browser.close();
}
}
getReviewPage = ((req, res) => {
printPDF()
res.sendFile(`${__dirname}/form.html`)
})
app.route('/preview').get(getReviewPage);
Since printPDF is an async function returning Promise, you need to wait for it to finish, with then for example:
const getReviewPage = (req, res) => {
printPDF().then(() => res.sendFile(`${__dirname}/form.html`)
}

How do I split my Jest + Puppeteer tests in multiple files?

I am writing automated tests using Jest & Puppeteer for a Front-end application written in Vue.js
So far I managed to write a set of tests, but they all reside in the same file:
import puppeteer from 'puppeteer';
import faker from 'faker';
let page;
let browser;
const width = 860;
const height = 1080;
const homepage = 'http://localhost:8001/brt/';
const timeout = 1000 * 16;
beforeAll(async () => {
browser = await puppeteer.launch({
headless: false, // set to false if you want to see tests running live
slowMo: 30, // ms amount Puppeteer operations are slowed down by
args: [`--window-size=${width},${height}`],
});
page = await browser.newPage();
await page.setViewport({ width, height });
});
afterAll(() => {
browser.close();
});
describe('Homepage buttons', () => {
test('Gallery Button', async () => {
// navigate to the login view
await page.goto(homepage);
await page.waitFor(1000 * 0.5); // without this, the test gets stuck :(
await page.waitForSelector('[data-testid="navBarLoginBtn"]');
await page.click('[data-testid="navBarLoginBtn"]'),
await page.waitForSelector('[data-testid="navBarGalleryBtn"]');
await page.click('[data-testid="navBarGalleryBtn"]'),
// test: check if we got to the gallery view (by checking nr of tutorials)
await page.waitForSelector('.card-header');
const srcResultNumber = await page.$$eval('.card-header', (headers) => headers.length);
expect(srcResultNumber).toBeGreaterThan(1);
}, timeout);
});
describe('Register', () => {
const btnLoginToRegister = '#btn-login-to-register';
const btnRegister = '#btn-register';
const btnToLogin = '#btn-goto-login';
test('Register failed attempt: empty fields', async () => {
// navigate to the register form page via the login button
await page.goto(homepage);
await page.waitForSelector(navLoginBtn);
await page.click(navLoginBtn);
await page.waitForSelector(btnLoginToRegister);
await page.click(btnLoginToRegister);
// test; checking for error messages
await page.waitForSelector(btnRegister);
await page.click(btnRegister);
const errNumber = await page.$$eval('#errMessage', (err) => err.length);
expect(errNumber).toEqual(3);
}, timeout);
test('Register failed: invalid char count, email format', async () => {
// fill inputs
await page.waitForSelector('#userInput');
await page.type('#userInput', 'a');
await page.waitForSelector('#emailInput');
await page.type('#emailInput', 'a');
await page.waitForSelector('#emailInput');
await page.type('#passInput', 'a');
await page.waitForSelector(btnRegister);
await page.click(btnRegister);
// test: check if we 3 errors (one for each row), from the front end validations
const err = await page.$$eval('#errMessage', (errors) => errors.length);
expect(err).toEqual(3);
}, timeout);
test('Register: success', async () => {
await page.click('#userInput', { clickCount: 3 });
await page.type('#userInput', name1);
await page.click('#emailInput', { clickCount: 3 });
await page.type('#emailInput', email1);
await page.click('#passInput', { clickCount: 3 });
await page.type('#passInput', password1);
await page.waitForSelector(btnRegister);
await page.click(btnRegister);
// test: check if go to login link appeared
await page.waitForSelector(btnToLogin);
await page.click(btnToLogin);
// await Promise.all([
// page.click(btnToLogin),
// page.waitForNavigation(),
// ]);
}, timeout);
test('Register failed: email already taken', async () => {
// navigate back to the register form
await page.waitForSelector(btnLoginToRegister);
await page.click(btnLoginToRegister);
await page.click('#userInput');
await page.type('#userInput', name2);
await page.click('#emailInput');
await page.type('#emailInput', email1); // <- existing email
await page.click('#passInput');
await page.type('#passInput', password2);
await page.click(btnRegister);
const err = await page.$eval('#errMessage', (e) => e.innerHTML);
expect(err).toEqual('Email already taken');
}, timeout);
});
I would like to be able to have a single test file that does the beforeAll and afterAll stuff, and each test suite: HomepageButtons, Register, etc. to reside in it's own test file. How would I be able to achieve this?
I've tried splitting tets into:
testsUtils.js that would contain the beforeAll and afterAll hooks and code but it doesn't guarantee that it runs when it needs: the beforeAll code to fire before all other test files and the afterAll code to fire after all the test files finished.
Sorry, I'd rather comment on your question, but I don't have reputation for that. Anyway, I think that you are looking for something like a "global beforeAll" and "global afterAll" hooks, right? Jest has it alread. It's called "globalSetup" and "globalTeardown".
Take a look at globalSetup. Excerpt:
This option allows the use of a custom global setup module which
exports an async function that is triggered once before all test
suites.
The Global Teardown one goes the same.
I think you'll have a headache trying to get a reference to the page or browser in globalSetup/globalTeardown and I confess that I never try this. Maybe the answer for that problem (if you have it) is on this page, under "Custom example without jest-puppeteer preset section.
Also there is a repo that tries to facilitate Jest + Puppeteer integration. Maybe you find it util: repo.
Good luck. :)