I'm trying to build a scraping script to get a bunch of Discord server's total members. I actually did that with Puppeteer like below but I think my IP address has been banned because I'm getting "Invite Invalid" error from Discord even though invite links are working.
My question is that does Discord have APIs to get any server's total member count? Or is there any 3rd party library for that purpose? Or any other method?
const puppeteer = require('puppeteer')
const discordMembers = async ({ server, browser }) => {
if (!server) return
let totalMembers
const page = await browser.newPage()
try {
await page.goto(`https://discord.com/invite/${server}`, {
timeout: 3000
})
const selector = '.pill-qMtBTq'
await page.waitForSelector(selector, {
timeout: 3000
})
const totalMembersContent = await page.evaluate(selector => {
return document.querySelectorAll(selector)[1].textContent
}, selector)
if (totalMembersContent) {
totalMembers = totalMembersContent
.replace(/ Members/, '')
.replace(/,/g, '')
totalMembers = parseInt(totalMembers)
}
} catch (err) {
console.log(err.message)
}
await page.close()
if (totalMembers) return totalMembers
}
const asyncForEach = async (array, callback) => {
for (let i = 0; i < array.length; i++) {
await callback(array[i], i, array)
}
}
const run = async () => {
const browser = await puppeteer.launch({
headless: true,
args: ['--no-sandbox']
})
const servers = ['tQp4pSE', '3P5K3dzgdB']
await asyncForEach(servers, async server => {
const members = await discordMembers({ server, browser })
console.log({ server, members })
// result
// { server: 'tQp4pSE', members: 57600 }
// { server: '3P5K3dzgdB', members: 159106 }
})
await browser.close()
}
run()
Update: Mar 22, 2022
Thanks for #Vaviloff's answer we can actually access Discord's private APIs but the problem is it's only accessible over browser. I'm getting Request failed with status code 400 issue from Axios. Is it a CORS issue? How do we get the results in a Node.js app?
const axios = require('axios')
const discordMembers = async ({ server }) => {
try {
const apiResult = await axios({
data: {},
method: 'get',
url: `https://discord.com/api/v9/invites/${server}?with_counts=true&with_expiration=true`
})
console.log(apiResult)
} catch (err) {
console.log(err)
}
}
discordMembers({ server: 'tQp4pSE' })
A lot of modern web applications have their own internal APIs. Oftentimes you can spot frontend making requests to it, by using Networking tab in Devtools (filter by Fetch/XHR type):
Such API endpoints can change any time of course, but usually the last for a long time and is a rather convenient way of scraping
Currently Discord uses this URL for basic instance description:
https://discord.com/api/v9/invites/tQp4pSE?with_counts=true&with_expiration=true
By accessing it you get the desired data:
Update
To make your code work don't send any data in the request:
const apiResult = await axios({
method: 'get',
url: `https://discord.com/api/v9/invites/${server}?with_counts=true&with_expiration=true`
})
Related
New to Svelte and am running into some issues.
Currently doing the following in +page.server.js
I would like to poll this API every couple hundred milliseconds, I am unsure how to do that. I have tried using set Interval here to no avail.
export async function load({params}) {
const response = await fetch(
`http://localhost:9595/api/v1/chrysalis/example?uid=${params.uid}`
)
const site = await response.json()
const siteData = site[0]
console.log(siteData)
return {
uid: params.uid,
partitions: siteData.partitions,
zones: siteData.zones,
zTypes: siteData.zTypes,
zStates: siteData.zStates,
zNames: siteData.zNames
}
}
For example, I've built this in next.Js using SWR with refreshInterval: 1.
const {data, error, isLoading} = useSWR(
'http://localhost:9595/api/v1/chrysalis/example',
(url) => {
const searchParams = new URLSearchParams();
searchParams.append("uid", body.uid)
const newUrl = `${url}?${searchParams.toString()}`
const options = {
method: 'GET',
headers: {'Content-Type': 'application/json'},
}
return fetch(newUrl, options).then(res => res.json())
},
{
refreshInterval: 1
}
);
I have also tried to do the following onMount of the +page.svelte but when trying to hit the API from the client I get CORS error.( ran into this before if +page.js was not +page.server.js
let x;
onMount(async () => {
setInterval(async () => {
const response = await fetch(
`http://localhost:9595/api/v1/chrysalis/example?uid=${data.uid}`
)
const site = await response.json()
x = site[0]
console.log(x)
}, 3000)
})
The CORS error results because +page.svelte/+page.js are run in the browser. So you need to proxy the call through a service that allows being called from the browser. (Or relax the CORS restrictions on http://localhost:9595)
You can use SvelteKit itself to proxy the call by creating an internal endpoint. So:
The internal endpoint simply fetches http://localhost:9595/... and returns the results. (You can just forward the response object from fetch())
+page.svelte calls that internal endpoint from setInterval().
API
(https://api.mosmarts.com/truck/v0/api.php)
The API is scripted in PHP and accepts GET & POST commands and in return it responds back with a JSON response data.
To retrieve data the API requires “functionality” and “action” among other params as show below.
Command for retrieving all truck
Command for retrieving all truck
Payloads
{
"functionality" : "get",
"action" : "get_all_truck"
}
Command to retrieving truck inspection details by id
Payloads
{
"functionality" : "get",
"action" : "get_inspection_history",
"truckId" : "1"
}
NB: you will get truckId from command "get_all_truck" above
What’s expected from you
As the software developer you are tasked to design and develop a web-based backend solution that will have:
Dashboard: -
• Retrieve and indicate total number of trucks
• Retrieve and indicate number of inspection repairs requested 2. List all Trucks: -
• Implement search option
Inspection List: -
• Implement filter by truck
i have some code using express.js bt i get is a 404 error, no data retrieved.
app.js
const apiCallFromRequest = require('./Request')
const apiCallFromNode = require('./NodeJsCall')
const http = require('http')
http.createServer((req, res) => {
if(req.url === "/request"){
apiCallFromRequest.callApi(function(response){
//console.log(JSON.stringify(response));
res.write(JSON.stringify(response));
res.end();
});
}
else if(req.url === "/node"){
apiCallFromNode.callApi(function(response){
res.write(response);
res.end();
});
}
// res.end();
}).listen(3000);
console.log("service running on 3000 port....");
NodeJsCall.js
const https = require('https');
_EXTERNAL_URL = 'https://api.mosmarts.com/truck/v0/api.php';
const callExternalApiUsingHttp = (callback) => {
https.get(_EXTERNAL_URL, (resp) => {
let data = '';
// A chunk of data has been recieved.
resp.on('data', (chunk) => {
data += chunk;
});
// The whole response has been received. Print out the result.
resp.on('end', () => {
return callback(data);
// console.log(JSON.stringify(data));
});
}).on("error", (err) => {
console.log("Error: " + err.message);
});
}
module.exports.callApi = callExternalApiUsingHttp;
Request.js
const request = require('request');
_EXTERNAL_URL = 'https://api.mosmarts.com/truck/v0/api.php';
const callExternalApiUsingRequest = (callback) => {
request(_EXTERNAL_URL, { json: true }, (err, res, body) => {
if (err) {
return callback(err);
}
return callback(body);
});
}
module.exports.callApi = callExternalApiUsingRequest;
Hey Gerald you can find a simple response for this kind of question on google.
if you are a real beginner I would propose you the Axios npm.
here is an example of a really simple GET request with axios.
axios.get('https://api.github.com/users/mapbox')
.then(response => {
console.log(response.data.created_at);
});
I working with localStorage token in my next.js application. I tried to get the localStorage on page getInitialProps but, it returns undefined.
Here is an example,
Dashboard.getInitialProps = async () => {
const token = localStorage.getItem('auth');
const res = await fetch(`${process.env.API_URL}/pages/about`, {
headers: { 'Authorization': token }
});
const data = await res.json();
return { page: data };
}
For the initial page load, getInitialProps will run on the server
only. getInitialProps will then run on the client when navigating to a
different route via the next/link component or by using next/router. Docs
This means you will not be able to access localStorage(client-side-only) all the time and will have to handle it:
Dashboard.getInitialProps = async ({ req }) => {
let token;
if (req) {
// server
return { page: {} };
} else {
// client
const token = localStorage.getItem("auth");
const res = await fetch(`${process.env.API_URL}/pages/about`, {
headers: { Authorization: token },
});
const data = await res.json();
return { page: data };
}
};
If you want to get the user's token for the initial page load, you have to store the token in cookies instead of localStorage which #alejandro also mentioned in the comment.
I am trying to login via the webfront end and trying to intercept a cookie and then using that in the subsequent API request. I am having trouble getting the cookie back into the GET request. Code posted below.
import https from 'https';
import { bitbucketUser } from "../userRole.js"
import { ClientFunction } from 'testcafe';
fixture `Request/Response API`
// .page `https://myurl.company.com/login`
.beforeEach(async t => {
await t.useRole(bitbucketUser)
});
test('test', async t => {
const getCookie = ClientFunction(() => {
return document.cookie;
});
var mycookie = await getCookie()
const setCookie = ClientFunction(mycookie => {
document.cookie = mycookie;
});
var validatecookie = await getCookie()
console.log(validatecookie)
const executeRequest = () => {
return new Promise(resolve => {
const options = {
hostname: 'myurl.company.com',
path: '/v1/api/policy',
method: 'GET',
headers: {
'accept': 'application/json;charset=UTF-8',
'content-type': 'application/json'
}
};
const req = https.request(options, res => {
console.log('statusCode:', res.statusCode);
console.log('headers:', res.headers);
let body = "";
res.on("data", data => {
body += data;
});
res.on("end", () => {
body = JSON.parse(body);
console.log(body);
});
resolve();
});
req.on('error', e => {
console.error(e);
});
req.end();
});
};
await setCookie(mycookie)
await executeRequest();
});
I have tried several examples but am quite not able to figure what is it that I am missing.
When you call the setCookie method, you modify cookies in your browser using the ClientFunction.
However, when you call your executeRequest method, you run it on the server side using the nodejs library. When you set cookies on the client, this will not affect your request sent from the server side. You need to add cookie information directly to your options object as described in the following thread: How do I create a HTTP Client Request with a cookie?.
In TestCafe v1.20.0 and later, you can send HTTP requests in your tests using the t.request method. You can also use the withCredentials option to attach all cookies to a request.
Please also note that TestCafe also offers a cookie management API to set/get/delete cookies including HTTPOnly.
I'm developing a Vue.js application which has only frontend (no server) and send a lot of requests to different APIs. The originally quite simple app became more complex. And there are problems with some APIs, because browsers do not accept the responses due to CORS. That is why I'm trying to test, if I can migrate the app to Nuxt.js.
My approach is as follows (inspired by this comment), but I expect, that there is probably a better way to send the requests from the client over the server.
pages/test-page.vue
methods: {
async sendRequest(testData) {
const response = await axios.post('api', testData)
// Here can I use the response on the page.
}
}
nuxt.config.js
serverMiddleware: [
{ path: '/api', handler: '~/server-middleware/postRequestHandler.js' }
],
server-middleware/postRequestHandler.js
import axios from 'axios'
const configs = require('../store/config.js')
module.exports = function(req, res, next) {
let body = ''
req.on('data', (data) => {
body += data
})
req.on('end', async () => {
if (req.hasOwnProperty('originalUrl') && req.originalUrl === '/api') {
const parsedBody = JSON.parse(body)
// Send the request from the server.
const response = await axios.post(
configs.state().testUrl,
body
)
req.body = response
}
next()
})
}
middleware/test.js (see: API: The Context)
export default function(context) {
// Universal keys
const { store } = context
// Server-side
if (process.server) {
const { req } = context
store.body = req.body
}
}
pages/api.vue
<template>
{{ body }}
</template>
<script>
export default {
middleware: 'test',
computed: {
body() {
return this.$store.body
}
}
}
</script>
When the user makes an action on the page "test", which will initiate the method "sendRequest()", then the request "axios.post('api', testData)" will result in a response, which contains the HTML code of the page "api". I can then extract the JSON "body" from the HTML.
I find the final step as suboptimal, but I have no idea, how can I send just the JSON and not the whole page. But I suppose, that there must be a much better way to get the data to the client.
There are two possible solutions:
Proxy (see: https://nuxtjs.org/faq/http-proxy)
API (see: https://medium.com/#johnryancottam/running-nuxt-in-parallel-with-express-ffbd1feef83c)
Ad 1. Proxy
The configuration of the proxy can look like this:
nuxt.config.js
module.exports = {
...
modules: [
'#nuxtjs/axios',
'#nuxtjs/proxy'
],
proxy: {
'/proxy/packagist-search/': {
target: 'https://packagist.org',
pathRewrite: {
'^/proxy/packagist-search/': '/search.json?q='
},
changeOrigin: true
}
},
...
}
The request over proxy can look like this:
axios
.get('/proxy/packagist-search/' + this.search.phpLibrary.searchPhrase)
.then((response) => {
console.log(
'Could get the values packagist.org',
response.data
)
}
})
.catch((e) => {
console.log(
'Could not get the values from packagist.org',
e
)
})
Ad 2. API
Select Express as the project’s server-side framework, when creating the new Nuxt.js app.
server/index.js
...
app.post('/api/confluence', confluence.send)
app.use(nuxt.render)
...
server/confluence.js (simplified)
const axios = require('axios')
const config = require('../nuxt.config.js')
exports.send = function(req, res) {
let body = ''
let page = {}
req.on('data', (data) => {
body += data
})
req.on('end', async () => {
const parsedBody = JSON.parse(body)
try {
page = await axios.get(
config.api.confluence.url.api + ...,
config.api.confluence.auth
)
} catch (e) {
console.log('ERROR: ', e)
}
}
res.json({
page
})
}
The request over API can look like this:
this.$axios
.post('api/confluence', postData)
.then((response) => {
console.log('Wiki response: ', response.data)
})
.catch((e) => {
console.log('Could not update the wiki page. ', e)
})
Now with nuxtjs3 :
nuxtjs3 rc release
you have fetch or useFetch no need to import axios or other libs, what is great, automatic parsing of body, automatic detection of head
fetching data
you have middleware and server api on same application, you can add headers on queries, hide for example token etc
server layer
a quick example here in vue file i call server api :
const { status } = await $fetch.raw( '/api/newsletter', { method: "POST", body: this.form.email } )
.then( (response) => ({
status: response.status,
}) )
.catch( (error) => ({
status: error?.response?.status || 500,
}) );
it will call a method on my server, to init the server on root directory i created a folder name server then api, and a file name newsletter.ts (i use typescript)
then in this file :
export default defineEventHandler(async (event) => {
const {REST_API, MAILINGLIST_UNID, MAILINGLIST_TOKEN} = useRuntimeConfig();
const subscriber = await readBody(event);
console.log("url used for rest call" + REST_API);
console.log("token" + MAILINGLIST_TOKEN);
console.log("mailing list unid" + MAILINGLIST_UNID);
let recipientWebDTO = {
email: subscriber,
subscriptions: [{
"mailingListUnid": MAILINGLIST_UNID
}]
};
const {status} = await $fetch.raw(REST_API, {
method: "POST",
body: recipientWebDTO,
headers: {
Authorization: MAILINGLIST_TOKEN,
},
}).then((response) => ({
status: response.status,
}))
.catch((error) => ({
status: error?.response?.status || 500,
}));
event.res.statusCode = status;
return "";
})
What are the benefits ?
REST_API,MAILING_LIST_UNID, MAILING_LIST_TOKEN are not exposed on
client and even file newsletter.ts is not available on debug browser.
You can add log only on server side You event not expose api url to avoid some attacks
You don't have to create a new backend just to hide some criticals token or datas
then it is up to you to choose middleware route or server api. You don't have to import new libs, h3 is embedded via nitro with nuxtjs3 and fetch with vuejs3
for proxy you have also sendProxy offered by h3 : sendProxy H3
When you build in dev server and client build in same time(and nothing to implement or configure in config file), and with build to o, just don deploy your project in static way (but i think you can deploy front in static and server in node i don't know)