Get countinuous json data from server before connection close in ktor client - kotlin

I'm using ktor client in my android project and I want to get countinous data before I can close the connection. Here is the example of nodeJS app that give data countinuosly.
const app = require("express")()
let count = 0
app.get("/", (req, res) => {
const headers = {
'Content-Type': 'text/event-stream',
'Connection': 'keep-alive',
'Cache-Control': 'no-cache'
}
res.writeHead(200, headers)
setInterval(() => {
res.write({data: count})
count++
}, 2000);
})
app.listen(3000, () => console.log("ok"))
I don't know how to implement this, and I couldn't find anything on the internet also. Any help will be appreciated.

You can read data from a response stream as soon as it becomes available for the client. Here is an example:
val client = HttpClient(CIO) {
install(HttpTimeout)
}
client.prepareGet("http://localhost:3000/") {
timeout {
requestTimeoutMillis = Long.MAX_VALUE
}
}.execute { response ->
val channel = response.bodyAsChannel()
while (!channel.isClosedForRead) {
val buffer = ByteBuffer.allocateDirect(1024)
val read = channel.readAvailable(buffer)
if (read == -1) break
buffer.flip()
val jsonString = Charsets.UTF_8.decode(buffer).toString()
println(jsonString)
}
}

Related

How to get total member count of any Discord server?

I'm trying to build a scraping script to get a bunch of Discord server's total members. I actually did that with Puppeteer like below but I think my IP address has been banned because I'm getting "Invite Invalid" error from Discord even though invite links are working.
My question is that does Discord have APIs to get any server's total member count? Or is there any 3rd party library for that purpose? Or any other method?
const puppeteer = require('puppeteer')
const discordMembers = async ({ server, browser }) => {
if (!server) return
let totalMembers
const page = await browser.newPage()
try {
await page.goto(`https://discord.com/invite/${server}`, {
timeout: 3000
})
const selector = '.pill-qMtBTq'
await page.waitForSelector(selector, {
timeout: 3000
})
const totalMembersContent = await page.evaluate(selector => {
return document.querySelectorAll(selector)[1].textContent
}, selector)
if (totalMembersContent) {
totalMembers = totalMembersContent
.replace(/ Members/, '')
.replace(/,/g, '')
totalMembers = parseInt(totalMembers)
}
} catch (err) {
console.log(err.message)
}
await page.close()
if (totalMembers) return totalMembers
}
const asyncForEach = async (array, callback) => {
for (let i = 0; i < array.length; i++) {
await callback(array[i], i, array)
}
}
const run = async () => {
const browser = await puppeteer.launch({
headless: true,
args: ['--no-sandbox']
})
const servers = ['tQp4pSE', '3P5K3dzgdB']
await asyncForEach(servers, async server => {
const members = await discordMembers({ server, browser })
console.log({ server, members })
// result
// { server: 'tQp4pSE', members: 57600 }
// { server: '3P5K3dzgdB', members: 159106 }
})
await browser.close()
}
run()
Update: Mar 22, 2022
Thanks for #Vaviloff's answer we can actually access Discord's private APIs but the problem is it's only accessible over browser. I'm getting Request failed with status code 400 issue from Axios. Is it a CORS issue? How do we get the results in a Node.js app?
const axios = require('axios')
const discordMembers = async ({ server }) => {
try {
const apiResult = await axios({
data: {},
method: 'get',
url: `https://discord.com/api/v9/invites/${server}?with_counts=true&with_expiration=true`
})
console.log(apiResult)
} catch (err) {
console.log(err)
}
}
discordMembers({ server: 'tQp4pSE' })
A lot of modern web applications have their own internal APIs. Oftentimes you can spot frontend making requests to it, by using Networking tab in Devtools (filter by Fetch/XHR type):
Such API endpoints can change any time of course, but usually the last for a long time and is a rather convenient way of scraping
Currently Discord uses this URL for basic instance description:
https://discord.com/api/v9/invites/tQp4pSE?with_counts=true&with_expiration=true
By accessing it you get the desired data:
Update
To make your code work don't send any data in the request:
const apiResult = await axios({
method: 'get',
url: `https://discord.com/api/v9/invites/${server}?with_counts=true&with_expiration=true`
})

RN "TypeError: Network request failed" - production - random

I know some questions about the subject has been opened here and there, but my issue is different :
all the other ones appear in dev mode, in my case it's in production,
a very big percentage of requests pass, a few of them is TypeError: Network request failed - but sometimes for critical requests
it's random, not always the same request. Sometimes it passes, sometimes not.
it appears to three on my projects, one is on AWS the other one on Clever-Cloud, both are projects between 1000 and 5000 users, servers are quite too big for what they do - I think I removed the risk of a server fault. Even if... I can reproduce locally when I don't start the api locally. So it's like the api is not responding, but as I said, I don't think so.
I have no clue where to dig anymore...
I can give you my API.js service file, maybe you'll find what's wrong ?
import URI from 'urijs';
import { Platform } from 'react-native';
import NetInfo from '#react-native-community/netinfo';
import { getUserToken, wipeData } from '../utils/data';
import { SCHEME, MW_API_HOST } from '../config';
import deviceInfoModule from 'react-native-device-info';
import { capture } from '../utils/sentry';
const unauthorisedHandler = (navigation) => {
wipeData();
navigation.reset({ index: 0, routes: [{ name: 'Auth' }] });
};
const checkNetwork = async (test = false) => {
const isConnected = await NetInfo.fetch().then((state) => state.isConnected);
if (!isConnected || test) {
await new Promise((res) => setTimeout(res, 1500));
return false;
}
return true;
};
class ApiService {
host = MW_API_HOST;
scheme = SCHEME;
getUrl = (path, query) => {
return new URI().host(this.host).scheme(this.scheme).path(path).setSearch(query).toString();
};
execute = async ({ method = 'GET', path = '', query = {}, headers = {}, body = null }) => {
try {
const config = {
method,
headers: {
'Content-Type': 'application/json',
Accept: 'application/json',
appversion: deviceInfoModule.getBuildNumber(),
appdevice: Platform.OS,
currentroute: this.navigation?.getCurrentRoute?.()?.name,
...headers,
},
body: body ? JSON.stringify(body) : null,
};
const url = this.getUrl(path, query);
console.log('url: ', url);
const canFetch = await checkNetwork();
if (!canFetch) return;
let response;
// To try to avoid mysterious `TypeError: Network request failed` error
// that throws an error directly
// we try catch and try one more time.
try {
response = await fetch(url, config);
} catch (e) {
if (e?.toString().includes('Network request failed')) {
// try again
await new Promise((res) => setTimeout(res, 250));
console.log('try again because Network request failed');
response = await fetch(url, config);
} else {
throw e;
}
}
if (!response.ok) {
if (response.status === 401) {
const token = await getUserToken();
if (token) unauthorisedHandler(API.navigation);
return response;
}
}
if (response.json) return await response.json();
return response;
} catch (e) {
capture(e, { extra: { method, path, query, headers, body } });
return { ok: false, error: "Sorry, an error occured, technical team has been warned." };
}
};
executeWithToken = async ({ method = 'GET', path = '', query = {}, headers = {}, body = null }) => {
const token = await getUserToken();
if (token) headers.Authorization = token;
return this.execute({ method, path, query, headers, body });
};
get = async (args) => this.executeWithToken({ method: 'GET', ...args });
post = async (args) => this.executeWithToken({ method: 'POST', ...args });
put = async (args) => this.executeWithToken({ method: 'PUT', ...args });
delete = async (args) => this.executeWithToken({ method: 'DELETE', ...args });
}
const API = new ApiService();
export default API;
Talking with experts here and there, it seems that it's normal : internet network is not 100% reliable, so sometimes, request fail, for a reason that we can't anticipate (tunnel, whatever).
I ended up using fetch-retry and I still have a few of those, but much less !

Verify HMAC Hash Using Cloudflare Workers

I'm trying to verify a HMAC signature received from a WebHook. The details of the WebHook are https://cloudconvert.com/api/v2/webhooks#webhooks-events
This says that the HMAC is generated using hash_hmac (PHP) and is a SHA256 hash of the body - which is JSON. An example received is:
c4faebbfb4e81db293801604d0565cf9701d9e896cae588d73ddfef3671e97d7
This looks like lowercase hexits.
I'm trying to use Cloudflare Workers to process the request, however I can't verify the hash. My code is below:
const encoder = new TextEncoder()
addEventListener('fetch', event => {
event.respondWith(handleRequest(event.request))
})
async function handleRequest(request) {
const contentType = request.headers.get('content-type') || ''
const signature = request.headers.get('CloudConvert-Signature')
let data
await S.put('HEADER', signature)
if (contentType.includes('application/json')) {
data = await request.json()
await S.put('EVENT', data.event)
await S.put('TAG', data.job.tag)
await S.put('JSON', JSON.stringify(data))
}
const key2 = await crypto.subtle.importKey(
'raw',
encoder.encode(CCSigningKey2),
{ name: 'HMAC', hash: 'SHA-256' },
false,
['sign']
)
const signed2 = await crypto.subtle.sign(
'HMAC',
key2,
encoder.encode(JSON.stringify(data))
)
await S.put('V22', btoa(String.fromCharCode(...new Uint8Array(signed2))))
return new Response(null, {
status: 204,
headers: {
'Cache-Control': 'no-cache'
}
})
}
This will generate a hash of:
e52613e6ecebdf98bb085f04ca1f91bf9a5cf1dc085f89dcaa3e5fbf5ebf1b06
I've tried use the crypto.subtle.verify method, but that didn't work.
Can anyone see any issues with the code? Or have done this successfully using Cloudflare Workers?
Mark
I finally got this working using the verify method (I had previously tried the verify method, but it didn't work). The main problem seems to the use of request.json() wrapped in JSON.stringify. Changing this to request.text() resolved the issue. I can then use JSON.parse to access the data after verifying the signature. The code is as follows:
const encoder = new TextEncoder()
addEventListener('fetch', event => {
event.respondWith(handleRequest(event.request))
})
async function handleRequest(request) {
const signature = request.headers.get('CloudConvert-Signature')
const key = await crypto.subtle.importKey(
'raw',
encoder.encode(CCSigningKey2),
{ name: 'HMAC', hash: 'SHA-256' },
false,
['verify']
)
const data = await request.text()
const verified = await crypto.subtle.verify(
'HMAC',
key,
hexStringToArrayBuffer(signature),
encoder.encode(data)
)
if (!verified) {
return new Response('Verification failed', {
status: 401,
headers: {
'Cache-Control': 'no-cache'
}
})
}
return new Response(null, {
status: 204,
headers: {
'Cache-Control': 'no-cache'
}
})
}
function hexStringToArrayBuffer(hexString) {
hexString = hexString.replace(/^0x/, '')
if (hexString.length % 2 != 0) {
return
}
if (hexString.match(/[G-Z\s]/i)) {
return
}
return new Uint8Array(
hexString.match(/[\dA-F]{2}/gi).map(function(s) {
return parseInt(s, 16)
})
).buffer
}

Make a get request until a response comes from a request post vue js

I should upload file in my vue.js app. When I browse a file, I make post request and until I get a response, I need to make get request every 2 seconds for example. How can I do this?
uploadFile(event) {
this.isLoadingProcess = true;
console.log(this.isLoadingProcess);
let data = new FormData();
this.file = event.target.files[0];
data.append('name', 'uploaded-file');
data.append('file', event.target.files[0]);
const options = {
headers: {
'Content-Type': event.target.files[0].type,
},
onUploadProgress: function (progressEvent) {
const {loaded, total} = progressEvent;
let percentCompleted = Math.floor((loaded * 100) / total);
//console.log(percentCompleted);
if (percentCompleted !== 100) {
axios.get(url, {data: progressId})
.then(response => {
this.progress = response.data.progress;
// console.log(this.progress);
console.log(response.data.progress);
})
}
},
};
axios.post(
url,
data,
options
)
.then(response => {
console.log(response);
})
.finally(()=> {
this.isLoadingProcess = false;
})
what about this?
It asks api about update more than every 2 seconds, but It's better for my opinion in case, when endpoint isn't available
var processTimeoutFunction = null;
function checkProgress() {
axios.get('ckeck-process', ...).then(response => {
if (response.processIsActive) {
processTimeoutFunction = setTimeout(() => checkProgress(), 2000);
} else {
clearTimeout(processTimeoutFunction);
this.isLoadingProcess = false;
})
}

How to passing result of http request inside async in ExpressJS?

I have below code
async send(user, data) {
const postData = {
'data': 'john',
'secret': 'secret'
};
const dataJson = JSON.stringify(postData);
const options = {
hostname: 'example.com',
path: '/send',
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Content-Length': dataJson.length
}
};
const req = https.request(options, (res) => {
let data = '';
console.log('Status Code:', res.statusCode);
res.on('data', (chunk) => {
data += chunk;
});
res.on('end', () => {
console.log('Body: ', JSON.parse(data));
});
}).on("error", (err) => {
console.log("Error: ", err.message);
});
req.write(dataJson);
req.end();
//---------------
let postResult = // HERE I WANT TO GET WHAT HTTP POST REQUESTED (e.g dataJson.body?)
//---------------
let result;
try {
result = await this.users.collection('users').updateOne(
{
_id: user
},
{
$set: {
// I WANT TO USE THAT HERE
data1 : postResult,
data2 : data2
}
},
{ maxTimeMS: consts.DB_MAX_TIME_USERS }
);
} catch (err) {
log.error('DB', 'UPDATEFAIL id=%s error=%s', user, err.message);
err.message = 'Database Error, failed to update user';
err.code = 'InternalDatabaseError';
throw err;
}
return { success: true };
}
How to get those data to outside variable?
I almost crazy about this, been searching on google and not found anything
I am using express and native-http to make http request, are there any native-curl maybe?
thank you very much for all the help
Your current code is using callback to retrieve result, so you can initiate data variable to outside callback function
let data = '';
const req = https.request(options, (res) => {
console.log('Status Code:', res.statusCode);
res.on('data', (chunk) => {
data += chunk;
});
res.on('end', () => {
console.log('Body: ', JSON.parse(data));
});
})
And also there are other easier way to make http request with nodejs. you can check axios that support Promise and async/await.
you can use syntax like this with axios
const response = await axios.get('/user?ID=12345');
way more easier.