I am developing a backend API using Cloudflare Workers to cache the tokens into respective individual pages, something like http://cache.api/[tokenId] with token value itself as body content, using Cache API.
const tokenId = 'fakeJWT';
const internalUrl = ''.concat(
'http://cache.api/',
tokenId // the query string
);
const cacheUrl = new URL(internalUrl);
const cacheKey = new Request(cacheUrl.toString());
const cache = caches.default;
let response = new Response(tokenId);
response.headers.append('Cache-Control', 's-maxage=86400'); // 24 hours
await cache.put(cacheKey, response.clone());
I've configured Cache-Control header with 24 hours expiry. Then, I wrote another API in the same Cloudflare Workers to check the existence of the cache, it exists after 10 minutes but does not exist after 15 minutes.
const internalUrl = ''.concat(
'http://cache.api/',
tokenId // the query string
);
const cacheUrl = new URL(internalUrl);
const cacheKey = new Request(cacheUrl.toString());
const cache = caches.default;
let response = await cache.match(cacheKey);
if (!response) {
console.log(
`Cache url for ${cacheUrl} is not present`
);
return unauthorised(`${pathName}: This user session has expired! cache url is ${cacheUrl}`);
}
else
{
let response = new Response(tokenId);
response.headers.append('Cache-Control', 's-maxage=86400'); // in seconds
await cache.put(cacheKey, response.clone());
}
Did I miss anything?
What may be happening is a cache miss rather than an expiration. When using Cloudflare for caching, you shouldn't expect caching to be guaranteed, in particular using the cache API the docs mention it being data-center specific (not propagated globally) -
https://developers.cloudflare.com/workers/runtime-apis/cache/#background
Caching should be mechanism for improving performance, but not relied upon for guaranteed storage in other words. If your use-case requires this, it would be better to use the Workers Key Value -
https://developers.cloudflare.com/workers/learning/how-kv-works/
Related
I want to ask about increase performance when i do multiple future http request in single page. In case , i want to build a dashboard page. In dashboard, i've 4 endpoints url that return different result in every endpoint and should be shown in dashboard page.
here example code when load data
var client = new http.Client();
Future main() async {
var newProducts = await client.get("${endpoint}/product?type=newly&limit=5");
ProductListResponse newProductResponse = ProductListResponse.fromJson(json.decode(newProducts.body));
var bestSeller = await client.get("${endpoint}/product?type=best-seller&limit=5");
ProductListResponse bestSellerResponse = ProductListResponse.fromJson(json.decode(bestSeller.body));
var outOfStock = await client.get("${endpoint}/product?type=out-of-stock&limit=5");
ProductListResponse outOfStockResponse = ProductListResponse.fromJson(json.decode(outOfStock.body));
var lastRequest = await client.get("${endpoint}/product-request?type=newly&limit=5");
ProductRequestListResponse productRequestResponse = ProductRequestListResponse.fromJson(json.decode(lastRequest.body));
}
every endpoint when i hit manually using postman it takes 200ms for return the result. But when i implement in flutter app, it took almost 2s.
can i improve performance when getting data?
The reason why your code run so slow is that you are making those HTTP requests one by one. Each await will take quite some time.
You can either not use await and implement the logic using callbacks (.then) or you can combine the Futures into one using Future.wait and use await for that combined Future.
Your code will look something like this:
var responses = await Future.wait([
client.get("${endpoint}/product?type=newly&limit=5"),
client.get("${endpoint}/product?type=best-seller&limit=5"),
client.get("${endpoint}/product?type=out-of-stock&limit=5"),
client.get("${endpoint}/product-request?type=newly&limit=5")
]);
I wanted to add a feature to my site similar to this:
When a new session starts look at the utm_source/utm_medium querystring values and also the referrer. Based on that display a differnt phone number of the site, so for example google cpc, bing cpc, google organic, bing organic would have different numbers.
The number of calls to each number should then give an indication of which traffic source generated the calls.
The problem is, because we're using clouldflare, if a user is served a page from the cache then there is no session_start event on the origin server.
Is there a solution to get around this? Is there anyway to do this on cloudflare itself, perhaps using its "workers"?
Thanks
Cloudflare workers can be used to accomplish that. The worker script will first need to determine which phone number to show. This can be done by checking the query params or a cookie or any other aspect of the request. Then the worker script can take the original response body (from the cache or the origin server) and replace all occurences of the original phone number with the new phone number.
Here's an example worker script that does that. To determine which phone number to show, it will first check the query parameters, as you mentioned. When it sees the utm_source query param, it will also set a cookie that can then be checked in all subsequent requests to show the same phone number.
// The name of the cookie that will be used to determine which phone number to show
const cookieName = "phone_num_id";
// The list of all phone numbers to use
const phoneNumbers = [
{
id: "google-cpc",
utmSource: "google",
utmMedium: "cpc",
phoneNumber: "222-222-2222"
},
{
id: "bing-cpc",
utmSource: "bing",
utmMedium: "cpc",
phoneNumber: "333-333-3333"
}
];
// This adds a "fetch" event listener which will be called for all incoming requests
addEventListener("fetch", event => {
event.respondWith(handleRequest(event.request));
});
async function handleRequest(request) {
// Forward the incoming request and get the original response. If Cloudflare has already cached
// this request, this will return the cached response, otherwise it will make the request to
// the origin
let response = await fetch(request);
// Check the content type of the response and fallback to an empty string
// if there is no content-type header
let contentType = response.headers.get("content-type") || "";
// We're only interested in changing respones that have a content-type starting
// with "text/html". Anything else will be returned without any modifications
if (/^text\/html/.test(contentType)) {
// `newPhoneNumberData` will be the new phone number to show (if any)
let newPhoneNumberData;
// searchParams are the query parameters for this request
let searchParams = new URL(request.url).searchParams;
// If the request has a `utm_source` query param, use that to determine which phone number to show
if (searchParams.has("utm_source")) {
let utmSource = searchParams.get("utm_source") || "";
let utmMedium = searchParams.get("utm_medium") || "";
// Lookup the phone number based on the `utmSource` and `utmMedium`
newPhoneNumberData = phoneNumbers.find(
phoneNumber =>
phoneNumber.utmSource === utmSource &&
phoneNumber.utmMedium === utmMedium
);
// If we found a match, set a cookie so that subsequent requests get the same phone number
if (newPhoneNumberData) {
// In order to modify the response headers, we first have to duplicate the response
// so that it becomes mutable
response = new Response(response.body, response);
// Now set a cookie with the id of the new phone number to use. You should modify the properties
// of the cookie for your use case. See this page for more information:
// https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie
response.headers.append(
"Set-Cookie",
`${cookieName}=${newPhoneNumberData.id}; Max-Age=2147483647`
);
}
}
// If we weren't able to determine the new phone number based on the query params, try
// checking the cookies next
if (!newPhoneNumberData) {
let cookieHeader = request.headers.get("cookie") || "";
// split each of the cookies and remove leading/trailing whitespace
let cookies = cookieHeader.split(";").map(str => str.trim());
// Find the phone number cookie
let phoneNumberCookieString = cookies.find(cookieString =>
cookieString.startsWith(`${cookieName}=`)
);
// If the request has the phone number cookie, use that
if (phoneNumberCookieString) {
// Extract the phone number id from the cookie
const phoneNumberId = phoneNumberCookieString.split("=")[1];
// Lookup the phone number data based on the ID
newPhoneNumberData = phoneNumbers.find(
phoneNumber => phoneNumber.id === phoneNumberId
);
}
}
// If we found a matching phone number to use, now we'll need to replace all occurences
// of the original phone number with the new one before returning the response
if (newPhoneNumberData) {
// Get the original response body
let responseBody = await response.text();
// Use a regex with the `g` flag to find/replace all occurences of the original phone number.
// This would look for a phone number like "(111)-111-1111" but you can modify this to fit
// however your original phone number appears
responseBody = responseBody.replace(
/\(?111\)?[-\s]*111[-\s]*1111/g,
newPhoneNumberData.phoneNumber
);
// Create a new response with the updated responseBody. We also pass the original `response` as the
// second argument in order to copy all other properties from the original response (status, headers, etc)
response = new Response(responseBody, response);
}
}
return response;
}
I'm trying to get the sessionID from express-session when a new WebSocket connection comes in from a user. I'm able to find the sessionID, I just have a question about its format.
When I make a HTTP request to my messenger page say I get 'X' as a sessionID, if I then made a WebSocket connection I can find the session ID 'AXB', the session ID X is in there, but also surrounded with other information.
var express = require('express');
var app = express();
var server = require('http').createServer(app);
var session = require('express-session');
var io = require('socket.io')(server);
var store = new MemoryStore({
checkPeriod: 86400000
});
app.use(session({
store: store,
secret: 'jpcs-0001080900DRXPXL',
saveUninitialized: false,
resave: true
}));
// ...
app.get('/messenger/:uid', authorizationRedirect, (req, res) => {
console.log(req.sessionID);
// prints "EIVUudPTckmojrkv6FN9Cdb5NAQq5oQU"
// ...
});
io.set('authorization', (data, accept) => {
if (data && data.headers && data.headers.cookie) {
console.log(data.headers.cookie);
cookies_str = data.headers.cookie;
cookies_arr = cookies_str.split(';');
cookies = {};
for (index in cookies_arr) {
cookie = cookies_arr[index].split('=');
key = cookie[0].replace(/ /g,'');
val = cookie[1];
cookies[key] = val;
}
sessionId = cookies['connect.sid'].split('.')[0];
console.log(sessionId);
// prints "s%3AEIVUudPTckmojrkv6FN9Cdb5NAQq5oQU.AQkvP..."
// ...
});
So basically, in io.set('authorization', ...) I get:
s%3AEIVUudPTckmojrkv6FN9Cdb5NAQq5oQU.AQkvPsfoxieH3EAs8laFWN28dr1C%2B9zIT%2BMXtKTRPBg
But in app.get('/...', ...) I get:
EIVUudPTckmojrkv6FN9Cdb5NAQq5oQU
You can notice that the string from socket.io does contain the session id in this format: "s%3A" + sessionID + ".xxxxxxxxxxx..."
So obviously I can get the sessionID from here, but I'm curious why the sessionID is shown like this when I get socket connections? Will it ALWAYS be shown like this regardless of browser, WebSocket implementations, etc? What does the other information contained mean? I mostly want to make sure that this is a reliable way to get the sessionID. Thanks!
I would first like to clarify that io.set('authorization',...) has been deprecated. Here's the updated version Documentation
So obviously I can get the sessionID from here, but I'm curious why the sessionID is shown like this when I get socket connections? Will it ALWAYS be shown like this regardless of browser, WebSocket implementations, etc?
It's not reserved for socket connections at all. That is simply how it is fixed on the browser. So yes, it will always be shown like that.
What does the other information contained mean? I mostly want to make sure that this is a reliable way to get the sessionID. (s%3AEIVUudPTckmojrkv6FN9Cdb5NAQq5oQU.AQkvPsfoxieH3EAs8laFWN28dr1C%2B9zIT%2BMXtKTRPBg)
The first three characters are just encoded, and I believe every sessionID containts that. DecodedURIComponent("s%3A") = "s:"
After that is the sessionID itself (EIVUudPTckmojrkv6FN9Cdb5NAQq5oQU)
Now, after the dot(AQkvPsfoxieH3EAs8laFWN28dr1C%2B9zIT%2BMXtKTRPBg) is the signature portion. That verifies the authenticity of the cookie and is actually given when you sign the cookie. AND yes, I would say it is a trusted and reliable way.
Is there a way to determine the timestamp of the last app launch or log the timestamp the last API request from a user on parse server, on the server side without adding code to the client?
To do this without modifying the client, you could think of a request your clients always perform, like a find query and then set a hook on your server like this one from the docs.
Parse.Cloud.beforeFind('MyObject', function(req) {
let query = req.query; // the Parse.Query
let user = req.user; // the user
let triggerName = req.triggerName; // beforeFind
let isMaster = req.master; // if the query is run with masterKey
let isCount = req.count; // if the query is a count operation (available on parse-server 2.4.0 or up)
let logger = req.log; // the logger
let installationId = req.installationId; // The installationId
});
Then just persist the data you want somewhere like the user model and save it.
I am making a simple API request to Github to get all the repositories. The problem is that Github has a limitation and the max that it can send is 100 per request. There are users that have more than 100 repositories and I don't know how to access it or how to make pagination.
I am making GET request with Axios like this:
https://api.github.com/users/<AccountName>/repos?per_page=100
I can also put page number like so
https://api.github.com/users/<AccountName>/repos?page=3&per_page=100
But how do I make this work in app without making 10 API requests? I wouldn't even know how many requests I should make because I don't know what is the number that gets returned, does somebody have 100 or 1000 repos? I would like for everything to be returned and saved in array, for example.
EDIT:
Example: I am passing in accountName
var config = {
headers: {'Authorization': `token ${ACCESS_TOKEN}`}
}
const REQUEST: string = 'https://api.github.com/users/'
const apiCall = {
getData: async function (accountName) {
const encodedAccountName = encodeURIComponent(accountName)
const requestUrl = `${REQUEST}${encodedAccountName}`
const user = await axios.get(requestUrl, config)
// This return user and inside of user there is a link for fetching repos
const repo = await axios.get(`${user.data.repos_url}?per_page=100`, config)
...
You can get the repo count by requesting from the user account URL first. For example here is mine:
https://api.github.com/users/erikh2000
The response there includes a "public_repos" value. Bam! That's the magic number you want.
You next need to make multiple fetches if the repo count is over 100. I know you didn't want to, but hey... can't blame web services for trying to conserve their bandwidth. The good news is you can probably put them in a Promise.all() block and have them all fetch together and return at once. So code like...
const fetchAllTheRepos = (userName, repoCount) => {
const MAX_PER_PAGE = 100;
const baseUrl = 'https://api.github.com/users/' + userName +
'/repos?per_page=' + MAX_PER_PAGE;
//Start fetching every page of repos.
const fetchPromises = [], pageCount = Math.ceil(repoCount /
MAX_PER_PAGE);
for (let pageI = 1; pageI <= pageCount; ++pageI) {
const fetchPagePromise = fetch(baseUrl + '&page=' + pageI);
fetchPromises.push(fetchPagePromise);
}
//This promise resolves after all the fetching is done.
return Promise.all(fetchPromises)
.then((responses) => {
//Parse all the responses to JSON.
return Promise.all( responses.map((response) => response.json()) );
}).then((results) => {
//Copy the results into one big array that has all the friggin repos.
let repos = [];
results.forEach((result) => {
repos = repos.concat(result);
});
return repos;
});
};
//I left out the code to get the repo count, but that's pretty easy.
fetchAllTheRepos('erikh2000', 7).then((repos) => {
console.log(repos.length);
});
Simultaneously fetching all the pages may end up being more than Github wants to let you do at once for those accounts with lots of repos. I would put some "good citizen" limit on the number of repos you'll try to get at once, e.g. 1000. And then see if api.github.com agrees with your definition of a good citizen by watching for HTTP error responses. You can get into throttling solutions if needed, but probably a "grab it all at once" approach like above works fine.
On the other hand, if you are spidering through multiple accounts in one session, then maybe design the throttling in from the beginning just to you know... be nice. For that, look at a queue/worker pattern.