API Request Pagination - api

Question

I am making a simple API request to Github to get all the repositories. The problem is that Github has a limitation and the max that it can send is 100 per request. There are users that have more than 100 repositories and I don't know how to access it or how to make pagination.
I am making GET request with Axios like this:
https://api.github.com/users/<AccountName>/repos?per_page=100
I can also put page number like so
https://api.github.com/users/<AccountName>/repos?page=3&per_page=100
But how do I make this work in app without making 10 API requests? I wouldn't even know how many requests I should make because I don't know what is the number that gets returned, does somebody have 100 or 1000 repos? I would like for everything to be returned and saved in array, for example.
EDIT:
Example: I am passing in accountName
var config = {
headers: {'Authorization': `token ${ACCESS_TOKEN}`}
}
const REQUEST: string = 'https://api.github.com/users/'
const apiCall = {
getData: async function (accountName) {
const encodedAccountName = encodeURIComponent(accountName)
const requestUrl = `${REQUEST}${encodedAccountName}`
const user = await axios.get(requestUrl, config)
// This return user and inside of user there is a link for fetching repos
const repo = await axios.get(`${user.data.repos_url}?per_page=100`, config)
...

Answer 1

You can get the repo count by requesting from the user account URL first. For example here is mine:
https://api.github.com/users/erikh2000
The response there includes a "public_repos" value. Bam! That's the magic number you want.
You next need to make multiple fetches if the repo count is over 100. I know you didn't want to, but hey... can't blame web services for trying to conserve their bandwidth. The good news is you can probably put them in a Promise.all() block and have them all fetch together and return at once. So code like...
const fetchAllTheRepos = (userName, repoCount) => {
const MAX_PER_PAGE = 100;
const baseUrl = 'https://api.github.com/users/' + userName +
'/repos?per_page=' + MAX_PER_PAGE;
//Start fetching every page of repos.
const fetchPromises = [], pageCount = Math.ceil(repoCount /
MAX_PER_PAGE);
for (let pageI = 1; pageI <= pageCount; ++pageI) {
const fetchPagePromise = fetch(baseUrl + '&page=' + pageI);
fetchPromises.push(fetchPagePromise);
}
//This promise resolves after all the fetching is done.
return Promise.all(fetchPromises)
.then((responses) => {
//Parse all the responses to JSON.
return Promise.all( responses.map((response) => response.json()) );
}).then((results) => {
//Copy the results into one big array that has all the friggin repos.
let repos = [];
results.forEach((result) => {
repos = repos.concat(result);
});
return repos;
});
};
//I left out the code to get the repo count, but that's pretty easy.
fetchAllTheRepos('erikh2000', 7).then((repos) => {
console.log(repos.length);
});
Simultaneously fetching all the pages may end up being more than Github wants to let you do at once for those accounts with lots of repos. I would put some "good citizen" limit on the number of repos you'll try to get at once, e.g. 1000. And then see if api.github.com agrees with your definition of a good citizen by watching for HTTP error responses. You can get into throttling solutions if needed, but probably a "grab it all at once" approach like above works fine.
On the other hand, if you are spidering through multiple accounts in one session, then maybe design the throttling in from the beginning just to you know... be nice. For that, look at a queue/worker pattern.