How to avoid to fetch a list of followers of the same Twitter user that was displayed before - api

Question

I'm very new at coding and I'm having some issues. I'd like to display the followers of followers of ..... of followers of some specific users in Twitter. I have coded this and I can set a limit for the depth. But, while running the code with a small sample, I saw that I run into the same users again and my code re-display the followers of these users. How can I avoid this and skip to the next user? You can find my code below:
By the way, while running my code, I encounter with a 401 error. In the list I'm working on, there's a private user, and when my code catches that user, it stops. Additionally, how can I deal with this issue? I'd like to skip such users and prevent my code to stop.
Thank you for your help in advance!
PS: I know that I'll encounter with a 429 error working with a large sample. After fixing these issues, I'm planning to review relevant discussions to deal with.
public class mainJava {
public static Twitter twitter = buildConfiguration.getTwitter();
public static void main(String[] args) throws Exception {
ArrayList<String> rootUserIDs = new ArrayList<String>();
Scanner s = new Scanner(new File("C:\\Users\\ecemb\\Desktop\\rootusers1.txt"));
while (s.hasNextLine()) {
rootUserIDs.add(s.nextLine());
}
s.close();
for (String rootUserID : rootUserIDs) {
User rootUser = twitter.showUser(rootUserID);
List<User> userList = getFollowers(rootUser, 0);
}
}
public static List<User> getFollowers(User parent, int depth) throws Exception {
List<User> userList = new ArrayList<User>();
if (depth == 2) {
return userList;
}
IDs followerIDs = twitter.getFollowersIDs(parent.getScreenName(), -1);
long[] ids = followerIDs.getIDs();
for (long id : ids) {
twitter4j.User child = twitter.showUser(id);
userList.add(child);
getFollowers(child, depth + 1);
System.out.println(depth + "th user: " + parent.getScreenName() + " Follower: " + child.getScreenName());
}
return userList;
}
}

Answer 1

I guess graph search algorithms can be implemented for this particular issue. I chose Breadth First Search algorithm because visiting root user's followers at first would be better. You can check this link to additional information about algorithm.
Here is my implementation for your problem:
public List<User> getFollowers(User parent, int startDepth, int finalDepth) {
List<User> userList = new ArrayList<User>();
Queue<Long> queue = new LinkedList<Long>();
HashMap<Long, Integer> discoveredUserId = new HashMap<Long, Integer>();
try {
queue.add(parent.getId());
discoveredUserId.put(parent.getId(), 0);
while (!queue.isEmpty()) {
long userId = queue.remove();
int discoveredDepth = discoveredUserId.get(userId);
if (discoveredDepth == finalDepth) {
continue;
}
User user = twitter.showUser(userId);
handleRateLimit(user.getRateLimitStatus());
if (user.isProtected()) {
System.out.println(user.getScreenName() + "'s account is protected. Can't access followers.");
continue;
}
IDs followerIDs = null;
followerIDs = twitter.getFollowersIDs(user.getScreenName(), -1);
handleRateLimit(followerIDs.getRateLimitStatus());
long[] ids = followerIDs.getIDs();
for (int i = 0; i < ids.length; i++) {
if (!discoveredUserId.containsKey(ids[i])) {
discoveredUserId.put(ids[i], discoveredDepth + 1);
User child = twitter.showUser(ids[i]);
handleRateLimit(child.getRateLimitStatus());
userList.add(child);
if (discoveredDepth >= startDepth && discoveredDepth < finalDepth) {
System.out.println(discoveredDepth + ". user: " + user.getScreenName() + " has " + user.getFollowersCount() + " follower(s) " + (i + 1) + ". Follower: " + child.getScreenName());
}
queue.add(ids[i]);
} else {//prints to console but does not check followers. Just for data consistency
User child = twitter.showUser(ids[i]);
handleRateLimit(child.getRateLimitStatus());
if (discoveredDepth >= startDepth && discoveredDepth < finalDepth) {
System.out.println(discoveredDepth + ". user: " + user.getScreenName() + " has " + user.getFollowersCount() + " follower(s) " + (i + 1) + ". Follower: " + child.getScreenName());
}
}
}
}
} catch (TwitterException e) {
e.printStackTrace();
}
return userList;
}
//There definitely are more methods for handling rate limits but this worked for me well
private void handleRateLimit(RateLimitStatus rateLimitStatus) {
//throws NPE here sometimes so I guess it is because rateLimitStatus can be null and add this conditional expression
if (rateLimitStatus != null) {
int remaining = rateLimitStatus.getRemaining();
int resetTime = rateLimitStatus.getSecondsUntilReset();
int sleep = 0;
if (remaining == 0) {
sleep = resetTime + 1; //adding 1 more second
} else {
sleep = (resetTime / remaining) + 1; //adding 1 more second
}
try {
Thread.sleep(sleep * 1000 > 0 ? sleep * 1000 : 0);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
in this code HashMap<Long, Integer> discoveredUserId is used to prevent program checking same users repeatedly and storing in which depth we faced with this user.
and for private users, there is isProtected() method in twitter4j library.
Hope this implementation helps.