Why do I see duplicate object ids when using git_odb_foreach? - libgit2

Question

I'm running a simple object id gatherer on a large git repo (in this case linux-2.6) in preparation for storing said ids in a sqlite database.
Pseudo-code:
// Table holding the SHA1 of each object in the database, ensure ids are unique
CREATE TABLE objs(key INTEGER PRIMARY KEY, id BLOB UNIQUE);
// For each id, insert into objs table, rc can tell us if we violate uniqueness constraint
int callback(const git_oid *oid, void *payload) {
// note that 'oid' in the following string is really the id value in real code
int rc = sqlite3_exec("INSERT INTO objs(id) VALUES(oid);");
if (rc == SQLITE_CONSTRAINT) {
// code to print type and oid
}
}
int main() {
// sqlite and git initialization
git_odb_foreach(...callback...);
// cleanup
return 0;
}
Out of ~4 million objects, there are ~70000 non-unique objects I end up encountering along the way. Interestingly, when running 'git rev-list --objects --all | wc -l', this count matches the number of unique objects from the foreach code.
Can someone explain why the git_odb_foreach function would produce these non-unique ids?

Answer 1

Git objects might be stored in more than one packfile, in addition to existing as a loose object. That's simply something that might happen and that git implementations have to deal with.
While git/libgit2/whatever will generally check whether an object exists before creating it, there is no way to make this determination when you are speaking to a remote.
If a remote has some of the same objects in history you are downloading but there is no way to detect this via the negotiation (which only exchanges a few commit ids) then the remote might send you objects you already have. These objects come in a packfile and there is no logic to get rid of duplicate ones outside of garbage collection, which can take quite some time.
Having the same object in different packfiles could even increase performance as you would be able to use objects which are closer together and share delta chains instead of going to open a different packfile.
All git_odb_foreach does is go through each backend and call the callback with whatever it finds. It doesn't try to deduplicate as it can't know that's what you want to do. So if an objects exists both as loose and packed, or in multiple packfiles, then that's what it will return.
Note, however, that your command git rev-list --objects --all | wc -l is doing something rather different from the git_odb_foreach call. The command is asking for all objects reachable from the references whereas the function call is getting all existing objects and in many cases (possibly most) those numbers won't match. E.g. if you ever do git add, there will be a blob in the object db that won't be reachable from any reference.