I'm trying to build a website that supports git version control. I'm using libgit2 for the backend. But I think filesystem is not so easy to scale and to guarantee data integrity. I noticed that libgit2 has custom backend (https://github.com/libgit2/libgit2-backends) support where I can use a database for some storage.
Initially I hoped that I can completely get rid of filesystem by saving everything git related into a database. But after I tried the sqlite backend, It seems that libgit2 still need to generate a .git folder on my filesystem? Can I remove the .git folder when I use a database as the backend?
There are a few "limitations" to having a working, "in-memory" repository using libgit2, even though there's support for "some" in-memory-ness. As you found out, there are API endpoints for both custom object databases (.git/objects/) and the refdb (.git/refs/), and the config subsystem can work in-memory. But that is not the case for all other things that can go in .git/, because there's no customization point at the repo-level — and IMHO this is not git we're talking about anymore when you're completely fs-less — work isn't really going in this direction.
Feature-request: https://github.com/libgit2/libgit2/issues/4671
Discussion on a fix + some branches with "preparatory design work": https://github.com/libgit2/libgit2/pull/4967
Related
A merge from a feature branch to trunk took over 45 minutes to complete.
The merge included a whole lot of jars (~250MB), however, when I did it on the server with the file:// protocol the process took less than 30 seconds.
SVN is being served up by Apache over https.
The version of SVN on the server is
svn, version 1.6.12 (r955767)
compiled Sep 3 2013, 17:49:49
My local version is
svn, version 1.7.7 (r1393599)
compiled Oct 8 2012, 20:42:17
On checking the Apache logs I made over 10k requests and apparently each of these requests went through an authentication layer.
Is there a way to configure the server so that it caches the credentials for a period and doesn't make so many authentication requests?
I guess the tricky part is making sure the credentials are only cached for the life of single svn 'request'. If svn merge makes lots of unique individual https requests, how would you determine how long to store the credential for without adding potential security holes?
First of all I'd strongly suggest you upgrade the server to a 1.7 or 1.8 versions since 1.7 and newer servers support an updated version of the protocol that requires fewer requests for many actions.
Second, if you're using path based authorization you probably want SVNPathAuthz short_circuit in your configuration. Without this for secondary paths (i.e. paths not in the request URI) as may happen for many recursive requests (especially log) when the authorization for those paths are run it runs back through the entire Apache httpd authentication infrastructure. With the setting instead of running the entire authentication/authorization infrastructure for httpd, we simply ask mod_authz_svn to authorize the action against the path. Running through the entire httpd infrastructure can be especially painful if you're using LDAP and it needs to go back to the LDAP server to check credentials. The only reason not to use the short_circuit setting is if you have some other authentication module that depends on the path, I've yet to see an actual setup like this in the wild though.
Finally, if you are using LDAP then I suggest you configure the caching of credentials since this can greatly speed up authentication. Apache httpd provides the mod_ldap module for this and suggest you read the documentation for it.
If you provide more details of the server side setup I might be able to give more tailored suggestions.
The comments suggesting that you not put jars in the repository are valuable, but with some configuration improvements you can help resolve some of your slowness anyway.
The merge included a whole lot of jars (~250MB)
That's your problem! If you go through your network via http://, you have to send those jars via http://, and that can be painfully slow. You can increase the cache size of Apache httpd, or you can setup a parallel svn:// server, but you're still sending 1/4 gigabyte of jars through the network. It's why file:// was so much faster.
You should not be storing jars in your Subversion repository. Here's why:
Version control gives you a lot of power:
It helps you merge differences between branches
It helps you follow the changes taking place.
It helps identify a particular change and why a particular change took place.
Storing binary files like jars provide you none of that. You can't merge binary files, and you can't track their changes.
Not only that, but version control systems usually use diffs to track changes. This saves a lot of space. Imagine a 1 kilobyte text file. In 5 revisions, six lines are changed. Instead of taking up 6K of space, only 1K plus those six changes are stored.
When you store a jar, and then a new version of that jar, you can't easily do a diff, and since jar format is zip, you can't really compress them either, store five versions of a jar in Subversion, and you store pretty close to five times the size of that jar. If a jar file is 10K, you're storing 50K of space for that jar.
So, not only are jar files taking up a lot of space, and they don't give you any power in storage, they can quickly take over your repository. I've seen sites where over 90% of a 8 gigabyte repository is nothing but compiled code and third party jars. And, the useful life of these binary files is really quite limited too. So, in these places, 80% of their Subversion repository is wasted space.
Even worse, you tend to lose where you got that jar, and what is in it. When users put in a jar called commons-beans.jar, I don't know what version that jar is, whether that jar was built by someone, and whether it was somehow munged by that person. I've see users merge two separate jars into a single jar for ease of use. If someone calls that jar commmons-beanutils-1.5.jar because it was version 1.5, it's very likely that someone will update it to version 1.7, but not change the name. (It would affect the build, you have to add and delete, there is always some reason).
So, there's a massive amount of wasted space with little benefit and almost no information. Storing jars is just plain bad news.
But your build needs jars! What should you do?
Get a jar repository like Nexus or Artifactory. Both of these repository managers are free and open source.
Once you store your jars in there, you can fetch the revision of the jar you want either through Maven, Gradel, or if you use Ant and want to keep your Ant build system, Ivy. You can also, if you don't feel like being that fancy, fetch the jars via an Ant <get/> task. If you use Jenkins, Jenkins can easily deploy the built jars for other projects to use in your Maven repository.
So, get rid of the jars. Merging will then be a simple diff between text files. Merging branches will be much quicker, and less information has to be sent over the network. If you don't want to switch to Maven, then use Ivy, or simply update your builds with the <wget> task to fetch the jars and the versions you need.
I want information about all commits done on my BitBucket repository.
I dont need an XML Feed. I want just a list of commits and the order in which they are committed and possibly information about merging.
On bitbucket websites information is shown in paginated form and is not sufficient for me. It would be nice if could obtain a pdf or something listing all my commits.
Is there a way to do so ?
Any help would be appreciated.
Thanks,
Pankaj.
You should make a local clone and then access the local repository using whatever tools you have that can generate such a PDF. That way you'll have full access to the information you need about commits and merges. Since Mercurial is a distributed version control system, this is really the way you're supposed to access it: locally, not via screen-scraping on Bitbucket.
If you're using Java or Python, then you can use JavaHg or python-hglib to access the repository. See the full list of libraries.
I'm using subversion to develop a website with a designer and I'm using a remote server to the Subversion database storage.
Now I need to make visible on the Apache(same machine that Subversion server) all the changes/commits that we make to the repository.
This is possible to do?
I have searched inside the repository files but nothing looks like the name of the files of the project(PHP Project).
Can someone give me a clue on how can I make visible the changes to the repository in a website? Is there a way to connect the changes that I do the the repository to the website?
Best Regards,
Assuming you can already browse the repository using HTTP, simply add something like Repos Style for a nice layout and folder/file history features.
You could shell out to svnadmin.
I want to set up a production and development environment for a wordpress website. The Wordpress site uses a lot of plugins. These plugins consist of php files which I upload to a directory within the main wordpress directory.
When activating and modifying the plugins, much of the configurations are stored in the mysql database. Developing on wordpress means much of my "code" and "changes" are stored in the database, as opposed to physical files that can be committed to and updated from a CVS repository. So instead of having a CVS keep track of my changes, I have to frequently and manually do mysqldumps of the database on production and import it into the development server.
Is there a better way to set up a development and production environment where much of the development changes are happening in the database?
Use CVS's hooks to perform the dump whenever you commit. I'm not familiar with how CVS does them, since I've only done it with SVN. However, from reading a bit of the manual, it seems pretty similar.
This section of the CVS manual explains what happens, script-wise, when you commit. Take a look at the section further down, "C.4.4.2 Keeping a checked out copy". You should be able to modify this example to put in your mysqldump command.
The nexus book: http://www.sonatype.com/books/nexus-book/reference/. Does not seem to spend any time on how one should go about backing up a nexus repository. If I am installing my snapshot and releases into this local repository, it seems that it would behoove me to back it up. However, I'm not really interested in backing up anything that can easily be downloaded from a remote repository.
Some google searches do not seem to reveal the canonical answer either, so perhaps for posterity it can be recorded here.
Thanks,
Nathan
When you install Nexus, you'll end up with two directories:
nexus-webapp-1.3.1.1/
sonatype-work/
We've separated the application from the data and configuration. The Nexus application is in nexus-webapp-1.3.1.1/ and the data and configuration is in sonatype-work/nexus. This was mainly done to facilitate easier upgrades, but it also has the side-effect of making it very easy to backup a Nexus installation.
The Simple Answer
Nexus doesn't store repositories in a database or do anything that would preclude a simple backup of the file system under sonatype-work/nexus. If you need to create a complete backup, just archive the contents of the sonatype-work/nexus.
Better Answer
If you want a more intelligent approach to backing up a Nexus installation, you will certainly want to backup everything under sonatype-work/nexus/conf, sonatype-work/nexus/storage, sonatype-work/nexus/template-store. If you want to backup the metadata and file attributes that Nexus keeps for proxy repository, backup sonatype-work/nexus/proxy, although this isn't required as the information about the proxy repository will be generated on-demand as attributes are requested.
You don't need to backup sonatype-work/nexus/logs and you don't need to backup the Lucene indexes in sonatype-work/nexus/indexer.
Nexus Pro Answer
There is a Nexus Professional plugin which can automate the process of creating a backup of the Nexus configuration data. This plugin is going to address the contents of the sonatype-work/nexus/conf directory. If you need to backup the sonatype-work/nexus/storage directory, you will need to configure some backup system to backup the contents of that filesystem. Once again, as with Nexus Open Source, there is currently no real benefit in backing up the contents of sonatype-work/nexus/indexer or sonatype-work/nexus/logs.
Excluding Storage for Remote Repositories
In your question you mention that you want to exclude the storage devoted to the local cache of a remote repository. If you are interested in doing this, you'll have to take a further level of granularity and just exclude the directories under sonatype-work/nexus/storage that correspond to the remote repositories.
Do you need to shut Nexus down for a backup?
Brian Fox told me no, the only real chance for file contention is going to be the files in the indexer/ directory. You shouldn't have a problem backing up the sonatype-work filesystem with a running instance of Nexus.
BTW, thanks for the question, this answer will likely be incorporated into the next version of the Nexus book.
afaik nexus (free version) does not have any backup features, but it should be as simple, as knowing your companies groupId and grabbing it from the storage directories in nexus
but i would schedule a complete repository backup too, you never know when the remote repositories are down, when you need them the most