It's hard to formulate the question actually so I just explain the situation.
I'm working on a application that consists of multiple sub applications. The main app just provides an navigation bar and some basic functionality like configuration of users and permissions while the sub applications provide the actual functionalities.
Now this is a Rails 2 application and the sub applications get embedded in frames, it's not really nice design and pretty complex to setup.
Fortunately we have Engines now and that would be the saner solution for this application.
Until now everything lives in subversion and can be updated at once, shared code uses externals. We would like to move to git while we're at restructuring and refactoring.
I've been searching the web the past few days about bundler, git submodules and git subtrees but I haven't found a good description how to properly manage a large project which consists of multiple Engines/Gems when you are developing on all of them the same time.
In particular I would like to be able to:
use Bundler to manage dependencies
do not install our own Gems and Engines into the global gem path but relative to the main app, as an git repository
have our own Gems and Engines setup as git repository (maybe with Bundler's local path override)
an easy way to fetch all dependencies (bundle install) which pulls the latest version of our own Gems and Engines, if that's not possible then one command to git pull all own Gems and Engines (maybe an rake task?)
make it easy for new developers to setup the entire development enviroment fast (git clone the app, bundle install dependencies including all own Gems and Engines, locally)
deploy with Capistrano, easily
What I already thought about:
including everything into one repository, seems to defeat the purpose of separate Gems/Engines for me, also I think it wouldn't allow us to manage the dependencies of the main app on our Engines via Bundler
using submodules, I read too many posts about why it's bad, and with our number of developers it's only a matter of time until somebody commits a submodule pointer to a commit that only exists in his local repo
git subtree utility, seems quite complex to me
So has anybody of you a similar setup and how do you manage it to make updating and committing changes as easy as possible? Where do you put your Engine/Gem code on which the application depends?
TL;DR How do manage a large rails project which consists of multiple Engines and Gems?
We have a similar (but probably less complex) case at my company. What we do (as for now) and that could work for you too :
Put your Rails app in its own git repository. The various gems each get their own repository also (while it is possible to do otherwise, the "one gem = one git repository" will make your life easier).
Then in your Rails app Gemfile, you have several options
Default should be to refer each gem to its git repository (so that bundle will load them from there)
When working locally on some of the gems and the Rails app, either change the Gemfile to use the local path (http://gembundler.com/v1.2/gemfile.html) or better, by overriding the path locally (see: http://gembundler.com/v1.2/git.html). Be careful that those two options are different : the first one use the path, the second the local git repository (so a new uncommitted change will be visible by the first, not by the second).
For updating all your gems easily, I would create a small .sh script (just to launch the various clone or update operations, and the bundle install so that everything comes out clean), and commit it with the main app. I would also get a "standard folder organization" among the team (ie, that everyone use a base folder of their choice, with under it folders for the Rails app and each gem), to make the procedure easier.
I hope this can help or get you ideas (your question is quite complex and manifold, so I'm not 100 % sure this is what you are looking for).
How to manage your Gem dependencies?
Bundler via Gemfile.
How to manage your Engines?
Bundler via Gemfile.
Architect your Engines as Gems and provide their git repo location in your Gemfile. If you want examples, check out how to include the https://github.com/radar/forem gem in your Gemfile.
Also, this helped me learn Rails Engines, http://edgeguides.rubyonrails.org/engines.html.
Are you coming from Java Land?
Rails does have a learning curve, but not like the Java mountain cliff drop off.
Related
I'm using git submodule to build and shared components between projects. The project is not in production yet, so, at this point submodule is serving well.
But I'm concerned about maintenance and deploy, would be a good idea transform it into a npm package ?
An npm package will allow fragmentation across different package versions. On the other hand, git submodules have a bit of a learning curve, and the tooling is really not that good. With git submodules, you have all the source in one folder.
If it's at all possible, I'd recommend using a plain monorepo for all projects. You may need to create build time variables (via babel plugin/s), you may need some sort of "live config" get served from the backend. I worked with git submodules for a year and I've recently worked with a project that uses npm to share code.
I would recommend using only one git submodule, for all shared code, instead of several submodules. I would strongly consider using lerna, and use your one git submodule to track lerna's packages directory. And if the team decides they don't like git submodules, you can easily make this repo a sibling git repo, instead of a submodule. However, above all this, I'd recommend using a plain monorepo.
Here's a great talk on monorepo's from Netflix: https://www.youtube.com/watch?v=VNqmHJtItCs (strong focus on discouraging npm-style packages)
Here's google's infamous monorepo talk: https://www.youtube.com/watch?v=W71BTkUbdqE
This is a great site to read to help you think about good development flows: https://trunkbaseddevelopment.com/ (it primarily advocates for the monorepo approach)
If you are developing software for different clients(different people/companies paying you for similar projects), and have some agreement that they should be at least ~80% the same, you may really enjoy using build flags to help get started on splitting functionality, but I'm sure you should very proactively keep the code around the build flags clean, and refactor into re-usable components/packages. Give each client some sort of build-flags.json. Build flags should be named for features only, which in theory can all be individually toggled. Some code may be totally custom for each project, in this case, you may want to consider using dynamic imports, but generally this is a pain point I have yet to fully cross, although I have plenty of unrefined ideas around this.
If a monorepo is just not happening, I would actually recommend using npm packages+separate repos over git submodules, assuming you can do good semantic versioning of the package. (And, yalc seems to be a good tool for linking together packages, as opposed to the standard npm/yarn link)
My findings after trying both lerna, npm workspaces and git submodules. I find it is not a case of the one vs the other.
The reason why I say this is because one can have submodules that are part of the monorepo. Doing exactly this made my development experience better as I could clone an existing project and actively develop it within the bigger project (monorepo). I could then contribute back to the cloned project once satisfied with the changes. This is something that you cannot do with npm workspaces alone. Hence my argument that it is not a case of one vs the other. They solve different problems and can therefore complement each other.
Before using npm workspaces I would use npm link all the time. npm workspaces makes this use-case of developing with multiple packages more convenient. Even when the team you work with does not use a monorepo; you could use one to develop multiple packages and test them in conjunction. Once satisfied, you can push the individual repos. This is something you cannot do with git alone.
Maybe you can think of more novel ways of combining the features of npm and git.
when I was created a new rails app with rails new app_name. I found that many gems were installed and I suspect I will not use many of them. It is possible to have a minimal set of gems when I do a rails new?
Thank you.
Additional information:
I'm concerned about this because when I push a fresh new rails app onto heroku, I realized many gems are installed, and I'm not using most of them.
What ending up within your application in controlled through the "Gemfile" in your root directory. Running bundle / bundle install will install the gems and their dependencies.
But having gems available in the system doesn't mean your application will rely on them.
Further reading: http://gembundler.com/gemfile.html
I feel that using Git submodules is somehow troublesome for my development workflow. I've heard about Git subtree and Gitslave.
Are there more tools out there for multiple repository projects and how do they compare ?
Can these tools run on Windows ?
Which is best for you depends on your needs, desires, and workflow. They are in some senses semi-isomorphic, it is just some are a lot easier to use than others for specific tasks.
gitslave is useful when you control and develop on the subprojects at more of less the same time as the superproject, and furthermore when you typically want to tag, branch, push, pull, etc all repositories at the same time. gitslave has never been tested on windows that I know of. It requires perl.
git-submodule is better when you do not control the subprojects or more specifically wish to fix the subproject at a specific revision even as the subproject changes. git-submodule is a standard part of git and thus would work on windows.
git-subtree provides a front-end to git's built-in subtree merge strategy. It is better when you prefer to have a single-repository "unified" git history. Unlike the subtree merge strategy, it is easier to export changes to the different (directory) trees back out to the original project, but it is not as automatic as it is with gitslave or even git-submodule.
repo is in theory similar to gitslave, but not as well documented for non-android operations that I have found. It is fairly dedicated to the Google Android development model and only natively supports a handful of git commands (though you can run arbitrary commands) and the limited native support doesn't support, for example, a centralized repository to push to and checking out a branch seems fairly difficult.
kitenet's mr is what you would want to use if you have multiple version control systems in use, but is mostly limited for git-only superprojects due to its lowest common denominator approach. There are ways to run arbitrary commands, but they are not as well integrated.
For some use cases, I have liked each of the following two simple approaches:
Nested repositories. If your software project has a plugin mechanism, with each plugin in its own sub-directory, it can make sense to git-ignore these plugin directories and, in your local filesystem, to make each of them into its own git repository. This way, all your files form a single directory tree, but are managed in different git repositories. It will not confuse git.
Per-package repositories. For software projects where you use some kind of source code package management system (gem / bundler, npm, pear or the like) it can make sense to put your re-used code into separate git repositories, then to make source packages from them, and then to install them with the package management tool into the parent project. Your parent project's git repository would only contain a reference to the required packages and their versions, while the actual code of these packages will be git-ignored as done with all other packages and external libraries as well. Compared to the nested repositories proposed above, this is a more elaborate approach as it allows to specify which package version is to be installed.
I currently use submodules for development and not just relating 3rd party libraries. There are some ways that you can make life easier with submodules, especially when they are the source of merge or rebase conflicts. Look to ls-tree to get the 2 commits involved on a conflict in the submodule. This is probably the most difficult part of submodules for people to deal with. For now scripting will make this much easier to work with. Future versions of Git should have better native support for dealing with them.
Hope this helps.
We encountered a similar issue when using Git submodules in projects where we had dependencies in a variety of languages. To deal with them, we built and open-sourced a tool called MDLR ("Modular") that gives you declarative version-controlled Git dependencies with similar functionality to Git submodules, but without the annoying workflow. You can install it and manage your dependencies with the instructions/downloads on the GitHub repo
Our Rails application pulls feeds from multiple sources. The workers that pull these feeds need gem dependencies for rmagick, oracle databases, and many other gems. In short, they have very different dependency needs than the main web application. Until Rails 3 and Bundler, life was good.
These worker gem dependencies are irrelevant to our actual production website. Under Rails 3, one Gemfile is expected to contain all these dependencies. This has the nasty side effect of requiring all gem dependencies to be loaded within the production app, which would cause pointless bloat, possible security issues, memory leaks, complicate deployment, and other ills. Sadly, Bundler breaks the standard require mechanism, which would have provided a way out of the quagmire by allowing us to simply require the necessary gems only in the worker and have them somewhere on the system, not in the bundle. The workers use our rails models to file their data.
Can anyone suggest solutions to make the system practical in Rails 3? I am tempted to make the Gemfile use conditional environment variables in places to drive the gem commands, however, it seems the Gemfile.lock could make this problematic going from working on one worker script (for the feeds) to the next, which would have different dependencies. Help???
I've been contemplating a similar problem, and although I don't have a solution in use anywhere yet, your question did make me think it out some more. I think you should be able to use a group to accomplish this. You can add something like this to your Gemfile:
group :workers do
gem "extra_gem_1"
gem "extra_gem_2"
end
Then, you can call
Bundler.require(:default, :workers)
and that should load your gems. How this works will depend on your setup, you might be able to add logic to config/application.rb, or you might need to do this elsewhere. This might be hackish, but it works in the console anyway.
When installing your gems, you can call:
bundle install --without workers
to exclude those gems from production.
Alternatively, you can use two Gemfiles, but that seems like a mess as well since presumably there's some crossover.
I have made minor modifications to mislav's famous will_paginate gem. I would like to "pack" those changes along with the rest of the gems I have so I can deploy on other machines easily. What I'm looking for is not a github commit as I only need those changes on my own app for my own purposes.
When I use "bundle pack" it indeed packs all my gems, but the original gem files that were downloaded and not the modifications i've made to the raw files.
How does one go about packing the current gems and then deploying them somewhere else along with changes made to them?
thanks!
You say you are not looking for a github commit, but honestly I think that is the easiest way to do it. Fork the repo, make the changes, commit and push to github and then add it to your Gemfile and point it to your fork. Bundler can install the gem directly from github. I don't even think you will have to regenerate the gemspec, as long as you are not adding new files. In the Gemfile you do this:
gem 'will_paginate', :git=>'git://github.com/your_username_goes_here/will_paginate.git'
So in summary (in case you are not that familiar w/ github):
Login or sign up for github account.
Go to the will_paginate repo and click the "Fork" button
Copy the URL from your new fork
local_shell> git clone git://github.com/your_username_goes_here/will_paginate.git
Make your changes
local_shell> git commit -a
local_shell> git push
Add the git fork URL to your Gemfile
local_shell> bundle install
If you already have a github account, aside from making changes to the code, this whole process will probably take you about 2 minutes. Easy and you will be able to keep track of what you have changed on github. IMO its a bad idea to make changes locally and not commit them to a remote repo... things can get messy and confusing, especially if you are deploying these changes to prod.