I tend to build Yocto builds for various boards and versions(sumo, warrior, etc.,). To speedup the builds I use separate DL_DIR & SSTATE_DIR. The structure I followed is given below:
Single download dir for all the builds & boards (as most of the builds for different boards use same tool-chain)
Separate sstate cache dir for each board
My question is, Can I use single SSTATE_DIR to improve the performance and reducing the folder size? Have anyone tried it?
Suggestions and personal experiences are also appreciated.
You can share DL_DIR among all build directories and Yocto versions. Those are just tarballs of sources.
You can share SSTATE_DIR among all build directories, though it is beneficial only for builds sharing the same Yocto version, otherwise no size or speed improvement at all. This makes sense because many native packages and "base" target packages of machines of the same architecture (in your case aarch64) are identical for said machines and will make use of the shared SSTATE_DIR. For non-matching cases, new entries will be added to your SSTATE_DIR. When you have different Yocto versions, some package recipes have been updated and most often glibc, binutils and gcc are updated. Those are the packages used by almost all other package recipes, which effectively makes it impossible to re-use sstate-cache from earlier (or later) Yocto versions for package recipes that haven't changed since the package recipes they depend on are part of the sstate-cache for said recipes. Also, the Yocto version is in the filename of a sstate-cache entry so anyway they'll be rebuilt.
Related
Please allow me two questions to the use in Conan.io in our environment:
We are developing automotive embedded software. Usually, this includes integration of COTS libraries, most of all for communication and OS like AUTOSAR. These are provided in source code. Typical uC are Renesas RH850, RL78, or similar devices from NXP, Cypress, Infinion, and so on. We use gnumake (MinGW), Jenkins for CI, and have our own EclipseCDT distribution as standardized IDE.
My first question:
Those 3rd party components are usually full of conditional compilation to do a proper compile-time configuration. With this approach, the code and so the resulting binaries are optimized, both in size and in run-time behavior.
Besides those components, we of course have internal reusable components for different purposes. The Compile-time configuration here is not as heavy as in the above example, but still present.
In one sentence: we have a lot of compile-time configuration - what could be a good approach to set up a JFrog / Conan based environment? Stay with the sources in every project?
XRef with Conan:
Is there a way to maintain cross-reference information coming from Conan? I am looking for something like "Project xxx is using Library lll Version vvv". In that way, we would be able to automatically identify other "users" of a library in case a problem is detected.
Thanks a lot,
Stefan
Conan recipes are based on python and thus are very flexible, being able to implement any conditional logic that you might need.
As an example, the libxslt recipe in ConanCenter contains something like:
def build(self):
self._patch_sources()
if self._is_msvc:
self._build_windows()
else:
self._build_with_configure()
And following this example, the autotools build contains code like:
def _build_with_configure(self):
env_build = AutoToolsBuildEnvironment(self, win_bash=tools.os_info.is_windows)
full_install_subfolder = tools.unix_path(self.package_folder)
# fix rpath
if self.settings.os == "Macos":
tools.replace_in_file(os.path.join(self._full_source_subfolder, "configure"), r"-install_name \$rpath/", "-install_name ")
configure_args = ['--with-python=no', '--prefix=%s' % full_install_subfolder]
if self.options.shared:
configure_args.extend(['--enable-shared', '--disable-static'])
else:
configure_args.extend(['--enable-static', '--disable-shared'])
So Conan is able to implement any compile time configuration. That doesn't mean that you need to build always from sources. The parametrization of the build is basically:
Settings: for "project wide" configuration, like the OS or the architecture. Settings values typically have the same value for all dependencies
Options: for package specific configuration, like a library being static or shared. Every package can have its own value, different to other packages.
You can implement the variability model for a package with settings and options, build the most used binaries. When a given variant is requested, Conan will error saying there is not precompiled binary for that configuration. Users can specify --build=missing to build those from sources.
We are struggeling hard with how to use features the correct way.
Let’s say we have the plug-in org.acme.module which depends on org.thirdparty.specific and org.acme.core.
And we have the plug-in org.acme.other which depends on org.acme.core.
We want to create an application from these, which includes a target file and a product file. We have the following options:
One feature per module:
org.acme.core.feature
org.acme.core
org.acme.module.feature
org.acme.module
org.acme.other.feature
org.acme.other
org.thirdparty.specific.feature
org.thirdparty.specific
This makes the target and product files gigantic, and the dependencies are very hard to manage manually.
One feature per dependency group:
org.acme.module.feature
org.acme.core
org.acme.module
org.thirdparty.specific
org.acme.other.feature
org.acme.core
org.acme.other
This approach makes the dependencies very easy to manage, and the target and product files are easy to read and maintain. However it does not work at all. The moment org.acme.core changes, you need to change ALL the features. Furthermore, the application has no say in what to package, so it can’t even decide to update org.acme.core (because of a bugfix or something).
Platform Feature:
org.acme.platform.feature
org.acme.core
org.acme.other
org.thirdparty.specific (but could be its own feature)
org.acme.module.feature
org.acme.module
This is the approach used for Hello World applications and Eclipse add-ons - and it only works for those. Since all modules' target platforms would point to org.acme.platform.feature, every time anything changes for any platform plug-in, you'd have to update org.acme.platform.feature accordingly.
We actually tried that approach with only about 50 platform plug-ins. It's not feasible to have a developer change the feature for every bugfix. (And while Tycho supports version "0.0.0", Eclipse does not, so it's another bag of problems to use that. Also, we need reproducibility, so having PDE choose versions willy-nilly is out of the question.)
Again it all comes down to "I can't use org.acme.platform.feature and override org.acme.core's version for two weeks until the new feature gets released.
The entire problem is made even more difficult since sometimes more than one configuration of plug-ins are possible (let's say for different database providers), and then there are high level modules using other child modules to work correctly, which has to be managed somehow.
Is there something we are missing? How do other companies manage these problems?
The Eclipse guys seem to use the “one feature per module” approach. Not surprisingly, since it’s the only one that works. But they don’t use target platforms nor product files.
The key to a successful grouping is when to use "includes" in features and when to just use dependencies. The difference is that "includes" are really included, i.e. p2 will install included bundles and/or included features all the time. That's the reason why you need to update a bundle in every feature if it's included. If you don't update it, you will end up with multiple versions in the install.
Also, in the old day one had to specify dependencies in features. These days, p2 will mostly figure out dependencies from the bundles. Thus, I would actually stop specifying dependencies in features but just includes. Think of features as a way to specify what gets aggregated.
Another key point to grouping is - less is more. If you have as many features as bundles chances a pretty high that you have a granularity issue. Instead, think about what would a user install separately. There is no need to have four features for things that a user would never install alone. Features should not be understood as a way of grouping development/project structures - that's where folders in SCM or different SCM repos are ok. Think of features as deployment structures.
With that approach, I would recommend a structure similar to the following example.
my.product.base
base feature containing the bare minimum of the product
could be org.acme.core plus a few minimum
my.product.base.dependencies
features with 3rd party libraries for my.product.base
my.addon.xyz
feature bundling an add-on
separate features for things that can be installed separately
my.addon.xyz.dependencies
3rd party libraries for add-on dependencies
Now in the product definition I would list just my.product.base. There is no need to also list the dependencies features. p2 will fetch and install the dependencies automatically. However, if you want to bind your product to specific versions of the dependencies and don't want p2 to select any matching one, then you must include the my.product.base.dependencies feature.
In the target definition I would include a "my.product.sdk" feature. That feature is an aggregation feature of all other features. It makes target platform management easier. I typically create an sdk feature with everything.
Another feature that is also very often seen is a "master" feature. This is an "everything" feature that maybe used for creating a p2 repository during the build. The resulting p2 repository is then used for assembling products.
For a more real world example see here:
http://git.eclipse.org/c/gyrex/gyrex-server.git/tree/releng/features
Features and Continuous Delivery
There was a comment regarding frequent updates to feature.xml. A feature.xml only needs to be modified when there is a change in structure. No updates need to happen when the bundle version is modified. You should reference bundles in features with version 0.0.0. That makes Tycho to fill in the proper version at build time. Thus, all you need to do is commit a change to any bundle and then kick off a rebuild. Tycho also takes care of updating the feature qualifier based on the qualifiers of the contained bundles. Thus, the new feature qualifier will be different than in a previous build.
In a project where some targets are to be build and run on the build platform and other targets are to be build for a cross platform; what options do we have, when using cmake?
Currently I use CMAKE_BUILD_TYPE to define tool chain, build type and platform (for example -D CMAKE_BUILD_TYPE=arm_debug). In one place in the build, I switch tools (compilers, linke etc.), command line flags, libraries etc. according to the value of CMAKE_BUILD_TYPE. For every build type, I create a build directory.
This approach has it's drawbacks: multiple build directories and no easy way to depend one target from one build type on a target in an other build type (some kind of precompiler needed on the build platform by the build for the cross platform for example).
As currently every build targets has a single tool chain to be used I would love to associate a target with a target platform / tools set. This implies that some libraries have to be build for more than one target platform with different tool sets.
The 'one build type and platform per CMake run' limitation is fundamental and I would strongly advise against trying to work around it.
The proper solution here seems to me to split the build into several stages. In particular, for the scenario where a target from one build type depends on a target from another build type, you should not try to have those two targets in the same CMake project. Proper modularization is key here. Effective use of CMake's include command can help to avoid code duplication in the build scripts.
The big drawback of this approach is that the build process becomes more complex, as you now have several interdependent CMake projects that need to be built in a certain order with specific configurations. Although you already seem to be way beyond the point where you can build your whole system with a single command anyway. CMake can help manage this complexity with tools like ExternalProject, that allows you to build a CMake project from within another. Depending on your particular setup, a non-CMake layer written in your favorite scripting language might also be a viable alternative for ensuring that the different subprojects get built in the correct order.
The sad truth is though that complex build setups are hard to manage. CMake does a great job at providing a number of tools for tackling this complexity but it cannot magically make the problem easier. Most of the limitations that CMake imposes on its user are there for a reason, namely that things would be even harder if you tried to work without them.
My question is the following:
do I need to recompile all the VxWorks O.S. libraries for the target architecture each time I switch between my projects, one for Pentium M and one for Atom?
If this is the case, is there some smart strategy to shorten the procedure?
In the case you have sources of vxworks in your distribution you can create source build projects for each BSP and build them. These projects produces the complete set of libraries. You can create VIP (vxworks image project) based on source build project and your image will be linked with the libraries you produced. One thing to remember - the source build project doesn't contain sources, but only links on them, so if you will change the source in one source build project it will be changed in all projects and the next build will change the produces library. So be aware of it. Hope this will help you.
The trick is to use SCCS (source code control system, like git, cvs, clearcase, whatever), then create two local workspaces from the same repo (one for Pentium M, and the other for Atom). When you make changes in one workspace you commit the change to the repo, and then merge those changes into the other workspace.
I'm currently building a product distributed through MSI Windows Installer. That product is being integrated by our customers using different forms such us inside their own MSI, using bootstrapper/chainner like WiX Burn or authoring tools like InstallShield.
Having this scenario in mind, I always wanted to know what are the limitations and/or benefits to use Merge Modules (MSM) instead of keeping an MSI, and also what are the nowadays recommended approach about choosing one against the other.
On paper merge modules are fine, but in the real world I find them clunky to update and hence error prone since they may be merged into many setups before being discovered to be defective. As a result I do not recommend merge modules at all. I prefer a single MSI that can be run as a batch process via a bootstrapper or batch file and that can also be updated easily. This avoids all kinds of problems that are not generally intuitive.
I want to add that merge modules work well for truly shared files installed in locations in the file system that are meant for shared files and that change infrequently. These are generally OS-runtimes. These merge modules are generally heavily tested and work ok. However, often I see people use merge modules for files that end up changing frequently and that they then end up installing in different locations in different flavors in an ad-hoc fashion. This kind of use is a total mess and a hugely wasted effort.
Having said all that - I have indeed used merge modules successfully when I have needed advanced release management with repetitive, non-changing inclusion of a set of files via a merge module into several setups. Even then I ran into a version issue after a while with a couple of files needing update, and subsequent, minor errors with the wrong merge module being used when I left the project to someone else. I also experienced having to rebuild all setups due to a minor merge module bug fix. All setups then had to go through QA again. Very frustrating with such tight coupling.
If your requirements are simple and you are not taking on a huge multi-product release project sharing a bunch of files, use MSI instead of MSM. Easier to comprehend, generally less work to deal with, more atomic updates and less risk of introducing the same error in many setups due to merge module update or design problems.
There's nothing that wrong with merge modules. Their primary use (which hasn't been mentioned) is sharing. If you want the same set of shared files in multiple MSI files, they need the same set of component guids to preserve the sharing rules. Or if you are giving files to clients for them to use (like Microsoft) in their MSI builds then give them merge modules. That's one of the reasons MS and other vendors redistribute merge modules so that everyone can build their MSIs and install them on the same system without file sharing disasters. I've also seen merge modules used as a common UI for MSI files. But mainly they are essential to make sure that shared files are used correctly. I'll tell you from experience that the disaster resulting from incorrect use of shared files is much worse that any perceived difficulty with using merge modules. Note also that they are universal and can be included in all tools that build MSI files.
I've never found merge modules difficult to patch, version, or fix. Major upgrades aren't a problem. The only potential issue I've seen is with build processes that rebuild all the binaries in a merge module during creation of a patch (.msp) build. If only one binary needs a fix but you compile them all, their versions and internals may change enough so that the patch process (the delta between two MSI files and their content) will tell you that they need to be included in the patch because they've changed, but this issue can be avoided if it is in fact an issue.