I would like to distribute an application that depends on several PyPI-packaged libraries. I have carefully selected certain versions of some of these libraries as newer versions (in some cases) are incompatible. My installer downloads them (with pip) at install-time and sets up the environment for the application. But how long are those versions going to be available? 6 hours? 2 years? Anything in between?
I'm basically looking for some sort of policy that tells me how long those versions of libraries are going to be hosted on PyPI (and who makes that decision).
In-before-"distribute them yourself": That is an answer to a different question.
This is really about how PyPI works, not how I distribute my application.
The friendly people in #python tell me that authors can delete any version of their packages at any time.
The only way to indemnify yourself against a version of something becoming nuked is to (assuming their license allows it) ship it yourself.
There is an argument for continuous integration against the latest versions on PyPI but that does assume there will be a new version and that the author doesn't just delete the whole thing. CI is just a good practice here, not a panacea.
Related
I have a fairly popular open source project that has a minimum cmake version set that is fairly old (~Oct 2013).
cmake_minimum_required(VERSION 2.8.12)
This is great for supporting a wide variety of users, but it makes adding new things to our cmake builds challenging, since we cannot use any of the new features in later cmake versions. This leads to complicated cmake files and more maintenance burden to emulate future improvements made to cmake.
I would like to support two CMakeLists.txt files for the project that have different minimum versions, one legacy one at the 2.8.12 version, and another more modern one at a later 3.x version. Is there any canonical way to do this? It doesn't look like there is a way to support multiple CMakeFiles.txt at the root of the project.
The naive way is to just rename the legacy file CMakeFiles_Legacy.txt and have users that depend on the older cmakes manually switch out the CMakeFiles.txt on their own.
tl;dr: Just don't support old versions.
CMake is absurdly easy to upgrade. Literally every halfway-modern development platform natively packages at least CMake 3.16+, but Windows and macOS are both way ahead of the curve with constant updates to the included version.
Visual Studio receives periodic updates and 2022 includes CMake 3.21. The version on Chocolatey is always up to date.
On macOS the Homebrew version is always up to date.
On Debian derivatives, Kitware provides an official APT repo that is always up to date. There is also a Snap package that is always up to date.
On most other Linux distros, one can use the official binaries directly from Kitware without sudo access at all. They have x86_64 and aarch64 binaries. The binaries are statically linked and require only libc6 as a dependency. Glibc has been ABI-stable since 1997. It will work on your system. But even if it doesn't, building a recent CMake version isn't difficult anyway. Your users maintaining such archaic systems shouldn't be your liability.
(aside: I wrote an entire blog post on this... https://alexreinking.com/blog/how-to-use-cmake-without-the-agonizing-pain-part-1.html)
I want to know what is the difference between installing HortonWorks HDP vs installing the components directly from Apache projects? One thing I can think of is that Horton works probably has the packages aligned so that the version of each component is compatible with that of the others within the suite, while getting them directly from Apache projects, I may have to handle version compatibility myself. Is that correct? Is there any other difference involved ignoring the support subscription aspect of it.
Thanks.
There are a lot of differences between "roll your own" and using a distribution. Some of the most obvious include:
All of the various components and versions have been tested and built to work together - incompatibility between versions (e.g. Hive, Hadoop, Spark, etc.) can be a painful problem to sort through on your own
Most distribution providers, including Hortonworks, will bring patches in from unstable releases into stable releases, so even for the "same" version (e.g. Hive 1.2.1) you're getting a better release than vanilla - these can include both bug fixes and "safe" feature changes
Most distribution providers, including Hortonworks, provide some flavor of centralized platform management. I'm a big fan of Ambari (the one that comes with HDP), for example - it makes configuration and monitoring significantly easier than coordinating a vanilla install
I would strongly recommend against trying to deploy vanilla, unless it's just for learning and playing. HDP community edition is free (both definitions) and a major improvement over doing it yourself. My last deployment of HDP was entirely based on the community edition.
In CMakeLists.txt scripts, it is customary to have a:
cmake_minimum_required(VERSION x.y)
for the appropriate x.y version number. But - how can you tell what that minimum version is? I don't work with older versions; and as my distribution gets updated, so does CMake, so I may have even introduced commands requiring newer versions over time and I just don't know it.
I could theoretically require the versions I've tested this CMakeLists.txt with - but they're pretty new, and I don't want to restrict users of my code to just that.
There is no automatic way to do this. Here are some approaches I've used:
If you have used CMake for a while, you may roughly remember what features are newer and older. Select some of the features you use which you think are newer, and read the documentation to discover what version supports them.
If you can install just one or two old versions, it may be enough. CMake 2.8.x is very popular, so installing that and testing to see if it works (perhaps fixing it so it does) would be a nice service to some users.
Don't worry about it. Set the minimum to 2.8.10 and then accept patches or bug reports if users have specific issues. It's not likely that setting this number too low will cause any serious harm--typically it will just result in a different error message than "The CMake version is too old."
tl;dr: Try building with different CMake versions.
Download the CMake binary from https://cmake.org/files/ according to the claimed minimal required version. You can unpack them in some directory and use this CMake to ensure it is still the minimal required version. You can have as many versions as you wish in parallel. Don't forget to delete the build directory.
A problem you did not mention, but is also important: Check your code with newer version. New policies can lead to dozens of warnings, many projects try avoiding warnings in releases. So it might be good to have the latest version with the same procedure.
I want to define a minimum version to CMake with "cmake_minimum_required" facility.
I have seen that some project set minimum version 2.8 some others set 3.0 or 3.2.
I would like to learn your opinions and best practices about the topic.
The cmake_minimum_required() function is used to avoid any cryptic error messages due to the CMakeLists.txt assuming a later version of CMake than the one installed on the current host.
As an example, failing early, and with a clear message...
CMake 3.2 or higher is required. You are running version 2.8.12.2
...is to be preferred over something more cryptic (much) later on...
In file included from /home/foouser/src/testprj/string.cpp:1:0:
/home/foouser/src/testprj/string.hpp:94:29: error: ‘std::is_same’ has not been declared
...just because, in this example, the older CMake version does not support set( CMAKE_CXX_STANDARD 11 ). I am sure you'll agree.
The ideal setting would be:
The oldest version with all the features your script needs.
Maximal compatibility with people running older versions, as well as with your script. But it requires testing which version exactly it was that first supported your constructs. So it usually boils down to:
The oldest version you have tested that has all the features your script needs.
That's probably good enough for most projects. And if you are the only one actually working on the project, and testing for CMake compatibility is really low on your list, you will probably end up with:
The version you are currently using.
This latter approach has a serious drawback once somebody else attempts to compile your project. Quite a few people are not using the latest version of everything. On Linux in particular, the default is to use whatever the package manager gives you. Ubuntu wily, for example, is currently at version 3.2.2 -- you may have a later version, but unless you need a later version, you shouldn't require it (as that means people won't be able to build your project without first installing a newer version of CMake, manually).
What you should not be doing is...
Requiring a very old version, but not actually testing against that old version (NO!).
The reasons should be obvious -- building could fail, without the user getting any hint as to why things went wrong.
With the new firefox we are shipping more and more libraries as the XPom interfaces we interact with are changing. We are at 10 dlls and increasing, each with a size of almost 2M.
This size is a concern for some users.
While we look at restructuring the library to seperate the parts we can make common between them, we are thinking about how we might reduce space on the disk while supporting version upgrades.
For instance, user has FireFox 3.6 and 4.0 installed and when our product is installed we install a dll for each version. When Firefox is 4.0 upgraded (say to 6.0) how might we now install from the msi the missing dll for 6.0 support.
Any ideas on how we could achieve this?
Are we worrying for no reason?
My first thought was we 'AllowAdvertise' and when FF tries to load the dll as directed by chrome it will cause the install, it doesn't seem to work.
My first reaction is to suggest that you move away from XPCOM and towards js-ctypes. After all, this is the direction that Mozilla is pushing extension developers (see Wladimir Palant's comments for example). If there isn't anything in your binary code that absolutely positively requires use of XPCOM, you'll be much happier to ship a DLL that interfaces with JS when needed via js-ctypes.
I guess that your extension is Windows-only so supporting multiple platforms is not an issue. A possible short-term solution:
Have a separate extension package for each Firefox version, mark it as compatible with this Firefox version only (e.g. minVersion 4.0 and maxVersion 4.*).
When your extension is installed, install the version that is compatible with user's installed Firefox version.
Make sure that your extensions have an updateURL entry that is pointing to your server. It is important to have %APP_VERSION% in the URL.
Make sure to test Firefox betas and prepare a new extension version in time for the next Firefox release (releases are scheduled on Tuesdays every 6 weeks, next release being on September 27th).
Configure your server to indicate different packages as updates depending on the Firefox version used. So an update check with %APP_VERSION% 4.0.1 would be sent to extension-ff4.xpi while %APP_VERSION% 6.0 would get extension-ff6.xpi.
Firefox will always check for extension updates when the application is updated. If you can give it a compatible update it will install it. But preparing new packages every six weeks requires tons of effort and I guess that you want to refactor your code/move to js-ctypes ASAP. Oh, and I think that you need to ignore the unlikely scenario that some user has more than one Firefox version installed.