How do I tell ReadTheDocs to build my project packages from a sub-directory? - setup.py

I have a repository that contains three python packages: a main package, and two addon packages, with shared documentation. Each project is in its own directory, with its own setup.py, as so:
Repository
Main Project
setup.py
Addon One
setup.py
Addon Two
setup.py
Documentation
RST files, RTD conf, etc.
Previously, I was using setuptools.find_packages() to build my packages, but was having issues with the contents of the packages bleeding together, as they shared namespaces. So I switched to specifying the packages I wanted to build, such as
packages=["Main Package"]
However, this broke my ReadTheDocs auto-build, where I had specified
- method: setuptools
path: Main Project
in .readthedocs.yml, with RTD now complaining my package (inside the Main Project directory) doesn't exist, as it attempts to build it.
In my project, I use a script to build the packages, where I move into each directory, run its setup, then move out. Works fine, my packages and documentation all build locally. However, it looks like RTD is only using the defined path to prepend my setup.py script, and therefore not finding the source package as the working directory is the parent directory (but I could be wrong!).
I've read through the documentation, including https://github.com/readthedocs/readthedocs.org/issues/4354 where the feature was originally added, but I have not been able to find a solution myself yet.
How can I tell RTD to change directory before building the packages, or is there an alternative approach that will support my repo structure?
Repository in question is here.

I found a solution:
I changed my local build script to use the root project directory, as per-RTD. I then added the directive package_dir={"": "[directory]"} to the setuptools.setup() calls in each project's setup.py.

Related

How to handle package dependencies for some build targets when building with Copr?

I want to have a rpm package build with Copr1. My current build target list is Fedora 35, 36, rawhide and Centos 7 and Stream 8. I have not yet created the copr project.
Compiling on one of my machines, the package builds successfully on the Fedora variants with mock. The problem is that on Centos variants one of the build dependencies and some of its dependencies are not available. I have found appropriate srpm files and compiled them on one of my machines with the Centos Stream 8 (one of them required two custom patches). With those custom dependencies I am able to successfully compile the original package.
So just to be clear, the problem is that the spec file contains for example
BuildRequires: libsomething
where libsomething is available as a plain upstream package in some of the build targets while needs an additional custom repo for some other build targets.
The FAQ says the following about dependencies:
Can I depend on other packages, which are not in Fedora/EPEL?
Yes, they just need to be available in some yum repository. It can either be another Copr repo or a third-party yum repo (e.g jpackage). Click on “Edit” in your project and add the appropriate repositories into the “Repos” field. Packages from your project are available to be used at build time as well, but only for the project you are currently building and not from your other projects.
But this sounds like an all or nothing approach, and I absolutely do not want to override the already existing upstream packages, only provide them when they are missing.
So what strategy do people use to handle this?
Update: I have now created copr projects and made some attempts at building (after resolving dependencies of dependencies in several levels), but the problem is as I describe above. If I add copr://hlovdal/projectname as a build dependency then epel-8-x86_64 compiles fine because it is provided with the missing dependencies while fedora-35-x86_64 fails because the repository does not have any fedora packages. If I remove the repo epel fails while fedora succeeds.
I also attempted to add the base url from the corresponding /etc/yum.d.repo file, and only hardcode epel instead of $distname hoping that the fedora builds would just ignore non-existing/wrong repo setting, but the build does not like that and still fails.
1 Copr is Fedora's freely available build system.

Built two different Debian packages for different Build Types using CMake

I have a small CMake project with different Build Types debug and release. I'm also providing a Debian package for this project. Building the Debian Package for release and providing it on my own Debian repository works perfect.
Now I also want to provide another Debian package for debug, due to debugging purposes, with a different package name. For example, my project is called myproject, and the debugging package should be myproject-debug.
I already read documentation about how to solve this in the debian/control file. I want to use Replaces: ... on each package vice versa, so that you can install only one of the both packages at a time. So either myproject or myproject-debug, but not both at the same time, to use the exact same files and filenames but only the binary has more debugging informations and debug prints in the myproject-debug package. Everything else should be the same. Same filename, same paths, etc.
Now the problem is that I don't know how the debian/rules file should look like, to first build the myproject package in a folder and then build the myproject-debug with different CMake options (-DCMAKE_BUILD_TYPE=debug) in a different folder, so the filenames can and should stay the same.
There is this CMake tutorial in the Debian documentation, but this doesn't fit my requirements. Because in this tutorial everything will be built in only one folder, and in this one folder there are different files. Then different .install files will be used to copy the needed files to each package. But since I have the same binary filename for each package myproject and myproject-debug this tutorial does not really fit my needs.
I already have the following lines in my debian/rules file:
override_dh_auto_configure:
dh_auto_configure -- -DCMAKE_BUILD_TYPE=release
But how can I run two different builds with two different build types?
For example, something like this, to split it up:
override_dh_auto_configure_release:
dh_auto_configure -- -DCMAKE_BUILD_TYPE=release
override_dh_auto_configure_debug:
dh_auto_configure -- -DCMAKE_BUILD_TYPE=debug
And run both in different folders so I can add both folders to two different packages.
Or maybe there is even a better solution I cannot imagine yet?

Conan, C++ package manager, don't work for boost

I run: conan install Boost/1.64.0#conan/stable, and it fails.
Output:
C:\temp>conan install Boost/1.64.0#conan/stable
Boost/1.64.0#conan/stable: Not found in local cache, looking in remotes...
Boost/1.64.0#conan/stable: Trying with 'bintray'...
Boost/1.64.0#conan/stable: Trying with 'conan.io'...
ERROR: Unable to find 'Boost/1.64.0#conan/stable' in remotes
Trying other package, works:
C:\temp>conan install fmt/4.0.0#bincrafters/stable
fmt/4.0.0#bincrafters/stable: Not found in local cache, looking in remotes...
fmt/4.0.0#bincrafters/stable: Trying with 'bintray'...
fmt/4.0.0#bincrafters/stable: Trying with 'conan.io'...
Downloading conanmanifest.txt
[==================================================] 121B/121B
Downloading conanfile.py
[==================================================] 1.8KB/1.8KB
fmt/4.0.0#bincrafters/stable: Installing package
Requirements
fmt/4.0.0#bincrafters/stable from conan.io
Packages
fmt/4.0.0#bincrafters/stable:63da998e3642b50bee33f4449826b2d623661505
fmt/4.0.0#bincrafters/stable: Retrieving package 63da998e3642b50bee33f4449826b2d623661505
fmt/4.0.0#bincrafters/stable: Looking for package 63da998e3642b50bee33f4449826b2d623661505 in remote 'conan.io'
Downloading conanmanifest.txt
[==================================================] 938B/938B
Downloading conaninfo.txt
[==================================================] 491B/491B
Downloading conan_package.tgz
[==================================================] 159.8KB/159.8KB
fmt/4.0.0#bincrafters/stable: Package installed 63da998e3642b50bee33f4449826b2d623661505
Any idea why the package isn't found?
How to debug it?
Conan is a decentralized package manager (kind of git-like style), so it can have many remotes. By default it comes configured with 2 remotes:
conan-transit: Is a read-only copy of the old conan.io repository, which contains many different Boost packages, from different authors. Quality is variable, so some packages might work only for certain OS, or might fail for some configurations.
conan-center: It is a moderated/reviewed repository, package creators can submit inclusion requests to share their packages with the community.
So far conan-transit contains several Boost/1.64 packages, so can check it with:
$ conan search Boost* -r=conan-transit
$ conan search Boost* -r=conan-center
As you can see the package you are trying to install doesn't exist in these repositories.
As I said above, conan is decentralized, so you can use different remotes. For example, the "bincrafters" community has a bintray repo that can be added with:
$ conan remote add bincrafters https://api.bintray.com/conan/bincrafters/public-conan
$ conan search Boost* -r=bincrafters
You will see they have a large number of Boost/1.64 packages, because they have created a modularized version of boost, in which every library lives in a different package, so you only get installed what you need.
UPDATE: Packages in the central repository are being renamed by the community to lowercase. Try with boost lowercase in the above if necessary.

Integrating NuGet into a large, existing code base, with a shared packages directory and a shared solutions directory

Q: Is it possible/feasible, to have a multiple solutions stored in a single 'Solutions' directory, and multiple NuGet packages stored in another single 'Packages' directory, and for everything to work nicely with different versions?
Further details...
For example: I have 2 projects. ProjectA requires Newstonsoft.Json.4.5.11, ProjectB requires Newstonsoft.Json.5.0.6.
For sake of example I have a solution file for both. I need all my solution files in the same directory (this is just the process that is followed, all the solutions in a directory are built in turn).
By default NuGet will create a packages directory alongside each solution file.
I have created a nuget.config file to allow me to store packages in a single directory, called 'SharedPackages', following this answer: Nu-Get & issue with project level dependences for projects referenced by multiple solutions
<settings>
<repositoryPath>..\SharedPackages</repositoryPath>
</settings>
This works great so far, so my structure is:
\Projects\ProjectA
\Projects\ProjectB
\Solutions
\SharedPackages
If I create ProjectB, it has Json.NET 4.5.11 by default. If I go to Manage NuGet Packages for Solution I have the option to update it to version 5.0.6. This is great as ProjectB needs the newer version. What is even better is now in my Shared Packages directory I have a directory for both versions of Json.NET side-by-side, so ProjectA can use the older version.
However, now I want to create ProjectC as a full MVC4 Web Application. For JQuery, you get version 1.8.2 currently when creating an ASP.NET MVC4 application in VS2012. I also get Knockout 2.2.0.
My process is, I delete the default packages directory, move the new solution to the Solutions directory alongside the existing nuget.config and edit the new solution file to update the relative path to the new .csproj file. Then when I build, NuGet Package Manager restores the extra packages I need (that weren't in use by ProjectA and ProjectB) to the Shared Packages directory. However... I get build errors, it cannot resolve some references including DotNetOpenAuth, WebGrease, System.Spatial... the references are pointing to the packages directory, not the SharedPackages directory...
As an aside: if I Enable Package Restore for solution, then it also tries to restore them to a packages folder within the Solutions directory by default, instead of restoring them to the SharedPackages directory.
Around about this point I realise that just creating the nuget.config file wasn't enough for ProjectA and ProjectB either, although they appeared to be working originally, the references in the .csproj. file are pointing to the bin folder beneath the project file, instead of my SharedPackages directory.
So I manually 'Find and Replace' ..\packages with ..\..\SharedPackages for all the references. I have to do this for ProjectA, ProjectB and ProjectC. Now everything builds and seems to work OK, new packages go into the right place.
Now, if I go back to ProjectA, and add the Knockout package, this is version 2.3.0. This installs happily alongside the other Knockout package in use by Project C which is version 2.2.0. Doing this also installs JQuery 2.0.3, alongside JQuery 1.8.2. So far so good.
Just as a sanity check, I create another Web Application - ProjectD, move the files around, update the references in the solution and the project. This time everything builds first time. I try and update WebGrease in ProjectD to see if it will retain the older version for ProjectC. This results in more issues, it installs it to the packages directory instead. WebGrease seems to have a separate config setting as well <WebGreaseLibPath>... it won't seem to restore...
I then go back to ProjectB and try 'Update All' - it looks like the files that already exist are updated in SharedPackages, with new version directories alongside the existing ones, but any new dependencies (e.g. now I have a reference to Owin.dll) get placed in the packages folder :( If I delete the packages folder, and the bin folder within ProjectB, then build the ProjectB solution, understandably I get build errors, the packages aren't automatically restored to the SharedPackages directory at any point.
Is it even possible to set NuGet to update packages in a common directory other than packages alongside the solution?
Would it be easier to just use the default packages folder, instead of SharedPackages, or would I still have problems?
This is turning into way too many questions. To try and keep it in scope, has anyone attempted a similar setup, what obstacles did they overcome and how did they manage it, or did they give up altogether? If you gave up, how did you end up using NuGet to manage packages in a massive code base?
I appreciate this is close to this question, which was well answered for that particular question, however the use case here is slightly different: NuGet and multiple solutions. It is also pretty much identical to this question: Setting up a common nuget packages folder for all solutions when some projects are included in multiple solutions, but I have decided to add this anyway as that question is more focused on the having different configurations for different solutions, whereas here I want all the packages in one place, I just want to implement it and see if it is possible. Also I think the troubleshooting and research time may be useful to someone.

How can I get Eclipse to use my IVY_HOME variable when downloading ivy dependencies?

My company uses extensive use of ivy to download dependencies. Some of these dependencies are huge (~500MB) and take a while to download from the remote repositories.
To build our application we have an ant script that will first resolve all the dependencies and the deploy to the server.
I have set an "IVY_HOME" environment variable so that all the dependencies are downloaded to D:\ivy_home instead of C:\Users\.ivy2\ - this is because D: is my SSD which is significantly faster, and it is where my local server directories are located - so copying files from ivy_home to the server is super fast.
But for some reason when I am using IvyDE plugin inside eclipse - it always wants to download a separate copy of all the dependencies and puts them into my C:\ which is causing several issues:
Local publishes from the ant script will not be picked up in eclipse since they are placed into a different location
Dependencies already downloaded in D: will not get picked up which makes the ivy Resolve inside eclipse much slower than it needs to be
The dependencies are in a slower drive in eclipse so performing searches, and executing these jars is also slower
How about creating symlink to replace the .ivy2 in Users to D? I've tried it on my own and it's looks working fine.
Open cmd as root, and then execute this line
mklink /d C:\Users\{username}\.ivy2 D:\.ivy2
I'd create an ivysettings.xml file and specify the location of my cache using the caches directive. See the following answer for example:
can I turn off the .ivy cache all together?
Why don't you set up IVY globally with the ivysettings.xml along with a property file.
This property file could have this:
ivy.default.ivy.user.dir=D:\ivy_home
For individual projects you could uncheck "enable project specific settings" for each IvyDE library management, so they would use IVY global settings, with one extra eclipse environment configuration.