Trying to download a public file from SharePoint via CMake's FetchContent.
My URL is like the following:
https://myorg.sharepoint.com/:u:/s/myfolder/EdajXJq3IV5HrSs9bKhEFoYByaMZHBYHyftA9GKLAGZ5wA?e=QPdu1N&Download=1
Note I added &Download=1 to the path given by SharePoint to access the file directly. However, my link gets redirect every time I use it. I'm able to download the file using wget & curl:
curl -v -L --cookie tmp.cookie 'https://link.from.above' --output myfile.txt
wget 'https://link.from.above'
Now trying do the same using CMake:
FetchContent_Declare(${MY_TARGET}
URL ${FILE_URL}
)
But that doesn't work. I guess it has something to do with redirection / cookies.
Same problem here. We are currently using the following workaround (which is far from perfect):
Check if the file exists in a specific location in the source directory
If it does not exist, download the file using curl
Pass the path to the downloaded file to FetchContent_Declare
If anyone has better ideas, I would like to hear them!
Related
Trying to get the sidekick image built and having some issues. Is there any documentation other than the README.md file?
My current problem is with getting the JRE requirement working but there are others. The page says "download Oracle JRE and place it inside the working directory. Optionally if you have a company wide distribution url, use that one at a later step." and the help says "Java (JRE) download url or path inside working directory". Have not been able to get this to work.
I went to the JRE link provided and was presented with options to download a rpm file or a tar.gz file. Which is expected (was unable to get either one working)?
It says to place the file in the "working directory" but not sure where exactly. Tried in sidekick folder and in sidekick/jre both without success no matter what I used after the -j command. Is this just the path or should the filename be included as well? Can I get an example?
I'm running this script using my login but noticed the output folder is being created with root user and group. I see no indication that this should be run with sudo. What is the correct way to run this script?
Using debug, I see the function "download if not cached". Can I save these files (JRE, Bamboo jar file, etc.) somewhere so I don't have to worry about downloading them? If so, where should they go? Looks like I might have a problem with the wget to d/l the jar file so would like to just be able to place all these in a folder and be done with it.
It looks like the major problem is the script didn't clean up after itself if it fails. The issue was the first time it failed then that caused subsequent issues as the output folder was already there. Removing this directory between each attempt help.
As for the correct syntax for the -j JRE option I manually downloaded the JRE and placed in a folder called per-build-container/sidekick/stuff/. For the command line it is not just the path but the file name as well (the tar.gz and not the RPM). For my case it was
-j stuff/jre-8u251-linux-x64.tar.gz
Note I also ran the script as sudo. Wasn't stated but seemed to work OK.
Another issue I ran into was the download of the agent jar file. There is a redirect in the wget file that was not working for us. I ended up editing the script and replacing the Altassian based url with the redirected one.
This addresses all the issues I ran into with the initial question.
I am trying to download a bunch of PDF's from the federal reserve archives but I have to click on a link and then view the PDF before I can download. Is there a way to automate this?
Example: https://fraser.stlouisfed.org/title/5170#521653 is a link to speeches and then you have to click the title, then view pdf, then the actual download button.
All of the remote .pdf files follow the path format:
https://fraser.stlouisfed.org/files/docs/historical/frbatl/speeches/guynn_xxxxxxxx.pdf
where each x is a placeholder for a digit.
So, yes, it's very easy to download a bunch of these PDFs in one go using the command-line in Terminal or whatever shell program you have access to.
If you're in a *nix-based operating system (including MacOS), that's good because your shell probably already has a command utility called curl installed. Windows may have it too, I'm not sure; I don't use Windows.
If you're using Windows, you'll have to make some tweaks to the code below, because the folder structures and file naming conventions are different, so the first couple of commands won't work.
But, if you're happy to proceed, open up a Terminal window, and type in this command to create a new directory in your Downloads folder, into which the .pdf files will be downloaded:
mkdir ~/Downloads/FRASER_PDFs; cd ~/Downloads/FRASER_PDFs
Hit Enter. Next, If there's no error, copy-n-paste this long command and then hit Enter:
curl --url \
"https://fraser.stlouisfed.org/files/docs/historical/frbatl/speeches/guynn_{"$(curl \
https://fraser.stlouisfed.org/title/5170#521653 --silent \
| egrep -io -e '/files/docs/historical/frbatl/speeches/guynn_\d+\.pdf' \
| egrep -o -e '\d+' | tr '\n' ',')"}.pdf" -O --remote-name-all
You can see this uses the URL you supplied in your question, from which that command retrieves all the .pdf links. If you need to do the same with other similar pages, provided they all use the same URL format, you can just substitute 5170#521653 with whatever page reference contains another list of .pdfs.
I have to download all site content and then parse the downloaded folder for "*.pdf" files. I am downloading site using wget -r --no-parent http://www.example.com/ But the problem is that sometimes link looks this
http://www.foodmanufuture.eu/dpubs?f=K20
and the dowloaded pdf is downloaded with name "dpubs?f=K20" and file format is not specified, it does not look like this "dpubs?f=K20.pdf", is there a way to check how many pdf files I have in this folder?
Have you tried the --content-disposition flag? From the man page:
If this is set to on, experimental (not fully-functional) support for "Content-Disposition" headers is enabled. This can currently result in extra round-trips to the server for a "HEAD" request, and is known to suffer from a few bugs, which is why it is not currently enabled by default. This option is useful for some file-downloading CGI programs that use "Content-Disposition" headers to describe what the name of a downloaded file should be.
So it tries to ask the server for a filename. I tried it for the URL you gave and it seemed to work.
You could use the command
file filename
Like this:
file pdfurl-guide
pdfurl-guide: PDF document, version 1.5
You could use:
file *
To know exactly which files in your folder are pdf files
I am trying to install Yii framework via archive file in Windows 7...after downloading the file I tried extracting it into a folder in my C:\wamp\www\bid location, however, I encounter an error saying this:
! C:\wamp\www\bid\yii-basic-app-2.0.2.tgz: Cannot create symbolic link C:\wamp\www\bid\basic\vendor\bin\markdown
A required privilege is not held by the client.
! C:\wamp\www\bid\yii-basic-app-2.0.2.tgz: Cannot create symbolic link C:\wamp\www\bid\basic\vendor\bin\yii
A required privilege is not held by the client.
I thought that perhaps Winrar can't extract tgz files so I downloaded 7-zip...Using 7-zip it extracted into a tar file and the tar file was extracted with no errors...For some reason though I do not seem to have the framework folder that seem to be in other people's directory structure...Moreover after trying create my own framework folder and performing this command:
yiic webapp C:\wamp\www\bid
It states that yiic is not recognized as an internal or external command
Can someone tell me what I'm doing wrong...I've tried setting this up numerous times and failed
You need to run 7-Zip File Manager in administrator mode.
Right-click the icon of 7-Zip File Manager, and then click "Run as administrator".
you need to call the command with php since it is actually a external command, i do it like this
php ./yiic webapp NameOfApp
This way your terminal understand that yii is a program that runs with php.
Another solution is to add the yii path to your environmental variable, regards
I am having trouble with compiling one of the open source libraries (libopekele OpenID lib).
The problem is I don't have sudo access on the system where I need to compile this.
Ran the configure. It complained of missing htmltidy lib.
Installed the htmltidy at a non-standard path /home/geekgod (as I dont have access to the /usr and /usr/local).
Now the problem is how do I make the configure script of libopekele to pick the the headers from /home/geekgod/include.
Poking into the configure.ac script of libopkele, it is using AC_CHECK_HEADERS to search for tidy.h or tidy/tidy.h.
I am pretty sure it is looking for these at standard location (/usr/include).
How do I add /home/geekgod to the standard include dir?
try this:
./configure CPPFLAGS=-I/home/geekgod/include --prefix=... --etc