Download (all) files via FTP with explicit TLS/SSL encryption - ssl

I've been trying to download files from FTP with explicit TLS/SSL encryption from one server to another using Debian. I tried a lot of commands like ftp and wget but none of them worked and said Login is incorrect. I searched whole Stack Overflow and Google.
I tried ftp and wget like this:
wget -m --user=username --password=password ftp://ip
and
ftp user#ip
Thanks in advance.

wget must be version 1.18 or above. The following was tested on 1.19.1
wget -r --level=5 -m --no-remove-listing --reject "index.html" -c --progress=dot -N --secure-protocol=auto --no-proxy --no-passive-ftp --ftp-user=XXXXX --ftp-password=YYYYY --no-check-certificate ftps://ZZZZZZZZ.com:21
Here is the link on how to build wget http://www.linuxfromscratch.org/blfs/view/8.1/basicnet/wget.html

Related

How to download a shared item on google colab using wget?

I have a shared link like the below
https://drive.google.com/file/d/1tfBFavijh4UTG4cGqIKwhcklLXUDuY0D/view
but I am unable to download it google colab using the command
wget https://drive.google.com/file/d/1tfBFavijh4UTG4cGqIKwhcklLXUDuY0D/view
Few suggestion on the internet were to use the below command
wget --no-check-certificate 'https://docs.google.com/uc?export=download&id=FILEID' -O FILENAME
but this didn't work. Any help is appreciated!
You can use gdown
!gdown --id 1tfBFavijh4UTG4cGqIKwhcklLXUDuY0D
Add a retry option after certificate check to the command. An example below
wget --no-check-certificate -r 'https://drive.google.com/uc?export=download&id=1tfBFavijh4UTG4cGqIKwhcklLXUDuY0D' -O FILENAME.zip

The web server not permitted for chmod under WSL Ubuntu

I use Ubuntu 18 as WSL and everything was running well. Today I run the apache and started the application. When the app tried to perform chmod() on a file which was submited through form inside the folder project (I use Laravel), I received the following error:
chmod(): Operation not permitted
I have notice that this error happen when I try running chmod() from web server (www-data user). In the cli I dont have problems.
From other posts over the net, I understand that Windows has some changes regarding WSL permissions and drive mounts. But I didnt get answer or didnt succeed to resolve that issue.
sudo umount /mnt/c
sudo mount -t drvfs C: /mnt/c -o metadata
Reference: https://github.com/Microsoft/WSL/issues/3172#issuecomment-389157376
sudo umount /mnt/c
sudo mount -t drvfs C: /mnt/c -o metadata,uid=1000,gid=1000,umask=22,fmask=111
did the trick for me.
Ref: https://devblogs.microsoft.com/commandline/chmod-chown-wsl-improvements/

How to automatically setup SSH key pass on first ansible command run for each new server?

Today I started learning ansible and first thing I came across while trying to run the command ping on remote server was
192.168.1.100 | UNREACHABLE! => {
"changed": false,
"msg": "(u'192.168.1.100', <paramiko.rsakey.RSAKey object at 0x103c8d250>, <paramiko.rsakey.RSAKey object at 0x103c62f50>)",
"unreachable": true
}
so I manually setup the SSH key, I think I faced this as no writeup or Tutorial by any devops explains the step why they don't need it or if they have manually set it up before the writing a tutorial or a video.
So I think it would be great if we can automate this step too..
If ssh keys haven't been set up you can always prompt for an ssh password
-k, --ask-pass ask for connection password
I use these commands for setting up keys on CentOS 6.8 under the root account:
cat ~/.ssh/id_rsa.pub | ssh ${user}#${1} -o StrictHostKeyChecking=no 'mkdir .ssh > /dev/null 2>&1; restorecon -R /root/; cat >> .ssh/authorized_keys'
ansible $1 -u $user -i etc/ansible/${hosts} -m raw -a "yum -y install python-simplejson"
ansible $1 -u $user -i etc/ansible/${hosts} -m yum -a "name=libselinux-python state=latest"
${1} is the first parameter passed to the script and should be the machine name.
I set ${user} elsewhere, but you could make it a parameter also.
${hosts} is my hosts file, and it has a default, but can be overridden with a parameter.
The restorecon command is to appease selinux. I just hardcoded it to run against the /root/ directory, and I can't remember exactly why. If you run this to setup a non-root user, I think that command is nonsense.
I think those installs, python-simplejson and libselinux-python are needed.
This will spam the authorized_keys files with duplicate entries if you run it repeatedly. There are probably better ways, but this is my quick and dirty run once script.
I made some slight variations in the script for CentOS 7 and Ubuntu.
Not sure what types of servers these are, but nearly all Ansible tutorials cover the fact that Ansible uses SSH and you need SSH access to use it.
Depending on how you are provisioning the server in the first place you may be able to inject an ssh key on first boot, but if you are starting with password-only login you can use the --ask-pass flag when running Playbooks. You could then have your first play use the authorized_key module to set up your key on the server.

ssh wget download jdk

I've SSHed into one of my VPS and I'm trying to install java on it. Not really sure how to go about downloading but I am trying to use wget to download and install JDK7 from the Oracle website.
this file in particular: http://download.oracle.com/otn-pub/java/jdk/7u40-b43/jdk-7u40-linux-x64.rpm
To download the file, it requires authentication and I don't know how to do that through wget.
SOmeone please help
# rpm
wget --no-cookies \
--no-check-certificate \
--header "Cookie: oraclelicense=accept-securebackup-cookie" \
"http://download.oracle.com/otn-pub/java/jdk/7u55-b13/jdk-7u55-linux-x64.rpm" \
-O jdk-7-linux-x64.rpm
# ubuntu
wget --no-cookies \
--no-check-certificate \
--header "Cookie: oraclelicense=accept-securebackup-cookie" \
"http://download.oracle.com/otn-pub/java/jdk/7u55-b13/jdk-7u55-linux-x64.tar.gz" \
-O jdk-7-linux-x64.tar.gz
# then
tar -xzvf jdk-7-linux-x64.tar.gz
refer: https://gist.github.com/scottvrosenthal/11187116
To download the file, it requires authentication and I don't know how
to do that through wget.
To be more accurate it requires license agreement.
If you have SSH access to the server you can copy file using scp command
scp /<path to the file>/jdk-7u40-linux-x64.rpm user-name#server-name:/tmp/
Then you go to the server by ssh and find it in /tmp/ folder

How to install wkhtmltopdf on a linux based (shared hosting) web server

I have tried in all ways to get wkhtmltopdf installed on our web server but unfortunately it is not getting installed. I cannot access user/bin folder as stated in a tutorial on installation.
On the server in public_html folder there is a sub folder _vti_bin, I copied the file wkhtmltopdf-i386 from wkhtmltopdf-0.9.1-static-i386, but I am not able to execute it.
How to install wkhtmltopdf on (shared hosting) web server and get it working?
I've managed to successfully install wkhtmltopdf-amd64 on my shared hosting account without root access.
Here's what i did:
Downloaded the relevant static binary v0.10.0 from here: http://code.google.com/p/wkhtmltopdf/downloads/list
EDIT: The above has moved to here
via ssh on my shared host typed the following:
$ wget {relavant url to binary from link above}
$ tar -xvf {filename of above wget'd file}
you'll then have the binary on your host and will be able to run it regardless of if its in the /usr/bin/ folder or not. (or at least i was able to)
To test:
$ ./wkhtmltopdf-amd64 http://www.example.com example.pdf
Note remember that if you're in the folder in which the executable is, you should probably preface it with ./ just to be sure.
Worked for me anyway
If you have sudo access...
Ubuntu 14.04 / 15.04 / 18.04:
sudo apt-get install wkhtmltopdf
# or
sudo apt install wkhtmltopdf
Others
Look at the other answers.
If its ubuntu then go ahead with this, already tested.:--
first, installing dependencies
sudo aptitude install openssl build-essential xorg libssl-dev
for 64bits OS
wget http://wkhtmltopdf.googlecode.com/files/wkhtmltopdf-0.9.9-static-amd64.tar.bz2
tar xvjf wkhtmltopdf-0.9.9-static-amd64.tar.bz2
mv wkhtmltopdf-amd64 /usr/local/bin/wkhtmltopdf
chmod +x /usr/local/bin/wkhtmltopdf
for 32bits OS
wget http://wkhtmltopdf.googlecode.com/files/wkhtmltopdf-0.9.9-static-i386.tar.bz2
tar xvjf wkhtmltopdf-0.9.9-static-i386.tar.bz2
mv wkhtmltopdf-i386 /usr/local/bin/wkhtmltopdf
chmod +x /usr/local/bin/wkhtmltopdf
Debian 8 Jessie
This works
sudo apt-get install wkhtmltopdf
Chances are that without full access to this server (due to being a hosted account) you are going to have problems. I would go so far as to say that I think it is a fruitless endeavor--they have to lock servers down in hosted environments for good reason.
Call your hosting company and make the request to them to install it, but don't expect a good response--they typically won't install very custom items for single users unless there is a really good reason (bug fixes for example).
Lastly, depending on how familiar you are with server administration and what you are paying for server hosting now consider something like http://www.slicehost.com. $20 a month will get you a low grade web server (256 ram) and you can install anything you want. However, if you are running multiple sites or have heavy load the cost will go up as you need larger servers.
GL!
Latest update for CentOS:
sudo yum install -y libpng libjpeg openssl icu libX11 libXext libXrender xorg-x11-fonts-Type1 xorg-x11-fonts-75dpi
wget https://github.com/wkhtmltopdf/wkhtmltopdf/releases/download/0.12.4/wkhtmltox-0.12.4_linux-generic-amd64.tar.xz
tar -xvf wkhtmltox-0.12.4_linux-generic-amd64.tar
sudo mv wkhtmltox/bin/* /usr/local/bin/
check installation success: wkhtmltopdf -V
rm -rf wkhtmltox
rm -f wkhtmltox-0.12.4_linux-generic-amd64.tar
Place the wkhtmltopdf executable on the server and chmod it +x.
Create an executable shell script wrap.sh containing:
#!/bin/sh
export HOME="$PWD"
export LD_LIBRARY_PATH="$PWD/lib/"
exec $# 2>/dev/null
#exec $# 2>&1 # debug mode
Download needed shared objects for that architecture and place them an a folder named "lib":
lib/libfontconfig.so.1
lib/libfontconfig.so.1.3.0
lib/libfreetype.so.6
lib/libfreetype.so.6.3.18
lib/libX11.so.6 lib/libX11.so.6.2.0
lib/libXau.so.6 lib/libXau.so.6.0.0
lib/libxcb.so.1 lib/libxcb.so.1.0.0
lib/libxcb-xlib.so.0
lib/libxcb-xlib.so.0.0.0
lib/libXdmcp.so.6
lib/libXdmcp.so.6.0.0
lib/libXext.so.6 lib/libXext.so.6.4.0
(some of them are symlinks)
… and you're ready to go:
./wrap.sh ./wkhtmltopdf-amd64 --page-size A4 --disable-internal-links --disable-external-links "http://www.example.site/" out.pdf
If you experience font problems like squares for all the characters, define TrueType fonts explicitly:
#font-face {
font-family:Trebuchet MS;
font-style:normal;
font-weight:normal;
src:url("http://www.yourserver.tld/fonts/Trebuchet_MS.ttf");
format(TrueType);
}
List of stable versions wkhtmltopdf: http://wkhtmltopdf.org/downloads.html
Installing wkhtmltopdf on Debian 8.2 (jessie) x64:
sudo apt-get install xfonts-75dpi
sudo apt-get install xfonts-base
sudo wget http://download.gna.org/wkhtmltopdf/0.12/0.12.2.1/wkhtmltox-0.12.2.1_linux-jessie-amd64.deb
sudo dpkg -i wkhtmltox-0.12.2.1_linux-jessie-amd64.deb
Shared hosting no ssh or shell access?
Here is how i did it;
Visit https://wkhtmltopdf.org/downloads.html and download the appropriate stable release for Linux. For my case I chose 32-bit
which is wkhtmltox-0.12.4_linux-generic-i386.tar.xz
Unzip to a folder on your local drive.
Upload the folder to public_html (or whichever location fits your need) using an FTP program just like any other file(s)
Change the binary paths in snappy.php file to point the appropriate files in the folder you just uploaded.
Bingo! there you have it. You should be able to generate PDF files.
A few things have changed since the top answers were added. They used to work out for me, but not quite anymore, so I have been hacking around for a bit and came up with the following solution for Ubuntu 16.04. For Ubuntu 14.04, see the comment at the bottom of the answer. Apologies if this doesn't work for shared hosting, but it seems like this is the goto answer for wkhtmltopdf installation instructions in general.
# Install dependencies
apt-get install libfontconfig \
zlib1g \
libfreetype6 \
libxrender1 \
libxext6 \
libx11-6
# TEMPORARY FIX! SEE: https://github.com/wkhtmltopdf/wkhtmltopdf/issues/3001
apt-get install libssl1.0.0=1.0.2g-1ubuntu4.8
apt-get install libssl-dev=1.0.2g-1ubuntu4.8
# Download, extract and move binary in place
curl -L -o wkhtmltopdf.tar.xz https://github.com/wkhtmltopdf/wkhtmltopdf/releases/download/0.12.4/wkhtmltox-0.12.4_linux-generic-amd64.tar.xz
tar -xf wkhtmltopdf.tar.xz
mv wkhtmltox/bin/wkhtmltopdf /usr/local/bin/wkhtmltopdf
chmod +x /usr/local/bin/wkhtmltopdf
Test it out:
wkhtmltopdf http://www.google.com google.pdf
You should now have a file named google.pdf in the current working directory.
This approach downloads the binary from the website, meaning that you can use the latest version instead of relying on package managers to be updated.
Note that as of today, my solution includes a temporary fix to this bug. I realize that the solution is really not great, but hopefully it can be removed soon. Be sure to check the status of the linked GitHub issue to see if the fix is still necessary when you read this answer!
For Ubuntu 14.04, you will need to downgrade to a different version of libssl. You can find the versions here. Anyways, be sure to consider the implications of downgrading libssl before doing so on any production server.
I hope this helps someone!
After trying, below command work for me
cd ~
yum install -y xorg-x11-fonts-75dpi xorg-x11-fonts-Type1 openssl git-core fontconfig
wget https://downloads.wkhtmltopdf.org/0.12/0.12.4/wkhtmltox-0.12.4_linux-generic-amd64.tar.xz
tar xvf wkhtmltox-0.12.4_linux-generic-amd64.tar.xz
mv wkhtmltox/bin/wkhtmlto* /usr/bin
Version 12.5 of wkhtmltopdf only lists DEB files on their download page now. Being a mac user and not knowing much linux or what DEB files were I couldn't use the solutions posted.
This page helped me get past the knew twist of downloading a DEB file: http://www.g-loaded.eu/2008/01/28/how-to-extract-rpm-or-deb-packages/
Basically what I did was:
Downloaded from https://wkhtmltopdf.org/downloads.html
Unzipped the DEB file.
Unzipped data.tar.xz
Uploaded the binary in the unzipped 'usr' folder from step 3 (usr/local/bin/wkhtmltopdf)
Then I found out that the 'exec' function was disabled on my host. So make sure you can specifically run 'exec' if you're using PHP to run this. "Can I run the wkhtmltopdf binary" isn't specific enough. My fault.