I want to download all .mp4 files from a "Index of" page using wget.
The target directory is http://ia600409.us.archive.org/27/items/MIT18.01JF07/
I've tried some commands (which I copied and pasted from websites), but they don't seem to work.
e.g.:
wget -r -l0 -nd --no-parent -A .mp4 -R ".*"
http://ia600409.us.archive.org/27/items/MIT18.01JF07/
Btw, I think the directory is Apache.
wget -e robots=off -r -nd -np -A mp4 http://ia600409.us.archive.org/27/items/MIT18.01JF07
Related
How to write scp user#url:/tmp/foo ./ with options instead of one-liner ?
I tried this, but unsuccessfully:
scp -l user url /tmp/foo ./
Send file
scp -o user=myuser foo.txt myserver:/my/folder
Get file
scp -o user=myuser myserver:/my/folder/foo.txt ./
im trying to use wget for FTP download ( auth )
this is the command i used to download the file bat.bat in appdata dir ...
i got my file inside new folder created by wget named ( website.com )
wget -r --ftp-user="user" --ftp-password="pass" ftp://website.com/bat.bat -P %appdata%
when i checked appdata directory i found my file here :
C:\Users\ev\AppData\Roaming\website.com\bat.bat
i dont need it to create new dir i need it here :
C:\Users\ev\AppData\Roaming\bat.bat
Try the -nH parameter:
wget -r -nH --ftp-user="user" --ftp-password="pass" ftp://website.com/bat.bat -P %appdata%
From wget --help:
Directories:
-nd, --no-directories don't create directories.
-x, --force-directories force creation of directories.
-nH, --no-host-directories don't create host directories.
--protocol-directories use protocol name in directories.
-P, --directory-prefix=PREFIX save files to PREFIX/...
--cut-dirs=NUMBER ignore NUMBER remote directory components.
I am trying to download a dataset from a website but I can't download the whole folder .. I have to download each file separately which will need a lot of time. I am wondering if there is anyway to download the whole folder at a time??
The website link: http://www.physionet.org/pn4/eegmmidb/
Use wget with the -r switch to turn on mirroring.
This command will do what you want:
wget --no-parent -r http://www.physionet.org/pn4/eegmmidb/
It'll produce a mirror copy of everything from that directory on down.
These two for cycles run in bash should do it:
for S in S{001..109}; do
mkdir ${S}
cd ${S}
for R in R{01..14}; do
file="http://www.physionet.org/pn4/eegmmidb/${S}/${S}${R}.edf"
wget "$file"
wget "${file}.event"
done
cd ..
done
I've SSHed into one of my VPS and I'm trying to install java on it. Not really sure how to go about downloading but I am trying to use wget to download and install JDK7 from the Oracle website.
this file in particular: http://download.oracle.com/otn-pub/java/jdk/7u40-b43/jdk-7u40-linux-x64.rpm
To download the file, it requires authentication and I don't know how to do that through wget.
SOmeone please help
# rpm
wget --no-cookies \
--no-check-certificate \
--header "Cookie: oraclelicense=accept-securebackup-cookie" \
"http://download.oracle.com/otn-pub/java/jdk/7u55-b13/jdk-7u55-linux-x64.rpm" \
-O jdk-7-linux-x64.rpm
# ubuntu
wget --no-cookies \
--no-check-certificate \
--header "Cookie: oraclelicense=accept-securebackup-cookie" \
"http://download.oracle.com/otn-pub/java/jdk/7u55-b13/jdk-7u55-linux-x64.tar.gz" \
-O jdk-7-linux-x64.tar.gz
# then
tar -xzvf jdk-7-linux-x64.tar.gz
refer: https://gist.github.com/scottvrosenthal/11187116
To download the file, it requires authentication and I don't know how
to do that through wget.
To be more accurate it requires license agreement.
If you have SSH access to the server you can copy file using scp command
scp /<path to the file>/jdk-7u40-linux-x64.rpm user-name#server-name:/tmp/
Then you go to the server by ssh and find it in /tmp/ folder
I am using Mac OSX 10.6 and doing web development on it. I know a small amount about writing shell scripts, but I am not really versed in them as of yet.
What I would like to do is to write a shell script that will simply ask for a local site alias and the document directory and it will then append the new alias onto hosts with something like "127.0.0.1 mysite.local" on a new line at the bottom of etc/hosts.
Then the script would append Apache's httpd-vhosts.conf file with something like this:
<VirtualHost *:80>
DocumentRoot "/Repositories/myproject/mysite.com/trunk/htdocs"
ServerName mysite.local
ServerAlias mysite.localhost
</VirtualHost>
Then it would finally run the command to restart my Apache server. Now I know the terminal command to restart Apache, that is simple enough. I also know how to read in the site name and path from the user running the script. Such as below:
#!/bin/bash
read -p "New local site name: " site
read -p "Site path (ex:/Repositories/myproject/mysite.com/trunk/htdocs): " sitepath
What I don't know how to do is to append text to a file from terminal.
Any thoughts or helpful ideas?
Thanks,
Patrick
Untested, but it should work:
#!/bin/bash
read -p "New local site name: " SITE
read -p "Site path (ex:/Repositories/myproject/mysite.com/trunk/htdocs): " SITEPATH
#/etc/hosts
cp /etc/hosts /etc/hosts.original
echo -e "127.0.0.1\t${SITE}.local" >> /etc/hosts
#httpd-vhosts.conf
VHOSTSFILE="/etc/apache2/httpd-vhosts.conf"
cp $VHOSTSFILE ${VHOSTSFILE}.original
echo "<VirtualHost *:80>" >> $VHOSTSFILE
echo -e "\tDocumentRoot \"${SITEPATH}\"" >> $VHOSTSFILE
echo -e "\tServerName ${SITE}.local" >> $VHOSTSFILE
echo -e "\tServerAlias ${SITE}.localhost" >> $VHOSTSFILE
echo '</VirtualHost>' >> $VHOSTSFILE
#restart apache
>> redirects the output to the given file, appending the contents to the file. I’m also using -e to allow \t to be expanded to a tab character.
Note that you need to run this script with sudo. I've also included commands to backup the original files before modifying them, just in case.
I made some tweaks and did some extra stuff in the above answer, because this didn't work for me but it helped me to come up with another solution. This answer is only for Mac users.
First go in your /etc/apache2/httpd.conf and uncomment virtual host reference, which is this line Include /private/etc/apache2/extra/httpd-vhosts.conf
Now create a bash file I did that in my user’s home named as imran, you need to replace it with your username.
I placed it inside /Users/imran named as create_new_site.sh. I gave it executeable permissions, so it can be easily executed using chmod +x create_new_site.sh
Code for the script is as below:
#!/bin/bash
read -p "New local site name (prefix to .local): " SITE
SITEPATH=$SITE
sudo chmod -R a+w /Users/imran/Sites/web
SITEPATH="/Users/imran/Sites/web/${SITEPATH}"
mkdir -p $SITEPATH
#/etc/hosts
cp /etc/hosts /etc/hosts.original
echo -e "127.0.0.1\t${SITE}.local" >> /etc/hosts
#httpd-vhosts.conf
VHOSTSFILE="/etc/apache2/extra/httpd-vhosts.conf"
cp $VHOSTSFILE ${VHOSTSFILE}.original
echo "<VirtualHost *:80>" >> $VHOSTSFILE
echo -e "\tDocumentRoot \"${SITEPATH}\"" >> $VHOSTSFILE
echo -e "\tServerName ${SITE}.local" >> $VHOSTSFILE
echo '</VirtualHost>' >> $VHOSTSFILE
echo '<?php phpinfo();' > "${SITEPATH}/phpinfo.php"
sudo chmod -R a+w $SITEPATH
#restart apache
sudo apachectl restart
echo "All done! visit, let's visit http://${SITE}.local/phpinfo.php"
Once that is done you can start creating new sites using sudo ./create_new_site.sh. Remember that you need to be in your home directory, which you can go via cd ~ command. Now let's suppose you created a site with name test. You should be able to visit http://test.local/phpinfo.php to see your vhost working.