I have a Centos server.
In SSH, I downloaded multiple mp4 files in a txt with this command:
wget -i urls.txt
Now I have a list of mp4 files in a directory. The problem is those mp4 files are named:
65464.mp4?md5=sdokj7SRuM-OKatFmdCJJg&expires=1570102642
45421.mp4?md5=sdokj56SRuM-OKatFmdCJJg&expires=157010277842
etc...
How do I remove ? and everything after in all files in this directory ( For example ?md5=sdokj7SRuM-OKatFmdCJJg&expires=1570102642 ) so it would change files name to:
65464.mp4
45421.mp4
etc...
Thank you.
The solution will depend upon what shell your CentOS system is running or has available. If you're running bash, you can do something as simple as:
for x in * ; do
mv $x ${x%%\?*}
done
This uses bash string variable manipulation techniques.
Another possible solution, still dependent upon shell features:
for x in * ; do
mv $x $(echo $x | sed "s/\?.*//")
done
Related
Files are being written to a directory using the COPY query:
Copy (SELECT * FROM animals) To '/var/lib/postgresql/data/backups/2020-01-01/animals.sql' With CSV DELIMITER ',';
However if the directory 2020-01-01 does not exist, we get the error
could not open file "/var/lib/postgresql/data/backups/2020-01-01/animals.sql" for writing: No such file or directory
PostgeSQL server is running inside a Docker container with the volume mapping /mnt/backups:/var/lib/postgresql/data/backups
The Copy query is being sent from a Node.js app outside of the Docker container.
The mapped host directory /mnt/backups was created by Docker Compose and is owned by root, so the Node.js app sending the COPY query is unable to create the missing directories due to insufficient permissions.
The backup file is meant to be transferred out of the Docker container to the Docker host.
Question: Is it possible to use an SQL query to ask PostgreSQL 11.2 to create a directory if it does not exist? If not, how will you recommend the directory creation be done?
Using Node.js 12.14.1 on Ubuntu 18.04 host. Using PostgreSQL 11.2 inside container, Docker 19.03.5
An easy way to solve it is to create the file directly into the client machine. Using STDOUT from COPY you can let the query output be redirected to the client standard output, which you can catch and save in a file. For instance, using psql in the client machine:
$ psql -U your_user -d your_db -c "COPY (SELECT * FROM animals) TO STDOUT WITH CSV DELIMITER ','" > file.csv
Creating an output directoy in case it does not exist:
$ mkdir -p /mnt/backups/2020-01/ && psql -U your_user -d your_db -c "COPY (SELECT * FROM animals) TO STDOUT WITH CSV DELIMITER ','" > /mnt/backups/2020-01/file.csv
On a side note: try to avoid exporting files into the database server. Although it is possible, I consider it a bad practice. Doing so you will either write a file into the postgres system directories or give the postgres user permission to write somewhere else, and it is something you shouldn't be comfortable with. Export data directly to the client either using COPY as I mentioned or follow the advice from #Schwern. Good luck!
Postgres has its own backup and restore utilities which are likely to be a better choice than rolling your own.
When used with one of the archive file formats and combined with pg_restore, pg_dump provides a flexible archival and transfer mechanism. pg_dump can be used to backup an entire database, then pg_restore can be used to examine the archive and/or select which parts of the database are to be restored. The most flexible output file formats are the “custom” format (-Fc) and the “directory” format (-Fd). They allow for selection and reordering of all archived items, support parallel restoration, and are compressed by default. The “directory” format is the only format that supports parallel dumps.
A simple backup rotation script might look like this:
#!/bin/sh
table='animals'
url='postgres://username#host:port/database_name'
date=`date -Idate`
file="/path/to/your/backups/$date/$table.sql"
mkdir -p `dirname $file`
pg_dump $url -w -Fc --table=$table -f $file
To avoid hard coding the database password, -w means it will not prompt for a password and instead look for a password file. Or you can use any of many Postgres authentication options.
I want to copy files remotely in a script from windows machine to Linux machine.
On the Linux machine I run the below command
scp user#remotehost:\D\mySrcCode\somefile.cpp .
I am getting an error
scp: DmySrcCodesomefile.cpp: No such file or directory
The file somefile.cpp is located at D:\mySrcCode on windows side.
Any ideas on what I am missing ?
You probably should quote or backslash the backslashes in the path.
If your interactive shell is GNU bash, read its §3.1.2 quoting chapter.
You could try:
scp user#remotehost:\\D\\mySrcCode\\somefile.cpp .
Consider also using other (more appropriate) tools, like rsync or git.
You might also use exec(3) from your C program to run /usr/bin/ssh, or look into libssh.
You could change your login shell (see chsh(1) and /etc/shells so shells(5)) to more user friendly alternatives such as zsh or fish. They could give you some warning (depending on how they are configured or used) or some autocompletion (with the tabkey).
PS. Your problem is not ssh specific. You might replace scp with echo to understand it more.
docker-machine has an scp command, but docker-cloud doesn't seem to have any way to transfer a file from my local machine to the cloud container or vice-versa.
I'm submitting an answer below that I've finally figured out (in hopes that it will help someone), but I'd love to hear better answers if there are any!
(I realize docker-cloud is going away, but perhaps this will be helpful for other cloud platforms as well)
To transfer a file from your local machine to a docker-cloud instance that is running linux with the tee command available:
docker-cloud container exec id12345 tee filename.ext < file_to_copy.ext > /dev/null
(you'll want to redirect output to /dev/null as shown unless you want the entire contents of the file to be echoed to the terminal... twice)
To transfer a file to your local machine, is somewhat easier:
docker-cloud container exec id12345 cat file_to_copy.ext > filename.ext
Note: I'm not sure this works for binary files, and it can even cause issues with linefeed characters in text files, based on terminal settings, etc. - but it's the best answer I've got short of using an external service like https://transfer.sh
I want to take backup of my website which is hosted on godaddy.
I used pscp command from my windows dos and try to download whole public_html folder.
my command is :
pscp -r user#host:public_html/ d:\sites\;
Files are downloading properly and folders also. But the issue is public_html and other subfolders has two folder like "./" and "../". Due to these two folders my copy is getting failed and I am getting
"security violation: remote host attempted to write to " a '.' or '..' path!"error.
Hope any one can help for this.
Note : I have only ssh access and have to download it from ssh commands itself.
Appending a star to the source should fix it, e.g.
pscp -r user#host:public_html/* d:\sites\;
Also you can do same thing by not adding '/' at the end of your source path.
For eg.
pscp -r user#host:public_html d:\sites
Above command will create public_html directory if not exists at your destination (i.e. d:\sites).
Simply we can say using above command we can make a as it is clone of public_html at d:\sites.
One important thing: You need to define the port number over here "-P 22".
pscp -r -P 22 user#host:public_html/* D:\sites
In my case, it works when I use port number 22 with the above script.
I've been advised to remove the return command from my bashrc file in order to allow Ruby Version Manager to function properly. Do I simply delete the return command, or do I replace it with some other command? I am hesitant to mess with my System-wide shell without some proper direction. But I would really like to get RVM working as it is a time saver.
My bashrc is located in the etc directory and looks like this:
# System-wide .bashrc file for interactive bash(1) shells.
if [ -z "$PS1" ]; then
return
fi
PS1='\h:\W \u\$ '
# Make bash check its window size after a process completes
shopt -s checkwinsize
if [[ -s /Users/justinz/.rvm/scripts/rvm ]] ; then source /Users/justinz/.rvm/scripts/rvm ; fi
The last line, is an insert, described in the RVM installation.
I wouldn't. That return is probably there for a good reason. It obviously doesn't want to execute anything after that if the PS1 variable is empty.
I would just move the inserted line up above the if statement.
In addition, if that's actually in the system-wide bashrc file, you should be using something like:
${HOME}/.rvm/scripts/rvm
rather than:
/Users/justinz/.rvm/scripts/rvm
I'm sure Bob and Alice don't want to run your startup script.
If it's actually your bashrc file (in /Users/justinz), you can ignore that last snippet above.
The last line uses a file in a specific user's home directory, and as such should not be in the system-wide bashrc, since only root and that user will have access to that file. Best to place it in that user's ~/.bashrc instead.