My data stored in a remote server and update every day like 16062021.csv , 15062021.csv etc. I would like to copy file from remote server to my local pc only with a specific range via shell .
This snippet will copy all the data to my computer but I want only n weeks data not the whole n year.
How can I specify a date range here and only copy 16042021.csv to 16062021.csv?
ROBOCOPY "\\103.241.144.137\files" "C:\Users\Documents\my_path" /mir
If using Robocopy from Windows, it has options:
MAXimum file AGE - exclude files older than n days/date.
Also /MINAGE:n :
MINimum file AGE - exclude files newer than n days/date.
This relies on backup files that have timestamps similar to names.
Options are easy to see, so if this is not a solution, please explain.
Related
I have a Pentaho job which has '10' ids as input in first transformation. In the next job I have a SQL query which needs to loop through each id from input.
So, I am using copy rows to results in first transformation and get rows from result in next job and selected execute for every input row in job properties of second job to loop through every 'id' from the first transformation. In the next steps data from the SQL query in second job is stored as files into local machine dynamically in different folders based on a specific field from the telesis query.
Next, I need to send these files in different folders of the local machine into sftp server through 'sftp put' step.
Now I want to track the logs with columns as:
1.number of files loaded into local machine for each id from input.
2.number of files loaded to sftp successfully from local machine to check whether all files loaded into local machine are sent successfully to sftp or not.
3.If a file is not sent to sftp for any reason, I need the name of the file which failed to load into sftp.
Thanks in advance..
We have large number of batch jobs which generate ascii log files. We browse these logs using commands/tools such as GREP or VI/VIM editors.
Simple text searches are fine. But if we have to search these log files for a particular string and compare date/times when the string was generated - the task become unwieldy. It means finding the file which has the string and manually noting down modification date/time of the file.
Are there any tools which can ingest an entire directory structure on linux and store it into an Oracle table with following attributes:
1. Full name of directory
2. Filename
3. File Modification Date/time
4. Line number
5. Line Text
I created a backup cmd file with this code
EXPDP system/system EXCLUDE=statistics DIRECTORY=bkp_dir DUMPFILE=FULLDB.DMP LOGFILE=FULLDB.log FULL=Y
it works good, but, when I run the backup again, it finds that the file exists
and terminate the process. it will not run unless I delete the previous file or rename it. I want to add something to the dumpfile and logfile name that creates a daily difference between them, something like the system date, or a copy number or what else.
The option REUSE_DUMPFILES specifies whether to overwrite a preexisting dump file.
Normally, Data Pump Export will return an error if you specify a dump
file name that already exists. The REUSE_DUMPFILES parameter allows
you to override that behavior and reuse a dump file name.
If you wish to dump separate file names for each day, you may use a variable using date command in Unix/Linux environment.
DUMPFILE=FULLDB_$(date '+%Y-%m-%d').DMP
Similar techniques are available in Windows, which you may explore if you're running expdp in Windows environment.
I am new to using SQL, so please bear with me.
I need to import several hundred csv files into PostgreSQL. My web search has only indicated how to import many csv files into one table. However, most csv files have different column types (all have one line headers). Is it possible to somehow run a loop, and have each csv imported to a table with the same name as the csv? Creating each table manually and specifying columns is not an option. I know that COPY will not work as the table needs to already by specified.
Perhaps this is not feasible in PostgreSQL? I would like to accomplish this in pgAdmin III or the PSQL console, but I am open to other ideas (using something like R to change the csv to a format more easily entered into PostgreSQL?).
I am using PostgreSQL on a Windows 7 computer. It was requested that I use PostgreSQL, thus the focus of the question.
The desired result is a database full of tables, that I will then join with a spreadsheet that includes specific site data. Thanks!
Use pgfutter.
The general syntax looks like this:
pgfutter csv
In order to run this on all csv files in a directory from Windows Command Prompt, navigate to the desired directory and enter:
for %f in (*.csv) do pgfutter csv %f
Note that the path for the downloaded program must be added to the list of accepted paths for Environmental Variables.
EDIT:
Here is the command line code for Linux users
Run it as
pgfutter *.csv
Or if that won't do
find -iname '*.csv' -exec pgfutter csv {} \;
In the terminal use nano to make a file to loop through moving csv files under my directory to postgres DB
>nano run_pgfutter.sh
The content of run_pgfutter.sh:
#! /bin/bash
for i in /mypath/*.csv
do
./pgfutter csv ${i}
done
Then make the file executable:
chmod u+x run_pgfutter.sh
I am backuping our servers with rsnapshot on daily bases.
Everything works fine and I have my data in daily.0, daily.1 ... daily.6
Now, I am using rsync to backup the backups from one to another NAS server.
The problem is that with rsync, on my 2nd backup server (NAS) I have the same data structure with all the daily from 0 to 6.
The big proble is that the NAS is recognizing the data from each daily as sing le physical files, so I have my data multiplied by 7 at the end.
My question is: Is there any possibility to use rsync and have on my 2nd server single files only, without all the daily.0 to daily.6, so I can avoid that the linux system thinks that I have 6 times more data that I have realy.
Rsync should only take files that have been modified, so you only have to backup the old data once.
But... (I'm not sure your OS or environment), you can pass only the most recent file to rsync like this
latest=`ls -t|head -1` ; rsync $latest backup_location
(my source)