There are set of log files that have a pattern xxxxxYYY where xxxx -> some text and YYY is a sequence number increasing sequentially by one and wrapping around. Only the last n number of files are available at a given time.
I would like to write a foolproof script that would make sure that all the log files are backed up in another server (via ssh/scp).
Can somebody please suggest a logic/code snippet(perl or shell) for it?
=> The script can run every few minutes to ensure bursts of traffic do not cause log files to miss getting backed up.
=> The roll over needs to be detected so that files are not overwritten in the destination server/directory.
-> I do not have super user either in source or destination boxes. The destiantion box does not have rsync installed and would take too long to get it installed.
-> Only one log file gets updated at a time.
I would look at having cron run an rsync --backup command.
Related
a few weeks ago my computer stopped working and I lost a lot of important data stored in a postgres DB (should have backed it up but, I didn't ** sigh ** ).
I was able to access the hard drive and extract the program files for postgres. How can I try and restore the DB?
Here is what I have tried so far:
download postgres (same version as the lost DB) on a separate computer and swap out the program files with the ones I am trying to recover. I am using a PC, so I stop service->swap out all the files-> restart service. Restarting the service is never successful.
Use PGAdmin -> creat new db (ensure path for binary files is correct) -> restore-> Here i get stuck figuring out what the correct files are
I am using Grails 2.5. We are using Camel. I have folder called GateIn. In this delay time is 3minutes. So Every 3minutes , it will look into the folder for file. If the file exists, it will start to process. If the file is processed within 3 minutes, file get deleted automatically. Suppose my file takes 10minutes,file is not deleted.Again and again, it process the same file. How to make file get deleted whether it is small or bulk file. I have used noop= true to stop reuse of file. But i want to delete the file too once it is preocessed. Please give me some suggestion for that.
You can check the file size using camel file language and decide what to do next.
Usually, in this kind of small interval want to process a large size of file, it will be better to have another process zone (physical directory), you have to move the file after immediately consuming it to that zone.
You can have a separate logic or camel route to process the file. After successful process, you can delete or do appropriate step according to your requirement. Hope it helps !!
I am using cfprint from ColdFusion to print multiple PDFs from a directory. The problem I am having is that when the files are spooled to the printer the size of the file dramatically increases and slows down everything. The file in the folder is 125K and when it is in the printer spool it increases up to 15.7MB. Here is the ColdFusion code:
<cfprint
source="[FILELOCATION]/[FILE].pdf"
color="yes"
printer="[printer name]">
The files will eventually print but it can take upwards of 15-20 minutes. Does anyone have any solutions for this issue? I have tried with both CF generated PDFs and ones that I have created from scratch. Thanks
Queue up two to five at a time. Pause to allow processing. Mark them as printed, move or delete them, move to the next batch...Time this out yourself to see how much time you need to allow. That way you don't compound a bunch of work for the server and create a bottleneck on your CF server.
If you are just doing this with one server consider using a secondary low priority server and run a developer edition fully paid for EULA compliant registered version of Coldfusion (or Railo) and dedicate that server for just printing so your other server can do useful things.
Edit
So the OP has a Coldfusion print bottleneck. In your server that does the printing (same as your CF server I assume?) and IF this is a windows server (not sure your server version), there is print queue folder. Provided you have access to this folder, you can do a few things. You can create a method for FTP-ing your files to this folder (or copy if it is the same server). The printer will queue up the job and off it goes. You can do some functions like check the print queue for file count. If the file count is greater zero, check back in 15 minutes. If the count is zero, copy over a few more files.
You be creating a scheduled task in your CFAdmin and automate. There is a getprinterInfo() so you can check if the printer is off line and do other things like check for another printer somewhere else if you need to reroute print jobs. You can also set up several print servers and attach printers to them and hit several print servers and check print queue folders.
The magic is endless, goal is to offset work to something other than your Coldfusion server.
So to recap:
Seperate concerns by not doing cfprint
Create escape routes to other priters if you can.
If you must use coldfusion then queue up a dedicated Coldfusion server for print management stuff.
Use getPrinterInfo() and dump out things to see what you can use, trap etc.
Ben forta has a tool that can check for several printers, consider incorperating this.
Next use cfftp (or cffile if you are on the same server) provided you have access and copy files to print queue folders, doing no cfprint at all.
Here is a link on print spool stuff (another link in the doc shows you how you can change the spool location).
When it is over you are going to be the coldfusion print master with escape routes and checks and everything.
I have been working on an algorithm in Python, and I was using Vim to edit this file. I opened it up, did a save, and it came up with an Error something like it occasionally does:
"WARNING: YOUR FILE CANNOT BE SAVED! ALL CHANGES WILL BE LOST! CANNOT WRITE THE FILE!"
As this happens occasionally, I did what I normally do, and I hit :q! to quit without writing any changes. No harm, no foul. When I looked at my file, everything had been erased! Everything!
I talked around the office, and it seems that the nfs mount was full, and so that was why I couldn't save anything. There was a huge script generating a ton of data, which caused the mount to be full temporarily. I believe the NFS mount is from NetApp. I found 2 files in my current directory.
One was last saved two days ago, and one was today. They are in the format of:
.nfs.xxxxxxxxxxx
When I try to attempt to open up this file, I see some of my code, here and there, splattered among unknown characters. Apparently, this must be a binary representation of the state of the file.
Is there any way to recover this file from this NFS mount? If there is a shortcut to recover this file in Emacs, I will switch to Emacs from vim!
So, I did find a way to recover the file. I found two ways, in fact. Since it was on a NetApp NFS mount, I was able to use the snapshots feature. When you are in a directory just do
ls .snapshot
And this will pull up any snapshots that your system administrators have set. For us, we have an hourly.0, hourly.1, and nightly.0, and nightly.1 backups. So, we can go back two days, and in the same day, we can go back one hour (the current hour, and the previous).
The other way was to rename the file to a vim swap file like this.
mv .nfs.xxx my_vim_file.cpp.swp
vim my_vim_file.cpp.swp
Then attempt to open it up in Vim, and it should ask you if you want to Recover the swap file, say yes, and it should be back!
Apparently your Netapp uses NFS to mount its volumes (as opposed to iSCSI, for example). Generally, each VM is stored on a unique volume (aka datastore) on the Netapp filer. To find out the volumes and snapshots, and then restore a snapshot, here are the commands to execute at the command line:
# list all volumes, snapshots are taken of volumes
vol status
# list the snapshots available for a particular volume
snap list <vol_name>
# restore a snapshot, nightly.1 for example
snap restore <vol_name> nightly.1
That's it. All that's left is to turn the VM back on and see if you've restored far back enough. If not, then do another "snap restore" but with an older snapshot.
Note that this procedure assumes your administrator didn't disable snapshots (Netapp has a snapshot schedule by default) and that the Netapp is licensed for snaprestore (use the "license" command to verify). This procedure can further be simplified if you have the Netapp OnCommand System Manager, which is a GUI for managing the Netapp. Reverting a snapshot in the GUI is simple:
Go to Storage > Volumes > click on a volume > click on Snapshot Copies (at the bottom)
Choose a snapshot and restore
Currently working on an application, that help you to take backup of the files in you machine at the server (hosted by the company itself), so that you can recover data after any hdd crash. I have implemented Single Instance feature, across the users.
Single Instance : A file uploaded already at the server, wouldn't be uploaded again. Whenever any other instance of the exact file uploaded will not be actual upload but some database changes and linked to the same previously uploaded file.
Issue arise when same file (that has not already been uploaded before) is uploaded simultaneously by more than one users, On Start file wouldn't be detected for an instance (as database is updated only after successful upload/backup). All are running, at once. What will be the best way to implement single instance in this way.
I am thinking when I let all the instance upload as it is. So more than one instance of the file will reside at the server. But whenever another backup of the same file will be taken afterwards, I will remove all the previous instances and link them up with the one. This will not let user double uploads and also less complex on the cost of some disc space that too for a while probably (till next upload of the same file will be done)
Thanks for your thoughts in advance.
Calculate the hash (signature) of the file before upload and store it in the DB.
Then - start uploading.
if a similar file will be mark for uploading during the upload of the first file (you will know b/c you already saved the hash) - you will hold the 2nd file upload, until the first one finish successfully, and then link, if the 1st on fails, you can go to the 2nd source and upload it.