So, as part of my daily jobs, I have to transfer a one file from our customers server to our internal server and any responses back.
Each customer effectively has one file up and one file down each day.
I have an SFTP server here that I can use and is already used manually for a few sites.
I'm looking to automate as many sites as possible using batch files on a scheduled task.
Initially, I'm looking at automating the internal side of the process.
We simple have a requests folder that needs to import from the SFTP (then delete the original on the SFTP) and a response folder which needs to copy to a 'sent' folder and then export to the SFTP (also, deleteing the original)
On the SFTP server I have a "to site" and "from site" folder. Each file is site specific followed by a variable. So SiteNameImport.<variable> and SiteNameExport.<variable>
EDIT:
I'm asking this as I'm a novice at scripting and basically have no idea what to do.
I've tried reading the automation guide on WinSCP website but a lot of it means nothing to me.
Filezilla doesn't support automation, You're better off with WinSCP. They have some scripting examples here as well as any other information you'll need to build the scripts functionality. You'll just need to add the specifics (Like deleting sent files and so on) CuteFTP is also another solution you can script with but I believe you have to pay for a licence. I suggest VBscript, Examples can be found Here for vbscript.
Related
WE have a call recordings everyday that are posted in FTP. As a backup i wanted to copy those recordings from FTP in to local drive on daily basis.Is it possible to copy entire folder as a whole and paste the folder in my local and rename it ?
thanks in advance
BIDS in 2008 has an FTP Task, but I never use it. There are also several third party extensions for SFTP and the like, but I don't use those.
[Edit: Since the above links are to CodePlex, which is shutting down, I'll provide more detail. The first link is to a project titled "SSIS SFTP Task Control Flow Component" and the second is to a project titled "SSIS Extensions - SFTP Task, PGP Task, Zip Task". As of June 2017 the projects do not appear to have an obvious rehosting.]
I tend to use WinSCP with a script file. It's certainly the least Microsoft-y approach, but, IMX, it's more foolproof. This has the advantage of working for FTP, FTPS, and SFTP. It has the disadvantage that you're storing the password in plain text. You can store it in the registry with WinSCP's session manager, but I believe that's a per-user registry which makes for lots of possible fun when you have the SQL Agent using an SSIS Proxy account.
Create your script file with the commands you want to use. Be sure to specify option batch abort and option confirm off. You also might want to specify option failonnomatch, but I haven't tested that one yet myself since I'm still on a slightly older version that doesn't support that option.
Mine tend to look like this:
# Set batch settings
option batch abort
option confirm off
# Connect
open sftp://user:password#sftp.server.com -hostkey="ssh-rsa 2048 00:11:22:33:44:55:66:77:88:99:aa:bb:cc:dd:ee:ff"
# Change remote and local directories
lcd "M:\SSIS\eSchoolPlus to Clever\Output"
# Transfer files
put -nopreservetime -nopermissions -transfer=ascii file1.csv
put -nopreservetime -nopermissions -transfer=ascii file2.csv
put -nopreservetime -nopermissions -transfer=ascii file3.csv
put -nopreservetime -nopermissions -transfer=ascii file4.csv
put -nopreservetime -nopermissions -transfer=ascii file5.csv
close
exit
The options on put are due to how the remote server works for the script I happened to grab. get works about the same.
Then you use an Execute Process Task. The executable is C:\Program Files (x86)\WinSCP\WinSCP.exe and the Arguments I use are:
-console -script="X:\Path\To\Script\WinSCPScript.txt" -xmllog="X:\Path\To\Script\WinSCP-!S-!Y-!M-!D-!T.log" -xmlgroups`
That creates a daily log of just the WinSCP information. The XML logs are more readable and useful than the older log format, IMO, which contains a lot of mostly debugging information.
You can also go the route of using the WinSCP .NET library with an Execute Script Task, but that's obviously a lot more effort. They do have a HOWTO for that, however.
Yes - you would use a File System Task and set the Operation to Copy Directory (before Microsoft changed it with Win 95, folders were called directories). You can change the name in the destination connection where you choose the destination folder. You can use an expression to add a date if you want.
https://msdn.microsoft.com/en-us/ms140185.aspx
Here's some good step-by-step instructions if you need them since Microsoft gives vague descriptions with no examples:
http://oops-solution.blogspot.com/2011/11/file-operation-copy-move-rename-and.htmlhttp://oops-solution.blogspot.com/2011/11/file-operation-copy-move-rename-and.html
I am looking to have users upload files to an FTP server using a "pre-configured" FTP client. By that I mean; the FTP client's connection settings have already been set in the client they have downloaded. Ideally users should be able just to drag and drop the files into a simple window and the file uploads.
I have found two applications which allow me to do this;
"FTP Maker" (softhing.com/ftpmaker.html)
This allows you to configure the FTP connection details and add a logo. You then hit a button which generates the "uploader application". This can then be distributed to users where they have to configure nothing. While this works, it doesn't have as many features as...
"FTP Uploader Creator" (devzerog.com)
Same as above, except the application can zip before uploading, can resume uploads and can also send an email after upload has completed. These are handy features I wish FTP Maker had. The issue with this application is it's developer seems to have gone out of business and only the thirty-day trial is available...
Another application is "FTPcreator" (ftpcreator.com). Unfortunately this is a little outside my price range.
I am also aware of options such as dropbox, ftpbox etc.
Do you know of any super-lite FTP clients which I could pre-configure before sending out to users? Ideally it should have the features of "FTP Uploader Creator" above. I believe this sort of tool might be used in IT to allow users to send files directly to an FTP server.
I know I could do this through a webpage. However, the files will be particularly large and well over the limits of my hosting apache upload file size limit.
I have spent literally hours looking for alternative! Any help would be greatly appreciated!
I was trying to work out the best method of synchronising an update of files via vb.net. I want to automatically update some files in a few folders from an FTP site to a local drive. This would ideally be the same folder structure, but on a specified local folder. I have looked at the FTP methods in .net and am able to transfer individual files successfully, but wondered if there was an elegant method to collecting the entire folder from the FTP and it's contents to be re-created locally. I have established the reading of an XML via linq from the ftp to establish an update single and decide whether the local machine is up-to-date or not, it's just the transfer of the folders that i'm stuck on as it's different to using a IO method.
Can any gurus confirm that this is the best way of approaching this task?
cheers CS
Your approach looks fine to me. You will have to transfer the files/folders individually if you are using the base class library.
There may be 3rd party libraries available which will do this on one shot.
Does anyone know of a script or program that can be used for backing up multiple websites?
Ideally, I would like the have it setup on a server where the backups will be stored.
I would like to be able to add the website login info, and it connects and creates a zip file or similar that it would then be sent back to the remote server to be saved as a backup etc...
But it would also need to be able to be set up as a cron so it backed up everyday at least?
I can find PC to Server backups that a similar, but no server to server remote backup scripts etc...
It would be heavily used, and needs to be a gui so the less techy can use it too?
Does anyone know of anything similar to what we need?
HTTP-Track website mirroring utility.
Wget and scripts
RSync and FTP login (or SFTP for security)
Git can be used for backup and has security features and networking ability.
7Zip can be called from the command line to create a zip file.
In any case you will need to implement either secure FTP (SSH secured) OR a password-secured upload form. If you feel clever you might use WebDAV.
Here's what I would do:
Put a backup generator script on each website (outputting a ZIP)
Protect its access with a .htpasswd file
On the backupserver, make a cron script download all the backups and store them
I would like to make a complete backup of my whole joomla 1.5 based site from time to time. How would this ideally be done? Are there any common pitfalls? Not that I only have ftp access to the hosting server. Is there a step by step tutorial somewhere? I am using latest Joomgallery and Kunena 1.0.9 (Legacy mode).
Maybe there is a good way to automate this?
There's two parts of the backup you have to worry about, the database and the files.
The first part is the database. It can be backed up using something like phpMyAdmin. If you don't have this available on your server already, it's not too hard to upload and get it going yourself. From there, you can just Export the entire database to a gzip file.
The second part is the code and uploaded files. The code base shouldn't change too often, so you could probably just make one backup of this. There's a number of ways. The simplest is to just download the entire folder via FTP, though if you're Linux, I'm sure someone will know a single command line to get all the changed files (rsync?).
The database is the main thing you have to worry about though: everything else should be able to be rebuilt just by reinstalling.
I think this: http://www.joomlapack.net/ is what you need. I use it myself and it works like a charm. Both for backups and for moving my Joomla installations from developer sites and to the real site.
get an FTP synchronisation tool and keep an up-to-date copy of your site locally. Then you could run the batch script
mysqldump -hhost -uuser -p%1 schema > C:\backup.sql
to create a backup of your mysql tables at various points in time.
edit
you would have to have MySQL Server installed on your local machine and path to its bin directory in you PATH, in order to run the mysqldump command without much hassle. -p%1 would take the command-line provided password, as you wouldn't want to store passwords in your batch script.
If you only have FTP access you are in a bit of a problem, as beside all files you'll also have to backup the database. Without accessing the database, a full-backup won't do you any good.
Whatever backup strategy you choose - be sure it can handle UTF-8 correctly. Joomla 1.5 stores all content with UTF-8, even when the database charset is set on 'iso-5589-1' - so when the backup solution is detecting the database charset, some characters like € or é will result in "strange" ¬ / é - not really what you'll want.
I absolutely endorse using Joomlapack - it works great. The optional remote tools allow you to initiate the backup from a Windows desktop machine - it performs the backup and downloads it. The remote has a scheduler, and you can also set it off to backup and download a list of sites.
Joomlapack also provides a file "kickstart.php" which you copy to your empty server account along with the backup, which automates the restore procedure. You do have to create an empty database with PHPMyAdmin or similar, and you are given the opportunity to supply the database parameters (host, database, username, password) during the process.
One pitfall I did run into with this though is that some common components can have absolute URLs in their configuration - e.g. SOBI2, Virtuemart. It's then just a matter of finding the appropriate configuration file, editing it and re-uploading it.
Another problem was one archive file (either ZIP or their JPA format) got a filename with a "?" character in it (from a Linux server) and this caused a bit of a problem trying to install it locally on a Windows WAMP stack - the extract process on the ZIP file failed, and it stopped the process completing cleanly.
I suggest using automatic backup service by http://www.everlive.net
Update:
Ok, here is some more information. EverLive.net is a website where you can create a free account. Enter your website details and you are ready to take your backups withe just one click. Restore is also possible in the same way.
Further you can use automatic backup option to take automatic backups at defined intervals. Other than that, you can use the website health check service to inform you if your website is not available.