Automating File Transfers - Multiple Directories / Lots of Files

One of the big disadvantages of FILEZILLA is its poor command line interface (CLI) support.  This makes it very unsuitable for use in an environment where you need to script automated file transfer routines

For example working on a recent project I've had to design a routine to automate the transfer of files (300,000 approx) from a local server across several remote servers.

Based on a defined local directory structure these files would be transferred in to specific remote folders on each remote server.

So I needed something that would allow me

1. Connect to a specific remote server and authenticate
2. Prevent interactive prompts i.e. asking user to confirm, retry or abort
3. Programatically select a local directory on the local server
4. Programatically select a local directory on the remote server
5. Transfer specific file types from the local server to that remote server
6. Close to the connection
7. Move to the next folder on the local server and repeat .....

While I could accomplish this with an enterprise class Message Broker by defining FileInput Nodes and FTP Nodes that seemed to be overkill and would be expensive.

A free and open source solution that offers an easy to use CLI / scripting language to accomplish this is WinSCP 

Simple create your script file e.g. myscript.scr and then use the /script command line option for WinSCP to read commands e.g.
winscp.com /script=myscript.scr

For example to connect to an FTP server www.myserver.com

# Set confirmation off during batch processing
option confirm off
# change username/password and host i.e. myserver
open ftp://username:password@www.myserver.com
# Then chnage in to remote directory
cd <remote directory>
# Then change in to local directory
lcd <local directory>
# Then transfer specific file types
put *.txt ./
# Then exit
close
....
Repeat as required

You can create a simple VB.NET or Java/SWING desktop style application to make it easier for non-technical users to access this type of feature.

Insert placeholders in to the script above for example
open ftp://$username:$password@$host

Then once the user keys in the values in to a nice to use form, simple do a find/replace within the template script file and save it to run.scr

Then perform a system call to WinSCP with the /script=run.scr option

Its simple, nothing fancy but it works and it automates a task and saves time.

Comments

  1. Auto FTP Manager is one of the best automated secure FTP client software. SFTP/SSH is supported. The clean interface is super easy to use. You can automate repetitive transfer tasks so your file transfers are handled quickly and efficiently and on schedule every time. It also supports FXP for superfast server to server transfers.

    ReplyDelete

Post a Comment

Popular posts from this blog

Basic Send Message to MQ with Java and IBM MQ JMS

Basic Receive Message to MQ with Java and IBM MQ JMS

Creating a simple Alert / Success Message with ASP.NET/VB using Bootstrap