8 reasons to why a file transfer may have failed
I wrote an article a while back about how to monitor a Managed File Transfer system, but I didn’t really discuss any of the many things that can go wrong and ruin your day. Here are some non-product specific thoughts and ideas on the automation aspect of MFT; the scheduled or triggered transfer of data between networks.
In general, the first place where something goes wrong is the actual triggering of an action. Depending on the system that you are working with, this may be an event, job, task or similar. Actions are generally triggered by an event matching a rule – for a scheduled action this is a time event corresponding to a specific time. More commonly however, the event will be the arrival of a file and the rule will be a filename or folder match. Common errors in this area include conflicting rules in multiple actions and files arriving just after a scheduled transfer.
Sources and Targets
Broadly speaking, the most common role of automation software is to move files from one place to another – unfortunately, this is where things most often go awry. Here are some common problem that occur.
Firewall incorrectly configured – affecting both inbound and outbound traffic. Caused by either some or none of the required ports being opened, or even incorrect NATing of the automation components IP address
Whitelisting and Blacklisting – Unfortunately as the automation administrator, you don’t necessarily have a view of this. It is however worthwhile being able to validate that your IP address hasn’t changed unexpectedly.
Password, Key or Certificate Expiry – There is always a play-off between security and operability, but invariably in more secure environments non-expiring certificates, keys or passwords are disallowed. Be aware that many secure transfer servers will not confirm that this is the cause of the problem, so it may not be immediately obvious. You should also note that repeated failures may result in IP blacklisting or a locked account.
Connectivity – The internet is a great place to get lost in and we all expect to have occasional issues reaching certain destinations. The same can often hold true inside your own network however. Remember that you may sometimes need to flush a DNS cache in order to make the right connections (especially true after a post maintenance DNS change). For those connections that just can’t be made (whether internal or external), you will need to put a plan in place to reconnect when the target server becomes available again.
Space – Maybe not the final frontier, but often the last straw. Many platforms place a hard limit on the maximum permissible file size; some network administrators have a hard limit on how big a file they allow through their network (especially during peak times). And of course, inevitably there will always be a disk running out of space somewhere.
Programs and Scripts
Most MFT systems will provide you with the opportunity to execute a program or script at some point during the transfer process. This allows you perform some basic transformation or processing of a file during transit. Invariably the MFT software will not by default provide all of the debug information that you need, so be prepared to write in extra logging, or redirect STDERR to STDOUT and capture it somewhere.
Often, MFT software cannot/will not report back failures to send email notifications as they potentially may be stopped somewhere outside of the MFT system. To counter this it is common to send yourself a daily status email, the arrival of which demonstrates that at least some emails are leaving the MFT system. If the MFT software does not provide the opportunity to test emails, try logging on to the server and running telnet on port 25 to test SMTP (Windows); on Linux systems you may use sendmail instead.
These are of course just a handful of the kind of problems that MFT administrators have to face on an almost daily basis; hopefully though, just by thinking a little about these may help to avoid some issues, or even give some pointers to resolution.