Blog Archives - Managed File Transfer Solutions | Secure File Transfer Software | UK

8 Common Reasons Why Managed File Transfer Fails

8 Common Reasons Why Managed File Transfer Fails

I wrote an article a while back about how to monitor a Managed File Transfer system, but I didn’t really discuss any of the many things that can go wrong and ruin your day.  Here are some non-product specific thoughts and ideas on the automation aspect of MFT; the scheduled or triggered transfer of data between networks.

Triggering

In general, the first place where something goes wrong is the actual triggering of an action.  Depending on the system that you are working with, this may be an event, job, task or similar.  Actions are generally triggered by an event matching a rule – for a scheduled action this is a time event corresponding to a specific time.  More commonly however, the event will be the arrival of a file and the rule will be a filename or folder match.  Common errors in this area include conflicting rules in multiple actions and files arriving just after a scheduled transfer.

Sources and Targets

Broadly speaking, the most common role of automation software is to move files from one place to another – unfortunately, this is where things most often go awry.  Here are some common problem that occur.

Firewall incorrectly configured – affecting both inbound and outbound traffic. Caused by either some or none of the required ports being opened, or even incorrect NATing of the automation components IP address

Whitelisting and Blacklisting – Unfortunately as the automation administrator, you don’t necessarily have a view of this.  It is however worthwhile being able to validate that your IP address hasn’t changed unexpectedly.

Password, Key or Certificate Expiry – There is always a play-off between security and operability, but invariably in more secure environments non-expiring certificates, keys or passwords are disallowed.  Be aware that many secure transfer servers will not confirm that this is the cause of the problem, so it may not be immediately obvious.  You should also note that repeated failures may result in IP blacklisting or a locked account.

Connectivity – The internet is a great place to get lost in and we all expect to have occasional issues reaching certain destinations.  The same can often hold true inside your own network however.  Remember that you may sometimes need to flush a DNS cache in order to make the right connections (especially true after a post maintenance DNS change).  For those connections that just can’t be made (whether internal or external), you will need to put a plan in place to reconnect when the target server becomes available again.

Space – Maybe not the final frontier, but often the last straw.  Many platforms place a hard limit on the maximum permissible file size; some network administrators have a hard limit on how big a file they allow through their network (especially during peak times).  And of course, inevitably there will always be a disk running out of space somewhere.

Programs and Scripts

Most MFT systems will provide you with the opportunity to execute a program or script at some point during the transfer process.  This allows you perform some basic transformation or processing of a file during transit.  Invariably the MFT software will not by default provide all of the debug information that you need, so be prepared to write in extra logging, or redirect STDERR to STDOUT and capture it somewhere.

Notifications

Often, MFT software cannot/will not report back failures to send email notifications as they potentially may be stopped somewhere outside of the MFT system.  To counter this it is common to send yourself a daily status email, the arrival of which demonstrates that at least some emails are leaving the MFT system.  If the MFT software does not provide the opportunity to test emails, try logging on to the server and running telnet on port 25 to test SMTP (Windows); on Linux systems you may use sendmail instead.

Summary

These are of course just a handful of the kind of problems that MFT administrators have to face on an almost daily basis; hopefully though, just by thinking a little about these may help to avoid some issues, or even give some pointers to resolution.

Resources Available For You

The Expert Guide to Managed File Transfer

Includes definitions, requirements assessment, product comparison, building your business case and much more.

Managed File Transfer Needs Analysis

200 essential questions to consider before implementing your chosen Managed File Transfer solution.

Managed File Transfer Comparison Guide

A full feature comparison of Managed File Transfer solutions from the eight leading vendors in the industry.

Are Your FTP Servers Still Under Manufacturers’ Support?

Are Your FTP Servers Still Under Manufacturers’ Support?

We’re now a week on from the first reports of WannaCry infecting computers across the globe. The initial media  furore has subsided, the world hasn’t ended and IT can get back on with their jobs of keeping the organisation running!

Many software vendors jumped on the bandwagon using WannaCry as an opportunity to position the need for their software, I’m sure you received a tonne of emails. We thought about it but resisted, instead we thought that a concise view of whether your FTP server(s) are supported by the manufacturer, the current version and what operating systems they supported would be of more use.

Naturally if you need any help upgrading any of these servers our technical team are here to help. However it would be advisable checking in with the pre-sales team to ensure that the process runs smoothly; some vendors will require a completely new licence key to go to the latest versions.

If you’ve got any other FTP servers you’d like us to add to the list, please get in touch.

Resources Available For You

The Expert Guide to Managed File Transfer

Includes definitions, requirements assessment, product comparison, building your business case and much more.

Managed File Transfer Needs Analysis

200 essential questions to consider before implementing your chosen Managed File Transfer solution.

Managed File Transfer Comparison Guide

A full feature comparison of Managed File Transfer solutions from the eight leading vendors in the industry.

Password Security in Managed File Transfer

Password Security in Managed File Transfer

Last week was “World Password Day”, a day designed to get people thinking about password security and hopefully change their passwords. I was surprised to see an article from Sophos that the average person has 19 passwords to remember and almost a third struggle with strong passwords.  With the raft of work systems, private emails, social media, online shopping and banking passwords I thought it would be many more. I did a quick tally of my online passwords and worked out I have in excess of 30 passwords, although most of the private account passwords are variations on 4 main passwords.  I worked for one very large organisation who insisted passwords were changed every month but suggested that you simply add the month digit to the end of your password, negating the password security almost entirely.

The full article from Sophos can be found here.

Having strong passwords and authentication methods for file transfer accounts is very important. There are several approaches for user authentication that are supported by most Managed File Transfer (MFT) solutions.

These are

  • Application Controlled
  • External source (AD / LDAP / Other source)
  • Advanced Authentication using RADIUS or a One Time Password system
  • Private key authentication

With application controlled authentication, the MFT solution will control the length, complexity, password history and password expiry using internal systems. Usually users will be prompted to change their passwords either by getting an email, or when they login.

This works well but, for users inside the organisation, passwords can drift out of sync and this can lead to increased issues as users are asked to remember more and more passwords to access different systems. In this case, we usually recommend that the MFT solution uses the internal Active Directory or LDAP source. This allows the user to use the same credentials that they login to their computers with. Responsibility for changing the password then resides with the AD/LDAP system and the MFT solution will not normally track the passwords. When a user presents their credentials to login to the MFT solution, the system will pass the username and password to the AD/LDAP source for verification. If the AD/LDAP system confirms the credentials are correct the MFT solution lets the user in. As there is usually no caching of credentials, if a user changes their password on the AD/LDAP system then that password is reflected instantly in the MFT system.

Increasing the security of using AD/LDAP to authenticate user credentials, RADIUS solutions using time limited one-time password tokens or even SMS messages can be integrated to provide an extra level of security.

In RADIUS authentication, the user or device sends a request to the MFT system to gain access to a particular network resource, then the system passes a RADIUS Access Request message to the RADIUS server, requesting authorization to grant access via the RADIUS protocol. RADIUS servers vary, but most can look up client information in text files, LDAP servers, or databases. The RADIUS server can respond with an Access Reject, Access Challenge, or Access Accept. If the RADIUS server responds with an Access Challenge, additional information is requested from the user or device, such as a secondary password. Access Accept and Access Reject allow or reject the user access respectively.

Using AD/LDAP authentication or RADIUS authentication works well for users who are logging into the system interactively using either a web interface or file transfer client such as FileZilla, but do not work well for accounts which are used as a part of file transfer scripts.

The most popular method of securing these is to use “private key” or “key pair” authentication. With this the account does not use a defined password, but rather the MFT solution encrypts a token and sends that as a challenge to the client. This token is decrypted using the private half of the key at the client end and sent back unencrypted. If the tokens match the MFT solution accepts the user as verified and allows the account access. In this way any scripts which need to access the MFT solution do not need to have passwords encoded into them in raw text. Key pair authentication works with SSH keys for SFTP and SSL Certificates for FTPS and HTTPS connections.

With many more password breaches coming not from brute force attacks but from compromised authentication databases, experts are now advocating not making passwords longer or more complex but to implement Two Factor Authentication (2FA). This can be achieved using a combination of password and Private Key authentication or RADIUS in your MFT solution and works well for users and scripts.

Now maybe a good time to review your MFT password policies and maybe time I change some of my passwords too!!!

Resources Available For You

The Expert Guide to Managed File Transfer

Includes definitions, requirements assessment, product comparison, building your business case and much more.

Managed File Transfer Needs Analysis

200 essential questions to consider before implementing your chosen Managed File Transfer solution.

Managed File Transfer Comparison Guide

A full feature comparison of Managed File Transfer solutions from the eight leading vendors in the industry.

File Transfer Enables Formula 1 Teams To Go Even Faster

File Transfer Enables Formula 1 Teams To Go Even Faster

The world of Formula 1 (F1) racing is one of high speeds, high stakes and leading edge technology. Drivers like Lewis Hamilton or Sebastian Vettel, are just the public face of a much larger team working together to build a car to go faster than their competitors. With stakes so high, racing teams are looking for ways in which they can gain a competitive edge.

And so data plays a key part in F1. Each car has hundreds of sensors monitoring every aspect of the car, generating data on fuel load, tyre temperature, brake performance and much, much more. During a race this data is typically streamed back to race HQ enabling in-race, real-time decisions to be made about race strategy.

Managed File Transfer in Formula 1

However, the sensors also create a large amount of data during pre-season testing, in simulation environments, during free practice before each race and during qualification, all of which needs analysing to gain the split second advantages that make the difference between first and second place.

The data produced is valuable, very valuable. The performance of the team has a clear, measurable impact on the prize money received but what’s not quite so easy to quantify is the effect on the brand. The prize money available to the racing teams on its own is astronomical.   Last season nearly $965m of prize money was split between the top 10 teams.   Ferrari won the prize money race, with $192m, even though they finished second in the constructors championship due to a special agreement with F1.

F1 2016 Prize Money

Typically, racing teams spend much of the year in countries around the world where Internet connectivity isn’t always brilliant. Coupled with the distance back to their HQ and factories, any data transfers suffer with long round trip times (RTT), high packet loss and latency on the connection. This combination of factors results in poor data throughput, which is where Managed File Transfer comes in.

Pro2col was approached by a Formula 1 team to help them solve the problems they were having with file transfers. They needed a system that was easy for the teams travelling the globe to send large data sets from their laptops back to their HQ in the UK. It needed to be able to cope with the challenging network environments that they would encounter before and during the race season, and of course, security was of paramount importance.

 

Conducting A Thorough Needs Analysis

Pro2col conducted a thorough needs analysis with the Formula 1 team to understand their exact requirements, which allowed us to then review the available technologies in the marketplace, scoring them against the defined requirements.

Whilst a variety of products would have ‘done the job’, the devil was in the detail of not whether the software would do it but how. Eventually, two technologies were recommended for evaluation. Pro2col’s technical team provided assistance in setting up the software and establishing the success criteria for the proof of concept, which resulted in both technologies performing as expected.

Working with the Formula 1 team, we established that one of the technologies had a much more competitively per-user, pricing structure with a clear growth path as they looked to expand it into other areas of the business, for example when sharing data with customers and suppliers.

Now seven years since the original implementation of the software, the F1 team continues to perform well on the track and grow the footprint of its Managed File Transfer solution, adding further licences when the need arises.

Resources Available For You

The Expert Guide to Managed File Transfer

Includes definitions, requirements assessment, product comparison, building your business case and much more.

Managed File Transfer Needs Analysis

200 essential questions to consider before implementing your chosen Managed File Transfer solution.

Managed File Transfer Comparison Guide

A full feature comparison of Managed File Transfer solutions from the eight leading vendors in the industry.

Backing up your Cisco Unified Communications Manager through SFTP

Backing up your Cisco Unified Communications Manager through SFTP

The Cisco Unified Communications Manager (CUCM) is in use at many organisations to integrate data, voice and video applications.  It’s a nice product which provides a good balance between security and functionality.

As is often the case however, this sort of product very quickly becomes a critical piece of the infrastructure and consequently needs to be treated as such.  It’s therefore important to ensure that the configuration is routinely backed up in a secure fashion in order to recover your system should the need arise.

CUCM allows you to backup the configuration to a location on your network; because there will be credentials contained within the backup, it requires that you transfer the backup using a secure mechanism – SFTP.

Any file transfer server that provides the SFTP protocol is fine to use – some file transfer vendors even publish simple guides on how to configure for their specific software, however the steps to successful implementation are straightforward.

First, create an account on the SFTP server that you will use to receive the backup and set up a folder for it.  Even though CUCM allows the use of SFTP, it does not  permit connection using an SSH key – you must create a user that has the ability to connect using just a password instead.  If your SFTP server has the ability to automatically forward on files to another location, you may wish to set this up at this time.

Next go to CUCM and log into the Disaster Recovery System.  From here, select Backup, then Backup Device.  This is where you provide the details of your SFTP server.  Click on “Add New” and provide a friendly name for your SFTP server.  Beneath this, there is an area marked “Select Destination” – here you can enter the SFTP server details, path and credentials.  You can also select how many backups you want to keep in the SFTP server – handy if your SFTP server lacks automation capabilities.

Once you have done this, you can schedule the backup.  Go to Backup, the Scheduler and click “Add New” to create a new schedule.  As you might expect, you can now add the frequency that you want to send the backup to the SFTP server, including the day of the week and time of day.  Finally, save the schedule and click on “Enable Schedule”.

Et Voila!  Your CUCM configuration is now being securely backed up to your SFTP server.

These SFTP servers are supported and recommended by Cisco:

 

Resources Available For You

The Expert Guide to Managed File Transfer

Includes definitions, requirements assessment, product comparison, building your business case and much more.

Managed File Transfer Needs Analysis

200 essential questions to consider before implementing your chosen Managed File Transfer solution.

Managed File Transfer Comparison Guide

A full feature comparison of Managed File Transfer solutions from the eight leading vendors in the industry.

Six Managed File Transfer Experts Give Their Top File Transfer Automation Tips

Six Managed File Transfer Experts Give Their Top File Transfer Automation Tips

One of the biggest driving factors behind businesses implementing Managed File Transfer (MFT) is to ‘improve productivity’ and one of the biggest ways in which MFT can improve productivity is through File Transfer Automation.

So we decided to ask six file transfer experts from Cleo, Coviant Software, Globalscape, Ipswitch, JSCAPE and South River Technologies for their ‘Top three file automation tips.’ Unsurprisingly our experts had diverse views on what they believed to be important about file transfer automation, but we didn’t quite expect the variety of answers we received.

Joe Dupree, VP Marketing, CleoJoe Dupree, VP Marketing, Cleo

“Manual workflow functions hold businesses hostage. MFT automation can make your organization faster and more nimble by:

Delivering at a greater scale: There is a growing need for file transfer solutions that address all types of data management and integration use cases. Automation eases deployment and operation. Even better, solutions that are agnostic to all data formats and transfer protocols provide more versatility. But in today’s business that’s not enough. Automated MFT needs to massively scale as more digitized business workflows require rapid movement of more data to parallel the movement of their business processes, team members, goods, and services.

Achieving high availability: Customers expect full uptime, but how do you get there? Can your solution automate failover, and can you implement it without a small army of consultants?

Onboarding faster: How long does it take to onboard one new trading partner? What if you need to add lots of new connections? Ditch the manual configuring with a solution that comes pre-loaded with 900 preconfigured connections to the world’s largest trading hubs. Choose a proven solution that streamlines operations by enabling faster onboarding.

Don’t get caught doing things manually while revenue awaits and productivity drains. An advanced and automated MFT solution enables greater operational efficiency.”

Pam Reid, CEO, Coviant Software

Pam Reid, CEO, Coviant Software

“When a file transfer automation application is deployed, business users may have the false expectation that file transfers will always be successful. File transfer automation does greatly reduce the number of file transfer errors, but it cannot eliminate them altogether. File transfer errors occur for many reasons – network outages, locked files, a new password or encryption key that was not updated in the file transfer automation application and many other issues.

So, my three top tips on how to live up to high business user expectations is to look for a solution that:

Reduces overall file transfer errors. When transient errors occur, such as a network outage or a locked file, jobs should automatically wait and retry the connection before failing the job.

Minimizes the time to diagnose and resolve errors. Look for a solution that sends all of the diagnostic information directly to an IT support email account, which enables IT support to identify and attempt to resolve the issue right away.

Gives IT support time to resolve errors before business users are notified.  Assume a weekly payroll file is supposed to be ready for pick-up at 2pm on Thursday and must be delivered by 5pm. If the file transfer job runs at 2pm and does not find the file, the job should notify only IT support that the file is not ready – which gives IT support several hours to resolve the problem before business users are notified of a failure.”

Matt Goulet, Senior VP, Sales & Marketing, Globalscape

Matt Goulet, CEO, Globalscape

“Automation is an effective way to improve file transfer efficiency. Manual, intermittent batch processes are an unreliable method of data delivery because of limitations such as lack of staff, turnover, and human error. Below are just a few ways you can streamline the file transfer process.

Program Workflows: Create programmatic workflows that can be used to trigger events and actions based on specific conditions. From being aware of multiple invalid logins to processing and scanning all incoming files, being able to program workflows saves time and minimizes errors.

Streamline Business Processes: Automated workflow tools that are used with managed file transfer solutions can help optimize business processes, even when sophisticated and complex workflows are needed. MFT automation tools help administrators perform complex tasks and remove the possibility of human error.

Process Files with Added Security Measures: Antivirus scanners and Data Loss Prevention tools can permit or prevent file transfers based on your policies and help keep your network free of infected files, and can help comply with regulatory standards by preventing personally identifiable information from being transferred.

If you’re ready to simplify your file transfer process, automate with an MFT solution.”

Kevin Conkin, VP of Product Marketing, Ipswitch

“Do you find yourself spending too much time on manual tasks related to file transfer such as tracking lost files, reworking custom scripts, or creating audit trails? Automated file transfer can make your job a lot easier to do, while improving the IT team’s positive impact on the business. Here are three ways automating file transfer helps IT teams:

IT productivity improves. Many file transfers are initiated on a recurring basis. IT teams can get bogged down confirming transfers to meet service level agreements. The automation that comes with a Managed File Transfer (MFT) solution promptly pushes data to the right person at the right time. This means that the IT team doesn’t have to think twice and can remain focused on other tasks.

IT makes users happy. Automated, managed file transfer will help you find a new way to work with your end users—to give them an easy-to-use solution that integrates seamlessly with what they’re already using.

IT becomes compliant. The automation that comes through Managed File Transfer can identify an audit trail, and at a moment’s notice, provide a real-time status of the transfer. As a result, MFT helps the organization become more compliant, especially when it is integrated with security controls such as encryption and data loss prevention technology.”

Van Glass, CEO, JSCAPE

Van Glass, CEO, JSCAPE

Simplify: Many legacy file transfer implementations are home grown solutions that have evolved into an ever growing number of disparate and undocumented scripts and processes. These scripts are often written in arcane programming languages and are difficult to maintain. This can be particularly troublesome in cases when the original authors leave the organization and this knowledge is lost. When automating file transfers it is important to try and centralize all file transfers under a single application that has a holistic view of all file transfer processes and does not require specialized programming knowledge.

Failures: It’s inevitable that some automated file transfers are going to fail. This can happen for a variety of reasons including invalid credentials, network connectivity issues or hardware failures. Regardless of the reason, it’s important that you be able to automatically respond to these failures in the most efficient way possible. Network connectivity issues are common but often short-lived, so rather than give up immediately, it is common practice to retry failed transfers with a certain wait period between each attempt. For example, you may configure your file transfers to retry up to 3 times with a wait period of 60 seconds between each retry. This gives the target server or network time to recover from the failure. Additionally, provided the protocol used supports it, retries should resume file transfers from the last byte successfully transferred instead of attempting to transfer the entire file in order to reduce bandwidth consumption and complete the transfer more quickly. This is especially important in very large file transfers of several GB or more. The ability to notify administrators via email or other means in the event of a hard failure is also key.

Track Results: Automating your file transfers is just one of the first steps in implementing a Managed File Transfer solution. Tracking the results of file transfers is equally important in that by doing so you gain improved visibility into your file transfers. This information will help you not only in troubleshooting failed file transfers but also in better understanding file transfer trends which can prove useful when scaling up hardware or network resources. The results of your file transfers should at a minimum be able to answer the following questions: Who initiated the file transfer? What is the path and size of the file transferred? Was the file transfer successful? When was the file transfer initiated and if successful when was it completed?”

Michael Ryan, CEO, South River Technologies

“For a Managed File Transfer implementation to be successful and truly improve productivity, it must consider both automated transfers and user initiated transfers.  Both aspects should be balanced, while also considering security implications.

Easy User Access.  Make sure that your Managed File Transfer solution makes it easy for users to access and collaborate on files.  If there are additional security steps that users have to perform, they may work around the security measures.  Security should be invisible to users, who need to focus on accomplishing their work.

Integration capability.  It’s important to consider solutions that will easily integrate with both existing corporate applications, as well as desktop user applications.  This extends your investment in existing technologies and reduces the training requirements for your MFT implementation.

Provide optimal performance.  Load balancing and automated failover are key requirements for performance and high availability, but also consider the server architecture.  Native 64-bit applications take full advantage of the resources of each individual server, which makes a load-balanced implementation even more effective in achieving high throughput and performance.”

In conclusion, we can see that our experts have differing opinions on what are important considerations for file transfer automation. All of them are very valid and I’d suggest that you take into account as many as possible when considering a Managed File Transfer solution.

Of course, there are many solutions in the marketplace, all with their slight differences in features, benefits and total cost of ownership. If you’re considering implementing Managed File Transfer software to provide file transfer automation, or reviewing what you already have in place, then we’d be happy to help. You can speak to one of our friendly MFT experts by calling 0333 123 1240 or get in touch here.

Resources Available For You

The Expert Guide to Managed File Transfer

Includes definitions, requirements assessment, product comparison, building your business case and much more.

Managed File Transfer Needs Analysis

200 essential questions to consider before implementing your chosen Managed File Transfer solution.

Managed File Transfer Comparison Guide

A full feature comparison of Managed File Transfer solutions from the eight leading vendors in the industry.