Moving files, fast. Really fast!

FAST-FILES

Most of the companies that contact us do so looking for guidance on automating their file transfer. Automation is primarily for B2B and predominately involves using SFTP, a simple but typical example might be that an insurance company receives files from a bank and the files need to be moved to another internal system for processing. Of course there are usually a wide range of other complex processes that accompany it, but you get the idea.

Making files fast BIIn recent times however many of our conversation have gone along the lines of, “How can we move REALLY large files?” With the continuing explosion in data creation companies are having problems moving the large sets of data, many that are gigabytes in size.  All variations of FTP can deliver large volumes of data, but when you add in to the mix that it is in a far flung location and you’ve got a dodgy internet connection, suffering from latency and packet loss then FTP becomes pretty much ineffectual. Latency on the line, measured in round trip times (RTT), is a measure of how long it takes for a packet of data to get from point A to B and back again. We won’t go into the reasons in this blog, but suffice to say the longer the RTT the less efficient FTP becomes as can be seen in the graph here.

However all is not lost, there is an resolution to this data transfer problem. A select few vendors have built proprietary protocols based upon the open standard UDP to move data faster. A lot faster! Their protocols generally work the same way in that they maximise the utilisation of the bandwidth available to them by flooding the connection with data. Of course, controls are built in to ensure other network traffic doesn’t suffer. This approach can increase the speed by up to 1,000 times depending upon the network conditions and bandwidth available.

These UDP based solutions are now reaching a level of maturity enabling the software to be used in many scenarios, for example:

  • Disaster recovery and business continuity
  • Content distribution and collection, e.g., software or source code updates, or CDN scenarios
  • Continuous sync – near real time syncing for ‘active-active’ style HA
  • Supports master slave basic replication, but also more complex bi-directional sync and mesh scenarios
  • Person to person distribution of digital assets
  • Collaboration and exchange for geographically-distributed teams
  • File based review, approval and quality assurance workflows

Not only are files getting bigger but the environments within which they are implemented are evolving, the technologies with which they need to interface are changing too. Our expert team analyse complex requirements daily, providing companies of all shapes and sizes with solutions to their tricky file transfer conundrums. If your business needs to share large or sensitive data either in an automated of manual process, we can help. Contact one of our file transfer experts on +44 1202 433415 or get in touch via the web site.