Distance no object for big data in the cloud

Distance no object for big data in the cloud

Steve Jones from IBM Cloud – Aspera explains trends in high-speed transport of big data in the cloud and why it’s so important.

Innovative organisations are seizing the promise of cloud. A media company wanting to use the cloud to distribute massive amounts of entertainment content…. a national meteorological agency aiming to model, understand and predict weather patterns… a life sciences institute, with researchers across the globe, needing to share huge data sets or medical images in the cloud…

Whatever their sector, these organisations often deal with data sets measuring in tens of terabytes or even in petabytes. And therein lies the problem: How do they quickly and securely migrate files to the cloud to get big data in and out of the cloud service provider’s datacentre?

Traditional file-sharing tools such as FTP and HTTP were never designed for this kind of task. Performance seriously degrades over distance, as network latency kicks in. A few numbers show the problem isn’t going to go away:

  • Cisco’s annual Visual Network Index predicts that video will account for a whopping 82% of all Internet Protocol (IP) traffic by 2021.
  • 85% of managers with responsibility for data storage in healthcare and life sciences plan to increase cloud compute resources, but 72% think IT infrastructure will create research bottlenecks.
  • 95% of healthcare and life sciences organisations say their existing file collaboration tools don’t meet their needs.

So, what’s the answer?

It may come as a surprise in this day and age, but some big-data-in-the-cloud users are still ‘hand-carrying’ data by shipping hard disk drives to their cloud providers. In taking this route, they run all the risks of introducing time-lags into projects and suffering loss from theft or mishandling. What’s more, there is still the problem of moving the data within the cloud, between remote storage and across compute nodes.

To realise the promise of the cloud, these organisations need the means to transport large volumes of data securely, at high speed to, from and across cloud infrastructures.

Some, such as the hugely successful Netflix (approaching 104 million subscribers worldwide!), have already gained the capability to shorten file transfer times from many hours to mere minutes.

Steve Jones

IBM Cloud – Aspera



To do this, you don’t necessarily need to be using a TCP based transfer protocol like FTP, but can use alternative protocols based on UDP instead.

UDP allows you to move big datasets much more quickly than TCP, regardless of size, distance or network conditions. Unlike TCP, which only sends subsequent data packets once the previous packet is received, UDP continually sends data packets without waiting to see if they are received successfully. This is significantly faster, allowing businesses to speed up processes and operate more efficiently.

In traditional UDP transfers, there are some trade-offs. If a data packet doesn’t reach its destination it is ultimately lost. For example, in a live video stream the sound might distort slightly or a few frames may freeze. As a consequence, in certain circumstances (like video streaming), the advantages greatly outweigh this.

Where maintaining the integrity of the data is critical however, the best results are gained from a combination of both UDP and TCP.  UDP is used to transfer the data, whilst TCP is used to ensure that any packets lost are resent.  An agent at the receiving station is used to reconstruct the data after the transfer.

Some examples of effective use of this method include:

  • Banking, where vast amounts of data is captured and moved around each day.
  • Researchers needing to share large volumes of scientific and clinical research data.
  • The manufacturing industry, sharing large files and data sets to global development teams.

Find out more about Aspera’s combined UDP and TCP technology – FASP – on the Aspera vendor page. You can also download the Aspera White Paper ‘Taking Big Data to the Cloud’.

This is the latest in a series of guest blog posts from the leading vendors, highlighting how a file transfer solution can add value to your organisation.

Do you need to quickly and securely migrate files to the cloud? Our Needs Analysis Service will help scope your requirements and find the right solution for you.

Visit the Aspera page to download the White Paper ‘Taking Big Data to the Cloud’.

Resources Available For You

Do you need a File Transfer solution?

Questions regarding need for File Transfer 


Find out your File Transfer requirements!

“Needs Analysis Service for File Transfer”


Compare the software on the market!

“Managed File Transfer Comparison guide”


Webinar – Move Big Data up to 100 Times Faster

Webinar – Move Big Data up to 100 Times Faster


Event Type: Live Webinar
Event Date: Thursday 28th April 2016
Event Time: 10AM – 10.30AM


IBM Aspera Customers have experienced;
  • 20 x reduction in process times – a customer transfer was reduced from 10 hours to 26 minutes
  • 97% increase in network utilisation from 2% to 99%
  • 4x reduction in process turnaround – a media customer reduced their process turnaround from 6 to 1.5 days

So Join Pro2col and IBM for a short webinar to see how Aspera moves big data up to 100s of times faster.

This webinar will cover how IBM Aspera:
  • Globally moves data over standard wide area networks
  • Achieve multi-Gbps speeds over high performance global networks
  • Securely move and access big data – from ANY LOCATION, to ANYWHERE and with ANYONE
  • Provide enterprise-grade user access controls and encryption
  • Utilise ANY infrastructure combination – On Premises, AWS, Google, Azure, SoftLayer

FASP 3.6 Aspera Transfer Platform Now Available

FASP 3.6 Aspera Transfer Platform Now Available

Aspera have released version 3.6 of its Aspera Transfer Platform, a major new release that is built into the core of all the latest Aspera products and Aspera On Demand offerings. In this release, Aspera’s core products; Enterprise Server, Connect Server, Point-to-Point Client and Desktop Client – have all undergone a significant revamping with support for multi-tenancy and many new notable features and capabilities.

Aspera LogoThe most significant highlights include:


New Multi-Tenant Access Control and Content Security

  • Enables multiple Aspera applications to securely authenticate to transfer clusters and individual transfer nodes with no access to underlying storage credentials, supporting all local and cloud storage types.
  • New server side encryption at rest stores content securely at rest based on server defined / queried secrets, in addition to client side encryption at rest.

New Aspera On Demand Capabilities

  • New native cluster support for invoking transfers that use all nodes in a cluster and New node API for querying cluster transfer status through one cluster address.
  • New autoscale service for the Transfer Cluster Manager (ATCM) that automatically scales the number of Aspera Server nodes for client transfer demands up and down based on user-defined policies.
  • New super fast HVM images running on Amazon Web Services cloud infrastructure, supports single transfers sessions at up to 1.5 Gigabits per second throughput using the latest VM technology.
  • Many new Direct-to-Cloud storage advanced transfer capabilities supporting all major cloud storage providers including, IBM SoftLayer Swift, AWS S3 and GovCloud, Microsoft Azure Blob, Akamai Netstorage, Limelight Orchestrate Cloud Storage, Google Cloud Storage, HDFS, Ceph, and coming soon, CleverSafe.

New Aspera Watch (Asperawatchd) Service

  • A new ultra fast service which provides immediate change detection and snapshot service designed for speed, scale and distributed change watching.

New FASPStream binary and APIs

  • Server and client products include a new binary for sending and receiving ‘streams’ of data or growing files over wide area networks at the stream/file generation rate with nearly 0 delay at the receiver.

New ASCP4 Binary

  • New ascp4 binary has a new architecture for ultra high-speed transfers of very small files, able to achieve 1 million files per minute over Global WANs for 10 Gigabit per second sustained transfers of files of <=100KB over long distance.

New Aspera Sync (ASYNC) Features

  • Aspera Sync incorporates the Asperawatchd service eliminating bottlenecks of scanning on very large file systems (>20 million files) and scales with number of synchronising users by eliminating per-user file system access.

New Aspera Watchfolder (Beta Release)

  • This beta release of the new Aspera WatchFolder includes in all Linux and Windows core products and designed to meet the scaling and management requirements of modern workflows. This can be used in place or in addition to the Windows asperasync.exe or simplesync services on Linux.


For more information on this new Aspera release please contact an Aspera engineer on 0207 118 9640.

IBM Aspera Connect Browser Plug-in

IBM Aspera Connect Browser Plug-in

Aspera introduces the release of the IBM Aspera Connect Browser Plug-in. Aspera Connect 3.6.0 is a significant new release that includes a new native plug-in for Chrome browsers (allowing Connect to work in Chrome browsers in which NPAPI is disabled). In addition, Connect 3.6.0 includes a new JavaScript API. The new API supports all major browsers, including Chrome browsers in which NPAPI is disabled.


These changes are described in detail below:

Chrome Web browser updates

  • Prior to version 3.6.0, Connect required support for NPAPI plug-ins, which Chrome supported by default. However, Chrome version 42.0 now disables NPAPI plug-in support by default.
  • This release of Connect no longer uses NPAPI for Chrome, and so preserves compatibility with the new Chrome behaviour.
  • Note: If you want to use Chrome without upgrading Connect, you must perform the Chrome configuration workaround described in the Aspera Support Knowledgebase article here.

Connect’s new JavaScript API

  • In the new Connect API, many previously synchronous functions are now asynchronous, improving overall responsiveness and flexibility.
  • In order to use the new API, you must adjust your code accordingly. Note that there are certain parameters that, if in use in previous versions of the Connect API, you must update if you intend to use Connect 3.6.0.
  • For a complete listing of new, changed, and removed parameters in this release, see the information on Connect in the Aspera Developer Network here.


While Aspera recommends that all users upgrade to Connect 3.6.0, you must upgrade if:

  • You run Aspera web applications on the Chrome browser.
  • You plan to upgrade your Shares, Faspex, or Connect Server installation with the next release.

Please get in touch with the Pro2col team should you need any further information or links to install the new server.

Aspera Files SaaS Platform Cloud – Connects Multiple Storage Systems

Aspera Files SaaS Platform Cloud – Connects Multiple Storage Systems

Pay-As-You-Go Service is Expected to Appeal to Smaller Media Firms with Its Lower Cost of Entry

Aspera LogoAspera has introduced Aspera Files, a new software-as-a-service (SaaS) platform. Aspera parent company IBM said Aspera Files will allow companies to connect multiple storage platforms in order to exchange any size of file-based media or data across separate organizations. Content can be stored and accessed in storage systems in the cloud as well as on-site, with drag-and-drop capabilities for authenticated content sharing.

The new platform is built on Aspera’s FASP data-transfer technology, which is designed specifically to speed the transport of large data over the Internet by eliminating bottlenecks introduced in standard TCP-based file transfers.

aspera-munson-230x267Aspera co-founder and CEO Michelle Munson told StudioDaily that the new platform goes above and beyond what Aspera was previously able to provide, in part by reducing the barrier to entry for smaller facilities.
“This is our first time being able to offer pure software-as-a-service,” she said. “For a lot of media organisations, particularly small ones, it’s about being able to get up and running easily. This system is pay as you go. Customers run their own virtual machines on Amazon or Azure, but in this case it’s all put together by us, with exclusive sharing and interoperable management features.”

Companies can sign up for a branded web presence via Aspera Files, then create Workspaces for project-based file sharing. Shared content will automatically appear in the Workspaces of users who are authorised to access the files. Packages of content from connected storage locations can also be delivered to user or group Inboxes, where those users can browse files in a given package or download everything. Aspera said 4K media files can be transferred over long-haul global WANs to both on-site and cloud storage.

“We have built a permission layer into our core transfer platforms that is able to, on behalf of an application or an authenticated end user or group, implement permissions that support read and write, linking a file system across all types of file systems or storage,” Munson explained. “This is built into the Aspera server software layer, and there are a few huge technology leaps in this compared to the way we were doing it before. We now have direct transfer to and secure browsing of object storage by the platform.”

Aspera Files implements a distributed content access control system that securely authenticates user access through a delegated OAuth authorization and secure access key system, the company said. Client-side and server-side encryption is optional. And the system scales easily through the addition of new transfer server instances to meet demand. Automated clustering technology adds new nodes to handle increases in transfer loads, makes over-utilized nodes unavailable for new transfers, and spins down nodes when demand decreases.

“The classic cloud argument is starting to come of age,” Munson said. “Security and trust are built into it, it is not putting content at risk, and it is simplifying the experience for media professionals who are not dealing with IT struggles.”

Contact the Pro2col team to find out more about this product or to view a demo.

Aspera Delivers Major Advances in its Core Fasp High-Speed Transfer Platform

Aspera Delivers Major Advances in its Core Fasp High-Speed Transfer Platform

aspera_faspstreamAspera unveiled the next generation of its unique FASP software platform for moving Big Data and media at maximum speed and scale regardless of size, distance or network conditions.  Multiple breakthroughs bring new capabilities and advanced features, allowing enterprises of all sizes to meet variable data movement demands in real time with speed, control, security and ease, and all capabilities are designed to move data directly to cloud and on premise storage. Advances in the Core Transport

  • ascp4 is the next-generation Aspera file transfer binary introducing a new architecture for ultra high-speed transfer of small files in addition to large data sets. The architecture transfers up to one million files per minute for even the smallest file sizes (e.g. <10KB) and achieves > 5Gbps transfer rates for global RTT and packet loss conditions (200ms / 2%).
  • FASPStream transport expands Aspera’s FASP transport technology for “live” and “near live” data streaming. Aspera FASP is a patented transport protocol for achieving highly efficient bulk data transfer over IP networks, independent of distance and quality (round-trip latency and packet loss). FASPstream expands this capability to provide a fully reliable streaming protocol for in-order data delivery over Internet WANs with excellent quality and negligible start up delay.

Advances in Aspera Direct-to-Cloud Transfer and Autoscaling Aspera Direct-to-Cloud Storage transfer capability moves even the largest (4K) media formats from source directly to object storage destination with native I/O for all major cloud storage providers: IBM SoftLayer Swift, AWS S3, Microsoft Azure Blob, Akamai NetStorage, Limelight Orchestrate Cloud Storage, Google Cloud Storage, and HDFS (beta).  Version 3.6 brings many new capabilities to all Aspera On Demand products such as:

  • Server-side encryption at-rest in addition to client-side, as well as in-transit encryption
  • Automatic determination of the cloud storage part size allowing for the largest file sizes to be sent without specific configuration.
  • New clustered transfers enable 10 Gbps+ transfers in/out/between clouds.
  • A new Aspera Transfer Cluster Manager (ATCM) with Autoscale technology providing elastic scaling, a multi-tenant access key system for security and transfer reporting, and automatic high availability. The ATCM is a cloud-infrastructure independent service created by Aspera that allows for dynamic, real-time scale out of transfer capacity with automatic start/stop of transfer server instances, automatic balancing of client requests across available instances and configurable service levels to manage maximum transfer load per instance, available idle instances for “burst” and automatic decommissioning of unused instances.

Advances in Transfer Automation, Synchronization and Management

  • A new Aspera WatchFolder Service is specifically designed to power high-volume automated file and directory transfers with advanced features for media workflows and content distribution.

Built on asperawatchd, a new file system notification service, designed for speed, scale and distributed change watching, watch directories can now cover huge files systems and large numbers of watch folders. Collections of files and folders can now be grouped into a single “drop” and transferred to remote nodes as a single logical unit and controlling which files arrive last. “Growing”(in progress) files are fully supported. A RESTful API enables programmatic control for customized and automated processing.

  • Aspera Sync 3.6 also integrates asperawatchd technology to capture changes on any local or shared storage client host (CIFS, NFS, etc.) and aggregate all changes in real time in a single snapshot for speed on very large file systems. File attribute changes (Windows ACL, Unix ownership) are syncd, even when content is not updated, and LZ compression achieves high performance on low capacity networks. Bidirectional synchronization now works on cloud storage as well as block storage.
  • Aspera Console 3.0 for Centralized Management is built on a new architecture with the ability to precisely regulate reporting load as the number of transfer nodes increases, dramatically improving scalability, robustness, and timely transfer status for large deployments. Many new settings and options have been added, including advanced email notification triggers on any transfer attribute such as source and destination path, Aspera Shares and Faspex user attributes, and transfer directories; new managed node clusters allow managed nodes to be assigned to cluster groups with shared storage to support automatic load balancing and failover of transfers; and an advanced search of transfer history searches by transfer names, IDs, contacts, paths, and status.