0333 123 1240 info@pro2colgroup.com

Distance no object for big data in the cloud

Distance no object for big data in the cloud

Steve Jones from IBM Cloud – Aspera explains trends in high-speed transport of big data in the cloud and why it’s so important.

Innovative organisations are seizing the promise of cloud. A media company wanting to use the cloud to distribute massive amounts of entertainment content…. a national meteorological agency aiming to model, understand and predict weather patterns… a life sciences institute, with researchers across the globe, needing to share huge data sets or medical images in the cloud…

Whatever their sector, these organisations often deal with data sets measuring in tens of terabytes or even in petabytes. And therein lies the problem: How do they quickly and securely migrate files to the cloud to get big data in and out of the cloud service provider’s datacentre?

Traditional file-sharing tools such as FTP and HTTP were never designed for this kind of task. Performance seriously degrades over distance, as network latency kicks in. A few numbers show the problem isn’t going to go away:

  • Cisco’s annual Visual Network Index predicts that video will account for a whopping 82% of all Internet Protocol (IP) traffic by 2021.
  • 85% of managers with responsibility for data storage in healthcare and life sciences plan to increase cloud compute resources, but 72% think IT infrastructure will create research bottlenecks.
  • 95% of healthcare and life sciences organisations say their existing file collaboration tools don’t meet their needs.

So, what’s the answer?

It may come as a surprise in this day and age, but some big-data-in-the-cloud users are still ‘hand-carrying’ data by shipping hard disk drives to their cloud providers. In taking this route, they run all the risks of introducing time-lags into projects and suffering loss from theft or mishandling. What’s more, there is still the problem of moving the data within the cloud, between remote storage and across compute nodes.

To realise the promise of the cloud, these organisations need the means to transport large volumes of data securely, at high speed to, from and across cloud infrastructures.

Some, such as the hugely successful Netflix (approaching 104 million subscribers worldwide!), have already gained the capability to shorten file transfer times from many hours to mere minutes. Steve Jones

IBM Cloud – Aspera

 

UDP and TCP

To do this, you don’t necessarily need to be using a TCP based transfer protocol like FTP, but can use alternative protocols based on UDP instead.

UDP allows you to move big datasets much more quickly than TCP, regardless of size, distance or network conditions. Unlike TCP, which only sends subsequent data packets once the previous packet is received, UDP continually sends data packets without waiting to see if they are received successfully. This is significantly faster, allowing businesses to speed up processes and operate more efficiently.

In traditional UDP transfers, there are some trade-offs. If a data packet doesn’t reach its destination it is ultimately lost. For example, in a live video stream the sound might distort slightly or a few frames may freeze. As a consequence, in certain circumstances (like video streaming), the advantages greatly outweigh this.

Where maintaining the integrity of the data is critical however, the best results are gained from a combination of both UDP and TCP.  UDP is used to transfer the data, whilst TCP is used to ensure that any packets lost are resent.  An agent at the receiving station is used to reconstruct the data after the transfer.

Some examples of effective use of this method include:

  • Banking, where vast amounts of data is captured and moved around each day.
  • Researchers needing to share large volumes of scientific and clinical research data.
  • The manufacturing industry, sharing large files and data sets to global development teams.

Find out more about Aspera’s combined UDP and TCP technology – FASP – on the Aspera vendor page. You can also download the Aspera White Paper ‘Taking Big Data to the Cloud’.

This is the latest in a series of guest blog posts from the leading vendors, highlighting how a file transfer solution can add value to your organisation.

Do you need to quickly and securely migrate files to the cloud? Our Needs Analysis Service will help scope your requirements and find the right solution for you.
Visit the Aspera page to download the White Paper ‘Taking Big Data to the Cloud’.

FASP 3.6 Aspera Transfer Platform Now Available

FASP 3.6 Aspera Transfer Platform Now Available

Aspera have released version 3.6 of its Aspera Transfer Platform, a major new release that is built into the core of all the latest Aspera products and Aspera On Demand offerings. In this release, Aspera’s core products; Enterprise Server, Connect Server, Point-to-Point Client and Desktop Client – have all undergone a significant revamping with support for multi-tenancy and many new notable features and capabilities.

Aspera LogoThe most significant highlights include:

 

New Multi-Tenant Access Control and Content Security

  • Enables multiple Aspera applications to securely authenticate to transfer clusters and individual transfer nodes with no access to underlying storage credentials, supporting all local and cloud storage types.
  • New server side encryption at rest stores content securely at rest based on server defined / queried secrets, in addition to client side encryption at rest.

New Aspera On Demand Capabilities

  • New native cluster support for invoking transfers that use all nodes in a cluster and New node API for querying cluster transfer status through one cluster address.
  • New autoscale service for the Transfer Cluster Manager (ATCM) that automatically scales the number of Aspera Server nodes for client transfer demands up and down based on user-defined policies.
  • New super fast HVM images running on Amazon Web Services cloud infrastructure, supports single transfers sessions at up to 1.5 Gigabits per second throughput using the latest VM technology.
  • Many new Direct-to-Cloud storage advanced transfer capabilities supporting all major cloud storage providers including, IBM SoftLayer Swift, AWS S3 and GovCloud, Microsoft Azure Blob, Akamai Netstorage, Limelight Orchestrate Cloud Storage, Google Cloud Storage, HDFS, Ceph, and coming soon, CleverSafe.

New Aspera Watch (Asperawatchd) Service

  • A new ultra fast service which provides immediate change detection and snapshot service designed for speed, scale and distributed change watching.

New FASPStream binary and APIs

  • Server and client products include a new binary for sending and receiving ‘streams’ of data or growing files over wide area networks at the stream/file generation rate with nearly 0 delay at the receiver.

New ASCP4 Binary

  • New ascp4 binary has a new architecture for ultra high-speed transfers of very small files, able to achieve 1 million files per minute over Global WANs for 10 Gigabit per second sustained transfers of files of <=100KB over long distance.

New Aspera Sync (ASYNC) Features

  • Aspera Sync incorporates the Asperawatchd service eliminating bottlenecks of scanning on very large file systems (>20 million files) and scales with number of synchronising users by eliminating per-user file system access.

New Aspera Watchfolder (Beta Release)

  • This beta release of the new Aspera WatchFolder includes in all Linux and Windows core products and designed to meet the scaling and management requirements of modern workflows. This can be used in place or in addition to the Windows asperasync.exe or simplesync services on Linux.

 

For more information on this new Aspera release please contact an Aspera engineer on 0207 118 9640.

Aspera Delivers Major Advances in its Core Fasp High-Speed Transfer Platform

Aspera Delivers Major Advances in its Core Fasp High-Speed Transfer Platform

aspera_faspstreamAspera unveiled the next generation of its unique FASP software platform for moving Big Data and media at maximum speed and scale regardless of size, distance or network conditions.  Multiple breakthroughs bring new capabilities and advanced features, allowing enterprises of all sizes to meet variable data movement demands in real time with speed, control, security and ease, and all capabilities are designed to move data directly to cloud and on premise storage. Advances in the Core Transport

  • ascp4 is the next-generation Aspera file transfer binary introducing a new architecture for ultra high-speed transfer of small files in addition to large data sets. The architecture transfers up to one million files per minute for even the smallest file sizes (e.g. <10KB) and achieves > 5Gbps transfer rates for global RTT and packet loss conditions (200ms / 2%).
  • FASPStream transport expands Aspera’s FASP transport technology for “live” and “near live” data streaming. Aspera FASP is a patented transport protocol for achieving highly efficient bulk data transfer over IP networks, independent of distance and quality (round-trip latency and packet loss). FASPstream expands this capability to provide a fully reliable streaming protocol for in-order data delivery over Internet WANs with excellent quality and negligible start up delay.

Advances in Aspera Direct-to-Cloud Transfer and Autoscaling Aspera Direct-to-Cloud Storage transfer capability moves even the largest (4K) media formats from source directly to object storage destination with native I/O for all major cloud storage providers: IBM SoftLayer Swift, AWS S3, Microsoft Azure Blob, Akamai NetStorage, Limelight Orchestrate Cloud Storage, Google Cloud Storage, and HDFS (beta).  Version 3.6 brings many new capabilities to all Aspera On Demand products such as:

  • Server-side encryption at-rest in addition to client-side, as well as in-transit encryption
  • Automatic determination of the cloud storage part size allowing for the largest file sizes to be sent without specific configuration.
  • New clustered transfers enable 10 Gbps+ transfers in/out/between clouds.
  • A new Aspera Transfer Cluster Manager (ATCM) with Autoscale technology providing elastic scaling, a multi-tenant access key system for security and transfer reporting, and automatic high availability. The ATCM is a cloud-infrastructure independent service created by Aspera that allows for dynamic, real-time scale out of transfer capacity with automatic start/stop of transfer server instances, automatic balancing of client requests across available instances and configurable service levels to manage maximum transfer load per instance, available idle instances for “burst” and automatic decommissioning of unused instances.

Advances in Transfer Automation, Synchronization and Management

  • A new Aspera WatchFolder Service is specifically designed to power high-volume automated file and directory transfers with advanced features for media workflows and content distribution.

Built on asperawatchd, a new file system notification service, designed for speed, scale and distributed change watching, watch directories can now cover huge files systems and large numbers of watch folders. Collections of files and folders can now be grouped into a single “drop” and transferred to remote nodes as a single logical unit and controlling which files arrive last. “Growing”(in progress) files are fully supported. A RESTful API enables programmatic control for customized and automated processing.

  • Aspera Sync 3.6 also integrates asperawatchd technology to capture changes on any local or shared storage client host (CIFS, NFS, etc.) and aggregate all changes in real time in a single snapshot for speed on very large file systems. File attribute changes (Windows ACL, Unix ownership) are syncd, even when content is not updated, and LZ compression achieves high performance on low capacity networks. Bidirectional synchronization now works on cloud storage as well as block storage.
  • Aspera Console 3.0 for Centralized Management is built on a new architecture with the ability to precisely regulate reporting load as the number of transfer nodes increases, dramatically improving scalability, robustness, and timely transfer status for large deployments. Many new settings and options have been added, including advanced email notification triggers on any transfer attribute such as source and destination path, Aspera Shares and Faspex user attributes, and transfer directories; new managed node clusters allow managed nodes to be assigned to cluster groups with shared storage to support automatic load balancing and failover of transfers; and an advanced search of transfer history searches by transfer names, IDs, contacts, paths, and status.