Large file transfer: How do you move over a terabyte of data across the world quickly and securely?

Large file transfer: How do you move over a terabyte of data across the world quickly and securely?

This is a server-to-server MFT large file transfer use case for moving whole company data sets during a business acquisition.

A utilities company specialising in renewable energy needed to move very large files over long distances. They were acquiring businesses around the world including South America and Africa.

With files measuring over a terabyte, IT were writing data to disk and then shipping it to the head office in Norway. It was then taken to the data centre where it was uploaded to the server.

This was time intensive and costly. From start to finish, the process took over a week to get the data from A to B. Security was a concern too; theft or a shipping incident could result in a breach of that valuable data.

With business expansion plans expecting a further 30 acquisitions per year, the company needed a faster, easier and more secure process.

The IT team looked at ad hoc person-to-person solutions, but the TCP-based protocol is not geared up for files of that size. Transfers take a very long time and – if anything goes wrong part way through and checkpoint restart isnt available – it has to start again from the beginning rather than picking up where it left off.

A UDP-based solution

The company evaluated a solution with a UDP-based protocol and it took just five hours to move a test file of 800GB.

Businesses on both sides of the acquisition found the interface very user friendly. IT build a bespoke application, from a simple-to-use template. The acquired business drags and drops the files into the application.

The result is a much quicker, cost effective and secure method for transferring very large files.

Other large file use-cases

This is just one example of server-to-server large file transfer. There are many other use-cases, some requiring manual / ad hoc transfers like this one, and other recurring automated transfers.

Manual (ad hoc)

  • Video production companies sharing files in 8k resolution;
  • Design agencies sharing multi-layered design files eg: Photoshop, InDesign;
  • Architects or manufacturers moving CAD files;
  • Transferring big data for research purposes, such as gene sequencing (this can also be automated).

Automated

  • Any organisation needing to move very large databases;
  • Independent software vendors sharing support and log databases in a ‘follow-the-sun’ workflow, ie: providing a 24/7 support service;
  • HPC (high performance computers) sharing or backing-up large files from the data centre. Eg: satellite data which has been processed to create forecasts;
  • Any company carrying out mapping or scanning (eg: healthcare, utilities, survey organisations).

The size of the data set and the nature of the transfer determines the right solution for your requirements. Pro2col can advise you through the different stages of your file transfer project to make sure it’s a success. Whether that’s scoping your requirements, product demos and proof of concept / evaluation, negotiating the best deal with vendors or installations and training, call 0333 123 1240 or complete the online contact form.

Need further expertise? Download our FREE resources

Pro2col in Pink: Race for Life team

Pro2col in Pink: Race for Life team

Three Pro2col ladies are doing the ‘Pretty Muddy’ Race for Life as team ‘Pro2col in Pink‘.

Lindsay, Michelle and Sally will get stuck into this muddy challenge on Saturday 15th June, raising money for Cancer Research UK. The route goes through Poole Park, Whitecliff Recreational Ground and Harbourside, with a variety of muddy obstacles along the way.

Pro2col MD James Lewis said: “The team are passionate about making a difference to our community and to society in general. So alongside of beach cleans and crisp packet recycling, most of the staff are undertaking different challenges in June to help raise money for charity. They are causes very close to their hearts and I’m really proud of their dedication.”

ABOUT RACE FOR LIFE

Cancer survival rates have doubled over the past 40 years. Consistent progress is being made but improvements to technology and ground-breaking work offer new opportunities to find different ways to prevent, diagnose and treat cancer and improve survival rates even further. Help Cancer Research UK improve results even faster.

1 in 2 people will be diagnosed with cancer in their lifetime. Every single pound you donate makes a difference to Cancer Research UK’s groundbreaking work, so please sponsor us now.

 

MEET THE TEAM

Lindsay Lewis

Michelle Grimaldi

Sally Gates

Secure Mail

Secure Mail

It is very common for staff to share sensitive files via email, risking your organisations security and compliance. Or they need to share files which are too big to attach, so they resort to consumer-grade applications like Dropbox or WeTransfer.

GoAnywhere MFT provides a Secure Mail module, which addresses these challenges. It is a great way for businesses of any size to control and manage sensitive and valuable data exchanges via email to remote or third parties.

In a typical email setup, the servers do not encrypt the email attachment. The GoAnywhere Secure Mail solution allows email attachments to be encrypted and securely transferred. This feature can be accessed either via the Microsoft Outlook plugin or the GoAnywhere HTTPS web interface.

  

Microsoft Outlook plugin

When sending emails using the Outlook plugin, the attached file is not attached to the email itself. It is sent encrypted to the GoAnywhere MFT server and encrypted at rest on the storage. An encoded URL, which can also be password protected, is then embedded into the email body. The recipient can link directly to the stored file or files hosted on and controlled by the GoAnywhere MFT server.

 

GoAnywhere HTTPS client interface

The GoAnywhere HTTPS client interface provides an alternative method to send secure email attachments. The web portal provides a familiar and intuitive interface for users to control and manage this process.

The administration interface allows IT or delegated administrators to manage aspects of the Outlook plugin and secure mail features, giving them control over the size of files attached and enforced settings including: Security controls & protection levels for URL protection, password and certificate delivery.

Address rules allow you to lock down the email domains you can send to or receive from. This level of control is a great solution for business with known trading partners who need that additional level of security.

The Reporting feature provides visibility of activities, while Auditing and Package Manager allow administrators to control the package expiration and maximum downloads for file or files.

Would you like to see GoAnywhere or the Secure Mail module in action? Book a free demo or trial.

Did you find this useful?

Receive GoAnywhere tips and news in your inbox

Choosing the right Managed File Transfer protocol

This blog post answers your questions about Managed File Transfer protocols. Which are the most widely used file transfer delivery protocols? Which should you be using and how do you identify which solution uses which protocols?

A protocol is the set of rules that determines how files are transferred from one computer to another, through a network. That might be an internal network (from one computer to another within the same network) or more commonly a Wide Area Network such as the internet.

The nature of your data and its destination will determine the right protocol for the transfer. For example, personally identifiable data and credit card information will need a secure protocol.

BASIC PROTOCOLS

FTP
(File transfer protocol)

How can it be used?

Upload/download files
Rename and delete files
Create/delete folders
Execute custom commands on server
Check integrity of files

When can it not be used?

X Secure data at rest
X Secure data in transit (FTPS can)
X Work over just one firewall port
X Provide strong authentication

FTPS
(“FTP Secured” using SSL)

 Secure data in transit
Upload/download files
Rename and delete files
Create/delete folders
Execute custom commands on server
Check integrity of files
 Provide strong authentication

X Secure data at rest
X Work over just one firewall port

SFTP
(“Secure FTP” using SSH)

Secure data in transit
Upload/download files
Rename and delete files
Create/delete folders
 Provide strong authentication
Work over just one firewall port (22)

X Secure data at rest
X Check integrity of files
X Execute custom commands on server

SCP
(“Secure CoPy”)

Secure data in transit
Upload/download files
Work over just one firewall port (22)
 Provide strong authentication

X Rename and delete files
X Create/delete folders
X Check integrity of files
X Execute custom commands on server

ADVANCED PROTOCOLS

HTTP
(HyperText Transfer Protocol)

HTTP CAN ALWAYS
Download files
Work over one firewall port (80)

HTTP CAN SOMETIMES
Upload files
Rename and delete files
Create/delete folders
Execute custom commands on server
Check integrity of files

Secure data at rest
X Secure data in transit (HTTPS can)
X Provide strong authentication

HTTPS
(HTTP Secured with SSL)

HTTPS CAN ALWAYS
✓ Download files
✓ Work over one firewall port (443)
✓ Secure data in transit

HTTPS CAN SOMETIMES
– Upload files
– Rename and delete files
– Create/delete folders
– Execute custom commands on server
– Check integrity of files
Provide strong authentication

Secure data at rest

WebDav

WEBDAV CAN ALWAYS
Download/upload files
✓ Rename and delete files
✓ Create/delete folders
✓ Work over one firewall port (443)
✓ Secure data in transit

WEBDAV CAN SOMETIMES
 Provide strong authentication

Secure data at rest
X Execute custom commands on server
X Check integrity of files

EMAIL PROTOCOLS

SMTP
To send mail

Push files as attachments
✓ Be secured with SSL/TLS
✓ Often uses ports 25, 465 or 587

Pull files from other servers

POP3
To get mail

Pull files from servers as attachments
Delete original email from servers
Be secured with SSL/TLS
Often uses port 995

X Push files as attachments
X Synchronize email folder contents
X Not supported in all email environments
POP3 is becoming obsolete

IMAP
To get mail and sync mail folders

Pull files from servers as attachments
✓ Delete original email from servers
✓ Synchronize email folder contents
✓ Be secured with SSL/TLS
Often uses port 993

Push files as attachments
X Be trusted if its key mailbox is also accessed interactively

Which Managed File Transfer protocol?

Guidance on what constitutes a secure protocol will change, adapting to stay one step ahead of cybercrime. That’s why it’s important to choose a vendor that releases regular product updates. With Pro2col’s free Managed File Transfer comparison service, you submit your requirements via a questionnaire. Our experts compare them against the different solutions and recommend the right product for you. Our experts consistently review the marketplace and only select credible solutions from credible vendors, who provide excellent support and regular software updates.

Interested in a file transfer solution?

Secure online forms and automation

Secure online forms and automation

Businesses need information from their internal users, external customers and suppliers all the time and it is highly likely this will include personal or sensitive data.

It is definitely not appropriate to ask users to share this information over email, but we know that is still common practice for many organisations. Email is not secure, so you risk a breach of the General Data Protection Regulation (GDPR), plus there is no guarantee of delivery. Email cannot support large files either.

Online forms provide a secure, customisable mechanism for your customers, suppliers and internal users to submit information to your business. It is a popular feature of Managed File Transfer systems and can capture any type of information or file size. Fields can also be configured to trigger onward business processes or integrate with internal systems.

To demonstrate the versatility and functionality of secure online forms with automation, let’s look at a use case from the motor insurance industry.

Use case: Motor insurance company

A customer is involved in a collision and needs to claim on their insurance. Using the secure online form, they enter the information required: Policy number, personal details, vehicle information, details of the collision and images of the damage.

Once the data has been received, a number of tasks need to happen to progress the claim.

This is where the technology really comes into its own. Automated actions sitting behind the secure online form, can execute many of these tasks and you can find out more in this video.

Examples of automated workflows

Once the user submits the form, it can trigger a range of automated workflows, such as:

  • Check the policy number meets alphanumeric sequence
  • Validate customer name against policy number
  • Assign claim to a claims handler
  • Automatically input the description directly into the customer database
  • Rename images to a pre-determined format, eg: policynumber_date
  • Move images to the image server
  • Send an automated reply via email or SMS or other business communication platform. The template can be personalised to include the name, assigned claims handler, policy number etc.

All of these processes can take place without any human intervention, demonstrating just how more efficient these labour-intensive tasks can become. You can see how this would suit other industries, such as mortgage brokers, doctors and private healthcare providers, or any outsourced business service, such as HR or payroll.

These are fairly simple use-cases, but there’s no end to the automation capabilities that can be applied. We recently customised secure online forms for a company sequencing hundreds of thousands of anonymised records of biological data. Customised logic built around metadata in mandatory fields in the form triggered the next step in the process.

Can you see how your organisation could benefit from adding secure online forms to your infrastructure? Get in touch for a chat now. Alternatively, complete our Managed File Transfer (MFT) comparison; answer a series of questions about this and other business requirements and our technical experts will recommend the best solution to suit your needs.

Interested in a file transfer solution?

Improve security and efficiency for your clients with online forms and automated workflows

Improve security and efficiency for your clients with online forms and automated workflows

Businesses need information from their internal users, external customers and suppliers all the time and it is highly likely this will include personal or sensitive data.

It is definitely not appropriate to ask users to share this information over email, but we know that is still common practice for many organisations. Email is not secure, so you risk a breach of the General Data Protection Regulation (GDPR), plus there is no guarantee of delivery. Email cannot support large files either.

Online forms provide a secure, customisable mechanism for your customers, suppliers and internal users to submit information to your business. It is a popular feature of Managed File Transfer systems and can capture any type of information or file size. Fields can also be configured to trigger onward business processes or integrate with internal systems.

To demonstrate the versatility and functionality of secure online forms with automation, let’s look at a use case from the motor insurance industry.

 

Use case: Motor insurance company

A customer is involved in a collision and needs to claim on their insurance. Using the secure online form, they enter the information required: Policy number, personal details, vehicle information, details of the collision and images of the damage.

Once the data has been received, a number of tasks need to happen to progress the claim.

This is where the technology really comes into its own. Automated actions sitting behind the secure online form, can execute many of these tasks and you can find out more in this video.

Examples of automated workflows

Once the user submits the form, it can trigger a range of automated workflows, such as:

  • Check the policy number meets alphanumeric sequence
  • Validate customer name against policy number
  • Assign claim to a claims handler
  • Automatically input the description directly into the customer database
  • Rename images to a pre-determined format, eg: policynumber_date
  • Move images to the image server
  • Send an automated reply via email or SMS or other business communication platform. The template can be personalised to include the name, assigned claims handler, policy number etc.

 

All of these processes can take place without any human intervention, demonstrating just how more efficient these labour-intensive tasks can become. You can see how this would suit other industries, such as mortgage brokers, doctors and private healthcare providers, or any outsourced business service, such as HR or payroll.

These are fairly simple use-cases, but there’s no end to the automation capabilities that can be applied. We recently customised secure online forms for a company sequencing hundreds of thousands of anonymised records of biological data. Customised logic built around metadata in mandatory fields in the form triggered the next step in the process.

If you have clients who would benefit from secure online forms with automation, or you would like to add this technology to your product portfolio, get in touch for an informal chat. Just complete our Partnering with Pro2col form.

Do you have a client needing secure online forms right now?
Or would you like to add this technology to your product portfolio?
Work with Pro2col and make big margins on products you wouldn’t ordinarily sell.