What would a better COVID test system setup look like?

What would a better COVID test system setup look like?

The limitations of Microsoft Excel have been blamed for the glitch that led to nearly 16,000 cases of COVID-19 going unreported in England. Our experts explain what a better setup would look like.

 

The process as we know it

Commercial firms analysing the COVID swab tests filed their results in CSV format without problem. These were sent to Public Health England (PHE) by their preferred method, SFTP1. There were no reported problems here.

Multiple files from multiple locations were received by PHE, then processed and consolidated into a single file in order to “upload to NHS Test & Trace team as well as other government computer dashboards,” according to the BBC2. This shouldn’t be a complex process, but this is where the problems began.

The cause

PHE developers selected an inappropriate file type to process potentially large volumes of data. Selecting Microsoft Excel’s XLS format allows a maximum number of rows just north of 65.5k. With complex test results taking many rows, only around 1,400 cases could be stored in each file. The rest of the data simply wasn’t retained.

What should have been in place?

Richard Auger, Pro2col’s Technical Manager, says, “It’s well known that the XLS format is inferior to CSV, and it’s therefore so infrequently used that we haven’t created a workflow based on this format for at least ten years.”

Given that PHE were central to the Government’s estimation on 21st September that “cases could reach 50,000 by October,” it’s not unreasonable to expect systems and processes to have been built with this volume in mind, or at the very least, with scalability in mind3.

The biggest issue is that the process design was fundamentally flawed. Any critical process such as this, should have had some error handling and reporting capabilities built into it. James Lewis, Managing Director at Pro2col suggests, “Taking eight days to identify the problem is unreasonable, and unnecessary. There are a number of ways basic error handling mechanisms could have been built into their workflow processes that wouldn’t have impacted performance design.”

What should have been in place?

To handle data movement of this scale, the process should include:

 

  • Security controls – Test results are classified as personal, sensitive data.
  • Automated onward processing – This might include results alerts, moving to the Test & Trace team, and reporting.
  • Reporting / error handling – The governance of sensitive data is critical, even more so in this case.

So what would a better setup look like?

The supply of data to PHE appears to be without fault – structured data in CSV files, provided by a secured file transfer protocol, SFTP.

The CSV files were then presumably processed automatically. The contents of each CSV exported and written to the XLS file. This is where the error most likely occurred, and could have been avoided.

Whilst we don’t know what other steps might have existed in their process, it is possible to select multiple CSV files and merge into a single CSV file. This would offer much larger capacity and ensure that all data is imported.

There are also several steps that could be incorporated to mitigate risk:

  • A check on the file creation date to ensure that only the correct files were being processed would be the first step.
  • Error handling can take many forms. A simple process would be to read the number of lines of data contained in each CSV file, total the number of records across all files, and match that against the output file. The result? A mismatch is quickly identified and action can be taken to resolve it before, it makes an impact. This would also allow any receiving party to check the governance of the data received before delivering it to any onward process.
  • Uploading an XLS file to the NHS Test & Trace team would likely result in a more complex routine for extracting the data than if it was provided in CSV format, which is much easier to import into onward systems.

Summary

As practitioners of workflow best practice, we’d guess that whoever designed and delivered this process had limited experience of creating workflows, especially for such critical healthcare data. The technology wasn’t at fault. It was the lack of insight to build in error reporting, combined with the failure to test the process with the volume of data they’d been expecting for many weeks. This was an entirely avoidable situation, and not a particularly complicated process to implement correctly.

References

3 Powerful Examples of MFT and DLP Paired in Action

Three Powerful Examples of MFT and DLP Paired in Action

This guest blog from HelpSystems outlines how you can maximise the power of MFT by pairing it with a DLP solution

On paper, combining two robust software products to form a more complete solution is a no-brainer. But how does it actually work? First, let’s take a look at what the two solutions – adaptive data loss protection (A-DLP) and managed file transfer (MFT) – each offer users. Then, we’ll highlight a few specific use cases where the combination of DLP and MFT elevates an organisation’s cybersecurity position, better than each solution could alone.

DLP offers content inspection:

  • Replacing sensitive text – including PII, PCI, and HIPAA – with asterisks.
  • Detecting and redacting text contained in images, even scanned ones.
  • Redacting text but allowing for files to still be delivered, minus offending content.

MFT delivers secure file transfers:

  • Dashboard-friendly, centralised control
  • Auditing and reporting
  • Advanced workflow automation
  • Document collaboration
  • File transfer security

What Level of Data Security do DLP and MFT Provide Together?

  • With MFT in place, the movement of files inside and outside of your organisation is secure and automated, with the collaboration and data tracking functionality not available with DLP alone.
  • MFT securely transfers attachments between organisations and trading partners
  • The ICAP Gateway intercepts content when threat protection and DLP requirements exist
  • ICAP runs rule sets (keyword search, executable renaming, script removal, etc.).
  • If the content can be sanitised the transmission is allowed and continues
  • If content cannot be sanitised the transmission is blocked

Together, HelpSystems’ Clearswift A-DLP solution and MFT provide a layered, unified, and web-based solution – from the content contained, to the secure data transfer of the content. Both solutions are trusted worldwide for their ability to provide enterprise-level protection against data breaches. When software is paired from a single provider, it provides stability and superior customer service.

MFT and DLP Use Cases: Combined for End-to-End Security

Use Case #1: Defence sector vendor needs to transfer attachments between employees and trading partners.

Originally, this vendor used MFT to securely transfer the attachments sent between its employees and partners. They decided to also add deep content inspection to these file transfers by integrating their MFT with Secure ICAP Gateway, adding Structural Sanitisation and Anti-Virus for additional layers of security. With both solutions in place, the vendor can:

  • Intercept content when threat protection and DLP requirements exit.
  • Run the proper rule sets (keyword search, executable renaming, script removal, etc.).
  • Control whether content is allowed or blocked. If content can be sanitised, then transmission continues. However, if content cannot be sanitised then the transmission is blocked.

Use Case #2: A credit card vendor needed insight into when customer, employee and/or patient records in a file were being uploaded to MFT.

This credit card vendor needed to detect when records contained in files (for employees and/or patients) were being uploaded to MFT. They also needed to be able to count the records and maintain an audit log showing what data was being shared and with whom, and to have the ability to corroborate files from a single record. The vendor was able to do all of this by integrating the secure ICAP Gateway with MFT. In addition, with this combination of solutions, they can take advantage of Lexical Expression Qualifiers and trigger different actions based on the number of records detected – actions such as an audit, refer to sender, redaction or block).

Use Case #3: Government agency needed same level of security from uploads received as they got from their MFT transactions.

This government agency regularly needs to accept file uploads from partners and the public. However, because this action can be exploited to deliver active content or even viruses and malware, they knew they needed to achieve comparable threat protection to what they had in place with their MFT file transfers. The agency integrated the Secure ICAP Gateway with their MFT for a combined, secure solution. Now, any file can be scanned before it is either passed or accepted and infected content is then destroyed or redacted. They also have the choice of triple-AV (Kaspersky/Sophos/Avira) and constant Active Content Protection in place.

Get in touch today to learn more about how your business will benefit from integrating MFT with a DLP solution.

Alternatively, reserve your place to see a live demo via our webinar below.

Clearswift A-DLP with Managed File Transfer

29/9/202010:30pm BST30 minutes
Join us for a short webinar demo showing how Clearswift ICAP integrates with your MFT solution. This will extend your security, whilst minimising the impact on business processes. The demo will be led by Clearswift Engineer Steve Jeffery.

$40,000 of free training

$40,000 OF FREE TRAINING

Helping businesses adapt to the increased demand for secure remote working and workflow automation

As COVID-19 took a hold on the world, businesses everywhere were forced to switch their offices for home studies, and their board meetings for zoom calls. Behind the scenes, we recognised the growing need for secure file transfer and business process automation. So we embarked upon a plan to give away our Certified File Transfer Professional (CFTP) training programme, completely free of charge.

Over 136 engineers signed up and completed the course – which equates to over $40,000 of free training!

Thanks to this offer, a huge group of employees have been able to continue developing their careers in lockdown, and are now better equipped to tackle the lingering economic challenges of COVID-19.

Managing Director James Lewis says, “I am delighted with the success of this promotion and the positive impact it will have on so many businesses, both now and in the future.”

Although the course is no longer available for free, there are still a whole host of reasons why the CFTP is a fantastic investment for IT professionals wanting to learn new skills or certify their knowledge:

1. The CFTP certifies your existing file transfer knowledge
Adding this to your CV will make you stand out ahead of your next promotion. It will help you work more efficiently as demand for remote working and automation continues to ramp up. The certification is CPD accredited, so employers know it’s a reputable qualification from a trusted provider.

2. Learn new knowledge that’s transferable across vendors
It’s vendor-independent – the only course that is in fact – so what you learn is applicable across all Managed File Transfer solutions.

3. Learn safely from your own home at your own pace
There’s no need to go to an exam centre or classes. It’s all online and you can do it at any time to fit it around child care or other work and family commitments.

4. Maintain full lifetime access to the resources once you’ve finished
The video lessons and study guide are there for you to refer to whenever you need, even after completion.

5. It’s available for just £240!

Enrolling couldn’t be easier – just visit the CFTP training programme and add it to your basket. We’ll set you up with your unique learning profile so you can get started straight away.

Pro2col also run and host product-based training tailored to your Managed File Transfer solution and requirement. As experts solely focused on this niche, we are perfectly placed to ensure your staff are fully trained to be able to use and support your software. Find out about other training options.

MFT and COVID-19

MANAGED FILE TRANSFER & THE NEW COVID-19 'NORMAL'

MFT plays a unique role addressing the business fallout of the pandemic

As new cases of COVID-19 decline in the UK, lockdown restrictions are gently easing.  Business are edging back towards ‘normal’, with some staff returning from furlough and sales picking up in certain areas.

Yet COVID-19 isn’t going away. Organisations are having to adapt their IT infrastructure for this uncharted landscape. They are investing in new technology or extending existing solutions to support secure remote working, handle higher volumes of transfers and deliver automation to reduce manual processing.

Managed File Transfer (MFT) is a powerful business enabler that does just that. It allows transfers to take place in a centralised, controlled, secure fashion – inside and outside an organisation – between systems and / or users. Files are transferred more quickly and securely, enhancing productivity and providing visibility, with minimal human interaction.

Secure remote working

Sensitive data is no longer within the safety of the company’s firewall. MFT secures data, no matter where staff work from. It’s accessible too, giving remote employees the tools they need to ensure critical files get where they need to be. Benefits include:

  • Protected files in motion and at rest
  • Guaranteed delivery
  • Access control and authentication
  • Single-pane-of-glass visibility for IT, to maintain compliance (PCI, SOX, GDPR, ISO)
  • Accelerated transfers of large files between remote employees or locations
  • User friendly web-based interface and mobile application
  • Optional high availability (HA) and disaster recovery (DR)

Automation

Automated workflows can replace some manual processes that are suffering, due to staff absence during the pandemic. For example, batch processing of any repetitive process or secure data capture, to remove paper processing.

Automated file transfer also delivers file-based system to system transfers, either inside an organisation or between trading partners. Plus integration with other applications, both on-premise or in the cloud, e.g SharePoint, Amazon & Azure.

There are many benefits to workflow automation, including:

  • Reduced operational costs
  • Increased productivity
  • Increased reliability, by reducing human error
  • Optimised performance
  • Supporting a high availability infrastructure

Getting started with MFT

If you don’t have MFT, this is a great time to invest. You will quickly see a return on investment. The Managed File Transfer Buyer’s Guide is a great starting point for your research.

If you already use MFT, considering adding functionality to support the ‘new normal’ in a COVID-19 society:

  • Secure email
  • Secure folders
  • Secure forms
  • Cloud connectors
  • Mobile app
  • Advanced workflow automation
  • Auditing and reporting

You can find more information about additional modules and functionality in the the Managed File Transfer Buyer’s Guide. Or, just pick up the phone. Pro2col experts will be happy to talk through your requirements. Call us on 0333 123 1240 or contact us online.

Certified File Transfer Professional: Getting Started

FREE FILE TRANSFER TRAINING DURING COVID-19 LOCKDOWN

Five reasons why the CFTP is the ideal training programme for IT professionals right now

IT departments across the world have closed their offices and turned out the lights. Busy support desks are now dispersed across sofas and kitchen counter tops, and the board room has been replaced with the home study.

This global effort to keep people safe from COVID-19 and to prevent health services from becoming overwhelmed is affecting us all in different ways. While some staff have been furloughed, others are busier than ever.

Either way, the spike in home working and increased demand for automation capabilities has made secure file transfer all the more important.

That’s why we’re opening up our Certified File Transfer Professional (CFTP) training programme, completely free, for up to two people per organisation until the end of June 2020.

We want to equip organisations with the right skills to help weather the economic downturn and give furloughed employees the chance to gain new knowledge for when they get back into the workplace.

Here’s why the CFTP is the ideal choice for IT professionals wanting to learn new skills or certify their knowledge during lockdown.

      1. The CFTP certifies your existing file transfer knowledge. We know we’re heading for difficult times as the global economy recovers. Adding this to your CV will make you stand out ahead of your next promotion or job move. It’s CPD accredited, so employers know it’s a reputable qualification from a trusted provider.
      2. Learn new knowledge that’s transferable across vendors. It’s vendor-independent – the only course that is in fact – so what you learn is applicable across all Managed File Transfer solutions.
      3. Learn safely from your own home at your own pace. There’s no need to go to an exam centre or classes. It’s all online and you can do it at any time to fit it around child care or other work and family commitments.
      4. Maintain full lifetime access to the resources once you’ve finished. The video lessons and study guide are there for you to refer to whenever you need.
      5. You’ll be saving over £240! That’s what the CFTP usually costs for a single user, but we’re offering you full access the resources, exam and certification for free until 30th June 2020.
It’s easy to get started. Just visit the CFTP training programme and add it to your basket. Enter the code pro2col-free-gift at the checkout. We’ll set you up with your unique learning profile so you can get started.

Don’t forget this is only valid until the end of June. Don’t miss out.

Remember this is only available for up to two people per organisation.

Managed File Transfer software comparison

Enterprise Managed File Transfer Software Comparison

We review the MFT vendor landscape for 2020-21

There are a lot of out of date or inaccurate comparisons of enterprise Managed File Transfer (MFT) vendors online. People still use the Garter MFT Vendors Magic Quadrant, for example, but it’s over ten years old. And there are other comparisons from so called ‘experts’ who only sell one product – so of course they will say it’s the best.

So, why should you listen to us? At Pro2col, we are independent experts and have worked with secure and Managed File Transfer for over 15 years. In fact we’re often asked to assist vendors with product research and development. We are also providers and developers of the only vendor-independent training programme: The Certified File Transfer Professional (CFTP).

We’ve distilled this knowledge to create a top four Enterprise Managed File Transfer comparison, and give you an overview of the rest of the marketplace.

All the products in our top four support the major transfer protocols, deliver high security with PGP encryption and other measures. They all have sophisticated enterprise schedulers, workflow triggers and cloud connectivity, plus excellent visibility / SLA monitoring capabilities.

All these vendors bring out regular releases and are responsive to any issues. That is ultimately why they have made the list.

These are in alphabetical order – we haven’t listed them by which is best. They are all strong products and the right one will depend entirely on your business and budget. For example, do you need enterprise Managed File Transfer or have smaller requirements? Ultimately, there is no ‘best Managed File Transfer’, but the best fit for you.

I hope this Enterprise Managed File Transfer Software Comparison helps you.

James Lewis

Managing Director, Pro2col Ltd

Enterprise Managed File Transfer comparison top four

Coviant Software – Diplomat MFT 

Coviant Software - Diplomat MFT

Diplomat MFT is one of our top four solutions because it has particular areas of strength and is really good value for money.

It has always been strong on efficiency and versatility, but it solves some specific problems very well in a rapidly changing space; especially enterprise integration for cloud storage, web services and it is platform independent.

Diplomat MFT has an increasing number of application and cloud integration points. The product is benefiting from a new lease of life under Greg Hoffer, the recently appointed CEO. He is working directly with users to understand their specific requirements and extend the product capabilities. 

But Diplomat’s USP is definitely its value for money. The product is considered by the analysts as the value player in the market.

The company was formed in 2004 and is based in San Antonio, Texas, USA.

Globalscape Software – EFT Server

Globalscape Software - EFT Server

No MFT comparison would be complete without including Globalscape EFT Server. It has been a cornerstone of the MFT marketplace for well over a decade. It has always been a strong automation platform for Windows and they bring out a major new release each year.

There are three versions of Globalscape EFT, which stands for Enhanced File Transfer. They are EFT Express for smaller to medium-sized businesses or tactical deployments; EFT Enterprise for enterprise Managed File Transfer incorporating the whole infrastructure; and their cloud-based solution EFT Arcus.

.

Managed File Transfer software comparison

EFT has a modular structure, so organisations can easily scale their solution by adding new features.

Globalscape has recently realigned their business strategy and the release of EFT 2020 (version 8) includes some very advanced new functionality.  The privacy and compliance features are ideal for supporting GDPR or other regulations. This is – without a doubt – now the product’s USP. It includes settings which map to the sections of the GDPR, a built in risk assessment report to identify compliance fails, the ability to assign tasks to Data Protection Officer (DPO) and a lot more!

The CEO is Robert Alpert, Globalscape is listed on the New York Stock Exchange and the organisation is based in San Antonio, Texas, USA.

HelpSystems – GoAnywhere MFT

Helpsystems - GoAnywhere MFT

The original developers Linoma were acquired a few years back, bringing GoAnywhere MFT under the HelpSystems umbrella. Since this acquisition, they have continued to prove to be an innovative vendor.

GoAnywhere MFT is a cross-platform application and very strong on cloud connectivity. HelpSystems have put a considerable focus on out-of-the-box cloud connectors. We’ve seen how this has simplified cloud strategies for a lot of business – both global enterprises and small businesses.

.

Managed File Transfer software comparison

They bring out a major new release each year, and are continually adding new cloud connectors to their portfolio. GoAnywhere MFT is another easily scalable solution and a big advantage is its seamless operation with other HelpSystems products. GoAnywhere MFT, for example, works together with their RPA solution, Automate. Combined, the two products enhance efficiency within a business processes, with the potential to add more processes when required. This and the product’s cloud connectivity are the big strengths for GoAnywhere MFT.

The standout feature of GoAnywhere MFT, however, is the Secure Forms module. This enables you to securely collect data in multiple formats (including drag and drop for images), then automatically process the data using complex, logic-based automated workflows.

The recently promoted CEO is Kate Bolseth and the organisation is based in Minneapolis, Minnesota, USA.

Progress formerly Ipswitch  – MOVEit

Progress formerly Ipswitch - MOVEit

The MOVEit product suite includes four solutions:

  • MOVEit Transfer for tactical deployments.
  • MOVEit Automation for complete Managed File Transfer. Users can chooose either the Corporate or Enterprise tier – the latter providing more extensive functionality.
  • MOVEit Cloud, which is self-explanatory.

MOVEit has been known and loved for many years. It is very well established and has a loyal customer base. People trust it because it has always had a strong reputation for security. Its multitenancy capabilities for example, are among the most secure on the market.

MOVEit can work out more cost effective for users needing multiple agents because there is a considerable volume discount. Another plus is that MOVEit Automation (Enterprise) users can automate the movement of files between unlimited servers

Managed File Transfer software comparison

In all honesty, we’ve observed a lack of development in the last couple of years. But the vendor Ipswitch was acquired by Progress in 2019 for +$200m and it is rapidly gaining pace. Recent developments, for example, includes API enhancements and Powershell for custom scripts.

So what differentiates MOVEit? Unlike other enterprise level solutions, MOVEit Automation users can add an unlimited number of managed servers.

Ipswitch was formed in 1991 and the CEO is Michael Grossi and the organisation is based in Burlington, Massachusetts, USA.

Enterprise Managed File Transfer comparison: The rest of the market

Of course, there are many other solutions on the market that haven’t made our top four. So here’s a brief explanation of some of the other products you’re likely to come across in your Managed File Transfer comparison, again in alphabetical order:

Attunity Managed File Transfer

Attunity Managed File Transfer

Since acquiring the technology from Repliweb in 2011 there appears to have been little focus on this product.

Axway Secure Transport

Axway - Secure Transport

Axway acquired Tumbleweed, the developer of SecureTransport, in 2008 and has since made numerous further additions to their portfolio. Whilst still a significant player in the MFT industry their products are largely disparate. The biggest concern we hear is the extensive professional service costs to implement and upgrade versions.

Cleo Harmony

Cleo - Harmony

Harmony doesn’t make our top four, but would make our top five. A solid managed file transfer solution, providing integration capabilities through the acquisition of Extol a number of years ago. Cleo is led by a very impressive leadership team and it only doesn’t make our top four because of the way in which it’s priced.

JScape MFT Server

JScape MFT Server

Another good Managed File Transfer product which has stood the test of time. A java application providing cross platform server installations, JScape has extensive functionality. The only downside users report to us is that the interface isn’t intuitive.

Oracle Managed File Transfer

Oracle Managed File Transfer

Oracle MFT has limited features and capabilities. It was developed to protect their SOA/B2B install base and has been in maintenance mode for some years, with the focus shifting to the Oracle Integration Cloud.

Seeburger

Seeburger

Still an active MFT vendor with a major focus on their home market of Germany. We don’t know of any company in the UK that runs it and haven’t had any exposure to it so could not comment on either product or company.

Solar Winds Serv-U

Solarwinds - Serv-U

On the face of it Serv-U has all the features that define enterprise Managed File Transfer, but the product is lacking in a modern browser-based file transfer mechanism, still relying on a java applet for file transfers. Customer support is only by email and some customers report difficulties accessing it. Support is, of course, just as important as the product itself. This product has no ongoing development and is in maintenance mode.

South River Technologies – Cornerstone

South River Technologies - Cornerstone

SRT continues to develop Cornerstone, which is a modular solution at a significantly more competitive price than other options. An extension of their industry leading Titan SFTP server, Cornerstone provides a lot of functionality for its price.

Enterprise Managed File Transfer FAQS

Which is the best Managed File Transfer vendor?

As independent experts with over 15 years’ experience working with MFT, we’ve whittled it down to the four best (in alphabetical order).

  • Coviant Software – Diplomat MFT
  • Globalscape Software – EFT Server
  • HelpSystems – GoAnywhere MFT
  • Progress  – MOVEit MFT

Who are the Enterprise Managed File Transfer vendors?

Our top four are:

  • Coviant Software – Diplomat MFT
  • Globalscape Software – EFT Server
  • HelpSystems – GoAnywhere MFT
  • Progress  – MOVEit MFT

But there are over forty on the market including:

  • Attunity
  • Axway
  • Cleo

See full list here…

How do I compare Managed File Transfer software?

Online comparison articles can only tell you so much. The right solution is going to depend entirely on your unique business requirements. Our free comparison tool will identify the right solution for you.