We live in a fast-paced, data-driven world where technology has an impact on every aspect of our lives. But the sheer volume of data being processed by businesses on a day to day basis is growing exponentially - a trend that is particularly noticeable in the creative industry.
Take video files. Just the transition from 1080p to 4k resolutions are seeing file sizes grow by as much as 152% per hour of footage. Back in the 720p HD Ready days an hour of footage weighed in at between 1-4 GB. Today 8K resolution maxes out around 200GB per hour.
And that is just the tip of the iceberg. Gaming has seen a similar explosion in file size requirements. Back in 2007 the original Assassin's Creed game wowed audiences with an immersive open world and a 7GB file size. 2020's Assassins Creed Valhalla weighed in at a chunky 47GB - a 571% change in just over a decade.
For creative industries, the ability to move files quickly, efficiently and securely from A to B is essential. When you wrap filming on the next streaming blockbuster series, you have to get the raw footage across to your post-production team on the other side of the work to start editing and working on SFX. When you’re in software, game or app development you’ll likely be working with designers, developers and teams that are spread across the world. Even our hunger for a 24 hour news cycle requires a constant stream of footage to be piped across the world – often filmed in locations where access to stable, fast internet connections are a luxury journalists and production teams lack.
No matter the field, whether it’s Design, Music Production, Architecture, Film, Visual Effects, Video Game Development, TV, or Radio – all are generating massive amounts of data. And, with the ever-improving high definition and 4K quality the size of the data behind the content we consume is only going to get bigger.
All of this creates significant challenges for those working in the creative industry. Just how can you efficiently, quickly and securely move that volume of data around the world. Traditional file transfer methods just won’t cut it. The days of shipping hard drives around the world are long gone. Now the creative industry must turn to new accelerated secure file transfer systems to ensure that files of any size or format will reach its endpoint. In this article, we’ll look at some real-world examples in the creative industry and highlight how traditional data transfer methods fall short in keeping up with the growing need for large and fast file transfer. You’ll learn how accelerated file transfer is revolutionising the way data moves and changing the game for creative professionals.
Transmission Control Protocol and User Datagram Protocol
How is data transferred?
Let’s start with a quick snapshot of how the main communications standards used to move data from one place to another. Instead of transferring data or files as a whole file or object, it’s broken down into smaller parts called ‘packets’. These packets are then sent across the network or the internet and reassembled at their destination. The two main communication standards used to achieve this are Transmission Control Protocol (TCP) and User Datagram Protocol (UDP).
What is better for transferring large files: TCP or UDP?
TCP is the standard protocol for delivering data across a network. Think of it as a delivery driver who likes to double check everything before moving onto the next package. Imagine you order a self-assembly cabinet that arrives in ten different sized boxes. TCP acts like this driver, it will tell you it's delivering one of the boxes, waits for you to tell it you're ready, sends it and waits for you confirm you got the package. This process continues for each packet in a transfer until the whole file or object has been completely transferred.
Google Drive, We Transfer and Dropbox are recognised file sharing tools and are often the go-to methods when you need to transfer a file that is too big for an email attachment. Most of these solutions rely on TCP, which is also the underlying network communication protocol used in all standard Managed File Transfer (MFT) protocols. Dropbox does offer some UDP functionality, however this is only available when you are hosting your own Dropbox server. While these tools might suffice for the occasional large file transfer they remain at heart consumer grade tools. You are giving up crucial security practices such as content scanning, filtering and logging for the sake of an easy fix . In some cases the T’s & C’s of these personal file transfer tools contain worrying ownership clauses, which as creative content generators are unpalatable in the extreme. When it comes to the creative industry, where work is valuable and often confidential, you have to ask, is it worth risking a potential data breach?
UDP on the other hand, operates more like the cashier at Lidl, quickly scanning as many packets through checkout as fast possible, without checking if you’ve bagged the item, or if any packet has fallen on the floor. From this simple analogy, we understand that UDP offers a faster point-to-point transfer because it doesn’t require the extra steps of verification that can slow down the data transfer process.
This makes UDP particularly beneficial for environments with unstable connections, such as mobile networks or satellite links, where constant rechecking of packets (as TCP would do) would severely slow down the speed of transfer. A select few vendors have built proprietary protocols based on UDP to move data significantly faster. Their protocols can intelligently manage data flow, ensuring that while the connection is flooded with data, other network traffic is not impacted. Depending on the network conditions and bandwidth available, this approach can boost transmission speeds by up to 1,000 times compared to traditional methods, making UDP the go-to choice protocol for businesses seeking accelerated file transfer.
Accelerated File Transfer
Earlier, I mentioned accelerated file transfer is the go-to for organisations in the creative industry - but what exactly is it? Well, if you can’t tell by its name, accelerated file transfer refers to technologies and methods that significantly speed up the process of moving large files over a network. Leveraging UDP protocols means that Accelerated File Transfer tools don't waste time waiting for delivery acknowledgement from their endpoint - instead pushing large file volumes through as quickly as possible and at a reduced latency across long distances compared to TCP tools.
While there are several enterprise solutions designed for large file transfers, the hyper-fast, secure file transfer solution, Fortra FileCatalyst has emerged as a market leader in the industry.
FileCatalyst is designed as an internal file transfer solution, that allows you to move large files quickly around your IT estate. The solution is comprised of several elements that work together to deliver your files as fast and securely as possible. The Transfer Server, which serves as the primary file system and can be deployed for on-premises, on bare metal, or in cloud hosting environments such as AWS or Azure. To interact with this Transfer Server and manage your files transfers, there are two options. The first, is the Transfer Agent, which is a locally installed transfer client operated by you, through a web browser. It offers a user-friendly interface that will be familiar to anyone who has used an FTP/SFTP client before.
The second option starts to look at automation through a feature called HotFolder. This option allows for monitoring of a directory on the local system and has programable actions once new files are added. For example, if you’ve just finished editing the latest episode of a show and need to send it back to HQ before calling it a day you don’t need to manually trigger a file transfer. Instead, you can just drop the file in a folder called ‘Completed’, and HotFolder will automatically transfer it back to the Transfer Server without any user input required to connect.
The final element of the FileCatalyst is central management, known as FileCatalyst Central. This can be connected into all nodes within the solution, allowing you to configure and monitor all file transfers happening in real-time. If you deploy this solution across a wider user base, Central is a game changer. Now, we'll take a look at some use cases within the creative industry where accelerated file transfer is fixing your biggest pain points and revolutionising the way we work.
Transferring Large Files in Film Production - Quickly
Scenario: You’re part of a film production crew, working on location in Sydney, Australia. You’ve captured a crucial shot for the film and need to get it back to your headquarters 10,553 miles away in London, for the editors to begin their work. Time is money, and you need to move fast to meet your deadline.
Uncompressed DPX Data Rates - 4096x2304 Full Resolution
In this case, let's say you are using the ARRI Alexa 35, a 4K professional film camera. Depending on the settings, per second of film could be over a gigabyte of data. On this occasion, your shot amassed to a hefty 50GB of data.
Using a traditional TCP-based transfer, with a 500 Mbps connection, you’d be looking at around 2 days and 15 hours to transfer the file. That’s two days of lost productivity, and a two day delay before any additional work can be conducted on the files.
Leveraging an accelerated file transfer UDP platform like FileCatalyst, you would be able to complete that same file transfer in just 15 minutes. That is around 258x faster than traditional TCP technology. This incredible speed is achieved through FileCatalyst’s UDP technology, combined with its proprietary compression algorithm it uses to improve transfer time.
Transferring Files Around Global Studios for Video Game Development
Scenario: Game studios, particularly those creating blockbuster AAA experiences tend to no longer be housed in one office. Instead studios have opened up a global array of offices and partner with specialist developers for specific purposes. Developed during the height of the Covid pandemic, Borderlands: Tiny Tina’s Wonderlands saw parent developer Gearbox work with 40 external partners.
With top talent and specific expertise spread out across the world, game studios need to find effective ways to distribute and collaborate on game design. Game developers need to find a way to move files globally for testing by your staff simultaneously. Each build is a massive file and must be tested before launch day.
In this scenario, using traditional methods to send these large builds would be inefficient and time-consuming. With FileCatalyst, you can take advantage of its HotFolder technology to send out your gaming files to all your team, at the same time, taking full advantage of the UDP based technology, saturating your bandwidth and ensuring you’re fully utilising your internet speed, delivering your project files as fast as possible.
In an industry that now brings in larger revenues than the music and film industries combined, time really is money and the efficiencies provided by accelerated file transfer are increasingly vital in a globally distributed industry.
Fast File Transfer in Remote Locations for Journalists and the Media
Scenario: Journalists regularly find themselves in remote locations, or on the frontline of natural disasters or battlefields where communication infrastructure is in ruins... or quite simply didn’t exist in the first place.
Every news organisation is in a breakneck race to cover the latest story first. In the field, when you are covering a breaking news story from a remote location, many production teams will need to get video footage back to the newsroom with spotty internet coverage, or potentially satellite internet. Time is running out - your material needs to reach the editor before the evening news broadcast.
When you are facing satellite internet as your only option to transfer your footage back to your editor, you need a technology that will transfer data across the world fast, even in difficult conditions such as packet loss over satellite communication. In such conditions, traditional transfer methods cannot be relied on. Packet loss, short breaks in connectivity and low bandwidth will cripple traditional TCP file transfers.
A solution like FileCatalyst excels in transferring files under the pressure of challenging and remote environments. How? The solution maximises bandwidth connection and sends data in chunks, one part after the other, then verifies with the destination to check for any missing parts and resends them if needed.
If you’re sending files from Antarctica, the priority is to get as much of the data across as quickly as possible before satellite coverage changes and the connection is disrupted, this could be the difference between your article or video going out on that evening’s news or having to be dropped as it didn’t get back to studio in time.
Transferring 3D Renderings or other Architectural Projects
Scenario: Architecture, Engineering, and Construction (AEC) teams are often juggling multiple projects remotely, all of whom need to collaborate on detailed CAD drawings (often exceeding 200MB), blueprint plans, drafting files, and high-resolution 3D renderings. And, in a lot of cases, these enterprise teams are spread across the globe. Emailing architectural files is not only insecure, but simply impractical due to size limits.
For AEC projects that rely heavily on digital file transfer processes to share large volumes of data, manage project tasks, and meet tight deadlines, a fast and reliable file transfer solution is essential. FileCatalyst provides AEC teams with ultra-fast, multi-gigabit transfer speeds, allowing teams to share large project files and data sets quickly, regardless of their location. Not only that, but FileCatalyst offers a central storage location for one or more “web folders”, where files can be organised, managed, and easily accessed by everyone, from anywhere, involved in the project.
Find out how Fast File Transfer Could Help Your Business
In a world where data plays a huge role in every industry, for the creative sector, fast and secure file transfer of big data is more critical than ever. Whether you’re working on the next blockbuster hit, distributing a game build to globally, or sending breaking news footage from remote locations, the ability to move large files quickly and securely is essential.
With most of us being able to relate to the phrase “time is money”, having a confident solid solution you can rely on is key. Since 2004, Pro2col have been helping businesses within the creative industry find their perfect solution tailored to their needs. If you’re facing similar challenges to the ones we’ve explored in this article or have a challenge of your own you’d like to discuss, get in touch with one of our technical experts and start your journey towards accelerated file transfer.
|
About the Author |
|
|
Sean Holdstock is the Technical Consultant at Pro2col. As a Technical Consultant Sean keeps up to date with trends and changes in the IT marketplace to give him a clear outlook on how technology evolves. He’s not afraid to challenge his knowledge by learning new code languages and always looks for ways this could help his colleagues and customers. Find out more about Sean here. |
