Some Thoughts on TCP Speeds

As a consultant in File Transfer technologies, a common complaint that I find myself having to address is the speed that a file travels at between two servers. Generally speaking, many people expect that if they have two servers exchanging files on a dedicated traffic-free 1 Gbps line, then their transfer speed should be somewhere close to this.

TCP Vs UDP

One of the first things to consider is the way that TCP works compared to UDP. No matter which protocol is used, data is broken into packets when being sent to the receiving computer. When UDP (User Datagram Protocol) is used, the packets are sent ‘blind’; the transfer continues regardless of whether data is being successfully received or not. This potential loss may result in a corrupted file – in the case of a streamed video this could be some missing frames or out of sync audio, but generally will require a file to be resent in its entirety. The lack of guarantee makes the transfer fast, but unless combined with rigorous error checking (as per several large-file-transfer vendors) it is often unsuitable for data transfers.

In contrast, TCP (Transmission Control Protocol) transfers data in a carefully controlled sequence of packets; as each packet is received at the destination, an acknowledgement is sent back to the sender. If the sender does not receive the acknowledgement in a certain period of time, it simply sends the packet again. To protect the sequence, further packets cannot be sent until the missing package has been successfully transmitted and an acknowledgment received.

Deliverability over speed / Calculating the Bandwidth Delay Product 

This emphasis on guarantee rather than speed brings with it a certain degree of delay however; we can see this by using a simple ping command to establish the round trip time (RTT) – the greater the distance to be covered, the longer the RTT. The RTT can be used to calculate the Bandwidth Delay Product (BDP) which we will need to know when calculating network speeds. BDP is the amount of data ‘in-flight’ and is found by multiplying the Bandwidth by the delay, so a round trip time of 32 milliseconds on a 100Mbps line gives a BDP of 390KB (data in transit).

Window Scaling

The sending and receiving computers have a concept of windows (‘views’ of buffers) which control how many packets may be transmitted before the sender has to stop transfers. The receiver window is the available free space in the receiving buffer; when the buffer becomes full, the sender will stop sending new packets. Historically, the value of the receiver window was set to 64KB as TCP headers used a 16 bit field to communicate the current receive windows size to the sender; however it is now common practice to dynamically increase this value using a process called Window Scaling. Ideally, the Receive Window should be at least equal in size to the BDP.

TCP speed fluctuations

The congestion window is set by the sender and controls the amount of data in flight. The aim of the congestion window is to avoid network overloading; if there are no packets lost during transmission then the congestion window will continually increase over the course of the transfer. However, if packets are lost or the receiver window fills, the congestion window will shrink in size under the assumption that the capacity of either the network or receiver has been reached. This is why you will often see a TCP download increase in speed then suddenly slow again.

TCP Speeds Diagram

A quick calculation…

One point to remember is that when talking about bandwidth, we tend to measure in bits; when referring to storage (window size or BDP) we are measuring in bytes. Similarly, remember to make allowance for 1Mb = 1000Kb, but 1MB=1024KB.

So, given this, a 1Gbps connection with a 60 ms round trip time gives a BDP of 7.15 MB (1000*60/8/1.024/1.024). As I mentioned, to fully utilise the 1Gbps connection, we must increase the Receiver Window to be at least equal to the BDP. The default (non-scaling) value of 64 KB will only give us a throughput of 8.74 Mbps: 64/60*8*1.024 = 8.738Mbps

So what can you do to speed up the transfer?

Logically, you would probably want to have the largest Receive Window possible to allow more bandwidth to be used. Unfortunately, this isn’t always a great idea; assuming that the receiver is unable to process the data as fast as it arrives you may potentially have many packets queued up for processing in the Receiving Buffer – but if any packet has been lost, all subsequent packets in the receive buffer will be discarded and resent by the sender due to the need to process them in sequence.

You also need to consider the abilities of the computer at the other end of the connection – both machines need to be able to support window scaling and selective acknowledgements (as per RFC1323).

Another option that you can investigate is the ability of several products to perform multithreading. Multithreaded transfers theoretically move quicker than single threaded transfers due to the ability to send multiple separate streams of packets; this negates somewhat the delays caused by having to resend packets in the event of loss. However the transfers may still be impacted by full receive windows or disk write speeds; in addition any file that has been sent via multiple threads needs to be reassembled on an arrival, requiring further resources. In general, most large-file transfer software is written around multithreading principles, or a blend of UDP transfer with TCP control.

Finally, consider other network users – when using large Receive Windows, remember that as the amount of data in transit at any time increases, you may encounter network usage spikes or contention between traffic.

 

If you have any questions about the speed of your file transfers or your chosen file transfer technology and infrastructure design give our team of experts a call on 0207 118 9640.

Download a Comparison of 8 Leading Managed File Transfer Solutions!

 

MFT_Comparison Guide Img

In this essential pack you’ll also find…

 

  • Key features and frequently asked questions

  • Other business policies that will need to be considered

  • Access to additional resources

  • Side by side comprehensive comparison

    * Updated to include new vendors (October 2015)