Four Ways AI Machine Learning could be implemented in MFT

AI. Artificial Intelligence. Unless you have been living under a rock for the last year you’ll have heard about AI. ChatGPT projected AI into the public consciousness when OpenAI released it to great acclaim in November 2022. Since then, AI has dominated discussions in the world of tech. Can it drive efficiency? Are jobs at risk? How can it be used and is it safe or Skynet in disguise?

 

Similar discussions are happening in boardrooms across the globe as businesses grapple with how to deploy AI within their operations or examine the potential threat to their business model that is posed by AI.

 

All the while, in the ever-changing world of IT, we see AI (Artificial Intelligence) being included in many solutions we use every day. MFT (Managed File Transfer) is no exception to this and is an area that could benefit from AI inclusions; however, I think it’s important to first think about what we mean by AI.

 

AI in its current form is not true artificial intelligence but is actually a collection of different machine learning paradigms with machine learning being one branch of artificial intelligence science. Deep learning and neural networks are others worth noting; and we will be touching on some of these concepts in this article and how they could be used in MFT.

 

All AI models require the need to be “trained”, which can be either supervised or unsupervised. When we say trained, we mean taught the correct way of interpreting the data it’s ingesting.

 

So, if we were to introduce AI or machine learning to File Transfer systems what would it be best suited to do? How could it improve MFT infrastructure? We’ve identified four key areas where AI could be implemented in MFT.

 

Let’s start by looking at reinforcement learning. This model is a learning method that rewards positive behaviour and punishes negative behaviour. This model uses trial and error to work out the best way to achieve a result by defining what is positive and what is negative.

 

For example, let’s say we are looking to train the model to reduce the time an audio file takes to transfer by SFTP (SSH File Transfer Protocol). In this example, we will hypothesise that the file is an MP4 of 1GB in size. We could make use of a point system to define the positive and negative behaviour – so, if the baseline transfer could be 1 minute; we would then award 1 point for a successful transfer of less than 1 minute and deduct 1 point for a transfer which took more than 1 min.

 

The AI agent being awarded and deducted would continue to try different variations of protocols, compression, key exchange etc in order to maximise its score.

 

Ultimately, after numerous attempts the AI agent would not be able to increase its point value beyond a particular maximum, which would derive the best possible connection and transfer configuration for that file.

Different AI models are more efficient than others at interpreting types of data. As an example, if we were looking for pattens within logging data, we might want to investigate using a support vector algorithm.

 

These types of models are “supervise” learning models and work by humans providing the model with a training set - this being a set of pre-categorised data items which it can use as a basis to learn how to define the difference between two sets of data and how to categorise it.

 

If you were to apply this to the realm of MFT, a support vector algorithm could be used to perform pattern analysis on inbound and outbound transfers and consequentially tune future workflows to be optimised to better utilise system resources, such as bandwidth or CPU processing.

A major field of importance in MFT is data compression. The art of determining how small can we make our data, to reduce the time a transfer takes.

 

If we consider this from a machine learning point of view, we could take the original file and apply an algorithm to identify the optimum compression method and level. By using a neural network to identify the data type - for example video or text - and the parameters of the raw data itself, it could then provide the optimum compression method and level.

Let’s look at this idea in more detail. In this example we we have a video file. The file has the following parameters or attributes contained within in the file’s metadata:

 

The neural network would identify the compression algorithm with the smallest result, perform the compression and then compare the original file with the compressed file. Comparing the original with the newly decompressed file would allow it to determine if there was any data loss during the compression. The AI model would then be able to rank the compression algorithms based on the percentage of loss. This could then be used to build a logic tree that would make decisions based on the data type and other parameters we have spoken about above.

 

One area which could see a huge improvement from using AI methods like the ones we have spoken about above would be Service Level Agreement (SLA) reporting in MFT systems. Using transfer log data and a neural network, we could train this AI model to identify repetitive transfers based on the source and type of files transferred.

 

This would be matched against the average time of day you receive these files historically. This would allow us to create an SLA with +/- limit of around 1 hour. If this is breached, the system could trigger an alert to administrators to let them know that they have not received the expected files.

We could take it further - if the SLA is breach it could trigger a workflow first which could run a diagnostic connection test to make sure that not only can you reach the sender, but you are able to receive files. Then alert the administrator that the system is healthy and that the issue is external in nature. Where the problem is with the connection, it could detail that either the sender cannot be reached or that your system is having issues.

 

Moore's Law dictates that computing power is set to double every two years. The advances in AI (together with the funding levels that come with mass adoption) means we should question whether Moore’s Law will remain in place.

So just how quickly are we likely to see AI in the MFT marketplace? It’s hard to say, but we predict it will arrive sooner than you might think. A number of the concepts I have spoken about in this blog may just be around the corner or who knows, they could be further away. But what is a guarantee is that MFT is not going to avoid the AI revolution which is enveloping IT now.

 

About the Author     Sean Holdstock is the Technical Consultant at Pro2col. As a Technical Consultant Sean keeps up to date with trends and changes in the IT marketplace to give him a clear outlook on how technology evolves. He’s not afraid to challenge his knowledge by learning new code languages and always looks for ways this could help his colleagues and customers.  Find out more about Sean here.