I’m sure that just like me, you have known technicians who have come up with often ingenious ways to address the day-to-day business challenges that we all encounter. Whilst it’s great to have these people around who can provide a quick solution and support it when needed, it’s far too easy to become dependent upon them – then one day they are no longer available to you, having moved on from their role.
Broadly speaking, we rely on these people to have documented their solutions so that someone else can pick up where they left off if needed. Although this is not always the case, it’s not the only area where you can come unstuck with using a homegrown solution to transfer your files; we’ll dig into this a bit deeper once we’ve considered what constitutes a homegrown system and why we might be using it.
Just like everything else that we ever do, file transfer solutions are either tactical or strategic choices. Often they start off as a tiny tactical plan to address an immediate issue – for example, you might just need to send a file to a trading partner once a week. They have an SFTP server, so you download a freeware client and send the files as you need to. A little later, you decide to automate the process with a simple script using a crontab or Windows task schedule. It runs on a server and doesn’t need anyone to look after it – this is a typical tactical solution that we see put in place in many environments. Soon, a second transfer is needed and then a third and so on; each time, your technician updates the transfer script or adds another schedule, sometimes adding logic into a script while at other times simply replicating it.
There was never an intention to invest in a strategic solution, but the more complicated the tactical solution becomes, the more the strategic solution is needed. Until the strategic solution is in place, there is a real risk of running into one of the main pitfalls of homegrown tactical file transfer solutions. In no particular order, they are: Management, Reporting, Security and Cost. Let’s look at each of these in turn.
Poor Management System
Who looks after a homegrown system? In my experience, it tends to be a technician who is an expert at scripting and most likely senior in the business. They will have started the process of building the system and with each subsequent request developed the system to include complex schedules, pattern matching and logic. They are likely to have built a library of scripts for different workflows and is also likely to be the only person who really understands them. If a file has not been transferred, they are likely the only people to see.
The problem here is that if they are unavailable to look at a failure, or even if they are available but unable to resolve an issue, then there is no recourse for further assistance. Similarly, when they are unavailable, someone else has to understand the process of adding a new workflow – how are the workflows being recorded? In a document somewhere that everyone has access to, or just in the crontab or schedule alone? If the latter, what happens if the schedule is lost or corrupted? Homegrown systems rarely enforce their administrators to follow any specific rule for documenting the workflows, mostly speaking the documentation goes as far as some comments in a script.
In each of these potential issues, the first concern is most likely time – how long will it take to put back. For how long can you afford to be without a certain transfer? Will that have a knock-on effect to other systems? Almost certainly the answer to this is yes, otherwise why transfer files at all?
You also need to consider that without proper management of a system, you cannot be sure that it is doing everything that it is supposed to do. If you need to know whether or not you are transferring files to a given third party but no one is available who understands the system or can find the documentation, will that harm the reputation of the IT department (or the company itself)?
MFT systems can address this by centralising management and standardising documentation around your workflows. Visibility can be granted to business users through non-technical means – for example opening a web page rather than reading a technical script. Although there may still be technicians managing the system, it will be a system that is properly supported and no longer requires the same degree of specialist knowledge (knowledge which can be found in a manual or training program).
Lack of Reporting Funcionality
Where it’s possible, most homegrown systems include some form of logging. Frequently this is in the form of a flat file to which a line or two is appended every time there is a transfer. Sometimes the client being used does not provide the capability to log anything more useful than a timestamp, filename, and result. If you ask the technician how many files have been sent to a specific third party or was a certain file transferred last week, it may require some in-depth investigation on their behalf in order to pull information which may exist in a number of logfiles. Homegrown systems generally do not have a reporting system but instead rely on the logging for all of the information that the administrator needs. Nor do they tend to have any kind of audit trail where the administrators can easily see who performed an action, for example adding or removing a workflow.
This can be a problem when faced with an audit of either transfers or administrator activities. Certain regulations (for instance GDPR, SOX or PCI) require you to record this information and, depending upon your business, retain the records for a certain period of time – not having this information could potentially result in a financial impact for the business.
Off-the-shelf MFT systems include reporting functions that take care of this for you, from the movement of files to the tracking of administrator activities. In general, reports can be executed as and when required, or scheduled as a workflow in its own right within the system. These scheduled reports are then invariably sent to the people who need to see them. Many file transfer systems allow administrators to create custom reports which are tailored to specific purposes (for example, all activity pertaining to a certain third party or folder), additionally there are reporting products that can be used to further refine reporting capabilities which flawlessly integrate to the file transfers systems own reporting function.
Security Threats of Homegrown File Transfer
The security of a homegrown system is probably the main driver behind the need to consider moving to an established strategic solution. Security is a blanket term that covers a few areas that you should be aware of. Some of these you will already have considered and put something in place accordingly.
Consider first the access to the homegrown system; most likely all system administrators who have access to the server where the workflows operate can make changes, either deliberately or inadvertently. We’ve already mentioned lack of reporting, so there won’t be any visibility on who makes those changes. Security also covers credentials – not just passwords which may be either hardcoded in a script or else in an easy to find .netrc file – but also keys and certificates. These have to be stored as files somewhere that the homegrown system can access them and could be at risk of deletion by cleanup jobs for example. Security includes cipher and protocol version selection – during handshake processes the client and target server do not necessarily select the most secure of either by default, which can be an issue as older ciphers and protocols are considered less secure over time. Many workflows (especially those to financial organisations) require that you PGP encrypt the files for security; this means that someone needs to be mindful of key expiry dates and manage them accordingly. A final consideration is whether there is a continuity solution for the homegrown system; most likely this will be something like ‘restore from backup’, and the system is unlikely to be highly available.
Any of these security concerns can be seem as being detrimental to your business and can easily result in security breaches or unplanned downtime for either a specific business process or else the entire system. Regulatory compliance may also be a concern especially if any personally identifiable data goes through the system; GDPR has some rules around controlling access which if transgressed carries financial penalties
Replacing a homegrown system with a reliable MFT system allows you to control and regulate administrator access, select the protocol versions and ciphers, and store credentials securely – passwords are stored encrypted, keys and certificates are kept in key vaults. Administrators can be informed when keys approach their expiry and can proactively change or rotate them. In addition, MFT systems invariably come with some form of high availability to keep downtime to an absolute minimum.
Higher Costs in the Long Run
One of the biggest mistakes that people with homegrown systems make is to believe that their system is cheap or even free – there are, after all, several freeware solutions as well as writing your own. The problem is that when it comes to support, you get what you pay for. In a homegrown system, it is very much a case of ‘the buck stopped here’ for the technician managing it. Mostly speaking, if he gets stuck then his only recourse will be internet forums; naturally, he cannot share too many details about his issue, so it may be some time before someone offers a solution. This can mean many hours wasted trying to resolve an issue, hours that have to be accounted for. Just the administration aspect alone may be time consuming to a point that it becomes someone’s main activity, and as already mentioned, in homegrown systems this tends to be the job of a senior technician.
There may be a financial cost for workflows failing to run as well, this might be contractual or even represent missed business. If ever there is an exposure through using a homegrown system then not only could there be fines, but there will almost certainly be reputational damage too.
MFT systems obviously have a price tag that at face value is much higher than a homegrown system, but they bring benefits in terms of support, reduced time for workflow management, and providing a simplified interface that doesn’t require the same skill set as the homegrown system, freeing up time for more technical staff.
Still thinking about a homegrown file transfer solution?
Homegrown systems have a place in the IT world, but just as a tactical solution. Frequently, effort that you put in to developing and expanding this solution simply makes the system more expensive to run and more of a security risk. In many ways, it’s not too different to customising a car – adding a roof rack and a roof box to add extra capacity is a good tactical solution, but when you start tying luggage to the bonnet, then it’s time to invest in a van before something bad happens.
Pro2col have assisted many customers in their journey from a homegrown system to MFT, from reviewing their needs and business processes, to product selection, implementation and support. We understand the needs of businesses of all sizes and work with hundreds of businesses on a daily basis. Book a call with one of our experts and let Pro2col take care of everything.
|
