Excel Hell II: If the sickness can't be fixed, it must be contained

Opinion Three years ago almost to the day, El Reg reported on how Excel had contributed to pandemic chaos. Some 16,000 cases had vanished between labs and the UK health service due to a combination of file format failures, outdated software, and lack of error messages. Now comes a fresh hell.

Last week, the Curse of Excel struck the health service again, this time culling a cohort of trainee anesthetists by marking them as unappointable whether they'd aced their applications or not. The reason – yet another botched data transfer between spreadsheets, in a manual amalgamation of non-standard sheets from multiple sources.

Once is a chance. Twice is enemy action. When you get to so many newsworthy spreadsheet errors they need a spreadsheet of their own, it's systemic failure. As we said at the time of the COVID-19 cock-up, the modern spreadsheet as exemplified by Excel is a bubbling swamp of bad ideas, straining complex data into a crude, highly constrained grid, intermingling that data, its inter-relationships, and operators through a letterbox of an interface.

For an industry hell-bent on serving us AI-generated experiences through hyper-real virtual reality simulations, this insistence that all business data is best represented by a big ol' 2D matrix you have to manipulate directly – and that this is appropriate for everyone from fishmongers to financial directors – is bleak farce.

We get the message. Microsoft owns the market in 2023 so can't be bothered to fix this concept from the 1970s, despite decades of evidence that it's a danger to business. Perhaps it just can't think of anything better. Spreadsheet reinvention is about as sexy as COBOL compiler optimization in the hot, status-driven world of career paths. If you can't get the talent, you don't get the awesome. So let's make fixing spreadsheets the sexiest tech task on the planet.

Ironically, medicine is an excellent guide here. Being a doctor is the legitimate ambition of some of the smartest brains around, despite it being mostly a task of fixing mistakes in biology. Fixing Excel might not live up to, say, oncology, but coming up with ideas that save untold pain and money across hundreds of thousands of businesses does have its own appeal.

Cancer is as good a place as any to start. It's apparent by now that we can't conceptually "cure cancer" as a single disease because it's a huge array of misdirection of basic biological systems essential to life. All cancer treatments keep those systems going while finding ways to nobble each misdirection. So let's treat Excel as the basic biology, and all its error-inducing features as its vulnerabilities to going cancerous. We can't fix everything at once, so how would we cure the latest sickness, the spreadsheet that, left untreated, would eat medical careers?

• Musk's first year as Twitter's Dear Leader is nigh
• The home Wi-Fi upgrade we never asked for is coming. The one we need is not
• Getting to the bottom of BMW's pay-as-you-toast subscription failure
• Last rites for the UK's Online Safety Bill, an idea too stupid to notice it's dead

In this case, assume the recruitment system has been running successfully for a while – probably since spreadsheets became ubiquitous. There's a large body of good data from the years of healthy functioning: how many applicants succeed, what the spread of interview scores are, how the two are related. All medical diagnosticians work by identifying significant deviations from good health; a sexy way to do this with an Excel-based process would be to train an AI with historical data and ask it to alert on deviation.

An even better way would have the system learning the data collation methods well enough to spot row or column transpositions. Excel can't know which transpositions are valid or bad, any more than DNA can tell whether a transposed sequence is an essential part of a gene regulatory network or one that contributes to a cancerous mutation.

A trained eye, human or machine, can diagnose the error from its outcome, allowing the development of a fix.

The very best thing, and one that would cure a lot of digital cancers, would be to design and package that AI so it could work with minimal expert configuration across multiple systems that had historically good behaviors to train on. It could even be applicable to migrations, all of which go through a phase where the new system closely emulates the old. There are many ways forward for such a tool – suggesting optimizations or fixes, generating automation scripts, finding verification methods – that make the idea worth exploring.

We've already adopted biological metaphors in security – viruses, worms, infections – so extending that to the rather buggy but inescapable systems that sit at the heart of digital evolutionary history seems no great stretch. At least no great stretch to realists; vendors whose projected image depends on not admitting unfixable naffness in current products may have problems here. That's OK, someone else can do the job.

Becoming doctors of the digital will never give techies the status of healing people, nor should it. But it will make life better for fishmongers and financial directors, it will let us all finally accept the truth that there's a lot of crap floating around in the industry and always will be, and you can smell the money from here.

Sadly, COBOL compiler optimizers are on their own. ®