Imperial College’s COVID-19 Coding Is Unintelligible, “Quite Possibly the Worst Production Code I Have Ever Seen”

In the past politicians may have consulted the Oracle of Delphi. Now they're consulting incomprehensible pieces of code written in C++

It may be surprising to those outside the bubble, but it is no great secret within the academic community that the quality of most computer codes used is very poor. Prof. Neil Ferguson’s group at Imperial have now published their pandemic modelling code, and it is even by those standards exceptionally abysmal.

What is even more alarming is that the published code has supposedly been improved over a number of weeks to be less dire. So what it looked like back in March when Prof. Ferguson was predicting hundreds of thousands of deaths is anyone’s guess.

But code quality is not merely stylistic. It goes to the heart of the problem — if others cannot read what you have written, then they cannot verify that it is correct. I have squinted and stared at this code for a day now and I am still none the wiser what is going on in many parts. Egyptian hieroglyphics may well be easier to interpret. Latin scribal abbreviations certainly are.

But this begs the question: just why are academic codes (in the modelling community the noun is codes not programmes) so invariably dreadful? That is not to say that they are all incorrect, of course. Dreadfully written just means verifying correctness is vexing to the point of impossibility. Or in other words, we simply can’t know if the ICL code is correct. The point of a clearly written code is that others can audit it and understand it.

Almost no scientific programmers are trained as such. Instead, C, or FORTRAN, or whichever language they use, is simply a tool to solve problems with which they learn on the go. Thus no one ever imparts good practices at the beginning. Then, by working alone or in small groups, there is no external driver to improve matters either. I only learned good practices when I started teaching C to others, for example.

Why does this chronic problem arise? Because of the secondary problem sometimes called “publish or perish”. It is actually better for one’s career to publish a result quickly, from one of these aforementioned unverified codes, and then just say in a future paper “oh, yeah, due to a bug, night is actually day”. I did exactly this, albeit unintentionally, back when I was a PhD student. Lewis, Bate, and Price (2015) was essentially revoked by an erratum (Lewis, Bate, Price 2017a), where a bug which had lurked in our magnetohydrodynamics code since about 2005 fatally compromised the results. And those of at least seven other papers.

But in the academic world, I could still list the 2015 paper (and scurrilously omit the erratum) on my CV, and list the second paper I published in 2017 which was the “correct” — whether you believe that is another matter — version. Worse, the 2015 paper still had “impact”, that all important academic metric. Indeed, people still sometimes cite it now, even with an erratum. Which also makes you wonder how often people read the papers they cite. Also, what about all the papers whose authors were not so dispassionate and did not publish errata?

Even worse than this, even if you were an academic who decided to spend some time writing a decent, well documented and verifiable code, there are precious few rewards for doing so. It is all but impossible to make a career out of ‘code papers’ (that is, papers which describe in detail not an algorithm or its results, but how it is implemented on a computer). Believe me, I tried. Instead, work spent improving a code is seen by the rest of the community often as wasted effort. After all, if it “works” and produces “results” then that is all that matters. Whether those results are necessarily correct or not matters less than you might expect.

So not only is there no reward for good coding, there is actually a perverse incentive to not worry too much about the quality of one’s code at all. And that’s before we talk about writing tests. Worse, spending time fixing or improving a code is not only not rewarded, but by reducing one’s output of impactful papers, it actually has a negative effect on career prospects.

In other words, the status quo rewards the production of incomprehensibly dreadful codes like the ICL one, and would actively punish anyone who spent a period of time trying to make it better.

This could be fixed if academics published their codes. But a few minor exceptions aside, there is no compulsion to do so from journals. And there is little internal incentive to do so either — all the more so if others can then see just how bad things are. The narrative around the ICL group’s model in March would have been very different had this code been published with it, after all.

But it must be fixed. There must be an expectation that when the results of a model are published, the tool used to produce them is too. We already require publication of the mathematical description of the algorithm — rightly a paper which did not do so would be rejected. Hence, it’s a small jump to also include the implementation. Some journals already require the publication of input parameters and output data files, so this is, if anything, simply completing the circle.

Some academics will be concerned that they will be “scooped” if they publish their codes. But to be so sees the problem entirely in reverse: yes others might use the code, if it is clear and intelligible, of course. This is better than the other group writing their own, equally secret, likely equally dire, code. To do so is wasted effort. Equally, the argument that the code must not be published lest others use it “badly” holds no water either. Others can equally use your mathematics badly, and we still publish that.

If it was up to me, I would go further still and require those codes to also have published test cases and proof that the version used in the paper passed them. Additionally, we need proper career progression for academics who code.

Imperial College’s Covid-19 code is quite possibly the worst production code I have ever seen. I don’t say this lightly, I have used fluid dynamics codes with a heritage from the 1980s.

This should be a wake-up call for the whole of academia, that it needs to sort out coding practices or, rightly, people will start to question the results. For theoretical astronomy, this is an anodyne problem. For epidemiology, it is, quite literally, lethal.

Source: The Critic

19 Comments
  1. Ivan M says

    The code is in C, not C++, although the files are name with the .cpp extension. A C file is a valid CPP file, but not vice-versa. There are no specific C++ constructs such as class or template.

  2. wardropper says

    I very much doubt that politicians are even consulting “incomprehensible pieces of code written in C++” It’s way above their heads. This is concentrated power-lust, and it doesn’t leave enough remaining mental energy for actual study of anything at all. They simple open their mouths, and the words of “coding” and “scientific” experts come out.

  3. Undecider says

    They can blame it on their Asian programmers.

    1. nick1111 says

      Some of them hardly finished 4 classes

  4. voza0db says

    Luckily for them The Herd doesn’t care about this!

    The MSM is their brain.
    https://uploads.disquscdn.com/images/ccda132f645270ce58611b1a3120ab0dc080f1745b6211a53b9bebe3bfe8758f.jpg

  5. Mensch59 says

    Even worse than this, even if you were an academic who decided to spend some time writing a decent, well documented and verifiable code, there are precious few rewards for doing so. It is all but impossible to make a career out of ‘code papers’ (that is, papers which describe in detail not an algorithm or its results, but how it is implemented on a computer). Believe me, I tried. Instead, work spent improving a code is seen by the rest of the community often as wasted effort. After all, if it “works” and produces “results” then that is all that matters. Whether those results are necessarily correct or not matters less than you might expect.

    So not only is there no reward for good coding, there is actually a perverse incentive to not worry too much about the quality of one’s code at all. And that’s before we talk about writing tests. Worse, spending time fixing or improving a code is not only not rewarded, but by reducing one’s output of impactful papers, it actually has a negative effect on career prospects.

    Imperial College’s Covid-19 code is quite possibly the worst production code I have ever seen. I don’t say this lightly, I have used fluid dynamics codes with a heritage from the 1980s… For epidemiology, it is, quite literally, lethal.

    Damning. And this is what the British government based its need for lockdown — here in Canada called shelter-in-place or “Enough is enough. Go home and stay home” (The Right Honorable Justin Trudeau PM) — on.

    1. Collectivist says

      (‘Removed’ from Alternet):

      I see . . .

      However, for the ‘life’ of, you, and me, I’ve never understood this argument for downplaying the seriousness of this current pandemic:

      “Prior to Covid, how many people die worldwide each year from seasonal flu? Globally, the World Health Organization (WHO) estimates that the flu kills 290,000 to 650,000 people per year. According to worldometer, the number of covid-19 deaths currently is at 281,000.”

      First of all, again, I understand, once again😥, why commercial/government media would sensationalize the crisis. 1) the ‘ratings’ (quantifiable, fixated ears & eyeballs, 24/7) must be astronomical! 2) “Now that we have your attention, on this matter, a word from the mayor, governor and a tweet from . . .”

      “NOTHING else is going on, that you need to know about” . . .?

      However, (unnecessary) deaths are deaths.
      (The, ultimately, unresolvable contradiction of life.)Many sicknesses can lead to death. Death often has myriad social affects. Families are impacted. Destabilized.
      Humans, understandably, want and seek to prevent that, everywhere we can.

      290, 000 deaths a year from seasonal flu – if avoidable – is, or should indicate, a global health crisis, to all conscientious, if not conscious, people, tantamount, arguably greater, perhaps, than obviously related, ever-increasing fascism.

      Ears and eyeballs, at least, must stay fixated on BOTH.

      The FACT that not enough has been done about any of these global ‘crises’, esp. the nature-precipitate, capitalist-compounded health crises, IN THE PAST and:

      “So few seemed to be concerned about them before”

      does not, in ANY way, minimize or negate the necessity to ACT NOW on them. . . .and connect the material and immaterial, political and economic, physical andmental, biospheric dots . . .

      https://www.facebook.com/82

      1. NightriderXP1 says

        the World Health Organization (WHO) estimates that the flu kills 290,000 to 650,000 people per year. According to worldometer, the number of covid-19 deaths currently is at 281,000.

        Here’s the part we’re supposed to overlook. The common flu kills between 290,000 to 650,000 people per year. COVID-19 has killed 281,000 people in 2-3 months. The US has shed nearly 80K since the beginning of April…

      2. Mensch59 says

        Regarding this statistic “Prior to Covid, how many people die worldwide each year from seasonal flu? Globally, the World Health Organization (WHO) estimates that the flu kills 290,000 to 650,000 people per year. According to worldometer, the number of covid-19 deaths currently is at 281,000”: why aren’t we getting the numbers about this season COVID-19 is killing x number of people and seasonal flu is killing y number of people and other acute respiratory infections are killing z number of people? “Why?”, I ask you. I suspect that COVID-19 deaths (the x variable) are including other causes of death (the y & z variables). Why? The maximize the fear factor. To hype up some new virus as “an extremely grave and dangerous new risk to public health”. I suspect that if COVID-19 deaths were separated from deaths caused by the seasonal flu, deaths caused by pneumonia, deaths caused by other viral respiratory infections, deaths from chrronic lower respiratory disease; then COVID-19 would be demonstrated to be an order of magnitude (at least) less lethal. This isn’t “minimizing” the pandemic. It’s challenging those who minimize the damage caused by the lockdown.

        The idea that viral infections are preventable is pseudoscience. Viruses have been infecting eukaryotic cells since life was ancient. Human hubris however is a much more recent and sociopathic (cultural) evolutionary development.

        I fundamentally disagree with @newestbeginning:disqus that having healthy people go back to work and school and play is what the ruling class¹ wants. I am convinced that the ruling class wants to keep us locked down for as long as possible. Why? So they can use the lockdown to establish additional overt totalitarian² social control mechanisms. It’s what they do. The response of the ruling class to every crisis — be it a terrorist attack or warfare or climate-change-causing-destruction-of-ecosystems or pandemics or famines or the threat of societal collapse or an economic depression — will always be in the service of every sort of totalitarian purpose.
        ————-
        ¹ Leftists throw around terms like “the ruling class”, the powers that be, the capitalists, the rentiers, the lords of finance capital, the financiers, the central bankers, the landlords (aka the landed aristocracy), the bosses, the slavers, the rulers, multinational corporations, wealthy & powerful (philanthropic) foundations, etc like they’re abstractions. How about we start naming names instead of talking so much about the politicians?
        ² We’ve already accepted the theory and practice of “inverted totalitarianism” (Sheldon Wolin). Why do we refuse to discuss overt totalitarian measures being implemented? I think it’s because the state is implementing these measures on behalf of the rulers and the vast majority of leftist authoritarians actually crave the power of the state for themselves. Then there are the leftist authoritarians who deny that authoritarian leftists even exist. {shaking my head & shrugging} Leftist statists would rather kill leftist anarchists than leftist statists are willing to surrender their craving for power. Interesting dialectics, eh?

        1. Mike the human says

          Where’s the money in establishing additional overt totalitarian social control mechanisms?
          Most of us already contribute to the economic machine.

          The motivations you project onto “the ruling class” is nothing short of mustache-twirling villainy. Not saying that some people don’t get off on that kind of thing, but what kind of international conspiracy, exactly, are you proposing?

        2. Collectivist says

          “Then there are the leftist authoritarians who deny that authoritarian leftists even exist.”

          Clearly, there are more than a few left ‘authoritarians’, esp. on the internet.

          NOTHING to fear here but fear itself😎

          “I am convinced that the ruling class wants to keep us locked down for as long as possible. Why? So they can use the lockdown to establish additional overt totalitarian² social control mechanisms”

          Of course. Why wouldn’t they? As you said, that’s what they do.

          Here’s what you’re not, dialectically, factoring into your analysis, and why Trumpinc, and other rightwing forces, want to, if they can, END the ‘lockdown’*:

          – As long as most people are home all day, they are not producing surplus value.

          – The government will have to take care of many more of these workers. More of tax and U.S. treasury fiat-generated money will have to be distributed to people; anathema to capitalists who want that money to go
          to corporations.

          – As long as most people are not preoccupied with economic survival – assuming some long term relief is wrung from the plutocrats – many more are likely to be more susceptible, and drawn to appeals, for ALTERNATIVES to the rule of capital.

          1. Mensch59 says

            Trump Inc and other authoritarian & pro-private property forces could end the lockdown and address an unemployment-generated recession/depression immediately by launching a massive public works project. If trillions of dollars are available from the Treasury and the Fed to rescue corporations who are laying off workers, but no percentage of those moneys can be dedicated to a public works project, then it’s a safe bet that Trump Inc and other authoritarian & pro-private property forces have no desire to end the lockdown.
            Maybe they’ve decided to go along with Gates.
            “One of the questions I get asked the most these days is when the world will be able to go back to the way things were in December before the coronavirus pandemic. My answer is always the same: when we have an almost perfect drug to treat COVID–19, or when almost every person on the planet has been vaccinated against coronavirus.” ~ Bill Gates, https://www.gatesnotes.com/Health/What-you-need-to-know-about-the-COVID-19-vaccine?WT.mc_id=20200430165003_COVID-19-vaccine_BG-TW&WT.tsrc=BGTW&linkId=87665522

            Sometimes it’s difficult to escape from the tunnel vision that the huge conglomerate called “the ruling class” only exists to make more money as per “[direct welfare payments from the Treasury are] anathema to capitalists who want that money to go to corporations.” I’m going to hammer this point home. This economic crisis benefits the imposition of social control mechanisms. The lockdown is what’s driving the crisis. The capitalists already have their money. They were and will continue to be bailed out. Now they’re imposing a new social — i.e. political-economic — order.

            Alternatives to the rule of capital — right here, right now — are a pipedream, a fanciful notion. Don’t get me wrong. I love a good fantasy. I’m dealing with what IS, not what IF — even if it means giving up hope (temporarily) for a socialist revolution. I’m simply dealing with the reality of capitalism and (what I see as) the non-reality of implementing the alternative to industrial capitalism regressing/devolving into neo-feudalism.

            It’s like when the watchers on the Titanic spotted the iceberg. The laws of motion had already determined it was too late to avoid a collision. It’s too late to stop what’s happening, even though it would be wonderful if there was a chance for an alternative happening. That being said, industrial capitalism regressing/devolving into neo-feudalism isn’t the end of history.

      3. Mensch59 says

        If the global economy didn’t go into lockdown due to various flu pandemics, why ought this coronavirus pandemic be special — especially when it’s considerably less lethal. I smell a rat. It’s not because humanity-as-a-whole or the ruling class has suddenly & magically become more compassionate & benevolent. “Something is rotten in the state of Denmark” (a line from the play Hamlet, by William Shakespeare, spoken by an officer of the palace guard after the ghost of the dead king appears, walking over the palace walls). I think it’s a mass psychosis to believe that humans can win a war against viruses.
        As I posted to @newestbeginning:disqus — In his book Shrinking the Technosphere, Dmitry Orlov wrote “The reason for extending life for as long as possible, no matter how little sense this makes, is to be found in the abstract teleology of total control. The technosphere’s compulsion is to control everything. It is unacceptable to it for old people to decide when to die all on their own. Death cannot be left up to a subjective judgment; it must be the objective outcome of a technical, measurable process.”
        The technosphere and the state and the ruling class work together to control everything.
        The response of the ruling class to every crisis — be it a terrorist attack or warfare or climate-change-causing-destruction-of-ecosystems or pandemics or famines or the threat of societal collapse or an economic depression — will always be in the service of every sort of totalitarian purpose.

        Why I’m more anarchist than statist is because states gravitate toward totalitarian purposes, authoritarian controls.

        1. Collectivist says

          “I think it’s a mass psychosis to believe that humans can win a war against viruses.”

          I think it’s ludicrous to expect people not to seek to survive viral infections.

          “Why I’m more anarchist than statist is because states gravitate toward totalitarian purposes, authoritarian controls”

          That’s your prerogative.

          Let’s see . . .Ah, Paul Street:

          https://www.counterpunch.org/2020/05/08/live-and-let-die/

          1. sabateur says

            it’s the terrain, in which there is NO profit, re money, or power.

          2. Mensch59 says

            Viruses are ancient. Studies at the molecular level have revealed relationships between viruses infecting organisms from each of the three domains of life (archaea, bacteria, and eukaryote), suggesting viral proteins that pre-date the divergence of life and thus infecting the last universal common ancestor.
            The idea that “viral infections” need to be defeated in order to insure human survival is pseudoscience.

          3. sabelmouse says

            seeing survival from viruses as war, is where the trouble starts

            1. Collectivist says

              Elaborate please.

            2. sabelmouse says

              strengthen the terrain/support the eco system/cooperation is how nature works best.
              there is NO profit in that re money, or power + [ male ] ego/human stupidity

Reply To Mensch59
Cancel Reply

Your email address will not be published.

Anti-Empire