The Magical App

...the innovative process used to make life and death guideline decisions.

The National Institute for Care and Excellence “provide national guidance and advice to improve health and social care” in the UK. During the pandemic, the recommendations NICE made were crucial in determining which treatments were used in the fight against COVID-19. NICE produced a ‘rapid guideline’ document which was very similar to the ‘living guidelines’ document the World Health Organisation produced.

They recommended against using Ivermectin outside of clinical trials, and there’s a similar pattern to that recommendation. First, let’s start with the rationale which is quoted below. The decision was first published on December 1st 2021.

Overall, there is a high degree of uncertainty about whether ivermectin is more effective than control for managing COVID-19 in hospital or community settings. The panel raised concerns about the quality of the studies on ivermectin.

Sound familiar?

The data they’ve gathered to make this determination is presented back to “clinicians and patients” through the same “MAGIC app” the World Health Organisation uses in presenting their own recommendations. That same app is used by the UK, the WHO, the British Medical Journal, and the Australian “Clinical Evidence Taskforce’.

Same process, same results.

Taken at face value, the MAGIC app is “enabling clinicians and patients to make well-informed healthcare decisions”. In reality, the MAGIC App and the process that underlies it seem to be designed to neatly present cherry-picked findings. A much better ‘at-a-glance’ presentation of data would be to use industry-standard forest plots. The MAGIC App’s purpose seems to obfuscate rather than illuminate.

Let’s look at how.

The interface hides the actual data, something forest plots don’t do. A key example is the app’s “decision aids”. Using this feature, decision-makers can click various endpoints, like All-cause mortality, and be presented with ‘revision style’ flashcards summarising the numbers. If the decision aids shown below were all the data you had on Ivermectin, it’s easy to predict what your regulatory decision on its roll-out might be. Is that the purpose of the so-called MAGIC App?

Ivermectin ‘decision aids’

When you try to get into the data behind these assertions, the app gets in the way. It just wants to give you distilled numerical comparisons. For example, in the three decision aids shown above, we’re not told how big the sample size is, we’re only given an extrapolation of events ‘per 1000’. We’re told there were 0 fewer deaths in the Ivermectin treatment group, but we could be looking at a sample size of 10 in just one study for all we know. Confidence intervals? Forget it.

Dare I say it, but for a moment, forget the data itself. I want you to look at this purely as a piece of communication. In a little more than double the screen real estate of the Magic App’s ‘decision aids’, look at how much information is succinctly communicated in a forest plot. Below are the findings from the 2021 Bryant et al study on Ivermectin, published in the American Journal of Therapeutics two months prior to the November ‘MAGIC App’ data used by NICE.

Bryant et al 2021 forest plot on all-cause mortality benefit of Ivermectin

If we were trying to help decision-makers get quickly up to speed with ‘what the data currently says’ regarding….anything…then you could do much worse than to use a forest plot like the one above. Why not use this brilliant form of data communication? Had decision-makers instead been presented with the graphic above, along with a link to the full text of the study, might their decision have been different? As chance would have it, one peer review comment on the NICE decision raised this exact question. See if you can make any sense out of NICE’s response…

‘An organisation’ asks if NICE have considered the Bryant meta-analysis on Ivermectin & NICE notes their action taken.

I’ve unleashed a can of worms there. I’ll return to this in due course…for now, let’s return to our Magical App.

Frustrated with the lack of detail, curious users might click on ‘Research Evidence’ and here we can see some more detail. We are now given the number of studies used to calculate the effect and the total number of participants. We have many more words thrown at us, but the scale of the data used to make these conclusions is still somewhat elusive.

Unlike a forest plot, where we can quickly look up the studies used, in our Magic App we can’t do any further interrogation of the claims because the studies used to support the claims are not listed. As a matter of fact, the reference used on this particular “All-cause mortality” endpoint points us to a dead end. The “source of evidence” is cited as a systematic review called “Ivermectin versus standard care for COVID-19”, but there’s a problem; it has no date, it has no author, and it doesn’t have a journal or a reference number to help us look it up. Have a look at the image below, I’ve highlighted the issue so you can see it more clearly.

There are 18 endpoints shown in the ‘MAGIC app’ and every single one cites the same reference. Try as I might, I can’t find this study anywhere at all. Decision-makers wanting to interrogate the data properly would now be forced ‘off-piste’, in my case I have filed an FOI request to seek access to the document. It appears that NICE did their own systematic review, referenced it to support their claims, but failed to publish it in any of the places you’d expect a systematic review to be published.

PRISMA have very clear guidelines on systematic reviews and pre-registering is a requirement. It’s so that readers can see what you intended to study and how, but this paper doesn’t appear on ClinicalTrials.gov or the WHO trial search. By not publishing their paper, we can’t check the conflicts of interest of the authors. Who are the authors? How did the authors choose which studies to include and which to exclude? If there’s no pre-registered method, the systematic review is open to bias. Authors could choose studies arbitrarily, picking only those that suit a particular viewpoint. Since they are not listed as authors, sloppy or biased methodologies are not linked back to them.

Just a few of the PRISMA guidelines on systematic reviews

Instead of a pre-registered, published systematic review, NICE instead provide a document with an entirely different name; “The effectiveness and safety of ivermectin for acute symptoms and complications of COVID-19 evidence review”. Quite why they reference a non-existent, totally different systematic review to support their claims is something I’ll return to in future articles.

Before we get into the data itself, I want to focus on the process. Why? Because this whole process reminded me of how the World Health Organisation came to its own decision on Ivermectin in March 2021, nine months earlier. They used the same app, and they used the same process. An evidence base is prepared, it’s presented to a panel of some kind using the Magic App, and they make a decision by looking at that evidence as presented through the….limiting lens of the Magic App.

When the WHO recommended against Ivermectin, the evidence used was also prepared in the same strange; some kind of systematic review is put together but not published or peer-reviewed in the ‘normal’ way. It’s then selectively communicated via the magical app, and a regulatory decision can be more easily predicted. When investigating that process at the WHO, I discovered that Professor Andrew Owen, who I credibly linked to the ‘lobby edited’ Andrew Hill paper, had put together the evidence base from which the World Health Organisation ‘Guideline Development Panel’ made their decision.

As much as this all feels familiar, is there anything concrete in the NICE process, which produced identical results from near-identical methods, that brings this all together? For fear that this will become too long, I’ll publish that tomorrow. As ever, your support in sharing this article to create an audience is greatly appreciated.