A Level-Headed Look at The Florida Vaccine Study

Last Friday October 7th, Dr. Joe Ladapo, Surgeon General of Florida, released a vaccine safety analysis along with updated guidance from the Florida Department of Health. The press release stated Dr. Ladapo “recommends against males aged 18 to 39 from receiving mRNA COVID-19 vaccines” has caused, predictably, quite the controversy.

As full disclosure: I am someone who has been working as a consultant epidemiologist for the Florida Department of Health, who knows the Florida Epi team and is friends with Dr. Ladapo. At the same time, I was not one of the authors of the study, and I will be expressing my opinions.

https://twitter.com/FLSurgeonGen/status/1578515633159180289

Joe's tweet, announcing the study results and new guidance now has over 90k likes and was accompanied by the additional information: Their “analysis found that there is an 84% increase in the relative incidence of cardiac-related death among males 18-39 years old within 28 days following mRNA vaccination. With a high level of global immunity to COVID-19, the benefit of vaccination is likely outweighed by this abnormally high risk of cardiac-related death among men in this age group. Non-mRNA vaccines were not found to have these increased risks”

First, let’s look at this study:

An 84% increase definitely sounds scary. But 84% increase from what? Many assumed this was compared with unvaccinated controls, but actually this was compared with other 18–39-year-old males who had also received either one or two doses of mRNA vaccine and also died… but just died later than 28 days after the vaccine but no later than 25 weeks.

Both on Twitter and in the mainstream media, people were quick to try to criticize the study design for inappropriate controls or for lack of information on benefits of the vaccine. But the so-called “self-controlled case series (SCCS)” method has a very smart built-in control group of only people who have experienced the adverse event and is simply not designed to assess benefits.

The SCCS method was actually created to assess vaccine safety. It was first introducedby Paddy Farrington, Elizabeth Miller and team in The Lancet in 1995 as a way to look for association of the adverse events of febrile seizure and immune thrombocytopenia from DPT and MMR vaccines. 

It is a well-established epidemiologic method for evaluating safety and has been used numerous times already to evaluate the Covid-19 vaccines, in the UK (1,2), France and Nordic countries.

The idea of the SCCS is delightfully simple in that it assesses whether or not the risk of a certain adverse event (in the case of the Florida study “all cause mortality” or “cardiac-related mortality”) occurs more often than expected shortly after vaccination compared with baseline risk.

An unvaccinated control group was not necessary and the usual confounding variable of differences in health between vaccinated and unvaccinated disappears with this method. It’s a good alternative method to explore safety signals when you can’t get adequate information from randomized trials.

But what did the Florida study actually teach us?

It taught us there was a higher-than-expected rate of death in the first 28 days after their last dose of mRNA vaccine in 18–39-year-old males by 84% compared with weeks 4-25 post-vaccination. It did not prove this was due to the vaccine. 

It is noteworthy that this increased relative incidence in young males was not seen in other groups besides the >60-year-old males post-mRNA; this argues against this signal being due to people being sicker/hospitalized at the time of vaccination (though, the study may have lacked power to detect the signal in other groups and the authors could have provided more information on location of vaccination or provided other information to provide information on the underlying health of newly-vaccinated individuals, which I will discuss below).

Just using the unadjusted raw numbers, one can calculate the expected numbers of deaths in the 18-39 year old males post mRNA vaccine would have been 10 instead of 20 the first 28 days if risk were equal over the entire 25 weeks, so the signal we are talking about is 10 extra deaths total and an approximate doubling of expected number of cardiac-related deaths proximal to the vaccine (I spoke with Joe about the 84% reported and this was arrived at adjusting for seasonal variation in death rate). However, given the wide confidence intervals, this may have been just 1 or 2 deaths away from expected… so it’s a small number of deaths and a very uncertain signal.

The Florida study’s findings were inconsistent with a very similar UK preprint study. The UK analysis using Office of National Statistics data looked at young people ages 12-29 who all died post vaccination did not find a signal of increased risk of death in the first 6 weeks post-vaccination compared with the second 6 weeks.

The UK study did not look specifically  at the mRNA vaccines in young males. Also, the practice in the UK is to space the two vaccine doses by 12 weeks. The follow up period was shorter, which is important in a SCCS as the use of different risk and baseline periods can greatly alter the results. For example, what if the true vaccine risk period extends to three months post-vaccine and the UK study simply compared two periods of nearly equal risk? What if the risk period only lasts only a week and it was watered down?

On the other hand, confounders could also have been in play in the Florida study. What if some external factor unrelated to the vaccine decreased young males’ chances of dying due to CV disease in Florida during the baseline period?  For example, what if some of the young males actually had missing death registrations from the end of the 25 weeks? It would be nice to know if Florida ran tests to look for temporal biases and inconsistencies.

In a very nice, large Nordic SCCS of Covid-19 vaccine safety, they compared 28 days post vaccine with a pre-vaccination period. They did not specifically look at cardiac death and the included population was almost entirely over 50 years old.

They examined several outcomes, which appeared significantly increased post-vaccination. Relevant to cardiac risk, they found a borderline increase in corona artery disease (CAD) from Moderna only. One would not expect CAD to develop in a brief period after vaccination, but underlying disease could have been brought to attention due to new complaints of chest pain, palpitations, shortness of breath etc. 

(Getting back to the Florida study limitations, Dr. Panthagani made the good point that not all ICD-10 codes that cause cardiac death were included in the Florida study; specifically, ischemic heart disease was missing, though the explanation may have been as simple as ischemic heart disease is unlikely to develop shortly after vaccination due to the vaccine. So perhaps the exclusion of this particular code by the Florida team was appropriate.)

The Nordic study also identified a very strong signal for central venous sinus thrombosis (rate ratio of 12), which is a well-known safety signal by now, as well as other signals shown above related to the AZ vaccine. For the mRNA vaccines, the most notable signal was an increase in intracranial hemorrhage post Moderna vaccine (RR 2.19; adjusted p <0.001) in the 50 and older age group, which they say should be, among other potential findings, explored in future analyses. They highlighted the uncertainty around this finding as they found overall an elevated risk of hip fracture post vaccine in the mRNA vaccines with a rate ratio of around 2. While the authors conclude this elevated rate of hip fracture post-vaccine suggested poorer health around the date of vaccination, if patients were feeling unwell or dizzy post-vaccination, falls leading to hip fracture may have actually been due to the vaccination, so I am not sure this was the ideal way to detect confounding. But the potential of newly vaccinated to either be more or less healthy than the baseline period is an important potential confounder in SCCS studies.

Furthermore, the Nordic and Florida studies ran a large number of tests, and if a p value of <0.05 is used this would mean 5% would be significant just by chance. You can see above that the Nordic study corrected for this using a “false discovery rate adjusted p-value”. Florida could have considered such an adjustment for running tests in so many demographics but did not.

On the other hand, the opposite may be true. There may have been a true difference in outcomes in more groups in the Florida, but, because the number of total deaths were so small, the study may have failed to detect some true signals.

Finally there is the issue of a missed Covid diagnosis in the Florida study. The research team excluded people who they knew had COVID-19 through their medical records or death records but could not rule out undiagnosed Covid before or after the vaccine. This is a potentially important confounder. In other words, a young man could have gone out for a vaccine and caught Covid a couple days later and this may not have shown up in his death certificate or health record.

So, where does this leave us, the readers of Sensible Medicine?

We are left with a Florida study with a high degree of uncertainty that found 10 deaths (9 with their seasonal adjustment) above expected in the 28 days post mRNA vaccine among males 18-39. This is a signal which has not been found previously with the SCCS method in arguably better studies. A significant correlation with cardiac arrest calls and rollout of first and second doses of the Pfizer vaccine has been seen previously in Israel, in a study with small numbers.

And the demographic found to be at risk in the Florida analysis is known to have elevated risk of myocarditis post second dose as high as around 1/2000 range per Kaiser Permanente and as high as 1/1200 per Ontario data (when not spacing doses out and mixing Pfizer and Moderna). We know of deaths caused by vaccine-related myocarditis have been  reported in the peer reviewed literature, including in the NEJM.

The politicization of the topic can distract from the big question: Should young healthy people continue to get vaccinated/boosted against Covid? Or specifically, how many doses of mRNA vaccine should healthy 18–39-year-old males get and based on what? Should recommendations for previously infected be different?

My own research group found mRNA boosters carry at least 18x the risk of severe adverse event as hospitalizations from covid they prevent at a population level for people 18-29. For males one can expect on average 1.3-3 cases of post-booster myocarditis per covid-19 hospitalization prevented. A large Nordic study in JAMA Cardiology found post-vaccination myocarditis can occur up to 28x more frequently than post-Covid myocarditis in males 16-24 with the Pfizer-Moderna combination without increased interval between doses.

Denmark, Sweden and Norway have all stopped offering boosters to people <50 unless they are found to be at high risk from Covid. Last year, numerous countries recommended against using Moderna in males <30 due to increased cardiac risk.

The debate over Covid vaccine safety and benefits has been uniquely polarized in the United States, often involving personal attacks based on political affiliation. It is worth noting that the parallel debate has been largely civil in Scandinavia and that the recommendations to not continue to vaccinate young healthy people are largely accepted by most doctors and reseachers there. In Scandinavia, all along there has been open, transparent debate about pros and cons and side effects of vaccination types and number of doses in different demographics. There, because people remain trustful of public health authorities (with good reason), they are also less likely to jump to irrational fears about vaccine side effects.

Attacking physicians or scientists for spreading supposed “misinformation” is not the solution. Young scientists should not be fearful of reporting their results if they don’t fit with the current narrative or might be viewed as “political”. Or even worse, scientific journal editors should not avoid publishing certain findings for fear of online repercussions. If the public sees silencing and censorship, they fear an important truth is being hidden.

I hope we can get beyond the idea there are heroes and villains in the Covid pandemic and Covid vaccination story. We are setting a precedent now for how online scientific debate will happen in the future. So let us respectfully and openly discuss vaccine risks and benefits as if we were sitting in the same room together, all simply trying to solve a problem and give good individualized public health guidance.

In that vein, let’s debate! What did I get wrong in my analysis of the study and what do you have to add?

Reprinted from Sensible Medicine