To the best of our knowledge, this research work is the first study conducted with an aim to compare accessible models developed by both the public and private sectors. This review provides insights into differences and similarities of model attributes and key assumptions to understand disease dynamics and key drivers of the results. Importantly, having a full understanding why different tools produce different results will help identify data needs and pitfalls important to the users of such models. Although the models we reviewed were developed to determine value for money of vaccination program, different vaccines were assessed: PneumoADIP assessed the CE of PCV, TriVac assessed the CE of PCV, Haemophilus influenzae B (Hib) and rotavirus vaccines and SUPREMES appraised pneumococcal Haemophilus influenzae protein D-conjugate vaccine (PHiD-CV).
The different model approaches translated in different data input requirements, which made it a challenge to use a standardized data set for the tool comparison exercise. The incidence-based cohort approaches adopted in the PneumoADIP and the PAHO TriVac models are more data demanding and may be more intuitively appealing than the SUPREMES model. The latter model evaluates under static conditions an entire population at specific time point using the total population size and it specific age distribution to estimate the impact of a new vaccine program on total population health during a fixed time period (usually 1 year) and therefore requires fewer data inputs. This feature makes the SUPREMES model useful if the vaccine causes intermediate and important changes to the population and if budget-impact estimates after vaccine introduction are of interest to the decision maker. In addition, the model is easier use and can be applied widely to those countries with limited data but may be more difficult to justify as users would need to agree with several implicit assumptions embedded in the modeling approach.
It is important to note that while all three models were at an advanced stage of development at the time of the comparison exercise, models of this kind are prone to continued extension and elaboration, and weaknesses or absent features listed in this article may be corrected or improved with time. This comparison process can serve a useful purpose in this regard, by giving modelers the opportunity to identify key omissions and improve the accuracy, transparency and usability of their models. For example, since the evaluation in August 2009 the TRIVAC model now includes otitis media outcomes, advanced options for probabilistic sensitivity analysis and scenario analysis, and key assumptions about waning efficacy and age-appropriate vaccination. The present version of the PneumoADIP model allows discount rates to be varied while GSK now has developed a cohort-based version of the SUPREMES model to deal with the different vaccine impact on disease events using cohort-based or population-based model methods . In addition, while none of the three models had been through a validation process at the time of the comparison exercise, the TRIVAC model has since been evaluated by an external expert panel . As models and cumulative effects analysis (CEA tools evolve continued appraisal is needed.
The differences between these models are not only due to the approach and data requirements, but are difficult to distinguish without fully transparent user manuals. The TriVac model has an advanced user-friendly and self-guided function enabling users to properly interpret the research findings.
All three models include the possibility to incorporate the herd immunity effects of the vaccination program, however there are important differences in how this is incorporated and the underlying assumptions in each model. Herd immunity effects have already been shown to be influential in various analyses published before 2006 . However, the closely related effects of serotype replacement, which can partly nullify the beneficial impacts of herd immunity are not taken into account in any of these models. All three models have shown that they acknowledge an important component of outcome, the morbidity or health-related quality of life, because they present the results in terms of incremental costs per DALY averted (PneumoADIP and TriVac) or per QALY gained (SUPREMES).
From the user's perspective, it is very important to be able to identify influential parameters. A series of sensitivity analyses using tornado diagrams has shown that vaccine efficacy against pneumonia, serotype coverage and replacement, vaccine coverage, and disease burden parameters including incidence and case fatality of pneumococcal diseases are the key driver of the outcomes. Unfortunately, evidence to inform one or more of these parameters is often lacking in low-income and middle-income countries and frequently parameters are populated through generalizations made using data from industrialized countries. Vaccination costs was also a parameter of major importance (though not included in the standard sensitivity analyses).
The outputs for TriVac and PneumoADIP were found to be very similar, while the outputs of the SUPREME model were markedly different from the other two models. Given the differences in various aspects such as parameters included in the models, the baseline value used in the model, and model structure, it is not possible to distinguish the sources that could explain the differences in findings. Because discounting is very influencing in this long-term outcome modeling, the lack of discounting in the SUPREME model could be considered a crucial factor that might explain, to a certain extent, the differences in QALYs gained and cost differences. To be more specific, the future outcomes earned by cohort vaccinated previously are measured at the same time for cohort currently being vaccinated.