BMC Medicine BioMed Central

Background Most Ministries of Health across Africa invest substantial resources in some form of health management information system (HMIS) to coordinate the routine acquisition and compilation of monthly treatment and attendance records from health facilities nationwide. Despite the expense of these systems, poor data coverage means they are rarely, if ever, used to generate reliable evidence for decision makers. One critical weakness across Africa is the current lack of capacity to effectively monitor patterns of service use through time so that the impacts of changes in policy or service delivery can be evaluated. Here, we present a new approach that, for the first time, allows national changes in health service use during a time of major health policy change to be tracked reliably using imperfect data from a national HMIS. Methods Monthly attendance records were obtained from the Kenyan HMIS for 1 271 government-run and 402 faith-based outpatient facilities nationwide between 1996 and 2004. A space-time geostatistical model was used to compensate for the large proportion of missing records caused by non-reporting health facilities, allowing robust estimation of monthly and annual use of services by outpatients during this period. Results We were able to reconstruct robust time series of mean levels of outpatient utilisation of health facilities at the national level and for all six major provinces in Kenya. These plots revealed reliably for the first time a period of steady nationwide decline in the use of health facilities in Kenya between 1996 and 2002, followed by a dramatic increase from 2003. This pattern was consistent across different causes of attendance and was observed independently in each province. Conclusion The methodological approach presented can compensate for missing records in health information systems to provide robust estimates of national patterns of outpatient service use. This represents the first such use of HMIS data and contributes to the resurrection of these hugely expensive but underused systems as national monitoring tools. Applying this approach to Kenya has yielded output with immediate potential to enhance the capacity of decision makers in monitoring nationwide patterns of service use and assessing the impact of changes in health policy and service delivery.


Background
Since its development, meta-analysis has become a powerful tool for informing clinical practice. Performed correctly, meta-analysis is superior to a purely narrative approach of summarizing medical research. As such, robust conclusions may sometimes be reached from serial, otherwise underpowered small studies [1,2]. Nonetheless, there are substantial limitations and pitfalls in meta-analysis. Publication bias, reliance on subjective summary results rather than individual patient data and the mishandling of important heterogeneity can all lead to erroneous conclusions [1][2][3][4][5][6][7][8]. This possibility is underscored by the occasional lack of concordance between meta-analyses and subsequent large randomized, controlled trials [3,9].
We assembled a meta-analysis of NAC efficacy in preventing CIN. Like previous attempts, we encountered significant heterogeneity that was not explained using a comprehensive meta-regression approach. A modified L'Abbé plot [86] followed by the application of a modelbased, unsupervised clustering algorithm [87] resolved the trials into two significantly different populations. Clinical practices aimed at preventing CIN are discussed and recommendations are made regarding future trials of NAC.

Methods
This meta-analysis was completed in accordance with the Quality of Reporting of Meta-analyses (QUOROM) statement [2].

Literature search
We searched MEDLINE (PubMed and Dialog), EMBASE, International Pharmaceutical Abstracts, Derwent Drug File, Adis R&D Insight, Adis Clinical Trials Insight, Biological Abstracts and CINAHL (OVID), the Web of Science and The Cochrane Library. Searches included: controlled vocabulary for acetylcysteine, contrast media/adverse, toxic and poisoning effects; free text for acetylcysteine and contrast; and MeSH terms acetylcysteine and contrast media. Retrieved records from the Cochrane CENTRAL file were re-checked in Web of Science to identify subsequent publications. Search dates were from the inception of the databases until September 30, 2004. Conference proceedings from the American Society of Nephrology, National Kidney Foundation, American Heart Associa-tion, American College of Cardiology, Society of Interventional Radiology, Radiologic Society of North America and International Society of Nephrology were also reviewed over the past five years. There were no restrictions on language or publication status. Over 450 citations and abstracts were screened by two authors to assemble a preliminary set of possibly relevant reports. New publications after September 30, 2004 were periodically monitored using the same search criteria up to March 1, 2007.

Selection criteria
Studies were limited to prospective, randomized, controlled trials (PRCTs) investigating the efficacy of NAC in preventing CIN. Trials with confounded, non-concurrent or otherwise improperly constructed control groups were prospectively excluded from further analysis. Outcome data were solicited from the authors if not found in the publication. Trials that still lacked outcome data necessary for planned analyses were excluded.

Quality assessment, data retrieval and clinical endpoints
Two of the authors evaluated each trial using the Jadad scoring device, under unmasked conditions [88]. Each PRCT included in the analysis scored at least 1 on the fivepoint scale, with higher scores indicating greater trial quality. Data were extracted independently into a standardized form. Results were compared and disagreements were resolved by discussion. The primary outcome measures were the development of CIN as defined in the studies [10-31] and change in creatinine (ΔCre). The occurrence of acute kidney injury requiring dialysis was recorded. When not reported in the publication, we contacted the authors for post-contrast dialysis information.

Meta-analysis and heterogeneity testing
Treatment effects were quantified by relative risk (RR) using a random-effects model (Comprehensive Meta-Analysis, Biostat Inc, Englewood, NJ). Statistical heterogeneity was assessed by means of a Mantel-Haenszel derived Cochran's Q statistic and associated I 2 value. Cochran's Q is used to test the null hypothesis that all treatment effects are equivalent [89]. Calculated from the Q-statistic and degrees of freedom, I 2 represents the proportion of treatment effect variation owing to trial heterogeneity, rather than simple sampling error [4, 89,90]. Statistical heterogeneity is present when this variation in results exceeds the amount expected from chance alone. The quantitative pooling of such studies may lead to erroneous conclusions [4].

Publication bias and meta-regression analysis
Evidence of publication bias was formally tested using multiple methods including those of Begg and Mazumdar [6], Egger et al. [5] and Higgins and Thompson [4]. Stand-ard meta-regressions of the effect size expressed as log RR were performed against trial factors including publication date, size and Jadad score. Well-known patient-related risk factors associated with increased rates of CIN were also evaluated by meta-regression including mean age, diabetes mellitus (%), gender (% female), mean contrast volume and mean baseline creatinine concentration [91][92][93][94]. Likewise, total NAC dose was examined for its relationship with outcome. A separate meta-regression examined the log odds of developing CIN in the treatment versus the control groups. This was used to detect whether NAC efficacy was affected by the rate of CIN in the control population [95,96]. All meta-regressions were weighted by the inverse variance of each study.

Jackknife-k sensitivity analysis, modified L'Abbé plot and unsupervised clustering: detection of trial subpopulations
A sensitivity analysis for heterogeneity was completed by means of a jackknife-k [97] procedure in order to detect studies that contributed most to heterogeneity. A prespecified p-value greater than 0.2 for Cochran's Q statistic and an I 2 of less than 10% indicated homogeneity. Every possible one-, two-and three-study combination was removed.
The method of L'Abbé et al. [86] was used to visualize heterogeneity in our set of trials. As originally described, the L'Abbé plot graphs the control group outcome rate along the x-axis and the treatment group outcome rate along the y-axis for each trial. To correct for differences in the definition of CIN across studies, we modified the L'Abbé plot by substituting ΔCre, a continuous variable, for the CIN rate. Compared with a standard L'Abbé plot (data not shown), the modified plot was similar, but was better at separating studies that were low and high contributors to heterogeneity.
We then analyzed our modified L'Abbé plot using an unsupervised, model-based clustering method that creates a best-fit Gaussian model and finds the number of clusters that maximize the Bayesian information criterion. All members of the data set are then classified using iterative expectation-maximization methods and group membership likelihoods are calculated [87]. The study and patient characteristics of each cluster were then compared using Wilcoxon rank sum tests. The decomposed Breslow-Day test was used to determine whether the identified clusters had significantly different treatment effects.
Additional information required for analysis was requested from trial authors; when unsuccessful in the case of one abstract [18], data were extracted from other meta-analyses. We included the more complete, updated data from manuscripts that were published after our cutoff date [55][56][57] if these studies had been available in the form of abstracts [19-21] before September, 2004. Table 1 lists the characteristics of the 22 trials meeting our prospective selection criteria [10-31]. Figure 2 shows a forest plot ordered by time of publication, with RR and confidence intervals (CIs) of developing CIN if treated with NAC. A summary statistic is not shown owing to the significant heterogeneity (I 2 = 37%; p = 0.04) that precluded the pooling of these trials.

Publication bias and meta-regression analysis
Although non-significant (p ≤ 0.11, but p > 0.05 when applying any one of the three methods used for analysis), a visual inspection of a funnel plot suggested publication bias with four studies [10,11,14,25] contributing most to the apparent asymmetry (shown with open circles on the left-hand side of Figure 3). An extensive meta-regression analysis of patient and study characteristics found no study-specific characteristic (publication date, size, quality as measured by Jadad score or total NAC dose) or   patient-related characteristic (age, diabetes, gender, contrast volume, baseline creatinine or CIN event rate in the control group) that significantly co-varied with NAC efficacy (Table 2).

Sensitivity analysis
A jackknife-k sensitivity analysis [97] identified 10 studies that decreased heterogeneity when individually removed (right-hand side of Figure 4). Removal of any one of the remaining 12 studies increased heterogeneity (left-hand side of Figure 4). The four small studies [10,11,14,25] that individually contributed the most to heterogeneity are shown as open circles in Figure 4 (circle size is proportional to inverse variance). Removal of any single study or all possible two-study combinations failed to adequately resolve heterogeneity. In contrast, the removal of multiple three-study combinations (combinations [11,14,25][10,11,14][11,14,21] and [11,14,17]) reached our pre-defined target for homogeneity (after the removal of any one of the three-study groups above, I 2 ≤ 9.5% and p ≥ 0.34). These four three-study groups represent only 7.9%, 9.4%, 12.0% and 13.7% of the entire study population, respectively.

Increase Decrease
Funnel plot of precision versus log RR   beneficial, relatively small studies [10,11,14,25]. These same four studies had caused the appearance of asymmetry in the funnel plot and were associated with heterogeneity by jackknife-k analysis. As suggested by the L'Abbé plot, a box plot ( Figure 5B) of creatinine change clearly shows that these four studies have relatively large creatinine increases in control patients (p = 0.02; open boxes on the left-hand side) and relatively large creatinine decreases in NAC-treated patients (p = 0.07; open boxes on the right-hand side).

Examination of new studies published after our cut-off
Cluster analysis based on changes in creatinine  One study of both high-and low-dose intravenous NAC in patients with acute myocardial infarctions [64] did not group strongly with either cluster (probabilities of group membership: low-dose arm, 39% for cluster 1 and 61% for cluster 2; high-dose arm, 49% for cluster 1 and 51% for cluster 2). Based on these results, this study [64] was found to be an outlier (p < 0.05; Dixon test) [98] compared with other trials assigned to either cluster 1 or 2.

Hemodialysis risk model
We tested for a correlation between CIN and the clinically more rigorous outcome of dialysis. The correlation was weighted by the inverse variance of each study. Of the 22 trials in our meta-analysis and the nine more recent studies, hemodialysis events occurred in a total of nine trials [12,15,16,[18][19][20][21]58,64]. Figure 7 shows that the RR of CIN, as defined in each trial, is positively correlated with the RR of requiring dialysis post-contrast (r = 0.66; p = 0.038). However, the regression equation is shifted upwards from the line of identity. For the RR of dialysis to be on the side of benefit (RR < 1.0), the RR of CIN would need to be substantially below one (RR < 0.67 for CIN).
In fact, observing a RR of CIN this low in any future clinical trial is unlikely based on our cluster analysis, because it lies outside the 95% CI for cluster 1. A trial enrolling 32 200 patients, as described in the power analysis, would also have a moderate likelihood of showing a harmful effect of NAC on the need for post-contrast dialysis (RR = 1.29).

Discussion
The limited ability of meta-analysis to address unexplained heterogeneity has been explored in a well-known data set that has been subjected to a large number of previous investigations. CIN is a common and important complication of diagnostic imaging that has a substantial impact on morbidity and mortality [91][92][93][94]. While hydration is clearly beneficial in preventing CIN [99,100], NAC has been investigated in many trials and subsequent metaanalyses with no consistent answer as to its efficacy. This meta-analysis of 22 studies, like previous meta-analyses [72][73][74][75][76][77][78][79][80][81][82][83], has demonstrated significant heterogeneity. The inconsistency across studies was systematically explored. Funnel plots [4-6] and a reiterative sensitivity analysis [97] both identified subsets of studies that appeared to be most strongly associated with this problem. However, a standard meta-regression approach [1,2,84] failed to identify a single study or patient-related characteristic that correlated with or fully explained variability in the NAC treatment effect. Ultimately, a modified L'Abbé plot [86] that substituted change in creatinine, a directly measured continuous endpoint, for CIN event rates, an all-or-nothing outcome that was variably defined across trials, indicated the possibility of distinct trial subpopulations within the overall results. Borrowing from our experience in functional genomics research, unsupervised, modelbased clustering [87] was applied to demonstrate that the data set represented two homogeneous, significantly different trial populations. This novel approach allowed us to directly compare trials that populated each of the two dissimilar clusters and provided a reliable aggregate point estimate for performing a formal power analysis.
NAC prophylaxis for the prevention of CIN was first introduced in 2000 [10] and although definitive proof of efficacy has been elusive, the use of NAC prophylaxis has become widespread. NAC trials have mainly been conducted in stable patient populations with at least one risk factor for the development of CIN [10-68]. Small doses of NAC given orally have been the most frequently investigated regimen despite evidence that the drug is poorly absorbed and undergoes significant first-past metabolism [101]. Although vigorous hydration has been demonstrated as an effective preventive strategy [99], NAC trials have typically been conducted using no more than maintenance infusions (1 ml/kg/h) of half-normal or normal saline . Whether the small, non-significant benefit of NAC in cluster 1 of our meta-analysis would persist if hydration were individually optimized is questionable. Importantly, a large PRCT of unselected patients undergoing elective coronary angiography found that normal compared with half-normal saline reduced the incidence of CIN almost threefold [100]. Merten et al. [102] reported a negligible incidence of CIN in subjects treated with a sodium bicarbonate infusion at 3 ml/kg/h before contrast followed by 1 ml/kg/h after contrast. These studies suggest that fluid administration regimens have a large impact on CIN risk. It is worth noting that all four highly beneficial studies in cluster 2 of our meta-analysis [10,11,14,25] employed protocols specifying half-normal saline infusions at 1 ml/kg/h.

Hemodialysis risk model
Changes in serum creatinine levels have invariably been used to diagnose CIN in trials of NAC. However, serum creatinine is a poor surrogate marker for glomerular filtration rate (GFR) because creatinine is influenced by diet, endogenous production, renal filtration, secretion and reabsorption [103,104]. Contrast agents themselves may decrease creatinine secretion and thereby raise serum creatinine levels, independently of changes in GFR [105]. Conversely, NAC in the absence of contrast has been shown to decrease serum creatinine levels in normal volunteers [106] and patients [66]. Hoffmann et al. [106] detected significant NAC-induced decreases in serum creatinine that were not associated with similar changes in cystatin C. As cystatin C is not secreted by renal tubule cells it may be a more accurate indicator of GFR [107,108]. Interestingly, in our meta-analysis, three out of the four cluster 2 studies [10,11,14] and one cluster 1 study [17], shown by sensitivity analysis to make a relatively large contribution to heterogeneity, all reported substantial NAC-induced decreases in serum creatinine. This response to NAC may be a drug effect independent of changes in GFR.
The four highly beneficial studies (cluster 2) represent only 11% of patients in our meta-analysis. These trials were significantly different from cluster 1 studies in that they had early publication dates, were small in size and of low quality. Furthermore, cluster 2 studies uniformly employed an inferior hydration regimen that may have exaggerated any effects of NAC treatment. Cluster 2 studies were characterized by relatively large serum creatinine increases in control patients and similarly large creatinine decreases in NAC-treated patients.
A power analysis of cluster 1 studies indicated that 32 200 patients would be needed in a single PRCT to have an 80% chance of detecting benefit using definitions of CIN based on serum creatinine. Importantly, dialysis use was not decreased by NAC treatment across the 2746 patients in our meta-analysis. The large PRCT just proposed would have a moderate likelihood of demonstrating harm as measured by the more rigorous clinical endpoint of dialysis. Based on this investigation, low-dose oral NAC has not been shown to prevent CIN and should not be routinely recommended.
Eight of the nine new trials published since we closed our meta-analysis [58][59][60][61][62][63]70,71] were found to group with cluster 1 and support our overall findings. One of the trials was an outlier and not only reported significant reductions in CIN rates, but also decreases in dialysis use and mortality [64]. In this study, very ill patients with acute myocardial infarctions were treated with intravenous NAC boluses during angioplasty [64]. As noted by the authors, these single-center results require confirmation.
As survival improved in their trial, Marenzi et al. speculated about possible benefits of NAC beyond the simple prevention of CIN [64]. Alternatively, the relatively high mortality in control subjects might also be explained by hidden imbalances created during randomization. In contrast to this highly beneficial trial, other studies in highrisk patients undergoing coronary bypass [109] or abdominal aortic surgery [110] did not find that intravenous NAC reduced the incidence of postoperative renal dysfunction or mortality.

Conclusion
Our meta-analysis does not support the use of NAC for reducing rates of acute kidney injury due to intravascular iodinated contrast. In several overly influential trials showing large beneficial effects, NAC decreased serum creatinine levels, suggesting possible drug effects independent of true changes in GFR. Dialysis use across all studies occurred infrequently, but did not indicate that NAC was efficacious. Future clinical trials of therapies to prevent CIN should incorporate primary endpoints other than change in creatinine.