Revisiting the technical validation of tumour biomarker assays: how to open a Pandora's box

Although great emphasis is given to the discovery and validation of the clinical significance of the biomarker, the technical validation of assays for novel biomarkers has not been embraced with the same enthusiasm, probably because of its more technical and apparently less rewarding nature. Nonetheless, the process of assay validation is critical for the introduction of a new biomarker in routine clinical practice. This minireview focuses on the technical issues related to validation of biomarkers analysed directly in human tumour tissue samples, with breast cancer pathology serving as a paradigm. It should be noted, however, that the concepts discussed in this review are applicable to biomarkers based on other types of samples (for example, circulating tumour cells, blood, serum, urine and other bodily fluids).

A biomarker often fails to be incorporated into clinical practice, not because of flawed science underpinning its discovery but because of poor choice of the assay used for its detection and inadequate validation of its accuracy [6, 7]. For the successful use of a biomarker assay in clinical practice, it is of paramount importance that the testing of the assay employ robust reagents and be based on a reliable and robust technology. Several false dawns in translational research have stemmed from attempts to introduce a technology that was not sufficiently mature at the time and from the sources of technical bias not being entirely known (for example, mass spectrometry-based serum proteomic analysis for the diagnosis of ovarian cancer) [8, 9].

The early technical validation of a new biomarker assay is critical to minimise waste of resources, generation of misleading results and possibly disrepute. In this context, however, full validity is often difficult to prove, given that these assays come directly from research assays for which a gold standard is usually not available.

Immunohistochemistry, fluorescence and chromogenic in situ hybridisation, expression profiling, either microarray-based or performed by quantitative real-time reverse transcriptase-polymerase chain reaction (qRT-PCR), and mutation analysis represent the main techniques currently being introduced into everyday practice in pathology laboratories. Among these techniques, immunohistochemical tests remain the most widely used in routine practice and, importantly, in the assessment of biomarkers in translational research endeavours. We therefore focus primarily on their validation.

In the case of immunohistochemistry, a typical 'quick and easy' approach is illustrated in Figure 1. It should be noted, however, that such an approach carries with it far more pitfalls than many investigators assume. Although immunohistochemistry has been introduced in pathology laboratories for more than 25 years and is a relatively user-friendly technique, one should not forget that the results of immunohistochemical analysis may be affected by a series of preanalytical and analytical parameters [11–15].

Preanalytical variables are the 'weakest link' of immunohistochemistry, as the properties of the tissues analysed and, consequently, the results, may be affected by several factors, including the time to collection (that is, the length of time that tissues are subjected to warm ischaemia between removal of the tissue at surgery and fixation), details of fixation (type of fixative agent used and length and conditions of fixation), dehydration steps, and conditions for paraffin-embedding (temperature of the paraffin). These preanalytical parameters are beyond the control of investigators, are most often unrecorded, and constitute a major potential source of bias, in particular in multicentre, retrospective studies. For example, the time to fixation and its length have been branded as the 'Achilles heel' of phosphoprotein assessment in clinical specimens [16] as recently shown by Pinhel and colleagues [16, 17], who found consistently significantly lower levels of the phosphoepitopes in surgical specimens compared with those found in core biopsies.

Four main analytical issues require special attention. Antigen retrieval (that is, a method that enables immunohistochemistry to be applied to formalin-fixed, paraffin-embedded (FFPE) samples), type of detection system, the choice of antibody, and the material to be used [11–13]. Enzyme-based and heat-induced epitope retrieval are available, for which strict protocols should be followed to obtain accurate results [11]. Excellent reviews have described the pitfalls of antigen retrieval and how they can be overcome [11–15]. Suffice it to say that in the absence of optimal positive and negative controls and a 'gold standard', changing antigen retrieval settings can render a given case positive or negative.

Arguably the most challenging aspect of any immunohistochemical assay is the choice of antibody. Before applying 'research-only' antibodies to FFPE tissues, their sensitivity and specificity need to be determined. It should be emphasised that this cannot be achieved merely by evaluating the sensitivity and specificity of the antibody by means of Western blot analysis; these parameters need to be assessed in FFPE samples and using optimal controls (Figure 2). It is always difficult to find the optimal balance between sensitivity and specificity, and, in this respect, the antigen retrieval method and antibody titration are critical, as is the availability of optimal controls to be used as a gold standard. One helpful and informative approach uses in-house or publicly available expression array data sets to identify cell lines that can be exploited as positive controls and then to validate their use as positive controls on the basis of qRT-PCR assay and Western blot analysis. Negative controls can be generated by siRNA (short interfering RNA) knockdown of the gene of interest in the cell line. Subsequently, antibody testing can be performed on FFPE cell pellets, and finally optimisation of immunohistochemistry on FFPE tissues will follow (Figure 2). In this process, the type of detected subcellular localisation may also help the researcher understand whether the antibody is specific. Finally, when dealing with biomarkers used in routine diagnostic practice, audit of the results from annual workload can provide supporting evidence of the actual performance of an optimised assay used in routine practice.

Another crucial aspect is the type of tissue to be used. Tissue microarrays (TMAs) have become very popular in studies aiming to determine the distribution of a given marker in a cohort of samples [18, 19]. Although TMAs have proven to be excellent tools, they should be employed for the testing of biomarkers whose expression is relatively homogeneous and should be used only if the concordance between the results of the analysis of a given marker on whole tissue sections and TMAs is close. Regrettably, assessing the latter appears not to be a common practice. It should be noted that this issue is applicable even to well-validated tests in breast cancer research and practice (for example, the discordant results in the analysis of PR expression in TMAs vs. whole tissue sections [20]).

In recent years, microarray gene expression profiling and its derivatives have been widely applied to the molecular and biological classification of breast cancers, and several prognostic and predictive signatures have been reported, some of which have been introduced into clinical practice (for example, Oncotype DX Breast Cancer Assay (Genomic Health, Inc., Redwood City, CA, USA) and MammaPrint assay (Agendia BV, Amsterdam, the Netherlands); for reviews, please see Weigelt et al. [21], Sotiriou and Pusztai [22], and Reis-Filho et al. [23]). The types of analysis and the data they generate pose a major challenge for the translation of their findings into assays that can be used in routine clinical practice, as the reliability, reproducibility and stability of some have been called into question [23–25]. In terms of preanalytical variables, most of the parameters that affect immunohistochemistry also affect gene expression profiling (for example, time to tissue fixation, time to freezing, length of fixation, type of fixative used and tissue storage). In addition, given that these technologies are based on nucleic acid extracts from tissue homogenates, the variable tumour content may also constitute a confounding factor. Cleator et al. [26] demonstrated that varying amounts of non-neoplastic cells in samples subjected to gene expression increases the error rates of multigene predictors, providing direct evidence that the non-tumour content of breast cancer samples has a significant effect on gene expression profiles.

Data analysis of these 'mega-parameter' profiles also poses huge challenges. Microarray technology and bioinformatics/statistics applied to microarray analysis have developed at disparate speeds and this may have led to inappropriate conclusions being drawn and contributed to the first wave of over-optimism and, then, to the subsequent wave of (over)scepticism with this type of technology experienced in the last 10 years [24]. Indeed, when these signatures were first described, there was little awareness of problems of data 'overfitting' and methods for power calculation for microarrays [21, 23]. This field has developed rapidly, however, and guidelines regulating how a therapeutically significant gene signature should be developed and validated are now available [27, 28]. The chances of success in developing and validating gene signatures will be significantly increased if these guidelines are strictly followed.

Technical validation and qualification of a biomarker assay may not be as glamorous as biomarker discovery; however, together they comprise the critical lynchpins of translational cancer research. Guidelines for the development and validation of biomarker assays are available. It should be noted, however, that not all aspects of these guidelines may be applicable to the assay to be developed. In these cases, imaginative approaches should be sought while erring on the side of caution. To avoid waste of resources and the use or publication of misleading data, full awareness of the technical challenges in assessment of the robustness of assays and careful evaluation of the context of the assay to be developed are required [29]. Finally, in the reporting of the results, uncertainties should be disclosed and all caveats ought to be voiced. While these issues are undoubtedly challenging and consideration of the numerous potential pitfalls may be discouraging, awareness and avoidance of these problems can allow clinically relevant work of great value to be conducted [29].