Evaluation of three 3ABC ELISAs for foot-and-mouth disease non-structural antibodies using latent class analysis

Foot-and-mouth disease (FMD) is a highly contagious viral disease of even-toed ungulates caused by Foot-and-mouth disease virus (FMDV), which is a member of the genus Aphthovirus and the family Picornviridae [1]. A number of non-structural protein (NSP) immunoassays, designed to detect aphthovirus-specific antibodies to replicating virus in animals irrespective of vaccination status or for screening multiple-serotype infected/exposed animals, have been under development in recent years [2–9].

The OIE Reference Laboratory for the Americas, PANAFTOSA/PAHO/WHO, has developed and validated a surveillance tool, to accompany the progress of eradication campaigns in South America, based on a screening NSP ELISA (the I-ELISA 3ABC), followed by an enzyme linked immunoelectotransfer blot test (EITB) to clear up false positive reactions [10]. Because vaccination prevents clinical disease but does not necessarily prevent infection or persistence [11–13], it is useful to be able to identify animals which are infected or have been exposed to live virus irrespective of their vaccination status. The NSP tests offer an alternative to the virus neutralization test (VNT) [14] and liquid phase blocking ELISA (LPBE) [15–17]. The VNT and LPBE detect antibodies to FMDV structural proteins and require separate testing for each of the seven serotypes of FMDV and for some subtypes. These tests are time consuming to perform, require virus containment facilities and cannot differentiate vaccinated from convalescing animals [18]. Furthermore the VNT may not be an appropriate comparison for NSP ELISAs since antibodies to the structural proteins are detected slightly sooner and for longer than NSP antibodies [19].

The need for a mass screening test for use in vaccinated populations to detect circulating infection gained impetus in Europe with the outbreak of FMD in the UK in 2001. The mass slaughter of millions of sheep and cattle caused public outrage and has added a political element to the arguments that vaccination should be used in future FMD outbreaks. The United Kingdom came close to vaccinating dairy herds in Cumbria prior to turn-out of cattle in the spring of 2001 but eventually opted not to amid fears of prolonging trade bans and consumer resistance to drinking milk from vaccinated cattle [20]. EU regulations have since been modified to allow for the emergency use of vaccination (EU Council Directive 85/511/EEC). The World Organization for Animal Health (O.I.E.) trade regulations at the time required 12 months of disease freedom following the end of emergency vaccination before a country could be declared free. However, in 2004 the regulations were altered and countries can now regain 'FMD free' status six months after the end of vaccination providing they have carried out surveillance with an NSP test (Article 2.2.10.7, O.I.E. Terrestrial Animal Health Code, 2004). This is three months later than could be achieved following a slaughter policy without vaccination.

In order to design and implement an effective vaccination program and subsequent sero-surveillance, it is necessary to know the properties of the test(s) intended to be used. The properties of a diagnostic test are often described using the sensitivity (Se), i.e. the proportion of true positives that the test identifies and the specificity (Sp), i.e. the proportion of true negatives that the test identifies. When available, these parameters are estimated using animals with known infectious status. For NSP tests the values have been obtained using controlled experimental and field models [8, 10, 21–23]. Alternatively, studies can be carried out to compare the diagnostic sensitivity and specificity of an assay against a standard of comparison i.e. a "gold standard" test.

Previously some of the authors compared two FMD 3ABC ELISAs using the VNT as the "gold-standard" [24], i.e. it was assumed that the Se and Sp of the VNT were both 100%. However, there might be reasons to question the justification of using the VNT as a "gold-standard". It appears, that reliable estimates of the duration of the different antibody responses in cattle are lacking. Thus, there might be justification in disregarding the VNT and compare the NSP tests to each other without assuming that the true status of the animal is known (i.e. the disease status is latent). The latent class approach does not assume any of the tests is a "gold standard" [25–27]. This paper follows on from our previous evaluation of two NSP tests using a conventional approach with the VNT as a "gold-standard" [24].

The objective of the current study was to estimate sensitivity and specificity of two previously studied NSP tests, the CHEKIT-FMD-3ABC bo-ov (CHEKIT) ELISA (Bommeli Diagnostics/Intervet) and the 3ABC blocking ELISA (C-ELISA) developed in Denmark, as well as the O.I.E. index NSP test, the I-ELISA 3ABC developed by PANAFTOSA [10] without assuming that a 'gold standard' was available. The test properties were estimated for each of the three tests in a cattle population from Cameroon where serotypes O, A and SAT2 were known to be circulating and where at the time of the original study in 2000, there was no vaccination [28, 29]. The analysis used a Bayesian formulation of the Hui-Walter latent class model for test evaluation in the absence of a "gold-standard" [26]. Since this work was carried out the CHEKIT-FMD-3ABC bo-ov has been reformulated as a mark II kit and the Danish C-ELISA has been reformulated as a commercial kit and marketed under the name Ceditest^® FMDV-NS (Cedi Diagnostics B.V.).

The key issue in traditional test parameter estimation is the availability of a "gold-standard" test, i.e. a test whose outcome can be directly interpreted as presence or absence of disease/infection. Quite often however, the need for such a test seems to determine the definition of the disease and this may introduce selection bias in the test evaluation [30] or force a disease definition, which does not necessarily reflect the essential properties of the disease, but merely the detectable properties. The estimates for the relative Se and Sp of the C-ELISA test when compared to the VNT as a "gold standard" for this cattle population were 71% and 90% respectively [24]. The new best estimate of 96.9% for the Se of the C-ELISA is much higher, although the estimate for the Sp did not change (90.0%). This difference is likely to reflect the problem of defining what a seropositive animal means. Using the VNT as a "gold standard" results in animals exposed to FMD virus up to several years previously [22, 31–33], being classified as seropositive due to the long half-life on neutralising antibodies [32]. By contrast, NSP antibodies do not appear to persist for more than 6–12 months [6, 22] though there seems to be much variation in these estimates [34] and so a different set of animals will be classified as seropositive by an NSP test. This difference in test classification using a poor "gold-standard" could result in the comparative test (in this situation the NSP test) appearing to perform poorly. However, from a disease control and a decision maker's point of view, what is important is evidence of recent exposure to an FMD virus or continuing sub-clinical disease post-vaccination which the NSP tests are directed at.

Using a latent class analysis circumvents the need for an available "gold-standard" test hence providing an alternative to the classical analyses where the problems with selection bias or improper disease definitions. However, there are explicit and implicit assumptions of the latent class analysis, which require that the results of the analysis are scrutinized. Specifically, there is a need to explore the definition of the disease imposed by applying a set of tests. Essentially the definition of disease in a latent class analysis can be perceived as whatever the tests agreed upon. The impact of this is that the definition of disease might depend on the subset of tests used in the analysis. This is very clearly illustrated in Toft et al. [35], where the choice of tests in the latent class analysis dramatically changes the properties of the tests, due to a changed perception of what constitutes a "diseased" individual. The problem is, essentially, that a test result rarely is a direct measure of what we are looking for in the test subject. An example is the use of antibody tests which may detect past exposure rather than current infection with the agent.

In order to explore how the choice of tests might impact the definition of disease in this cattle population, we did a pair-wise comparison of the 3ABC ELISA tests to compare the test properties as well as the estimated prevalences when using all three tests opposed to any combination of just two tests (Tables 3). Given the uncertainty associated with the estimates, there are no discernable differences between the estimates of Se, Sp and prevalence across the combinations of test. This leads us to conclude that the disease definition remains the same regardless of which tests are used for the analysis.

However, for the juvenile sub data set (Table 4), there seems to be some inconsistency between estimates when doing pair-wise comparisons, most noticeably in the estimate of the Se of the CHEKIT ELISA. Comparing the scenario with all three tests to the C-ELISA verses the I-ELISA there are no relevant differences, but when either of these two tests are evaluated alone against the CHEKIT ELISA the estimates seem to shift towards something that indicates a changed perception of what constitutes a "diseased" case. However, it is the author's experience that this shift might also be explained by the combination of the poor Se of the CHEKIT ELISA and the relatively small sample size.

The assumption of conditional independence between the tests given disease status is vital to the presented analysis. The disease status defined by three NSP ELISAs for FMD virus must be considered a measure of the presence of NSP antibodies. Conditional on such a 'disease status' there is no reason to assume that the three tests are not conditionally independent. It is possible to assess pair-wise conditional dependence, i.e. assuming that one tests is conditionally independent given disease status of the other two potentially dependent tests. However, a model assuming that all three tests are conditional dependent given disease status is unidentifiable and can as such not be evaluated without specifying informative priors. Furthermore, it would not make sense to assume such a correlation structure as it would merely imply that none of the tests were measuring what we are looking for. For completeness, we compared the three models allowing for pair-wise conditional dependence to the model assuming conditional independence between tests given disease status using the Deviance Information Criteria (DIC) [36]. The DIC did not suggest that any of the models allowing conditional dependence between two tests given disease status were to be preferred over the model assuming conditional independence between the three tests given disease status.

The prevalence estimates from this analysis follow a similar pattern across the five Divisions to the herdsmen reported outbreaks for the 12 months prior to the interview based on a structured questionnaire. The herdsman reported estimates were 77%, 57%, 37%, 44% and 73% for the Vina, Mbere, Djerem, Mayo Banyo and Faro et Deo Divisions respectively [37, 38]. The model assumes that the tests perform the same in all the sub-populations (i.e. Divisions) and that only the true prevalence differs. These Divisions were the obvious way to divide the population but others could have been used. The effects of using different ways to divide the population were not explored. The prevalence estimates based on the juveniles only were significantly lower than for the full population sample. This would be expected since only relatively recent exposure is captured in this subpopulation.

The estimates of Se and Sp for the NSP tests presented here are lower than those obtained from experimental and field models in South American livestock populations for the I-ELISA [9, 10, 21] as well as for the C-ELISA and CHEKIT tests (unpublished results). Studies using sera panels from FAO have also suggested very high parameter estimates of 98% Se and 99.7% Sp in non-vaccinated cattle for the Ceditest^® (C-ELISA) and 98% Se and 97.2% Sp for the CHEKIT ELISA [39]. Estimates for the C-ELISA give similar ranges of specificity of 95.1% before and 99.4% after heat inactivation in naïve cattle though these drop slightly to 85.3 and 95.7 in vaccinated cattle [40]. A more recent intensive validation exercise has produced estimates of Sp of 97.3% for the I-ELISA, 98.2% for CHEKIT and 97.2% for the Ceditest (the C-ELISA). These are higher than the estimates from this study with the exception of the CHEKIT test [41]. Estimates of diagnostic Se varied with time since infection but were 100% for the I-ELISA more than 100 days post infection compared to the 50% for both the CHEKIT and Ceditest [41]. It is clear from this that the estimates of diagnostic Se depends in part on the time from infection/exposure, the mix of animals sampled and their exposure history. These population specific factors will influence the performance of these test when applied to real populations in the field compared to panels of experimentally derived sera and may in part explain some of the variation in estimates being reported. It is not clear how significant these differences are in practical terms. In addition, it is also important to remember that test parameters are population specific and it is a well recognized problem that diagnostic test can have specificity problems when applied in an African context [42].

The use of latent class models for the evaluation of diagnostic tests in the absence of a "gold-standard" test constitute a useful alternative or addition to the classic analysis for evaluation of tests, such as antibody tests for FMD. This work has also highlighted the problem of defining "disease" on the basis of serological tests, particularly when serological results have to be interpreted without additional epidemiological information. It also suggests that the C-ELISA and I-ELISA are both highly sensitive tests for use in cattle populations with multiple exposures to different FMD virus serotypes. However, these NSP tests all lack specificity and this may create difficulties in designing sero-surveillance strategies as the number of herds with false positives may overwhelm the available resources to deal with them unless a confirmatory test such as the EITB is also used.