Sponsorship bias and quality of randomised controlled trials in veterinary medicine

In order to effectively practice veterinary medicine in an evidence-based way, it is imperative that accurate scientific evidence is available so that the evidence base is complete, reliable, and therefore not misleading. Randomised controlled trials (RCTs), along with their synthesis in the form of systematic reviews, are considered to be the gold standard method for assessing the efficacy of treatment interventions and are a valuable source of information on which to base clinical decisions [1]. The results of RCTs can however be affected by many biases including selection, performance, detection, attrition and reporting biases [2, 3]. The presence of bias can lead to misinterpretation of treatment efficacy or harms, and mislead clinicians when putting the evidence into practice.

Sponsorship bias (the influence of funding source on the reporting of trial results) is an additional potential problem when assessing the reliability of RCTs. The medical literature contains differing reports over whether financial conflicts of interest influence the reported results of a trial. Some studies report a greater likelihood of positive results for industry funded trials [4, 5], while some report no difference between industry and non-industry sponsored trials [6, 7]. A recent overview of medical literature in a Cochrane systematic review concluded that drug and medical device studies were more likely to report favourable results when the study was sponsored by a manufacturer [8].

There have been several studies examining the methodological and reporting quality of clinical trials in the published veterinary literature [9–11]. Such studies have highlighted issues with the reporting of RCTs and have shown how these reporting deficiencies are associated with an increased likelihood of a trial reporting one or more positive outcomes [10]. To our knowledge, no studies to date have examined the influence of funding source on the likelihood of reporting positive outcomes in the veterinary RCT literature.

The aim of this study was to examine the relationship between funding source and proportions of positive outcome reporting in veterinary RCTs involving a pharmaceutical intervention published in a single calendar year (2011). A secondary aim was to assess the risk of bias of veterinary RCTs published in the same time period.

A cross-sectional study of veterinary RCTs was conducted. The target population was feline, canine, equine, bovine and ovine RCTs where a pharmaceutical agent was the intervention of interest and efficacy was assessed. The sample population was feline, canine, equine, bovine and ovine RCTs published in 2011 within journals indexed in PubMed.

Search strategy and filtering of results

A structured search of PubMed was conducted in June 2013 using the “clinical trial” Publication Type combined with the relevant species MeSH heading e.g. “clinical trial” [publication type] AND cats [mh]. This was done for each of the 5 species studied: cats, dogs, horses, cattle and sheep (Fig. 1). The search was limited to one calendar year with a PubMed filter: 01/01/11–31/12/11. Search results were exported into EndNote® software for filtering. Papers indexed as RCTs by PubMed (“randomised controlled trials” [publication type]) were extracted, investigators then confirmed if they were RCTs according to the Cochrane definition below (http://www.cochrane.org/glossary/):

“An experiment in which two or more interventions, possibly including a control intervention or no intervention, are compared by being randomly allocated to participants. In most trials one intervention is assigned to each individual but sometimes assignment is to defined groups of individuals (for example, in a household) or interventions are assigned within individuals (for example, in different orders or to different parts of the body).”All publications containing trials confirmed by the investigators as being RCTs, published in 2011, and relevant to the species of interest were then categorised into four intervention subcategories based on the main intervention of interest of the study (Table 1 - Level 1 exclusion criteria):

1. 1.

   Pharmaceutical – consisting of an active pharmaceutical ingredient, including anthelmintics and vaccines

    
2. 2.

   Nutritional

    
3. 3.

   Para-pharmaceutical – including probiotics, prebiotics, synbiotics, nutraceuticals and supplements/vitamins/minerals if not considered part of the total dietary ration

    
4. 4.

   Other – including surgical interventions, management/husbandry interventions, non-medicinal shampoos, studies relating to diagnostic tests.

    

Level 1: Inclusion criteria for publications	Level 1: Exclusion criteria for publications
Species of interest is cats, dogs, horses, cattle or sheep	Not about cats, dogs, horses, cattle or sheep
Published in 2011	E published only in 2011 if full publication occurred in a different calendar year
RCT according to PubMed publication types and the Cochrane definition	Not an RCT (not indexed as an RCT by PubMed or not fulfilling Cochrane definition of an RCT)
Treatment of interest is a pharmaceutical intervention (including anthelmintics and vaccines)	Treatment of interest is not a pharmaceutical agent e.g. nutritional, surgical, animal husbandry etc
Level 2: Inclusion criteria for analysis of pharmaceutical RCTs	Level 2: Exclusion criteria for analysis of pharmaceutical RCTs
Primary aim is to assess efficacy	Primary aim was not to assess efficacy (pharmacokinetic/dynamic studies, safety studies, physiological effects, resistance testing, testing routes of administration only, testing timing of administration only)
Identifiable treatment or protocol of interest	Treatment or protocol of interest could not be identified
Single dose of the treatment of interest used	Multiple doses of the treatment of interest used/dose finding studies
Published in English	Not available in English

Only publications within the ‘Pharmaceutical intervention’ subcategory were included in this study; these were assessed for further eligibility for analysis according to the second level of inclusion and exclusion criteria in Table 1.

Publications included in the analysis were therefore single dose efficacy studies of pharmaceutical interventions in cats, dogs, horses, cattle or sheep published in 2011. In the case of a publication containing more than one trial, each trial was included independently in the analysis if it met all inclusion criteria.

This study found a significantly higher proportion of positive outcomes reported in RCTs with pharmaceutical funding (56.9%) or involvement compared to those with declared non-pharmaceutical funding (34.9%) or with no funding source stated (29.1%) within the sample of literature studied. There was a correspondingly lower proportion of negative outcomes reported in trials within the pharmaceutical funding group (23.5%) compared to the other two groups (37.6% and 37.1%). When assessing the trials for risk of bias across the five main categories using the Cochrane risk of bias tool, a large proportion were at an ‘unclear’ risk indicating significant reporting deficiencies. A high risk of bias was most predominantly seen for selective outcome reporting (reporting bias), and more moderately for incomplete outcome data (attrition bias) and blinding (detection bias). Proportions of trials at high, low or unclear risk of bias for the different quality criteria were largely similar across funding categories.

The sponsorship bias detected in this study is in accordance with many reports in the medical literature where an association between funding source and positive results has been demonstrated, most notably in a Cochrane Review of drug and medical devices [8]. There are many reasons why such a bias may be present in the published literature including differences in the methodological quality of trials; inherent biases in trial conduct to favour a treatment; a genuinely greater likelihood that pharmaceutical companies would be testing pharmaceutical agents that are likely to perform well; and inadequacies in trial reporting which favour a treatment. Additionally, publication bias may play a role through researchers within different environments potentially being more or less likely to publish trials demonstrating a positive effect compared to trials showing a ‘negative’ result. Further studies are required to examine this finding and its potential causative factors in more detail, in particular whether there are correlations between quality criteria and funding source, something which this study did not investigate.

There are a variety of methods that could have been utilised for the current study. For example, in medical literature reviewing the presence of sponsorship bias, it is common to report one overall conclusion for a paper (i.e. overall the paper has a positive/negative/not significantly different outcome) determined either by the reviewers, based on the assertions of the authors or on the statistical analysis of one primary outcome of the study [4, 7, 12]. The method we have used, whereby we have extracted each outcome and its result, is more achievable in the veterinary literature, as primary outcomes are often unspecified [10, 13], but different results would potentially be obtained using a different approach. Of note in this study is the potential for differences between species, and potential clustering of some types of trials, e.g. anthelmintic efficacy trials, to have skewed the data; these limitations will be discussed in more detail below. To date, we have found no other publications examining the association of funding source with positive outcome reporting in the veterinary literature with which to compare our results. The group of trials with no funding source stated are particularly difficult to assess in this study as no assumptions can be made as to which of the two other groups they would most appropriately belong to. Within the results, they appear to be most like the non-pharmaceutical group of trials in their characteristics, but this in itself highlights a continuing problem of poor reporting of clinical trials (20% of trials in this study did not report a funding source).

Selective outcome reporting, for example not reporting, or incompletely reporting, results for pre-specified outcomes, or reporting outcomes that were not pre-specified, can introduce reporting bias into a study and influence the overall results [2, 3]. A striking feature of our data was the high proportion of outcomes that were described only (18.9%) or were mentioned in the materials and methods then not reported in the results (10.3%). This could partly be due to manuscripts not detailing clearly which of the parameters being measured were intended to be outcomes used to assess efficacy, leading us to misclassify the information, highlighting again the issue of poor reporting. A previous study reporting quality criteria and outcome data from a sample of dog and cat trials also reported a high percentage of outcomes with no formal statistical analysis (31%) and a lower percentage not reported at all (3.1%) [10]. The proportions of outcomes in these two categories contribute to the overall high risk of reporting bias (selective outcome reporting) found in this study. Research has shown that outcomes that are not reported, or incompletely reported are more likely to be statistically insignificant [14, 15]. This highlights the need for pre-specified primary and secondary outcomes to be explicitly stated in the methods and adhered to when reporting results. One approach which should help to combat this problem is for all clinical trial protocols to be registered in advance, so a comparison can be made with the final report; this approach is being championed by the AllTrials campaign in human medicine. AllTrials aims to ensure that all clinical trials are registered before they commence and that all are fully reported [16] (www.alltrials.net). A similar initiative is currently underway for veterinary clinical trials [17]; these schemes should also help to combat publication bias. Publication bias, meaning negative studies are less likely to be published than positive ones, is a problem that has been identified across scientific publishing generally and which can lead to over estimates of treatment effects [3, 15]. The potential impact of publication bias on our study results would depend on who was funding any unpublished trials.

The high proportions of ‘unclear’ risk of bias for the five quality criteria assessed in this study indicate a significant issue with poor reporting, a feature which has also been described in previous quality assessments of veterinary clinical trial literature [9, 10, 13]. This does not necessarily equate to poor methodological trial conduct, but a lack of complete reporting means that the methodology cannot be adequately assessed [18, 19]. This study did not set out to assess the impact of risk of bias on levels of positive outcome reporting. However, it has previously been shown in both veterinary and medical literature that incomplete or inadequate reporting of certain quality criteria (e.g. method of randomisation) is linked to an exaggeration of treatment efficacy [10, 13, 20, 21].

The CONSORT reporting guideline was developed in order to improve the reporting of RCTs, making it easier to ascertain what was done, identify possible sources of bias, and evaluate the reliability of a study [22, 23]. In general, the adoption of the CONSORT checklist has improved the reporting of RCTs in the medical literature, but there are still reporting deficits [24, 25]. In veterinary medicine the REFLECT statement is also available, which is an extension to the CONSORT reporting guideline specifically developed for RCTs involving livestock [26, 27]. Strict adherence to such reporting guidelines [28] should have reduced all the ‘unclear’ assessments of bias made in this study and would have allowed us to identify the funding source of all the trials. Most importantly, this would allow more reliable assessments of treatment efficacies to be made, meaning more effective translation of evidence into clinical practice. A recent survey assessing the awareness of reporting guidelines amongst veterinary editors reported that 35.1% of journal editors said reporting guidelines were referred to in their instructions to authors [29]. An improvement in the endorsement of reporting guidelines by journals could help to improve the reporting quality of the veterinary clinical trial literature as it has done for medicine.

A significant limitation of this study is that there were a relatively small number of trials included in the analysis, and due to the large proportion of pharmaceutical trials in the sample (68%), the groups for comparison were unbalanced and the non-pharmaceutical group small. Another, larger study would be extremely beneficial in assessing the presence of sponsorship bias in the veterinary clinical trials literature. In particular, an exploration of potential differences between species, or between companion animal versus production animals, warrants further investigation with larger sample sizes (no significant differences were found in the current study, see Table 2). Results of this type of study can be very dependent on the methods, including what types of studies are included (e.g. we have only included pharmaceutical interventions), which outcome classifications are used, the way in which outcomes are extracted (e.g. we did not include results for outcomes which were not mentioned in the materials and methods) and how funding categories are divided, meaning results across studies could be very different. Another limitation of this study is that the authors were not blinded to any manuscript details during data extraction potentially leading to biased interpretation. The lack of inclusion of efficacy studies where multiple doses of the test treatment were used is another significant limitation of the study. On balance it was felt that inclusion of these could potentially skew the results due to multiple entries for the trial by including each dose, or selecting only one of the doses. The inclusion of multiple trials within one publication may also skew results, as the methods, and therefore assessment of quality, tend to be identical for all the included trials. As most multiple trial papers were in the pharmaceutical category, this could potentially lead to clustering of information. Of particular influence in this study were RCTs assessing anthelmintic agents as these often contained multiple similar trials with an overwhelming proportion of positive outcomes. As they fulfilled our initial inclusion criteria they remained in our sample but their impact on the overall results may be substantial. The subjective assignment of a single outcome result for an outcome which was assessed at multiple time points is another limitation which was necessary for practicality. Limits to the initial sample size were needed due to cost and time constraints; a single calendar year search in PubMed was chosen to give a representative, recent sample of trials, rather than selecting certain journals to search. Using PubMed also allowed us to search by publication type. Not including studies unavailable in English was a necessary cost and time limitation but only one paper was excluded on this basis so this is unlikely to have affected the study outcomes.

This study found a positive association between pharmaceutical funding or involvement and increased positive outcome reporting. Consistently poor reporting of trials, including non-identification of funding source was identified, which hinders the assessment and use of the limited evidence available to the profession.