fiddle: a tool to combat publication bias by getting research out of the file drawer and into the scientific community

Many laboratories have a ´file drawer’ [1] of unpublished data from well-designed experiments. There are many reasons why data may end up in the file drawer [2,3]. For example, the research team may not have the time or expertise required to analyze the entire data set. Lower priority datasets may remain unpublished, as lab members focus on preparing manuscripts containing the results of high priority experiments. Parts of the study may be missing or incomplete. The study may be a failed replication attempt. Personnel responsible for the project may have left the laboratory before writing a manuscript, or the authors may have published some parts from a larger study, but not others. Alternatively, editors may have rejected the manuscript because the findings were not exciting enough for publication in the authors’ journal of choice.

Regardless of the reasons, failing to publish data from well-designed experiments creates problems for individual researchers, the scientific community and the public. Scientists in preclinical and translational research have invested time and research funds to design and conduct studies yielding valuable data that they have either chosen not to publish, have not been able to publish, or have only partially published [4–6]. Funding agencies and the public do not learn anything from research that is not shared; hence the resources used to complete the work are wasted [7]. Other laboratories, who have no way of knowing that the research was ever conducted, may invest additional time and funding to repeat the same types of studies. Additional problems depend on the type of study. When data from animal studies are not published or shared, animals suffer or are killed without benefits to scientists or society [8–10]. Publication bias can also create risks for patients. Unpublished preclinical data can lead to flawed decisions about whether a potential therapy should advance to clinical trials, exposing patients to unnecessary burdens and risks [11]. When results from neutral or negative clinical trials are not published, clinicians’ decisions and recommendations about patient care are based on incomplete evidence [11].

Studies with neutral and null results are more likely to end up in the file drawer than studies with statistically significant findings [12]. This publication bias (Box 1) leaves scientists, funding agencies and clinicians with a distorted view of the scientific evidence, which can lead to poor decisions about what research directions are most promising and should be funded or what medical treatments should be recommended to patients [13]. Such practices can have detrimental consequences. During the 1980s, over 100,000 people died after receiving lorcainide-class like drugs. These antiarrhythmic medications were routinely prescribed to patients after a heart attack. A publication with data on the lethal side effects of lorcainide was repeatedly rejected and ultimately not published for 13 years, as the authors did not interpret the death rates in their small study as conclusive evidence and journals repeatedly refused to publish these null results [14–16]. While this example is extreme, it illustrates the potential harmful effects of publication bias. Selective reporting of results can create similar problems.

Unfortunately, scientists don’t know what proportion of data are never published because there is no comprehensive registry of all planned studies. A study in social science found that two thirds of survey-based experiments that produced null results ended up in the file drawer, whereas nearly all experiments with statistically significant results supporting the underlying hypothesis were published [18]. Many reports confirm the same phenomenon in the medical field, where negative results are less likely to be published [12,19,20]. Clinical trials offer another unique opportunity to assess publication bias, as journal editors began requiring trial registration in 2005 [21,22]. Estimates from AllTrials (http://www.alltrials.net/), based on comparisons of registered versus published trials, suggest that approximately 50% of clinical trials results remain unpublished [23,24].

New publication formats (Box 2) make it easier for scientists to share research, regardless of the outcome (Table 1), while also ensuring that the data become a part of the permanent scientific record. Tables 1 and 2 and fiddle explain and compare these different formats. fiddle also provides links to websites for publishers of each publication format, which researchers can use to find sample publications that may be relevant to their field. Fiddle focuses on generalist publishers that publish papers from many different fields; it does not provide a comprehensive list of discipline specific publishers. The tool does include links to curated lists designed to help readers identify specialized repositories and discipline-specific databases (i.e. re3data.org, fairsharing.org, and Nature’s list of recommended repositories). Users who are interested in discipline specific repositories or databases can use these links to identify suitable options once they have chosen a publication format.

Some of the new publication formats follow the same format as traditional peer-reviewed research articles, but make it easier to publish manuscripts that would typically be rejected from journals where editorial and peer review often prioritizes exciting results. Preprints, for example, are unpublished manuscripts that have not been peer-reviewed and are shared immediately with the scientific community. Platforms that publish preprints include bioRxiv, medRxiv, and Open Science Framework Preprint Services. Publication platforms, such as F1000 Research and Open Research Central, publish articles immediately without editorial filtering. Open peer review occurs after publication. Journals that are open to null results are traditional journals that publish peer-reviewed manuscripts but welcome all studies regardless of outcome. Such journals have clear public policies to publish manuscripts describing well-designed studies with null results, results that appear to contradict those of previous publications [32] or other research outcomes that are hard to publish. Examples include PeerJ and PLOS ONE.

Other new publication formats facilitate publication of data or results that would be difficult to share in an Introduction-Method-Results-Discussion format. Micropublications, for example, are very short publications designed for unpublished observations, neutral or null results or other research that does not require a scientific narrative [33]. Platforms that publish micropublications include Science Matters and BMC Research Notes. Data repositories, such as figshare, Zenodo or Dryad, allow scientists to upload small or large research datasets to make them citable and reusable. Data journals, such as Scientific Data or Data, publish journal articles that present a dataset, metadata explaining the dataset and the data collection methods.

As the use of preprints [25] and other alternative publication formats continues to grow, even researchers who choose not to use these formats will benefit from understanding how they work. Knowing where these research outputs are indexed and whether they are peer reviewed, for example, is essential to finding and evaluating materials that are relevant to one’s area of research. Scientists who are unaware of data repositories and data journals may miss opportunities to use datasets relevant to their work. Researchers who don’t know about rapidly growing preprint servers may not find out about important studies until papers are published, often many months after the preprints were first posted. Tools that help researchers to understand different publication formats and identify those formats that are most appropriate for the dissemination of their data are thus urgently needed [34].

fiddle is a free, open source ´matchmaking’ tool designed to help researchers to identify the publication format that will work best for a particular dataset or study that may be hard to publish in traditional journals (RRID:SCR_017327, available at: http://s-quest.bihealth.org/fiddle/). The tool includes a link to a brief video tutorial. Researchers can use this shiny (RRID:SCR_001626) [35] app to quickly compare characteristics of different publications formats and search for a format that best meets their needs. Fiddle is not discipline-specific and can be used for any life science field where publication occurs and where research results from well designed and executed studies remain hidden in the file drawer. Once users have identified a publication format in fiddle, they can click on links to visit websites of relevant publishers or platforms, or see examples of this particular format.

There are two ways to search for publication formats (Figure 1). The first filtering option is to search by important characteristics describing the dataset and the researcher's publishing-related preferences. Users can find suitable publishing platforms by answering the questions below that are most relevant to them:

1. What type of unpublished information do you have (unanalyzed dataset, rejected manuscript, etc.)?

2. Amount of funding available for publication costs

3. Where should the publication or dataset be indexed?

4. Do you want the publication or dataset to be peer reviewed?

5. Do you want the publication or dataset to appear immediately?

The other, alternative filtering option is to search by scenarios that describe the reason why the information is unpublished. Example scenarios include ´I don’t have enough time to prepare a publication’, ´I have data that may be useful to others, but am not able to analyze everything’, and ´My study is completed, but the findings aren’t novel or exciting’. The tool highlights publication formats that meet the user's requirements. Users can review detailed information on each type of publication format and then click on links to visit websites for different publishers. Users can also compare all publication formats. All options in fiddle provide a permanent, citable and findable link to the data or manuscript. Many formats are also peer-reviewed. The source code for fiddle is available at https://github.com/quest-bih/fiddle.

Scientists should focus on the quality of the study methods, rather than the desirability of the results, when deciding which file drawer data to publish. While data from well-designed experiments often ends up in the file drawer, many file drawers also contain data from poorly designed, badly conducted or insufficiently documented experiments that are unlikely to be reproducible or useful. fiddle and other efforts to reduce publication bias and selective reporting encourage authors to publish data from well-designed experiments that may be useful to the scientific community or the public, regardless of whether the findings were statistically significant. This applies also for datasets that are too small to yield reliable conclusions, however may be informative when combined with many other datasets using techniques such as meta-analysis.

fiddle is not intended to promote publication of data from poor quality studies that are unlikely to be useful or informative. Authors should consult study design and reporting guidelines when designing studies and preparing publications to increase the likelihood that data will be transparent, rigorous and reproducible. Table 3 lists guidelines for common types of studies in many fields, including observational studies, animal studies, randomized controlled trials, and systematic reviews and meta-analyses. Guidelines for other types of studies can be found through the EQUATOR network website (RRID:SCR_012861).

The goal of publishing file drawer data is to make these research outputs available to the scientific community; therefore, scientists should ensure that the information is shared in a form that others can understand and use. The list below outlines some important features that should be reported for most, if not all, formats listed in fiddle. Additional information may be needed, depending on the publication format, study design, experimental methods, and type of data that is generated. The lack of time is one reason for not publishing file drawer data [3]; therefore there may be trade-offs between efforts to reduce publication bias by introducing shorter publication formats that take less time to prepare and attempts to improve transparency and reproducibility by encouraging authors to report detailed information required to assess study quality. Information that scientists need to interpret and use scientific data include the following:

1. Research question: The material provided should clearly specify the research question that the study was designed to answer, along with any hypotheses.

2. Participants, subjects, specimens or samples: The material should specify who the participants or subjects were, and how specimens or samples were obtained. When appropriate, the authority that gave regulatory study approval should be stated (i.e. institutional review board, animal care and use committee, etc.). Human studies should state how patients were consented.

3. Study design: The material should specify the study design, and state whether the study was exploratory or confirmatory. Important design features needed to assess the risk of bias should be reported. These include whether the measurements and analyses were performed in a blinded fashion, whether participants or subjects were randomized to the different conditions and how randomization was performed, a power calculation or sample size justification, and details on the number of excluded observations and reasons for exclusion [37,38].

4. Data: A scientist without prior knowledge of the experiment should be able to interpret and use the dataset based on the meta-data provided. The dataset should be compliant with the respective Minimum Information for Biological and Biomedical Investigations (RRID:SCR_002042, https://fairsharing.org/collection/MIBBI) and include a data dictionary that clearly explains what each variable is, what the measurement units are and how the variables were measured (https://dataedo.com/blog/different-types-of-tools-you-can-use-to-create-data-dictionary). Data should have a license specifying any conditions for re-use. Authors who share data should consult the FAIR data principles [39] and plan their data documentation [40]. When depositing data obtained from human samples or patient data, regional data protection laws and legislations apply and need to be considered prior to the start of the project to found out which form of consent, de-identification procedures or data access restrictions may apply. Some research institutions employ a data protection or open data specialist to help researchers with open data issues. Investigators working with patient data should contact their institutional review board for guidance.

5. Results: Readers should know what was measured, be able to determine sample sizes for each group and/or analysis and know what summary statistics are reported.

6. Analysis: If the data were analyzed, the material should provide enough information to determine how the analysis was conducted. This could include code for the analysis. The SAMPL guidelines [41] recommend providing enough detail so that a reader who understands statistics could reproduce the analysis if he or she had access to the data.

7. Limitations: The limitations of the data or study should be clearly explained.

8. Contact person: If the uploading author is not the best person to answer additional questions, the name and contact information for one or two people with such knowledge should be provided.

Researchers can take several steps to reduce publication bias and accelerate scientific discovery. The first step is to plan ahead. Research teams should ask all collaborators to commit to publishing all study results, regardless of the perceived importance of the results and whether the results support the hypothesis. An additional strategy is to pre-register a study by posting a publicly available, time stamped protocol that outlines the study objectives and hypotheses, data collection procedures and planned analyses. Cite this pre-registration when publishing the study, regardless of which publication format is used, and provide an explanation if the final study differs from the pre-registered protocol. Pre-registration addresses publication bias by allowing researchers to identify studies that were conducted, but not published. Studies can be pre-registered on sites like AsPredicted (RRID:SCR_018789, https://aspredicted.org) and the Open Science Framework (RRID:SCR_003238, https://osf.io)

Once the study is complete, researchers should share all findings with the scientific community using traditional peer-reviewed publications or alternative publication formats described in fiddle. Specific actions that researchers can take include using repositories and other platforms to share data and protocols, and avoiding ´data not shown’ statements. Scientists who have a sound scientific reason not to report data should specify this when publishing or sharing such study results. Investigators might report, for example, that one variable measured was not reported due to device malfunctioning on the day the test was performed. Finally, scientists should talk to their colleagues about the consequences of publication bias and selective reporting. These conversations are especially important when co-authors, reviewers or editors encourage selective reporting.

The open source fiddle tool is a match-making Shiny app designed to help researchers identify the publication format that is most appropriate for their publication or dataset. Users can search for a publication format that meets their needs, compare and contrast different publication formats, and find links to publishers and examples. This tool will assist scientists in getting otherwise inaccessible data from well-designed experiments out of the file drawer and into the scientific community to reduce bias in the scientific literature. Finally, funding agencies, journals, and hiring and promotion committees need to incentivize and reward publication of all research from well-designed experiments, regardless of the form of publication. Some investigators may be reluctant to publish studies that are unlikely to be accepted by journals with high impact factors due to concerns that funding agencies or promotion and tenure committees may devalue this work, adversely affecting career advancement. This perception bolsters publication bias by encouraging scientists to publish only their most interesting and impactful research, to the detriment of the scientific community and the public. We hope that this paper and the tool will raise awareness of the negative consequences of publication bias and selective reporting, and encourage the scientific community to work towards individual and systemic change.

The authors declare that there are no competing interests associated with the manuscript.

R.B. was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy – EXC-2049 – 390688087. T.L.W. was funded by American Heart Association [grant number 16GRNT30950002]. This publication was made possible by CTSA [grant number UL1 TR000135] from the National Center for Advancing Translational Sciences, a component of the National Institutes of Health. The content is solely the authors’ responsibility and does not necessarily represent the official views of the NIH. The writing of the manuscript and the decision to submit it for publication were solely the authors’ responsibilities.

fiddle

file drawer data liberation effort