Quality data and results are the primary product of any R&D-intensive organization. What’s even more valuable is being able to find all this critical information when you need it later during development. Organizations that generate better, reproducible data, and can build on those results more efficiently, have a competitive advantage.
As part of a recent webcast series designed to provide emerging R&D-intensive organizations with best practices, I had the pleasure of hosting April Clyburne-Sherin, co-founder and executive director of Reproducibility for Everyone (R4E), a community-led reproducibility education project for researchers; and Founder & CEO of protocols.io Lenny Teytelman, for an engaging discussion on establishing and managing protocols, materials and data generation for reproducibility from the very start.
For those new to the term, reproducibility is about the ideas, tools and methods needed to create more rigorous reproducible research at scale and there are four different modes to consider. When we talk about reproducible research, we’re talking about when a particular researcher provides all the necessary materials, including data, codes, methods and protocols to run the analysis again in order to recreate the same results. This is especially important within your own lab, so that everyone is able to reproduce each other’s work. A replication is when a study arrives at the same scientific findings as another, collecting new data and completing new analysis. If using the same data but playing around with different methods and achieving the same results, we are referring to robust research. The gold standard in any research is generalizable research, in which we are able to gather new data and different methods but achieve stable findings over time.
There is currently a gap in education for people conducting research in understanding the main facets of reproducibility including organization, documentation, automation and dissemination of the research.
Why is this important? In establishing best practices around reproducibility to manage and organize your research process better, researchers become more efficient and save critical time. This is especially important for startups with limited staff and resources, and for companies facing turnover to ensure that as one employee leaves, prior work is accurately captured for the next researcher to pick up and be able to reproduce the same results successfully. Lenny suggests that when thinking about reproducibility, many automatically think of a published paper. When in fact, the challenges of reproducibility begin much sooner in the process, so it is important to capture the research protocols from the start or companies can lose months, even years trying to recreate the original methods and similar outcomes.
Another common challenge is that as multiple researchers collaborate on an experiment and the resulting amount of data increases, the experimental outcomes are often stored in varying formats using different software, formats and file types. This makes it quite difficult for researchers to locate and compare data and methods, and most certainly impacts research outcomes and creates inefficiencies. April advises, however, that it is not only about sharing your protocols with fellow researchers:
“Your most likely collaborator is you in six months.”
For various reasons including budget, competing priorities or a leave of absence, researchers can have lengthy interruptions during a study. How confident are you that you could step away from your research for months at a time and be able to pick up exactly where you left off?
In working to achieve successful reproducibility, a first and critical step is in organizing and sharing protocols. Consider a protocol as a brief, modular, self-contained scientific publication. Then, it is necessary to put the protocol in context with a brief abstract. Protocols should include as much detail as possible including duration per step, reagent amount, vendor names, catalogue numbers, etc., as well as capture:
- The expected results
- Safety information and requirements
- The type of software package used
- A chronology of steps
- Additional notes and tips and tricks
To achieve this level of detail in a way that is fully accessible to fellow researchers, teams should consider collaboration tools that digitize and centralize research protocols. It is also important to document in a way that tracks versioning to ensure others are using the most recent version and have a history of prior versions for reference.
Organizations that create and follow standards for protocol tracking, data storage and experimental sharing have a decided advantage. Employees turn over, old results become important and reproducible results accelerate research. Thinking about these steps from the very start of your R&D organization will save you valuable time in the long run.
Learn more about reproducibility and protocols by viewing the full webcast recording and others from my Optimizing Culture and Information Management webcast series for emerging R&D organizations here.
You may also enjoy these related posts:
Building a Culture of Success from the Start: Focus on Management Excellence
Fostering an Inclusive Culture from the Start
Visit CCC’s solutions page for emerging life science organizations and also learn how to reduce the time-consuming article retrieval process, facilitate collaboration across teams, maximize the value of content investments and simplify copyright compliance.