5 Experimental Errors That Prevent Your Flow Cytometry Data From Passing The Peer Review Process

Reproducibility is a critical component of the scientific process. One cannot publish data if the experiments cannot be replicated are are full of errors.

Unfortunately, as Begley and Ellis pointed out in a commentary in Nature that when Amgen attempted to reproduce 53 “landmark” papers in the area of cancer research, only 6 papers were “confirmed.”

What does that mean to you, the flow cytometry researcher?

To avoid publishing errors that reviews despise, it’s important to follow and promote the best practices in the field, thus ensuring that your data is reproducible to investigators attempting to validate your research.

In particular, you must follow these 5 experimental tips before embarking into the peer review process…

Error 1: Know what your hypothesis is and design the experiments to test your hypothesis.

It may seem obvious, but the goal of the experiments is to prove some hypothesis that you are testing. In examining the hypothesis, there are several questions that should be on your mind…

A. Is flow cytometry the right tool to answer the question?—Sometimes flow isn’t the best way to answer the question so it’s better to make the decision before starting experiments.

B. Is flow cytometry a path to another technique?—For example, will sorted cells be used to perform experiments such as RNAseq, Proteomic analysis, cell culture etc? If so, what controls will you need to build into the experiment to help evaluate the handoff between technologies?

C. Is there another technology that will confirm the findings?—It is always useful to have data from two different technologies that can confirm each other. For instance, flow cytometry brings statistical power to examining cell populations from other experiments.

Error 2: Know which statistical tests are appropriate for your data.

Proving the hypothesis will require statistical analysis of the data. The experiments must be designed to withstand the statistical tests that you will be using. It is a best practice to decide on which of the statistical tests you will be performing BEFORE you start your experiments. This means you must do the following…

A. Define your threshold—The alpha value is typically set to 0.05 based on work by RA Fisher. However, this is only a guideline and should be evaluated for each experiment.

B. Define your statistical test—Depending on what is being compared and what assumptions are being made about the data (e.g. Gaussian distribution or not), the appropriate test should be identified before starting.

C. Determine the appropriate sample size—The number of replicates, number of samples is critical to determine before performing the experiments. As this little vignette shows, it is more than just “always using 3 patients.” As the power of an experiment increases, the chance of a Type II (false negative) error decreases.

Error 3: Know your instrument.

The operation and characterization of the instrument is essential for good reproducible data. The individuals who manage the instruments (core directors, shared-resource facility managers) tend to establish and run quality control on the instruments. This is so they can identify and repair issues before they impact the end-user.

However, there is a lot of other information that the end-user needs to be able to design good multi-color experiments, run at optimal voltages, maintain consistency between runs and the like, including…

A. Understand the configuration—It may seem obvious, but the first step is knowing the instrumentation configuration, such as the excitation lines, the power of the lasers, the emission filters and PMTs. For example, if the instrument has a high-power 488 nm laser PerCP is not recommended. Also, when using fluorochromes on multi-lasers instruments it helps to model the excitation and emission profile, to see where there may be issues. Spectral viewers from various vendors are quite useful in this. Here are some useful links to these viewers…

• BD Spectral Viewer
• Biolegend Spectral Viewer
• eBioscience Spectral Viewer
• ThermoFisher (Invitrogen) Spectral Viewer

B. Understand the PMT (Part I)—With analog systems, investigators were taught to put the negative populations in the first decade. With newer digital systems, it is a best practice to determine the optimal voltage settings of your instrument. This was described in a paper by Maecker and Trotter. There are other methods out there, but the “peak 2 (RCP-30-5A-2)” method is instrument and vendor agnostic. Using this method helps establish where the PMTs are most sensitive.

C. Understand the PMT (Part II)—When the first 17 color flow cytometry flow cytometry panel was published, the authors discussed how to determine which PMTs were most sensitive and which PMTs received the most error from the fluorochromes used in the panel. Determining PMT sensitivity helps determine which fluorochromes to use and which to avoid. This work has been expanded recently in two articles that carefully evaluate spectral spillover—Nguyen et al., (2013) and Perfetto et al., (2014).

Error 4: Know your antibody panel.

Your antibody panel is the lifeblood of your flow cytometry analysis. Properly designed, optimized and executed, and the answers will be revealed. Poorly designed panels or a lack of optimization will limit the utility of the panel for interpretation. To avoid this, make sure you follow these four tips…

A. Automate your antibody panel—Starting with the hypothesis and knowing the intimate details of the machine you will be using is the start of the process. Next becomes the selection of the antibodies and fluorochromes. This can be a tedious process of searching through multiple catalogues, knowing fluorochrome names (e.g. TriColor = CyChrome + PE-Cy5), and keeping up-to-date with the latest releases on the market. Instead, it’s best to consider using software automation for flow cytometry panel design. These automated programs will help you search the various databases and ensure that you find fluorochromes that (1) are available (2) work on your instrument. The three most common tools for antibody panel automation are (in alphabetical order)…

• Chromocyte
• Fluorish
• Fluorofinder

B. Know how bright your fluorochromes are—Unless you keep updating your staining index panel as new fluorochromes come out, you might want to look to these lists of brightness for different fluorochrome—BD Biosciences and Biolegend.

C. Match fluorochromes to the right antibodies—The goal of panel design is to maximize the signal on the markers where critical measurements are to be made, while minimizing the effects of spectral spillover in those channels. Remember, the more fluorochromes used in a panel, the greater the spread of the data and impact on sensitivity. Design the panel based on the following rules…

• Highly expressed antigens paired with dimmer fluorochromes
• Antigens for subset determination paired with mid-bright fluorochromes
• Unknown and/or low expression antigens paired with brightest fluorochromes
• Always strive to minimize spectral overlap that will reduce sensitivity in your panel.
• Don’t forget a dump channel and viability dye – slot those in where you have room, but don’t forget them.

D. Research and use the proper OMIP—The Optimized Multicolor Immunofluorescent Panel is a journal type describing polychromatic panels that have been vetted and optimized. OMIP panels are a great place to start if you’re in a quandary on how to build a panel to answer the hypothesis you’re working on. There are over 25 published OMIPs, and each are required to provide the sample types the OMIP has been validated on, list the antibodies and fluorochromes used, and provide an instrument configuration along with a typical analysis.

Error 5: Know your data.

After all your cells are stained and run on the instrument, it’s time for another important step in the process, gating and data analysis. Extracting the critical information that will be used in statistical analysis to prove (or disprove) the hypothesis follows a path from beginning to end. As you approach the end, keep these four things in mind…

A. Begin with the right controls—Proper data interpretation requires the correct controls to be run as part of the experiment. One of the best articles on flow cytometry controls is this one by Hulspas et al. Without the right controls, the data will be difficult (even impossible) to interpret. The right controls include…

• Compensation controls
• Gating controls (aka – FMO controls)
• Stimulation controls
• Reference controls
• Instrument controls

B. Gate generously—Gating is the process of excluding data from the final analysis. Thus, it is critical to be careful with your gating. Don’t rely on forward and side scatter alone to define your populations. Instead, let the antigens in the panel do the required cutting and fine-tuning of the data. Here are four gates you should become familiar with…

• Viability gate—This gate is used to define the live cells in your populations. This can be combined with a dump gate to give the viability + dump-population. Here, the reagents are doing the heavy lifting helping to get rid of the things that you don’t want to analyze.
• Singlet gate—This gate is used to define the non-clumping cells based on pulse geometry. Don’t forget to collect the height and width measures for the physical parameters, as well as area, when you set up the instrument.
• FSCxSSC gate—This gate is used to remove debris, including cell fragments and other “smutz” in lower left-hand portion of the plot, as well the very large (off-scale) debris usually found in the upper right-hand portion of the plot.
• Phenotypic antigen gate—This gate is used to define your final populations of interest. Use these gates carefully as if you were a sculptor trying to uncover the beauty beneath the rough.

C. Backgate and analyze the ancestry—Backgating is a process that allows you to review the gates and the shape of your final populations of interest. These gates allow you to determine if you’ve missed part of the population. A typical example would be in a simulation experiment, where the cells became larger and are therefore missed because of a too restrictive forward and side scatter gate. Likewise, if the cells are fixed, they can get smaller and again be missed with restrictive gating.

D. Grab the right statistics—With all that work done, make sure you collect the right statistics to answer your hypothesis.

E. Share your flow cytometry data—consider using the MiFlowCyt standard to communicate the essential pieces of your flow experiments to the world. This will add a level of detail to your process that makes peer reviewers happy. You can also share your data with the greater scientific community using the Flow Repository. This repository allows curious researchers to review your data and perform independent analyses on it, increasing their confidence in your conclusions. Data repositories are not a new idea, and have been used for many years in the protein and nucleic acid research communities, but have only recently (2012) come to the field of flow cytometry.

In the peer review process when reviewers evaluate article submissions, they look at the above criteria and that helps guide their review of the material. If there are flaws, cut corners, or missing information—then the material is destined to be rejected. Those papers that answer the questions above, and demonstrate a complete understanding of the process will be reviewed positively. By answering the above questions and following flow cytometry best practices, your experiments will be next in line to be published.

To learn more about publishing your flow cytometry data, and to get access to all of our advanced materials including 20 training videos, presentations, workbooks, and private group membership, get on the Flow Cytometry Mastery Class wait list.