The Right Way To Read A Flow Cytometry Scientific Paper

Scientific reproducibility and the public’s confidence in scientific results is critically important. One has to look no further than the website Retraction Watch to learn of the published papers that are being retracted for a host of reasons. Reasons for retractions can range from plagiarism to scientific misconduct. The cost of these retractions is more than losing a paper, it can lead to financial penalties and even prison. In March, 2019 Duke University settled a lawsuit brought by the US government for over 100 million dollars arising from fraudulent data being used by a researcher in grants and papers from that institution over the years.

These types of stories erode public confidence in scientific data and lead to issues like the discredited link between Autism and vaccination. If you couple this the issues of reproducibility that have been discussed in several articles like this one.

But right now science is facing a reproducibility crisis and the public’s trust in scientific results is faltering. As scientists, it is our responsibility to do everything we can to address these issues.

Recently, a colleague of mine forwarded me a tweet that had been sent by a journal sharing a paper this publication recently published. The tweet implied that this type of research was cutting-edge and so was the journal.

And he asked me, “What do you think of the flow data?” And I was aghast that this paper had been published based upon the quality and the lack of information, and the lack of controls, and the lack of consistency in the data.

It was disappointing to see that a paper like this made it through peer-review. Of course, it doesn’t help when stories like this one highlight another issue in science. It can be easy to set up a website and start a journal in an effort to make easy money. These predatory journals are characterized by charging fees to the scientists seeking to get published. There is a list of these journals here.

When you are reading a paper, look at three areas that should get a critical read in helping to assess the quality of the data and commitment to reproducibility that the authors have.

1. Methods section.

The methods section can reveal a lot about the paper. Here is where the details of how the experiments were performed and the information needed to assess how much confidence one should have in the data. When reading papers, there are a couple of critical areas to check out.

1. Did the authors describe the instrument? Understanding the instrument characteristics is important to help assess the quality of the fluorescent data. Sub-optimal excitation lines or poorly chosen emission filters can call into question conclusions based upon poor sensitivity the instrument in question would have to the target fluorochrome.
2. Did the authors mention antigen and clone name? Mentioning CD3 from Company X in the methods section doesn’t provide enough information to reproduce the experiment. Take the common target anti-Human CD3k there are at least 7 different commercially available antigens. Using two different sources – Benchsci.com and the OMIPs, there is no clear clone that is used more than another so if the paper says CD3, which one are the authors referring to?

Figure 1: The distribution of anti-CD3 clones used in publications and the OMIPs

1. What controls were used? Gating data to identify the populations of interest is a critical step in the data analysis process. Proper gating requires the use of the best controls. These include FMO controls, reference controls, unstimulated controls and more. These should be spelled out and how they were used. In the figure below, shows the comparison of the unstimulated, FMO and isotype control in setting gates on a simulated sample.

Figure 2: Comparison of three different controls to determine the appropriate gate. The data is from Maecker and Trotter (2006). Red and blue lines are added for illustration.

The FMO control is designed to identify the spreading of signal in the channel of interest and is illustrated by the red dashed line above. The isotype control is supposed to address the background staining and is illustrated by the blue dashed line. The isotype control is not recommended.

On the fully stained sample, the FMO control would overestimate the number of positive cells. However, the unstimulated sample (quadrants) addresses both the spectral issues and any NSB issues for the target antigen. This is the type of detail that needs to be in the methods section to improve the reproducibility of the data by other researchers.

Other important details in the methods section include was a viability dye used, what software was used for analysis and more. The methods section should provide sufficient detail that the data can be reproduced.

2. Results section.

Moving to the figures and results section, the information from the methods section should prepare the reader to know what information should be presented in results. One of the most common mistakes made in papers is how the figures are labeled.

Don’t rely on the labels that come with the FCS header. Each machine has its unique labeling system which may not be correct when it comes for publication. Take, for example, an instrument that has a channel labeled PerCp, and the published figure has PerCp on the axis of the bivariate plot – but there is no PerCp mentioned in the methods section. What does this mean? The axes should be labeled with the antigen name and ideally the excitation line and emission filters to make sure the reader knows the correct details.

A second point to look at on the axes of bivariate plots. Looking at the axes can indicate if the data has been transformed in some manner. This is a big issue in papers, that they do not reveal the transformation or indicating that all data had the same transformation applied to the data.

Another area to review in the results section is the gating strategy. The use of pulse geometry is a powerful way to help remove doublets. However, if the ASF is set wrong, cells of interest can be lost. Especially if you are using larger cells, the ASF needs to be adjusted. The consequences of poorly set ASF are discussed in this paper.

Statistical analysis is another area to examine closely. The hypothesis should clearly be stated, along with the threshold and the type of test that was used. The number of replicates should also be listed. This information will help the reader assess the strength of any results.

3. MIFlowCyt standard and the Flow Repository.

As cytometrists, we have a tool that can be used to help improve the communication of experimental information. This is the Minimum Information for a Flow Cytometry Experiment or the MIFlowCyt standard. The development of this standard was under the auspices of the ISAC Data Standards Taskforce and represents a trend in biomedical sciences for these comprehensive checklists to ensure the authors include critical information.

Using the MIFlowCyt standard requires the researcher to include descriptions about the sample, about the instrument, and the experimental overview in the data analysis.

At the present time, the MIFlowCyt is only used by a handful of journals from publishers Wiley-Blackwell and Nature Publishing group. For articles submitted to Cytometry A, if the paper is submitted and is MIFloCyt compliant, it gains a special distinction, and a badge to indicate the paper is compliant.

In the case of publishing in another journal, investigators can encourage other journals to adopt this standard.

Another way for researchers to improve the quality of publications and confidence in data is to share the information in a way that others can access it. The flow repository that was started as a partnership between ISAC with the support of the Wallace H. Coulter Foundation. Investigators can upload their data to this database. The data can remain embargoed until publication, but a special access link can be provided to the reviewers so they can have access to the full data as they make a decision on the paper.

Once a paper is published, the data can be released and anybody can access it. This allows researchers to review the data a paper is based on. Further, it can be used to train new researchers in analytical techniques using published data.

Looking to see if a paper is MIFlowCyt compliant, and the data is archived in the Flow Repository is an excellent way to gain confidence in the conclusions found in the publication, or identify how the analysis was skewed in one way or the other.

When you are reviewing papers, Glen Begley’s rules, which were discussed by Bruce Booth are an excellent source of the way to review a paper. These 6 rules are a powerful tool to aid the researcher and help demonstrate the reproducibility of the data.

In the end, no one wants to be the headline in Retraction Watch. No one wants to see their institution to be fined 100 million dollars because of data fraud. Read every paper with a critical eye, and doubly so for your own manuscripts. Even if not submitting a paper to a journal that doesn’t use the MiFlowCyt standard, it is a good practice to annotate each paper in this manner. In so doing, the authors will go through and make sure none of the critical information is missing. With estimates of the wasted money due to irreproducible results in the 70-90% range, it is essential that researchers do better.

To learn more about The Right Way To Read A Flow Cytometry Scientific Paper, and to get access to all of our advanced materials including 20 training videos, presentations, workbooks, and private group membership, get on the Flow Cytometry Mastery Class wait list.