Assessing Citation Bias in Scientific Literature

Short Communication

As we have discussed elsewhere [1], there are many types of research fraud, including authorship fraud, plagiarism, data falsification, publication fraud, grant fraud, and data fabrication [2], under a wider rubric of research misconduct or questionable research practices (QRPs), all of which can distort the scientific record. If the scientific record is distorted, then future research and related policy recommendations can be biased and ultimately ineffective. The QRP under investigation here is the tendency of researchers to cite information that supports their own viewpoints, often known as citation bias, a subset of confirmation bias, and to less often cite information that clashes with their own worldviews. Our view would agree with Thomas Jefferson (1820) who said “…. We are not afraid to follow truth wherever it may lead, nor to tolerate any error so long as reason is left free to combat it.” [3].

Duyx and colleagues [4] have noted that, “Citation bias is considered to be a questionable research practice (QRP). QRPs are suboptimal and undesirable behaviors of scientists that lie between responsible conduct of research and research misconduct or fraud (fabrication, falsification, and plagiarism). (p. 92)” and that QRPs “may have a strong negative impact on the development of knowledge (p. 92)” as well as concluding that “The scientific process stands or falls by a balanced representation of the available research. Citation bias distorts this balanced representation and may lead to false beliefs (p. 98).” Where citation bias exists, there is a risk that literature reviews will reflect that bias and become distorted, even misleading. The net result might be that scientific consensus could be incorrect, leading to problems with future research funding, clinical practice, legislation and policy making [5].

The objective of this study was to test for citation bias/ confirmation bias for a particular set of articles by the same author. Ideally, one would want not only the same author from the same institution but the ability to compare articles that were similar in year of publication, sample size, sample characteristics, and research topics. Elsewhere we have looked at two other cases of this problem [1].

Methods

It is not easy to test for confirmation/citation bias. In an ideal world, one might find two articles that were by the same authors, using the same data, published in the same journal, in the same year. Sometimes only less than ideal data are available. If the major difference between citation rates for two such articles is the positivity of the outcome and the positive article has significantly more citations than the article with the less favorable outcome, then we may have statistical evidence for citation or confirmation bias. If citation bias is not in operation, then the citation rates for two such articles should not be significantly different. Elsewhere we have considered two sets of articles that were nearly identical [1]. Here we will consider another set of articles that were not identical but may represent a conservative test because the more popular article has had far less time to have been cited, biasing the relative citations in a direction of greater equality.

The objective outcome measure used for comparing the papers was the Google citation count per article as of 01 June 2020.

Following [1], we used two analytic approaches. First, a one-sample chi-square test was used to test the null hypothesis that papers were cited with similar frequency regardless of the favorability of the findings (50/50 in the case for comparing two articles). Second, setting a binomial probability to 0.500 (comparing two articles), a binomial test was used to test the null hypothesis that the papers with favorable and unfavorable findings would be cited with similar frequencies. In both cases, the alternative hypothesis was that the papers with favorable findings would be cited more than those which were less favorable. Binomial tests were performed from a website calculator (www.socscistatistics. com) while we used SPSS to calculate one-sample chi-square tests.

The first study was based on a 1997 dissertation by Sirota [6] which compared 68 adult daughters of gay fathers with 68 adult daughters of heterosexual parents on numerous characteristics. Of the gay fathers’ group, 63 were gay, three were bisexual, and two were sometimes gay and at other times bisexual; all of their daughters knew of their father’s sexual orientation. She reported that the groups did not differ significantly in terms of age, education, occupation, income, degree of religiosity, attitudes towards homosexuals, attitudes towards their fathers, or current lifestyle/marital status. The two groups did not differ with respect to their fathers’ ages, incomes, or occupations. A significantly higher percentage of the daughters of gay fathers were white (87.7% vs. 69.7%, p < .01) or reported no religious affiliation (55.3% vs. 14.7%, p = .001). Daughters of gay fathers were more likely to have LGB relatives (37/66, 56% vs. 12/68, 18%, p = .001) or LGB friends (63/68, 93% vs. 41/67, 61%, p = .001).

Daughters of heterosexuals did report closer relationships with their mothers (65/68, 96%) than did daughters of gay fathers (49/66, 74%, p = .001). The parents of daughters with gay fathers were more likely to have divorced or been separated (39/68, 57%) than were those of daughters with heterosexual parents (17/68, 25%, p = .001). Roughly equal numbers of parents had died from each group (14/68 vs. 15/68), although none of the mothers of the daughters of heterosexual parents had died compared to five of the mothers of the daughters of gay fathers.

The former were more likely (23/67, 34.3%) than the latter (2/67, 3.0%) to identify as lesbian or bisexual (d = 0.87, p < .001) and, if currently heterosexual, more likely to have questioned their sexual orientation while growing up (30/43, 69.8% vs. 14/60, 23.3%; d = 1.32, p < .001). Sirota also found a significant (d = 0.68, p < .05) difference in having problems with alcohol or drug use (30/68, 44% vs. 10/68, 15%; table 13 on page 77 incorrectly reports 7.3%). The daughters of gay fathers were more likely to report one or both parents as having had problems using drugs or alcohol (40/68, 59%) than were the daughters of heterosexual fathers (21/67, 31%, p = .001). Later, Sirota [7] reported significant differences in insecure attachment (78% vs. 44%, d = 0.72, p < .001) and with respect to feeling uncomfortable with close relationships (44% vs. 12%, d = 0.75, p < .001). The differences could not be explained away by differentials in parental divorce rates [8], although later scholars [9] have implied or asserted, without evidence, that the differentials were divorce-related. The Google citation count for this first article was 28.

The comparison study was also by the same author and also concerned research on homosexuality in which problems with homophobic bias among nurses was investigated [10]. The sample was larger and the article had fewer years during which it could have been cited than the earlier comparison article [7]. The Google citation count for the second article was 52.

Results

Comparing Sirota’s two articles’ citation counts of 28 and 52, by the binomial test, we obtained z = - 2.57 (p < .006). The onesample chi-square test (df = 1) obtained was 7.20 (p = .007). After eliminating citations by known conservative scholars from the two sets of citations, the one-sample chi-square test was 17.75 (p < .001) while z = - 4.09 (p < .001) for the binomial test.

Discussion

In the case of the two articles compared, citation rates were significantly higher for the more positive article than for the more negative article. This type of pattern has the potential to bias literature reviews, lower their quality [11] and create a false scientific consensus [12]. Publication bias may exacerbate the problem, if positive manuscripts are more likely to be published as journal articles than negative manuscripts. It is possible that the dissertation discussed here was largely unpublished for such reasons.

Conclusion

It is our hope that by informing fellow scientists about the risks of citation bias and confirmation bias, those biases may be reduced in the future, even if little can be done about past issues. One advantage of our methodological approach is that the situation can be re- evaluated every few years to check if citation bias patterns are similar, decreasing, or increasing over time. Ultimately, we hope that literature reviews will more accurately reflect the “facts on the ground” and thereby draw more accurate conclusions about scientific and social phenomena, even if that means correcting the scholarly record [13,14] in some cases.

References

• acSchumm WR, Catherine RP, Crawford DW (2020) Forty years of confirmation bias in social science: Two case studies of selective citations. Internal Medicine Review.
• Justin TP (2020) The Stewart retractions: A quantitative and qualitative analysis. Econ Watch Journal 17(1): 152-190.
• Ingrid JH (2020) Ideological asymmetries in social psychological research: rethinking the impact of political context on ideological epistemology. Psychological Inquiry 31(1): 29-34.
• Bram D, Urlings MJE, Swaen GMH, Bouter LM, Zeegers MP (2017) Scientific citations favor positive results: A systematic review and meta-analysis. Journal of Clinical Epidemiology 88: 92-101.
• Daniele Fanelli (2013) Positive results receive more citations, but only in some disciplines. Scientometrics 94: 701-709.
• Sirota T (1997) A comparison of adult style dimensions between women who have gay or bisexual fathers and women who have heterosexual fathers.
• Sirota T (2009) Adult attachment style dimensions in women who have gay or bisexual fathers. Archives of Psychiatric Nursing 23(4): 289-297.
• Schumm W (2008) Re-evaluation of the “no difference” hypothesis concerning gay and lesbian parenting as assessed in eight early (1979-1986) and four later (1997-1998) dissertations. Psychological Reports 103: 275-304.
• Carneiro FA, Fiona Tasker, Francis SQ, Isabel Leal, Costa PA (2017) Are the fathers alright? A systematic and critical review of studies on gay and bisexual fatherhood. Frontiers in Psychology 8(1636): 1-13.
• Sirota T (2013) Attitudes among nurse educators toward homosexuality. Journal of Nursing Education 52(4): 219-227.
• Schumm WR, Crawford DW (2019) Evaluating the quality of literature reviews in the social sciences: developing a measure of quality with an illustration. Psychology Research and Applications 1(2): 47-69.
• Schumm WR, Crawford DW (2019). Scientific consensus on whether LGBTQ parents are more likely (or not) to have LGBTQ children: an analysis of 72 social science reviews of the literature published between 2001 and 2017. Journal of International Women’s Studies 20(7): 1-12.
• Schumm WR (2020) Changes over the decades in selected LGBTQ research findings. JSM Sexual Medicine 4(2): 1029.
• Schumm WR (2020) Avenues for future LGBT theory and research. JSM Sexual Medicine 4(2): 1-7.