Six Persistent Misconceptions in Scientific Research – Kenneth J. Rothman

Below is the translation of his article³The original piece is here https://doi.org/10.1007/s11606-013-2755-z.

Summary

Scientific knowledge changes rapidly, but the concepts and methods for conducting research change more slowly. To encourage discussion about outdated ways of thinking about conducting research, I list six misconceptions about research that persist long after their shortcomings have become apparent.

The misconceptions are:

1. There is a hierarchy of research designs; randomized trials offer the greatest validity, followed by cohort studies, while case-control studies are the least reliable.
2. An essential element for a valid generalization is that the subjects constitute a representative sample of a target population.
3. If a term indicating the product of two factors in a regression model is not statistically significant, there is no biological interaction between those factors.
4. When categorizing a continuous variable, a reasonable scheme for choosing categorical cutoffs is to use percentile-defined cutoffs, such as quartiles or quintiles of the distribution.
5. One should always report P values ​​or confidence intervals corrected for multiple comparisons.
6. Significance research is useful and important for data interpretation.

These misconceptions have persisted in magazines, classrooms, and textbooks. They persist because they are intellectual shortcuts that avoid a more thoughtful approach to research problems. I hope that highlighting these misconceptions will spark the necessary discussions to bury these outdated ideas for good.

Kenneth J. Rothman, DrPH
Research Triangle Institute, Research Triangle Park, NC, VS; Boston University School of Public Health, Boston, MA, VS.

KEYWORDS: research design; data interpretation; epidemiological methods; representativeness; evaluation of interaction; multiple comparisons; percentile limits; statistical significance tests.

PMID:24452418 | PMCID:PMC4061362 | DOI:10.1007/s11606-013-2755-z
© The Author(s) 2014. This article is published open access on Springerlink.com

There are still a surprising number of misconceptions about conducting research with human subjects. Some misconceptions persist despite the fact that the opposite is taught, and others because of the fact that the opposite should be taught. To stimulate discussion about these issues, here I list six persistent misconceptions about research and provide a brief summary of the problems each of these misconceptions poses.

Misconception 1. There is a hierarchy of study designs: randomized trials offer the greatest validity, followed by cohort studies, while case-control studies are the least reliable.

Gerandomiseerde studies worden vaak beschouwd als de 'gouden standaard' onder de onderzoekstypes, maar zijn zelfs in theorie niet perfect. Bovendien is de veronderstelling dat de vergelijkende validiteit van onderzoeksresultaten kan worden afgeleid uit het type onderzoek onjuist.

Although some believe that evidence from a randomized trial is as compelling as logical evidence, no empirical study can provide absolute certainty. If randomized trials were perfect, how could they produce divergent results? In fact, they are subject to several errors¹. Uiteraard is er sprake van willekeurige fouten, zoals te verwachten is bij een onderzoek op basis van willekeurige toewijzing. Maar er is ook sprake van systematische fouten, of vertekening. Gerandomiseerde onderzoeken worden bijvoorbeeld meestal geanalyseerd aan de hand van het 'intent to treat'-principe, waarbij de groepen die aanvankelijk willekeurig zijn toegewezen, worden vergeleken, ongeacht eventuele latere niet-naleving. Niet-naleving leidt tot een onderschatting van het effect van de behandeling. Deze vertekening wordt meestal als aanvaardbaar beschouwd, omdat deze wordt gecompenseerd door de voordelen van willekeurige toewijzing. Onderschatting van effecten is echter niet aanvaardbaar in een veiligheidsonderzoek dat gericht is op het blootleggen van bijwerkingen van de behandeling. Een andere belangrijke bron van vertekening in een gerandomiseerde studie zijn fouten bij de beoordeling van de uitkomst, zoals het te laag tellen van uitkomstgebeurtenissen. Ook als randomisatie aan het begin van de studie zorgt voor een evenwicht tussen de risicofactoren van de groepen, kunnen de studiegroepen door langdurige follow-up steeds meer uit balans raken door differentiële uitval of veranderingen in de verdeling van risicofactoren. Bij langdurige onderzoeken kunnen de voordelen van willekeurige toewijzing daarom na verloop van tijd afnemen.

In short, studies are far from perfect. Furthermore, both cohort and case-control studies yield valid results when properly designed and conducted. It is therefore incorrect to without thinking give more validity to a study based on a hierarchy of research designs^2,3. For example, the association between cigarette smoking and lung cancer is well established based on findings from cohort and case-control studies. This connection has never been clearly demonstrated in a randomized study. It is not easy to randomly assign people to a smoking or non-smoking group, but when smoking cessation was examined as part of a multifaceted intervention in the Randomized Multiple Risk Factor Intervention Trial⁴, those who were encouraged to quit smoking actually developed more lung cancer than those who were not encouraged to quit. The results of the study did not reverse the findings of the many cohort and case-control studies conducted without randomization. Rather, the discrepancy was attributed to problems with the study.

In another striking example, the results of large cohort studies pointed out^5,6 indicated that the risk of coronary heart disease was reduced in postmenopausal hormone users, but later results from two randomized trials indicated no association or an increased risk.^7,8The reaction in the scientific community and the popular press⁹was to discredit the results of the cohort studies, on the assumption that they had been refuted by the randomized studies. Many continue to adhere to that interpretation, but in an elegant reanalysis, Hernan et al.¹⁰indicated that the study groups in the cohort studies and the randomized studies were different and that the effects of hormone use after menopause varied greatly depending on age and time since menopause. When the studies were limited to new hormone users, Hernan et al. showed that differences in the distribution of age and time since menopause could explain all apparent discrepancies. Although it is common to attribute such discrepancies to inherent weaknesses of the non-experimental studies, it is simplistic to assign validity based on an assumed hierarchy of study types.¹¹

Likewise, discrepancies between cohort studies and case-control studies should not be superficially explained away by an assumed validity advantage of cohort studies over case-control studies. Well-designed case-control studies will yield the same results as well-designed cohort studies. When conflicts arise, they may arise from problems in either or both types of studies. Although case-control studies have long been dismissed as outdated versions of cohort studies, which start from the disease and look back at possible causes, epidemiologists now view case-control studies as conceptually identical to cohort studies, except for an efficiency gain that comes from sampling the denominators rather than conducting a full count. Indeed, these efficiency gains allow more resources to be devoted to exposure assessment or case validation in case-control studies, resulting in less bias than in corresponding cohort studies of the same relationship.

Those who view case-control studies as outdated versions of cohort studies sometimes make the false analogy that the controls must be very similar to the cases, except that they do not have the disease that defines the case. In fact, the control group in a case-control study is intended to be a sample of the population denominator that gives rise to the cases, a substitute for the full denominators obtained in a cohort study. The control group must therefore resemble the entire research group, and not the cases.^12,13 When well designed, case-control studies can achieve the same excellent validity as well-designed cohort studies, while a poorly designed study can be unreliable. The type of study should not be used as a measure of the validity of a study.

Misconception 2. An essential element for making valid generalizations from a study is that the subjects are a representative sample of a target population.

This misconception is related to the view that scientific generalization involves a mechanical extrapolation of results from a sample to the source population. But that describes statistical generalization; scientific generalization is slightly different: it is the process of establishing a correct statement about the way nature works.

Scientific generalization is the ultimate goal of scientific research, but a prerequisite for this is designing a study with internal validity, which is enhanced by holding all confounding variables constant. When have we ever heard of animal researchers looking for a statistically representative sample of animals? Instead, their approach is almost the opposite of the pursuit of representativeness. For example, biologists studying mice prefer mice that are homogeneous in terms of genes and environment, and differ only in terms of the experimentally manipulated variable. Unlike the statistical generalization of polls or sampling, which requires only an extrapolation from the sample to the source population, scientific generalization is through educated guesses, but only from the safe platform of a valid survey. Consequently, studies are stronger if they limit the variability of confounding factors, rather than aiming for representativeness. Doll and Hill¹⁴studied mortality among male British doctors in relation to their smoking habits. Their findings were considered broadly generalizable, despite the fact that their study group was not representative of the general population of tobacco users in terms of gender, race, ethnicity, social class, nationality, and many other variables.

When there is a legitimate question about whether an overall association varies by subgroup of a third variable, such as age or ethnic group, it may be necessary to include people from a wide range of values ​​of that third variable, but even then it is counterproductive if the study group is representative of the source population for that variable. The goal in that case would be to include subjects evenly distributed across the range, or in a distribution that increases the overall efficiency of the study. A sample that is representative of the source population will be suboptimal.^15,16

Misconception 3. If a term indicating the product of two factors in a regression model is not statistically significant, there is no biological interaction between those factors.

De term 'biologisch' moet hier in brede zin worden opgevat, zodat het biochemische, psychologische, gedragsmatige en fysieke interacties omvat. Het probleem is dat interactie meestal wordt geëvalueerd aan de hand van regressiemodellen, waarin de productterm betrekking heeft op statistische interactie in plaats van biologische interactie.

Biological interaction refers to two or more causes acting in the same mechanism, with effects that are interdependent. It describes a natural state. When base effects are measured as changes in disease risk, a synergistic (i.e., positive) biological interaction occurs when the joint effect of two causal factors is greater than the sum of their separate effects.¹⁷Statistical interaction, on the other hand, does not describe nature, but a mathematical model. This is usually assessed with a product term for two variables in a regression model. Its magnitude depends on the choice of measures and the measurement scale. Statistical interaction only implies that the basic functional form of a specific mathematical model is not an appropriate description of the relationship between variables. Two factors that interact biologically may or may not interact statistically depending on the model used.

Product terms in regression models have units that are difficult to interpret. If one variable is fat consumption, measured in grams per day, and another is the number of pack-years of cigarettes smoked, how to interpret a variable that has the unit gram/day multiplied by pack-years? The challenge of interpreting such product term coefficients has led to a focus on the p-value associated with the coefficient, rather than the size of the coefficient itself. Focusing on the p-value, or on whether the coefficient of a product term is statistically significant, only worsens the problem of confusing statistical interaction with biological interaction (see misconception 6). A more meaningful assessment of interaction would be to focus on the percentage of cases of a disease that can be attributed to biological interaction.^17,18

Take a simple example from the TREAT (Trial to Reduce Cardiovascular Events with Aranesp Therapy) study¹⁹), waarin het risico op een beroerte werd geëvalueerd bij 4.038 patiënten met diabetes mellitus, chronische nierziekte en bloedarmoede die willekeurig werden ingedeeld in een groep die darbepoëtine alfa kreeg of een groep die een placebo kreeg. Bij patiënten zonder voorgeschiedenis van een beroerte was het risico op een beroerte tijdens de onderzoeksperiode 2% bij patiënten die placebo kregen en 4 % bij patiënten die darbepoëtine alfa kregen. Bij patiënten met een voorgeschiedenis van een beroerte waren de overeenkomstige risico's 4 % en 12 %. De auteurs merkten op dat de risicotoename groter was voor darbepoëtine alfa bij patiënten met een voorgeschiedenis van beroerte, maar zij verwierpen deze interactie omdat de productterm in een logistisch regressiemodel niet statistisch significant was. Het verhoogde risico dat aan darbepoeitin alfa kon worden toegeschreven, was 2 % bij patiënten zonder voorgeschiedenis van een beroerte en 8 % bij patiënten met een voorgeschiedenis van een beroerte, wat wijst op een sterke biologische interactie tussen darbepoeitin alfa en een voorgeschiedenis van een beroerte. Als de risico's louter additief waren, zou het risico bij patiënten met beide risicofactoren 6 % bedragen, in plaats van de werkelijke 12 %. De helft van het risico bij patiënten met beide risicofactoren lijkt toe te schrijven te zijn aan biologische interactie, ondanks de bewering van de auteurs dat er geen interactie was.

Misconception 4. When categorizing a continuous variable, a reasonable scheme for choosing category boundaries is to use percentile-defined boundaries, such as quartiles or quintiles of the distribution.

There are two reasons why using percentiles is a poor method for choosing category boundaries. First, these boundaries may not correspond to the parts of the distribution where biologically significant changes occur. Suppose you are conducting a study on vitamin C intake and the risk of scurvy in the United States. If you were to divide vitamin C intake into five groups, you would see that the entire association between vitamin C consumption and scurvy was limited to the lowest group, and within that group to only a small proportion of people who had exceptionally low vitamin C intake. 10 mg of vitamin C per day can prevent scurvy, but those who take less than that represent only a fraction of 1% of the population in the United States.²⁰Using percentile-based categories would make it impossible to determine the effect of inadequate vitamin C intake on scurvy risk, because all intake above 10 mg/day is essentially equivalent. If we routinely use percentile cut points, we may not know if we are facing the same problem as when researching vitamin C and scurvy. A more effective alternative would be to start with many narrow categories and merge adjacent categories until significant differences in risk become apparent.

The second problem with percentile-based categories is that it is difficult to compare results between studies because categories between studies that use percentile category boundaries are unlikely to match. This problem can be avoided by expressing cut-off points in the natural units of the variable (such as mg/d for vitamin C intake). It is also useful to report means or medians within categories.

Misconception 5. One should always report P values ​​or confidence intervals corrected for multiple comparisons.

Traditional adjustments for multiple comparisons involve inflating the P-value, or the width of a confidence interval, based on the number of comparisons performed. When analyzing biological data that is full of real relationships, the premise for traditional adaptations is shaky and the adaptations are difficult to defend. The concern about multiple comparisons stems from the fear of finding falsely significant findings (type I errors in the jargon of statistics). In Misconception 6, we discuss the problems associated with using statistical significance tests for data analysis. But before we look at those issues, let's consider the reasons for adjusting reported results for multiple comparisons.

Despite the fact that a single test of significance is intended to have a 5% chance (at the conventionally used level) of being significant when the null hypothesis is true, and therefore multiple tests, if performed correctly, should each have this property, there is concern that performing multiple tests increases the chance of a false positive result. Of course, as the number of tests increases, the chance that one or more tests will be false positive also increases, but that is only because there are many tests being performed. Adjustments for multiple comparisons reduce these types of Type I errors, but at the cost of increasing Type II errors, which are nonsignificant test results in the presence of a real relationship. When observed relationships are all the result of chance, type I errors can occur, but type II errors cannot occur. Conversely, when the observed relationships all reflect actual relationships, Type II errors can occur, but Type I errors do not. The context of any analysis thus has fundamental implications for the interpretation of the data. In particular, it is absurd to make adjustments that reduce Type I errors at the expense of increasing Type II errors, without an evaluation of the estimated relative costs and frequency of each type of error.

If scientists studied random numbers instead of biological data, any significant results they report would be type I errors, and adjustments for multiple comparisons would be meaningful; some skeptics believe that studies of genome-wide association scans approach this situation.²¹ But when scientists study biological relationships rather than random numbers, the assumption that Type I errors are the biggest problem may be incorrect.²²A more rigorous evaluation of the need for multiplicity adjustments would begin with an assessment of the tenability of the claim that the data are essentially random numbers. When studying experiments on paranormal phenomena, skepticism about the results could be an argument for multiplicity adjustments. When studying the physiological effects of pharmaceutical agents, real associations are to be expected and the adaptations are more difficult to defend. Studying single nucleotide polymorphisms in relation to a particular disease could be a middle path. An approach to this issue that is more theoretically defensible is a Bayesian approach, in which ex ante credibility is assigned to different levels of relationships and adjustments are made using Bayes' theorem to calculate posterior credibility.^23,24

Misconception 6. Significance research is useful and important for data interpretation.

Significance research has led to far more misunderstandings and misinterpretations than clarity in interpreting research results.^25–28A significance test is a deteriorated version of the P-value, a statistical quantity that combines precision and effect size, confusing two essential aspects of data interpretation. Measuring effect size and precision as separate tasks is a more direct and clear approach to data interpretation.

For studies that aim to measure relationships and conclude whether these reflect causal relationships, the focus on the magnitude of these relationships should be paramount: estimating effects is clearly preferable to statistical testing. Ideally, a study estimates the size of the effect and analyzes the possible errors that could have distorted the effect. Systematic errors, such as distortion by measured factors, can be addressed with analytical methods; other systematic errors, such as the effects of measurement error or selection bias, can be addressed with sensitivity analyzes (also called bias analysis). Random errors are typically expressed in confidence intervals, which provide a range of parameter values ​​that are consistent with the data up to a certain level.

It is unfortunate that a confidence interval, from which both an estimate of effect size and measurement accuracy can be derived, is typically only used to assess whether it contains the null value or not, thus turning it into a significance test. Significance testing is a poor classification system for research results; strong effects may be incorrectly interpreted as null findings because authors incorrectly interpret the lack of statistical significance as a lack of effect, or weak effects may be incorrectly interpreted as important because they are statistically significant. Rather than being used as surrogate tests of significance, confidence intervals should be interpreted as quantitative measures that indicate the magnitude of the effect size and the degree of precision, with little attention paid to the precise location of the boundaries of the confidence interval. This advice is supported by the Uniform Requirements for Manuscripts Submitted to Biomedical Journals, but is nevertheless often overlooked, even by reviewers and editors of journals that support these requirements.²⁹

Veel misvattingen komen voort uit het vertrouwen op statistische significantietests. De focus op de statistische significantie van interactietermen in plaats van het meten van interactie, zoals hierboven besproken, is daar een voorbeeld van. De evaluatie van dosis-respons trends door simpelweg te verklaren dat er al dan niet een significante trend is, in plaats van de omvang en idealiter de vorm van die trend weer te geven, is een ander voorbeeld. Nog een ander voorbeeld is het advies dat soms wordt gegeven om de power van een studie te berekenen bij het rapporteren van resultaten, vooral als die resultaten niet statistisch significant zijn. Het rapporteren van de power van een onderzoek als onderdeel van de resultaten wordt 'post-hoc'-powerberekening genoemd.³⁰Power calculations are based on a hypothesis about the level of association to be distinguished from a null association, but when the research results are in hand, it is no longer necessary to formulate a hypothesis about the magnitude of the association because you now have an estimate of it. A confidence interval for the estimated association reflects all relevant information; a capital calculation no longer yields anything.

The unfortunate consequence of the focus on statistical significance testing is that it has created a dichotomous view of relationships that are better assessed in quantitative terms. This distinction is more than a subtlety. Every day there are significant, regrettable, and avoidable misinterpretations of data that result from the confusing fog of statistical significance testing. Most of these errors could be avoided if the focus were shifted from statistical testing to estimation.

Conclusion

Why do such important misconceptions about research persist? These misconceptions are largely a substitute for more thoughtful and difficult tasks. It is easier to resolve a discrepancy between a trial and a non-experimental study in favor of the trial, without performing the laborious analysis that Hernan et al.¹⁰Het is gemakkelijk om te verklaren dat een resultaat statistisch niet significant is, waardoor ten onrechte wordt gesuggereerd dat er geen aanwijzingen zijn voor een verband, in plaats van kwantitatief te kijken naar het bereik van verbanden dat de gegevens daadwerkelijk ondersteunen. Deze misvattingen zijn een makkelijke weg, maar wanneer die weg vol is met anderen die dezelfde weg bewandelen, is er misschien weinig reden om de route in twijfel te trekken. Deze misvattingen worden inderdaad vaak in tijdschriften, klaslokalen en studieboeken in stand gehouden. Ik ben van mening dat de beste kans op verbetering ligt in het vergroten van het bewustzijn over deze kwesties, met een redelijk debat. Max Planck zei ooit: "Een nieuwe wetenschappelijke waarheid zegeviert niet door haar tegenstanders te overtuigen en hen het licht te laten zien, maar omdat haar tegenstanders uiteindelijk sterven en er een nieuwe generatie opgroeit die ermee vertrouwd is."³¹To the extent that this cynical view is correct, we can expect outdated concepts to disappear slowly at best. I hope that highlighting these misconceptions will spark needed discussions and be a catalyst for change.

Acknowledgments: I have received useful criticism from Susana Perez, Andrea Margulis, Manel Pladevall in Jordi Castellsague.

Conflict of interest: The author declares that he has no conflict of interest.

Corresponding author: Kenneth J. Rothman, DrPH; Research Triangle Institute, Research Triangle Park, NC, VS (e-mail: KRothman@rti.org).

Open Access This article is distributed under the terms of the Creative Commons Attribution License, which authorizes any use, any dissemination and any reproduction in any medium, provided the original author(s) and source are credited.

REFERENCES

1. Hernán MA, Hernández-Díaz S, Robins JM. Accorded studies analyzed as observational studies. Ann Internal Med. 2013:59:560–2. doi:10.7326/0003-4819-159-8-201310150-00709
2. Ioannidis JPA. Why most published research results are incorrect. PLoS With. 2005;2(8):e124.
3. Hiatt WR. Observational studies on drug safety – aprotinin and it lack of transparency. N Engl J Med. 2006;355:2171–3.
4. Shaten BJ, Kuller LH, Kjelsberg MO, Stamler J, Ockene JK, Cutler JA, Cohen JD. Lung cancer mortality at 16 years among MRFIT participants in intervention and usual care groups. Multiple Risk Factor Intervention Trial. Ann Epidemiol 1997;7:125–36.
5. Grodstein F, Manson JE, Colditz GA, et al. A prospective, observational study of postmenopausal hormone therapy and primary prevention of cardiovascular disease. Ann Intern Med. 2000;133:933–41.
6. Varas-Lorenzo C, García-Redríguez LA, Pérez Gutthhann S, et al. Hormon-reflying therapy and incidence of acute myocard infarct. Circulation. 2000;101:2572–8.
7. Hulley S, Grady D, Bush T, Furberg C, Herrington D, Riggs B, Vittinghoff E. Penadized study to estrogen plus progestage of coronary prevention in postminopausal women. Heart and Estrogen/progestin Restruction Study (HERS) Research Group. JAMA. 1998;280:605–13. doi:10.1001/jama.280.7.605.
8. Manson JE, Hsia J, Johnson KC, et al. Estrogen plus progestin and the risk of coronary heart disease. N Engl J Med. 2003;349:523–34.
9. Taubes G. Do we really know what makes us healthy? New York Times, 16 september 2007.
10. Hernán MA, Alonso A, Logan R, Grodstein F, Michels K, Willett WC, Manson JE, Robins JM. Observational studies analyzed as randomized experiments: an application to postmenopausal hormone therapy and coronary heart disease. Epidemiology. 2008;19:766–79. doi:10.1097/EDE.0b013e3181875e61
11. Concato J. Observational versus experimental studies: what is the epidemiological evidence for a hierarchy? NeuroRx. 2004;1:341–7.
12. Vandenbroucke JP, Pearce N. Case–control studies: basic concepts. Int J Epidemiol. 2012;41:1480–9. doi:10.1093/race/dys147.
13. Rothman KJ. Chapter 5, Types of epidemiological studies, in Epidemiology, An Introduction, 2e editie. Oxford University Press, New York, 2012.
14. Doll R, Hill AB. Mortality among physicians in relation to their smoking habits: a preliminary report. Br Med J 1954;ii:1451–5.
15. Rothman KJ, Gallacher J, Hatch EE. Why representativeness should be avoided. Int J Epidemiol. 2013;42:1012–4. doi:10.1093/walk/dys223
16. Rothman KJ, Gallacher J, Hatch EE. When it comes to scientific deductions, sometimes a cigar is just a cigar. Int J Epidemiol. 2013;42:1026–8. doi:10.1093/walk/dyt124
17. Rothman KJ. Chapter 11, Measuring Interaction, in Epidemiology, An Introduction, 2e editie. Oxford University Press, New York, 2012.
18. Knol MJ, van der Tweel I, Grobbee DE, Numans ME, Geerlings MI. Estimation of interaction on an additive scale between continuous determinants in a logistic regression model. Int J Epidemiol. 2007;36:1111–8.
19. Skali H, Parving HH, Parfrey PS, Burdmann EA, Lewis EF, Ivanovich P, Keithi-Reddy SR, McGill JB, McMurray JJ, Singh AK, Solomon SD, Uno H, Pfeffer MA. TREAT Investigators: Stroke in patients with type 2 diabetes mellitus, chronic kidney disease, and anemia treated with darbepoetin alfa: the Reducing Cardiovascular Events with Aranesp study -therapie (TREAT). Circulation. 2011;124:2903–8.
20. Recommended daily allowances for vitamin C, vitamin E, selenium and carotenoids. Institute of Medicine, The National Academies Press, Washington, D. C., 2000.
21. Dudbridge F, Gusnanto A. Estimation of significance thresholds for genomic association scans. Genet Epidemiol. 2008;32:227–34.
22. Rothman KJ. No adjustments are necessary for multiple comparisons. Epidemiology. 1990;1:43–6.
23. Greenland S, Robins J. Empirical-Bayesian adjustments for multiple statistical comparisons are sometimes useful. Epidemiology. 1991;2:244–51.
24. Greenland S, Poole C. Empirisch-Bayesiaanse en semi-Bayesiaanse benaderingen van toezicht op beroeps- en milieurisico's. Arch Environ Health. 1994;48:9–16.
25. Rothman KJ. A vote of confidence (editorial). N Engl J Med. 1978;299:1362–3.
26. Poole C. Beyond the Confidence Interval. Am J Public Hlth. 1987;77:195–9.
27. Rothman KJ. In search of significance (editorial). Ann Int Med. 1986;105:445–7.
28. Gelman A, Stern H. Het verschil tussen 'significant' en 'niet significant' is op zichzelf niet statistisch significant. Amer Statistician. 2006;60:328–31.
29. Uniform Requirements for Manuscripts Submitted to Biomedical Journals, http://www.icmje.org/manuscript_1prepare.html (consulted on 2 mei 2013)
30. Smith AH, Bates MN. Confidence limit analyses should replace power calculations in the interpretation of epidemiologic studies. Epidemiology. 1992;3:449–52.
31. Planck M. Scientific autobiography and other articles, Philosophical Library, New York, 1968, vert. F. Gaynor (New York, 1949), pp. 33–34

Footnotes

• 1
  It's all over the place sometimes, see Het-academic bankruptcy in four companies
• 2
  See also the article from June 2021 about scientific integrity
• 3
  The original piece is here https://doi.org/10.1007/s11606-013-2755-z