Introduction
Some of the most well recognized names in the “Paleosphere” surprisingly maintain few professional, academic, or even experiential credentials which would qualify them as scientists, researchers or even lay experts in the discipline. These self proclaimed, charismatic authorities have influenced and continue to influence hundreds of thousands of people based upon nothing more than their untested subjective opinions and limited understanding of the scientific, peer review literature.
Most have never been trained in the research process, few maintain anything more than a bare bones understanding of the scientific method and don’t have even the slightest inkling of the statistical or research design issues that can make or break the validity and generalizability of any scientific study. Universally, none of these influential Paleo bloggers have an extensive publication record in the scientific peer review literature relating to Paleo diets or anything else.
Accordingly, their blogs have no origins in their own prior refereed scientific writings (because they don’t have any). Unfortunately, these bloggers can utter just about anything they desire about contemporary Paleo diets because virtually no objective system of checks and balances underlie their writings and opinions.
The peer review process
The difference between charismatic bloggers and published research scientists is that the latter must present their ideas, work and experiments before a panel of scientific peers prior to publication. The peer review process certainly is not infallible and clearly does not always insure the accuracy, generalizability or validity of any experiment or idea. Nevertheless, it generally does insure that the paper or concept has been examined by a panel of scientists and experts who usually are trained in the discipline, but who also are typically trained in universal research design and statistical concepts and procedures, without which experiments and data are meaningless and un-interpretable.
Almost universally, charismatic bloggers have little or no understanding of how research design and statistical issues can make or break the interpretation of any experiment or hypothesis, yet as I will show you they proudly offer their opinions regardless.
Research design and statistics
I graduated from the University of Utah in 1981 with a Ph.D. in Health Sciences (emphasis: Exercise Physiology). Besides course work, one of the requirements for the Ph.D. was the successful completion of an experimental project and the subsequent write-up of this research via a Doctoral Dissertation. During my years of coursework, I took almost two quarters worth of specific graduate level classes that focused upon 1) a variety of statistical procedures, 2) research design issues, and 3) computer assisted data compilation and interpretation. Whew! These classes were not fun, and I struggled to get through some of them. But without the knowledge and experience I gained from these classes, I wouldn’t have had a clue about designing, statistically analyzing, and writing up the physiological/respiratory experiment that eventually became my Doctoral Dissertation.
When I finally completed my two year long experiment, wrote the dissertation and finally graduated, I breathed a sigh of relief in the mistaken belief that I would never again have to go through this ordeal. Wrong! At the time, little did I realize the research design, statistical and computer skills I had utilized for my Ph.D. project would never leave me, and that I would have to repeat this process again, again and again on a regular basis for the next 32 years.
It is sometimes said that the best way to better learn about any topic or skill is to have to teach it to others. As a rookie, Assistant Professor at Colorado State University in the fall of 1981, I was immediately assigned to teach a graduate course in Research Design and Statistics to both Master’s and Ph.D. students. As it turned out, I would go on to teach this course for the next 32 years, but more importantly I continually honed my research design and statistical tools not only for my own research, but also as I taught my graduate students to implement their research projects. Increasingly as my career developed I fully appreciated the magnitude of these powerful scientific tools as I served as a reviewer for scientific journal articles and governmentally sponsored grants.
“By using it, you will not lose it,” or so goes the truism. In the case of research design and statistics, almost all charismatic bloggers, never learned these scientific tools in the first place, so their Paleo diet interpretations of the scientific literature and subsequent subjective pronouncements need to be rigorously evaluated if we are to place any credence whatsoever upon their writings.
Below are just a few key questions almost any scientist familiar with research design and statistical procedures would be able to answer. I suspect that none of our charismatic Paleo bloggers whose names you all recognize would be able to answer any of these questions off the top of their heads. Familiarity with these concepts is essential in correctly interpreting and fully understanding the scientific literature.
Questions
- What is statistical power and how does it influence hypothesis testing?
- What is the null hypothesis. Can it be answered in either the affirmative or negative or only singly and why does it matter?
- What is a two-tailed statistical test? How does it affect alpha and subsequently hypothesis testing?
- What is the relationship between alpha (a type 1 statistical error) and beta (a type 2 statistical error) and how does sample size (n) interact with these concepts to affect hypothesis testing?
- Why is sample size crucial when evaluating the internal and external validity of an experiment?
- What are the four levels of data and how does this information influence the type of statistic to be employed in the analysis and why?
- What are the differences among 1) pre-experimental, 2) quasi experimental and 3) true experimental designs. How do these considerations influence internal validity and generalizability of the experiment?
- When is a repeated measures ANOVA used to analyze data and why should multiple t-tests not be used in making repeated comparisons?
- What are the differences between parametric and non-parametric statistical tests and how does the level of data influence their choice?
- Do descriptive statistics show causality? How about inferential statistics? What are common differences between the two?
- Is it possible to generate a standard deviation greater than the mean? How are large standard deviations generally interpreted with small sample sizes? How about with large sample (n) sizes?
- When should the standard error of the mean (SEM) be employed in lieu of the standard deviation?
- With the inclusion of more and more variables into a forward, stepwise multiple regression equation, what is the effect upon “R”; what is the effect upon “p”. Why does this matter?
- How does the use of partial correlation techniques help to unravel relationships among a series of variables?
OK, OK – ENOUGH! You get my point; we could go on endlessly with these obscure statistical and research design concepts. For most of you, not only can you not answer these questions, the answers are irrelevant anyway.
What you want to find out from your charismatic blogger is a simple answer to a simple question – should I drink milk or not? How about kefir? Should I regularly consume legumes and beans? How about sea salt – is it OK? Do contemporary Paleo diets require supplements?
I’ll give you some insight into your charismatic blogger – off the top of their heads, without the input of skilled professionals, they could not answer these research design and statistical questions either – they simply lack the training. Like you, they are barely even familiar with these terms and concepts known to most research scientists.
Without the knowledge or understanding of research design and statistical notions our charismatic, influential Paleo bloggers simply cannot understand the subtleties, limitations and flaws in the scientific papers they may read. Accordingly, their advice and pronouncements about a variety of Paleo Diet issues are at best incomplete, and at worst flat out wrong.
Cause and effect
One of the challenges faced by nutritional scientists when they ultimately make recommendations regarding what we should and should not eat is to establish cause and effect between a dietary element and the subsequent development or prevention of disease. Some foods and some dietary habits promote good health whereas others promote disease. Figure 1 demonstrates the four primary procedures by which causality is established between diet and disease.1
