What Is the Evidence Pyramid? (Part 1)

The Evidence Pyramid includes all types of scientific research, ranked by the strength of the evidence of the results. Scientific research is divided into categories: expert opinions, case reports, in vitro, animal research, observational, experimental, systematic reviews and meta-analyses. These categories reflect the way the evidence came about, and the strength of the evidence determines the place in the pyramid.

This is a follow-up to 'What is nutrition science?'.

When we make statements (claims/assertions) about food and health, evidence has to be provided, just like in a court case. So scientific research is not ‘the truth’ in itself, but a method to provide evidence to support or refute theories about the truth. But not all evidence is equally strong, just like camera footage of a murder is stronger evidence than eyewitnesses. The Evidence Pyramid was developed to visualise the hierarchy of the strength of evidence. In it, the different types of research are ranked in categories to show how strong the evidence is that the results provide. The higher a type of research is in the pyramid, and so in the hierarchy, the stronger the strength of the evidence is. Several factors play a role here, such as the population of the study (animal or human), how much control there is over important factors (observational or experimental) and whether it is primary research (collecting the data yourself) or secondary research (research based on other research). The Evidence Pyramid consists of the following categories: expert opinions, case reports, animal research, in vitro, observational, experimental, systematic reviews and meta-analyses.

What is expert opinion?

Expert opinion speaks for itself, more or less. But who are experts? That is a good question and it will mean something different to everyone. Still, the pyramid mainly refers to people with authority, for example professors, chairs and scientists. You can find the opinion of these experts mainly in books, published in scientific journals or, these days, more and more often on the internet or in podcasts. It is then often about a subject that this expert has gained a lot of experience and knowledge of during his or her career, because he or she has done a lot of scientific research on the subject. The opinion can also come as the position of an organisation or expert panel, both published in a scientific journal. For example a position stand of the International Society of Sports Nutrition on the effect of caffeine on sports, or an expert panel that gives a summary of the current literature on the relationship between blood values and health. Even though these people or organisations are experts, the ‘opinions’ are at the bottom of the pyramid. The hierarchy of the pyramid is based on the method behind the evidence, not on where the evidence comes from. An argument from authority is simply not worth much as scientific evidence. First, because an expert also has to provide evidence for his or her positions, and the position is only as strong as the evidence that is provided. Second, because the pyramid is about the scientific method, which has to make the evidence as objective as possible. An opinion based on experience or literature is perhaps valuable, but very subjective. A narrative review (literature review) is officially not mentioned in this category. This is different in my Evidence Pyramid. As far as I am concerned, a narrative review is nothing more than an expert's opinion published in a scientific journal. Experts use literature to strengthen their opinion on a theory, but there is little transparency about the method behind the review. That is why, as far as I am concerned, it belongs at the bottom of the ladder. It is very sensitive to deception or bias, a good example of this is: Plantaardige of plantkwaadaardige oliën?(deel 1).​

Is the opinion of an expert worth nothing, then? Of course not. It is not strong scientific evidence, but it certainly adds something to the total body of evidence. Experts are interesting to invite to a talk show or debate. A discussion between experts can be very interesting and informative. They can also give direction to future research or help build theories. However, when you judge the opinion of an expert as scientific evidence, it has a low strength of evidence. This is because the scientific method has not been used. So you have no idea what exactly a professor bases the position on and which factors have influenced this opinion. Maybe the expert suffers enormously from Confirmation bias and finds it hard to change their mind in the light of new evidence after years of doing research. Even if they state which literature they use, you will always have to judge the literature itself.

What is a case report?

Case reports are scientific studies in which (usually) one person in a certain context is described in detail. This involves an extensive description of the context (who, what, where and how) and relevant outcomes (health) that are based on one human (so one case). This often happens with people in a unique situation, for example a rare disease.

Case report example

For example, researchers followed a 27-year-old man who fasted for 382 days under the supervision of a medical staff. To safeguard the man's health, blood glucose, electrolytes, magnesium and urine excretion were monitored by a medical staff. In addition, they described the whole process, with a focus on the effects of fasting on this person's health and weight. The 27-year-old man lost 125 kg and the authors conclude that therapeutic fasting is relatively safe, with a significant reduction in weight as the result.​

When several of these ‘cases’ are compared and combined, you also speak of a case series. The results from this type of research can help to formulate a hypothesis that can later be tested in larger groups of people, in order to be able to say something about larger populations.

What is animal research?

In animal research, scientific research is done with animals into the effect of food on health. A common animal that is experimented on is the mouse. Mice have a very fast metabolism and a short life span. As a result, the effect of food on health becomes visible quickly, and a formula can be used to convert it back to years of human life. In addition, mice can easily be genetically manipulated. In this way, certain genes can be switched on or off, which makes the effect of these genes visible.

Animal research example

A good example of this kind of research is the research of Satchidananda Panda, professor at the Regulatory Biology Laboratory at the Salk Institute. Panda has been doing research on our biological clock and fasting for years. Panda has investigated with his research which genes determine our biological clock and how disastrous a disrupted biological clock (through light or food) is for the health of mice. Panda mainly uses mice in his research because mice live in captivity (so the eating pattern can be fully controlled) and they are easy to manipulate genetically. He is now busy testing his hypotheses in humans.

Controlling complete eating patterns and genetic manipulation are things that none of us can simply do with humans. But results from animal research cannot simply be taken over for humans. So animal research mainly gives direction to research and is used to investigate mechanisms and to formulate hypotheses. This hypothesis can then be tested in humans.

What is In Vitro research?

In vitro research is a technique that investigates the effect of food on, for example, cells, by means of test tubes or laboratory glassware, outside the body of animal and human. In this way the cells or blood of people can be used to see what effect certain substances have on our body.

In Vitro research example

Researchers added antioxidants to human blood. They then looked at the effect of the antioxidants on free radicals in the blood. With this research, the researchers want to uncover a mechanism without experimenting on humans themselves. The authors say that observational research had already shown that consuming products high in antioxidants was associated with good health. With their in vitro research, they nicely show what the mechanism behind this could be.

Animal and in vitro research is also often used for research that cannot be done in humans in the first instance, such as on toxic substances. But also, of course, to investigate mechanisms. When in vitro research uncovers a mechanism, it is not yet guaranteed that it works the same way in humans. Our body is a very complex system that you reduce to something simple when you take cells or blood out of the body and study them. The mechanisms must therefore always be investigated in humans as well.

What is observational research?

Observational research consists of ecological, cross-sectional, case-control and cohort research. A characteristic of observational research is that no intervention is carried out on people. Groups of people are observed, measurements are taken and, based on this information, associations between different factors are investigated.

Ecological research

Ecological research is also known as geographical research. It can be used to show patterns of disease and associated factors in a large population (for example at national level).

Ecological research example

A good example of ecological research is a study that looked at the raw materials per country (import and export) of different countries and, on that basis, made an estimate of the omega-3 and omega-6 intake of the inhabitants. It was then examined whether there are associations between omega-3 intake (in relation to omega-6 intake) and cardiovascular and mental diseases, in order to then draw up advice on omega-3 consumption for the inhabitants of the United States. Often there is little change in the eating pattern of the inhabitants of a country over a long time, and the differences between countries are very large. This is an advantage for ecological research, because it means strong correlations can be found. However, many other factors that influence the correlation, such as genetic predisposition, other dietary factors, including the availability of total energy intake, and other environmental or lifestyle factors, cannot be included in the analysis, which makes them very crude correlations. This kind of research does give good information for formulating a hypothesis.

Cross-sectional research

In cross-sectional research, data are collected on a population at one single point in time to investigate the relationship between food and a certain outcome. So this is mainly based on a snapshot and can never demonstrate a causal relationship. Potential confounders can be included in the analysis, though. This way you can look statistically at what the association is between two variables, with the effect of other factors taken into account. However, you never know whether you have included all potential confounders. In addition, you also do not know which way the association goes.

Cross-sectional research example

An example of this is skipping breakfast and overweight. Cross-sectional research shows that when a snapshot is taken, skipping breakfast is certainly associated with being overweight. However, when you look at cohort studies, which follow people over a longer time, this association disappears. This is a good example of how cross-sectional research sometimes shows associations that disappear once stronger evidence such as cohort studies is included. See here the importance of the body of evidence. The chance is namely large that skipping breakfast in itself does not result in overweight, but that people with overweight skip breakfast more often and then compensate with more unhealthy energy-dense food. Cross-sectional research is a way to test a hypothesis in large groups of people. However, the evidence is never strong enough to draw definitive conclusions about a causal relationship between two factors.

Cohort research

Cohort research evaluates a possible link between the exposure (for example to saturated fat) and an outcome (for example cardiovascular disease) by observing a group of people over a certain period (often years). This can be done by looking back in time (retrospective) or by following people forward in time (prospective). This looks at who does and who does not develop the outcome. A cohort is a group of people who share a common characteristic, such as employees of the same factory or a group of older people of the same age and living conditions.

Case-control research

In case-control research, people are placed in groups based on the outcome of the study. The outcome is often a disease, where having or not having this disease determines which group you are in. The groups are then compared with each other. Often all kinds of data are available on these people because, for example, they have been coming to a hospital for their health for some time. The collected data can then be analysed, which makes it possible to look back in time to see whether there are differences between the groups (for example in eating pattern). In other words, a retrospective cohort study. However, a case-control study can also be cross-sectional, because the two groups are compared on the basis of a snapshot, or prospective, by following the people for an x amount of time.

Case-control research example

In a study on the relationship between fruit, vegetables and diabetes, researchers split a group of people into two groups: people without diabetes and with pre-diabetes. They then had the people fill in a Food Frequency Questionnaire (FFQ) in which people stated how much fruit and vegetables they had eaten in the past year. This was to then see whether there was a difference in fruit and vegetable consumption between the two groups. A big disadvantage of this kind of research is the large chance of recall bias. Just think about what you yourself ate in the past year; many people cannot remember it well and start guessing. Or they can remember one category better than other categories. Still, it is often the case that people fairly stably eat too little or enough vegetables and fruit. This means that a questionnaire, at least for large groups of people, gives a reasonable estimate of roughly how many vegetables and fruit they ate.

Prospective cohort research

The results from a prospective cohort study have the strongest strength of evidence of observational research. In this research, people are followed over a longer time (often years) to see what they do in the area of food and health, with all kinds of factors being measured. Then the difference is looked at between people who, after a number of years, did or did not develop certain diseases or other outcomes. Often the same measurements are also taken at several points in time to follow the developments in the exposure (food) and outcomes (health). By excluding people who already have the outcome (e.g. a disease) from the study, you have a better indication of the associations you are investigating, and there is more certainty about the direction and causality of the link.

Prospective cohort research example

For example, researchers looked at the relationship between the consumption of fats and mortality over several decades. More than eighty thousand women and forty thousand men were followed from 1980 to 2012, and they were free of disease at the start. They were then checked every 2 to 4 years on diet, lifestyle, risk factors and diseases. In a study like this you can already speak much more of a causal relationship, because the study follows people over a longer time and the people were free of the outcome at the start of the study. This makes you more certain of the chronological order and you know for sure that the exposure to a certain eating pattern comes before the outcome, which is not possible with cross-sectional research. So prospective cohort research removes many problems of other observational research, but it is not as strong as experimental research.

In PART 2 of ‘What is the Evidence Pyramid?’ I discuss experimental research, reviews and meta-analyses.

What did you think? Let me know in the form of a comment or an email: [email protected]

Have you come across a claim on the internet or social media and are you curious about an assessment of the support behind it? Let me know and I will dive into it!

Do you want to support me and my nutrition science adventure? Share my articles or the podcast on your socials!

Everything you read on this website is my opinion based on knowledge and experience. Chances are that I sometimes overlook something, or that something could be better. I would love to hear it! We do science together.

Terms

Narrative review: A narrative or systematic review is a literature review that summarises primary research. A systematic review distinguishes itself from a narrative review by its systematic approach. In a systematic review it is fully described how the literature was searched for and how the literature was assessed for quality. In a narrative review this does not happen.

Bias: Bias stands for a distortion of the results. In other words: the results are different from what they really are. Distortion is a difficult concept that can occur in all kinds of forms. For example, it can be in the interpretation of the results, because the person interpreting does not look at the results objectively due to prejudices (confirmation bias). But distortion can also occur through the design of a study. For example when the sample is not representative of the population (selection bias) or when the people studied have to provide information based on their memory (recall bias).

Body of Evidence: All results of all (kinds of) studies on a certain subject together. When we want to draw a conclusion about the effect of food on our health, the evidence from all categories in the Evidence Pyramid has to be brought together into one story. In other words, the body of evidence.

Association/correlation: The relationship between two factors.

Causal relationship: When one factor is the direct cause of another factor, we call that a causal relationship.

Recall bias: A form of bias (distortion) in which the results are influenced because the participants of a study have to answer questions based on their memory. Our memory sometimes lets us down, for example when we have to remember what we ate, which means results can be distorted.