Methods for Analysing Literature Data to Assess Effects of Production Systems On Food Composition.

In experimental science it is customary to consider a study successful if the primary outcome effect is significant. Unfortunately, the high costs limits the number of studies that are large enough to definitively quantify effects of complex agronomic management factors such as different types of production systems.

Similar issues in medicine led to the development and subsequent proliferation of systematic reviews and meta-analyses, primarily for randomised placebo-controlled intervention trials. Systematic reviews and meta-analyses can however also be used to ‘mine’ data from studies and surveys with different designs. This can be used to identify common patterns and effects, which affect the outcomes of large numbers of studies, despite low or inconsistent significance within individual studies.

The principle of systematic reviews is to include all published datasets that meet certain pre-specified criteria for relevance and reliability. For a meta-analysis, each dataset is transformed into specific metrics, which then can be compared directly between trials. However there are many different ways to do this, and some of the choices have large effect on the outcomes of the analysis, so enthusiastic scientists might be tempted to choose methods that result in a particular preferred type of outcome. To control this risk, a set of quality criteria has been developed, where researchers are expected to define and publish a protocol for how they intend to analyse the literature and the data, justify each method they use and in particular any changes to the protocol, and make their data available for other scientists to check both the origin and the use of the data.

The presentation will include examples of good and not so good practice in this area, focusing on studies investigating potential implications for human health of the differences between organic and conventional production systems, by comparing the composition of foods produced in each system.