• lead
• neurobehavioural testing

Whether or not low to modest levels of exposure to lead have a detrimental effect on cognition is an important issue given the growing attention, for example, in the United States, that has recently been paid towards potentially revising downward the levels of lead exposure allowed in the workplace. Thus, we read with interest the meta-analysis of studies on this topic that appeared in this journal by Goodman and colleagues.¹ Unfortunately, we believe that the authors' conclusions are not valid. Specifically, the authors state that “the data available to date are inconsistent and are unable to provide adequate information on the neurobehavioural effects of exposure to moderate blood concentrations of lead”. We found no direct support for this conclusion in the publication. Moreover, numerous flaws in their method limit any specific inferences that can be made. In general, we found that the meta-analysis combined evidence from studies of widely varying quality and did not account for significant confounding within and between studies. Given these and other flaws, it was predictable that the authors did not find an association between blood lead levels and neurobehavioural test scores.

Specific concerns that we had with the methods include: (1) The authors offer no evaluation of the quality of the evidence from available studies based on study design and analytical method. (2) The authors combine data from poorly done studies with data from well done studies, clouding any effects that are observable from the better conducted studies. (3) Although age and education adjustment within studies is assessed, six studies were included that did not adjust for age, and another three studies did not adjust for education. These are the two most well established predictors of neurobehavioural test scores and the most important potentially confounding variables. (4) Even among the remaining studies that did adjust for age and education, the authors do not address the confounding in the meta-analysis that is caused by variation in age and education across study populations. (5) The authors' main effect measure is an exposed versus control comparison. Among the options that could have been pursued, this is the effect measure with the lowest power. It is unable to assess a dose-effect relation, and it is also the one most prone to selection bias. (6) Relatively few of the 22 studies listed in table 2 contribute to the estimate of the effect size for each neurobehavioural outcome. Moreover, the authors do not state which studies contributed to the effect estimate.

It is important to note that several recent studies, all published before this article was accepted for publication, reported that blood lead was associated with neurobehavioural test scores in multiple cognitive domains. One study of 803 Korean lead workers is the largest study reported to date and observed consistent associations of blood lead with test scores in the domains of executive abilities, manual dexterity, and peripheral motor strength at blood lead levels as low as 18 μg/dl.² In another study of former organolead manufacturing workers, tibia lead was associated with test scores at cross section and with longitudinal declines in test scores.³ These findings suggest that lead may have both short term and progressive influences on neurobehavioural performance.

We elaborate on our main concerns, below.

(1) No evaluation of the quality of the evidence available from studies, and (2) Data from poorly done studies were combined with data from well done studies. It is traditional in meta-analysis to establish a priori criteria for what defines acceptable evidence from studies. The authors only had three inclusion criteria, none of which refer to the quality of the study designs, analytical method, adjustment for confounding, evaluation of bias in selection of exposed and non-exposed subjects, and other such methodological factors. There is apparently no consideration for this arguably single most important step in meta-analysis. The meta-analytical results could simply reflect wide heterogeneity in the quality of the evidence that was combined. This factor alone could account for the overall conclusion of no association.

(3) Inclusion of studies that did not control for age and education. Age and education are the two most important predictors of neurobehavioural test scores in working populations. In the absence of adjustment for these confounders there should be convincing evidence that the two groups being compared were equivalent in age and education. Eight of the included studies did not adjust for age and/or education. The authors offer no explanation for why these studies should be included in the meta-analysis.

(4) No adjustment for age, education, or lead dose differences across studies. By not adjusting for age and education differences across studies, the authors make an implicit assumption that age and education do not modify the relation between blood lead and neurobehavioural test scores. This may or may not be true. In the meta-analysis, the authors also implicitly assume a fixed difference in blood lead levels between exposed and non-exposed groups. Table 1 clearly indicates that this assumption does not hold.

(5) Reliance on exposed versus control comparisons. This is a weak test and a test that is not germane to the conclusions that the authors make. The authors conclude that blood lead levels, that are described as “moderate” in one location in the manuscript and “low” in another, are not associated with neurobehavioural test scores. All studies included exposed workers with a range of blood lead levels, from very low to high. More appropriate approaches could have been considered, for example, only including studies that reported beta coefficients for the blood lead versus test score relation, or adjusting for mean blood lead levels in exposed and non-exposed groups.

(6) Reliance on a small number of unspecified studies for effect estimates. Table 2 of the meta-analysis reports the number of studies that were combined to derive effect estimates, but does not specify which studies were combined. This omission does not allow the reader to determine whether solid evidence was combined with more questionable evidence, or to evaluate whether any of the issues described above were germane to the effect estimates reported.

Two more concerns exist regarding the authors' treatment of the issue of cumulative versus ongoing lead exposure, as well as the identification of the source of funding for this study. In their introduction, the authors quote the review by Balbus-Kornfeld et al, which noted that “the current scientific evidence is flawed because of inadequate estimation of cumulative exposure to or absorption of lead ...” but fail to acknowledge this issue in the interpretation of their own meta-analysis. In fact, as has been widely reported in the literature, methods are now available to non-invasively measure bone lead levels as a reliable and accurate measure of cumulative lead dose. Several studies^3–5 suggest that cumulative lead dose, as measured by tibia lead levels, is a very important biological marker that may be related to cognitive decrements not predicted by blood lead levels. With regard to funding, the authors note that they are mainly from the Exponent Health Group in Alexandria, Virginia, and Menlo Park, California; however, they fail to describe what motivated the study or sources of funding for the study. We believe this information would be of interest to scientists and policy makers engaged in work on this topic.