METHODS

Literature review and study selection

Study selection involved Medline searches using the keywords: “lead AND neurotoxic AND occupational exposure”; “lead AND neurotoxic AND occupation”; “blood lead AND neurobehavioural effects”; “lead AND health effects AND occupation”. A research librarian conducted additional searches of other relevant databases, including those referencing ongoing research. These searches were supplemented by a review and retrieval of references from the Toxicological profile for lead published by the Agency for Toxic Substances and Disease Registry (ATSDR).¹

As well as studies published in English, potentially relevant reports were translated from Hebrew, Chinese, Japanese, German, and Danish. In situations where information was missing, attempts were made to contact the author(s) to obtain the missing data. Descriptive information on each neurobehavioural test was reviewed from standard textbooks.^8–10

The following inclusion criteria were used for the meta-analysis:

• Central tendency for lead exposure was less than 70 μg/dl

• Numbers of exposed and unexposed were reported

• Test score arithmetic means and measures of dispersion were reported for both the exposed and the unexposed workers.

Data extraction

Pertinent data from the studies were entered into an electronic database including central tendency measures for blood lead concentrations among the occupationally exposed and unexposed groups. The outcome information included mean test scores and their corresponding units and measures of dispersion for the exposed and unexposed (control) groups. Before data entry, blood lead concentrations given in μmol/l were converted to μg/100 ml and all measures of dispersion were converted to SDs. Some studies included test score information for subgroups of exposed workers with a mean blood lead concentration >70 μg/dl. For these studies, only information on those subgroups of exposed workers with an average blood lead concentration <70 μg/dl were extracted. Several studies provided test scores for certain strata within the exposed and unexposed groups. In these instances, information on each stratum was entered separately. Thus, some articles provided more than one study group.

Data were extracted only for neurobehavioural tests included in three or more studies. We omitted tests involving self rating of affect such as profile of mood states and multiple adjective affect checklist. Only studies reporting non-overlapping data were included.

Data analysis

The analysis of the data included two steps. The first step involved a review of the studies to assess their quality using the following criteria:

• Evaluation of the pre-exposure status

• Adjustment for age

• Adjustment for other occupational exposures

• Adjustment for alcohol use

• Adjustment for socioeconomic confounding factors such as income level, education, etc

• Use of blinding procedures.

The second step included a quantitative meta-analysis of the pooled data. The statistical analysis protocol is illustrated in figure 1. There are two approaches for combining the data: a fixed effects model and a random effects model. The fixed effects method assumes no heterogeneity between studies and attributes all observed variation between results to sampling error alone.¹¹ The random effects model assumes that the study specific effect sizes come from a random distribution of effect sizes with a certain mean and variance.

Although several tests of heterogeneity have been proposed, their interpretation is problematic because: (a) the power of the test may be insufficient to detect significant heterogeneity¹²; (b) the studies may be subject to similar design flaws leading to consistent bias and providing a false impression of homogeneity; and (c) the effect sizes may seem to be more consistent than they really are if studies with zero or negative effects are less likely to be published.¹³

Because of the wide variation in testing procedures and scoring practices among the studies, as well as variations in the selected study populations, assumptions of homogeneity, even in the presence of a non-significant test, were generally inappropriate. Therefore, we used both the fixed and the random effects models for each of our analyses. Where the variance between studies was negligible (high level of homogeneity), the random effects models reduced to fixed effects models. To correct for measurement error, whenever possible, all results were adjusted for test-retest reliability.^8,14–17

Using the fixed effects assumption, the general formula for the weighted average effect size of k studies iswhere T_i is the effect size estimate of the ith study and w_i is the weight associated with it. The weights that minimise the variance of T. are given bywhere v_i is the conditional variance in each study. The average effect size T. has a conditional variance v. given by

The effect size of each study (T_i ) was estimated from the standardised mean difference statistic with the small sample bias correction applied as proposed in Hedges and Olkin.¹⁴ The statistic d_i is calculated as follows:where X^exp_i is the mean of the exposed group, X^c_i is the mean of the control group, s_i is the pooled SD of the two groups, N_i is the total sample size of the two groups, and r_i is the reliability of the psychological test used in the study. The conditional variance of d_i is calculated as follows:where n^exp_i and n^c_i are the sample sizes of the exposed and control groups respectively.

A test of whether the fixed effects assumption holds uses the following statistic:

When the assumption holds, Q has a χ² distribution with k-1 degrees of freedom.

Equations for variance change for the random effects assumption. The total variance of an effect size estimate is given by:where σ² is the random effects variance and v_i is the conditional variance already given. There are two methods for estimating σ². The first uses the ordinary unweighted sample estimate for the variance of the effect sizes computed as:

The random variance is then estimated by:

If this estimate is negative, it is assumed to be zero and the fixed effects model applies.

The second method for estimating the random variance uses Q, which is taken as an estimate of the weighted sample estimate of the unconditional variance of T_i. In this method, the random variance is estimated by:

This second method will give a non-zero estimate only when Q is greater than its expected value. Otherwise, it is assumed to be zero and the fixed effects model applies.

When the random effects model applies, the average effect size T. and its variance v. are calculated from equations 1–3; however, v*_i is substituted for v_i.

When a study provided data for several subgroups, to prevent its overweighting in the meta-analysis, we combined subgroups to calculate a single effect size and then included the result in the final meta-analysis. If the strata represented different control and exposure groups, the combined effect size was calculated from the fixed effects model where k=number of strata. If the strata represented the same exposure and control groups that were tested two or more times with different test versions—such as simple reaction time with different types of stimuli—the effect size was calculated with the fixed effects model where k=number of tests. If a study consisted of one control group and two or more exposure groups, then the exposure groups were combined to calculate a common mean (SD), provided the exposure level fit the criteria.

Publication bias was examined by calculating the fail safe N, defined as the number of studies with a non-significant result (p>0.05) that would bring a significant pooled analysis to non-significant levels. The calculations were based on Rosenthal's application of the Stouffer-Liptar inverse normal method of combining p values.¹⁸ To evaluate observer bias, studies that implemented blinding procedures were included in a separate analysis.

RESULTS

Review of the literature

About 140 papers, reports, and books were reviewed. Twenty two studies met the inclusion criteria (table 1). Lead concentrations among study subjects ranged from 24 to 63 μg/dl for exposed and from 0 to 28 μg/dl for unexposed workers. However, none of the studies had an overlap in terms of mean blood lead concentration.

There was considerable variation in quality among the 22 studies that met the inclusion criteria. None of the studies directly compared the test scores in their study populations before and after exposure. However, Bolla et al,¹⁹ recognising the importance of controlling for premorbid verbal intelligence, used the vocabulary subtest score of the WAIS-R in her regression model.

Five studies (Baker et al,³ Campara et al,²⁰ Chia et al,²¹ Maizlish et al,²² and Williamson and Teo²³) adjusted results for the confounders age, education, and alcohol use. One study did not control for any of these variables.²⁴ The rest of the studies showed varying and inconsistent attention to controls for these potential confounders (table 1).

There seemed to be no consistent pattern for adjustment for other potential confounders. For example, Chia et al²¹ adjusted for smoking history and ethnicity, Campara et al²⁰ matched the study group and referents on “domestic status” (marital status), distance from work, and number of chronic illnesses, and Pasternak et al³² eliminated subjects with evidence of current illness or injury.

Six of the studies (Bolla et al,¹⁹ Campara et al,²⁰ Chia et al,²¹ Johnson et al,³⁰ Pasternak et al,³² and Valciukas et al⁴) indicated that they had used observer blinding procedures.

Meta-analysis

Twenty two neurobehavioural tests met the inclusion criteria. The results of the meta-analysis for each test are presented in table 2. Two tests (digit symbol and D-2 errors) showed a significant effect for all three models used. The result for one test (D-2 speed) was significant when using the fixed effects model and weighted random effects model, but was not significant when using the unweighted random effects model. Also, several tests (simple reaction time, grooved pegboard, trail making A and B, picture completion, visual reproduction, eye-hand coordination, and vocabulary) showed a significant effect, but for the fixed effects model only.

Correction for reliability coefficients did not seem to affect the results substantially. When the data from different subgroups in a single study were combined as opposed to included separately, the results of the meta-analysis changed in some instances, but only for the fixed effects model. The visual reproduction test result changed from statistically non-significant to statistically significant, whereas the result for paired associates remained the same.

Separate inclusion of strata led to overestimation of homogeneity. For example, the random effects model for the visual reproduction test with Ryan et al³⁶ included as four separate strata was reduced to a fixed effects model because the variance between studies was estimated to be zero. By contrast, inclusion of Ryan et al³⁶ as a single estimate (after combining the results of the four strata) produced a meaningful random effects model result.

DISCUSSION

Our evaluation of the literature indicates that none of the studies reviewed allow definitive conclusions about the presence or absence of adverse neurobehavioural effects at low (<70 μg/dl) current blood lead concentrations. Perhaps the most difficult aspect of observational cross sectional studies involving neurobehavioural outcomes is their limited ability to control for wide interpersonal variability of test results.

Only one study (Mantere et al³⁷) attempted to use testing results before employment as a background measure of neurobehavioural performance. The authors then re-evaluated the lead workers and controls for the number of people whose testing improved, declined, or stayed the same. They also performed a linear regression analysis of the relation between blood lead time weighted average and performance score. However, these results were reported selectively for two tests only: Santa Ana coordination and block design. Unfortunately, the study by Mantere et al³⁷ reported the mean test scores and SD only before the start of exposure and, therefore, was not included in our analysis.

Various attempts to control for factors such as age, education, and alcohol use, although useful, are probably insufficient because of the many confounding variables. Therefore, our review of the literature indicates a need for prospective studies that would directly compare the neurobehavioural function before exposure to test results after exposure.

Our meta-analysis shows that changes in study selection criteria or methods of analysis may greatly affect the results. For example, the results for the visual reproduction test changed several times depending on the model used. This finding indicates a need for caution in interpreting the results of meta-analyses in general, and meta-analyses of studies evaluating minor subclinical changes in particular.

In conducting meta-analyses, it is necessary to also consider the possibility of publication bias. Easterbrook et al³⁸ surveyed 487 research projects approved by the Central Oxford Research Ethics Committee between 1984 and 1987. By 1990, only 52% of these studies were published. Studies with significant positive findings were far more likely to be published. The observational studies were at particularly high risk of such publication bias.

We considered publication bias with two approaches. Firstly, we attempted to contact authors of published studies to identify unpublished and ongoing research. Although we became aware of several potentially relevant studies, we were unable to obtain the data. Secondly, we calculated a fail safe N. These calculations showed that in some instances the results might have been attributed to publication bias.

Our review of individual study results showed substantial variability, even in the absence of a significant homogeneity test. In some cases, the differences in mean values reported by different studies leads one to question whether these studies did, indeed, use the same tests. For example, for simple reaction time (non-preferred hand), Braun and Daigneault,²⁶ Haenninen et al,²⁸ and Repko et al³⁴ reported means for the exposed groups of 178.78, 1310, and 29.62, respectively. The source of such discrepancy is not apparent, although it is possible that different studies used different numbers of items.

In some cases, the source of discrepancy is more evident. For example, the articles by Araki et al,²⁵ Baker et al,³ Bolla et al,¹⁹ Campara et al,²⁰ Chia et al,²¹ Hogstedt et al,²⁹ Jeyaratnam et al,² Maizlich et al,²² and Valciukas et al,⁴ with means for the digit symbol test ranging from 29.4 to 59.9, indicate that this test is taken from the WAIS, whereas the papers by Parkinson et al,⁵ Pasternak et al,³² and Ryan et al,³⁶ with means ranging from 8.3 to 9.83 for the digit symbol test, indicate that the source for the test was the WAIS-R.

We think that in instances when the tests are evaluating the same psychological construct, although perhaps using different procedures and scoring, they legitimately may be combined in a meta-analysis using a random effects model. One methodological feature of this paper is the fact that analyses were conducted and presented with both the fixed effects and random effects models so that the results of these two models can be directly compared. Our reluctance to rigidly follow a certain analytical model is related to the fact that each of the proposed approaches has its limitations.

A recent meta-analysis by Meyer-Baron and Seeber³⁹ evaluated 12 tests from 22 studies using similar selection criteria, but a different analytical approach. For each test, a fixed effects model was assumed, the effect sizes calculated, and a test of homogeneity performed. If the test for homogeneity was significant, then a random effects model was used.

With this approach, the results for the block design, logical memory, and Santa Ana (preferred hand only) tests showed significant effects. By splitting the studies that included the digit symbol test into two groups and dropping one study,⁵ two homogeneous groups were formed and the effect sizes of both groups were significant. The authors concluded that “The evidence of neurobehavioural deficits at a current blood lead concentration of ∼40 μg/100 ml is obvious,” however, several concerns make this conclusion seem far reaching. Their meta-analysis included studies with a mean current lead concentration of <70 μg/dl. Therefore, many of the observed effects could be attributed to the inclusion of people whose lead concentrations were much higher than those claimed to be the focus of the study. The authors proposed that their calculated effect sizes for lead exposure might correspond to the effects of aging. This statement is not supported by the data. Moreover, as noted by Balbus-Kornfield et al,⁶ the effects of current lead exposure may reflect only transient and reversible changes, whereas the interpretation offered by Meyer-Baron and Seeber assumes permanent effects.

Also, concerns about study selection, inclusion and exclusion of tests, and choice of statistical analyses cast doubt on the conclusions of their meta-analysis. For example, Lindgren et al⁴⁰ reported score results for cumulative exposure categories only. Therefore, it is inappropriate to combine Lindgren et al⁴⁰ with other studies. Exclusion of the Lindgren study would exclude the digit span, trail making, and logical memory tests in accordance with their study criteria.

For the aiming test, the authors used the eye-hand coordination test from Repko et al³⁴ and the pursuit aiming test from Maizlish et al²² and Chia et al²¹ However, eye-hand coordination measures time “between successive pulses,” whereas pursuit aiming measures the “number of dotted circles.” Thus, the study by Repko et al³⁴ should have been excluded, thereby eliminating the aiming test from their meta-analysis.

Their block design test meta-analysis included data from Lindgren et al,⁴⁰ Haenninen et al,²⁸ Valciukas et al,⁴ Campara et al,²⁰ and Mantere et al³⁷ As already stated, the study by Lindgren should have been excluded, but papers by Baker et al³ and Parkinson et al⁵ that contain block design data should have been included.

Comparison of our results with those of Meyer-Baron and Seeber³⁹ indicates that all three tests (block design, Santa Ana and logical memory) reported as showing significant effects were no longer significant in our analysis. However, our analysis of the digit symbol test with a fixed effects model, unlike the result of Meyer-Baron and Seeber, was significant. In conducting analyses involving a long list of tests, it would be expected that at least some tests would show significant effects by chance. To interpret the results as reflecting an underlying biological process, consistency of findings across studies would be expected. A lack of consistency should caution against far reaching conclusions about the true biological effects of low to moderate lead concentrations.