METHODS

Study design

The EUROHAZCON study is a multicentre case–control study which uses data from seven existing regional, population based congenital malformation registers in five European countries (Belgium, Denmark, France, Italy, UK). The methodology of the study and findings regarding risk of congenital anomaly in relation to distance of residence from sites are described in detail elsewhere for non-chromosomal¹ and chromosomal anomalies.¹²

In summary, we identified 21 landfill sites which contained “hazardous” waste of non-domestic origin (as defined in the EC directive 91/689 on Hazardous Waste¹³) in the regions covered by the participating centres. Each of the participating centres had found a collaborating landfill specialist in their region who could help them identify eligible sites and gather relevant information. Study areas were defined as 7 km zones around each site. If study areas of two or more sites were overlapping, study areas were combined to form one large study area. In this way, 15 study areas were defined around the 21 landfill sites, with three study areas containing more than one site. One of these study areas (area 14) was excluded because geographic site coordinates used in initial analyses proved incorrect, resulting in a total of 20 landfill sites in 14 study areas. (Exclusion of site 14 did not change findings published previously: the odds ratio for living within 3 km of a site including site 14 was 1.33 (95% CI 1.11 to 1.59) for non-chromosomal anomalies¹; excluding site 14 this estimate was 1.34 (95% CI 1.12 to 1.60).) Within each study area, a 0–3 km “proximate” zone was defined on the advice of the collaborating specialists, to represent the zone of most likely exposure. In analyses this 3 km proximate zone was compared with a 3–7 km “distant” zone.

Cases included all malformed live births, stillbirths, and fetal deaths from 20 weeks gestation, and termination of pregnancy following prenatal diagnosis, born to mothers resident in one of the study areas, and born before 31 December 1994 and after five years of operation of the landfill site.¹ Cases with neoplasms, metabolic diseases, familial syndromes, minor malformations, and deformations were not included. Controls were non-malformed live births, approximately two per case, and selected from the same year of birth and same 7 km study area as the case.¹ They were either selected randomly from population registers, or by using one-to-one matching to facilitate the selection of controls in the absence of suitable population registers. Cases and controls were located geographically using addresses or postcodes at birth, with an accuracy of 100 m or better.

Landfill questionnaire

A questionnaire was completed for each of the study sites by the local waste authority responsible for the sites or their regulation. This questionnaire aimed to collect information that was readily available from existing documentation held by the waste site regulator, operator, inspector, and other relevant parties. Site visits were not carried out. Table 1 lists items included in the questionnaire.

The landfill questionnaire gave reasonably complete information on age and size of the EUROHAZCON study sites, soil type, and engineering and monitoring practices. Response rates for these items varied between 85% and 100% (table 1). Items related to total quantity of waste in place, contamination of ground or surface water, off site migration of landfill gas, and complaints about smells and odours were least well completed (40–55% response). For the majority of sites some monitoring results of either leachate, ground water, surface water, or landfill gas were available, but this type of information was not easily comparable between sites: monitoring was carried out for different substances, with different frequencies, on and off site, and in different years either during the study period or before. Summary reports of site investigations and monitoring were available for only six sites. Sites had all been reported to contain hazardous waste (as defined through the EC directive), but the amount of detail in the information deposited collected through the questionnaire on exact types and quantities of wastes was very variable (table 2). In most cases information on hazardous wastes deposited was limited to the types of industries from which the wastes originated.

Expert panel scoring

A panel of four was established from the group of collaborating regional landfill specialists on the basis of their varying geographic origin and interest in contributing expertise in hazard potential issues. Their collective expertise included fields of environmental chemistry, environmental and landfill engineering, hydrogeology, soil and ground water pollution, and risk assessment. This “expert panel” consisted of two landfill specialists who worked for regional environment agencies (in Scotland and Denmark) and two who worked for waste disposal companies (in England and Italy). The expert from Scotland had first hand knowledge for sites 15a and 15b and the expert from Denmark for sites 1 and 2. The other two experts had no first hand knowledge about any of the study sites, beyond that which they gained participating in the study. None of the experts were directly involved in the operation of any of the sites.

Results of the landfill questionnaire were summarised in a site description document and sent to the members of the expert panel. Each expert was asked to score each landfill site on the basis of the information provided in the site descriptions. Experts were blind to results of analyses of risk of congenital anomaly in relation to distance from each site. Sites were scored on a scale from 1 (low hazard) to 5 (high hazard) in three independent categories: water, air, and overall hazard table 3). The water hazard scoring aimed to reflect the ease with which hazardous materials can escape via the water route (groundwater and surface water), and the potential for the nearby population to come into contact with the water (via drinking water, surface water, recreation). The air hazard scoring aimed to reflect the ease with which hazardous substances in both vapour and particle form may be emitted into the air. The overall hazard scoring aimed to reflect a site’s overall potential to cause exposure of nearby residents relative to other sites. A large, old, badly managed site with many reported problems, for example, would receive a higher overall score than a well managed, small site.

In a subsequent meeting the members of the expert panel discussed differences between their scores for each site and were given the chance to consult additional documentation on the sites such as inspection and monitoring reports and site maps. During the meeting, initial scores were changed when discussion between experts led to a consensus, when first hand knowledge from one of the experts changed the opinion of the others, or when the information given in the summary description proved to have been misinterpreted by one or more of the experts. As an example of the latter, one site was judged of low air hazard by one expert because a gas collection system was present, whereas the other experts had noted that the gas collection system was installed after the study period ended and scored the air hazard higher. The first expert increased his score at the meeting. As an example of first hand knowledge leading to a change in scores, a site for which groundwater pollution had been detected and which was near a private drinking water well was judged of high water hazard by three experts. First hand knowledge of the fourth expert clarified that the groundwater flow was away from the drinking water well and the others lowered their scores.

Final scoring and ranking

Final hazard scores (after changes were made) of the four experts were averaged to form the final overall, water, and air hazard scoring. In study areas containing more than one site, different hazard scores were given to different sites, which made the assignment of one score to the exposure zone in those study areas problematic. Study area matching of cases and controls meant that only one score could be assigned to each study area. Within study area classifications were not possible. It was decided that if 3 km “proximate” zones around sites did not overlap in these multiple site areas, the average hazard score of the sites, weighted by the proportion of controls in the proximate zone around each site, most accurately represented the hazard of the proximate zone in the study area. If the 3 km zones did overlap, the score of the highest scoring site was applied to the 3 km proximate zone. This algorithm was developed after consultation with members of the expert panel, but it was noted that it was not possible to be confident about how hazards from multiple sites would affect exposure of residents in an area.

High, medium, and low hazard categories were created using tertiles of the hazard scores as cut off points. This resulted in categories of five study areas each. After exclusion of site 14, low hazard categories for overall, water, and air hazard contained four study areas each.

Analysis

In order to assess the agreement between experts in both initial and final scores, intraclass correlation coefficients (ICC) were calculated by analysis of variance. In addition, the reliability of the average expert scores (ICC_k) was calculated. ICC and ICC_k are calculated as follows¹⁴:

ICC = variance between sites/(variance between sites + variance within sites)

= inter-rater agreement = reliability of single rater

ICC_k = variance between sites/((variance between sites + variance within sites)/k)

= reliability of mean of k raters = reliability of the average score.

An intraclass correlation coefficient (ICC) of 1 reflects perfect agreement between experts.

In order to investigate whether the hazard potential of a site modified the odds ratio for residence within 3 km from a site, odds ratios for living within 3 km from a waste compared to further away (3–7 km) were calculated in each of the three hazard categories (high, medium, low). All odds ratios were stratified by study area and year of birth and adjusted for maternal age and socioeconomic status using logistic regression models.¹ The likelihood test for the interaction term between hazard category as a numerical variable (1=low, 2=medium, 3=high hazard) and distance zone (0–3 versus 3–7 km) was then used to test for the statistical significance of the trend in odds ratio from low to high hazard category. In addition, the interaction term between continuous hazard score (for 14 study areas) and distance zone (0–3 versus 3–7 km) was used to test for linear trend in odds ratio with continuous hazard score.

Hazard scoring analyses were carried out for all non-chromosomal anomalies combined, all chromosomal anomalies combined, and the three malformation subgroups which showed statistically significant increased risks related to residence within 3 km from landfill sites in our previous work (neural tube defects, cardiac septal defects, and malformations of the great arteries and veins).¹

RESULTS

Expert scoring

Initial scores

Few sites were given the score of 1 (low hazard)—the only ones were three sites which scored 1 for air hazard. Air hazard was generally scored lower than water hazard. Agreement between experts as measured by the intraclass correlation coefficient, was better for overall (ICC = 0.50) and water hazard scores (ICC = 0.44) than for air hazard (ICC = 0.20). The differences between the lowest and the highest expert score (measured on scale 1 to 5) given to a site also reflect this. For the majority of sites the difference between expert scores is one point or less in the overall (16 sites) and water scoring (12 sites), whereas in the air scoring only seven sites show one point or less difference between experts. Three sites show a difference of three points or more in the air hazard scoring.

Final scores

Table 4 shows the final hazard scores assigned by each expert to the study sites, as well as the average scores and the hazard category (low, medium, high) of each site. Scores that were changed during the expert panel meeting are emboldened in table 4. Few scores were changed in the overall and water hazard scoring: six and eight respectively. Differences between experts were greater for the initial air hazard scoring and 18 air scores for 11 sites were changed. As expected, changes made at the expert panel meeting improved agreement between experts. The intraclass correlation coefficient (ICC) for overall hazard scores increased from 0.50 to 0.60, water hazard scores from 0.44 to 0.56, and air hazard scores from 0.20 to 0.53. The number of sites differing by 1 point or less is 18 in the final overall hazard scoring, 15 in the final water hazard scoring, and 14 in the final air hazard scoring. Differences of two or more points are found for site 5 and 7b in the water score, and sites 1, 2, 7b, and 11 in the air score. The difference between the lowest and highest scoring expert was never more than 2.5 points in the final scores. The reliability of the average of the final scores of the four experts was high for overall (ICC_k = 0.86), water (ICC_k = 0.83), and air hazard scores (ICC_k = 0.82). Average scores covered a limited range with overall scores ranging from 2.50 to 4.63, water scores from 2.50 to 4.75, and air scores from 2.25 to 4.50 (table 3). The average final overall and water scores were highly correlated, with a correlation coefficient of 0.86. Correlations between overall and air (0.76) and water and air (0.62) were not as strong. All correlation coefficients were statistically significant (p < 0.01).

Relation between risk of congenital anomaly and hazard score

In table 5 the odds ratios (ORs) for living within 3 km from a landfill site compared to living further away from a site are presented pooled for all study areas and by low, medium, and high hazard categories. There was no evidence for a trend of increasing odds ratios with increasing overall hazard or air hazard. Odds ratios by water hazard category show an increasing trend of borderline statistical significance (p = 0.06) from 0.79 (0.51–1.21) in the low hazard category, 1.43 (1.10–1.86) in the medium, to 1.60 (1.16–2.21) in the high water hazard category. Testing the linear trend in the odds ratios for the 14 study areas with continuous hazard scores gave broadly similar results, although water hazard now suggests only a very weak and not statistically significant increasing trend (p > 0.2).

Odds ratios for chromosomal anomalies showed a similar pattern over the various hazard categories to those for non-chromosomal anomalies (table 6). Again only water hazard showed some weak, and not statistically significant, suggestion of an increase in odds ratios with hazard category.

In analyses of neural tube defects, cardiac septal defects, and malformations of the great arteries and veins, numbers of cases in different hazard categories were often small and confidence intervals wide, giving very limited power to test for differences between odds ratios (table 7). For neural tube defects odds ratios increased with air hazard category from 0.46 (95% CI 0.10 to 2.09) for low hazard, 1.93 (95% CI 1.23 to 3.02) for medium hazard, to 3.81 (95% CI 1.01 to 14.43) for high hazard, but this trend did not reach statistical significance (p = 0.06 for trend in three ORs). Odds ratios for malformations of cardiac septa increased with water hazard (low hazard OR: 0.99, 95% CI 0.50 to 1.99; medium hazard OR: 1.57, 95% CI 1.02 to 2.42; high hazard OR: 2.02, 95% CI 1.07 to 3.83), but again this trend did not reach statistical significance (p =0.16 for trend in three Ors).

DISCUSSION

Evaluation of scoring method

The interpretation of our results concerning the presence or absence of a relation between risk of congenital anomaly and hazard potential is dependent on an evaluation of the validity of the hazard scoring system. The hazard potential of a landfill site is dependent on many factors, and importantly also their interrelation. In the context of this study we believe it inappropriate to categorise sites by individual site characteristics (for example, age, size, waste type, containment, or engineering method). Analysis of a large number of such individual characteristics would lead to interpretational problems related to multiple statistical testing and small numbers, particularly in the absence of strong independent evidence of the degree to which these characteristics each individually determine hazard potential. Instead we aimed to capture hazard potential by combining information on many characteristics in a single scoring method.

Little is known in the published literature about the validity of existing hazard potential ranking systems, even of well used systems such as the USEPA Hazard Ranking System.⁹ A method for external validation of our expert panel assessment was not available. It was not feasible to take measurements of individual chemicals in air, water, or soil near the study sites and there was no reliable documented information on such measurements. In the absence of reliable and feasible methods for determining exposure to landfills, the expert panel scoring of hazard potential of landfill sites, even though crude, was the best proxy available. We compared the expert panel scoring to an adaptation of one published ranking system which was developed for use with existing site documentation and did not require information from site visits.¹⁵ Where there were differences, reasons could usually be found that pointed out deficiencies in the published ranking system. In particular, expert judgements were more able than the published ranking system to take into account the interrelations between factors. For example, the presence of a gas collection system was particularly important if significant biodegradable waste was present.

Expert panels have not been commonly used to assess environmental hazards, but they have proven useful in occupational settings to estimate exposures from job descriptions and titles where direct exposure measurements were not available.^16–19 The agreement between experts on a panel can give some indication of the reliability and therefore validity of a scoring method. Also, the more experts on a panel, the more repeatable, and therefore reliable, an average score will be (average score reliability for final scores was between 0.82 and 0.86 for our four member panel).^20,21 Agreement between experts in this study, measured by the interclass correlation coefficient, ranged from 0.20 (for air hazard) to 0.50 (for overall hazard) in the initial hazard ranking, increasing to 0.53 (for air hazard) to 0.60 (for overall hazard) after they had a chance to meet and discuss. Values of interrater agreement (reliability) between 0.40 and 0.75 have been reported as fair to good, values above 0.75 as excellent, and values below 0.40 as poor.²² Interrater agreements reported in occupational studies rarely exceed the value of 0.7.²² The final agreement found in this study falls within the range of interrater agreements reported, for example, in studies of pesticide applicants (0.4–0.8¹⁹), exposures of sawmill workers (0.40–0.68¹⁸), workers in various manufacturing industries (0.5–0.7¹⁶), and is higher than found in expert panels assessing metal exposures (0.2–0.5¹⁷) and various occupational chemical exposures (0–0.6²²). Comparisons are problematic of course, since different methods for expert assessment have been applied in these different studies.

Landfill questionnaires gave reasonably complete information on site characteristics such as size, age, engineering, and management practices, but there was little documented data on actual waste types deposited, chemicals present in the site, and off site migration of substances from the sites. Information on types of waste present (that is, chemical composition) would probably have been of limited use to differentiate sites, even if available. The vast majority of sites took a mixture of chemicals and our ability to judge the relative teratogenic potential of different waste types is very limited. There are no strong prior hypotheses about which specific chemicals or chemical mixtures may cause congenital malformation, although many chemicals commonly present in landfills (organic solvents, heavy metals, pesticides) have shown teratogenic potential.²³ Also, teratogenic potential depends on dose and there exists insufficient information on this. Moreover, the composition of wastes entering a site may bear very little resemblance to that of trace contaminants present in leachate and landfill gas emissions from sites. Indeed, in order to judge a site’s potential to generate landfill gas an estimate of the amount or proportion of biodegradable waste present in each site would have been of more use than a more detailed breakdown of waste types. First hand knowledge of sites was considered quite valuable by our expert panel. For example, additional knowledge of the direction of groundwater flow was used to judge potential risk to drinking water wells. First hand knowledge therefore addressed gaps in the questionnaire. First hand knowledge was only available for four sites and there were relatively more changes in scores for these sites (13 of 48 scores) than for other sites (19 of 192 scores). Ideally in future studies, site visits by one or more of the panel experts may give a better idea of the management of sites and adequacy of some pollution prevention measures, but cost effectiveness of such visits would need further evaluation.

Incomplete information may have resulted in either over or underestimation of the true relative hazard of sites. Where information that should normally be available is missing, this may indicate poorly managed sites with less pollution controls and therefore higher hazard potential. A study of the US EPA Hazard Ranking System on the other hand showed that missing information usually led to underestimation of hazard potential.²⁴ In our study, most data items were well completed, and the main issue was the limited scope of information available.

It was difficult to classify the hazard potential of study areas containing multiple sites with differing hazard potential scores. Experts agreed on an algorithm to classify these sites but the algorithm could not be validated. In future studies, dispersion modelling using meteorological, topographical, and hydrogeological information may be valuable in mapping patterns of relative exposure around landfill sites and could underpin the hazard potential assessments in multiple site areas as well as the definition of distance based exposure zones.

Interpretation of results regarding risk of congenital anomaly

Previous EUROHAZCON findings have shown an increased risk of congenital anomaly for residents living close to (within 3 km of) a hazard waste landfill site.^1,12 Potential sources of bias in the relation between distance of residence from sites and risk of congenital anomaly, including misclassification of exposure, ascertainment bias, migration bias, occupational and industrial exposures, and socioeconomic confounding, are discussed in our previous paper in detail.¹ The current findings show little evidence for relative risk of congenital anomaly close to landfill sites to be associated with the estimated hazard potential of landfill sites. Data show some evidence, although not statistically significant, of a trend of increasing relative risk of congenital anomaly with increasing water hazard of sites. The relation with water hazard could be a chance finding, as indicated by its low statistical significance. It provides suggestive evidence, however, that water is a more important exposure pathway than air for sites in this study, or that water is an equally or less important pathway than air, but easier to measure. There may be some reason to believe that water hazard was easier to classify than air hazard from the information available to the experts since agreement between experts on the initial water hazard scoring was considerably better than the initial air hazard scoring. Knowledge about pathways of potential exposure to landfill sites is as yet severely limited, adding to difficulties in interpretation of these findings.

Malformation subgroups analysed in relation to the hazard potential classification showed different patterns of risk with hazard potential: neural tube defects showed some evidence of a trend with air but not with water hazard, cardiac septal defects showed some evidence of a trend with water but not with air hazard. Although these findings may be caused by chance (again the trends reported were not statistically significant), they may alternatively indicate risks of different malformations occurring through different possible exposure pathways, possibly through exposures to different substances. This can only be resolved in a larger study with more detailed exposure assessment.

If misclassification of the relative hazard potential of one or a few sites occurred, this could have had an important impact on results regarding the risk of congenital anomaly risk near sites in each hazard category, especially if multiple site areas (for example, areas 13 and 15) and sites in the more densely populated study areas were misclassified. Such misclassification would reduce our ability to detect any true relation between risk of congenital anomaly and hazard potential.

Conclusion

We have shown the development of an expert panel hazard potential scoring method for an environmental exposure, and indicated ways in which the method could be improved in future studies of environmental exposures in general and landfill in particular. It is recognised that the hazard potential assessment presented in this paper has many limitations, the main one being the absence of an external validation method. However, the assessment method presented forms a basis for further developments and indeed expert assessment may be the only feasible way to assess the potential hazard of landfill sites in epidemiologic surveillance based studies in Europe.

Using the expert assessment method, we find little evidence for a relation between risk of congenital anomaly among proximate relative to distant residents and hazard potential of sites. This finding does not add support to a causal interpretation of the relation between distance from a waste site and risk of congenital anomaly. In the absence of external validation of the hazard potential scoring method, interpretation should be cautious. The extent of misclassification of hazard potential of sites is difficult to estimate and such misclassification may have reduced our ability to detect any true dose–response effect.