Skip to main content

Reproducibility and comparative validity of a food frequency questionnaire for Australian children and adolescents

Abstract

Background

Dietary intake during childhood and adolescence is of increasing interest due to its influence on adult health, particularly obesity, cardiovascular disease and diabetes. There is a need to develop and validate dietary assessment methods suitable for large epidemiologic studies of children and adolescents. Limited large scale dietary studies of youth have been undertaken in Australia, due partly to the lack of a suitable dietary intake tool. A self-administered, semi-quantitative food-frequency questionnaire (FFQ), the 'Australian Child and Adolescent Eating Survey' (ACAES), was developed for youth aged 9-16 years. This study evaluated reproducibility and comparative validity of the ACAES FFQ using assisted food records (FRs) as the reference method.

Methods

The ACAES FFQ was completed twice (FFQ1 and FFQ2) at an interval of 5 months, along with four one-day assisted FRs. Validity was evaluated by comparing the average of the FRs with FFQ2 (n = 113) as well as with the average of FFQ1 and FFQ2 (n = 101). Reproducibility was evaluated by comparing FFQ1 and FFQ2 (n = 101). The two methods were compared using correlations, Kappa statistics and Bland-Altman plots.

Results

Correlation coefficients for comparative validity ranged from 0.03 for retinol to 0.56 for magnesium for transformed, energy-adjusted, deattenuated nutrient data, with correlation coefficients greater than 0.40 for total fat, saturated fat, monounsaturated fat, carbohydrate, sugars, riboflavin, vitamin C, folate, beta-carotene, magnesium, calcium and iron. Correlation coefficients for reproducibility ranged from 0.18 for vitamin A to 0.50 for calcium for transformed, energy-adjusted, deattenuated nutrient data. The ACAES FFQ ranked individuals reasonably accurately, with the comparative validity analysis showing that over 50% of participants were classified within one quintile for all nutrients, with only a small percentage grossly misclassified (0-7%).

Conclusion

The ACAES FFQ is the first child and adolescent specific FFQ available for ranking the dietary intakes of Australian children and adolescents for a range of nutrients in epidemiologic research and public health interventions.

Background

Developing efficient, cost-effective and valid tools for assessing the dietary intakes of children and adolescents are key research priorities [1, 2]. The accurate assessment of dietary intake is critical to understanding the associations between food intake and obesity [3] as well as the influence of childhood dietary intake on chronic disease risk in adulthood [4–6]. The prevalence of obesity further necessitates the need for tools to assist in monitoring food consumption patterns. Ten per cent of the world's school-aged children are overweight or obese, with dramatically higher rates in developed regions, and prevalence continues to rise rapidly worldwide [7]. In New South Wales, Australia, the prevalence of overweight and obesity in children and adolescents doubled between 1985 and 1997 [8] with recent estimates at 25% [9].

Dietary assessment presents an ongoing challenge, particularly in large populations, with specific challenges for assessing the intakes of children and adolescents [10–12]. Although 24-hour recalls and food records (FRs) have been used successfully, the time and economic constraints of these methods make them unsuitable for most large scale studies [13]. Food frequency questionnaires (FFQs) have a lower respondent burden, are less time-consuming, less intrusive, relatively inexpensive and do not require trained interviewers, rendering them more practical for large-scale epidemiologic studies [14].

There has been limited nutrition data collected in large, representative samples of Australian children and adolescents. Only three national surveys have been conducted in the past 50 years: the 1985 National survey of schoolchildren (aged 10-15 years) [15], the 1995 National Nutrition Survey [16] which included children aged 2-18 years and the 2007 Australian National Children's Nutrition and Physical Activity Survey (ages 2-16 years) [17]. Yet the long-term nature of the development of diet-related chronic disease necessitates the collection of longitudinal dietary data. Frequency data can explain much of the variation in dietary intake and FFQs provide sufficient accuracy to relate individual diets in childhood and adolescence to subsequent health outcomes [18, 19]. FFQs have been used in adults to predict associations between dietary intake and disease specific mortality and morbidity including, colon cancer, heart disease and diabetes [20]. In the US, the association between diet and health outcomes has been monitored in the Growing Up Today Study (GUTS), a longitudinal study of 16,882 9-14 year olds using the Youth-Adolescent Questionnaire (YAQ). Data collected in GUTS by the YAQ FFQ has been used to investigate the relationship between BMI and dietary intake [21], BMI and snack foods [22] and has demonstrated a relationship between BMI and intake of sugar-sweetened beverages [23] and BMI and family mealtimes [24]. There is currently no widely accepted FFQ for school-aged children in Australia. The Australian Child and Adolescent Eating Survey (ACAES) was developed in response to these gaps.

The aim of this study was to assess the reproducibility and comparative validity of a semi-quantitative, self-completed FFQ designed for school-aged children aged 9 to 16 years in the Hunter region, New South Wales, Australia.

Methods

Development of the ACAES FFQ

The ACAES FFQ is based on the YAQ, validated in the US by Rocket et al. [25]. Focus groups and pre-testing were undertaken with students aged 9-16 years to develop a comprehensive food list and refine the format of the FFQ. Five focus groups were held with a representative sample of 61 students: 31 primary students (9-12 years) and 30 secondary students (13-16 years), from two primary and two secondary schools. Students were asked what they usually ate for breakfast, snacks, lunch and dinner. A list was compiled and they were asked to add to this list until all the food and drinks usually consumed were included. Focus group results were used to modify the food list of the YAQ to reflect the Australian food supply, the local vernacular and the foods frequently consumed by the target population.

Once the food list was developed, a draft of the ACAES FFQ was pre-tested with a convenience sample of 41 primary and secondary students. Students were asked to complete the FFQ and circle anything that was not clear. On completion of the survey, the students discussed with a research assistant, any aspects of the FFQ they did not understand. As a result of the pre-testing, small modifications were made to the food list and ranges within response categories.

The final version of the ACAES FFQ is a 120-item FFQ with 15 supplementary questions regarding age, use of vitamin supplements, food behaviours and sedentary behaviours. The FFQ is designed as a self-administered tool, to collect information about the dietary intake of 9-16 year olds over the previous 6 months. An individual response for each food, or food type, is required, with frequency options ranging from 'Never' to '4 or more times per day', but varied depending on the food. The ACAES is semi-quantitative with a standard portion size provided for each food item and determined using 'natural' serving size (eg. slice of bread). In the absence of a natural serving size, portion sizes were derived from the 1995 National Nutrition Survey (NNS) (unpublished data purchased from the ABS). These data were for two age groups: 9-13 years and 13-16 years for boys and girls separately. Although boys had larger serve sizes than girls and the older age group had larger serve sizes than the younger, the differences were small. The average portion size for each item was calculated and used for all participants. This was the approach used by Rockett et al in the development of the Youth Adolescent Questionnaire (YAQ) [26] on which the ACAES FFQ was modelled. There were eight items without a 'natural' serving size or NNS data. For these foods, either FoodWorks 'Unspecified' serve sizes were used (5 items) or packet serve sizes (3 items). For composite items (those including more than one food) the NNS data was used and the serving size was weighted according to the NNS consumption data so that the foods consumed by the largest numbers of this age group were weighted more heavily. Due to the seasonal availability of some fruits, a separate section was included in the FFQ for seasonal fruit. The availability of seasonal fruit was determined by obtaining supermarket literature and contacting Sydney Markets, the largest fruit and vegetable market in New South Wales, which is 180 kilometres from Newcastle. Both sources indicated the number of months each year that seasonal fruit was available. The frequency categories for seasonal fruit were listed as for other food items and participants were asked to answer the question based on intake when the fruit is available. The average daily intake of seasonal fruit was calculated by adjusting for the number of months per year the fruit was available.

Participants for the comparative validity and reproducibility study

The protocol for this study was approved by the University of Newcastle Human Research Ethics Committee (Approval No. H-498-0203) and the NSW Department of Education and Training (SERAP No 03.48). A total of 224 students in years 4, 6, 8 and 10 (aged 9-16 years) from seven schools in the Hunter region, New South Wales, Australia, were invited to participate in the study. Two rural primary schools, two urban primary schools, two rural secondary schools and one urban secondary school were included. Socio-economic status was measured using the Socio-Economic Indexes for Areas (SEIFA) allocated on the basis of school postcode and compared to the NSW average for the Index of Relative Socio-Economic Advantage/Disadvantage [27].

Collection of dietary data

The ACAES FFQ was administered at the beginning and end of the five month study period (Figure 1). The reference method of four assisted food records (FRs) was undertaken during the same period. The protocol was designed to capture weekly and seasonal variation.

Figure 1
figure 1

Timeline for data collection of four assisted food records (FR) and two ACAES FFQs for the comparative validation and reproducibility of the ACAES FFQ in students aged 9-16 years in the Hunter region, 2003.

The ACAES FFQ was completed by the children and adolescents, at school. Blank forms for the FRs were distributed to participants at school two school days prior to each data collection visit. The students were asked to complete the FR for the following day (or the day after if it was to be recorded for a Sunday). When the research team visited the school two school days later, the completed FRs were reviewed individually with the participants to clarify all food types and quantities. Any participants that had not completed a FR, instead completed a 24-hour recall (37% of records) with a research assistant following a standardised proforma. The FRs and 24-hour recalls were completed for weekdays (75% of records) and weekends (25% of records) to capture differences in eating habits. To help participants give accurate portion sizes, incremental cup and spoon measures, glasses and a ruler were used. Data from FFQ1, FFQ2 and the FRs/24-hour recalls were entered and nutrient intakes computed in FoodWorks (Version 3.02.581) using the following databases: Australian AusNut 1999 database (All Foods) Revision 14 and AusFoods (Brands) Revision 5 [28].

Data cleaning

A total of 63 FFQs and 10 FRs were excluded from the analysis due to implausible energy intakes. The cut-offs were less than 2,090 kilojoules or greater than 20,900 kilojoules for an individual record, as used in previous studies [25].

The distribution of intakes of all nutrients, except energy, were examined for extreme values using scatterplots generated for the average nutrient values of the FRs plotted against the average of FFQ1 and FFQ2. The original FRs and/or FFQs for these participants were checked and a decision made about whether or not to exclude the record from the analysis. The two main reasons for exclusion were more than 20 main meals a week reported on the FFQ (n = 5) and food records where intakes were reported to be 'less than usual' for 3 or 4 food records (n = 2).

FFQ records were excluded if more than 10% (12 items) of the questions did not have a response (n = 36), with the number of missing items ranging from 12 - 112. Food record data was not included in the analysis if participants had completed less than three FRs. A total of 61 food records were excluded from the analysis, for 37 participants that completed less than three food records.

Statistical Analysis

All statistical analysis was performed using STATA version 8 (2003, Stata Corporation, College Station, TX, USA) [29].

The raw data for most nutrients were not normally distributed but were positively skewed, as commonly found with dietary data [30]. Statistical and graphical assessment was used to determine the most appropriate transformation to achieve normal distribution. Logarithmic and square root transformations were attempted, with the square root transformation producing the most normalised distributions for all but a few nutrients. For simplicity, the square root transformation was used for all nutrients [30]. The nutrients were energy-adjusted using the method of Willett and Stampfer [31].

As recommended [19, 32–34], comparative validity of the FFQ was evaluated by comparing the average of the FRs to FFQ2 as well as to the average of FFQ1 and FFQ2. Reproducibility was assessed by comparing the nutrient intake data from FFQ1 and FFQ2.

Means, standard deviations, medians and interquartile ranges were calculated for 22 nutrients. Spearman correlation coefficients were calculated using the unadjusted nutrient data. Pearson correlation coefficients were calculated on the data after statistical adjustment (transformation, energy-adjustment and de-attenuation). To adjust for within-person variation in the FRs, deattenuated correlation coefficients were calculated [35]. Misclassification error was assessed using the Kappa statistic [36] to compare classification of nutrient data into quintiles. In most applications, the number of categories used for calculating kappa statistics varies from two to five [37, 38]. In validation studies of dietary intake, tertiles [39], quartiles [40] and quintiles [41] have all been used in the calculation of kappa statistics. The proportion grossly misclassified was also estimated.

The Bland Altman method was used to evaluate the agreement between the FFQ and FRs and between FFQ1 and FFQ2 [42]. The Bland Altman method plots the individual differences between the two measurements against the mean of the measurements. For comparative validity, the difference between the average of FFQ1 and FFQ2 and the average of the food records [FFQavg-FRavg]s was plotted on the y-axis and the mean of the FFQ1/FFQ2 average and the FR average [(FFQavg-FRavg)/2] was plotted on the x-axis. For reproducibility, the difference between FFQ1 and FFQ2 [FFQ1-FFQ2] was plotted on the y-axis and the average of FFQ1 and FFQ2 [(FFQ1+FFQ2)/2] was plotted on the x-axis. These plots indicate the direction of bias and whether it is constant across levels of intake. The 'limits of agreement' or LOA (the mean difference ± 2SDs for the difference between the two measurements) determine whether the agreement between the two methods is acceptable [43]. The differences were checked for normality as calculation of the 95% LOA assumes differences are normally distributed [44]. Where Bland Altman plots showed a tendency for the differences to increase as the magnitude of the measurement increased, the data was log-transformed and replotted. If the relationship remained, the differences were regressed onto the means. Interpretation of the Bland-Altman results was based on the categories defined by Tang [45]. 'Good agreement' is shown when the difference between the two measurements is approximately equal to one standard deviation of the average nutrient intake from the reference method. 'Fairly good agreement' is when the difference between the two measurements is approximately equal to two standard deviations of the average nutrient intake from the reference method and 'Bad (or poor) agreement' occurs if the difference between the two measurements is approximately equal to three standard deviations of the average nutrient intake from the reference method.

Results

A total of 224 students were recruited to the study. Following data cleaning, records for 113 participants remained for the comparative validity analysis (50% of those recruited) using the assisted FRs and the FFQ2 data. Complete records for 101 participants remained for the validity analysis using the average of FFQ1 and FFQ2, with the same number for the reproducibility analysis (45% of those recruited). Table 1 reports the baseline characteristics for age, height, weight, BMI and BMI z-score for the study participants. There were no significant differences between the baseline characteristics of the participants and those excluded from the analysis.

Table 1 Baseline characteristics of study participants (n = 113, 36% male)

The SEIFA codes for all schools were below the NSW average for the Index of Relative Socio-Economic Advantage/Disadvantage [27].

The median and interquartile ranges for energy and nutrients are presented in Table 2. The median values for FFQ2 and the FRs were exactly the same for monounsaturated fat, niacin equivalent, folate and iron. For all other nutrients, the values for FFQ2 were higher than those of the FRs, with the exception of polyunsaturated fat, carbohydrate, thiamin and vitamin C. The interquartile range of most nutrients tended to be wider for the FFQ data than the FRs. The dietary intakes estimated using FFQ1 were slightly higher than for FFQ2 (data not shown). This resulted in higher median values for all nutrients for the average of FFQ1 and FFQ2 when compared to the FRs.

Table 2 Summary statistics for the ACAES FFQ2, the average of ACAES FFQ1 and ACAES FFQ2 and the average of (3 or 4) assisted food records (FRs)

Table 3 shows the correlation coefficients for comparative validity.

Table 3 Correlation coefficients for the food frequency questionnaires (FFQ1 and FFQ2) and assisted food records (FRs) for comparative validity

When the unadjusted data for FFQ2 and the average of the FRs was compared, Spearman's correlation coefficients ranged from 0.09 (retinol) to 0.35 (calcium). The median correlation for the unadjusted nutrient intakes of the two methods was 0.25. After transformation, the mean Pearson correlation was 0.24, ranging from 0.13 (niacin equivalent) to 0.37 (calcium). Deattenuation of the correlations for the transformed data increased the mean correlation to 0.27, ranging from 0.03 (retinol) to 0.41 (calcium). When the transformed data was energy-adjusted, the mean correlation coefficient increased to 0.34, with substantially larger coefficients for most nutrients except retinol, niacin equivalent, fibre and polyunsaturated fat. Deattenuation of the correlations for the transformed, energy-adjusted data increased the mean correlation to 0.39, ranging from 0.17 (niacin equivalent) to 0.56 (magnesium). The mean correlation for FFQ1 and the average of the FRs was higher than for FFQ2 for the crude data, the transformed data and the de-attenuated, transformed data (0.32, 0.34 and 0.38, respectively). However, the correlations for FFQ1 and the average of the FRs dropped following energy adjustment, resulting in lower correlations (data not shown) than for the energy-adjusted data for FFQ2.

Reproducibility was evaluated by calculating correlation coefficients for FFQ1 and FFQ2, as shown in Table 4. Spearman correlations for the unadjusted data ranged from 0.34 (sugars) to 0.53 (niacin equivalent). The median correlation for the unadjusted nutrient intakes of the two FFQs was 0.46. After transformation, the mean correlation using Pearson correlation coefficients was lower (0.44), ranging from 0.34 (sugars) to 0.51 (niacin equivalent). When the transformed data was energy-adjusted, the mean correlation coefficient decreased from 0.44 to 0.32, with substantially smaller coefficients for all nutrients, except magnesium and calcium which increased.

Table 4 Correlation coefficients for the ACAES FFQ1 and ACAES FFQ2 for reproducibility

Table 5 shows the results of the kappa analysis undertaken as part of the comparative validity.

Table 5 Kappa statistics for food-frequency questionnaires (FFQ1 and FFQ2) and assisted food records (FRs) for comparative validity

For the comparative validity analysis, the weighted kappa values for the nutrient intake data from FFQ2 and the average of the FRs ranged from 0.09 (retinol) showing 'slight' agreement, to 0.36 (iron), showing 'fair' agreement. All other nutrients showed 'slight' to 'fair' agreement. The proportion of individuals correctly classified into the same quintile was highest for sugars (31%) and lowest for zinc and folate (19%). The percentage classified within one quintile was highest for beta-carotene and calcium (65%) and lowest for zinc (51%). The percentage grossly misclassified (those ranked in the lowest quintile for the FFQ data but the highest quintile for assisted FRs) was small (1-7%).

For the reproducibility analysis of the nutrient intake data from FFQ1 and FFQ2, niacin (0.54) showed the strongest weighted kappa value and was classified as having 'moderate' agreement, while sugars had the lowest weighted kappa value (0.36) indicating 'fair' agreement. As shown in Table 6, with the exception of carbohydrate, sugars and vitamin A, all nutrients showed 'moderate' agreement with values between 0.41 and 0.60. The proportion of individuals correctly classified into the same quintile was highest for niacin equivalent (39%) and lowest for sugars (23%). The percentage classified within one quintile was highest for protein (79%) and lowest for vitamin A (63%). A very small percentage of individuals were grossly misclassified, ranging from none for Vitamin C and 5% for fibre.

Table 6 Kappa statistics for food-frequency questionnaires (FFQ1 and FFQ2) for reproducibility

When applying the Bland Altman method to the comparative validity data, the mean difference between the methods using the raw nutrient intake data was positive for most nutrients, indicating that when compared to FRs, the ACAES FFQ provides higher estimates for the intake of all nutrients, except polyunsaturated fat and thiamin.

Figure 2 shows the Bland-Altman plot for comparative validity analysis using the raw data for beta-carotene. The plot is scattered, indicating the bias is constant across levels of intake. The limits of agreement are approximately equal to two standard deviations of the FR data, showing fairly good agreement [45] between the FRs and the average of the FFQs.

Figure 2
figure 2

Bland Altman plot of the difference between beta-carotene intake measured by FFQ1 and FFQ2 and the average of the assisted 24-hr recalls, plotted against the mean beta-carotene intake of the two methods, for the comparative validity analysis. The solid horizontal line in the centre indicates the mean difference between the two methods (1184 μg) and the solid lines above and below this indicate ± 2SDs (-3200 μg to 5613 μg)

The Bland Altman plots showed a tendency for the differences to increase as the magnitude of measurement increased for all nutrients, except beta-carotene, polyunsaturated fat, thiamin and vitamin C. The data for nutrients showing this trend were log transformed in an attempt to remove this relationship [42, 44]. For calcium, folate, sugars, riboflavin, thiamin and vitamin A this created a plot with a more consistent bias across levels of intake as shown in Figure 3 for calcium. However, the wide 95% LOA (antilog values of -53% to 336%) indicated discrepancies between the two methods for some individuals.

Figure 3
figure 3

Bland Altman plot of the difference between log-transformed data of calcium intake measured by FFQ1 and FFQ2 and the average of the assisted 24-hr recalls, plotted against the mean calcium intake of the two methods, for the comparative validity analysis. The solid horizontal line in the centre indicates the mean difference between the two methods (34% greater for the FFQ - antilog) and the solid lines above and below this indicate ± 2SDs (-53% to 336% - antilog).

Log-transformation did not remove this relationship for energy, protein, total fat, saturated fat, monounsaturated fat, carbohydrate, fibre, niacin, retinol, iron and zinc. For these nutrients, the absolute residuals from a fitted regression line method was used [44] and this regression analysis showed significant relationships existed for these nutrients.

Bland Altman analysis of the reproducibility data showed a small, positive mean difference for all nutrients, indicating that FFQ1 tended to give slightly higher estimates of nutrient intake than FFQ2. The plots for all nutrients were scattered, showing consistent bias across all levels of intake. The LOA were wide (greater than ± 2SDs of the assisted FRs) indicating poor agreement between the two FFQs across the range of intakes.

Discussion

Comprehensive FFQs have been developed to assess the total diets of children and adolescents in the United States [26], Norway [46], the United Kingdom [47] and Italy [48] with a limited number of validation studies undertaken. These studies demonstrated that for these populations, dietary intakes could be measured reasonably well using an FFQ. Differences in study methods, populations and between-person variation make it difficult to compare validity across studies internationally. However, it is worth noting that the results from the present study are similar to those of previous FFQ validation analyses for children and adolescents [25, 47, 49–53].

The validity of the ACAES FFQ was assessed by comparing the nutrient estimates from the average of the FRs with ACAES FFQ2 as well as comparing the average of the FRs with the average of ACAES FFQ1 and ACAES FFQ2. Although ACAES FFQ2 represents the conceptually appropriate time sequence, the process of keeping FRs might alter awareness of food intake and artificially improve accuracy in completing it [19]. However, comparing ACAES FFQ1 with the FRs, would tend to underestimate validity because ACAES FFQ1 asked about diet prior to the study period. Therefore, the use of the average of ACAES FFQ1 and ACAES FFQ2 provides a combination of minimal and maximal estimates and is likely to be a more accurate indication of true validity [19].

Food frequency questionnaires tend to estimate higher nutrient intakes when compared to 24-hour recalls and FRs [54]. This was shown in the present study, with the ACAES FFQ providing higher estimates of intakes for all nutrients, except polyunsaturated fat, carbohydrate, thiamin, niacin and vitamin C, when compared to assisted FRs.

The correlation results from this study are comparable to those for the YAQ [25]. This was anticipated given that ACAES was modified from the YAQ and both studies had a similar design. When the correlation results for relative validity of the ACAES FFQ are compared to the YAQ [25], the correlation for carbohydrate is equivalent (r = 0.47), while the correlations for fat, saturated fat, folate, iron and riboflavin (Vitamin B2) are marginally lower. The correlations for beta-carotene, magnesium and zinc were higher for the ACAES FFQ compared to the YAQ. Correlations for all other nutrients were either lower (energy, protein, monounsaturated fat, fibre, thiamin, vitamin C, vitamin A, calcium), not significant (polyunsaturated fat, niacin and retinol) or not common to both studies (sugars, niacin equivalent).

The mean correlation coefficients for the common nutrients between both studies (without vitamin supplementation) were calculated, excluding those nutrients in our study with correlation coefficients that were not significant (>0.05) (polyunsaturated fat, niacin and retinol). Rockett observed a mean correlation of 0.42 between the 2nd YAQ and the mean of three 24-hour recalls [25], compared to 0.39 for this study, indicating similar correlations between the two methods in both studies.

When compared to the correlations for other comparative validity studies, the ACAES FFQ showed similar results. The ACAES FFQ had higher correlations than the Block Kids Questionnaire (BKQ) for fibre and calcium, but not for energy [49]. Our study had almost identical correlations for calcium as two FFQs designed specifically to assess calcium intake in children and adolescents in the US and Italy [50, 48, 51]. As found in the present study, others have also shown poor agreement for vitamin A [32], polyunsaturated fat [52, 55] and protein [47, 53]. This is probably due to the large day-to-day variation in the intakes of these nutrients by children and adolescents, particularly girls. To reflect the usual intakes of vitamin A, polyunsaturated fat and protein, a greater number of recording days is necessary, with as many as 46 days recording required to estimate the usual vitamin A intake for females aged 5-17 years [56].

In order to detect associations between diet and disease, it is suggested that correlations need to be at least 0.3 or 0.4 [32]. The present study had significant correlations greater than 0.3 for the deattenuated data of all nutrients except protein and vitamin A.

The reproducibility results of the YAQ [26] were similar to the present study. The mean correlation for the YAQ was 0.41 for the log-transformed, energy adjusted data for the 7 nutrients presented. For the 21 nutrients examined in the ACAES FFQ reproducibility analysis, the mean correlations were 0.46 for raw data, 0.44 for transformed, 0.32 for transformed, energy-adjusted data. In the present study, the correlations for many nutrients dropped following transformation and again following energy adjustment. The square-root transformation was applied to all nutrients due to its appropriateness for the majority of nutrients for the ACAES FFQ1 and FFQ2 data. However, it was not the most statistically appropriate transformation for some nutrients in ACAES FFQ2 and is likely to be the reason the correlation coefficients for these nutrients dropped following transformation. The reduction in the reproducibility correlations following energy adjustment has occurred in other studies of reproducibility [57–60] and is likely to be due to systematic errors of over- and under-estimation between ACAES FFQ1 and FFQ2. Although we have no way to assess this directly, adjustment for energy increases correlation coefficients when the variability in nutrient intake is related to energy intake, but results in lower correlations when the nutrient variability depends on systematic errors of overestimation and underestimation [35]. The correlation coefficient for reproducibility for total fat intake in our study was higher for crude data (0.49) when compared to the reproducibility correlation (0.41) for an FFQ for children aged 2-5 years developed specifically to assess fat intake [61] demonstrating good validity for total fat intake.

Bland Altman analyses have not often been reported in studies comparing FFQs with diet records in children and adolescents [53]. In the only other study reporting Bland Altman results for children (11-13 years) [53], the findings were similar. The mean differences were positive and of very similar magnitude to the current study, with the limits of agreement wide for all nutrients presented (energy, fat, sugar, calcium and protein). Similarly, the Bland Altman plots from the raw data showed strong trends of increasing difference with increasing intakes. The Bland Altman results for the present study show that the ACAES FFQ is not suitable for estimating absolute intakes for children and adolescents, but is appropriate for ranking intakes. The similarity of results between the present study and that of Lietz et al [53], suggests that large variation in the agreement between methods may be characteristic of child/adolescent populations and that these results may be due to the usual variability of dietary intake of children and adolescents. Agreement between FFQs and 24-hour recalls or food records may be lower in children and adolescents than in adults due to the greater day to day variation in their dietary intake [56]. Adolescents have highly variable food patterns, with possibly half of the foods they eat varying greatly [62].

The reference method of choice for FFQ validation studies is weighed food records or diet records [34]. Although 24-hour recalls have less respondent burden, their sources of error tend to be more correlated with the error in an FFQ due to reliance upon memory, conceptualisation of portion sizes and distortion of reported diet [34]. There was a high proportion of children aged 9-12 years (71.7%) in the sample and it was expected that some of these participants would be less likely to complete a food record. The 24-hour recall method was undertaken with participants that had not completed their food records due to it's suitability to participants with limited cooperation or literacy [34]. Interestingly, approximately the same proportion of younger children (9-12 years) and older children (13-16 years) completed 24-hour recalls (36.8% and 38.1% respectively) because they had not completed a food record.

Standard portion sizes were applied to the ACAES FFQ because frequency has been found to be more discriminatory than portion size [63, 64]. Adding open-ended questions regarding portion size can actually reduce validity of an FFQ due to sources of error in conceptualising serve sizes and large, within-person variations in serving sizes when the same food is consumed on different occasions [20].

A general limitation of validation studies is that the results are not necessarily transferable to another population [65]. A sample size of at least 50 is desirable for each demographic group [33], and ideally between 100 and 200 participants [19]. The sample size in the present study was inadequate to compare the validity and reproducibility of subsets for age, gender or BMI category. Performance of the ACAES FFQ would need to be tested in populations of different SES and ethnicity.

A total of 43 participants were excluded due to implausible energy intakes reported on the ACAES FFQ, representing 19% of the number recruited. A total of 36 of these were for high intakes, while seven were for implausibly low intakes. This is a high proportion and we feel that the main reason was the completion of the ACAES FFQ in a class environment. It seemed that by completing the FFQ as a group, there may have been more distractions compared to completing it on their own or with a parent present. Some of the excluded students marked the highest intake category for almost all food items. In a subsequent study by the same research team using the ACAES FFQ with primary school children, only 2 out of 60 participants (3.3% of the sample) were excluded due to implausible energy intakes (data not shown). These children completed the FFQ independently at the school, rather than in a class environment. In future studies, it is recommended that the ACAES FFQ is completed by the child independently, rather than in a class environment.

Despite these limitations, there are many aspects of the study design and analysis that are likely to contribute to an underestimation of the true validity and reproducibility of the ACAES FFQ. Food records were used predominantly (63% of records) and combined with 24-hour recalls (37% of records). Due to differences in the biases of food records and FFQs, the predominant use of food records is likely to underestimate validation coefficients when compared to 24-hour recalls as the reference method [19]. The day of the week is also likely to influence the results of validation studies. In the present study, the proportion of records collected on weekdays (75%) and weekends (25%) is close to the actual proportion of weekdays and weekend days (71% and 29%, respectively). When comparing 14 days versus 2 days of food records in Australian children, Jenner et al found that correlation coefficients were slightly higher when both days of the two day records were taken on weekdays [66]. It is possible that this is due to a perceived 'usual' intake during the week and greater variation on weekends. Participants reported a 'usual' intake for 64% of the records collected. Of the remaining 36% of records, half were reported as 'less than usual' with the other half reported as 'more than usual'. This large proportion of 'more' or 'less' than usual intakes may be typical of the intakes of this age group, but is likely to contribute to reduced agreement with the FFQ, where they record their perceived 'usual' intake. The conservative approach to excluding records from the data analysis also contributes to underestimating the validity of the FFQ. For example, respondents with food records that were inconsistent with their FFQs were not excluded if their responses were plausible. The most extreme examples of these were intakes of liver in the FFQ, but none reported on the FR days (resulting in particularly poor agreement for vitamin A and retinol) and reporting items in their food records that were not on the FFQ (eg. oysters, popcorn, slurpee, added sugar, chocolate topping, sherbet).

Although 24-hour recalls capture rich information on food consumption, they measure episodically consumed foods poorly. Recent statistical modelling has suggested that combining FFQ data with a limited number of 24-hour recalls may provide the most accurate method of estimating usual dietary intake at the individual level, supporting the use of both methods in national surveillance [67].

Conclusion

This study evaluates the strengths and limitations of a self-administered FFQ developed for Australian children and adolescents aged 9 to 16 years. When compared to assisted FRs, the ACAES FFQ overestimates nutrient intakes and is therefore not suitable for estimating absolute intakes for individuals in this age group. However, with the exception of polyunsaturated fat and retinol, it demonstrates an acceptable ability to correctly classify participants into quintiles of intake.

This FFQ has comparable results across a wide range of nutrients to other FFQs designed for children and adolescents evaluated for validity and reliability. It provides an important contribution to the tools available for assessing usual intakes in children and is a useful tool for estimating the dietary intakes of total fat, saturated fat, carbohydrate, sugars, fibre, vitamin C, folate, beta-carotene, calcium and iron in clinical practice, epidemiologic research and public health interventions among Australian youth and adolescents.

References

  1. Potischman N, Cohen BE, Picciano MF: Dietary recommendations and identified research needs for The National Children's Study. J Nutr. 2006, 136 (3): 686-9.

    CAS  Google Scholar 

  2. Taren D, et al: Dietary assessment methods: where do we go from here?. Public Health Nutr. 2002, 5 (6A): 1001-3.

    Article  Google Scholar 

  3. Davison KK, Birch LL: Childhood overweight: a contextual model and recommendations for future research. Obes Rev. 2001, 2 (3): 159-71. 10.1046/j.1467-789x.2001.00036.x.

    Article  CAS  Google Scholar 

  4. Andersen LB, et al: The relationship between physical fitness and clustered risk, and tracking of clustered risk from adolescence to young adulthood: eight years follow-up in the Danish Youth and Sport Study. Int J Behav Nutr Phys Act. 2004, 1 (1): 6-10.1186/1479-5868-1-6.

    Article  Google Scholar 

  5. WHO: Expert consultation on diet, nutrition and the prevention of chronic diseases. WHO Technical Report Series No. 916. 2003, WHO: Geneva

    Google Scholar 

  6. Dietz WH: The obesity epidemic in young children. Reduce television viewing and promote playing. Bmj. 2001, 322 (7282): 313-4. 10.1136/bmj.322.7282.313.

    Article  CAS  Google Scholar 

  7. Lobstein T, Baur L, Uauy R: Obesity in children and young people: a crisis in public health. Obes Rev. 2004, 5 (Suppl 1): 4-104. 10.1111/j.1467-789X.2004.00133.x.

    Article  Google Scholar 

  8. Booth ML, et al: The epidemiology of overweight and obesity among Australian children and adolescents, 1995-97. Aust N Z J Public Health. 2001, 25 (2): 162-9. 10.1111/j.1753-6405.2001.tb01840.x.

    Article  CAS  Google Scholar 

  9. Booth M, et al: NSW Schools Physical Activity and Nutrition Survey (SPANS) 2004: Full Report. 2006, NSW Department of Health: Sydney

    Google Scholar 

  10. McPherson RS, et al: Dietary assessment methods among school-aged children: Validity and reliability. Preventive Medicine. 2000, 31: S11-S33. 10.1006/pmed.2000.0631.

    Article  Google Scholar 

  11. Livingstone MB, Robson PJ, Wallace JM: Issues in dietary intake assessment of children and adolescents. Br J Nutr. 2004, 92 (Suppl 2): S213-22. 10.1079/BJN20041169.

    Article  CAS  Google Scholar 

  12. Persson LA, Carlgren G: Measuring children's diets: evaluation of dietary assessment techniques in infancy and childhood. Int J Epidemiol. 1984, 13 (4): 506-17. 10.1093/ije/13.4.506.

    Article  CAS  Google Scholar 

  13. Buzzard M: 24-hour dietary recall and food record methods. Nutritional Epidemiology. Edited by: Willett W. 1998, Oxford University Press: Oxford

    Google Scholar 

  14. Willett W: Food-frequency methods. Nutritional Epidemiology. Edited by: Willett W. 1998, Oxford University Press: Oxford

    Chapter  Google Scholar 

  15. Department of Community Services and Health: National dietary survey of schoolchildren (aged 10-15 years): 1985. No. 2 Nutrient intakes. 1988, Canberra: AGPS

    Google Scholar 

  16. ABS: National Nutrition Survey: Nutrient Intakes and Physical Measurements. 1998, Australian Bureau of Statistics: Canberra

    Google Scholar 

  17. CSIRO: 2007 Australian National Children's Nutrition and Physical Activity Survey: Main Findings. Canberra. 2008

    Google Scholar 

  18. Rockett HRH, Colditz GA: Assessing diets of children and adolescents. American Journal of Clinical Nutrition. 1997, 65 (suppl): 116S-22S.

    Google Scholar 

  19. Willett W, Lenart E: Reproducibility and validity of food-frequency questionnaires. Nutritional Epidemiology. Edited by: Willett W. 1998, Oxford University Press: Oxford

    Chapter  Google Scholar 

  20. Willett WC: Invited commentary: comparison of food frequency questionnaires. Am J Epidemiol. 1998, 148 (12): 1157-1159. 10.1006/pmed.2001.0850.

    Article  CAS  Google Scholar 

  21. Rockett HR, et al: Cross-sectional measurement of nutrient intake among adolescents in 1996. Prev Med. 2001, 33 (1): 27-37. 10.1038/sj.ijo.0802762.

    Article  CAS  Google Scholar 

  22. Field AE, et al: Snack food intake does not predict weight change among children and adolescents. Int J Obes Relat Metab Disord. 2004, 28 (10): 1210-6. 10.1038/oby.2004.94.

    Article  CAS  Google Scholar 

  23. Berkey CS, et al: Sugar-added beverages and adolescent weight change. Obes Res. 2004, 12 (5): 778-88. 10.1038/oby.2005.104.

    Article  Google Scholar 

  24. Taveras EM, et al: Family dinner and adolescent overweight. Obes Res. 2005, 13 (5): 900-6. 10.1006/pmed.1997.0200.

    Article  Google Scholar 

  25. Rockett HR, et al: Validation of a youth/adolescent food frequency questionnaire. Prev Med. 1997, 26 (6): 808-16. 10.1016/S0002-8223(95)00086-0.

    Article  CAS  Google Scholar 

  26. Rockett HR, Wolf AM, Colditz GA: Development and reproducibility of a food frequency questionnaire to assess diets of older children and adolescents. J Am Diet Assoc. 1995, 95 (3): 336-40. 10.1016/S0002-8223(95)00086-0.

    Article  CAS  Google Scholar 

  27. Trewin D: Socio-Economic Indexes for Areas 2039.0. 2001, Canberra: Australian Bureau of Statistics

    Google Scholar 

  28. Xyris Software (Australia) Pty Ltd: FoodWorks Professional Version 3.02.581. Brisbane. 2004

    Google Scholar 

  29. StataCorp: Stata Statistical Software: Release 8.2. 2003, College Station, TX: Stata Corporation LP

    Google Scholar 

  30. Kirkwood BR, Sterne JAC: Essential Medical Statistics. 2003, Oxford: Blackwell Science, Second

    Google Scholar 

  31. Willett WC, Stampfer MJ: Total energy intake: Implications for epidemiologic analyses. American Journal of Epidemiology. 1986, 124 (1): 17-27. 10.1079/NRR200370.

    CAS  Google Scholar 

  32. Cade J, et al: Development, validation and utilisation of food-frequency questionnaires - a review. Public Health Nutr. 2002, 5 (4): 567-87. 10.2307/2529310.

    Article  Google Scholar 

  33. Cade JE, et al: Food-frequency questionnaires: a review of their design, validation and utilisation. Nutrition Research Reviews. 2004, 17: 5-22. 10.1079/NRR200370.

    Article  CAS  Google Scholar 

  34. Ministry of Agriculture Fisheries and Food Project: Consensus document on the development, validation and utilisation of food frequency questionnaires. 2000, Nuffield Institute for Health, University of Leeds Institute of Human Nutrition, University of Southampton

    Google Scholar 

  35. Willett W: Correction for the effects of measurement error. Nutritional Epidemiology. Edited by: Willett W. 1998, Oxford University Press: Oxford

    Chapter  Google Scholar 

  36. Landis JR, Koch GG: The measurement of observer agreement for categorical data. Biometrics. 1977, 33 (1): 159-74. 10.1097/00001648-199603000-00016.

    Article  CAS  Google Scholar 

  37. Sim J, Wright CC: The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Phys Ther. 2005, 85 (3): 257-68. 10.1038/sj.ejcn.1602124.

    Google Scholar 

  38. Brenner H, Kliebsch U: Dependence of weighted kappa coefficients on the number of categories. Epidemiology. 1996, 7 (2): 199-202. 10.1079/BJN20051671.

    Article  CAS  Google Scholar 

  39. Vereecken CA, et al: Young adolescents' nutrition assessment on computer (YANA-C). Eur J Clin Nutr. 2005, 59 (5): 658-67. 10.1079/PHN2004604.

    Article  CAS  Google Scholar 

  40. Huybrechts I, et al: Validity and reproducibility of a semi-quantitative food-frequency questionnaire for estimating calcium intake in Belgian preschool children. Br J Nutr. 2006, 95 (4): 802-16. 10.1079/BJN20051671.

    Article  CAS  Google Scholar 

  41. Flood VM, et al: Issues in assessing the validity of nutrient data obtained from a food-frequency questionnaire: folate and vitamin B12 examples. Public Health Nutr. 2004, 7 (6): 751-6. 10.1191/096228099673819272.

    Article  Google Scholar 

  42. Bland JM, Altman DG: Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 1986, 1 (8476): 307-10.

    Article  CAS  Google Scholar 

  43. Gibson RS: Principles of Nutritional Assessment. 2005, Oxford: Oxford University Press, Second

    Google Scholar 

  44. Bland JM, Altman DG: Measuring agreement in method comparison studies. Statistical Methods in Medical Research. 1999, 8: 135-160. 10.1017/S1368980099000762.

    Article  CAS  Google Scholar 

  45. Tang HK: Diet, physical activity, environments and their relationship to the emergence of adolescent overweight and obesity in Ho Chi Minh City, Vietnam. PhD thesis. 2005, University of Newcastle: Faculty of Health

    Google Scholar 

  46. Andersen LF, et al: Evaluation of a quantitative food frequency questionnaire used in a group of Norwegian adolescents. Eur J Clin Nutr. 1995, 49 (8): 543-54. 10.1097/01.MPG.0000153004.53610.0E.

    CAS  Google Scholar 

  47. Robinson S, et al: Assessing the diet of adolescent girls in the UK. Public Health Nutr. 1999, 2 (4): 571-7. 10.1016/j.jada.2008.02.015.

    Article  CAS  Google Scholar 

  48. Bertoli S, et al: Validation of food frequency questionnaire for assessing dietary macronutrients and calcium intake in Italian children and adolescents. J Pediatr Gastroenterol Nutr. 2005, 40 (5): 555-60. 10.1016/j.jada.2004.02.031.

    Article  Google Scholar 

  49. Cullen KW, Watson K, Zakeri I: Relative reliability and validity of the Block Kids Questionnaire among youth aged 10 to 17 years. J Am Diet Assoc. 2008, 108 (5): 862-6. 10.1016/j.jada.2007.12.006.

    Article  CAS  Google Scholar 

  50. Jensen JK, et al: Development of a food frequency questionnaire to estimate calcium intake of Asian, Hispanic, and white youth. J Am Diet Assoc. 2004, 104 (5): 762-9. 10.1016/j.jada.2004.02.031.

    Article  Google Scholar 

  51. Wong SS, et al: Evaluation of a computerized food frequency questionnaire to estimate calcium intake of Asian, Hispanic, and non-Hispanic white youth. J Am Diet Assoc. 2008, 108 (3): 539-43. 10.1079/PHN2002344.

    Article  Google Scholar 

  52. Stein AD, et al: Consistency of the Willett semiquantitative food frequency questionnaire and 24-hour dietary recalls in estimating nutrient intakes of preschool children. Am J Epidemiol. 1992, 135 (6): 667-77.

    CAS  Google Scholar 

  53. Lietz G, et al: Can the EPIC food-frequency questionnaire be used in adolescent populations?. Public Health Nutr. 2002, 5 (6): 783-9. 10.1186/1471-2458-5-135.

    Article  Google Scholar 

  54. Block G, et al: Comparison of two dietary questionnaires validated against multiple dietary records collected during a 1-year period. J Am Diet Assoc. 1992, 92 (6): 686-93.

    CAS  Google Scholar 

  55. Baer HJ, et al: Use of a food frequency questionnaire in American Indian and Caucasian pregnant women: a validation study. BMC Public Health. 2005, 5: 135-10.1186/1471-2458-5-135.

    Article  Google Scholar 

  56. Nelson M, et al: Between- and within-subject variation in nutrient intake from infancy to old age: estimating the number of days required to rank dietary intakes with desired precision. Am J Clin Nutr. 1989, 50 (1): 155-67. 10.1038/sj.ejcn.1601588.

    CAS  Google Scholar 

  57. Willett WC, et al: Reproducibility and validity of a semiquantitative food frequency questionnaire. Am J Epidemiol. 1985, 122 (1): 51-65. 10.1590/S0102-311X2007000900026.

    CAS  Google Scholar 

  58. Slater B, et al: Validation of a semi-quantitative adolescent food frequency questionnaire applied at a public school in Sao Paulo, Brazil. Eur J Clin Nutr. 2003, 57 (5): 629-35. 10.1038/sj.ejcn.1601588.

    Article  CAS  Google Scholar 

  59. Marchioni DM, et al: Reproducibility of a food frequency questionnaire for adolescents. Cad Saude Publica. 2007, 23 (9): 2187-96. 10.1006/pmed.2000.0701.

    Article  Google Scholar 

  60. Rimm EB, et al: Reproducibility and validity of an expanded self-administered semiquantitative food frequency questionnaire among male health professionals. Am J Epidemiol. 1992, 135 (10): 1114-26.

    CAS  Google Scholar 

  61. Dennison BA, Jenkins PL, Rockwell HL: Development and validation of an instrument to assess child dietary fat intake. Prev Med. 2000, 31 (3): 214-24. 10.1006/pmed.2000.0701.

    Article  CAS  Google Scholar 

  62. Frank GC, et al: A food frequency questionnaire for adolescents: defining eating patterns. J Am Diet Assoc. 1992, 92 (3): 313-8.

    CAS  Google Scholar 

  63. Hunter DJ, et al: Variability in portion sizes of commonly consumed foods among a population of women in the United States. Am J Epidemiol. 1988, 127 (6): 1240-9. 10.1093/ije/dyg310.

    CAS  Google Scholar 

  64. Flegal KM: Evaluating epidemiologic evidence of the effects of food and nutrient exposures. Am J Clin Nutr. 1999, 69 (6): 1339S-1344S.

    CAS  Google Scholar 

  65. Plummer M, Kaaks R: Commentary: An OPEN assessment of dietary measurement errors. Int J Epidemiol. 2003, 32 (6): 1062-3. 10.1016/j.jada.2006.07.002.

    Article  Google Scholar 

  66. Jenner DA, et al: A comparison of methods of dietary assessment in Australian children aged 11-12 years. Eur J Clin Nutr. 1989, 43 (10): 663-73. 10.1002/sim.4780111005.

    CAS  Google Scholar 

  67. Subar AF, et al: The food propensity questionnaire: concept, development, and validation for use as a covariate in a model to estimate usual food intake. J Am Diet Assoc. 2006, 106 (10): 1556-63. 10.1016/j.jada.2006.07.002.

    Article  Google Scholar 

  68. Cole TJ, Green PJ: "Smoothing reference centile curves: the LMS method and penalized likelihood.". Stat Med. 1992, 11 (10): 1305-19. 10.1002/sim.4780111005.

    Article  CAS  Google Scholar 

  69. Cole TJ, Pan H: LMS Growth (computer program), Medical Research Council. 2002

    Google Scholar 

Download references

Acknowledgements

This research project was funded by the Hunter Medical Research Institute; however, the views expressed in this manuscript are those of the authors. The authors acknowledge the nutrition and dietetics students who assisted with data collection; G Berry and S Noble for assistance with data entry; the support of the parents and staff at participating schools; and the students who participated in the study.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Jane F Watson.

Additional information

Competing interests

A scannable version of the ACAES FFQ is currently under development and will be available for research application.

Authors' contributions

JW collected the data, performed the statistical analysis and drafted the manuscript. JW, CC, DS, MD and MG developed the study, and participated in its design and coordination. CC and DS provided assistance for the statistical analysis and helped to draft the manuscript. All authors read and approved the final manuscript.

Authors’ original submitted files for images

Rights and permissions

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article

Watson, J.F., Collins, C.E., Sibbritt, D.W. et al. Reproducibility and comparative validity of a food frequency questionnaire for Australian children and adolescents. Int J Behav Nutr Phys Act 6, 62 (2009). https://doi.org/10.1186/1479-5868-6-62

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/1479-5868-6-62

Keywords