Level 2 Evaluate Statistical Report Template

Title: Sunbed users in NZ

Source of statistically based report:

This is a NZ government survey from 2010, carried out by the Research and Evaluation Unit at HSC (Health Sponsorship Council). The survey is a 2 yearly Health and Lifestyle Survey (HLS).

Purpose of the report:

HSC wanted to find out more information about Sunbed use in NZ following information released by the World Health Organisation (WHO). Because it is a government institution (Crown entity) they are likely to be more open and neutral, as they are only seeking information, not selling a product.

Summary of the report - a one paragraph summary of the report, including the purpose of the report and identification of the population of interest.

This report finds out information regarding the use of sunbeds by New Zealanders 15 years and older. It looks into demographics (age, gender, ethnicity), sunburn, awareness of Vitamin D and the rate of sunbed use by New Zealanders.

The population of interest is New Zealanders 15 years and older.

They paid for and carried out this investigation as a result of reports from the World Health Organisation and Consumer New Zealand about the health concerns as a result of sunbed usage and the non-compliance of standards on sunbeds in NZ.

A description of measures and variables representation:

Whether or not used a sunbed in the last 12 months: Yes / No

Ethnicity: Asian, European, Other, Maori, Pacific

Skin types: Medium, Olive, Fair, Other

Age groups: 15 - 17, 15 - 24, 25 - 34, 55 and over, etc

Area that people live: Auckland, Urban etc.

Applied fake tan: Yes / No

Tried to get a suntan: Yes / No

Had a sunburn in the last 12 months: Yes / No

Severity of sunburn: Severe, Some degree, etc

Vitamin D report read in the last 6 months: Yes / No

Evaluation:

Remembering information over a 12 month period can be inaccurate. Additionally some people may not feel comfortable telling a stranger that they use sunbeds as they may feel judged by this. So this data may be inaccurate.

Collecting information from New Zealanders about their Ethnicity should be reasonably accurate, as this is a very common question for people to be asked and people can identify their family background and culture easily.

In terms of skin types, I am unsure of what the difference between Medium skin type and Olive skin type is. I am confident about identifying that Fair skin type is very pale. There is no information about how they collected data on skin type. For example, they may have asked people to select a category themselves. If this was the way they collected the data, then other people like myself may also find it difficult to know what category of skin type they are. However, if they gave people a colour chart and used this to categorise which skin type they have, then this would make the data more accurate and reliable.

People are more likely to feel comfortable stating what age group they are in than stating their actual age. So the fact that they were asked their age group means that the data is more accurate and reliable instead of the alternative which would be having missing data when people refuse to answer.

When people were asked if they had a sunburn in the last 12 months, and remembering how severe this was, relies on people's memories being accurate.Trying to remember events up to a year ago is unreliable and leads to inaccuracies in this data. It is likely that people will have underestimated the severity of the sunburn, as they may not feel confident speaking to a complete stranger about this issue.

A description of the sampling method(s):

The sampling method is not stated in the report, and I was unable to find it online.

The data was weighted to ensure that it is representative of New Zealanders.

Evaluation:

Not knowing whether a random or biased sampling method was used means that I cannot be sure that the data is representative. A biased method such as convenience or self-selected survey would not give representative data. However a random method such as simple random sampling, stratified or systematic would give data that was representative as each person in NZ would have an equal chance at being selected. This would lead to data that has a mixture of characteristics, such as age, gender, ethnicity, sunbed use, etc.

However, with the data having being weighted to ensure that it is representative, this is a good thing. That means that in terms of the balance of age, gender and ethnicity, the number of people in these categories will be roughly proportional to the population of NZ in 2010.

A description of the survey method(s):

Face-to-face in home survey was used.

Evaluation:

Advantages:

One advantage of a face-to-face survey is that people are more truthful, particularly about sensitive topics (such as health). This will lead to more accurate data. By going to people’s homes increases the reliability of the data also, as people are more likely to respond as they are in an environment that they feel very comfortable and even though health might be a sensitive subject, they may be more open and truthful in their answers.

The response rate for a face-to-face survey is usually higher than other methods such as paper and online surveys, as it is much harder to say no to someone to their face, compared with deleting an email.

Disadvantages:

The cost of doing face-to-face surveys is much higher than telephone calls, using the internet, or printing out paper surveys. This is because the company has to pay people to go to NZ’s homes, talk with them, and at times return to people’s homes if they were not home the first time they visited. As the time required to collect the data is much higher than other methods, this raises the costs quite a lot.

Geographical coverage is a disadvantage in this survey.The interviewers have to travel to people’s homes, which limits the coverage across New Zealand. For example, there will be some small towns that were not visited at all, which means that the data may not be representative of all NZ.

Interviewer bias may be present, as people may give answers that they think the interviewer wants to hear rather than the truth. For example, when asking about sensitive topics such as whether they use sunbeds, and an elderly man was asking the questions, I might not feel comfortable admitting this to the interviewer and respond that I didn’t use sunbeds when in fact I may have. This will reduce the accuracy and reliability of the data.

Accessibility bias may be a concern. If the interviewers are visiting home between 9am and 5pm, then there are several groups that will be over or under represented in the data. For example, during these hours, retired people, parents with very young children (below the age of 5), people who are ill or disabled are more likely to be home, and therefore these groups are likely to be over-represented. The opposite of this, is that business people, students and middle-aged people are likely to be at work or school, and these groups would then be under-represented in the data.

Improvement: If they collected data using a second method as well, that could increase the response rate. For example, if they were to also do a telephone survey in the evenings or weekends this would increase the number of people who they collected data from, and get a broader range of responses from other groups of people.

A description of the sampling and possible non-sampling errors:

Response rate = 57%

Non-response rate = 100% - 57% = 43%

Evaluation:

Non-sampling errors:

Non response error:

The response rate of 57% is quite high for a survey. Normally surveys only have a 20-30% response rate, so this means that they have information from 57% of the people they asked to do the survey. This is a reasonable coverage.

However, the non-response rate is 43%, which means that of the people who were asked to do the survey, we have no information on 43% of them.

This is a problem, as the people who did not respond may have different views and different data on sunbed usage compared with those who did respond. For example, if may be that people who didn’t respond are far more likely to use sunbeds and get sunburns than those that did respond.

Response error:

People were asked if they used a sunbed in the last 12 months, and if they had a sunburn in this time, and how severe it was if they did. This depends on people having very accurate memory recall over a 12 month period, which is a very long time. People’s memories are not likely to be accurate over this long period and therefore the data collected will have errors and be inaccurate. Moreover, it is likely that people underestimated the severity of their sunburn as they will be trying to look healthy in front of the interviewer.

Sampling error:

As soon as a sample is taken sampling error is present. It is only by doing a census and collecting information from the entire population that you can eliminate sampling error. However, as the sample size is reasonable (as discussed below), this minimises the sampling error.

A description of the sample size:

1740 New Zealanders aged 15 and over were sampled.

Breaking this down by ethnicity gives:

1067 European / Other

460 Maori

326 Pacific

124 Asian

Evaluation:

1740 people giving information is quite a lot. This will increase the reliability of the data, leading to more accurate estimates (such as the proportion of males and females who use sunbeds).

The sample sizes of Maori, Pacific and Asian ethnicity people is much smaller, meaning that we have less information about people of these ethnicities. With less data, this leads to lower reliability and accuracy.

One improvement that I would suggest if this survey was done again, would be to have bi/multilingual interviewers, so that people who didn’t have fluent English may feel more confident to answer the questions in their own language. This is likely to lead to a higher response rate, more representative data from a wider group of people, alongside increased reliability and accuracy of the data. This will then lead to more accurate estimates and mean that some ethnic groups are not under-represented.

An evaluation of the findings of the survey.

One finding in the report is that there is no difference between the percentage of males and females who use sunbeds. The percentage of males is 2.8% and females percentage is 2.7%. Given how close these percentages are, I agree with the finding that there is no difference between males and females. However, this surprised me, as I would have expected that there would be a higher percentage of females using sunbeds than males. The reason I expected this is that I know many females are very conscious of how they look, particularly at special events such as weddings, 21st birthdays, etc. This makes me wonder if females felt less comfortable admitting to the interviewers that they use sunbeds as they may have felt that they would be judged as “unhealthy” for admitting this.

Much of the findings of the survey have been presented in sentences. I would recommend that an improvement would be to create an infographic with much of this information shown. This makes the information more accessible to more people, and more visually appealing. This will be likely to increase the number of people who would read the report and learn from the findings.

An overall evaluation of the effectiveness of the statistically based report.

Putting it all together. Key ideas:

• Observational study - therefore inferential/suggestive language should be used.
• Response rate is 57%
• Sample size is 1740
• Survey method is face-to-face in the home
• Sampling method is unknown
• Author: NZ government organisation HSC
• Purpose: collecting information to inform NZ’s

Evaluation:

Overall, I think this is a reasonable report, leading me to have a reasonable level of confidence and trust in its findings. There are several reasons for this. The reasonably high response rate of 57%, along with the sample size of 1740 and data collected using face-to-face interviews all increase the reliability and validity of the data, along with the findings. Because the research was paid and carried out by a government organisation, this gives me confidence in their neutrality as they don’t have a hidden purpose such as trying to sell me a product.

However, not knowing the sampling method, along with the smaller sample sizes in some ethnic groups concerns me. Some data also requires people to have accurate memories over a 12 month period of time, which makes the data less reliable. Finally, as the report was collected and produced in 2010, this is nearly 10 years ago. Some of the information will no longer be relevant to today, as there will be differences in the technology used in sunbeds, along with better awareness in the population of being sunsmart and protecting ourselves from sunburn.

Therefore I feel that there is some useful findings in this report, and I have some confidence in the reliability and accuracy of the data, but this is only an observational study and a sample, therefore their findings can only be suggestive, not causative.