SOST60421 Survey Research Methods – Assessment Instructions
Assessment Part 2 – Individual Essay
Task: Write a structured essay about how you would construct a survey on a specific topic (different
topic to the group work assessment). Explain the entire survey design process and its different steps.
Instructions for essay:
Find a research question which can be answered/studied with the help of a survey (different
from group work topic)
Find a matching target population (no limitations)
Outline how you would design a survey in order to answer your research question(s)
Assume reasonable/realistic limitations on (1) time, (2) budget and (3) other resources
Explain your choices for survey type, data collection methods and sampling strategy
Discuss potential ethical issues regarding the research question, target population and data
collection method
Discuss the advantages and disadvantages of your choices and possible alternativesin relation
to your topic and target population
Explain what kind of practical difficulties you might encounter in your surveys and how you
would attempt to avoid them beforehand or deal with them in post-survey processing
Make references to methodological literature where appropriate (not lecture slides)
Present up to 3 key questions, you would include in order to answer your research question
and evaluate their format and design
Assessment criteria:
How well do the research question(s) and the target population match (15%)
What ethical issues should be considered given the chosen research topic and the survey
population (15%)
How well do the sampling strategy and data collection methods match the target population
(15%) and how well are the strengths and weaknesses of alternative methods discussed (15%)
Which practical difficulties are considered (10%) and what design choices are explained to
avoid them (15%)
How well are the 3 key questions designed to answer the research question (15%)
Formatting Specifications:
Essay shouldn’t exceed 2500 words
Double-spaced
Font size 12 point in Times New Roman or Arial
Use the Harvard system of referencing
The assignment is about how you would (hypothetically) conduct a survey and to demonstrate your
understanding of the whole survey design process. Please DO NOT CONDUCT ANY SURVEY OR
COLLECT DATA of any kind.
Submission date: 16th January 2023, 2pm
Contribution to total mark: 80%
—
伴侣下单表
为了论文的准确度，我们强烈强烈强烈建议您附件中提供以下内容：
• 请把该论文相关的课程课件提供给我们（请尽可能提供准确的课件），以便写手根据所学知识进行进一步运用，英文填写下单表。
• 请提供该论文或作业的评分标准（Criteria）
• 请提供范文（Model paper）（尽量提供）
订单编号（客服提供）Order numbeir:
BL-23010501
作业字数/PPT页数Number of Words/Number of slides:
2500
需要的日期（请以当地时间为准，注明地区，例如：美国，英国）Paper Urgency:
2023/01/10 UK
作业种类 (论文，报告，演讲稿，润色)Type of homework:
ESSAY
学校/专业/论文科目university/professional/Subject Area:
University of Manchester/Social research methods and statistics/ Survey Research Methods
引文类型(APA，HARVARD，MLT…没有特殊要求可不写）Paper Style:
HARVARD
引用数量（常规为每千字5个,如需添加需额外收费）The number of References:
12/13
论文具体说明.Instructions（请写下您需要写手注意的需求，文档类资料与本订单一起发送到客服邮箱）:
I chose “The correlation between Chinese students’ socializing and drinking in the UK” as the research question. It’s just a thought. If you have something better, please ignore it.
Apply the concepts mentioned in the powerpoint as much as possible
More critical analysis
Low repetition rate
下单说明：
1.本论文仅用于学术交流，不得用于学术成绩获得，交易结束前论文的版权归我司所有，交易结束后，论文所有知识产权都为客户所有，版权移交后我司不为论文的用途负责；
2.强烈建议客户尽量用英文提出写作要求，如果写作要求用中文提出，我们将安排专业英文翻译免费进行翻译（我们将对翻译进行高标准严要求，争取达到信达雅，在认知相同的情况下，尽力与客户要求一致）；
3.我们的写手均毕业于牛剑、MIT、伯克利、帝国理工、UCL等顶级大学，作业写作只是他们研究和工作外创收的一部分，因此：
（1）我们建议您尽量只把与作业相关的材料发给我们，如果材料过多（多于千字2.5个），由于写手时间极为宝贵，可能造成少读或跳读（如果您材料过多，无法筛选，请和我们进一步联系，我们尽力为您和写手沟通）；
（2）写手将免费为客户提供多轮次的常规修改服务以满足教授要求，但如客户有和原始订单较大偏差的改动，我们将免费为客户和写手沟通，争取在最少的费用下完成作业；
4.我们希望圆满完成您的作业，在下单到约定交稿时间的前一半，我们免费接收您添加材料；超过一半时间添加材料，请和客服联系，我们会为您酌情处理；
5.我们是学术服务的搬运工，写手高昂的费用在下单后都是我们提前为客户垫付的，在交付稿件后请您交付尾款，并通知客服（经沟通逾期未交付，我们将适度收取滞纳金）；
SURVEY RESEARCH METHODS
SOST60421 2022-23
Acquiesce
2
Overview
3
Session Date Topic Lecturer
1 27th Sep Introduction Tina
2 4th Oct Survey Inference and Errors Tina
3 11th Oct Questions and Answers Kingsley
4 18th Oct Questionnaire Development Kingsley
5 25th Oct Pretesting Kingsley
1st Nov Reading Week
6 8th Nov Mode of Data Collection Tina
7 15th Nov Computer-Helped Data Collection Kingsley
8 22nd Nov Interviewing Tina
9 29th Nov Nonresponse Tina
10 6th Dec Sampling Kingsley
11 13th Dec Post-Survey Processing; Estimation Tina
Survey error
Bias
Types of non response
Practical exploratory work
4 Reminder from last week…….
Week 3 Survey Questions and Answers
6 Recent Survey Example…..
This Week’s Main Issues
7
Cognitive processes in answering questions
Problems in answering survey questions
Surveys – from a design perspective
8
Measurement
Construct
Response
Target Population
Respondents
Sampling Frame
Sample
Post survey Adjustments
Survey Statistic
Edited Response
Measurement Representation
Source: Groves et al. (2009)
Understanding How Respondents
Answer Questions
9
¨ Knowing how Respondents produce answers, e.g. their mental
processes, helps us understand where things can go wrong
(we try to reduce error)
¨ Writing better questions can be supported, by understanding the
response process
¨ Can help explain “response effects” in existing data
¤ E.g. question order effects
Types of questions
10
¨ Behaviours and facts vs. attitudes
¤ Have you ever been a regular cigarette smoker, that is, at least one
pack a week?
¤ To reduce cigarette smoking some countries have increased tax on the
purchase of cigarettes. Do you support or oppose this kind of tax?
OK lets think about asking questions…
11
¨ How happy are you?
OK lets think about asking questions…
12
¨ How happy are you?
¨ Make a note of the process your brain went through
What did you first think about?
13
¨ Sensitive (or threatening) vs. non-sensitive topics
How many pets do you have?
How much money to do your parents earn?
How many sexual partners have you had in your life?
¨ Closed vs. open questions
What political party do you identify with: Republican party,
Democratic party, Independent or something else?
Why do you identify with that party?
Types of Questions
Survey Example From Brazilian Election Polling
14
https://www.washingtonpost.com/world/2022/10/03/bolsonaro-brazil-polls-trump/
https://www.bloomberg.com/news/articles/2022-10-03/pollsters-fail-to-capture-bolsonaro-support-again-in-brazil-vote
Simple Model of the Response Process
15
Comprehension
of the question
Retrieval of
information
Judgement
and estimation
Reporting
an answer
Simple Model of the Response Process
16
¨ Sequential although respondents can backtrack
¨ Models ideal performance, but:
¤ Respondents may misunderstand question
¤ Event(s) may not be recorded (“encoded”) in Respondent’s memory
¤ Respondents may forget relevant events: recall error
¤ Respondents may take shortcuts: satisficing, acquiescence
¤ Respondents may intentionally misreport: social desirability
Comprehension
of the question
Retrieval of
information
Judgement
and estimation
Reporting
an answer
Comprehension Processes
17
¨ Comprehension usually assumed to involve analysis at various
levels:
¤ sensory: segment speech stream into words, recognize printed
characters
¤ lexical: retrieve meaning(s), pronunciation, part of speech, etc.
¤ syntactic: string words into a grammatical sequence
¤ semantic: determine overall sentence (question) meaning
¤ pragmatic: infer speakers’ intentions
Retrieval Processes
18
¨ Consensus view is that respondents search their long-term
memories on basis of words in questions or “retrieval cues”
¨ Successful extension of the cue in question, if it matches indexes,
with which the event is stored in memory
Retrieval Processes – Cues
19
¨ Cues can affect what people recall:
¤ Couper, Tourangeau & Kenyon (2004) asked questions like “How many
sporting events have you attended in the past year?” and presented
images (in web survey)
¤ Images were either low frequency (Premiership) or high frequency
(local game)
¤ Low frequency images led to lower reports than high frequency
images
20
¨ Cues can remind people of events:
¤ National Crime and Victimisation Survey (NCVS) item for shopping
includes: “drugs, clothing, grocery, hardware and convenience stores”
¨ In general, exact times and dates are poor cues
Retrieval Processes – Cues
Estimation and Judgment Processes
21
¨ In the past two years, how many times did you donate blood?
¨ For someone who donates regularly in the same workplace
location, there is little to distinguish one donation from the next so
it’s hard to recall and count up each one
¨ How much alcohol did you drink last week?
Reporting An Answer
22
¨ People have to choose available options for answers
¤ when response options are read out loud, people show preference for
the last one read (“recency effect”) possibly due to limits of working
memory capacity
n Shuman & Presser (1981) study on housing, older respondents showed
recency effects while younger respondents showed little evidence of the
effect
¤ when response options self-administered, people show a preference
for the first one read (“primacy effect”)
n Krosnick and Alwin (1987) interpret as evidence of “survey satisficing” –
choosing the first acceptable option as a time-saving behavior
Let’s go back to the happiness question….
23
How happy are you?
What process did you notice your brain going through?
Did you think of today, recently or in general?
The Cognitive Process……
24
¨ Developing a survey involves defining and translating concepts and issues
into a form that is measurable (De Vaus, 2013).
¨ Respondents engage in four cognitive operations when assessing survey
items (comprehension, recall, judgement and response) (Tourangeu 1984).
¨ Holbrook et al. (2003: 82) state that ‘a respondent must interpret the
meaning and intent of each question, retrieve all relevant information from
memory, integrate that information into a summary judgment, and report
that judgment accurately’. Many social surveys are conducted by an
interviewer and so there is a dialogical nature to survey questions.
Discourse Framing
25
¨ How is the question framed?
¨ Example – Question about attitudes towards immigration.
¨ Try and write a question on this subject.
ESS Example Question
26
ESS Example Question – Critique
27
Problems in Answering Survey Questions
28
Types of problem that can cause errors:
1. Failure to encode the information sought
2. Misinterpretation of the question
3. Forgetting and other memory problems
4. Inaccurate or unwarranted judgment/estimation strategies
5. Problems in formatting an answer
6. More or less deliberate misreporting
7. Failure to follow instructions
Encoding
29
¨ Some events are less likely to be encoded in memory than others
¨ Lee et al. (1999) demonstrated that parents’ poor memory for
children’s vaccinations due to not encoding the events
¤ Assessed parents’ recall accuracy by comparing reports to medical
records
¤ Parents reported that vaccinations were up-to-date for 80% of those
children who were not up-to-date
¤ Childhood injections are frequent, not particularly distinctive and occur
in batches
¤ Parents may simply have not encoded enough to accurately recall and
report the events
Misinterpretation
30
Types of comprehension problems
¤ Ambiguity and Conceptual Misalignment
¤ Excessive complexity
¤ Faulty presupposition
¤ Vague concepts
¤ Vague quantifiers
¤ Unfamiliar terms
¤ False inferences
Comprehension Problems:
Two kinds of ambiguity
31
(1) Lexical ambiguity:
Q: “The best way to prevent cancer is to catch it early!”
A: strongly agree; somewhat agree; somewhat disagree; strongly
disagree
¨ Unclear which sense of “catch” is intended
¤ word label corresponds to more than one meaning
32
(2) Referential ambiguity
Interviewer: Last week, did you have more than one job including
part-time, evening or weekend work?
¨ Unclear how to apply “more than one job” to one’s circumstances
¤ which circumstances are included and which are not?
Comprehension Problems:
Two kinds of ambiguity
Two kinds of solutions
33
¨ First case (Lexical ambiguity)
¤ pretesting, e.g. “cognitive interviewing,” should uncover problems
¤ can be improved with less ambiguous question wording
n e.g. replace “catch” with “detect”
¨ Second case (Referential ambiguity )
¤ pretesting cannot anticipate all borderline circumstances, especially if
there are many
¤ question cannot be worded to address all of them
¤ can be improved with clarification during interview
n e.g. “in this survey, we count that as one job”
Try and write a question yourself….
34
¨ Subject: the impact of children’s exposure to violence on
TV
35
Do you think children suffer any ill effects from watching programmes
with violence in them?
¨ Belson (1981) determined that respondents interpreted children,
ill effects, and violence in numerous ways
¤ e.g. “children”: < 8 years, < 19 – 20 years
¤ children as students
¤ only 8% interpreted question as intended
¨ Additional words can clarify the intended meaning, but this may
lead to unwieldy, convoluted questions
¤ tradeoff between clarity and complexity
Comprehension Problems:
Conceptual Variability
How complex can you get?
36
Example:
Do you think that children suffer any ill effects from watching TV
with violence in them, other than ordinary Westerns? By children, I
mean people under age 14; by ill effects, I mean increased
aggression at school or at home, increased nightmares, inability to
concentrate on routine chores, and so on; by violence, I mean
graphic depictions of individuals inflicting physical injuries on
themselves or others, depictions of individuals want only damaging
property or possessions, abusive behaviors or language to others,
and so on.
Amended question
37
¨ Trying to strike a balance between complexity and clarity,
perhaps it could be better to ask openly about the ill effects and
then code the spontaneous answers:
What ill effects do you think children (under 14) suffer from
watching TV with graphic depictions of violence?
¨ A pilot questionnaire should be used to uncover any problems
Comprehension Problems:
Presupposition
38
¨ Some sentences (or questions) refer to what has previously been
said or implied
¨ What if presupposition is not accepted by respondent?
Family life often suffers because parents concentrate too much on their
work. Do you strongly agree, somewhat agree, etc.?
¨ “Don’t know” option does not solve the problem because can imply
acceptance of presupposition
Comprehension Problems:
Vague quantifiers
39
¨ Non-numerical terms for quantity have different numerical
interpretations
¤ What does a few years mean?
Comprehension Problems:
Vague quantifiers
40
Belson (1981) found that the quantifier “few” in “over the last few years”
meant:
¤ “no more than two years” (7/59)
¤ “seven or more years” (19/59)
¤ “ten or more” (11/59)
41
¨ Particularly problematic in response options
How often do you feel really excited? Very often, pretty often, not too
often or never?
¤ If respondents says more than very often, Schaeffer & Bradburn
asked for number
¤ For those with higher level qualifications and younger respondents,
“pretty often” and “very often” were associated with larger numbers
Comprehension Problems:
Vague quantifiers
Forgetting and other memory problems
42
¨ Different forms of memory failure:
1. Mismatch between terms in question and terms used to encode
events initially
2. Distortions in the representation over time
3. Retrieval Failure
4. Reconstruction Errors
Memory problems: Mismatch
43
“How often do you do light or moderate activities for at least 10
minutes that causes only light sweating or a slight to moderate
increase in breathing or heart rate?” (NHIS)
¨ If respondent did not include walking as “light to moderate
activity”……. Under reporting?
Memory problems: Distortion over time
44
¨ Hard to distinguish whether information was actually experienced
or added through retelling of or thinking about event afterward
45
¨ Even inferred aspects of events are hard to distinguish from
actual aspects of events:
¤ Experimental participants watched film of traffic incident
¤ “How fast was the car going when it went through the stop sign”
¤ Led to reports of stop sign in original traffic event on a subsequent
memory test even when one was not present (Loftus, 1979)
Memory problems: Distortion over time
Memory Problems: Retrieval failure
46
¨ Interference
¤ The longer the time period in question (e.g. 1 year vs. 1 month) the
more likely other similar events will have occurred
¤ Hard to distinguish details of one event from others
¤ Tend to blend into single generic memory
¨ Decay
¤ The more time since the events, the weaker the memory
¤ Forgetting is most rapidly in period immediately after event
¤ However, forgetting continues after as many as 50 years(!)
Estimation Problems
47
“Now think about the past 12 months, from [DATE] through today. We
want to know how many days you’ve used any prescription tranquilizer that
was not prescribed to you or that you took only for the experience or the
feeling it caused during the past 12 months.” (NSDUH)
¨ Telescoping: Events in the past seem closer to present than they
actually are
Estimation Problems:
Behavioral Frequency Questions
48
¨ At least three broad strategies could be taken by respondents,
each leading to different types of error
¤ recall and count: underestimation
n more likely to forget than invent an event
¤ rate-based estimation: overestimation
n for regularly occurring events, rates can be quite accurate except when
respondents fail to take exceptions (e.g., not doing the behavior) into account
¤ impression-based estimation: overestimation
n translation of impression to number cannot be any lower than 0 but is
unbounded on the high end
Judgment Problems: Opinion Questions
49
“Now turning to business conditions in the country as a whole — do you think that during the next 12 months we’ll have good
times financially, or bad times, or what?” (SCA)
Assuming Respondent has not already formed an opinion, set of
strategies similar to those used for factual questions
¤ vague impression
¤ construct Assessment (count and compare specific beliefs about the issues)
Harder (or maybe impossible) to assess the truth of attitude than
behavioural reports
50
¨ Problems with Open Numerical format
“Now thinking about your physical health, which includes illness and injury,
for how many days during the past 30 days was your physical health not
good?” (BRFSS)
¤ May be hard to convert vague impression into number
¤ Rounded numbers may indicate difficulty with conversion or
unwillingness to be precise because truthful response is
embarrassing/fear of consequences
Formatting Problems
Formatting Problems:
Ordered Response Scales
51
¨ Problems with Ordered Response scales
¨ Bias:
¤ Respondents tend to endorse more positive than negative values
¤ Schwarz, et al. (1991) suggested more extreme values with numerical
labels lead to more negative interpretation of low-end verbal labels
Positivity Bias
52
¨ Schwarz, et al. (1991):
“How successful would you say you have been in life?”
¤ 34% of 0 to 10 group responded in lower half (0 to 5)
¤ 13% of -5 to +5 group responded in lower half (-5 to 0)
¤ zero scale anchor combined with label to mean “absence of success”
¤ negative scale values combined with label to mean “presence of failure”
Other Problems….
53
6. Deliberate misreporting
7. Failure to follow instructions
54 Recent Government Survey….
Week 3 – Summary
55
¨ Developing a survey involves defining and translating concepts and issues
into a form that is measurable (De Vaus, 2013).
¨ A number of factors may affect the answers given by survey respondents
including: interviewer effects (relating to the interaction between
respondent and interviewer); the wording and ordering of the questions
and response options; acquiescence bias; primacy or recency effects and
response fatigue (Bryman, 2018; Lavrakas, 2008).
¨ Surveys are political and can be conducted for political purposes.
Week 3
56
Any questions?
Readings For Week 4
57
Main
Groves, R.M. et al. (2009). Chapter 7
Also try the journal: International Journal of Social Research Methodology
https://www.tandfonline.com/journals/tsrm20
Recommended:
¤ Bradburn, N., Rips, L, and Shevell, S. (1987). Answering autobiographical questions: The impact of memory and inferences on surveys. Science, 236, 157-161.
¤ Bryman, A. (2018) Social Research Methods. Oxford: Oxford University Press.
¤ De Vaus, D. (2013) Surveys in Social Research. 6th ed. London: Routledge.
¤ Converse, J., and Presser, S. (1986) Survey Questions: Handcrafting the Standardized Questionnaire, Newbury Park: Sage.
¤ Holbrook AL, Green MC and Krosnick JA (2003) Telephone versus Face-to-Face Interviewing. Public Opinion Quarterly, 67(1): 79-125.
¤ Tourangeau, R., Rips, L., and Rasinski, K. (2000). The Psychology of Survey Response. New York: Cambridge University Press. Chapter 1.
¤ Tourangeau, R. (1984) Cognitive science and survey methods: A cognitive perspective. In T. Jabine, M. Straf, J. Tanur, & R. Tourangeau (Eds.), Cognitive aspects of survey
methodology: (pp. 73–100). Washington, DC: National Academy Press.
¤ Tourangeau, R. (2017) The Survey Response Process from a Cognitive Viewpoint Improving the Quality of Data Collection in Large Scale Assessments.
58 Quiz Question
Do you support with the Monetary Control Bill?
59 Practical week 3
Practical Tasks
60
Last Week – Find your specific research question(s)
¤ In your broad topic:
n What is the most interesting aspect?
n Which specific question(s) can be answered with your survey?
n What existing surveys/questions exist?
This Week – Identify survey questions and necessary
background information that are needed to answer your
research questions.
Take notes on your ideas and discussion and post it on
BB.
