Research Quality in Empirical Research with Human
Participants
Fall semester
2023
The course starts on September 7 at 1:15 p.m. in seminar room
Allen Newell (IDA)
The schedule will be decided at the first seminar to reduce
the risk of schedule conflicts for the participants. We will initially see each
other once a week for 8-10 weeks, preliminary on Thursdays at 13:15.
We expect each seminar to be approximately one hour, but this
may vary depending on the topic.
Nils Dahlbäck (IDA)
nils.dahlback@liu.s
https://www.ida.liu.se/~nilda08/e
Fredrik Stjernberg (IKOS)
fredrik.stjernberg@liu.se
The course is primarily aimed at doctoral students and
teachers who are doing empirical research, and primarily for those who do
studies with people as informants/subjects. This includes not only basic
psychological research, but also all forms of applied cognitive research such as
e.g. human-computer interaction, human-machine interaction, etc., and not only
controlled experimental lab studies but also qualitative research approaches
such as anthropological field studies and quasi-experimental studies in more or
less natural situations.
The course addresses and highlights factors that affect the
quality of the research and the ability to draw conclusions from the
observations obtained, but which are rarely or never addressed in standard
methods courses.
The objective of the course is not to establish what is
"right" in any general sense, but instead to become aware of which problem
dimensions that exist and to reflect on which approaches exist in different
situations and how that knowledge can be used in one's own research. In this
way, it is more like a research ethics course than a regular methods course.
The course literature consists of both classic and current
articles on various aspects of research quality, where an important goal is to
give the participants the opportunity to reflect on quality aspects also in
research fields and methods other than their own, and to what extent they are
applicable to their own research. Additional literature may be added during the
course.
After an introductory meeting, the course consists of a
series of seminars where in the first part we discuss the articles relevant to
the seminar, and in the second part we discuss to what extent the points of view
presented are applicable to our own research.
The seminars take place on site at IDA.
To pass the course, 80% participation in the seminars is
required, including approved proposals for discussion questions for the seminars
(se below on examination).
The course requires basic knowledge in experimental and
non-experimental research with human participants, corresponding approximately
to an introductory course at bachelor's level. Own experience of empirical
research of this kind is desirable.
Interested students who do not fulfil these requirements can
apply to participate in parts of the course that treat more general
issues in empirical research. Applications will be decided on individual cases
after discussion with the course instructors.
The examination consists of two parts.
1. Before each seminar, each participant must submit two
discussion questions about the seminar's articles. These shall be
sent to the course teachers no
later than 12 noon the day before the seminar.
2. A concluding essay addressing two questions.
1. Which
parts of the course have been most and least relevant to your own research and
why you think that is the case.
2. Examples
from your own research area of how the quality of the research could be
increased based on what they learned on the course.
This is a preliminary outline of the seminars but please note
that this can and probably will be modified during the course depending on the
interests of the participants.
1. Introduction
• An
overview of common (but rarely discussed) problems in empirical research
• Why
it is important to know the strengths and problems with other methods than the
ones you work with yourself
• Differences
in quality criteria in different research fields
• Readings:
o Dahlbäck
(2003) If cognitive science is multidisciplinary, which are the
disciplines
o Dahlbäck
(2016) Tvärvetenskapens kvalitetsproblem
2. Why
self-correcting does not always work in science
· Stroebe
et al (2012) Scientific misconduct and the Myth of self-correction in science
· Nosek
et al (2012) Scientific utopia: II. Restructuring incentives and practices to
promote truth over publishability
· Ioannidis
(2012) Why science is not necessarily self-correcting
3. Demand
characteristics – How the experimental setting differs
from the situations one wants to generalize the results to
· Orne Demand characteristics
· Clark Social
Actions, Social Commitments
· (For those who are not familiar with it, we also recommend Milgram, Stanley (1963). "Behavioral Study of Obedience". Journal of Abnormal and Social Psychology. 67 (4): 371–8, or the Wikipedia entry)
4. Non-statistical genralization problems
(this might be two seminars)
· The
participants differ from the intended generalization population
i. Rosenthal
ii. Smart
iii. Karsvall & Dahlbäck
iv. WEIRD
v. Mechanical Turk
1. Buhrmester et al
2. Paolacci & Chandler
5. Statistical generalization problems
· The
materials used are also a sample from a population (random effects v. fixed
effects)
i. Clark
(1973) The language as fixed-effect fallacy, Journal of Verbal
Learning and Verbal Behavior, 12, 335-359
iii. Clark,
H.H., Cohen, J., Smith, J.E.K., & Keppel, G. (1976) Discussion of Wike &
Church’s comments, Journal of Verbal Learning and Verbal
Behavior, 15, 257-266
6. The
Replication crisis
· Ioannidis,
J. (2005) Why most published research findings are false
· Ioannidis video
från YouTube
https://www.youtube.com/watch?v=GPYzY9I78CI
· One
or two follow-up papers
7. The Geralization Crisis
· Yarkoni,
T (2022) The Generalizability Crisis. Behavioral and Brain
Sciences
· Selection
of comments and Yarkoni’s response
8. Concluding
seminar (details will be provided later)
Please note that this literature list is incomplete and
will be updated before the course starts and probably also during the course.
Most of the papers can be downloaded from the university
library. The papers with Nils Dahlbäck as one of the authors can also be
downloaded from https://www.ida.liu.se/~nilda08/publications.html or
by the link provided in the list below.
If you have problems with downloading any paper, please
contact Nils Dahlbäck for a copy.
· Buhrmester,
Tracy Kwang, and Samuel D. Gosling (2011) Amazon’s Mechanical Turk: A New Source
of Inexpensive, Yet High-Quality, Data? Perspectives on
Psychological Science 6(1) 3–5
· Clark,
H. (1973) The language as fixed-effect fallacy, Journal of
Verbal Learning and Verbal Behavior, 12, 335-359
· Clark,
H.H., Cohen, J., Smith, J.E.K., & Keppel, G. (1976) Discussion of Wike &
Church’s comments, Journal of Verbal Learning and Verbal
Behavior, 15, 257-266
· Clark,
H. (2006) Social actions, social commitments. In Enfield and Levinson (eds.) Roots
of Human Sociality Oxford: Berg Publishers
· Dahlbäck
(2003) If cognitive science is multidisciplinary, which are the disciplines:
Cognitive Science as Three Methodological Cultures. In: Proceedings
of the European Conference on Cognitive Science (EuroCogSci'03), September
10-12, 2003, Osnabruck, Germany.
· Dahlbäck
(2016) Tvärvetenskapens kvalitetsproblem
https://www.ida.liu.se/~nilda08/papers/Tvärvetenskapens%20kvalitetsproblem%20(4).pdf
· Dahlbäck
and Karsvall, Personality Bias in Volunteer Based User Studies? Proceedings
of HCI 2000 In: Proceedings of HCI 2000
· Heinrich
et el (2010) The Weirdest
People in the World? Behavioral and Brain Sciences
· Ioannidis,
J. (2005) Why most published research findings are false
· Ioannidis
(2012) Why science is not necessarily self-correcting
· Nosek
et al (2012) Scientific utopia: II. Restructuring incentives and practices to
promote truth over publishability
· Orne
(1962) On the social psychology of the psychological experiment with particular
reference to demand characteristics and their implications. American
Psychologist, 17(11), 776–783
· Paolacci and
Chandler (2014) Inside the Turk: Understanding Mechanical Turk as a Participant
Pool
· Rosenthal
(1967) Covert communication in the psychological experiment. Psychological
Bulletin, 1967, Vol 67, No. 5, 356-367
· Stroebe
et al (2012) Scientific misconduct and the Myth of self-correction in science
· Wike,
E.L. & Church, J.D. (1976) Comments on Clark’s “The language-as-fixed-effect
fallacy”. Journal of Verbal Learning and Verbal Behavior,
15, 249-256.
· Yarkoni,
T (2022) The Generalizability Crisis. Behavioral and Brain
Sciences
More papers can be added during the course.