Hide menu

Research quality in empirical studies with human participants


Status Active - open for registrations
School IDA-gemensam (IDA)
Division COIN
Owner Nils Dahlbäck

  Log in  

Course plan

No of lectures

Eight lectures/seminars

Recommended for

Ph D and master's students who do studies with human participants as part of their thesis work, or anyone that is interested in quality aspects other than those usually adressed in courses in research methods

The course was last given

This is the first time


To be able to reflect on how seemingly external factors and "irrelevant" influence the results and the generalizability of the obtained results of a study.


No formal requirements but some familiarity with quantitative and/or qualitative research methods is probably necessary.


A series of combined lectures and seminars, where the first part is primarily devoted to discusing the texts for the seminar, and the first time is devoted on how the arguments apply or not apply to the partiticipants' own research


Factors that often are not discussed inpublished papers but which are known (or suggested) to influence the participants be3havior and/or the validity of the generalizations from the obtained results.


Curent and classical papers on quality aspects of empirical research. P
reliminary literature list:

Dahlbäck (2003) If cognitive science is multidisciplinary, which are the disciplines
Dahlbäck (2016) Tvärvetenskapens kvalitetsproblem
Orne (1962) On the social psychology of the psychological experiment With particular reference to demand characteristics and their implications. American Psychologist, 17(11), 776–783
Rosenthal (1967) Covert communication in the psychological experiment. Psychological Bulletin, 1967, Vol 67, No. 5, 356-367
• Stroebe et al (2012) Scientific misconduct and the Myth of self-correction in science
Nosek et al (2012) Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability
Ioannidis (2012) Why science is not necessarily self-correcting
Nils Dahlbäck and Arvid Karsvall, Personality Bias in Volunteer Based User Studies? Proceedings of HCI 2000
Heinrich et el (2010) The Weirdest People in the World? Behavioral and Brain Sciences
Michael Buhrmester, Tracy Kwang, and Samuel D. Gosling (2011) Amazon’s Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data? Perspectives on Psychological Science 6(1) 3–5
Gabriele Paolacci1 and Jesse Chandler (2014) Inside the Turk: Understanding Mechanical Turk as a Participant Pool
Clark, H. (1973) The language as fixed-effect fallacy
Ioannidis, J. (2005) Why most published research findings are false
Yarkoni, T (2022) The Generalizability Crisis. Behavioral and Brain Sciences


1. Different explicit or implcit quality criteria in different scientific fields.
2. Demand characteristics of the experimental situation
3. Why self-correction does not work as it should i today's research
4. Problems with non-statistical generalizations I
5. Problems with non-statistical generalizations II
6. Problems with statistical generalizations
8. The replication crisis
9. The generalization crisis.
10. Course summary and evaluation


Two questions submitted to each seminar

A written ezam


Nils Dahlbäck & Fredrik Stjernberg (IKOS)


6 p


We are a bit uncertain on how many credit points the students should receive and would appreciate comments on this before making this final.

We plan to build a home page for the course that we aim to have ready before the start of the fall semester, or earlier if deemed necessary

Page responsible: Director of Graduate Studies