Tackle advanced challenges in experimental design by analyzing blinding procedures, operationalization decisions, ecological validity, randomization failures, and the replication crisis through detailed real-world research scenarios. You will build the ability to spot subtle methodological weaknesses that can invalidate even well-intentioned, well-funded studies and to evaluate whether a study's conclusions actually follow from its design.
Experimental design is the craft of arranging an investigation so that the conclusion is earned by the data, not smuggled in some other way. The advanced skill isn't memorizing study types. It's reasoning about which threats to validity a given design controls and which it doesn't, and what kind of conclusion the design can actually support. This exercise drills the moves methodologists use to match a claim's confidence to a design's strength.
Blinding, operationalization, ecological validity, and randomization failures each determine whether a study can support its conclusion. This exercise trains you to spot design flaws that invalidate results, so you can judge whether a finding measures what it claims rather than something correlated with it.
Background
The randomized controlled trial is the gold standard because random assignment severs the tie between treatment and any hidden factor. But trials aren't always possible or ideal, so researchers approximate with quasi-experiments, comparing before and after, exploiting a policy change or a natural disaster as an accidental randomizer. Each design has specific strengths and specific blind spots, and the skill is knowing which is which.
A few advanced traps: analyze people in the group they were assigned to, even if they dropped out, or you sneak selection bias back in. Subgroup findings are usually underpowered and riddled with multiple-comparison problems. And a claim about a mechanism ('X works because of pathway Z') needs evidence about the pathway, not just the input and output. The advanced habit is matching the claim's confidence to what the design can bear. See Scientific Thinking.
Questions
0 of 6 answered
Question 1
A medical school is testing whether a new minimally invasive surgical technique for knee cartilage repair reduces recovery time compared to the standard open procedure. Patients are randomly assigned to receive either the new (n = 85) or standard (n = 87) surgery. Surgeons obviously know which technique they perform. The physical therapists evaluating recovery milestones (range of motion, weight-bearing ability, return to activity) also know each patient's surgical group. A biostatistician reviewing the protocol says the study needs partial blinding. What specific bias concerns her most, and what is the best feasible solution?
Question 2
A university study investigates whether an 8-week mindfulness-based stress reduction (MBSR) program reduces anxiety in college students (n = 120, randomly assigned 60 per group). The MBSR group attends weekly 90-minute classes with an experienced instructor, practices guided meditation, receives homework assignments, and builds relationships with fellow participants. The control group is placed on a waiting list and receives no contact. After 8 weeks, the MBSR group shows significantly lower anxiety scores (d = 0.58, p < 0.001). The researchers conclude that mindfulness meditation reduces anxiety. What confounding variable has a critic most likely identified?
Question 3
A research team uses a large epidemiological database (n = 200,000) to investigate relationships between 50 dietary variables and cardiovascular disease risk. They test each variable individually at p < 0.05 and find that consumption of a specific fermented food is significantly associated with lower CVD risk (p = 0.008, HR = 0.82). They publish a paper focused exclusively on this finding, with the title framing it as a novel discovery. They do not mention the other 49 variables tested. A statistician accuses them of p-hacking. Why?
Question 4
A cognitive psychology lab publishes a study finding that holding a warm beverage makes people rate strangers as having "warmer" personalities (n = 41, p = 0.04, d = 0.48). The experiment was conducted in a university laboratory where undergraduates briefly held either a warm or cold cup, then rated a fictional person described in a short paragraph. The authors claim this demonstrates embodied cognition, meaning physical warmth activates psychological warmth concepts. A critic raises concerns about ecological validity. What is the core problem?
Question 5
In 2015, the Reproducibility Project: Psychology attempted to replicate 100 published studies from three top psychology journals. Only 36% of replications produced statistically significant results (vs. 97% of originals), and average effect sizes dropped by approximately 50%. A psychology professor argues: "This does not mean 64% of psychology findings are wrong. Some failures may reflect genuine contextual differences between original and replication samples, settings, or time periods." How should a critical thinker evaluate this defense?
Question 6
A research team studying the effect of class size on student achievement compares standardized test scores across 500 schools in a state. They find that schools with smaller average class sizes (under 20 students) have significantly higher test scores than schools with larger classes (over 30 students), with p < 0.001 and d = 0.45. The state education board proposes spending $2 billion to reduce class sizes statewide. A policy researcher urges caution. What is the most important concern?
