Read the article and answer the question
Read Kim (2021) and answer the following questions.
What are the independent and dependent variables?
How was the dependent variable of the study operationalized and measured?
What are the constants?
Vafaee, A&HL5575 (Class 2-Part A)
2
Vafaee, A&HL5575 (Class 2-Part A)
3
Source of Knowledge
As Authority
As Belief
e.g., religion
Doughty and Pica (1986)
said that according to
Long (1981), claims that
such activities promote
optimal conditions for
students to adjust their
input to each other’s level
of comprehension … and
thereby facilitate their L2
acquisitions.
Vafaee, A&HL5575 (Class 2-Part A)
A priori
Empirical
Doughty and Pica (1986)
said that in keeping with
SLA theory, such
modified interaction is
claimed to make input
comprehensible to learners
and to lead ultimately to
successful classroom
acquisition.
According to McDonald
(1987), for all
constructions tested, it was
found that with increasing
exposure, cue usage in L2
gradually shifted from that
appropriate to the L1 to
that appropriate to the L2.
4
Identifying
the problem
Reviewing
relevant
information
Vafaee, A&HL5575 (Class 2-Part A)
Collecting
data
Analyzing
data
Drawing
conclusions
5
Vafaee, A&HL5575 (Class 2-Part A)
6
Vafaee, A&HL5575 (Class 2-Part A)
7
Vafaee, A&HL5575 (Class 2-Part A)
8
Vafaee, A&HL5575 (Class 2-Part A)
9
Vafaee, A&HL5575 (Class 2-Part A)
10
Vafaee, A&HL5575 (Class 2-Part A)
11
Empirical Research
Basic/Theoretical
e.g., universals of relative
clauses
Vafaee, A&HL5575 (Class 2-Part A)
Applied
Practical
e.g., order of acquisition
of relative clauses in
typologically similar and
dissimilar languages from
English
e.g., evaluation of relative
clause teaching materials
12
AL Research Types
Secondary
Library
Research
Primary/Empirical
Lit. Review
Qualitative
Ethnography
Identifying problems and
reviewing information
Discourse
Analysis
All Qualitative
Vafaee, A&HL5575 (Class 2-Part A)
Survey
Interviews
Questionnaires
Qualitative or Quantitative
Mixed Methods
Statistical
Descriptive
Exploratory
Quasiexperimental
experimental
All Quantitative
13
Qualitative
Quantitative
Concerned with understanding human
behavior from the actor’s own frame of
reference
Seeks facts or causes/effects/relationships of
social/psychological phenomena without
regard to the subjective states of the
individuals
Observer-participant Interaction
Detached role of researcher
Subjective
Objective
Grounded, discovery-oriented, exploratory,
descriptive.
Ungrounded, verification- oriented,
confirmatory, reductionist, inferential, and
Hypothetical- deductive
Naturalistic and uncontrolled observation
Obtrusive and controlled measurement
Holistic Inquiry
Focused on individual variables
Context-specific
Context- free
Inductive
Hypothetical-deductive
‘real’, ‘rich’, and ‘deep’ data; ungeneralizable
‘hard’ and replicable data ; generalizable
Narrative description
Statistical analysis
Process- oriented
Outcome- oriented
Vafaee, A&HL5575 (Class 2-Part A)
14
AL Research Type
Data Collection
Method
Type of Collected
Data
Type of Data
Analysis
Experimentally vs. Nonexperimentally
Qualitative vs.
Quantitative
Statistical vs.
Interpretative
Paradigm 1:
exploratory-interpretative
Paradigm 3: experimentalqualitative-interpretative
Paradigm 5: exploratory
qualitative-statistical
Paradigm 7: exploratoryquantitative-interpretive
Non-experimental design
Experimental design
Non-experimental design
Non-experimental design
Qualitative data
Qualitative data
Qualitative data
Quantitative data
Interpretative analysis
Interpretative analysis
Statistical analysis
Interpretative analysis
Paradigm 2:
analytical-nomological
Paradigm 4: experimentalqualitative-statistical
Paradigm 6: exploratoryquantitative-statistical
Paradigm 8: experimentalquantitative-interpretive
Experimental design
Experimental design
Non-experimental design
Experimental design
Quantitative data
Qualitative data
Quantitative data
Quantitative data
Statistical analysis
Statistical analysis
Statistical analysis
Interpretative analysis
Vafaee, A&HL5575 (Class 2-Part A)
15
Session 3: Quantitative Research in AL
Research Literacy (A&HL 5575)
Quantitative Research in AL
AL Research Types
Secondary
Qualitative
e.g., Ethnography
Primary/Empirical
Quantitative
e.g., Critical
Discourse Analysis
Correlational/
associational
(Quasi)
experimental
Vafaee, A&HL5575 (Class 3)
2
Brief Historical Overview
! Quantitative social research was inspired by the progress of natural
sciences in the 19th century.
! Social research started adopting the “scientific method”.
! Scientific method postulates three key stages in research:
1) observing a phenomenon or identifying a problem;
2) generating an initial hypothesis;
3) testing the hypothesis by using empirical data.
! The scientific method is closely associated with numerical values,
and statistics became a subdiscipline of math by the end of the 19th
century.
! Major developments both in scientific method (e.g., the works of
Karl Popper) and statistics (e.g., Spearman and Pearson) in the first
half of the 20th century.
Vafaee, A&HL5575 (Class 3)
3
Brief Historical Overview
! Thanks to the progress in psychometrics, quantitative methodology
became the dominant research methods in social sciences in the
middle of 20th century.
! The hegemony started to change in the 1970s by the challenges
posed by qualitative research.
! Currently, in many areas of social sciences, the two methods of
have peaceful coexistence.
! In Applied Linguistics, there was a significant increase in the
number of quantitative research between 1970 to 1985.
! Quantitative methods still maintain the dominance although
qualitative research method is gaining a fast momentum.
! Out of 524 empirical studies published between 1991 to 2001, 86%
were quantitative, 13% were qualitative, and 1% was mixed method.
Vafaee, A&HL5575 (Class 3)
4
Main Characteristics of Quantitative Research
! Using numbers
! A priori categorization
! Variables rather than cases
! Statistics and the language of statistics
! Standardized procedures to assess objective reality
! Quest for generalizability and universal laws
Vafaee, A&HL5575 (Class 3)
5
Strengths of Quantitative Research
! Systematic
! Focused
! Tightly controlled
! Precise measurement with reliable and replicable data
! Generalizable to other contexts
! The statistical analytical apparatus can be evaluated easily
! Relatively quick and less costly
6
Vafaee, A&HL5575 (Class 3)
Weaknesses of Quantitative Research
! Impossible to do justice to the subjective variety of individuals
! Not always sensitive to the underlying processes
! Can be overly simplistic, decontextualized and reductionist
Vafaee, A&HL5575 (Class 3)
7
Research Design and Variance
! Research design is a plan for answering research questions by telling
us how to explain variance in the outcome variables of interest.
! In other words, the research design helps us to identify the sources
that contribute to the variance in the outcome variables of interest.
! Or, the research design helps us to identify the main independent
variables that contribute to the variance in the depended variable(s).
! Research design also helps us to identify other sources that
contribute to the variance in the depended variable(s).
! In our research design we can control for the effect of these
variables.
Vafaee, A&HL5575 (Class 3)
8
Research Design and Variance
! To have valid conclusions, research designs include more control
variables.
! In other words, the researcher should think about both the main
independent variables and intervening and/or moderating variables.
The effect of intervening and/or moderating variables should be
controlled; otherwise, they will act as confounding variables.
! However, no matter how many independent variables are included in
a research design, still there will be some confounding variables.
! The researcher should be aware of as many confounding variables as
possible and acknowledges the limitations of her/his findings.
Vafaee, A&HL5575 (Class 3)
9
Research Design Types
! Correlational (Associational) Research
” The goal of associational research is to determine whether a
relationship exists between variables and, if so, the strength of that
relationship.
” This is often tested statistically through correlations, which allow a
researcher to determine how closely two variables (e.g., motivation and
language ability) are related in a given population.
” Associational research is not concerned with causation, only with co-
occurrence.
Vafaee, A&HL5575 (Class 3)
10
Research Design Types
! Correlational (Associational) Research
” Correlation can be used in different ways: for example, to test a
relationship between or among variables, and to make predictions.
” Predictions are dependent on the outcome of a strong relationship
between or among variables. That is, if variables are strongly related,
we can often predict the likelihood of the presence of one from the
presence of the other(s).
Vafaee, A&HL5575 (Class 3)
11
Research Design Types
! Experimental and Quasi-Experimental Research
” In experimental studies, researchers deliberately manipulate one or
more variables (independent variables) to determine the effect on
another variable (dependent variable).
” This manipulation is usually described as a treatment and the
researcher’s goal is to determine whether there is a causal relationship.
” Many types of experimental research involve a comparison of
pretreatment and posttreatment performance.
” Randomization is usually viewed as one of the hallmarks of
experimental research.
” Design types can range from truly experimental (with random
assignment) to what is known as quasi-experimental (without random
assignment).
Vafaee, A&HL5575 (Class 3)
12
Research Design Types
! Experimental and Quasi-Experimental Research
” A typical experimental study usually uses comparison or control groups
to investigate research questions.
” Many second language research studies involve a comparison between
two or more groups.
” This is known as a between-groups design.
” This comparison can be made in one of two ways: two or more groups
with different treatments; or two or more groups, one of which, the
control group, receives no treatment.
Vafaee, A&HL5575 (Class 3)
13
Session 4: Experimental Research Design_Part 1
Research Literacy (A&HL 5575)
Experimental Research
! Experimental research is the way of determining the effect of something
on something else.
! In experimental research, we manipulate at least one variable, while
controlling for the effect of other variables, to determine the effect of
manipulation on the outcome variable.
! Example: Whether focusing a learner’s attention on some aspect of language increases
that individual’s uptake of that aspect of language.
! Experimental research is based on the Rationalist worldview.
” Rationalist approach is a theory-then-research or deductive approach.
! Research questions must be stated explicitly and must have some basis
on previous literature.
! Example: Does focused attention on noun-adjective agreement in Italian promote
learning to a greater extend than focused attention on wh-movement in Italian for
beginning learners of Italian?
! There are always hypotheses about the results of experiments.
! Example: Focused attention on noun-adjective agreement in Italian will promote
learning to a greater extend than focused attention on wh-movement in Italian for
beginning learners of Italian.
Vafaee, A&HL5575 (Class 4)
2
Explaining and Controlling Variance
!
Experimental research is conducted for the purpose of explaining or controlling variance.
Independent
Variable
Dependent Variable
Group 1
Method 1
Group 1
T-Scores 1
Group 2
Method 2
Group 2
T-Scores 2
Group 3
Method 3
Group 3
T-Scores 3
Variance in
scores
Controlling variance allows us to minimize the effects of extraneous variance. As a result, the
dependent variable can be interpreted without bias.
! Construct-relevant variance (Good variability or systematic variance).
!
! Variance related to the construct being investigated (e.g., variance in scores related to the
effect of method).
!
Construct-irrelevant variance (Bad variability or error/unsystematic or unwanted variance).
! Variance unrelated to the construct being investigated coming from factors other than the
effect of instruction (e.g., pre-existing difference in ability level)
Vafaee, A&HL5575 (Class 4)
3
Five Ways of Controlling the Unwanted Variance
1. Randomization
2. Holding conditions or factors constant (e.g., controlling for
proficiency, only one teacher)
3. Statistical adjustments (adjusting pretest-posttest differences in
gain scores–ANCOVA)
4. Building conditions or factors into the research design as
independent variables (e.g., incorporating proficiency level
the design—advanced/intermediate/beginners)
into
5. The combination of all four methods
Vafaee, A&HL5575 (Class 4)
4
1. Randomization
Unsystematic
variance
Systematic
variance
Random, error,
unsystematic
variance due
to other factors
(ability level,
motivation,
anxiety, etc.)
Variance due
to methods
Total variance
Vafaee, A&HL5575 (Class 4)
5
1. Randomization
Independent
Variable
Dependent Variable
G1: 20 SS
Method 1
Group 1
T-Scores 1
G2: 20 SS
Method 2
Group 2
T-Scores 2
G3: 20 SS
Method 3
Group 3
T-Scores 3
Variance in
scores
60 SS randomly assigned
to each of the 3 groups
! It equalizes the groups with respects to variables other than the main
independent variable.
! It aims at reducing the effect of “random”, “inherent” or “ within
groups” variance.
Vafaee, A&HL5575 (Class 4)
6
2. Holding Factors Constant
Independent
Variable
Same
Teacher
Dependent Variable
G1: 20 SS
Method 1
Group 1
T-Scores 1
G2: 20 SS
Method 2
Group 2
T-Scores 2
G3: 20 SS
Method 3
Group 3
T-Scores 3
Variance in
scores
60 SS randomly assigned
to each of the 3 groups
! Disadvantage:
! Reduces external validity
Vafaee, A&HL5575 (Class 4)
7
3. Statistical Control
! The effect of pre-existing variables is removed statistically.
Independent
Variable
Same
Teacher
G1: 20 SS
Method 1
Group 1
T-Scores 1
G2: 20 SS
Method 2
Group 2
T-Scores 2
G3: 20 SS
Method 3
Group 3
T-Scores 3
60 SS randomly assigned
to each of the 3 groups
Vafaee, A&HL5575 (Class 4)
Dependent Variable
Variance in
scores
Proficiency
scores from
all 60 SS
Transformed
test scores or
dependent
variable
Variance in
scores
8
4. Building Factors into the Research Design
Independent Variable
Same
Teacher
Vafaee, A&HL5575 (Class 4)
Group 1
Group 2
Group 3
Group 4
Group 5
Group 6
10 SS- Low
10 SS- High
10 SS- Low
10 SS- High
10 SS- Low
10 SS- High
Method 1
Method 2
Method 3
60 SS randomly assigned
to each of the 6 groups
Dependent Variable
Group 1
Group 2
Group 3
Group 4
Group 5
Group 6
T-scores 1
T-scores 2
T-scores 3
T-scores 4
T-scores 5
T-scores 6
Variance in
scores
9
4. Building Factors into the Research Design
Unsystematic
variance
Variance due
to ability
level
Systematic
variance
Variance due
to methods
Variance due
to other
factors
(motivation,
anxiety, etc.)
Vafaee, A&HL5575 (Class 4)
Total variance
10
5. Using Four Ways in Combination
Independent Variable
Dependent Variable
20 SS- Low
20 SS- High
20 SS- Low
20 SS- High
20 SS- Low
20 SS- High
20 SS- Low
20 SS- High
20 SS- Low
20 SS- High
20 SS- Low
20 SS- High
T-scores
T-scores
T-scores
T-scores
T-scores
T-scores
T-scores
T-scores
T-scores
T-scores
T-scores
T-scores
Teacher 1
Teacher 2
Same School
Teacher 1
Teacher 2
Teacher 1
Teacher 2
Method 1
Method 1
Method 2
Method 2
Method 3
Method 3
WM scores
for all 120 SS
Transformed
test scores or
dependent
variable
120 SS randomly assigned
to each of the 6 groups
Variance in
scores
Vafaee, A&HL5575 (Class 4)
11
Working Example
!
Which kind of corrective feedback is more effective? Explicit or implicit?
”
”
!
Which of the two corrective feedback types of metalinguistic explanation and recast is more
beneficial?
”
!
What morphosyntactic target structures? ➔
easy or difficult
Who are the learners? Children, young adults or adults
Which of the two corrective feedback types of metalinguistic explanation and recast is more
beneficial for learning the English article system and the past perfect tense among adult
learners?
”
!
morphosyntax, pronunciation, collocation
Which of the two corrective feedback types of metalinguistic explanation and recast is more
beneficial for learning the English article system and the past perfect tense?
”
!
Learning what? ➔
Which of the two corrective feedback types of metalinguistic explanation and recast is more
beneficial for learning morphosyntax?
”
!
Explicit like metalinguistic explanation
Implicit like recast
What is the setting? Classroom learning or naturalistic learning?
Which of the two corrective feedback types of metalinguistic explanation and recast is more
beneficial for learning the English article system and the past perfect tense among adult
learners who learn English in the formal classroom learning setting?
Vafaee, A&HL5575 (Class 4)
12
What are the Variables in the Working Example?
X
Variables
Constants
Main Independent
Intervening
Manipulated or predictor
How and why
Metalinguistic explanation
and recast
Dependent/predicted
Posttest on knowledge of articles and tense
Underlying mechanism
Moderating
Age and motivation
Noticing the feedback, so
individual differences in WM
capacity and L2 aptitude
Vafaee, A&HL5575 (Class 4)
13
Working Example
X
Variables
Constants
Main Independent
Intervening
Dependent/predicted
How and why
Manipulated or predictor
Posttest on knowledge of articles and tense
Control
Binary/classifying
Moderating
Underlying mechanism
Metalinguistic explanation
and recast
Learning setting, age and motivation
Continuous
Noticing the feedback, so
individual differences in WM
capacity
No Control
Adults
Classroom Setting
Vafaee, A&HL5575 (Class 4)
Measures of WM and Motivation
Statistically (Co-variate)
Extraneous or
Confounding
14
Amount of exposure outside the experimental
setting, stress level, how the feedback is delivered,
length of experiment, proficiency level, first
language, previous knowledge
Working Example
!
Controlling for individual differences in WM and motivation, which of the two corrective
feedback types of metalinguistic explanation and recast is more beneficial for learning the English
article system and the past perfect tense among adult EFL learners at the intermediate level?
Independent Variable
Group 1
Article
Past Tense
Same
Teacher
Group 2
Dependent Variable
Recast
Group 1
Article
Past Tense
Article Test
PP Test
Article Test
Meta_ L_ E
Group 2
Select 100 adult EFL learners
from the intermediate level
and assign them randomly to
the two groups
Motivation
and WM
scores
PP Test
Variance in
scores
Transformed
test scores or
dependent
variable
Variance in
scores
Vafaee, A&HL5575 (Class 4)
15
Working Example
!
Controlling for individual differences in WM and motivation, which of the two corrective feedback types of
metalinguistic explanation and recast is more beneficial for learning the English article system and the past
perfect tense among adult learners who learn English in the formal classroom learning setting?
Amount of exposure outside the experimental
setting, stress level, how the feedback is delivered,
length of experiment, proficiency level, first
language, previous knowledge
Extraneous or Confounding
Let’s control them
Amount of exposure and practice outside the experimental setting
Constant: Run a single-session experiment
Statistical: Include a questionnaire
Stress level
Statistical: Include a questionnaire
How the feedback is delivered
Constant: use the same materials and
experimenter
Length of experiment
proficiency level
Constant and statistical: make the
experiment long and include several
posttests (posttests and delayed posttests)
Constant: just the intermediate
Statistical: Include a test
First Language
Constant: just one first language
Statistical: Include a questionnaire
Previous knowledge
Constant: pretest and screen
16
Statistical: pretest and a covariate
What about little differences in previous knowledge or all the other
variables we are aware of?
Randomization
Working Example
!
What kind of corrective feedback is more useful?
!
Controlling for individual differences in WM and motivation, which of the two corrective
feedback types of metalinguistic explanation and recast is more beneficial for learning the English
article system and the past perfect tense among adult learners who learn English in the formal
classroom learning setting?
!
In a true (vs. quasi) experiment with a pretest, immediate posttest and delayed posttest design, the
effectiveness of metalinguistic explanation versus recast for correcting errors related to the use of
English articles and past perfect tense was studied. This study was carried out among Persian
learners of English who were all above 18 years old. These participants are learning English in an
EFL context and rarely have exposure to English out of the classroom setting. Also, during the
length of experiment which lasted for ten one-hour sessions, and by the time they took the delayed
posttest, which was one month after the last session of the experiment, these participants had no
practice of English outside of the experimental setting. After each of the experimental sessions, the
leaners took a posttest, so ten posttests were included in the study. In this study, to control for the
effect of motivation and stress on learning, questionnaires for both of these psychological variables
were included and their results were used as covariates in the statistical analyses. Also, the effect
of individual differences in WM on the learning outcomes were accounted for statistically.
Vafaee, A&HL5575 (Class 4)
17
Nonexperimental Quantitative Research
Ex Post Facto
One-group Posttest Only
Observational Approach
Explanatory
Predictive
Survey Approach
Cross-sectional
Trend
Vafaee, A&HL5575 (Class 6)
Longitudinal
Cohort
Panel
2
Vafaee, A&HL5575 (Class 6)
3
Vafaee, A&HL5575 (Class 6)
4
Ex Post:
Cons: Post test only, no control group may fact internal validity, no random assign
Pros:
Vafaee, A&HL5575 (Class 6)
5
N= 34 Study abroad group
N= 26 Study at home group
Study
abroad
Vafaee, A&HL5575 (Class 6)
6
Vafaee, A&HL5575 (Class 6)
7
Vafaee, A&HL5575 (Class 6)
8
Vafaee, A&HL5575 (Class 6)
9
Vafaee, A&HL5575 (Class 6)
10
Vafaee, A&HL5575 (Class 6)
11
Vafaee, A&HL5575 (Class 6)
12
Vafaee, A&HL5575 (Class 6)
13
Vafaee, A&HL5575 (Class 6)
14
Planning
Sampling
Instrumentation
Identifying the research problem
Identifying the variables
Literature Review
Operational definition the variables
Designing the study
Defining the population and sub-groups
Detailed sampling procedure
Sampling
Identifying existing instruments
Constructing tests and questionnaires
Designing interviews
Piloting the instruments
Data Collection
Measurement administration
Inter-rater and intra-rater reliability
Data Preparation
Data coding
Inter-rater and intra-rater reliability
Data set up
Data Analysis
Checking the measurement quality
Checking the assumptions
Conducting the analysis
Interpretation
Vafaee, A&HL5575 (Class 6)
Avoid going beyond data
15
Vafaee, A&HL5575 (Class 6)
16
Vafaee, A&HL5575 (Class 7)
2
Vafaee, A&HL5575 (Class 7)
3
Vafaee, A&HL5575 (Class 7)
4
Strata
Strata Size
Proportional Sample size by strata
B1 & B2
50
50/2= 25
B3 & B4
100
100/2= 50
I1 & I2
150
150/2= 75
I3 & I4
60
60/2= 30
Advanced
40
40/2= 20
N= 400
N= 200
Vafaee, A&HL5575 (Class 7)
5
Population: 15
advanced
classes
Random Selection of classes
Sample
of 5
classes
All members of
each class get
measured
Generalization of results based on logical basis
Vafaee, A&HL5575 (Class 7)
6
Vafaee, A&HL5575 (Class 7)
7
Vafaee, A&HL5575 (Class 7)
8
Vafaee, A&HL5575 (Class 7)
9
Vafaee, A&HL5575 (Class 7)
10
Vafaee, A&HL5575 (Class 2-Part A)
2
Vafaee, A&HL5575 (Class 2-Part A)
3
Source of Knowledge
As Authority
As Belief
e.g., religion
Doughty and Pica (1986)
said that according to
Long (1981), claims that
such activities promote
optimal conditions for
students to adjust their
input to each other’s level
of comprehension … and
thereby facilitate their L2
acquisitions.
Vafaee, A&HL5575 (Class 2-Part A)
A priori
Empirical
Doughty and Pica (1986)
said that in keeping with
SLA theory, such
modified interaction is
claimed to make input
comprehensible to learners
and to lead ultimately to
successful classroom
acquisition.
According to McDonald
(1987), for all
constructions tested, it was
found that with increasing
exposure, cue usage in L2
gradually shifted from that
appropriate to the L1 to
that appropriate to the L2.
4
Identifying
the problem
Reviewing
relevant
information
Vafaee, A&HL5575 (Class 2-Part A)
Collecting
data
Analyzing
data
Drawing
conclusions
5
Vafaee, A&HL5575 (Class 2-Part A)
6
Vafaee, A&HL5575 (Class 2-Part A)
7
Vafaee, A&HL5575 (Class 2-Part A)
8
Vafaee, A&HL5575 (Class 2-Part A)
9
Vafaee, A&HL5575 (Class 2-Part A)
10
Vafaee, A&HL5575 (Class 2-Part A)
11
Empirical Research
Basic/Theoretical
e.g., universals of relative
clauses
Vafaee, A&HL5575 (Class 2-Part A)
Applied
Practical
e.g., order of acquisition
of relative clauses in
typologically similar and
dissimilar languages from
English
e.g., evaluation of relative
clause teaching materials
12
AL Research Types
Secondary
Library
Research
Primary/Empirical
Lit. Review
Qualitative
Ethnography
Identifying problems and
reviewing information
Discourse
Analysis
All Qualitative
Vafaee, A&HL5575 (Class 2-Part A)
Survey
Interviews
Questionnaires
Qualitative or Quantitative
Mixed Methods
Statistical
Descriptive
Exploratory
Quasiexperimental
experimental
All Quantitative
13
Qualitative
Quantitative
Concerned with understanding human
behavior from the actor’s own frame of
reference
Seeks facts or causes/effects/relationships of
social/psychological phenomena without
regard to the subjective states of the
individuals
Observer-participant Interaction
Detached role of researcher
Subjective
Objective
Grounded, discovery-oriented, exploratory,
descriptive.
Ungrounded, verification- oriented,
confirmatory, reductionist, inferential, and
Hypothetical- deductive
Naturalistic and uncontrolled observation
Obtrusive and controlled measurement
Holistic Inquiry
Focused on individual variables
Context-specific
Context- free
Inductive
Hypothetical-deductive
‘real’, ‘rich’, and ‘deep’ data; ungeneralizable
‘hard’ and replicable data ; generalizable
Narrative description
Statistical analysis
Process- oriented
Outcome- oriented
Vafaee, A&HL5575 (Class 2-Part A)
14
AL Research Type
Data Collection
Method
Type of Collected
Data
Type of Data
Analysis
Experimentally vs. Nonexperimentally
Qualitative vs.
Quantitative
Statistical vs.
Interpretative
Paradigm 1:
exploratory-interpretative
Paradigm 3: experimentalqualitative-interpretative
Paradigm 5: exploratory
qualitative-statistical
Paradigm 7: exploratoryquantitative-interpretive
Non-experimental design
Experimental design
Non-experimental design
Non-experimental design
Qualitative data
Qualitative data
Qualitative data
Quantitative data
Interpretative analysis
Interpretative analysis
Statistical analysis
Interpretative analysis
Paradigm 2:
analytical-nomological
Paradigm 4: experimentalqualitative-statistical
Paradigm 6: exploratoryquantitative-statistical
Paradigm 8: experimentalquantitative-interpretive
Experimental design
Experimental design
Non-experimental design
Experimental design
Quantitative data
Qualitative data
Quantitative data
Quantitative data
Statistical analysis
Statistical analysis
Statistical analysis
Interpretative analysis
Vafaee, A&HL5575 (Class 2-Part A)
15
Measures of Central Tendency
Measures of Dispersion
Reliability
9
Reliability
10
When two sets of values go up or down together, the correlation is positive
When one set of values go up but the other go down, the correlation is negative
Vafaee, A&HL4088 (Class 8)
11
Vafaee, A&HL4088 (Class 8)
12
An independent-samples t-test was run to determine if there were differences in engagement to a lesson
between males and females. The lesson was more engaging to male viewers (M = 5.56, SD = 0.35) than
female viewers (M = 5.30, SD = 0.35), a statistically significant difference, M = 0.26, 95% CI [0.04,
0.48], t(38) = 2.365, p = .023.
Vafaee, A&HL4088 (Class 8)
13
A paired-samples t-test showed that the carbohydrate-protein drink elicited a statistically
significant increase in distance run in two hours compared to a carbohydrate-only drink, t(19) =
6.352, p < .001.
Vafaee, A&HL4088 (Class 8)
14
Vafaee, A&HL4088 (Class 8)
15
A one-way ANOVA was conducted to determine if the ability to cope with workplace-related stress (CWWS
score) was different for groups with different physical activity levels. Participants were classified into four
groups: sedentary (n = 7), low (n = 9), moderate (n = 8) and high levels of physical activity (n = 7). CWWS score
was statistically significantly different between different physical activity groups, F(3, 27) = 8.316, p < .0005.
CWWS score increased from the sedentary (M = 4.15, SD = 0.77) to the low (M = 5.88, SD = 1.69), moderate (M
= 7.12, SD = 1.57) and high (M = 7.51, SD = 1.24) physical activity groups, in that order. Tukey post hoc analysis
revealed that the mean increase from sedentary to moderate was statistically significant (2.97, p = .002), as well
as the increase from sedentary to high (3.35, p = .001), but no other group differences were statistically
significant.
Vafaee, A&HL4088 (Class 8)
16
Vafaee, A&HL4088 (Class 8)
17
After adjustment for pre-test scores, the results of an ANCOVA showed that there was
a statistically significant difference in post-test scores, F(2, 41) = 105.512, p < .001.
Vafaee, A&HL4088 (Class 8)
18
Vafaee, A&HL4088 (Class 8)
19
Vafaee, A&HL4088 (Class 8)
20
APA: Students in Schools A, B and C scored higher in their English exam (M = 75.6, SD =
8.2; M = 63.6, SD = 6.6 and M = 59.8, SD = 4.6, respectively) than their math exam (M =
43.9, SD = 8.5; M = 40.8, SD = 8.2 and M = 30.8, SD = 7.7, respectively).
Vafaee, A&HL5199 (Classes 11 & 12)
21
Although Wilks' Lambda is usually recommended, Pillai's Trace is more robust and is
recommended when you have unequal sample sizes and also have a statistically significant Box's
M result
APA: There was a statistically significant difference between the schools on the combined
dependent variables, F(4, 112) = 17.675, p < .0005; Wilks' Λ = .376; partial η2 = .387.
Vafaee, A&HL5199 (Classes 11 & 12)
22
APA: A one-way multivariate analysis of variance was run to determine the effect of students'
previous schooling on academic performance. Two measures of academic performance were
assessed: English and math's end-of-year exam scores. Students arrived from three previous
schools: School A, School B and School C. Preliminary assumption checking revealed that data
was normally distributed, as assessed by Shapiro-Wilk test (p > .05); there were no univariate
or multivariate outliers, as assessed by boxplot and Mahalanobis distance (p > .001),
respectively; there were linear relationships, as assessed by scatterplot, no multicollinearity
(r = .393, p = .002); and there was homogeneity of variance-covariance matrices, as assessed
by Box’s M test (p = .003). Students in Schools A, B and C scored higher in their English exam
(M = 75.6, SD = 8.2; M = 63.6, SD = 6.6 and M = 59.8, SD = 4.6, respectively) than their math
exam (M = 43.9, SD = 8.5; M = 40.8, SD = 8.2 and M = 30.8, SD = 7.7, respectively). The
differences between the schools on the combined dependent variables was not statistically
significant, F(4, 112) = 1.254, p =.365; Wilks’ Λ = .041; partial η2 = .005.
Vafaee, A&HL5199 (Classes 11 & 12)
23
To determine which dependent variable would appear to be contributing to the statistically
significant MANOVA, you can inspect the one-way ANOVA result for each dependent variable.
These results are contained within the Tests of Between-Subjects Effects table, as shown
below:
APA: There was a statistically significant difference in English exam scores between the
students from different previous schools, F(2, 57) = 30.875, p < .001; partial η2 = .520.
APA: There was a statistically significant difference in math exam scores between the
students from different previous schools, F(2, 57) = 14.295, p < .001; partial η2 = .334.
Vafaee, A&HL5199 (Classes 11 & 12)
24
For any of your univariate ANOVAs that are statistically significant, you can follow them up
with Tukey post-hoc tests (or another multiple comparison procedure of your choosing).
For example, if you had violated the assumption of homogeneity of variances, you might
prefer to run a Games-Howell post-hoc test.
Vafaee, A&HL5199 (Classes 11 & 12)
25
A one-way multivariate analysis of variance was run to determine the effect of students'
previous schooling on academic performance. Two measures of academic performance were
assessed: English and math's end-of-year exam scores. Students arrived from three previous
schools: School A, School B and School C. Preliminary assumption checking revealed that data
was normally distributed, as assessed by Shapiro-Wilk test (p > .05); there were no univariate
or multivariate outliers, as assessed by boxplot and Mahalanobis distance (p > .001),
respectively; there were linear relationships, as assessed by scatterplot; no multicollinearity
(r = .393, p = .002); and there was homogeneity of variance-covariance matrices, as assessed
by Box’s M test (p = .003). Students in Schools A, B and C scored higher in their English exam
(M = 75.6, SD = 8.2; M = 63.6, SD = 6.6 and M = 59.8, SD = 4.6, respectively) than their math
exam (M = 43.9, SD = 8.5; M = 40.8, SD = 8.2 and M = 30.8, SD = 7.7, respectively). The
differences between the schools on the combined dependent variables was statistically
significant, F(4, 112) = 17.675, p < .001; Wilks' Λ = .376; partial η2 = .387. Follow-up univariate
ANOVAs showed that both English scores (F(2, 57) = 30.875, p < .001; partial η2 = .520) and
math scores (F(2, 57) = 14.295, p < .001; partial η2 = .334.) were statistically significantly
different between the students from different previous schools, using a Bonferroni adjusted α
level of .025. Tukey post-hoc tests showed that for English scores, students from School A had
statistically significantly higher mean scores than pupils from either School B (p < .001) or
School C (p < .001), but not between School B and School C (p = .169). For math scores, Tukey
post-hoc tests showed that School C had statistically significantly lower mean scores than
students from either School A (p < .001) or School B (p = .001).
Vafaee, A&HL5199 (Classes 11 & 12)
26
Vafaee, A&HL4088 (Class 8)
27
A multiple regression was run to predict health from
gender, age, weight and heart rate. The multiple
regression model statistically significantly predicted
health, F(4, 95) = 32.393, p < .001, adj. R2 = .56. All four
variables added statistically significantly to the
prediction, p < .05. Regression coefficients and standard
errors can be found in Table 1 (below).
Vafaee, A&HL4088 (Class 8)
28
Session 3: Quantitative Research in AL
Research Literacy (A&HL 5575)
Quantitative Research in AL
AL Research Types
Secondary
Qualitative
e.g., Ethnography
Primary/Empirical
Quantitative
e.g., Critical
Discourse Analysis
Correlational/
associational
(Quasi)
experimental
Vafaee, A&HL5575 (Class 3)
2
Brief Historical Overview
! Quantitative social research was inspired by the progress of natural
sciences in the 19th century.
! Social research started adopting the “scientific method”.
! Scientific method postulates three key stages in research:
1) observing a phenomenon or identifying a problem;
2) generating an initial hypothesis;
3) testing the hypothesis by using empirical data.
! The scientific method is closely associated with numerical values,
and statistics became a subdiscipline of math by the end of the 19th
century.
! Major developments both in scientific method (e.g., the works of
Karl Popper) and statistics (e.g., Spearman and Pearson) in the first
half of the 20th century.
Vafaee, A&HL5575 (Class 3)
3
Brief Historical Overview
! Thanks to the progress in psychometrics, quantitative methodology
became the dominant research methods in social sciences in the
middle of 20th century.
! The hegemony started to change in the 1970s by the challenges
posed by qualitative research.
! Currently, in many areas of social sciences, the two methods of
have peaceful coexistence.
! In Applied Linguistics, there was a significant increase in the
number of quantitative research between 1970 to 1985.
! Quantitative methods still maintain the dominance although
qualitative research method is gaining a fast momentum.
! Out of 524 empirical studies published between 1991 to 2001, 86%
were quantitative, 13% were qualitative, and 1% was mixed method.
Vafaee, A&HL5575 (Class 3)
4
Main Characteristics of Quantitative Research
! Using numbers
! A priori categorization
! Variables rather than cases
! Statistics and the language of statistics
! Standardized procedures to assess objective reality
! Quest for generalizability and universal laws
Vafaee, A&HL5575 (Class 3)
5
Strengths of Quantitative Research
! Systematic
! Focused
! Tightly controlled
! Precise measurement with reliable and replicable data
! Generalizable to other contexts
! The statistical analytical apparatus can be evaluated easily
! Relatively quick and less costly
6
Vafaee, A&HL5575 (Class 3)
Weaknesses of Quantitative Research
! Impossible to do justice to the subjective variety of individuals
! Not always sensitive to the underlying processes
! Can be overly simplistic, decontextualized and reductionist
Vafaee, A&HL5575 (Class 3)
7
Research Design and Variance
! Research design is a plan for answering research questions by telling
us how to explain variance in the outcome variables of interest.
! In other words, the research design helps us to identify the sources
that contribute to the variance in the outcome variables of interest.
! Or, the research design helps us to identify the main independent
variables that contribute to the variance in the depended variable(s).
! Research design also helps us to identify other sources that
contribute to the variance in the depended variable(s).
! In our research design we can control for the effect of these
variables.
Vafaee, A&HL5575 (Class 3)
8
Research Design and Variance
! To have valid conclusions, research designs include more control
variables.
! In other words, the researcher should think about both the main
independent variables and intervening and/or moderating variables.
The effect of intervening and/or moderating variables should be
controlled; otherwise, they will act as confounding variables.
! However, no matter how many independent variables are included in
a research design, still there will be some confounding variables.
! The researcher should be aware of as many confounding variables as
possible and acknowledges the limitations of her/his findings.
Vafaee, A&HL5575 (Class 3)
9
Research Design Types
! Correlational (Associational) Research
" The goal of associational research is to determine whether a
relationship exists between variables and, if so, the strength of that
relationship.
" This is often tested statistically through correlations, which allow a
researcher to determine how closely two variables (e.g., motivation and
language ability) are related in a given population.
" Associational research is not concerned with causation, only with co-
occurrence.
Vafaee, A&HL5575 (Class 3)
10
Research Design Types
! Correlational (Associational) Research
" Correlation can be used in different ways: for example, to test a
relationship between or among variables, and to make predictions.
" Predictions are dependent on the outcome of a strong relationship
between or among variables. That is, if variables are strongly related,
we can often predict the likelihood of the presence of one from the
presence of the other(s).
Vafaee, A&HL5575 (Class 3)
11
Research Design Types
! Experimental and Quasi-Experimental Research
" In experimental studies, researchers deliberately manipulate one or
more variables (independent variables) to determine the effect on
another variable (dependent variable).
" This manipulation is usually described as a treatment and the
researcher's goal is to determine whether there is a causal relationship.
" Many types of experimental research involve a comparison of
pretreatment and posttreatment performance.
" Randomization is usually viewed as one of the hallmarks of
experimental research.
" Design types can range from truly experimental (with random
assignment) to what is known as quasi-experimental (without random
assignment).
Vafaee, A&HL5575 (Class 3)
12
Research Design Types
! Experimental and Quasi-Experimental Research
" A typical experimental study usually uses comparison or control groups
to investigate research questions.
" Many second language research studies involve a comparison between
two or more groups.
" This is known as a between-groups design.
" This comparison can be made in one of two ways: two or more groups
with different treatments; or two or more groups, one of which, the
control group, receives no treatment.
Vafaee, A&HL5575 (Class 3)
13
Session 4: Experimental Research Design_Part 1
Research Literacy (A&HL 5575)
Experimental Research
! Experimental research is the way of determining the effect of something
on something else.
! In experimental research, we manipulate at least one variable, while
controlling for the effect of other variables, to determine the effect of
manipulation on the outcome variable.
! Example: Whether focusing a learner's attention on some aspect of language increases
that individual’s uptake of that aspect of language.
! Experimental research is based on the Rationalist worldview.
" Rationalist approach is a theory-then-research or deductive approach.
! Research questions must be stated explicitly and must have some basis
on previous literature.
! Example: Does focused attention on noun-adjective agreement in Italian promote
learning to a greater extend than focused attention on wh-movement in Italian for
beginning learners of Italian?
! There are always hypotheses about the results of experiments.
! Example: Focused attention on noun-adjective agreement in Italian will promote
learning to a greater extend than focused attention on wh-movement in Italian for
beginning learners of Italian.
Vafaee, A&HL5575 (Class 4)
2
Explaining and Controlling Variance
!
Experimental research is conducted for the purpose of explaining or controlling variance.
Independent
Variable
Dependent Variable
Group 1
Method 1
Group 1
T-Scores 1
Group 2
Method 2
Group 2
T-Scores 2
Group 3
Method 3
Group 3
T-Scores 3
Variance in
scores
Controlling variance allows us to minimize the effects of extraneous variance. As a result, the
dependent variable can be interpreted without bias.
! Construct-relevant variance (Good variability or systematic variance).
!
! Variance related to the construct being investigated (e.g., variance in scores related to the
effect of method).
!
Construct-irrelevant variance (Bad variability or error/unsystematic or unwanted variance).
! Variance unrelated to the construct being investigated coming from factors other than the
effect of instruction (e.g., pre-existing difference in ability level)
Vafaee, A&HL5575 (Class 4)
3
Five Ways of Controlling the Unwanted Variance
1. Randomization
2. Holding conditions or factors constant (e.g., controlling for
proficiency, only one teacher)
3. Statistical adjustments (adjusting pretest-posttest differences in
gain scores--ANCOVA)
4. Building conditions or factors into the research design as
independent variables (e.g., incorporating proficiency level
the design—advanced/intermediate/beginners)
into
5. The combination of all four methods
Vafaee, A&HL5575 (Class 4)
4
1. Randomization
Unsystematic
variance
Systematic
variance
Random, error,
unsystematic
variance due
to other factors
(ability level,
motivation,
anxiety, etc.)
Variance due
to methods
Total variance
Vafaee, A&HL5575 (Class 4)
5
1. Randomization
Independent
Variable
Dependent Variable
G1: 20 SS
Method 1
Group 1
T-Scores 1
G2: 20 SS
Method 2
Group 2
T-Scores 2
G3: 20 SS
Method 3
Group 3
T-Scores 3
Variance in
scores
60 SS randomly assigned
to each of the 3 groups
! It equalizes the groups with respects to variables other than the main
independent variable.
! It aims at reducing the effect of “random”, “inherent” or “ within
groups” variance.
Vafaee, A&HL5575 (Class 4)
6
2. Holding Factors Constant
Independent
Variable
Same
Teacher
Dependent Variable
G1: 20 SS
Method 1
Group 1
T-Scores 1
G2: 20 SS
Method 2
Group 2
T-Scores 2
G3: 20 SS
Method 3
Group 3
T-Scores 3
Variance in
scores
60 SS randomly assigned
to each of the 3 groups
! Disadvantage:
! Reduces external validity
Vafaee, A&HL5575 (Class 4)
7
3. Statistical Control
! The effect of pre-existing variables is removed statistically.
Independent
Variable
Same
Teacher
G1: 20 SS
Method 1
Group 1
T-Scores 1
G2: 20 SS
Method 2
Group 2
T-Scores 2
G3: 20 SS
Method 3
Group 3
T-Scores 3
60 SS randomly assigned
to each of the 3 groups
Vafaee, A&HL5575 (Class 4)
Dependent Variable
Variance in
scores
Proficiency
scores from
all 60 SS
Transformed
test scores or
dependent
variable
Variance in
scores
8
4. Building Factors into the Research Design
Independent Variable
Same
Teacher
Vafaee, A&HL5575 (Class 4)
Group 1
Group 2
Group 3
Group 4
Group 5
Group 6
10 SS- Low
10 SS- High
10 SS- Low
10 SS- High
10 SS- Low
10 SS- High
Method 1
Method 2
Method 3
60 SS randomly assigned
to each of the 6 groups
Dependent Variable
Group 1
Group 2
Group 3
Group 4
Group 5
Group 6
T-scores 1
T-scores 2
T-scores 3
T-scores 4
T-scores 5
T-scores 6
Variance in
scores
9
4. Building Factors into the Research Design
Unsystematic
variance
Variance due
to ability
level
Systematic
variance
Variance due
to methods
Variance due
to other
factors
(motivation,
anxiety, etc.)
Vafaee, A&HL5575 (Class 4)
Total variance
10
5. Using Four Ways in Combination
Independent Variable
Dependent Variable
20 SS- Low
20 SS- High
20 SS- Low
20 SS- High
20 SS- Low
20 SS- High
20 SS- Low
20 SS- High
20 SS- Low
20 SS- High
20 SS- Low
20 SS- High
T-scores
T-scores
T-scores
T-scores
T-scores
T-scores
T-scores
T-scores
T-scores
T-scores
T-scores
T-scores
Teacher 1
Teacher 2
Same School
Teacher 1
Teacher 2
Teacher 1
Teacher 2
Method 1
Method 1
Method 2
Method 2
Method 3
Method 3
WM scores
for all 120 SS
Transformed
test scores or
dependent
variable
120 SS randomly assigned
to each of the 6 groups
Variance in
scores
Vafaee, A&HL5575 (Class 4)
11
Working Example
!
Which kind of corrective feedback is more effective? Explicit or implicit?
"
"
!
Which of the two corrective feedback types of metalinguistic explanation and recast is more
beneficial?
"
!
What morphosyntactic target structures? ➔
easy or difficult
Who are the learners? Children, young adults or adults
Which of the two corrective feedback types of metalinguistic explanation and recast is more
beneficial for learning the English article system and the past perfect tense among adult
learners?
"
!
morphosyntax, pronunciation, collocation
Which of the two corrective feedback types of metalinguistic explanation and recast is more
beneficial for learning the English article system and the past perfect tense?
"
!
Learning what? ➔
Which of the two corrective feedback types of metalinguistic explanation and recast is more
beneficial for learning morphosyntax?
"
!
Explicit like metalinguistic explanation
Implicit like recast
What is the setting? Classroom learning or naturalistic learning?
Which of the two corrective feedback types of metalinguistic explanation and recast is more
beneficial for learning the English article system and the past perfect tense among adult
learners who learn English in the formal classroom learning setting?
Vafaee, A&HL5575 (Class 4)
12
What are the Variables in the Working Example?
X
Variables
Constants
Main Independent
Intervening
Manipulated or predictor
How and why
Metalinguistic explanation
and recast
Dependent/predicted
Posttest on knowledge of articles and tense
Underlying mechanism
Moderating
Age and motivation
Noticing the feedback, so
individual differences in WM
capacity and L2 aptitude
Vafaee, A&HL5575 (Class 4)
13
Working Example
X
Variables
Constants
Main Independent
Intervening
Dependent/predicted
How and why
Manipulated or predictor
Posttest on knowledge of articles and tense
Control
Binary/classifying
Moderating
Underlying mechanism
Metalinguistic explanation
and recast
Learning setting, age and motivation
Continuous
Noticing the feedback, so
individual differences in WM
capacity
No Control
Adults
Classroom Setting
Vafaee, A&HL5575 (Class 4)
Measures of WM and Motivation
Statistically (Co-variate)
Extraneous or
Confounding
14
Amount of exposure outside the experimental
setting, stress level, how the feedback is delivered,
length of experiment, proficiency level, first
language, previous knowledge
Working Example
!
Controlling for individual differences in WM and motivation, which of the two corrective
feedback types of metalinguistic explanation and recast is more beneficial for learning the English
article system and the past perfect tense among adult EFL learners at the intermediate level?
Independent Variable
Group 1
Article
Past Tense
Same
Teacher
Group 2
Dependent Variable
Recast
Group 1
Article
Past Tense
Article Test
PP Test
Article Test
Meta_ L_ E
Group 2
Select 100 adult EFL learners
from the intermediate level
and assign them randomly to
the two groups
Motivation
and WM
scores
PP Test
Variance in
scores
Transformed
test scores or
dependent
variable
Variance in
scores
Vafaee, A&HL5575 (Class 4)
15
Working Example
!
Controlling for individual differences in WM and motivation, which of the two corrective feedback types of
metalinguistic explanation and recast is more beneficial for learning the English article system and the past
perfect tense among adult learners who learn English in the formal classroom learning setting?
Amount of exposure outside the experimental
setting, stress level, how the feedback is delivered,
length of experiment, proficiency level, first
language, previous knowledge
Extraneous or Confounding
Let’s control them
Amount of exposure and practice outside the experimental setting
Constant: Run a single-session experiment
Statistical: Include a questionnaire
Stress level
Statistical: Include a questionnaire
How the feedback is delivered
Constant: use the same materials and
experimenter
Length of experiment
proficiency level
Constant and statistical: make the
experiment long and include several
posttests (posttests and delayed posttests)
Constant: just the intermediate
Statistical: Include a test
First Language
Constant: just one first language
Statistical: Include a questionnaire
Previous knowledge
Constant: pretest and screen
16
Statistical: pretest and a covariate
What about little differences in previous knowledge or all the other
variables we are aware of?
Randomization
Working Example
!
What kind of corrective feedback is more useful?
!
Controlling for individual differences in WM and motivation, which of the two corrective
feedback types of metalinguistic explanation and recast is more beneficial for learning the English
article system and the past perfect tense among adult learners who learn English in the formal
classroom learning setting?
!
In a true (vs. quasi) experiment with a pretest, immediate posttest and delayed posttest design, the
effectiveness of metalinguistic explanation versus recast for correcting errors related to the use of
English articles and past perfect tense was studied. This study was carried out among Persian
learners of English who were all above 18 years old. These participants are learning English in an
EFL context and rarely have exposure to English out of the classroom setting. Also, during the
length of experiment which lasted for ten one-hour sessions, and by the time they took the delayed
posttest, which was one month after the last session of the experiment, these participants had no
practice of English outside of the experimental setting. After each of the experimental sessions, the
leaners took a posttest, so ten posttests were included in the study. In this study, to control for the
effect of motivation and stress on learning, questionnaires for both of these psychological variables
were included and their results were used as covariates in the statistical analyses. Also, the effect
of individual differences in WM on the learning outcomes were accounted for statistically.
Vafaee, A&HL5575 (Class 4)
17
Vafaee, A&HL5575 (Class 5)
2
Experimental Variables & Levels
An experimental variable usually has from 2 to 5 different levels or
experimental treatments.
Experimental Variable
Types of Corrective Feedback
Levels or Experimental Treatments
• Explicit with metalinguistic
explanation
• Explicit without metalinguistic
explanation
• Recast
Vafaee, A&HL5575 (Class 5)
3
Independent
Experimental Design
Types of Feedback
Randomly Assigned
Dependent Variable
Explicit + Explanation
N = 40 ESL learners
Explicit – Explanation
N = 40 ESL learners
# of correct answers
on a test
Recast
N = 40 ESL learners
Experimental procedure
R
R
R
G1
G2
G3
Xexp+ex
Xexp-ex
Xrecast
N = number of participants
R = randomization
X = experimental variable (manipulated)
I= Independent variable
G = group
O = Observation (assessment)
O1
O2
O3
20.03
23.20
29.90
Compare the mean scores of the groups.
◦ We use a t-test to compare the differences in
the mean scores of 2 groups.
◦ We use ANOVA (analysis of variance) to
compare the means of 3 or more groups).
Vafaee, A&HL5575 (Class 5)
4
Posttest-Only Control-Group Design
R G1
R G2(c)
X1
-
O1
O2
◦ Used when a pretest is not available, not used, too costly or interacts
with treatment.
◦ Randomization causes no need for pre-test especially with large
samples.
Vafaee, A&HL5575 (Class 5)
5
Pretest-Posttest Control Group Design
R G1
R G2(c)
O1
O3
X1
-
O2
O4
◦ It is a little tricky because pretests and posttests should be equal forms
or identical.
◦ In a randomized experiment, pretests are not used to create equal
groups.
◦ However, the pretests can reveal improbable results.
◦ Also, pretest results can be used for statistical control. ANCOVA is
used for data analysis for this purpose.
◦ The scores from pretests to posttests can be used to compute “gain”
scores.
6
Vafaee, A&HL5575 (Class 5)
Randomized Solomon Four-Group Design
R
R
R
R
G1E O1
G2C O3
G3E G4C -
X
X
-
O2
O4
O5
O6
◦ This design can show if pretests had an interaction with the treatment
or influenced the posttest’s scores.
Vafaee, A&HL5575 (Class 5)
7
Factorial Design
In the previous designs, we examined the effect of an independent
variable on a dependent variable. For example, we looked at the
effectiveness of three different types of corrective feedback.
● But the effectiveness of three different types of corrective feedback
might be mediated or moderated by other variables such as the
students’proficiency level.
●
●
●
There might be an interaction between the different types of corrective
feedback and the different ability groups.
To examine the effectiveness of three different types of corrective
feedback accounting for variations in the students’ proficiency level,
we can design a study using ONLY the intermediate level students.
And then, we design a similar study for ONLY the advanced level
students.
● But it is more efficient if we designed ONE study that accounts for
different types of corrective feedback in relation to different proficiency
levels of the learners.
●
When there is more than one independent variable in the deign of a study,
the design is called the factorial design.
Vafaee, A&HL5575 (Class 5)
8
Factorial Design
• A “complete” factorial design involves 2 or more independent variables
(factors) in the design
1) types of corrective feedback
2) proficiency levels
• Also, the independent variables must have at least 2 levels for each
variable
1) Three types of corrective feedback
2) Two proficiency levels (e.g., intermediate & advanced)
• This is a 3 x 2 factorial design, i.e., 2 independent variables and two
and three levels for each, respectively.
Vafaee, A&HL5575 (Class 5)
Intermediate
(Xexp + ex)
Intermediate
(Xexp - ex)
Intermediate
(Recast)
Advanced
(Xexp + ex)
Advanced
(Xexp - ex)
Advanced
(Recast)
9
Factorial Design
Practice 1:
• A factorial design with 3 independent variables:
1) types of pragmatics instruction
(explicit & implicit)
2X3X3
2) proficiency levels
(beginning, intermediate & advanced)
3) length of residence
(Low: 0-2 years; medium: 3 – 5 years; long: more than 5 years)
Practice 2 :
3X3X4
4X2
2X5X2
3X2X2X2
Vafaee, A&HL5575 (Class 5)
10
Repeated Measures Design: Extension of Posttest-Only
Vafaee, A&HL5575 (Class 5)
R G1
X1
O1
O2
O3
R G1
R G2(c)
X1
-
O1
O4
O2
O5
O3
O6
11
Repeated Measures Design: Extension of Pretest-Posttest
R G1 O1
O2
O3
X1 O4 O5 O6
R G1 O1
R G2 O6
O2 X1 O3 O4 O5
O7 X2 O8 O9 O10
R G1 O1 O2 X1 O3 O4 O5
R G2 O6 O7 X2 O8 O9 O10
R GC O11 O12 - O13 O14 O15
R G1 O1 X1a O2 X1b O3 X1c O4 O5
R G2 O6 X2a O7 X2b O8 X2c O9 O10
R GC O11 - O12 - O13 - O14 O15
Vafaee, A&HL5575 (Class 5)
12
Quasi-Experimental Research
1. Experimental design involves:
• One or more independent & dependent variables
• An experimental variable that is manipulated
• Random assignment to groups.
2. Quasi-experimental design involves:
• All the above
• Except for the Random assignment of participants to
experimental treatment groups.
• There are intact groups.
Vafaee, A&HL5575 (Class 5)
13
Threats to validity
Threats to internal validity:
• Differential selection of subjects (difficulty to argue that the
effect was due to treatment if groups are different)
Threats to external validity:
• Lack of random selection puts into question the
generalizability of the results
Solution:
• To argue for representativeness of the sample on a logical
basis
• The equivalence of groups can be checked statistically and if
not equivalent, the effect can be adjusted for.
Vafaee, A&HL5575 (Class 5)
14
Post-Test Only, Non-Equivalent, Multi-Group Design
G1(e)
G2(e)
X1
X2
O1
O2
RQ: What is the effect of explicit vocabulary instruction on lexical knowledge?
Non randomly Assigned
Types of Instruction
Dependent Variable
N = 20
Explicit vocabulary instruction
Post-test
RQ: What are the effects of explicit and incidental vocabulary teaching methods?
Non randomly Assigned
Types of Instruction
Dependent Variable
N = 20
Explicit vocabulary instruction
Post-test
N = 20
Incidental vocabulary instruction
Post-test
Vafaee, A&HL5575 (Class 5)
15
Post-Test Only, Non-Equivalent, Control Group Design
G1 (e)
G2 (e)
G3 (c)
X1
X2
-
O1
O2
O3
RQ: What is the effect of explicit vocabulary instruction on lexical knowledge?
Non randomly Assigned
Types of Instruction
Dependent Variable
N = 20
Explicit vocabulary instruction
Post-test
N = 20
-
Post-test
RQ: What are the effects of explicit and incidental vocabulary teaching methods?
Non randomly Assigned
Types of Instruction
Dependent Variable
N = 20
Explicit vocabulary instruction
Post-test
N = 20
Incidental vocabulary instruction
Post-test
N = 20
-
Post-test
Vafaee, A&HL5575 (Class 5)
16
Pretest-Posttest, Nonequivalent, Control Group Design
Pretest-Posttest, Nonequivalent, Multi-Group Design
Non randomly Assigned
Pretest
G1
O1
G2
G3
O3
O5
Gc or G4
O7
Types of treatment
Posttest 1 Posttest 2
X1
X2
O2
O9
O4
O10
X3
O6
O11
O8
O12
- Or X4
Use the pretest scores to:
1. check the similarity of the groups. If groups are not similar, the pretest scores can be used
to adjust the posttest scores statically.
2. generate gain scores.
Vafaee, A&HL5575 (Class 5)
17
Repeated Measures Design
G1
O1
O2
G1
G2
O1
O6
O2 X1 O3 O4 O5
O7 X2 O8 O9 O10
G1
G2
GC
O1 O2 X1 O3 O4 O5
O6 O7 X2 O8 O9 O10
O11 O12 - O13 O14 O15
G1
O1 X1a O2 X1b O3 X1c O4 O5
O6 X2a O7 X2b O8 X2c O9 O10
O11 - O12 - O13 - O14 O15
G2
GC
Vafaee, A&HL5575 (Class 5)
O3
X1 O4 O5 O6
18
Single-Subject Designs
Repeated measures of the dependent variable
Measurement must be highly standardized
Single variable rule
Traditional or normal condition is called the baseline
Baseline should be long enough for the dependent variable to gain stability
O1
Ok
O1
Ok
B
A
O1
Ok
A
Vafaee, A&HL5575 (Class 5)
O1
Ok
B
O1
Ok
A
19
Criteria for a well-designed experiment
1. Adequate Experimental Control: we control some variables (e.g.,
proficiency level) for meaningful interpretation of the results (to avoid
confounding); other variables can be controlled via randomization or by
building variables into the design.
2. Lack of artificiality: If we want to generalize to a non-experiment setting,
the results should apply to the real world. The artificial nature of the
experiment should not cause the experimental effects.
3. Adequate basis for comparison: there must be a way to determine the
experimental effect. The experiment may or need a control group (e.g., no
new treatment).
4. Precise data: the data must be reliable (e.g., the ratings from the 2 raters need
to be consistent); codings must have agreement; a test/ questionnaire should
had high Cronbach's alphas.
Vafaee, A&HL5575 (Class 5)
20
Criteria for a well-designed experiment
5. No contaminated data & no bias: no interactions between the data (e.g.,
raters must rate the data independently) (part of reliability); no interaction
between variables (raters are more consistent on some topics or with some
examinees)
6. No confounding of the data: no two variables contribute to the score in a
way that is difficult to separate their effect (proficiency and age) (validity
issue)
7. Sample representativeness: random selection to enhance generalizability
(of learners; also of items, tasks, contexts)
8. Parsimony: a simple research design/ underlying model is better than
complex one—when possible (!) BUT, the lack of complexity might not
represent reality.
Vafaee, A&HL5575 (Class 5)
21
Experimental Validity
1. Construct validity: The extent to which the constructs measured in the
independent or dependent variables are meaningful & appropriate as a
result of empirical supports.
2. Internal Validity: The extent to which the results of the experiment can be
attributed to the treatment. In other words, the extraneous variables are not
having an undue effect on the link between the independent & dependent
variable. They are being controlled--or are part of the design.
3. External Validity: The extent to which the results of the experiment can be
generalized to other populations (ESL/EFL, urban/rural), or conditions (class
size, computer-mediated/ face-to-face).
4. Statistical Conclusion Validity: The extent to the statistical procedures used
in the study are appropriate & meaningful for decision making. Are the
measures sufficiently reliable? Is the statistical procedure appropriate for the
data? Have the assumptions underlying the statistical procedure been met?
Vafaee, A&HL5575 (Class 5)
22
Threats to Construct Validity
1. Inadequate pre-operational explanation of constructs: Inadequate
theoretical & operation definition of the dependent or independent (i.e.,
how is “instruction” defined and operationalized/implemented by two
different teachers?)
2. Mono-operation bias: Only one form of experimental variable is
implemented (e.g., only written feedback is given, not oral)
3. Mono-method bias : Only one method is used to measure the dependent
variable (e.g., only a GJT)
4. Hypothesis guessing within the experimental conditions: Participants
behave differently when they know they’re part of an experiment
(Hawthorne Effect)
5. Confounding constructs & levels of constructs: Drawing conclusions
about constructs when some levels are absent.
Vafaee, A&HL5575 (Class 5)
23
Threats to Internal Validity
1. History: unanticipated events have occurred that affect performance on the
dependent variable.
2. Maturation: the participants mature/change (over time)
3. Testing/Pre-testing: the effects of test familiarity
4. Instrumentation/Measuring instruments: inconsistent use of a test (e.g.,
the instructions changed from the pre to the post-test)
5. Statistical regression: participants with extreme scores cause other scores
to regress toward the average score (e.g., with gain scores)
6. (Differential) selection (of a sample): no randomization, causing
unequivalent groups
7. (Experimental) Mortality: participants drop out
8. Interaction of selection & maturation, etc.: inconsistent maturation of
participants across groups (e.g., some students spent the summer in an
English speaking country)
Vafaee, A&HL5575 (Class 5)
24
Threats to External Validity
1. Interaction effect of pre-testing & the experimental variable : pretesting
interacts with the experimental variable causing ungeneralisable results.
2. Interaction effect of selection biases & experimental treatment: the effect
of choosing one group to give treatment to, which would produce a
different effect if the groups were randomized.
3. Reactive effect of experimental arrangements: the effect occurs when an
experiment takes place in a setting that would not produce the same effect
in different settings (generalizing across settings)
4. Multiple treatment interference: the carry over effect of participating
in multiple treatments (generalizing across time to the experimental
effects).
Vafaee, A&HL5575 (Class 5)
25
Threats to Statistical Conclusion Validity
1. Low statistical power: Sample size too small to detect differences between
groups
2. Violation of assumptions: failure to meet the assumptions or understand
the effect of violations (robust or not)
3. Fishing & error rate problem: Capitalizing on chance factor findings
4. Low reliability: Imprecise measures
Vafaee, A&HL5575 (Class 5)
26
Vafaee, A&HL5575 (Class 5)
2
Experimental Variables & Levels
An experimental variable usually has from 2 to 5 different levels or
experimental treatments.
Experimental Variable
Types of Corrective Feedback
Levels or Experimental Treatments
• Explicit with metalinguistic
explanation
• Explicit without metalinguistic
explanation
• Recast
Vafaee, A&HL5575 (Class 5)
3
Independent
Experimental Design
Types of Feedback
Randomly Assigned
Dependent Variable
Explicit + Explanation
N = 40 ESL learners
Explicit – Explanation
N = 40 ESL learners
# of correct answers
on a test
Recast
N = 40 ESL learners
Experimental procedure
R
R
R
G1
G2
G3
Xexp+ex
Xexp-ex
Xrecast
N = number of participants
R = randomization
X = experimental variable (manipulated)
I= Independent variable
G = group
O = Observation (assessment)
O1
O2
O3
20.03
23.20
29.90
Compare the mean scores of the groups.
◦ We use a t-test to compare the differences in
the mean scores of 2 groups.
◦ We use ANOVA (analysis of variance) to
compare the means of 3 or more groups).
Vafaee, A&HL5575 (Class 5)
4
Posttest-Only Control-Group Design
R G1
R G2(c)
X1
-
O1
O2
◦ Used when a pretest is not available, not used, too costly or interacts
with treatment.
◦ Randomization causes no need for pre-test especially with large
samples.
Vafaee, A&HL5575 (Class 5)
5
Pretest-Posttest Control Group Design
R G1
R G2(c)
O1
O3
X1
-
O2
O4
◦ It is a little tricky because pretests and posttests should be equal forms
or identical.
◦ In a randomized experiment, pretests are not used to create equal
groups.
◦ However, the pretests can reveal improbable results.
◦ Also, pretest results can be used for statistical control. ANCOVA is
used for data analysis for this purpose.
◦ The scores from pretests to posttests can be used to compute “gain”
scores.
6
Vafaee, A&HL5575 (Class 5)
Randomized Solomon Four-Group Design
R
R
R
R
G1E O1
G2C O3
G3E G4C -
X
X
-
O2
O4
O5
O6
◦ This design can show if pretests had an interaction with the treatment
or influenced the posttest’s scores.
Vafaee, A&HL5575 (Class 5)
7
Factorial Design
In the previous designs, we examined the effect of an independent
variable on a dependent variable. For example, we looked at the
effectiveness of three different types of corrective feedback.
● But the effectiveness of three different types of corrective feedback
might be mediated or moderated by other variables such as the
students’proficiency level.
●
●
●
There might be an interaction between the different types of corrective
feedback and the different ability groups.
To examine the effectiveness of three different types of corrective
feedback accounting for variations in the students’ proficiency level,
we can design a study using ONLY the intermediate level students.
And then, we design a similar study for ONLY the advanced level
students.
● But it is more efficient if we designed ONE study that accounts for
different types of corrective feedback in relation to different proficiency
levels of the learners.
●
When there is more than one independent variable in the deign of a study,
the design is called the factorial design.
Vafaee, A&HL5575 (Class 5)
8
Factorial Design
• A “complete” factorial design involves 2 or more independent variables
(factors) in the design
1) types of corrective feedback
2) proficiency levels
• Also, the independent variables must have at least 2 levels for each
variable
1) Three types of corrective feedback
2) Two proficiency levels (e.g., intermediate & advanced)
• This is a 3 x 2 factorial design, i.e., 2 independent variables and two
and three levels for each, respectively.
Vafaee, A&HL5575 (Class 5)
Intermediate
(Xexp + ex)
Intermediate
(Xexp - ex)
Intermediate
(Recast)
Advanced
(Xexp + ex)
Advanced
(Xexp - ex)
Advanced
(Recast)
9
Factorial Design
Practice 1:
• A factorial design with 3 independent variables:
1) types of pragmatics instruction
(explicit & implicit)
2X3X3
2) proficiency levels
(beginning, intermediate & advanced)
3) length of residence
(Low: 0-2 years; medium: 3 – 5 years; long: more than 5 years)
Practice 2 :
3X3X4
4X2
2X5X2
3X2X2X2
Vafaee, A&HL5575 (Class 5)
10
Repeated Measures Design: Extension of Posttest-Only
Vafaee, A&HL5575 (Class 5)
R G1
X1
O1
O2
O3
R G1
R G2(c)
X1
-
O1
O4
O2
O5
O3
O6
11
Repeated Measures Design: Extension of Pretest-Posttest
R G1 O1
O2
O3
X1 O4 O5 O6
R G1 O1
R G2 O6
O2 X1 O3 O4 O5
O7 X2 O8 O9 O10
R G1 O1 O2 X1 O3 O4 O5
R G2 O6 O7 X2 O8 O9 O10
R GC O11 O12 - O13 O14 O15
R G1 O1 X1a O2 X1b O3 X1c O4 O5
R G2 O6 X2a O7 X2b O8 X2c O9 O10
R GC O11 - O12 - O13 - O14 O15
Vafaee, A&HL5575 (Class 5)
12
Quasi-Experimental Research
1. Experimental design involves:
• One or more independent & dependent variables
• An experimental variable that is manipulated
• Random assignment to groups.
2. Quasi-experimental design involves:
• All the above
• Except for the Random assignment of participants to
experimental treatment groups.
• There are intact groups.
Vafaee, A&HL5575 (Class 5)
13
Threats to validity
Threats to internal validity:
• Differential selection of subjects (difficulty to argue that the
effect was due to treatment if groups are different)
Threats to external validity:
• Lack of random selection puts into question the
generalizability of the results
Solution:
• To argue for representativeness of the sample on a logical
basis
• The equivalence of groups can be checked statistically and if
not equivalent, the effect can be adjusted for.
Vafaee, A&HL5575 (Class 5)
14
Post-Test Only, Non-Equivalent, Multi-Group Design
G1(e)
G2(e)
X1
X2
O1
O2
RQ: What is the effect of explicit vocabulary instruction on lexical knowledge?
Non randomly Assigned
Types of Instruction
Dependent Variable
N = 20
Explicit vocabulary instruction
Post-test
RQ: What are the effects of explicit and incidental vocabulary teaching methods?
Non randomly Assigned
Types of Instruction
Dependent Variable
N = 20
Explicit vocabulary instruction
Post-test
N = 20
Incidental vocabulary instruction
Post-test
Vafaee, A&HL5575 (Class 5)
15
Post-Test Only, Non-Equivalent, Control Group Design
G1 (e)
G2 (e)
G3 (c)
X1
X2
-
O1
O2
O3
RQ: What is the effect of explicit vocabulary instruction on lexical knowledge?
Non randomly Assigned
Types of Instruction
Dependent Variable
N = 20
Explicit vocabulary instruction
Post-test
N = 20
-
Post-test
RQ: What are the effects of explicit and incidental vocabulary teaching methods?
Non randomly Assigned
Types of Instruction
Dependent Variable
N = 20
Explicit vocabulary instruction
Post-test
N = 20
Incidental vocabulary instruction
Post-test
N = 20
-
Post-test
Vafaee, A&HL5575 (Class 5)
16
Pretest-Posttest, Nonequivalent, Control Group Design
Pretest-Posttest, Nonequivalent, Multi-Group Design
Non randomly Assigned
Pretest
G1
O1
G2
G3
O3
O5
Gc or G4
O7
Types of treatment
Posttest 1 Posttest 2
X1
X2
O2
O9
O4
O10
X3
O6
O11
O8
O12
- Or X4
Use the pretest scores to:
1. check the similarity of the groups. If groups are not similar, the pretest scores can be used
to adjust the posttest scores statically.
2. generate gain scores.
Vafaee, A&HL5575 (Class 5)
17
Repeated Measures Design
G1
O1
O2
G1
G2
O1
O6
O2 X1 O3 O4 O5
O7 X2 O8 O9 O10
G1
G2
GC
O1 O2 X1 O3 O4 O5
O6 O7 X2 O8 O9 O10
O11 O12 - O13 O14 O15
G1
O1 X1a O2 X1b O3 X1c O4 O5
O6 X2a O7 X2b O8 X2c O9 O10
O11 - O12 - O13 - O14 O15
G2
GC
Vafaee, A&HL5575 (Class 5)
O3
X1 O4 O5 O6
18
Single-Subject Designs
Repeated measures of the dependent variable
Measurement must be highly standardized
Single variable rule
Traditional or normal condition is called the baseline
Baseline should be long enough for the dependent variable to gain stability
O1
Ok
O1
Ok
B
A
O1
Ok
A
Vafaee, A&HL5575 (Class 5)
O1
Ok
B
O1
Ok
A
19
Criteria for a well-designed experiment
1. Adequate Experimental Control: we control some variables (e.g.,
proficiency level) for meaningful interpretation of the results (to avoid
confounding); other variables can be controlled via randomization or by
building variables into the design.
2. Lack of artificiality: If we want to generalize to a non-experiment setting,
the results should apply to the real world. The artificial nature of the
experiment should not cause the experimental effects.
3. Adequate basis for comparison: there must be a way to determine the
experimental effect. The experiment may or need a control group (e.g., no
new treatment).
4. Precise data: the data must be reliable (e.g., the ratings from the 2 raters need
to be consistent); codings must have agreement; a test/ questionnaire should
had high Cronbach's alphas.
Vafaee, A&HL5575 (Class 5)
20
Criteria for a well-designed experiment
5. No contaminated data & no bias: no interactions between the data (e.g.,
raters must rate the data independently) (part of reliability); no interaction
between variables (raters are more consistent on some topics or with some
examinees)
6. No confounding of the data: no two variables contribute to the score in a
way that is difficult to separate their effect (proficiency and age) (validity
issue)
7. Sample representativeness: random selection to enhance generalizability
(of learners; also of items, tasks, contexts)
8. Parsimony: a simple research design/ underlying model is better than
complex one—when possible (!) BUT, the lack of complexity might not
represent reality.
Vafaee, A&HL5575 (Class 5)
21
Experimental Validity
1. Construct validity: The extent to which the constructs measured in the
independent or dependent variables are meaningful & appropriate as a
result of empirical supports.
2. Internal Validity: The extent to which the results of the experiment can be
attributed to the treatment. In other words, the extraneous variables are not
having an undue effect on the link between the independent & dependent
variable. They are being controlled--or are part of the design.
3. External Validity: The extent to which the results of the experiment can be
generalized to other populations (ESL/EFL, urban/rural), or conditions (class
size, computer-mediated/ face-to-face).
4. Statistical Conclusion Validity: The extent to the statistical procedures used
in the study are appropriate & meaningful for decision making. Are the
measures sufficiently reliable? Is the statistical procedure appropriate for the
data? Have the assumptions underlying the statistical procedure been met?
Vafaee, A&HL5575 (Class 5)
22
Threats to Construct Validity
1. Inadequate pre-operational explanation of constructs: Inadequate
theoretical & operation definition of the dependent or independent (i.e.,
how is “instruction” defined and operationalized/implemented by two
different teachers?)
2. Mono-operation bias: Only one form of experimental variable is
implemented (e.g., only written feedback is given, not oral)
3. Mono-method bias : Only one method is used to measure the dependent
variable (e.g., only a GJT)
4. Hypothesis guessing within the experimental conditions: Participants
behave differently when they know they’re part of an experiment
(Hawthorne Effect)
5. Confounding constructs & levels of constructs: Drawing conclusions
about constructs when some levels are absent.
Vafaee, A&HL5575 (Class 5)
23
Threats to Internal Validity
1. History: unanticipated events have occurred that affect performance on the
dependent variable.
2. Maturation: the participants mature/change (over time)
3. Testing/Pre-testing: the effects of test familiarity
4. Instrumentation/Measuring instruments: inconsistent use of a test (e.g.,
the instructions changed from the pre to the post-test)
5. Statistical regression: participants with extreme scores cause other scores
to regress toward the average score (e.g., with gain scores)
6. (Differential) selection (of a sample): no randomization, causing
unequivalent groups
7. (Experimental) Mortality: participants drop out
8. Interaction of selection & maturation, etc.: inconsistent maturation of
participants across groups (e.g., some students spent the summer in an
English speaking country)
Vafaee, A&HL5575 (Class 5)
24
Threats to External Validity
1. Interaction effect of pre-testing & the experimental variable : pretesting
interacts with the experimental variable causing ungeneralisable results.
2. Interaction effect of selection biases & experimental treatment: the effect
of choosing one group to give treatment to, which would produce a
different effect if the groups were randomized.
3. Reactive effect of experimental arrangements: the effect occurs when an
experiment takes place in a setting that would not produce the same effect
in different settings (generalizing across settings)
4. Multiple treatment interference: the carry over effect of participating
in multiple treatments (generalizing across time to the experimental
effects).
Vafaee, A&HL5575 (Class 5)
25
Threats to Statistical Conclusion Validity
1. Low statistical power: Sample size too small to detect differences between
groups
2. Violation of assumptions: failure to meet the assumptions or understand
the effect of violations (robust or not)
3. Fishing & error rate problem: Capitalizing on chance factor findings
4. Low reliability: Imprecise measures
Vafaee, A&HL5575 (Class 5)
26
