Read the article and answer the question

Read Kim (2021) and answer the following questions.

What are the independent and dependent variables?

Don't use plagiarized sources. Get Your Custom Essay on
Read the article and answer the question
Just from $13/Page
Order Essay

How was the dependent variable of the study operationalized and measured?

What are the constants?

  • What kind of experimental study is this?
  • Who are the participants and what were the sampling methods?
  • What are the language target forms? Why were two (rather than just one) target forms were used in this study?
  • Create a table and summarize all the statistical analyses conducted in the study. Explain why each of the analyses was done.
  • Did the study report reliability estimates and effect sizes?
  • What are some of the limitations of this study?
  • Vafaee, A&HL5575 (Class 2-Part A)
    2
    Vafaee, A&HL5575 (Class 2-Part A)
    3
    Source of Knowledge
    As Authority
    As Belief
    e.g., religion
    Doughty and Pica (1986)
    said that according to
    Long (1981), claims that
    such activities promote
    optimal conditions for
    students to adjust their
    input to each other’s level
    of comprehension … and
    thereby facilitate their L2
    acquisitions.
    Vafaee, A&HL5575 (Class 2-Part A)
    A priori
    Empirical
    Doughty and Pica (1986)
    said that in keeping with
    SLA theory, such
    modified interaction is
    claimed to make input
    comprehensible to learners
    and to lead ultimately to
    successful classroom
    acquisition.
    According to McDonald
    (1987), for all
    constructions tested, it was
    found that with increasing
    exposure, cue usage in L2
    gradually shifted from that
    appropriate to the L1 to
    that appropriate to the L2.
    4
    Identifying
    the problem
    Reviewing
    relevant
    information
    Vafaee, A&HL5575 (Class 2-Part A)
    Collecting
    data
    Analyzing
    data
    Drawing
    conclusions
    5
    Vafaee, A&HL5575 (Class 2-Part A)
    6
    Vafaee, A&HL5575 (Class 2-Part A)
    7
    Vafaee, A&HL5575 (Class 2-Part A)
    8
    Vafaee, A&HL5575 (Class 2-Part A)
    9
    Vafaee, A&HL5575 (Class 2-Part A)
    10
    Vafaee, A&HL5575 (Class 2-Part A)
    11
    Empirical Research
    Basic/Theoretical
    e.g., universals of relative
    clauses
    Vafaee, A&HL5575 (Class 2-Part A)
    Applied
    Practical
    e.g., order of acquisition
    of relative clauses in
    typologically similar and
    dissimilar languages from
    English
    e.g., evaluation of relative
    clause teaching materials
    12
    AL Research Types
    Secondary
    Library
    Research
    Primary/Empirical
    Lit. Review
    Qualitative
    Ethnography
    Identifying problems and
    reviewing information
    Discourse
    Analysis
    All Qualitative
    Vafaee, A&HL5575 (Class 2-Part A)
    Survey
    Interviews
    Questionnaires
    Qualitative or Quantitative
    Mixed Methods
    Statistical
    Descriptive
    Exploratory
    Quasiexperimental
    experimental
    All Quantitative
    13
    Qualitative
    Quantitative
    Concerned with understanding human
    behavior from the actor’s own frame of
    reference
    Seeks facts or causes/effects/relationships of
    social/psychological phenomena without
    regard to the subjective states of the
    individuals
    Observer-participant Interaction
    Detached role of researcher
    Subjective
    Objective
    Grounded, discovery-oriented, exploratory,
    descriptive.
    Ungrounded, verification- oriented,
    confirmatory, reductionist, inferential, and
    Hypothetical- deductive
    Naturalistic and uncontrolled observation
    Obtrusive and controlled measurement
    Holistic Inquiry
    Focused on individual variables
    Context-specific
    Context- free
    Inductive
    Hypothetical-deductive
    ‘real’, ‘rich’, and ‘deep’ data; ungeneralizable
    ‘hard’ and replicable data ; generalizable
    Narrative description
    Statistical analysis
    Process- oriented
    Outcome- oriented
    Vafaee, A&HL5575 (Class 2-Part A)
    14
    AL Research Type
    Data Collection
    Method
    Type of Collected
    Data
    Type of Data
    Analysis
    Experimentally vs. Nonexperimentally
    Qualitative vs.
    Quantitative
    Statistical vs.
    Interpretative
    Paradigm 1:
    exploratory-interpretative
    Paradigm 3: experimentalqualitative-interpretative
    Paradigm 5: exploratory
    qualitative-statistical
    Paradigm 7: exploratoryquantitative-interpretive
    Non-experimental design
    Experimental design
    Non-experimental design
    Non-experimental design
    Qualitative data
    Qualitative data
    Qualitative data
    Quantitative data
    Interpretative analysis
    Interpretative analysis
    Statistical analysis
    Interpretative analysis
    Paradigm 2:
    analytical-nomological
    Paradigm 4: experimentalqualitative-statistical
    Paradigm 6: exploratoryquantitative-statistical
    Paradigm 8: experimentalquantitative-interpretive
    Experimental design
    Experimental design
    Non-experimental design
    Experimental design
    Quantitative data
    Qualitative data
    Quantitative data
    Quantitative data
    Statistical analysis
    Statistical analysis
    Statistical analysis
    Interpretative analysis
    Vafaee, A&HL5575 (Class 2-Part A)
    15
    Session 3: Quantitative Research in AL
    Research Literacy (A&HL 5575)
    Quantitative Research in AL
    AL Research Types
    Secondary
    Qualitative
    e.g., Ethnography
    Primary/Empirical
    Quantitative
    e.g., Critical
    Discourse Analysis
    Correlational/
    associational
    (Quasi)
    experimental
    Vafaee, A&HL5575 (Class 3)
    2
    Brief Historical Overview
    ! Quantitative social research was inspired by the progress of natural
    sciences in the 19th century.
    ! Social research started adopting the “scientific method”.
    ! Scientific method postulates three key stages in research:
    1) observing a phenomenon or identifying a problem;
    2) generating an initial hypothesis;
    3) testing the hypothesis by using empirical data.
    ! The scientific method is closely associated with numerical values,
    and statistics became a subdiscipline of math by the end of the 19th
    century.
    ! Major developments both in scientific method (e.g., the works of
    Karl Popper) and statistics (e.g., Spearman and Pearson) in the first
    half of the 20th century.
    Vafaee, A&HL5575 (Class 3)
    3
    Brief Historical Overview
    ! Thanks to the progress in psychometrics, quantitative methodology
    became the dominant research methods in social sciences in the
    middle of 20th century.
    ! The hegemony started to change in the 1970s by the challenges
    posed by qualitative research.
    ! Currently, in many areas of social sciences, the two methods of
    have peaceful coexistence.
    ! In Applied Linguistics, there was a significant increase in the
    number of quantitative research between 1970 to 1985.
    ! Quantitative methods still maintain the dominance although
    qualitative research method is gaining a fast momentum.
    ! Out of 524 empirical studies published between 1991 to 2001, 86%
    were quantitative, 13% were qualitative, and 1% was mixed method.
    Vafaee, A&HL5575 (Class 3)
    4
    Main Characteristics of Quantitative Research
    ! Using numbers
    ! A priori categorization
    ! Variables rather than cases
    ! Statistics and the language of statistics
    ! Standardized procedures to assess objective reality
    ! Quest for generalizability and universal laws
    Vafaee, A&HL5575 (Class 3)
    5
    Strengths of Quantitative Research
    ! Systematic
    ! Focused
    ! Tightly controlled
    ! Precise measurement with reliable and replicable data
    ! Generalizable to other contexts
    ! The statistical analytical apparatus can be evaluated easily
    ! Relatively quick and less costly
    6
    Vafaee, A&HL5575 (Class 3)
    Weaknesses of Quantitative Research
    ! Impossible to do justice to the subjective variety of individuals
    ! Not always sensitive to the underlying processes
    ! Can be overly simplistic, decontextualized and reductionist
    Vafaee, A&HL5575 (Class 3)
    7
    Research Design and Variance
    ! Research design is a plan for answering research questions by telling
    us how to explain variance in the outcome variables of interest.
    ! In other words, the research design helps us to identify the sources
    that contribute to the variance in the outcome variables of interest.
    ! Or, the research design helps us to identify the main independent
    variables that contribute to the variance in the depended variable(s).
    ! Research design also helps us to identify other sources that
    contribute to the variance in the depended variable(s).
    ! In our research design we can control for the effect of these
    variables.
    Vafaee, A&HL5575 (Class 3)
    8
    Research Design and Variance
    ! To have valid conclusions, research designs include more control
    variables.
    ! In other words, the researcher should think about both the main
    independent variables and intervening and/or moderating variables.
    The effect of intervening and/or moderating variables should be
    controlled; otherwise, they will act as confounding variables.
    ! However, no matter how many independent variables are included in
    a research design, still there will be some confounding variables.
    ! The researcher should be aware of as many confounding variables as
    possible and acknowledges the limitations of her/his findings.
    Vafaee, A&HL5575 (Class 3)
    9
    Research Design Types
    ! Correlational (Associational) Research
    ” The goal of associational research is to determine whether a
    relationship exists between variables and, if so, the strength of that
    relationship.
    ” This is often tested statistically through correlations, which allow a
    researcher to determine how closely two variables (e.g., motivation and
    language ability) are related in a given population.
    ” Associational research is not concerned with causation, only with co-
    occurrence.
    Vafaee, A&HL5575 (Class 3)
    10
    Research Design Types
    ! Correlational (Associational) Research
    ” Correlation can be used in different ways: for example, to test a
    relationship between or among variables, and to make predictions.
    ” Predictions are dependent on the outcome of a strong relationship
    between or among variables. That is, if variables are strongly related,
    we can often predict the likelihood of the presence of one from the
    presence of the other(s).
    Vafaee, A&HL5575 (Class 3)
    11
    Research Design Types
    ! Experimental and Quasi-Experimental Research
    ” In experimental studies, researchers deliberately manipulate one or
    more variables (independent variables) to determine the effect on
    another variable (dependent variable).
    ” This manipulation is usually described as a treatment and the
    researcher’s goal is to determine whether there is a causal relationship.
    ” Many types of experimental research involve a comparison of
    pretreatment and posttreatment performance.
    ” Randomization is usually viewed as one of the hallmarks of
    experimental research.
    ” Design types can range from truly experimental (with random
    assignment) to what is known as quasi-experimental (without random
    assignment).
    Vafaee, A&HL5575 (Class 3)
    12
    Research Design Types
    ! Experimental and Quasi-Experimental Research
    ” A typical experimental study usually uses comparison or control groups
    to investigate research questions.
    ” Many second language research studies involve a comparison between
    two or more groups.
    ” This is known as a between-groups design.
    ” This comparison can be made in one of two ways: two or more groups
    with different treatments; or two or more groups, one of which, the
    control group, receives no treatment.
    Vafaee, A&HL5575 (Class 3)
    13
    Session 4: Experimental Research Design_Part 1
    Research Literacy (A&HL 5575)
    Experimental Research
    ! Experimental research is the way of determining the effect of something
    on something else.
    ! In experimental research, we manipulate at least one variable, while
    controlling for the effect of other variables, to determine the effect of
    manipulation on the outcome variable.
    ! Example: Whether focusing a learner’s attention on some aspect of language increases
    that individual’s uptake of that aspect of language.
    ! Experimental research is based on the Rationalist worldview.
    ” Rationalist approach is a theory-then-research or deductive approach.
    ! Research questions must be stated explicitly and must have some basis
    on previous literature.
    ! Example: Does focused attention on noun-adjective agreement in Italian promote
    learning to a greater extend than focused attention on wh-movement in Italian for
    beginning learners of Italian?
    ! There are always hypotheses about the results of experiments.
    ! Example: Focused attention on noun-adjective agreement in Italian will promote
    learning to a greater extend than focused attention on wh-movement in Italian for
    beginning learners of Italian.
    Vafaee, A&HL5575 (Class 4)
    2
    Explaining and Controlling Variance
    !
    Experimental research is conducted for the purpose of explaining or controlling variance.
    Independent
    Variable
    Dependent Variable
    Group 1
    Method 1
    Group 1
    T-Scores 1
    Group 2
    Method 2
    Group 2
    T-Scores 2
    Group 3
    Method 3
    Group 3
    T-Scores 3
    Variance in
    scores
    Controlling variance allows us to minimize the effects of extraneous variance. As a result, the
    dependent variable can be interpreted without bias.
    ! Construct-relevant variance (Good variability or systematic variance).
    !
    ! Variance related to the construct being investigated (e.g., variance in scores related to the
    effect of method).
    !
    Construct-irrelevant variance (Bad variability or error/unsystematic or unwanted variance).
    ! Variance unrelated to the construct being investigated coming from factors other than the
    effect of instruction (e.g., pre-existing difference in ability level)
    Vafaee, A&HL5575 (Class 4)
    3
    Five Ways of Controlling the Unwanted Variance
    1. Randomization
    2. Holding conditions or factors constant (e.g., controlling for
    proficiency, only one teacher)
    3. Statistical adjustments (adjusting pretest-posttest differences in
    gain scores–ANCOVA)
    4. Building conditions or factors into the research design as
    independent variables (e.g., incorporating proficiency level
    the design—advanced/intermediate/beginners)
    into
    5. The combination of all four methods
    Vafaee, A&HL5575 (Class 4)
    4
    1. Randomization
    Unsystematic
    variance
    Systematic
    variance
    Random, error,
    unsystematic
    variance due
    to other factors
    (ability level,
    motivation,
    anxiety, etc.)
    Variance due
    to methods
    Total variance
    Vafaee, A&HL5575 (Class 4)
    5
    1. Randomization
    Independent
    Variable
    Dependent Variable
    G1: 20 SS
    Method 1
    Group 1
    T-Scores 1
    G2: 20 SS
    Method 2
    Group 2
    T-Scores 2
    G3: 20 SS
    Method 3
    Group 3
    T-Scores 3
    Variance in
    scores
    60 SS randomly assigned
    to each of the 3 groups
    ! It equalizes the groups with respects to variables other than the main
    independent variable.
    ! It aims at reducing the effect of “random”, “inherent” or “ within
    groups” variance.
    Vafaee, A&HL5575 (Class 4)
    6
    2. Holding Factors Constant
    Independent
    Variable
    Same
    Teacher
    Dependent Variable
    G1: 20 SS
    Method 1
    Group 1
    T-Scores 1
    G2: 20 SS
    Method 2
    Group 2
    T-Scores 2
    G3: 20 SS
    Method 3
    Group 3
    T-Scores 3
    Variance in
    scores
    60 SS randomly assigned
    to each of the 3 groups
    ! Disadvantage:
    ! Reduces external validity
    Vafaee, A&HL5575 (Class 4)
    7
    3. Statistical Control
    ! The effect of pre-existing variables is removed statistically.
    Independent
    Variable
    Same
    Teacher
    G1: 20 SS
    Method 1
    Group 1
    T-Scores 1
    G2: 20 SS
    Method 2
    Group 2
    T-Scores 2
    G3: 20 SS
    Method 3
    Group 3
    T-Scores 3
    60 SS randomly assigned
    to each of the 3 groups
    Vafaee, A&HL5575 (Class 4)
    Dependent Variable
    Variance in
    scores
    Proficiency
    scores from
    all 60 SS
    Transformed
    test scores or
    dependent
    variable
    Variance in
    scores
    8
    4. Building Factors into the Research Design
    Independent Variable
    Same
    Teacher
    Vafaee, A&HL5575 (Class 4)
    Group 1
    Group 2
    Group 3
    Group 4
    Group 5
    Group 6
    10 SS- Low
    10 SS- High
    10 SS- Low
    10 SS- High
    10 SS- Low
    10 SS- High
    Method 1
    Method 2
    Method 3
    60 SS randomly assigned
    to each of the 6 groups
    Dependent Variable
    Group 1
    Group 2
    Group 3
    Group 4
    Group 5
    Group 6
    T-scores 1
    T-scores 2
    T-scores 3
    T-scores 4
    T-scores 5
    T-scores 6
    Variance in
    scores
    9
    4. Building Factors into the Research Design
    Unsystematic
    variance
    Variance due
    to ability
    level
    Systematic
    variance
    Variance due
    to methods
    Variance due
    to other
    factors
    (motivation,
    anxiety, etc.)
    Vafaee, A&HL5575 (Class 4)
    Total variance
    10
    5. Using Four Ways in Combination
    Independent Variable
    Dependent Variable
    20 SS- Low
    20 SS- High
    20 SS- Low
    20 SS- High
    20 SS- Low
    20 SS- High
    20 SS- Low
    20 SS- High
    20 SS- Low
    20 SS- High
    20 SS- Low
    20 SS- High
    T-scores
    T-scores
    T-scores
    T-scores
    T-scores
    T-scores
    T-scores
    T-scores
    T-scores
    T-scores
    T-scores
    T-scores
    Teacher 1
    Teacher 2
    Same School
    Teacher 1
    Teacher 2
    Teacher 1
    Teacher 2
    Method 1
    Method 1
    Method 2
    Method 2
    Method 3
    Method 3
    WM scores
    for all 120 SS
    Transformed
    test scores or
    dependent
    variable
    120 SS randomly assigned
    to each of the 6 groups
    Variance in
    scores
    Vafaee, A&HL5575 (Class 4)
    11
    Working Example
    !
    Which kind of corrective feedback is more effective? Explicit or implicit?


    !
    Which of the two corrective feedback types of metalinguistic explanation and recast is more
    beneficial?

    !
    What morphosyntactic target structures? ➔
    easy or difficult
    Who are the learners? Children, young adults or adults
    Which of the two corrective feedback types of metalinguistic explanation and recast is more
    beneficial for learning the English article system and the past perfect tense among adult
    learners?

    !
    morphosyntax, pronunciation, collocation
    Which of the two corrective feedback types of metalinguistic explanation and recast is more
    beneficial for learning the English article system and the past perfect tense?

    !
    Learning what? ➔
    Which of the two corrective feedback types of metalinguistic explanation and recast is more
    beneficial for learning morphosyntax?

    !
    Explicit like metalinguistic explanation
    Implicit like recast
    What is the setting? Classroom learning or naturalistic learning?
    Which of the two corrective feedback types of metalinguistic explanation and recast is more
    beneficial for learning the English article system and the past perfect tense among adult
    learners who learn English in the formal classroom learning setting?
    Vafaee, A&HL5575 (Class 4)
    12
    What are the Variables in the Working Example?
    X
    Variables
    Constants
    Main Independent
    Intervening
    Manipulated or predictor
    How and why
    Metalinguistic explanation
    and recast
    Dependent/predicted
    Posttest on knowledge of articles and tense
    Underlying mechanism
    Moderating
    Age and motivation
    Noticing the feedback, so
    individual differences in WM
    capacity and L2 aptitude
    Vafaee, A&HL5575 (Class 4)
    13
    Working Example
    X
    Variables
    Constants
    Main Independent
    Intervening
    Dependent/predicted
    How and why
    Manipulated or predictor
    Posttest on knowledge of articles and tense
    Control
    Binary/classifying
    Moderating
    Underlying mechanism
    Metalinguistic explanation
    and recast
    Learning setting, age and motivation
    Continuous
    Noticing the feedback, so
    individual differences in WM
    capacity
    No Control
    Adults
    Classroom Setting
    Vafaee, A&HL5575 (Class 4)
    Measures of WM and Motivation
    Statistically (Co-variate)
    Extraneous or
    Confounding
    14
    Amount of exposure outside the experimental
    setting, stress level, how the feedback is delivered,
    length of experiment, proficiency level, first
    language, previous knowledge
    Working Example
    !
    Controlling for individual differences in WM and motivation, which of the two corrective
    feedback types of metalinguistic explanation and recast is more beneficial for learning the English
    article system and the past perfect tense among adult EFL learners at the intermediate level?
    Independent Variable
    Group 1
    Article
    Past Tense
    Same
    Teacher
    Group 2
    Dependent Variable
    Recast
    Group 1
    Article
    Past Tense
    Article Test
    PP Test
    Article Test
    Meta_ L_ E
    Group 2
    Select 100 adult EFL learners
    from the intermediate level
    and assign them randomly to
    the two groups
    Motivation
    and WM
    scores
    PP Test
    Variance in
    scores
    Transformed
    test scores or
    dependent
    variable
    Variance in
    scores
    Vafaee, A&HL5575 (Class 4)
    15
    Working Example
    !
    Controlling for individual differences in WM and motivation, which of the two corrective feedback types of
    metalinguistic explanation and recast is more beneficial for learning the English article system and the past
    perfect tense among adult learners who learn English in the formal classroom learning setting?
    Amount of exposure outside the experimental
    setting, stress level, how the feedback is delivered,
    length of experiment, proficiency level, first
    language, previous knowledge
    Extraneous or Confounding
    Let’s control them
    Amount of exposure and practice outside the experimental setting
    Constant: Run a single-session experiment
    Statistical: Include a questionnaire
    Stress level
    Statistical: Include a questionnaire
    How the feedback is delivered
    Constant: use the same materials and
    experimenter
    Length of experiment
    proficiency level
    Constant and statistical: make the
    experiment long and include several
    posttests (posttests and delayed posttests)
    Constant: just the intermediate
    Statistical: Include a test
    First Language
    Constant: just one first language
    Statistical: Include a questionnaire
    Previous knowledge
    Constant: pretest and screen
    16
    Statistical: pretest and a covariate
    What about little differences in previous knowledge or all the other
    variables we are aware of?
    Randomization
    Working Example
    !
    What kind of corrective feedback is more useful?
    !
    Controlling for individual differences in WM and motivation, which of the two corrective
    feedback types of metalinguistic explanation and recast is more beneficial for learning the English
    article system and the past perfect tense among adult learners who learn English in the formal
    classroom learning setting?
    !
    In a true (vs. quasi) experiment with a pretest, immediate posttest and delayed posttest design, the
    effectiveness of metalinguistic explanation versus recast for correcting errors related to the use of
    English articles and past perfect tense was studied. This study was carried out among Persian
    learners of English who were all above 18 years old. These participants are learning English in an
    EFL context and rarely have exposure to English out of the classroom setting. Also, during the
    length of experiment which lasted for ten one-hour sessions, and by the time they took the delayed
    posttest, which was one month after the last session of the experiment, these participants had no
    practice of English outside of the experimental setting. After each of the experimental sessions, the
    leaners took a posttest, so ten posttests were included in the study. In this study, to control for the
    effect of motivation and stress on learning, questionnaires for both of these psychological variables
    were included and their results were used as covariates in the statistical analyses. Also, the effect
    of individual differences in WM on the learning outcomes were accounted for statistically.
    Vafaee, A&HL5575 (Class 4)
    17
    Nonexperimental Quantitative Research
    Ex Post Facto
    One-group Posttest Only
    Observational Approach
    Explanatory
    Predictive
    Survey Approach
    Cross-sectional
    Trend
    Vafaee, A&HL5575 (Class 6)
    Longitudinal
    Cohort
    Panel
    2
    Vafaee, A&HL5575 (Class 6)
    3
    Vafaee, A&HL5575 (Class 6)
    4
    Ex Post:
    Cons: Post test only, no control group may fact internal validity, no random assign
    Pros:
    Vafaee, A&HL5575 (Class 6)
    5
    N= 34 Study abroad group
    N= 26 Study at home group
    Study
    abroad
    Vafaee, A&HL5575 (Class 6)
    6
    Vafaee, A&HL5575 (Class 6)
    7
    Vafaee, A&HL5575 (Class 6)
    8
    Vafaee, A&HL5575 (Class 6)
    9
    Vafaee, A&HL5575 (Class 6)
    10
    Vafaee, A&HL5575 (Class 6)
    11
    Vafaee, A&HL5575 (Class 6)
    12
    Vafaee, A&HL5575 (Class 6)
    13
    Vafaee, A&HL5575 (Class 6)
    14
    Planning
    Sampling
    Instrumentation
    Identifying the research problem
    Identifying the variables
    Literature Review
    Operational definition the variables
    Designing the study
    Defining the population and sub-groups
    Detailed sampling procedure
    Sampling
    Identifying existing instruments
    Constructing tests and questionnaires
    Designing interviews
    Piloting the instruments
    Data Collection
    Measurement administration
    Inter-rater and intra-rater reliability
    Data Preparation
    Data coding
    Inter-rater and intra-rater reliability
    Data set up
    Data Analysis
    Checking the measurement quality
    Checking the assumptions
    Conducting the analysis
    Interpretation
    Vafaee, A&HL5575 (Class 6)
    Avoid going beyond data
    15
    Vafaee, A&HL5575 (Class 6)
    16
    Vafaee, A&HL5575 (Class 7)
    2
    Vafaee, A&HL5575 (Class 7)
    3
    Vafaee, A&HL5575 (Class 7)
    4
    Strata
    Strata Size
    Proportional Sample size by strata
    B1 & B2
    50
    50/2= 25
    B3 & B4
    100
    100/2= 50
    I1 & I2
    150
    150/2= 75
    I3 & I4
    60
    60/2= 30
    Advanced
    40
    40/2= 20
    N= 400
    N= 200
    Vafaee, A&HL5575 (Class 7)
    5
    Population: 15
    advanced
    classes
    Random Selection of classes
    Sample
    of 5
    classes
    All members of
    each class get
    measured
    Generalization of results based on logical basis
    Vafaee, A&HL5575 (Class 7)
    6
    Vafaee, A&HL5575 (Class 7)
    7
    Vafaee, A&HL5575 (Class 7)
    8
    Vafaee, A&HL5575 (Class 7)
    9
    Vafaee, A&HL5575 (Class 7)
    10
    Vafaee, A&HL5575 (Class 2-Part A)
    2
    Vafaee, A&HL5575 (Class 2-Part A)
    3
    Source of Knowledge
    As Authority
    As Belief
    e.g., religion
    Doughty and Pica (1986)
    said that according to
    Long (1981), claims that
    such activities promote
    optimal conditions for
    students to adjust their
    input to each other’s level
    of comprehension … and
    thereby facilitate their L2
    acquisitions.
    Vafaee, A&HL5575 (Class 2-Part A)
    A priori
    Empirical
    Doughty and Pica (1986)
    said that in keeping with
    SLA theory, such
    modified interaction is
    claimed to make input
    comprehensible to learners
    and to lead ultimately to
    successful classroom
    acquisition.
    According to McDonald
    (1987), for all
    constructions tested, it was
    found that with increasing
    exposure, cue usage in L2
    gradually shifted from that
    appropriate to the L1 to
    that appropriate to the L2.
    4
    Identifying
    the problem
    Reviewing
    relevant
    information
    Vafaee, A&HL5575 (Class 2-Part A)
    Collecting
    data
    Analyzing
    data
    Drawing
    conclusions
    5
    Vafaee, A&HL5575 (Class 2-Part A)
    6
    Vafaee, A&HL5575 (Class 2-Part A)
    7
    Vafaee, A&HL5575 (Class 2-Part A)
    8
    Vafaee, A&HL5575 (Class 2-Part A)
    9
    Vafaee, A&HL5575 (Class 2-Part A)
    10
    Vafaee, A&HL5575 (Class 2-Part A)
    11
    Empirical Research
    Basic/Theoretical
    e.g., universals of relative
    clauses
    Vafaee, A&HL5575 (Class 2-Part A)
    Applied
    Practical
    e.g., order of acquisition
    of relative clauses in
    typologically similar and
    dissimilar languages from
    English
    e.g., evaluation of relative
    clause teaching materials
    12
    AL Research Types
    Secondary
    Library
    Research
    Primary/Empirical
    Lit. Review
    Qualitative
    Ethnography
    Identifying problems and
    reviewing information
    Discourse
    Analysis
    All Qualitative
    Vafaee, A&HL5575 (Class 2-Part A)
    Survey
    Interviews
    Questionnaires
    Qualitative or Quantitative
    Mixed Methods
    Statistical
    Descriptive
    Exploratory
    Quasiexperimental
    experimental
    All Quantitative
    13
    Qualitative
    Quantitative
    Concerned with understanding human
    behavior from the actor’s own frame of
    reference
    Seeks facts or causes/effects/relationships of
    social/psychological phenomena without
    regard to the subjective states of the
    individuals
    Observer-participant Interaction
    Detached role of researcher
    Subjective
    Objective
    Grounded, discovery-oriented, exploratory,
    descriptive.
    Ungrounded, verification- oriented,
    confirmatory, reductionist, inferential, and
    Hypothetical- deductive
    Naturalistic and uncontrolled observation
    Obtrusive and controlled measurement
    Holistic Inquiry
    Focused on individual variables
    Context-specific
    Context- free
    Inductive
    Hypothetical-deductive
    ‘real’, ‘rich’, and ‘deep’ data; ungeneralizable
    ‘hard’ and replicable data ; generalizable
    Narrative description
    Statistical analysis
    Process- oriented
    Outcome- oriented
    Vafaee, A&HL5575 (Class 2-Part A)
    14
    AL Research Type
    Data Collection
    Method
    Type of Collected
    Data
    Type of Data
    Analysis
    Experimentally vs. Nonexperimentally
    Qualitative vs.
    Quantitative
    Statistical vs.
    Interpretative
    Paradigm 1:
    exploratory-interpretative
    Paradigm 3: experimentalqualitative-interpretative
    Paradigm 5: exploratory
    qualitative-statistical
    Paradigm 7: exploratoryquantitative-interpretive
    Non-experimental design
    Experimental design
    Non-experimental design
    Non-experimental design
    Qualitative data
    Qualitative data
    Qualitative data
    Quantitative data
    Interpretative analysis
    Interpretative analysis
    Statistical analysis
    Interpretative analysis
    Paradigm 2:
    analytical-nomological
    Paradigm 4: experimentalqualitative-statistical
    Paradigm 6: exploratoryquantitative-statistical
    Paradigm 8: experimentalquantitative-interpretive
    Experimental design
    Experimental design
    Non-experimental design
    Experimental design
    Quantitative data
    Qualitative data
    Quantitative data
    Quantitative data
    Statistical analysis
    Statistical analysis
    Statistical analysis
    Interpretative analysis
    Vafaee, A&HL5575 (Class 2-Part A)
    15
    Measures of Central Tendency
    Measures of Dispersion
    Reliability
    9
    Reliability
    10
    When two sets of values go up or down together, the correlation is positive
    When one set of values go up but the other go down, the correlation is negative
    Vafaee, A&HL4088 (Class 8)
    11
    Vafaee, A&HL4088 (Class 8)
    12
    An independent-samples t-test was run to determine if there were differences in engagement to a lesson
    between males and females. The lesson was more engaging to male viewers (M = 5.56, SD = 0.35) than
    female viewers (M = 5.30, SD = 0.35), a statistically significant difference, M = 0.26, 95% CI [0.04,
    0.48], t(38) = 2.365, p = .023.
    Vafaee, A&HL4088 (Class 8)
    13
    A paired-samples t-test showed that the carbohydrate-protein drink elicited a statistically
    significant increase in distance run in two hours compared to a carbohydrate-only drink, t(19) =
    6.352, p < .001. Vafaee, A&HL4088 (Class 8) 14 Vafaee, A&HL4088 (Class 8) 15 A one-way ANOVA was conducted to determine if the ability to cope with workplace-related stress (CWWS score) was different for groups with different physical activity levels. Participants were classified into four groups: sedentary (n = 7), low (n = 9), moderate (n = 8) and high levels of physical activity (n = 7). CWWS score was statistically significantly different between different physical activity groups, F(3, 27) = 8.316, p < .0005. CWWS score increased from the sedentary (M = 4.15, SD = 0.77) to the low (M = 5.88, SD = 1.69), moderate (M = 7.12, SD = 1.57) and high (M = 7.51, SD = 1.24) physical activity groups, in that order. Tukey post hoc analysis revealed that the mean increase from sedentary to moderate was statistically significant (2.97, p = .002), as well as the increase from sedentary to high (3.35, p = .001), but no other group differences were statistically significant. Vafaee, A&HL4088 (Class 8) 16 Vafaee, A&HL4088 (Class 8) 17 After adjustment for pre-test scores, the results of an ANCOVA showed that there was a statistically significant difference in post-test scores, F(2, 41) = 105.512, p < .001. Vafaee, A&HL4088 (Class 8) 18 Vafaee, A&HL4088 (Class 8) 19 Vafaee, A&HL4088 (Class 8) 20 APA: Students in Schools A, B and C scored higher in their English exam (M = 75.6, SD = 8.2; M = 63.6, SD = 6.6 and M = 59.8, SD = 4.6, respectively) than their math exam (M = 43.9, SD = 8.5; M = 40.8, SD = 8.2 and M = 30.8, SD = 7.7, respectively). Vafaee, A&HL5199 (Classes 11 & 12) 21 Although Wilks' Lambda is usually recommended, Pillai's Trace is more robust and is recommended when you have unequal sample sizes and also have a statistically significant Box's M result APA: There was a statistically significant difference between the schools on the combined dependent variables, F(4, 112) = 17.675, p < .0005; Wilks' Λ = .376; partial η2 = .387. Vafaee, A&HL5199 (Classes 11 & 12) 22 APA: A one-way multivariate analysis of variance was run to determine the effect of students' previous schooling on academic performance. Two measures of academic performance were assessed: English and math's end-of-year exam scores. Students arrived from three previous schools: School A, School B and School C. Preliminary assumption checking revealed that data was normally distributed, as assessed by Shapiro-Wilk test (p > .05); there were no univariate
    or multivariate outliers, as assessed by boxplot and Mahalanobis distance (p > .001),
    respectively; there were linear relationships, as assessed by scatterplot, no multicollinearity
    (r = .393, p = .002); and there was homogeneity of variance-covariance matrices, as assessed
    by Box’s M test (p = .003). Students in Schools A, B and C scored higher in their English exam
    (M = 75.6, SD = 8.2; M = 63.6, SD = 6.6 and M = 59.8, SD = 4.6, respectively) than their math
    exam (M = 43.9, SD = 8.5; M = 40.8, SD = 8.2 and M = 30.8, SD = 7.7, respectively). The
    differences between the schools on the combined dependent variables was not statistically
    significant, F(4, 112) = 1.254, p =.365; Wilks’ Λ = .041; partial η2 = .005.
    Vafaee, A&HL5199 (Classes 11 & 12)
    23
    To determine which dependent variable would appear to be contributing to the statistically
    significant MANOVA, you can inspect the one-way ANOVA result for each dependent variable.
    These results are contained within the Tests of Between-Subjects Effects table, as shown
    below:
    APA: There was a statistically significant difference in English exam scores between the
    students from different previous schools, F(2, 57) = 30.875, p < .001; partial η2 = .520. APA: There was a statistically significant difference in math exam scores between the students from different previous schools, F(2, 57) = 14.295, p < .001; partial η2 = .334. Vafaee, A&HL5199 (Classes 11 & 12) 24 For any of your univariate ANOVAs that are statistically significant, you can follow them up with Tukey post-hoc tests (or another multiple comparison procedure of your choosing). For example, if you had violated the assumption of homogeneity of variances, you might prefer to run a Games-Howell post-hoc test. Vafaee, A&HL5199 (Classes 11 & 12) 25 A one-way multivariate analysis of variance was run to determine the effect of students' previous schooling on academic performance. Two measures of academic performance were assessed: English and math's end-of-year exam scores. Students arrived from three previous schools: School A, School B and School C. Preliminary assumption checking revealed that data was normally distributed, as assessed by Shapiro-Wilk test (p > .05); there were no univariate
    or multivariate outliers, as assessed by boxplot and Mahalanobis distance (p > .001),
    respectively; there were linear relationships, as assessed by scatterplot; no multicollinearity
    (r = .393, p = .002); and there was homogeneity of variance-covariance matrices, as assessed
    by Box’s M test (p = .003). Students in Schools A, B and C scored higher in their English exam
    (M = 75.6, SD = 8.2; M = 63.6, SD = 6.6 and M = 59.8, SD = 4.6, respectively) than their math
    exam (M = 43.9, SD = 8.5; M = 40.8, SD = 8.2 and M = 30.8, SD = 7.7, respectively). The
    differences between the schools on the combined dependent variables was statistically
    significant, F(4, 112) = 17.675, p < .001; Wilks' Λ = .376; partial η2 = .387. Follow-up univariate ANOVAs showed that both English scores (F(2, 57) = 30.875, p < .001; partial η2 = .520) and math scores (F(2, 57) = 14.295, p < .001; partial η2 = .334.) were statistically significantly different between the students from different previous schools, using a Bonferroni adjusted α level of .025. Tukey post-hoc tests showed that for English scores, students from School A had statistically significantly higher mean scores than pupils from either School B (p < .001) or School C (p < .001), but not between School B and School C (p = .169). For math scores, Tukey post-hoc tests showed that School C had statistically significantly lower mean scores than students from either School A (p < .001) or School B (p = .001). Vafaee, A&HL5199 (Classes 11 & 12) 26 Vafaee, A&HL4088 (Class 8) 27 A multiple regression was run to predict health from gender, age, weight and heart rate. The multiple regression model statistically significantly predicted health, F(4, 95) = 32.393, p < .001, adj. R2 = .56. All four variables added statistically significantly to the prediction, p < .05. Regression coefficients and standard errors can be found in Table 1 (below). Vafaee, A&HL4088 (Class 8) 28 Session 3: Quantitative Research in AL Research Literacy (A&HL 5575) Quantitative Research in AL AL Research Types Secondary Qualitative e.g., Ethnography Primary/Empirical Quantitative e.g., Critical Discourse Analysis Correlational/ associational (Quasi) experimental Vafaee, A&HL5575 (Class 3) 2 Brief Historical Overview ! Quantitative social research was inspired by the progress of natural sciences in the 19th century. ! Social research started adopting the “scientific method”. ! Scientific method postulates three key stages in research: 1) observing a phenomenon or identifying a problem; 2) generating an initial hypothesis; 3) testing the hypothesis by using empirical data. ! The scientific method is closely associated with numerical values, and statistics became a subdiscipline of math by the end of the 19th century. ! Major developments both in scientific method (e.g., the works of Karl Popper) and statistics (e.g., Spearman and Pearson) in the first half of the 20th century. Vafaee, A&HL5575 (Class 3) 3 Brief Historical Overview ! Thanks to the progress in psychometrics, quantitative methodology became the dominant research methods in social sciences in the middle of 20th century. ! The hegemony started to change in the 1970s by the challenges posed by qualitative research. ! Currently, in many areas of social sciences, the two methods of have peaceful coexistence. ! In Applied Linguistics, there was a significant increase in the number of quantitative research between 1970 to 1985. ! Quantitative methods still maintain the dominance although qualitative research method is gaining a fast momentum. ! Out of 524 empirical studies published between 1991 to 2001, 86% were quantitative, 13% were qualitative, and 1% was mixed method. Vafaee, A&HL5575 (Class 3) 4 Main Characteristics of Quantitative Research ! Using numbers ! A priori categorization ! Variables rather than cases ! Statistics and the language of statistics ! Standardized procedures to assess objective reality ! Quest for generalizability and universal laws Vafaee, A&HL5575 (Class 3) 5 Strengths of Quantitative Research ! Systematic ! Focused ! Tightly controlled ! Precise measurement with reliable and replicable data ! Generalizable to other contexts ! The statistical analytical apparatus can be evaluated easily ! Relatively quick and less costly 6 Vafaee, A&HL5575 (Class 3) Weaknesses of Quantitative Research ! Impossible to do justice to the subjective variety of individuals ! Not always sensitive to the underlying processes ! Can be overly simplistic, decontextualized and reductionist Vafaee, A&HL5575 (Class 3) 7 Research Design and Variance ! Research design is a plan for answering research questions by telling us how to explain variance in the outcome variables of interest. ! In other words, the research design helps us to identify the sources that contribute to the variance in the outcome variables of interest. ! Or, the research design helps us to identify the main independent variables that contribute to the variance in the depended variable(s). ! Research design also helps us to identify other sources that contribute to the variance in the depended variable(s). ! In our research design we can control for the effect of these variables. Vafaee, A&HL5575 (Class 3) 8 Research Design and Variance ! To have valid conclusions, research designs include more control variables. ! In other words, the researcher should think about both the main independent variables and intervening and/or moderating variables. The effect of intervening and/or moderating variables should be controlled; otherwise, they will act as confounding variables. ! However, no matter how many independent variables are included in a research design, still there will be some confounding variables. ! The researcher should be aware of as many confounding variables as possible and acknowledges the limitations of her/his findings. Vafaee, A&HL5575 (Class 3) 9 Research Design Types ! Correlational (Associational) Research " The goal of associational research is to determine whether a relationship exists between variables and, if so, the strength of that relationship. " This is often tested statistically through correlations, which allow a researcher to determine how closely two variables (e.g., motivation and language ability) are related in a given population. " Associational research is not concerned with causation, only with co- occurrence. Vafaee, A&HL5575 (Class 3) 10 Research Design Types ! Correlational (Associational) Research " Correlation can be used in different ways: for example, to test a relationship between or among variables, and to make predictions. " Predictions are dependent on the outcome of a strong relationship between or among variables. That is, if variables are strongly related, we can often predict the likelihood of the presence of one from the presence of the other(s). Vafaee, A&HL5575 (Class 3) 11 Research Design Types ! Experimental and Quasi-Experimental Research " In experimental studies, researchers deliberately manipulate one or more variables (independent variables) to determine the effect on another variable (dependent variable). " This manipulation is usually described as a treatment and the researcher's goal is to determine whether there is a causal relationship. " Many types of experimental research involve a comparison of pretreatment and posttreatment performance. " Randomization is usually viewed as one of the hallmarks of experimental research. " Design types can range from truly experimental (with random assignment) to what is known as quasi-experimental (without random assignment). Vafaee, A&HL5575 (Class 3) 12 Research Design Types ! Experimental and Quasi-Experimental Research " A typical experimental study usually uses comparison or control groups to investigate research questions. " Many second language research studies involve a comparison between two or more groups. " This is known as a between-groups design. " This comparison can be made in one of two ways: two or more groups with different treatments; or two or more groups, one of which, the control group, receives no treatment. Vafaee, A&HL5575 (Class 3) 13 Session 4: Experimental Research Design_Part 1 Research Literacy (A&HL 5575) Experimental Research ! Experimental research is the way of determining the effect of something on something else. ! In experimental research, we manipulate at least one variable, while controlling for the effect of other variables, to determine the effect of manipulation on the outcome variable. ! Example: Whether focusing a learner's attention on some aspect of language increases that individual’s uptake of that aspect of language. ! Experimental research is based on the Rationalist worldview. " Rationalist approach is a theory-then-research or deductive approach. ! Research questions must be stated explicitly and must have some basis on previous literature. ! Example: Does focused attention on noun-adjective agreement in Italian promote learning to a greater extend than focused attention on wh-movement in Italian for beginning learners of Italian? ! There are always hypotheses about the results of experiments. ! Example: Focused attention on noun-adjective agreement in Italian will promote learning to a greater extend than focused attention on wh-movement in Italian for beginning learners of Italian. Vafaee, A&HL5575 (Class 4) 2 Explaining and Controlling Variance ! Experimental research is conducted for the purpose of explaining or controlling variance. Independent Variable Dependent Variable Group 1 Method 1 Group 1 T-Scores 1 Group 2 Method 2 Group 2 T-Scores 2 Group 3 Method 3 Group 3 T-Scores 3 Variance in scores Controlling variance allows us to minimize the effects of extraneous variance. As a result, the dependent variable can be interpreted without bias. ! Construct-relevant variance (Good variability or systematic variance). ! ! Variance related to the construct being investigated (e.g., variance in scores related to the effect of method). ! Construct-irrelevant variance (Bad variability or error/unsystematic or unwanted variance). ! Variance unrelated to the construct being investigated coming from factors other than the effect of instruction (e.g., pre-existing difference in ability level) Vafaee, A&HL5575 (Class 4) 3 Five Ways of Controlling the Unwanted Variance 1. Randomization 2. Holding conditions or factors constant (e.g., controlling for proficiency, only one teacher) 3. Statistical adjustments (adjusting pretest-posttest differences in gain scores--ANCOVA) 4. Building conditions or factors into the research design as independent variables (e.g., incorporating proficiency level the design—advanced/intermediate/beginners) into 5. The combination of all four methods Vafaee, A&HL5575 (Class 4) 4 1. Randomization Unsystematic variance Systematic variance Random, error, unsystematic variance due to other factors (ability level, motivation, anxiety, etc.) Variance due to methods Total variance Vafaee, A&HL5575 (Class 4) 5 1. Randomization Independent Variable Dependent Variable G1: 20 SS Method 1 Group 1 T-Scores 1 G2: 20 SS Method 2 Group 2 T-Scores 2 G3: 20 SS Method 3 Group 3 T-Scores 3 Variance in scores 60 SS randomly assigned to each of the 3 groups ! It equalizes the groups with respects to variables other than the main independent variable. ! It aims at reducing the effect of “random”, “inherent” or “ within groups” variance. Vafaee, A&HL5575 (Class 4) 6 2. Holding Factors Constant Independent Variable Same Teacher Dependent Variable G1: 20 SS Method 1 Group 1 T-Scores 1 G2: 20 SS Method 2 Group 2 T-Scores 2 G3: 20 SS Method 3 Group 3 T-Scores 3 Variance in scores 60 SS randomly assigned to each of the 3 groups ! Disadvantage: ! Reduces external validity Vafaee, A&HL5575 (Class 4) 7 3. Statistical Control ! The effect of pre-existing variables is removed statistically. Independent Variable Same Teacher G1: 20 SS Method 1 Group 1 T-Scores 1 G2: 20 SS Method 2 Group 2 T-Scores 2 G3: 20 SS Method 3 Group 3 T-Scores 3 60 SS randomly assigned to each of the 3 groups Vafaee, A&HL5575 (Class 4) Dependent Variable Variance in scores Proficiency scores from all 60 SS Transformed test scores or dependent variable Variance in scores 8 4. Building Factors into the Research Design Independent Variable Same Teacher Vafaee, A&HL5575 (Class 4) Group 1 Group 2 Group 3 Group 4 Group 5 Group 6 10 SS- Low 10 SS- High 10 SS- Low 10 SS- High 10 SS- Low 10 SS- High Method 1 Method 2 Method 3 60 SS randomly assigned to each of the 6 groups Dependent Variable Group 1 Group 2 Group 3 Group 4 Group 5 Group 6 T-scores 1 T-scores 2 T-scores 3 T-scores 4 T-scores 5 T-scores 6 Variance in scores 9 4. Building Factors into the Research Design Unsystematic variance Variance due to ability level Systematic variance Variance due to methods Variance due to other factors (motivation, anxiety, etc.) Vafaee, A&HL5575 (Class 4) Total variance 10 5. Using Four Ways in Combination Independent Variable Dependent Variable 20 SS- Low 20 SS- High 20 SS- Low 20 SS- High 20 SS- Low 20 SS- High 20 SS- Low 20 SS- High 20 SS- Low 20 SS- High 20 SS- Low 20 SS- High T-scores T-scores T-scores T-scores T-scores T-scores T-scores T-scores T-scores T-scores T-scores T-scores Teacher 1 Teacher 2 Same School Teacher 1 Teacher 2 Teacher 1 Teacher 2 Method 1 Method 1 Method 2 Method 2 Method 3 Method 3 WM scores for all 120 SS Transformed test scores or dependent variable 120 SS randomly assigned to each of the 6 groups Variance in scores Vafaee, A&HL5575 (Class 4) 11 Working Example ! Which kind of corrective feedback is more effective? Explicit or implicit? " " ! Which of the two corrective feedback types of metalinguistic explanation and recast is more beneficial? " ! What morphosyntactic target structures? ➔ easy or difficult Who are the learners? Children, young adults or adults Which of the two corrective feedback types of metalinguistic explanation and recast is more beneficial for learning the English article system and the past perfect tense among adult learners? " ! morphosyntax, pronunciation, collocation Which of the two corrective feedback types of metalinguistic explanation and recast is more beneficial for learning the English article system and the past perfect tense? " ! Learning what? ➔ Which of the two corrective feedback types of metalinguistic explanation and recast is more beneficial for learning morphosyntax? " ! Explicit like metalinguistic explanation Implicit like recast What is the setting? Classroom learning or naturalistic learning? Which of the two corrective feedback types of metalinguistic explanation and recast is more beneficial for learning the English article system and the past perfect tense among adult learners who learn English in the formal classroom learning setting? Vafaee, A&HL5575 (Class 4) 12 What are the Variables in the Working Example? X Variables Constants Main Independent Intervening Manipulated or predictor How and why Metalinguistic explanation and recast Dependent/predicted Posttest on knowledge of articles and tense Underlying mechanism Moderating Age and motivation Noticing the feedback, so individual differences in WM capacity and L2 aptitude Vafaee, A&HL5575 (Class 4) 13 Working Example X Variables Constants Main Independent Intervening Dependent/predicted How and why Manipulated or predictor Posttest on knowledge of articles and tense Control Binary/classifying Moderating Underlying mechanism Metalinguistic explanation and recast Learning setting, age and motivation Continuous Noticing the feedback, so individual differences in WM capacity No Control Adults Classroom Setting Vafaee, A&HL5575 (Class 4) Measures of WM and Motivation Statistically (Co-variate) Extraneous or Confounding 14 Amount of exposure outside the experimental setting, stress level, how the feedback is delivered, length of experiment, proficiency level, first language, previous knowledge Working Example ! Controlling for individual differences in WM and motivation, which of the two corrective feedback types of metalinguistic explanation and recast is more beneficial for learning the English article system and the past perfect tense among adult EFL learners at the intermediate level? Independent Variable Group 1 Article Past Tense Same Teacher Group 2 Dependent Variable Recast Group 1 Article Past Tense Article Test PP Test Article Test Meta_ L_ E Group 2 Select 100 adult EFL learners from the intermediate level and assign them randomly to the two groups Motivation and WM scores PP Test Variance in scores Transformed test scores or dependent variable Variance in scores Vafaee, A&HL5575 (Class 4) 15 Working Example ! Controlling for individual differences in WM and motivation, which of the two corrective feedback types of metalinguistic explanation and recast is more beneficial for learning the English article system and the past perfect tense among adult learners who learn English in the formal classroom learning setting? Amount of exposure outside the experimental setting, stress level, how the feedback is delivered, length of experiment, proficiency level, first language, previous knowledge Extraneous or Confounding Let’s control them Amount of exposure and practice outside the experimental setting Constant: Run a single-session experiment Statistical: Include a questionnaire Stress level Statistical: Include a questionnaire How the feedback is delivered Constant: use the same materials and experimenter Length of experiment proficiency level Constant and statistical: make the experiment long and include several posttests (posttests and delayed posttests) Constant: just the intermediate Statistical: Include a test First Language Constant: just one first language Statistical: Include a questionnaire Previous knowledge Constant: pretest and screen 16 Statistical: pretest and a covariate What about little differences in previous knowledge or all the other variables we are aware of? Randomization Working Example ! What kind of corrective feedback is more useful? ! Controlling for individual differences in WM and motivation, which of the two corrective feedback types of metalinguistic explanation and recast is more beneficial for learning the English article system and the past perfect tense among adult learners who learn English in the formal classroom learning setting? ! In a true (vs. quasi) experiment with a pretest, immediate posttest and delayed posttest design, the effectiveness of metalinguistic explanation versus recast for correcting errors related to the use of English articles and past perfect tense was studied. This study was carried out among Persian learners of English who were all above 18 years old. These participants are learning English in an EFL context and rarely have exposure to English out of the classroom setting. Also, during the length of experiment which lasted for ten one-hour sessions, and by the time they took the delayed posttest, which was one month after the last session of the experiment, these participants had no practice of English outside of the experimental setting. After each of the experimental sessions, the leaners took a posttest, so ten posttests were included in the study. In this study, to control for the effect of motivation and stress on learning, questionnaires for both of these psychological variables were included and their results were used as covariates in the statistical analyses. Also, the effect of individual differences in WM on the learning outcomes were accounted for statistically. Vafaee, A&HL5575 (Class 4) 17 Vafaee, A&HL5575 (Class 5) 2 Experimental Variables & Levels An experimental variable usually has from 2 to 5 different levels or experimental treatments. Experimental Variable Types of Corrective Feedback Levels or Experimental Treatments • Explicit with metalinguistic explanation • Explicit without metalinguistic explanation • Recast Vafaee, A&HL5575 (Class 5) 3 Independent Experimental Design Types of Feedback Randomly Assigned Dependent Variable Explicit + Explanation N = 40 ESL learners Explicit – Explanation N = 40 ESL learners # of correct answers on a test Recast N = 40 ESL learners Experimental procedure R R R G1 G2 G3 Xexp+ex Xexp-ex Xrecast N = number of participants R = randomization X = experimental variable (manipulated) I= Independent variable G = group O = Observation (assessment) O1 O2 O3 20.03 23.20 29.90 Compare the mean scores of the groups. ◦ We use a t-test to compare the differences in the mean scores of 2 groups. ◦ We use ANOVA (analysis of variance) to compare the means of 3 or more groups). Vafaee, A&HL5575 (Class 5) 4 Posttest-Only Control-Group Design R G1 R G2(c) X1 - O1 O2 ◦ Used when a pretest is not available, not used, too costly or interacts with treatment. ◦ Randomization causes no need for pre-test especially with large samples. Vafaee, A&HL5575 (Class 5) 5 Pretest-Posttest Control Group Design R G1 R G2(c) O1 O3 X1 - O2 O4 ◦ It is a little tricky because pretests and posttests should be equal forms or identical. ◦ In a randomized experiment, pretests are not used to create equal groups. ◦ However, the pretests can reveal improbable results. ◦ Also, pretest results can be used for statistical control. ANCOVA is used for data analysis for this purpose. ◦ The scores from pretests to posttests can be used to compute “gain” scores. 6 Vafaee, A&HL5575 (Class 5) Randomized Solomon Four-Group Design R R R R G1E O1 G2C O3 G3E G4C - X X - O2 O4 O5 O6 ◦ This design can show if pretests had an interaction with the treatment or influenced the posttest’s scores. Vafaee, A&HL5575 (Class 5) 7 Factorial Design In the previous designs, we examined the effect of an independent variable on a dependent variable. For example, we looked at the effectiveness of three different types of corrective feedback. ● But the effectiveness of three different types of corrective feedback might be mediated or moderated by other variables such as the students’proficiency level. ● ● ● There might be an interaction between the different types of corrective feedback and the different ability groups. To examine the effectiveness of three different types of corrective feedback accounting for variations in the students’ proficiency level, we can design a study using ONLY the intermediate level students. And then, we design a similar study for ONLY the advanced level students. ● But it is more efficient if we designed ONE study that accounts for different types of corrective feedback in relation to different proficiency levels of the learners. ● When there is more than one independent variable in the deign of a study, the design is called the factorial design. Vafaee, A&HL5575 (Class 5) 8 Factorial Design • A “complete” factorial design involves 2 or more independent variables (factors) in the design 1) types of corrective feedback 2) proficiency levels • Also, the independent variables must have at least 2 levels for each variable 1) Three types of corrective feedback 2) Two proficiency levels (e.g., intermediate & advanced) • This is a 3 x 2 factorial design, i.e., 2 independent variables and two and three levels for each, respectively. Vafaee, A&HL5575 (Class 5) Intermediate (Xexp + ex) Intermediate (Xexp - ex) Intermediate (Recast) Advanced (Xexp + ex) Advanced (Xexp - ex) Advanced (Recast) 9 Factorial Design Practice 1: • A factorial design with 3 independent variables: 1) types of pragmatics instruction (explicit & implicit) 2X3X3 2) proficiency levels (beginning, intermediate & advanced) 3) length of residence (Low: 0-2 years; medium: 3 – 5 years; long: more than 5 years) Practice 2 : 3X3X4 4X2 2X5X2 3X2X2X2 Vafaee, A&HL5575 (Class 5) 10 Repeated Measures Design: Extension of Posttest-Only Vafaee, A&HL5575 (Class 5) R G1 X1 O1 O2 O3 R G1 R G2(c) X1 - O1 O4 O2 O5 O3 O6 11 Repeated Measures Design: Extension of Pretest-Posttest R G1 O1 O2 O3 X1 O4 O5 O6 R G1 O1 R G2 O6 O2 X1 O3 O4 O5 O7 X2 O8 O9 O10 R G1 O1 O2 X1 O3 O4 O5 R G2 O6 O7 X2 O8 O9 O10 R GC O11 O12 - O13 O14 O15 R G1 O1 X1a O2 X1b O3 X1c O4 O5 R G2 O6 X2a O7 X2b O8 X2c O9 O10 R GC O11 - O12 - O13 - O14 O15 Vafaee, A&HL5575 (Class 5) 12 Quasi-Experimental Research 1. Experimental design involves: • One or more independent & dependent variables • An experimental variable that is manipulated • Random assignment to groups. 2. Quasi-experimental design involves: • All the above • Except for the Random assignment of participants to experimental treatment groups. • There are intact groups. Vafaee, A&HL5575 (Class 5) 13 Threats to validity Threats to internal validity: • Differential selection of subjects (difficulty to argue that the effect was due to treatment if groups are different) Threats to external validity: • Lack of random selection puts into question the generalizability of the results Solution: • To argue for representativeness of the sample on a logical basis • The equivalence of groups can be checked statistically and if not equivalent, the effect can be adjusted for. Vafaee, A&HL5575 (Class 5) 14 Post-Test Only, Non-Equivalent, Multi-Group Design G1(e) G2(e) X1 X2 O1 O2 RQ: What is the effect of explicit vocabulary instruction on lexical knowledge? Non randomly Assigned Types of Instruction Dependent Variable N = 20 Explicit vocabulary instruction Post-test RQ: What are the effects of explicit and incidental vocabulary teaching methods? Non randomly Assigned Types of Instruction Dependent Variable N = 20 Explicit vocabulary instruction Post-test N = 20 Incidental vocabulary instruction Post-test Vafaee, A&HL5575 (Class 5) 15 Post-Test Only, Non-Equivalent, Control Group Design G1 (e) G2 (e) G3 (c) X1 X2 - O1 O2 O3 RQ: What is the effect of explicit vocabulary instruction on lexical knowledge? Non randomly Assigned Types of Instruction Dependent Variable N = 20 Explicit vocabulary instruction Post-test N = 20 - Post-test RQ: What are the effects of explicit and incidental vocabulary teaching methods? Non randomly Assigned Types of Instruction Dependent Variable N = 20 Explicit vocabulary instruction Post-test N = 20 Incidental vocabulary instruction Post-test N = 20 - Post-test Vafaee, A&HL5575 (Class 5) 16 Pretest-Posttest, Nonequivalent, Control Group Design Pretest-Posttest, Nonequivalent, Multi-Group Design Non randomly Assigned Pretest G1 O1 G2 G3 O3 O5 Gc or G4 O7 Types of treatment Posttest 1 Posttest 2 X1 X2 O2 O9 O4 O10 X3 O6 O11 O8 O12 - Or X4 Use the pretest scores to: 1. check the similarity of the groups. If groups are not similar, the pretest scores can be used to adjust the posttest scores statically. 2. generate gain scores. Vafaee, A&HL5575 (Class 5) 17 Repeated Measures Design G1 O1 O2 G1 G2 O1 O6 O2 X1 O3 O4 O5 O7 X2 O8 O9 O10 G1 G2 GC O1 O2 X1 O3 O4 O5 O6 O7 X2 O8 O9 O10 O11 O12 - O13 O14 O15 G1 O1 X1a O2 X1b O3 X1c O4 O5 O6 X2a O7 X2b O8 X2c O9 O10 O11 - O12 - O13 - O14 O15 G2 GC Vafaee, A&HL5575 (Class 5) O3 X1 O4 O5 O6 18 Single-Subject Designs Repeated measures of the dependent variable Measurement must be highly standardized Single variable rule Traditional or normal condition is called the baseline Baseline should be long enough for the dependent variable to gain stability O1 Ok O1 Ok B A O1 Ok A Vafaee, A&HL5575 (Class 5) O1 Ok B O1 Ok A 19 Criteria for a well-designed experiment 1. Adequate Experimental Control: we control some variables (e.g., proficiency level) for meaningful interpretation of the results (to avoid confounding); other variables can be controlled via randomization or by building variables into the design. 2. Lack of artificiality: If we want to generalize to a non-experiment setting, the results should apply to the real world. The artificial nature of the experiment should not cause the experimental effects. 3. Adequate basis for comparison: there must be a way to determine the experimental effect. The experiment may or need a control group (e.g., no new treatment). 4. Precise data: the data must be reliable (e.g., the ratings from the 2 raters need to be consistent); codings must have agreement; a test/ questionnaire should had high Cronbach's alphas. Vafaee, A&HL5575 (Class 5) 20 Criteria for a well-designed experiment 5. No contaminated data & no bias: no interactions between the data (e.g., raters must rate the data independently) (part of reliability); no interaction between variables (raters are more consistent on some topics or with some examinees) 6. No confounding of the data: no two variables contribute to the score in a way that is difficult to separate their effect (proficiency and age) (validity issue) 7. Sample representativeness: random selection to enhance generalizability (of learners; also of items, tasks, contexts) 8. Parsimony: a simple research design/ underlying model is better than complex one—when possible (!) BUT, the lack of complexity might not represent reality. Vafaee, A&HL5575 (Class 5) 21 Experimental Validity 1. Construct validity: The extent to which the constructs measured in the independent or dependent variables are meaningful & appropriate as a result of empirical supports. 2. Internal Validity: The extent to which the results of the experiment can be attributed to the treatment. In other words, the extraneous variables are not having an undue effect on the link between the independent & dependent variable. They are being controlled--or are part of the design. 3. External Validity: The extent to which the results of the experiment can be generalized to other populations (ESL/EFL, urban/rural), or conditions (class size, computer-mediated/ face-to-face). 4. Statistical Conclusion Validity: The extent to the statistical procedures used in the study are appropriate & meaningful for decision making. Are the measures sufficiently reliable? Is the statistical procedure appropriate for the data? Have the assumptions underlying the statistical procedure been met? Vafaee, A&HL5575 (Class 5) 22 Threats to Construct Validity 1. Inadequate pre-operational explanation of constructs: Inadequate theoretical & operation definition of the dependent or independent (i.e., how is “instruction” defined and operationalized/implemented by two different teachers?) 2. Mono-operation bias: Only one form of experimental variable is implemented (e.g., only written feedback is given, not oral) 3. Mono-method bias : Only one method is used to measure the dependent variable (e.g., only a GJT) 4. Hypothesis guessing within the experimental conditions: Participants behave differently when they know they’re part of an experiment (Hawthorne Effect) 5. Confounding constructs & levels of constructs: Drawing conclusions about constructs when some levels are absent. Vafaee, A&HL5575 (Class 5) 23 Threats to Internal Validity 1. History: unanticipated events have occurred that affect performance on the dependent variable. 2. Maturation: the participants mature/change (over time) 3. Testing/Pre-testing: the effects of test familiarity 4. Instrumentation/Measuring instruments: inconsistent use of a test (e.g., the instructions changed from the pre to the post-test) 5. Statistical regression: participants with extreme scores cause other scores to regress toward the average score (e.g., with gain scores) 6. (Differential) selection (of a sample): no randomization, causing unequivalent groups 7. (Experimental) Mortality: participants drop out 8. Interaction of selection & maturation, etc.: inconsistent maturation of participants across groups (e.g., some students spent the summer in an English speaking country) Vafaee, A&HL5575 (Class 5) 24 Threats to External Validity 1. Interaction effect of pre-testing & the experimental variable : pretesting interacts with the experimental variable causing ungeneralisable results. 2. Interaction effect of selection biases & experimental treatment: the effect of choosing one group to give treatment to, which would produce a different effect if the groups were randomized. 3. Reactive effect of experimental arrangements: the effect occurs when an experiment takes place in a setting that would not produce the same effect in different settings (generalizing across settings) 4. Multiple treatment interference: the carry over effect of participating in multiple treatments (generalizing across time to the experimental effects). Vafaee, A&HL5575 (Class 5) 25 Threats to Statistical Conclusion Validity 1. Low statistical power: Sample size too small to detect differences between groups 2. Violation of assumptions: failure to meet the assumptions or understand the effect of violations (robust or not) 3. Fishing & error rate problem: Capitalizing on chance factor findings 4. Low reliability: Imprecise measures Vafaee, A&HL5575 (Class 5) 26 Vafaee, A&HL5575 (Class 5) 2 Experimental Variables & Levels An experimental variable usually has from 2 to 5 different levels or experimental treatments. Experimental Variable Types of Corrective Feedback Levels or Experimental Treatments • Explicit with metalinguistic explanation • Explicit without metalinguistic explanation • Recast Vafaee, A&HL5575 (Class 5) 3 Independent Experimental Design Types of Feedback Randomly Assigned Dependent Variable Explicit + Explanation N = 40 ESL learners Explicit – Explanation N = 40 ESL learners # of correct answers on a test Recast N = 40 ESL learners Experimental procedure R R R G1 G2 G3 Xexp+ex Xexp-ex Xrecast N = number of participants R = randomization X = experimental variable (manipulated) I= Independent variable G = group O = Observation (assessment) O1 O2 O3 20.03 23.20 29.90 Compare the mean scores of the groups. ◦ We use a t-test to compare the differences in the mean scores of 2 groups. ◦ We use ANOVA (analysis of variance) to compare the means of 3 or more groups). Vafaee, A&HL5575 (Class 5) 4 Posttest-Only Control-Group Design R G1 R G2(c) X1 - O1 O2 ◦ Used when a pretest is not available, not used, too costly or interacts with treatment. ◦ Randomization causes no need for pre-test especially with large samples. Vafaee, A&HL5575 (Class 5) 5 Pretest-Posttest Control Group Design R G1 R G2(c) O1 O3 X1 - O2 O4 ◦ It is a little tricky because pretests and posttests should be equal forms or identical. ◦ In a randomized experiment, pretests are not used to create equal groups. ◦ However, the pretests can reveal improbable results. ◦ Also, pretest results can be used for statistical control. ANCOVA is used for data analysis for this purpose. ◦ The scores from pretests to posttests can be used to compute “gain” scores. 6 Vafaee, A&HL5575 (Class 5) Randomized Solomon Four-Group Design R R R R G1E O1 G2C O3 G3E G4C - X X - O2 O4 O5 O6 ◦ This design can show if pretests had an interaction with the treatment or influenced the posttest’s scores. Vafaee, A&HL5575 (Class 5) 7 Factorial Design In the previous designs, we examined the effect of an independent variable on a dependent variable. For example, we looked at the effectiveness of three different types of corrective feedback. ● But the effectiveness of three different types of corrective feedback might be mediated or moderated by other variables such as the students’proficiency level. ● ● ● There might be an interaction between the different types of corrective feedback and the different ability groups. To examine the effectiveness of three different types of corrective feedback accounting for variations in the students’ proficiency level, we can design a study using ONLY the intermediate level students. And then, we design a similar study for ONLY the advanced level students. ● But it is more efficient if we designed ONE study that accounts for different types of corrective feedback in relation to different proficiency levels of the learners. ● When there is more than one independent variable in the deign of a study, the design is called the factorial design. Vafaee, A&HL5575 (Class 5) 8 Factorial Design • A “complete” factorial design involves 2 or more independent variables (factors) in the design 1) types of corrective feedback 2) proficiency levels • Also, the independent variables must have at least 2 levels for each variable 1) Three types of corrective feedback 2) Two proficiency levels (e.g., intermediate & advanced) • This is a 3 x 2 factorial design, i.e., 2 independent variables and two and three levels for each, respectively. Vafaee, A&HL5575 (Class 5) Intermediate (Xexp + ex) Intermediate (Xexp - ex) Intermediate (Recast) Advanced (Xexp + ex) Advanced (Xexp - ex) Advanced (Recast) 9 Factorial Design Practice 1: • A factorial design with 3 independent variables: 1) types of pragmatics instruction (explicit & implicit) 2X3X3 2) proficiency levels (beginning, intermediate & advanced) 3) length of residence (Low: 0-2 years; medium: 3 – 5 years; long: more than 5 years) Practice 2 : 3X3X4 4X2 2X5X2 3X2X2X2 Vafaee, A&HL5575 (Class 5) 10 Repeated Measures Design: Extension of Posttest-Only Vafaee, A&HL5575 (Class 5) R G1 X1 O1 O2 O3 R G1 R G2(c) X1 - O1 O4 O2 O5 O3 O6 11 Repeated Measures Design: Extension of Pretest-Posttest R G1 O1 O2 O3 X1 O4 O5 O6 R G1 O1 R G2 O6 O2 X1 O3 O4 O5 O7 X2 O8 O9 O10 R G1 O1 O2 X1 O3 O4 O5 R G2 O6 O7 X2 O8 O9 O10 R GC O11 O12 - O13 O14 O15 R G1 O1 X1a O2 X1b O3 X1c O4 O5 R G2 O6 X2a O7 X2b O8 X2c O9 O10 R GC O11 - O12 - O13 - O14 O15 Vafaee, A&HL5575 (Class 5) 12 Quasi-Experimental Research 1. Experimental design involves: • One or more independent & dependent variables • An experimental variable that is manipulated • Random assignment to groups. 2. Quasi-experimental design involves: • All the above • Except for the Random assignment of participants to experimental treatment groups. • There are intact groups. Vafaee, A&HL5575 (Class 5) 13 Threats to validity Threats to internal validity: • Differential selection of subjects (difficulty to argue that the effect was due to treatment if groups are different) Threats to external validity: • Lack of random selection puts into question the generalizability of the results Solution: • To argue for representativeness of the sample on a logical basis • The equivalence of groups can be checked statistically and if not equivalent, the effect can be adjusted for. Vafaee, A&HL5575 (Class 5) 14 Post-Test Only, Non-Equivalent, Multi-Group Design G1(e) G2(e) X1 X2 O1 O2 RQ: What is the effect of explicit vocabulary instruction on lexical knowledge? Non randomly Assigned Types of Instruction Dependent Variable N = 20 Explicit vocabulary instruction Post-test RQ: What are the effects of explicit and incidental vocabulary teaching methods? Non randomly Assigned Types of Instruction Dependent Variable N = 20 Explicit vocabulary instruction Post-test N = 20 Incidental vocabulary instruction Post-test Vafaee, A&HL5575 (Class 5) 15 Post-Test Only, Non-Equivalent, Control Group Design G1 (e) G2 (e) G3 (c) X1 X2 - O1 O2 O3 RQ: What is the effect of explicit vocabulary instruction on lexical knowledge? Non randomly Assigned Types of Instruction Dependent Variable N = 20 Explicit vocabulary instruction Post-test N = 20 - Post-test RQ: What are the effects of explicit and incidental vocabulary teaching methods? Non randomly Assigned Types of Instruction Dependent Variable N = 20 Explicit vocabulary instruction Post-test N = 20 Incidental vocabulary instruction Post-test N = 20 - Post-test Vafaee, A&HL5575 (Class 5) 16 Pretest-Posttest, Nonequivalent, Control Group Design Pretest-Posttest, Nonequivalent, Multi-Group Design Non randomly Assigned Pretest G1 O1 G2 G3 O3 O5 Gc or G4 O7 Types of treatment Posttest 1 Posttest 2 X1 X2 O2 O9 O4 O10 X3 O6 O11 O8 O12 - Or X4 Use the pretest scores to: 1. check the similarity of the groups. If groups are not similar, the pretest scores can be used to adjust the posttest scores statically. 2. generate gain scores. Vafaee, A&HL5575 (Class 5) 17 Repeated Measures Design G1 O1 O2 G1 G2 O1 O6 O2 X1 O3 O4 O5 O7 X2 O8 O9 O10 G1 G2 GC O1 O2 X1 O3 O4 O5 O6 O7 X2 O8 O9 O10 O11 O12 - O13 O14 O15 G1 O1 X1a O2 X1b O3 X1c O4 O5 O6 X2a O7 X2b O8 X2c O9 O10 O11 - O12 - O13 - O14 O15 G2 GC Vafaee, A&HL5575 (Class 5) O3 X1 O4 O5 O6 18 Single-Subject Designs Repeated measures of the dependent variable Measurement must be highly standardized Single variable rule Traditional or normal condition is called the baseline Baseline should be long enough for the dependent variable to gain stability O1 Ok O1 Ok B A O1 Ok A Vafaee, A&HL5575 (Class 5) O1 Ok B O1 Ok A 19 Criteria for a well-designed experiment 1. Adequate Experimental Control: we control some variables (e.g., proficiency level) for meaningful interpretation of the results (to avoid confounding); other variables can be controlled via randomization or by building variables into the design. 2. Lack of artificiality: If we want to generalize to a non-experiment setting, the results should apply to the real world. The artificial nature of the experiment should not cause the experimental effects. 3. Adequate basis for comparison: there must be a way to determine the experimental effect. The experiment may or need a control group (e.g., no new treatment). 4. Precise data: the data must be reliable (e.g., the ratings from the 2 raters need to be consistent); codings must have agreement; a test/ questionnaire should had high Cronbach's alphas. Vafaee, A&HL5575 (Class 5) 20 Criteria for a well-designed experiment 5. No contaminated data & no bias: no interactions between the data (e.g., raters must rate the data independently) (part of reliability); no interaction between variables (raters are more consistent on some topics or with some examinees) 6. No confounding of the data: no two variables contribute to the score in a way that is difficult to separate their effect (proficiency and age) (validity issue) 7. Sample representativeness: random selection to enhance generalizability (of learners; also of items, tasks, contexts) 8. Parsimony: a simple research design/ underlying model is better than complex one—when possible (!) BUT, the lack of complexity might not represent reality. Vafaee, A&HL5575 (Class 5) 21 Experimental Validity 1. Construct validity: The extent to which the constructs measured in the independent or dependent variables are meaningful & appropriate as a result of empirical supports. 2. Internal Validity: The extent to which the results of the experiment can be attributed to the treatment. In other words, the extraneous variables are not having an undue effect on the link between the independent & dependent variable. They are being controlled--or are part of the design. 3. External Validity: The extent to which the results of the experiment can be generalized to other populations (ESL/EFL, urban/rural), or conditions (class size, computer-mediated/ face-to-face). 4. Statistical Conclusion Validity: The extent to the statistical procedures used in the study are appropriate & meaningful for decision making. Are the measures sufficiently reliable? Is the statistical procedure appropriate for the data? Have the assumptions underlying the statistical procedure been met? Vafaee, A&HL5575 (Class 5) 22 Threats to Construct Validity 1. Inadequate pre-operational explanation of constructs: Inadequate theoretical & operation definition of the dependent or independent (i.e., how is “instruction” defined and operationalized/implemented by two different teachers?) 2. Mono-operation bias: Only one form of experimental variable is implemented (e.g., only written feedback is given, not oral) 3. Mono-method bias : Only one method is used to measure the dependent variable (e.g., only a GJT) 4. Hypothesis guessing within the experimental conditions: Participants behave differently when they know they’re part of an experiment (Hawthorne Effect) 5. Confounding constructs & levels of constructs: Drawing conclusions about constructs when some levels are absent. Vafaee, A&HL5575 (Class 5) 23 Threats to Internal Validity 1. History: unanticipated events have occurred that affect performance on the dependent variable. 2. Maturation: the participants mature/change (over time) 3. Testing/Pre-testing: the effects of test familiarity 4. Instrumentation/Measuring instruments: inconsistent use of a test (e.g., the instructions changed from the pre to the post-test) 5. Statistical regression: participants with extreme scores cause other scores to regress toward the average score (e.g., with gain scores) 6. (Differential) selection (of a sample): no randomization, causing unequivalent groups 7. (Experimental) Mortality: participants drop out 8. Interaction of selection & maturation, etc.: inconsistent maturation of participants across groups (e.g., some students spent the summer in an English speaking country) Vafaee, A&HL5575 (Class 5) 24 Threats to External Validity 1. Interaction effect of pre-testing & the experimental variable : pretesting interacts with the experimental variable causing ungeneralisable results. 2. Interaction effect of selection biases & experimental treatment: the effect of choosing one group to give treatment to, which would produce a different effect if the groups were randomized. 3. Reactive effect of experimental arrangements: the effect occurs when an experiment takes place in a setting that would not produce the same effect in different settings (generalizing across settings) 4. Multiple treatment interference: the carry over effect of participating in multiple treatments (generalizing across time to the experimental effects). Vafaee, A&HL5575 (Class 5) 25 Threats to Statistical Conclusion Validity 1. Low statistical power: Sample size too small to detect differences between groups 2. Violation of assumptions: failure to meet the assumptions or understand the effect of violations (robust or not) 3. Fishing & error rate problem: Capitalizing on chance factor findings 4. Low reliability: Imprecise measures Vafaee, A&HL5575 (Class 5) 26

    Achiever Essays
    Calculate your paper price
    Pages (550 words)
    Approximate price: -

    Why Work with Us

    Top Quality and Well-Researched Papers

    We always make sure that writers follow all your instructions precisely. You can choose your academic level: high school, college/university or professional, and we will assign a writer who has a respective degree.

    Professional and Experienced Academic Writers

    We have a team of professional writers with experience in academic and business writing. Many are native speakers and able to perform any task for which you need help.

    Free Unlimited Revisions

    If you think we missed something, send your order for a free revision. You have 10 days to submit the order for review after you have received the final document. You can do this yourself after logging into your personal account or by contacting our support.

    Prompt Delivery and 100% Money-Back-Guarantee

    All papers are always delivered on time. In case we need more time to master your paper, we may contact you regarding the deadline extension. In case you cannot provide us with more time, a 100% refund is guaranteed.

    Original & Confidential

    We use several writing tools checks to ensure that all documents you receive are free from plagiarism. Our editors carefully review all quotations in the text. We also promise maximum confidentiality in all of our services.

    24/7 Customer Support

    Our support agents are available 24 hours a day 7 days a week and committed to providing you with the best customer experience. Get in touch whenever you need any assistance.

    Try it now!

    Calculate the price of your order

    Total price:
    $0.00

    How it works?

    Follow these simple steps to get your paper done

    Place your order

    Fill in the order form and provide all details of your assignment.

    Proceed with the payment

    Choose the payment system that suits you most.

    Receive the final file

    Once your paper is ready, we will email it to you.

    Our Services

    No need to work on your paper at night. Sleep tight, we will cover your back. We offer all kinds of writing services.

    Essays

    Essay Writing Service

    No matter what kind of academic paper you need and how urgent you need it, you are welcome to choose your academic level and the type of your paper at an affordable price. We take care of all your paper needs and give a 24/7 customer care support system.

    Admissions

    Admission Essays & Business Writing Help

    An admission essay is an essay or other written statement by a candidate, often a potential student enrolling in a college, university, or graduate school. You can be rest assurred that through our service we will write the best admission essay for you.

    Reviews

    Editing Support

    Our academic writers and editors make the necessary changes to your paper so that it is polished. We also format your document by correctly quoting the sources and creating reference lists in the formats APA, Harvard, MLA, Chicago / Turabian.

    Reviews

    Revision Support

    If you think your paper could be improved, you can request a review. In this case, your paper will be checked by the writer or assigned to an editor. You can use this option as many times as you see fit. This is free because we want you to be completely satisfied with the service offered.

    Live Chat+1(978) 822-0999EmailWhatsApp

    Order your essay today and save 20% with the discount code RESEARCH

    slot online