Research & Program Evaluation

🔑 Key Vocabulary & Foundational Concepts

1. Qualitative vs. Quantitative Research

• Quantitative Research:

  • Focuses on numerical data, statistical analysis.

  • Seeks to test hypotheses, examine relationships, and establish causality.

  • Deductive reasoning (starts with a theory and tests it).

  • Example: Comparing math test scores across two teaching methods.

• Qualitative Research:

  • Focuses on non-numerical data like themes, experiences, and context.

  • Inductive reasoning (gathers data to build theories).

  • Example: Interviewing teachers to explore experiences with remote learning.

• Both are valid: Quantitative research is often more generalizable; qualitative provides depth and rich understanding.

🔬 Quantitative Research Designs

Non-Experimental Designs (No manipulation of variables)

1. Survey Research:

   • Collects self-reported data from participants.

   • Example: School climate questionnaire for high school students.

2. Descriptive Research:

   • Describes characteristics or functions.

   • Example: Recording the average GPA of college freshmen.

3. Comparative Research:

   • Compares two or more groups on a specific variable.

   • Example: Comparing stress levels of first-year vs. senior college students.

4. Correlational Research:

   • Examines relationships between variables (no causation).

   • Example: Relationship between sleep quality and academic performance.

5. Ex Post Facto Research:

   • Looks at existing conditions retrospectively.

   • Example: Studying effects of childhood trauma on adult anxiety levels.

Experimental Designs (Manipulation of variables)

1. True Experimental Research:

   • Random assignment, control groups, manipulation of IV.

   • Control group: No treatment or receives placebo.

   • Experimental group: Receives the independent variable.

2. Quasi-Experimental Research:

   • No random assignment; uses naturally formed groups.

   • Example: Studying effect of curriculum change in two different schools.

🧠 Qualitative Research Designs

1. Naturalistic Inquiry:

   • Observing naturally occurring events without interference.

2. Interactive Designs:

   • Case Studies: In-depth analysis of a single subject or case.

   • Ethnography: Studies cultures and groups in their natural setting.

3. Non-Interactive Designs:

   • Content analysis, historical research, artifact review.

🔄 Mixed Method Designs

• Combines qualitative and quantitative data for a more comprehensive view.

• Benefits:

  • Richer interpretation

  • Validates results across data types

  • Enhances credibility

🔍 Specialized Designs

1. Single Subject Research: Focus on one participant; ABAB design.

2. Action Research: Conducted by practitioners to improve practice.

3. Pilot Study: Small-scale preliminary study.

4. Longitudinal Research: Follows participants over time.

5. Cross-Sectional Research: Observes different groups at one time point.

6. Meta-Analysis: Statistically combines results from multiple studies.

🧪 Validity

Internal Validity (Confidence that IV caused changes in DV)

• Threats:

  • Selection bias: Groups not equivalent at start.

  • Maturation: Changes in participants over time.

  • Mortality (Attrition): Dropouts affect results.

  • Experimental bias: Researcher influence.

  • History: External events affect outcomes.

  • Statistical regression: Extreme scores tend to regress toward mean.

External Validity (Generalizability of findings)

• Threats:

  • Selection of subjects: Non-representative sample.

  • Ecological validity: Results may not apply outside the setting.

  • Subject reactivity: Behavior changes due to being studied.

  • Placebo effect: Belief in treatment impacts outcomes.

  • Novelty effect: Newness of treatment affects results.

  • Disruption: Events that interfere with treatment delivery.

📏 Levels of Measurement

1. Nominal: Categories with no order (e.g., gender).

2. Ordinal: Ordered categories (e.g., class rank).

3. Interval: Ordered, equal intervals, no true zero (e.g., IQ).

4. Ratio: Equal intervals with true zero (e.g., height, weight).

🔁 Sampling Techniques

1. Random Sampling: Equal chance for all members.

2. Stratified Sampling: Subgroups are proportionally represented.

3. Proportional Sampling: Same as stratified, but exact proportions.

4. Cluster Sampling: Groups (clusters) randomly selected.

5. Purposeful Sampling: Handpicked subjects for specific traits.

6. Convenience Sampling: Easily accessible subjects (less generalizable).

Sample Size Guidelines

• Quantitative: Minimum 30 participants per group for statistical analysis.

• Qualitative: Smaller, based on saturation (often 10–20).

• Meta-analysis: 10+ studies ideal.

📊 Statistical Analysis

Descriptive Statistics

• Mean, median, mode

• Standard deviation, range, variance

Inferential Statistics

• Generalizes results from sample to population

Parametric Tests (Interval/ratio data, normal distribution)

• T-Test: Compares means of 2 groups.

• ANOVA:

  • One-Way ANOVA: 1 IV with 3+ levels.

  • Factorial ANOVA: 2+ IVs.

  • MANOVA: Multiple DVs.

  • ANCOVA: Controls for covariates.

Non-Parametric Tests (Nominal/ordinal data or non-normal)

• Chi-Square: Categorical variable comparison.

• Mann-Whitney U, Kruskal-Wallis: Alternatives to t-tests and ANOVA.

Other Terms:

• Four-Group Design: Solomon Four-Group to test interaction between treatment and pre-testing.

• Multiple Regression: Predicts DV from multiple IVs.

• Scatterplots: Show relationship between variables.

• Factor Analysis: Identifies groupings/clusters of variables.

• Likert Scales: Measures attitudes on a scale (e.g., 1–5 strongly disagree to strongly agree).

❓ Research Components

• Research Question: Broad inquiry

• Hypothesis: Predictive statement about relationship

• Independent Variable (IV): Manipulated variable

• Dependent Variable (DV): Measured outcome

🔬 Significance & Error

• Alpha Level (Significance): Probability threshold (commonly 0.05)

• Type I Error (α): Rejecting a true null hypothesis (false positive)

• Type II Error (β): Failing to reject a false null hypothesis (false negative)