Study Detail

View Revisions

Pfannenstiel, J. (2015). Evaluation of the i3 validation of improving education outcomes for American Indian children. Unpublished manuscript. Overland Park, KS: Research & Training Associates, Inc.

Model(s) Reviewed: Baby FACE, an adaptation of PAT
Study Screening Details
Screening Decision Screening Conclusion
Study passes screens Eligible for review
Study Design Details
Rating Design Attrition Baseline Equivalence Reassignment Confounding Factors
Moderate Randomized controlled trial High

Established on race/ethnicity and SES; outcomes not feasible to assess at baseline

None

None

Notes:

This study reports findings for both a RCT and a quasi-experimental design (QED). In the RCT, outcomes that were not assessable at baseline received a moderate rating. Outcomes that were assessable at baseline received a low rating because the study did not establish baseline equivalence for these outcomes.

In the QED, all outcomes received a low rating because treatment and comparison groups were not enrolled in the study at the same time, creating a confounding factor.

Study Characteristics
Study Participants The study recruited families from the catchment areas of 20 Bureau of Indian Education schools that had not previously implemented the FACE program (Baby FACE is the home visiting component of FACE). One of the 20 programs withdrew from the grant after two years and is not included in the study. The study included a QED analysis and a smaller RCT. The QED analysis sample included families from the 19 remaining schools. The QED study included 1,329 participants: 853 in the treatment group and 476 in the comparison group. Within the QED, 49 percent of children in the treatment group and 55 percent of children in the comparison group were male. In the treatment group, 25 percent of mothers had completed less than a 12th grade education and 37 percent had a high school diploma or GED. Three percent had a bachelor’s degree or higher. In the comparison group, 21 percent of mothers had completed less than a 12th grade education and 32 percent had a high school diploma or GED. Seven percent had a bachelor’s degree or higher. A majority of children were from two-parent households. Seventy-three percent of treatment and comparison groups received Supplemental Nutrition Assistance Program benefits. The RCT sample included families from one of the 19 Bureau of Indian Education school catchment areas that were included in the QED analysis. The site was selected for the RCT because it had more eligible families than space in the program. Within the RCT, 129 families were randomly assigned, with 63 families assigned to the treatment group (Baby FACE) and 66 assigned to the comparison group (business-as-usual).
Setting The QED study included 20 Bureau of Indian Education school catchment areas located across the following U.S. states: Arizona, Idaho, New Mexico, North Carolina, South Dakota, and Washington. The RCT took place in one of the QED study sites that was oversubscribed.
Home Visiting Services The Baby FACE program is based on the Parents as Teachers (PAT) model. Participants in the treatment group received home visits by certified parent educators, developmental screenings, group meetings with other parents (known as Family Circle), and referrals to community resources. The initial home visits focused on the foundational topics of the model, and the home visitor selected subsequent topics for home visiting sessions based on the family's goals and needs.
Comparison Conditions The comparison group was eligible to receive existing business-as-usual services that were available to them within their communities.
Staff Characteristics and Training Parent educators received training to conduct the foundational plans that are part of the Baby FACE curriculum. In addition to the face-to-face training, home visitors continued receiving professional development via conference calls.
Funding Sources Institute of Education Sciences (IES) U.S. Department of Education, Grant #U396B100189.
Author Affiliation Study authors are not model developers or distributors. They were contracted by the implementers to conduct an independent evaluation of the Baby FACE program.
Study Registration:

Clinicaltrials.gov Identifier: None found. Study registration was assessed by HomVEE beginning with the 2014 review.

Study Effects Details

Outcome domain: Child Development and School Readiness Outcome Effects
Rating Outcome Measure Effect Sample Timing of Follow-up Sample Size Program Group Comparison Group Group Difference Effect size Statistical significance Outcome Type Notes
Moderate

Boehm Test of Basic Concepts -3 Preschool, Total Score

FavorableUnfavorableNo Effect

RCT: BabyFACE vs. comparision

3 years

67 families Adjusted mean = 26.84 Adjusted mean = 23.86 MD = 2.98 Study reported = 0.34

Not statistically significant, p = 0.09

Primary
Moderate

Boehm Test of Basic Concepts-3 Preschool - Total Score

FavorableUnfavorableNo Effect

RCT: BabyFACE vs. comparision

3 years

67 families Not reported Not reported Not Reported Not available

Not statistically significant, p=0.050

Primary

Although authors do not report the effect size, they indicate the finding is significant and favorable. Covariates included child's birth, household poverty, intergenerational living arrangement, and frequency of English spoken to children.

Moderate

Devereux Early Childhood Assessment (DECA) - Attachment

FavorableUnfavorableNo Effect

RCT: BabyFACE vs. comparision

2 years

66 families Adjusted mean = 26.63 Adjusted mean = 26.47 MD = 0.16 Study reported = 0.07

Not statistically significant, p = 0.69

Primary
Moderate

Devereux Early Childhood Assessment (DECA) - Attachment

FavorableUnfavorableNo Effect

RCT: BabyFACE vs. comparision

3 years

68 families Adjusted mean = 26.37 Adjusted mean = 25.90 MD = 0.47 Study reported = 0.15

Not statistically significant, p = 0.26

Primary
Moderate

Devereux Early Childhood Assessment (DECA) - Behavioral concerns

FavorableUnfavorableNo Effect

RCT: BabyFACE vs. comparision

3 years

68 families Not reported Not reported Not Reported Not available

Statistically significant, p = 0.03

Primary

Although authors do not report the effect size, they indicate the finding is significant and favorable. Covariates included child's birth, household poverty, intergenerational living arrangement, and frequency of English spoken to children.

Moderate

Devereux Early Childhood Assessment (DECA) - Behavioral concerns

FavorableUnfavorableNo Effect

RCT: BabyFACE vs. comparision

3 years

68 families Adjusted mean = 10.73 Adjusted mean = 13.00 MD = -2.27 Study reported = -0.47

Not statistically significant, p = 0.56

Primary
Moderate

Devereux Early Childhood Assessment (DECA) - Behavioral concerns

FavorableUnfavorableNo Effect

RCT: BabyFACE vs. comparision

2 years

66 families Adjusted mean = 11.06 Adjusted mean = 11.83 MD = -0.77 Study reported = -0.19

Not statistically significant, p = 0.22

Primary
Moderate

Devereux Early Childhood Assessment (DECA) - Initiative

FavorableUnfavorableNo Effect

RCT: BabyFACE vs. comparision

2 years

66 families Adjusted mean = 32.03 Adjusted mean = 30.37 MD = 1.66 Study reported = 0.27

Not statistically significant, p = 0.56

Primary
Moderate

Devereux Early Childhood Assessment (DECA) - Initiative

FavorableUnfavorableNo Effect

RCT: BabyFACE vs. comparision

3 years

68 families Adjusted mean = 32.35 Adjusted mean = 30.58 MD = 1.77 Study reported = 1.04

Not statistically significant, p = 0.19

Primary
Moderate

Devereux Early Childhood Assessment (DECA) - Self-control

FavorableUnfavorableNo Effect

RCT: BabyFACE vs. comparision

2 years

66 families Adjusted mean = 20.26 Adjusted mean = 20.53 MD = -0.27 Study reported = -0.38

Not statistically significant, p = 0.82

Primary
Moderate

Devereux Early Childhood Assessment (DECA) - Self-control

FavorableUnfavorableNo Effect

RCT: BabyFACE vs. comparision

3 years

68 families Adjusted mean = 20.42 Adjusted mean = 20.00 MD = 0.42 Study reported = 0.10

Not statistically significant, p = 0.35

Primary
Outcome domain: Positive Parenting Practices Outcome Effects
Rating Outcome Measure Effect Sample Timing of Follow-up Sample Size Program Group Comparison Group Group Difference Effect size Statistical significance Outcome Type
Moderate

Frequency of home literacy activity

FavorableUnfavorableNo Effect

RCT: BabyFACE vs. comparision

3 years

67 families Adjusted mean = 60.08 Adjusted mean = 55.45 MD = 4.63 Study reported = 0.64

Statistically significant, p = 0.01

Secondary
Moderate

Hours parent reads to child

FavorableUnfavorableNo Effect

RCT: BabyFACE vs. comparision

3 years

67 families Adjusted mean = 64.00 Adjusted mean = 33.73 MD = 30.27 Study reported = 0.49

Statistically significant, p = 0.04

Secondary
Moderate

Protective Factors Survey - Nurturing and Attachment

FavorableUnfavorableNo Effect

RCT: BabyFACE vs. comparision

3 years

68 families Adjusted mean = 26.87 Adjusted mean = 26.70 MD = 0.17 Study reported = 0.13

Not statistically significant, p = 0.56

Secondary
Moderate

Protective Factors Survey - Nurturing and Attachment

FavorableUnfavorableNo Effect

RCT: BabyFACE vs. comparision

2 years

66 families Adjusted mean = 27.06 Adjusted mean = 26.87 MD = 0.19 Study reported = 0.08

Not statistically significant, p = 0.55

Secondary
Moderate

Protective Factors Survey - Parent's Child Development Knowledge

FavorableUnfavorableNo Effect

RCT: BabyFACE vs. comparision

3 years

68 families Adjusted mean = 31.11 Adjusted mean = 30.03 MD = 1.08 Study reported = 0.27

Not statistically significant, p = 0.13

Secondary
Moderate

Protective Factors Survey - Parent's Child Development Knowledge

FavorableUnfavorableNo Effect

RCT: BabyFACE vs. comparision

2 years

66 families Adjusted mean = 30.43 Adjusted mean = 29.57 MD = 0.86 Study reported = 0.16

Not statistically significant, p = 0.21

Secondary

Study Outcome Measure Summary

Outcome Domain: Child Development and School Readiness Outcome Measures
Rating Outcome Measure Description of measure Data collection method Properties of measure Outcome Type
Moderate

Boehm Test of Basic Concepts-3

Assesses 26 basic concepts relevant to children ages 3 to 5

Child assessment

Internal consistency and test reliability range from .85 to .92.

Primary
Moderate

Devereux Early Childhood Assessment (DECA) - Attachment

A standarized behavior rating scale with initiative, self-control, attachment and behavior concerns scales.

Parent report

Cronbach's alpha is .81 and median test-retest reliability is .64.

Primary
Moderate

Devereux Early Childhood Assessment (DECA) - Behavioral concerns

A standarized behavior rating scale with initiative, self-control, attachment and behavior concerns scales.

Parent report

Cronbach's alpha is .71 and median test-retest reliability is .64.

Primary
Moderate

Devereux Early Childhood Assessment (DECA) - Initiative

A standarized behavior rating scale with initiative, self-control, attachment and behavior concerns scales.

Parent report

Cronbach's alpha is .86 and median test-retest reliability is .64.

Primary
Moderate

Devereux Early Childhood Assessment (DECA) - Self-control

A standarized behavior rating scale with initiative, self-control, attachment and behavior concerns scales.

Parent report

Cronbach's alpha is .83 and median test-retest reliability is .64.

Primary
Outcome Domain: Positive Parenting Practices Outcome Measures
Rating Outcome Measure Description of measure Data collection method Properties of measure Outcome Type
Moderate

Frequency of home literacy activity

A list of 14 literacy activities rated on a 5-point scale from rarely/never to daily/several times a day

Parent report

Cronbach's alpha is .88

Secondary
Moderate

Hours parent reads to child, RCT

Hours read to child

Parent report

Not applicable

Secondary
Moderate

Protective Factors Survey - Nurturing and Attachment

The Protective Factors Survey is a self-administered survey with five subscales: family functioning/resiliency, social support, concrete support, nurturing and attachment, and child development knowledge.

Parent report

Cronbach's alpha is .78

Secondary
Moderate

Protective Factors Survey - Parent's Child Development Knowledge

The Protective Factors Survey is a self-administered survey with five subscales: family functioning/resiliency, social support, concrete support, nurturing and attachment, and child development knowledge.

Parent report

Cronbach's alpha is .69

Secondary