Assessing Evidence of Effectiveness
After completing all impact study reviews for a model, the HomVEE team evaluated the evidence across all studies of the model that received a high or moderate rating and measured outcomes in at least one of the eligible outcome domains. The review included outcomes for pregnant women or families with children ages 0-5. If the sample included children outside of that age range, only subgroup analyses of children or families with children in the targeted age range were used in the review. In addition to assessing whether each model met the HHS criteria for an evidence-based early childhood home visiting service delivery model, the HomVEE team examined other aspects of the evidence for each model, including the following:
- Quality of Outcome Measures. HomVEE classified outcome measures as primary if data were collected through direct observation, direct assessment, or administrative records; or if self-reported data were collected using a standardized (normed) instrument. Other self-reported measures are classified as secondary.
- Duration of Impacts. To provide information on the length of follow-up, HomVEE noted when the outcomes were measured.
- Sustained Impacts. HomVEE classified impacts as sustained if they were measured at least one year after program enrollment.
- Replication of Impacts. HomVEE classified impacts as replicated if favorable, statistically significant impacts were shown in the same outcome domain in at least two non-overlapping analytic study samples.
- Subgroup Findings. HomVEE reports subgroup findings if such findings are replicated in the same outcome domain in at least two studies using different analytic samples.
- Unfavorable Impacts. In addition to favorable impacts, HomVEE reports unfavorable, statistically significant impacts on full sample and subgroup findings. While some outcomes are clearly unfavorable (such as an increase in children’s behavior problems), others are ambiguous. For example, an increase in the number of days mothers are hospitalized could indicate an increase in health problems or increased access to needed health care due to participation in a home visiting program.
- Evaluator Independence. HomVEE reported the funding source for each study and whether any of the study authors were model developers.
- Magnitude of Impacts. HomVEE reported effect sizes when possible, either those calculated by the study authors or HomVEE computed findings.