Review Process

Prioritizing Models for Review

Each year, HomVEE releases new review results for models. This includes reviews of studies on models that have not previously been reviewed, updates to previously reviewed models, or both. Decisions on the number of models to review depend on (1) the number of studies that are identified for review about each model and (2) the available project resources. The process by which HomVEE selects models for review is called the prioritization process.

Prioritization Process

HomVEE selects models for the annual review by calculating a prioritization score for each model and then reviewing models with the highest scores. The prioritization score is based on points assigned at the study and model levels. HomVEE divides reviews into two tracks. Track 1 is for models that are not evidence-based (that is, models that either HomVEE has never reviewed before or that HomVEE has reviewed but did not meet the criteria for evidence of effectiveness). Track 2 updates the review of literature on evidence-based models. HomVEE prioritizes models separately in each track, but the process is largely similar for both. The prioritization process occurs in six steps:

1. Identify studies eligible for review, which includes the HomVEE Literature Search and Screening Studies activities.

2. Apply study-level criteria

3. Apply model-level criteria

4. Calculate prioritization scores

5. Refine prioritization scores

6. Prioritize models

Below, we describe Steps 2 through 5 in the prioritization process and how it differs for models in Tracks 1 and 2.

Step 2. Apply study-level criteria. HomVEE reviews the titles and abstracts of impact studies identified for each model and assigns points based on HomVEE’s prioritization criteria. This process is identical for both tracks. Models can earn up to 5.75 points for each eligible impact study (Table 1). HomVEE assesses each study (manuscript) separately and then sums the points for all studies about a model. Therefore, models with more eligible studies tend to receive more study-level points. Whether a model is already evidence based determines which studies are included in that model’s study-level point total:

  • If a model is not evidence based (Track 1), the total includes study-level points for studies that HomVEE reviewed in previous years and assigned a high or moderate rating as well as studies that HomVEE has not previously reviewed.
  • If a model is already evidence based (Track 2), the total includes points only for studies that HomVEE has not reviewed yet.
Table 1. HomVEE study-level prioritization criteria and associated points
Criterion Points Notes

Number and design of impact studies

2 to 3 per study

3 points for each randomized controlled trial, single-case design, or regression discontinuity design

2 points for each matched-comparison group designa

Sample size

1 per study

Study sample contains 250 or more pregnant women and/or families

Outcomes of interest

1 per study

Study examines outcomes in one or more of the following domains: reductions in child maltreat­ment; reductions in juvenile delinquency, family violence, or crime; linkages and referrals; and family economic self-sufficiencyb

Study sample

0.5 per study

Study sample lives in the United States or is an indigenous population in or outside of the United States

Priority population

0.25 per study

The entire sample belongs to one or more priority populations named in the Maternal, Infant, and Early Childhood Home Visiting (MIECHV) statutec

Note: HomVEE applies these points at the study level based on information that study authors provide in the title and abstract. HomVEE assesses each study separately and then sums the points for all studies to create a study-level total for the model.

a Details of HomVEE’s standards for these designs are available on the HomVEE website: https://homvee.acf.hhs.gov/review-process/Producing%20Study%20Ratings.

b More information about these outcomes is available at https://homvee.acf.hhs.gov/outcomes.

c According to 42 U.S.C. § 711 (d)(4), priority populations are as follows:

  • Low-income families
  • Families with pregnant women who have not reached age 21
  • Families that have a history of child abuse or neglect or have had interactions with child welfare services
  • Families that have a history of substance abuse or need substance abuse treatment
  • Families that have users of tobacco products in the home
  • Families that are or have children with low student achievement
  • Families with children with developmental delays or disabilities
  • Families that include individuals who are serving or formerly served in the Armed Forces, including such families that have members of the Armed Forces who have had multiple deployments outside of the United States

 

Step 3. Apply model-level criteria. HomVEE assigns model-level points based on information from study titles and abstracts, model websites, and previous HomVEE reviews. HomVEE may contact study authors or model developers to confirm publicly available information. This process is identical for Tracks 1 and 2 (models that are and are not evidence based). Models can earn up to 4 points in this step:

  • 1 point if the model is associated with a national organization or institution of higher education (organizations can be in or outside the United States)
  • 1 point if the model is currently serving or available to serve families
  • 1 point if the model has been implemented for at least three years (even if it is not currently active)
  • 1 point if support is available to implement the model in the United States

These model-level factors are specific MIECHV-relevant criteria, in order to more closely align HomVEE with the MIECHV Program.

Step 4. Calculate prioritization scores. After assigning study- and model-level points, HomVEE sums all points to calculate a model’s point total. For models that are not yet evidence based (Track 1), the total is the final model prioritization score. For models that are evidence based (Track 2), there is one additional step: HomVEE assigns a weight to the Track 2 model score based on the number of years since HomVEE last reviewed the model and released a report. The weight is calculated using the following formula:

Weight = [1 + 0.1 * (current year – release date of prior report)]2

For example, a model considered for review in 2019 for which its most recent Hom­VEE report was released in 2015 would receive a weight of [1 + 0.1 * (2019 – 2015)]2 = 1.96.

After calculating weights, a model’s final prioritization score is then calculated as: Prioritization score = Model point total * Weight

The weights permit Track 2 (evidence-based) models with new research to be updated periodically. Models that were reviewed longer ago have a higher weight than models reviewed more recently. This increases the relative likelihood that a model that has not been reviewed recently will be prioritized for review.

Step 5. Refine prioritization scores. After calculating prioritization scores, HomVEE sorts models from highest to lowest score separately within each track. The team then conducts a second, focused database search on model names to identify additional studies about top-scoring models in each track. The model’s prioritization score is adjusted to add the study-level points for the newly identified studies. Additionally, HomVEE examines the full texts of all screened-in studies about top-scoring models and then adjusts the study-level point totals (and the model’s corresponding prioritization score) based on information available from the full text.

Step 6. Prioritize models. HomVEE re-sorts models from highest to lowest using the adjusted prioritization scores and identifies models with the highest scores. After prioritizing a model, HomVEE reviews all new impact studies about that model, with two exceptions:

  • In years when resources are limited, HomVEE will not review research conducted outside the United States if it is about a model that is already evidence based. (Research with indigenous communities outside of the United States will still be reviewed.) If studies conducted outside of the United States are not reviewed, the model report on the HomVEE website will clearly indicate which research was and was not included in the updated report.
  • Regardless of whether the model has previously been found to be evidence-based or not, HomVEE will not update results for previously reviewed models every year. Models are only considered for updates every two years at the earliest. For example, if review results for a model were updated in 2019, that model would not be considered for additional updating until 2021 or later.

For more information about the review process and subsequent steps, see https://homvee.acf.hhs.gov/review-process/Overview.

HomVEE’s prioritization process reflects HomVEE’s emphasis on identifying new evidence-based home visiting models while continuing to update reports on models that are already evidence based. The annual prioritization effort may yield more models in the highest point category than can be reviewed that year. The number of models reviewed each year depends on the available project resources and the number of studies identified to review for each model. Regardless of whether a model is reviewed in a given year, all models will be included in the prioritization process in subsequent years. The MIECHV Program may coordinate with HomVEE to prioritize review of promising approaches implemented and evaluated under a MIECHV grant.1

1 Under federal law, a home visiting service delivery model that qualifies as a promising approach conforms to a “promising and new approach” to achieving specified benchmark areas and participant outcomes; has been developed or identified by a national organization or institution of higher education; and will be evaluated through a well-designed and rigorous process (see Social Security Act, Title V, § 511 (d); https://www.ssa.gov/OP_Home/ssact/title05/0511.htm).

View Revisions