Chapter 11. Do Certain Interventions Work Better When
11.2 Differential Impacts Depending on
Exhibit 11-3 illustrates possible differential impacts of in - terventions based on number of psychosocial challenges by showing the estimated size of the impact of each contrast (for example, SUB versus UC) at the 20th and 80th percen-tiles of challenge. The 20th percentile is 0 challenges (that is, more than 20 percent of families reported no challenges) and the 80th percentile is 4.0 challenges. The asterisks in the exhibit reflect whether the variation in impact by level of each psychosocial challenge is statistically significant.
For example, the first row considers impacts for the con-firmatory outcome of at least 1 night homeless or doubled
up in the past 6 months or at least 1 night in shelter in the past 12 months. In the first pair of columns, the impact of SUB in comparison with UC is estimated to be a reduction of 28.54 percentage points in this outcome for those with low numbers of challenge and 26.73 percentage points for those with high numbers of challenges. The average effect is very large and significant, as reported previously in Chapter 6.
The two impact estimates are nearly the same, however—
the variation in impact based on number of psychosocial challenges is trivial.
As in other chapters of this report, we consider findings that are statistically significant at the .10 level or better as evidence of differences in impact magnitude for families with different degrees of challenges. Moreover, as in other chapters, we take into account the number of statistically significant findings relative to the number of tests conducted. For each policy comparison, such as SUB versus UC, we test for variation in impact for each of the 18 key outcome variables included in the executive summary, and so might expect 2 of the 18 tests conducted to reach statistical significance on the basis of
Exhibit 11-2. Percentage of Families Reporting That a Condition Was a Big or Small Problem in Finding a Place to Live at the Time of Study Enrollment (for families interviewed at 18 months)
Housing Barriers Percent Reporting Big or Small Problem
Not enough income to pay rent 96.7%
Inability to pay a security deposit or first/last month’s rent 94.2
Not currently employed 79.9
Poor credit history 73.3
Lack of transportation to look for housing 65.5
No reference from past landlords 43.9
Past eviction 39.6
No rent history at all 39.0
Recently moved to community and no local rent history 32.6
Problems with past landlords 19.6
Three or more children in the household 17.8
Racial discrimination 17.4
Past lease violations 16.8
Someone in the household less than 21 years old 8.5
Teenagers in the household 5.5
Source: Family Options Study baseline survey
Short-Term Impacts of Housing and Services Interventions for Homeless Families 130 When Offered to Families Who Face Greater Difficulties?
Exhibit 11-3. Impacts Moderated by Psychosocial Challenges Index
Outcome SUB vs. UC CBRR vs. UC PBTH vs. UC SUB vs. CBRR SUB vs. PBTH CBRR vs. PBTH Impact at Low vs. High
Challenge Low High Low High Low High Low High Low High Low High
Housing stability
At least 1 night homelessa or doubled up in past 6 months or in shelter in past 12 months (%)
– 28.54 – 26.73 – 1.17 – 7.29 – 3.63 – 8.16 – 35.84 – 20.68* – 39.16 – 23.21 – 6.74 14.38*
At least 1 night homelessa or
doubled up in past 6 months (%) – 19.75 – 27.39 2.90 – 9.27 1.92 – 7.62 – 27.17 – 16.03 – 34.19 – 19.42 – 3.89 13.92 Number of places lived in past
6 months – 0.31 – 0.39 0.03 – 0.17 – 0.05 – 0.07 – 0.40 – 0.12 – 0.37 – 0.34 – 0.11 0.10
Any stay in emergency shelter in
months 7 to 18 after RA (%) – 17.50 – 10.02 – 5.01 – 0.75 – 9.87 – 7.06 – 18.46 – 9.21 – 19.81 – 7.99 3.63 – 0.59 Family preservation
Family has at least one child
separated in past 6 monthsb (%) – 8.53 – 5.87 2.46 – 5.14 – 1.46 – 0.04 – 7.30 3.57* – 8.67 – 4.06 8.36 – 5.71**
Spouse/partner separated in past 6 months, of those with spouse/
partner present at RAc (%)
– 0.02 1.49 17.06 2.95 – 3.16 3.51 – 23.77 – 8.98 – 15.13 12.10 14.79 – 1.24
Family has at least one child reunified, of those families with at least one child absent at RAd (%)
8.50 3.22 10.25 3.02 0.20 7.32 1.23 – 1.54 18.03 30.18 – 15.86 6.90
Adult well-being
Health in past 30 days was poor
or fair (%) 6.68 – 5.21* – 2.44 – 4.27 – 0.43 3.50 – 1.96 1.53 3.13 – 8.90 – 3.90 – 11.74
Psychological distresse – 0.47 – 1.27 – 0.29 – 0.86 – 0.50 – 0.12 – 0.76 – 0.11 – 0.76 – 1.62 – 1.55 – 1.80 Alcohol dependence or drug
abusef (%) – 3.15 – 5.59 – 1.61 – 5.01 0.65 – 0.22 0.71 0.12 – 1.66 – 9.49 – 2.44 – 10.62
Experienced intimate partner
violence in past 6 months (%) – 0.56 – 11.37** – 2.68 0.21 2.00 – 5.16 – 1.33 – 11.54* – 2.96 – 3.66 – 2.11 0.72 Child well-being
Number of schools attended since
RAg – 0.16 – 0.25 – 0.03 – 0.05 – 0.03 – 0.11 – 0.02 – 0.42*** – 0.07 – 0.20 – 0.24 0.24**
Childcare or school absences in
last monthh – 0.14 – 0.16 0.05 – 0.26* 0.04 0.07 – 0.14 0.06 – 0.03 – 0.18 0.26 – 0.44***
Poor or fair health (%) 0.28 0.64 – 2.11 1.87 2.79 2.34 2.50 – 2.95* – 2.30 – 0.06 – 8.92 0.06**
Behavior problemsi – 0.09 – 0.16 – 0.12 – 0.07 – 0.08 – 0.18 0.14 0.03 0.10 0.23 – 0.08 0.17
Self-sufficiency
Work for pay in week before
survey (%) – 2.99 – 6.70 – 6.81 6.46** 5.53 5.77 – 1.23 – 6.91 – 14.99 – 9.65 – 15.46 – 1.30
Total family income ($) – $1,146 $176 $682 $1,379 $649 $1,374 – $843 – $1,184 – $1,536 – $1,886 – $1,176 $1,490 Household is food insecure (%) – 11.21 – 7.40 – 2.19 – 10.88 – 0.12 – 3.16 – 6.40 – 2.29 1.28 – 11.90 1.54 – 13.35 CBRR = community-based rapid rehousing. PBTH = project-based transitional housing. SUB = permanent housing subsidy. UC = usual care.
RA = random assignment.
*/**/*** Impact magnitude varies significantly with level of psychological challenge at the .10, .05, and .01 levels, respectively, using a two-tailed t-test.
a The definition of homeless in this report includes stays in emergency shelters and places not meant for human habitation. It excludes transitional housing.
b Percentage of families in which a child who was with the family at baseline became separated from the family in the 6 months before the 18-month survey.
c Percentage of families in which a spouse or partner who was with the family at baseline became separated from the family in the 6 months before the 18-month survey.
d Percentage of families in which at least one child was separated from the family at baseline and no child was reunited with the family at the time of the 18-month survey.
e Psychological distress is measured with the Kessler 6 (K6) scale and ranges from 0 to 24, with higher scores indicating greater distress. Impacts shown as standardized effect sizes. Effect sizes were standardized by dividing impacts by standard deviation for the UC group.
f Measures evidence of alcohol dependence or drug abuse using responses to the Rapid Alcohol Problems Screen (RAPS-4) and six items from the Drug Abuse Screening Test (DAST-10).
g Number of schools outcome is topcoded at four or more schools.
h Absences outcome is defined as 0 = no absences in past month; 1 = 1–2 absences; 2 = 3–5 absences; 3 = 6 or more absences.
i Behavior problems outcome is measured as the standardized Total Difficulties score from the Strengths and Difficulties Questionnaire (SDQ).
Notes: The Low estimate is calculated at the 20th percentile of the moderator in the full sample and the High estimate is calculated at the 80th percentile of the moderator. Impact mean difference estimates are regression-adjusted for baseline characteristics and are weighted to adjust for survey nonresponse. See Chapter 5 and Appendix B for outcome definitions.
Source: Family Options Study 18-month followup survey
chance alone. Where more than 2 test results exceed the .10 threshold of significance for a given policy comparison—but only modestly more—we also consider whether the statisti-cally significant findings conform to the hypothesis that the more intensive intervention in a given comparison will have larger impacts at higher levels of challenges or barriers. If this pattern does not hold, the existence of modestly more than the number of significant findings expected by chance is not credited as evidence of real impact variation.
11.2.1 SUB Versus UC
SUB, as the more intensive intervention, would be expected to have greater impact relative to UC for families with higher numbers of psychosocial challenges. With only 2 statistically significant test findings among 18 tests conducted—the num ber expected by chance alone if no true impact variation occurred—we cannot conclude that impacts of SUB in com - parison with UC differ for families with different numbers of psychological challenges. Instead, the results are best inter - preted as random variation. In general, the levels of psycho-social challenge that families experience do not moderate the substantial average differences between SUB and UC shown in previous chapters.
11.2.2 CBRR Versus UC
CBRR, as the least intensive of the active interventions, would be expected to work better in comparison with UC for families with low psychosocial challenges rather than those with high challenges. With only 2 statistically significant test findings among 18 tests conducted—fewer than the number expected by chance alone if no true impact variation occurred—we cannot conclude that the size of impacts families experience from CBRR compared with UC vary by their number of psychosocial challenges. In particular, no evidence shows that CBRR worked better for families with lower numbers of challenges.
11.2.3 PBTH Versus UC
As the more intensive intervention, and one that addresses psychosocial challenges directly, PBTH would be expected to have greater impact relative to UC for families with higher numbers of these challenges, but this was not the case. No differential effects met the threshold of statistical significance for any of the 18 outcomes examined.
11.2.4 SUB Versus CBRR
SUB, as the more intensive intervention, would be expected to have greater impact than CBRR for families with higher numbers of psychosocial challenges even if SUB does not
address these challenges directly. Of 5 statistically significant results (of 18 tests total), 2 point in the opposite direction:
SUB’s effects grows less favorable relative to CBRR as the num - ber of challenges increases for the confirmatory outcome (any stay in emergency shelter in the 12 months before the survey or a stay in a place not meant for human habitation or doubling up in the 6 months before the survey) and child separations from the family. In fact, the positive point esti-mate for the latter result implies that SUB causes more child separations relative to CBRR for families with a high number of challenges, which is counterintuitive. Of the 3 significant results that conform to the expected pattern—(1) experience of intimate partner violence, (2) number of schools attend-ed, and (3) children’s poor or fair health—one seems to be contradictory (for high-challenge families, SUB has a smaller effect on homelessness than CBRR but a larger reduction in the number of schools attended) and another involves a per - verse effect (for low-challenge families, SUB worsens children’s health relative to CBRR). This inconsistent pattern of effects prevents us from concluding that the size of impacts families experience from SUB compared with CBRR varies by number of psychosocial challenge. These results should be interpreted as spurious, the result of random variation in the data.
11.2.5 SUB Versus PBTH
Because PBTH addresses psychosocial needs more directly than SUB, proponents would expect it to be especially ben-eficial for families with higher numbers of these challenges.
Proponents of SUB make the opposite prediction. With no statistically significant test finding among 18 tests conducted, the evidence does not confirm either point of view. We con - clude that psychological challenges do not moderate the size of impacts families experience from PBTH compared with SUB.
11.2.6 CBRR Versus PBTH
As the more intensive intervention, PBTH would be expected to have greater impact relative to CBRR for families with higher psychosocial challenges. Of 5 statistically significant findings (of 18 tests total), 2 point in the opposite direction:
CBRR’s effects grow more favorable relative to PBTH as the number of challenges increases for child separations and child absences from childcare or school. The effect of PBTH grows more favorable relative to CBRR as challenges increase in 3 significant results: (1) the confirmatory homeless out - come, (2) the number of schools attended, and (3) poor or fair child health. For health, however, the result simply brings the two interventions to parity (at high levels of chal-lenge) from an initial advantage favoring CBRR (at low levels of challenge). With only 2 of 5 results conforming well to the postulated pattern, we are cautious in interpreting the
Short-Term Impacts of Housing and Services Interventions for Homeless Families 132 When Offered to Families Who Face Greater Difficulties?
evidence as anything other than chance differences that hap-pened to occur in this instance at more than the expected rate (5 in 18 times, rather than 2 in 18).