Predictions Regarding What Works - What Works for Whom?

Chapter 3. Hypotheses About Intervention Effects

3.5 What Works for Whom?

3.5.1 Predictions Regarding What Works

The study team developed two hypotheses about the relative effects of the interventions on families with varying levels of housing barriers and psychosocial challenge (see box).

Hypotheses About Which Interventions Work for Whom

From the Perspective of SUB Proponents

H10. Relative to all other interventions, SUB will have larger effects on stability for families with higher scores on the housing barriers index and may have larger effects on stability for families with higher scores on the psycho-social challenges index. Differential effects may extend to self-sufficiency, adult well-being, child well-being, and family preservation.

From the Perspective of PBTH Proponents

H11. Relative to all other interventions, PBTH will have larger effects on stability, self-sufficiency, and adult well- being for families with higher scores on the psychosocial challenges index and may have larger effects on these outcomes for families with higher scores on the housing barriers index. Differential effects may extend to child well-being and family preservation.

Short-Term Impacts of Housing and Services Interventions for Homeless Families 34

CHAPTER 4. DATA SOURCES AND METHODOLOGY

T

his chapter describes the data sources and method-ology employed in analysis of the 20-month impact estimates for the Family Options Study. The first section describes the data sources used to (1) measure pro - gram use by study families, (2) analyze the impacts of study interventions on families, and (3) describe study interventions and their costs. The second section provides an overview of the method used to estimate the impacts of interventions.

Additional detail about the impact methodology is provided in Appendix C.

4.1 Data Sources

The study has collected data from a wide range of sources, including study families, intervention providers, and admin-istrative data systems. Exhibit 4-1 shows the sources of data used in this report and provides information about how they were collected and their content. Key aspects of the data collection are described in the subsections that follow.

Exhibit 4-1. Data Sources Used in the Report (1 of 2)

Data Source Collection Process Data Source Collects or Measures…

From study implementation Random assignment enrollment data

(n = 2,282) • Recorded in web-based enrollment and random assignment tool, based on information entered by field interviewer and point-in-time intervention availability

• Name, date of birth, and Social Security number of family head and spouse or partner

• Eligibility screening responses

• Intervention availability at random assignment

• Random assignment result From study families

Baseline survey (n = 2,282) • In-person survey (40 minutes) conducted immediately before random assignment

• Completed for the full sample of families randomly assigned

• Demographic characteristics

• Preshelter housing

• Housing barriers

• Homelessness history

• Employment

• Family composition

• Income and income sources

• Family head: physical health

• Family head: mental health and experiences of trauma

6- and 12- month tracking surveys

(n = 1,671; n = 1,632) • Telephone survey (10 minutes) conducted 6 and

12 months after random assignment • Family composition

• Current housing status

• Use of homeless and housing programs 18-month followup adult survey

(n = 1,857) • In-person or telephone survey (60 minutes) conducted

at least 18 months after random assignment • Current housing status

• Experience of homelessness

• Use of homeless and housing programs

• Housing quality and affordability of current unit

• Employment and earnings

• Income and income sources

• Material hardship

• Family composition and preservation

• Adult well-being

• Child well-being (for up to two focal children)

• Receipt of services 18-month followup child assessments • In-person child assessments (50 minutes) conducted

for focal children who were ages 3 years, 6 months to 6 years, 11 months

• Collection attempted only if family head responded to followup adult survey

• Verbal ability (Woodcock-Johnson III letter-word identification test; n = 876)

• Math ability (Woodcock-Johnson III applied problems test; n = 846)

• Self-regulation (Head Toes Knees Shoulders assessment; n = 780)

Exhibit 4-1. Data Sources Used in the Report (2 of 2)

Data Source Collection Process Data Source Collects or Measures…

From study families (continued) 18-month followup child survey

(n = 945) • In-person or telephone survey (30 minutes) conducted for focal children who were ages 8 to 17 years

• Collection attempted only if family head responded to followup adult survey

• Mental health

• Experiences of traumatic events

• Substance use

• School effort

• Arrests or police involvement From study intervention providers

Enrollment verification data • Study team verified (by telephone and e-mail) whether families enrolled in the programs to which they were referred

• Conducted from September 2010 to September 2012

• Use of assigned intervention

Program information • Study team conducted site visits and staff interviews

• Conducted from June 2011 to April 2012 • Provider information

• Characteristics of housing assistance

• Characteristics of services Program cost information • Study team conducted site visits and staff interviews

• Collected audited expense statements, program budgets, staffing lists, partner commitment letters, and program staff estimates of costs not reflected in expense statements

• Conducted from November 2012 to August 2013

• Overhead costs

System (HMIS) • Individual-level records collected from community and government administrators of the Homeless Management Information System (HMIS; one or more per site)

• Participation in homeless assistance programs covered in HMIS (including emergency shelter, rapid re-housing, transitional housing, and permanent supportive housing)

HUD Public and Indian Housing

Information Center (PIC) • Individual-level data collected from HUD • Receipt of housing assistance through HUD’s Housing Choice Voucher (HCV) and public housing programs

HUD Tenant Rental Assistance

Certification System (TRACS) • Individual-level data collected from HUD • Receipt of housing assistance through project-based Section 8 programs

From combination of sources

Program usage data • Combines data from seven sources: enrollment verification; 6-, 12-, and 18-month surveys; HMIS; HUD PIC; and the Tenant Rental Assistance Certification System (TRACS)

• Participation in seven types of homeless and housing assistance programs (by calendar month after random assignment)

Notes: All surveys conducted with family head collected or updated family contact information for tracking purposes. Additional information about data collection from study families provided in Appendix A. Additional information about program cost data collection provided in Appendix G.

4.1.1 Baseline Data Collection

Local field interviewers enrolled families into the study in person at the emergency shelters where the families were staying. Enrollment began in September 2010 and was com-pleted in January 2012. During enrollment, the interviewer would first inform the family about the study. If the family consented to participate in the study, the interviewer would then ask eligibility screening questions for programs that had available program slots. Finally, the interviewer would administer the baseline survey using Computer-Assisted Personal Interviewing, or CAPI, software. All family heads who consented to participate in the study completed the baseline survey before random assignment.

In families with only one adult present, that individual was interviewed. For families headed by couples, the study team interviewed women. There were two reasons for this prefer-ence: (1) Some homeless assistance programs exclude men,

and in cases of family separations the children are more likely to remain with the mother; and (2) some outcome measures such as psychological distress have different distributions for men and women in the population at large, so this pref-erence results in having greater homogeneity in the sample.

4.1.2 Followup Data Collection

The followup data collection effort was conducted from July 2012 through October 2013. The study team attempted to contact families for the study’s followup survey beginning in the 18th month after random assignment. The median time from random assignment to the followup survey was 20 months. The followup period referred to in the report thus covers a period of 20 months after random assignment, al-though the survey is sometimes referred to as the 18-month followup survey. The data collection consisted of three components.

Short-Term Impacts of Housing and Services Interventions for Homeless Families 36 Chapter 4. Data Sources and Methodology

• Followup adult survey.

• Followup child assessments.

• Followup child survey.

The study attempted to complete the adult survey with all 2,282 family heads. For families headed by couples, the same adult interviewed at baseline was interviewed at followup.

Up to two focal children per family were randomly selected during the administration of the adult survey.³⁶ Information about the well-being of the focal children was collected from the parent in the adult survey and directly from appropriately aged children in either the child assessments or the child survey. The adult survey needed to be completed before attempts were made to complete child assessments and the child survey with focal children (both so parental consent could be given and so focal children could be selected).

Focal children needed to—

1. Have been either identified in the baseline survey as part of the family or born after random assignment.

2. Have been ages 1 to 17 years at the time of the followup adult survey.

3. Have been with the family at the time of the followup adult survey or had enough contact with the family head so that their parent (the family head) was knowledgeable about key aspects of their lives.³⁷

The focal child selection process oversampled children who were ages 3 to 17, and with the family at both baseline and followup, in order to maximize the number of children from whom data were directly collected (in the child assessments and child survey).³⁸

4.1.3 Construction of Program Usage Data

To construct a new dataset with information about program usage, the study team combined data from seven sources:

enrollment verification; 6-, 12-, and 18-month surveys; the Homeless Management Information System (HMIS); the HUD Public and Indian Housing Information Center (PIC); and the Tenant Rental Assistance Certification System (TRACS).

The dataset contains monthly participation information for the entire period of observation (from random assignment to the time of the followup survey), spanning seven program types. Exhibit 4-2 shows the seven program types and the data sources for each type.

The data are structured so that they count 1 month of par-ticipation in a program type if the family uses that program type for at least 1 night during the month. The data contain 3,573 spells of program use for the 1,857 followup survey respondent families.³⁹ The structure of the data could bias estimates of program duration upward (relative to true use) because (1) program entries are spread throughout the cal-endar month and (2) program exits for emergency shelter, transitional housing, and permanent supportive housing are typically spread throughout the calendar month (rather than occurring at the end of the calendar month). To address this upward bias, each program spell is adjusted slightly downward in measures of duration. (See Appendix A for full details.) These data are known to miss at least some program use.

The baseline stay in emergency shelter does not appear in the data for 20 percent of survey respondent families. The missing data rate for subsequent stays in emergency shelter is unknown. The study team expects that HMIS records on community-based rapid re-housing, transitional housing, and permanent housing to be at least as complete as the baseline emergency shelter records (at least 80 percent). Because the data on these three program types also rely on enrollment verification (for the referred program) and up to three self- reports, the study team expects the vast majority of program spells of these types to be captured in the data.

The data on use of subsidy, public housing, and project-based vouchers or Section 8 projects should be essentially complete because they are based on HUD administrative records. Addi - tional detail about the construction of the program usage data is provided in Appendix A.

The remainder of the chapter describes the methodology used to calculate the 20-month impact estimates in the report.

36 The survey software randomly selected the focal children immediately after the focal child screener, the first module in the adult survey. The subsequent module on child well-being then asked items about the focal child or focal children who had been selected.

37 About 530 children (out of about 4,200 total) screened for selection as focal children were living with the family head less than half the time. The family head was knowledgeable regarding only 60 of these 530 children. Thus, in accordance with the focal child selection protocol, these 60 children were selected as focal children (along with 2,724 other selected focal children who were living with the family head at least half of the time). During analysis, however, it was decided that such a small number of children would not allow estimates to generalize to the whole group of largely absent children. Therefore, these 60 children were not included in impact analyses. As a result, the child impact results generalize only to children living with the family head half of the time or more at the time of the adult survey.

38 The oversampling criterion of being with the family at baseline was included so that oversampled children would be directly affected by the study’s random assignment.

Children needed to be with the family at followup for the study team to attempt collection of child assessments or the child survey. (The study did not attempt to locate children separated from the family.)

39 A spell of program use is a continuous period of use, with a single starting month and a single ending month.

Exhibit 4-2. Program Types and Their Data Sources in the Program Usage Data

Program Type Data Sources

Emergency shelter • HMIS records