Predictions Regarding What Works

문서에서 Family Options Study (페이지 65-0)

Chapter 3. Hypotheses About Intervention Effects

3.5 What Works for Whom?

3.5.1 Predictions Regarding What Works

The study team developed two hypotheses about the relative effects of the interventions on families with varying levels of housing barriers and psychosocial challenge (see box).

Hypotheses About Which Interventions Work for Whom

From the Perspective of SUB Proponents

H10. Relative to all other interventions, SUB will have larger effects on stability for families with higher scores on the housing barriers index and may have larger effects on stability for families with higher scores on the psycho-social challenges index. Differential effects may extend to self-sufficiency, adult well-being, child well-being, and family preservation.

From the Perspective of PBTH Proponents

H11. Relative to all other interventions, PBTH will have larger effects on stability, self-sufficiency, and adult well- being for families with higher scores on the psychosocial challenges index and may have larger effects on these outcomes for families with higher scores on the housing barriers index. Differential effects may extend to child well-being and family preservation.

Short-Term Impacts of Housing and Services Interventions for Homeless Families 34

CHAPTER 4.

DATA SOURCES AND METHODOLOGY

T

his chapter describes the data sources and method-ology employed in analysis of the 20-month impact estimates for the Family Options Study. The first section describes the data sources used to (1) measure pro - gram use by study families, (2) analyze the impacts of study interventions on families, and (3) describe study interventions and their costs. The second section provides an overview of the method used to estimate the impacts of interventions.

Additional detail about the impact methodology is provided in Appendix C.

4.1 Data Sources

The study has collected data from a wide range of sources, including study families, intervention providers, and admin-istrative data systems. Exhibit 4-1 shows the sources of data used in this report and provides information about how they were collected and their content. Key aspects of the data collection are described in the subsections that follow.

Exhibit 4-1. Data Sources Used in the Report (1 of 2)

Data Source Collection Process Data Source Collects or Measures…

From study implementation Random assignment enrollment data

(n = 2,282) • Recorded in web-based enrollment and random assignment tool, based on information entered by field interviewer and point-in-time intervention availability

• Name, date of birth, and Social Security number of family head and spouse or partner

• Eligibility screening responses

• Intervention availability at random assignment

• Random assignment result From study families

Baseline survey (n = 2,282) • In-person survey (40 minutes) conducted immediately before random assignment

• Completed for the full sample of families randomly assigned

• Demographic characteristics

• Preshelter housing

• Housing barriers

• Homelessness history

• Employment

• Family composition

• Income and income sources

• Family head: physical health

• Family head: mental health and experiences of trauma

6- and 12- month tracking surveys

(n = 1,671; n = 1,632) • Telephone survey (10 minutes) conducted 6 and

12 months after random assignment • Family composition

• Current housing status

• Use of homeless and housing programs 18-month followup adult survey

(n = 1,857) • In-person or telephone survey (60 minutes) conducted

at least 18 months after random assignment • Current housing status

• Experience of homelessness

• Use of homeless and housing programs

• Housing quality and affordability of current unit

• Employment and earnings

• Income and income sources

• Material hardship

• Family composition and preservation

• Adult well-being

• Child well-being (for up to two focal children)

• Receipt of services 18-month followup child assessments • In-person child assessments (50 minutes) conducted

for focal children who were ages 3 years, 6 months to 6 years, 11 months

• Collection attempted only if family head responded to followup adult survey

• Verbal ability (Woodcock-Johnson III letter-word identification test; n = 876)

• Math ability (Woodcock-Johnson III applied problems test; n = 846)

• Self-regulation (Head Toes Knees Shoulders assessment; n = 780)

Exhibit 4-1. Data Sources Used in the Report (2 of 2)

Data Source Collection Process Data Source Collects or Measures…

From study families (continued) 18-month followup child survey

(n = 945) • In-person or telephone survey (30 minutes) conducted for focal children who were ages 8 to 17 years

• Collection attempted only if family head responded to followup adult survey

• Mental health

• Experiences of traumatic events

• Substance use

• School effort

• Arrests or police involvement From study intervention providers

Enrollment verification data • Study team verified (by telephone and e-mail) whether families enrolled in the programs to which they were referred

• Conducted from September 2010 to September 2012

• Use of assigned intervention

Program information • Study team conducted site visits and staff interviews

• Conducted from June 2011 to April 2012 • Provider information

• Characteristics of housing assistance

• Characteristics of services Program cost information • Study team conducted site visits and staff interviews

• Collected audited expense statements, program budgets, staffing lists, partner commitment letters, and program staff estimates of costs not reflected in expense statements

• Conducted from November 2012 to August 2013

• Overhead costs

System (HMIS) • Individual-level records collected from community and government administrators of the Homeless Management Information System (HMIS; one or more per site)

• Participation in homeless assistance programs covered in HMIS (including emergency shelter, rapid re-housing, transitional housing, and permanent supportive housing)

HUD Public and Indian Housing

Information Center (PIC) • Individual-level data collected from HUD • Receipt of housing assistance through HUD’s Housing Choice Voucher (HCV) and public housing programs

HUD Tenant Rental Assistance

Certification System (TRACS) • Individual-level data collected from HUD • Receipt of housing assistance through project-based Section 8 programs

From combination of sources

Program usage data • Combines data from seven sources: enrollment verification; 6-, 12-, and 18-month surveys; HMIS; HUD PIC; and the Tenant Rental Assistance Certification System (TRACS)

• Participation in seven types of homeless and housing assistance programs (by calendar month after random assignment)

Notes: All surveys conducted with family head collected or updated family contact information for tracking purposes. Additional information about data collection from study families provided in Appendix A. Additional information about program cost data collection provided in Appendix G.

4.1.1 Baseline Data Collection

Local field interviewers enrolled families into the study in person at the emergency shelters where the families were staying. Enrollment began in September 2010 and was com-pleted in January 2012. During enrollment, the interviewer would first inform the family about the study. If the family consented to participate in the study, the interviewer would then ask eligibility screening questions for programs that had available program slots. Finally, the interviewer would administer the baseline survey using Computer-Assisted Personal Interviewing, or CAPI, software. All family heads who consented to participate in the study completed the baseline survey before random assignment.

In families with only one adult present, that individual was interviewed. For families headed by couples, the study team interviewed women. There were two reasons for this prefer-ence: (1) Some homeless assistance programs exclude men,

and in cases of family separations the children are more likely to remain with the mother; and (2) some outcome measures such as psychological distress have different distributions for men and women in the population at large, so this pref-erence results in having greater homogeneity in the sample.

4.1.2 Followup Data Collection

The followup data collection effort was conducted from July 2012 through October 2013. The study team attempted to contact families for the study’s followup survey beginning in the 18th month after random assignment. The median time from random assignment to the followup survey was 20 months. The followup period referred to in the report thus covers a period of 20 months after random assignment, al-though the survey is sometimes referred to as the 18-month followup survey. The data collection consisted of three components.

Short-Term Impacts of Housing and Services Interventions for Homeless Families 36 Chapter 4. Data Sources and Methodology

• Followup adult survey.

• Followup child assessments.

• Followup child survey.

The study attempted to complete the adult survey with all 2,282 family heads. For families headed by couples, the same adult interviewed at baseline was interviewed at followup.

Up to two focal children per family were randomly selected during the administration of the adult survey.36 Information about the well-being of the focal children was collected from the parent in the adult survey and directly from appropriately aged children in either the child assessments or the child survey. The adult survey needed to be completed before attempts were made to complete child assessments and the child survey with focal children (both so parental consent could be given and so focal children could be selected).

Focal children needed to—

1. Have been either identified in the baseline survey as part of the family or born after random assignment.

2. Have been ages 1 to 17 years at the time of the followup adult survey.

3. Have been with the family at the time of the followup adult survey or had enough contact with the family head so that their parent (the family head) was knowledgeable about key aspects of their lives.37

The focal child selection process oversampled children who were ages 3 to 17, and with the family at both baseline and followup, in order to maximize the number of children from whom data were directly collected (in the child assessments and child survey).38

4.1.3 Construction of Program Usage Data

To construct a new dataset with information about program usage, the study team combined data from seven sources:

enrollment verification; 6-, 12-, and 18-month surveys; the Homeless Management Information System (HMIS); the HUD Public and Indian Housing Information Center (PIC); and the Tenant Rental Assistance Certification System (TRACS).

The dataset contains monthly participation information for the entire period of observation (from random assignment to the time of the followup survey), spanning seven program types. Exhibit 4-2 shows the seven program types and the data sources for each type.

The data are structured so that they count 1 month of par-ticipation in a program type if the family uses that program type for at least 1 night during the month. The data contain 3,573 spells of program use for the 1,857 followup survey respondent families.39 The structure of the data could bias estimates of program duration upward (relative to true use) because (1) program entries are spread throughout the cal-endar month and (2) program exits for emergency shelter, transitional housing, and permanent supportive housing are typically spread throughout the calendar month (rather than occurring at the end of the calendar month). To address this upward bias, each program spell is adjusted slightly downward in measures of duration. (See Appendix A for full details.) These data are known to miss at least some program use.

The baseline stay in emergency shelter does not appear in the data for 20 percent of survey respondent families. The missing data rate for subsequent stays in emergency shelter is unknown. The study team expects that HMIS records on community-based rapid re-housing, transitional housing, and permanent housing to be at least as complete as the baseline emergency shelter records (at least 80 percent). Because the data on these three program types also rely on enrollment verification (for the referred program) and up to three self- reports, the study team expects the vast majority of program spells of these types to be captured in the data.

The data on use of subsidy, public housing, and project-based vouchers or Section 8 projects should be essentially complete because they are based on HUD administrative records. Addi - tional detail about the construction of the program usage data is provided in Appendix A.

The remainder of the chapter describes the methodology used to calculate the 20-month impact estimates in the report.

36 The survey software randomly selected the focal children immediately after the focal child screener, the first module in the adult survey. The subsequent module on child well-being then asked items about the focal child or focal children who had been selected.

37 About 530 children (out of about 4,200 total) screened for selection as focal children were living with the family head less than half the time. The family head was knowledgeable regarding only 60 of these 530 children. Thus, in accordance with the focal child selection protocol, these 60 children were selected as focal children (along with 2,724 other selected focal children who were living with the family head at least half of the time). During analysis, however, it was decided that such a small number of children would not allow estimates to generalize to the whole group of largely absent children. Therefore, these 60 children were not included in impact analyses. As a result, the child impact results generalize only to children living with the family head half of the time or more at the time of the adult survey.

38 The oversampling criterion of being with the family at baseline was included so that oversampled children would be directly affected by the study’s random assignment.

Children needed to be with the family at followup for the study team to attempt collection of child assessments or the child survey. (The study did not attempt to locate children separated from the family.)

39 A spell of program use is a continuous period of use, with a single starting month and a single ending month.

Exhibit 4-2. Program Types and Their Data Sources in the Program Usage Data

Program Type Data Sources

Emergency shelter • HMIS records

• 6-, 12-, and 18-month surveys

Subsidy (housing choice voucher)a • HUD PIC and TRACS records

• Enrollment verification records (for referred program)

• 6-, 12-, and 18-month surveys

Community-based rapid re-housing • HMIS records

• Enrollment verification records (for referred program)

• 6-, 12-, and 18-month surveys

Transitional housingb • HMIS records

• Enrollment verification records (for referred program)

• 6-, 12-, and 18-month surveys

Permanent supportive housing • HMIS records

• 6-, 12-, and 18-month surveys

Public housing • HUD PIC and TRACS records

• 6-, 12-, and 18-month surveys

Project-based vouchers or Section 8 projects • HUD PIC and TRACS records

• 6-, 12-, and 18-month surveys

HMIS = Homeless Management Information System. HUD = U.S. Department of Housing and Urban Development. PIC = Public and Indian Housing Information Center.

TRACS = Tenant Rental Assistance Certification System.

a The subsidy program type represents housing choice vouchers plus site-specific programs that families were referred to when assigned to the permanent housing subsidy (SUB) intervention. The site-specific non-housing choice voucher programs were public housing in Honolulu, Hawaii, and project-based vouchers in Bridgeport, Connecticut. In other sites, these programs are coded separately.

b The transitional housing program type represents both project-based and scattered-site varieties of transitional housing, including transition-in-place units.

4.2 Methodology

This report presents separate impact estimates for each of the 6 pairwise comparisons of a single assignment arm to another assignment arm, plus 4 additional comparisons of pooled assignment arms to a single assignment arm (see Exhibit 1-1 and Chapters 6 through 10). All 10 comparisons have been analyzed separately using the same basic estimation model.

Pairwise Comparisons

SUB versus UC SUB versus CBRR CBRR versus UC SUB versus PBTH PBTH versus UC CBRR versus PBTH

Pooled Comparisons.

• What is impact of any kind of housing subsidy for homeless families (SUB + CBRR + PBTH) compared to usual care (UC)?

• What is the impact of a housing subsidy with heavy services on homeless families (PBTH) compared to a housing subsidy with light or no services (SUB + CBRR)?

• What is the impact of interventions that are more costly (PBTH + SUB) compared to a less costly intervention (CBRR)?

• What is the impact of a housing subsidy with no time limit (SUB) compared to a time-limited housing subsidy (PBTH + CBRR)?

The explanation of the estimation model begins with some terminology that describes how random assignment was implemented in this study. Enrollment and random assign-ment was a multistep process, as shown in Exhibit 2-7 (in Chapter 2). The PBTH, CBRR, and (in some sites) the SUB interventions had multiple service providers in each site.

Before random assignment, the number of slots currently available at all providers for each of the interventions was assessed. An intervention was deemed available if at least one slot at one provider of that intervention in the site was currently available. After an intervention was determined to be available, the interviewer asked the family a series of questions to assess provider-specific eligibility for the avail-able interventions and programs. A family was considered eligible for a particular intervention if the household head’s responses to the eligibility questions showed that the family met the eligibility requirements for at least one provider of that intervention that currently had an available slot. For example, some programs required that families have a source of income that would allow for them to pay rent on their own within a designated period of time. The study team thus asked families if they wanted to be considered for programs with such an income requirement. Other programs required families to pay a monthly program fee, and the screening question asked if families wanted to be considered for pro-grams with this type of requirement.

Other programs required participants to demonstrate sobri-ety, pass criminal background checks, or agree to participate in case management or other services. The study team asked

Short-Term Impacts of Housing and Services Interventions for Homeless Families 38 Chapter 4. Data Sources and Methodology

screening questions for these questions that ascertained families’ willingness to be considered for programs with these requirements.

To undergo random assignment, a family needed to be eligible for at least one available intervention in addition to UC.40 Based on this approach to random assignment, each family has a randomization set.

Randomization set. The set of interventions to which it was possible for a family to be assigned was determined by considering both the availability of the intervention and the assessed eligibility of the family. In the study, each family has one of seven possible randomization sets. These sets are {PBTH, SUB, CBRR, UC}, {PBTH, SUB, UC}, {PBTH, CBRR, UC}, {SUB, CBRR, UC}, {PBTH, UC}, {SUB, UC}, and {CBRR, UC}.

The randomization set of each family determines the pairwise comparisons in which the family is included. A family is in-cluded in the pairwise comparisons of its assigned interven-tion with the other interveninterven-tions in its randomizainterven-tion set.

For example, families assigned to PBTH with randomization set {PBTH, SUB, UC} are included in these two pairwise comparisons: PBTH versus UC; and SUB versus PBTH.

4.2.1 Impact Estimation Model for Family and Adult Outcomes

For each pairwise comparison, the study team estimated impacts for the sample of families who (1) had both inter-ventions in their randomization set and (2) were randomly assigned to one of the two interventions. The team used mul - tivariate regression to increase the precision of our impact estimates and to adjust for any chance imbalances between assignment groups on background characteristics (Orr, 1999).

Consider two interventions q and r (for example, PBTH ver-sus SUB), where the second option (r) is treated as the base case. Then, the impact on an outcome Y (for example, at least 1 night homeless or doubled up during past 6 months, working for pay in week before survey, or adult psycho-logical distress) of intervention q relative to intervention r is estimated through equation 1 for those families who had both options q and r as possible assignments, and were assigned to one of them. The estimation equation was—

(1) ,

where

Yi = outcome Y for family i,

Tq,i = indicator variable that equals 1 if family i was assigned to intervention q,

dq,r = average impact of being assigned to intervention q relative to being assigned to intervention r,

Xi = a vector of background characteristics41 of family i, Ik,i = indicator variable for “site-RA regime”42 k for family i, ei = residual for family i (assumed mean-zero and i.i.d.

[independently and identically distributed]), aq,r = a constant term, and

bq,r, fq,r,k = other regression coefficients.

The estimate of the impact parameter dq,r is the intention-to- treat, or ITT, estimate. For the pairwise comparisons, it is an estimate of the average effect of being offered intervention q rather than intervention r. The average effect is taken over all families in the q,r comparison, regardless of whether families actually participated in the intervention to which

The estimate of the impact parameter dq,r is the intention-to- treat, or ITT, estimate. For the pairwise comparisons, it is an estimate of the average effect of being offered intervention q rather than intervention r. The average effect is taken over all families in the q,r comparison, regardless of whether families actually participated in the intervention to which

문서에서 Family Options Study (페이지 65-0)