Three-Year Effects of the Youth Transition Demonstration Projects

Introduction

For youths with disabilities, the transition to adulthood can be especially difficult. Besides the host of issues facing all individuals at that age, young people with disabilities have additional challenges related to health, social isolation, service needs, the potential loss of program benefits, and lack of access to supports (Osgood, Foster, and Courtney 2010). These challenges complicate their planning, often leading to poor education and employment outcomes, dependence on public programs, and a possible lifetime of poverty (Davies, Rupp, and Wittenburg 2009).

The cost of providing disability benefits to young people is high. The Supplemental Security Income (SSI) and Disability Insurance (DI) programs, both administered by the Social Security Administration (SSA), are the primary federal programs that provide cash assistance to children and adults with disabilities. In December 2016, 1,095,000 individuals aged 13–25 received SSI payments with an annualized value of $8.4 billion (SSA 2017b, Table 35). In the same month, 178,000 individuals aged 25 or younger received DI benefits with an annualized value of $1.4 billion (SSA 2018, Tables 5.A1.2 and 5.A1.4).

Findings from earlier demonstration projects document the importance of customized supports and early interventions that meet the specific needs of youths with disabilities. Of particular note is the Transitional Employment Training Demonstration (TETD). Funded by SSA, TETD provided employment supports to SSI recipients in 13 communities; the recipients ranged in age from 18 to 40 and had intellectual disabilities. TETD operations began in June 1985; participants were enrolled through 1986 and services were provided through June 1987. Decker and Thornton (1995) found that TETD increased participants' cumulative earnings by 72 percent over the 6 years following their entry into the demonstration. Moreover, sites that delivered customized supports had better results than sites that provided uniform supports to all participants. Ivry and Doolittle (2003) found that mixed results from studies of other demonstration projects for youths with disabilities could be explained largely by the underenrollment of key subgroups of young people, inconsistent participation among enrollees, and high attrition rates. Their findings, as with those for TETD, underscore the importance of customizing employment supports to reflect the needs of specific youths rather than providing uniform supports.

Several more recent studies have pointed to additional factors that improve employment outcomes for youths with disabilities. Growing evidence indicates that work experience during the secondary-school years is a key predictor of postschool employment success (Luecking and Fabian 2000; Fabian 2007; Test and others 2009). Further, expectations and support from youths' families are linked to positive employment outcomes (Carter, Austin, and Trainor 2012; Simonsen and Neubert 2013), as is the provision of services designed to enhance youth self-determination (Wehmeyer, Field, and Thoma 2012). Summarizing findings from 22 studies, Test and others (2009) identified exposure to at least some general-education classes and participation in vocational education as effective strategies for improving postschool employment outcomes for youths with disabilities.

Youths receiving SSI face the same challenges that other youths with disabilities face, compounded by low income.¹ Recognizing the importance of helping young people with disabilities to achieve their full economic potential, SSA developed the Youth Transition Demonstration (YTD) (Fraker and Rangarajan 2009). Focusing on youths aged 14–25 who were either receiving SSI payments or DI benefits or were at high risk of receiving them in the future, SSA and its contractors developed, implemented, and evaluated strategies to promote self-sufficiency. YTD projects in multiple sites around the country offered services designed to assist young SSI recipients and DI beneficiaries facing the transition to adulthood.² SSA also provided waivers from certain restrictions on disability-program work incentives for YTD participants (Table 1). These waivers enhanced the existing incentives, for example by increasing the proportion of earnings that could be excluded from SSI countable income and delaying the loss of payments or benefits associated with negative disability redeterminations (SSA 2008).

The findings reported in this article address two research questions:

• Did the YTD projects provide participants with substantial levels of services, especially of services designed to promote employment?
• Did the YTD projects improve employment and other transition outcomes for participants in the third year after their enrollment in the evaluation,³ relative to what they would have experienced in the absence of the projects?

To answer these questions, this article summarizes a series of reports prepared for SSA by its YTD evaluation contractor, Mathematica Policy Research.⁴ The first question is addressed by analyzing project implementation and participant outcomes in the first year after enrollment in the evaluation. The second question is addressed through analysis of outcomes in the third year after enrollment.

The YTD Program Model

The YTD program model was based on existing research on effective approaches to promoting successful transitions to adulthood for youths with disabilities (Rangarajan and others 2009). In addition to the research cited earlier, two studies contributed promising insights; both centered on thorough reviews of existing research on the needs of youths in transition from secondary education to adulthood. The first was conducted by the National Alliance for Secondary Education and Transition (NASET) with input from more than 30 national advocacy groups, professional organizations, and education associations. The NASET study produced a set of standards, quality indicators, and research-based benchmarks for identifying critical needs for all youths, including those with disabilities (NASET 2005).

Building on the NASET framework, the National Collaborative on Workforce and Disability for Youth conducted its own review of research, demonstration projects, and recognized effective practices. From this review, it developed its Guideposts for Success (National Collaborative on Workforce and Disability for Youth 2005, 2009) to help practitioners and policymakers optimize service delivery for youths with disabilities. The guideposts involve providing school-based preparatory experiences, career-preparation and work-based experiences, youth development and leadership training, connections to programs and services, and encouragement of family involvement and support.

The YTD program model (Chart 1) included many of the components identified in Guideposts, although the YTD project customized the components to meet the particular needs of its target population (Luecking and Wittenburg 2009). First and foremost among those components was to provide participating youths with individualized work-based experiences and supports. The experiences included worksite tours; volunteer work; subsidized jobs; and, most notably, competitive paid employment in integrated settings, where people with disabilities work alongside nondisabled individuals. Another key component was to promote self-sufficiency by enabling participants to acquire the skills and knowledge needed to chart their own courses and advocate for themselves. This involved engaging youths in extensive planning that focused on education, employment, health care, and independent living. A third component—to encourage family involvement—included training, networking, and providing transition-related information to parents and other relatives. YTD also sought to link youths and their families with providers of social and health services and other potential supports. The final component of the model was counseling on SSA disability program benefits—and on the special waivers of restrictions on certain work incentives.

Another noteworthy feature of YTD was the provision of extensive programmatic technical assistance to project staff. TransCen, Inc., a nonprofit organization with expertise in designing and implementing employment programs for youths with disabilities, delivered the technical assistance via site visits, remote webinars and teleconferences, and annual meeting attendance. The technical assistance was primarily focused on helping front-line project staff conduct job development with employers and match participants with appropriate jobs. TransCen also assisted project managers in monitoring job development efforts and outcomes.

The YTD Project Sites

YTD projects were established and entered into the randomized controlled trial (RCT) evaluation in two distinct phases. SSA signed cooperative agreements with seven organizations in September 2003 to operate YTD projects in six states. Two years later, SSA selected a team of researchers and transition program specialists headed by Mathematica to assist the agency in refining the program model, provide technical assistance to the projects on model implementation, and conduct the RCT evaluation. Members of the Mathematica team visited each of the projects to observe the delivery of services and to interview staff. Based on an assessment of the quality of services and the willingness of staff to modify their interventions to include all components of the YTD program model, the Mathematica team recommended that two projects in New York (one in Bronx County and the other in Erie County) and one in Colorado participate in the first phase of the evaluation. SSA accepted the team's recommendations, and youths began to enroll in the evaluation in Colorado and Bronx County in August 2006 and in Erie County in February 2007. Services concluded in fall 2009 in Colorado and Erie County and in spring 2010 in Bronx County.

Three additional projects entered the evaluation in phase 2. Following the recommendations of the Mathematica team, SSA selected the projects from a group of five that were funded through the evaluation contract to deliver services on a pilot basis in 2007. The selection criteria included the number of youths recruited during the pilot year, the strength of services delivered, the degree of fidelity to the YTD program model, the quantity and quality of alternative services available in the project vicinity, and the size of the target population. The projects selected for full implementation in phase 2 were located in Miami-Dade County, Florida; Montgomery County, Maryland; and 19 counties in West Virginia. (Unlike New York's two distinct project sites, West Virginia was home to a single project with multiple field locations. In that respect, it was similar to the Colorado project, which operated in four counties.) Youths in each of the phase 2 sites began to enroll in the evaluation in March 2008, and SSA-funded YTD services ended in March 2012.

Table 2 lists the RCT project sites, arranged by phase and state, along with their lead organizations and target populations. Detailed descriptions of the six YTD projects that participated in the RCT evaluation appear in Martinez and others (2008).

The phase 1 projects entered the evaluation and began delivering services before TransCen was fully integrated into its role as the provider of programmatic technical assistance. Given that the phase 1 projects had independently developed their program models, they were only able to partially incorporate TransCen's technical assistance. By contrast, the phase 2 projects collaborated with TransCen in developing their program models and fully incorporated the technical assistance in delivering services. That assistance was thorough and consistent over the course of the evaluation. It focused on employment and provided guidance in assessing participants' strengths and challenges, engaging employers, placing youths in jobs, and delivering postemployment supports. Technical assistance also provided the projects with quantitative tools to use in conjunction with their case management information systems for monitoring participants' readiness for employment and their employment outcomes.

Enrolling Youths in the Evaluation

Five of the six sites (all except Maryland) restricted enrollment in the evaluation to youths who were SSI or DI beneficiaries; the Bronx County project further restricted enrollment to SSI recipients only. Interviewers at Mathematica contacted youths on the disability rolls via letter and telephone to describe the study and to enroll those who expressed interest. A young person enrolled by completing a baseline telephone survey and sending Mathematica a signed consent form affirming the enrollment decision. A youth without a legal guardian (generally, one aged 18 or older) could sign the consent form; otherwise, a legal guardian's signature was required. After a young person enrolled, Mathematica used a computer algorithm to randomly assign the youth to either the site's treatment group or its control group. Table 2 provides counts of evaluation enrollees and their treatment or control group status, by site.

In Maryland, eligibility was restricted to youths who were in—or had recently dropped out during—their last 2 years of high school and were considered by the county public school or mental health system to have a severe emotional disturbance or other significant mental illness. They were thus at high risk of receiving SSI payments as young adults. For youths who met these criteria, the Maryland YTD project staff conducted the initial outreach, primarily through presentations to students in high school transition classes and in transition-service information sessions held at the schools for parents and families. Interviewers at Mathematica then followed up with youths who had registered their interest and asked them to complete the baseline survey and provide written consent, after which the youths were randomly assigned to either the site's treatment group or its control group.

On average, 850 youths per site enrolled in the evaluation, for a total of 5,103 enrollees. By design, Mathematica randomly assigned slightly more than half (54 percent) of the enrollees to a treatment group. In the second stage of recruitment, project staff reached out to the treatment-group assignees and extended formal offers to participate in the YTD projects and receive the services that they were providing as well as the waivers from certain SSA work incentive restrictions. Table 2 provides counts of the treatment-group youths who signed the consent forms for this stage (or whose legal guardians signed for them) and were classified as YTD participants. Overall, 84 percent of treatment-group assignees became YTD participants.

Participants differed significantly from nonparticipants (that is, treatment group assignees who did not sign the second-stage consent forms) in several characteristics measured in the baseline survey (not shown). In at least half of the evaluation sites, participants had higher average family incomes, their mothers were more likely to have graduated from high school, they were more likely to have participated in job training, and a higher proportion of them expected to work for pay in the future. These are characteristics that one would expect to be positively associated with successful transition outcomes. However, following the standard approach in RCT evaluations for estimating effects on individuals targeted by the interventions, all treatment-group members, regardless of their participation status, were included in the YTD outcome analysis. This means that the participant-nonparticipant differences at baseline could not be a source of bias in the estimated intent-to-treat (ITT) effects of the YTD projects on postenrollment outcomes.⁵

Enrollee Characteristics

Table 3 presents the baseline characteristics of youths who enrolled in the evaluation at each of the six project sites. The characteristics of enrollees varied from site to site, with the Maryland and Bronx County projects exhibiting several important differences from the other four locations. Maryland was the only site in which the YTD project did not exclusively target youths who were SSI or DI beneficiaries. Only 21 percent of the enrollees there were receiving disability benefits when they entered the evaluation, compared with 100 percent in the other sites. The small share of beneficiaries among Maryland enrollees may help explain why comparatively high proportions of them had worked for pay in the previous year (57 percent, versus 37 percent or lower in the other sites) and reported being in excellent health (28 percent, versus 22 percent or lower in the other sites). Because the Maryland site targeted high school students and recent school leavers, the average age of its enrollees was younger (17.7) than that of enrollees in most of the other sites (around 19 or 20). The YTD project in Bronx County also targeted students; as a result, evaluation enrollees there were younger (average age 16.2) and more likely to be in school (94 percent) than their counterparts in the other sites. Bronx County enrollees also had somewhat lower socioeconomic status than enrollees in the other sites: only 18 percent were living with both parents when they enrolled in the evaluation (compared with 29 percent or higher in the other sites), and fewer than half of their mothers had graduated from high school (compared with about two-thirds or more in the other sites).

For some baseline characteristics, the similarities among evaluation enrollees across the sites are more notable than the differences. For example, at least 57 percent of enrolled youths were males, reflecting the preponderance of males among young disability beneficiaries nationwide; in December 2016, 67 percent of SSI recipients younger than 18 were male (SSA 2017b, Table 19). In all sites, the proportion of enrollees with physical primary disabling conditions was relatively low (24 percent in Colorado and 19 percent or lower in the other sites), as was the proportion with speech, hearing, or visual impairments (less than 13 percent in all sites). In addition, large shares of enrollees in all sites reported at baseline that, in the next 5 years, they expected to live independently (68 percent or higher), while even larger shares expected to work for pay (78 percent in West Virginia and 89 percent or higher in the other sites).

As noted earlier, youths who enrolled in the evaluation were first randomly assigned to either a treatment group or a control group. Those in the treatment group were given the opportunity to receive both the YTD waivers and specialized services, whereas those in the control group followed standard DI and SSI work-incentive restrictions and had access only to the non-YTD services already available in their communities. Because of random assignment, the two groups were expected to be statistically similar at the beginning of the study, so that any differences in postenrollment outcomes could be attributed to YTD. Consistent with this expectation, the treatment and control groups in each site did have statistically equivalent baseline characteristics. We conducted chi-square tests and t-tests for treatment-control differences in approximately 50 baseline characteristics per site. The number of statistically significant differences (those with p values less than 0.10) ranged from four to seven per site, as would be expected on the basis of random chance (Fraker and others 2014b).

All youths who enrolled in the evaluation were included in the analysis of YTD effects in the first and third years after their enrollment, contingent on the availability of follow-up data. The analysis thus included even the treatment-group assignees who did not ultimately participate in the YTD project services. We analyzed the projects' effects on all evaluation enrollees to preserve the integrity of the RCT design, thereby ensuring that any baseline differences between the treatment and control groups were attributable to chance.

Data Sources and Target Samples

The YTD evaluation included analyses of both the implementation of the individual projects and their effects on youth employment (and other outcomes) in the first and third years after enrollment. The implementation analysis and the outcome analysis differed in their data sources and target samples, as described below.

For quantitative data on the delivery of services to YTD participants, the implementation analysis relied on Efforts to Outcomes (ETO), a proprietary web-based management information system used at all of the sites. Project staff entered into ETO each service provided to YTD participants and the amount of time associated with its delivery. The ETO data pertained to treatment-group members only and, more precisely, to the 84 percent of treatment-group members who agreed to participate in the YTD projects. To supplement the ETO data, the Mathematica evaluation team collected data on project implementation during three visits to each site. The visits involved observations of project operations, interviews with project staff, and focus-group discussions with participating youths and their parents.

The analysis of outcomes in the first and third years after enrollment in the evaluation was based on data from SSA and Internal Revenue Service (IRS) administrative files and from two follow-up surveys of enrollees conducted by Mathematica. A distinguishing feature of these data is that they were available for control-group members as well as treatment-group members. The administrative data included monthly disability benefit amounts and annual earnings from employment.⁶ The survey data included information on service receipt, employment and earnings, educational progress, contact with the justice system, and other outcomes.

Implementation Analysis Measures

The ETO data and the Mathematica team's site visits yielded information on a broad range of measures of project implementation. This article presents findings from an analysis of ETO-based measures of services received by treatment-group youths during the first 12 to 15 months after they agreed to participate in the YTD projects. The measures pertain to (a) whether a youth received YTD services and (b) the number of hours of service received. Those two measures were applied first to services of any type and secondly to employment-promoting services in particular. The employment services included but were not limited to career exploration, soft-skills training, job-search assistance, development of work experiences, job placement, and postemployment follow-up including job coaching. Data for these measures were reported by project staff via ETO, rather than by participants themselves in the follow-up surveys.

Outcome Analysis Measures

Data from SSA and IRS administrative files were available for the evaluation enrollees who did not die between the date of enrollment and the end-dates for the two analysis periods. However, in the three phase 2 sites (Florida, Maryland, and West Virginia), IRS records were not available at the time of the analysis for youths who had entered the evaluation in the final year of enrollment (2010). Thus, for year-3 outcome measures that are based on IRS data, the sample sizes for the phase 2 sites represent between 58 percent and 83 percent of the surviving evaluation enrollees (Table 4).

Most but not all enrollees responded to the YTD follow-up surveys. Mathematica attempted follow-up interviews with all surviving enrollees, including control-group members as well as the members of the treatment groups who did not participate in project services; however, 13.3 percent of the enrollees did not respond to the 1-year follow-up survey (because they could not be located or declined to respond) and 17.7 percent did not respond to the 3-year follow-up survey. Consequently, the sample sizes for outcomes measured using survey data are smaller than the counts of the surviving evaluation enrollees.

The measures for the outcome analysis are discussed below. For each measure, we identify the data source as being either the SSA or IRS administrative files or the YTD follow-up surveys. Fraker and others (2014b) present sample sizes, mean values, and standard deviations for these measures by site and for treatment and control groups.

Receipt of Employment Services (Year 1)

Through individualized employment services, the YTD projects aimed to improve youth employability and employment outcomes. The measure of employment services for the outcome analysis is whether a youth received any of the following during the period from enrollment to the 1-year follow-up survey: career counseling, résumé preparation support, job-search assistance, job shadowing and apprenticeship, SSI and DI benefits and work incentives counseling, and other employment services. The measure differs in several respects from the measure of the receipt of employment services for the implementation analysis. First, it is based on youth reports in the 1-year follow-up survey of services received rather than on service data recorded by project staff in ETO. Second, the measure does not restrict the services to those provided by the YTD projects. Third, the measure was obtained for both treatment- and control-group members; hence, it can be included in the outcome analysis.

Hours of Services of Any Type (Year 1)

Treatment- and control-group members who responded to the 1-year follow-up survey identified the providers from whom they received various services in the year after they enrolled in the evaluation. For each provider, they reported the starting and ending dates of service, the frequency of service visits, and the typical length of a visit. From this information, we calculated the hours of services received from each provider and the total hours of services of any type from all providers.

Paid Employment (Years 1 and 3)

The YTD projects sought to help youths find paid employment in the short term and, by combining those experiences with other YTD services and the associated program waivers, to improve their paid-employment outcomes in the longer term. The evaluation's surveys provided measures of paid employment at any time between enrollment in the study and the 1-year follow-up and at any time during the year preceding the 3-year follow-up. IRS administrative files provided a measure of paid employment in the third calendar year following enrollment in the evaluation. The findings reported here treat paid employment in the year following enrollment as a service measure rather than as an outcome measure, as assistance in obtaining paid employment was a core component of the YTD program model. By contrast, paid employment in the third year following enrollment is treated as an outcome measure because YTD services typically lasted for only 12 to 18 months.

Earnings from Employment (Year 3)

The outcome analysis drew on two data sources for measures of annual earnings from employment. First, a survey-based measure captured earnings during the year preceding the 3-year follow-up survey reported by the enrolled youth. Youths reported jobs held, usual hours worked, and wage rates. Second, IRS files provided a measure of earnings during the third calendar year following enrollment in the evaluation. In principle, the survey-based measure should be more comprehensive than the IRS-based measure because it includes earnings from informal jobs for which employers did not report employee earnings to the IRS. On the other hand, the survey-based measure is subject to respondent recall error, whereas the IRS-based measure is not.

Disability Benefit Amount (Year 3)

Even though SSA expected that the YTD projects would reduce dependency on disability benefits in the long term, it had no expectation that the projects would achieve that objective during the YTD evaluation's 3-year follow-up period. The YTD waivers enabled youths receiving YTD services to (a) retain more of their benefits if they were working and (b) delay the effectuation of negative disability redeterminations. The waivers remained in effect for a YTD participant for 4 years after enrollment in the evaluation or until the youth reached age 22, whichever came later (SSA 2008). Because of the waivers, the YTD projects likely would increase the amount of benefits received by treatment-group youths during the evaluation's limited follow-up period. The outcome of interest is the total amount of SSI and DI benefits (as recorded in SSA program files) received in the third year following enrollment in the evaluation—in other words, the total amount of benefits received in months 25 through 36, where month 1 is the enrollment month.

Total Income (Year 3)

The YTD initiative was expected to improve youths' income by increasing their earnings from employment and providing them with waivers that allowed them to retain more of their benefits than would otherwise have been possible as their earnings increased. Thus, one of the important outcome measures to be analyzed was the total income received by youths from earnings and disability benefits in the third year following enrollment. This measure is the sum of yearly earnings as reported in the 3-year follow-up survey and total benefits received in the third year following enrollment as recorded in SSA program files.

Contact with the Justice System (Year 3)

None of the YTD project sites provided services specifically designed to reduce youth contact with the justice system. Nevertheless, by counseling participants (and, in some cases, their parents), engaging them in positive activities, assisting them with staying in school, and increasing their incomes, the projects might have reduced the likelihood of justice-system contact. In the outcome analysis, the measure of such contact was whether a youth reported an arrest or a charge of delinquency or criminal activity in the year preceding the 3-year follow-up survey.

Analytical Methods

When well-executed, random assignment ensures that comparing mean values of outcomes between treatment and control cases yields unbiased estimates of intervention effects. However, we used multivariate statistical models to improve the precision of our estimates. These models also allowed us to control for chance differences in baseline characteristics between treatment- and control-group members that could be correlated with outcome measures. We used ordinary least squares regression models to analyze continuous outcome measures and logistic regressions for binary outcomes. (Hereafter, we may use the term “regression models” to refer to models of both types.) The independent variables in the regression models were measures of age, race, sex, education, health, employment, expectations, family resources, and disability benefits from the evaluation's baseline survey or SSA files. The models also included an independent variable indicating whether a youth had been assigned to a control group or a treatment group. The regression coefficient on this independent variable is the estimated effect of the YTD project on the outcome measure. Fraker and others (2014b) provide detailed specifications of the regression models by site.

For all outcomes based on the evaluation's 1-year and 3-year follow-up surveys, we used weights in our regression models to account for survey nonresponse. To calculate the weights, we used logistic models to estimate the propensity of a surviving evaluation enrollee to respond to a survey as a function of his or her baseline characteristics. The surviving-enrollee response rates to the 1-year and 3-year follow-up surveys were 86.7 percent and 82.3 percent, respectively (Table 4). Treatment-group youths were slightly more likely than were control-group youths to respond to the surveys (not shown). The response rate differentials between the two groups for the 1-year and 3-year follow-up surveys were 2.2 percentage points and 3.3 percentage points, respectively.

All YTD participants (specifically, treatment-group youths who signed—or whose legal guardians signed—forms stating that they agreed to receive project services) were included in the implementation analysis of the receipt of services from the YTD projects. We used simple descriptive statistics to analyze the implementation analysis' four measures: receipt (yes or no) of YTD services of any type and of YTD employment-promoting services in particular; and hours of services in those two categories.

Analysis Results

In this section, we present results pertaining to both the receipt of services and the effects of the YTD projects on outcomes in the third year following enrollment.

Receipt of Services

Treatment-group youths in the YTD evaluation were more likely than their control-group counterparts to receive employment services; however, the extent of those services varied considerably across the project sites. We used two data sources and two analytical methods to investigate differences in the receipt of services. First, we used data from the evaluation's 1-year follow-up survey, in conjunction with the evaluation's RCT design, to assess whether the projects had positive effects on the receipt of employment services from any source (not just from the YTD projects) and on paid employment in the year following enrollment in the evaluation. Second, we used data entered by project staff into ETO to document the receipt of YTD services by the youths in the treatment groups who had agreed to participate in the projects. The latter data permitted a descriptive analysis not based on the RCT design. Among the participants who received YTD services, we analyzed the depth of those services, as measured in hours. Given that the data sources and methods for the two analyses differed, we had no reason to expect the results to be fully consistent.

Table 5 shows that all of the YTD projects had positive and statistically significant (p < 0.01) effects on youths' receipt of employment services from any source. Roughly two-thirds of treatment-group youths received employment services, with some variation among locations. The regression-adjusted difference in the receipt of employment services between treatment cases and control cases ranges from about 12 percentage points in Colorado and Florida to 30 percentage points in West Virginia. With the exception of Erie County, the YTD sites had no statistically significant effects on the total number of hours of services of any type. The pattern of results for those two measures indicates that five of the sites shifted the composition of all services received toward a concentration on employment services with no net increase in the total number of hours of services. Apparently, treatment-group youths substituted participation in the YTD projects, with their focus on employment services, for participation in more eclectic non-YTD services.

In the YTD program model, job placement or assistance in finding paid work is the most fundamental employment service. Among all treatment-group members, the rate of paid employment in the year following enrollment—as measured by the evaluation's 1-year follow-up survey—ranged from 23 percent in Florida to 53 percent in Maryland. The YTD projects in Bronx County and Florida had positive effects of about 9 percentage points on paid employment and the West Virginia project's effect was 19 percentage points; all three were statistically significant (p < 0.01). The other YTD projects had no statistically significant effects on paid employment in the year following enrollment.⁷

Almost all YTD participants received some YTD services, according to data entered into ETO by project staff; however, the extent of the services varied greatly across the project sites. Table 6 shows that at least 96 percent of the participants in each site received some type of YTD service. Viewed from the opposite perspective, less than 4 percent of participants were “no shows”—those who had formally agreed to participate but never made themselves available to receive YTD services. At one extreme, participants in the Colorado project received an average of only 7 hours of YTD services of any type, whereas participants in the Bronx County project received an average of 43 hours of services. The extent of YTD services of any type was generally higher for participants in the phase 2 projects, averaging about 30 hours. The receipt of employment-specific YTD services was less consistent across the projects and was higher in the phase 2 sites. Only 54 percent of participants in the Colorado project received YTD employment services, compared with 85 percent of participants in Erie County and more than 90 percent of participants in the other four projects. Among participants who did receive YTD employment services, the extent of those services varied greatly across the projects, with average amounts ranging from 4 hours in Colorado and 6 hours in Erie County to 21 hours in Bronx County and 24 hours in West Virginia.

In summary, all of the YTD projects increased the likelihood that youths who enrolled in the evaluation received employment services from any source, but only the projects in Bronx County, Florida, and West Virginia increased the likelihood that enrollees had paid work experiences within a year of enrollment. Participants in those three projects, along with participants at the Maryland site, had high rates of receipt of YTD employment services, and the number of hours of those services was high relative to the hours of employment services received by participants in the other two projects.

Outcomes in the Third Year after Enrollment

The phase 2 YTD projects generally had statistically significant effects on more outcome measures in the third year after enrollment than did the phase 1 projects. This finding is broadly consistent with the finding, noted above, that two of the three phase 2 projects had positive effects on paid employment in the year after enrollment, compared with just one of the three phase 1 projects. In addition, the phase 2 projects generally delivered more employment services and more services of any type.

Table 7 shows year-3 outcomes for the phase 1 projects. Only the Erie County project had a positive and statistically significant effect on paid employment. It had a positive effect on the employment rate of 8 percentage points (p < 0.05), as measured by the evaluation's 3-year follow-up survey. Our analysis of the survey data also found that the Erie County project increased mean earnings by $521; however, that estimated effect is just short of being statistically significant at the 0.10 level. Table 7 provides no evidence that the Bronx County and Colorado projects had any effects on employment and earnings in the third year.

Both of the New York YTD projects had positive and statistically significant effects on the amount of disability benefits received by evaluation enrollees in the third year and, consequently, on their incomes. The average total income (earnings plus benefits) received in the third year by treatment-group members relative to control-group members was higher by $1,729 in Bronx County and by $1,106 in Erie County (in both cases, p < 0.01). In Bronx County, the effect on total income was almost entirely due to the YTD project's positive and statistically significant effect on the disability benefit amount (p < 0.01). In Erie County, the effect on total income was the joint product of a statistically significant positive effect on the disability benefit amount (p < 0.05) and the previously noted positive but statistically insignificant effect on the survey-based measure of earnings.

The Bronx County project reduced youths' contact with the justice system in the third year after enrollment. Treatment-group members in that site were 3.8 percentage points less likely than were control-group members to have been arrested or charged with delinquency or criminal activity (p < 0.05). Like the other YTD projects, the Bronx County site did not provide services explicitly designed to reduce criminal activity; however, it is possible that the general counseling provided to youths and parents, combined with the project's positive effect on youth income, contributed to the favorable result. By contrast, treatment-group members in Colorado were 2.8 percentage points more likely to have been arrested or charged than their control-group counterparts were (p < 0.10). It is unclear which of the features of the Colorado project accounted for this unexpected result.

Table 8 shows that the phase 2 projects had statistically significant effects on a greater number of year-3 outcome measures than did the phase 1 projects, particularly in the case of the Florida site. That project had a positive and statistically significant effect of about 7 percentage points on paid employment when measured either with the evaluation's year-3 follow-up survey (p < 0.05) or with IRS records (p < 0.10). It also had a statistically significant positive effect of $615 on the survey-based measure of earnings in the third year (p < 0.05). That result, combined with a statistically significant positive effect on the disability benefit amount (p < 0.01), resulted in a statistically significant positive effect of $1,246 on total income (p < 0.01) in the third year after enrollment. The positive effect on income may have contributed to the project's statistically significant negative effect of 2.7 percentage points on the proportion of evaluation enrollees arrested or charged with delinquency or criminal activity in the third year (p < 0.05). Neither the Florida site nor any of the other YTD projects provided services explicitly designed to reduce criminal activity.

The Maryland YTD project had no effect on paid employment in the third year after enrollment; however, it did have a statistically significant positive effect of $1,162 on the survey-based measure of earnings (p < 0.10). The effect on earnings was the dominant factor behind that project's positive and statistically significant effect of $1,382 on total income (p < 0.05), as was expected because only one-fifth of the youths at this site were receiving disability benefits when they enrolled in the evaluation.

The YTD project in West Virginia had a positive and statistically significant effect of 8 percentage points on paid employment in the third year after enrollment, based on IRS data (p < 0.10). The estimated effect on paid employment based on data from the 3-year follow-up survey just missed the 0.10 threshold for statistical significance. The West Virginia project increased total income in the third year by a statistically significant $1,010 (p < 0.01), primarily because of its statistically significant positive effect of $748 on the disability benefit amount (p < 0.01).

Discussion

In the first year after enrollment in the evaluation, the proportion of youths with disabilities who received employment-promoting services was greater for treatment-group members than for control-group members in all six project sites. However, only three projects—those in Erie County, Florida, and West Virginia—had positive and statistically significant effects on paid employment in the third postenrollment year. Findings on the receipt of services provide insight into the positive year-3 employment results for the Florida and West Virginia projects. In those two sites, YTD employment-service design and delivery led to higher proportions of treatment-group youths, relative to their control-group counterparts, having paid employment in the year after enrollment. In addition, the implementation analysis found that these two projects delivered employment services and services of all types to virtually all of their participants and that the services provided were extensive. The Florida project delivered an average of 14 hours of employment services and 29 hours of services of any type to each participating youth. This project was characterized by comprehensive technical assistance for front-line staff on the delivery of employment services and by systematic quantitative monitoring of staff service efforts (Fraker and others 2018). The West Virginia project delivered an average of 34 hours of services of any type and 24 hours of employment services to each participating youth. That project placed special emphasis on delivering customized employment supports to youths in settings that were readily accessible, such as at the youth's workplace, school, or home (Cobb, Wittenburg, and Stepanczuk 2018).

The positive effect of the Erie County YTD project on the proportion of youths with paid employment in the third year after enrollment is surprising because that project provided participants with few hours of services and had no significant effect on employment in the first year after enrollment. Given the low intensity of services, we speculate that SSA's waivers for YTD may have contributed to the year-3 employment result.

The YTD project in Bronx County had no effect on employment in the third postenrollment year despite delivering a high average number of hours of services to participating youths and its positive effect on employment in the first postenrollment year. The Bronx County project was unique in two notable respects that help explain these seemingly contradictory findings. First, this project delivered almost all of its services in workshops and other group activities rather than on an individual basis, as did the other five YTD projects. It is likely that an hour of services received in a group setting represents less intensive exposure than an hour of individualized services. Second, this project placed its participants in 7-week part-time jobs through an arrangement with New York City's Summer Youth Employment Program (SYEP). YTD participants placed in those jobs were paid by the project or SYEP, rather than by their nominal employers, which typically were units of the City University of New York, where the project was housed (Fraker and others 2011). Those work experiences may have been less effective at promoting subsequent employment than the more conventional jobs that the other YTD projects helped their participants to find.

The Maryland YTD project provided a substantial depth of services but had no effect on paid employment in either the first or third year after enrollment, although it did increase youths' earnings in the third year. The lack of positive employment results for this project may be explained by two factors. First, Maryland's was the only YTD site where the target population did not consist exclusively of SSI or DI beneficiaries. In fact, 79 percent of the Maryland evaluation enrollees were not beneficiaries and therefore may not have faced consistently significant barriers to employment. Second, the services available to control-group youths in that site were relatively strong. These two factors imply that many of the Maryland youths who enrolled in the evaluation may not have needed help in finding jobs; but those who did need assistance, even those in the control group, had access to relatively strong services. Consequently, youths in both groups achieved high rates of employment—in fact, the highest rates across all of the evaluation sites.

The YTD projects in Bronx County and Florida provided many hours of services to participants and achieved statistically significant negative (desirable) effects on youth arrests and charges of delinquency or criminal activity in the third year after enrollment in the evaluation. By contrast, the Colorado project provided few hours of services and had a significant positive (undesirable) effect on encounters with the justice system. We do not know what components of the projects generated these results, but we conclude that well-designed and well-implemented interventions providing substantial hours of services may be able to reduce contact with the justice system among youths with disabilities.

Limitations and Implications for Research

The extent of the interventions, as measured by service hours, was uneven across the YTD projects. Consequently, fidelity to the YTD program model varied, especially between the phase 1 and phase 2 projects. It is therefore difficult to draw broad inferences from the findings across the sites. Further, relative to the phase 1 projects, the phase 2 projects received deeper technical assistance in delivering employment-related services. It is impossible to know whether the phase 1 projects would have generated more positive results if they had received and embraced deeper technical assistance designed to improve employment services. The fidelity of interventions to program models would be a useful area to examine in future research.

In a few instances, the evaluation failed to detect effects that were large enough to be policy-relevant. For example, in the West Virginia site, an estimated effect of 5.7 percentage points on the survey-based measure of paid employment in the third year (p = 0.11) just missed the 0.10 threshold for statistical significance. The evaluation was designed to have 80 percent power to detect effects of 7 percentage points on employment if based on data for all of a site's enrollees and of 8 percentage points if based on the survey respondents only (Rangarajan and others 2009). For effects smaller than the threshold for detection, the evaluation had an elevated risk of generating estimates that were not statistically significant.

SSA plans further analyses of the YTD evaluation enrollees to determine whether the projects' effects on employment, earnings, and program participation persist or change over time. This research will be based on SSA and IRS administrative data only, as the agency has no plans to conduct additional follow-up surveys of the enrollees. The research will include reestimating the year-3 effects on employment and earnings based on IRS data for 100 percent of the enrollees in the phase 2 sites. Recall that for the present analysis, the year-3 IRS data were unavailable for between 17 percent and 42 percent of the enrollees in those sites. Although there is no reason to expect that the point estimates based on the full data would differ from those presented in Table 8, they would likely have smaller standard errors because of the larger sample sizes. More importantly, the planned future analyses will produce estimates of project effects in periods more than 3 years after enrollment.

The evaluation design did not enable us to disentangle the effects of SSA's YTD waivers from the effects of project services. However, we can make two observations about the waivers as implemented in the YTD evaluation. First, Mathematica survey staff and YTD project staff reported that the waivers were a strong inducement for youths to complete the baseline survey and enroll in the evaluation and, if assigned to the treatment group, to formally agree to participate in project services. Second, the presence of the waivers throughout the evaluation's 3-year follow-up period meant that any positive effects of the projects on youth earnings were unlikely to be manifested as negative effects (reductions) in disability benefits. This is because several of the waivers were designed to moderate the loss of benefits associated with increases in earnings. SSA's planned follow-up analyses (discussed above) will extend the period of study to years after the waivers expired and so should provide a clearer picture of the intervention's potential reduction of disability benefits. SSA might also consider conducting a demonstration of the effects of the YTD waivers in isolation, without any additional services except perhaps enhanced benefits counseling. Such a demonstration would be relatively simple and inexpensive to implement and evaluate.

The youths who enrolled in the YTD evaluation were volunteers who were not representative of all YTD-eligible youths in the project locations. More specifically, in the five sites where recipients of disability benefits constituted the YTD target population, those who enrolled in the evaluation were not representative of all youths receiving disability benefits. Hence, it would be inadvisable to infer from these findings the effects of a hypothetical YTD-like intervention that would be mandatory for all youths receiving disability benefits. However, interventions for youths receiving disability benefits are more likely to be voluntary than mandatory. For example, the Department of Education's current PROMISE initiative is funding voluntary school-to-work transition programs for SSI recipients aged 14–16 in 11 states (Department of Education 2013b; Fraker and others 2014a). The YTD findings may be instructive regarding the likely effects of such voluntary interventions.

Implications for Policy and Practice

The findings presented in this article show that the delivery of substantial amounts of well-designed services to youths with disabilities, in conjunction with rule waivers that enable workers to retain more of their disability benefits, can improve employment and other key transition outcomes in the short- to medium-term. However, the estimated effects of the YTD projects, even those that are statistically significant, are not large in absolute size. For example, the statistically significant estimated effects on the paid employment rate in the third year range from about 6 percentage points to about 8 percentage points in the Erie County, Florida, and West Virginia study sites. Even if we adjust those estimates to reflect the fact that they are based on all treatment-group members rather than just those who participated in the YTD projects, the estimated effects on paid employment remain modest, ranging from about 7 percentage points to 9.5 percentage points. Hypothetical future YTD-like interventions would therefore be unlikely to dramatically reduce the SSI rolls. Nevertheless, a persistent employment effect of this magnitude would suggest that YTD-like interventions could modestly reduce SSI participation and payment amounts for some recipients, in addition to improving recipient well-being by increasing their labor-force engagement and increasing their total incomes.

These findings underscore the need for entities serving youths with disabilities to increase and redirect their efforts to focus on employment services and employment outcomes. Doing so may not only immediately improve employment outcomes (as evidenced by the year-1 findings for three of the project sites), they may also have a sustained effect (as evidenced by the year-3 findings for the Florida site and, to a lesser extent, the West Virginia and Erie County sites). These findings also indicate that such results may not require a net increase in services for youths, but rather a sharpened focus of services on employment. Fostering that focus may require technical assistance by professionals whose training and experience include a strong emphasis on engaging with employers and facilitating employment for youths with disabilities.

Research not only supports the value of employment-focused interventions for youths with disabilities, it also has shown that employment outcomes for young SSI recipients are markedly poor (Wittenburg and Loprest 2007) and that the longer individuals with disabilities remain out of the labor market, the more their likelihood of ever working is significantly diminished (Kraus and others 2001; Young 2010). This in turn implies that dependence on public income support will be lifelong for a substantial fraction of young SSI recipients (Davies, Rupp, and Wittenburg 2009; Rupp, Hemmeter, and Davies 2015). Simply put, youths need to be exposed to work opportunities to have a reasonable expectation of being employed as adults. In fact, such exposure is mandated in the PROMISE initiative (Department of Education 2013a).

In the three YTD study sites where the projects achieved statistically significant employment results in the third year (Erie County, Florida, and West Virginia), less than half of the treatment-group youths were employed in that year (Tables 7 and 8). Those proportions are substantially lower than the roughly 90 percent of evaluation enrollees who expected at baseline to be employed in the next 5 years (Table 3). The large gap between employment expectations and outcomes suggests that employment results could be greater than those achieved by even the most successful YTD projects. As we acknowledge the substantial efforts of those projects, future interventions should test additional ways to serve youths with disabilities and help them to more fully realize their own expectations for employment.

The Workforce Innovation and Opportunity Act of 2014, or WIOA (Public Law 113-128), provides an opportunity for many youths to receive more extensive employment and transition supports at younger ages than were previously available. Lessons learned from the YTD evaluation may be applicable to states as they implement WIOA. Most of the YTD projects struggled to develop and maintain employment-focused services. For several of the projects, technical assistance provided by TransCen facilitated the delivery and tracking of effective employment services. It will be interesting to see if WIOA services must likewise be supported by technical assistance to achieve equivalent or better results.