Effects of the Ticket to Work Program: Return on Investment and Overall Assessment of Outcomes Versus Design

Introduction

The Social Security Administration (SSA) has operated the Ticket to Work (TTW) program for 2 decades. TTW aims to enable SSA disability program beneficiaries to exit the program rolls and attain self-sufficiency through work by facilitating their access to and participation in employment training and support services. Many policymakers had high hopes for TTW when it was established and, for some, the program's results have been disappointing. This analysis examines the program's intentions, as described in the Ticket to Work and Work Incentives Improvement Act of 1999, and its outcomes through 2016. It also compares the program's costs with the benefit payments forgone for the beneficiaries the program served as intended.

TTW was based on the premise that many Supplemental Security Income (SSI) and Social Security Disability Insurance (DI) beneficiaries had the functional capacity and the desire to work but had too few options for employment services. Before TTW, all such services were provided by entities called state vocational rehabilitation agencies (SVRAs). It was thought that because SVRAs faced no competition for SSA disability program clients and received payments from SSA under a cost-reimbursement model, the existing vocational rehabilitation (VR) system did not give SVRAs incentives to provide the innovative solutions that could enable more of their clients to discontinue cash benefits through long-term employment. A Disability Policy Panel convened by the National Academy of Social Insurance in 1993 envisioned an expansion of employment service availability and choice by allowing public and private service providers, which would come to be known as employment networks (ENs), to compete with SVRAs for clientele. The panel also conceived a performance-based model for SSA payments to the service providers (Mashaw and Reno 1996). The existing payment system, entailing reimbursement for costs associated with providing employment services, would remain available; but providers would now have the option of receiving payments based on their clients' performance-based milestones and outcomes. A key feature of the new system would be the direct linkage of provider payments to beneficiary progress toward employment success and the resulting reductions in benefit payments. In this way, the program attempted to align the incentives of the employment service providers with those of SSA and the beneficiaries attempting to exit the disability rolls and achieve financial independence via work (Stapleton and Livermore 2003; Huynh and O'Leary 2003).

The Ticket to Work and Work Incentives Improvement Act, Public Law 106-170, was signed into law on December 17, 1999. A centerpiece of the act was the Ticket to Work and Self-Sufficiency Program, which was meant to comprehensively address longstanding issues that were seen as limiting employment for beneficiaries with disabilities. TTW created the new class of service providers, ENs, that the Disability Policy Panel had envisioned, and established the rules that aligned the SSA provider payments they received with the long-term success of beneficiaries. The legislation also protected beneficiaries from continuing disability reviews (CDRs, which affirm or deny ongoing eligibility for benefits) while they participated in TTW,¹ extended access to Medicare or Medicaid coverage for beneficiaries returning to work, and allowed for expedited benefit reinstatement if the work attempt ultimately failed. As the Ticket to Work and Work Incentives Advisory Panel noted in their final report (2007), the act “represented the culmination of efforts by diverse stakeholders who recognized the importance of removing barriers to employment and labor market activity for individuals with disabilities.” The panel further noted that the legislation “demonstrated the evolution of thinking and attitudes about the interest in and capacity of people with disabilities to work, contribute to our nation's economy, and reduce their reliance on public benefits.” Radtke (2000) described the act as “one of the most significant changes in disability policy in the last 20 years.”

Despite early optimism, as the program unfolded and the results of SSA's evaluations were released, many came to regard TTW as a failure (Butler 2018; Government Accountability Office [GAO] 2011; Khazan 2015; Lankford 2017; Lawler 2014; SSA 2008). Eimicke, Cohen, and Miller (2017) noted that “by December 2005…a mere 1.8% of disability beneficiaries were participating in the TTW program in the 13 Phase 1 states” and that “as of early 2007, ENs as well as SVRAs were losing interest in the program.” Autor and Duggan (2010) described the TTW program as “ineffective” and noted that “fewer than 1,400 tickets (0.01 percent) of 12.2 million tickets issued in the first seven years of the…program led to successful workforce integration.”

It seems clear that the TTW program has not achieved the lofty changes many had anticipated for it. It is less clear, however, whether one should regard the program as a failure. Improving the rate at which beneficiaries leave the disability program rolls and become self-sufficient was an obvious goal of the program, but it was not the only goal, and one should consider all of TTW's objectives in assessing its level of success. That is a complex undertaking, as measures of program success are open to interpretation. SSA hired an independent contractor to evaluate TTW, which produced extensive findings on TTW activities, outcomes, and effects. In seven reports,² the evaluation delves into topics ranging from SSA's implementation of TTW, participation by ENs and beneficiaries, the effect of regulatory changes implemented in 2008, crossover effects between SVRAs and ENs, program costs, and outcomes involving work attempts, earnings, and disability benefit receipt. Although the reports describe successes, failures, and adjustments, SSA did not design the evaluation to reach a single conclusion on TTW. Instead, the agency intended the evaluation to provide detailed findings on the various program aspects to inform decision makers and the public. As such, readers can reach a variety of conclusions from the reports.

To provide an overall TTW program assessment, we examine two basic research questions: Does the program achieve the stated objectives of its authorizing legislation, and has it produced benefits that exceed its costs? Although these questions are conceptually simple, overall costs and benefits can be difficult to measure.

The analysis is structured as follows. We start with an overview of expectations for the TTW program as outlined in the legislation that created it. We then review the findings from the TTW evaluation in the context of the legislative goals. We then extend those findings in two ways. First, we analyze service delivery, program costs, and employment and benefit outcomes without regard to what would have happened in the absence of TTW. Second, we assess employment outcomes using multivariate regressions, focusing on DI beneficiaries who have completed a trial work period (TWP)—during which a beneficiary can accrue earnings without risking benefit termination, under program rules—as a means of addressing selection bias among those who choose to use TTW services.

Overview of TTW Expectations and Perceived Results

The text of Public Law 106-170 stated that the primary purpose of TTW was

to establish a return to work ticket program that will allow individuals with disabilities to seek the services necessary to obtain and retain employment and reduce their dependency on cash benefit programs.

Subsequent passages, designated as “findings” 1 and 10–12 of the law's text, suggested additional goals of the program:

(1) It is the policy of the United States to provide assistance to individuals with disabilities to lead productive work lives.

(10) Eliminating…barriers to work by creating financial incentives to work and by providing individuals with disabilities real choice in obtaining the services and technology they need to find, enter, and maintain employment can greatly improve their short and long-term financial independence and personal well-being.

(11) In addition to the enormous advantages [this law's] changes promise for individuals with disabilities, redesigning government programs to help individuals with disabilities return to work may result in significant savings and extend the life of the Social Security Disability Insurance Trust Fund.

(12) If only an additional one-half of one percent of the current Social Security Disability Insurance and Supplemental Security Income recipients were to cease receiving benefits as a result of employment, the savings to the Social Security Trust Funds and to the Treasury in cash assistance would total $3,500,000,000 over the worklife of such individuals, far exceeding the cost of providing incentives and services needed to assist them in entering work and achieving financial independence to the best of their abilities.

Thus, although the primary goal to TTW was to reduce benefit payments by significantly increasing the rate at which benefits can be suspended or terminated because of work (STW), the act clearly included at least three other, though arguably secondary, objectives:

1. Expand the services available to and used by beneficiaries that might allow them to obtain and retain employment.
2. Improve the lives of beneficiaries by enabling greater self-sufficiency, with increased earnings allowing the reduction or elimination of dependence on cash benefits.
3. Bring about benefit reductions in amounts that exceed the cost SSA pays to providers for employment services.

Since the completion of the TTW evaluation in 2013, stakeholders have tended to focus on the finding that the program has not made significant progress toward the primary goal of reducing the disability rolls. Less attention has been paid to the substantial evidence, presented below, that TTW has (1) expanded the range of services available to beneficiaries, (2) had a positive effect on participants, and (3) done so with costs that are no higher than those for the SVRA-only services it replaced, and are potentially lower than the benefit reductions that have ensued.

TTW Effects on Benefit Receipt

As noted above, SSA funded a comprehensive TTW evaluation, as required in the authorizing legislation. SSA contracted with the Lewin Group in 2000 to design the TTW evaluation, and with Mathematica (then known as Mathematica Policy Research) in 2002 to implement the evaluation. SSA also contracted with Mathematica in 2003 to conduct a survey to collect information needed for the evaluation that was not available in agency administrative data. Mathematica conducted the survey in four rounds from 2004 through 2010, collecting information from more than 20,000 beneficiary respondents.

The evaluation spanned 11 years and produced seven reports, each comprising from one to nine separately published studies, and several policy briefs. The fourth evaluation report (Stapleton and others 2008) and a study from the seventh report (Stapleton, Mamun, and Page 2013) specifically addressed TTW's net effect on the size of SSA's disability beneficiary population. If that outcome is measured by the rate at which TTW enabled DI disabled-worker beneficiaries to resume work at an earnings level that constitutes substantial gainful activity (SGA)—and thereby allows them to forgo cash benefits—then Table 1 appears to show that TTW had little or no effect.³ Although the benefit termination rates for SGA were marginally lower than 0.55 percent in years after 2001, the last full year before TTW Phase 1 implementation, the rates varied little. However, this simple measure does not account for differences over time in the beneficiary population or other environmental factors. The results in Table 1 are generally regarded as “gross outcomes,” as they do not attempt to account for related factors that could accentuate or offset any TTW effects. Results of analyses that attempt to remove or “net out” these related factors are generally regarded as “net effects.” To account for these other factors, more sophisticated statistical methods are needed. The evaluation contractor therefore attempted to measure the net increase in the rate at which beneficiaries reduced their dependence on disability benefits that was specifically attributable to the TTW program. Stapleton and others (2008) reported the results of the first of these analyses, and Stapleton, Mamun, and Page (2013) revised and updated those results using improved analytical techniques and additional data. The findings of these two analyses were consistent, and because the 2013 analysis was more robust, we focus on that latter report here.

SSA implemented TTW in three phases, each encompassing a different group of states, from the beginning of 2002 through the end of 2004. The 9/11 attacks disrupted the Phase 1 rollout in New York, leaving only Phases 2 and 3 as viable for analysis. In each phase, SSA divided randomly selected beneficiaries into 10 “mail-month” groups. SSA mailed tickets, which program participants could exchange for employment services with an SVRA or an EN, to the first group in November of that phase's year, sent no tickets in December, and sent tickets to one of the remaining groups each month from January through September of the next year. Thus, a random subset of beneficiaries in each phase got their tickets as much as 11 months sooner than those in the latter mail-month groups. Stapleton, Mamun, and Page (2013) used this randomness to estimate the difference in TTW effects between the early and late mailing groups.⁴ However, because all mail-month groups in each phase received tickets within an 11-month span, the period available for measuring TTW effects was limited, which in turn limited the precision of this evaluation method.

Both the 2008 and 2013 analyses concluded that TTW had generated a statistically significant net increase in employment service enrollment with both ENs and SVRAs, but it did not increase, at statistically significant levels, the proportion of beneficiaries with disabilities who were able to transition from cash benefits to STW status—that is, to self-sufficiency. In particular, the evaluators found evidence that receiving a ticket 11 months sooner than other beneficiaries in Phase 2 increased the attainment of STW status by 7 percent, and the number of months in STW status by 5 percent, as of 48 months after the phase began. However, for Phase 3, the evaluators found no significant effect on STW status, and the effect on the number of STW-status months was similar to that for Phase 2 but in the opposite direction. They also found that TTW's effects on the completion of TWP months were insignificant for Phase 2, which cast further doubt on the positive Phase 2 STW-status findings because improved STW-status outcomes should be associated with improved TWP outcomes, as the latter is a precursor to the former. The evaluators concluded that the effects they measured were effectively random and therefore not statistically different from zero.

Although the evaluators' finding was disappointing, two points are worth considering. First, Congress designed the TTW program to supplement the cost reimbursement system that SSA had used for SVRAs since 1981. Under that system, SVRAs were reimbursed if a client attained an earning objective, with or without reaching STW status. TTW added ENs as eligible service providers and allowed ENs and SVRAs to choose whether to be reimbursed under the traditional system or under one of two alternatives: either a “milestone plus outcome” or an “outcome only” system, which tied payments to the client's STW status in the current month. Following standard practice, the evaluator analyzed the TTW's effects relative to the prior system. In other words, the net effect of TTW was the difference between the combination of EN and SVRA services and SVRA services alone, the services which had been available prior to TTW. The evaluation did not, and could not, measure the effect of TTW by comparing outcomes for beneficiaries receiving services from an SVRA or an EN to outcomes for beneficiaries who had no access to services.

The TTW evaluation did not attempt to measure the effect of employment services on the beneficiaries who used them, but other studies have, albeit with several limitations. For example, Dean, Dolan, and Schmidt (1999) and Dean and others (2001, 2014) used the earnings of SVRA clients to measure success, and compared annual earnings to service costs to assess their performance. Those studies found that beneficiaries generally increase employment and earnings after receiving SVRA services and that their earnings exceed the cost of services.⁵ Earnings relative to service costs, however, tell us little about effectiveness relative to disability benefit receipt. One analysis of earnings relative to SGA found that such earnings are not sufficient to discontinue disability benefits (GAO 2007). However, because SVRA-provided supports represent an ongoing national program, it is not possible to use an experimental design to study their effects. Researchers are thus left with only quasiexperimental methods that compare participants with characteristically similar nonparticipants such as VR service dropouts. Whether the comparison groups have been similar enough to participants to address the inherent selection bias implicit in completing VR services is unclear and this limits the strength of the findings in these studies. We will discuss selection bias in more detail later.

Second, the evaluation notably did not estimate TTW effects on participants only. By necessity, Mathematica evaluated TTW's effects on all beneficiaries (this is known as the “intent-to-treat” approach in the economic literature). Because TTW serves a relatively small proportion of DI beneficiaries and SSI recipients with disabilities, and many such beneficiaries and recipients forgo benefits for work without using employment services, TTW's effects on participants would need to be substantial to have a measurable effect on the larger population of beneficiaries and recipients. Mathematica did not find a large effect, but a small positive effect would have been undetectable because of the limitations of the methods available.

Both evaluation-design elements—the assessment of TTW effects relative to the prior system and the use of intent-to-treat methodology—made sense, given the way Congress required SSA to roll TTW out and the program's main goal of substantially increasing exits from the disability benefit rolls. However, these design parameters limited the methods available to the evaluator, and those methods limit the implications of the results. We know that TTW did not lead to a large increase in disability benefit exits, but we do not know if smaller effects were achieved. In fact, Stapleton, Mamun, and Page (2013) found that “the statistical power of the projections for STW [status] and the number of STW [status] months is insufficient to rule out the possibility that TTW had impacts of at least five percent on each outcome for Phases 2 and 3 pooled.” More precisely, their analysis found that TTW increased STW status incidence by less than 30 percent and the number of months in STW status by less than 9 percent, but they could not rule out smaller effects that approached those boundaries. This is not to say that the evaluation found evidence that positive outcomes occurred. Instead, we merely note that, given the inherent limitations of the evaluation methodology, effects of less than these magnitudes could have occurred without statistically significant detection. The methodological restrictions simply prevented a more precise result.

The evaluation also could not assess TTW's effect on other outcomes of interest. In particular, it did not address whether EN and SVRA services significantly increased employment among participants. Because employment by itself does not directly affect benefit eligibility, SSA does not track employment data. SSA tracks earnings, but only if they are high enough to affect benefits. Further, pre- and post-TTW earnings data are inconsistent because the quality of the earnings tracking changed after TTW implementation, as the program required SSA to improve its earnings data to calculate EN payments accurately. As such, the earnings and employment data for disability program beneficiaries that would allow a comprehensive TTW assessment are in some instances unavailable, or are available with gaps, especially for the pre-TTW comparison period. These limitations were not flaws of the evaluation design. Rather they reflect what was possible given the information available to the evaluator, the characteristics of the TTW program, and the way Congress required SSA to implement it.

Even the finding that TTW did not greatly affect the disability roll exit rate is more complicated than it first appears. The evaluation found that the success rate that resulted from combining EN and SVRA services was statistically no different from offering only SVRA services. To the extent that SVRA services had been effective prior to TTW, they continued to have a similar level of success after TTW, and the success rates for ENs were similar to the SVRA results. In fact, the success rate was higher for ENs, meaning that the new ENs were at least as effective as SVRAs. This does not necessarily mean, however, that ENs are better at serving beneficiaries than SVRAs. Because ENs can choose their clients while SVRAs cannot, adding ENs to the service mix under TTW may have simply shifted some of the easier-to-serve cases from SVRAs to ENs.⁶ Regardless, combined EN and SVRA services reach the same level of success that SVRAs alone had previously achieved. We explore this finding in further detail later.

Other Effects of TTW

As noted earlier, increasing the rate at which beneficiaries leave the disability rolls for work and thereby lower the amount of cash benefits paid by SSA was not TTW's only objective. Other objectives included expanding the availability and use of employment services, increasing the self-sufficiency of people with disabilities through expanded earnings, and spending less on employment services than the dollar value of benefits that would be paid if beneficiaries remained on the rolls.

TTW clearly increased consumer choice for beneficiaries. Further, under TTW, SVRAs could choose, case by case, whether to serve a client under either the existing cost reimbursement model or one of the new TTW payment options—in the latter cases, operating essentially as ENs. In addition to these SVRAs operating under both the new and old payment rules, approximately 600 ENs joined the ranks of employment service providers under TTW.⁷ Many of those ENs were VR vendors that had provided services under contract with SVRAs, but others offered new service models. These included “consumer-directed service ENs, which share a portion of the TTW payments they receive with their clients; employer ENs receiving TTW payments based on work activity of their TTW participant employees; [and] state workforce agency ENs comprising local workforce investment boards and One-Stop Career Centers” (Schimmel and others 2013). Beneficiaries have responded to the expanded choice by using EN services at an increasing rate since they were first offered, and this increase in choice also likely contributed to some of the increase in participation in return-to-work services that Mathematica found. Using data from SSA's Disability Analysis File (DAF), Chart 1 shows that although SVRAs have dominated the TTW ticket assignments, EN assignments have risen steadily since that option was fully implemented in 2005.⁸ Although SVRAs received 93 percent of ticket assignments in 2005, their market share had dropped to 72 percent by 2019. Further, while the numbers of tickets assigned to SVRAs have remained relatively constant since 2012 (with slight declines since 2015), EN assignments have increased by 80 percent since 2012, indicating progress toward Congress' intent of expanding service alternatives.

Mathematica also found that TTW increased the level of interest in work among disability program beneficiaries. The National Beneficiary Survey (NBS), sponsored by SSA and conducted by Mathematica, includes questions that probe beneficiaries' expectations about future employment. Comparing results from the first NBS in 2004 (Thornton and others 2006) with those from the round completed in 2015 (SSA 2018a) shows an increase in beneficiary interest in substantial work across multiple measures:

• Goals include work or career advancement (from 30 percent to 37 percent);
• Sees oneself working for pay
  • in the next 2 years (from 20 percent to 25 percent)
  • in the next 5 years (from 26 percent to 28 percent);
• Sees oneself working and earning enough to stop receiving disability benefits
  • in the next 2 years (from 7 percent to 11 percent)
  • in the next 5 years (from 15 percent to 17 percent).⁹

This increase in interest in work is consistent with the net increase in participation in employment services noted above. Stapleton, Mamun, and Page (2013) found that employment service enrollment increased by about 15 percent over the 48-month observation period in the Phase 2 and Phase 3 states. Increases in reported interest in work and in employment service participation would seem to be a significant positive result, even without increases in disability roll exits. Congress intended TTW, at least partly, to improve opportunities for disability program beneficiaries and to change expectations about work, so it seems reasonable to regard higher interest in work and employment service participation as TTW successes.

Not all of the additional participants assigned their tickets to ENs: They distributed their tickets among both provider types, and the TTW evaluations found differences between participants who sought EN services and those who sought SVRA services (Thornton and others 2006; Stapleton and others 2008). For example, Stapleton and others found that

participants were more likely to assign their Ticket to an EN if they had low benefits, were African American, had minor children, or had been on the rolls for a year or less. Those with [intellectual disability, with] sensory or other nervous system disorders, [with] severe mental health problems, or in need of assistance to perform a daily activity were more likely to assign their Ticket to an SVRA than to an EN.

Thus, TTW likely led a broader range of beneficiaries to attempt employment and gave them greater chances of finding services that fit their situations. This, combined with the overall increase in employment service use, suggests that TTW prompted certain beneficiary groups to consider work and to pursue employment goals in greater numbers.¹⁰ The fact that participation increased without an increase in the proportion reaching STW status implies that these new participants were less successful overall than those who had participated before TTW implementation. This suggests that TTW broadened the range of beneficiaries interested in work, drawing participants from groups facing marginally greater challenges to work than those who had received SVRA services prior to TTW. These additional participants from the margin likely had less success than other beneficiaries. Moreover, even if we saw no increase in STW status for these beneficiaries, they may have improved their self-worth by attaining some level of work and likely improved their quality of life by adding some earnings to their benefit income.

Livermore, Hoffman, and Bardos (2012) found that most TTW participants were satisfied with the services they received during the program's early rollout. About 60 percent of EN users considered themselves “very” or “somewhat” satisfied with those services. Satisfaction was higher for SVRA clients, at about 70 percent. Regression-adjusted satisfaction results improved for both EN and SVRA users after SSA implemented modified TTW regulations in 2008, although only the increase for ENs was statistically significant, at 36 percent.¹¹

Each year, many SVRAs do not have enough resources to serve all their applicants. When that occurs, those SVRAs enter “order of selection” status, wherein they must ration services and prioritize individuals with the most significant disabilities.¹² SVRA clients include individuals who are not beneficiaries of SSA programs; and although SSA program beneficiaries tend to have the most significant disabilities among SVRA applicants, they do not constitute all the clients who are prioritized under orders of selection (Honeycutt and Stapleton 2013). Thus, by adding ENs as service providers, TTW has helped to expand the availability of employment services for beneficiaries despite tight fiscal resources among SVRAs. Without ENs, either SVRAs would need to increase the level of services they provide to absorb the demand currently fulfilled by ENs or unmet demand for employment support services would increase.

Although TTW has not significantly increased the number of beneficiaries who reach self-sufficiency relative to an SVRA-only system, it has increased the overall availability of employment services, consumer choice, and beneficiary participation, and beneficiaries are generally satisfied with the services they receive through TTW.

Measuring TTW Costs and Benefits

Assessing whether TTW's net effects were “substantial” is a matter of interpretation. The most policy-relevant question is whether TTW provided benefit savings (and other effects) sufficient to offset program costs.

TTW participants represent a relatively small proportion of disability beneficiaries. As a result, Thornton (2012) estimated that relatively modest program effects, on the order of a net increase of 2,000 to 3,000 disability roll exits, would be sufficient to offset the costs of the TTW program. Unfortunately, measuring such a small effect was precluded by the evaluation's limited precision:

The fact that we did not find statistically significant impacts on STW [incidence or months in STW status] does not by itself rule out the possibility that TTW under the initial regulations had impacts on these outcomes that were sufficiently large for the program to be “self-financing”—that is, for savings from a net reduction in benefits to be sufficient to pay for TTW payments to providers and all administrative costs attributed to the program. Thornton (2012) suggests that only a very small impact—an increase of 3,000 or so in the number of all beneficiaries experiencing STW [status] for the first time in each year—might be sufficient for the program to be self-financing. An annual impact on first-time [STW incidence] that is as small as 3,000 might correspond to such a small impact on STW [incidence] as of month 48 for new, young [DI] beneficiaries that the evaluation would be unable to differentiate between that impact and no impact at all (Stapleton, Mamun, and Page 2013).

Stapleton, Mamun, and Page estimated that 3,000 new instances of STW status in 2007 as a result of TTW would have represented a 5 percent increase in those cases. This leaves us with a rather ambiguous evaluation result. We know that the program did not lead to large-scale changes—TTW increased STW incidence by less than 30 percent and it increased the number of months in STW status by less than 9 percent—but we cannot rule out the possibility that it had effects that were of policy relevance. An increase as modest as 5 percent in DI and SSI exits would be sufficient to cover the cost of the program, but we cannot tell if such a change occurred.

We can look at cost in the broader sense as well. Because the employment service participation rate increased without a corresponding increase in the rate of disability roll exits, we would expect that the cost of providing such services rose. This in fact took place, but not necessarily because EN services were, on average, more costly. As we detail in Appendix A, ENs and SVRAs have different cost structures related to their different client populations, and this complicates the comparison of costs between the two provider types. Thornton (2012) found that the total provider payments per STW case were notably lower for ENs than for SVRAs. He estimated that SSA-financed return-to-work assistance per STW case was $11,500 (in 2009 dollars) for ENs and $14,035 for SVRAs.¹³ “The $2,535 difference represents the savings SSA receives when it pays for exits under the new TTW system, which makes outcome payments only while beneficiaries remain off of cash benefits, rather than under the traditional payment system” (Thornton 2012, 40). This result should be viewed with caution as it defines costs narrowly for both ENs and SVRAs, but it does suggest that costs could be higher if SSA reverted to an SVRA-only system and SVRAs served all beneficiaries currently receiving services through ENs.¹⁴

Even under Thornton's narrow definition of costs, the actual increase in SVRA costs relative to those for ENs would likely be less than the $2,500 per STW case noted above because the lower EN costs partly reflect the ability of ENs to choose to serve only lower-cost beneficiaries. If those beneficiaries used SVRAs, they would likely generate lower costs and SSA reimbursements, bringing down the average SVRA cost per STW case. Still, as we describe in Appendix A, it seems safe to conclude that TTW has expanded service availability without increasing costs over what would be required to serve the same number of beneficiaries under an SVRA-only system.

Comparison of TTW Costs and Benefits

Regardless of whether TTW expanded service availability and use or changed net disability benefit receipt, we would like to know whether TTW services are effective and provide efficient results for participating beneficiaries. In other words, did TTW services decrease SSA benefit payments relative to what participants would have received in the absence of those services, and were the costs of TTW services less than the net reductions in benefits? Stapleton, Mamun, and Page (2013) focused on whether TTW changed the rate at which disability program beneficiaries return to work relative to the SVRA-only system that preceded it. Their study did not assess whether either the SVRA or EN components of TTW led to reduced benefit payments relative to no services. This is an important distinction. We showed above that TTW expanded service delivery and had the same success as the SVRA-only program that preceded it. We now address whether services provided through SVRAs or ENs under TTW reduce reliance on cash benefits. This is a crucial question, as nothing is gained in expanding an ineffectual program.

Although TTW subsumed the national SVRA services program for SSA program beneficiaries with disabilities, many aspects of SVRA and EN services remain separate. We therefore explore the costs and benefits of these two provider types separately, assess their efficiencies, and compare their results.

To estimate the beneficial effects of TTW, we focus on the reductions in disability benefits that have resulted from participation. Ideally, we would estimate reductions in disability benefits that have been adjusted to account for the counterfactual: what would have happened if neither EN nor SVRA services were available under TTW. As noted in our discussion of the 2008 and 2013 evaluation reports, however, the methods available for estimating these effects have shortcomings. We therefore start by estimating gross benefit reductions without considering the counterfactual. Because gross benefit reductions are essentially an upper-bound estimate of TTW effects, they can be compared with costs to get a sense of whether net benefits exist at all. If costs are greater than any associated reductions in benefit payments, the program cannot pay for itself. Conversely, if the magnitude of benefit reductions grows relative to program costs, the opportunity for actual savings increases. So, although the outcome analysis cannot definitively show whether TTW provides savings, it can compare program costs to the benefits forgone for work (BFW) to provide an upper bound on TTW effects and thus set context for analyzing whether TTW may offset all or most of its costs.¹⁵

TTW Costs

The two components of TTW program-related expenses are the administrative cost of running the program and the SSA provider payments to ENs and SVRAs. Agency administrative data closely track the provider payments as part of managing the program. SSA's Office of the Inspector General reports these amounts by the year when the payments occur (SSA 2018b). This accounting method has problems because it is affected by budgetary and administrative priorities that are not directly relevant to TTW operations. Reporting cost by date of disbursement tends to show lower payment levels in years when staffing for SSA payment processing was tight, followed by higher payment levels in subsequent years as processing catches up with backlogs. Recording payments by the dates when they are due—when the service provider completes the documentation that supports that payment—would better represent the timing of the beneficiary's work activity that triggered the payment.

Table 2 shows the number and value of payments made to employment service providers from 2002 through 2016.¹⁶ The numbers of payments under the new TTW systems prior to 2006 were much lower than traditional-system reimbursements as SSA phased in the new program. The low figures in the early years also reflect the lag between the start of EN services and the point at which participants began working at a level sufficient to initiate a payment for the service provider. Payment frequencies under the TTW systems picked up starting in 2006 and then rose significantly after 2008 when SSA implemented new TTW regulations. In the later years, providers using the new milestone- and outcome-oriented payment systems established by TTW received far greater numbers of payments than did SVRAs using the traditional reimbursement model. This reflects the nature of the two payment structures, wherein traditional reimbursement is essentially a “lump-sum” payment after service completion while multiple payments under the newer TTW systems occur over time as beneficiaries reach employment milestones. By 2016, total annual new TTW-system payment amounts had risen to more than $63 million. Yet because the new TTW-system payment amounts are relatively small on average, their total value each year is still lower than that of traditional-system reimbursements. Payment amounts under the traditional system have averaged around $125 million annually, although payments in 2003, 2010, and 2011 were noticeably lower, likely because those years were affected by recessions. All payments under the traditional system went to SVRAs; most of the payments under the TTW's new systems went to ENs, but about 20 percent went to SVRAs (not shown).

Table 2 includes an interesting dynamic in that the total payment amounts for the new TTW systems relative to the traditional system climbed steadily over time, growing from nearly nothing in 2002 to nearly 40 percent in 2011. Since 2011, TTW-system payment amounts have apparently settled at about one-third the amount of traditional-system payments, although differences in payment lag times between the systems might skew the proportions for 2015 and 2016. By 2015, total provider payments had reached about $250 million, with indications that they would continue to rise.

In addition to the payment costs to providers, SSA has incurred administrative costs to run TTW. Some of these costs can be viewed as the continuation of the SVRA administrative costs that existed before TTW subsumed the old system. TTW was not funded under a separate appropriation so there is no line item in the budget that can be used to cite TTW administrative costs. Because there also is no administrative need to track TTW costs separately from those for related functions, SSA does not do so. Still, stakeholders overseeing TTW have expressed interest in quantifying those costs. In response, we used SSA data on staffing levels and estimated the hours spent on TTW-related tasks to compute TTW administrative costs for 2017 in an unpublished SSA report (O'Leary and Roessel 2018).¹⁷ For this analysis, we adjust those 2017 cost estimates to 2016 dollars so that they align with our data on EN and SVRA provider payments.

The absence of TTW cost estimates for all program years poses alignment issues for this cost-benefit analysis, which covers TTW operations and beneficiary participation beginning in 2006. We make the simplifying assumption that the administrative costs that occurred in 2006 are the only administrative costs for this cohort.¹⁸ Ideally, 2006 costs would have been estimated shortly after the close of 2006 and then adjusted to 2016 dollars for this analysis, but no such estimates are available, and recall issues would make new estimates for that year unreliable. Instead, we use the 2017 cost estimate, adjusted to 2016 dollars, which implicitly assumes that annual TTW administrative costs have been flat since 2006. This poses a limitation, which we discuss further below.

In this analysis, the TTW administrative cost estimates account only for SSA staffing, printing, mailing, and contract resources specifically dedicated to the TTW program. Costs are separately calculated for operations related to EN and SVRA ticket assignments. Staffing costs include those for headquarters, regional office, and field office personnel and are based on workload assessments for a representative sample of staff. The administrative cost estimates do not include costs for activities not specifically related to the TTW program. They also do not include any costs for information systems except those managed by the Ticket Program Manager (TPM) under the TPM contract. Further, they exclude any costs associated with beneficiaries who remain on the rolls because SSA did not conduct a CDR, under the TTW provision protecting participating beneficiaries from undergoing one (which might lead to benefit cessation).¹⁹ Finally, we also exclude any potential excess costs of unrecovered overpayments for TTW participants and associated administrative costs.²⁰

We estimate total TTW administrative costs for 2016 of $18.8 million, of which $12.3 million were for operations related to EN ticket assignments and $6.5 million were related to SVRA assignments (Table 3). Based on the 2016 figures from Tables 2 and 3, the administrative costs related to SVRA ticket assignments under the traditional payment system represent about 3 percent of total costs of operating that part of the program ($6.5 million ÷ [$199.5 million + $6.5 million]), and administrative costs related to EN assignments under the TTW payment systems represent about 16 percent of total costs of operating that part of the program ($12.3 million ÷ [$63.1 million + $12.3 million]). Notably, the costs related to SVRA assignments do not account for the substantial SVRA funding provided by non-SSA federal block grants and state funds, which we discuss in Appendix A.²¹

Benefits of TTW

Prior analyses of the utility of employment services generally measured success based on the earnings of the client after the completion of services (for example, Ashenfelter 1978; Couch 1992; Dean, Dolan, and Schmidt 1999; and Dean and others 2001, 2015). Although a client's postservice earnings are of interest to SSA, they are not an ideal measure of TTW's effectiveness and efficiency. To SSA, the more important question is whether these services lead to a decreased reliance on cash benefits (Dean and others 2014), and more specifically, whether the reduction in benefits linked to TTW services is greater than what SSA pays for those services. SSA recently developed BFW, a new measure that uses an algorithm to calculate the cash benefits given up over time when beneficiaries start working (Mathematica 2022). BFW has been incorporated into the DAF, enabling us to use it for this analysis. Linking TTW service data with the BFW data in the DAF allows us to compare the cost of these services with any subsequent decrease in cash benefits because of work.

We examine the effect of TTW services by following a client cohort over time. A successful progression through employment services generally begins with enrollment (ticket assignment), followed by service provision, then employment, possibly involving several attempts. Because the traditional cost reimbursement system and the newer TTW payment systems are all outcome-oriented, some level of employment success is necessary before SSA begins to incur costs in the form of employment service provider payments. For reimbursement under the traditional payment system, an SVRA's client must attain earnings of SGA level or higher for at least 9 months in a 12-month period.

The TTW payment systems are more complex.²² Recall that SVRAs that opt for payment under one of these systems are effectively ENs. ENs can qualify for a phase 1 milestone payment, a phase 2 milestone payment, and an outcome payment.²³ An EN qualifies for a phase 1 milestone payment when the beneficiary reaches 1, 3, 6, and 9 months of earnings at the TWP level ($810 in 2016). An EN qualifies for a phase 2 milestone payment each month that the beneficiary earns at or above SGA level. Outcome payments require that the beneficiary be in STW status.

For an SSI recipient, earnings of more than $85 per month can trigger a benefit reduction. As such, a reduction can take effect rather quickly, although the dollar reduction is usually modest. For DI beneficiaries, by contrast, there are no partial reductions in benefit amounts, and they must complete a TWP before attaining STW status, although the benefit reduction amounts are typically more substantial. Because of these differences, it is unclear whether TTW is more cost-effective for SSI recipients or DI beneficiaries. DI beneficiaries are better candidates for employment because of their work histories, but they typically are also older, are more likely to have physical disabilities that are associated with poorer return-to-work outcomes, and must sustain employment longer to generate benefit reductions. Conversely, SSI recipients tend not to have extensive work histories, but they are younger, are more likely to have mental conditions that are associated with better return-to-work outcomes, and can generate benefit reductions sooner (SSA 2020).

Regardless of the payment system, positive employment outcomes take time to occur. Clients might take a year or two to complete employment services, with another year to find stable employment, and, for DI beneficiaries, still another year to complete their TWP. Only then does the beneficiary enter an extended period of eligibility during which benefits are suspended because of work. Thus, substantial benefit reductions are likely to take 3–4 years after enrollment to emerge, and even longer if the labor market sours or the participant does not quickly find stable employment. Further, because the cost of employment services can be high, and SSA pays at least a portion of those costs before the client accrues substantial work duration, a long time may pass before the value of benefits not paid because of work covers the cost of services.

Because it takes time to see whether TTW outcomes may have offset costs, we examine a 2006 cohort of TTW participants and follow them through receipt of employment services and any attainment of STW status that ensues through 2016.²⁴ We selected the 2006 cohort because pre-2006 cohorts participated in TTW while its rollout was not yet complete, and the experiences of the 2007–2009 cohorts were more directly affected by the anomalous economic conditions of the Great Recession. Employment outcomes for the 2008–2011 cohorts were lower than those for 2006, but beginning with the 2013 cohort, inflation-adjusted outcomes again approached the 2006 levels. The 2006 cohort should thus provide representative results for a period of relative economic stability.²⁵

Under both the TTW and traditional payment systems, some payments occur before a beneficiary maintains work for a substantial period. This means SSA pays providers for some beneficiaries whose work effort never reduces benefit payments enough to fully offset the service costs. However, service providers must submit payment requests with documentation, as appropriate, in a timely manner to be eligible to receive those payments. Because service providers do not always complete these payment-request steps within program-required limits, some beneficiaries receive services and attain employment that reduces SSA's benefit costs without generating SSA payments to ENs or SVRAs.²⁶ We track TTW participants by the type of provider to which they assigned their ticket and by whether SSA made payments to the provider. Table 4 shows the extent to which 2006 DI beneficiaries or SSI recipients aged 18–65 in January 2006 participated in TTW services during 2006 and had their cash benefits suspended because of work that followed employment-service receipt.²⁷

SSA's DAF indicates STW status as well as the BFW measure mentioned above. SSA created STW status as a research variable because the agency's administrative data do not specifically identify work as a reason for a suspension or termination. SSA uses an algorithm to compare an individual's benefit status with information on work, earnings, other income, medical reviews, retirement, and death to determine whether benefit suspension or termination resulted from work or for one of these other reasons (Mathematica 2022).

Table 4 shows that very few beneficiaries enrolled in TTW in 2006. Of the 11.1 million beneficiaries on the rolls at some point in 2006, only 1.2 percent enrolled for SVRA services that year and only 0.04 percent enrolled for EN services. Yet those who enrolled for services had substantially better employment outcomes through the end of 2016 than those who did not. Of those who could have enrolled in TTW in 2006 but did not, only 3.2 percent reached STW status for at least 1 month by the end of 2016. For those who enrolled in SVRA services in 2006, the proportion was 18.7 percent and for those entering EN services in 2006, it was 28.6 percent. As might be expected, those who generated SSA payments to their service provider had even higher rates of STW status by the end of 2016. Of those generating SVRA payments as of year-end 2016, 76.3 percent attained STW status for at least 1 month, while of those generating EN payments, 62.7 percent reached STW status for at least 1 month.

Table 5 shows the values of BFW and the costs incurred for those beneficiaries under TTW. Table 4 showed that 3.2 percent of the 2006 cohort was able to reach STW status without the help of EN or SVRA services and Table 5 shows that this small share of beneficiaries nonetheless generated significant BFW. This is because 3.2 percent of such a large base (nearly 11 million) is a substantial number of (current and former) beneficiaries. Through 2016, those who were eligible for TTW services in 2006 but did not use them generated nearly $16 billion in gross BFW—benefits that were not paid because those beneficiaries were in STW status. Because those beneficiaries received no services, there are no service costs and the net BFW are this same $16 billion.

The smaller proportions of beneficiaries that used EN and SVRA services are reflected in the lower BFW results. Overall, each dollar spent by SSA on SVRA services resulted in more than $6 in gross BFW, while each dollar spent on EN services generated more than $2 in gross BFW. SVRA clients who enrolled in 2006 ultimately generated $711.7 million in gross BFW, $6.5 million in administrative costs, and $108.9 million in provider reimbursement costs, leaving $596.3 million in net BFW. Those who enrolled with ENs generated $54.2 million in gross BFW, $12.3 million in administrative costs, and $10.3 million in provider payments, leaving $31.6 million in net BFW.

Interestingly, Table 5 also shows substantial gross BFW in cases that never generated a payment to a service provider: $454.5 million for SVRA users and $8.8 million for EN users. This can occur for a variety of reasons, but the most likely cause is service providers losing track of some participants who find work. Unaware of their clients' employment success, the providers are not timely in their request for payment from SSA (Stapleton and others 2010). With minimal administrative costs and no provider reimbursements in these instances, the net BFW are nearly equal to the gross BFW for these SVRA users. For EN users, the administrative costs more than offset the gross BFW, leaving a net loss of $0.3 million in these cases. For clients who generate provider payments, combined administrative costs and provider payments reduce gross BFW by 42 percent for SVRA enrollees and by 30 percent for EN enrollees, leaving net BFW of $147.9 million and $31.8 million, respectively.

Although the total amounts show the scale of BFW, they are not useful for examining TTW's relative efficiency. Table 6 repeats Table 5 with per-beneficiary averages in place of total amounts. Beneficiaries in 2006 who never enrolled in TTW services generated a relatively modest average net BFW of $1,446. SVRA clients who generated no payments to their providers averaged $3,504 in net BFW. SSA's cost for EN clients who generated no provider payments exceeded gross BFW by $99 per beneficiary.

The average net BFW for SVRA clients who generated a provider payment were nearly $20,000 and the average net BFW for EN enrollees who generated a provider payment were about 50 percent higher, at nearly $30,000, despite ENs' higher administrative costs. This is because average gross BFW are significantly higher for EN enrollees than for SVRA enrollees, while average EN provider payment amounts are significantly lower. For TTW enrollees overall, disregarding provider payment status, EN clients generated substantially higher average net BFW ($7,324) than SVRA clients ($4,405).

Although it is tempting to conclude from Table 6 that EN services are more expensive than SVRA services but attain significantly better outcomes, that is a misleading oversimplification. EN services are more costly to SSA but differences in the SVRA and EN cost structures and client populations, detailed in Appendix A, result in a significant portion of SVRA costs being borne by state and federal governments through block grants.²⁸ Further, costs are not distributed evenly across TTW participants; those with greater employment success incur higher administrative and service costs than those with less success. Further still, because SVRAs and ENs are subject to different enrollment rules, SVRAs receive far more ticket assignments than ENs and are reimbursed for a much smaller fraction of their clients (5 percent, versus 26 percent for ENs). Because costs are linked to success, average costs for SVRAs are significantly lower than those for ENs. Although this poses no problem for comparing administrative and provider-payment costs with BFW, as in Table 6, simple averages cannot tell us what the expected costs for a typical EN participant would be under an SVRA assignment or vice versa. Would the typical EN (or SVRA) participant be more (or less) successful than the typical SVRA (or EN) participant and thereby generate costs to SSA that are higher (or lower) than those of the other provider type, assuming similar results?

In Appendix A we adjust for the differences in SVRA and EN clientele and show that the average costs by provider type are similar—even if we ignore the effects of block grants. These adjustments are imperfect, but they imply that shifting clients from SVRAs to ENs and vice versa is unlikely to significantly change the overall cost of providing employment services to SSA disability program beneficiaries.

Beyond this, the higher BFW values attained by EN clients also reflect differences stemming from the enrollment rules under which the ENs and SVRAs operate. ENs are allowed to choose which beneficiaries they will serve while SVRAs are required to serve anyone they are able to help and, if they are over capacity, must serve those with the most severe disabilities first. Thus, SVRAs could be expected to have a harder-to-serve clientele, on average, than ENs have. In the same way that one would not expect the average cost for one provider type to apply to the other, one would not expect the average success rate for one provider type to apply to the clientele of the other: an SVRA's (or EN's) success would improve (or decline) if it served the other provider type's clientele. Rather than comparing costs and outcomes between ENs and SVRAs, the point to be drawn from Table 6 is that participants in both ENs and SVRAs attain BFW that are significantly greater than their costs to SSA, and the results are far better than those for beneficiaries who get no SSA-funded services.

Although the outcome comparisons in Table 6 are interesting, the comparisons between service recipients and nonrecipients are likely to include significant bias because seeking services is itself a strong signal of interest in work. That bias would tend to make TTW outcomes look better than they really are because TTW participants represent a more motivated subpopulation among beneficiaries. We can adjust for this bias by limiting the analysis to DI beneficiaries who have completed a TWP. That group is not representative of all beneficiaries; it consists only of those who received DI benefits (either with or without concurrent SSI payments) and worked enough to earn at the trial work level (in 2016, at least $810 per month) for 9 or more months within a span of 12 consecutive months. The TWP duration and earnings criteria remove much of the bias in Table 6 in that they require the willingness and the ability to work at a substantial level, regardless of whether the beneficiary seeks employment services. However, a serious interest in work is not the only potential bias related to service use. Those who receive services presumably believe they need them. To the extent they are correct, one might expect these beneficiaries, in the absence of services, to have poorer employment outcomes than beneficiaries who forgo services, all else being equal. So, although we adjust for motivation toward work by restricting the analysis to beneficiaries who complete a TWP, the negative bias of service users needing those services likely remains.

Using TWP completion to control for work effort and aptitude narrows our examination to a subset of beneficiaries; therefore, any results for that group cannot be extrapolated for the larger group. Nevertheless, restricting to those with TWP completion allows us to intuitively account for bias and reveal more accurate net effects of TTW for at least this subset of beneficiaries. Table 7 shows the average BFW and costs for beneficiaries who were eligible for TTW services in 2006 and subsequently completed a TWP. Of the beneficiaries who never enrolled in TTW, only 3.0 percent (333,659) ever completed a TWP; and while only 3.2 percent of TTW nonenrollees overall ever attained STW status, 48.8 percent of those who completed a TWP did. Average net BFW also differed dramatically, at $1,446 for TTW nonparticipants overall (Table 6) and $27,387 for those who eventually completed a TWP (Table 7). Controlling for TWP completion also changes the STW-status and BFW numbers for TTW participants, although by significantly smaller proportions. Comparing Tables 4 and 7 shows that STW status increases from 18.7 percent for SVRA participants overall to 50.2 percent for those who completed a TWP, and from 28.6 percent to 60.4 percent for EN participants. Among those who completed a TWP, average gross BFW for SVRA clients are $17,622 and for EN clients they are $31,812. Although the average gross BFW for all SVRA clients in Table 7 are lower than those for TTW nonparticipants, the average gross BFW for TTW participants whose service providers received SSA payments are much higher than those for nonparticipants ($43,742 for SVRA clients and $53,410 for EN clients). Accounting for TTW costs drops the average net BFW further below the level for nonparticipants (who had no service costs) for both SVRAs and ENs overall, but for cases that generated SSA payments for services, the average net BFW still exceed those for nonparticipants.

Table 7 may also indicate the effect of differing levels of need for services in the difference between outcomes for ENs, which can choose whom they serve, and SVRAs, which cannot. Thus, the poorer outcomes for SVRAs could reflect a clientele that not only needs services to succeed, but also needs more intensive services than those provided by ENs. These findings suggest that, even when we restrict our analysis to beneficiaries who completed their TWP, significant differences remain between those who do and do not receive services, and between those who receive services from ENs versus SVRAs.

Table 8 compares the characteristics of DI beneficiaries who completed a TWP for each of the three types of employment service use (none, SVRA, EN). EN clients were the quickest to start working and had higher average initial earnings than SVRA clients. Although it took SVRA clients longer than EN clients to start work after assigning their tickets, the median time differed by only 6 months.

Relative to the other service-type groups, service nonusers were significantly more likely to have musculoskeletal- or organ-system impairments, SVRA clients were significantly more likely to have intellectual or sensory-system disabilities or to receive concurrent DI and SSI benefits, and EN clients were less likely to have disabilities from injuries or to have no work experience prior to starting benefits. SVRA clients were also about half as likely to die during the observation period as were those in the other service-type groups, which likely reflects the younger average age of SVRA users. Consistent with Table 7, which shows that EN clients tend to attain higher BFW and are more likely to reach STW status than nonparticipants and SVRA users, Table 8 shows that EN clients also maintain STW status for a longer period.

Multivariate Analyses

Differences in beneficiary characteristics are likely to account for the differences in TTW outcomes that remain after controlling for TWP completion. We can account for these differences by using the following simple ordinary least square (OLS) multivariate regression:

where:

$Y$ is our outcome,

$X$ is our vector of observable characteristics,

$T$ is a dummy indicating whether the beneficiary received TTW services, and

$\epsilon$ is our error term.

Table 9 shows the regression results for total earnings, STW status attainment and duration, and gross BFW for DI beneficiaries who completed a TWP, controlling for the observable differences.

Given the large dataset (358,393 beneficiaries), it is not surprising that many of the parameter estimates are statistically significant. We therefore focus on those that are both significant and of greater magnitude. Across all four models, intellectual disability is significantly negatively associated with positive employment outcomes, while neoplasms have a significant positive relationship. Disability because of injury has significant positive effects on earnings, months in STW status, and BFW, and a modest positive effect on attaining STW status. Concurrent-benefit status is negatively associated with employment outcomes in all models but the magnitude of the effect on attaining STW status is small.

Longer periods between last employment and first benefit are negatively associated with earnings and BFW, although only the earnings magnitudes are large. Disabled-worker beneficiaries must by definition have work histories, so DI beneficiaries without prior earnings must qualify either as disabled widow(er)s or disabled adult children with eligibility based on the work history of a spouse or parent who is disabled or has retired or died. These results suggest that, among beneficiaries who completed a TWP, surviving spouses and adult children tend to have the highest earnings; but among those with work histories, recency of work is associated with higher earnings. For the two STW-status regressions, the signs in the results are more in line with expectations in that those who worked within 2 years of benefit commencement had positive STW-status outcomes, but the magnitudes are small. Further, among TWP completers, both age and sex have limited effects on outcomes, although results are consistent with other research, as employment success declines with age and men are more likely than women to attain employment.

After accounting for these characteristics, our results are roughly consistent with those of the simple outcome analysis in Table 7. Table 9 also shows that those who received EN services attained $24,214 more in total earnings than those who received no services. Further, they were 10.7 percent more likely to reach STW status, accrued 7.07 more months in STW status, and accounted for $3,381 more in gross BFW through the end of 2016 (versus the $4,425 difference indicated in Table 7). Relative to those who did not receive employment services, SVRA clients had lower total earnings, attained and retained STW status at about the same levels, and accounted for $10,601 less in gross BFW (compared with the $9,765 difference indicated in Table 7).

We can also use TWP completion status to model gross BFW on the full DI population using a Heckman two-stage selection model. The Heckman model was developed to produce regression estimates—for example, of wages—for an entire population when input data are available for only a subset of the population—for example, those who work (Heckman 1974, 1976). Without such a model, any estimate based on only the wages of workers would be biased and tend to overstate wages for nonworkers. Whether individuals work is related to wage rates: They will work only if they can earn above a particular amount. The Heckman model uses information on the work decision (which, unlike that for wages, is available for everyone) to adjust the wage equation for those who do work. It solves the bias problem by estimating a first-stage equation on the probability of work (that is, the probability of having wages) based on characteristics that drive the work decision. From this equation, one can calculate the inverse Mills ratio, $\lambda$, which provides a measure of the selection bias driving whether an individual works, and thus has earnings, for the wage equation. By including $\lambda$ in the second-stage wage equation, the Heckman model can produce unbiased wage estimates for the entire population.

Although Heckman developed this technique in the context of wages and work, it can apply to other situations where outcome data are only available for a subpopulation that is defined by a participation decision. For our purposes, we can use it to estimate dollar outcomes (BFW) that are subject to exceeding a threshold value.

We use the following model to estimate BFW:

where:

$Y$ is the BFW outcome,

$X$ is the vector of observable characteristics,

$T$ is a dummy indicating whether the beneficiary receives TTW services,

$A$ is aptitude for self-supporting work, and

$N$ is need for assistance in attaining self-supporting work.

$A$ and $N$ are unobservable, yet they determine whether we have data on $Y$. If we run our analysis on the full population and do not account for these unobservable variables, we will have biased results. In the regressions in Table 9, we adjusted for $A$ by including only those who had completed a TWP. Under the Heckman model, we include all DI beneficiaries and modify equation 2 as follows:

We drop the unobservable quantity $A$ from equation 3 and add equation 4, the selection equation, with $B$ as the threshold dummy variable indicating whether we see outcome $Y$. For our analysis, only a person with sufficient aptitude and desire for self-supporting work generates BFW; that is, when $\gamma X^{\prime }+\upsilon \ge 0$. We substitute $T$, the completion of the TWP, for $B$ as our selection variable. Both $B$ and $T$ are indicators of self-supporting aptitude; but whereas $B$ indicates that self-supporting work has occurred, $T$ indicates the broader condition that self-supporting work is possible and the beneficiary is sufficiently motivated to attain it.²⁹ Another advantage of using TWP completion is that it is an SSA program variable (meaning that administrative data track the count precisely) while BFW is a constructed variable, constituting an estimate generated from an algorithm (Mathematica 2022).

Also, note that our selection equation only addresses selection that is due to aptitude for self-supporting work. The variable $N$, the need for assistance to get to self-supporting work, remains unobservable because our Heckman model does not have a proxy for it, so it is omitted and our results will retain some bias. Note also that $A$ and $N$ work in opposite directions: omitting $A$ will tend to overstate the effect of TTW services, $\delta$, and omitting $N$ will tend to understate $\delta$.

Equation 4 is our first-stage selection equation on the probability of completing the TWP, and equation 3 is our second-stage estimate of BFW, adjusted for those who do complete their TWP and thus are unlikely to have nonzero BFW. Notice that we use the vector of independent variables,$X^{\prime }$, in the selection equation rather than $X$. To implement the Heckman approach properly, we need exclusion restrictions both to minimize correlation between $\lambda$ and the vector of independent variables and to facilitate model identification.³⁰ To do so, we want to include independent variables in stage one that predict selection into the TWP-completion group but do not predict the BFW, at least not directly. Although equations 3 and 4 include many of the same independent variables, only the selection equation includes whether the beneficiary died before the end of the observation period. This variable is meant to proxy for health (as distinct from disability) in that it seems likely that many beneficiaries with short life horizons will be aware of that likelihood and will be less likely to pursue work at substantial levels as a result. We also exclude the two variables that, based on Table 9, are related more to earnings (stage 2) than to the decision to work at a substantial level (stage 1): most recent work before benefits and benefit type (DI only versus concurrent DI/SSI). The Heckman model results are shown in Table 10.

In the first-stage (probit) regression, congenital anomalies and intellectual disability have strong negative correlations, relative to musculoskeletal system impairments, with completing a TWP. The same is true, to a lesser degree, for nervous system impairments. Not surprisingly, death before the end of the observation period is also strongly and negatively associated with TWP completion. Alternatively, blood disorders/infectious diseases and neoplasms are positively, and strongly, associated with TWP completion. Being male, older, and with longer time on the rolls are all negatively associated with TWP completion, but the effects are relatively small.

In the second-stage (OLS) regression, recipients of concurrent DI/SSI payments have significantly lower gross BFW than DI-only beneficiaries, by $13,509 on average. Relative to musculoskeletal impairments, sensory system disorders and blood disorders/infectious diseases all significantly reduce gross BFW. Injuries and congenital anomalies are both associated with higher BFW relative to musculoskeletal impairments, but neoplasms are associated with the highest relative increase in BFW at $23,588. Higher monthly benefit amounts are also associated with higher BFW: An increase of just $100 in the monthly benefit is associated with a $3,100 increase in BFW. Given that DI benefits generally vary between $1,000 and $2,000 per month, this effect can be substantial. Not surprisingly, men are more likely to attain higher BFW levels than women, but the relative difference is modest ($3,550). Similarly, not having worked since 5 or more years before starting benefits has the expected negative effect, but again it is modest relative to those with no prior work ($3,672).

The Heckman model results of greatest interest are the effects of SVRA and EN services. EN service use increased gross BFW over the period by $3,544 per beneficiary (relative to TTW nonparticipants) while the effect of SVRA service use under TTW was to reduce BFW over the period by $10,692. Both estimates are very close to the Table 9 OLS regressions on BFW ($3,381 and −$10,601, respectively). The parameter estimate for the inverse Mills ratio ($\lambda$) in the Heckman model is significant and, as expected, negative. The magnitudes of TTW effects estimated with the Heckman model are much smaller than those that would have resulted from running the regression without any selection adjustment for serious interest in work.³¹

We also find that the simple estimations of effects (Table 7) are close to the net effects resulting from the Heckman regression controlling for seriousness toward work and observable characteristics. In Table 10, those who received EN services attained $3,544 more on average in gross BFW through the end of 2016 than nonparticipants (compared with $4,425 more in Table 7), and those who received SVRA services attained $10,692 less in gross BFW per beneficiary than those with no services (compared with $9,765 less in Table 7). The simple analysis overstates the effect of TTW, but not by much.

The costs shown in Table 7 are for beneficiaries completing a TWP. However, the Heckman estimate is for all recipients of DI or concurrent DI/SSI benefits regardless of TWP status. For that larger group, the TTW costs were $5,680 per EN client and $854 per SVRA client (not shown). Combining those figures with the gross BFW results in Table 10 suggests that neither EN nor SVRA services reduced SSA's costs through BFW in the years following a beneficiary's TTW participation. For EN clients, BFW through the end of 2016 recouped about two-thirds of costs and for SVRA clients, with negative net BFW, per participant costs sum to $11,546.³²

However, as noted above, this is not the full story. Our Heckman model adjusts for serious interest in work, but it does not adjust for the fact that many who seek employment services do so because they think those services are necessary for them to succeed. If this self-assessed need is reasonably accurate, our regression—which ignores that assessment—will underestimate the effects of service use. The fact that SVRA service use has a negative, large, and significant effect for beneficiaries who are serious about work and complete their TWP clearly seems to indicate that SVRA clients may have greater barriers to employment success than other TWP completers do.

Also noteworthy is that the stage 2 regression model has a low R² value, indicating that much of what drives these effects is not accounted for. This, in turn, means that even after limiting our analysis to those who have completed their TWP, significant quantities of information remain unobserved in the model. At least part of this missing information is likely due to the service needs of those who enroll in employment service programs.

Charts 2A and 2B track the earnings and employment outcomes of 2001 DI awardees to further examine potential bias related to beneficiaries' employment-service need. We shift the study cohort from 2006 beneficiaries to 2001 awardees to provide enough lead time for beneficiaries to have sorted themselves into the three types of employment service use: none, EN, and SVRA. We define the user groups by their status as of 2006, thus allowing beneficiaries 5 years from benefit start to decide whether to seek services, followed by a 10-year observation period that lets us track and compare their earnings and employment outcomes. To provide visual comparisons, Charts 2A and 2B also track earnings and employment outcomes for service nonusers who did not work until after 2006. However, our discussion focuses only on EN users, SVRA users, and service nonusers who worked as of 2006.

The charts cover 2001–2016; but because some 2001 awardees had prebenefit earnings in 2001, our first year of interest is 2002. As might be expected, beneficiaries who worked and did not seek services in the first 5 years on the benefit rolls had the highest average earnings in 2002, slightly above those of EN clients and substantially higher than those of SVRA clients (Chart 2A). For service nonusers who worked, earnings increased steadily over the observation period, and they had the highest average earnings, by a small margin, in the last few observation years. Although service users started the period with lower average earnings, EN clients reached and sustained higher earnings over time. SVRA clients started with average earnings well below the other groups and, over time, they reduced the earnings gap with service nonusers who worked, but did not equal them. By 2005, earnings for EN clients had surpassed those of service nonusers who worked, and by 2008, EN users' average earnings eclipsed the annualized SGA level. SVRA clients had closed about half the earnings gap with service nonusers who worked by 2006 and although they didn't further close the gap thereafter, their average earnings continued to climb, finally edging above SGA level by 2015.

All three of the service-use groups that Chart 2B tracks before 2007 had similarly low proportions of beneficiaries working in 2002. For service nonusers who worked, the rate was 36 percent in 2002 and declined thereafter. That downward trend likely reflects the workers who were unable to sustain employment, and as they dropped out, the culling effect likely played a significant part in the increase in earnings for service nonusers who work (Chart 2A), in that the beneficiaries who continued to work were pared down over time to only the highest earners in the group. Among EN users, only 32 percent worked in 2002 (Chart 2B), but the rate rose steadily and significantly through 2006. SVRA users had a higher employment rate in 2002 (41 percent) and the rate rose steadily, but less sharply than that of EN users, through 2006. More importantly, both the EN and SVRA users had earnings and employment-rate increases: Each year from 2003 to 2006, more enrollees worked, and they had higher earnings. Although the employment rates declined after 2006, they were, for both groups, about 10 percentage points higher in 2016 than the rate for service nonusers who had worked as of 2006.

Although the evidence is not definitive, Charts 2A and 2B support the hypothesis that beneficiaries who seek EN and (especially) SVRA services do so because they believe that they need those services, without which, such beneficiaries would be expected to have poorer employment outcomes than those who never seek services. The charts also suggest that EN services can fill clients' needs and SVRA services can at least ameliorate them.

Limitations

This analysis focuses on the costs and effects of the TTW program and does not attempt to identify or account for any costs and effects related to other provisions of the authorizing legislation. Examples of the latter include costs associated with the elimination of work activity as a basis for reviewing an individual's disabled status, expedited reinstatement of benefits after a failed work attempt, outreach and support for the TTW Work Incentives Planning and Assistance initiative, and extended access to medical benefits, along with the previously mentioned protection of TTW participants from CDRs and the exclusion of unrecovered overpayments. Some of these costs are not specific to TTW and could be modified or eliminated irrespective of the program. As such, we treated them as general SSA costs that apply to all beneficiaries who work. Similarly, we did not account for computer system development costs that were related to or prioritized by TTW implementation, other than those included in the TPM contract. Although we do not attempt to account for these general SSA costs, we acknowledge that some of them may be used disproportionately for TTW participants.

We have reliable data on costs for services associated with beneficiaries who assigned their tickets in 2006 but we do not have 2006 data on administrative costs. We therefore substituted 2017 administrative costs adjusted to 2016 dollars. This substitution assumes that annual TTW administrative costs have been flat since 2006, which is not accurate; but we think it is reasonably close after we remove the nonsteady-state costs associated with TTW startup. To the extent that our estimates are lower than the true costs, program effects will be overstated.

Although our Heckman analysis adjusted for selection bias based on work aptitude and interest, it had no adjustment for service need. The negative need-for-service bias is likely to be substantial based on three findings in the analysis. First, we found that SVRA participation had a large, negative, and statistically significant effect on BFW. This defies common sense. Although it seems possible that helping someone build skills and find work has no discernible effect, it seems unlikely that a beneficiary who receives such services is worse off than one who receives no services at all. Perhaps some SVRA clients delayed a work start without getting anything useful from the SVRA during the delay. However, given that average gross BFW for SVRA clients are fully one-third less than those for service nonusers, and the median time from ticket assignment to work start is only 6 months longer for SVRA clients than for EN clients, it is hard to believe that the difference is due only to an employment delay caused by SVRA participation.

Second, we cannot determine how much of the difference between SVRA and EN outcomes is attributable to the success of the new service models under TTW and how much is due to bias from client self-sorting, based on need and ability, into ENs or SVRAs. We know from the contractor's evaluation that EN and SVRA services differ (Schimmel and others 2013) but we also know that ENs can choose which beneficiaries they serve while SVRAs cannot. Given that EN payments follow a fixed payment structure while SVRA payments use a reimbursement model, it seems likely that the beneficiaries with the greatest need end up at SVRAs. We cannot quantify differences in service need among TTW participants, but those unmeasured differences in client needs and abilities are likely to be the primary factors driving the differences between EN and SVRA outcomes. This seems evident in the earnings profiles of beneficiaries in the years after benefits begin, as those who seek SVRA services start out with significantly lower average earnings than either EN clients or service nonusers. Although SVRA clients' earnings improve significantly over time, and they narrow the gap with service nonusers, they remain the group with the lowest average earnings across the entire study period.

The selection of beneficiaries based on service need also suggests a limitation of the Heckman model for this analysis. The model assumes we can capture bias through the first-stage regression, but this works only if a single threshold variable can be used to capture the bias entirely. That seems unlikely in this instance, where the bias moves in two directions: increased aptitude for self-supporting work makes success more likely, while the need for assistance makes it less likely.

We have not attempted to build a comprehensive model of the factors affecting a beneficiary's likelihood of returning to work or leaving the disability benefit rolls. Instead, our analysis uses the available administrative data from the DAF on variables that are thought to be potential correlates with this behavior. Yet SSA data do not track several characteristics that could illuminate the analysis. These missing beneficiary characteristics include educational attainment, job characteristics prior to benefit start (such as industry, occupation, and job tenure), the stage at which benefits were awarded (initial determination, reconsideration, appeal), residual functional capacity, proximity to services, and access to transportation. Because SSA introduced an automated application processing system called the Electronic Disability (eDib) System, some of this information is available for beneficiaries applying from 2008 forward, but there are concerns about the completeness of the eDib data. In attempting to use eDib data for this analysis we encountered differences between eDib and DAF data, the causes of which we could not determine. Although we did not use eDib information in our analysis, its breadth remains an attractive extension for future study.

Another limitation is that our Heckman model excludes SSI recipients who do not receive concurrent DI benefits. This is because we use TWP completion as our threshold variable, and the TWP does not apply to SSI-only recipients. An alternative would be to use the STW-status variable, which applies to both SSI and DI and is conceptually similar to TWP completion. As noted earlier, however, we chose TWP completion because it is a program variable while the STW-status variable is constructed for the DAF. In one sense, STW status is a better choice in the Heckman setup: If one's benefits are suspended for work, BFW must be nonzero, in the same way that if work occurs, earnings are nonzero under the Heckman theoretical framework (Killingsworth 1983). However, STW status is a higher threshold than TWP completion for DI beneficiaries and a lower one for SSI. For DI, STW status implies TWP completion and at least 1 month of SGA after the end of the TWP. For SSI, STW status does not necessarily indicate a sustained work effort because it can occur for any month in which earnings are approximately double the SSI payment amount.³³ Further analysis of the use of STW status as the threshold variable is worth examining in future research.

We made several assumptions in estimating TTW administrative costs. As noted in Appendix A, those costs are spread out over several years, so we approximate a single year's costs associated with the enrollee cohort. Although costs have changed over time, we use 2017 as a representative year for administrative costs because it is the best available estimate and represents a year in which TTW was operationally mature. It is likely that actual costs for 2006 were higher, but we think that many of those costs are attributable to TTW's startup status. We do not think the range of possible steady-state cost values for 2006 would materially affect our basic findings.³⁴

Summary

The TTW program was implemented with high hopes that it would significantly broaden access to employment services, efficiently fund those services, reduce dependency on cash benefits, and increase the proportion of beneficiaries who are able to exit the disability benefit rolls and live self-sufficiently, thereby generating savings to the Social Security trust funds (for DI) and the general fund of the treasury (for SSI).

Although some observers regard TTW as a failure, that conclusion is based narrowly on findings that TTW did not substantially increase disability program exits. Broader evidence shows that TTW accomplished other goals. It expanded the service options available to SSA disability program beneficiaries and increased enrollment in employment services overall. There is also evidence that TTW participation, for both EN and SVRA clients, has yielded positive earnings results, improving lives and reducing dependency on cash benefits. In addition, long-term net-outcomes analysis shows substantial BFW for TTW participants.

Finding that the TTW's effects were not as substantial as its designers had hoped does not mean that new EN services added under the program had failed. Because TTW superseded a preexisting SSA program to provide VR services, Mathematica's evaluation compared TTW's effects with those of that prior program—it did not assess the TTW effects relative to a no-services counterfactual. The evaluation found that TTW expanded service provision and participation without the decline in disability program exit rates that might have been expected as more beneficiaries (perhaps with poorer work prospects) enrolled for services. This means that services provided by ENs, combined with those offered by the preexisting SVRAs, were no less effective than those that were previously available to fewer potential clients, presumably with better median employment prospects, from SVRAs alone. Also of note: Although Mathematica did not find that TTW had a large effect on the disability roll exit rate, a smaller effect, not measurable with the methods available, may have occurred. Specifically, TTW increased STW status by less than 30 percent and the number of months in STW status by less than 9 percent—but the evaluation could not determine, with the available data, how closely the actual figures approached those ceilings.

Further, separate models assessing EN and SVRA services found better outcomes for ENs. TTW's introduction of ENs thus expanded service options and availability and reshuffled the service landscape while achieving the same benefit exit rate as the prior SVRA-only system. With increased service availability, beneficiaries were able to choose the providers that best met their needs.

The Mathematica evaluation examined TTW's marginal effect, but not the overall effect of the employment services. We therefore examined the net outcomes—program-associated costs and savings—for all beneficiaries. Analyzing program outcomes, without adjustments to account for selection biases, we found that TTW participants using either EN or SVRA services had much better employment results and, on average, accrued much higher BFW than nonparticipants. Then, adjusting for bias related to work interest and capability among employment service users, we limited the analysis to DI beneficiaries who had completed a TWP. Among this work-motivated group, we found that EN and SVRA clients accrued high average gross BFW, but average net BFW, after adjusting for average service costs, were lower for TTW service users than for TWP completers who received no services.

We next examined the effect of TTW services using more complex regression analyses to account for differences between service users and nonusers. Our Heckman model found that TTW employment service use, relative to nonuse of any services, increased gross BFW per beneficiary by $3,544 for EN users and reduced per-beneficiary gross BFW by $10,692 for SVRA users over the 2006–2016 study period. We also found that the simple adjusted outcomes, when restricted to DI beneficiaries who were TWP completers, were close to the more complex regression results. This is a useful finding: Because the simple model is intuitive and easy to explain, and provides similar results, it serves as a proxy for the more accurate and complex econometric models and can illustrate the results to a broader audience.

Conclusions

Although this analysis clarifies TTW's effects, significant questions remain. We show that simple outcome measures can be misleading because they do not account for the biases inherent in who does and does not seek services. Restricting the analysis to DI beneficiaries who completed a TWP and applying the Heckman model both seem to work reasonably well in adjusting for serious interest in work. However, neither of these approaches account for bias related to the differences in service needs between service users and nonusers, so the actual effects are likely greater.

By adjusting for interest in work but not for self-assessed need of services, our estimates appear to represent a lower bound for TTW effects. Unfortunately, this finding does not address whether savings from benefit reductions cover TTW costs: accounting for the beneficiary's need for service would likely increase the estimated benefits of TTW, and would likely erase the apparent negative outcomes of SVRA services, but perhaps not enough to fully offset EN and SVRA service costs. The absence of clear findings after nearly 2 decades of observation, evaluation, and analysis underscores the difficulty in estimating definitive effects without a well-designed randomized control trial. The quasiexperimental methods used thus far simply do not seem capable of capturing the complex biases inherent in the return-to-work decisions facing beneficiaries with disabilities.

Still, our analysis finds that EN services, for which service need is less likely to be a limiting factor, significantly boost client earnings, STW status, and BFW for DI beneficiaries. This, along with Mathematica's finding that the combined employment outcomes for EN and SVRA services under TTW were similar to the pre-TTW SVRA-only outcomes, suggests that SVRAs are likely more successful than the Heckman analysis implies: DI beneficiaries who use EN services would have been as successful under the old system as they are under TTW. If this were not the case, then overall employment success would have improved when SSA switched from SVRA services only to the hybrid SVRA/EN model under TTW. It may be that pre-TTW SVRAs succeeded only with clients who have self-sorted into using ENs under TTW, but that seems unlikely. The likelier scenario is that SVRAs have a range of successes across the continuum of beneficiaries they serve and that SVRA clients with lower service needs have positive outcomes like those attained by EN clients. Other SVRA clients with greater service needs probably have poorer outcomes, but it seems unlikely that those outcomes are worse than they would have been with no services (as the Heckman model implies) when the SVRAs seem capable of achieving significant positive results for other clients. Further, the Heckman model's large negative finding for SVRA users suggests that the service-need bias that remains in the model could be quite substantial and that removing the bias could lead to significantly larger estimated effects for both ENs and SVRAs.

We also find that it seems unlikely for TTW or any alternative return-to-work program to significantly improve on the TTW's results. TTW offered many innovations to address perceived shortfalls of previous programs:

• It enables payment for outcomes rather than reimbursing only for costs. Reimbursements to SVRAs are predicated on the client reaching SGA in any 9 out of 12 consecutive months, while EN payments are tied more directly to employment outcomes and long-term success.³⁵
• It removes several barriers to work that had been identified as keeping beneficiaries from leaving the rolls. TTW significantly extended health insurance coverage for former beneficiaries after leaving the rolls for work, eased a return to benefits if the beneficiary's work attempt failed, and offered protections from CDRs while enrollees participated in the program.
• It expanded service choice by offering services from ENs as an alternative to SVRAs. Although many ENs were VR vendors offering services as ENs much like those they offered under contract with SVRAs, some, such as consumer-directed service ENs and employer ENs, offered entirely new service models. TTW also likely expanded the availability of services, as ENs were able to absorb excess demand from SVRAs that reached capacity and went into order of selection status.

Combined, these TTW innovations affected employment service participation, but they have not led to large increases in disability program exits. It is hard to imagine how further changes in incentives, or the removal of additional barriers, could both remain within the current benefit rules and result in significantly more program exits than we have seen under TTW.

The 2008 change in TTW regulations is also instructive. As discussed in Livermore, Hoffman, and Bardos (2012), the new regulations increased TTW participation and seem to have drawn beneficiaries with greater work challenges on average than those who participated before 2008. Schimmel and others (2013) found that these newly induced participants were less successful in finding work than prior TTW participants had been, which led to an overall decline in STW status:

The recession almost certainly explains some and possibly most of the adjusted decline in the percentage [of participants] with at least one [STW-status] month [−39 percent], but other explanations are possible. In particular, the regulatory changes increased incentives to serve beneficiaries with a lower likelihood of sustaining high levels of earnings.

The authors found similar results for BFW.

Together, these findings suggest that TTW may have reached the maximum number of beneficiaries who are able and wish to work and can be served by employment service providers. This does not mean that employment programs for disability program beneficiaries cannot be improved, but that doing so is likely to have only marginal effects, and it is hard to imagine that a large new program addressing these same incentives and barriers would dramatically move the needle in the way that the TTW legislation envisioned. Instead, significant changes in beneficiary work activity would seem to require structural changes to the disability program rules that would change beneficiary incentives for work.

Still, this analysis shows that TTW has had some significant positive outcomes. It has improved choice and induced more beneficiaries to try to improve their lives with work. TTW employment services have produced definitive improvements in earnings, STW status, and BFW, at least for DI beneficiaries who are served by ENs; it likely had similar effects for SVRA clients as well. At a minimum, these benefit reductions offset some employment service costs, although further research is needed to determine the extent of the net effects. These are all important and valid achievements that SSA should build on. Although TTW has not brought the sea change its designers hoped for, there is substantial evidence that it has, and will continue to have, a significant positive effect on the lives of the people it serves.

Appendix A: Alternative Average Cost Calculations for EN and SVRA Services

Table 6 presents estimated costs per EN and SVRA enrollee, but comparing the average costs of providing EN and SVRA services under TTW is not straightforward for many reasons, of which these two are among the more prominent:

1. Because SVRAs are also funded by block grants, SSA provider payments cover only part of their costs. In 2017, block grants funded $3.97 billion of SVRA payments, with $3.12 billion (78.7 percent) representing the non-SSA federal share and $0.85 billion (21.3 percent) representing the state share. According to our review of Case Service Reports from the Department of Education's Rehabilitation Services Administration, beneficiaries of SSA disability programs make up at least 30 percent of SVRA enrollees. Therefore, at least some of these federal funds are spent on SSA beneficiaries, even though SSA does not pay them.
2. Although ENs are allowed to choose which beneficiaries they will serve, SVRAs are required to serve anyone they are able to help. As a consequence, many SVRA clients are not expected to return to work (VR services may target functional improvement rather than employment as a goal), and these clients are not considered to be TTW participants who are likely to generate SSA reimbursement for the provider. Although SSA counts these clients as TTW participants while services are ongoing, SSA funds are highly unlikely to be used for them.

For these and other reasons, SVRAs and ENs use different approaches to enrolling and serving beneficiaries, which likely drives some of the differences between EN and SVRA outcomes seen in Tables 4 and 5. For example, Table 5 indicates that SVRAs receive provider payments for only about 5 percent of the beneficiaries they enroll while ENs receive payments for 26 percent of their clients. This is significant because the different approaches SVRAs and ENs use are reflected in their costs.

Some TTW administrative costs involve services that are available to all beneficiaries. These include outreach and support (for example, the telephone help line provided under the TPM contract) and training for field and regional office staff about the TTW program. Some costs, such as those for managing ticket assignments and unassignments, relate only to TTW participants. Other costs apply only to currently active EN and SVRA clients who are actively participating in TTW; examples include benefit planning queries and other questions handled at field and regional offices. Lastly, the cost of administering TTW provider payments to ENs and SVRAs relates only to TTW participants who generate payments. Because of operational differences, ENs and SVRAs have different proportions of the TTW subpopulations to which these costs apply, so simply averaging the total cost among a single subpopulation, such as all enrollees, might provide misleading comparative results. We therefore seek estimates that account for these differences, to let us compare the cost for beneficiaries who received services from one provider type with the hypothetical cost they would generate had they used the other provider type and attained similar results.

We cannot sort these subpopulations into mutually exclusive categories; but summary statistics compiled from Tables 3, 4, and 5 on total enrollees, enrollees who reach STW status, and enrollees who generate provider payments can capture important differences in the SVRA and EN clientele and associated SSA costs (Table A-1). In Table A-2, we use the subpopulations and cost estimates shown in Table A-1 as the basis for illustrating five alternative methods of calculating average costs for EN and SVRA services. Each calculation method includes a caveat.

Table A-2 also includes a column that addresses the fact that SVRAs receive some non-SSA funds to serve SSA program beneficiaries. With those beneficiaries accounting for at least 30 percent of SVRA clients, one might presume that SSA payments represent a similar share of total SVRA funding. Because we cannot determine the proportion of block grant funds that SVRAs use to serve SSA program beneficiaries, we instead identify the percentage of SVRA block grant funding that would equalize average EN and SVRA costs under each of the five cost calculations. For example, the total EN cost per enrollee in Table A-2, calculation 1, exceeds the SVRA cost by $4,395. If, on average, SVRAs spent this difference of $4,395 from their block grant funds on each of their 135,363 enrollees who were SSA program beneficiaries in 2016, that cost would have totaled about $595 million, which is equal to about 15 percent of the $3.97 billion in block grants they received for that year.³⁶ Under average-cost calculation 1, if SVRAs spend more than 15 percent of their non-SSA resources on SSA program beneficiaries, then SVRA services are more expensive on average than EN services. If the figure is less than 15 percent, then SVRA services are less expensive than EN services. Among the five calculation methods, the greater the difference between the EN and SVRA average costs, the higher the percentage of block grant spending that would be needed to equalize these costs. For calculation 5, average costs are lower for ENs than for SVRAs, so the percentage becomes negative and EN services are less expensive regardless of what proportion of the non-SSA funds are spent on SSA program beneficiaries. In no  case, however, does the block grant percentage approach 30 percent, so unless SVRAs consistently spend considerably less than 30 percent of their non-SSA resources on the 30 percent of clients who are SSA program beneficiaries, SVRA services are more expensive on average than EN services when all SSA payments, federal block grants, and state block grant shares are included.

We explored the five alternative calculations to find an allocation method that reasonably adjusts for service intensity, such that providers whose clients require more intensive services get a higher share of the costs. The caveats in Table A-2 address potential strengths or limitations of each allocation method. In particular, the variability in the calculations shows that averaging all costs by a single group likely leads to imprecise estimates because administrative costs and provider payments are focused on different subsets of the beneficiary population. Averages per enrollee overall (calculation 1) likely underestimate both administrative costs and provider payments while averages per STW exit (calculation 2) likely overestimate administrative costs and underestimates payments.

Calculations 4 and 5 provide the best comparative average cost estimates for ENs and SVRAs by using different populations for averaging administrative and payment costs. Only in calculation 5 are the average total costs lower for ENs than for SVRAs. Still, neither of these estimates is entirely accurate and we conclude the true costs likely lie somewhere between calculations 4 and 5.

Regardless of calculation method for average costs, total administrative costs are greater for ENs than for SVRAs. Table A-3 looks at the types of costs that account for the differences. Administrative costs for headquarters operations and provider payments are similar for the two provider types. Regional and field office operational costs are higher for ENs because those services entail more TTW training for SSA staff and more inquiries from providers and participants than SVRA services do. TPM contract costs also diverge widely, mostly because the larger of the two components of TPM costs—for provider outreach/recruitment and timely completion of client progress reviews—is specific to EN services. The other TPM costs, for operating the TTW call center and managing ticket assignments, are higher for SVRA services because they are allotted by the relative size of the EN and SVRA participant populations.

Some of these costs are not strictly administrative in that they are not necessary to run the TTW program. For instance, responding to participant queries and completing timely progress reviews can respectively be viewed as broader beneficiary services and program integrity workloads that are not central to TTW operations. To estimate how “administrative” costs that are not strictly administrative in nature could affect the difference between average EN and SVRA costs, in Table A-4 we reconsider calculations 4 and 5 from Table A-2, identifying the EN costs that, if removed, would essentially eliminate the differences. Calculation 6 in Table A-4 shows that if $7 million in EN costs are not essential to TTW operations³⁷ and we remove those costs, EN and SVRA costs equalize, even though this method adopts calculation 4's denominator (enrollees who reach STW status), which likely overstates EN average costs. This is not to say there are $7 million in EN costs that are not essential to TTW operations. TTW mandated many costs, and if $7 million of those costs were unnecessary, EN and SVRA costs would be the same.

Using the methodology of calculation 5 in Table A-2 and a rough estimate of the likely distribution of administrative costs among service providers with high and low TTW activity, calculation 7 in Table A-4 shows a “best guess” of average costs and the actual provider payment amounts. Even in this scenario, costs are lower for ENs than for SVRAs (ignoring block grant resources), although the differences are smaller than those for calculation 5 in Table A-2.

This appendix shows that when we analyze TTW costs in a manner that addresses the inherent differences in how each provider type operates, it is reasonable to conclude that average EN and SVRA costs are similar. It also shows how the simple average can be misleading. Table A-5 shows the simple average costs for 2016 of $835 for SVRAs and $5,204 for ENs. If all EN clients had been served by SVRAs, total cost, as estimated using simple averages, would be $116.6 million. This suggests that moving clients from ENs to SVRAs would significantly lower costs. However, these EN clients are more successful than the average SVRA client so the SVRA cost for serving these clients would be higher than the simple SVRA average cost. If we use only SVRA costs for both EN and SVRA enrollees but compute average administrative costs using all enrollees and provider payments using only those enrollees generating payments (as in calculation 5), the estimated total cost is $129.3 million. This higher estimate adjusts for the higher success rate for EN clients and the higher costs associated with that success. This estimate is slightly lower than the combined EN and SVRA costs ($135.5 million) because it assumes that none of the $3.97 billion in non-SSA funds were used to serve these SVRA clients.

Because EN and SVRA costs are so similar when properly calculated, removing either the EN or SVRA side of TTW services would not materially change the overall cost of providing employment services to SSA disability program beneficiaries. The costs would simply shift to the other provider type and the net effect would be minimal. In other words, if SSA were to stop supporting EN services, SVRA costs would go up by an amount similar to the resulting reduction in EN costs, as the ENs' former clients moved to the SVRAs. The only way for costs to decline in this case would be if the change resulted in an overall decrease in service provision.