Participation in Programs Designed to Improve Employment Outcomes for Persons with Psychiatric Disabilities: Evidence from the New York WORKS Demonstration Project

Summary

The New York WORKS demonstration project was designed to improve employment outcomes for persons with psychiatric disabilities receiving Supplemental Security Income (SSI) disability payments. This article shows how the individual characteristics of participants were related to outcomes at each stage of the multistage recruitment process used in the project and how those characteristics contributed to enrollment. The findings are important to program administrators who are interested in ensuring that SSI recipients receive equal access to employment-related services and who want to improve recruitment strategies for future demonstration projects.

The New York WORKS recruitment process used administrative records from the Social Security Administration (SSA) to identify the population of over 68,000 SSI recipients with a diagnosis of a psychiatric disability in Erie County and New York City. Staff involved in the project documented the results of each stage of the recruitment process. The New York WORKS project included four stages: (1) the provision of information (sending a letter and information packet); (2) demonstrated awareness of the project (response to a letter containing an overview of the project); (3) expression of interest (indication of interest in the project, using a postmarked form returned to New York WORKS project staff); and (4) participation (actual enrollment in the program). The project staff members were also able to identify data from administrative records that described the characteristics of the population, including age, sex, type of psychiatric diagnosis, the number of months that the person collected benefits before the recruitment process, employment experience before the recruitment process, and annual earnings in the year before the recruitment process.

The data on outcomes at each stage of the recruitment process and the characteristics of SSI recipients were analyzed using an empirical method recently suggested by Heckman and Smith. The analysis identified the relationship between the characteristics of SSI recipients and the outcomes at each stage of the recruitment process and demonstrated how those characteristics contributed to the overall likelihood of enrollment.

Demographic characteristics, information about diagnosis, and characteristics related to work history had different effects on outcomes at different stages of the recruitment process. For example, younger SSI recipients were less likely to reply to the information letter but more likely to express an interest in the project and more likely to enroll, given that interest. This result suggests that there may be an information barrier for the younger group of SSI recipients during the early recruitment stages.

There were also interesting differences by psychiatric diagnosis and by recent employment experience at each stage of the process. Most notably, persons with anxiety disorders were less likely to express an interest in the project and less likely to enroll in the project if they expressed an interest. This finding suggests that project administrators may need to examine more effective methods to accommodate persons with an anxiety disorder at the enrollment stage of the recruitment process. Persons with relatively low earnings in the year before the project were more likely to respond, to express interest in the project, and to enroll than were those with no earnings and those with relatively high earnings. This finding suggests that the recruitment strategy used by the New York WORKS project is more effective at enrolling a subset of the population that has some demonstrated work capacity and that, of those with a demonstrated work capacity, New York WORKS is enrolling those who are likely to have the most to gain from the project. This finding has implications for the development of sample designs for other demonstration projects, in particular the SSA-proposed mental health treatment study.

A number of strategies may lead to improvements in future evaluations of participation in SSA projects and programs. For example, the New York WORKS data did not contain information for all SSI recipients on race, education, or literacy. These factors have been shown to be important in the Job Training Partnership Act literature and are likely to play a role in participation. The inclusion of these data in future evaluations may provide important information on participation in SSA projects. Another useful strategy for future evaluations includes the identification of random samples of those who choose not to enroll at different stages in the process and the collection of information on the reason for the decision. These additional data may help project administrators to gain a clearer description of the reasons for differences in outcome that occur at each stage of the process, assess the overall performance of the recruitment process, and improve the recruitment processes used in future projects.

Introduction

The Social Security Administration's (SSA's) disability programs—the Social Security Disability Insurance (DI) program and the means-tested Supplemental Security Income (SSI) program—provide income and medical care to people who are unable to work as a result of a health condition that is expected to last at least 12 months or result in death. Although these programs play an important role in providing income security to persons with a disability, they have been criticized for not doing enough to assist program participants in their efforts to return to work. This criticism is clearly documented in a 1996 report produced by the National Academy of Social Insurance, which found that in the early 1990s the rate at which people left the SSI and DI programs because of a return to work or a medical recovery was "at an all-time low" and that the low rate was partly due to "the level of investment in efforts to assist beneficiaries return to work" (Mashaw and Reno 1996, 19). SSA responded to these findings in the late 1990s by initiating several new efforts designed to increase employment rates for SSI and DI beneficiaries. One of these efforts was a set of new research demonstration projects, called the State Partnership Initiative, that enabled states to test a variety of innovative methods designed to assist SSI and DI beneficiaries with their efforts to return to work.

New York was one of 12 SSA-sponsored states to take part in the State Partnership Initiative. The New York State project, called New York WORKS, focused exclusively on SSI recipients who were awarded benefits on the basis of a primary diagnosis of a psychiatric disability. The project recognized that persons with a psychiatric illness make up the largest diagnostic category of working-age persons who receive SSI or DI (McAlpine and Warner 2001). Moreover, among individuals with a psychiatric illness, those who receive SSI or DI benefits tend to be the most severely disabled (Estroff and others 1997). In spite of this, many of these individuals are able to work (McAlpine and Warner 2001), and indeed most recognize the potential benefits of employment and want to work (Baron and others 1996).

The overall objective of the New York WORKS project was to examine the impact of three interventions on the employment outcomes of SSI recipients with a primary diagnosis of a psychiatric disability. Those three interventions are benefits advisement (obtaining expert counseling on the impact of employment on all social service benefits), employment coordination (coordinating existing resources available to participants from various federal, state, and local employment programs), and improvements to the SSA work incentive provisions.

This article examines the relationship between characteristics of SSI recipients, outcomes at each stage of the recruitment process, and the likelihood of enrollment in the project. The New York WORKS project staff worked with the Social Security Administration to identify the population of more than 68,000 noninstitutionalized SSI recipients with a primary diagnosis of a psychiatric impairment in Erie County and the five counties that constitute New York City (Bronx County, Kings County, New York County, Queens County, and Richmond County). They documented the results of each of the four stages of the recruitment process:

• provision of information—the receipt of a letter from the New York State Commissioner of Labor informing the identified SSI recipients about the project and requesting a response on a postmarked form regarding their level of interest in the project,
• demonstration of awareness—a response to this letter,
• expression of interest—indication of interest in the project using a form returned to New York WORKS project staff, and
• participation—actual enrollment in the program of those who indicated they are interested in the program.

The project staff members were also able to extract data from administrative records that describe the characteristics of the population. The characteristics included age, sex, type of psychiatric diagnosis, number of months that the person collected benefits before the recruitment process, employment experience before the recruitment process, and annual earnings in the year before the recruitment process.

The data on outcomes at each stage and the data on characteristics of SSI recipients were analyzed using an empirical method recently suggested by Heckman and Smith (2004). The analysis identified the relationship between characteristics of SSI recipients and recruitment outcomes at each stage and showed how the characteristics contributed to the overall likelihood of enrollment.

The analysis is important for several reasons. First, it identifies the point in the recruitment process at which certain segments of the SSI recipient population are losing the opportunity to enroll in the project.¹ This information is important to policymakers interested in equal access to the employment services and work incentive provisions offered by the New York WORKS project.² The results show that younger SSI recipients were more likely to lose the opportunity to participate because they did not respond to the information letter. This finding suggests that SSA may need to explore other methods of delivering information to younger SSI recipients. The results also show that SSI recipients with anxiety disorders who expressed an interest in the project were more likely to lose the opportunity to participate at the enrollment stage than were SSI recipients with other psychiatric disorders. This finding suggests that program administrators may need to explore other methods to accommodate persons with anxiety disorders at the later stages of the recruitment process.

Second, the analysis describes the characteristics of those interested in the New York WORKS services. This information may be used in designing cost-effective sampling strategies for future demonstration projects such as SSA's proposed mental health treatment study.³ If certain segments of the SSI recipient population are more interested in the employment services and SSA work incentive provisions, then a stratified sample design may allow SSA to sample SSI recipients for similar projects in a cost-effective manner.

Third, the analysis may allow evaluators to differentiate between the impact of the project for those who successfully navigate the recruitment process and the impact that may occur for SSI recipients if the project were to become part of the SSI program. SSI recipients are likely to obtain information and react to information differently in an ongoing program than in a research demonstration project. Information on how the recruitment strategy influences the decision to participate in the project may help evaluators assess the degree to which the results may change in an ongoing program.

The New York WORKS Project

In September 2000, the New York WORKS project began the 30-month recruitment process that randomly selected SSI recipients each month who met specific criteria (that is, they were 21 years old, were not institutionalized or in a nursing home, and so on) and sent them a letter from the New York State Commissioner of Labor with information about the availability of a research demonstration project called New York WORKS. The letter included a brochure that described the project and a postmarked form to indicate the degree of interest in the project.⁴ The postmarked form provided recipients with three options: to indicate that they had no interest in the project, to express interest in the project, or to postpone interest in the project for 3 months.

With the exception of a small number of referrals,⁵ all persons who were on the list that SSA gave to New York WORKS were randomly assigned to one of three experimental groups: (1) a full-service treatment group, (2) an enhanced-service treatment group, and (3) a control group.⁶ Persons who agreed to participate and who were initially assigned to the control group received basic information about available vocational rehabilitation services, SSA work incentives, and, when the Ticket to Work program rolled out in New York State, information about the Ticket program. Persons assigned to the two treatment groups who agreed to participate received services and policy waivers that were designed to support their efforts to return to work or maintain employment. Enrollment in New York WORKS took longer than anticipated, lasting from September 2000 to March 2003; the project and its services ceased in March 2004. Components of the interventions are listed in Table 1 and are briefly summarized below.

Maintaining Access to Medicaid—1619 Work Incentive Session

The potential loss of medical coverage has been cited as a significant barrier to employment among SSI and DI beneficiaries (Mashaw and Reno 1996, 26). Persons with chronic or debilitating illnesses have a great need for health care services, and the high costs of health care combined with demand for these services make access to medical coverage critical. The SSI 1619b provision extends Medicaid coverage to SSI recipients who lose SSI cash benefits because of earned income and who meet other 1619b criteria. Although the 1619b provision allows SSI recipients to maintain their access to Medicaid while they work, many recipients are either unaware of or do not understand the provision and its importance in efforts to enter or reenter the labor force.

Many recipients are also unaware of the 1619a provisions. These provisions allow SSI recipients to work at earnings above the level considered to be substantial gainful activity and to continue receiving SSI benefits. Participants in the New York WORKS project attended a mandatory intensive information session about the 1619a and 1619b provisions immediately following enrollment.⁷

Intensive Benefits Advisement

Many SSI recipients do not understand the complex public benefit programs well enough to determine the impact of employment on all of their benefits, are unaware of work-related provisions or exemptions that are available, and may not know how to resolve outstanding issues with their SSA benefits such as overpayments.⁸ New York WORKS participants received intensive benefits advisement services from a specially trained benefits advisor who addressed these issues.⁹

The role of the benefits advisor in the project was that of a trained expert on the benefits system. A benefits advisor examined each recipient's case individually and developed a plan that allowed the recipient to determine what would happen to benefits once he or she began paid employment. The advisor also assisted recipients in accessing other benefits and waivers that were available to them. In the New York WORKS project, benefits advisors worked closely with the Social Security Administration's PASS (Plan for Achieving Self-Support) Cadre to expedite the resolution of SSA-related problems, such as overpayments, and to administer the special work incentive provisions tested in the demonstration.¹⁰

Employment Coordination

The full-service treatment model included employment coordination, which was implemented by an employment coordinator who performed three major tasks. The first task was to help the participant obtain access to existing job training services, vocational rehabilitation services, and employment services.

The second task was to help the participant lead a team that worked together to achieve the participant's employment goal. At a minimum, the team consisted of the participant, the employment coordinator, and the benefits advisor. The participant was allowed to choose additional members for the team, which could include family members, therapists, case managers, and friends. If the participant had a vocational rehabilitation counselor, that counselor also became part of the team.

The third task was to maintain contact with the participant, help put the necessary supports in place to obtain and maintain employment, and, if the person was employed, continually assess the individual's relationship with employers and others in the workplace. If necessary, the employment coordinator helped the participant adapt to the demands of colleagues and coworkers and the demands of the job itself. The employment coordinator was the first point of contact when a participant encountered problems with the employer.

Improvements to SSI Work Incentive Provisions

New York WORKS participants were offered four waivers for specific SSI rules, resulting in significant financial benefits to employment. Access to the waivers was a coordinated effort between the New York WORKS benefits advisor and the SSA PASS Cadre.

One-for-Four Earned Income Exclusion. Under existing program rules, SSI recipients who work have their monthly benefits reduced by $1 for every $2 of their countable earnings. Countable earnings are the person's gross earnings minus an earnings disregard amount determined by SSA.¹¹ For recipients who explore the possibility of employment or attempt a return to work, the loss in benefit income that results from work reduces the financial value of employment and may discourage SSI recipients from seeking employment. To reduce this potential financial disincentive to work, participants in the project's two treatment groups were permitted to keep a larger share of their benefits as they increased their earnings; they were given a waiver that reduced the offset to $1 for every $4 of their countable earnings. Thus, rather than losing $0.50 of benefits for each dollar earned, participants would lose only $0.25 of benefits for each dollar earned.

Independence Account. New York WORKS participants were allowed to establish an independence account in addition to the current $2,000 resource limit. With that account, participants could have a bank account for saved wages. The saved wages would not count as resources that affect their SSI benefits. The money in the account could not be commingled with other money and was limited to deposits of up to 50 percent of gross earnings, not to exceed $8,000 per year. This work incentive provision allowed for a 24-month spend-down period following the end of the project. The account was intended to allow participants who were employed to attain some measure of financial independence and to put aside funds for large expenditures or emergencies.

Suspension of Continuing Disability Reviews. SSA periodically assesses whether an SSI recipient's primary disability has improved over time. This assessment is referred to as a continuing disability review. If SSA finds that the recipient's primary disability has improved to the point where he or she no longer meets the SSA definition of disability, then the recipient may be removed from the program. Recipients may fear that employment would be used during a continuing disability review as evidence of medical improvement and that employment therefore may result in the loss of SSI benefits. Thus, these reviews are generally viewed as a disincentive to employment.

To reduce this disincentive, New York WORKS participants were granted a suspension of continuing disability reviews while enrolled in the project. This suspension did not apply to participants with disabilities that were determined by SSA to be the most likely to improve, referred to as "medical improvement expected" cases.¹²

Unearned Income Due to Work Activity Disregard. Unearned income that resulted from employment-related activities was treated in a similar fashion to the $1-for-$4 earned income provision described above. Unearned income that resulted from employment-related activities included workers' compensation benefits, unemployment insurance benefits, and New York State disability benefits. SSI benefits were reduced by $1 for every $4 of countable unearned income from these sources rather than the $1-for-$1 reduction under the existing rules.

Vocational Rehabilitation Services

Expedited entry and presumed eligibility for New York State vocational rehabilitation services were offered to participants in the two treatment groups. Participants were eligible for expedited entry into the state's vocational rehabilitation system and were assigned to a dedicated New York WORKS vocational rehabilitation counselor.

Other Waivers and Incentives

Other incentives offered to the treatment groups included transportation assistance and a housing waiver that put a freeze on rent contributions when employment began. These other waivers and incentives were designed to provide SSI recipients with additional reasons to pursue employment or to further benefit recipients who were already employed.

Participation in Projects Designed to Improve Employment Outcomes

Several studies have documented the different processes used to recruit and enroll participants in demonstration projects. This study of the New York WORKS recruitment process builds on the findings from past Social Security demonstration projects, as well as on the innovative strategies used in the evaluation of the Job Training Partnership Act (JTPA) projects.

SSA's Demonstration Projects

The Social Security Administration conducted a number of demonstration projects designed to improve employment outcomes for beneficiaries of SSI, DI, or both programs. However, aside from the evaluation of participation in SSA's Project NetWork demonstration project (Rupp, Wood, and Bell 1996), there is relatively little information on recruitment and participation in SSA demonstrations or work incentive programs. In some cases, evaluations of these projects have focused on outcomes rather than on the recruitment and participation process. In other cases, the results have not been published or have not been analyzed. Table 2 summarizes information from a variety of sources and lists the recruitment and enrollment outcomes from SSA projects that were intended to promote employment among participants. Developments in the evaluation of recruitment and enrollment in other social programs, most notably the Job Training Partnership Act, are also described in this section.

Transitional Employment Training Demonstration. One of SSA's first demonstration projects designed to promote employment was the Transitional Employment Training Demonstration (Thornton and Decker 1990). The project was conducted between June 1985 and June 1987 and was designed to provide transitional employment services to SSI recipients between the ages of 18 and 40 with mental retardation. Recruitment into the project included a mailing to 13,800 SSI recipients who met the age and diagnosis criteria, follow-up letters, telephone calls, and outreach to service providers in the communities selected for the demonstration. Of the more than 13,800 invitations, 745 persons ultimately agreed to participate, for a take-up rate of about 5 percent. The evaluation of the demonstration focused on employment outcomes and did not include a rigorous analysis of participation.

Research Demonstration Projects. Another set of SSA demonstration projects, carried out in the late 1980s, were the Research Demonstration Projects. These projects consisted of a number of small grants distributed to a variety of organizations. SSA never published an overall evaluation of these projects, and little information on the projects is publicly available (see Department of Health and Human Services 1994).

One of the few individual Research Demonstration Projects that produced a well-documented evaluation was conducted by S.L. Start and Associates (Kuhta 1990). They conducted two projects that focused on return-to-work interventions. One, the Rapid Intervention Employment Project, targeted DI applicants. The other, Expedited Referral and Intervention Validation, targeted DI applicants as well as beneficiaries scheduled for a continuing disability review.

Rapid Intervention Employment Project. In this project, 42 applicants were selected who were likely to be awarded benefits and whose impairments were determined not to be too severe. Recruitment of the applicants was initiated by S.L. Start and Associates by mail, with at least two follow-up phone calls. Of the 42 applicants, 6 agreed to participate, for a take-up rate of 14 percent. The three major reasons given by those who chose not to participate were (1) they believed that their disability was too severe, (2) they did not want to work, and (3) they reported that their condition was medically unstable.

Expedited Referral and Intervention Validation. The recruitment process for this project was initiated at the SSA district office level during the DI application process. As part of the recruitment process, 434 participants were identified and 95 participated in the intervention, for a take-up rate of about 22 percent. The researchers attributed the increase in the take-up rate in this project compared with that in the Rapid Intervention Employment Project to the direct involvement of the SSA district office claims representatives in the recruitment process. The three most frequent reasons for not wanting to participate were (1) the disability would not allow them to work or their attending physician had not released them for work, (2) the participant was advised not to work by his or her attorney, and (3) fear that participating in a return-to-work project would jeopardize their DI benefits.

Project NetWork. In 1991, SSA began a new, large-scale demonstration project called Project NetWork. This project used an experimental design to test different methods that may be used to provide rehabilitation and employment services to DI beneficiaries and SSI disability applicants and recipients (Rupp, Bell, and McManus 1994). Project NetWork used four different variations of a case management approach to administer services designed to improve employment outcomes.

Project NetWork differed from SSA's prior demonstrations in that it included a rigorous evaluation of the recruitment process (Rupp, Wood, and Bell 1996). The evaluation went beyond identifying that only 4.5 percent of the 145,404 who were eligible for the project chose to participate (Burstein, Roberts, and Wood 1999). It examined the participation rates among several key subgroups and identified three mutually exclusive groups of SSA disability beneficiaries on the basis of the likelihood that they would participate in Project NetWork. The key findings from the evaluation of participation rates among subgroups were as follows:

• Concurrent beneficiaries (those receiving both SSI and DI) had higher participation rates (5.4 percent) than did SSI recipients (4.1 percent) and DI beneficiaries (4.7 percent).
• Those between the ages of 31 and 40 had the highest participation rates among age categories (5.6 percent), and those aged 50 or older had the lowest (3.0 percent).
• Higher participation rates were found among those with some college education (6.5 percent) and a college degree (6.7 percent).
• Participation rates were higher for those with recent work experience (6.8 percent) and those who worked at least 30 hours per week (11.0 percent).
• Higher participation rates were associated with positive attitudes toward work (8.1 percent) and life (6.0 percent) and having an internal locus of control (6.0 percent).

In addition to the analysis of participation rates among subgroups, the Project NetWork evaluation identified three mutually exclusive groups on the basis of the likelihood of participation in the project. The first group—those who were the least likely to participate—consisted of those with severe activities of daily living impairments (ADLs), those with severe instrumental activities of daily living impairments (IADLs), those who never worked, and those who stated that they were unable to work. This group made up 73 percent of the eligible population and had a participation rate of 2.6 percent. The second group—those with a relatively average likelihood of participation in the project—comprised those who did not have severe ADLs or IADLs, those who reported that they were able to work, and those who did not work at least 30 hours per week in the year before the Project NetWork survey. This group made up 24 percent of the eligible population and had a participation rate of 9.2 percent. The third group was made up of those who were most likely to participate in the project—those who worked for more than 30 hours per week in the year before the Project NetWork survey, those who did not have severe ADLs or IADLs, and those who reported that they were able to work. This group made up 3 percent of the eligible population and had a participation rate of 12.2 percent.

State Partnership Initiative. This SSA demonstration project was initiated in 1998 and was completed in 2004.¹³ It consisted of 12 states sponsored by the Social Security Administration and 6 states sponsored by the Rehabilitation Services Administration (RSA).¹⁴ Its purpose was to support innovative strategies designed by the participating states to improve employment outcomes for persons with disabilities. The strategies included a combination of enhancements to work incentive provisions in government programs, employment-related services, job training, rehabilitation services, information on work incentives through individualized benefits counseling, and other services. Each state designed its own strategy for achieving improved employment outcomes, targeted a different group or geographic location for the project, used different outreach activities to identify target group members, and used different enrollment procedures.

In most cases, the outreach and recruitment strategies made it difficult to identify the size of the eligible population and, as a result, it is not possible to provide a reasonable estimate of the overall participation rate among the 12 SSA-sponsored states. In addition, precise information on the size of the eligible population and participation rates for individual states has not been published. However, the 12 SSA-sponsored states identified their goals for the number of participants to be enrolled in the State Partnership Initiative. At the beginning of the project, the 12 states anticipated a target enrollment of 6,860 participants. As of March 31, 2003, 6,506 participants had actually enrolled in the project (Peikes and Paxton 2003).

Job Training Partnership Act Programs

A more sophisticated body of research on recruitment and participation was developed in the evaluation of the Job Training Partnership Act (JTPA) programs. Initial studies focused on how performance standards that emphasized high job placement rates may have affected the choices that JTPA administrators made in selecting participants for the program. That is, they attempted to identify whether the performance standards led JTPA to serve "those who were most employable at the expense of those most in need" (Anderson, Burkhauser, and Raymond 1993). The studies used a regression corrected for selection bias to estimate how job placement outcomes would have differed under the assumption that those eligible would have been randomly included, rather than selected, to participate. They concluded from the analysis that although JTPA administrators appeared to be selecting the most employable individuals, that bias was not as prevalent as critics of JTPA suggested. Moreover, the research suggested that if policymakers want to promote a more equitable distribution of job training across the pool of eligible participants, they must recognize that targeting individuals who are hard to serve will probably come at the expense of relatively lower job placement rates. SSA administrators need to be aware of a similar trade-off when setting performance measures for their demonstration projects and return-to-work programs.

Recent research has more carefully examined the participation process by decomposing it into several stages. For the JTPA program, Heckman and Smith (2004) decomposed the process into eligibility, awareness, application, acceptance, and enrollment. Their analysis of participation in JTPA distinguished between choices made by JTPA administrators and choices made by potential participants. It also identified the stage in the participation process in which differences in the distribution of JTPA services occur. For example, they found that low rates of participation among those with low levels of education were not entirely driven by the decision of JTPA administrators about who they accepted for the program. Rather, they found that lack of awareness of the program among the less educated contributed to lower participation rates. Decomposing the recruitment process for SSA's demonstration projects can provide similar insights.

An Empirical Model of the Multistage Recruitment Process

A variant of the sequential response model described in Maddala (1983, 49) and used by Heckman and Smith (2004) was used to examine the impact of characteristics of SSI recipients on participation in the New York WORKS project. This approach shows how individual characteristics affect responses at each stage of the recruitment process and how they contribute to the overall likelihood of enrollment. Put differently, it describes the point in the multistage recruitment process at which a segment of the SSI recipient population with a particular characteristic loses the opportunity to enroll in the project.

The sequence of events in the New York WORKS project was as follows:

• the successful delivery of the information letter to SSI recipients,
• a response to the letter given that the letter was received,
• expression of interest in the project given that the person responded to the letter, and
• enrollment in the project given that the person expressed an interest in the project.

Equation 1 describes the decomposition of the probability of being enrolled in the project (Enr) given that the person is eligible (El). In this equation, P(G=1|El=1,X) represents the probability that the information letter regarding the project was delivered to a person with a good address (G) given that the person with a set of characteristics (X) was eligible (El); P(R=1|G=1,El=1,X) represents the probability that a person responded to the information letter (R) given that the person with a set of characteristics (X) was eligible and had a good address; P(I=1|El=1,G=1,R=1,X) represents the probability that a person expressed interest in the project given that the person with a set of characteristics (X) was eligible for the project, had a good address, and responded to the letter; and P(Enr=1|El=1,G=1,R=1,I=1,X) represents the probability that a person enrolled in the project given that the person with a set of characteristics (X) was eligible, had a good address, responded to the letter, and expressed interest in the project. Having a "good address" means that the information letter that was sent was not returned to the New York WORKS project staff because the person on the list no longer lived at that address. The SSI recipients' characteristics (X) include age, sex, geographic location, type of psychiatric diagnosis, the number of months that the person collected benefits before the recruitment process, employment experience before the recruitment process, and annual earnings in the year before the recruitment process.

Equation 1

The four conditional probabilities shown on the right-hand side of equation 1 were estimated separately using logits (see Maddala 1983). The probability of a good address given that the person was sent a letter was estimated using the entire group that was sent a letter. The probability of responding to the letter given that a letter was sent was estimated using the entire group that received a letter. The probability of expressing interest in the program given a response to the letter was estimated using the entire group that responded to the letter. Finally, the probability of enrollment in the project was estimated using the entire group that expressed interest in the project. The same set of characteristics is included in each of the logits.

The resulting coefficients of each of the logit models were used to estimate the impact of a change in a characteristic on the probability that a particular event within the multistage recruitment process will occur. We used the average of the marginal effects for each person in the data as recommended by Bartus (2005). For the dummy variables, we replaced the marginal effect with the change in the probability associated with each value of the variable. Standard errors for the impact of a characteristic on the change in the probability that a particular event in the multistage recruitment process will occur were computed using the delta method.¹⁵

Equation 2 describes the decomposition of the change in the probability of enrollment associated with a change in each characteristic at each stage of the recruitment process.¹⁶ Equation 2 is the derivative of the probability of enrollment given that the person was eligible with respect to characteristics X in equation 1. For the dummy variables in the model, derivatives were replaced with discrete changes. Each of the four terms on the right-hand side of equation 2 was calculated using the impact of characteristics X on the change in the probability of the particular event in the recruitment process, as described in the previous paragraph. The estimated predicted probability of the particular event in the recruitment process was the mean of the predicted probabilities for each person at that stage of the recruitment process.

Equation 2

The empirical model provides a description of what actually happened in the New York WORKS process. The estimated parameters in the later stage of the model might change if changes were made to the recruitment process at an early stage of the project that resulted in increased enrollment. For example, if the process changed so that there was a large increase in the number of young persons who responded to the information sent about the demonstration, we could not rely on the parameters at the interest and enrollment stages to remain the same because we do not know how the additional younger persons who responded would behave at these two stages. What our empirical model shows is how the New York WORKS recruitment process, which is similar to processes used in other demonstration projects, led to differences in enrollment and where in the process those differences occurred.

Data

The target group for the project was defined as SSI recipients living in Erie County or New York City and having a primary medical diagnosis of psychiatric illness. In addition, participants were required to be over the age of 21 and not hospitalized or incarcerated. SSA provided a list of 68,489 individuals meeting these criteria. However, we excluded 448 of them from the analysis because they were directly referred to the project. The remaining 68,041 people who experienced the recruitment process were randomly assigned to the full-service group, the enhanced-service group, or the control group. From this SSA list, 59,764 persons (87.2 percent) were randomly selected and sent an information letter that described the project as having potential benefits for people with disabilities who wanted to work.¹⁷

The data for the project came from three sources: New York WORKS project data, SSA administrative data, and New York State unemployment insurance data. Frequencies from each stage of the recruitment process came from the New York WORKS project. The SSA administrative data came from the Supplemental Security Record, which is the system used by SSA to administer the Supplemental Security Income program. The Supplemental Security Record data extract includes information on age, sex, primary medical diagnosis, participation in SSI work incentive programs (for example, 1619a and 1619b), and date of eligibility for SSI benefits. Data on quarterly earnings came from the New York State unemployment insurance records.

The focus for the analysis of the recruitment process was on persons who were assigned to the full- and enhanced-service groups.¹⁸ These two groups experienced the last stage in the recruitment process, which involved being enrolled in an intervention. We combined them for our empirical analysis and refer to them as the treatment group because they were involved in an ongoing intervention.¹⁹ Members of the control group were never offered such an intervention and thus were not involved in the enrollment stage of the process. Definitions of variables for the stages of the recruitment process and the characteristics of persons involved in the process are shown in Table 3.

Size of the Sample at Various Stages of the Recruitment Process

Of the 68,041 persons who were on SSA's list and included in the analysis, 47,209 were assigned to the treatment group and 20,832 were assigned to the control group (Table 4). For the 47,209 persons assigned to the treatment groups, 41,431 were randomly chosen to receive a letter, 37,415 did not have the letter returned to New York WORKS because of a bad address, 17,275 of the 37,415 responded to the letter, 4,187 of the 17,275 who responded reported that they were interested in participating in New York WORKS, and 900 ultimately enrolled in the project.

The distribution of responses at each stage of the recruitment process as a percentage of the total number of persons who were involved in the stage is also shown in Table 4. For example, 90.3 percent of treatment group members received the letter and 9.7 percent did not. For the other stages of the recruitment process, 46.2 percent of the treatment group responded to the letter and 53.8 percent did not. Of those who responded, 24.2 percent reported that they were interested in the project and 75.8 percent did not. Of those who reported interest, 22.1 percent were enrolled in the project and 77.9 percent were not. The percentages for the control group members were similar. These numbers provide a reference for the predicted probabilities presented later in this article.

Finally, Table 4 shows the percentage of people who were sent a letter and who survived each stage of the recruitment process. For those in the treatment group who were sent a letter, 90.3 percent did not have the letter returned to New York WORKS because of a bad address, 41.7 percent sent in a postmarked form that indicated whether or not they were interested in the program, 10.1 percent expressed an interest in the program, and 2.2 percent were enrolled in one of the treatment groups. The pattern for the control group was similar. These numbers suggest that most of the persons eligible for the program leave the recruitment process by not responding to the letter or by expressing that they are not interested in participating.

Characteristics of the Sample at Each Stage of the Recruitment Process

As mentioned above, the characteristics associated with the results of the recruitment process are shown in Table 3. The variables fall into four general categories:

• demographic characteristics,
• psychiatric diagnosis,
• past employment and earnings, and
• program participation.

The variables used to describe demographic characteristics include sex, age, and location. Age is divided into the following five categories: ages 21–29, 30–39, 40–49, 50–59, and 60 or older.

The variables used to describe psychiatric diagnosis are drawn from SSA administrative records and are based on the diagnosis used by SSA disability examiners to establish a person's eligibility for the SSI disability program. The variables include schizophrenia, affective disorder, anxiety disorder, and other psychiatric disorders.²⁰

Data on past employment and earnings come from the New York State unemployment insurance administrative records and are reported on a quarterly basis (that is, they cover a 3-month period). The employment variables include whether the person was employed in the year before the date that the information letter was sent, in the quarter before the date the information letter was sent, or in the quarter in which the information letter was sent. Annual earnings in the year before the date that the information letter was sent are grouped into six categories: no earnings, earnings of $5,000 or less, earnings of $5,001–$10,000, earnings of $10,001–$15,000, earnings of $15,001–$20,000, and earnings of more than $20,000.²¹

Finally, variables for program participation include concurrent beneficiary status and months from the date of enrollment in SSI to the date that the information letter was sent. Concurrent beneficiaries collect SSI and DI benefits at the same time: they meet the means test to collect SSI benefits and the insured status requirements to collect DI benefits. The number of months from the date of enrollment in SSI to the date that the letter was sent is grouped into six categories. Five of the categories are grouped into 24-month intervals up to 120 months, and the sixth category represents more than 120 months.

Changes to the distribution of characteristics at each stage of the recruitment process and the rationale for the decomposition suggested by Heckman and Smith (2004) are shown in Table 5. The demographic characteristics are shown for persons on the SSA list who were sent a letter, those who were not sent a letter, and those who were not sent a letter but were referred to the New York WORKS project.²² The distribution by sex was similar for those who were sent an information letter and those who were not sent an information letter and were not referred: 39.4 percent were men and 60.6 percent were women.

An examination of the "Yes" columns at each stage after the information letter was sent shows how the composition of men changed at each stage of the process. The percentage of SSI recipients who were men changed from 39.4 percent who were sent a letter, to 38.4 percent who received it, to 35.1 percent who responded, to 46.4 percent who expressed interest in the project, and to 50.0 percent who enrolled. More important, the change in composition that occurred at each stage suggests that different stages of the recruitment process had a different impact on the final composition of those who enrolled.

The remainder of Table 5 shows how the composition changed for each of the other sets of characteristics that were included in the analysis. Two observations emerge from the descriptive statistics and suggest the need for further analysis, as described in the next section. First, the composition of characteristics for those who were initially sent an information letter differed from those who enrolled. That is, there are systematic differences in the observed characteristics between these two groups. Second, the descriptive statistics suggest that these differences occur at different stages of the recruitment process.

Results from the Empirical Model

The descriptive statistics in Table 5 do not account for the underlying relationships between the characteristics of SSI recipients. For example, female SSI recipients tend to be older than male SSI recipients. The descriptive statistics show that the percentage of women who enrolled was smaller (50 percent) than the percentage of woman who were sent a letter (60.6 percent). They also show that the percentage of enrollees aged 60 or older was smaller (3.9 percent) than the percentage who were sent a letter (24.9 percent). On the one hand, the change in the composition of women could simply be due to the fact that women tend to be older and those who are older are less likely to enroll. On the other hand, the change in the composition of those who are older could be due to the fact that older persons tend to be women and women may be less likely to enroll. The descriptive statistics in Table 5 do not distinguish between these two competing explanations. The decomposition model accounts for the underlying relationships and, for example, distinguishes between the impact of sex and the impact of age on enrollment. The model also shows how these characteristics have a different impact at each stage of the recruitment process and how the differences at each stage affect the likelihood of enrollment.

The decomposition used three steps:

1. Estimation of a logit for each step of the recruitment process,
2. Estimation of the impact of the characteristics on the probability of the outcome based on logit coefficients, and
3. Use of the impact and the predicted probability of each outcome to decompose the probability of enrollment, as shown in equation 2.

The results from the logit models are shown in Table A-1.²³ The impact of the characteristics on the probability of an outcome at each stage of the recruitment process was estimated using the logit coefficients and provided more meaningful information than the logit coefficients alone. Therefore, this article focuses on the impact of characteristics on the probability of enrollment at each stage of the recruitment process, as shown in Table 6. The decomposition of the multistage recruitment process is shown in Table 7.

Impact of Determinants on Conditional Probability

The predicted probability of a positive response at each stage given that the person survived to the particular stage in the recruitment process is shown in the last row of Table 6. Of those sent a letter, the predicted probability that it was sent to a good address was 90.8 percent. Of those with a good address, the predicted probability that the person responded was 46.2 percent. Of those who responded, the predicted probability that the person expressed interest in the project was 24.2 percent. The predicted probability of enrollment among those who expressed interest in the project was 22.1 percent. The predicted probabilities are very similar to the sample proportions shown in Table 4. They provide a reference point for the estimated effect of each of the characteristics.

Table 6 also shows the estimated change in the probability of a response at each stage of the recruitment process for a change in each variable, holding the other variables in the model constant.²⁴ It provides an estimate of the magnitude of each variable's effect on a response at each stage for the group of SSI recipients who survived to that stage. The estimates in Table 6 focus on differences that arise at each stage.

Impact of Demographic Characteristics. Compared with women, men had a 3.0-percentage-point reduction in the probability of a good address, a 5.0-percentage-point reduction in the probability of a response given a good address, a 4.7-percentage-point increase in the probability of expressing interest given a response, and a 2.1-percentage-point increase in the probability of enrollment given that they expressed interest in the project.

The impact of age on a response at each stage of the recruitment process is substantial, especially the impact on a response to the letter given that it was sent to a good address and on interest in the project given that the person responded. Differences in the probability of a good address are relatively small and indicate that younger persons were relatively less likely to have the letter successfully delivered. Compared with persons aged 50–59, the probability of a good address was 1.7 percentage points lower for those aged 21–29, 3.4 percentage points lower for those aged 30–39, 1.7 percentage points lower for those aged 40–49, and 0.7 percentage points higher for those aged 60 or older.

Younger persons were significantly less likely to respond to the letter given that the letter was sent to a good address. Compared with those aged 50–59, the probability of a response was 13.6 percentage points lower for those aged 21–29, 10.1 percentage points lower for those aged 30–39, and 3.4 percentage points lower for those aged 40–49.

Among those who responded to the letter, the probability of an expressed interest in the project was significantly higher for young persons. Compared with those aged 50–59, the probability was 25.3 percentage points higher for those aged 21–29, 18.1 percentage points higher for those aged 30–39, and 9.7 percentage points higher for those aged 40–49.

Finally, the probability of enrollment given that the person expressed interest in the project was significantly higher for young persons. The probability was 8.0 percentage points higher for those aged 21–29 than for those aged 50–59. The estimated impact on enrollment was about the same for those aged 30–39 and 40–49 (7.4 percentage points and 7.9 percentage points higher, respectively).

The magnitude of the difference between Erie County and New York City was large and statistically significant for each stage of the recruitment process except for the probability of a good address. In Erie County, the probability of a response given a good address was 5.8 percentage points higher, the probability of interest given a response was 10.1 percentage points higher, and the probability of enrollment given interest was 12.1 percentage points higher.

Impact of Psychiatric Diagnosis. Psychiatric diagnosis has a statistically significant impact on the probability of a good address and the probability of interest in the project given that the person responded. Compared with those in the "Other" psychiatric diagnosis category, the probability of a good address was 1.7 percentage points higher for persons with an affective disorder and 2.7 percentage points higher for persons with an anxiety disorder, holding all else equal. The probability of interest given a response to the information letter was 4.1 percentage points higher for persons with a diagnosis of schizophrenia and 3.0 percentage points lower for persons with an anxiety disorder compared with persons in the "Other" psychiatric diagnosis category, holding other variables in the model constant. Although not statistically significant, the estimated probability of enrollment given that the person expressed interest in the project was 2.7 percentage points lower for those with an anxiety disorder compared with those in the "Other" psychiatric diagnosis category.

Impact of Past Employment and Earnings. The estimated impact of employment in the quarter before the date that the information letter was sent was relatively small for all recruitment stages and was not statistically significant in almost all cases. A notable exception was the relatively large impact on the probability of enrollment given interest in the program: those who were employed had a 5.1-percentage-point increase in the probability of enrollment compared with those not employed at that time.

Earnings in the year before the date that the information letter was sent had a significant impact on the probability of a response given a good address, the probability of interest given a response to the letter, and the probability of enrollment given interest in the project. The results for the probability of a good address were relatively small, not statistically significant, and did not illustrate a clear pattern between earnings and a good address. Compared with those with no earnings, the probability of a response given a good address was 3.2 percentage points higher for those earning $5,000 or less and 8.7 percentage points lower for those earning more than $20,000, holding other factors in the model constant. This finding is not surprising given that the employment services in the New York WORKS project provide very important benefits to those with very low earnings and provide very limited benefits to those earning above $20,000 per year.

Compared with persons with no earnings, the probability of interest in the project given a response was higher for all earnings levels. It was largest for those with the lowest earnings and declined as earnings increased. It was 25.1 percentage points higher for those earning $5,000 or less, 20.5 percentage points higher for those earning $5,001–$10,000 and $10,001–$15,000, 5.1 percentage points higher for those earning $15,001–$20,000, and 1.7 percentage points higher for those earning more than $20,000.

Persons with relatively low earnings also had a higher probability of enrollment given that they expressed interest in the project. Compared with those who were interested but had no earnings, the probability of enrollment given interest in the project was 10.7 percentage points higher for those earning $5,000 or less, 7.6 percentage points higher for those earning $5,001–$10,000, and 7.5 percentage points lower for those earning $15,000–$20,000. This finding is consistent with the idea that the benefits of the project are more attractive for those with low earnings than they are for those with higher earnings.

Impact of Program Participation. The impact of program participation variables on the probability of each event in the recruitment process is also shown in Table 6. Compared with individuals receiving SSI only, those receiving SSI and DI benefits concurrently were more likely to have a good address, respond to the information letter, and enroll in the project given that they expressed an interest. For concurrent beneficiaries, the probability of a good address was 2.4 percentage points higher, the probability of a response to the information letter given a good address was 15.7 percentage points higher, and the probability of enrollment given interest in the project was 2.8 percentage points higher than for SSI recipients only. The probability of interest in the project given a response to the information letter was 7.0 percentage points lower for SSI and DI concurrent beneficiaries.

Finally, the only stage at which time enrolled in the SSI program had a noteworthy impact was on the probability of a good address. Compared with those in the program for 24 months or less, the probability of a good address was 3.4 percentage points lower for persons on the program for 25–48 months, 8.4 percentage points lower for 49–72 months, 9.1 percentage points lower for 73–96 months, 7.6 percentage points lower for 97–120 months, and 6.0 percentage points lower for more than 120 months.

Decomposition of the Recruitment Process

The results of the decomposition of the probability of enrollment are shown in Table 7.²⁵ The average probability of participation among eligible SSI recipients who were sent an information letter was 2.2 percent. The 2.2 percent rate may seem small in absolute terms, but it is important in that it exceeds the 0.5 percent rate at which SSI and DI beneficiaries leave the programs because they return to work.²⁶ Moreover, policymakers have estimated that doubling the current 0.5 percent rate could lead to $3.5 billion in savings over the work life of such persons.²⁷ Given the emphasis that policymakers have placed on the 0.5 percent rate, we use differences of 0.5 percent, as well as the 2.2 percent predicted probability of enrollment, to highlight important differences in the probability of enrollment. Because the differences in enrollment are relative to a reference group, we compare these two reference points to differences in the absolute value of the probability of enrollment.

Impact of Characteristics on Probability of Enrollment Given That a Letter Was Sent. Table 7 also shows the impact of each of the characteristics on the overall probability of enrollment for those who were sent a letter. The differences in the probability of enrollment for each characteristic compared with the reference category are important in many cases and in some cases are larger than the average predicted probability of enrollment. For example, compared with persons aged 50–59 and holding the other characteristics in the model constant, the predicted probability that SSI recipients enrolled was 2.5 percentage points higher for those aged 21–29, 1.8 percentage points higher for those aged 30–39, 1.5 percentage points higher for those aged 40–49, and 1.9 percentage points lower for those aged 60 or older. These differences are large relative to the 2.2 percent average predicted probability of enrollment and the 0.5 percent rate of exit from the disability programs because of earnings from work. Erie County had relatively higher enrollment rates than New York City, as indicated by a probability of enrollment that was 2.4 percentage points higher in Erie County than in New York City. Compared with persons with affective disorders or other psychiatric disorders, those with schizophrenia had a 0.4-percentage-point increase in enrollment, and those with anxiety disorders had a 0.5-percentage-point reduction in enrollment.

Relatively large differences in enrollment rates were also related to employment experience. Persons employed in the quarter before the information letter was sent had a 0.8-percentage-point increase in enrollment compared with those who were not employed. Compared with persons with no earnings in the year before the information letter was sent, the probability of enrollment was 3.5 percentage points higher for persons earning $5,000 or less, 2.8 percentage points higher for those earning $5,001–$10,000, 1.5 percentage points higher for those earning $10,001–$15,000, and 0.8 percentage points lower for those earning $15,001–$20,000. There was a comparatively small increase (0.2 percentage points) in enrollment for those earning more than $20,000 compared with those with no earnings.

The only noteworthy difference in the SSA program variables was the 0.5-percentage-point increase in the probability of enrollment for persons receiving SSI and DI concurrently compared with those receiving SSI only. The differences across the time enrolled in the SSI program were comparatively small. Table A-2 shows that the differences in enrollment estimated in Table 7 from the decomposition are similar to the marginal effects that result from a logit for the overall probability of enrollment for the treatment group members who were sent a letter.

Impact of Each Stage on Differences in Probability of Enrollment for Each Characteristic. Table 7 also shows the decomposition described in equation 2, that is, the contribution of each stage of the recruitment process to differences in the probability of enrollment for each characteristic in the model. The decomposition shows how characteristics can have a very different impact on the probability of enrollment at different stages. In some cases, a stage in the recruitment process may reduce the overall differences in enrollment across characteristics, and in other cases it may significantly contribute to the difference.

This information is important for at least three reasons. First, this information is important to policymakers who are concerned with promoting equal access to services.²⁸ For example, differences that occur at the enrollment stage of the process may indicate that there are barriers that systematically affect certain segments of the SSI recipient population within the final enrollment procedures of the New York WORKS project. Identifying and addressing such barriers is important in order to improve participation.

Second, the information allows one to distinguish between differences that arise purely because of interest in the project (that is, self-selection) and differences that may arise from project administration. As demonstrated in the JTPA evaluation literature, project administrators may address the former differently from the latter.

Finally, program administrators often focus solely on the characteristics of those who express interest in SSA's demonstration projects. The decomposition indicates that for some characteristics such an analysis is probably affected by selection bias. For example, policymakers who focus only on the impact of the concurrent beneficiary status on the probability of being interested in the program may be puzzled by the low probability of interest this group shows compared with that of persons who only receive SSI. Our analysis shows that the difference in probability of expressing interest in the project across the two groups may be related to differences that occur at the response stage of the recruitment process.

Good or Bad Address. The first stage described in the decomposition shows that few important differences arise as the result of a bad address.²⁹ Only the length of time enrolled in the SSI program is related to a bad mailing address. This relationship is not surprising given that many SSI payments are made through direct deposit and that participants have little incentive to report address changes to SSA. However, the contribution is small.

Response. Significant differences occur at the response stage of the process. The contribution of the response stage to the probability of enrollment is 0.2 percentage points lower for men compared with women. Compared with persons aged 50–59, the response stage tends to reduce overall differences in enrollment, with the exception being persons aged 60 or older. This effect is evident by estimates of 0.7 percentage points lower for persons aged 21–29, 0.5 percentage points lower for persons aged 30–39, and 0.2 percentage points lower for persons aged 40–49 and 60 or older. Differences at the response stage contributed to an increase in the enrollment probability of 0.3 percentage points for persons living in Erie County.

The response stage had relatively little effect on differences across psychiatric disorders or differences by employment in the quarter before the date the information letter was sent, but it did contribute to differences across categories of past earnings. Compared with individuals with no earnings in the previous year, it slightly increased the probability of enrollment by 0.2 percentage points for those earning $5,000 or less and by 0.1 percentage points for those earning $5,001–$10,000. It reduced the probability of enrollment by 0.3 percentage points for those earning $10,001–$15,000 and by 0.4 percentage points for those in the two highest earnings categories ($15,001 or more).

The largest impact on the difference in the probability of enrollment occurred for persons receiving both SSI and DI benefits. The probability was 0.8 percentage points higher for concurrent beneficiaries.

It is important to keep these results in mind when focusing on results at the next stage of the process. That is especially true for results that have the opposite effect on the overall probability of enrollment.

Interest. The interest stage of the recruitment process tends to make the largest contribution to differences in the probability of enrollment. The differences that occur at this stage are consistent with characteristics of persons who are the most likely to benefit from the New York WORKS services. Younger individuals have a longer working life, and the benefits of investing in an employment-related program are likely to occur over a longer period of time. All else being equal, men are less likely to leave the labor market to raise children and thus have greater expected gains in lifetime earnings as a result of an employment-related program. Persons who were employed before the project have demonstrated that they have the capacity to work and are more likely to benefit from the project's changes to the SSI work incentive provisions. Finally, among those who were employed before the project, the provisions of the New York WORKS project are likely to be of greater value to low earners than to high earners. Low earners are less likely to have jobs with health insurance benefits and are less likely to have experience with or knowledge of the SSI 1619b provisions. They are more likely to be in a position where they would like to increase their work and earnings but may have concerns about how it will affect their benefits and may therefore have a greater need for benefits advisement. Low earners may also be more likely to be constrained by other means-tested programs, such as housing subsidies, or to face barriers resulting from transportation needs. Thus, low earners are more likely to benefit from the housing waivers and transportation subsidies provided by the New York WORKS project. These results remain important even if one assumes that part of whether or not they were interested was already reflected in the response stage of the recruitment process.

The interest stage contributes to a lower probability of enrollment for persons with an anxiety disorder (compared with persons in the "Other" psychiatric diagnosis category) and for concurrent beneficiaries. To some degree, the result for concurrent beneficiaries may be explained by the response stage of the process, which has an almost equal and opposite impact on the probability of enrollment. However, because the actual reason for nonresponse among SSI recipients is not known, it is not possible to infer from the data whether the difference at the response stage was due to differences in interest in the program, whether it was a function of the administration of the project, or whether it was due to some other reason.

The estimated contribution of the interest stage to differences in the probability of enrollment was 0.4 percentage points higher for men, 2.3 percentage points higher for persons aged 21–29, 1.7 percentage points higher for persons aged 30–39, 0.9 percentage points higher for persons aged 40–49, 0.7 percentage points lower for persons aged 60 or older, 0.9 percentage points higher for persons living in Erie County, 0.4 percentage points higher for persons with schizophrenia, and 0.3 percentage points lower for persons with an anxiety disorder (Table 7). Compared with persons with no earnings, the contribution was 2.3 percentage points higher for those earning $5,000 or less, 1.9 percentage points higher for those earning $5,001–$10,000 and $10,001–$15,000, and 0.5 percentage points higher for those earning $15,001–$20,000. Finally, compared with SSI-only recipients, the impact of the interest stage on the probability of enrollment was 0.7 percentage points lower for concurrent beneficiaries.

Enrollment. Differences in the probability of enrollment that are the result of the enrollment stage of the recruitment process are also shown in Table 7. Differences that arise at this final stage are important in many cases, and in some cases they are more important than the contribution at the interest stage. Like the findings at the interest stage, the findings about the differences that arise at the enrollment stage are consistent with the characteristics of persons who are more likely to benefit from the New York WORKS services. The enrollment stage contributes to a higher probability of enrollment for men, younger persons, and those who were employed with low earnings in the year before the date the information letter was sent.

The enrollment stage contributes to a lower probability of enrollment for persons with anxiety disorders and for persons earning between $15,001 and $20,000. In the former case, it may be worthwhile for project administrators to carefully examine whether their recruitment process is less likely to accommodate persons with an anxiety disorder compared with persons with all other types of psychiatric disorders. It is also somewhat surprising that the contribution to the probability of enrollment for those earning $15,001–$20,000 is in the opposite direction in the enrollment stage (a reduction of 0.8 percentage points) from that of the interest stage (an increase of 0.5 percentage points) of the recruitment process. The contribution of the final enrollment stage to the probability of enrollment is 0.2 percentage points higher for men compared with women. Compared with persons aged 50–59, it is 0.8 percentage points higher for persons in all age groups except those aged 60 or older, for whom it is 1.0 percentage points lower. It is 1.2 percentage points higher in Erie County compared with New York City, 0.3 percentage points lower for persons with anxiety disorders, 0.5 percentage points higher for persons employed in the quarter before the date that the information letter was sent, 1.1 percentage points higher for persons earning $5,000 or less, 0.8 percentage points higher for persons earning $5,001–$10,000, and 0.8 percentage points lower for persons earning $15,001–$20,000.

Discussion

Our analysis of the literature shows that participation rates in SSA demonstration projects that target SSI and DI beneficiaries tend to be between 4.5 percent and 5.0 percent. These numbers have been described as low when compared with the nearly 50 percent of SSI and DI beneficiaries who report that they would be ashamed if they did not try to work (Rupp, Wood, and Bell 1996; Newcomb, Payne, and Waid 2003, 65). The demonstration projects that targeted DI applicants—the Rapid Intervention Employment Project and the Expedited Referral and Intervention Validation—had relatively higher participation rates (14 percent and 22 percent, respectively). Although the sample sizes for these demonstrations were small, they provide some indication that higher participation rates may be possible if projects target applicants rather than beneficiaries. A key finding from the review of the literature is that the strongest predictor of program participation is recent or current work experience.

An important and often overlooked finding from past SSA demonstration projects is discussed in the evaluation of the Expedited Referral and Intervention Validation project. That is, the way that information was delivered by SSA played an important role in the recruitment process. The finding suggests that any evaluation needs to look at the entire process and identify the key points in the process that affect the probability of participation. Such an approach to evaluation can provide insight into how to make improvements that will lead to a more equitable distribution of employment services and, potentially, lead to increased participation and more employment. The recent work of Heckman and Smith (2004) suggests an approach to analyzing a recruitment process and applies the approach to JTPA.

Our analysis of the New York WORKS recruitment process builds on these findings. We examine the characteristics associated with enrollment identified from past SSA demonstrations in an empirical model that was developed within the JTPA literature by Heckman and Smith. Our results highlight the importance of their approach when evaluating characteristics associated with participation in programs designed to improve employment outcomes for persons with disabilities. As shown in Table 7, the characteristics make a different contribution to enrollment at different points in the recruitment process.

Our results show that important differences did not arise at the first stage of the process. If the objective is to ensure equal access to employment services, devoting extra resources to ensuring that the letter is sent to the right address is unlikely to have an important impact.

Important differences in the probability of enrollment begin to occur at the response stage. In many cases these differences are in the opposite direction of differences that occur at subsequent stages in the process. Although this pattern may be consistent with the notion that those who did not respond were not interested, it is not possible to make these inferences from our data. Because we do not observe the reason for nonresponse, it is possible that differences at this stage resulted for other reasons. We recommend that future evaluations of the recruitment process identify a sample of the nonrespondents and perform a follow-up analysis of their reasons for not responding.

The largest contribution to differences in the probability of enrollment occurs at the interest stage of the process. This is evident in the differences across sex, age, location, psychiatric disorder, and earnings in the year before the date that the information letter was sent. These results are consistent with the characteristics of persons who are most likely to benefit from the services of the New York WORKS project. However, to some degree these contributions may not have fully measured the importance of interest in the project on the overall unconditional probability of enrollment. If the differences that occur at the response stage are partly due to interest, it is possible that the differences at the interest stage may have understated the true contribution of interest in the project.

The final stage of the process—the enrollment decision—shows that this stage also makes an important contribution to differences in the probability of enrollment. The pattern is similar to the interest stage of the process. The individuals who are most likely to benefit from the project are the ones who are most likely to successfully enroll in the project. For policymakers interested in equal access to employment services, the differences that occur at this stage may be the most troubling. Presumably, all persons at this stage are interested in employment services, yet some groups are more successful in obtaining the services than others.

Several areas of our study need to be extended in future research. First, our results are limited to the characteristics that are available in the data. Information on race, literacy, and education level is not available in the data, but all of these characteristics are likely to affect the probability of enrollment differently at different stages of the process. Project administrators should consider including data on race and education level for future evaluations of the recruitment process. Second, if similar processes are used in future demonstration projects, we recommend identifying random samples of persons who choose not to enroll at different stages in the process and collecting information on the reason for their decision not to enroll. The additional data can help project administrators gain a clearer description of the reasons that differences occur at each stage of the process and make changes to the process accordingly.

Appendix

Additional information on the analysis presented in this article are in Tables A-1 through A-5. Table A-1 shows the estimated parameters of the logit model. These parameters were used to estimate the marginal effects in Table 6. Table A-2 shows estimates based on a logit for enrollment in the project for all persons who were sent a letter and who were assigned to the treatment group. These estimates do not show what happens at each stage of the recruitment process; rather they show the final result of the process. Tables A-3 through A-5 show results of fully interacted models to test for parameter differences by the type of treatment (full service compared with enhanced service), by sex, and by location (Erie County compared with New York City). The purpose of this analysis is to examine the extent to which the parameter estimates may differ by treatment group, sex, and location.