An Overview of the National Survey of SSI Children and Families and Related Products

Summary

The National Survey of SSI Children and Families (NSCF) is the first nationally representative survey since 1978 of noninstitutionalized children and young adults who currently receive or formerly received Supplemental Security Income (SSI). Over 8,500 interviews were completed between July 2001 and June 2002. The primary objective of the NSCF is to provide data to support research and policy evaluation on the current cross section of children (ages 0 to 17) and young adults (ages 18 to 23) receiving SSI. Following that objective, the survey was designed to answer questions such as those presented below.

• What are the general characteristics of children and young adults receiving SSI and their families?
• What are the patterns of access to and utilization of health care among children and young adults receiving SSI?
• What services are utilized by children and young adults receiving SSI?
• What are the unmet health care and service needs of children and young adults receiving SSI?
• What costs are associated with caring for a disabled child?
• What is the impact on the family of having a disabled child?
• What is the status of young adults with disabilities as they make the transition to adulthood?
• How well are they prepared for that transition?

In addition, the NSCF questionnaire and sample were designed to be comprehensive enough and large enough to address numerous additional policy issues as they emerge. The NSCF fills a gap in the data available to policy analysts by addressing a wide range of topics that cannot be addressed with SSI administrative data and by providing a large sample in contrast to major national survey databases that cover this target population fairly sparsely. A companion article to this overview describes general characteristics of SSI beneficiary children and their families (see Rupp and others 2005/2006, in this issue). Other topics being examined include disability-related expenditures for SSI children and young adults and labor force participation of the parents of SSI children.

The NSCF data files are accompanied by a detailed User's Manual, which includes a detailed codebook and information about the NSCF sample design, questionnaire design and content, data collection procedures, variable construction, editing, and variance estimation procedures. In order to facilitate research, the Social Security Administration published the NSCF Public-Use File and survey documentation on its Web site. These products are available at http://www.socialsecurity.gov/disabilityresearch/nscf.htm. The NSCF is an outstanding tool for conducting research and policy analysis regarding children and young adults receiving SSI.

Introduction

The National Survey of SSI Children and Families (NSCF) is a nationally representative survey of noninstitutionalized children and young adults who are current or former recipients of Supplemental Security Income (SSI). The primary objective of the NSCF is to provide data to support research and policy evaluation on the cross section of children aged 0–17 and young adults aged 18–23 who were receiving SSI in December 2000.¹ Two groups were excluded from the sample universe: recipients in Alaska, Hawaii, and the Northern Mariana Islands, for logistical and cost considerations; and the almost 13,000 children and young adults receiving SSI, or 1.5 percent of the childhood caseload in December 2000, who lived in a Medicaid institution.

The survey yielded 8,535 completed survey interviews.² The interviews provide detailed data on topics including health, functional limitations, and disability status; health care utilization; service utilization, perceived needs, and expenses; education and training; employment and earnings; family income and assets; and health insurance. The data were collected between July 2001 and June 2002 through telephone and in-person interviews. Interviews were conducted in English or Spanish. The Social Security Administration (SSA), in collaboration with Mathematica Policy Research, Inc., designed the NSCF, and Mathematica collected the data and prepared the data files and documentation.

The NSCF fills a gap in the data available to policy analysts for two reasons. First, it addresses a wide range of topics that cannot be addressed with SSI administrative data. Second, it provides a large sample in contrast to major national survey databases that cover this target population fairly sparsely (see Ireys and others (2004) for detailed comparisons). In order to facilitate research, SSA published the NSCF Public-Use file and survey documentation at SSA's Web site. These products are available at http://www.socialsecurity.gov/disabilityresearch/nscf.htm.

The SSI program was designed to protect the disabled and elderly from extreme poverty. When the original legislation was drafted, Congress heavily debated whether to include children in the SSI program. If the intent was to cover out-of-pocket expenses associated with the child's disability, perhaps Medicaid coverage would be sufficient and cash benefits unnecessary. If families of disabled children incur other expenses, such as lost wages of a parent who stays at home to care for the disabled child, then cash benefits in addition to Medicaid coverage would be justified.

In the end, Congress decided to include children in the program and to provide cash benefits. The first SSI payments were issued in January 1974. In most states, Medicaid coverage for children receiving SSI is either automatic or very closely related to the SSI eligibility criteria. Although initially a small portion of the overall SSI program, the childhood rolls have grown steadily over the years. In December 1974, only 70,900 children under the age of 18 received SSI payments, representing 3.8 percent of the total SSI caseload; by December 2005, that number had grown to 1,036,498, or 14.6 percent of the SSI caseload.

The NSCF is the first national survey since 1978 of noninstitutionalized children and young adults receiving SSI. General social surveys such as the Survey of Income and Program Participation and the Current Population Survey typically do not contain enough observations to support detailed analyses of this population. SSA administrative records cover all SSI recipients and contain detailed programmatic information, but they lack data on the demographic and economic characteristics of SSI recipients and their families. Some researchers have conducted case studies and focus groups with children and young adults receiving SSI and their families, but the samples are small and not nationally representative (see, for example, Rogowski and others 2002; Lazear and Worthington 2002). The NSCF fills this critical data gap by combining current and credible survey data on the health and well-being of a nationally representative sample of the target population with current and historical administrative records on program status and receipt of SSI payments.

The NSCF data files are accompanied by a detailed User's Manual, which includes a detailed codebook and information about the NSCF sample design, questionnaire design and content, data collection procedures, variable construction, editing, and variance estimation procedures. The NSCF has the potential to be the premier source of data for policymaking, program planning, and general research on children and young adults receiving SSI.

This article

• presents the major groups of interest, sample sizes, and interview completion rates;
• discusses the design and content of the questionnaire;
• describes the procedures for collecting data;
• provides an overview of the procedures for imputing values for missing data, developing sampling weights, and estimating variance; and
• introduces the major NSCF data products.

Groups of Interest, Sample Sizes, and Completion Rates

The overall sample design of the NSCF provides a framework for analyzing a wide range of policy issues of current and future interest. During the design phase, SSA identified two major groups for analysis—children and young adults receiving SSI in December 2000 and children and young adults receiving SSI in December 1996—and, within these two groups, several subgroups of interest. The design also included a group of people not receiving SSI as a potential tool for making various comparisons with the two major analysis groups. The groups were identified on the basis of programmatic variables defining SSI receipt in SSA's administrative records. The groups were reorganized into mutually exclusive sampling strata, and a complicated algorithm was developed to optimize the sample allocation to meet SSA's analytic needs subject to a number of precision and cost constraints. The major analysis groups and important subgroups are described below, followed by a discussion of sample sizes and completion rates. Potter (2001) and Potter and Diaz-Tena (2003) provide additional details.

Children and Young Adults Receiving SSI in December 2000

The first major analysis group includes children and young adults identified in SSA administrative records as SSI recipients in December 2000. This group provides an excellent basis for addressing issues about the current caseload that are of interest to policymakers: the group reflects a fairly recent point in time, and no major changes in program rules affecting child and young adult recipients have occurred since December 2000.

Because of potential differences in the way various program features affect subgroups of interest to policymakers, the December 2000 cross section was designed to provide a reasonable representation of children and young adults receiving SSI by key variables such as age group, sex, and type of impairment. For example, comparisons by sex will be important, particularly in light of the contrast between the substantial overrepresentation of males among children receiving SSI compared with that of females among elderly beneficiaries. The type of impairment—specifically, mental versus physical impairment—is also of great interest. The distribution of children receiving SSI by age group (0–5, 6–12, 13–17) will play a role in some analyses, for example, when looking at special education, parental employment, or impairment differences among children awarded benefits at different ages. Child-to-adult transition issues are particularly important. Therefore, several subgroups were defined within each major analysis group to ensure that a sufficient number of sample cases would be available for comparative analyses.

To reduce the variability of the sample, these subgroups were proportionately represented through a sequential selection procedure rather than through explicit stratification by these subgroup characteristics. Overall, it was necessary to oversample 17- to 18-year-olds in December 2000 and recipients with a mental disorder. The analysis group of December 2000 recipients can be followed using administrative records to track important outcomes such as SSI participation, receipt of benefits from the Social Security Disability Insurance program, earnings, and mortality.

Children and Young Adults Receiving SSI in December 1996

The second major analysis group includes children and young adults identified in SSA administrative records as SSI recipients in December 1996. By including this subgroup, the survey provides a snapshot of the December 1996 caseload at the time of the survey in 2001–2002—about 5 to 6 years later. The survey also includes many characteristics not available from administrative records. Important policy questions relate to longer-term outcomes of SSI program participation, and the inclusion of this analysis group provides a limited longitudinal perspective on the dynamics of the SSI caseload. For example, this group allows researchers to track the experiences of the subset of the child caseload that was potentially affected by the 1996 welfare reform legislation. It may also facilitate other analyses of longer-term outcomes, including comparisons with the sample of SSI children who were not affected by that legislation. For studies regarding welfare reform, it is desirable to examine childhood redeterminations separately from age-18 redeterminations. Therefore, children aged 17–18 in December 1996 and those subject to childhood redeterminations were oversampled in the second analysis group.

Other Groups

In addition, SSA identified two groups—former SSI recipients and applicants who had been denied—as potentially important comparison groups for certain analyses. Former SSI recipients are defined as children who had applied for SSI benefits in 1992 or later and were not receiving benefits at the time of the survey or at welfare reform (1996) but who had received benefits for at least 1 month since 1992. Denied applicants are children who had applied for SSI benefits in 1992 or later but had never received benefits between 1992 and the time of the survey. The inclusion of these groups provides an opportunity for policy analyses that require information on children who had some contact with the SSI program in the past but did not receive benefits in either December 2000 or December 1996. For example, denied applicants can be used as a comparison group for SSI awardees.

Sample Size and Completion Rates

The unweighted and weighted sample sizes of the major groups described above can be seen in Table 1. A total of 5,006 interviews were completed with nonincarcerated children and young adults who were on the SSI rolls in December 2000; 5,033 interviews were completed with those on the SSI rolls in December 1996. The auxiliary group of former recipients (other than those on the rolls in December 1996) and denied applicants contains 1,767 completed interviews. Characteristics of the sample of children and young adults receiving SSI in December 2000 and the sample of children and young adults receiving SSI in December 1996—their age and sex, and their main health condition and general health status as reported by the respondent—are shown in Table 2.

Mathematica Policy Research, Inc., calculated survey completion rates. The unweighted completion rate is the number of completed interviews and ineligible sample members divided by the number of attempted interviews.³ The weighted completion rate represents the proportion of the estimated NSCF study universe with a completed interview. The overall weighted completion rate for the NSCF was 74.4 percent. The completion rate was highest for those who were SSI recipients at the time of the survey, largely because they were more likely to be located. The weighted completion rate for SSI recipients in December 2000 was 84.1 percent, compared with 76.8 percent for recipients in December 1996. Completion rates were lower for those not receiving SSI in December 2000 (71.1 percent) and those aged 17–18 in December 1996 (70.0 percent) (authors' calculations based on Potter and Diaz-Tena 2003, 32).

Design and Content of the Questionnaire

Two versions of the NSCF questionnaire were designed—one for children and one for young adults. For survey implementation, the two versions were combined into a single, computerized instrument with fairly complex skip patterns. Although the majority of questions are the same on each version of the questionnaire, which allows the two populations to be analyzed together for most issues, there are some important differences. Perhaps the most important difference has to do with the selection of the respondent. The parent or representative payee is the respondent for sample members under the age of 18 and for young adults aged 18 or older living at home. Young adults living in their own home are the respondent, but they can designate a proxy respondent if they are unable to complete the interview because of a disability. Another important difference between the children's questionnaire and the one for young adults is that the latter contains more detailed questions about preparations for the transition to adulthood. For example, both versions ask about involvement with special education programs and whether an individual education plan was prepared. The questionnaire for young adults also asks about involvement in vocational training programs and other programs geared toward providing youth-to-adult transition services for individuals with disabilities. It also gathers information about the young adult's employment, earnings, and other sources of income. (This information is also gathered for the parents if the young adult is living at home.) In contrast, the children's questionnaire focuses on the employment, earnings, and income sources of the parents.

The major sections of the questionnaire include disability status and functional limitations; health care utilization; health insurance; education and training; programs and services; impact on family, household members, and self; SSI experience; employment and earnings; work and child care; unearned income and assets; and housing and transportation. (For additional details about the content of each section of the questionnaire, see Box 1.)

The interviews took approximately 70 minutes on average to complete. Interviews with children who were living with both parents took slightly longer because employment and earnings data were collected for both parents. Interviews were significantly shorter with respondents who reported that the sample member was not disabled, did not receive SSI benefits, or made relatively little use of the health care system.

Although the questionnaire contains many questions that were designed to address topics specific to children and young adults with disabilities who receive SSI or topics of particular interest to SSA, many questions were drawn from existing surveys. For example, questions regarding employment, earnings, other income sources, and assets were taken from the Survey of Income and Program Participation. Questions regarding health status and functional limitations were drawn from the National Health Interview Survey. The purpose of using questions from these other sources is to preserve comparability between the NSCF and other major surveys. Nonetheless, comparisons between the NSCF and other surveys must be made with great caution because of different modes of interviewing and different sampling, weighting, and imputing techniques.

Data Collection

NSCF interviews were conducted between July 2001 and June 2002. Most interviews were conducted during the second half of 2001. A number of steps were taken to maximize the quality of the data collected in the NSCF. For example, the survey was designed for mixed-mode data collection to improve response rates and to avoid potential biases resulting from telephone-only surveys. Costly efforts to locate sample members were also undertaken to help achieve a high survey completion rate. These efforts are discussed briefly below. See the Appendix A for a discussion of the randomized experiment that was conducted to test different response incentives to boost cooperation among interview subjects who were located.

Mixed-Mode Data Collection

The primary method for collecting data was computer-assisted telephone interviewing (CATI). Of the 8,726 interviews (8,535 completed interviews plus 191 abbreviated interviews with sample members who were incarcerated at the time of the NSCF interview), 7,285 were completed by CATI. For sample members who could not be reached by telephone (for example, a correct telephone number could not be found) or who could not complete the interview by telephone (for example, the respondent's disability prevented him or her from responding by telephone, language barriers), field interviews were attempted using computer-assisted personal interviewing (CAPI). The remaining 1,441 interviews (16.5 percent of the total) were conducted using CAPI. The same version of the questionnaire was administered for both CATI and CAPI interviews to minimize potential mode effects.

Locating Sample Members

The biggest challenge in conducting the survey was locating sample members. Although SSI administrative data were used to identify the telephone number and address of sample members, about 70 percent of the sample had addresses that were no longer valid. Even among SSI recipients at the time of the survey, nearly 50 percent had invalid telephone numbers or addresses. Numerous methods were used to locate sample members, including searches of commercially available databases and ground work by NSCF field interviewers in the sample member's last known neighborhood. In the end, about 84 percent of the NSCF sample was located for interviewing. Despite the overall success of these efforts, nearly three-quarters of final nonresponses in the NSCF were because the sample member could not be found.

Cooperation of Sample Members

Once sample members had been located, obtaining their cooperation was very successful. Observations of a small number of interviews suggest that respondents were quite willing to answer the NSCF questions and that many seemed happy to "tell their story" to an interested listener. To encourage cooperation, letters sent in advance of the NSCF interview informed sample members that they would receive a response incentive of $10 upon completion of the interview (see Appendix A for details). The cooperation rate is the number of completed interviews as a percentage of the number of sample members who were located. The weighted cooperation rate was 90.6 percent overall and 93.7 percent among SSI recipients at the time of the survey.

An important issue with respect to the quality of the survey data is the pattern of survey noncompletion. Random patterns of noncompletion produce no selection bias in survey estimates, although they lower efficiency because they reduce the sample size. However, if survey noncompletion is systematically and substantially associated with relevant measured or unmeasured characteristics, concern arises about whether the sample is representative. Appendix B provides three (weighted) measures relevant to assessing potential selectivity bias by a number of characteristics. Those measures are the percentage who were located, the percentage who cooperated among those who were located, and the overall completion rate, which is the product of the first two sources of sample selectivity.

Overall, 81.9 percent of the sample members were located. Among those who were located, 90.6 percent completed the interview. The product of these two factors is the overall completion rate; 74.4 percent is a slightly adjusted version of this product.⁴ The data also show that there was substantially more variation in the proportion who were located than in the proportion who cooperated among those who were located. The rate located was relatively low for nonrecipients (those who, according to administrative records, had no payment) and those who had recently moved (78.2 percent and 77.2 percent, respectively) and relatively high for those with a grandparent as a representative payee (91.3 percent). Both sources of noncompletion contributed to the substantial overall difference (15 percentage points) in the completion rate between SSI recipients and sample members who were not receiving SSI at the time of the survey.⁵ They also contributed to the relatively high completion rate among sample members for whom a mental impairment was listed as the reason they applied for SSI, but the differences were less marked. The subgroup differences for most variables were relatively modest. Note that these data reflect differences before the nonresponse adjustment and weight adjustments that were used to address these issues, as discussed below.

Procedures for Imputations, Weights, and Variance Estimation

Mathematica statisticians developed imputation procedures, weights, and variance estimation procedures. Imputed values allow the use of observations with item nonresponse in statistical and econometric analyses. Weights are necessary for unbiased estimation of population means and other statistics. The variance estimation procedures developed by Mathematica are important tools for deriving unbiased estimates of the precision of statistical estimates (sampling variance) in the presence of the complex NSCF sample design, which makes the assumption of simple random sampling in variance estimation inaccurate.

Imputing Values for Missing Data Items

The rates of item nonresponse and the methods used to impute missing values are important indicators of the quality of the survey data files. In the NSCF, item nonresponse rates were generally low, and sophisticated imputation procedures were used. Administering the questionnaire by using computer-assisted instruments substantially reduced the extent of item nonresponse. With computer-assisted instruments, interviewers are automatically routed through the questionnaire, questions cannot be left blank, and automated checks of consistency and reasonableness are implemented. Nevertheless, missing data are sometimes generated because questions are mistakenly not answered and respondents sometimes provide "don't know" responses or refuse to answer a question. In addition, the value of derived variables is missing if any of the component variables were subject to item nonresponse.

Because of the length of the NSCF questionnaire and the number of variables on the data file, the cost of imputing values for missing data on all variables appeared to be prohibitive. Mathematica therefore focused its imputation efforts on variables for earned and unearned income, assets, and out-of-pocket expenditures. Not only are those variables of great importance for planned analyses with the NSCF data, but item nonresponse was relatively high for some of them. In addition, summary variables, such as total income and assets, were derived from a number of survey items, each one of them a potential source of a missing value for the summary measure. For these reasons the potential payoffs of high-quality imputations were especially large for data items on income, assets, and out-of-pocket expenditures. In contrast, variables related to the sample member's type of impairment and functional limitations, for example, are poor candidates for imputation because their correlation with other observed characteristics is inherently complex.

Altogether, 97 variables were imputed using one or more of the following methods: logical imputation, hot-deck imputation, and regression-based imputation. For each variable for which missing values were imputed, an imputation flag was created and added to the NSCF database. For each observation, this flag indicates whether or not the value for that observation was imputed and the imputation method that was used. Potter and Diaz-Tena (2003) provide additional details on the imputation methods. Analysts interested in using variables for out-of-pocket expenditures, health care utilization, employment, unearned income, and assets are encouraged to consult Potter and Diaz-Tena (2003, Table IV.2) to assess both the extent of item nonresponse and the resulting portion of missing values for derived variables. The NSCF data documentation also provides detailed information related to item nonresponse and imputations for other survey variables in the file.

Weighting the Sample

Mathematica developed analysis weights for the NSCF using a three-step process. First, they developed sampling weights, which are simply the inverse of the probability of selection for each sample member. The sampling weights thus account for both the selection of primary sampling units and the individual selection probabilities within each stratum. The initial sampling weights ranged from 29.7 to 510.7.

Second, they made two adjustments to the initial sampling weights for nonresponse: one was based on the ability to locate the sample member, and the other was based on the cooperation of sample members once they had been located. Weighted logistic regression models were estimated to develop a propensity score for locating the sample member and a propensity score for response among those located. Covariates used in the logistic regression models include the sample member's personal characteristics, current and previous SSI status, geographic region and urban/rural status, and the personal characteristics of an adult living with the sample member. Potter and Diaz-Tena (2003) provide detailed results from the logistic regression models. The nonresponse adjustment is the inverse of each propensity score, which was then multiplied by the initial sampling weight to obtain the response-adjusted weight.

Finally, Mathematica poststratified the response-adjusted weights to match the weighted sums across 11 analytic populations and by sex, with control totals for groups calculated from SSA administrative data. The weight variable included on the data file provides the final analysis weight for each sample member, reflecting the adjustments described above.

Variance Estimation

The NSCF sample design uses a complex, multistage procedure for sample selection. As a result, the standard variance calculations assuming a simple random sample (SRS) will underestimate the true variance. Therefore, Mathematica developed two sets of variance estimation specifications for the NSCF.⁶ Sample code for implementing each procedure is included in the NSCF User's Manual (Gillcrist and Edson 2004).

Means and standard errors calculated using the Taylor-series linearization for several NSCF variables of interest are shown in Table 3, which also presents the Taylor-series standard errors as a percentage of the SRS standard errors. The SRS standard errors are smaller than the standard errors that account for the complex NSCF sample design and thus would lead to overly optimistic conclusions about the precision of estimates. Accounting for the complex sample design clearly makes a substantial difference in most cases, and the magnitude of the difference varies greatly depending on the particular variable in question. If the bias involved in using the uncorrected SRS estimate of standard errors was roughly similar across the board, a single value of the design effect could be used as a rough way to correct for the complex survey design. Although this assumption may be correct in some surveys, the nature of the NSCF sample design would make it incorrect in this survey. For example, Table 3 shows that the SRS estimate of standard errors underestimates the true standard errors for health insurance coverage among children receiving SSI in December 2000 by a factor of 1.25; in contrast, the corresponding underestimate of the percentage who are black is much larger, 4.08. Because of the importance of accounting for the complex NSCF survey design, analysts should use appropriate software to calculate corrected standard errors. Potter and Diaz-Tena (2003) explain the application of two widely used techniques—the Taylor-series linearization procedure and the balanced repeated replication (BRR) procedure—for the NSCF data set.

Data Products

Mathematica prepared several major data-related products for use by NSCF researchers—a master data file, a User's Manual, a report on the quality of the NSCF data, a report on weighting and imputations, and a report comparing data from the NSCF with data on children receiving SSI from other national surveys. They also prepared a public-use version of the NSCF data set and documentation that is available to researchers studying disabled children enrolled in the SSI program. The data set is available at http://www.socialsecurity.gov/disabilityresearch/nscf.htm.

SSA's master data file contains all of the information collected by the NSCF. It also contains administrative data items that were used to design and control the sample and numerous derived and recoded variables to make the data more user friendly. For example, responses to the open-ended questions on type of impairment were recoded following the International Classification of Diseases (ICD) coding scheme and the SSA diagnosis coding scheme. Administrative data from other SSA systems can be appended to the NSCF as needed for individual SSA projects.

The User's Manual prepared by Mathematica (Gillcrist and Edson 2004) is a critically important resource for successfully using the NSCF for research and analysis. In addition to the detailed codebook, the User's Manual includes a discussion of the NSCF sample design, the questionnaire design and content, and procedures for collecting data, constructing and editing variables, and estimating variances. It also provides instructions for subsetting the data to identify various subpopulations of potential interest to researchers. Used in conjunction with three other reports—on data quality (Gillcrist, Kasprzyk, and Mitchell 2004); on weighting, nonresponse adjustments, and imputation (Potter and Diaz-Tena 2003); and comparing estimates from the NSCF to estimates for children receiving SSI from other national surveys (Ireys and others 2004)—users will be prepared to study the wealth of information on children receiving SSI that is available in the NSCF.

Appendix A. Experiment with Response Incentives

Offering an incentive to participate in surveys like the National Survey of SSI Children and Families is standard practice. In most cases, a small payment by check is offered in advance to sample members and is paid once they have completed the survey. In addition to this standard approach, Mathematica proposed an experiment to test two alternative incentives—a point-of-sale (debit) card and a prepaid telephone card. About 70 percent of NSCF sample members were randomly selected to receive the standard check payment, 15 percent to receive a debit card, and the remaining 15 percent a prepaid telephone card. The experiment did not include a no-incentive control group, based partly on the presumption that the offer of an incentive payment cannot produce a negative effect on the willingness of sample members to cooperate. Sample members were sent a letter in advance telling them which type of payment they would receive and its value ($10).⁷ Mathematica tracked the completion rate, usage rate, administrative issues, and costs of each type of payment.

For two reasons, the key outcome of interest from the government's perspective is the effect on the final survey completion rate. First, increasing the survey completion rate is believed to improve the overall quality of the survey product, presumably by reducing the potential for selectivity bias that arises from patterns of survey noncompletion.⁸ Second, response incentive payments, if successful, may be highly cost-effective relative to other methods of increasing survey completion rates. For example, shifting from a mixed mode of data collection to face-to-face interviews may be rather costly.

The final completion rate was statistically significantly higher for sample members who received a debit card (79 percent) than for those who received a telephone card (75 percent). The differences between the response rate for debit cards and the rate for checks (77 percent) and between the rates for checks and telephone cards were not statistically significant. Given this overall pattern and assuming that the offer of telephone cards did not reduce the willingness to cooperate relative to the hypothetical "no incentive payment" counterfactual, these results suggest that the use of debit cards was successful and should be regarded as the preferred method. Although the exact magnitude of this effect is highly uncertain, based on this evidence the best point estimate is an increase of 4 percentage points attributable to the use of debit cards versus telephone cards. This difference is substantial, suggesting that using a debit card as an incentive is clearly cost-effective.

Our conclusions differ from those reached by Mathematica's Mitchell, Lamothe-Galette, and Potter. They conclude (2003, 4) that "In the end, POS [debit] and telephone cards did not perform as dependably or economically overall as checks." The differences stem primarily from differences in the key outcomes of interest. Whereas the Mathematica authors focused on operational difficulties and the unit cost of the three interventions from the research firm's perspective, we focus on the key outcome of interest for the government—the net effect on survey completion rates.

In addition, there are three areas of disagreement between our interpretation and theirs. First, with respect to cost, Mitchell, Lamothe-Galette, and Potter (2003, 3) conclude that "costs loom large" on the basis of differences between their estimate of the total unit cost of debit cards ($12.21) and checks ($11.20). We argue that the $1 difference in unit cost is minuscule in comparison with the potential government savings from using this low-cost method to boost survey completion rates as opposed to using more expensive methods (for example, shifting from telephone to field interviews). In addition, Mitchell and colleagues ignore cost differentials arising from differences in respondents' use of the three types of incentive payments. Once the probability of using the incentive payment is considered, the pattern of unit cost estimates dramatically shifts in favor of the debit cards.

Second, Mitchell, Lamothe-Galette, and Potter take the relatively low usage rate of debit cards (47 percent) versus checks (85 percent) as prima facie evidence of the ineffectiveness of using debit cards. This interpretation is problematic because these types of experiments involve a "deadweight loss" (that is, behavior may be unaffected by the response incentive). If a person would complete the interview whether he or she received an incentive payment or not, there is no gain in completion rate attributable to the incentive—the $10 incentive payment is wasted. It is the marginal respondent—the one whose behavior is positively altered by the offer of the incentive—who counts. In this study, the deadweight loss may be as high as 75 percent. The experimental results suggest that the debit cards were more successful with marginal respondents.

Finally, our interpretation of the statistical evidence is different. In our interpretation the key result is that the completion rate using debit cards (79 percent) was statistically significantly higher than the rate with telephone cards (75 percent), which can be taken as an upper-bound estimate of the "no response incentive" counterfactual. The fact that neither number is significantly different from the response rate using checks (77 percent) suggests that one cannot reject the hypothesis that the true effect of the debit cards may be only 2 percentage points. That would still be a substantial gain in completion rates given the low cost of any of the three interventions. Our interpretation is also supported by the fact that the completion rate using debit cards was 5 percentage points higher than the rate with telephone cards at the end of the first month of data collection—the time when behavior is most likely to have been affected by the experiment with incentives. By contrast, there is no difference between the rates for checks and telephone cards at the end of the first month. The completion rate among debit card recipients clearly exceeds that of check and telephone card recipients for months 1 through 11 of data collection, as reported by Mitchell, Lamothe-Galette, and Potter (2003, Table 1).

Appendix B.   Location and Completion Rates

Table B-1 provides weighted location and completion rates for NSCF sample members by characteristics of the sample members. The table was adapted from Table III.3 of Potter and Diaz-Tena (2003).