When Every Dollar Counts: Comparing Reported Earnings of Social Security Disability Program Beneficiaries in Survey and Administrative Records

Introduction

The Social Security Administration (SSA) aims to make the best use of administrative and survey data for research and program operations, particularly in measuring employment and earnings. Both sources offer advantages in monitoring program operations, capturing beneficiary characteristics, and measuring the effects of demonstration projects. However, information on how the sources might produce differing estimates is limited. A better understanding of these differences might be critical to identifying important evaluation outcome measures and/or designing interventions to customize supports for SSA disability program beneficiaries.

This article compares employment and earnings outcomes for disability program beneficiaries based on linked data from SSA's National Beneficiary Survey (NBS) and administrative records from the agency's Master Earnings File (MEF). NBS respondents are a nationally representative sample of people who received benefits from Social Security Disability Insurance (DI), Supplemental Security Income (SSI), or both. We use NBS data to construct annual measures of employment and earnings for 2003 through 2005 that we then compare with linked annual employment and earnings reports in the MEF (which are not accessible to the general public). We present detailed comparisons of survey and administrative data on employment and earnings for DI beneficiaries, comprising those who receive DI benefits only and those who receive DI and SSI benefits concurrently. We also summarize similar comparisons for individuals who receive SSI payments but no DI benefits, and present detailed tables for that population in Appendix A.

We find that employment rates and earnings levels are higher in administrative records than in survey reports for SSA disability program beneficiaries overall and for all major sociodemographic subgroups. The proportional differences between the administrative and survey records can be substantial because the employment rates of DI and SSI beneficiaries are low. For example, employment rates for DI beneficiaries in our sample were about 40 percent higher in administrative records than in survey data, although the absolute difference is only 5.6 percentage points (19.2 percent versus 13.6 percent). Among subgroups, we find that the largest relative differences between administrative and survey data are for beneficiaries with survey-reported employment rates and earnings levels that are lower than the average survey-reported employment rate and earnings level of DI beneficiaries overall. For example, the survey and administrative data differ significantly for beneficiaries with a musculoskeletal primary disabling condition,¹ and that subgroup's employment rate and earnings are considerably lower than those of DI beneficiaries overall. Absolute differences between two relatively low employment rates or earnings levels would thus be proportionally greater than similar absolute differences among subgroups with higher employment rates or earnings.

Background

Measuring employment outcomes poses challenges for both survey and administrative data collection. Several studies suggest using both survey and administrative data to identify potential underreporting associated with one source or the other (Abowd and Stinson 2011; Barnow and Greenberg 2014; Ford and others 2014). Using both sources can be especially advantageous for data on subpopulations that may be underrepresented or prone to reporting error in one of the sources. As Davies and Fisher (2009) noted, researchers have used matched survey and administrative data to assess the accuracy of the survey data and used the resulting information to adjust for error in the survey-based estimates.

Several studies have shown that administrative data produce higher estimated employment rates and earnings levels than survey data (Coder and Scoon-Rogers 1996; Pedace and Bates 2000; Gottschalk and Huynh 2005). Most authors have speculated that administrative earnings records are higher because survey respondents tend to underreport earnings from seasonal or temporary jobs (Moore, Marquis, and Bogen 1996; Kornfeld and Bloom 1999). For example, survey respondents might be prone not to recall earnings from each of multiple part-time or seasonal jobs, whereas those earnings would be recorded in the administrative data whenever they were provided by employers. Incomplete recall by respondents might also explain why administrative estimates of earnings tend to be higher than survey-based estimates for short-term, marginal, or overlapping jobs (Bridges, Del Bene, and Leonesio 2003; Hurd, Juster, and Smith 2004; Abraham and others 2009) as well as for highly variable or unpredictable sources of income. Alwin, Zeiser, and Gensimore (2013) found that administrative data–based estimates for irregular earnings, such as those from short-term employment, were higher than estimates from surveys. Bound and others (1994) also speculated that administrative data–based earnings estimates are higher than survey estimates because respondents might have trouble accurately recalling earnings for hourly work, which are automatically recorded in the administrative data.

In general, proportional differences between administrative and survey data on earnings tend to be larger among subgroups with lower earnings—again, with estimated earnings based on administrative data tending to be higher than corresponding estimates from the survey data. The lower estimates in survey data could reflect recall error for work at sporadic or multiple jobs (Rodgers, Brown, and Duncan 1993; Pischke 1995; Gottschalk and Huynh 2005). Recall error aside, people who work more sporadically tend to have lower earnings than other workers, and a $50 difference between two earnings estimates for a low earner is proportionally greater than a $50 difference between two earnings estimates for a higher earner.

Information on differences in administrative and survey records by demographic or disability characteristics is scarce. A key challenge in measuring these differences is that the sample sizes of many of the subgroups in national surveys that are linked to administrative data are small (Kornfeld and Bloom 1999; Monti and Gathright 2013). For example, many of the studies mentioned above draw on data from national surveys such as the Current Population Survey and the Survey of Income and Program Participation that have been linked to administrative records. Although those surveys have large samples overall, the samples of specific subgroups—such as DI and SSI beneficiaries—are limited.

Program Descriptions, Data Sources, and Methodology

To qualify for DI or SSI benefits, an applicant must demonstrate an inability to engage in substantial gainful activity (SGA) because of a medically determinable impairment that is expected to last at least 12 months or to result in death. SGA is determined by a monthly earnings level; each year, SSA adjusts the SGA definition, if needed, based on changes in the national average wage. In 2018, SSA defines SGA as monthly earnings of $1,180 or more for nonblind applicants and $1,970 or more for blind applicants.

DI and SSI eligibility rules differ in ways that might influence employment outcomes. DI eligibility is contingent on having sufficient levels of recent and lifetime Social Security–covered employment. By contrast, SSI is a means-tested program, with eligibility subject to strict income and asset limits. Individuals may qualify for both programs if their income (including DI benefits) and assets are low enough to meet the SSI limits. Perhaps not surprisingly, DI beneficiaries tend to be older and have more extensive work histories than SSI recipients (Mamun and others 2010). Finally, there are important differences in DI and SSI benefit reductions resulting from work earnings. DI beneficiaries face a “cash cliff” (that is, benefit payments stop altogether) for earnings above a certain threshold, whereas SSI payments decline incrementally, generally being reduced by $1 for every additional $2 of earnings.

Previous studies have examined the employment and earnings outcomes of DI beneficiaries and SSI recipients using survey and administrative data separately. Livermore (2009) documented the work activities of all DI beneficiaries and SSI recipients using 2004 survey data and found that 13 percent reported working during the previous year and 9 percent were working as of the date of the interview. A much higher proportion had aspirations of work: 40 percent of working-age disability program beneficiaries reported having work goals or expectations. Mamun and others (2010) used administrative earnings records to find that 12 percent of disability program beneficiaries had at least $1,000 in earnings in 2007.

Our analysis extends the literature by comparing the survey data on employment and earnings from the NBS to similarly constructed measures in the administrative data. The NBS, which is sponsored by SSA and was developed and implemented as part of the agency's Ticket to Work (TTW) program evaluation, collects cross-sectional data from a national sample of DI beneficiaries and SSI recipients and a sample of TTW program participants. Its primary purpose is to provide information on work-related activities of SSI and DI beneficiaries, particularly as they relate to TTW implementation. The survey collects information on respondent employment status, employment services used in the past year, disability status, experience with SSA programs, health and functional status, health insurance, earnings, income support, and sociodemographic characteristics (Livermore and others 2011). Proxy respondents are permitted for individuals whose disabilities make the interview prohibitively challenging and for people who cannot be contacted and interviewed directly.

We pooled data from three rounds of the NBS—fielded in 2004, 2005, and 2006—to obtain a larger sample of beneficiaries with earnings than a single round would provide. Our total pooled sample includes 7,987 observations for individuals aged 21–64 at the time of the NBS interview. We generated statistics using survey weights (adjusted for pooling); all standard errors used to compute tests of statistical significance account appropriately for the complex NBS sampling design.

We used the MEF to generate administrative estimates of employment and earnings. The MEF contains annual earnings data from Internal Revenue Service Form W-2, quarterly earnings records, and annual income tax forms (Olsen and Hudson 2009). Annual earnings are defined as the maximum of (1) Social Security–taxable wages and self-employment earnings (capped at $127,000 in 2017), (2) Medicare-taxable wages and self-employment earnings, or (3) total compensation (which captures earnings taxable by neither Social Security nor Medicare). Earnings not reported on a W-2 are not included in the underlying data and are thus not included in the analysis. Additionally, earnings from known “third-party payers”—that is, insurance, pensions, and similar nonwork earnings—are subtracted from our earnings measure.

We linked MEF data to survey results for each NBS respondent. Survey questions covered the year prior to the interview; for example, the 2004 NBS covered employment and earnings for 2003. Therefore, we merged 2003 MEF earnings data with the 2004 NBS results, and so forth, to produce comparable survey- and administrative data–based estimates.

We developed an annual frame for definitions of employment and earnings to facilitate comparisons between the survey and administrative data. The choice of an annual frame was necessary to allow the survey data to be consistent with the annual reporting in the administrative data described above. The NBS includes questions about earnings from each respondent's work that lasted for at least 30 days in the prior year. Hence, we defined annual employment as the presence of any earnings from a job the NBS respondent reported working that year. The sum of earnings for each reported job establishes the annual earnings measure.² The earnings values in both the survey and administrative data are reported in nominal dollars (that is, they are not adjusted for inflation).

The earnings estimate we derive from the administrative data is likely to be higher than that from the survey data for two reasons. First, the survey questions potentially impose a burden on respondents to recall detailed information about all jobs in which they received earnings during the past year, which might be more difficult for those who held multiple jobs. Respondents who were frequently paid on a piece-rate basis rather than by the hour or with an annual salary might face similar recall challenges. Second, given the potential benefit reductions for SSI and DI beneficiaries with work earnings, NBS respondents may choose to underreport job and earnings information out of concern that full disclosure in a survey sponsored by SSA could jeopardize their disability program benefits.

Table 1 provides descriptive statistics for our sample of DI beneficiaries. These statistics provide context for the types of jobs held, the resultant earnings, and the nature of employment overall for this population. The majority of individuals in the sample (74.8 percent) received only DI benefits; the other 25.2 percent received SSI payments as well. Of the entire sample, 20.7 percent had a musculoskeletal primary condition, 18.3 percent had a psychiatric condition, 5.7 percent had an intellectual condition, 3.4 percent had a sensory condition, and 47.5 percent had conditions of other types. Most beneficiaries were middle-aged or older: 84.7 percent were older than 39 and 59.5 percent were older than 49. The division by sex was almost equal, with slightly more women than men. Nearly three-fourths of the sample was white only, 18.2 percent was black or African American only, and the remainder represented other racial groups. The majority of survey respondents self-reported data; less than 20 percent of the interviewees were proxies.

In Tables 2–5, we examine differences in employment and earnings statistics between administrative and survey data. In each table, we find that administrative data produce higher estimates than survey data. We measure absolute differences in percentage-point or dollar terms by subtracting the survey estimate from the administrative estimate. We measure proportional differences in percentage terms by dividing the absolute difference by the survey estimate. The absolute differences are relatively uniform across beneficiary subgroups while the proportional differences offer the context of relative magnitude, which is relevant to understanding the extent of the disparity between the survey estimate and the administrative data. The proportional difference is therefore particularly useful for researchers interested in understanding the potential undercount of earnings if survey records are the only available source of earnings information.

Findings

Table 2 presents employment-rate estimates for DI beneficiaries based on administrative and survey data, overall and by subgroup. For the full sample, we find a higher employment rate in the administrative data (19.2 percent) than in the survey data (13.6 percent). Given the relatively low employment rate of this population, the absolute difference of 5.6 percentage points means that the administrative estimates are 41.3 percent higher than the survey estimates.

The absolute differences between administrative data– and survey-based employment rates are relatively consistent among subgroups. With two minor exceptions (the “missing data” disabling condition and “other” race), administrative estimates of employment are from 3.6 to 7.1 percentage points higher across all subgroups. However, the proportional differences vary substantially because the employment rates themselves can vary by subgroup. The largest proportional differences (70 percent or more) are seen for subgroups that have some of the lowest employment rates (beneficiaries with musculoskeletal conditions and those aged 50 or older). Notably, this pattern would hold if we defined “proportional difference” as the absolute difference divided by the administrative estimate. In any event, although the absolute differences are relatively small, the proportional differences for many of the subgroups are quite large. In summary, we find that the subgroups of beneficiaries with the lowest estimated employment rates (regardless of the data source) tend to exhibit higher proportional differences between the survey- and administrative data–based estimates of their employment rates. This is consistent with prior literature citing potential issues related to recall error or underreporting of earnings in surveys.

Table 3 compares administrative data– and survey-based mean annual earnings estimates for DI beneficiaries overall and by subgroup. The data cover all DI beneficiaries regardless of employment status and hence include many individuals who have no earnings. The average earnings of the overall sample are $1,125 based on the administrative data and $514 based on survey data. Consistent with the employment-rate results, we find that estimated earnings based on administrative data are higher than those based on survey data for all subgroups. The consistency in patterns between Tables 2 and 3 is not surprising given that the definitions for the employment and earnings measures are directly related. In absolute dollar amounts, the differences in the earnings estimates by subgroup range from $262 (individuals receiving concurrent DI/SSI benefits) to $778 (beneficiaries with a sensory condition).

Repeating the patterns seen for employment rates, subgroups with the highest average earnings have most of the lowest proportional differences between administrative data– and survey-based estimates. For example, higher-earning subgroups include those with sensory conditions, for whom we find a 57.6 percent relative difference between administrative and survey data; and beneficiaries aged 21–29, for whom we find a 43.1 percent relative difference. Conversely, subgroups with lower average earnings tend to have higher proportional differences, including beneficiaries with musculoskeletal or “other” impairment conditions, those aged 50–64, and those who are Hispanic (all with relative differences exceeding 160 percent).

In Table 4, we restrict our analysis to beneficiaries who have nonzero annual earnings reported in both administrative and survey data. This table provides a useful contrast to Table 3 because it includes only those survey respondents who recall having some earnings. Hence, all else being equal, we would expect survey-estimated earnings to have closer concordance with administrative data for these beneficiaries than we would see for a sample that might include individuals who had earnings they did not recall.

As in Table 3, administrative records produce higher average-earnings estimates than survey records do ($6,402 versus $4,181, respectively, for employed beneficiaries overall); but not surprisingly, the proportional difference (53.1 percent) is much lower than that in Table 3 (119.0 percent). In a change from previous tables, we find that the relationship between average earnings levels and proportional differences between administrative data– and survey-based estimates varies across subgroups in Table 4. For example, employed beneficiaries with musculoskeletal disabilities had both the highest MEF-based mean earnings and a very high proportional difference between the data sources, with administrative data–based earnings that were 79.5 percent higher than the survey-based estimates.³ Other higher-earning subgroups, however, exhibit the opposite relationship: For beneficiaries with sensory disabilities, administrative earnings estimates were only 31.1 percent higher than survey estimates. Among the subgroups with the lowest earnings, beneficiaries with intellectual disabilities had the smallest difference between data sources in absolute terms but the proportional difference of 38.5 percent is in the middle range of proportional differences among the subgroups.

We also examined median earnings estimates to assess whether the differences between survey and administrative data were consistent with the patterns seen for mean earnings (not shown). Overall and by subgroup, the median absolute differences were similar in magnitude to the mean differences. For some subgroups, we observed some proportional differences in the median earnings estimates that departed substantially from the mean proportional differences. However, those departures arose because the survey medians were substantially lower than the survey means, given that many beneficiaries had low earnings.

Finally, we examined whether the differences shown in Table 4 persisted if we measured earnings reported in either administrative or survey data, but not necessarily in both (not shown). Specifically, we ran the same estimates shown in Table 4 for (1) any earnings reported in the survey and (2) any earnings reported in the administrative records. As with our findings described above, we found that mean earnings estimated with administrative data were consistently higher than those estimated with survey data.

Results for SSI-Only Recipients

Table 5 provides a summary comparison of survey- and administrative data–based earnings differences for the total DI beneficiary sample and the SSI-only recipient sample. (Appendix Tables A-1 through A-4 repeat Tables 1 through 4 for the SSI-only sample.) SSI-only recipients have lower employment rates than do DI beneficiaries—13.6 percent versus 19.2 percent, respectively, based on administrative data. The difference based on survey data is smaller (11.6 percent versus 14.2 percent).

As with the patterns for DI beneficiaries discussed above, the administrative estimates of employment rates and earnings for SSI-only recipients are higher than the survey estimates. Specifically, administrative records for SSI-only recipients consistently show a higher employment rate (14.2 percent) and average earnings ($786) than survey records do (11.6 percent and $603, respectively).

However, administrative- and survey-reported earnings of SSI-only recipients differ less than those of DI beneficiaries, particularly when viewed in proportional terms. For example, average earnings among all SSI-only recipients is 30.3 percent higher in administrative sources than in survey sources; for all DI beneficiaries, the administrative estimate is 119.0 percent higher than the survey estimate. These widely differing proportions might reflect several substantive differences between DI and SSI, including differences in the characteristics of the beneficiary populations and in program requirements for reporting earnings. For example, SSI recipients tend to be younger and have considerably less work experience than DI beneficiaries. In addition, SSI recipients are required to report their earnings more frequently than DI beneficiaries do, because SSI has lower caps on allowable earnings and lower maximum benefit levels.

Discussion

We find that estimated employment rates and earnings levels are consistently higher in administrative data than in survey data, particularly among beneficiary subgroups with employment rates below the national beneficiary average. Further, the divergence in administrative data– and survey-based estimates is greater, in both absolute and proportional terms, for subgroups with lower survey-reported employment rates than that for beneficiaries overall.

We speculate that survey respondent recall error is the main factor driving the differences. Complete and accurate recall may be particularly difficult for individuals who work part-time and/or part-year. Furthermore, although all respondents are guaranteed confidentiality, some may be wary of potential negative consequences of fully disclosing their earnings. Hence, some respondents may be reluctant to provide accurate earnings information if they believe that it could jeopardize the benefits they receive from the same agency that sponsors the NBS.

The findings here can inform decisions about when and where to make the best use of survey questions related to employment and earnings. For example, administrative reports on earnings might be a valid substitute for survey data when the variable of interest is annual earnings. This substitution might be desirable to free up questionnaire space when a limited amount of survey data can be collected within a project's scope or budget. Such considerations may be especially useful for projects in which administrative records on annual earnings are accessible to program staff and can be linked to survey respondents.

These findings can also inform policymakers considering whether to use surveys to identify target populations for future demonstration projects to support working beneficiaries. Our findings indicate that administrative records might provide more reliable information on employment and earnings, but they do not diminish the need for survey information on measures not recorded in administrative data (for example, health status) and even on some employment-related measures.

Although administrative data appear to offer greater precision than survey data in measuring annual employment and earnings, they are extremely limited in measuring other characteristics of employment. For example, administrative data do not include information on hours, wage rates, monthly earnings, occupations, or the specific time period or duration of employment. Thus, researchers and policymakers can benefit from levering both sources to improve overall data quality and expand the coverage—from any source—for important employment-outcome estimates.

Future analyses could examine differences between administrative data and estimates from other SSA-sponsored surveys of beneficiaries. With the survey-sponsor variable held constant, such work could isolate certain aspects of the other surveys, such as an emphasis on employment or a linkage to a specific employment-focused demonstration project, and examine their potential effects on reported employment and earnings. Focusing on surveys conducted as part of employment-focused demonstration projects could provide insight into whether treatment- and control-group differences in reported earnings change with differing survey contexts. Recent examples of such SSA-sponsored surveys have been conducted for the Accelerated Benefits Demonstration, the Youth Transition Demonstration, and the Mental Health Treatment Study. Each survey focuses on employment with its own context. Comparing the results of these surveys with the administrative data could thus add contextual depth to our understanding of the differences in employment-outcome estimates.

Appendix A. Tables for SSI-Only Recipients