PERSPECTIVES: The Alignment Between Self-Reported and Administrative Measures of Disability Program Application and Benefit Receipt in the Health and Retirement Study

Introduction

Understanding the circumstances that lead to disability program application and the postapplication outcomes for beneficiaries and denied applicants is important for assessing the effects of changes to the benefit determination process, program rules, and benefit generosity. The Social Security Administration (SSA) collects from applicants only the information that is necessary to make benefit determinations or to administer monthly benefits. This information includes work history, education, health status, income, and assets, but does not always include applicants' living arrangements, other income sources, or receipt of other forms of public or private assistance. SSA periodically requires beneficiaries to update the information on their health condition (to determine whether benefit eligibility will continue) and earnings (if they exceed the level that denotes substantial gainful activity), but, in general, the information available to the agency on beneficiary characteristics is limited.

Therefore, researchers and policymakers turn to other sources of information on disability program applicants and beneficiaries to understand their needs more fully. For example, surveys collect detailed information on a range of subjects including respondent disability status and benefit receipt. Many surveys compile only their respondents' self-reported information, but some link respondents' self-reported information to administrative data. SSA has used such data linkages to analyze its disability and retirement programs. Recent research has capitalized on administrative-data linkages to better understand the accuracy of self-reported survey data and to identify the best way to combine information from the two potentially differing sources. As we discuss later, the research findings vary by disability program and survey (for example, Meyer and Mittag 2019; Chen, Munnell, and Sanzenbacher 2018; Bee and Mitchell 2017).

In this article, we compare survey data from the University of Michigan's Health and Retirement Study (HRS) with administrative data from SSA on Social Security Disability Insurance (DI) and Supplemental Security Income (SSI) applicants and beneficiaries. The HRS is a nationally representative longitudinal survey of noninstitutionalized adults aged 51 or older. The study started in 1992 and is known for the richness of its data on the health, income, and other characteristics of older adults. After entering the survey sample, respondents are interviewed every other year until they die or otherwise exit the study. To maintain a consistent age distribution, the HRS replenishes the survey sample with a new cohort of respondents aged 51–56 every 6 years. During the initial interview and each subsequent survey wave, respondents report their own program participation history, including DI and SSI applications filed and benefit receipt. Respondents are periodically asked for consent to have their information linked to SSA records on earnings and benefits. Not all HRS respondents consent to the linkage, but for those who do, it is possible to compare their self-reports with their administrative records.

There are several reasons why comparing the HRS results with administrative records is important, even with the breadth of existing literature based on other survey data sources. First, health shocks occur more frequently with age (Smith 2003), so HRS respondents will be more likely to apply for and receive disability program benefits than the younger adults who are typically included in other national surveys. Second, the programs administered by SSA offer a critical—but potentially confusing—mix of benefits to an individual in the years just before retirement. Old-Age and Survivors Insurance (OASI) retired-worker benefits can be claimed on reaching age 62, and DI benefits are available to insured workers at any age until the worker's full retirement age (FRA—65 to 67, depending on year of birth), when they automatically convert to OASI benefits. Individuals may qualify for SSI payments at any age, although the eligibility requirements change at age 65. Thus, respondents misreporting their program participation may be more common in the HRS than in other national surveys.

This study examines the accuracy of self-reported disability program participation as collected in the HRS survey and the potential strengths and limitations of using matched administrative data. We answer the following questions:

• What share of HRS respondents consented to the data linkage to measure DI and SSI application and receipt? How did the likelihood of consenting vary by cohort and over time?
• How do HRS respondents who consented to the administrative-data linkage differ from those who did not, both sociodemographically and in their reporting of DI and SSI application and receipt?
• How does the prevalence of DI and SSI application and receipt vary by cohort, time, and age? How do the aggregate rates differ if a researcher uses the self-reported data instead of the administrative records, and what factors might explain the difference?
• Among respondents who consented to the linkage, how accurate are self-reports, and what are the characteristics of respondents whose reports are incorrect?

The majority of HRS respondents have consented to administrative-data linkages, but rates of consent differ by survey cohort and over time. This, in part, reflects changes to how the HRS has obtained consent over the years. Consistent with earlier studies (Olson 1999; Haider and Solon 2000), we find that demographic, employment, and health-related characteristics differ between respondents who consent to the linkage and those who do not.

We also find that the share of respondents who report having DI or SSI application or receipt is generally lower than SSA records indicate. The pattern of underreporting is generally consistent across respondent age groups (regardless of their cohort or the survey year); however, there is no consistent pattern across survey cohorts. As we saw with respondents consenting and not consenting to a data linkage, the demographic, socioeconomic, and health characteristics of respondents whose self-reports diverge from their administrative records differ from those whose self-reports agree with SSA data.

Background: The Accuracy of Self-Reported Disability Program Participation in National Surveys

Surveys offer a depth of information that is not available in administrative sources alone. Longitudinal surveys can provide a detailed look at the characteristics, outcomes, and trajectories of individuals before, during, and after application for or receipt of DI or SSI benefits. Davies and Fisher (2009) documented potential uses of SSA-and-survey linked data, while also assessing earlier work (Huynh, Rupp, and Sears 2002; Koenig 2003) on DI and SSI reporting in surveys versus administrative sources. They summarized the literature based on analysis of data from older adults in the 1990s as showing that Current Population Survey (CPS) respondents slightly underreported their Old-Age, Survivors, and Disability Insurance (OASDI) benefits and significantly underreported their SSI payments, while Survey of Income and Program Participation (SIPP) respondents slightly overreported their OASDI income and were mixed on reporting their SSI payments. Schimmel Hyde and others (2018) used 2008–2009 CPS and SIPP data for a working-age population and found that, compared with the administrative record, many beneficiaries misreported their beneficiary status and benefit income. The authors also found that discrepancies appeared to be larger than those in the earlier studies cataloged by Davies and Fisher.

More recent research has sought to augment survey self-reports with administrative data to better understand income received from government programs more broadly. Beginning with Meyer, Mok, and Sullivan (2015), these studies suggest that misreporting is not uncommon and that underreporting is more common than overreporting. Meyer and Mittag (2019) found that income from government benefits among working-age CPS respondents was dramatically understated. Bee and Mitchell (2017) similarly documented underreporting of income among older adults in the CPS, driven primarily by misreported defined benefit pension income and retirement account withdrawals. Chen, Munnell, and Sanzenbacher (2018) extended Bee and Mitchell's work to other data sources and found that the CPS was an outlier in terms of retirement income misreporting. For example, when compared with administrative data, the CPS captured 61 percent of retirement income, while the SIPP captured 93 percent and the HRS captured 96 percent.

Although the linkage to administrative data from SSA has been available for two decades, to our knowledge, no research to date has assessed the accuracy of HRS respondents' self-reported DI and SSI application and receipt (Schimmel Hyde and Stapleton 2017). This article bridges that gap by comparing HRS self-reports with administrative records. Although it is easy to assume that deviations between the two sources reflect respondent misreporting, self-reports may be more current or complete than the administrative records for several reasons. We discuss those reasons later in our findings to allow interested researchers to assess the relative strengths and limitations of self-reports versus administrative data.

Data and Measures

In this section, we further describe our data sources, the sample selection, the cohorts we include in our analysis, and the measures we use to document DI and SSI application and receipt. We also discuss how the HRS collects consent for the administrative linkage from its respondents, how nonconsent affects the sample size, and how we adjusted the sample weights to account for nonconsenters.

Data Sources

We combined information from publicly available HRS survey data with restricted-access SSA records for HRS respondents. The latter are available with permission from the HRS following an in-depth application and review process. We drew on four source files for our analysis.

The RAND-HRS Longitudinal File is a cross-wave, consistent file developed to facilitate research (Phillips 2003/2004; Bugliari and others 2021). Largely derived from the information in each HRS interview, it simplifies information collected about DI and SSI benefits over many years of the study. As we will discuss, the RAND-HRS file occasionally uses longitudinal information to impute or infer information across waves, particularly when survey design changes limit cross-wave comparisons. The RAND-HRS file is continually updated as new data become available; our analysis used the version of the file that contained data through 2018.

SSA Form 831 Respondent Records (the “831 file”) is an administrative data file that contains information on initial applications for DI and SSI benefits.¹ The file we used contained data from 1988 (when SSA began storing application information in the 831 file) through 2016.² Because the 831 file contains only applications from 1988 onward, it undercounts the number of applications filed by HRS respondents in all years, including those who filed before they joined the HRS sample. This mostly affects participants in earlier HRS waves. The discrepancy diminishes with each subsequent HRS cohort, as the number of respondents with initial applications before 1988 declines.

The 831 file is limited to initial applications that received a medical review. Although disability benefit applications can go through multiple levels of adjudication, the initial application is the “original” and therefore reflects the starting point for each disability claim. The 831 file, however, does not include initial applications that: (1) have not yet received an initial review, or (2) were denied for nonmedical reasons; that is, because they did not meet the financial eligibility criteria of SSA's disability programs (“technical denials”). Most initial reviews occur relatively quickly—within a few months—although a few HRS respondents could have filed an application that had not yet received an initial decision by the time of the interview. Technical denials, however, could account for many more undercounts in the 831 file. During our analysis period, technical denials represented up to one-third of initial DI applications each year (SSA 2022).³ With a technical denial, an HRS respondent would correctly report an application that would not be recorded in the 831 file. We do not know how technical denial rates vary with age, but they might be lower for older DI applicants, who are more likely to have accumulated a work history necessary for benefit eligibility.

The Disability Analysis File (DAF) combines SSA data from multiple administrative sources to produce monthly information about the receipt of DI and SSI benefits for all beneficiaries with at least 1 month of benefits since 1996 (Mathematica 2022).⁴ The file also includes information on monthly benefit amounts and a range of other factors related to the periodic continuing disability reviews that beneficiaries must undergo to retain benefit eligibility, but we do not include that information in our analysis. The version of the file we used contained data through 2018.

The HRS-SSA Permissions Consent History file provides information about whether an HRS respondent consented to having his or her information linked to SSA records and whether a matching administrative record was found (HRS 2021a). We used this file to determine which respondents might be expected to have information available on disability program participation in the administrative records. Respondents who did not consent to the administrative linkage will not have that information available.

Sample Selection

To align with the availability of administrative records, we use data for 1996–2016, spanning four HRS respondent cohorts: the original HRS cohort (born during 1931–1941, first interviewed in 1992), the War Baby cohort (born during 1942–1947, first interviewed in 1998), the Early Baby Boom cohort (born during 1948–1953, first interviewed in 2004), and the Middle Baby Boom cohort (born during 1954–1959, first interviewed in 2010).⁵ We include age-eligible sample members in each cohort, meaning that younger spouses who were selected because they lived in a household with an older age-eligible respondent are included in our analysis once they themselves age into the survey. Members of the original HRS cohort were first interviewed when they were aged 51–61, but all younger cohorts were first interviewed when they were aged 51–56. To provide a study sample comprising four similarly structured cohorts, our analysis includes only the younger members of the original HRS cohort, born during 1936–1941 and first interviewed at ages 51–56 in 1992. For simplicity, we refer to this younger subset as the “original HRS cohort” hereafter, while noting that we found that the outcomes for the younger and older subsets of the full original HRS cohort differed.

For three of the four cohorts, we used the data collected every other year from the initial interview through 2016 (Table 1). The exception is the original HRS cohort, which initially was surveyed in 1992, but we excluded results from survey waves before 1996 to align with the availability of DAF data on disability benefit receipt. Because DI benefits are converted to OASI retired-worker benefits when the beneficiary reaches FRA, and SSI recipients transition from the “disabled” eligibility category to “aged” at age 65, we stop tracking respondents' DI or SSI status at FRA.⁶ In comparing results across cohorts, we categorize respondents in each wave into four groups—(1) interviewed and younger than FRA; (2) interviewed, but reached FRA; (3) not interviewed (no indication of death); and (4) not interviewed (died before interview).⁷

HRS Consent Requirements and Implications for Sample Selection

The administrative data are available only for HRS respondents who consented to the linkage and who provided requisite information (accurate Social Security number, name, and date of birth). The HRS consent process has changed over the years. Importantly for our analysis, the SSA 831 file and DAF are available only for respondents who consented to the linkage in 2006 or later, when the HRS moved from a retrospective permission approach (consent covered all data through the consent year) to a prospective approach (consent allowed linkages with data for past years as well as for a prespecified number of years in the future). This change meant that members of earlier cohorts initially consented under a retrospective permission system but may not have provided the requisite prospective permission necessary to be in our analysis.⁸

Table 2 shows the full unweighted sample size for each cohort in our analysis, as well as the share of each cohort who consented, at any time or in 2006 or later, to the linkage to their administrative records. For each subsequent cohort, the share of cohort members consenting to a linkage at any time has declined, from 88.0 percent in the original HRS cohort to 78.7 percent of Middle Baby Boomers. Despite the overall decline, the rate of consent granted in 2006 or later increased across the cohorts, from 49.0 percent among the original HRS cohort to 77.4 percent of the Middle Baby Boomers. The lower rate in earlier cohorts reflects the fact that more time elapsed for those cohorts from survey entry through 2006, during which many respondents left the sample, died, or did not reconsent. In this article, we use “consenter subsample” to refer to respondents who consented in 2006 or later, noting that this excludes those who consented in an earlier year.⁹

Consistent with earlier work (HRS 2021a), we found that the characteristics of respondents in the consenter subsample differ from those of the full HRS sample. Consenters are more likely to be White, female, and employed, and to have higher educational attainment and lower rates of chronic conditions, including heart disease, lung disease, diabetes, and stroke. Consenters also report lower rates of smoking, fewer difficulties with activities of daily living (ADLs), and fewer hospital stays and doctor visits (Table 3).

Weighting Process

Because of the differences in the characteristics of consenters and the full HRS sample, simply using the administrative records without reweighting would undermine the comparability of the consenter subsample and the full HRS sample. To adjust the sample weights for our analysis, we followed the HRS process to develop analysis weights for its SSA data.¹⁰ Specifically, we predicted the likelihood of consenting in 2006 or later using a logistic regression in each survey wave. Our model included variables for sex, race and ethnicity (indicators for Black and Hispanic), marital status (indicators for married, divorced, and widowed), education (indicators for high school graduate, some college, college graduate, and advanced degree), being employed, categories of self-rated health status, and quintiles of household income and wealth.

We used the predicted values from the logistic regression models to generate inverse probability weights (IPWs) for each record in the consenter file. We then applied the IPWs and the HRS sampling weights to the consenter subsample. This allowed us to generate a consenter subsample that approximated the full HRS sample on the observable characteristics in the IPW model, and with the HRS sampling weight applied, it yielded a weighted sum of interviewed consenters in each wave that equals the weighted sample size of interviewed respondents in that wave from the full HRS. We use this IPW-adjusted survey weight to produce nationally representative estimates based on the administrative data. This allows us to compare our consenter subsample with nationally representative estimates based on self-reports (using the HRS sampling weights alone). Like all such weighting algorithms, our method does not fully account for unobserved variation in consenters and nonconsenters or for observed factors on which consenters differ but were not included in the model. To the extent that those differences also affect the likelihood of applying for disability program benefits, our weighted estimates might be biased. Because our analysis was designed to broadly replicate how HRS data users might use the HRS-provided weights, we did not explore more sophisticated weighting approaches.

Chart 1 shows the weighted distribution of respondents by interview and consent status in each cohort and wave, from the year of survey entry through 2016, applying the wave-specific IPW to the baseline weights for each cohort. In the chart, the dark green bar segments show the share of respondents interviewed in each wave who provided consent for the administrative-data linkages in 2006 or later, enabling their inclusion in our analysis. The IPW reweighting process for analyzing the administrative data means that the weighted sum of the post-2005-consenter subsample (dark green bar segments) equals the weighted sum of the total number interviewed in each wave who have not reached FRA (the combination of the dark blue, light blue, and dark green bar segments). Over time, the share of the non-FRA sample that is interviewed declines because of attrition via FRA attainment, death, or withdrawal from the HRS. All respondents in the original HRS and War Baby cohorts reached FRA before 2016, while only some in the Early Baby Boom cohort did (and none in the Middle Baby Boom cohort did). Because respondents in a 2-year birth cohort attain FRA over more than one survey wave, it is important to note that a cohort's age composition changes as its members approach FRA, as shown in Table 1.

Measuring DI and SSI Application and Receipt

In this subsection, we describe our approach to measuring applications for and receipt of DI and SSI benefits. Box 1 defines our measures. Self-reported values are defined using cross-wave, consistent measures in the RAND-HRS file. Administrative information on applications comes from SSA's Form 831 records linked to the HRS, while administrative records on benefit receipt are derived from the DAF. If HRS respondents consented to the SSA linkage but did not have information available in their 831 file for either DI or SSI, we conclude that they had not applied for benefits from the relevant program. We follow a similar approach if they consented to the administrative linkage but did not have a record in the DAF, counting those respondents as nonbeneficiaries for the relevant program.

Application. For self-reported applications, we use the data available in the RAND-HRS file to identify whether the person had reported ever applying for DI and/or SSI by the date of the HRS interview. This information is based on the respondent's recollection of his or her application status, including the date of initial application. As described earlier, there are reasons why individual self-reports of application may not align with administrative records in the 831 file and why the 831 file undercounts applications that respondents might report. Based on the HRS questions, there are several scenarios under which a respondent would correctly report an application without having an analogous record in the 831 file. Although pending applications might eventually generate an 831 record in a future HRS wave, the 831 file will never include applications filed before 1988 nor does it include technical denials. Allowing for additional time to pass before analyzing the administrative data will not substantially reduce the magnitude of the disparity.

Conversely, the 831 file might also contain applications that respondents do not self-report. First, a respondent who applies for SSI may not know that SSA will also process an application for DI if the applicant meets the latter program's financial eligibility criteria. Second, SSA may consider the SSI eligibility of DI applicants based on information that SSA collects on that initial application.¹¹ In these cases, HRS respondents may report an application only for the program from which they sought benefits, even though the SSA record might show applications for both programs. Similarly, SSA may consider the DI eligibility of OASI retired-worker benefit claimants younger than FRA if their initial application indicates that they have a long-lasting impairment that limits their ability to work. As with concurrent applications, we believe that many HRS respondents may not report a DI application in this case, even though one might appear in the 831 file. Third, entities such as hospitals can file for SSI on behalf of uninsured patients who might be eligible for Medicaid once granted SSI; in these cases, an application may appear in the SSA record that HRS respondents are not aware of. We do not know the frequency of these scenarios, either in absolute terms or relative to the reasons that the 831 file might undercount applications.

Finally, until the 2016 survey wave, the HRS questions about DI and SSI application and receipt history were limited to respondents who reported having a health condition or impairment that limited their ability to work. Thus, some respondents who did not indicate a disability would not have been asked the question and therefore would be counted as nonapplicants based on self-reporting. We return to this point later.

For both the self-reports and administrative data, we assume that if the respondent is a beneficiary in the current wave (based on the comparable self-report or administrative measure), then he or she applied for those program benefits at some point before that interview. We do so even when the survey and administrative data do not affirmatively indicate that the individual had applied. This may, to some extent, mitigate undercounts of applications. For example, applications filed before 1988 will be counted if they were approved and subsequently resulted in benefits (while denied applications will not).

Benefit receipt. We measured the receipt of DI and SSI benefits at the time of the HRS interview. For self-reports, we used wave-specific measures in the RAND-HRS file indicating that the respondent was currently receiving benefits from DI and/or SSI. For the administrative data, we used the DAF to measure benefit receipt, identifying individuals who were in current-payment status in the month or months of the HRS interview.¹² There are fewer reasons to expect misalignment between survey reports and administrative records on benefit receipt than there are for application data. Nonetheless, HRS self-reports undercount benefit receipt for beneficiaries who do not report a work-limiting health condition or impairment, because they are not asked about benefit receipt in that instance.

From 1992 through 2000, HRS respondents were asked about DI and SSI together. As such, respondents may have reported benefits from one of the programs but were unsure which one. We opted not to incorporate information on those whose responses were unsure. For example, respondents who did not clarify whether they received benefits from DI or SSI were classified as not being beneficiaries. After 2000, the survey questions on disability benefit receipt addressed the programs separately, so that an affirmative response was available separately for each program (we excluded information for respondents who replied “don't know” for both programs).¹³ Based on our review of patterns over time, our exclusion of the “unsure” group through 2000 also understates program participation during that time, yet we found that the alternative of including the “unsure” group would have dramatically overstated program participation.¹⁴

Patterns in DI and SSI Application and Receipt by Time and Cohort

To start, we consider the aggregate alignment of survey and administrative reports in each year, incorporating all four cohorts. This analysis shows the patterns of DI and SSI application and receipt derived from each data source over the years of the HRS. Chart 2 shows the results. The blue line shows self-reports, while the red line shows administrative data values; both have been weighted to produce nationally representative estimates of the sample in each year, as described earlier. Despite differences between survey and administrative data in the levels of DI and SSI applications and receipt, the rates of new applications and benefit receipt over time (indicated by the slopes of the lines) are generally similar. In other words, information on the prevalence of DI and SSI application and receipt from the two data sources differ, but the data on their incidence largely agree.

Both self-reported and administrative measures of DI benefit receipt generally increase over the period, as would be expected as a cohort ages and its members are more likely to meet the work-history and health-condition criteria for program eligibility. An individual receiving benefits in one survey wave might not receive benefits again in the next wave, but terminations for reasons other than death or reaching FRA are rare. Self-reported data on DI benefit receipt are always lower than the measure using administrative data at the same time, with the former increasing from about 3 percent in 1996 to just under 10 percent in 2016 and the latter increasing from 7 percent to just over 10 percent over the same period. The addition of new, younger cohorts in 1998, 2004, and 2010 obscures some of the patterns reflecting the aging of the earlier cohorts, shown by the small dip in DI receipt in those years when younger cohorts are added to the sample. Some of the lower levels of self-reported DI application and receipt in earlier years reflect our decision to include only those who reported benefit receipt for a specific program, although this was not an issue after 2000.

The share of claimants who have ever applied for DI benefits is lower in self-reports than in administrative records each year through 2004, then quite close through 2008, after which the self-reported rates are higher than administrative data values. Our measure of having ever applied for benefits is cumulative through each year. Around 4 percent of respondents self-reported having applied for DI benefits in 1996 and around 16 percent self-reported having applied by 2016, while administrative records show that about 9 percent had applied in 1996 and about 14 percent had applied by 2016. The pattern over time is consistent with a growing share of technical denials over the period, meaning that administrative records would exclude an increasing share of applications in the later years of our analysis. It is also possible that toward the later years of the study period, respondents were reporting on applications for which an initial decision was still pending, although we expect this to represent relatively few applications.

Despite differences between survey and administrative data in the levels of DI application and receipt, both sources show similar patterns in new applications and benefit receipt over the period, which can be seen from the slopes of the lines. The slopes for DI receipt are relatively similar at most points after 2000 (when the HRS question scheme changed), except during the Great Recession of 2008, when the administrative data had more marked changes than the self-reports. After 2000, the slopes are quite similar for DI application as well.

In general, self-reported values of SSI application and receipt are also lower than those from the administrative data. The share of respondents who had ever applied for SSI increased from 1.1 percent in 1996 to almost 8 percent by 2016 based on self-reports compared with a change from 4.6 percent to nearly 10 percent based on administrative data. We suspect that the wider differences prior to 2000 largely reflect the HRS questions that combined DI and SSI. After 2000, the difference between self-reported and administrative data values narrows, fluctuating between 1.0 percent and 2.3 percent.

We next disaggregate the data shown in Chart 2 to highlight differences in self-reported and administrative data for each cohort in our analysis. The annual values in Chart 2 combine patterns over time based on secular patterns in experience with SSA's disability programs, differences in patterns across HRS cohorts (reflecting a range of factors including labor market conditions and sufficient labor force participation to be insured for DI), and the aging of HRS cohorts over time. Chart 3 highlights the same four outcome measures as Chart 2, but the horizontal axis replaces calendar years with HRS interview waves, starting with each cohort's respective first interview. Recall that the original HRS cohort was first interviewed in 1992, the War Baby cohort in 1998, the Early Baby Boom cohort in 2004, and the Late Baby Boom cohort in 2010. Because the DAF began in 1996, the first wave recorded in the chart for the original HRS cohort is its wave 3.

In Chart 3, the solid line for each cohort tracks the self-reported value over the successive waves, while the dashed line of the same color represents the value from the administrative data. The patterns by cohort are not consistent across all four measures, whether comparing across cohorts or comparing self-reports to administrative records. More recent cohorts tend to report higher rates of DI application than their administrative records show, aligning with the pattern shown in Chart 2, where self-reported application exceeds that of the administrative record in the later years.

Patterns are less clear for DI benefit receipt and for SSI application and receipt, which may reflect a combination of the factors discussed so far. Although there is modest evidence that the self-reports for the original HRS and War Baby cohorts “catch up” to the administrative records after 2000 following the introduction of the new survey question sequence (waves 5 and 2, respectively), a similar convergence in survey and administrative data is seen for the other cohorts, so that pattern may reflect other factors. Those cohorts also may be misreporting SSI as OASI at those points, although we did not explore that possibility.

Maestas, Mullen, and Strand (2015) found increased DI participation during and following the Great Recession of 2008; we would expect to see this reflected primarily in the 2010 wave given the HRS survey timing. This corresponds to wave 7 for the War Baby cohort and wave 4 for the Early Baby Boom cohort. We do not see notable deviations from the previous trend in DI or SSI application or receipt at that point for those cohorts, either in the self-reported or administrative data. By wave 7 for the War Baby cohort, much of the sample had passed the earliest age of retirement eligibility at 62, so it may be that much of the cohort claimed OASI early and did not meet the criteria for DI.

Age Profiles of DI and SSI Applicants and Beneficiaries

Next, we examine reporting of DI and SSI application and receipt by age, an alternative way to consider the HRS data. Because DI and SSI application and receipt are relatively rare events, some researchers may opt to combine data from multiple cohorts and look at a pooled sample of, say, all respondents aged 51–52. The results in this section highlight how similar survey responses would be to administrative records in that case, and they hold age constant while allowing the cohort to vary. We acknowledge that there are cohort and year differences in DI and SSI outcomes that may be important to account for in some contexts that we do not investigate in this exercise.

In Chart 4, we reorient the data such that all respondents are grouped by age,¹⁵ regardless of the corresponding cohort or year. This structure allows direct comparisons by age but does not consider compositional effects of cohorts or time. These results are weighted using our IPW method, as described earlier. However, in this case, we reran the IPW model within age bands instead of by HRS survey wave. We then applied the IPWs to the wave weight available in the RAND-HRS file for the respondent at the relevant age. These estimates are therefore nationally representative of the age group across all the survey years—an artificial cohort, but one that allows for closer inspection of benefit patterns within ages and across cohorts.

Chart 4 displays the percentage of respondents who self-reported DI and SSI application and/or receipt alongside corresponding percentages from administrative records at each age. One important caveat is that this restructuring does not yield a rectangular dataset—in our study design, we do not have data for each age in all four cohorts. Rather, the values shown include all respondents at each age for whom data were available. As we discussed earlier, Table 1 highlights the years from which we identified respondents of a particular age and cohort. For example, respondents aged 51–54 from the original HRS cohort were interviewed in 1992 and 1994 but are omitted from our analysis because the administrative data we use for comparison, the DAF, began in 1996. On the other end of the age range, the Middle Baby Boomers were last interviewed at ages 57–62. We include the information we have available for each age group, not all of which are represented in all four cohorts.

Chart 4 consists of panels for each of six measures. Each panel contains three sets of dots for each age group. The light blue dots represent the application or receipt rate, as applicable, reported by the full HRS sample; this is the value that is available to HRS data users without access to the administrative linkage. The red dots represent the corresponding rate as shown in the administrative data, which are limited to consenters. The light blue and red dots mirror information shown by year (in Chart 2) and by cohort (in Chart 3), instead shown by age. The dark blue dots represent the self-reported rate among only those HRS respondents who consented to the administrative-data linkage. This set of dots allows us to compare the self-reports of consenters with both the self-reports of the full HRS sample and the consenters' administrative records. Although researchers are unlikely to study this group, we include them here to highlight the accuracy of consenters' self-reports.

Chart 4 shows that HRS respondents' self-reported DI and SSI application and receipt rates are generally lower than those reflected in the administrative records for both the full sample and for the subset who have consented to the administrative-data linkage. This is most notable for SSI applications, for which we expected the administrative data counts to be lower than the self-reports. The pattern is similar for DI and SSI benefit receipt, except for SSI receipt at age 65 or older.¹⁶ For DI applications, self-reported rates are higher than those in the administrative records at ages younger than 60, after which the pattern switches. This may reflect DI applications that are considered because the applicant reported a work-limiting health impairment when claiming OASI retired-worker benefits, which can occur as early as age 62. Despite finding that respondents who consent to the SSA data linkage differ on several demographic and health characteristics, the aggregate patterns of reporting on disability program application and benefit receipt do not differ substantially between consenters and the full HRS sample.

Chart 4's dark blue dots show that, in general, consenters are less likely to self-report DI application and receipt than the members of the full HRS sample are. For SSI, the rates are more similar than those for DI, and in some cases, consenters are more likely to self-report application or receipt. Chart 4 also shows results for measures that combine DI and SSI application and receipt. These combined measures account for individuals who may know they have interacted with a disability program administered by SSA but may incorrectly recall the program. If misreporting reflects respondent confusion between the programs, this combined measure will more closely align with SSA records than either of the individual program measures.

There is not a significant age gradient in the observed gaps between self-reports and administrative records, in either the individual or combined program measures. There is some evidence that misreporting of DI benefits increases as respondents reach the earliest age of eligibility for Social Security retirement benefits (62). For example, self-reports and administrative records of DI benefit receipt are much closer for respondents aged 55–56 than for those aged 63–64 or 65–66. We do not observe a similar pattern for SSI, nor do we see that combining DI and SSI results in differences between self-reported and administrative data that are substantially smaller. This again may reflect applicants who initially claim OASI benefits but are ultimately awarded DI benefits. Unlike OASI benefits, DI benefits claimed before FRA are not actuarially reduced. Further, Medicare coverage is available to DI beneficiaries after a 2-year waiting period, potentially before age 65, but not to OASI beneficiaries before age 65. Given these facts, it would be unlikely that a DI-eligible claimant would prefer OASI benefits.¹⁷ Nonetheless, it is possible that DI beneficiaries who initially claimed OASI benefits may misreport their benefit status when interviewed. Because the composition of the sample is changing with age (given the availability of data at older ages for more recent cohorts), we cannot definitively conclude that self-reports at older ages reflect (or do not reflect) confusion over the program from which benefits are being claimed.

Accuracy of Individual Self-Reported DI and SSI Application and Benefit Receipt Responses

Having described patterns of reporting in the aggregate—by wave, cohort, and age—we now describe the accuracy of individual self-reports. We focus on results for two specific ages: 55, the age for which data are likely to be available for the greatest number of respondents; and 63, which is past the earliest retired-worker benefit eligibility age (62) but is younger than FRA for all cohorts. The misreports we discuss are not weighted; we are interested solely in the likelihood of misreporting by groups of respondents, and nationally representative estimates are not appropriate in that context.

Examining responses separately for DI and SSI as well as for application and receipt, we categorize the accuracy of self-reports into one of four groups:

• Correct negative means that a respondent reports not having applied for or received DI or SSI benefits, and the corresponding administrative record concurs.
• Correct positive means a concurrence in self-reports and administrative records for respondents who report they have applied for or received benefits.
• False positive indicates that a respondent reports having applied for or received benefits, but the administrative record does not.
• False negative indicates that a respondent reports not having applied for or received benefits, but the administrative record indicates application or receipt.

When interpreting these values, we assume that the administrative record is correct—although, as noted earlier, there are reasons why this may not be true, especially for applications.

Chart 5 displays the distribution of respondents aged 55 and 63 who reported ever submitting a DI or SSI application, by accuracy category. Because most adults do not interact with disability programs, correct negatives constitute the largest of the four categories, representing 81–90 percent of the respondents, depending on the program and respondent age. Correct positives are the second largest category, but they occur far less frequently than correct negatives simply because relatively few adults apply for benefits. Together, the false positives and false negatives represent the share of respondents who misreported their benefits, which is small relative to the full sample; 7 percent to 9 percent of HRS respondents misreport DI and SSI application at ages 55 and 63.

Another way to consider the magnitude of misreporting is to consider false reports as a share of total positive or negative reports. This allows for a much closer inspection of the effect of misreporting on aggregate values. For example, consider DI applications reported at age 63: 15.6 percent of respondents either self-reported having applied (10.9 percent) or had a false negative (4.7 percent), meaning that the administrative record indicated that the respondents filed but they did not report an application. The share of false positives (3.9 percent) is close to the share of false negatives (4.7 percent) overall; yet false negatives as a percentage of total self-reported negatives is far lower than false positives as a share of total self-reported positives. This means that positive self-reports are more likely to be wrong relative to the administrative record; 26.4 percent of positive self-reports were incorrect (3.9 percent of 14.8 percent) compared with only 5.5 percent of negative self-reports (4.7 percent of 85.2 percent). We can consider false negatives to be the share of actual applications that were not reported and led to an undercount of total applications. Conversely, false positives represented an opposite influence, toward overcounting; but other than DI applications reported at age 55, false negatives constituted larger shares of the self-reports than false positives.

Chart 6 displays similar results and patterns for benefit receipt for respondents aged 55 and 63. Overall, correct negatives are the largest category of self-reports, consistent with the relative infrequency of disability program participation. Misreports are a smaller share of total reports for benefit receipt than for application (reflecting that many applicants do not ultimately become beneficiaries), but false positives again constitute a much greater share of total positive reports than false negatives relative to all negative self-reports. As with applications, false negative reports of benefit receipt are more common than false positives.

It is helpful to compare the distributions in Charts 5 and 6 with the total misreports indicated in Chart 4. In Chart 6, actual receipt is the sum of correct positives and false negatives. For example, the percentage of respondents aged 55 who reported receiving DI benefits from Chart 6 is 7.1 percent—5.1 percent (correct positives) plus 2.0 percent (false positives). The most proximate value in Chart 4 is represented by the dark blue dot indicating self-reported DI receipt among respondents aged 55–56 who consented to the linkage (and therefore have a corresponding administrative record from which we can assess misreporting). In Chart 4, 6.4 percent of respondents aged 55–56 receive DI benefits. Because the values in Chart 4 are weighted and those in Charts 5 and 6 are unweighted, we expect these values to be similar—as they are—but not necessarily identical.

As we alluded to in discussing Chart 4, it may be useful to consider the overlap in misreporting across programs to try to determine whether misreports reflect confusion about the multiple programs administered by SSA. To evaluate whether a respondent may be correct in reporting receipt of some disability benefit but mistaken on which program, we considered a false positive in one program and a false negative in the other (Table 4). Although there is some overlap that might suggest that respondents are misreporting participation in one program as participation in the other, the share of respondents in this category is relatively small and without a clear pattern. Among false positives for DI receipt, more respondents aged 55–56 reported a false negative for SSI receipt than a correct positive for SSI. Considering the opposite scenario—a false positive report for SSI receipt—we do not see a clear concentration of false negative reports for DI. It appears likely that respondents who misreport benefit receipt for one program may report correctly for the other. Thus, we find some evidence that misreports are the result of respondents mistaking the program from which they receive benefits.

Tables 5 and 6 examine whether misreporting is concentrated in certain demographic and health condition subgroups. It is possible, for example, that misreporting is more (or less) common among those who are less healthy, as they may have had more opportunities to interact with (or learn about) disability programs. In Table 5, we compare characteristics of respondents aged 55–56 and 63–64 with correct positive and false positive reports of DI and SSI benefit receipt. In Table 6, we compare characteristics of those with correct negative and false negative reports.

The tables contain several simplifications to aid in interpretation. First, we omit results for application to focus on benefit receipt.¹⁸ Second, we focus on characteristics in which we identified statistically significant differences between those who report correctly and those who misreport in at least one of the outcomes we considered. To simplify further, we focus on groups of variables (for example, race includes White, Black, and “all other responses,” where we tested the difference in the racial distribution of the groups). A check indicates that the mean or distribution of the variable category shown was statistically different between the correct- and false-report groups.¹⁹

Table 5 shows that there are differences between respondents who misreported receiving benefits (false positives) and those who correctly reported receiving benefits (correct positives). We do not observe consistent patterns in the characteristics correlated with misreporting across program or age. Respondents aged 55–56 who misreported DI receipt differed from respondents who reported correctly by ethnicity and educational attainment. Respondents aged 55–56 with false positive reports for DI had worked for fewer years and were more likely to report poorer health (with a higher prevalence of high blood pressure). Respondents aged 63–64 with a false positive report of DI receipt were twice as likely to be Hispanic, had less education (by almost 1 year, on average), and were employed for 6 fewer years (on average).

We also observe demographic and health differences between respondents aged 55–56 with false positive and correct positive reports of SSI receipt, but they are not the same differences we find for DI beneficiaries. SSI misreporters differ from correct reporters by race and ethnicity, as well as by average income and assets. Notably, false positive reporters are more likely to have higher incomes and assets (which might be expected, given the income and assets limits for SSI). There are also health differences between respondents who reported false positives and correct positives; those with false positive reports tend to have better health behaviors but report worse health. Specifically, those with false positive reports are less likely to be smokers, report drinking fewer alcoholic drinks per day, and are less likely to report having a psychological problem, but they have had more hospital stays in the last 2 years and higher out-of-pocket medical expenditures. In general, the patterns of differences between correct reporters and false reporters for SSI receipt among respondents aged 63–64 reflect a different set of characteristics than those for respondents aged 55–56.

Table 6 reveals that there are consistent differences between false negative and correct negative reporters, across ages and programs. We find statistically significant differences across most individual characteristics, which may not be particularly surprising for two reasons. The first is sample size; correct negatives include all respondents who have no interaction with DI or SSI, which, as shown in Chart 6, is most of the sample. As such, the larger sample sizes may better detect statistically significant differences in characteristics. The second reason involves the eligibility factors underlying program participation. False negative reporters receive benefits, meaning that their financial and health characteristics meet the program eligibility requirements. Because beneficiaries have significant health and functional impairments and are generally out of the labor force, the differences in socioeconomic and health characteristics are to be expected.

Discussion

We began this project seeking a definitive answer to whether researchers should use the HRS self-reported data or the administrative records from SSA. Based on our analysis, the answer is that it depends. In many cases, the self-reported data may be accurate enough—if receipt of SSI is simply a control variable, the difference between 2.0 percent (self-reported data) and 2.5 percent (administrative records), for example, may not be important (Chart 4). Moreover, the consistency of benefit self-reporting along with other self-reported data in the HRS may make the potential bias relative to administrative records derived from another source acceptable. Administrative records may contain information that differs from a respondent's correct self-report, especially on application data, for known reasons. For example, SSA data do not track applications that result in technical denials, which has the effect of undercounting applications. Because administrative data are available only for a subset of HRS respondents who consent to the linkage—and especially for targeted population subsets that can constitute a small sample—using the self-reported data is a sensible choice in many cases, despite its limitations.

If the research question involves establishing beneficiary status, administrative data from SSA should, on their own, provide an accurate representation; however, the administrative linkage to the HRS may be tremendously powerful. Because tracking an individual's disability program interactions is notoriously complex, especially for interactions after reaching retirement age, administrative data about the application process may be valuable. For research projects that intend to use information about denied or allowed applications, such as time to initial decision or reason for denial, administrative data are almost certainly preferable. Yet even then, the 831 file does not contain information on all benefit applications submitted to SSA, nor does it contain the full determination path for those applications. In the next several years, SSA intends to incorporate more complete information on disability applications from its Structured Data Repository into the DAF, which would provide substantially more complete information about applications than is currently available in that file or in the Form 831 records, although it would encompass only applications from 2007 forward (Mathematica 2022).

We found that among the approximately 15 percent of HRS respondents who indicated interactions with SSA's disability benefit programs, about half of their responses to survey questions about DI or SSI application or receipt do not align with the administrative record maintained by SSA for that individual. In general, we found that it is more likely that respondents fail to report benefits they are receiving than to report benefits they are not receiving. As a result, on net, the overall prevalence of DI and SSI application and receipt (when weighted to be nationally representative in the HRS) is lower if based on self-reports than if based on the administrative data. We found that this is generally true across HRS respondent ages and cohorts.

Despite differences between survey and administrative data in the prevalence of reported interactions with SSA's disability benefit programs, the patterns of incidence—new applications and new benefit receipt—across ages and interview waves in the self-reported and administrative data look generally similar. In other words, the differences between self-reported and SSA data that we observe for respondents when they first enter the survey appear generally to remain over future waves, although we observe some differences by HRS cohort that may be important to consider in some research contexts. Overall, we found that the availability of OASI early retirement benefits at age 62 likely does not seem to exacerbate misreporting. We found some evidence that suggested that respondents were reporting DI program interactions when they meant SSI.

We do not fully understand the causes of misreporting beyond those caused by known issues such as changes in some of the HRS questions over the years and the omission of pending applications and technical denials in the 831 administrative file. In some instances, information in the administrative record may not match what is salient to an individual. For example, an applicant may not know that he or she was also considered for DI when applying for SSI or that the lack of a cash payment in a given month does not mean beneficiary status has ended. As we described, most of the reasons we might expect a mismatch between the data sources would result in self-reports of program interactions that are higher than the administrative records indicate, but we generally found the opposite. We found that misreports are nonrandom and differ across race, sex, income, employment history, and several health conditions and behaviors.

We also found—as others have with older versions of the files—that consenting to the administrative-data linkage is nonrandom. We attempted to account for this using a simple IPW scheme that the HRS also uses for its other SSA data linkages, although a more in-depth approach to reweighting, such as exactly matching participants on certain characteristics, may be warranted in other research contexts. More importantly, though, researchers considering using the linked data should be able to use our analysis to take stock of the effects on sample size. The richness of the HRS questionnaire should not be understated, but the small sample for low-frequency events such as disability program benefit receipt becomes still smaller as some respondents decline consent to the data linkage, which may make certain research studies infeasible. Understanding the sample size loss may lead some researchers to accept the loss of precision in the self-reports to preserve record count.

Another reason that researchers may avoid using administrative records is a very high barrier to entry. Although the HRS has streamlined and simplified the process to access the linked SSA data in recent years, the documentation required to understand and link the files to the core survey remains complex and limited. Even with the addition of the DAF—which was designed to support research on disability programs by linking information contained in SSA's other administrative files (many of which can also be accessed by HRS users with permissions)—a detailed knowledge of SSA programs and program data is required to work with the linked data. We have attempted to fill some of that gap with this article. However, the administrative records were not designed primarily to support research, and utmost caution is required to avoid misinterpretation of the information they contain.

Because of the high barriers of access to administrative records, we suspect that self-reported HRS survey results will remain the dominant source of information on disability program benefit receipt. Despite their misalignment with the administrative records, there are several reasons why this may be advantageous. First, the HRS is continually improving the information it collects from respondents. For example, in 2016, the HRS began asking all new respondents—not only those reporting a health-related work limitation—about their receipt of DI and SSI benefits, recognizing that a share of beneficiaries would not report such limitations. Beginning with the 2022 survey wave, all respondents are asked these questions. Second, the HRS collects a large volume of information about disability onset that goes beyond program participation. For example, the survey asks respondents about the nature of their limitations, the timing of new onsets, and their own and their employer's responses to new health conditions. To the extent that self-reported information about program participation aligns with the respondent's recall about the other disability measures, self-reported data across the board may be preferable to information combined from other sources.

A third advantage of using self-reported information is that the RAND-HRS files have converted data drawn from a complex question sequence that has varied over the three decades of HRS data collection to a streamlined, quickly accessible set of measures of DI and SSI program participation. The herculean effort that went into producing cross-wave, consistent measures of program participation should not be understated, and we suspect that many studies of those measures would not have been conducted if the researchers had been faced with developing the measures independently, using the core HRS files. The HRS has significantly advanced knowledge about older workers with new disabling conditions because of its rich, longitudinal data collection and its care in preserving measures as much as possible over time to produce cross-wave consistency. The RAND-HRS files have built on that extensive data collection to make the information widely accessible to the research community. Without both components, we suspect that our understanding of disabilities among older workers would be substantially less robust.

Conclusion

In this article, we investigated differences between HRS survey results and administrative data on DI and SSI application and benefit receipt, as well as differences between those who consent to having their survey responses linked with administrative data and those who do not. We find that aggregate self-reported percentages of DI and SSI application and benefit receipt are lower than those reported in linked HRS-SSA data at nearly all ages, but patterns of new applications and benefit receipt are similar over time and across ages. Moreover, there are cohort differences in the self-reported and administrative data on DI and SSI application and benefit receipt, but no consistent pattern in the difference between the two data sources across the cohorts. Individual misreporting represents a minority of cases, and false negatives (that is, reporting no application or receipt despite administrative records indicating otherwise) tend to be more common than false positives, especially at older ages. For respondents whose administrative data indicate that they misreported their program interactions, some characteristics differ from those whose self-reports concur with administrative records. Those differences depend on the program and the respondent age, but include race, income, assets, education, health conditions, and health behaviors.

Taken together, we find that both data sources can be useful for research pertaining to DI and/or SSI applicants or beneficiaries, depending on the research question. Using HRS self-reported data is likely to result in lower estimates of program application and receipt than linked HRS-SSA data would provide. Estimated distributions of applicants and beneficiaries by demographic, employment, income, and health characteristics might also differ. As such, care should be taken in interpretations of applicant or beneficiary characteristics when using self-reports. Still, the use of linked data may not be feasible for some research purposes. When data linkage may not be practical, self-reports can still be informative in many research applications. These can include, and are not limited to, longitudinal analysis of employment or health characteristics in relation to SSA programs, or the use of beneficiary status as a covariate or control in statistical analysis.

Appendix A