Results of the Office of Policy's 2001 User Satisfaction Survey

Summary

In the summer of 2001, individuals who had an active professional interest in research or policy issues related to Social Security or to Supplemental Security Income (SSI) were surveyed by the Gallup Organization, under contract to the Social Security Administration (SSA). The survey had two goals. One was to determine the extent to which SSA's research, statistical, and policy analysis work was focusing on topics and issues of widespread concern to users. The other was to gauge user satisfaction with the products of that work. Most of SSA's research and analysis is carried out by the Office of Policy.

Responses were obtained from 1,043 persons out of a sample of 1,800. Respondents were most likely to have been interested in Social Security or SSI issues for at least 10 years, to have used information from SSA multiple times during the preceding 2 years, and to have worked for the government, a college or university, or a nonprofit service organization.

Results suggest that most users were satisfied with the quality of SSA's research, statistical, and policy analysis information products.

• Overall, 86 percent of those who had used products from SSA in the preceding 2 years were very or somewhat satisfied, 10 percent were neither satisfied nor dissatisfied, and only 4 percent were somewhat or very dissatisfied.
• 70 percent to 89 percent of users were satisfied with the accuracy, clarity, comprehensiveness, objectivity, timeliness, usefulness, and ease of finding the information. They were most satisfied with accuracy and objectivity and least satisfied with timeliness and the accessibility of products.
• 62 percent of respondents were very or somewhat satisfied with SSA's identification of and work on new and emerging research and policy issues over the past 2 years, 24 percent were neither satisfied nor dissatisfied, and 15 percent were somewhat or very dissatisfied.

Respondents were asked to characterize their professional interests in two ways--as related primarily to Social Security, SSI, or both programs, and as related primarily to retirement issues, disability issues, or issues in both areas. Satisfaction varied, sometimes substantially, with professional interests. On all measures, Social Security specialists were more satisfied than SSI specialists, and retirement specialists were more satisfied than disability specialists. Note that of respondents interested primarily in the SSI program, 93 percent were also more interested in disability issues.

The survey asked users to recommend ways of improving SSA's research, statistical, and policy analysis, and to recommend issues for SSA's research and policy agenda in the near future. Only 39 percent of users offered recommendations for improvement, and of those recommendations, no single issue predominated. In general, the most common recommendations were for increased data analysis in particular areas and for improved dissemination of information.

A large proportion of respondents (77 percent) suggested issues that SSA should include in its research and policy agenda. Although the recommendations were quite diverse, two types appeared to dominate--disability issues and programs, and Social Security solvency and reform proposals. Some of the suggestions for disability research and policy work seem to suggest that the lower satisfaction ratings of specialists in this area reflect generalized dissatisfaction with the structure and administration of disability programs themselves rather than dissatisfaction with the quality of SSA's research and policy products.

The Office of Policy is developing specific proposals for improving products and services, based on the survey's findings. By improving the quality and content of its statistical and analytical information, SSA will continue its efforts to meet the needs of a diverse research and policy community.

Introduction

An initial analysis of the data from the 2001 User Satisfaction Survey was provided by the Gallup Organization in December of that year and is posted on the Office of Policy's Web site at www.socialsecurity.gov/policy. The analysis consists primarily of percentage distributions of responses to each item in the questionnaire, with only limited attention given to relationships between variables. In general, the survey found relatively high levels of satisfaction, although satisfaction varied somewhat for different measures.

This article explores those and other survey findings in greater depth--in particular, how selected characteristics and professional interests of respondents may be related to their degree of satisfaction and their recommendations for improvements in the Social Security Administration's (SSA's) research and analysis efforts.

Survey Design

The target population for the survey--users and potential users of SSA's research, statistical, or policy information--cannot be easily identified or listed for sampling. The population would include individual subscribers to relevant SSA publications, but it would also include nonsubscribers who are active in Social Security-related policy and research and who may gain access to SSA's information in other ways--for example, in libraries, on the Internet, by contacting SSA staff, or through SSA briefings and presentations. To select a sample, the Office of Policy and Gallup developed four unduplicated lists: individual subscribers to a cross-section of publications, nonsubscribers compiled from lists of professional associations and attendees at various research and policy-related conferences, individuals on an agency stakeholders list who had expressed interest in receiving statistical publications, and a small group of decision makers--individuals in high-level positions at SSA and related federal agencies.¹

A sample of 1,800 persons was selected from the four lists, and a questionnaire was mailed to them in July 2001. Respondents were given the option of completing a Web-based questionnaire, and telephone interviews were attempted for those who had not responded by mail or Internet within several weeks. Responses were obtained from 1,043 individuals (for a response rate of 58 percent). Almost all of the respondents (88 percent) were identified as active--that is, as having had a professional interest in research, statistics, or policy issues related to Social Security or to Supplemental Security Income (SSI) in the past 24 months. And almost all of the active respondents (92 percent) were identified as users of SSA's information--that is, as having received any research, statistical, or policy information from SSA publications, Web sites, or staff in the past 24 months. The final group of 840 users included 45 decisionmakers, 462 subscribers, 185 nonsubscribers, 145 stakeholders, and 3 respondents whose status was unknown.²

The questionnaire was designed to be relatively brief, to minimize the burden on respondents and encourage a high response rate. Testing indicated that most respondents were able to complete it in about 10 minutes. Keeping the questionnaire short limited the number and complexity of questions that could be included. The core content of the questionnaire was divided into two parts. The first included a series of questions about topics and issues of concern to the 913 respondents who were classified as active in the area of Social Security or SSI policy and research. The second part focused on the subset of 840 users of SSA's research, statistical, or policy information in the 2 years preceding the survey. This group was asked about the types of products they had used and about their satisfaction with various aspects of those products. A final question asked about their satisfaction with the overall quality of the information. In addition to the core content, the questionnaire included a few questions about the respondents themselves and an open-ended question eliciting their recommendations for improving SSA's research, statistical, or policy products and services.³

Characteristics of Respondents

Although the survey included only a few questions about respondents' characteristics, some interesting patterns emerged. Over 70 percent of respondents were employed in one of four categories: the executive branch of the federal government, colleges or universities, nonprofit service organizations, or state or local governments (see Chart 1). About one-half of the federal workers (11 percent of respondents) were employees of SSA. Roughly three-fourths of respondents had been interested in Social Security or SSI issues for at least 10 years, and almost half reported such interests for more than 20 years (see Chart 2).

Those who had used information from SSA typically reported multiple uses during the preceding 2 years. Thirty-seven percent said they had sought or received research, statistical, or policy information three to nine times; almost half had done so even more frequently (see Chart 3). Users received the information in a variety of ways, the most common being published or hard copy materials, SSA's web sites, and individual assistance from SSA staff (see Chart 4). The majority used information in the form of both statistical tables and analytical reports; less than one-fifth used one or the other exclusively.

Professional and Programmatic Interests

Because users and potential users of SSA's research and policy work could be expected to have a range of professional interests, respondents were asked to characterize their interests in two ways--first, as related primarily to Social Security, SSI, or both programs, and second, as related primarily to retirement issues, disability issues, or issues in both areas. On the basis of their answers, the respondents were divided into program specialists (the 29 percent more interested in Social Security and the 17 percent more interested in SSI) and program generalists (the 55 percent interested in both programs). Respondents were divided similarly along areas of interest: specialists were the 25 percent more interested in retirement issues and the 36 percent more interested in disability issues; generalists were the 39 percent interested in both areas.

Although the majority of respondents were interested in both programs and in both areas, dividing them into specialists and generalists provided some useful insights. The overwhelming majority of respondents interested primarily in the SSI program were concerned particularly with disability issues (see Table 1). In contrast, 60 percent of respondents interested primarily in Social Security were concerned particularly with retirement issues. Of respondents interested in both Social Security and SSI, over half were interested in both retirement and disability issues.

Work Affiliation

Specialist and generalist groups varied somewhat by work affiliation. As indicated in Chart 5, the largest group of generalists worked in the executive branch of the federal government. The largest group of Social Security or retirement specialists were employed by universities, and the largest group of SSI specialists were employed by state or local governments. Disability specialists appeared to be more evenly divided among the federal government, state and local governments, nonprofit service organizations, and universities.

Interest in Particular Topics and Specific Populations

The survey also measured interest in specific topics. Overall, respondents showed the greatest interest in current programs or proposals for change, the economic well-being of the aged and disabled, work-related issues, and the economic impact of the Social Security or SSI program.

Interest in some of the topics differed substantially by program and area, however. Consistent with earlier findings, the topics of greatest interest to SSI specialists tended to be similar to those of greatest interest to disability specialists, and the topics that most interested Social Security specialists tended also to interest retirement specialists. The four topics with the greatest spread in interest are shown in Table 2. Note, for example, that whereas Social Security financing ranks only fifth in interest overall (at 63 percent), it is a topic of great concern to Social Security and retirement specialists (at 79 percent and 80 percent, respectively).⁴ Specialists in Social Security or retirement issues showed much greater interest in private sources of income security and in other countries' income security systems than did SSI or disability specialists. Generalists, whether interested in both Social Security and SSI or in both retirement and disability, showed more interest in other government income security programs in the United States than the total sample did; SSI specialists showed less interest (23 percentage points less).

The questionnaire then measured respondents' interest in selected populations. Overall, respondents showed the most interest in disabled adults, women, low-wage workers, racial or ethnic minorities, and disabled children, as indicated in Table 2. Again, there were some wide differences between specialists and generalists. Social Security or retirement specialists exhibited considerable interest in women, older workers, and early retirees, whereas SSI or disability specialists exhibited far less. As would be expected, retirement and disability generalists exhibited a greater interest in most topics than the total sample did.

Satisfaction with SSA's Research and Policy Agenda and Suggested Issues for the Future

For the satisfaction questions in the survey, five responses were provided: very satisfied, somewhat satisfied, neither satisfied nor dissatisfied, somewhat dissatisfied, and very dissatisfied. Overall, 62 percent of respondents were very or somewhat satisfied with SSA's performance in identifying and working on new and emerging research and policy issues over the preceding 2 years. Satisfaction varied by specialty area, with 76 percent of retirement specialists, 67 percent of Social Security specialists, and 52 percent of disability or SSI specialists indicating satisfaction (see Table 3). Of the five largest employment categories (Chart 1), only respondents affiliated with a private research or policy organization exhibited greater satisfaction (74 percent) than the total sample; none of the other four groups exhibited notably less. In general, those who used information from SSA at least 20 times in the preceding 2 years were more satisfied than those who used information less than 20 times. Satisfaction with SSA's performance on new and emerging issues did not vary much by length of interest in issues relevant to Social Security or SSI.

After exploring respondents' interests and their satisfaction with SSA's recent research and policy agenda, the survey asked them to list the most important research or policy issues they thought SSA should be working on in the next year or two. Twenty-three percent did not respond to this open-ended question; those who did cited diverse and wide-ranging issues (see Table 4). About one-fourth of respondents identified program solvency and Social Security reform issues as the most important research or policy areas for the near term. Other priorities were disability, income security, and work-related issues, with the latter two often linked to concerns for the disabled.

Response rates and suggestions varied somewhat, largely according to satisfaction with SSA's performance in addressing new and emerging issues (data not shown), although concerns about disability and Social Security solvency or reform continued to dominate. Among the 114 respondents who were somewhat or very dissatisfied with SSA's performance, only 10 did not answer. The most frequent suggestion from the dissatisfied (54 instances) focused on some aspect of the disability program, including both SSI and Disability Insurance. This was followed by suggestions related to Social Security trust fund solvency and the effects of proposed Social Security reforms such as individual accounts and privatization (25 instances). Specific suggestions regarding the disability program focused most often on the disability determination process, including the definition of disability and the adjudication process, followed by employment of the disabled and incentives or disincentives to work, disabled children, the Ticket to Work program, and financing of disability programs.

Among the 189 respondents who were neither satisfied nor dissatisfied with SSA's performance, 76 percent offered suggestions. Most frequently, they referred either to some aspect of disability programs (52 instances) or to Social Security trust fund solvency and the effects of proposed reforms such as individual accounts and privatization (50 instances). Specific suggestions related to disability focused most often on the determination process and on work and disability.

Over 80 percent of the 141 respondents who were very satisfied with SSA's performance offered suggestions. The most frequent suggestions mentioned Social Security trust fund solvency or the effects of proposed Social Security reforms such as individual accounts and privatization (79 instances). Smaller numbers mentioned disability-related issues (16 instances).

Satisfaction with Particular Aspects of SSA's Products

Users of SSA's research, statistical, or policy information were asked to rate their satisfaction with the agency's statistical tables and analytical articles or reports. Respondents evaluated seven specific aspects of these products: accuracy, clarity, comprehensiveness, objectivity, timeliness, usefulness, and accessibility (see Chart 6). Users of both statistical tables and analytical products were most satisfied with the accuracy of the information and least satisfied with the effort it took to find what they were looking for (accessibility). On all seven aspects, slightly more users were very satisfied with statistical tables than with analytical articles. Total satisfaction (very and somewhat satisfied) was roughly equal between users of statistical tables and users of analytical articles on all seven aspects.

Satisfaction varied somewhat by professional interest. The largest differences overall concerned objectivity and accessibility. Specifically, 75 percent of SSI specialists and 78 percent of disability specialists were very or somewhat satisfied with the objectivity of statistical tables, compared with 88 percent of Social Security specialists and 92 percent of retirement specialists. In addition, 60 percent to 63 percent of disability and SSI specialists were very or somewhat satisfied with the ease of finding information (statistical tables), compared with 80 percent of Social Security or retirement specialists.

Some differences were even more pronounced when the measure was restricted to those who were very satisfied. In rating the accuracy, objectivity, and clarity of information, Social Security and retirement specialists were more likely to be very satisfied with statistical and analytical materials than were SSI and disability specialists:

• Accuracy: Among users of statistical tables, 81 percent of Social Security specialists, 83 percent of retirement specialists, 54 percent of SSI specialists, and 53 percent of disability specialists were very satisfied with the accuracy of the information. Among users of analytical articles or reports, 65 percent of Social Security specialists, 71 percent of retirement specialists, 40 percent of SSI specialists, and 37 percent of disability specialists were very satisfied with accuracy.
• Objectivity: Among users of statistical tables, 71 percent of Social Security specialists, 71 percent of retirement specialists, 46 percent of SSI specialists, and 50 percent of disability specialists were very satisfied with the objectivity of the information. Among users of analytical articles or reports, 62 percent of Social Security specialists, 60 percent of retirement specialists, 33 percent of SSI specialists, and 38 percent of disability specialists were very satisfied with objectivity.
• Clarity: Among users of statistical tables, 56 percent of Social Security specialists, 54 percent of retirement specialists, 37 percent of SSI specialists, and 35 percent of disability specialists were very satisfied with the clarity of information. Among users of analytical articles or reports, 48 percent of Social Security specialists, 46 percent of retirement specialists, 26 percent of SSI specialists, and 28 percent of disability specialists were very satisfied with clarity.

Differences on the remaining aspects (comprehensiveness, timeliness, usefulness, and accessibility) were not as large, although in all cases more Social Security or retirement specialists were very satisfied than SSI or disability specialists. In order to keep the survey questionnaire brief, the Office of Policy did not ask any follow-up questions; therefore, the survey provides few clues as to what respondents found unsatisfactory about particular aspects of quality. However, in an open-ended question that appeared later in the survey, several respondents asked for more nontechnical or nonstatistical discussions of issues and for the use of more bold print and bullets to improve the clarity of information from SSA.

Satisfaction with Specific Publications

Respondents were asked to indicate their overall satisfaction with 10 selected Office of Policy publications they might have used in the preceding 2 years. In contrast to the other satisfaction questions in the survey, Gallup did not have a screening question to determine this question's applicability to each individual; instead, a "did not use" category was added at the end of the satisfaction response categories for each of the publications. This design feature, intended to keep the questionnaire shorter and less complex, may have inadvertently introduced some measurement error.

The possibility of measurement error is suggested by the data in Table 5. On the one hand, the "did not use" option appears to have worked as intended for a large number of respondents: that is, the more specialized publications (such as the statistical compilation OASDI Beneficiaries by State and County) were used least often, and the publications with broader interest (such as the Social Security Bulletin) were used most often. However, some respondents who had not used a particular publication may have simply skipped that rating, and still others who were uncertain about their usage may have taken what they felt was a neutral position. In fact, the rate of "missing" responses was much higher on this question than on the survey's other satisfaction questions, and the proportion choosing the middle response on the satisfaction scale ("neither satisfied nor dissatisfied") was also higher. Perhaps more important, the incidence of middle responses for particular publications was roughly correlated with "did not use" responses--that is, the less frequently used a publication was, the more likely it was to have missing responses and ambiguous ratings on the satisfaction scale.

In light of these concerns, caution should be used in interpreting the satisfaction ratings for publications. An initial analysis suggests fairly wide differences among them, ranging from a high of 83 percent who were very or somewhat satisfied with the Social Security Bulletin and the Annual Statistical Supplement to a low of 64 percent who were very or somewhat satisfied with Social Security Programs Throughout the World (Table 5). However, if calculations are restricted to respondents who took a clear position (excluding the "neither satisfied nor dissatisfied" category), the differences between publications narrow considerably, ranging from 98 percent who were very or somewhat satisfied with the Social Security Bulletin to 92 percent for two of the more specialized publications (see lower panel, Table 5). With either approach, the proportion expressing clear dissatisfaction with the 10 publications was relatively small, ranging from 2 percent to 8 percent who were somewhat or very dissatisfied.

An effort was made to explore the reasons for variations in satisfaction by examining respondents' ratings of specific aspects of analytical and statistical products--the possibility, for example, that a respondent's lower satisfaction with a particular publication might be related to his or her lower satisfaction with timeliness or some other attribute. However, almost all respondents reported use of multiple publications, making it impossible to establish a link between a given publication and evaluations or recommendations made elsewhere in the survey.

Satisfaction with the Overall Quality of Information from SSA

Altogether, 86 percent of users indicated that they were very or somewhat satisfied with the overall quality of SSA's research, statistical, or policy information, 10 percent were neither satisfied nor dissatisfied, and 4 percent were very or somewhat dissatisfied. Ratings varied somewhat by professional interest, length of time interested in Social Security or SSI issues, frequency of use of information from SSA, work affiliation, and satisfaction with SSA's work on new and emerging issues. Overall quality ratings did not vary much among those who used only statistical tables, only analytical articles or reports, or both.

Differences by specialty were not great on the measure of total satisfaction (very satisfied and somewhat satisfied), but they were fairly pronounced on the very satisfied measure (Chart 7), with more retirement and Social Security specialists than disability or SSI specialists reporting great satisfaction. The two generalist categories were intermediate in their ratings of very satisfied. Dissatisfaction was quite low for all groups of respondents.

Total satisfaction was high and tended to rise with years of interest in Social Security or SSI issues (see Table 6). In fact, users with more than 20 years' interest accounted for the largest percentage of people who were very satisfied with the overall quality of SSA's information. Again, dissatisfaction was quite low for all groups.

Although satisfaction varied somewhat with frequency of use of SSA's products, it was difficult to discern a consistent pattern (Table 6). Respondents who had used information more than 50 times in the past 24 months accounted for both the greatest proportion who were very satisfied and the greatest proportion who were somewhat or very dissatisfied. As a result, total satisfaction among the most frequent users is closer to satisfaction among the least frequent users than it is to the total satisfaction of intermediate users.

Over 80 percent of users in the five largest employment categories--private research or policy organizations, university and college employees, state and local employees, nonprofit service organizations, and the executive branch of the federal government--were very or somewhat satisfied with the quality of SSA's information (Table 6). Employees of SSA expressed roughly the same levels of satisfaction as all users. Private research organizations had the highest percentage of users who were very satisfied, and nonprofit service organizations had the lowest.

Finally, there appears to be a correlation between satisfaction with SSA's recent work on new and emerging issues and satisfaction with the overall quality of SSA's information (Table 6). The correspondence is highest among those who expressed greatest satisfaction with SSA's work on new issues and those who were most satisfied with overall quality. A much smaller proportion was dissatisfied on both measures. More interesting, however, is the finding that a majority of those dissatisfied with SSA's work on new issues were satisfied with the overall quality of SSA's information.

Recommended Improvements

The survey's final question was open-ended, seeking recommendations for improving SSA's research, statistical, or policy products and services. Overall, 61 percent of users did not respond or indicated that they had no recommendations (see Table 7). The suggestions that were offered were wide-ranging and often quite specific. Grouped into general categories, the greatest number of users suggested expanding the content of information, improving the dissemination of information, or improving the quality of information. Some differences can be seen between the recommendations of the somewhat or very dissatisfied and other users, but the sample size of the dissatisfied is too small to reach any conclusions.

A few users repeated the concerns they had expressed earlier in the survey about increasing access to SSA's administrative data files. Such data files are maintained for internal, administrative purposes, and the law restricts access to them by outside researchers and analysts in the interests of maintaining citizens' confidentiality. Assuming appropriate safeguards could be implemented, users particularly wanted access to public-use data files in four areas mentioned in the questionnaire: SSA data linked to survey data (73 percent), characteristics of Social Security beneficiaries (68 percent), characteristics of SSI recipients (62 percent), and the earnings of covered workers (50 percent). Differences in interest by specialty were not surprising; for example, SSI specialists were not particularly interested in the characteristics of Social Security beneficiaries, and Social Security or retirement specialists were not as interested in the characteristics of SSI recipients.

Potential Users of Information from SSA

The survey defined potential users of research, statistical, or policy information as respondents who, in the preceding 2 years, had had a professional interest in research, statistics, or policy issues related to Social Security or SSI but who had not received any research, statistical, or policy information from SSA publications, SSA Web sites, or SSA staff. In total, 72 respondents (8 percent of the sample) were identified as potential users. Although the sample size was small, the following characteristics of potential users emerged:

• Potential users were slightly more likely to be employees of a state or local government, Congressional agency, or business or industry than the total group of respondents. They were slightly less likely to be employees of the executive branch of the federal government, a university, or a nonprofit service organization.
• Potential users had, on average, slightly shorter career interests in Social Security or SSI issues than the total group of respondents. For example, 15 percent of potential users had less than 5 years of interest, compared with 9 percent of all respondents, and 36 percent had more than 20 years of interest, compared with 45 percent of all respondents.
• Professional interests of potential users were roughly the same as those of the total sample; that is, the same percentage of individuals were interested in Social Security, SSI, or both Social Security and SSI, and the same percentage were interested in retirement, disability, or both retirement and disability.
• Only 69 percent of potential users expressed an opinion with regard to SSA's performance on new and emerging issues. Of those expressing an opinion, 40 percent were very or somewhat satisfied with SSA's performance, 46 percent were neither satisfied nor dissatisfied, and 15 percent were very or somewhat dissatisfied.
• Only 35 percent of potential users offered recommendations for improving SSA's research, statistical, or policy products and services. The small sample size makes analysis difficult, but the greatest number of those who responded (12) recommended improvements in the dissemination of information.
• The most important research or policy issues suggested by potential users were similar to those suggested by the total group of respondents. The most common recommendations were related to disability (eligibility or work) and to Social Security solvency or the effects of proposed reforms such as individual accounts and privatization.
• On average, potential users were only about 10 percentage points less interested than all respondents in SSA's public-use data files. Interest among potential users ranged from 64 percent for SSA data linked to survey data to 30 percent for earnings histories of covered workers.

The survey also asked these respondents why they had not used SSA's information in the past 24 months. Of those who gave a reason, the most common was that the information was "not related to my needs," as indicated in Chart 8. A similar proportion did not know the information was available or had not received it. About one-half of all potential users gave no particular reason (or did not respond to the question) for not using SSA's information.