In July 2012, we signed an interagency agreement (IAA) with the Bureau of Labor Statistics (BLS) to develop and test occupational data collection methods that could lead to the development of a new Occupational Information System (OIS). The OIS will replace the Dictionary of Occupational Titles (DOT) that we currently use in our disability adjudication process. The new OIS will include many occupational descriptors similar to those adjudicators currently use in the DOT. However, the OIS will expand on DOT information by describing basic mental and cognitive work requirementsand providing more details about occupations’ exertional and skill ratings.
(See OIS Project History for more background information.)
BLS Occupational Requirements Survey
Under the IAA, BLS is using the Occupational Requirements Survey (ORS) to collect occupational characteristics data that is critical to our disability adjudication process. ORS data, such as the physical demands, skill requirements, and environmental conditions of work, mirror the occupational descriptors found in the DOT.
BLS Fiscal Year 2013 Testing
In fiscal year (FY) 2013, the BLS ORS team conducted three phases of data collection field-testing to evaluate survey design options; to develop, test, and refine collection protocols and aids; and to assess the feasibility of using the National Compensation Survey (NCS) as a platform to collect the occupational data. In the first phase, BLS collected data from 27 establishments. By Phase 3, BLS was collecting data from 690 establishments for about 3,259 occupations.
BLS Fiscal Year 2014 Testing
In Fiscal Year 2014, BLS continued testing the feasibility of using the NCS platform to capture data relevant to the Social Security disability program. The primary goals were improving data quality, minimizing the burden on employers, and resolving outstanding issues from the FY 2013 testing. Specifically, BLS tested the collection of the mental and cognitive data elements and worked with SSA to revise these elements following testing. In fiscal year 2014, BLS completed six feasibility tests to refine the Occupational Requirements Survey (ORS) methodology. Five tests were conducted nationally, across all six BLS regions. The New Data Elements Test was conducted in two metropolitan areas: Washington, D.C. and San Diego, CA.
BLS Fiscal Year 2015 Testing
The results of FY 2013 and 2014 testing were encouraging and indicated that BLS could move forward to pre-production testing in FY 2015.
In FY 2015, BLS:
- Conducted a large-scale nationwide pre-production test to complete the evaluation of whether ORS can produce estimates of sufficient quantity and quality to meet our needs and prepare to move the survey into a production environment;
- Continued to work on issues from the FY 2014 testing, including further refinement and testing of the new mental and cognitive data elements, survey design processes, protocols, aids, and collection procedures; developed best practices to collect data from large employers; and conducted additional research on methods of collection and approaches for testing the validity and reliability of occupational requirements data; and
- Contracted with an outside expert to produce a methodological report that provides advice on achieving high levels of reliability, validity, and accuracy given the ORS’ parameters. Specific concerns include the relative merits of different sources of information and modes of data collection. The central recommendation is that BLS conduct its own systematic tests and validation studies in order to ensure that the final survey design aims for the most reliable, valid, and accurate measures possible given the various alternatives. (See Occupational Requirements Reliability and Validity Literature Research)
In FY 2015, SSA plans to:
- Use the limited data from the pre-production test to assess the need for updated or revised adjudication policy, and to design preliminary testing which could include determining the usability of the front-end IT tool.
- Continue to study the components of O*NET to determine which of its elements can be incorporated into our OIS with the data collected by BLS.
- Assess the budgetary and operational feasibility of various OIS maintenance and update options.
BLS Fiscal Year 2016 Activities
Pending the Outcomes from FY 2015 testing, SSA and BLS have the following activities planned for FY 2016:
- Begin collecting OIS production data;
- Continue identifying the impact of the new occupational data on our disability policy and begin making changes to prepare for rolling out the OIS;
- Develop an appropriate testing plan that may include comparing new occupational data with existing information provided by claimants or contained in O*NET and DOT descriptions, and side by side adjudication testing to assess potential effects on the disability system, including allowance and denial rates.
- Begin issuing any necessary guidelines on use of the new OIS IT platform and begin to design training for adjudicators on the new system.
The DOT contains discrete and well-established descriptions of the physical demands of jobs, but it does not provide information on their mental and cognitive requirements. The current ORS testing represents a unique opportunity to establish the comparable mental and cognitive requirements of work so that we can add this data to the new OIS.
In FY 2013, we organized an internal workgroup of disability vocational policy experts and contracted psychiatric and psychological consultants to develop data elements that describe the mental and cognitive requirements of work. The workgroup started with the Disability Evaluation Constructs -- a list of all of the possible elements that may be useful to disability adjudication that we gathered from external experts and public comment from the Occupational Information Development Advisory Panel -- and a list of the mental and cognitive data elements from O*NET. The workgroup also reviewed the results of an internal study that gathered information from 5,000 disability claims, which is a representative sample of claims decided at steps 4 and 5 of sequential evaluation at the initial and hearings levels. To ensure that the mental and cognitive elements met the needs of our disability programs, the workgroup used the results from the study showing the most commonly assessed mental limitations in claims with mental impairments to develop a manageable set of mental and cognitive data elements focused on the demands of lower skilled work.
During FY 2014 feasibility testing, BLS tested the mental and cognitive survey questions. OIS project staff and staff from the Office of Disability Policy observed several data collection appointments. We worked with BLS, whose field economists have experience collecting similar mental and cognitive occupational requirements for the NCS, to make refinements and adjustments to the questions.
The test captured the following mental and cognitive data:
- The complexity of the work tasks;
- The frequency and nature of social interactions; and
- The frequency of required changes in work tasks, work schedules, and work location.
Results from testing and feedback from stakeholders have resulted in changes to the mental and cognitive survey questions and responses. In the summer of 2015, BLS will test the revised mental and cognitive questions.
Working with DOL’s Employment and Training Administration (ETA)
The Department of Labor (DOL) last updated the DOT in 1991, and, in its place, launched O*NET in 1998. O*NET does not contain the detailed physical job requirements needed to adjudicate disability claims under current Social Security regulations and policy.
However, there are aspects of O*NET that we could effectively integrate into the OIS to avoid duplicating work already done by DOL. We are working with DOL’s ETA to identify and incorporate into the OIS O*NET task descriptions, lay titles, and information on occupational tools and technology.
OIS Technology Platform
In FY 2015, we began work on an easy-to-use, web-based platform to support the new OIS. We assembled a workgroup of the agency's occupational information users, including representatives from the Office of Disability Adjudication and Review and adjudicators from State disability determination services, to help outline system requirements. When we complete development, the OIS will be available to the public to use online at no cost.
For more information regarding the development of the OIS, please contact us at firstname.lastname@example.org.