Can AI solve unemployment? How states can transform workforce data into optimized career opportunities

Unemployment is a complex and multifaceted problem. And it’s one that big information gaps are contributing to. Many jobseekers simply don't have access to the context needed to make well-informed decisions for their lives and careers.

For example, imagine for a moment that you are a 45-year-old office administrator with a high school education who was recently laid off. You have 20 years of experience in office management and certifications in Microsoft Office and QuickBooks. However, administrative jobs in your area are decreasing due to automation and remote work trends. You courageously decide to consider a career change. But how do you figure out the occupations in which your skills and certification will transfer? What additional training and certifications will you need? And will that job still exist in 10 years? 

This is information jobseekers almost never have. But while the unemployed workforce doesn’t have access to this information, states do. 

Why and How AI will Transform How States Leverage SLDS 

Nearly every state has a statewide longitudinal education and workforce data system (SLDS). An SLDS collects information about an individual’s education and training experiences to track employment and wage outcomes. These are also known as P20W systems, short for “preschool through 20^th grade plus workforce.” Maintaining these data systems is common practice. Traditionally, states have used SLDS to report on student success and assess the effectiveness of education systems.  

But the advent of AI and machine learning has created enormous opportunities for SLDS that many states are only just beginning to get their arms around.  

Combined with AI, a robust SLDS can generate incredible insights for individuals as well as agency users. AI applies algorithms to large data sets, allowing the software to identify and learn from patterns and trends. Its iterative nature mimics human intelligence by learning from experience and monitoring its own performance. Because AI can process massive amounts of information and complete tasks quickly, it can solve complex problems, answer hard questions, and make informed predictions. More importantly, AI can delve into understanding the “human condition” and the nuanced drivers of achievement and performance disparities. With this understanding at their fingertips, states can tailor and optimize programs and interventions for a person’s specific background and goals—particularly jobseekers—while minding privacy and security considerations.  

Pivot: Indiana’s Workforce Recommendation Engine 

The Indiana Department of Workforce Development is putting AI to work on its SLDS. Indiana’s workforce is comprised of more than 3 million people across 830 occupations. Every quarter, employers report each of those individuals’ wage records to the state. Over time, this ever-expanding dataset elucidates how earnings change as people move through their careers. 

Sensing the opportunity latent in the data they already collected, Indiana DWD developed Pivot, a first-of-its-kind career recommendation engine.  

The tool applies AI to wage and education data to offer jobseekers personalized career recommendations based on outcomes of similar personas. So, recalling our fictitious office administrator, Pivot would provide the context needed to make a career change. It would provide insights into where individuals with comparable education levels and experience have succeeded, with all data de-identified and anonymous. And, eventually, the tool would help jobseekers chart workforce skill-up programming and training options needed to succeed in certain career paths. 

Since Pivot launched in November 2023, the response from jobseekers has been overwhelmingly positive. The tool includes an optional feedback mechanism which asks users to indicate if a recommendation resonates with them. So far, the tool has seen thousands of positive reactions to these occupation recommendations. 

Access to Pivot is currently isolated to those receiving unemployment insurance, but Indiana DWD hopes to make it available to anyone engaging in a job search in the future. 

More Use Cases and Opportunities for AI and SLDS 

Indiana’s workforce recommendation engine is one example of a micro-application of AI and SLDS, of which there are many. States can also leverage the workforce recommendation concept for other high-need populations, like formerly incarcerated individuals and veterans. Another obvious application is in K-12 education. In tracking student outcomes in the workforce post-graduation, states are better equipped to evaluate school and curriculum performance. On the student side, transparency around the labor market can help students make more informed decisions about their future careers.  

But the applications extend to understanding macro-effects of programs and services. For example, how investment in reducing dropout rates impacts long-term consumption of need-based services. This is true of many open questions across education and workforce, like, “What would have happened if this person or subgroup of people didn’t receive a particular service?” The pharmaceutical industry handles this with placebo control groups. It's difficult to do this purposefully in education and workforce spaces, but we can almost always gather and organize data on representative groups that didn’t participate in a program after the fact. That allows states to answer the ‘what if’ questions and calculate incremental contributions to creating an outcome. 

In both micro and macro instances, agencies need to be able to couple the individual (real people with nuances) with the broader environment. Applying AI to SLDS can help accomplish this.  

Many applications of AI in government have yet to be seen, but the integration of AI with SLDS is a practical yet transformative leap forward. In leveraging AI, states can unlock the full potential of workforce and education data they’re already collecting. Indiana's workforce recommendation engine Pivot exemplifies this. As states refine and expand these systems, the prospect of reshaping public services and programs becomes increasingly tangible. Embracing these advancements is a critical step toward solving the complex issue of unemployment and fostering a more informed, adaptable and resilient workforce. 

John Roach is the president of Resultant, a leading consulting firm shaping how local, state and federal government agencies use data.