How the procurement process can help agencies acquire responsible AI

Artificial intelligence is a hot commodity, but government agencies need to be cautious about buying it, experts say.

The biggest consideration is security. “It’s a challenge. There’s no other way to say it,” said Meredith Ward, deputy executive director of the National Association of State Chief Information Officers. “Cybersecurity needs to be baked into the process from the beginning, and quite honestly, that’s not happening.”

One reason why is the newness of AI, particularly generative AI solutions such as ChatGPT. State chief information security officers are proceeding with more caution than private-sector companies when it comes to AI, she said. 

Efforts to ensure safe and trustworthy AI are happening, though. For instance, StateRAMP, an organization providing a security maturity assessment approach for cloud services, is participating in the National Institute of Standards and Technology’s Generative AI Public Working Group. Established last July, the group studies best practices for AI security.

One way agencies can limit their vulnerability to commercial AI solutions that may have unintended or harmful consequences is by using the procurement process to acquire the most responsible AI possible.

Bolstering security through procurement is two-fold, said Leah McGrath, StateRAMP’s executive director. Buying AI solutions is about setting new policies but also applying tried-and-true processes.

“When you’re going through procurement, it’s understanding how sensitive is the input, the outcome, what we’re asking this AI tool to do, and what is the learning model based on,” McGrath said. “We want to understand … potential influences that could negatively influence the outcome that we’re trying to achieve.”

But security isn’t the only aspect of AI procurement that agencies need to consider. After the technology debuted on its State CIO Top Ten Priorities list for 2024 at No. 3, NASCIO issued an AI blueprint for agencies thinking of adopting it. The blueprint  lists 12 recommendations, including  developing acquisition and procurement guidelines and identifying potential use cases.

Tying AI’s potential to a specific problem is important, said Zac Christensen, deputy chief cooperative procurement officer on the NASPO ValuePoint team, the contracting arm of the National Association of State Procurement Officers. 

Agencies should first ask how AI can benefit the organization, he said. Then, they should look at the various tools, such as generative AI, machine learning and robotic process automation. 

They then need a strategy for how they will “incorporate AI into their current processes,” Christensen said. If agencies “put together that plan [that goes] as far as identifying ‘we can use AI to help automate these tasks’ or ‘we can use it to help provide insight or additional data,’ then they can actually go forward and build that and have a really successful implementation.”

As an example, he pointed to Michigan, where the central procurement office is using AI for vendor contract negotiation. When a business wants to modify its terms, the office runs the request through the algorithm to determine whether it had accepted such a term in the past. With that application, the procurement team found that they “just escalated their negotiation process substantially,” Christensen said.

McGrath said that to formulate an AI procurement strategy, agencies should ask whether the purchase meets agency policies for AI use and what learning model the tool uses, including what datasets it pulls in and where potential negative influencers exist.

Having the right people in place is also essential, Jamia McDonald, principal at Deloitte Consulting, told Route Fifty in an email. A leading best practice in AI procurement is having a leader with “a clear vision for what AI will achieve and the internal authority to actively participate in the implementation effort,” she said. Agencies will also need “a multidisciplinary team with experienced professionals who understand the technology.”

Procuring AI is an important decision that requires work to get right. “State and local governments should also consider using market research, convening forums and communities of practice, and utilizing requests for information for additional clarification, where needed, before developing the procurement instrument,” McDonald said.

But even with controls, governance, policies and strategies in place, the length of the procurement process could make it hard to get effective AI into agencies in time for it to be effective. “Things are being invented every day, and the procurement process is 18 months long,” NASCIO’s Ward said. “By the time we get to the end, the technology they wanted to procure might be obsolete or it might be outdated.”