Comptroller: Agencies need better AI guidance, oversight "This audit is a wake-up call"

Albany — The NYS Comptroller's Office released their audit of four State agencies and their usage of AI this week, finding "insufficient central guidance and oversight".

The agencies included: the Office for the Aging, the Department of Corrections and Community Supervision, the Department of Motor Vehicles, and the Department of Transportation (DOT).

CBS6 spoke with Tina Kim, Deputy NYS Comptroller, State Government Accountability, to break down the audit.

The Office for the Aging uses an AI Companion to combat loneliness.

"While the agency found that it was effective in combating loneliness, one of the concerns that we we had was the data so, so somebody's talking to this AI, and there is actually that information is being kept, and that data is actually owned by the company. It's actually not owned by the agency. And so that can potentially lead to issues related to somebody's privacy, and whether people understand that or not, is, you know, something that, basically, you actually also would want to understand," Kim said.

DOCCS has a policy which allows incarcerated individuals to communicate with a controlled list of family and friends. To facilitate that it uses a voice biometric software called "Investigator Pro". The software validates the voice of the inmate to identify if an individual is using someone else's personal identification number.

"So DOCCS had some of the data concerns. They own the data. A lot of AI applications, you worry about bias. You worry that some group is unfairly being targeted. And the concern that we had about DOCCS was they were keeping the data to understand how many times that when they were using voice recognition, the wrong person was identified so they did have somebody checking to see, you know when, when? Basically, the software identified somebody who they thought was not the wrong person, but you didn't keep data to tell me...Well, is it basically unfairly picking out a group of people who basically, and it might be, again, be biased, so they weren't keeping data. So that would, that would enable you to make that evaluation," Kim said.

The DMV has been using facial recognition since 2008, with the hope of improving safety and deter identity fraud through driver's license image matches.

"So DMV, and that's an interesting case, because DMV, in terms of, you know, governments, had probably the best program that we saw. They had their own policies and procedures, but they had a different definition of AI than what the state actually has. And under their definition, facial recognition, this facial recognition application, what they didn't consider it artificial intelligence, which basically, under the state definition, it clearly is. So they were not, you know, putting this application through the controls that they actually established. And again, they, of all of the agencies that we looked at, they had the best program, but yet they didn't consider this facial recognition application as AI, which, again, based on the state definition, it clearly was," Kim said.

The DOT is currently piloting three AI systems.

"That was an interesting application, because it was unclear based on our review, and basically what they told us whether that really was AI. So one of the things we recommended to them is that they have a conversation related with ITS to discuss that," Kim said.

To see the audit, click here.

"One of the things that you know that [Comptroller DiNapoli] is going to be doing going forward is proposing legislation so that our office audits this on a continuous basis for exactly the point that you're making...that technology evolves over time. Governance has to evolve over time. How you address risk, you have to move as the risks change, and the Comptroller wants to make sure that, you know, the state can achieve all the benefits that this technology can offer, but does it in a safe and responsible way, and that's the reason why he's putting forth legislation," Kim says.

In their response the Office for the Aging said initially they were not aware of the State's AI policy, and that it didn't apply to their usage of AI. (The Comptroller's Office argued otherwise).

The Office for the Aging went on to say that they reviewed the finding and would move to enact the recommendations listed.

DOCCS accepted the recommendations, saying they would move forward on implementation.

The DMV said disagreed with some of the audit. "DMV in fact has developed procedures to test its Document AI system, and has documented policies and procedures for training, testing and monitoring its performance to evaluate its accuracy." The DMV also argued their facial recognition program didn't fall under the definition of AI (which the Comptroller disagreed with).

The DOT agreed with recommendations and said they were taking steps to implement them.