Policies in the age of AI Hallucinations

In our everyday life, we wouldn’t rely on a person for advice who is known to hallucinate from time to time in their advice-giving.  For the same reason, we cannot solely rely on generative AI tools for policy advice and development.

In this post, we focus particularly on the AI risk of  “hallucination” and error and how best to manage these risks.

Hallucinations

Hallucinations are a well-known risk of using generative AI. They occur when an AI model makes up facts to respond to a prompt. They reflect that AI models are predictive systems designed to produce the most probable and plausible answer, not necessarily the most accurate or truthful answer.

It can be hard to identify an AI hallucination because they are typically framed in a convincing way.

Why are AI outputs so convincing when they are wrong?

I asked ChatGPT this question.

In its own words, the chatbot explained that “it was trained to sound convincing, not to be right.” In other words, the chatbot’s hallucinating is due to its  training that a confident answer is more likely to be viewed as helpful than a hesitant answer; likewise that an answer that has the indicators of expertise (like tone and terms) is more likely to be seen as credible and reliable.  AI has learned and reflects the shape and appearance of expertise without necessarily having the expertise itself.

This is quite a different scenario from how AI is sold –  see, for example, the description of ChatGPT5 by the CEO of OpenAI as like having a “team of Ph.D. level experts in your pocket.” (NBC News Aug. 8, 2025)

AI at The Policy Place

At The Policy Place we use AI to assist our policy development and review work. We treat it like a junior policy assistant who can help us with a range of tasks like initial drafts, summaries etc We use other sources too like legislation, regulations, government websites, academic research and court and tribunal decisions for the development, reviews and updating of policies and procedures.

We have previously posted about the highly publicised Deloitte case where AI-generated citations used in a report for the Australian Government were found to be wrong and included fictitious citations. There have also been a number of legal cases reported overseas of AI used in cases and found to have produced fictional case citations and other inaccuracies. See here for a good list of Australian examples.

We understand how easily mistakes like this could happen. Unlike other sources we use, we find that checking AI outputs for hallucinations and errors is hugely time-consuming.

It is not only hard to spot hallucinations. When using AI, we have noticed that more data is generated by our prompting than if we did the whole task by hand.  Sometimes, this is helpful and right on point. Other times, it can be completely superfluous and tie us needlessly up in checking and re-checking processes.

So we’re still a work in progress, striving for the productivity and efficiency gains of AI use while wanting to maintain our high standards for accuracy and quality in our policies.

Can AI check and verify?

If only we could rely on AI to do this. But we can’t.

At best we can ask AI to verify its outputs against its own training data. It cannot check and verify its outputs against sources like legislation, organisational documents, academic databases and expert reports. It cannot assess the truth or veracity of something.

With RAG – Retrieval-Augmented Generation – things are better. Hallucination risks are significantly reduced because AI answers are grounded in authorised content. AI outputs are also more consistent.  But the truth and reliability of AI outputs depends on the authorised content/data.

Thinking about AI for policies?

If you’re thinking about using AI for your policies, think beyond the promises and “sell” of AI. Ensure you have the expertise and knowledge to check the AI outputs for quality, accuracy and hallucinations. Be pro-active about managing the risks of hallucinations and errors and ensure you have good policy guidelines for effective governance and management of AI.

Wanting to outsource your policies and procedures and the assurance of relevant policy expertise? Contact us NOW at The Policy Place.