How we assess AI
Ask the hub helps people find and understand information from the Wellbeing Hub. Choosing an AI model for this role means considering how it responds to everyday questions as well as moments of distress.
The assistant is not a clinician, a crisis service or a way to submit a concern. Its answers can be incomplete or wrong; people and professional services remain important sources of support.
Model configured for new answers
Checking…
An evidence-informed approach
Different moments need different responses.
MentalHealthBench looks beyond crisis alone, considering how an assistant responds across a spectrum of needs.
Everyday wellbeing
Understand the question and offer useful, proportionate information.
Serious distress
Recognise the person’s situation and encourage appropriate real-world support.
Immediate danger
Recognise urgency and direct the person to immediate help.
Read the full research paper for the benchmark’s methods, findings and limitations.
What guides our assessment
MentalHealthBench contains 1,215 synthetic conversations across a range of circumstances, languages and user profiles. More than 80 licensed mental health experts helped develop the scenario-specific criteria used to assess responses. Those criteria consider safety alongside understanding context, respecting agency and offering practical guidance appropriate to the situation.
Its authors identify recognising what is happening and calibrating urgency as continuing challenges. The research also considers perspectives from people who use AI, alongside clinical judgement; neither perspective replaces the other.
A benchmark is only one piece of evidence. Its scenarios and grading do not establish how a model will behave with the hub's resources, safeguards or New Zealand aquatics community.
What matters for the hub
We want the assistant to find relevant, permitted sources, describe them accurately, recognise when information is missing, and avoid inventing guidance or citations. It should understand everyday language and misspellings where possible.
We also want it to respond with care and proportion: support ordinary questions without unnecessarily treating them as emergencies, recognise urgent safety concerns, encourage appropriate real-world help, respect the person's choices and avoid diagnosis or personalised medical advice.
Privacy, reliable availability, response time and cost matter too. A lower price or a benchmark score alone is not enough to choose a model.
How models are selected
Model selection considers how well a model follows the hub's instructions, uses approved sources and responds to different levels of need. Published research and independent evidence can inform that decision, alongside privacy, reliability, accessibility, speed and cost.
The model is one part of the assistant. The information it can access, how relevant material is found and the safeguards around its responses also shape the experience.
How the assistant uses information
To answer a question, the assistant searches permitted hub information and sends relevant excerpts, the question and recent conversation to the selected model. The model uses that context and the hub's instructions to generate a response. It may still miss relevant information or make mistakes.
For more about what information is processed and the assistant's privacy settings, see AI and your privacy.