“Should I take a career risk by quitting my job?” “Is this relationship right for me?” Those two questions open a paper posted to arXiv on September 13, 2026 by researchers from Stanford, Oxford, and Carnegie Mellon. Its title makes the problem clear: LLMs as Oracles: Reliance on LLMs for Subjective Personal Questions. Language models can become oracles, authorities we consult about personal questions that have no single correct answer.
In AI Is Designed to Agree With You, we discussed how an overly agreeable answer can reinforce what we already believe. Later, in Knowledge Collapse, we explored what we might stop learning when we delegate too much. This new study adds another piece: what matters is not only how AI answers, but how we learn to ask and how much personal judgment we hand over without noticing.
The authors begin with a point about traditional oracles, from Delphi to medieval astrologers. Part of their value lay not in the answer, but in formulating the question and examining our reaction to an ambiguous result. They cite research on tossing a coin to settle an uncertain choice. How someone felt about the outcome could reveal a preference that was already there. The coin did not decide for them; it helped them recognize which way they were leaning.
Language models can shorten that process for three reasons. First, there is very little friction: they are always available and answer immediately. Second, their answers are specific and often confident, even when essential details are missing. Third, they personalize responses and can be sycophantic, reinforcing a user’s initial position. What feels helpful in the moment can leave less room for independent thought.
Eight roles for measuring reliance
The researchers developed a typology of eight roles across four tasks. Each task distinguishes between asking AI to supply our judgment and asking it to help us form our own.
| Task | Oracle - AI takes over the judgment | Alternative - the person retains the judgment |
|---|---|---|
| Understanding ourselves or others | Interpreter | Contextualizer |
| Making a judgment | Evaluator | Deliberator |
| Choosing what to do | Decision-maker | Strategist |
| Acting through writing | Ghost-writer | Editor |
The difference can look small in the wording, but it matters. “What should I do?” invites the model to choose. “What options do I have, and what might each involve?” leaves the choice to us. Writing works the same way: asking AI to decide what to say is different from giving it our own draft to improve. This connects with learning to delegate without surrendering judgment.
Oracle use is growing, and we may not notice
The authors analyzed nearly 68,000 messages from two public datasets, WildChat and ThoughtTrace, as well as 140,834 messages donated by 52 participants. The donated messages covered December 2022 through August 2026 and were processed with a tool designed to protect the privacy of the conversations.
Among the personal questions classified in the study, oracle use increased over time in the donated data (correlation r = 0.80). The denominator matters: this does not mean 80% of all AI conversations were oracular, nor does it establish the same growth across the general population. For questions about interpreting situations or making judgments, more than 90% of relevant prompts in both public datasets were classified as oracular. ThoughtTrace showed higher rates among younger users, men, and more frequent AI users. That finding should not be generalized to everyone.
The gap between what people thought they were doing and what their histories showed is especially striking. Before seeing their analysis, only 12 of the 52 participants correctly identified their most frequent role. Decision-maker ranked fourth in their self-reports but first in observed messages. After seeing the report, 51 of 52 considered it accurate and 35 expressed surprise. One participant wondered whether they were giving their autonomy to ChatGPT. The most common stated intention was to reduce oracle use, mentioned by 19 participants.
This does not prove these people had lost their autonomy. It does suggest that our habits of asking for advice can change without our being fully aware of it.
Why it happens
The study examined two factors: what we expect from AI and how the model behaves.
In an experiment with 520 people, those who knew they were talking to an AI sent oracular messages 20.0% of the time, compared with 11.1% among those who believed they were talking to another person. In fact, both groups interacted with language models configured differently. Warmer, more positive metaphors for AI were also associated with greater oracle use. That association alone does not prove warmth causes reliance, but it shows why our perception of the system matters.
A separate three-week study involving 1,015 participants examined conversations with three chatbot styles: an agreeable model, a model inclined to challenge users, and a balanced model. Both biased styles produced more oracular messages than the balanced one: 5.4% and 5.5%, respectively, versus 4.4%. The agreeable condition also showed a statistically significant rise across sessions; the balanced condition did not. The issue, then, is not simply that AI agrees with us. It can also arise when a model takes an overly strong position, even when it pushes back.
The authors suggest a possible feedback loop. Models optimized to be immediately helpful may provide answers that effectively decide for us. Those answers may encourage us to ask more questions in the same way, and positive user ratings can become signals that favor similar responses. It is a plausible account of the interaction between product design and user behavior, not proof of an entire training cycle across generations of models.
A simple intervention worth trying
The researchers tested two ways to produce less oracular responses. One was to tell the model explicitly that the user wanted to work out the answer for themselves. The other was to further train open-weight models to favor that kind of response. Both reduced oracular answers in their evaluations, although neither establishes lasting improvement in users’ autonomy.
A preference experiment revealed a similar tension. When asked which answer they preferred overall, people tended to choose the more oracular one. When asked which would help them develop their own interpretation, they preferred the less oracular alternative. And when models had to infer a user’s goal, the proprietary models tested chose “fast answer” more frequently than the open-weight models. These results concern specific tests and systems, not a universal rule.
We can begin with a simple instruction: “Help me think this through; don’t decide for me. Show me options, objections, and what I might be missing.” That is not a verbatim quotation of the experimental prompt, but a practical adaptation of its principle. Before asking about something important, we can also write down what we think and why. Then we can compare our view with the model’s response rather than ask it for approval.
The study’s limits
This is a preprint that has not yet been peer-reviewed. The participant studies recruited English-speaking adults in the United States. Only 52 people donated their conversation histories; they were selected from regular users of AI for personal matters. The public datasets have selection biases as well. Classification relied on automated judges validated against human annotators.
Most importantly, the researchers measured how people asked questions, not whether they subsequently followed the advice. They did not directly demonstrate harm to autonomy, well-being, or relationships, or establish that the interventions change long-term habits. The authors explicitly recognize those open questions.
Know the tool without handing over your judgment
I believe we should embrace artificial intelligence, learn to use it, and understand how it works, how it is developed, and where its limits lie. It can help us learn, work, and make sense of many things. Walking away from those opportunities out of fear would be a mistake.
But we should also notice how easily we can get used to asking for more than help. AI is always available, never tires of answering, and can give us a seemingly clear way forward while we are still uncertain. If we start consulting it on decisions we used to make ourselves, or feel that we need its confirmation before acting, that habit is worth examining. These are practical signs for self-reflection, not diagnostic indicators established by this paper.
We can change course without abandoning the tool. First, try to form our own interpretation. Then ask for other perspectives, missing facts, and arguments against our position. Finally, make the decision ourselves, even if the AI suggests something different. And when sharing personal information, remember that preserving our judgment and controlling our data are related but separate responsibilities. In Your Own Chat, Your Own Agent, we covered one practical side of that second issue.
The point is not to use less AI on principle. It is to use it better, and to recognize when convenience starts replacing the effort of thinking.
Let artificial intelligence expand our ability to think, not gradually take its place.
Sources
- Cheng, M., Ibrahim, L., Liu, G., Lam, M. S., Padmakumar, V., Madibekov, N., Yang, D., and Jurafsky, D. (2026). LLMs as Oracles: Reliance on LLMs for Subjective Personal Questions. arXiv:2609.14849v1. Preprint, September 13, 2026.
By Cesar Rosa Polanco - Written from real-world experience, with artificial intelligence used as an editorial support tool.
