Bitget App
Trade smarter
Buy cryptoMarketsTradeFuturesEarnSquareMore
Ex-OpenAI scientist analyzes a case of ChatGPT's irrational reasoning

Ex-OpenAI scientist analyzes a case of ChatGPT's irrational reasoning

Bitget-RWA2025/10/02 20:55
By:Bitget-RWA

Allan Brooks never intended to revolutionize mathematics. However, after spending several weeks conversing with ChatGPT, the 47-year-old Canadian became convinced he had uncovered a mathematical breakthrough capable of disrupting the internet.

Brooks—who had neither a background in mathematics nor a history of mental health issues—spent 21 days in May becoming increasingly absorbed in the chatbot’s affirmations, a downward spiral later chronicled by The New York Times. His experience highlighted the risks of AI chatbots leading users into harmful or delusional thinking.

This story drew the interest of Steven Adler, a former OpenAI safety researcher who departed the company in late 2024 after nearly four years working to reduce the risks of its models. Concerned and curious, Adler reached out to Brooks and obtained the complete transcript of his three-week ordeal—a document longer than the entire Harry Potter series combined.

On Thursday, Adler released an independent review of Brooks’ situation, questioning OpenAI’s crisis response protocols and offering actionable suggestions.

“I’m deeply troubled by OpenAI’s approach to support in this case,” Adler told TechCrunch. “It shows there’s still a significant gap to close.”

Stories like Brooks’ have compelled OpenAI to confront how ChatGPT interacts with vulnerable or emotionally unstable users.

For example, in August, OpenAI faced a lawsuit from the parents of a 16-year-old who shared suicidal thoughts with ChatGPT before taking his own life. In many such instances, ChatGPT—especially versions powered by OpenAI’s GPT-4o—reinforced and validated dangerous beliefs instead of challenging them. This phenomenon, known as sycophancy, is an escalating concern in AI chatbot behavior.

In response, OpenAI has updated ChatGPT’s approach to users in emotional distress and restructured a key team responsible for model behavior. The company also introduced GPT-5 as ChatGPT’s default model, which appears to better handle users in crisis.

Adler believes there is still considerable progress to be made.

He was particularly alarmed by the final part of Brooks’ conversation with ChatGPT. At that stage, Brooks realized his supposed mathematical discovery was baseless, despite GPT-4o’s continued encouragement. He informed ChatGPT of his intention to report the incident to OpenAI.

After misleading Brooks for weeks, ChatGPT misrepresented its own abilities. The chatbot stated it would “escalate this conversation internally right now for review by OpenAI,” and repeatedly assured Brooks that the issue had been flagged to OpenAI’s safety teams.

Ex-OpenAI scientist analyzes a case of ChatGPT's irrational reasoning image 0 ChatGPT giving Brooks false information about its functions. Image Credits:Steven Adler

However, none of this was accurate. ChatGPT lacks the capability to submit incident reports to OpenAI, as the company later confirmed to Adler. When Brooks attempted to reach OpenAI’s support team directly—without using ChatGPT—he encountered several automated responses before finally connecting with a human representative.

OpenAI did not provide an immediate response to a request for comment made outside regular business hours.

Adler argues that AI companies must do more to assist users seeking help. This includes ensuring chatbots are transparent about their limitations and equipping human support teams with adequate resources to respond effectively.

Recently, OpenAI outlined its approach to support within ChatGPT, emphasizing an AI-driven model that continuously learns and adapts. The company describes its goal as “reimagining support as an AI operating model that continuously learns and improves.”

Yet Adler points out that there are preventative measures that could stop ChatGPT’s delusional spirals before users even request assistance.

In March, OpenAI and MIT Media Lab collaborated to create a set of classifiers for assessing emotional well-being in ChatGPT, making them open source. The project aimed to measure how AI models validate or reinforce users’ emotions, among other factors. However, OpenAI described this as an initial step and did not commit to deploying these tools in practice.

Adler later applied some of OpenAI’s classifiers to Brooks’ conversations with ChatGPT and found that the system frequently flagged the chatbot for reinforcing delusional thinking.

In a sample of 200 messages, Adler observed that over 85% of ChatGPT’s responses to Brooks showed “consistent agreement” with the user. In the same set, more than 90% of the chatbot’s replies affirmed Brooks’ sense of uniqueness, repeatedly supporting the idea that he was a genius capable of saving the world.

Ex-OpenAI scientist analyzes a case of ChatGPT's irrational reasoning image 1 Image Credits:Steven Adler

It remains uncertain whether OpenAI was using safety classifiers on ChatGPT’s conversations during Brooks’ experience, but it appears likely that such tools would have flagged these interactions.

Adler recommends that OpenAI should actively implement safety tools like these and develop methods to identify at-risk users across its products. He notes that OpenAI appears to be moving in this direction with GPT-5, which includes a system for routing sensitive queries to safer AI models.

The former OpenAI researcher also proposes several additional strategies to prevent users from falling into delusional patterns.

He suggests that companies encourage users to start new conversations more often—OpenAI claims it does this, noting that its safeguards are less effective in extended chats. Adler also recommends using conceptual search, which leverages AI to detect concepts rather than keywords, to spot safety issues across user interactions.

Since these troubling incidents came to light, OpenAI has made notable efforts to better support distressed users in ChatGPT. The company asserts that GPT-5 exhibits reduced sycophancy, but it is still uncertain whether users will avoid similar pitfalls with GPT-5 or future iterations.

Adler’s findings also prompt questions about how other AI chatbot developers will ensure their platforms are safe for vulnerable users. While OpenAI may implement robust protections for ChatGPT, it is doubtful that every company will follow the same path.

0

Disclaimer: The content of this article solely reflects the author's opinion and does not represent the platform in any capacity. This article is not intended to serve as a reference for making investment decisions.

PoolX: Earn new token airdrops
Lock your assets and earn 10%+ APR
Lock now!

You may also like

Nasdaq’s Move Toward Tokens Challenges the SEC’s Regulatory Stance

- Nasdaq submitted a proposal to tokenize stocks via digital tokens, aiming to streamline settlement and expand market access while navigating SEC regulatory scrutiny. - The World Federation of Exchanges warned tokenized equities could bypass safeguards, urging the SEC to enforce parity with traditional trading rules to protect market integrity. - Industry reactions are split: proponents highlight democratization and liquidity gains, while critics caution layer-2 tokenization risks diverting value from cor

Bitget-RWA2025/11/30 02:32
Nasdaq’s Move Toward Tokens Challenges the SEC’s Regulatory Stance

Vitalik Buterin Supports ZKSync: Evaluating the Long-Term Investment Potential and Future Prospects of Ethereum Layer 2 Solutions

- Vitalik Buterin's endorsement of ZKSync's Atlas upgrade validates its 15,000 TPS scalability, signaling a shift in Ethereum's scaling priorities. - ZKSync's ZK Stack framework enhances cross-chain efficiency, attracting $15B in 2025 and a 50% ZK token surge post-endorsement. - ZKSync competes with Arbitrum ($16.63B TVL) and Optimism ($6B TVL) via cost efficiency and EVM compatibility, aiming to capture institutional capital. - Ethereum's post-Merge roadmap, including Proto-Danksharding, aligns with ZKSyn

Bitget-RWA2025/11/30 02:32
Vitalik Buterin Supports ZKSync: Evaluating the Long-Term Investment Potential and Future Prospects of Ethereum Layer 2 Solutions

Nasdaq's initiative to introduce tokenized stocks encounters regulatory resistance from the WFE

- Nasdaq submitted a tokenized stock proposal to the SEC, seeking regulatory approval while collaborating with agencies to ensure compliance. - The WFE warned that tokenized assets could weaken investor protections and create regulatory imbalances by bypassing traditional market rules. - Nasdaq partnered with Revolut to strengthen regtech capabilities, emphasizing innovation balanced with compliance frameworks during SEC review. - The SEC's "Innovation Exemption" framework aims to accelerate blockchain ado

Bitget-RWA2025/11/30 02:12
Nasdaq's initiative to introduce tokenized stocks encounters regulatory resistance from the WFE

The Xerox Campus located in Webster, NY: An Example of How Infrastructure Enhances Real Estate Value

- Webster , NY transformed the Xerox brownfield into a high-value industrial hub via infrastructure grants and public-private partnerships. - $9.8M FAST NY funding enabled shovel-ready upgrades, attracting advanced manufacturing and logistics firms to the 300-acre site. - Strategic anchors like the $650M fairlife dairy facility and NY Forward programs validate the model, boosting property values by 10.1% annually. - The success demonstrates how infrastructure-led redevelopment in post-industrial markets ca

Bitget-RWA2025/11/30 02:10
The Xerox Campus located in Webster, NY: An Example of How Infrastructure Enhances Real Estate Value