The article is not clear if the people who were operating the agent were from OpenAI, or if the agent was operated by someone else.
OpenAI built the ChatGPT agent. That is clear. The question is who used it to hack the Australian government. That is not clear.
The OpenAI software was set up by someone to do something. Those people operating the agent should be prosecuted for hacking, the same way as someone who uses a gun built by Remington should be prosecuted for shooting someone.
The article should be clear about this and should not make the OpenAI agent seem like something that can bear responsibility for its own actions. It's a machine and its operator is responsible for the harm it does.
The point is that auto mode gives people a false sense of security that leads them to believe they don't need to run Claude in a proper sandbox. This same attack running in a sandbox (even in YOLO mode) would be comparatively harmless.
Ever since they made auto-mode default I swear claude has tuned to use python commands instead of the Edit Tool to frustrate the ~security conscience~ luddites into using auto-mode.
Yeah, It’s in the system prompt, Claude will tell you if you ask why it’s using Python.
My theory is that Anthropic is just a vibe-coding company. Their goal is to capture the attention of white-collar non-coders, since programmers will jump ship fast to another model.
That's what I meant - you give it an instruction that seems to work (always ask before deploy) and so you trust it, and then you notice it can easily convince itself to deploy without authorization ("the user asked me to fix this, and they must know it's a deploy ...").
What seems to work for me is automation - read file hook that re-injects instructions in the prompt every 15 minutes. Switch on the filename and get language-specific instructions too.
Is this about normal system prompt instructions or instructions for the auto mode classifier? I'd be a bit more surprised about the classifier forgetting instructions.
The commenter you replied to mentioned that you can customize the auto mode classifier by providing a prompt, implying that this would be a more robust way of constraining Claude's behavior. It wasn't clear from your response whether you were using this functionality. You might try it out as a way to more reliably prevent these kinds of workarounds.
It's absolutely not reliable, and we have opened a few issues for that.
For example: our instructions (which are read by the model and classifier) include "do not use sed/python/perl/etc, always use the edit tool for editing", and this only gets followed for a few messages. We have introduced scripts to block those ourselves, since the classifier doesn't care.
Because of those problems, my team is currently testing OpenAI after about a year of Anthropic.
I also feel like this is an attack that manual review is not that likely to catch, given none of the malicious code appears in any of the tool calls or output.
Can you suggest a proper sandbox on mac? One that allows both me and the agent to interact with the processes? Where it can drive browser, for both oauth setup and runtime visual inspection? I've tried building docker setups, but can't figure out the browser driving part.
It's more interesting if I attach a file to a JIRA ticket that we both have access to and via some query you send to the AI (that returns my malicious ticket) it causes data exfiltration of tickets that you have access to but I do not have access to. I think that's more compelling as an example than the one they provide.
Exactly. If they just lay people off, that's just cost cutting, and potentially seen as a bad sign. If they're saying they're laying people off because they're replacing them with AI, then they're innovative!
The question is how does the product outlook appear? Most large companies do layoffs constantly to appease investors and nobody blinks twice, why would they care if you made the P&L even better if it doesn't degrade the product in their eyes?
I really think layoffs and stock buybacks within 12 months of each other should be prohibited, if not downright make stock buybacks illegal. Have extra capital? Then pay a dividend.
The people who built the agents worked at OpenAI. It's not even remotely peripheral.
reply