> This access seems to have only allowed the agents to make ‘GET’ requests, meaning they could fetch and read websites, but not interact with them, submit forms, or send data to them
The authors of this (very interesting) analysis should really not state the sandbox's wrong assumptions in their own voice.
GET absolutely allows you to interact with sites. And of course GET can also send information. It's all up to the server that receives the GET to decide what it let's callers do with it.
This is like the number one mistake I see juniors making with security. If I had a penny for every time someone was like "we don't need any backend checks because we have checks on the frontend"... If you don't have an adversarial mindset then you're not doing security.
Article says that it _can_ be calibrated, it just isn't out of the box. So the title seems contradicted by the body.
> If you want calibrated probabilities you’ll still need to recalibrate Jev’s probabilities on your own data. The good news is that is cheap. A few hundred labeled examples from your actual data can be enough to fit a Platt scaling on top of Jev’s scores.
Yes, good point. I meant that the Jev can't be calibrated for everyone out of the box. I find it important because this is one of their main claims. They literally say
> Outcomes assigned a probability of 0.2 should occur about 20% of the time.
I suspect they meant it as "they claim it is already calibrated, but it can't be, because there is no universally correct calibration", and not "it is not possible to calibrate Jev", but I read it as the latter at first too.
Hah! You independently picked exactly the same sentences I flagged (I know you posted this 11min before me but the comment only appeared after I had submitted mine).
That's the standard annoying pattern though: "Rewrites are declared by the publisher, never inferred from overlap." and "NULL means dirty, and DELETE is the fence." - still the same LLMisms. I didn't expect them to disappear, but it's not a radical improvement either.
This one is pretty terrible (right after “The worst outcome is a spurious rerun, never lost work.”). We’ve got lands, several "no X", hyphenation, strange noun/verb sentence order and an unnecessary analogy word (swallowed).
> There's no read-modify-write and no truncation of the dirty period, so a write that lands during a job can't be swallowed.
That's the main reason I'm using GPT models. I'll ask Fable to analyze something, then pipe its output straight through Astra without even looking at it first.
Of course it was exposed - not sure it's explicit or not. Why wouldn't HackerNews comments be part of the training data? And Simon's blog and the many discussions about Pelicans? It'd be hard to miss. Doesn't mean Anthropic has made this an explicit goal in training.
reply