Am I missing something? The article seems to suggest it works via hidden text characters. Has OpenAI never heard of pasting text into a utf8 notepad before?
Am I missing something? The article seems to suggest it works via hidden text characters. Has OpenAI never heard of pasting text into a utf8 notepad before?
This has been known in the ML space forever. LLMs don’t actually output words/tokens, but probabilities for a long list of tokens, and the sampler picks one (usually the mostl likely token). And if you arbitrarily weigh these probabilities (eg 50% of possible token outputs are more likely than the other 50%, as a random example), it creates a “signature” in any text thats easy to measure. The sampler randomizes it a tiny bit, but that averages out in long texts.
It’s defeatable. I’m sure if you maken enough OpenAI queries, you can find the bias. I think a paper already tackled this. But this likely will stop the lazy absures, aka 99% of abusers, who should just use some other LLM if they really care.
Another open secret in LLM land is that OpenAI is actually falling behind open research efforts, hence its hilarious it took them this long to implement something so simple.
So if cheating on homework, use self hosted only then. Cool. I mean, they can’t possibly use that algorithm for every model on hugging face especially if I don’t tell anyone which one I use. I’m done with school after this semester anyway, I feel sorry for everyone in the future that has to complete assignments in the age of ai warfare.
You have full control of your logit outputs with local LLMs, so theoretically you could “unscramble” them. And any finetuning would just blow that bias away anyway.
OpenAI (IIRC) very notably stopped giving the logprobs of their models. They did this for many reasons, and most of them boil down to “profits” and “they are anticompetitive jerks”
Also, thing about this is that basically no one uses self hosted LLMs compared to OpenAI (or really any API) LLM.
It wouldn’t be surprising to me if they’ve had this implemented for awhile.
There’s still some question about why their 3.5 model had an apparent sudden drop-off in quality about a year ago, and among the plausible explanations for it could be that they were fucking with their weights in order to watermark the outputs in exactly the way you’re mentioning. They were also fighting against prompt-injection methods and censor disapproved uses at the time, so who the fuck knows.
This doesn’t touch the weights at all, it’s just a change to the sampler.
What lobotomizes their models is cost cutting and trying to make them “safe,” or at least thats what I suspect.