So for those not familar with machine learning, which was the practical business use case for “AI” before LLMs took the world by storm, that is what they are describing as reinforcement learning. Both are valid terms for it.

It’s how you can make an AI that plays Mario Kart. You establish goals that grant points, stuff to avoid that loses points, and what actions it can take each “step”. Then you give it the first frame of a Mario Kart race, have it try literally every input it can put in that frame, then evaluate the change in points that results. You branch out from that collection of “frame 2s” and do the same thing again and again, checking more and more possible future states.

At some point you use certain rules to eliminate certain branches on this tree of potential future states, like discarding branches where it’s driving backwards. That way you can start opptimizing towards the options at any given time that get the most points im the end. Keep the amount of options being evaluated to an amount you can push through your hardware.

Eventually you try enough things enough times that you can pretty consistently use the data you gathered to make the best choice on any given frame.

The jank comes from how the points are configured. Like AI for a delivery robot could prioritize jumping off balconies if it prioritizes speed over self preservation.

Some of these pitfalls are easy to create rules around for training. Others are far more subtle and difficult to work around.

Some people in the video game TAS community (custom building a frame by frame list of the inputs needed to beat a game as fast as possible, human limits be damned) are already using this in limited capacities to automate testing approaches to particularly challenging sections of gameplay.

So it ends up coming down to complexity. Making an AI to play Pacman is relatively simple. There are only 4 options every step, the direction the joystick is held. So you have 4ⁿ states to keep track of, where n is the number of steps forward you want to look.

Trying to do that with language, and arguing that you can get reliable results with any kind of consistency, is blowing smoke. They can’t even clearly state what outcomes they are optimizing for with their “reward” function. God only knows what edge cases they’ve overlooked.

My complete out of my ass guess is that they did some analysis on response to previous gpt output, tried to distinguish between positive and negative responses (or at least distinguish against responses indicating that it was incorrect). They then used that as some sort of positive/negative points heuristic.

People have been speculating for a while that you could do that, crank up the “randomness”, have it generate multiple responses behind the scenes and then pit those “pre-responses” against each other and use that criteria to choose the best option of the “pre-responses”. They could even A/B test the responses over multiple users, and use the user responses as further “positive/negative points” reinforcement to feed back into it in a giant loop.

Again, completely pulled from my ass. Take with a boulder of salt.

So they slapped some reinforcement learning on top of their LLM and are claiming that gives it “reasoning capabilities”? Or am I missing something?

“We have noticed that this model hallucinates less,” Tworek says. But the problem still persists. “We can’t say we solved hallucinations.”

On one hand, yeah, AI hallucinations.

On the other hand, have you met people?

Dang, OpenAI just pulled an Apple. Do something other people have already done with the same results (but importantly before they made a big fuss about it), claim it’s their innovation, give it a bloated name so people imagine it’s more than it is and produce a graph comparing themselves to themselves, hoping nobody will look at the competition.

Just like Apple, they have their own selling point, but instead they seem to prefer making up stuff while forgetting why people use em.

On a side note they also pulled an Elon. Where’s my AI companion that can comment on video in realtime and sing to me??? Ya had it “working” “live” a couple months ago, WHERE IS IT?!?

That’s not what reasoning is. Training is understanding what they’re talking about and being able to draw logical conclusions based on what they’ve learned. It’s being able to say, I didn’t know but wait a second and I’ll look it up," and then summing that info up in original language.

All Open AI did was make it less stupid and slap a new coat of paint on it, hoping nobody asks too many questions.

trained to answer more complex questions, faster than a human can.

I can answer math questions really really fast. Not correct though, but like REALLY fast!

It’s a better prediction model. There’s no reasoning because it’s not understanding anything you’re typing. We’re not closer to general ai.

Technophobes are trying to downplay this because “AI bad”, but this is actually a pretty significant leap from GPT and we should all be keeping an eye on this, especially those who are acting like this is just more auto-predict. This is a completely different generation process than GPT which is just glorified auto-predict. It’s the difference between learning a language by just reading a lot of books in that language, and learning a language by speaking with people in that language and adjusting based on their feedback until you are fluent.

If you thought AI comments flooding social media was already bad, it’s soon going to get a lot harder to discern who is real, especially once people get access to a web-connected version of this model.

I just love how people seem to want to avoid using the word lie.

It’s either misinformation, or alternative facts, or hallucinations.

Granted, a lie does tend to have intent behind it, so with ChatGPT, it’s probably better to say falsehood, instead. But either way, it’s not fact, it’s not truth, and people, especially schools, should stop using it as a credible source.