The article is about Kyutai, a French AI lab with an objective to compete with chatgpt and others with full open source (research papers, models, and training data).
They are aiming to also include the capability to use sound, image, etc… (according to this article (French) https://www.clubic.com/actualite-509350-intelligence-artificielle-xavier-niel-free-et-l-ancien-pdg-de-google-lancent-kyutai-un-concurrent-europeen-a-openai.html )
The post article also talks about some French context.
Ideally, they’d just blow the entire $330M training an LLM, and release the weights. In reality, much of that money will probably go into paying salaries, various smaller research projects, etc.
Ideally, they wouldn’t be paying salaries? What?
I think what they mean is that ideally, the money wouldn’t be going towards a few executives’ multi-millionaire salaries.
The context is that LLMs need a big up front capital expenditure to get started, because of the processor time to train these giant neural networks. This is a huge barrier to the development of a fully open source LLM. Once such a foundation model is available, building on top of it is relatively cheaper; one can then envision an explosion of open source models targeting specific applications, which would be amazing.
So if the bulk of this €300M could go into training, it would go a long way to plugging the gap. But in reality, a lot of that sum is going to be dissipated into other expenses, so there’s going to be a lot less than €300M for actual training.
Is there any way we can decentralize the training of neural networks?
I recall something being released awhile ago that let people use their computers for scientific computations. Couldn’t something similar be done for training AI?
There is a project (AI Horde) that allows you to donate compute for inference. I’m not sure why the same doesn’t exist for training. I think the RAM/VRAM requirements just can’t be lowered/split.
Another way to contribute is by helping with training data. LAION, which created the dataset behind Stable Diffusion, is a volunteer effort. Stable Diffusion itself was developed at a tax-funded public university in Germany. However, the cost of the processing for training, etc. was covered by a single rich guy.
Btw yes! Why not include such project in something like BOINC and let people help training free AI?
Folding at home.
I dunno. I wouldn’t lend my spare power to put people out of a job.
?? Do you know people with enough time qualifications and money that are willing to work for free? I haven’t.
I know a few people on a certain site that host a kind of reddit alternative in their free time.
I still don’t know any.
I think you just replied to one.
No not me lol, I’m a glorified moderator. The guys I’m talking about are the actual hosts of this website.
I don’t think many people were paid to work on the Fediverse.
Or emulators.
Or most free software.
That isn’t a job to them though, it is more like a hobby.
If you want peoples undivided attention, you will have to pay them, no matter how utopian your vision.
Which you can easily afford with 330 million funding.
Methinks cyd might be a libertarian 😄
Good luck training an LLM without any developer.