In light of the recent Crowdstrike crash revealing how weak points in IT infrastructure can have wide ranging effects, I figured this might be an interesting one.
The entirety of wikipedia is periodically uploaded here, along with many other useful wikis and How To websites (ex. iFixit tutorials and WikiHow): https://download.kiwix.org/zim
You select the archive you want, then the language and archive version (for example, you can get an archive with no pictures, to save on space). For the totality of the english wikipedia you’d select the “wikipedia_en_all_maxi_2024-01.zim”
The archives are packed as .zim files, which can be read with the Kiwix app completely offline.
I have several USBs I keep that have some of these archives along with the app installer. In the event of some major catastrophe I’d at least be able to access some potentially useful information. I have no stake in Kiwix, and don’t know if there are other alternative apps and schemes, just thought it was neat.
This saved my ass at my engineering chemistry exam (still a requirement, even for software engineers) where only offline tools were allowed. Love Kiwix!
LOL… Malicious compliance at its best…
Is there a git repo for it or do I have to redownload the whole thing to do an update?
The text version of Wikipedia*
The images and other media are a hell of a lot more.
The 100Gb version mentioned above does only have thumbnails/lowres pictures, yeah. Better than nothing for some types of articles, but not everything. The true text-only version is actually only ~53Gb though.
Some of the high res photos are ridiculous.
Like a 8000x9000 uncompressed image of someone’s hand and weighs about 22mb.
I know that because I use a lot of royalty free images.
Is there an index of the images or something like that?
Without images Wikipedia is a “mere” 22.14gb.
I’ve installed game patches that were larger than this.
and you should donate to wikipedia if you are gonna do that
I couldn’t afford to donate for a long time but I used it near daily. So now I do monthly, probably larger than average, contribution to make up for sibs from other cribs that can’t afford it. Pay it forward is indeed a golden rule.
Do you wear a cape? Or are you one of those who doesn’t wear one?
No cape. I’m brown so I’m on the radar bad enough as it is as soon as I leave major cities lol.
So something akin to this joke image I saw the other day is actually feasible for Wikipedia?
I mean, you can self-host your own local LLMs using something like Ollama. The performance will be bound by the disk space you have (the complexity of the model you’re able to store), and the performance of the CPU or GPU you are using to run it, but it does work just fine. Probably as good results as ChatGPT for most use cases.
We do this at work (lots of sensitive data that we don’t want Openai to capitalize on) and it works pretty well. Hosted locally, setup by a data security and privacy sensitive admin, who specifically runs the settings to not save any queries even on the server. Bit slower than chatgpt but not by much
Chatgpt is also probably around 50-100GB at most
Probably a lot less, keep in mind that whenever it answers a question the whole model is traversed multiple times, going through multiple GBs is not possible in the matter of seconds the model answers.
Plus input data?
No, but it’s the model after the input that you need.
DYK that Kiwix was actually created by Wikipedia? Back in the late 2000s there was this gigantic effort to select and improve a ton of articles to make an offline “Wikipedia 1.0” release. The only remains of that effort are Kiwix, periodic backups, and an incredibly useful article-rating system.
Can you write more about the rating system you mentioned?
- There is a set of criteria to rate an article B, C, Start or Stub. These are called classes. Similarly, articles can be rated to be of 1 of 4 importance values to a particular WikiProject.
- There’s a banner on every article’s talk page. Any editor can change an article’s rating between one of the above classes boldly; if a revert happens, they discuss it according to the criteria.
- Some WikiProjects have their own criteria for rating articles. Some of them even have process to make an article A-class.
- Before this system, Wikipedia already had processes to make an article a Good Article or Featured article.
- With GAs, a nominator should put a candidate onto backlog. Later, a reviewer will scrutinize the article according to criteria. Often, the reviewer asks the nominator to fix quite a bit of issues. If these issues are fixed promptly, or the reviewer thinks that there are only nitpicks, the article passes. If they aren’t fixed in a week or the reviewer thinks that there are major problems, the article fails.
-
- As with other processes, the nominator and reviewer can be anyone, though reviewers are usually experienced.
- With FAs, a nominator brings the candidate to a noticeaboard. Editors there then come to a consensus about whether the article should pass.
- Both processes display a badge directly on passed articles.
- Both processes have an associated re-review process where editors come to a consensus whether the article should fail if it were nominated today
- There’s also an informal process called “peer review”, where someone just puts an article at a noticeable and anyone can comment about its quality.
- Articles are automatically sorted into categories by their rating and importance. Editors usually look at these to decide which articles to focus on nowadays.
I did! I do! Also all public domain books as part of the project Gutenberg
The benefit of text not taking up much space.
Imagine downloading it just after some troll changed critical information lmao
I imagine you could also download with all the history of every article
I tried to download it but couldn’t get it to work :(
Download the kiwix app for whatever OS you’re using, then go into Kiwix and click on the folder icon in the app and navigate to where the .zim file you downloaded is located. If you click it it should automatically pop-up and be viewable.
If you did that and it’s still failing, is it giving you a specific error or anything?
i remember a time when it was only 2gb for all of wikipedia. usain bolt had just burst onto the world stage at the time.
And by now he’s exited the solar system at incomprehensible speeds.
I am currently reading on terrorists while in the states. But something tells me I will get my IP banning me. But I have read a shitton and I highly doubt its just 100gb. Otherwise you would see it more on piracy sites.
But it’s freely and easily available to download, why would it be on piracy sites?
China is making a copy. For… reasons.
How high were you when you wrote this?
Currently where I am working can’t get high. But can get drunk. But was neither when I wrote that. My ISP is very brutal on looking up stuff or downloading shit.
Otherwise you would see it more on piracy sites.
What on Earth do you mean? Piracy sites share things which aren’t available easily for free otherwise.
https://en.m.wikipedia.org/wiki/Wikipedia:Database_download
And the text only version of Wiki is just 22.14gb.
Aside from the text clarification, this is also only the US version of Wikipedia.
What worries me though is that most videos linked on Wikipedia are hosted on YouTube. That’s a pretty dangerous choke point.
You mean the English version? There is no US version, thank god.
I never even noticed any videos on Wikipedia. Maybe for some cinema articles.
Videos aren’t an essential part of an encyclopedia.
Ten year old me would beg to differ.
Videos turned Encarta 95 from being an encyclopedia to the encyclopedia!
I jest - a multimedia experience helps but I agree that the text knowledge is the big draw.
The real ones remember wandering around that damn maze answering questions while managing limited torches to see the map.
I liked to look at the genitals of everyone possible
Thank you encarta
My brain immediately thought archive.org but after the last incident, I kinda feel like archive org is going to get lawsuited into oblivion
I tried searching but found nothing. What incident?
It’s already been done: https://m.youtube.com/watch?v=1lRI35gKSPA
What’s already been done…?
Sorry, I meant to reply to the commenter with the chatgpt on a dvd pic saying that it’s actually feasible for Wikipedia.
