There are many
official dumps of Wikipedia available for download. Most are intended to be loaded into a MediaWiki database to run a Wikipedia clone. However, there are also
static HTML dumps: every page is pre-rendered to HTML using MediaWiki's ordinary parser, so you can just dump it all on a web or file server.
Why is this useful? To set up fast read-only copies on intranets not connected to the Internet, or when your connection is slow or sporadic. On your laptop, say.
One caveat: the static HTML dump is about 5.5 GB large (with 7zip), but comes out to roughly 80 GB uncompressed, with many millions of files. (78GB actual disk usage on a reiser3 FS, YMMV.)
Why so large? Well, it turns out that this dump really does include
every page in Wikipedia. Including every talk page. After removing those, as well as user pages, image-description pages, and miscellania like AFD pages and WP: and Wikipedia: namespace stuff, I'm down to 36GB (in some 3.5 million files IIRC). Not a real problem nowadays.
I compressed it again with tar/bzip2, and it went down to 3.7GB - fitting nicely on a DVD. 7zip probably would have been better, but as it requires scanning all the files-to-be-compressed as a first pass before actually compressing them, my poor 2GB RAM soon ran out and I had to kill the process. I guess my workstation isn't in the league of the mighty Wikipedia servers.
Of course, all this doesn't include the actual images. Those are available as a
separate download (made to match the DB dumps, not the static HTML dump) which was 76GB large back in 2005. These days it's too large for a simple download, and is distributed as a > 200GB torrent. At least I hope it wouldn't be any bigger uncompressed. Still, I've got the entire English Wikipedia on a DVD!
Now there's just one thing marring my joy: the latest available static HTML dump is from October 2006. In contrast, the latest DB dumps are from April 04 2007 - less than a month ago. So the obvious next step is to setup a local MediaWiki installation with a WP DB dump and run the static-dump script from that. I wonder if it would finish running before a new official dump becomes available. Or before
Wikipedia 1.0 is released.
Owner login