/b/stats 2.0: beta edition
since 27 Jan 2013
Please send questions/comments/spam/death threats/love letters to webmaster (at) b-stats.org
All times EST/EDT
Active development is on GitHub
blackbox is dead
posted by terrance on 2016-07-09 8:30 pm
blackbox, the trusted full image server that I've had since 2011, seems to have kicked the bucket earlier today. All attempts to turn it on have failed. I hope it is a simple power supply issue, but it may be that the motherboard has finally given out.
No data was lost and the hard drives are fine. However, the cause for the server's death has still not been identified.
I now have the server running in a virtual machine until I can fix the old server or get a new one. What fun!
EDIT 3: (2016-07-12)
What was once blackbox is now raspberrypi. The hard drives have been placed into usb caddies and connected to a Raspberry Pi 2 Model B. This should go well.
For those curious, blackbox was an AMD Phenom II X2-based system with 8GB of DDR3 RAM, originally running Debian 6 but later upgrading to 7 and finally 8. It is still unknown what led to blackbox's premature demise. Perhaps running 24/7 for 5 years on cheap desktop-class hardware had something to do with it.
posted by terrance on 2016-05-17 5:07 am
I just noticed something strange that's preventing the /lgbt/ archiver from running.
The archiver reports that the `no` value is not set for some posts, which shouldn't happen, causing it to crash. I'm looking into it.
Update: the threads in question (6203746, 6200223, 6201726) have nearly-empty posts? I've no idea what's up with that. I pushed a quick fix to ensure that required values are always set.
Full Images Return
posted by terrance on 2016-05-04 6:18 pm
The full-image server is back up now.
Archiver Downtime :(
posted by terrance on 2016-05-01 4:18 am
Sorry folks, I've been in the midst of moving house the past two days and apparently in the middle of it all the archivers decided to crap out!
Because of this we are missing a little over one day's worth of posts on all boards, from near the end of the day (EST) on April 29 to a little after midnight (EST) on May 1.
I'm currently thinking of a way to restore the missing data.
On a related note, the full-image server is down for a bit and is not currently saving images. The thumbnail server has been going strong throughout all of this (although it has probably been very upset and lonely, having downloaded no new thumbs for the past 24 hours...)
posted by terrance on 2016-04-14 9:47 pm
We've got some new domains!
They're both currently pointing to the same server as archive.b-stats.org, but I may end up doing something else with them in the future. For now, they make it easy to get to an archived thread (change 'a' to '4', or 'org' to 'pw').
posted by terrance on 2016-04-14 3:47 pm
I'm currently generating thumbnails for /f/. All swfs should have thumbs within a day or two.
posted by terrance on 2016-04-04 10:54 pm
I've successfully imported 4ch.be's /trash/ archive, so it now goes back all the way to post 1! Thankfully the new DB schema on this site made it pretty easy to convert from FoolFuuka's format to my own. The only difference is that FoolFuuka doesn't save full HTML post content, so I had to fake that (with MariaDB's built-in regexp_replace). It should be pretty good, though.
posted by terrance on 2016-03-31 8:50 am
For the past week(s) I've been moving servers around quite a bit. I'm happy to say that the new servers are working great! The frontend server now has ample CPU and RAM, so I plan on implementing Sphinx search soon. For now, I've enabled page-based access to all past threads on all boards, as well as searching for image reposts.
I've also been somewhat hard at work rewriting a lot of the backend code, which you can follow on GitHub if you're into that. Hopefully this will make the archive easier to maintain as well as faster to use.
posted by terrance on 2016-02-17 7:12 am
Whoops. Yep, it's been over three years of archival. That's a lotta data.
posted by terrance on 2016-01-06 5:26 am
/trash/ is now being archived. In addition, I am attempting to import 4ch.be's /trash/ archive for a more complete record (theirs seems to go back to the board's beginning, or at least very close). More on that soon.
(/cm/ + /hm/) ÷ 2 = /y/
posted by terrance on 2015-10-30 12:57 am
I added /y/ to the list of archived boards, after Yaoi's recent portrayal in South Park.
Also, I fixed the issue mentioned in the previous news article.
Good News and Bad News
posted by terrance on 2015-09-02 6:07 am
Welp, I have some good news and some bad news.
The good news is that the full-image server now has 4TB (3.64TiB) hard drive, so we can save images for quite a bit longer than we could before.
The bad news is I've managed to crash the table that holds the 4 million+ thumbnails we've collected since 2013. There are some backups, and I'm attempting to run a repair of the table right now, but it is possible that most of them are lost. Fortunately, we still have the image hashes, and many thumbs have their full-image counterparts, so theoretically we could re-generate the thumbs from the fulls. For the time being, I'm hoping the table repair goes smoothly.
posted by terrance on 2015-01-27 6:00 am
It's that time of year again: b-stats's anniversary! And a very special one, too, because it's number two.
Stupid puns aside, it has indeed been another rocky year for the b-stats archive, although it managed to pull through. The main server suffered outages during the summer during a transition to the new cloud-hosted server, and the full-image server had to be moved around a bit (it now has twice the capacity!).
What matters, though, is that the website lives on even though I have personally become less involved with the 4chan community lately. Don't worry, I won't pull a m00t and retire. That would be silly.
Again, here's to another great year!
(as always, special thanks to moot, milky, noface, νοκοσαγ͞α, etc)
/cm/ - here to stay?
posted by terrance on 2014-10-29 2:43 am
A day or two ago, I added /cm/ to the list of boards that are archived here. This was just to test that the 'generic' archive script I was writing actually worked correctly. Well, it turns out it does! So in honor of my successes, the /cm/ board is here to stay (and be publicly accessible) for a while! It's a relatively low-volume board, down there with /hm/, so I don't foresee a significant performance impact.
As for images, I will only be archiving thumbnails for /cm/ until I decide otherwise. The 1TB (917GiB) full-image-drive is on course to reach its capacity in a few weeks, and I'd like to focus on archiving the full images from boards that I actually visit.
Enjoy the new board.
I Fixed it, Mostly
posted by terrance on 2014-07-24 6:54 am
It's been a tough week-and-a-half, but you need not worry anymore. The site is back up to its normal status, except it's better than before.
The most significant change is that the main server is now hosted on a Linode VPS (nicknamed 'lemmiwinks') so that I am less likely to completely screw things up in the future. Also, uptime should no longer be an issue. However, because images are taking up close over 750GiB, they're staying on the original server (which, if you didn't know, is nicknamed 'blackbox'). New images and thumbnails missed in the past 9 days will be grabbed from other archives if possible.
Another change is that you can now see the last time the boards were updated. This is a nice feature and I hope you all enjoy it thoroughly.
Thanks for not being too mad and as a token of my gratitude I will make /lgbt/ a publicly visible board now, in addition to /f/.
posted by terrance on 2014-07-14 3:39 pm
Well, I've sure done it now.
I've managed to accidentally bring the main b-stats server's network interface down, so I'm afraid that will be out of commission for a week or two. Until then this temporary server is configured to continue the archiving duties so that we don't lose too many threads. There may be a few hours of missing threads on each board, but that was unfortunately unavoidable.
The main databases are all fine, nothing to worry about there. The server has just gone offline temporarily because I fucked up.
Just hold tight and we'll be back in full force by 24 July at the latest. In the meantime, I'll be updating a lot of the website's code, so that should be exciting...
posted by terrance on 2014-03-29 4:42 am
I've recently been on /lgbt/ and was surprised to see that nobody was archiving it yet, so I took it upon myself to start an /lgbt/ archive. This also means I updated my archive script (again) to hopefully lessen its impact on my and moot's bandwidths.
It's unfortunate that it's starting nearly a year after /lgbt/ first opened, but I'm sure that it will record a significant portion of /lgbt/'s history.
On a related note, fgts.eu seems to have begun archiving /hm/ and 4plebs is now archiving /f/. So you can check those out if you want.
posted by terrance on 2014-01-27 5:00 am
Yep, I have officially wasted one entire year of my life with this. I can't believe it either.
A year ago I thought I was hot shit for making my own archive. Not much has changed, but I have definitely learned a good deal from the experience, namely about how to run a website and optimize code.
The site has moved around a few times, going from terrance-is.tk to b-stats.org to archive.b-stats.org, and from shared hosting to a dedicated server, but through it all, it survived. It now has a year's worth of talk, porn, and drama. That's a lot of data. Just the plain text amounts to 1.7 GiB; thumbnails to over 8.7 GiB, and full images (since July 16) take up a whopping 300 GiB. Flashes use an additional 37 GiB of storage. But, as they say, storage is cheap, and there's still room for more, so we'll see where the future takes us.
In all, it was a good year. Here's to another.
(speshul thanks to moot, milky, noface, νοκοσαγ͞α, kittens, etc)
page took 0.0017 seconds to execute