Web offline

 Following our move to the new Tera-Byte data centre, one drive didn’t come back up. Email is back online, but web hosting is still offline.

drive replacement

We're temporarily offline for a drive replacement.

Update: We're back.

drive replacement

We're currently offline while replacing another failed drive.

Update: Back online. Any further updates will be posted to http://log.antiflux.org/system/

drive replacement

We're currently offline while replacing a drive that was reporting errors.

Update: We're back up with the new drive. Mail is still offline while everything is migrating. Further updates will be posted at log.antiflux.org/system .

drive installation report

Our new drive is installed, but the server's running into an error while booting. Hopefully our friends at Tera-Byte will be able to work it out soon.

Update: Problem identified. We're now waiting for the always-slow disk checks to finish and we should be back in business within the next hour.

Update 2: We're back online. If anything's not working, please let us (root@) know. Sorry for the extended downtime, but it was good to shake out some bugs before the long-anticipated OS update.

system upgrade

I'll be working on a long overdue system software upgrade later today and possibly tomorrow.

There shouldn't be any significant downtime. Mail may be slightly delayed. The web server may be down briefly.

Update 20:26 EST: real life got in the way and I have not managed to upgrade anything yet. Stay tuned.

new drive

okcomputer is down right now while we install a new 1 TB drive.

back to regularly scheduled programming

I think everything's restored now. If you notice anything missing, let us know by emailing root@antiflux.org. Future posts will be to log.antiflux.org/system except in the event of an outage.

[construction sign]

I'm working on some network settings right now. There will be another reboot in the next hour.

Update:after a couple of tries, we have a new kernel with proper DMA, SCSI, and IP filtering support.

reboots

Yesterday afternoon Apache freaked out and overloaded okcomputer. It didn't go down, but it was inaccessible for about half an hour.

We rebooted okcomputer a few times last night as Tim and I tried to sort out some issues with the new hard drives.

I will probably reboot it once more this evening (probably around 1900 MST) to install a new kernel that will speed up the disk access.

After that, okcomputer needs a newer, bigger, backup drive. It also needs to be moved within Tera-Byte's building at some point. We can probably combine those two into one reboot.

I'll post on this blog about future reboots so you guys will know what's going on.

Update: in the future, Apache will automatically be restarted if the load gets too high.

IMAP's working now so you should be able to download your mail. If your mail program is configured to check the mail/ directory, remove that. Just check the default directory now.

Webmail is partly working - inbox only for now.

web server now sort of working

I just got the web server working again. There's bound to be broken CGIs and things, but it's getting closer.

A few more bits of information:
- Our backup hard drive seems to be ok, so I don't think we lost any files.
- While we were offline, incoming mail should have been queued at the sender's ISP's mail server. Now that we're back online, it should get delivered so that nothing should wind up bouncing.
- Mailing lists are still offline, so they'll be bouncing for a while. I felt that normal email was important enough that it should come online before mailing lists work.
- Spam scanning isn't set up beyond some simple blacklisting, so you may notice more spam for a while.
- IMAP and webmail are still offline.
- I accidentally installed some incorrect packages and had to go back and downgrade/fix/remove them and install a different kernel, so we'll have to reboot later today to make sure everything still works.
- MySQL is working, but PostgreSQL is not.
- LDAP isn't set up, so services that rely on it (like the Antiflux wiki) won't authenticate
- I really want to thank Tera-Byte, who stepped up to the task and got us a new server, installed our backup drive, and got us back on our feet.

Feel free to leave comments here if you have any questions or notice anything else broken.

mail comes in and goes out

Mail should be working now. IMAP and webmail aren't set up yet, so you won't be able to retrieve your mail unless you log in with ssh and use pine/mutt/etc.

If you want to forward your mail somewhere else for now, put the forwarding address in a file called .forward in your home directory.

some hope

We have a new server and it looks like the files on the backup hard drive are ok. We're in the process of installing and configuring all the services to get things back to normal. I hope to have email working tonight.

okcomputer is hooped

That thing I always worried about in the back of my mind? It happened last night. Okcomputer went down and won't come back up. There's a LILO error when it tries to start and it kernel panicks when booting from a boot CD. It immediately became clear that we need a place outside antiflux.org to post messages like this during extended outages.

The good people at Tera-Byte are setting up a new server for us and putting in our backup drive so we can start restoring services. Our highest priority will be to get email working again, but it will be quite a while before everything's back to normal.