back to regularly scheduled programming

I think everything's restored now. If you notice anything missing, let us know by emailing root@antiflux.org. Future posts will be to log.antiflux.org/system except in the event of an outage.

[construction sign]

I'm working on some network settings right now. There will be another reboot in the next hour.

Update:after a couple of tries, we have a new kernel with proper DMA, SCSI, and IP filtering support.

reboots

Yesterday afternoon Apache freaked out and overloaded okcomputer. It didn't go down, but it was inaccessible for about half an hour.

We rebooted okcomputer a few times last night as Tim and I tried to sort out some issues with the new hard drives.

I will probably reboot it once more this evening (probably around 1900 MST) to install a new kernel that will speed up the disk access.

After that, okcomputer needs a newer, bigger, backup drive. It also needs to be moved within Tera-Byte's building at some point. We can probably combine those two into one reboot.

I'll post on this blog about future reboots so you guys will know what's going on.

Update: in the future, Apache will automatically be restarted if the load gets too high.

IMAP's working now so you should be able to download your mail. If your mail program is configured to check the mail/ directory, remove that. Just check the default directory now.

Webmail is partly working - inbox only for now.

web server now sort of working

I just got the web server working again. There's bound to be broken CGIs and things, but it's getting closer.

A few more bits of information:
- Our backup hard drive seems to be ok, so I don't think we lost any files.
- While we were offline, incoming mail should have been queued at the sender's ISP's mail server. Now that we're back online, it should get delivered so that nothing should wind up bouncing.
- Mailing lists are still offline, so they'll be bouncing for a while. I felt that normal email was important enough that it should come online before mailing lists work.
- Spam scanning isn't set up beyond some simple blacklisting, so you may notice more spam for a while.
- IMAP and webmail are still offline.
- I accidentally installed some incorrect packages and had to go back and downgrade/fix/remove them and install a different kernel, so we'll have to reboot later today to make sure everything still works.
- MySQL is working, but PostgreSQL is not.
- LDAP isn't set up, so services that rely on it (like the Antiflux wiki) won't authenticate
- I really want to thank Tera-Byte, who stepped up to the task and got us a new server, installed our backup drive, and got us back on our feet.

Feel free to leave comments here if you have any questions or notice anything else broken.

mail comes in and goes out

Mail should be working now. IMAP and webmail aren't set up yet, so you won't be able to retrieve your mail unless you log in with ssh and use pine/mutt/etc.

If you want to forward your mail somewhere else for now, put the forwarding address in a file called .forward in your home directory.

some hope

We have a new server and it looks like the files on the backup hard drive are ok. We're in the process of installing and configuring all the services to get things back to normal. I hope to have email working tonight.

okcomputer is hooped

That thing I always worried about in the back of my mind? It happened last night. Okcomputer went down and won't come back up. There's a LILO error when it tries to start and it kernel panicks when booting from a boot CD. It immediately became clear that we need a place outside antiflux.org to post messages like this during extended outages.

The good people at Tera-Byte are setting up a new server for us and putting in our backup drive so we can start restoring services. Our highest priority will be to get email working again, but it will be quite a while before everything's back to normal.