The Firebot Blog

This is a bit of short notice, but I’m planning a maintenance window for firebot tonight, January 12, 2012. Firebot will be moving from his current server to a more powerful machine.

As with any migration like this, the chances for regressions and unexpected problems is pretty high. I’ll post a follow-up when the migration is complete, after that, please contact me if there’s any abnormalities with firebot.

Note: As part of this same migration, the mail server which powers firebot’s bugmail will also be moving, so the bug reporting may be up or down separate from this whole-bot outage.

Thanks for your patience

– Wolf (firebot admin)

During routine maintenance firebot’s server has failed and will not come back up due to a very likely critical hardware failure

Firebot will be offline until further notice.

I’ll post more once I have more information.

Update (4:08pm): The critical services have been moved to a secondary machine – I’m going to fully examine the hardware tomorrow and see what’s gone wrong. In any case there’s a chance I’ll be able to move firebot to the same machine (load permitting) and bring it back up by tomorrow evening (may be late-evening).

Update (12/24 3:06am): The server has been restored for now. Firebot is up and running normally.

Firebot is down and will be down for several hours this morning while the server he’s located on undergoes maintenance to copy all the data from a failing hard disk drive to a new disk.

Firebot should be back up around 12pm EDT (4pm GMT)

Firebot may have appeared to be a bit flakey over the last hour or so, due to a problematic network connection between the switch his server is on and its router. Things seem to have stabilized now and the server is now back on the network reliably. The exact cause is not known at this time so a recurrence is possible.

Also, just a heads up, on Thursday, May 12th, there will be a short downtime while the router for firebot’s network is replaced.

Update: Looks like I was premature to consider this problem resolved earlier, it did happen again. Turns out there’s a faulty network cable between the switch and router – which has been replaced with a temporary drop. At some point in the near future there’ll be an outage so the cable and a discovered faulty junction box can be fully replaced.

Its definitely been noticed that in the last couple of months since firebot’s server upgrade, firebot’s performance has been less than stellar.He frequently seems to disconnect and crash.

The cause is not yet determined, the machine itself was largely unchanged except for upgrading to Windows Server 2008 R2 from 2003, which seems to factor in. Rolling back is not an option since firebot is just a passenger on the server. So going forward, it looks like its time to investigate firebot’s code, along with issues that may be present in Cygwin Perl to determine the cause.

As for the memory loss issues with regard to some channels, mozbot has a slow start-up process, which seems to be compounded by hundreds of disk-writes of its config file during startup. (once per-module per-channel, at the least…) If the bot fails to complete, it results in losing portions of the config file, as the channels get reset every time. Since firebot is having trouble starting up successfully and joining many channels without timing out, this is also factoring into his stability issues.

I’ll post more as I learn more.

– Wolf

So, after giving it some thought, I’ve decided that the status posts for firebot deserve their own space and not to be jumbled up with my own blog. This is the result. I’ll be fixing it up to replace the old pages that were lurking here, as well as getting it added to planet mozilla, but in the meantime. Welcome aboard.