The Firebot Blog

tl;dr; Firebot is moving servers, there may be downtime and you may see a second bot named “koalabot”, this is expected.

Two weeks ago, the server firebot was on underwent routine maintenance as its role had changed since firebot was migrated to it – this resulted in a lot of software being reinstalled, including firebot’s cygwin/perl environment . Since then, firebot’s performance has been less than stellar, frequently resulting in hung processes which cause him to experience ping timeouts if the process isn’t killed fast enough. These leftover processes have occurred since firebot was moved from Windows Server 2003 to 2008 R2 awhile ago, but never before did they actually affect his performance.

Since I originally started running firebot in 2004, technology has progressed that makes it much easier to move firebot back to a machine running Linux, which mozbot Is intended. (Thanks to the magic of VMs.) Originally, firebot ran on Windows because, for reasons outside of the scope of this post, the mail server his bugmail is processed on, is on Windows, and the bot needed to be able to get the mail piped to him. This is no longer a limitation and I’m now able to move the bot to be separate from the mail processor (in fact, firebot and the mail server are not running on the same machine and haven’t for some time now).

So, long story short, I’ve been testing running firebot on koala, a new Linux VM configured for him, and results have been good. Therefore, I plan on migrating firebot to be in production on this new VM and shutting down the old bot. There may be sporadic outages while I copy files around and so forth. Also, for a time, you might also see his twin running on the new VM “koalabot” if I need to run both instances at the same time.

Thanks for your patience – if you see any issues, please let me know.

Update: Firebot’s migration is now completed. If you see any problems, file a bug in bugzilla (webtools/mozbot) or contact me (Wolf) on IRC or by email.

This is a bit of short notice, but I’m planning a maintenance window for firebot tonight, January 12, 2012. Firebot will be moving from his current server to a more powerful machine.

As with any migration like this, the chances for regressions and unexpected problems is pretty high. I’ll post a follow-up when the migration is complete, after that, please contact me if there’s any abnormalities with firebot.

Note: As part of this same migration, the mail server which powers firebot’s bugmail will also be moving, so the bug reporting may be up or down separate from this whole-bot outage.

Thanks for your patience

– Wolf (firebot admin)

During routine maintenance firebot’s server has failed and will not come back up due to a very likely critical hardware failure

Firebot will be offline until further notice.

I’ll post more once I have more information.

Update (4:08pm): The critical services have been moved to a secondary machine – I’m going to fully examine the hardware tomorrow and see what’s gone wrong. In any case there’s a chance I’ll be able to move firebot to the same machine (load permitting) and bring it back up by tomorrow evening (may be late-evening).

Update (12/24 3:06am): The server has been restored for now. Firebot is up and running normally.

Firebot is down and will be down for several hours this morning while the server he’s located on undergoes maintenance to copy all the data from a failing hard disk drive to a new disk.

Firebot should be back up around 12pm EDT (4pm GMT)

Firebot may have appeared to be a bit flakey over the last hour or so, due to a problematic network connection between the switch his server is on and its router. Things seem to have stabilized now and the server is now back on the network reliably. The exact cause is not known at this time so a recurrence is possible.

Also, just a heads up, on Thursday, May 12th, there will be a short downtime while the router for firebot’s network is replaced.

Update: Looks like I was premature to consider this problem resolved earlier, it did happen again. Turns out there’s a faulty network cable between the switch and router – which has been replaced with a temporary drop. At some point in the near future there’ll be an outage so the cable and a discovered faulty junction box can be fully replaced.

Its definitely been noticed that in the last couple of months since firebot’s server upgrade, firebot’s performance has been less than stellar.He frequently seems to disconnect and crash.

The cause is not yet determined, the machine itself was largely unchanged except for upgrading to Windows Server 2008 R2 from 2003, which seems to factor in. Rolling back is not an option since firebot is just a passenger on the server. So going forward, it looks like its time to investigate firebot’s code, along with issues that may be present in Cygwin Perl to determine the cause.

As for the memory loss issues with regard to some channels, mozbot has a slow start-up process, which seems to be compounded by hundreds of disk-writes of its config file during startup. (once per-module per-channel, at the least…) If the bot fails to complete, it results in losing portions of the config file, as the channels get reset every time. Since firebot is having trouble starting up successfully and joining many channels without timing out, this is also factoring into his stability issues.

I’ll post more as I learn more.

– Wolf

So, after giving it some thought, I’ve decided that the status posts for firebot deserve their own space and not to be jumbled up with my own blog. This is the result. I’ll be fixing it up to replace the old pages that were lurking here, as well as getting it added to planet mozilla, but in the meantime. Welcome aboard.