The Firebot Blog

Firebot Maintenance–1/12/2012

wolf — Thu, 12 Jan 2012 23:58:21 +0000

This is a bit of short notice, but I’m planning a maintenance window for firebot tonight, January 12, 2012. Firebot will be moving from his current server to a more powerful machine.

As with any migration like this, the chances for regressions and unexpected problems is pretty high. I’ll post a follow-up when the migration is complete, after that, please contact me if there’s any abnormalities with firebot.

Note: As part of this same migration, the mail server which powers firebot’s bugmail will also be moving, so the bug reporting may be up or down separate from this whole-bot outage.

Thanks for your patience

– Wolf (firebot admin)

Firebot Outage (Unplanned) – December 22, 2011

wolf — Thu, 22 Dec 2011 20:22:54 +0000

During routine maintenance firebot’s server has failed and will not come back up due to a very likely critical hardware failure

Firebot will be offline until further notice.

I’ll post more once I have more information.

Update (4:08pm): The critical services have been moved to a secondary machine – I’m going to fully examine the hardware tomorrow and see what’s gone wrong. In any case there’s a chance I’ll be able to move firebot to the same machine (load permitting) and bring it back up by tomorrow evening (may be late-evening).

Update (12/24 3:06am): The server has been restored for now. Firebot is up and running normally.

Firebot Outage–5/27/2011

wolf — Fri, 27 May 2011 05:40:52 +0000

Firebot is down and will be down for several hours this morning while the server he’s located on undergoes maintenance to copy all the data from a failing hard disk drive to a new disk.

Firebot should be back up around 12pm EDT (4pm GMT)

Firebot Outage Report (5/9) & Scheduled Outage (5/12)

wolf — Mon, 09 May 2011 21:44:36 +0000

Firebot may have appeared to be a bit flakey over the last hour or so, due to a problematic network connection between the switch his server is on and its router. Things seem to have stabilized now and the server is now back on the network reliably. The exact cause is not known at this time so a recurrence is possible.

Also, just a heads up, on Thursday, May 12th, there will be a short downtime while the router for firebot’s network is replaced.

Update: Looks like I was premature to consider this problem resolved earlier, it did happen again. Turns out there’s a faulty network cable between the switch and router – which has been replaced with a temporary drop. At some point in the near future there’ll be an outage so the cable and a discovered faulty junction box can be fully replaced.

About Firebot’s Recent Stability Problems

wolf — Tue, 08 Mar 2011 00:51:52 +0000

Its definitely been noticed that in the last couple of months since firebot’s server upgrade, firebot’s performance has been less than stellar.He frequently seems to disconnect and crash.

The cause is not yet determined, the machine itself was largely unchanged except for upgrading to Windows Server 2008 R2 from 2003, which seems to factor in. Rolling back is not an option since firebot is just a passenger on the server. So going forward, it looks like its time to investigate firebot’s code, along with issues that may be present in Cygwin Perl to determine the cause.

As for the memory loss issues with regard to some channels, mozbot has a slow start-up process, which seems to be compounded by hundreds of disk-writes of its config file during startup. (once per-module per-channel, at the least…) If the bot fails to complete, it results in losing portions of the config file, as the channels get reset every time. Since firebot is having trouble starting up successfully and joining many channels without timing out, this is also factoring into his stability issues.

I’ll post more as I learn more.

– Wolf

Welcome to The Firebot Blog

wolf — Tue, 08 Mar 2011 00:45:56 +0000

So, after giving it some thought, I’ve decided that the status posts for firebot deserve their own space and not to be jumbled up with my own blog. This is the result. I’ll be fixing it up to replace the old pages that were lurking here, as well as getting it added to planet mozilla, but in the meantime. Welcome aboard.

Firebot Downtime –12/28/2010

wolf — Tue, 28 Dec 2010 16:35:05 +0000

I’m taking advantage of my winter break as well as the holiday season lull to upgrade firebot’s server today (12/28/2010). So he’ll be offline from now until that’s completed sometime tonight or tomorrow.

I’m also reconfiguring how my mail server is working so even after he comes back up, bugmail reporting will take longer to have working again.

Happy Holidays

– Wolf

How to report a problem or request for Firebot

wolf — Fri, 29 Jan 2010 16:39:15 +0000

In general, it seems as if its not obvious how to reach me regarding firebot. In fact, in seems like a lot of people don’t know who runs firebot at all. So hopefully this post will clear things up.

First off, for those who are new to IRC or just don’t have the time to dive into details.. Firebot, is the IRC robot that hangs around many of the channels on irc.mozilla.org. It’s maintained by me, Wolf, I can be found in many of the same channels. It runs the mozbot software, which is a mozilla.org project, which I also am the module owner of.

Mozilla’s Bugzilla: Feature requests and bugs for Mozbot itself belong in bugzilla.mozilla.org, in the webtools/mozbot component. This is the best place to track them. Just telling me will likely result in your request getting lost, its unfortunate, but I just don’t remember things that well.

Firebot’s Bugzilla: Configuration needs for firebot, (Examples: turning on a tree in Tinderbox, adding new Bug announcements, disabling a module in a channel, enabling a feature in a channel, making the bot join or leave a channel, etc) on the other hand, are frowned upon in that component (though I personally don’t care, the bugzilla overlords seem to.) therefore they belong in Firebot’s Bugzilla.. at https://bugzilla.pcfire.net/ ( Specifically the new bug form for firebot is here: [New Firebot Bug/Request Form] )

Via E-Mail: If you don’t want to go through the trouble to sign up for yet-another-bugzilla (ideas for better ways to deal with this are welcome), which I can understand, you can also e-mail them to me via firebot-admin@ . You can also ask firebot for it.. just “/msg firebot admin?” and I’ll file your bug and link you to the number so you can keep up.

Confused? The distinctions aren’t very clear and I’m not picky about them, so if you’re confused or not sure, just e-mail me Thanks.

Firebot Downtime – September 24th and 25th, 2009

wolf — Thu, 24 Sep 2009 22:21:40 +0000

Starting now, and continuing for at least several hours, the server that hosts firebot will be down for an upgrade.. since the other server here is being kind enough to donate its “old” CPU and motherboard…
The server will be being upgraded from an circa-2003 Athlon XP 2400+ to an Athlon 64 X2 4000+, plus moving to a SATA disk…
Firebot will be back soon.
Update: The CPU upgrade was successful, unfortunately, due to some time-consuming issues that were encountered, I’ve moved back my plans to migrate the system to the SATA disk, the disk is installed and ready, just have to plan more downtime to set up the system on it. Its tentatively planned for tomorrow, 9/25.

Firebot Outage Report: 7/14 – 7/16/2009

wolf — Thu, 16 Jul 2009 02:53:00 +0000

The last few nights, between 1am and ~~3am~~ 4am, firebot has been offline, because of an ongoing connectivity issue. At the current time, I have no eta for when the issues will be resolved. The situation is pretty straightforward though, the outages are due to construction along the highway where I live and firebot is hosted, its being expanded to multiple lanes resulting in the need for my ISP, TimeWarner cable / RoadRunner, to move their fiber optic lines to make way for the new road. Though this has been ongoing it would seem they’ve reached the point where the overnight customer-facing work is being done, as each night, like clockwork, the cable has been going out completely.

Other recent events affecting firebot’s stability and connectivity over the past few weeks which are not external that are:

Migrating to a new server. After 6 years of service, the machine he previously ran on was replaced as another more powerful machine became available. Firebot now runs on an AMD Athlon XP 2400+ with 2gb of RAM, 320gb disk running Windows Server 2003 R2 Enterprise Edition.. (Which, while definitely not current, is more than adequate.) Previously it was an Intel Celeron 1.7ghz with 1.2gb of ram, and 120gb disk, which was not performing well anymore. This is also the reason why at times there were two instances of the bot running.
New network hardware, the network has been upgraded to gigabit (and Wireless-N), requiring the replacement of both the switch his machine was on, as well as the router. This caused a nightly (twice daily actually) connection reset, due to a bug in the firmware of the router and how it handles static dhcp leases (it for some reason, issues a dhcpnack before reissuing the same address to the machine, resulting in a brief loss of connectivity.) I’ve worked around it for now. (For those curious, the router is a Netgear WNR3500.)