Our Blog

On Sunday, May 30th Saturday, June 12th we will be taking AmbitiousLemon down for six to eight hours to reinstall the operating system. The server will go down at about 11:00 AM Pacific time and will stay down until we have full functionality restored. We have full backups of data stored on the server, so while data loss is a possibility it is very unlikely.

We have been working on resolving our intermittent downtime issues and have decided that our attempts to band-aid the issue have not been successful. For this reason, we are going to install a fresh build of Linux. In addition, we have seen some attempts to break into AmbitiousLemon. Rebuilding the server from the ground up allows us to add additional layers of security to ensure that future attempts have less chance of success.

Update:
One of our admins is has had a family emergency and will not be available on Sunday, May 30th. Accordingly, we are rescheduling our downtime to ensure the reinstall goes as smoothly as possible.

Update #2:
The downtime is being delayed again. We are delaying to ensure that we are in the best position possible to reinstall with minimal downtime and to ensure that the experience is as seamless as possible for our users.

We were down for about eight hours today. The downtime occurred while we were upgrading system software trying to address the downtime and related issues from the past week. This was caused when a system component got upgraded before a planned kernel upgrade was completed. Unfortunately, this left the system in a state where we needed to be on-site to work on it.

After getting to the unit, we were able to diagnose and fix the issue, and while doing so completed some additional upgrades to the machine.

We are continuing to investigate the cause of our downtime; the server has not been performing as it should and we are investigating how to best and most effectively fix it.

We experienced approximately 3 hours downtime this morning. We are still investigating the cause.

On Saturday, March 27th, I will be taking AmbiousLemon down for a hardware upgrade. I will make every effort to minimize downtime, but the upgrades will require the server to be offline for a minimum of one hour over the course of the day.

Update:
Hardware upgrades are complete.

Last night we updated Passenger to version 2.2 from 2.1 and discovered some time afterwards that this update broke the implementation of rails hosting used by most of our rails users. This is apparently a deliberate change made by the Passenger developers to better accommodate Capistrano-style deployment setups. While we evaluate how to best handle this change we are attempting to roll back the update to again server rails apps using Passenger 2.1. This is not a long term solution, so we may need to find a way to allow rails users to easily and safely set their document root without using symlinks. This likely will mean that when we do update to Passenger 2.2 again, our rails users will need to take some action in order to transition their sites to the new system (this is vague because we don’t yet know how we will handle this transition). So we ask our Rails users to be extra watchful of announcements from us in the future.

We experienced a couple 2-3 minute periods of downtime this afternoon around 4PM EST.

It was caused by a script that runs after creating accounts for new members. We’ve corrected the problem, so it shouldn’t cause any issues in the future.

Last night’s downtime didn’t go entirely as expected. As expected things were up and down a bit from 2am to 2:22am. The unexpected bit came at 4:30am when we went down and didn’t come back up. Around 11am things came back online. This downtime was due to some changes at the carrier hotel rather than anything we were doing ourselves. No word from the carrier hotel what was going on. My best guess is that some of the new hardware that was being swapped in had some unexpected problem.

On Mar/21/2009-Mar/22/2009 from 10pm – 2am Pacific time the server will be unavailable for approximately 10-30 minutes.

We apologize for the late notice.