Amazon S3 - Outage

Many sites are currently feeling the pain of an Amazon S3 outage. According to http://status.aws.amazon.com/, S3 has been down since about 9AM PDT today.

Unfortunately, my company's site is one that 's impacted.

The AWS Service Health Dashboard is helpful but I'd prefer something that provides a more proactive notification when there's a service interruption. Something along the lines of an SMS once a service disruption is announced.

Perhaps there's something out there already that I haven't found?

UPDATE: As of 5PM PDT, Amazon is reporting that the system is back to normal.

UPDATE: Here's the latest summary of the chain of events from Amazon. Nice of them to not make every customer email for SLA credit.

reset_column_information in cruisecontrol.rb

As I described in a previous post, cruisecontrol.rb was failing with every other build because of the way our migrations were written (more on why they were written that way in a later post).

Today I was able to resolve the issue by adding a new task to our list of tasks that get run when cruise completes a build.

desc 'resets the column info for all models'
namespace :columns do
task :reset => :environment do
 ActiveRecord::Base.reset_column_information
 ActiveRecord::Base.send(:subclasses).each{|klass| klass.reset_column_information rescue nil}
end
end

Now, our cruise task looks like this:

task :cruise do
['test_env_init', 'db:rollback', 'columns:reset', 'db:migrate','test', 'spec:cruise:rcov'].each do |task|
 Rake::Task[task].invoke
end
...

And cruisecontrol.rb is much happier...as is our development team!

cruisecontrol.rb fails every other build

Why does cruisecontrol.rb report that every other build fails?

Looking at the log, you see things such as:

NoMethodError: undefined method `referring_friend' for #<user:0x2aaab3936628>

So...where does referring_friend get introduced?

251_add_promo_to_user.rb

class AddPromoToUser < ActiveRecord::Migration
 def self.up
   if User.table_exists? && !User.column_names.include?('referring_friend')
     add_column :users, :referring_friend, :string
   end
 end

 def self.down
   if User.table_exists? && User.column_names.include?('referring_friend')
     remove_column :users, :referring_friend
   end

 end
end

OK, clearly this works and we have some additional checks in place. Specifically, don't add the column if it already exists and don't drop it if it's not there. Doing either of these could cause some database errors.

Nice, but we still don't know why it's failing. Let's have a look at what cruise does

custom_cc.rake

...
task :cruise do
['test_env_init', 'db:rollback', 'db:migrate','test', 'spec:cruise:rcov'].each do |task|
...

See the problem yet?

Let's see what's happening in that cruise task.

• test_env_init - pretty simple, just sets the RAILS_ENV to test
• db:rollback - runs the down for the latest migration. So in this case we

    remove_column :users, :referring_friend

• db:migrate - runs the latest migrations. So in this case we

    add_column :users, :referring_friend, :string

However...we only drop the column if it exists and we only add it if it's not already there

OK, so let's try this in our good friend, script/console:

$ script/console
Loading test environment (Rails 2.0.2)
>> puts User.column_names.include?('referring_friend')
true
=> nil
>> exit
$ rake db:rollback
(in /usr/local/cruise/cruisecontrolrb-1.2.1/projects/release_1_7/work)
== 251 AddPromoToUser: reverting ==============================================
-- remove_column(:users, :referring_friend)
 -> 0.1270s
== 251 AddPromoToUser: reverted (0.3239s) =====================================

$ script/console
Loading test environment (Rails 2.0.2)
>> puts User.column_names.include?('referring_friend')
false
=> nil
>> exit
$ rake db:migrate
(in /usr/local/cruise/cruisecontrolrb-1.2.1/projects/release_1_7/work)
== 251 AddPromoToUser: migrating ==============================================
-- add_column(:users, :referring_friend, :string)
 -> 0.0739s
== 251 AddPromoToUser: migrated (0.2668s) =====================================

$ script/console
Loading test environment (Rails 2.0.2)
>> puts User.column_names.include?('referring_friend')
true
=> nil
>>

Hmm...that all looks correct. But that wasn't actually a good test of what's happening because we're "Loading test environment" every time we run script/console.

Let's try that again using 2 different sell sessions.

Session 1

$ script/console
>> puts User.column_names.include?('referring_friend')
true

Session 2

$ rake db:rollback
(in /usr/local/cruise/cruisecontrolrb-1.2.1/projects/release_1_7/work)
== 251 AddPromoToUser: reverting ==============================================
-- remove_column(:users, :referring_friend)
 -> 0.1799s
== 251 AddPromoToUser: reverted (0.3748s) =====================================

Session 1

>>  puts User.column_names.include?('referring_friend')
true

Session 2

$ psql -U cruise_test -d cruise_test_1_7
$ \d users;

*As we expected, there's no referring_friend column on the Uses table

Session 1

>> User.reset_column_information
=> nil
>>  puts User.column_names.include?('referring_friend')
false

Ah ha!!

So, if we update our User model after each change, the build should Colt 45 (it works every time)

updated 251_add_promo_to_user.rb

class AddPromoToUser < ActiveRecord::Migration
 def self.up
   if User.table_exists? && !User.column_names.include?('referring_friend')
     add_column :users, :referring_friend, :string
     User.reset_column_information
   end
 end

 def self.down
   if User.table_exists? && User.column_names.include?('referring_friend')
     remove_column :users, :referring_friend
     User.reset_column_information
   end

 end
end