Saturday, October 10, 2009

Customizing an EC2 AMI

Setup the Amazon EC2 API Tools

note: this assumes you already have Java installed with the JAVA_HOME environment variable set.

Download and unzip
$ cd
$ wget http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip
$ unzip ec2-api-tools.zip
Set EC2 environment variables (assuming bash shell)

add the following to your ~/.bash_profile

export EC2_HOME=$HOME/ec2-api-tools-X.X-XXXXX
export PATH=$PATH:$EC2_HOME/bin
export EC2_PRIVATE_KEY=$HOME/.ec2/pk-XXXXXXXXXXXXXXX.pem
export EC2_CERT=$HOME/.ec2/cert-XXXXXXXXXXXXXXX.pem

the EC2_HOME variable will point to the directory created where ec2-api-tools.zip was unzipped, replace the X's with the version numbers.

the EC2_PRIVATE_KEY and EC2_CERT are the paths to your AWS X.509 certificate files, again replace the X's to match your filenames.

if you have not generated/uploaded these files in your AWS account you can do so in the Security Credentials section of your AWS account. save the private key as well as the certificate file to the ~/.ec2 directory.

Reload your bash profile
$ source ~/.bash_profile
Start the base AMI

this can be done a number of ways but using Elasticfox or the AWS Management Console i feel are the easiest.

both will have you create a key pair in order to connect to the instance, remember where you saved this, you will need it later on.

Customize the AMI

connect to the running instance and customize it how you like.

Upload your private/public key to the instance

upload your X.509 private key and certificate files to the running instance's /mnt directory.

(local machine)

$ scp -i /path/to/ec2/ssh_key $EC2_PRIVATE_KEY $EC2_CERT root@public.dns.address:/mnt

the /path/to/ec2/ssh_key is the path to the ssh key you used to connect to the running EC2 instance (not the X.509 private key).

also replace public.dns.address with the public dns address of the running EC2 instance.

Bundle the Customized AMI

after you are done customizing the instance create a bundled Amazon Machine Image (AMI)

(EC2 instance)

$ ec2-bundle-vol -d /mnt -k /mnt/pk-XXXXXXXXXXXXXXX.pem -c /mnt/cert-XXXXXXXXXXXXXXX.pem -u <aws_account_id> -s 10240 -r i386

note: if you installed your own version of ruby and this command is failing, the easiest workaround is to just change into the /usr/lib/site_ruby directory and run the command.

your AWS account ID can be found in the Security Credentials section of your AWS account.

paramdescription
-dthe directory to create the bundle in (/mnt is ignored by default when bundling an AMI)
-kyour private key (uploaded in the previous step)
-cyour certificate file (uploaded in the previous step)
-uyour AWS account ID.
-sthe size (in MB) of the image file to create. this can be left out if you are not resizing the current AMI
-rthe architecture (i386 or x86_64)
Upload AMI bundle to S3

upload the AMI bundle to your S3 account

(EC2 instance)

$ ec2-upload-bundle -b bucket_name/key -m /mnt/image.manifest.xml -a <access_key_id> -s <secret_access_key>

your Access Key ID as well as yout Secret Access Key can be found in the Security Credentials section of your AWS account.

paramdescription
-bthe bucket name to upload the bundle to. you can use slashes to upload to a subdirectory (or key as Amazon calls them)
-mthe path to the bundle manifest file, this will be /mnt/image.manifest.xml if you did not change the destination directory or prefix in the previous step
-ayour aws access key ID
-syour aws secret access key
Register the AMI

the bundled AMI must be registered with Amazon before it can be used, by default the AMI will only be accessible from your account.

(local machine)

$ ec2-register bucket_name/key/image.manifest.xml

the bucket_name/key/image.manifest.xml is the bucket path on S3 the bundle was uploaded to in the previous step

if successful the AMI ID will be output and you will find it listed as an available AMI.

Saturday, February 14, 2009

Testing Rake Tasks with RSpec

Not sure if this is the correct way to go about testing rake tasks but it works pretty well. Hit up the comments if there is a better way.

So here is an example rake task I want to test:

[lib/tasks/app.rake]

namespace :app do
  namespace :options do
    desc "refreshes option values"
    task :refresh => :environment do
      options = YAML.load_file("config/options.yml")
      options.each do |attrs|
        option = Option.find_or_initailize_by_name(attrs["name"])
        option.update_attribute("value", attrs["value"])
      end
    end
  end
end

This is just loading in a yml file and refreshing some default values. Simple enough but I want it tested.

Here is what I came up with:

[spec/lib/tasks/app.rake_spec.rb]

require "/path/to/spec_helper"
require "rake"

describe "app rake tasks" do
  before do
    @rake = Rake::Application.new
    Rake.application = @rake
    Rake.application.rake_require "lib/tasks/app"
    Rake::Task.define_task(:environment)
  end

  describe "rake app:options:refresh" do
    before do
      @task_name = "app:options:refresh"
      YAML.stub!(:load_file).and_return([])
    end
    it "should have 'environment' as a prereq" do
      @rake[@task_name].prerequisites.should include("environment")
    end
    it "should load 'config/options.yml'" do
      YAML.should_receive(:load_file).with("config/options.yml").and_return([])
      @rake[@task_name].invoke
    end
    it "should create or update all records in the config file" do
      YAML.should_receive(:load_file).with("config/options.yml").and_return([
        { "name" => "option one", "value" => 10 },
        { "name" => "option two", "value" => 20 }
      ])
      option_one = mock(Option, :null_object => true)
      option_two = mock(Option, :null_object => true)
      
      Option.should_receive(:find_or_initialize_by_name).with("option one").and_return(option_one)
      Option.should_receive(:find_or_initialize_by_name).with("option two").and_return(option_two)

      option_one.should_receive(:update_attribute).with("value", 10)
      option_two.should_receive(:update_attribute).with("value", 20)
      @rake[@task_name].invoke
    end
  end
end

Let's walk through whats going on here...

@rake = Rake::Application.new
Rake.application = @rake

This is initializing a new rake application object and setting the current Rake.application to it. This is done so that each test is run with a fresh application object.

Rake.application.rake_require "lib/tasks/app"

This is requiring the rake file with the task I will be testing.

Rake::Task.define_task(:environment)

This is essentially stubbing the environment task which loads the rails environment. The rails env is already loaded when you are running your specs so this is unnecesary.

@rake[@task_name].invoke

This is what actually runs the rake task.

The rest of the 'it' blocks are pretty self explanatory, testing that the task has the correct prereqs and then just spec'ing the functionality.

Saturday, October 4, 2008

Regex Changes in RSpec 1.1.5 Stories

RSpec 1.1.5 changed how the Given, When, and Then match statements are parsed and can break some of your stories.

One of my Then statements was as follows:

Then("the event should have $count artists?") do |count|
   #...
end

The match statements are converted to regular expressions so the question mark would just make the 's' on artists optional so this would match both:

Then the event should have 4 artists

and...

Then the event should have 1 artist

After updating RSpec to 1.1.8 from 1.1.4 (the versions jumped so fast to align with Rspec-rails but 1.1.5 has most of the changes) I was seeing a bunch of my steps were now pending.

I ran a diff on the lib/spec/story/step.rb file between tags 1.1.4 and 1.1.5 in rspec and noticed the method building the matcher was updated.

-    def assign_expression(string_or_regexp)
-      if String === string_or_regexp
-        expression = string_or_regexp.dup
-        expression.gsub! '(', '\('
-        expression.gsub! ')', '\)'
-      elsif Regexp === string_or_regexp
-        expression = string_or_regexp.source
-      end
-      while expression =~ PARAM_PATTERN
-        expression.gsub!($2, "(.*?)")
-      end
-      @expression = Regexp.new("^#{expression}$")
+  def init_name(name)
+    @name = name
+  end
+
+  def init_expression(string_or_regexp)
+    if String === string_or_regexp
+      expression = string_or_regexp.dup
+      %w<? ( ) [ ] { } ^ !>.each {|c| expression.gsub! c, "\\#{c}"}
+    elsif Regexp === string_or_regexp
+      expression = string_or_regexp.source
+    end
+    while expression =~ PARAM_PATTERN
+      expression.sub!($2, "(.*?)")
     end
+    @expression = Regexp.new("\\A#{expression}\\Z", Regexp::MULTILINE)
+  end

I didnt realize the matcher was parsed differently based on if it was passed in as a String or a Regexp. Now if you give a string to match with any of the %w<? ( ) [ ] { } ^ !> characters they will be escaped to match those actual characters. Previously it was just parenthesis.

Anyway to fix your current matchers just send them in as regexs instead of strings.

Then(/the event should have $count artists?/) do |count|
   #...
end

Don't worry about escaping the dollar sign, the PARAM_PATTERN strips that out before it gets converted back into a regex.

Tuesday, April 29, 2008

gitweb + apache + gentoo

This is meant as a quick tutorial to get gitweb up and running on gentoo with apache as the web server.

USE FLAGS

since git version 1.4, gitweb is bundled with git and can be installed by building with the cgi and perl use flags.

add the following to your package.use file:

[/etc/portage/package.use]

dev-util/git perl cgi
EMERGE...

emerge all the required packages...

> sudo emerge dev-util/git
> sudo emerge apache
INSTALL

install a copy of gitweb to apache's cgi-bin directory

> sudo cp -r /usr/share/git/gitweb /var/www/localhost/cgi-bin
> sudo chown -R apache:apache /var/www/localhost/cgi-bin/gitweb

note: change the 'apache' user and group to whatever you have apache configured to run as.

APACHE CONFIG

create a new virtual host file in your apache vhosts.d directory to be used for accessing gitweb.

[/etc/apache2/vhosts.d/01_gitweb_vhost.conf]

<VirtualHost *:80>
     ServerName git.mydomain.com
     DocumentRoot /var/www/localhost/cgi-bin/gitweb 
     <Directory /var/www/localhost/cgi-bin/gitweb>
          Allow from all
          AllowOverride all
          Order allow,deny
          Options ExecCGI
          <Files gitweb.cgi>
               SetHandler cgi-script
          </Files>
     </Directory>
     DirectoryIndex gitweb.cgi
     SetEnv  GITWEB_CONFIG  /etc/gitweb.conf
</VirtualHost>

note: if you are just using this for localhost and/or you are not setting this up as a subdomain you can just overwrite the default VirtualHost defined in 00_default_vhost.conf with the above and change 'ServerName' to localhost

GITWEB CONFIG

create and edit your gitweb configuration file to point to the root of your projects directory. (/var/git in my case)

[/etc/gitweb.conf]

$projectroot = '/var/git';

note: additional configuration options can be found here: http://repo.or.cz

CREATE A TEST PROJECT
> cd /var/git
> mkdir test.git
> cd test.git
> git --bare init
> echo "this is a test" > description
GIT PUSH

push a project to the gitweb repo

> git push phil@mydomain.com:/var/git/test.git

note: if there is a permissions issue check the test.git repo permissions and ensure you have write access.

if you need more security or are looking to manage access and permissions without using system accounts you might want to check out gitosis for the shared repo.

FIRE IT UP
> sudo /etc/init.d/apache2 start

point your browser to the address you put in for the vhost config and you should be in business.

<update>
if your projects are not being listed in gitweb, try adding your apache user to the git group or whatever group the project dir belongs to. also try this if you updated git and your projects are now showing "no commits". i believe the default group permissions for a shared repo were changed/fixed. i noticed this at v.1.5.6.1 but didn't track down the actual commit.
</update>

if the owner names for the projects are not showing up you can add a name through the usermod command.

> sudo usermod -c "phil sergi" phil

Thursday, October 4, 2007

acts_as_messageable plugin released!

Problem:

Private messaging in Rails.

Solution:

acts_as_messageable

What it do?

acts_as_messageable is a plugin for enabling private messaging between users. Conversations, multiple recipients, trash, sentbox... its all in there.

to install:
> script/plugin install http://actsasmessageable.googlecode.com/svn/tags/acts_as_messageable-1.0.1 
> script/generate messageable 
> rake db:migrate
to use:
class User < ActiveRecord::Base 
  acts_as_messageable 
end
sending:
phil = User.find(3123) 
todd = User.find(4141) 
phil.send_message(todd, "whats up for tonight?", "hey guy")

sends a mail to todd's inbox and puts the sent message in phil's sentbox.

retrieving:
todd.mailbox[:inbox].unread_mail

returns an array of all unread mail messages in todd's inbox.

replying:
mail = todd.mailbox[:inbox].unread_mail[0] 
todd.reply_to_sender(mail, "not sure, probably having a few dozen cocktails.")

replys to the sender of the mail message. (Messages can be sent to multiple users, and are conversation based. See the RDoc.)

moving / deleting:
convo = mail.conversation
todd.mailbox.move_to(:trash, :conversation => convo)

moves all mail messages to the 'trash' that are part of the given conversation. There are 3 default mailboxes for each user (:inbox, :sentbox, :trash), although any name can be passed to this method if you wanted to implement folders.

bonus:
todd.mailbox[:inbox].latest_mail

returns the last message received for each conversation you are involved in. Pretty sweet for an inbox view.

Check the RDoc.

Saturday, June 30, 2007

Rails + Memcached = Undefined Class/Module

Problem:

When using memcached, if you store Model objects in the cache, when you go to load to object back out it doesn't know about your Model classes and throws an 'Undefined Class/Module' error. The old solution for this was to add the 'model' method to the top of your controller and list the models that were needed. This is now deprecated.

Solution:

before_filter  :preload_models
 
def preload_models()
  Model1
  Model2
  ...
  ...
  ...
  Model9
end

What it do?

This is more of a hack than a solution, but it works. Add the above code to your application controller and list all the models you are storing in memcached as well as any associations that are also being stored. Referencing the Model class name must trigger a load that loading from memcached does not.

Rails Page Caching and MIME Types

Problem:

Regardless of what your template file extension is, after a page is cached using page caching the content will be served as text/html. This is fine in most cases, but what if you are caching rss or ical feeds? Rails offers a 'page_cache_extension' attribute which allows you to set the cached file extension to whatever you like but this is a class attribute and will cause all cached pages to be of this type, pretty much useless.

Solution:

<LocationMatch \/(rss)\/?>
    ForceType text/xml;charset=utf-8
</LocationMatch>
<LocationMatch \/(ical)\/?>
    ForceType text/calendar;charset=utf-8
</LocationMatch>

What it do?

This solution assumes that you are running apache. Add the above to your apache config within the VirtualHost directive. This examines the requested location and forces a MIME type for locations that match the regex. Locations that match '/rss' or '/rss/' will be served as 'text/xml' and locations that match '/ical' or '/ical/' will be served as 'text/calendar'. Done.