skip main content

Posts Tagged ‘Ruby on Rails’

Building the Mythology Engine

posted by Duncan at 10:06 pm on March 30th, 2010

More than a few weeks ago now, we finished a project called The Mythology Engine. This was a prototype to see if we could build a framework for navigating drama on the BBC. Our first test case was Doctor Who. Mr Ferne has written an excellent post on the subject over on the BBC R&D blog. Rather than repeat what he said there, I advise you to go have a read.

This project was a wonderful experience for me, mainly due to the fact I got to build a lot of it. This post is a bookmark for me, and runs though the technologies we used.

Mythology Engine Prototype

As a prototyping team, we get to make technology decisions that help us build stuff quickly. This affords us a lot of flexibility. So for this project we used a fairly common set of tools paired with bits and pieces to make our lives easier.

Ruby on Rails: This choice was because I have a lot of experience with the framework, and it is an ideal candidate for prototypes. In fact over the years it has been used to build some great internal tools at the BBC. Oh and I know it’s fine for production too incase you were wondering.

MySql database: Again, we have a lot of experience with this, so why change.

Apache WebServer: Familiar, and easy to setup. We run Passenger as well to make Rails deployment a breeze. Because the prototype has streaming media playback, we needed to compile in the ( H264 Streaming Module). This was a fairly simple process with good documentation. I actually think if I did the project again, I would plump for Nginx instead of Apache. Just because it is so lightweight and fast, but Apache worked just fine.

Build for browsers that support CSS3: Here’s an interesting one. So when building the pages, I realised that some of the complex design could be implemented just using CSS3, without the need to chop up images and create complex layouts. This did mean that “browsers” like I.E (at least < 9) and the like would not display the designs as expected. But hey, this is a prototype, so let's just save come complexity and do it the easy way.

As well as the technologies above, we also used Flash for our visualisations and the custom media player. We powered them using an API we built into the main site. We are lucky to have great Flash experts like Mr Bowley to build this stuff, and I even got the chance to dust off my own AS3 skills with one of the visualisations. Oh and finally, we used the jQuery javascript library for any client side trickery.

So, nothing too unusual there, or in the build really. We decided on our data model (which was influenced by the OntoMedia and ideas from the team) and url design up front, and created the database and subsequent models. We initially thought we could populate the database using provided documents (exel etc) but quickly realised this would be inadequate, and needed something quick and easy to allow data entry. Along came Typus. I highly recommend this project. It creates a simple but extendable CMS based on your Rails applications models. You also get user admin for free. We made some change to suit our project (If any of the Typus people are reading, I tried to feed back changes we made but couldn’t get your tests to run). I also wrote some filters for Paperclip so we could get all images sizes and styles from one upload.

Finally we first built an initial basic site. This was to test out initial layouts and get some understanding of how the site felt when we had real data. This layout was then tweaked and polished once we were happy.

So that was a whistle stop tour of how we built the prototype. What we ended up with was a very polished prototype, which we are all rather proud of. Current I only have screenshots to show, but hopefully that will only be temporary.

One final thing to note. We build this with a maximum of 5 people. The majority of the team were cross discipline. This really helps. It means that everyone understands what the other is doing without everyone having to work on the same area of the project. This was especially key on this project, as we built in from scratch in a very short period of time. I have been fortunate enough to work on some recent influential projects within the BBC, and all those projects had around 5 people working on them. 5 is clearly, the magic number.

So that’s a brief overview. Like I say up top, read the proper explanation on the R&D blog.

expires_in Rails FileStore fragment caching in about 6 lines

posted by Duncan at 2:32 pm on April 3rd, 2009

I have been building a Rails app that creates RSS feeds based on a lot of screen scraping behind the scenes. I really needed a way to cache the data object that feeds those specific pages. In rails you can do page, action and fragment caching, using many different methods, and storing that cache data in many different locations.

After deciding I needed Fragment caching, it turns out that if you need your cache data to magically expire over time, you need to be using memcached (I couldn’t), other wise you have to roll you own solution, based on cache sweeping, or in-code testing of whether your content has expired.

I’m using FileStore for caching (cached files live on the file system) and I have no database, and all the feed data comes from external feeds that change over time, so unless I fetch it all again, I don’t know.

What I needed was the ability to have a expires_in parameter whilst using FileStore. So this is what the code below does in surprisingly few lines. The code is an amalgamation of many posts I found doing almost the same thing, but this is what works for me.

First we extend FileStore:

# stick this in a file somewhere in your path e.g lib/fs_extend.rb
class ActiveSupport::Cache::FileStore
  def read(name, options = nil) 
    ttl = 0
    ttl = options[:expires_in] if options.is_a?(Hash) && 
    fn = real_file_path(name)
    return if ttl > 0 && File.exists?(fn) && (File.mtime(fn) < ( - ttl)), 'rb') { |f| Marshal.load(f) } rescue nil

And then in your controller I have:

class FooController < ApplicationController 
  def do_stuff
    key = 'someuniquekey'
    expire_fragment(key) unless 
        @data = read_fragment(key, :expires_in => 1.hour) || 
                                  write_fragment(key, some_complex_data)
    respond_to do |format|

This has reduced many 5 second requests to 32ms requests, so hurrah!

BBC Radio Facebook app give-away

posted by Duncan at 2:54 pm on December 7th, 2008

A year and a half or so ago I wrote a Facebook app called BBC Radio. It’s main reason for being, was to easily let you listen to BBC Radio whilst on Facebook, showing friends which networks you liked and when you listened to them as well as now/next information, and for me to understand how to build an app using the then new Facebook Developers Platform.

BBC Radio facebookThis is what appeared on you Profile Page

The app has now been switched off, and the application page will be removed just as soon as I finish this post (here is a link to a screengrab of the BBC Radio Facebook App page that was). This is due to lack of permanent hosting and to be honest interest from me to keep it going. Whilst the app was running I had a constant barrage of emails asking me if I could build them similar for their radio station. Other than give people advice, I never did. Anyway, now it’s been switched off, I’d like to give the source out to anyone who wants it. Before I give out the location, first some facts and stats:

  • It’s a Ruby on Rails 2.0.1 app running via Mongrel
  • It was deployed with Capinstrano
  • It only uses publicly available BBC data
  • It ran for over a year without me touching it
  • It was hosted on Amazon EC2 without problems
  • Application stats before switch-off:
    • Total Users: 18000
    • Monthly Active Users: 885
    • About Page Fans: 885
    • Radio Listens by the app: 30000

So there you go. You can now go and download all the source from

[REALLY IMPORTANT] This source comes as is, with no support what so ever. It was written at speed and with no concern for code quality, so I take no responsibility for how good or bad the source is, how few tests there are, and whether you can get it working or not. If you do choose to use it for anything though, it would be nice if you gave me a mention in some small way.

New Omnigraffle stencil for Rails

posted by Duncan at 2:43 pm on July 22nd, 2006

Ruby on Rails Omnigraffle stencil by metaskills.netMetaskills have created a new Omnigraffle stencil for Ruby on Rails. Omnigraffle is a great OSX tool for creating visulisations of any kind. If you use OSX then its a must for this type of thing.

It says it’s semi-complete but it seems to have the basics you’ll need to create a rails project model schema.

Another tag cloud script for Ruby on Rails

posted by Duncan at 9:09 am on July 6th, 2006

Or any other Ruby program really. I know there are attempts already out there but none that really suited my purposes. I needed a simple way of creating that tag-cloud-look for many situations. I needed a tag cloud by size or by colour and I didn’t want it to do everything, just provide me with the style attributes I needed to get the job done. Below is an example of what gets returned by default:


Here’s the code; it comes with no warranty.

With all these tags cloud scripts you really only need 3 parameters:

  1. The total counts of items – For example you have 10 tags each that has been used numerous times. The total counts would be all of the uses added together
  2. The minimum count of items in a batch – Using the example above this would be a count of the uses for the tag that has been used least
  3. The maximum count of items in a batch – Using the example above this would be a count of the uses for the tag that has been used most

Below is the actual script and then below that I have stuck a quick example of how you could use it.

def font_size_for_tag_cloud( total, lowest, highest, options={} )
  return nil if total.nil? or highest.nil? or lowest.nil?
  # options
  maxf = options.delete( :max_font_size ) || 14
  minf = options.delete( :min_font_size ) || 11
  maxc = options.delete( :max_color ) || [ 0, 0, 0 ]
  minc = options.delete( :min_color ) || [ 156, 156, 156 ]
  hide_sizes = options.delete( :hide_sizes )
  hide_colours = options.delete( :hide_colours )
  # function to work out rgb values
  def rgb_color( a, b, i, x)
    return nil if i <= 1 or x <= 1
    if a & b
  # work out colours
  c = []
  (0..2).each { |i| c && rgb_color( minc[i], maxc[i], total, highest ) || nil }
  colors = c.compact.empty? ? minc.join(',') : c.join(',')
  # work out the font size
  spread = highest.to_f - lowest.to_f
  spread = 1.to_f if spread <= 0
  fontspread = maxf.to_f - minf.to_f
  fontstep = spread / fontspread
  size = ( minf + ( total.to_f / fontstep ) ).to_i
  size = maxf if size > maxf
  # display the results
  size_txt = "font-size:#{ size.to_s }px;" unless hide_sizes
  color_txt = "color:rgb(#{ colors });" unless hide_colours
  return [ size_txt, color_txt ].join

and now a quick example of it’s use in a rails view. This code assumes the db find call looks something like this:

@foo = Foo.find( :all,
  :select => '*, COUNT(DISTINCT( AS item_cnt',
  :joins => 'AS f INNER JOIN bars AS b ON',
  :group => '',
  :order => 'item_cnt DESC',
  :limit => 50

Below just loop through your db results and calls font_size_for_tag_cloud for each item, passing in the required parameters. There are a few options available but if you leave them out you should still see something useful.

<% for f in @foo -%>
<%= content_tag( 'li', f.some_label, :style => font_size_for_tag_cloud( f.bars.size, 
) ) %>
<% end -%>

That’s it! I guess I should turn it into a plugin but hey, there’s to much reading to be done on ActiveResource at the moment heyhey!

Deploying wordpress using capistrano

posted by Duncan at 2:47 pm on May 21st, 2006

[update] I have written a new post called Deploying WordPress to SliceHost using Capistrano and Git, which uses new versions of Capinstano and WordPress, and although also now uses Git as the SCM, may still be useful.

So I was running typo up until last week. I was gradually getting more and more hacked off with it’s problems, and decided to change over to WordPress. I feel like I’ve ran every blogging tool out there, but WordPress is the most polished I’ve seen so far, and I’ve a PHP background which helps.

Don’t get me wrong, I’m loving Ruby and RoR ( Hell, it’s all I’m doing at the moment at work ) but I just want to post stuff up fast and easily, and also make changes to the blog and post them up fast and easily. After reading this article on nubby on rails, I decided to try it with wordpress. I’d been using Capistrano to deploy my old Typo blog and at work we use it to deploy all our apps. It really is a clever bit of kit and I wanted to continue to use it even though I was not gonna be running a RoR app.

Well, no problem. Although Capistrano makes a bunch of assumtions i.e. you are running a RoR app, you are using Fast CGI etc you can overide all this stuff.

First, what I wanted from all this:

  1. Easy deploy and re-deploy of my WordPress blog
  2. The image uploads when posting stuff to be seperate from the app. This is so I could override the app whenever I wanted without having to worry about overwriting all the images

Step 1.

Created a folder called wp_<app_name> and stick a public and config folder in there. Put all the files and folders that make up wordpress into the public folder. In the config folder you put the deploy script. You can either use one created in a rails app ( this gives you lots of comments on what the various sections do ) or you can just create a file called deploy.rb and stick all the code in Step 2. inside. How I got this file the long was was creating a dummy rails app and running cap -A . inside to get the default deploy.rb created for me in /config/deploy.rb.

rails dummy_app
cd dummy_app
cap -A .
mv config/deploy.rb /path/to/wp_app_name/config/
cd ..
rm -rf dummy_app

Now you should have the file/folder structure as below. This is mine:

  |- config
  |    |- deploy.rb
  |- public
  |    |- <your wordpress files>

Step 2.

Edit the deploy.rb file to suit your setup. I changed the information in the various sections as seen below:

set :domain, ""
set :application, "wp_whomwah"
set :user, "duncan"
set :repository, "http://#{domain}/svn/repos/trunk/#{application}/"
role :web, "#{domain}"
role :app, "#{domain}"
role :db,  "#{domain}"
set :deploy_to, "/users/home/#{user}/sites/#{application}"
# setting use_sudo to false means you can use
# cap cleanup ok without needing to be a sudoer
set :use_sudo, false
desc "This is here to overide the original :restart"
task :restart, :roles => :app do
  # do nothing but overide the default
# create a symlink to where I store all my images on
# the server. 
desc 'Link to central uploads folder'
task :after_symlink do
  run "ln -nfs #{deploy_to}/#{shared_dir}/uploads/" 

Step 3.

You should now be ready to go. Check all this stuff back into SVN and the you should be able to run:

cap setup

You might want to look on your server to make sure capistrano has created your app and all the folders in your :application path as stated above. Once you are happy it’s there and in the correct place you can run:

cap deploy

this should then checkout a copy of your app from subversion and stick it in the releases folder. It will also create a symlink called current, pointing to the latest deployment. It is this current folder that you need to point the server at when displaying your site.
Update: I have stuck up a copy of the deploy.rb file I use. It differs slighly from the one above. When you run the cap deply line it will:

  1. deploy app
  2. symlink uploads folder
  3. dump a backup of the db
  4. run cleanup to leave a max of 5 releases

Happy Deploying!

Rails 1.1 released

posted by Duncan at 4:49 pm on March 28th, 2006

The Rails team have now made Rails 1.1 an official release. As DDH put in his post:

The biggest upgrade in Rails history has finally arrived. Rails 1.1 boasts more than 500 fixes, tweaks, and features from more than 100 contributors. Most of the updates just make everyday life a little smoother, a little rounder, and a little more joyful.

I’ve had it on my machine for a while, but have not really delved into the new features. I look forward to trying out the new ActiveRecord stuff though as that’s an area where I’ve had to build lots of workarounds due to restrictions before. Plus the respond_to feature of ActionController also looks cool.

Switchtower gets new name

posted by Duncan at 9:02 am on March 6th, 2006

Due to a “cease and desist” from Raindance Communications, Inc. over the use of their registered trademark SWITCHTOWER on Friday, the guys have had to come up with a new name. The name Capistrano was chosen after much deliberation.

They will be making new Gems available in the next couple of days. You will now use:

cap foo

instead of the old

switchtower foo

Rails v1.0 finally released

posted by Duncan at 9:48 am on December 14th, 2005

15 months after the first public release, Rails has arrived at the big 1.0. They’ve even re-vamped the Ruby on Rails homepage in honour. Get it now and make your life complete.

Whomwah on rails

posted by Duncan at 9:04 pm on November 8th, 2005

Hurrah! I’ve finally got round to putting on rails. I it took only a morning to get the basic functionality of the old Movable Type powered site up and running, and I still have a way to go to add any complex stuff. But for now I can post stuff and manage that, and that’s a start.

I’ve also decided to use tags instead of categories as I felt a category is essentially a tag, at least it was the way I was using it before. I will add new stuff as I go along so keep on checking.

Oh and also, if you happen to subscribe to the RSS feed then it there is now a new RSS feed url, so update you subscriptions. I will stick a redirect on the old url for now.

back to the top