Last week I stole a ticket from my boss. It’s an interesting one involving streaming of XML data to another application. There are some issues with the concept and process, but it’s a legacy system and we can’t change it (other teams are busy building a replacement). We’ve also recently upgraded this project from Rails 2.2 to 2.3 and switched from Mongrel to Passenger. All this means I’ve been tasked with refactoring all of this streaming code from a custom Mongrel handler to a normal ActionController setup utilizing a render call with a Proc in the text attribute.

The XML streaming is actually a search API for our ETL system. It queries for very large chunks of data. The streaming requirement has a few positive side effects, it allows the extracts to take place quicker and it helps us run our queries and instantiate our models in batches, keeping our RAM usage under control. Check out the really simple example below.

def search
    response.content_type = Mime::XML

    render :text => Proc.new{ |resp, out|
      out.write ""
      1.upto(10){ |i| out.write "" }
      out.write ""
    }
end

I had the migrations from the Mongrel Handler to a regular controller mostly done so I decided it was time to get some tests together. I setup the first test, the simplest thing I could do was ask for the default set. It looked something like this:

test "default" do
    get :search
    assert_response :success
    assert_select "element", 10
end

I hit cmd-r in Textmate and something was wrong, I got the following exception: TypeError: can’t convert Proc into String. After about 45 minutes of debugging I found the source of my problem, line 16 of HTML::Document. After thinking it over, it seemed like the appropriate place to make the update was around line 490 in TestProcess. The update is pretty simple: check to see if @response.body responds to call, if so run it with a StringIO object, eventually passing its contents to HTML::Document.new. Check out my ticket and patch at lighthouse. Read it over, check out the patch, run the tests and give it a +1 if you don’t mind.