A new mentorship program

I recently returned from RailsConf 2012 in Austin.  I had a terrific time,  learned a ton, and most importantly: I was inspired.  I was inspired to return to making more contributions to Open Source and to help introduce others (kids and adults) to the computer world.

RailsConf had three double-sided whiteboards that (as far as I could tell) with nothing but “We’re hiring” ads on them.  I also stopped by a job fair where there were more companies hiring than applicants.  And finally, when all the attendees were asked if they worked for a company that was hiring Rails developers, at least two-thirds raised their hands!

So my first idea to get wings is a new mentorship program I’m starting in my town.  This is specifically a locally-oriented program.  I’m currently searching for 1-3 high school or college students who are interested in learning how to be the kind of well-rounded & highly-knowledgeable developers that these businesses are looking for.

Over about the next four months, we’ll cover the types of topics not typically covered at university, such as: git, TDD, setting up your own servers, and a complete understanding of the Rails stack.  This isn’t an internship (they’re doing their own work, not work for some other company), and it’s not a class (I’m not getting paid).  It’s just a chance to get some young new developers off on the right foot.  There will be a ton of material to cover, and there definitely will not be a speed limit!  Along the way, I hope to have each student come up with their own web app idea and fully implement it, start to finish.  In the end, I hope they’ll wind up with something even more impressive than a portfolio to be able to show potential employers.

I’m pretty excited and hope to find some equally excited students.  I’ll be periodically offering updates on how the program is going.  I also hope to open source my lesson plan as well for use by others, so check back for more!

RailsConf 2008 – Final Summary

Birds of a Feather

BoFs were a great part of RailsConf. I didn’t get a chance to write about them elsewhere, so I’ve tacked them on here:

BoF: PoolPartyRb

Saturday night, I sat in the BoF talk on pool-party. It’s a gem for automating deploying to Amazon EC2. It can handle a lot of things:

  • Automatic scaling based on demand
  • Starting/stopping instances
  • Provisioning and bootstrapping initial software
  • Setting up S3Fuse
  • Load balancing with HAProxy
  • Built in monitoring
  • Extensible with plugins (coming soon?)

The software is very new, but Ari Lerner already has a ton of functionality that I’m pumped to use. It also seems to fill a lot of the same niche as RightScale (at least for Rails) and it’s open source. His presentation slides are also available if you’re looking for more info.

BoF: Annoying IT

We sat down and had a great talk about how Rails developers should work within the IT system of a medium or large company. There is a often a large gap between the developers and the system administrators. It is important that developers communicate with the IT department from a very early stage to make the unique requirements of a Rails application known. It begets a bigger question that a ‘Rails’ developer is truly many different things: They are a developer, a sysadmin, a DBA, and a Business Analyst. The nature of the product, the Agile development methodology, etc means we must be jacks-of-all-trades in order to succeed.

This raises an interesting corrallary: We turn out to be better developers and better engineers just because the idiosyncrasies of Rails force us into working with all these other ‘puzzle pieces’ that previously were the domain of specialists. Now obviously not every Rails coder is a ninja or rockstar, but I’d say on the whole, our group is certainly cream-of-the-crop, thanks in large part to the requirement that we branch out and work on the ‘big picture’ of web app development and not just being ‘code monkeys’.

Vendors/Expo

There were a fair number of vendors in the Exhibit hall highlighting their latest projects and products. Nearly everyone gave away a t-shirt (for next year, if anyone needs an idea for schwag: how about soft cloths/screen cloths for laptops? I sure could have used one!). Most of the vendors were demoing/pitching their normal, well known product (RightScale, EngineYard, Amazon WS, etc). There were a few new releases too though:

  • FiveRuns released their Tune-Up tool. I didn’t really ‘get’ TuneUp at first when visiting the website, but after talking with Brian about it, I learned what an awesome product this is. It’s available for free and will make developing in Rails an even greater pleasure. If you’re using NewRelic RPM, it has much the same functionality minus the social-performance-sharing.
  • Pivotal Labs demonstrated their software for tracking stories/tasks looks incredibly cool (and pretty). I got to watch a little of it and talk to the guys. Unfortunately, now I can’t find any web info about the product, when it will be released, etc. If I find more info, I’ll be sure to post it.

Final Summary

The event as a whole was extremely well organized and absolutely amazing. Thanks to Chad Fowler, Rich Kilmer, Dave Black, O’Reilly, etc. for the great job. Some people complained about the ‘commercialization’ of the conference, but given the size, that should be no surpise. It’s not a bad thing…if you want tiny community conference go to MtnWest or Hoedown. I came expecting a very large forum for sharing ideas and talking about how to make Rails put food on the table, and I wasn’t disappointed.

The Conference ‘Theme’

The vast majority of RailsConf2008 was centered around improving yourself as a developer/engineer. All the keynotes and many of the talks were great insights and provided a ton of inspiration to the mantra I try to live by: The best way to be a Rails developer is to know about everything else besides Rails. I think for a lot of people at the conference this was a mind-blowing concept and I’m sure many of them were upset/angry/confused that we spent so much time talking about things other than Rails. The key to remember is that, just like good programming techniques, these styles of development and a thirst for knowledge is key to our way of life. These ideals transcend any given language or framework and will last us for years to come.

The MVC commercials

Easily one of the most talked about parts of the conference. These commercials were hilarious. I hope to see them posted online very soon!

Presentations

Many presenters had very good powerpoints. It’s nice to no that even though we are not all master designers, practically ever presenter had clear and concise presentations. There were basically three types of slides: One Sentencers, Full of Code, and Comedy. No un-readable gobbeldygook and no reading from slides here. All very good, nice job guys.

The InterWebs

Network access at the conference was the best ever. It was fairly reliable and amazingly responsive. For this type of conference, good internet access should the the absolute top priority, and the Ruby Central guys definitely gave it its due.

Complaints

No tables in the conference halls. Maybe this is a logistical issue (although the Portland convention center probably has capacity for twenty times as many people, I’m sure they can find/rent some tables).

Talks Format

I missed out on a LOT of talks I wanted to see. This is just how things work: there are only a certain number of hours in a day, so you have to pick and choose. Some conferences have been known to schedule every talk at two different times so that there are more ways to schedule your day and make sure you don’t miss any talks that you really want to see. This may or may not work for RailsConf, but I’d like to see them explore some alternatives for how to organize the talks.

Help me out

On Friday, I heard of a great Rails plugin that provides a suite of rake tasks for starting and stopping memcache/sphinx/etc. If anyone knows the name of this plugin, please put it in the comments.

Rails doesn’t scale: It does now!

This was without a doubt the year of scaling: There were 31 talks about deploying/hosting/scaling rails and 50% of the vendors were focused on hosting and scaling Rails apps. It became very clear that the true fact is that Rails now scales better than any other language/framework out there.

Some more quotes

  • “Screw the databases, why would you want to use a relational database” – Chad Fowler
  • “[On GitJour] People were trying to find out how to make 28 lines of code [turn into] not 28 lines of code” – Evan Phoenix
  • “I had a great time, so I assume it was awesome.”
  • Joel Spolsky:

  • “If Brad Pitt is an Apple iPod, then Ian Somerhalder is a Microsoft Zune”
  • “Ian Somerhalder gets a free gift certificate for SuperCuts”
  • “Take a digital picture and upload it to icanhazcheezburger”
  • “‘In order to serve you better’ should just be part of the message box API”
  • “The message there [Abercrombie site] is if you buy these clothes, you will get a girlfriend”
  • “The iPhone is sleek and stylish and you get the feeling that if you swallowed one, it’d go right down!”
  • “Contributing to open source projects is like networking for people with no social skills”
  • “I vaguely resemble TVs Burt Reynolds and that makes me somewhate of an authority”

See you all at RubyConf2009!!!!

RailsConf 2008 – Sunday Afternoon Summary

Advanced ActiveRecord Techniques

Chad Pytel

Chad’s talk was about refactoring your Rails code to use ActiveRecord the way it was meant to be used. It was a perfect example of a more advnaced (though still easy to understand), technical, & code-heavy talk. As such, I don’t have much in terms of notes (His slides will be available from Smashing Robots into Other Giant Robots at some point soon). He mentioned that in order to refactor effectively, you must have a good solid test suite or you will break things you never knew could break! He talked about moving code from the controller to the model, he talked about using callbacks, modules, and mixins where appropriate. I can’t really go into depth on the path he took to refactor a bunch of code, but it was a terrific opportunity to validate how I write code and know that what I do is sanctioned by others (or at least one other person). He had this nugget of wisdom that I thought sums up the talk quite well:

A great metric for your controllers (and code in general) is ‘can you put it up on a slide’. If you can’t, something’s wrong.

Keynote – Rails Panel

Chad opened the panel by announcing that RailsConf2009 will likely not be held in Portland next year. He mentioned Las Vegas as a possibility, but I don’t know if he was joking. He also said that RailsConf will be sponsoring CabooseConf next year for free. Just example #10001101011 of why the Ruby/Rails community is so amazing.

The core team started by answering a question about the roadmap of the framework. They said that although they want to do some cleanup and refactoring in ActiveRecord, there is no roadmap for what’s coming up. Then they answered a question addressing plugin support (breaking compatibility with popular plugins) and DHH pointed out that they have really had to start dealing with an issue where “Many people don’t want to build against edge because things can change or be added or removed, but edge can’t be released until it is tested“.

DHH plugged the docrails branch on github that makes it much easier for developers for contribute documentation about Rails. They also pointed out that github pull requests on the Rails core do not go anywhere, they just disappear. I thought that was good to note, as I was curious how they handled pull requests. DHH also mentioned that he isn’t happy with a log of the view helpers such as ‘error_messages_for’ and a few others, so if anybody was looking for something to work on in Rails core.

I got up and asked about how moving to git and lighthouse has impacted their work. I got a good response that both git and lighthouse have improved their process because they can better track issues using lighthouse and that git makes merging patching much easier. There was also a great question about handling security issues. When Rails had a major security issue discovered the first time, they were unprepared for the process they had to go through. They are more aware of that process now and have a more formal security handling protocol in place. Awww…Rails is growing up!

And that’s the end of the conference! Look for my final notes shortly!

RailsConf 2008 – Sunday Morning Summary

Dispelling the Myths about Rails performance

Lewis Cerne – NewRelic

This was another vendor talk that I decided to attend. I’ve been betatesting the New Relic RPM system for awhile. I decided to attend to see if they would talk about upcoming features that may change my opinion on what NewRelic has to offer. Lewis started by comparing a lot of the FUD about Ruby & Rails performance/scaling to the same statements made about Java and other technologies 10 years ago. He noted the key truth that Rails developers are always using to back up their product: “It’s okay that Ruby is slower. We make up our gains in other areas”. He also highlighted some key observations that I’ve heard myself say many times before: You need data to know what optimize. Otherwise you’re wasting time and money and often just making things worse.

Lewis moved on to describe the the systems that power RPM. Nothing surprising, but it looks to be a solid and scalable architecture (as it should be since their goal is to help you scale your app). Of course, none of this is a surprise given that their cluster is run by EngineYard. He highlighted a great point about putting Rails apps into production: “Testing is important but not enough – You can’t predict or reproduce what will happen in production”. This is an important lesson to remember because good testing/test coverage has nothing to do with how your app will handle load. Bugs and load are very different issues and you must handle them in different ways.

I really started to get interested when Lewis mentioned that you can build your own custom data handling for targetting performance that is very specific to your app. This starts to sound a lot like plugins for ScoutApp, munin, etc. It’s not clear just what kind of custom data you can collect (are you able to run any Unix command for data collection, or are you limited to something inside Rails?), but it certainly sounds like a step forward.

Lewis concluded by noting that they really eat their own dog food and use nothing but Rails for both their collectors and their ui. The collector portion was especially surprising given how easily this type of data collection function lends itself to using a much more specialized setup instead of Rails. So good for them.

Ward Cunningham, of the original WikiWiki and now AboutUs.org came up to talk about his experience in switching from PHP to Rails and using NewRelic to monitor their performance. He relayed a great story about doing a 10 minute live switch from PHP to Rails and then using the data collected by RPM to find out what the problems were with their deployment. After fixing those problems, they were able to run the site effectively and easily. Although I believe AboutUs could have done the same thing with FiveRuns or ScoutApp, the key is that this style of performance monitoring is critical to deploying any Rails app.

Overall, a good presentation and I’ll still be watching NewRelic closely because performance monitoring continues to be a critical part of developing Rails apps.

Oh the Fail I’ve known

Adam Keys

Adam produced absolutely one of the best talks of the conference. He presented a lot of ideas about how best to use fail, accept that it is the best way to learn, and protect yourself when you do fail. He started by mentioning some different types of failure:

  • Learning failure – The knowledge was out there already and you didn’t know about it.
  • Technological Failure – Not having a tool that can do what you need to do, or not choosing the correct tool.
  • Solve the right problem – Yak shaving. Adam specifically mentioned extracting pieces out as a framework when there is no need to: 3 strikes before you refactor.
  • Inappropriately applying the wrong technology – There’s a better tool out there and you’re not using it.

Adam then keyed in on the core of his talk: Learning. He said there are only kinds of learning: Learning from others & Learning by doing. You only learn by falling down. Nobody gets everything right the first time. Set yourself up to rapidly trying things until you find what’s right. Just like you don’t play hockey without padding: don’t develop without padding. Unit tests, exception notifier, cheap branching/merging in git, fast deploy with capistrano are all padding that mean you fear less and can try things that might hurt you because you can fix them fast.

Adam then moved on to some ways to ‘supersize’ your learning by mistakes:

  • Make bigger mistakes – Because you’re going to make mistakes anyway, you mind as well swing long so that when you do succeed: it’s spectacular (This is a great sentiment that I think applies very well to running a business too).
  • Getting into the groove – We all have issues with getting into the groove when programming without being distracted. Adam recommended that you need a jumpstart, because once you get started, you are likely to continue coding. Set small goals or fix a few small bugs to get the ball rolling.
  • Layer up, Layer down – Adam also mentioned that the best way to expand your learning scope is to work with things adjacent to your main area of work. He addressed specifically other parts of the Rails production stack: SQL, Web servers, etc. Again we find ourselves talking about being jack-of-all-trades by knowing about more than just our tiny little slice of the computing stack.
  • Learn a new language every year – The grass is always greener on the other side, so explore that ‘other side’. Know if there is a better way of doing something, or at least be sure that you’re still using the best tool out there
  • Passive Mentors – Adam talked about ‘shadowing’ many great developers, in IRC, on blogs, whatever. Learn what they have to share, how they operate, the mistakes they’ve made, etc.

“Beside writing code, learning is what we do as software developers” – Adam Keys

“Hurry up and lose your first 50 games.” – Go proverb presented by Adam Keys

RailsConf 2008 – Saturday Evening Summary

Keynote

Announcements

Chad announced that to help with the overflow issue, those talks from Friday that were overflowing (5 of them) have been scheduled to be re-presented again Sunday morning. Once again, the Ruby Central team has come through with the right priorities and great organizational skills (or deep pocketbooks, who knows…). This conference is definitely a success thanks to them.

Kent Beck

Kent Beck, commonly considered the ‘father’ of eXtreme Programming, was tonights keynote speaker. He started by describing that he really has no presentation, no slides, no title, no abstract. He came just to tell stories about the things of substance that he has done in the last 20 years. He said he has a lot of stories to tell, but basically, it is about 3 ideas: Tests, Patterns, and XP and all three had the impact they did because of talent, time, and luck. He related the concept that it truly takes 20 years before he can see the true impact of what was going on. After the first 5 years, something would be big, but the next 5, 10, 15 years would show an exponential increase that he could never foresee. This is especially relevent for Rails given that we are only about five years in at this point.

Kent gave some great stories and examples about how the early days of Patterns, Testing, & XP came about. He closed his talk by addressing Zed Shaw’s memo from a few months ago. He said that what he found in the memo (after wading through a lot of the junk in the memo) is that “Rails needs a transparant services market”. As I read the memo again, I realize that this is a terrific observation about where Zed Shaw’s true problem lies and brings to light what I hope to be a great opportunity for the Ruby & Rails community

A standing ovation is sort of a given these days because it’s pretty much a given, but Kent truly deserved it. He gave a very moving presentation about where ‘good’ programming is coming from and where it should be going. If you want to better yourself as a developer, it behooves you to learn everything Kent is talking about.

Some funny sayings of the day

“I did it, and it felt like I was cheating! [On TDD]” – Kent Beck

“I’m still making money writing smalltalk programs, but I’d say that’s probably not a method you want to follow” – Kent Beck

“You can determine the purpose of any business memo in constant time by flipping to the second last paragraph” – Kent Beck

“Obviously it’s in Japanese and that’s sorta a problem”. – Patrick Farley

“Because that’s how ninjas really are. If you ask them to speak, they’re like ‘I don’t even know what you’re talking about'” – Patrick Farley

RailsConf 2008 – Saturday Afternoon Summary

Lightning Talks

Pastie

The creator of Pastie, talked about the cool features of Pastie supporting the full suite of syntax highlighting. Pastie has textmate, Vim, & IRC integration. The IRC integration is particularly awesome because it supports monitoring the channel to request a pastie from people that are asked for one. He shared some nice stats and insights into Pastie. It’s a great service that I swear by if you’ve never used it before. Enjoy.

Arduino boards

There was a terrific talk by Ben Bleything at RubyConf2007 on building microcontroller code using Ruby. This was a nice quick demo about writing ruby code for the Arduino boards. This is something that I definitely hope to get into a lot me, so I always enjoy seeing people playing it. Check out http://rad.rubyforge.org

ScrumNinja

David Lowenfels – Internaut

I saw a paper on the bulletin board for something called ‘ScrumNinja‘. This talk by David introduced ScrumNinja. It uses the cards-on-a-board model for organizing stories, tasks, etc. It looked pretty nice, but in very early development. Pivotal Labs demoed at their booth a very similar product that they are releasing.

Rubber

Matt Connelly

Rubber is a rails for managing & running EC2 instances. Matt gave a really fast demo of starting and bootstrapping EC2 instances. It’s quite new, but I’ll be watching this product.

Acts_as_revisable

Rich Cavenaugh – withoutscope.com

Acts_as_revisable is a Rails plugin that handles very detailed rules for revising records. It supports branching, reverts, and some great state-transition work and other cool features. I can think of a lot of great places to use this type of extension and I look forward to checking it out.

earfl

Earfl (pronounced Ear-full) is an app that provides Phone voice IVR style functions without using Asterisk. Here’s a quote from their website:

“Add a phone number for collecting audio content to anything! This gem is a simple client to Earfl’s RESTful API. An earfl is a great audio moment. A story. A reaction. A review. An opinion. A once-in-a-lifetime event captured in real time.”

Code Gear

I’ll admit it, I’m an IDE junkie. It’s terrible I know, but I have yet to find a lightweight text editor that can do the things I want. The biggest lack is always code completion & inline API lookups. I’ve been using NetBeans almost exclusively sicne the very early version 6 betas (about a year) and I’m extremely happy with all the Ruby and Rails support that they’ve built into it now.

I’ve tried Aptana/RadRails, but it’s much too Eclipsey for me and has a ton of junk that gets in my way (ironic, I know). So I decided to attend the CodeGear presentation, where inevitably they will be talking about 3rd Rail (no matter what the title of the talk is). They started with a poll of what text editor was being used. As expected, there were a lot of textmate users, but also a lot of NetBeans & Aptana users, so I felt a little better.

One thing that was mentioned several times is that 3rdRails provides wizards for the common tasks but ‘teaches you how to use the command line’. I didn’t see any of that actually happen and the wizards felt a lot like “The company deciding what’s best for the developer”. 3rdRails also offers intelligent code completion, refactoring support, syntax checking, inline debugger, etc. It includes some features for supporting methods that don’t really exist (such as find_all_by_name).

The demo started by showing starting a Rails app and debugging an issue using a breakpoint and the dependency tracker. They also demonstrated the ruby syntax checking by showing an unreferenced variable. A lot of these things are very difficult for a Ruby IDE thanks to being dynamically typed. In order to provide context for the code you are editing, the IDE has to know what kind of object something is, even if the program itself doesn’t care. It’s not easy stuff, but I feel that NetBeans has already implemented this effectively.

As I watched the presentation some more, it became very obvious that the IDE is VERY smart when handling Rails code. It has a lot of custom handling for Rails-isms that NetBeans can’t handle. I walked in not really keen on 3rdRail given my previous experiences demoing it and my big preference for NetBeans, but I was hoping that they would convince me otherwise and prove that 3rdRail was really an amazing platform. I’m not yet sure it’s “amazing”, but I’m certainly ready to give them another try (and I’ll let you know what happens).

In the end, one thing I got to thinking about is that whether it’s NetBeans, Eclipse, 3rdRails, etc….we’re Ruby programmers writing Ruby in an IDE written in something else. Why do we not have a really good IDE written in Ruby. Sounds like I’ve got to get to work!

Update after the keynote

Kent Beck echoed this exact same thought…talk about validation of my ideas!

Technorati Tags: , , , ,

RailsConf 2008 – Saturday Morning Summary

Jeremy Kemper Keynote

Rails 2

Jeremy’s keynote was billed as the technical talk that DHH didn’t give. He started by saying that Rails 2 is all about working with resources. Rails2 introduced a lot of features involving a unified way of referencing an object, throughout the DOM, in the code itself, etc. He also mentioned that Rails2 dropped a lot of ‘fat’, pulled deprecated features, improved performance, etc and talked about making choices about what needed to be kept and what could be shed.

Then he moved to Rails 2.1, which has been in development over the last 6 months with 1400 contributors, 1600 patches, & 9000 comments. The core team wasn’t able to keep up, so they moved to Git, Github, & Lighthouse. Git provides the excellent ability to fork your own version of Rails and offers many ‘gateways’ to committing code, so that the core team members are no longer the sole ‘gatekeepers’.

Then Jeremy referenced the move to Lighthouse. He only mentioned it very shortly, which I think is unusual. Whereas the move to github was a natural, obvious, and powerful evolution of managing the github core, I think it still stands to be seen how things will work out with Lighthouse. Lighthouse is fairly new and has a limited featureset compared to something like Trac. I haven’t really heard any personal experiences about if Lighthouse is truly better for the core team or not, so if you have any insights, I’d love to hear them.

There are many new features of Rails 2.1:

  • Merging migrations. The new UTC based migrations and out-of-order migrations are one of the biggest new features in Rails 2.1 This is definitely going to be one of the biggest benefits to developers and is an exciting new feature.
  • But it’s not the MOST exciting new feature. That honor goes to the topic Jeremy discussed next: Time Zone support. Rails 2.1 now supports setting a timezone at the start of a request and all the times you see throughout the request will be given in that timezone. This is something I hope will eventually be moved to Ruby core because it is so mind-blowing in terms of how easy it makes handling time zones.
  • Jeremy also very quickly previewed the new change_table syntax. Then he moved on to a live demo (Where else will you find a conference with 2,000 attendees and the keynote speaker writes live code?). He demonstrated all these new features on a blank app with very few hiccups. Pretty cool.
  • Some news to me was that memcache-client is now bundled with Rails 2.1 for handling caching. This is an awesome improvement that will allow you to easily use memcache (including expiring cache) globally throughout Rails very easily.
  • Dirty checking & partial updates – You can know what DB columns have changed and only changed columns will be sent to the DB with an UPDATE. Jeremy noted that there is a nice side effect of cleaner logs. He also noted that there is a danger with partial updates where validations may involve the entire object, but if you only update part of the object, then concurrent updates may cause an invalid record. He recommended using optimistic locking for any cases like that.
  • Smart :inclues. This is about avoiding joins when two seperate lookups are faster (because with a join, you’ll likely return many duplicates: e.g. If you have 1000 messages that belong to only 3 users, all 1000 message records will return all the user data for each of those 1000 rows. So basically 2000 records get returned. Seperate the queries instead of joining and you get 1003 records).

Jeremy demonstrated Rails running on Rubinius, jRuby, ruby-1.9; which got some applause. He mentioned some off-the-cuff performance data of 20-60% increases with Rails using 1.9: not the ‘golden cow’ to worship, but decent none the less. He also said that mongrel, MySQL, Postregres, etc are all supporting or very very close to supporting 1.9.

Overall, a great new set of features that we’ve been reading about quite a bit recently. Of course, it was no surprise when Jeremy announced (to big applause) that Rails 2.1 is released as of today.

RubyConf2008

Rich Kilmer

Rich got up to note that RubyConf2008 has announced their venue: The Orlando, FL Omni Hotel & Resort. RubyConf2007 had some problems where a lot of us wanted to hang out at night, play werewolf, code, etc to the point where I actually wound up renting a conference room at a hotel down the block so we could be there into the night (thanks to all those who threw in a few dollars to cover the cost).

He made it clear that the priority was to have a space that we could basically use 24 hours a day. The resort has FOUR restaurants on the property, DS3 line on premise with 5mbps wireless everywhere, a pool that will be open all night for us, easy access to Disneyworld, and a firepit with chairs in a circle for what I believe will be some awesome werewolf. November 6-7-8. I’ll see you there!!!!

ESI

Aaron Batalion

Aaron’s talk is another presentation dealing with scaling Rails for large numbers of users. But he addresses a much different area. Instead of scaling hardware, the database, app servers, etc., he is talking about about what you can do inside your Rails app to optimize your views and handle more traffic on the same hardware you already have. I consider this as being pretty important because this should be one of your first steps in scaling your site before you start replicating and sharding your database.

Aaron starts by addressing a real issue with many web apps: Most pages are very personalized for each user. If you are using restful_authentication, then you are basically guaranteed to require custom content for each user. He proposes using Edge Side Includes (ESI) in order to avoid using your mongrels all together (as much as possible).

For those who aren’t familiar with ESI: It is an a set of XML tags that allow you to build a complex HTML by breaking it down into several smaller pieces, some of which can be cached long term (general content) and others that are more specialized to an individual request. For Rails developers: think cached render :partial that is assembled into a whole document by the ESI server instead of by Rails.

Aaron showed a great example of a page with some general content and a sidebar with some custom content for each user that demonstrated how the fragment caching works.

So why should we care? Rails already has a several methods of fragment caching (typically with memcache) that provides these services.

Well, ESI has a number of extra benefits:

  • Personalized Full Page Caching – Often times, the entire page can be cached for a given user
  • Slow/Broken Dependencies – By breaking up a page into several pieces that are assembled outside Rails, we can concurrently load those pieces. The request is faster (to the user, in exchange for more load on the server) and if a request should fail, ESI supports try/fail that will load a backup page instead.
  • Application Sharding – Different pieces of the app can be served by completely separate instances
  • Polyglot Assembly – An extention of sharding: ESI allows you to use multiple backends for different fragments and these fragments don’t need to be provided by the same app. Think of it as similar to RPC/SOAP requests from remote services giving you back whole fragments.
  • Cached New User Experience – The cache can be specific to users (using cookies and URL params), including brand new users. So your non-user pages are basically fully cached
  • Inline Invalidation – Easily invalidate caches when updates are made
  • FragmentFu – You get view helpers and other methods that will generate the ESI tags you need. Even better…you get a special respond_to format.fragment so you can use the same view and render both a full page or just a fragment depending on the request. This isn’t available yet, but Aaron says it’s coming.
  • Deployment – There are open-source, commerical, and CDN options:
    • Mongrel-esi
    • Squid 3.0 (partial support)
    • Varnish (basic esi support)
    • Commercial Oracle web cache
    • Commercial – Akamai – Most complete implementation
  • Pros and Cons
    • Cons
      • YAGNI – Since ESI is really only well supported by the ‘big dogs’ and that means big money.
      • Cache invalidation
      • Lack of mature open source
      • Cost of deployment
    • Pros
      • Concurrent execution
      • Efficient execution – Use Rails only when you need it. Putting pieces together is not what Rails should be doing.
      • Conditional/ AB testing
      • Forces you to split one ‘action’ into smaller restful pieces
      • Syndication for Free – You’ve already built it by splitting up the page
      • Geographically Distributed

Aaron’s presentation will be available from http://blog.hungrymachine.com

Overall, Aaron provided a good talk with some code examples to really help people understand how ESI can be used. Unfortunately, it’s obvious that the technology is still pretty commercial and there’s not a lot of open-source options. This means that only the largest of the large will need to worry about trying to use it. Anything smaller, will likely just use normal fragment caching.