Year 6 with Rails
In my sixth year with Rails the Ruby community has been experiencing a sea change: gem innovation has peaked and leading Rails developers have been moving on to JavaScript 6 Node, Dockerized microservices in Go, functional Clojure on the JVM or highly concurrent Elixir Phoenix on the fault tolerant Erlang VM. Despite these defections Ruby adoption has resurged (as, coincidentally, has Java) given new versions that have improved garbage collection and performance. Rails 5 has embraced concurrent-ruby and websockets, while HTTP/2 looks set to drive further significant change to all web application frameworks in the near future.
Meanwhile our in-house multi-currency e-commerce solution has grown to 45kloc, 2,100 tests and nearly 100 gem dependencies thanks to ever expanding integration with fraud detection services, fulfillment providers, payment gateways, email marketing solutions and an autodialer. Between daily business requirements we have been addressing technical debt by straggling to Ruby 2.2, Rails 4.2, RSpec 3.4 and ActiveAdmin 1.0.0.pre2. We have adopted CBRA and gemified the persistence core of the application. New gems adopted have included VCR, Wisper and React on Rails.
On the tools side this year has seen widespread adoption of Slack for distributed teams, but we have stuck with Atlasssian HipChat, configuring integration with Airbrake for production exceptions, Assembla for tickets, BitBucket for code reviews, CodeShip for CI and Coveralls for test coverage. Increasingly global sales have raised expectations of operations to provide 24×7 uptime, monitored by PagerDuty and requiring MySQL hot backups.
‘In Container’ Rails Development With Docker
Why Docker? Because it promises to make dev-ops easier. Dockerizing an existing production environment is not trivial, moving development to Docker is perhaps a useful stepping stone. It might also make developer on-boarding easier but I haven’t tried that yet.
Docker can be installed on OS X in various ways, most notably Docker Toolbox or homebrew. I was previously using docker-machine with Parallels but recently I discovered Dlite which simplifies using docker by creating a local OS X docker.sock that forwards requests to the docker VM. Dlite also sets up a an NFS server that exports /Users to the VM. Instead of VirtualBox or Parallels Dlite leverages a lightweight hypervisor called xhyve which requires Yosemite and hardware virtualization support (sysctl kern.hv_support). Note that as of writing Docker for Mac and Windows in public beta is using a similar approach.
With Dlite started I can get a usable bash session with:
docker run -v /Users:/home -it ruby:2.2 bash
I can then cd to the project directory and bundle.
To run browser based tests I was previously building a docker image with Xvfb, Chrome and Selenium Server installed. Recently, however, I found a different approach that uses standalone selenium-server as another docker service. On a Mac workstation installing selenium-server as a service is as simple as
brew install selenium-server-standalone chromedriver
selenium-server
Once tests are passing with a local selenium-server then adding selenium/standalone-chrome to docker-compose.yml and running browser based tests inside the container is quite achievable. The location of the selenium-server is exported to the app container as SELENIUM_PORT, eg. tcp://172.0.0.5:4444
TIOBE Index for May 2016: Ruby equals best position ever
TIOBE replaces their index content each month, so I quote:
“Ruby is currently at position 8 in the TIOBE index. This is equal to the highest position it reached in December 2008. This second peak is quite the opposite of the first one. In 2006 the Ruby evangelists were shouting the language to the top. There was no room for self reflection or discussion: Ruby and its programming environment Rails were about to conquer the world. And they succeeded in this to some extent. Ruby became the language for fancy start up companies with their tight time to market schedules. It was even awarded “Programming Language of 2006″ until its popularity peaked at the end of 2008. Then scalability appeared to be a problem. Performance dropped significantly if much data needed to be processed. In the beginning of 2010 one of those fancy start ups, Twitter, started to replace Ruby by Java for this same reason. The Ruby evangelists vanished to other cool new languages and Ruby seemed to be on its way out. But not really. After being out the limelight for quite some years, Ruby’s popularity started to grow again at the beginning of 2015. This time it seems to be a more sustainable growth based on actual usage of the language instead of shouting.”
RailsConf ’16, Kansas City, MO
Wednesday. Day 1. Keynote introduction by Jeremy Daer, nee Kemper, DHH thankfully racing. Heroku presenation on Kafka, Shopify presentations on shipit-engine, request_interceptor and BuildKite, Rafael França on Sprockets, and well attended presentation by Akira Matsuda with ideas for assorted Rails performance hacks. Closing keynote from Nickolas Means with a history of Lockheed’s original skunkworks.
Thursday. Day 2. Ruby Heroes including Avdi Grimm, Charles Nutter, Akira Matsuda and Richard Schneeman. Surprisingly few vendor booths, including Indeed Prime, Hired, Skylight, RubyMine, Stitch Fix and Codacy. Rails 5 presentations on ActionCable, Turbolinks and Sean Griffin on miscellaneous enhancements. Richard Schneeman on maintaining Sprockets and David Copeland of Stitch Fix on emergent software architecture. Lightening talks on Scrivito, RocketJob, Appfolio Farcy, Railscamp, Bundler Gemstash, multi_zip, Brakeman, RubyBookClub, Reinteractive Wallaby and Benjamin Fleischer on JSONAPI.
Friday. Day 3. Aaron Patterson, now at GitHub, workshops on Hashicorp tools and RedPotion and presentations on Percy and Sandi Metz on refactoring. Paul Lamere from Spotify with a wonderful presentation on music data analytics… in Python.
P.S. I never guessed the most interesting presentation on Rails would be by Justin Searls.
From Rails 3.2 to Rails 4.2 If We Really Must
Rails 5 is almost upon us and we’ve spent the last two months, amongst other things, moving our 45,000 lines of Rails e-commerce code from 3.2 to 4.2. For better or worse the move has been less about embracing new features and more about compliance and maintaining security bug fix support. The move has been a rapid succession of stages, from 3.2 to 4.0 in development, 4.1 in production, and finally 4.2 in production. Overall it has been a significant effort and a learning experience.
The biggest challenge we encountered was not fixing our 175 models and 40 controllers, but dealing with our nearly 100 open source gem dependencies. The project started from a proof of concept elegantly scaffolded with ActiveAdmin’s DSL and has accreted functionality and third party interfaces for the past four years. ActiveAdmin remains the leading Rails admin interface, but with the departure of Greg Bell momentum has slowed. Sean Linsley, Timo Schilling and others have put a significant amount of effort into answering questions and applying bug fixes, but the ambitious 1.0 milestone has remained unmet for two years, and gem releases have been infrequent. To update ActiveAdmin first we switched from stable 0.6 gems to the master branch in GitHub, replacing meta-search with ransack, then we worked our way down the master branch to 1.0.0.pre1 in 90 day increments, isolating and working around the issues that impacted us (#2233, #2679, #2771, #3038/#3324, #3145, #3147, #3486, #3591, #3776, #3784). Critical to this effort was our test suite coverage, which remains a disappointing, but so far adequate, 90% (and yes, we upgraded to RSpec 3.x as well).
Apart from gem dependencies the most dramatic issue updating Rails was the breakage of non-strict mode used with our old MySQL database (and thus nearly everything). For Rails 4 we needed to configure ‘strict: false’ to ignore our true mode setting in the database, then with Rails 4.2 this changed again and we needed to explicitly set variable sql_mode to the required value.
So what has Rails 4.x brought us? ActiveRecord foreign keys and enums, but we embraced the foreigner and enumerize gems long ago. Spring, but we so loved Zeus. secrets.yml but we already use environment variables and dotenv. Web Console, but we are happy with Better Errors and Pry. An improved cookie serializer that fouled up our ‘seamless’ rolling production upgrade. Autoloader vexation. Millisecond DateTime rounding errors (or is that due to OS X?). ActionMailer::Preview and ActionMailer#deliver_later, but we have already built out more sophisticated solutions. Strong parameters for sure. New bin/ scripts. Stricter routing with Patch. An end to ActiveRecord’s old dynamic finder API and tweaks to callbacks and ActiveRecord::Relation. Better performance, please, from Aaron’s AdequateRecord. ActiveJob as a wrapper for Resque or Sidekiq. ActiveSupport::Testing::TimeHelpers. Perhaps for us most importantly a step closer to greater concurrency with Puma (and LiveStreaming?), Sidekiq, Celluloid, who knows?
More detailed ActiveAdmin update notes
Year 5 with Rails
In year five with Rails I finally started contributing to high profile open source projects (ActiveAdmin, ActiveMerchant), albeit with mixed success. Understanding complex libraries written by others reacquainted me with debugging in RubyMine. In addition I published the first gems of my own using jeweler, taking my public GitHub contributions over 200 for the year. Working on OSS exposed me to Colloquy IRC and Travis CI. I contributed slightly more to StackOverflow and project related Google Groups.
At my employer our ‘mature’ Rails codebase has grown from 20 to 30KLOC, monitored soberly by Coveralls (87%), CodeClimate (2.8), Codeship CI, Rubocop, Gemnasium and Bliss. I remain committed to SublimeText, now with GitGutter and SublimeGit, and to RSpec, influenced heavily by a RSpec workshop at RailsConf 2014. New gems adopted this year included (tentatively) interactors, oink and thor. Documentation has been mildly improved using YARD.
Ruby Survey 2015
Rarebit (creators of Sass and Haml) recently published the 2015 results of their annual Ruby survey, now in its seventh year.
This year saw a significant shift to Ruby 2.2 (40%) following the shift to 2.1 last year (50%). Only about 15% of rubyists remain with Ruby 2.0 or below. JRuby adoption remains below 5%.
Rails remains the dominant web framework (85%) over Sinatra (10%). RSpec still retains 70% preference while many Test::Unit users have switched to Minitest (15%). 60% of rubyists still prefer JQuery, though there has been a 10% shift to other frameworks in the past year. Interestingly the utilization of Javascript remains around the same: 25% use it heavily, 25% rarely and the rest somewhere in-between. Amongst respondents to this survey Node.js utilization has remained around 20% the last couple of years. On the front end CoffeeScript is used more often than not (55%), Haml remains preferred by 45% and Sass by 80%.
For production the preferred web server remains Unicorn (40%) over Passenger (30%) though Puma adoption has reached 15% in its first year now that Rails 4 has better concurrency support.
If you are a full time Ruby developer I encourage you to participate at http://www.askr.me/ruby
Angel Investing Overview Half Day Workshop
Presentation yesterday at Kellog School of Management’s Coral Gables campus by Troy Knauss of Angel Resource Institute, sponsored by Accelerated Growth Partners, Kauffman Foundation and Greenberg Taurig. Angel investments are typically mid-six figure series A rounds in businesses with a low seven figure valuation. 60% of angel investors have a $1M – $2½M net worth and are looking to invest 10% or less of that across a dozen start-ups, so funding a round often requires syndicating across multiple angel funds or networks. Angels are significantly more prevalent than VCs, so most angel funded business are sold without going on to VC funding or an IPO. Angel portfolios have significant risk and volatility, but with adequate diversification a 20% return is achievable, though funds may be locked up for 3-10 years. Angel funding in the south-east US is less available, meaning investors can negotiate more restrictive terms, but less capital also puts funded businesses at a disadvantage against competitors from better funded regions. Topics covered included due diligence, term sheets, valuations and board involvement.
Scaling Complex Rails Applications
After ten years of development Rails is a mature framework, and a recurring theme at conferences is how to build and maintain Rails applications that are increasingly complex. Rails achieved early success as an opinionated framework that emphasizes MVC and convention over configuration, but as developers have tackled ever more ambitious projects there has been divergence as to how to structure them.
What is generally agreed is that business logic should reside in models, away from controllers, views and API implementations. This has led to ‘fat’ models and discussion of strategies to manage them such as this blog post from CodeClimate titled ‘7 Patterns to Refactor Fat ActiveRecord Models‘. An important point often missed when first learning Rails is that app/models is not intended exclusively for classes persisted with ActiveRecord as it may include other domain objects. In addition to domain objects other popular patterns used to structure code include form objects, presenters, decorators and service objects. Presenters and decorators intermediate between controllers and views, whereas service objects intermediate between controllers and models.
Presenters and decorators are probably less controversial in the Rails community and have done much to replace view helpers. Service objects, however, have been disparaged by DHH in favor of concerns, but many developers with a traditional OOP background are unconvinced by his approach. One example project that uses service objects is GitHub itself. More recently Collective Idea have published an ‘interactor’ gem that elaborates this approach.
These strategies all result in more classes, and probably the first step in managing a codebase with many classes is to introduce namespaces. The Rails class loader automatically searches sub-folders matching namespaces so source files can be arranged similarly to the namespace hierarchy. Given most developers nowadays navigate code using global search or ‘jump anywhere’ features in their text editor the neater project directory layout is of limited benefit, but namespacing can reduce name conflicts with libraries or between different areas of an application.
The next step beyond namespacing, suggested by Stephan Hagemann is to divide a project into internal engines or gems, an approach labelled Component Based Rails Architecture. An example project is provided at https://github.com/shageman/the_next_big_thing Each component has its own test suite so the scope of regression testing of any feature change is more limited. If a component proves reusable it can be split out from the original project altogether.
Beyond a component based architecture the final step to scaling a complex application is to break it apart into separate services interacting with an internal API, an approach referred to as Service Oriented Architecture (SOA). I will defer comment on the benefits and challenges of such an approach to a later post.
See also
- Railsconf ’14 presentation ‘Concerns, Decorators, Presenters, Service Objects, Helpers, Help Me Decide!’
- Railscast on Form objects
- Railscast on Presenters
- Railscast on Service Objects