All posts by Eileen M. Uchitelle

Running GitHub on Rails 6.0

Post Syndicated from Eileen M. Uchitelle original

On August 26, 2019, the GitHub application was deployed to production with 100 percent of traffic on the newest Rails version: 6.0. This change came just 1.5 weeks after the final release of Rails 6.0. Rails upgrades aren’t always something companies announce, but looking back at GitHub’s history of being on a custom fork of Rails 3.2, this upgrade is a big deal. It represents how far we’ve come in the last few years, and the hard work and dedication from our upgrade team made it smoother, easier, and faster than any of our previous upgrades.

At GitHub, we have a lot to celebrate with the release of Rails 6.0 and the subsequent production deploy. First, we were more involved in this release than we have been in any previous release of Rails. GitHub engineers sent over 100 pull requests to Rails 6.0 to improve documentation, fix bugs, add features, and speed up performance. For many GitHub contributors, this was the first time sending changes to the Rails framework, demonstrating that upgrading Rails not only helps GitHub internally, but also improves our developer community as well.

Second, we deployed Rails 6.0 to production without any negative impact to customers—we had only one Rails 6.0 exception occur during testing, and it was hit by a bot! We were able to achieve this level of stability for the upgrade because we were heavily involved with its development. As soon as we finished the Rails 5.2 upgrade last year, we started upgrading our application to Rails 6.0.

Instead of waiting for the final release, we’d upgrade every week by pulling in the latest changes from Rails master and run all of our tests against that new version. This allowed us to find regressions quickly and early—often finding regressions in Rails master just hours after they were introduced. Upgrading weekly made it easy to find where these regressions were introduced since we were bisecting Rails with only a week’s worth of commits instead of more than a year of commits. Once our build for Rails 6.0 was green, we’d merge the pull request to master, and all new code that went into GitHub would need to pass in Rails 5.2 and the newest master build of Rails. Upgrading every week worked so well that we’ll continue using this process for upgrading from 6.0 to 6.1.

In addition to ensuring that Rails 6.0 was stable, we also contributed to the new features of the framework like parallel testing and multiple databases. The code for these tools is used in our production application every day—it’s well-tested, GitHub-approved code in a public, open source framework. By upstreaming this tooling, we’re able to reduce complexity in our code base and set a standard where so many companies once had to implement this functionality on their own.

There are so many wins to staying upgraded that go beyond more security, faster performance, and new features. By staying current with Rails master, we’re influencing the future of the framework to meet our needs and giving back to the open source community in big ways. This process means that the GitHub code base evolves alongside Rails instead of in response to Rails. Investing in our application by staying up to date with the Rails framework has had a tremendous positive effect on our code base and engineering teams. Staying current allows us to invest in our community, invest in our tools for the long term, and improve the experience of working with the GitHub code base for our engineers.

Keep a look out for all the great stuff we’ll be contributing to Rails 6.1 and beyond—this is just the beginning.

The post Running GitHub on Rails 6.0 appeared first on The GitHub Blog.

Upgrading GitHub from Rails 3.2 to 5.2

Post Syndicated from Eileen M. Uchitelle original

On August 15th GitHub celebrated a major milestone: our main application is now running on the latest version of Rails: 5.2.1! 🎉

In total the project took a year and a half to upgrade from Rails 3.2 to Rails 5.2. Along the way we took time to clean up technical debt and improve the overall codebase while doing the upgrade. Below we’ll talk about how we upgraded Rails, lessons we learned and whether we’d do it again.

How did we do it?

Upgrading Rails on an application as large and as trafficked as GitHub is no small task. It takes careful planning, good organization, and patience. The upgrade started out as kind of a hobby; engineers would work on it when they had free time. There was no dedicated team. As we made progress and gained traction it became not only something we hoped we could do, but a priority.

Since GitHub is so important to our community, we can’t stop feature development or bug fixes in order to upgrade Rails.

Instead of using a long-running branch to upgrade Rails we added the ability to dual boot the application in multiple versions of Rails. We created two Gemfile.lock’s: one for the current version Gemfile.lock and one for the future version Gemfile_next.lock. The dual booting allows us to regularly deploy changes for the next version to GitHub without requiring long running branches or altering how production works. We do this by conditionally loading the code.

if GitHub.rails_3_2?
  ## 3.2 code (i.e. production a year and a half ago)
elsif GitHub.rails_4_2?
  # 4.2 code
  # all 5.0+ future code, ensuring we never accidentally
  # fall back into an old version going forward

Each time we got a minor version of Rails green we’d make the CI job required for all pushes to the GitHub application and start work on the next version. While we worked on Rails 4.1 a CI job would run on every push for 3.2 and 4.0. When 4.1 was green we’d swap out 4.1 for 4.0 and get to work on 4.2. This allowed us to prevent regressions once a version of Rails was green, and time for engineers to get used to writing code that worked in multiple versions of Rails.

The two versions that we deployed were 4.2 and 5.2. We deployed 4.2 because it was a huge milestone and was the first version of Rails that hadn’t been EOL’d yet (as an aside: we’d been backporting security fixes to 3.2 but not to 4.0+ so we couldn’t deploy 4.0 or 4.1. Rest assured your security is our top priority).

To roll out the Rails upgrade we created a careful and iterative process. We’d first deploy to our testing environment and requested volunteers from each team to click test their area of the codebase to find any regressions the test suite missed.

After fixing those regressions, we deployed in off-hours to a percentage of production servers. During each deploy we’d collect data about exceptions and performance of the site. With that information we’d fix bugs that came up and repeat those steps until the error rate was low enough be considered equal to the previous version. We merged the upgrade once we could deploy to full production for 30 minutes at peak traffic with no visible impact.

This process allowed us to deploy 4.2 and 5.2 with minimal customer impact and no down time.

Lessons Learned

The Rails upgrade took a year and a half. This was for a few reasons. First, Rails upgrades weren’t always smooth and some versions had major breaking changes. Rails improved the upgrade process for the 5 series so this meant that while 3.2 to 4.2 took 1 year, 4.2 to 5.2 only took 5 months.

Another reason is the GitHub codebase is 10 years old. Over the years technical debt builds and there’s bound to be gremlins lurking in the codebase. If you’re on an old version of Rails, your engineers will need to add more monkey patches or implement features that exist upstream.

Lastly, when we started it wasn’t clear what resources were needed to support the upgrade and since most of us had never done a Rails upgrade before, we were learning as we went. The project originally began with 1 full-time engineer and a small army of volunteers. We grew that team to 4 full-time engineers plus the volunteers. Each version bump meant we learned more and the next version went even faster.

Through this work we learned some important lessons that we hope make your next upgrade easier:

  • Upgrade early and upgrade often. The closer you are to a new version of Rails, the easier upgrades will be. This encourages your team to fix bugs in Rails instead of monkey-patching the application or reinventing features that exist upstream.
  • Keep upgrade infrastructure in place. There will always be a new version to upgrade to, so once you’re on a modern version of Rails add a build to run against the master branch. This will catch bugs in Rails and your application early, make upgrades easier, and increase your upstream contributions.
  • Upstream your tooling instead of rolling your own. The more you push upstream to gems or Rails, the less logic you need in your application. Save your application code for what truly makes your company special (i.e. Pull Requests), instead of tools to make your application run smoothly (i.e. concurrent testing libraries)
  • Avoid using private API’s in your frameworks. Rails has a lot of code that’s not private but isn’t documented on purpose. That code is subject to change without notice, so writing code that relies on private code can easily break in an upgrade.
  • Address technical debt often. It’s easy to think “this is working, why mess with it”, but if no one knows how that code works, it can quickly become a bottleneck for upgrades. Try to prevent coupling your application logic too closely to your framework. Ensure that the line where your application ends and your framework begins is clear. You can do this by addressing technical debt before it becomes difficult to remove.
  • Do incremental upgrades. Each minor version of Rails provides the deprecation warnings for the next version. By upgrading from 3.2 to 4.0, 4.0 to 4.1, etc we were able to identify problems in the next version early and define clear milestones.
  • Keep up the momentum. Rails upgrades can seem daunting. Create ways in which your team can have quick wins to keep momentum going. Share the responsibility across teams so that everyone is familiar with the new version of the framework and prevent burnout. Once you’re on the newest version add a build to your app that periodically runs your suite against edge Rails so you can catch bugs in your code or your framework early.
  • Expect things to break. Upgrades are hard and in an application as large as GitHub things are bound to break. While we didn’t take the site down during the upgrade we had issues with CI, local development, slow queries, and other problems that didn’t show up in our CI builds or click testing.

Was it worth it?


Upgrading Rails has allowed us to address technical debt in our application. Our test suite is now closer to vanilla Rails, we were able to remove StateMachine in favor of Active Record enums, and start replacing our job runner with Active Job. And that’s just the beginning.

Rails upgrades are a lot of hard work and can be time-consuming, but they also open up a ton possibilities. We can push more of our tooling upstream to Rails, address areas of technical debt, and be one of the largest codebases running on the most recent version of Rails. Not only does this benefit us at GitHub, it benefits our customers and the open source community.

The post Upgrading GitHub from Rails 3.2 to 5.2 appeared first on The GitHub Blog.