This is part 2 of many in my "Open-source as a project model for internal work" series and a follow up to "Empathetic Infrastructure Code Management: An Introduction"
So you've seperated your codebases by application and infrastructure code. Now what? One of the most dramatic changes you're now able to make comes from the idea of a push vs. pull release model.
When the app and infrastructure codebases were conjoined, every change released was immediately pushed out to all application code. That's great for getting features quickly released, but it's also unfortunately great for efficiently breaking your app.
Imagine that Team Hufflepuff has been working for three weeks on a branch for their next major piece of functionality. They've just completed their last bit of functional testing and are ready to prepare for a release.
Unfortunately for Team Hufflepuff, the day before, the infrastructure team released their own update that modified a crucial bit of code that Team Hufflepuff was expecting to work a specific way. Now the Hufflepuff team is in scramble mode because they didn't think to tell the infrastructure team about their functionality, because why would they need to know? The infrastructure team has already merged their code and Team Slytherin is relying on those changes to get their own release out. At this point, people start bracing for a long weekend of refactoring code.
There's a better way. Instead of having infrastructure code "pushed" out immediately to the various apps, allow the apps the room to "pull" in the code when their ready. Allow the teams to decide when they're ready to upgrade, based on their release timeline and what they're currently working on. This is only possible with a separate codebase for the infrastructure code (and a good versioning system which will be covered in the next post).
This isn't an uncommon model. Almost all open-source projects out there follow it. When you load jQuery on your site, you're pulling down a specific version (unless you're using the latest CDN file, in which case you need to re-evaluate your approach). Package manager's also help consumers follow this model (unless you're using the latest flag, which again, you probably shouldn't be).
The Benefits
For any project that has multiple teams depending on its code, switching to a pull model allows for a few key benefits. On large projects, these benefits far outweigh the drawbacks we'll cover in a bit.
By slowing down the release process, you're able release quicker. That may seem contadictory, but I assure you it's faster than ignoring the complexity of a large codebase and trying to rush code out the door.
Faster iteration due to smaller testing efforts
By not requiring the infrastructure team to regression test every app with every release (or coordinate every single change with every single team), they're able to release code quicker. Due to the shortened release cycles, each version change will be more bite-sized, allowing teams an easier upgrade.
This doesn't mean that the code isn't tested. The infrastructure team should have a series of regression tests that they run for any release. It also doesn't mean that app teams are on their own, or that the infrastructure team is free to make any change they want. Again, the teams should be there for each other.
By separating the code update from the code adoption, you can take feature validation in stages. You don't have to rush to get your feature built, tested and validated in one big swoop. This extension of time brings us to the next benefit:
Lower risk of breaking production with a release
Teams still need to regression test their code against the update, but with a pull model, they're able to do this at their own pace. This greatly reduces the risk of breaking production, as there isn't an artificial timeline to get the code tested.
This also allows you to inherintly test changes with small pilot groups. After a release, only one or two teams will be ready to immediately implement the update. This is your perfect chance to validate your changes in a real world scenario.
Instead of having to juggle information and feedback from every team at once, feedback trickles in and allows you time to react. You can even start doing pre-releases as a way to test code before saying it's production ready (this is more useful for major updates than small bug fixes).
If something breaks due to the release, each team has the option to delay the upgrade until it fits better, versus trying to cram in a quick-fix that will likely never be revisited.
A final hidden benefit of the "pull" model is the ability to rollback an update for an app if a bug is found in production. Because our infrastructure is versioned, we're not stuck using only the latest edition. They can be swapped out if a critical error is found, and again, this allows you time to find a perfect fix.
The Drawbacks
Slower Propogation of Features
Without a doubt, features go out slower with a pull model. It takes effort for a team to pull down code, compared to it automatically being pushed to them. This is the trade-off of the model and needs to be understood and accepted as part of the deal.
If the consumers of the app feel too much pain with this process, there will be mumbling and grumbling. Part of being empathetic is understanding this pain. You're requiring them to take extra effort so that you can continue moving forward. Have discussions with teams to see where the real pain points are.
Usually it's not about having to change a version number in a package management system. Perhaps their release process isn't automated well. Or maybe you're taking too much advantage of the "pull" method and introducing breaking changes with every release.
Either way, they're feeling a pain with every upgrade. It's easy to blame the pull model on this and therefore important to look deeper at the problem before the pull model gets replaced and all of this flexibility is lost.
UI/Visual Inconsistencies
This is the hardest pill to swallow, because it effects the app end-user. If your infrastructure code includes visual design, splintering versions means having an inconsistent experience across apps which are supposed to look similar. As each team upgrades their version separately, each app will have a slightly different look.
In the open-source world, this is okay, because users don't expect every Bootstrap website to be up-to-date with the latest version. In a company website or app made of several parts, the expectation is that a button on Page A will look similar to a button on Page B.
While I don't have a magic bullet answer, there are a couple of approaches to help mitigate this problem:
Plan Ahead by Bundling Visual Changes
If you know you have a series of visual updates that need consistency across the site, go through the effort to plan a combined release of these changes. Along with this, get in touch with the app teams and see if you can do a coordinated release for all of the apps. It does mean letting go of the benefits described above, but hopefully this isn't a common occurance.
Separate out the UI Code
In theory, you could have the UI code live in an entirely different repo with its own version. You could even follow a "push" release process for it, though I don't recommend it for the same reasons listed above. It also means you've got yet another codebase to maintain and app developers have another project they have to track against.
It's also difficult to do. In a website, it's tough to segregate functional code from visual code. Consider an accordian widget. There is a strong functional aspect to how the widget responds to interaction. But this functionality also depends on visual treatments being applied. It would be difficult to separate the two.
Take time to think through how you want to approach this situation and don't be afraid to adjust as you move along. Be sure to look at the whole picture, and don't get drawn in to an approach that seems like it solves everything. There are always trade-offs to every model, it's just a matter of finding the benefits and drawbacks that fit your project best.
The next post will cover a few more approaches that the open-source world takes that are applicable for internal projects. I'll get in to Semantic versioning, the Node Stability Index, and why you should keep a changelog.