The rule of three for generality

MP #60: In pursuit of correct abstractions.

Note: I’m currently in North Carolina at DjangoCon, so I haven’t had time to finish this week’s OOP post. I’m going to pause that series for a week. Instead I’ll share a takeaway from a project I’ve been working on for a couple years now, that’s finally getting a lot closer to a 1.0 release. There’s no code in this post; even if you don’t use Django, the main point should still be relevant to most people.

---

Deployment is more difficult than it needs to be

I’ve been working publicly on a project called django-simple-deploy off and on for about two years now. In case you’re not a web developer, “deployment” is a word that makes a lot of people in the web world groan. It’s the step where you take a project that works beautifully on your own system, and deploy it to a server. When you take this step, you cross your fingers and hope it will run just as well on the remote server as it does on your local system.

Deployment to a new platform is almost always a somewhat involved process, even for people who are knowledgeable about the process in general. You have to look at the new platform’s documentation, configure your project to work on that infrastructure, push your project, and hope you followed the configuration steps correctly. Even experienced people struggle here, because most platforms’ documentation tends to be slightly out of date in various ways. Or maybe the way they wrote the documentation doesn’t clearly map to how your project is structured. Often times you only find this out through a series of failed deployments.

With django-simple-deploy, most of that error-prone configuration work is automated. Assuming you have an account on the host’s platform and their CLI installed, deployment can be reduced to a three-step process:

$ pip install django-simple-deploy
# Add simple_deploy to INSTALLED_APPS
$ python manage.py simple_deploy --platform <platform-name> --automate-all

You install django-simple-deploy, add it to your list of apps, and run one command. The tool inspects your project, makes appropriate configuration changes, and makes the necessary CLI calls to push your project to the specified platform. Your project then appears in a new browser window, running on the platform’s servers.

It’s a platform-agnostic tool; you get to choose which platform to push to. I gave a talk about this project last year at DjangoCon, and made a last-minute decision on stage to push to a different platform than I had planned. The platform I intended to use had been flaky earlier in the day, and I didn’t want to risk a failed deployment during a live demo. That flexibility in deployment was really satisfying.

The home page of simple_deploy’s documentation.

The (planned) road to 1.0

I had a relatively clear plan to bringing this project to a stable 1.0 release:

• Support deployment to Heroku, because that’s the platform I’d been using the longest.

• Add support for Azure. I wanted to learn Azure deployment, and building support for a new platform boils down to writing a recipe for deployment to that platform. Building support for a second platform also helps distinguish all the platform-agnostic code from the code that’s specific to Heroku, or specific to Azure.

• Build support for one more platform, to prove that the abstractions I’d already identified did in fact generalize to a third platform, and likely any number of additional platforms.

This plan almost worked. I ended up dropping support for Azure and replacing it with Platform.sh. Deploying to a more general platform like Azure involves many more decisions than deploying to a more targeted host such as Platform.sh. I saw that I could build support for Azure at some point, but it wasn’t the most appropriate platform to target when sorting out pre-1.0 implementation issues.

By the 0.5.0 release, I had implemented preliminary support for three platforms: Heroku, Platform.sh, and Fly.io. If I could clean up all this work, a 1.0 release should be somewhat close. In case it’s unclear, version 1.0 in this project implies that anyone with a simple (but not necessarily trivial) project can reliably expect their deployment to work using this tool.¹

The roadblock

While preliminary support for three platforms felt really good, it turns out I had hit a significant roadblock in that third platform. I needed to show that simple_deploy could consistently produce successful, safe deployments for everyone. This roadblock only showed up as a result of that decision to support three platforms before declaring it stable and ready for widespread use.

When you deploy a typical web app, you need two main resources on the remote server: an app, and a database. Heroku and Platform.sh create both of these resources when you run a single command, such as heroku create or platform create. With hosts like this, the configuration work that follows is straightforward because the required resources have already been linked.

When you deploy to Fly.io, however, the app and database need to be created in two separate steps. After that, they need to be linked. If you’re doing this work manually, this process works because you know the names and locations of both resources. But if you’re trying to automate these steps, and the user has more than one database, it isn’t always clear which database should be used. An automated deployment tool is a dangerous, complete failure if it does anything to interfere with a pre-existing resource, especially one as critical as a database.

This was a blocking issue, and a critical one to solve before claiming a stable 1.0 status. It’s an issue that wouldn’t have arisen if I had just built support for two platforms and then considered the project stable. It also turned out to affect more than just deployments to Fly.io. This past summer, Heroku moved to the same model—they now require you to explicitly create a database if you need one, rather than creating one automatically every time a new app is created.²

Preliminary support for Fly.io in the 0.5.x series of releases worked only if you had no pre-existing projects on that platform. A 1.0 release needs to be able to support an arbitrary number of deployments on any given platform.

The solution

Heading to DjangoCon brought me back to a focus on this project, and some time away from the issue helped me see a pretty clear solution. When a user is logged into a platform’s account, you can get a list of the resources that have been provisioned. I’m planning to identify the most recently created database, and have the user confirm that this is the correct database to configure against. This is a much cleaner solution than having the user include the name of the database as an argument in the call to simple_deploy. I’ll probably show the other databases that were found, or maybe show the last few databases that were created, to emphasize the need to identify the correct resource.

I’m staying here for the sprints, and I’m hoping to implement this solution for both Fly.io and Heroku.³ If this approach works, all the remaining pre-1.0 issues should be relatively straightforward (and satisfying) to address. This is joyful work!

Conclusions

The core idea here of addressing three use cases before trusting that an abstraction or implementation is correct usually comes up in much smaller contexts. Here’s a typical example:

• You have a function with some code that another function needs.

• You pull the common code out into a helper or utility function, and generalize the code a little so it supports both of the calling functions’ needs.

If your new utility function is only called from two other areas in your codebase, it’s hard to fully trust that you’ve hit on the correct abstractions and set of parameters for a truly general helper function. It’s quite possible that a third call to the function will require an additional parameter, or another conditional block. Once your utility function serves three other parts of your codebase, however, there’s a fair chance that you’ve figured out an appropriate abstraction.

This rule of three that serves us well at the function level can work just as well at the project level. You can see this with users, for example. If you build a project that works for you, you’ve met the needs of one user. You’ll probably have to make some significant changes if you want your project to work just as well for someone else. If you can support a third person (who’s exercising the full capabilities of your project), then you’ve probably implemented an appropriate model for addressing your users’ needs.

There’ll always be more edge cases, and of course you’ll learn things at every new order of magnitude of users, but that simple rule of three carries a lot of weight. Keep this in mind when you’re working out the implementation of a code base that needs to support a variety of use cases.

1. Writing this article was easier because of the choice to include a changelog from the earliest release. It was rather satisfying to look back through the earliest releases of this project and see exactly how support for multiple platforms evolved in the project. For example, it was easy to scroll through and see that 0.2.0 was the first release that supported deployment to Azure.

If you haven’t created a changelog before, an excellent resource is Keep a Changelog.

2. Heroku has made some bad moves in recent years, but the decision to stop auto-provisioning Postgres databases seems perfectly reasonable to me. It’s not a good thing to create a database that never ends up being used. And once you have created a bunch of databases, you have to be really thoughtful about deleting them without explicit user permission.

3. If you haven’t heard of sprints before, they’re typically a few days after the end of a conference’s main activities, where you can work on open projects with other contributors. It’s a fantastic opportunity to work in person with other open source maintainers.

If you’re going to a Python conference and have a day to spend at the sprints, I highly encourage you to do so, even if you’re not planning to make a contribution. Just seeing how all this works in person is really interesting and motivating.