On systems development
You need a system to go fast, you need chaos to do something new. Chose well
Intro
Making something new is hard. If you ever tried to do any project from the ground up, you know for sure that the amount of decisions to make even for a small project can be really daunting.
Let's say you want to send an email. Here are just some questions:
What is the trigger for the email? How do you deploy it?
How do you know what has been sent in a particular situation?
How do you render the content?
How do you monitor the delivery status?
Which template would you use?
Would you do translations?
And a lot more of course
Now, imagine you're working on the same email in the situation where the setup is done and you just need to clone and adjust the template. It's still a new email, but the task went from being a huge one to trivial. What happens there is that given a framework with all the hard questions already answered, any additional emails come almost for free.
I often call the tasks of the first kind "zero to one" tasks. Decisions require a lot of energy, and that's the primary reason for updates on existing systems to be so much easier than any sort of greenfield development.
There is also a conclusion to make here: if you want to develop a product faster, do whatever it is in your power to get past the first phase right into the second one1.
This pattern can be defined like this:
In case a brand-new type of problem is encountered, you solve it in a way that makes the next case of the same kind trivial or much easier to accomplish.
As it happens with many things in this universe, the pattern is fractal and can be used to have an easy and standard way to show modal windows or to send satellites to the orbit. If you introduce many bits like this and make them play well together, you have a system.
System
Here's a definition I like (the last part is mine):
A regularly interacting or interdependent group of items forming a unified whole, each of them solving a particular class of problems.
The more problems are solved, the easier it is to implement a new feature given you stick to the existing solutions.
The good
Once a solution is found for a class of problems, the number of decisions to make for another feature of the same class drops drastically and as a consequence the difficulty of implementing another feature of the same class plummets as well. That's what you usually want because the speedup can be in the orders of magnitude.
Not only that, the application of existing solutions requires much less skill and effort and as a consequence less experienced developers can be employed for the task.
I have an approach that I’ve successfully used with many web applications: once a set of basic interactions and patterns is defined2, every new page becomes a trivial exercise of throwing these interactions on the page and wiring them up. You would be really surprised to see how far you can get with just a handful of building blocks. The important bit is to keep searching for repetitive patterns and turn them into reusable interactions.
You could also expect an explosion of usage of a given pattern right after it was introduced with a systematic approach. If sending an email is a pain, it's reasonable to have very few emails implemented. Not that nobody wants them of course, it's just too darn painful. Once it becomes easy, you can expect a flood of requests for all the emails that would have been implemented before.
Another conclusion to make is that if a unique type of feature happens a couple of times, it's a good time to make a guess whether it's just a one-off or you can expect more features of this kind.
The bad
While doing everything from scratch every single time is impossible, doing everything in a neat and tidy manner is not an option either, because implementing a solution for a class of problems usually takes more time than a one-off development. To make things worse it increases the cognitive load required to understand the whole system and maintenance costs to keep it running. When I said that you should make a guess I meant it, sometimes you get it right, sometimes you don't, and often you need to do fixes in both directions.
It doesn't help to have everything abstracted away and generic, since all the time will be wasted to understand the interconnections between different parts. Ideally you shouldn't have to jump through lot’s abstraction layers just to understand what's going on.
Another consideration - when a new subsystem or anything actually is built it's optimized to solve a particular set of scenarios, to make them easier to do. However, that also implies that some other scenarios are not considered, which makes them harder or impossible to implement.
Systems work well, however, sometimes they work too well. If you think about it, systems kill innovations in a particular area. Since the issue is solved™, it's hard to make a case to break a well-oiled system in favor of a risky new approach. However, this might leave you in the local maximum forever.
If your system is well split into subsystems you could argue that you can replace some of the parts with brand-new stuff and indeed that's a better scenario. However, in this way you cannot address the split itself! All the demarcation lines are especially hard to change, and the risk there is that you'll find yourself implementing an innovative solution for something that may not even need to exist in case the system was sliced differently.
It’s important to be aware that whenever you step outside of the systematic solution, you have a much higher chance of getting in trouble. Everything is more complex than it seems and the neat custom navigation that looked so cool on the mockups can take 90% of the implementation time and a lot of repercussions down the road. Next time you get a particularly creative design or product requirements for a new feature - try giving two estimates - one for an approach using the existing tools, systems and patterns and the other one for a new snowflake kind of development. With a big enough difference, chances are that magic will happen and some of the requirements will evaporate in front of your eyes.
The ugly
Everything is good in moderation and the same is true for building systems as well. Building them is so fulfilling that sometimes it's hard to stop. Overengineering has killed more than one good project, the usual smell is when your smart developers tell you how their super complex and flexible solution cannot solve a seemingly simple task.
There is also a mirror side of simplicity provided by existing systems - they start influencing product development since it's so much easier to follow them instead of doing whatever fits the customer's needs and there is a thin line there between unreasonable requirements from stakeholders and unreasonable limitations of existing solution.
Conway's law is real too and one ugly side-effect is that any organizational changes will inevitably result in some orphan systems that will never be anyone's priority. If you ever heard horror stories about mission-critical abandonware within a company, that was that.
So what?
First of all, it’s important to be aware of the systems in the first place. Knowledge of their capabilities, limitations and accepted practices will help a lot on all levels. If someone works on a web or mobile app, it doesn’t hurt to know how the platform was designed to work in the first place.
Second, it’s important to know the difference between the zero to one and iteration tasks, that’s where you get high velocity for your team. You also want to know who's capable of doing tasks of the first kind to make sure they can handle the analysis paralysis. If you're a developer you might want to practice this skill to make sure you can make yourself cross the chasm in a reasonable time.
Innovation flourishes in chaos, great inventions often pop up from the ashes of another war, and you probably want to allow some chaos to innovate. One way to do it is to have an isolation layer between different parts of the system3.
It's always possible to have the data as the interface - if a new service generates the same data as the old one, you could have both and a contender will die out or eat the legacy solution eventually. Of course, this approach can break down because of the organizational or power structure. If someone's career or job security depends on it, system evolution will be correspondingly distorted.
As for unique product solutions, your best bet would also be to isolate them as much as possible. Make sure the feature is confined to the smallest scope that makes sense - it will make it easier to clean up the mess in case it is not successful.
Outro
I think one bit I didn't mention is the target audience of the post. Many developers and designers are aware of this difference at least on the lowest level (DRY principle for example), however empirically very few people approach new tasks by asking - does it look like a standard building block? Can I reuse any existing ones?
Two important groups would be dev leads and product owners. It's the job of every dev lead to keep an eye on the overall system and make sure that it's cohesive and built up in a way to accelerate the development of new features rather than to hinder it and also pair with stakeholders to find the best solutions.
For product owners, it's equally important to be aware that their requests are not implemented in a vacuum and depending on the initial requirements a project can be a flop or a great success development-wise. Whenever you see your developer cringe every time you make some innocent request (e.g. let's send a push notification once the order is done) that's a sure sign that there is no systematic solution in place or it is rotten. A good question to ask here is not about this particular notification, you could get much more value by starting a discussion on how to make the notifications the simplest thing ever in your project. In the first case, you may get your notification once, in the second you will get a capability that may be transformational to your product.
Or, it's possible that what you ask for breaks current patterns for no big benefit and that's a really good point for discussion as well.
Big thanks to Borislav, Artem and Oleg for the feedback!
This post on Hacker News
One of the key features of the Google Chrome browser from the very beginning was its upgrade tech. What they started with was an app with an empty window and automatic updates that would push new versions to all users reliably, and this feature single-handedly teleported them into iteration mode.
Forms, ajax handling, modals, navigation, layout library, router etc
In the case of web app - if all pages are independent from each other, any new page can be used as a playground for experimentation within that scope.