Why internal documentation is lacking
I’ve noticed that at many (most?) companies, documentation on internal tools, APIs, and codebases tends to be lacking. Often the only “documentation” you have to follow are examples of how others are using it. Why is this? Is it because people are lazy, or inherently incapable of writing good documentation? Neither of those two superficial explanations seems to make sense.
In particular, it’s interesting to contrast the vast amount of publically available good documentation, tutorials, and guides on even complex subjects like GPTs. Why does this not exist for internal technologies?
Here is my pet-theory (hypothesis, but we’re using terms colloquially here) about why this condition exists, summarized in these points:
- Good documentation takes a lot of effort.
- The incentives are rarely there.
- Good documentation might not matter.
Let’s look at each point in detail.
Good documentation takes a lot of effort
Writing good documentation requires several factors which I’ll term as the documentation funnel. At any point, if one of the factors is missing, the criteria for being “good documentation” is lost.
The funnel in detail:
- Does the documentation exist?
- The most basic criteria. A certain percentage of things you want to find documentation for internally simply won’t exist.
- Is the documentation discoverable?
- Internally, companies usually rely on multiple ways to record down knowledge: Wikis, tickets, emails, internal documents of various formats, chat (Slack/Discord/Teams) etc.
- This creates a fragmented environment upon which documentation is scattered into islands.
- There rarely is a universally searchable index of all internal documentation.
- Even if there is a way to search, such internal search is usually pretty bad. Discoverability is a big issue, because even if you write the best documentation known to humanity, if no one can find it, does it really exist?
- Is it written well?
- Well-written documentation understands who the audience is, what their level of knowledge is, and what their goals are, and then gives them the information as concisely as possible, in the best possible order. (I’m a big fan of this explanation from Divio on structuring good documentation)
- By contrast, poorly-written documentation can fail in many ways: It may bombard the reader with all sorts of unimportant details, it may be vague or leave out information, or it may be inaccurate, or it may just be hard to read because it’s poorly-formatted.
- This is probably the step which takes the most effort and consequently is the one most often missed.
- Is it up to date?
- Some documentation may have started out very good, but over the time the code/API/tool it was documenting changed, but the documentation hasn’t been kept up to date.
- This turns the documentation into inaccurate documentation.
- Poorly-written or inaccurate/out of date documentation can actually waste readers’ time, as they go down the wrong path and then have to reverse once they find out about the mistakes they’ve made because of the bad documentation.
- Is there someone who can explain things to you?
- This is not part of the funnel per se, but rather a trump card: If there’s a subject matter expert (SME) who’s willing to engage with you, this basically solves the problem of not having good documentation.
- However, such engagement is usually very time-consuming for the SME, and doesn’t scale well to multiple users/clients.
Getting through the entire funnel takes a significant and continuing effort, and companies have limited resources. Without the appropriate incentives, this simply won’t get done.
By contrast, popular and publically available technologies (e.g. open source tools/frameworks) will have large communities of users that are incentivized to help each other through writing tutorials, improving documentation, and keeping that information up to date.
Which brings me to my next point: Incentives.
The incentives are rarely there
Consider the case of an internal tool/technology which lacks good documentation. We can consider the perspective of at least two groups of individuals who are in a position to create documentation for it:
- The team which owns the tool/technology (if such a team still exists!)
- The individuals who have to use the tool/technology.
For the team which owns the tool/technology, you could argue it is their responsibility to have good documentation for it. But how would “good” documentation be measured? The most obvious impact of missing or bad documentation is that it takes a user longer to do something. But the impact from the bad documentation can often be hard to measure. What cannot be measured, cannot be improved.
For example, the length of time it takes someone to get onboarded to a new technology is at least a function of:
- Their overall experience level.
- The exact task they’re trying to do.
- The quality of the documentation.
If factors (1, 2) contribute to the majority of the variance in time taken, then the impact of good documentation can be missed, especially if the number of new users of the technology is only a few per quarter. (This is basically a low-sample size test where the measured variable has high variance)
As a first step, one way to measure documentation quality might be to track how many questions a team gets for something they own, with the idea that questions reflect a lack of documentation. But this is an imperfect measure on its own, and is subject to many of the weaknesses outlined previously and also a few more.
For the individuals who have to use the tool/technology, some may eventually struggle through and figure out how to do their task. At this point, they have accumulated the knowledge that before was not properly documented, but they also have no incentive to improve someone else’s documentation.
This is in contrast to publically-available tools/technologies, where there are incentives for others to produce good documentation on how to accomplish a given task. This good documentation takes many forms such as detailed blog posts, guides, YouTube tutorials, etc. These benefit the producer of the documentation by creating pageviews/subscribers which can be monetized or used to build their personal brand.
Some examples are the extensive community around MidJourney AI’s text-to-image models, which smartly leveraged Discord to quickly build that community (you access the model via a Discord bot) which has given rise to detailed prompt engineering guides.
These incentives don’t exist internally at a company, and even if they did, there usually isn’t a large enough community of users within a company which can support such efforts.
It may not matter
If you’re operating a company where you are selling access to an API (like a payment API) you are incentivized to have good documentation because poor documentation will cause you to lose signups. If potential customers can’t find any documentation for your API, or there isn’t a simple tutorial explaining how to onboard, you are going to lose that sale. In this case, documentation matters because it directly impacts the perceived quality of your product.
By contrast this matters much less for an internal API. There aren’t a dozen different internal payment APIs at your company for you to choose from, and consequently no way to measure which one is “best” by the adoption rates across internal users. Instead, there is probably just one, and if you want to accept payments, you must use it. So you struggle through it, and eventually get things to work.
In other words, there are ways to workaround poor documentation that hide its effects: You find ways to deal with it, (e.g. finding someone who can explain things to you), as you struggle to eventually complete your task. Yes, you might have taken slower than you would have had there been proper documentation, but the work was not brought to a grinding halt nor did the company cease to exist because of a lack of documentation.
In that sense, it did not matter.
I’ve presented my observations and hypotheses on why internal documentation is often lacking. These are obviously biased toward my own personal experiences, but I believe such experiences aren’t uncommon.
In cases where I have seen good internal documentation exist, it was usually due to one of these reasons:
- A leader specifically asked for it, thus creating an explicit incentive. This can be a very powerful incentive, as it also sets the culture of the organization.
- There was an individual who took it upon themselves to write the good documentation, without expectation of being rewarded, simply because they felt it was the right thing to do.
My personal opinion is that a culture of good documentation can only come from leaders who explicitly convey the importance of good documentation and by setting a good example ourselves. Because it’s so hard to measure the impact, we can only justify the effort in good documentation by engraining it into a culture of being good neighbors to one another.
When we were taught as kids to always say “Please” and “Thank you", no one asked “but how can we measure the impact of such remarks?” I believe a similar situation exists in writing good documentation.