Every engineering team has knowledge that is not written down.
Some of it is harmless. A shortcut in a development environment or a preference for one debugging command does not need a formal process. The problem begins when important systems depend on information that lives only in somebody’s memory.
Which service must restart first? Why is that firewall rule present? Who knows how the certificate is renewed? What should happen when the deployment is partially successful? Which alert can be ignored, and which one means customer data is at risk?
When the answer is “ask that person,” the team has created a hidden dependency.
Tribal knowledge often looks efficient because writing things down takes time. In reality, it moves the cost into interruptions, slow onboarding, inconsistent decisions, fragile incidents, and fear around change.
The Human Single Point of Failure
Engineers are careful about single points of failure in infrastructure. We add replicas, backups, health checks, and failover plans. Then we allow one person to become the only reliable source of information about how the system works.
That person becomes the human single point of failure.
They are invited to every incident. They review every risky change. They answer the same questions in private messages. They cannot take a real holiday because the team may need context that nobody else has.
This is bad for the organization and unfair to the engineer. Being indispensable can feel valuable for a while, but it usually becomes a trap. The expert spends more time protecting old knowledge than creating new improvements.
A resilient team should be able to operate when any one person is unavailable.
Interruptions Are the First Bill
The most visible cost of tribal knowledge is interruption.
Someone cannot find a deployment step, so they ask the person who did it last time. A monitoring alert has no explanation, so the on-call engineer calls the service owner. A new colleague needs access, so three people search old chat messages to remember the correct approval path.
Each interruption looks small. The hidden cost is the context switch on both sides.
The person asking cannot continue. The person answering must stop their current work, reconstruct the context, and explain the answer. If the explanation stays in chat, the same cost appears again next week.
Teams often try to solve this by working harder. The better answer is to capture knowledge at the moment it is used.
If a question appears twice, it probably belongs in documentation, a runbook, an automated check, or the user interface of the platform itself.
Incidents Expose What the Team Does Not Know
Tribal knowledge becomes dangerous during incidents because time and attention are limited.
When a system is failing, people need clear signals and tested actions. They should not be discovering basic architecture from old tickets or waiting for one expert to wake up.
A useful runbook does not need to predict every failure. It should explain what the service does, where its important dependencies live, how to check health, what recent changes matter, which actions are safe, and how to escalate.
It should also state what not to do.
That last part is important. Experienced engineers often carry warnings in their heads: do not restart both nodes together, do not rotate this key before updating that consumer, do not restore this database without stopping writes. These constraints are exactly the knowledge that disappears when the expert is unavailable.
The best time to improve the runbook is after using it. If an incident required an undocumented command or decision, capture it during the postmortem.
Documentation Is Not the Only Answer
Not every knowledge problem should become another page in a wiki.
Documentation is useful for context, decisions, and procedures. But repeated operational knowledge is often better expressed through automation and system design.
If deployments must happen in a specific order, encode the dependency in the pipeline. If a required field is frequently missing, validate it before the workflow starts. If access must expire, automate expiry. If a dangerous action should never happen, restrict the permission instead of relying on a warning paragraph.
Good tools reduce how much people need to remember.
This is one reason infrastructure as code, CI/CD, GitOps, policy as code, and automated tests matter. They turn knowledge into repeatable behavior. A pull request can show why a configuration changed. A test can preserve an expected outcome. A reconciler can detect drift. A policy can enforce a rule every time.
Documentation explains the system. Automation helps the system remember.
Decisions Need History
One of the most expensive forms of tribal knowledge is the reason behind a decision.
Teams inherit unusual architectures, exceptions, and workarounds. Without context, the next engineer assumes the design is accidental and removes it. Sometimes that is the correct decision. Sometimes they rediscover the original failure in production.
Important decisions should leave a short record: what problem existed, which options were considered, why this option was chosen, what tradeoffs were accepted, and what conditions would justify revisiting it.
This does not need to become bureaucracy. A short architecture decision record stored near the code is often enough.
The value is not proving that the old decision was perfect. The value is giving the next person enough context to make a better decision.
Sharing Knowledge Is Part of Done
Teams usually define done around code: implemented, reviewed, tested, and deployed.
For operational changes, that definition is incomplete.
If a new service is deployed but nobody except the author can troubleshoot it, the work is not done. If an alert is created without a response path, the work is not done. If a migration changes recovery steps but the runbook remains old, the work is not done.
Knowledge transfer should be part of delivery:
- Update the runbook when operational behavior changes.
- Record important design decisions.
- Add dashboards and alerts that explain useful state.
- Automate repeatable steps.
- Pair on risky or unfamiliar work.
- Make more than one person capable of operating the system.
These activities are not separate from engineering. They are how engineering survives contact with time, growth, and staff changes.
Final Thought
Tribal knowledge feels cheap because the team does not receive an invoice for it.
The cost appears somewhere else: in repeated questions, delayed incidents, nervous deployments, slow onboarding, overloaded experts, and systems nobody wants to touch.
The goal is not to document every thought. The goal is to remove dangerous dependencies on memory.
Write down the context that people need. Encode repeatable rules in automation. Test recovery paths. Make knowledge visible while the people who understand it are still available.
A resilient system needs redundant infrastructure. A resilient team needs redundant understanding.