SharePlane candidate

The Lost Discipline of Fault Isolation

Modern IT can assemble an incident call faster than it can narrow a problem.

By Tony Malott

I start work early most days, which means I often finish earlier in the afternoon.

“Finish” is probably not the right word.

When technology is both your profession and your hobby, the boundary between work and not-work becomes difficult to locate. I can leave the corporate environment, walk into my lab, and continue experimenting with systems, automation, infrastructure, security, or AI. The equipment changes. The questions often do not.

That has distorted some of my expectations.

Because technology has been part of nearly my entire life, I sometimes assume that people who work in IT must be as interested in it as I am. That is not fair. IT is an enormous profession containing engineers, developers, architects, analysts, project managers, service owners, database specialists, application teams, security practitioners, and dozens of other roles.

Not everyone needs a home lab. Not everyone wants to spend Saturday evening tracing an API call or testing an operating system deployment because that somehow qualifies as recreation.

Passion is optional.

For people serving in technical resolver roles, diagnostic discipline should not be.

Modern IT has confused specialization with permission to abandon foundational troubleshooting. You do not need to understand every technology stack. You should know how to define a symptom, reduce uncertainty, isolate a fault domain, eliminate possible causes, and escalate with evidence.

When that discipline is missing, organizations substitute attendance for analysis.

Specialization changed the bargain

Modern technology is too broad and too complex for any one person to understand everything.

I do not expect a workstation engineer to be a database administrator. I do not expect a network engineer to understand the internals of every application. I do not expect an application owner to know every operating system policy, identity flow, firewall rule, service dependency, and cloud platform behind a business transaction.

That would be ridiculous. I certainly do not know all of it.

What I expect is more foundational.

A technical practitioner should be able to understand the rough path through a system, identify likely fault domains, form a plausible hypothesis, run a bounded test, interpret the result, and reduce the number of remaining possibilities.

That is not mastery of every technology.

It is troubleshooting.

The problem is that IT organizations have increasingly divided accountability into narrow components and service boundaries. The endpoint team owns the device. The network team owns the route. The identity team owns authentication. The application team owns the application. The database team owns the database. One vendor may own a support queue while another owns the platform underneath it.

Each boundary may be rational by itself.

The complete system still has to work.

When nobody is responsible for reasoning across those boundaries, diagnosis becomes a search for the team willing to accept the ticket.

Twelve people and one unresolved question

I was recently invited to a meeting concerning what appeared to be a combination of workstation and network behavior.

There were roughly twenty people on the invitation. At least a dozen attended.

Some had legitimate ownership interests. Some represented technologies that might have been involved. Others appeared to be present because someone had forwarded the invitation, one of the enterprise’s more dependable scaling mechanisms.

The problem was not that twelve people joined.

Some incidents genuinely require several teams. Complex systems cross organizational boundaries, and consequential failures may need technical responders, service owners, security specialists, business representatives, communications leads, vendors, and decision-makers.

The problem was that the number of participants was growing faster than the body of diagnostic evidence.

Which users were affected?
Was the behavior reproducible?
Did it occur on one workstation or several?
Was it specific to one site, network segment, identity, device configuration, or application route?
Could the endpoint reach the next dependency?
Did name resolution work?
Did authentication complete?
Did the request leave the device?
Did it arrive at the application?
Which parts of the path had actually been proven healthy?

Those questions are not glamorous. They will not transform the enterprise. Nobody is likely to establish a steering committee in their honor.

They narrow the problem.

That is the job.

Troubleshooting is not intuition

People who troubleshoot well can appear unusually intuitive. They often identify likely causes quickly, know which evidence matters, and ignore attractive distractions.

What looks like intuition is usually compressed experience operating through a method.

Google’s Site Reliability Engineering guidance describes troubleshooting as both learnable and teachable. Its model is hypothesis-driven: observe the system, understand how it should behave, identify plausible causes, and test those causes using evidence or controlled changes. It recommends systematic reduction and bisection when diagnosing failures across layered systems.[1]

That is the formal explanation for something technicians have practiced for generations.

Split the problem.

Determine which side contains the fault.

Split it again.

Repeat until a vague complaint becomes a bounded technical condition.

If a user cannot reach an application, do not immediately debate every component in the architecture. Determine whether the problem follows the user, the device, the location, the network path, or the application. Test from another endpoint. Test another identity. Test the next dependency directly. Compare a known-good path with the failing one.

Each useful test should remove uncertainty.

Activity is not the same as diagnosis.

Restarting random components is activity.

Changing several settings at once is activity.

Forwarding screenshots without timestamps or context is activity.

Inviting another team because its technology appears somewhere on an architecture diagram is activity.

Troubleshooting produces evidence.

A basic fault-isolation discipline

The exact method varies by system, but the underlying questions remain stable.

1. Define the symptom

“The application is down” is not a useful technical description.

What operation was attempted? What result was expected? What happened instead? Who experienced it? From where? At what time? Can it be reproduced?

A vague symptom creates a wide fault domain. Precision immediately makes it smaller.

2. Establish the boundaries

Determine the blast radius.

One user or many?

One endpoint or an entire device class?

One location or every location?

One function or the complete service?

One identity type or all identities?

Boundaries often reveal more than the original error message.

3. Map the path

Even a high-level transaction path is enough to begin:

user -> endpoint -> local policy -> network -> identity -> application -> service dependency -> database

The actual path may be more complicated, but someone must understand it well enough to ask where expected behavior stops.

Architecture diagrams help, assuming they describe the system currently in production rather than a more optimistic historical civilization.

4. Test a meaningful boundary

Choose a point that separates groups of possible causes.

Can another endpoint on the same network complete the transaction?

Can the affected endpoint reach the service using a different identity?

Can the application tier reach its database?

Does the request appear in downstream logs?

A useful test rules out several possibilities. A weak test merely produces another observation nobody knows how to interpret.

5. Change one variable

When testing actively, change one condition at a time whenever practical.

Otherwise, even a successful result may not reveal what fixed the problem. It tells you only that somewhere inside a small pile of simultaneous changes, reality became more cooperative.

Controlled testing prevents the investigation from becoming another source of failure.

6. Record negative results

A failed hypothesis is progress.

If a test shows that name resolution works, record it. If the same user succeeds from another device, record it. If network traffic reaches the application tier, record it.

Google’s troubleshooting guidance recommends documenting ideas, tests, and results so responders do not repeat work or leave the system in an unknown configuration.[1]

Knowing what the problem is not can be as valuable as knowing what it might be.

7. Escalate a narrowed problem

There is nothing wrong with escalation. The mistake is treating escalation as a substitute for investigation.

A weak escalation says:

The user still cannot connect. Please investigate the network.

A useful escalation says:

The issue reproduces on three managed endpoints in one location. The same users succeed from another site. DNS resolution and local authentication complete successfully. Traffic leaves the affected subnet but does not appear at the application gateway. No endpoint configuration differences have been identified. Please investigate the network path between these boundaries.

The second escalation respects the receiving team’s time and gives it somewhere rational to begin.

It may also mean the next meeting needs four people instead of fourteen, a dangerous reduction in calendar utilization but a meaningful operational improvement.

Why organizations lose this capability

It is easy to blame individuals for weak troubleshooting. Sometimes an individual simply lacks the skill. That should be acknowledged and corrected.

The larger problem is organizational.

Fragmented ownership encourages people to defend components rather than diagnose systems. Each team demonstrates that its piece appears healthy, then transfers the remaining uncertainty to someone else.

This creates an operating model in which everyone owns a component and nobody owns the diagnosis.

NIST’s current cybersecurity incident-response guidance treats response as a capability that must be integrated throughout risk-management activity. Its purpose includes better preparation and more efficient and effective detection, response, and recovery.[2]

That guidance addresses cybersecurity incidents, not every workstation or application failure. The broader operating lesson still applies: reliable response cannot depend entirely on whoever happens to join a call and sound confident.

The method has to exist before the failure.

Troubleshooting is also cognitively demanding. A 2026 software-engineering preprint based on interviews with 27 professional developers describes troubleshooting as the construction and refinement of a mental model of unexpected system behavior. The researchers found that the work places sustained demands on attention, working memory, and mental modeling.[3]

The paper does not study enterprise incident bridges or prove that larger calls produce worse outcomes.

My inference is narrower.

When diagnosis depends on maintaining and refining a coherent model of the failure, adding participants helps only when they contribute relevant evidence, system knowledge, or disciplined coordination. Additional voices can also introduce competing assumptions, repeated explanations, unbounded theories, and pressure to act before the fault is understood.

More people can increase available expertise.

They do not automatically increase diagnostic coherence.

Restoration and root cause are different jobs

Fault isolation does not mean allowing a production system to continue failing while everyone pursues the intellectually satisfying explanation.

In a major incident, restoration may take priority over root-cause analysis. Traffic may need to be redirected. A failing component may need to be isolated. A recent change may need to be rolled back. Evidence may need to be preserved before the system is altered.

Google’s SRE guidance states this distinction plainly: stop the immediate damage first, while preserving what will be needed for later analysis.[1]

This is another form of diagnostic discipline.

The responder must know which question is currently being answered:

How do we reduce impact?
How do we restore service?
How do we preserve evidence?
How do we identify the proximate failure?
How do we determine the deeper cause?
How do we prevent recurrence?

Confusing those questions creates its own delays. So does pretending that restoring service means the investigation is finished.

The minimum standard

I am not arguing that everyone in IT must be an engineer.

I am not arguing that every engineer must be a generalist.

I am not arguing that people need to spend personal time building labs, reading technical documentation, or dismantling perfectly functional systems to understand why they work. Some of us apparently chose that life voluntarily. There is no reason to make it a licensing requirement.

I am arguing that technical resolver roles require a minimum diagnostic standard.

That standard includes the ability to:

describe a failure precisely;
understand the basic dependency path;
distinguish evidence from assumption;
formulate plausible hypotheses;
design tests that eliminate possibilities;
avoid changing several variables blindly;
document what has been learned;
distinguish immediate restoration from root-cause analysis;
escalate with a bounded fault domain and usable evidence.

Those capabilities should be taught, practiced, observed, and assessed.

Organizations routinely train people on tools, ticket queues, technology platforms, certifications, change processes, and procedural compliance. They are less consistent about teaching the reasoning method connecting all of them.

The tool will change.

The platform will change.

The architecture will become more distributed, abstracted, automated, outsourced, cloud-based, AI-assisted, or whatever term appears next in the PowerPoint ecosystem.

The need to reduce uncertainty will remain.

Attendance is not diagnosis

There will always be difficult incidents.

Some failures involve several interacting conditions. Some are intermittent. Some hide behind incomplete telemetry. Some require deep expertise from multiple teams. Some must be stabilized before anyone can safely investigate the cause.

No troubleshooting framework removes complexity.

It prevents complexity from becoming an excuse for undisciplined response.

A large incident call may mean the problem is broad, consequential, and genuinely difficult.

Sometimes it only means nobody has narrowed it yet.

The maturity of an IT organization should not be measured by how quickly it can assemble twenty people. It should be measured by how quickly those people can establish what is known, eliminate what is not involved, identify the failing boundary, restore service safely, and leave behind evidence that makes the next diagnosis faster.

The incident bridge is not the troubleshooting method.

The troubleshooting method is what should make most of the bridge unnecessary.

Sources

[1] Chris Jones, “Effective Troubleshooting,” Site Reliability Engineering, Google. Source
[2] National Institute of Standards and Technology, SP 800-61 Revision 3: Incident Response Recommendations and Considerations for Cybersecurity Risk Management, April 2025. Source
[3] Arty Starr and Margaret-Anne Storey, “Theory of Troubleshooting: The Developer’s Cognitive Experience of Overcoming Confusion,” arXiv preprint 2602.10540, February 2026. Source

Capstone teaching visual: Diagnostic Essay

Mainline white prompt

Create a clean, high-density editorial systems infographic on a pure white field. Title: “The Lost Discipline of Fault Isolation.” Subtitle: “Attendance is not diagnosis.” Show a broad field of unresolved technical possibilities progressively narrowed through seven explicit stages: Define the symptom; Establish the boundaries; Map the path; Test a meaningful boundary; Change one variable; Record negative results; Escalate a narrowed problem. Use black text, deep red emphasis, neutral gray structure, and a precise left-to-right or top-to-bottom fault-domain reduction. Include the distinction “Passion is optional. Diagnostic discipline is not.” Show component ownership lanes for endpoint, network, identity, application, service dependency, and database, then expose the missing cross-system role: diagnostic ownership. End with: “The incident bridge is not the troubleshooting method. The troubleshooting method is what should make most of the bridge unnecessary.” No dashboards, no invented metrics, no people caricatures, no employer branding, no holograms, no glowing AI imagery, no tiny unreadable text. Static, dependency-free visual language, suitable for a SharePlane teaching artifact. Candidate-specific composition: Use a restrained editorial scroll with margin annotations, a large typographic thesis, and one clean progressive narrowing diagram. Preserve a generous reading column and treat the seven-step method as a full-width instructional interruption.

Dark expressive prompt

Create a cinematic but controlled technical field-guide infographic on a dark graphite background. Title: “The Lost Discipline of Fault Isolation.” Subtitle: “Attendance is not diagnosis.” Use off-white text, intentional deep red fault boundaries, muted amber evidence markers, and subtle schematic linework. Show an initially noisy incident field progressively narrowed by seven explicit diagnostic stages: Define the symptom; Establish the boundaries; Map the path; Test a meaningful boundary; Change one variable; Record negative results; Escalate a narrowed problem. Include the doctrine “Passion is optional. Diagnostic discipline is not.” Represent endpoint, network, identity, application, service dependency, and database as separate ownership lanes, then show the cross-cutting requirement for diagnostic ownership. Conclude visibly: “The incident bridge is not the troubleshooting method. The troubleshooting method is what should make most of the bridge unnecessary.” No neon clutter, no fake terminal screenshots, no humanoid AI, no robots, no fabricated incident metrics, no employer branding, no illegible microtext. Static editorial systems graphic, dark mode designed intentionally rather than inverted. Candidate-specific composition: Use a restrained editorial scroll with margin annotations, a large typographic thesis, and one clean progressive narrowing diagram. Preserve a generous reading column and treat the seven-step method as a full-width instructional interruption.