I made my network monitor a single point of failure - then fixed it

A while back I wrote up the home network monitoring stack I look at every day - Pi-hole and ntopng on a Raspberry Pi 5, with the Pi wired inline as a transparent Layer 2 bridge between the ISP modem and the Eero so ntopng could see WAN traffic on a mesh that offers no mirror port. It worked. It was also wrong in two ways I didn’t fully appreciate until people poked at it, and this is the rebuild.

The two things I got wrong

The Pi was a single point of failure for the entire house’s internet. It sat in the forwarding path, so if it died, everyone’s connection died with it. I had a story for handling that - a GPIO relay that would bypass the Pi on power loss - except the story didn’t survive contact with reality. A single-channel relay switches one conductor; gigabit ethernet needs all four pairs. The relay I’d speced couldn’t have done the job, and I never finished wiring it anyway. So the real failover was “go recable it by hand,” which is not a failover.

My health check watched the path, not the job. It confirmed the bridge was forwarding and that I could ping out. It never confirmed that ntopng was actually ingesting flows. The failure mode that should scare you - box healthy, NICs up, but the monitoring daemon silently wedged on a stalled process, a full disk, or a pipe broken by a log rotation - would have sailed straight past it. Nothing technically breaks, so nothing alerts, and you find out three weeks later when you go to read data that quietly stopped recording.

Fix #1: stop being inline

The cleanest solution to “the Pi is a single point of failure” turned out not to be a cleverer bypass. It was to take the Pi out of the path entirely.

A cheap managed switch - a TP-Link TL-SG105E, about $23 - goes inline between the modem and the Eero. It does the forwarding, which is the one job it has and is far better at than a Pi: no SD card, no OS, no services to wedge. Then I turn on port mirroring, so every packet crossing the modem-facing port is copied to a third port. The Pi hangs off that mirror port and ingests passively. It is no longer in the forwarding path at all.

The proof is the part I like. I did the recable with the Pi powered off, and the house stayed online. A Pi failure - or an ntopng crash, or me fat-fingering a config - can’t take the internet down anymore, because the box that forwards packets and the box that watches them are now different boxes. Same WAN visibility as before; no single point of failure.

(I confirmed this the hard way a few days later, when I had to pull the Pi’s power for an unrelated reason and the internet never noticed. The whole point, made involuntarily.)

One caveat carried over from the original setup, and it’s worth being honest about: the tap sits WAN-side of the Eero’s NAT, so ntopng sees aggregate flows and top destinations, not true per-device attribution - every device wears the Eero’s single WAN IP. Per-device data comes from a separate poller against the Eero’s API. The switch didn’t change that. It just removed the risk.

And the irony I’ll own: my original post advised “if you can’t mirror, bridge.” That was backwards. A consumer mesh won’t give you a mirror port, but a $23 switch in front of it will - and that’s the better answer. I had the right instinct and talked myself out of it.

Fix #2: watch the job, not the box

Removing the single point of failure unlocked the second fix. Once a stalled Pi can’t take the house offline, I can be aggressive about ingestion health without worrying that a false alarm pages me at 3am over nothing.

The check I landed on - shaped, honestly, by a sharp commenter on the original thread - is a closed loop:

• the Pi pings a fixed canary IP every minute, a known heartbeat;
• a packet capture on the tap confirms that heartbeat physically crossed the wire;
• then it asks ntopng, over its REST API, whether it actually counted that traffic.

The alert fires on the contradiction: the heartbeat is provably on the wire, but ntopng isn’t counting it. That’s “the daemon stopped doing its one job,” with live traffic to prove it - not a quiet network. And because the heartbeat guarantees there is always traffic to compare against, it catches the nastiest case: a genuinely idle network where ntopng also happens to wedge, which a naive “any flows lately?” check would read as perfectly normal. Three strikes and it restarts ntopng and pings me. A disk-space guard on the flow store covers the other common silent death.

I tested it the satisfying way - stopped ntopng and watched the probe notice the contradiction and restart it on its own. The first time your monitoring fixes itself in front of you is a good feeling.

The lesson

Here’s what I’d take from it. The failover isn’t a relay you’re hoping works. It’s that the Pi was never load-bearing in the first place. The reliability came from changing the topology so the fragile component can’t matter - not from bolting on a clever mechanism to rescue it after it fails.

The monitoring follows the same shape. Watching “is the box up” is easy and nearly useless. Watching “is it doing its one job” is the check that catches the failure you’d otherwise discover weeks too late.

None of this is novel - a switch with a SPAN port is how real networks have done it forever. The only interesting part was admitting the first version put the cleverness in the wrong place.