Projects · 04 · Hughes Network Systems

Ceph Storage Self-Healing Monitor

In production at Hughes Network Systems. The code belongs to my employer and is not public, so this page keeps to what my resume says and the general ideas behind it.

From my resume

  • Sole author (77 of 79 commits, 6,586 lines, 61 tests).
  • Monitors 24 storage nodes across 8 clusters on a 3-minute cycle and averages ~130 verified metadata-server failovers per month with no human involvement.
  • Failover gated by a four-verdict standby-capacity assessment so it never moves into a worse node, outcomes verified against the cluster log to distinguish a real failover from a same-node reclaim.
  • 7-table append-only event history, 19-panel Grafana dashboard, pytest in CI.

Stack

  • Python
  • uv
  • Fabric/SSH
  • PostgreSQL
  • Grafana
  • pytest

The problem, in general terms

A Ceph file system keeps its file names and directories in the care of a metadata server. When that server struggles, every client waits, and the cluster can still look healthy enough from a distance. The fix is well known: hand the work to a standby. But someone has to notice, decide the standby is fit to take it, make the move and check that it worked, at any hour.

The ideas behind it

Each of these is one of the resume lines above, said plainly.

  • It looks on a fixed cycle, not when someone is paged. Every three minutes it reads the state of every cluster again.
  • It checks where the work is going before it moves it. A failover onto a standby with no room turns one problem into two. The monitor assesses the standby first and reaches one of four verdicts, and only some of them allow the move.
  • It does not take the command’s word for it. After a failover it reads the cluster’s own log to see what actually happened, because the same node taking the work straight back can look like a success and is not one.
  • It writes down what it does. Events are appended to a history that is never rewritten, with a dashboard beside it.

What I took from building it

  • Automation that acts on its own needs a way to say no, and “no” has to be an outcome it reports, not an error.
  • “It ran” and “it worked” are different facts and need different evidence.
  • The count of about 130 failovers a month is worth quoting because each one was verified, not assumed.

The second idea is one I can show. The simulation below uses made-up clusters and none of this system’s code, names or numbers.

There is no screenshot, architecture sketch, name or threshold here: those are my employer’s. The full resume · Experience

Simulation · made-up clusters, not a real system

A monitor that refuses to make things worse

Each cluster has two active nodes, which share its work, and one standby. Press a node to fail it. Nothing happens at first: the monitor only looks every 6 seconds. On its next check it finds the failure, asks whether the standby has room for that node's work, and moves it. If the standby has no room it refuses, because overloading a healthy node turns one failure into two. Try c1: the standby in its cluster is busy.

Monitor checks every 6 s

Cluster A

serving

Each active node's work needs 30% of a node.

Cluster B

serving

Each active node's work needs 35% of a node.

Cluster C

serving

Each active node's work needs 45% of a node.

Monitor log

  1. The monitor starts when this section is on screen.