Nov 2021 – Present
Systems Engineer III (Software Engineering)
Hughes Network Systems · Germantown, MD
- Second of 21 contributors over the last year on the satellite gateway automation platform: 491 commits across 67 delivered tickets spanning monitoring, alerting, self-healing, and reporting for five ground-segment platforms
- Root-caused and eliminated a platform-wide logging defect that silently discarded log records, cutting the log footprint from 19.68 GB to 58 MB and reclaiming ~25 GB in production; swept all 146 scheduled jobs for the same pattern
- Root-caused two production failures: a 738 MB core dump traced with gdb to an unguarded deserialization path in a vendor gateway component, and a 177-process leak traced to a library deadlock at interpreter teardown; delivered both as actionable defect reports and shipped the supervision module that closed the leak
- Designed the platform's shared logging entry point, now used by every script and device module; introduced continuous integration to the organization; migrated the action-audit data model with 26,857 rows reconciled exactly
- Authored the platform's Kubernetes migration design covering 161 scheduled jobs and made the shared library cluster-ready (stdout log collection, service-DNS database routing, unit-tested), on a Patroni-managed high-availability PostgreSQL pair
- Lead analyst on infrastructure failure investigations and 24/7 on-call against a 99.9% availability SLA; rollback owner in formal change control, building incident timelines from core-dump metadata and process logs
Systems I built there
-
Ceph Storage Self-Healing Monitor
Sole author (77 of 79 commits, 6,586 lines, 61 tests). Monitors 24 storage nodes across 8 clusters on a 3-minute cycle and averages ~130 verified metadata-server failovers per month with no human involvement. Failover gated by a four-verdict standby-capacity assessment so it never moves into a worse node, outcomes verified against the cluster log to distinguish a real failover from a same-node reclaim. 7-table append-only event history, 19-panel Grafana dashboard, pytest in CI.
Python · uv · Fabric/SSH · PostgreSQL · Grafana · pytest
-
Config Integrity Self-Heal
Sole author (30 of 30 commits, 3,594 lines, 76 tests). Detects configuration drift across 18 gateways at 8 sites over SFTP and restarts the affected service, averaging ~295 remediations per month with 89% of episodes closing inside one 10-minute polling cycle. Modernized a blocking-I/O workload onto asyncio with semaphore-bounded fan-out rather than process pools.
Python · asyncio · SFTP · PostgreSQL
The failover idea, as a simulation with made-up clusters: on the Ceph monitor’s page.