IronSight AI

What it is, and what is deliberately not here

A private in-house machine-learning analytics platform, roughly 93 Python programs and 31,000 lines, every one of them mine. What follows is the engineering: the pipeline shape, the verification discipline and the reliability work. The model design, the inputs it reads and the method behind it are proprietary and are not described on this site — the point of showing this project is how it was engineered, not what it concluded.

Pipeline

Real-time data ingestion with an in-memory cache and a fallback path for anything the primary route misses, feeding a local service layer so nothing downstream reaches a vendor directly. The training pipeline produces ensembled models with strictly time-ordered train, validation and sealed-test splits. It is resumable, so a long run survives an interruption, degrades gracefully when a dependency is unavailable rather than failing outright, handles missing values natively instead of imputing data that was never there, and versions every artifact it saves.

Verification before serving — the part I would argue for in any ML system

A separate harness re-runs live inference against held-out data and refuses to promote a model whose live accuracy does not reproduce its test accuracy. This catches the failure that quietly kills machine learning in production: a train-serve skew where the offline number is real and the online number is not. Paired with versioned artifacts and one-step rollback, it means a bad model cannot become the live model by default.

Honest measurement

Every prediction is graded the same day it resolves, and the record is kept in full rather than curated — misses included. The validation methodology was rebuilt once a single hold-out split proved over-optimistic, and a candidate input was dropped outright when leakage was found in it. Being willing to throw away a feature that flattered the numbers is the whole discipline, and it is the habit I bring to evaluating anyone else’s model.

Reliability engineering

Atomic state writes — serialize, temp file, fsync, re-parse, replace — so a crash mid-write cannot corrupt state. A watchdog that detects process death, heartbeat stalls and stale output, with a restart-storm guard so a crash loop cannot become an outage. Structured logging throughout, and a safe default: the system does nothing unless explicitly enabled for the day, so the failure mode of every guard is inaction rather than a wrong action.

Built with