Hardening a ~3-billion-events/month client-side threat-detection pipeline on AWS

The analytics contract is demanding: security customers expect events to appear in their dashboards within a minute or two, aggregated hourly, forever queryable — while ingest volume swings by an order of magnitude as individual customer projects spike to half a billion events a month. Under review, the platform showed the standard pathologies of a grown-not-designed streaming estate: partition skew concentrating load, aggregate updates amplifying write costs, ungoverned index aging, and unit economics that scaled linearly with volume when the business needed them sub-linear.

VeUP mapped the full analytics topology — API edge, dual Kinesis streams (raw and enriched), the consumer fan-out, the OpenSearch aggregate tier, and the Firehose → S3 archive — and delivered a modernization plan where every driver has a named fix. Partition de-skew makes stream throughput additive across shards; entitlement and throttling move to the edge so the analytics tiers only process billable events; the aggregate layer moves to append-only writes with materialized hourly rollups; and index lifecycle management ages data down storage tiers with the S3 lake guaranteeing replayability. The plan also scoped an optional real-time OLAP pilot alongside OpenSearch for the workloads where columnar aggregation fits better. Success criteria were set like an analytics SLA: near-zero dropped events at peak, sub-60–90-second event-to-dashboard freshness plus hourly rollups, and a measured, per-component cost per ingested event trending down.