Hardening a ~3-billion-events/month client-side threat-detection pipeline on AWS

Serverless compute made the platform possible — no fleet to size for half-billion-event customer spikes — but at this volume its economics become the architecture’s central question. VeUP’s cost mapping put Lambda (both the write path onto the streams and the read path consuming them) at the top of the pipeline’s spend, with the surrounding failure modes compounding it: hot Kinesis shards concentrating invocations, OpenSearch write-amplification multiplying downstream work, and entitlement rejections arriving only after the functions had already run. The platform needed its serverless spine kept — and made efficient.

VeUP mapped the full production path — Amazon API Gateway → Lambda ingest → Kinesis Data Streams (raw + enriched) → ~15–20 Lambda consumers → OpenSearch hourly aggregates, with Firehose → S3 replay — and scoped a plan that attacks Lambda cost at its causes rather than its symptoms. Upstream, edge throttling and entitlement enforcement at API Gateway stop invocations for traffic the platform will reject; de-skewed Kinesis partition keys spread consumer invocations evenly so concurrency tracks real load, not shard luck. Within the functions, the plan scopes batch-size and concurrency tuning on the stream consumers and an append-only write pattern into OpenSearch that eliminates the update-churn each consumer was paying for. Per-ARN FinOps instrumentation ties every function’s spend to its throughput, making cost per ingested event a tracked KPI — with the success criteria holding sub-60–90-second freshness and near-zero drops at peak while the cost curve bends.