BrontoBytes runs 4-5 TB/day per customer on AWS Lambda + Kinesis Firehose

Log ingestion is a workload that punishes hand-built plumbing: it never stops, it bursts with every customer incident, and at multi-TB/day the difference between an efficient and an inefficient ingest path is a material line item. The platform’s Lambda log-forwarder chain carried operational burden and cost sensitivity — every forwarded batch invoked compute, and delivery semantics were the platform’s own problem. A subtle economic trap compounded it: Firehose bills in 5 KB increments, so a naive record-shaping strategy can pay for padding rather than payload. The ingest tier needed to become boring — managed, buffered, durable — with its billing behavior engineered, not discovered.

VeUP’s modernization put Amazon Kinesis Data Firehose at the front of the platform: customer log flows deliver into Firehose, which buffers, batches, and lands them in the S3 log store with managed delivery semantics — no forwarder fleet to operate, back-pressure and retry handled by the service. The design work concentrated where the money is: record sizing and batching tuned against Firehose’s 5 KB billing increment so the platform pays for log data, not for rounding; buffer intervals tuned to balance freshness against object efficiency in S3; and the flow-control posture hardened so tenant bursts degrade gracefully instead of cascading. Downstream, the platform’s hundreds of GraalVM-compiled Java Lambda functions continue to fan out over the S3 store for burst search, with the review’s FinOps program targeting a 30–40% reduction in per-search Lambda cost — the ingest and query tiers optimized as one economic system, instrumented in CloudWatch.