Blogsby GlyphNov 5
How we continuously validate formal specifications in production
Learn how Datadog built FETCh to continuously validate production behavior against TLA+ specifications and uncover a subtle bug that conventional testing had not exposed.
Datadog Engineering the engineering blog of Datadog: latest posts and feed
Datadog · Engineering blog · 2 a month · 104 posts since Jul 11, 2016
Oct 20
Beyond container image signing: How we enforce software supply chain policies at Datadog
Datadog built Turnstile to evaluate signed attestations and enforce software supply chain policies before container images are deployed.
Oct 1
How we extended Apache DataFusion to execute one query across many machines
Learn how Datadog built Distributed DataFusion to scale interactive Apache DataFusion queries across multiple machines.
Sep 30
How we built an async-aware Python profiler
Learn how we made Python profiling async-aware while reducing profiler overhead and preserving context across asyncio tasks.
Sep 1
Making Rust observability reliable at scale with OpenTelemetry
Learn how Datadog improved Rust tracing by building an opinionated OpenTelemetry-based library to help ensure consistent sampling, propagation, and trace quality at scale.
Aug 19
20× the CI traffic without getting slower: How we rebuilt Git serving at Datadog
Learn how Datadog built gitretriever to handle 20× more CI Git traffic while maintaining low latency and reducing backend CPU usage.
Aug 11
How we improved APM Java startup by encoding a prefix trie as a JVM constant
Learn how the Datadog APM team improved Java startup performance by encoding a prefix trie as a JVM string constant.
Jul 22
Unbiased Java CPU profiling with JFR in JDK 25
Modern Java profilers often rely on unsupported JVM internals for accurate CPU profiling. Here’s how engineers from Datadog, SAP, Amazon, and the OpenJDK community helped bring a new CPU profiling event to JDK 25.
Jul 1
How we measure data completeness at scale
Learn how Datadog helps ensure data completeness at scale, enabling accurate alerts and safer automated decisions across distributed pipelines.
Jun 23
How we migrated a live routing system using AI-assisted refactoring
Using AI-assisted refactoring, we migrated our live routing brain to a relational model, safely validating changes against live production traffic.
Jun 4
When failover isn’t safe: Building high-availability PostgreSQL on Kubernetes
During a reliability gameday, Datadog engineers discovered that their PostgreSQL clusters couldn’t safely fail over. Here’s how the team redesigned them for high availability using Patroni and synchronous replication.
Jun 2
From single pull requests to full software packages: Detecting malicious code at scale
By combining stacked LLM evaluations with tool-driven investigation, we scaled malicious code detection from pull requests to dependency packages without sacrificing accuracy or cost control.
Apr 21
Steganography at scale: Embedding share URLs in Datadog widget screenshots
Learn how we embed widget metadata into screenshots using invisible, resilient watermarks, enabling self-describing visualizations at scale.
Apr 7
How we built a real-world evaluation platform for autonomous SRE agents at scale
Find out how we built a scalable evaluation platform for Datadog’s Bits AI SRE agent that replays real incidents, detects regressions, and measures agent performance across production scenarios.
Mar 23
When upserts don’t update but still write: Debugging Postgres performance at scale
When a high-volume upsert doubled disk writes, Datadog engineers traced the issue to Postgres WAL behavior and rewrote the query to eliminate hidden costs.
Mar 9
When an AI agent came knocking: Catching malicious contributions in Datadog’s open source repos
Learn how Datadog detected and resolved issues from hackerbot-claw, an AI-powered automated attack campaign.
Mar 4
Designing MCP tools for agents: Lessons from building Datadog’s MCP server
We share lessons learned building Datadog’s MCP server, from designing agent-friendly tools and managing context windows to using queries instead of raw data retrieval.
Feb 18
How we reduced the size of our Agent Go binaries by up to 77%
Get an inside look at shrinking large Go binaries in the Datadog Agent through dependency and linker analysis.
Jan 7
Hardening eBPF for runtime security: Lessons from Datadog Workload Protection
Learn field-tested lessons for eBPF-powered workload protection.
Nov 18, 2025
Scaling real-time file monitoring with eBPF: How we filtered billions of kernel events per minute
Learn how Datadog scaled eBPF-powered file monitoring to handle more than 10 billion kernel events per minute while preserving full detection coverage.
Nov 4, 2025
Replication redefined: How we built a low-latency, multi-tenant data replication platform
Discover how Datadog engineered a scalable Change Data Capture (CDC) platform to replicate data across systems in near real time, reducing search latency by 87%, increasing availability, and powering diverse, multi-tenant use cases across...
Oct 21, 2025
Detecting malicious pull requests at scale with LLMs
Learn how Datadog’s SDLC Security team built an LLM-powered system to detect malicious pull requests at scale, without sacrificing developer velocity.
Oct 15, 2025
Failure is inevitable: Learning from a large outage, and building for reliability in depth at Datadog
After a major outage, we re-architected Datadog systems to degrade gracefully under failure. Here’s what we learned, and how we’re building forward.
Oct 1, 2025
Inside Husky’s query engine: Real-time access to 100 trillion events
See how Husky enables interactive querying across 100 trillion events daily by combining caching, smart indexing, and query pruning.
Sep 18, 2025
From hand-tuned Go to self-optimizing code: Building BitsEvolve
Learn how we turned hot-path optimizations into a system for continuous, AI-assisted performance improvements and saved thousands of cores in the process.
About this blog
How Datadog builds observability at scale: storage, Go, Rust and incidents.
Posts per week, last 26 weeks
2 a month over the last six months (or since the feed's oldest post)
Posts by month
Writes about
Developer tools (6), Reliability and performance (3), Databases (2) in the last six months.
Datadog
Datadog on Companies (companies.fru.dev)Across Glyph:Acquisitions·advisories·awards·calendar·Attested·Quarterly·Tier Zero·Downtime·Mainstage·Leaders·marketplaces·openperks·papers·Allied·Paydays·problems·quadrant·Changefeed·repos·salaries·Nines·socials·standards·Stories·TechConf·terms·Titlewave
History
Official checkSep 25
not reviewed to official: on datadoghq.com