Who put the Dog in the ClickHouse?

On Datadog Partnering with ClickHouse

Who put the Dog in the ClickHouse?

Well, leadership at Datadog did with this announcement at their customer conference earlier this summer: 

If you haven’t seen, Datadog and Clickhouse announced that now “organizations can route logs directly to ClickHouse through Datadog Observability Pipelines and search those logs from the Datadog Log Explorer, combining ClickHouse's economics and performance with the Datadog experience.”

Datadog is always announcing corporate partnerships or acquisitions – this one is different and why they made this move says something larger about the role of traditional observability pricing models in today’s market.

Waving the White Flag on Vendor Lock-In for Data Storage

You have to hand it to Datadog, this partnership is a very clever way to not cannibalise their core business - whose pricing model and structure is increasingly unfit for these times - whilst at the same time appeasing their large Enterprise customers who have already been moving AI workloads en masse to Clickhouse.

Mat Duggan recently wrote a much-discussed analysis on how Clickhouse’s columnar database was enabling it “win the observability wars” as data volumes climb. It’s for this reason that many observability vendors choose to just build on top of the Clickhouse database.

By leveraging Clickhouse, Datadog customers can now mitigate the trade-off between keeping full fidelity logs searchable and managing their skyrocketing observability costs at scale. In theory, the integration allows teams to store large volumes of data affordably in ClickHouse while seamlessly searching and managing it from the familiar Datadog user interface.

In practice, teams who opt for this tandem approach will now need to leverage two different tools for their observability - all to get around Datadog’s broken pricing model, which has customers threatening to churn as AI/agentic data volumes surge.

As we see it, the relationship between the two companies evolved due to distinct market pressures and technical realities:

1. The Churn Threat (Datadog's Defensively Motivated Move)

As everyone knows, Datadog’s dual ingestion-and-retention pricing model becomes incredibly expensive at scale. We wrote at length recently about why this vendor-centric model is unsustainable for customers.

As companies scale high-volume AI workloads, their engineering teams still often fight to keep Datadog for its premium UI, while finance teams demand cost cutting.

• The Threat: Hyperscale AI leaders (like OpenAI, Anthropic, and Character.ai) began migrating large log-management workloads off Datadog and onto ClickHouse—saving 70% to 90% in costs. Even ClickHouse's own engineering team built an internal platform called LogHouse to migrate away from their expensive Datadog setup.

• The Defense: Rather than losing these massive customers to complete churn, Datadog chose to partner. By allowing customers to route high-volume logs into ClickHouse, Datadog ensures that enterprises stay within the Datadog ecosystem instead of ripping it out completely.

This isn’t the first defensive move that Datadog has made to confront their data challenge. They introduced “Flex Logs” in 2023 to give customers the flexibility to choose which logs are indexed and sent to a particular storage tier. It’s great if you can see the future and know which logs will matter during an incident, and how much query capacity your team will need to search them. Once again, the burden is put on customers to navigate a broken system with increasingly complex plans.

2. The Storage Trap (ClickHouse's Enterprise Play)

ClickHouse is a column-oriented SQL database that is exceptionally good at ingesting and compressing billions of rows of telemetry data at a fraction of standard cloud warehouse costs. However, raw database infrastructure lacks the out-of-the-box alerting, dashboarding, and incident response UI that DevOps teams rely on every day.

• Partnering with Datadog allows ClickHouse to instantly anchor itself within standard enterprise engineering workflows.

• Customers get "full-fidelity" log data retention at ClickHouse’s low costs, but they don't have to build custom UIs or manually stitch queries together.

This two-fer approach might solve an immediate problem and work well when it's a limited number of high profile customers whose telemetry data volumes are insane and growing exponentially, but does it scale? As high volume workloads become more common across all enterprise companies and indeed midmarket companies too, surely this multi-vendor complexity isn’t the long-term answer for customers. 

This is a partnership of inconvenience - use this and this and still don't achieve your goal – but again hats off to Datadog for the clever marketing to, and management of their most 'spendy' customers but it's hard to see this translating beyond this group.

As ClickHouse leans further into the observability space, between the two, who would you rather be? At Bronto, we believe that data is king.

Solving the Observability Data Problem Directly

We built Bronto to skip past waiting for these traditional vendors to pivot themselves into a complete solution for customers.

Instead of ClickHouse’s columnar database, we built a custom polymorphic database that is purpose-built for observability data. We tackled the indexing and cardinality challenges head on so storing each type of signal is at least 100x more efficient and search is sub-second fast. You get full-fidelity logs, traces, and metrics; and instead of 30 day retention, you get 12 month always hot retention.

Instead of Datadog’s prohibitive ingestion-and-retention pricing, Bronto plans are based on straightforward rates $0.10 per GB ingested and $1 per TB searched. That’s it. No per-metric, per-host, or per-seat charges. Finally, you can predict your monthly costs without a quantum computer.

So yes, this partnership announcement signals that the traditional solutions are just not built for today’s data volumes. Just like sampling and shortened retention, this integration is another workaround you can adopt to put a bandaid on the observability data problem.

If you want to see how we’ve solved it instead, try Bronto today. All your data in full fidelity for a year, always hot and accessible to humans and agents. 

It's your data - have it your way.