Lessons from a year of Postgres CDC in production

Summary

It’s been about a year since we launched the private preview of the Postgres CDC connector in ClickHouse Cloud as part of ClickPipes, built on the integration of PeerDB, which we acquired in 2024. Since then, it has gone through several iterations, progressed through public beta, and reached GA in May. I want to reflect on that journey, the learnings and experiences from integrating a product post-acquisition, how customer adoption shaped the connector, the features I’m most excited about, the gaps we’re tackling next, and the key lessons along the way. So let’s get started. The primary goal of this connector was to make it easy for customers to integrate two open-source database technologies, each purpose-built for a specific use case, Postgres and ClickHouse.The idea was simple: give customers a straightforward way to sync transactional data from Postgres to ClickHouse and seamlessly offload analytics to ClickHouse. Keeping the above goal in mind, ClickHouse acquired PeerDB, a state-of-the-art open-source product built specifically for Postgres CDC, in July 2024. Within just four months, PeerDB was fully integrated into ClickHouse Cloud and became the engine powering the Postgres CDC connector in ClickPipes. As part of the acquisition, we made an important decision: not only to keep PeerDB open source, but also to maintain it as a distinct, modular component powering ClickPipes. In other words, you can think of PeerDB as the open-source DB CDC engine for ClickHouse, while ClickPipes CDC is the managed-service implementation of PeerDB in ClickHouse Cloud. In retrospect, this decision was critical for engineering velocity and management - allowing each piece to evolve independently yet cohesively, while ensuring that all enhancements made to ClickPipes flow back into the open-source project for the broader community. To validate the impact of this decision on velocity, I pulled up a comparison based on the GitHub activity of PeerDB versus another popular and well-kn...