Clickhouse Solutions Architect

We are looking for a ClickHouse Solutions Architect to join our team supporting the design and implementation of a greenfield, enterprise-scale ClickHouse observability platform for a global banking client. This is a genuine greenfield build at significant scale — there is no incumbent platform to inherit or work around. You will be shaping the architecture from first principles, in an environment where the performance, multi-tenancy and retention requirements are demanding enough that the design decisions made early will determine whether the platform holds up in production.

About the project:

• We are architecting a central repository for three classes of time-series telemetry — metrics, traces and logs — serving the bank's application estate.

The system requirements are:

• Ingest path: 3 million events per second sustained, each event carrying a timestamp and full transaction payload
• Read path: Approximately 1,000 concurrent users
• Several hundred real-time dashboards
• Multi-tenancy and workload isolation
• Hundreds of distinct application codes as tenants
• Throttling and quota control required on both read and write paths, so that no single tenant can degrade another

Standards and tooling constraints:

• OpenTelemetry-native throughout: native OTel agent and native OTel collector
• No proprietary components anywhere in the agent or collector pipeline

Retention and storage tiering:

Getting this right means solving for ingest throughput, query concurrency, tenant isolation and storage cost simultaneously — the interesting part of the work is that these pull against each other, and the architecture has to reconcile them rather than optimise for any one in isolation.

Key Activities:

• Discovery and architecture
• Run technical discovery with client stakeholders to establish workload characteristics — query patterns, latency targets, concurrency, ingest volume and velocity, retention and cost envelope
• Translate those into a target-state architecture: deployment model (ClickHouse Cloud vs self-managed on Kubernetes/VMs), cluster topology, shard and replica strategy, Keeper configuration, storage tiering
• Design the data model — table engine selection across the MergeTree family, primary/sort key design, partitioning, projections, materialized views, dictionaries, TTL and object-storage tiering
• Design the ingestion architecture — streaming (Kafka/Kinesis, Kafka table engine or ClickPipes), batch and CDC paths, idempotency and deduplication strategy, schema evolution
• Produce sizing and cost modelling, with options and trade-offs rather than a single answer
• Define the semantic and consumption layer — BI tool integration, API access patterns, downstream contracts

Build and validation:

• Stand up the reference environment and prove the architecture against representative data volumes and query shapes
• Build and tune the ingestion pipelines to agreed throughput and freshness SLOs
• Benchmark and tune query performance against the stated latency targets; iterate on the data model where benchmarks disprove the design
• Establish infrastructure-as-code, CI/CD and environment promotion for schema and configuration changes

Productionisation:

• Define and implement observability — system table monitoring, metrics, alerting thresholds, capacity headroom tracking
• Establish backup, restore and disaster recovery, and validate them by test
• Implement security and governance — RBAC, row/column-level access, encryption, network isolation, audit, and any sector-specific compliance requirements
• Define upgrade, patching and capacity-management procedures

Client enablement and handover:

• Set engineering standards and guardrails the client's teams work within — naming, modelling patterns, query review criteria, anti-patterns
• Coach and upskill the client's data engineers through pairing, review and structured sessions
• Run the handover to the client's run team against an agreed readiness checklist
• Act as escalation point for complex performance and design questions during transition

Advisory and practice contribution (weighted more heavily at principal level):

• Support pre-sales, bid and scoping work — estimating, technical qualification, solution shaping
• Build reusable assets, reference architectures and accelerators from delivered work
• Represent the practice credibly in client and community forums