Lead Data Engineer(Snowflake)

Lead Data Engineer(Snowflake)

Full-Time On-site
V

Is it Permanent / Contract: Open for Contract (6 months)/FTC (6 months)/Perm

Is it Onsite/Remote/Hybrid: 4 days per week from office

Key Responsibilities

Platform foundations

  • Build Snowflake environment structures – layered databases and schemas (Raw / Conformed / Consumption), environment separation (development / test / production) and naming conventions
  • Implement role-based access control, service accounts, virtual warehouse design, resource monitors and cost guardrails
  • Manage the platform as code – Snowflake objects, roles, grants and warehouses through infrastructure-as-code and version control
  • Establish Git workflows and CI/CD pipelines for data code, with automated build, test and deployment across environments

Data acquisition and CDC

  • Assess source systems and recommend fit-for-purpose acquisition patterns – full vs incremental, batch vs micro-batch vs event, and file vs API vs direct database vs message hub
  • Advise on where change data capture should be applied and owned – log-based CDC at source, application-emitted change events, or warehouse-side change detection – and implement the chosen patterns
  • Engineer CDC and replication from relational, document and time-series application databases, handling initial loads, deletes, schema evolution, lag and backfills
  • Ingest event streams from messaging platforms into Snowflake, covering ordering, idempotency, deduplication, replay and error handling
  • Configure and optimise managed ELT connectors (e.g. Fivetran), and build file- and API-based ingestion where required
  • Design and build a metadata-driven ingestion framework – sources, objects, load types (full, incremental, CDC), schedules and target mappings held as configuration, with control tables for watermarks, run status, audit and reprocessing – so new feeds are onboarded through metadata rather than new code
  • Make the framework reusable across ingestion channels (managed ELT, file, API, database CDC and event feeds), with generic, parameterised loading, schema-drift handling and standard error handling
  • Prove patterns through technical spikes, and record decisions as Architecture Decision Records

Transformation and modelling

  • Design and build dbt projects from the ground up – structure, layering, conventions, macros, packages, tests and documentation
  • Run dbt both natively in Snowflake (dbt Projects on Snowflake) and on dbt Cloud, building portable, convention-led projects that move between the two with minimal rework
  • Implement canonical data models – including industry-aligned models such as BIAN-based business objects – from source-to-canonical mapping specifications and governed reference data
  • Build dimensional and domain-oriented data products for reporting, regulatory, analytics and AI consumers
  • Engineer identity resolution, key management and cross-source reconciliation, and feed implementation findings back into the models and mappings

Quality, governance and operations

  • Embed data quality as code – tests, freshness checks, reconciliations and data contract enforcement
  • Implement Snowflake governance controls – tagging and classification, dynamic data masking, row access policies and PII handling aligned to GDPR – with lineage that supports BCBS 239
  • Build monitoring and alerting for pipeline failures, freshness, data quality and cost, and define run-books and support processes

Orchestration and scheduling

  • Design and implement orchestration and scheduling natively in Snowflake using Tasks and task graphs (DAGs) – dependencies, root and finaliser tasks, serverless vs warehouse-backed compute, schedules and event-triggered runs
  • Orchestrate ingestion, dbt execution (e.g. EXECUTE DBT PROJECT), data quality checks and downstream refreshes end to end within Snowflake, with retries, failure handling, alerting and notifications
  • Use Dynamic Tables where declarative, target-lag-driven pipelines are a better fit than task-based scheduling, choosing between incremental and full refresh modes and managing their cost
  • Build operational visibility over task and Dynamic Table runs – run history, latency, failures and credit consumption – and keep orchestration designs portable to external orchestrators where needed
  • Use Snowflake CoCo (formerly Cortex Code) to accelerate development, with generated code reviewed, tested and deployed through standard CI/CD
  • Apply Snowflake Cortex AI capabilities – AISQL functions, Cortex Search, semantic views, Cortex Analyst and Cortex Agents – where they add value, and design data products to be AI-ready

Leadership and collaboration

  • Work as the engineering counterpart to the Enterprise Data Architect, contributing to design decisions and ADRs
  • Collaborate with Integration, Solution and Application teams on source extracts, event contracts and data contracts
  • Define and own engineering standards, templates and reusable patterns
  • Lead level: lead, mentor and help recruit data engineers, own code review, and plan and report delivery

Experience & Profile

  • Senior: 10+ years in data engineering, including at least 3 years of hands‑on Snowflake delivery in production. Lead: 12+ years, including leading a data engineering team or workstream
  • Strong track record in data warehousing and data modelling, with experience implementing models designed by architects as well as designing their own
  • Has stood up a new data platform or major platform workstream – environments, access control, automation – rather than only extending an existing one
  • Has built a dbt project from the ground up, including standards, testing and CI/CD
  • Has designed and built a metadata-driven ingestion framework that has been used to onboard multiple sources in production
  • Strong hands‑on experience scheduling and orchestrating production pipelines natively in Snowflake with Tasks / task graphs and Dynamic Tables
  • Has designed and implemented data acquisition across heterogeneous sources, and can clearly explain the trade‑offs between full, incremental and CDC approaches
  • Comfortable reading and implementing architecture artefacts – canonical models, mapping specifications, ADRs – and offering well‑reasoned implementation feedback
  • Clear communicator, able to work with architects, application teams and business stakeholders

Lead Data Engineer(Snowflake) employer: VBeyond Europe

Join a forward-thinking company in London that prioritises innovation and quality in the Contact Centre and Telephony sector. As a Test Manager/Test Lead, you will thrive in a collaborative work culture that values employee growth through continuous learning and development opportunities. With a focus on automation and AI adoption, this role offers a unique chance to lead transformative projects while enjoying a supportive environment that champions excellence and teamwork.

V

Contact Details:

VBeyond Europe Recruitment Team