Is it Permanent / Contract: Open for Contract (6 months)/FTC (6 months)/Perm
Is it Onsite/Remote/Hybrid: 4 days per week from office
Key Responsibilities
Platform foundations
- Build Snowflake environment structures – layered databases and schemas (Raw / Conformed / Consumption), environment separation (development / test / production) and naming conventions
- Implement role-based access control, service accounts, virtual warehouse design, resource monitors and cost guardrails
- Manage the platform as code – Snowflake objects, roles, grants and warehouses through infrastructure-as-code and version control
- Establish Git workflows and CI/CD pipelines for data code, with automated build, test and deployment across environments
Data acquisition and CDC
- Assess source systems and recommend fit-for-purpose acquisition patterns – full vs incremental, batch vs micro-batch vs event, and file vs API vs direct database vs message hub
- Advise on where change data capture should be applied and owned – log-based CDC at source, application-emitted change events, or warehouse-side change detection – and implement the chosen patterns
- Engineer CDC and replication from relational, document and time-series application databases, handling initial loads, deletes, schema evolution, lag and backfills
- Ingest event streams from messaging platforms into Snowflake, covering ordering, idempotency, deduplication, replay and error handling
- Configure and optimise managed ELT connectors (e.g. Fivetran), and build file- and API-based ingestion where required
- Design and build a metadata-driven ingestion framework – sources, objects, load types (full, incremental, CDC), schedules and target mappings held as configuration, with control tables for watermarks, run status, audit and reprocessing – so new feeds are onboarded through metadata rather than new code
- Make the framework reusable across ingestion channels (managed ELT, file, API, database CDC and event feeds), with generic, parameterised loading, schema-drift handling and standard error handling
- Prove patterns through technical spikes, and record decisions as Architecture Decision Records
Transformation and modelling
- Design and build dbt projects from the ground up – structure, layering, conventions, macros, packages, tests and documentation
- Run dbt both natively in Snowflake (dbt Projects on Snowflake) and on dbt Cloud, building portable, convention-led projects that move between the two with minimal rework
- Implement canonical data models – including industry-aligned models such as BIAN-based business objects – from source-to-canonical mapping specifications and governed reference data
- Build dimensional and domain-oriented data products for reporting, regulatory, analytics and AI consumers
- Engineer identity resolution, key management and cross-source reconciliation, and feed implementation findings back into the models and mappings
Quality, governance and operations
- Embed data quality as code – tests, freshness checks, reconciliations and data contract enforcement
- Implement Snowflake governance controls – tagging and classification, dynamic data masking, row access policies and PII handling aligned to GDPR – with lineage that supports BCBS 239
- Build monitoring and alerting for pipeline failures, freshness, data quality and cost, and define run-books and support processes
Orchestration and scheduling
- Design and implement orchestration and scheduling natively in Snowflake using Tasks and task graphs (DAGs) – dependencies, root and finaliser tasks, serverless vs warehouse-backed compute, schedules and event-triggered runs
- Orchestrate ingestion, dbt execution (e.g. EXECUTE DBT PROJECT), data quality checks and downstream refreshes end to end within Snowflake, with retries, failure handling, alerting and notifications
- Use Dynamic Tables where declarative, target-lag-driven pipelines are a better fit than task-based scheduling, choosing between incremental and full refresh modes and managing their cost
- Build operational visibility over task and Dynamic Table runs – run history, latency, failures and credit consumption – and keep orchestration designs portable to external orchestrators where needed
- Use Snowflake CoCo (formerly Cortex Code) to accelerate development, with generated code reviewed, tested and deployed through standard CI/CD
- Apply Snowflake Cortex AI capabilities – AISQL functions, Cortex Search, semantic views, Cortex Analyst and Cortex Agents – where they add value, and design data products to be AI-ready
Leadership and collaboration
- Work as the engineering counterpart to the Enterprise Data Architect, contributing to design decisions and ADRs
- Collaborate with Integration, Solution and Application teams on source extracts, event contracts and data contracts
- Define and own engineering standards, templates and reusable patterns
- Lead level: lead, mentor and help recruit data engineers, own code review, and plan and report delivery
Experience & Profile
- Senior: 10+ years in data engineering, including at least 3 years of hands‑on Snowflake delivery in production. Lead: 12+ years, including leading a data engineering team or workstream
- Strong track record in data warehousing and data modelling, with experience implementing models designed by architects as well as designing their own
- Has stood up a new data platform or major platform workstream – environments, access control, automation – rather than only extending an existing one
- Has built a dbt project from the ground up, including standards, testing and CI/CD
- Has designed and built a metadata-driven ingestion framework that has been used to onboard multiple sources in production
- Strong hands‑on experience scheduling and orchestrating production pipelines natively in Snowflake with Tasks / task graphs and Dynamic Tables
- Has designed and implemented data acquisition across heterogeneous sources, and can clearly explain the trade‑offs between full, incremental and CDC approaches
- Comfortable reading and implementing architecture artefacts – canonical models, mapping specifications, ADRs – and offering well‑reasoned implementation feedback
- Clear communicator, able to work with architects, application teams and business stakeholders
Lead Data Engineer(Snowflake) employer: VBeyond Europe
Join a forward-thinking company in London that prioritises innovation and quality in the Contact Centre and Telephony sector. As a Test Manager/Test Lead, you will thrive in a collaborative work culture that values employee growth through continuous learning and development opportunities. With a focus on automation and AI adoption, this role offers a unique chance to lead transformative projects while enjoying a supportive environment that champions excellence and teamwork.