Senior Data Engineer in England

Senior Data Engineer – Contract

£600 per day | Outside IR35 | Fully Remote | 2-Month Contract | Start: 1st November

We are looking for a highly experienced Senior Data Engineer to support a critical data platform migration project for one of our clients.

This is an initial 2-month contract , working fully remotely, with a target delivery date of the end of December.

The successful consultant will be responsible for migrating an existing Azure Synapse Dedicated SQL Pool codebase into Databricks, ensuring the migrated solution delivers equivalent or improved performance, maintains data accuracy and supports ongoing business consumption.

We are specifically looking for someone with hands-on experience migrating complex SQL workloads from Synapse Dedicated SQL Pools to Databricks , rather than general Databricks engineering experience alone.

Key Responsibilities

• Migrate approximately 72 T-SQL stored procedures from Azure Synapse Dedicated SQL Pool into Databricks.
• Analyse existing SQL code, dependencies and execution patterns to determine the most effective migration approach.
• Refactor and optimise complex T-SQL logic for Databricks SQL, Spark SQL and/or PySpark.
• Identify, remove and rework dependencies on existing Claims dimensional model fact and dimension tables.
• Ensure migrated datasets and pipelines are available for downstream business consumption.
• Address architectural differences between Synapse's MPP SQL processing engine and Databricks' Spark-based execution.
• Optimise query execution, joins, data transformations and pipeline performance.
• Deliver equivalent or improved pipeline performance following migration.
• Perform comprehensive reconciliation and validation to ensure data accuracy and consistency with existing migration thresholds.
• Troubleshoot migration issues and deliver a production-ready solution within the agreed timeline.

Essential Skills & Experience

Azure Synapse Dedicated SQL Pool

• Strong hands-on experience with Azure Synapse Analytics, specifically Dedicated SQL Pools.
• Advanced T-SQL skills, including complex stored procedures, transformations and query optimisation.
• Deep understanding of MPP architecture, distributed tables, execution plans and performance tuning.

Databricks & Apache Spark

• Extensive commercial experience with Azure Databricks, Spark SQL, Databricks SQL and PySpark.
• Strong understanding of Apache Spark execution, data distribution, joins, shuffles and performance optimisation.
• Experience with Delta Lake and modern lakehouse architecture.

SQL / Synapse to Databricks Migration

• Demonstrable experience migrating SQL workloads from Synapse Dedicated SQL Pools or similar MPP SQL environments into Databricks.
• Strong understanding of the architectural and processing differences between Synapse SQL and Databricks.
• Experience converting complex T-SQL code into scalable, maintainable Databricks solutions.
• Ability to identify and resolve SQL compatibility, data transformation and performance challenges.

Data Quality & Validation

• Experience reconciling source and target datasets following complex migrations.
• Strong understanding of data validation, regression testing and data quality.
• Ability to work independently and take ownership of delivery.

Desirable Experience

• Previous experience migrating large volumes of SQL stored procedures.
• Insurance, Claims or financial services data experience.
• Experience working with dimensional data models, including fact and dimension tables.
• Databricks Workflows, orchestration and performance tuning.
• Experience delivering time-critical data platform migration projects.

Contract Details

• Role: Senior Azure Synapse to Databricks Migration Consultant
• Rate: Up to £600 per day
• IR35: Outside IR35
• Location: Fully Remote
• Duration: Initial 2 months
• Start Date: 1st November 2026
• Target Delivery: End of December 2026

#J-18808-Ljbffr