Senior Databricks Data Engineer

This is a key contract role focused on transforming and migrating legacy SQL Server-based data solutions into a modern Databricks architecture.

The successful candidate will assess the current data landscape, identify dependencies and data quality issues, redesign critical data structures, and support the migration of selected workloads into Databricks. You will work closely with business, accounting, and technology stakeholders to ensure the successful delivery of a scalable and supportable data platform.

Key Responsibilities

• Conduct a detailed inventory of the current data environment, including databases, schemas, tables, views, stored procedures, SQL Agent jobs, SSIS packages, feeds, reports, consumers, permissions, and operational processes.
• Document data ownership, lineage, business keys, relationships, calculations, transformations, refresh frequencies, and downstream dependencies.
• Perform data profiling and quality assessments, including completeness, accuracy, duplication, referential integrity, timeliness, and historical coverage.
• Collaborate with business and technical SMEs to determine which data structures, calculations, and reports should be retained, redesigned, consolidated, enhanced, or retired.
• Convert, redesign, and migrate T‑SQL code, stored procedures, SSIS packages, ETL processes, and reporting transformations into the target architecture.
• Design, build, and optimise Databricks solutions, including Delta tables, Spark SQL transformations, pipelines, quality controls, reconciliation processes, and monitoring capabilities.
• Create source‑to‑target mappings, migration strategies, validation plans, and technical documentation.
• Support business‑as‑usual activities, including SQL development, feed troubleshooting, reporting enhancements, testing, reconciliation, and production issue resolution.

Required Skills & Experience

• Extensive experience with SQL Server, T‑SQL, Databricks, and Spark SQL development.
• Strong expertise in data modelling, with a solid understanding of Medallion Architecture.
• Proven track record of migrating SQL Server, SSIS, stored procedures, ETL processes, and reporting workloads to Databricks or similar cloud‑based data platforms.
• Experience with data profiling, metadata management, lineage analysis, dependency mapping, source‑to‑target mappings, and data quality controls.
• Ability to assess complex legacy systems and determine the most effective approach for migration, redesign, consolidation, replacement, or retirement.
• Excellent documentation, stakeholder engagement, communication, and knowledge transfer skills.