Data Platform Engineer

Data Platform Engineer Permanent
Hybrid (3 days in the office, 2 days WFH)
London

McCabe & Barton are partnering with a leading financial services client to recruit an experiencedData Platform Engineer. This is an excellent opportunity to join a forward-thinking team driving innovation with modern cloud-based data technologies.

Role Overview

As a Data Platform Engineer, you will design, build, and maintain scalable cloud-based data infrastructure usingAzureandDatabricks. Youll play a key role in ensuring that data pipelines, architecture, and analytics environments are reliable, performant, and secure.

Key Responsibilities

Platform Development & Maintenance

• Design and implement data pipelines usingAzure Data Factory,Databricks, and related Azure services.

• Build ETL/ELT processes to transform raw data into structured, analytics-ready formats.

• Optimise pipeline performance and ensure high availability of data services.

Infrastructure & Architecture

• Architect and deploy scalable data lake solutions usingAzure Data Lake Storage.

• Implement governance and security measures across the platform.

• LeverageTerraformor similar IaC tools for controlled and reproducible deployments.

Databricks Development

• Develop and optimise data jobs usingPySparkorScalawithin Databricks.

• Implement themedallion architecture(bronze, silver, gold layers) and useDelta Lakefor reliable data transactions.

• Manage cluster configurations and CI/CD pipelines for Databricks deployments.

Monitoring & Operations

• Implement monitoring solutions usingAzure Monitor,Log Analytics, and Databricks tools.

• Optimise performance, ensure SLAs are met, and establish disaster recovery and backup strategies.

Collaboration & Documentation

• Partner with data scientists, analysts, and business stakeholders to deliver effective solutions.

• Document technical designs, data flows, and operational procedures for knowledge sharing.

Essential Skills & Experience

• 5+ years of experience withAzure services(Azure Data Factory, ADLS, Azure SQL Database, Synapse Analytics).

• Strong hands-on expertise inDatabricks,Delta Lake, andcluster management.

• Proficiency inSQLandPythonfor pipeline development.

• Familiarity withGit/GitHuband CI/CD practices.

• Understanding ofdata modelling,data governance, andsecurityprinciples.

Desirable Skills

• Experience withTerraformor other Infrastructure-as-Code tools.

• Familiarity withAzure DevOpsor similar CI/CD platforms.

• Experience with data quality frameworks and testing.

• Azure Data Engineer or Databricks certifications.