At a Glance
- Tasks: Lead the design and development of a cutting-edge Databricks data platform.
- Company: Join a forward-thinking team in the retail sector with a focus on innovation.
- Benefits: Enjoy a competitive salary, flexible remote work, and opportunities for professional growth.
- Other info: Dynamic role with excellent career advancement potential in a collaborative environment.
- Why this job: Make a real impact by shaping data architecture and mentoring a talented team.
- Qualifications: Strong experience in data engineering, Databricks, Python, and cloud platforms required.
The predicted salary is between 65000 - 65000 £ per year.
We are looking for a Lead Data Engineer to join the Retail Insight Data and Engineering team and own the architecture and development of our Databricks environment. This is a senior, hands-on role where you will design and build scalable data infrastructure that underpins our products, while guiding a small team of engineers and defining platform standards, data architecture and engineering best practices supporting analytics, machine learning and AI across the business.
Key responsibilities
- Lead the design, development and optimisation of a Databricks based data platform, ensuring reliability, performance and scalability.
- Design enterprise scale data architectures on Databricks and cloud platforms such as Azure, AWS or GCP.
- Define lakehouse standards, data modelling approaches and governance frameworks.
- Lead design of batch and streaming platforms using Delta Lake, Spark and related technologies.
- Establish security, access control, data quality and compliance standards.
- Optimise performance, scalability and cloud cost management.
- Lead and mentor a small team of data engineers, setting technical standards and reviewing code.
- Partner with data science, analytics and product teams to ensure the platform supports their work.
- Own data quality, governance and observability across pipelines.
- Support AI and ML enablement including feature stores and model deployment patterns.
- Translate business requirements into technical solutions.
- Contribute to platform roadmap, tooling selection and continuous improvement.
Required experience
- Strong experience as a senior or lead data engineer building and scaling production data platforms.
- Deep hands-on expertise in Databricks including Spark, Delta Lake and Databricks Workflows.
- Python and SQL proficiency.
- Infrastructure as code experience using Terraform, Bicep or CloudFormation.
- Cloud data platform experience, ideally Azure including Azure Data Factory, ADLS Gen2 and Azure DevOps.
- Strong understanding of data modelling, ETL and ELT patterns, distributed processing.
- Experience with security, governance and data platform operations.
- Experience leading or mentoring engineers, with strong stakeholder management.
- Rigorous approach to data quality and testing.
- Databricks certification.
Desirable
- Unity Catalog, MLflow and Databricks Asset Bundles.
- dbt or similar transformation frameworks.
- Retail or CPG data such as POS, inventory, supply chain.
- Streaming technologies such as Kafka or Event Hubs.
Education
- Degree in computer science, data engineering or related field, or equivalent practical experience.
Lead Data Engineer in London employer: FBI &TMT
As a Lead Data Engineer at our company, you will thrive in a dynamic and innovative environment that champions collaboration and professional growth. With the flexibility of remote or hybrid work options and a supportive culture that values mentorship, you'll have the opportunity to lead impactful projects while enjoying a competitive salary and benefits package. Join us in shaping the future of data engineering within the retail sector, where your expertise will be recognised and rewarded.
