At a Glance
- Tasks: Lead the design and implementation of a modern cloud-based data platform using Databricks on Azure.
- Company: Join a leading private equity firm with exciting projects in data engineering.
- Benefits: Hybrid work model, competitive salary, and opportunities for professional growth.
- Other info: Dynamic role with a focus on greenfield implementations and collaboration with stakeholders.
- Why this job: Make a real impact by building innovative data solutions from the ground up.
- Qualifications: 5+ years in Data Engineering, strong skills in Databricks, Python, and SQL.
The predicted salary is between 60000 - 80000 £ per year.
Location: London (Hybrid) Overview We are supporting a leading private equity firm in the build of a greenfield data warehouse and portfolio monitoring platform.
This is a key contract opportunity for an experienced Data Engineer to lead the design, architecture and implementation of a modern cloud-based data platform using Databricks on Azure.
The successful consultant will be responsible for establishing the core data infrastructure, building scalable ingestion frameworks, implementing reconciliation controls across multiple data sources, and ensuring robust governance and quality assurance processes are embedded from day one.
What you'll do Design and build a new Databricks environment on Azure from the ground up.
Define the architecture and technical standards for the data warehouse and portfolio monitoring platform.
Design and implement data ingestion pipelines from: Aztec (SFTP / file-based integrations) Canoe (API-based integrations) Build and maintain scalable ETL/ELT processes using Python and SQL.
Develop an automated reconciliation framework comparing Aztec and Canoe data sources.
Implement data quality controls, validation checks and exception management processes.
Establish formal QA, testing and approval gates across the data lifecycle.
Work closely with business stakeholders, operations teams and third-party vendors to gather requirements and deliver solutions.
Produce technical documentation and support knowledge transfer activities.
Ensure the platform is secure, scalable, auditable and production-ready.
What we're looking for 5+ years' experience in Data Engineering or Data Platform Engineering roles.
Strong commercial experience with Databricks and Azure.
Advanced Python development skills.
Strong SQL and data modelling experience.
Proven experience building cloud-based data warehouses and data platforms.
Experience designing and implementing ETL/ELT pipelines.
Experience integrating data from APIs and file-based/SFTP sources.
Strong understanding of data quality, governance and testing frameworks.
Experience delivering greenfield data platform implementations.
Ability to work independently and engage effectively with technical and business stakeholders. #J-18808-Ljbffr
