At a Glance
- Tasks: Design and maintain scalable data pipelines using Azure Databricks and Spark.
- Company: Join a leading financial services firm focused on data innovation.
- Benefits: Attractive salary, flexible working options, and opportunities for professional growth.
- Other info: Collaborative culture with a focus on continuous learning and development.
- Why this job: Make an impact by optimising data processes in a dynamic tech environment.
- Qualifications: 10+ years in data engineering with strong Python and Spark skills required.
The predicted salary is between 80000 - 100000 £ per year.
Responsibilities
- Design, develop, and maintain robust and scalable data pipelines for ingesting, transforming, and loading data from various sources (e. g., APIs, databases, financial data providers) into the Azure Databricks platform.
- Optimise data pipelines for performance, efficiency, and cost-effectiveness.
- Implement data quality checks and validation rules within data pipelines.
- Implement complex data transformations using Spark (Py Spark or Scala) and other; develop and maintain data processing logic for cleaning, enriching, and aggregating data.
- Ensure data consistency and accuracy throughout the data lifecycle.
- Work extensively with Azure Databricks Unity Catalog, including Delta Lake, Spark SQL, and other relevant services.
- Implement best practices for Databricks development and deployment.
- Optimise Databricks workloads for performance and cost.
- Integrate data from various sources, including relational databases, APIs, and streaming.
- Implement data integration patterns and best practices.
- Work with API developers to ensure seamless data exchange.
- Data Quality & Governance
- Hands on experience to use Azure Purview for data quality and data governance.
- Implement data quality monitoring and alerting processes.
- Work with data governance teams to ensure compliance with data governance policies and standards.
- Implement data lineage tracking and metadata management processes.
- Collaborate closely with data scientists, economists, and other technical teams to understand data requirements and translate them into technical solutions.
- Communicate technical concepts effectively to both technical and non-technical.
- Participate in code reviews and knowledge sharing sessions.
- Implement automation for data pipeline deployments and other data engineering tasks.
- Work with Dev Ops teams to implement and build CI/CD pipelines for environmental deployments.
- Promote and implement Dev Ops best practices.
Qualifications
- 10+ years of experience in data engineering, with at least 3+ years of hands‑on experience with
- Strong proficiency in Python and Spark (Py Spark) or Scala.
- Deep understanding of data warehousing principles, data modelling techniques, and data integration patterns.
- Extensive experience with Azure data services, including Azure Data Factory, Azure Blob Storage,
- Experience working with large datasets and complex data pipelines.
- Experience with data architecture design and data pipeline optimization.
- Proven expertise with Databricks, including hands‑on implementation experience and certifications.
- Experience with SQL and No SQL databases.
- Experience with data quality and data governance processes.
- Experience with version control systems (e. g., Git).
- Experience with Agile development methodologies.
- Excellent communication, interpersonal, and problem-solving skills.
- Experience with streaming data technologies (e. g., Kafka, Azure Event Hubs).
- Experience with data visualisation tools (e. g., Tableau, Power BI).
- Experience with Dev Ops tools and practices (e. g., Azure Dev Ops, Jenkins, Docker, Kubernetes).
- Experience working in a financial services or economic data environment.
- Azure certifications related to data engineering (e. g., Azure Data Engineer Associate).
- #J-18808-Ljbffr
Azure Data Engineer in London employer: Qualient Technology Solutions UK Limited
As a leading player in the investment banking sector, our company offers an exceptional work environment for Data Modelers, fostering innovation and collaboration. With a strong emphasis on employee growth, we provide access to cutting-edge tools and resources, alongside opportunities for professional development in a dynamic capital markets landscape. Our inclusive culture values diverse perspectives, ensuring that every team member can contribute meaningfully while enjoying a supportive and engaging workplace.
Contact Details:
Qualient Technology Solutions UK Limited Recruitment Team
We think you need these skills to ace Azure Data Engineer in London
SQL
Python
Data Pipeline Development
Problem-Solving Skills
Data Engineering
Communication Skills
API Integration
