At a Glance
- Tasks: Design and build scalable data pipelines using Databricks and cloud-native platforms.
- Company: Join a leading tech firm in London focused on innovative data solutions.
- Benefits: Permanent position with competitive salary and opportunities for professional growth.
- Other info: Collaborative environment with a focus on data quality and governance.
- Why this job: Make an impact by transforming legacy systems into modern data architectures.
- Qualifications: Strong experience with Databricks, streaming technologies, and AWS data services.
The predicted salary is between 80000 - 100000 £ per year.
Role –Senior Databricks Architect
Technology – Databricks (Spark, PySpark/SQL) and data lake architectures
Location – London, UK
Duration-Permanent
Job Description
We are looking for a Data Architect to design, build, and operate scalable data pipelines and data products, enabling a modern data mesh architecture. The role focuses on integrating legacy systems with cloud-native platforms to support analytics, reporting, and domain-owned data products.
Primary Skills
• Strong experience with Databricks (Spark, PySpark/SQL) and data lake architectures.
• Hands-on experience with streaming technologies, especially Kafka / Confluent Kafka.
• Experience integrating legacy data sources such as DB2 (mainframe) and SQL Server.
• Working knowledge of AWS data services, especially S3.
• Understanding of data product thinking, data mesh concepts, and modern data engineering best practices.
Key Responsibilities
• Build and maintain batch and streaming data pipelines from legacy mainframe DB2 and on prem SQL Server systems into an AWS-based data lake.
• Implement real-time ingestion using Confluent Kafka and land data in Amazon S3.
• Develop data transformations and analytics pipelines in Databricks using the medallion architecture (Bronze, Silver, Gold).
• Design and deliver high-quality, reusable data products aligned to data mesh principles.
• Ensure data quality, reliability, security, and governance across pipelines.
• Collaborate with domain teams, platform teams, and stakeholders to enable self-serve analytics and consumption.
Good to have
• Experience in banking or financial services data platforms.
• Familiarity with data governance, metadata management, and regulatory reporting
Databricks Architect in City of London employer: Avance Consulting
Avance Consulting is an exceptional employer that fosters a dynamic work culture in the heart of Edinburgh, offering employees the chance to lead innovative technological changes in customer communications. With a strong emphasis on professional development and cross-functional collaboration, team members are encouraged to grow their skills while working with cutting-edge tools in a supportive environment. The company values diversity and inclusion, making it a rewarding place for those looking to make a meaningful impact in sectors like Banking and Insurance.
