Overview
In this role you will engineer scalable data reconciliation pipelines on Databricks, migrating from Hadoop to cloud-native architectures on AWS. You will work at the intersection of data engineering and financial data operations, shaping a production-grade platform that supports downstream reporting, risk, and decision-making. The role focuses on modernising large-scale pipelines, improving data quality, and delivering reusable components across the data platform. You join a collaborative team that values practical engineering, architectural input, and continuous improvement, with mentoring and flexible work options.
Pay / Benefits
- 27 days annual leave (plus bank holidays)
- discretional annual bonus
- Private Medical Care & Life Insurance
- Employee Assistance Program
- Pension Plan
- Paid Parental Leave
Responsibilities
- Build and optimise large-scale Scala Spark data pipelines on Databricks for reconciliation
- Refactor legacy Hadoop pipelines to cloud-native Databricks/AWS architectures with Delta Lake and Workflows
- Tune Spark jobs to remove bottlenecks and standardise patterns
- Translate designs into detailed implementations, data models, and reusable components
- Collaborate with architects, platform, and DevOps teams on design and production releases
- Write and maintain tests (unit, integration, data validation) for quality and reliability
- Contribute to engineering standards via shared frameworks and libraries
Key requirements
- Experience with Scala Spark or PySpark or Java Spark
- Familiarity with AWS services (S3, Glue, IAM, CloudWatch, Step Functions)
- Exposure to data reconciliation or financial data domains
- Experience with CI/CD pipelines and DevOps practices
- Knowledge of data modelling concepts and orchestration patterns (beyond Databricks Workflows)
- Experience with large-scale Hadoop to cloud migration or legacy platform modernization
- Familiarity with streaming data processing (Kafka, Spark Structured Streaming)
- Experience contributing to shared engineering frameworks or reusable libraries in a platform team
- collaboration
- problem solving
- attention to detail
- Scala Spark
- PySpark
- Java Spark
Lead Data Engineer – Spark / Scala - Vice President in Belfast employer: Citigroup
Citi is an exceptional employer, offering a dynamic work environment in the heart of London where innovation meets collaboration. As a Sales Director, you will benefit from extensive training and mentorship opportunities, fostering your professional growth while working alongside industry leaders in FX Sales. With a strong commitment to integrity and inclusivity, Citi provides a supportive culture that empowers employees to excel and drive meaningful results for clients.
