At a Glance
- Tasks: Design and build secure data solutions using Databricks and Azure services.
- Company: Join a hyper-growth payments intelligence company in Manchester.
- Benefits: Performance-based bonuses, career advancement, and personalised development plans.
- Other info: Opportunity for equity and continuous training in the payments industry.
- Why this job: Make a real impact on innovative data engineering projects in a collaborative environment.
- Qualifications: 3+ years experience in data engineering with strong skills in Azure Databricks and Python.
The predicted salary is between 63000 - 77000 £ per year.
KEY INFORMATION
- Position: Senior Data Engineer
- Reporting to: Data Engineering Manager
- Location: Manchester
Overview of the Role:
As a Senior Data Engineer, you will be an integral part of a cross-functional feature development squad. You will play a key role in the evolution of our payments intelligence platform towards a Databricks-native and increasingly agentic operating model. You will design, build and operate secure data ingestion, orchestration and platform capabilities, using Databricks as the first-choice execution platform and Azure services where appropriate. The focus remains on production data engineering rather than AI model development: creating reliable, observable and well-governed foundations that can be operated by both engineers and automated/agentic workflows. You will work across client onboarding, data retrieval, ELT, CI/CD and platform reliability to deliver continuous business value.
JOB ROLE
Key accountability of this role: Delivery, support and continuous improvement of secure, scalable, Azure/Databricks data capabilities and infrastructure on our payments intelligence platform.
Key responsibilities will include:
- Collaborate within a cross-functional squad to define and deliver scalable, secure data and platform capabilities aligned with business outcomes.
- Design Databricks-native solutions as the default, using Spark/SQL, Lakeflow Jobs, Unity Catalog, Volumes and Databricks Asset Bundles, with Azure services used where they add value.
- Build and operate data retrieval and ingestion across APIs, SFTP/FTPS and browser automation, migrating suitable orchestration from external services into Databricks-native workflows.
- Own client data onboarding flows, including configuration, historical backfills, validation, data quality, monitoring and operational handover.
- Build reusable platform capabilities and guardrails that enable agentic workflows to safely automate onboarding, operational diagnosis and routine engineering tasks, with human approval where required.
- Engineer for production reliability through observability, alerting, retries/idempotency, failure recovery, performance and cost optimisation; investigate complex failures and drive root-cause fixes.
- Deliver changes through version control, automated testing, CI/CD and Infrastructure as Code, maintaining secure coding and deployment standards through code review.
- Apply strong security and governance practices across Unity Catalog, Entra/RBAC, secrets, credentials and data access.
- Use advanced Python with strong SQL/Spark skills to develop maintainable, reusable libraries, frameworks and data pipelines.
- Mentor data engineers and drive continuous improvement, evaluating new Databricks and automation capabilities against clear technical and business value.
WHAT WE’RE LOOKING FOR
You are a great match if:
- 3+ years professional experience as a Data Engineer, DevOps Engineer or Systems Engineer delivering production data platforms.
- Strong hands-on experience with Azure Databricks, including Spark/SQL, workflow orchestration and production lakehouse data engineering.
- Advanced proficiency in Python for data retrieval, processing, automation, integration and reusable engineering libraries.
- Experience designing and operating ETL/ELT pipelines with data quality controls, schema handling, monitoring and failure recovery.
- Expertise in CI/CD, version-controlled deployment and automated testing for data or platform workloads.
- Experience integrating data through APIs and secure file transfer (SFTP/FTPS), including secure handling of credentials and secrets.
- Strong Microsoft Azure knowledge, including ADLS, Azure DevOps and Entra/RBAC, with experience designing and deploying infrastructure.
- Strong production troubleshooting, problem-solving and operational support skills.
- Degree in Computer Science, Mathematics, Physics or other STEM related subject.
Benefits
What we offer…
- Excellent performance-based earning opportunity, including Objective-driven bonuses.
- Ability to advance career and expand professional experiences in a hyper-growth company.
- Future opportunity for equity, rewarded to high performers.
- Payments industry training and continuous training in respective role.
- Personalised individual development plan, aligned to professional goals.
Senior Data Engineer in Manchester employer: CMSPI
CMSPI is an exceptional employer that fosters a dynamic and collaborative work culture in its Manchester office, where employees are encouraged to thrive in a hybrid working environment. With a strong focus on personal and professional growth, the company offers competitive benefits including a performance-linked bonus, equity opportunities, and comprehensive wellbeing support, making it an ideal place for those looking to make a meaningful impact in digital marketing while enjoying a balanced work-life experience.
