Senior Data Engineer

Key Responsibilities:

• Data Pipelines and ETL Processes

• ETL Design: Design and implement ETL processes within MaPS architectural patterns to extract, transform, and load data from various source systems into our reporting solutions.
• Pipeline Development: Develop and configure meta-data driven data pipelines using data orchestration tools such as Azure Data Factory and engineering tools like Apache Spark to ensure seamless data flow.
• Monitoring and Failure Recovery: Implement monitoring procedures to detect failures or unusual data profiles and establish recovery processes to maintain data integrity.

• Utilise Azure Cloud based data technology platforms, Data Lake based storage & Cloud data pipeline/orchestration tools for efficient data storage, processing, and transfer.
• Ensure the scalability and reliability of data solutions using Azure's suite of services.

• Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions that meet their needs.
• Provide technical leadership and mentorship to junior data engineers, fostering a culture of continuous learning and improvement.

• Ensure data governance policies are adhered to and maintain high standards of data quality and security.
• Develop and enforce best practices for data management and integration.
• Ensure only thoroughly tested, high quality code, architecture and resulting Data Products are delivered to the MaPS business.

• Understanding & interest of broad architectural principles underlying modern data architecture. E.g. Data Lakes, External tables, medallion architecture, loose coupling.
• Reporting & analysis tooling. An understanding of report tooling such as Power BI & the needs of analysts consuming engineered data into these tools.

You will need to demonstrate the following skills and experience:

• Proven experience as a Data Engineer, with a focus on Azure services.
• Strong expertise in designing and implementing ETL processes.
• Experience in using SQL to query and manipulate data.
• Proficiency in Azure data tooling such as Synapse Analytics, Microsoft Fabric, Azure Data Lake Storage/One Lake, and Azure Data Factory.
• Spark/Pyspark or Python skills a bonus or a willingness to develop these skills.
• Experience with monitoring and failure recovery in data pipelines.
• Excellent problem-solving skills and attention to detail.
• Strong communication and collaboration skills.

About Us:

The Money and Pensions Service (MaPS) is based in Bedford, in a recently renovated modern office. This is a wonderful opportunity for you to become an integral part of a dynamic organisation, working to help people across the UK.

At the heart of the Money and Pensions Service are our values – caring, connecting, and transforming, which are the foundation of our success. They permeate every area of our work and define all our business relationships and the way we work with each other.

Our Inclusive Working Environment:

By fostering our values, we are immensely proud of the inclusive working environment that we have created. The diversity of our people is a strength that we embrace and wish to build upon, so we are committed to attracting people of all backgrounds.

Flexible Working:

At MaPS, we take pride in our flexible approach to work. As standard, we work on a hybrid basis with a minimum of 2 days in the office per week.

Career Development:

In MaPS, we take career development seriously. We actively encourage and support applications from our existing MaPS colleagues.

What We Offer:

This role will offer the opportunity to develop your career in a Cloud only environment with a high level of technical freedom, working with other highly technical and skilled engineers.