Senior Data Engineer in Leeds

Purpose of the Job

Design, build, and maintain robust data systems and pipelines that support data storage, processing, and analysis on the Cloud. Work with large datasets, ensuring data quality, scalability, and performance, while collaborating.

Key Accountabilities

• Design and implement scalable, efficient, and secure data architectures, ensuring optimal data flow across systems in order to achieve high service levels of support, maintenance and development.
• Own development and change projects to ensure requirements are met in the most cost-effective manner while minimising associated risk to expected standards.
• Responsible for cloud data platform development, data modelling, shaping and technical planning.
• Mentor among the owning decision making and evaluation of requirement suitability, facilitate reliable estimates, technical project management, stakeholder management with a project.
• Ensure that resource requirements are understood and planned/estimated effectively against demand, including identification of additional temporary resource capability within projects.
• Maintain appropriate process procedures, compliance and service level monitoring, performance reporting and vendor management.
• Implement best practices around data security, privacy, and compliance for the teams compliance with cyber security and data protection and supporting along with BI lead.
• Strong stakeholder management will be required for maintaining relationships with our business users to clarify and influence requirements.
• Mentor data engineers, supporting their professional growth and development.

Knowledge

• Broad data management technical knowledge so as to be able to work across full data cycle.
• Proven experience working with AWS data technologies (S3, Redshift, Glue, Lambda, Lake formation, Cloud Formation), GitHub, CI/CD.
• Coding experience in Apache Spark, Iceberg or Python (Pandas).
• Experience in change and release management.
• Experience in Database Warehouse design and data modelling.
• Experience managing Data Migration projects.
• Cloud data platform development and deployment.
• Experience of performance tuning in a variety of database settings.
• Experience of Infrastructure as code practices.
• Proven ability to organise and produce work within deadlines.

Skills

• Good project and people management skills.
• Excellent data manipulation and analysis skills using a variety of tools including SQL, Python, AWS services and the MSBI stack.
• Ability to prioritise and be flexible to change those priorities at short notice.
• Able to provide appropriate and understandable data to a wide ranging audience.
• Well-developed and professional communication skills.
• Strong analytical skills - ability to create models and analyse data in order to solve complex problems or reinforce commercial decisions.
• Able to understand business processes and how this is achieved/influenced by technology.
• Must be able to work as part of a collaborative team to solve problems and assist other colleagues.
• Ability to learn new technologies, programs and procedures.

Technical Essentials

• Expertise across data warehouse and ETL/ELT development in AWS preferred with experience in the following: Strong experience in some of the AWS services like Redshift, Lambda, S3, Step Functions, Batch, Cloud formation, Lake Formation, Code Build, CI/CD, GitHub, IAM, SQS, SNS, Aurora DB.
• Good experience with DBT, Apache Iceberg, Docker, Microsoft BI stack (nice to have).
• Experience in data warehouse design (Kimball and lake house, medallion and data vault) is a definite preference as is knowledge of other data tools and programming languages such as Python & Spark and strong SQL experience.
• Experience in building Data lake and building CI/CD data pipelines.
• A candidate is expected to understand and can demonstrate experience across the delivery lifecycle and understand both Agile and Waterfall methods and when to apply these.

Experience

• This position requires several years of practical experience in a similar environment.
• We require a good balance of technical and personal/soft skills so successful candidates can be fully effective immediately.
• Proven experience in developing, delivering and maintaining tactical and enterprise data management solutions.
• Proven experience in assessing the impact of proposed changes on production solutions.
• Proven experience in managing and developing a team of technical experts to deliver business outcomes and meet performance criteria.
• Exposure to Energy markets, Energy Supply industry sector.
• Developing and implementing operational processes and procedures.

Seniority level: Mid-Senior level

Employment type: Full-time

Job function: Staffing and Recruiting