Find a job 
Featured Companies 
Featured Universities 
For Companies At a Glance
- Tasks: Lead a data warehouse migration to a cloud-native Databricks Data Lake on Azure.
- Company: Scrumconnect is a top tech consultancy impacting UK public services and saving taxpayers millions.
- Benefits: Enjoy 28 days holiday, health cover, birthday leave, and on-the-job training.
- Why this job: Join a collaborative team making a real difference in public services with innovative data solutions.
- Qualifications: Expertise in data engineering, Azure, and experience in public sector data initiatives required.
- Other info: Flexible hybrid work environment with opportunities for mentorship and professional growth.
The predicted salary is between 42000 - 56000 £ per year.
Lead Data Engineer
Senior Level – SFIA5
Salary: 350- 400 Inside IR 35(dependent on experience)
Location: Coventry/Hybrid
About Scrumconnect:
Scrumconnect is a leading force in technology consultancy, proudly contributing to over 20% of the UK’s most significant citizen-facing public services. Our award-winning team has made a substantial impact, delivering more than 64 services in the past two years alone. This work has not only reached over 50 million citizens but also achieved considerable savings for the taxpayer, amounting to over £25 million. At Scrumconnect, we foster a community of talented consultants who thrive on collaboration, sharing knowledge, and continuous learning to address and solve complex challenges. Our mission is to combine advanced software engineering, human-focused design, and data-driven insights to deliver unparalleled service to our clients.
About the role:
We are seeking a highly skilled and experienced Lead Data Engineer to lead a critical data warehouse migration project from an on-premises environment to a cloud-native Databricks Data Lake on Azure. This role will be instrumental in designing, developing and implementing robust data solutions that will power critical decision-making processes within the public sector.
This role requires a proactive individual who thrives in an agile environment and can drive the product process from research to execution.
Key Responsibilities
- Technical Leadership:
- Provide technical leadership and guidance to a team of data engineers and analysts.
- Define and enforce data engineering best practices, standards, and methodologies.
- Oversee the design, development, and implementation of data pipelines, ETL processes, and data models.
- Data Warehouse Migration:
- Lead the migration of the existing data warehouse to a cloud-native Databricks Data Lake on Azure.
- Assess the current data warehouse infrastructure and identify migration strategies.
- Design and implement data migration plans, including data extraction, transformation, and loading (ETL) processes.
- Optimize data pipelines for performance and scalability.
- Data Lake Design and Implementation:
- Design and implement a robust data lake architecture on Azure Databricks.
- Define data ingestion, storage, and processing strategies.
- Ensure data quality, security, and privacy standards are met.
- Data Engineering and Development:
- Develop and maintain data pipelines using Databricks, Azure Data Factory, and other relevant tools.
- Write efficient and scalable data processing code in Python, or SQL (PySpark and SparkSQL).
- Collaborate with data analysts and business stakeholders to understand data requirements and translate them into technical solutions.
- Data Governance and Security:
- Implement data governance policies and procedures to ensure data quality and integrity.
- Enforce data security best practices, including access controls, encryption, and data masking.
- Monitor data pipelines and systems for performance and security issues.
- Power BI Integration:
- Integrate Databricks with Power BI to enable advanced analytics and reporting.
- Develop custom visualizations and dashboards to provide actionable insights.
- Stakeholder Management:
- Communicate effectively with technical and non-technical stakeholders.
- Present complex technical concepts in a clear and concise manner.
- Manage project timelines and deliverables.
Skills needed for this role:
Skills needed for an SFIA Level 5 Lead Data Engineer in the UK public sector typically include:
Technical Skills:
- Expert-level: Data development process, Data integration design, Data modelling
- Practitioner-level: Data analysis and synthesis, Data innovation, Metadata management, Problem resolution (data), Programming and build (data engineering), Technical understanding
- Communicating between the technical and non-technical: Practitioner level
Soft Skills:
Leadership and Collaboration
- Team Collaboration: Ability to work effectively with multidisciplinary teams, including developers, designers, delivery managers, and stakeholders.
- Stakeholder Management: Strong skills in communicating technical concepts to non-technical audiences and influencing decisions.
- Mentorship: Capability to guide and support less experienced team members, fostering knowledge sharing and skill development.
Agile and Delivery Focus
- Agile Working: Experience working in agile teams, contributing to ceremonies, and integrating technical goals into iterative delivery cycles.
- Delivery Management: Ability to prioritise tasks, manage technical risks, and ensure timely delivery of high-quality solutions.
Analytical and Problem-Solving Skills
- Requirements Analysis: Proficiency in translating business and user needs into technical requirements.
- Problem Solving: Skilled in diagnosing complex technical issues and implementing effective solutions.
- Data-Driven Decisions: Ability to use data, analytics, and performance insights to guide architectural decisions.
Knowledge of Public Sector Standards
- Government Digital Service (GDS): Familiarity with GDS service standards and the Technology Code of Practice.
These skills reflect the need for both technical depth and the ability to navigate the unique demands of the UK public sector environment.
Desired Qualifications
- Certifications in Azure, Databricks, or related technologies. Example
- Experience with public sector data initiatives and compliance requirements.
- Knowledge of machine learning and artificial intelligence concepts.
What our offer includes
- 28 days holiday inc. bank holidays
- 1 day Birthday leave after 1 year service
- 2 additional days after 2 years service
- Pension: 4% employee, 3% employer
- BUPA Health Cover
- AIG Life Cover
- Rewards Gateway
- On job training
Where you’ll work
Your working time at Scrumconnect will be split between multiple locations, including from our HQ and hub locations, client site or home. Travel requirements vary in frequency and take into account requirements of your work, our clients and the team.
We welcome candidates from all identities, attributes and backgrounds to thrive with us. The diversity of our people should be reflected in the impact we deliver.
Join us at Scrumconnect, where your highly demonstrable skills and expertise will drive the future of user-centred design in public services.
Lead Data Engineer employer: Scrumconnect Consulting
Contact Detail:
Scrumconnect Consulting Recruiting Team
StudySmarter Expert Advice 🤫
We think this is how you could land Lead Data Engineer
✨Tip Number 1
Familiarize yourself with Azure and Databricks, as these are crucial for the role. Consider taking online courses or certifications to deepen your understanding and showcase your commitment to mastering these technologies.
✨Tip Number 2
Highlight your experience in leading data migration projects, especially in cloud environments. Be prepared to discuss specific challenges you faced and how you overcame them during the interview.
✨Tip Number 3
Demonstrate your ability to communicate complex technical concepts to non-technical stakeholders. Prepare examples of how you've successfully managed stakeholder expectations and facilitated collaboration in previous roles.
✨Tip Number 4
Showcase your leadership skills by discussing your experience mentoring junior team members. Highlight any initiatives you've taken to foster a collaborative environment and promote knowledge sharing within your team.
We think you need these skills to ace Lead Data Engineer
Some tips for your application 🫡
Tailor Your CV: Make sure your CV highlights relevant experience in data engineering, particularly with cloud technologies like Azure and Databricks. Emphasize your leadership skills and any experience you have in agile environments.
Craft a Compelling Cover Letter: In your cover letter, express your passion for data engineering and how your skills align with Scrumconnect's mission. Mention specific projects or experiences that demonstrate your ability to lead data migration and implement data solutions.
Showcase Technical Skills: Clearly outline your technical skills related to data development processes, data integration design, and programming languages such as Python and SQL. Provide examples of how you've applied these skills in previous roles.
Highlight Soft Skills: Don't forget to mention your soft skills, especially in leadership, collaboration, and stakeholder management. Provide examples of how you've effectively communicated complex technical concepts to non-technical audiences.
How to prepare for a job interview at Scrumconnect Consulting
✨Showcase Your Technical Leadership
As a Lead Data Engineer, it's crucial to demonstrate your ability to provide technical leadership. Prepare examples of how you've guided teams in implementing best practices and methodologies in data engineering.
✨Discuss Migration Strategies
Be ready to talk about your experience with data warehouse migrations, especially to cloud-native environments like Azure Databricks. Highlight specific strategies you've used for data extraction, transformation, and loading (ETL) processes.
✨Emphasize Collaboration Skills
Scrumconnect values collaboration across multidisciplinary teams. Share instances where you've effectively communicated complex technical concepts to non-technical stakeholders and how you managed project timelines.
✨Prepare for Problem-Solving Scenarios
Expect questions that assess your analytical and problem-solving skills. Be prepared to discuss how you've diagnosed complex technical issues and the solutions you implemented, particularly in an agile environment.
