Data Engineer

About the role

The Data Engineer sits within the Data Innovation team and works day‑to‑day on the CFlakehouse, our Databricks platform that holds the routine healthcare data behind our analytical and product work. The role is hands‑on and delivery‑focused: writing SQL and Python, building and maintaining automated pipelines, and turning raw data into structured, production‑ready material that consultants and clients can rely on. CF runs an apprentice model of development. You will learn actively from senior engineers, contribute to the team’s collective knowledge, and take on greater scope as you grow. The work spans data cleansing and validation for client engagements, building and maintaining pipelines, and contributing to CF’s technical products, including through hackathons. This is a good fit for an engineer with a couple of years of experience who wants to build a broad technical foundation in a data‑rich healthcare consultancy, work close to real client problems, and see how technical work turns into client value.

Responsibilities

• Build and maintain automated data pipelines using PySpark and Spark, with appropriate monitoring.
• Build data models and pipelines under the guidance of senior colleagues, producing production‑ready code or client‑ready material.
• Develop data quality, validation and consistency checks, and carry out data cleansing.
• Develop unit, functional and integration tests, and follow the team’s Git and GitHub workflow.
• Participate in agile ways of working: keep user stories and tasks up to date, and contribute to stand‑ups, retros and show‑and‑tells.
• Flag early when work is deviating from plan, and help identify and deliver mitigations.
• Build an understanding of healthcare data and the CFlakehouse: what is collected, how it is structured, and how new sources are brought in.
• Work with teams to bring analytical insights to client problems.
• Communicate technical solutions to non‑technical colleagues, and give and receive feedback well.
• Learn best‑practice development processes from senior colleagues, and seek out new tools and techniques to apply.
• Use AI tools to improve code quality and efficiency.
• Support business development and bid writing on technical detail, and contribute to product development through hackathons and thought leadership.

Qualifications

• Around 2 to 3 years of experience in a data engineering or comparable technical data role.
• Working SQL, used day‑to‑day on the lakehouse.
• Intermediate Python for data transformation and automation.
• Experience building automated data pipelines using PySpark or Spark, with appropriate monitoring.
• Ability to write unit, functional and integration tests.
• Comfort working to a Git and GitHub workflow.
• Curiosity about healthcare data and how it relates to care delivery and policy.
• A clear communicator who knows when to ask for support and works well in a team.

Desirable Skills

• Databricks lakehouse experience: notebooks, Workflows and Jobs, Delta Lake.
• Unity Catalog basics (will be trained).
• Awareness of cloud basics such as storage and access control, and of medallion structure, bronze to silver to gold.
• Exposure to dbt and basic dimensional modelling.

Flexible Working

We follow a hybrid working model that balances in‑person connections and remote work to drive exceptional client impact. Team members can work from home one day per week as standard. Additionally, we offer 44 remote working days per year which can be used to top up your working‑from‑home days and enable you to work from home up to two days per week, subject to client needs. Core in‑person working hours are from 10am until 4pm.

Diversity & Inclusion

We are committed to building an inclusive and supportive culture where diversity thrives. We only recruit, promote and reward our people based on their skills and contribution, without regard to gender, race, disability, religion, nationality, ethnicity, sexual orientation, age, marital status, or other characteristics. We are Disability Confident Accredited.

Benefits

• Holiday entitlement: 25 days/year for staff and 30 days/year for leadership; increases by 1 day for every year of service up to a maximum of 35 days.
• Pension: we contribute 7% of your salary, you contribute 3% (or more if you like).
• Flexible benefits programme: increase pension contributions, access a cash plan, or benefit from a ClassPass subscription.
• Annual leave purchase: employees with less than 35 days entitlement can purchase additional leave days.
• Income protection: 75% of salary in event of long‑term incapacity with a qualifying claim.
• Enhanced sick pay beyond Statutory Sick Pay (up to 12 weeks in any 12‑month period).
• Life insurance: four times basic salary in a tax‑free lump sum payable to beneficiaries in case of death in service.
• Enhanced family leave: additional pay for parents who have a baby or adopt.
• Interest‑free loan up to £10,000.
• Interest‑free season ticket loan repayable in 12 monthly instalments.
• Workplace nursery scheme: tax and NI savings.
• Employee assistance and wellness programme: telephone counselling, life coaching, interactive tools, digital content.
• Seasonal flu jabs.
• Eye care tests and vouchers at Vision Express.
• Ride to work scheme: savings on bikes and cycling accessories.
• Membership to the Health Service Journal (HSJ).