Senior Data Engineer - Nectar (SN) in London

As a Senior Data Engineer, you will play a pivotal role in designing, building and optimising the data platforms, pipelines and services that enable scalable machine learning solutions across the organisation. You will partner closely with Data Scientists, to ensure data is reliable, accessible and production‑ready. You will also contribute to engineering excellence by driving best practices, mentoring other engineers, and shaping the technical direction of data and ML workflows across our domain.

Key Responsibilities

• Lead the design and build of high-quality, scalable and reusable data pipelines using Sainsbury’s engineering standards and best practices.
• Integrate and manage data from multiple sources, ensuring consistency, integrity and quality throughout the data lifecycle.
• Provide guidance for the junior & mid Data Engineers on the best practices when building and managing data infrastructure, including data lakes, warehouses, and distributed processing systems (e.g., PySpark, Hadoop).
• Collaborate with data scientists to prepare and transform raw data into formats suitable for machine learning, including feature engineering and data augmentation.
• Implement automation tools and frameworks (CI/CD) to streamline the deployment and monitoring of machine learning models in production.
• Optimise data processing workflows and storage solutions to improve performance and reduce costs.
• Work closely with cross-functional teams, including data science, engineering, and product management, to deliver data solutions that meet business needs.
• Mentorship: junior and mid-level data engineers and provide technical guidance on best practices and emerging technologies in data engineering and machine learning and helping to enhance their skills and career growth.
• Promote a culture of knowledge sharing within the engineering teams by organising regular technical workshops, brown bag sessions, and code reviews.
• Foster a collaborative and inclusive team environment that encourages continuous learning and improvement.

Essential Criteria

• Expertise with PySpark or PyTorch for large-scale distributed data processing, including optimisation, partitioning, and debugging on managed Spark clusters (AWS EMR).
• Experience with containerisation and orchestration tools (e.g., Docker, Airflow, Kubernetes).
• Hands-on expertise with Snowflake as a cloud data warehouse, including writing and optimizing SQL and integrating securely into pipelines.
• Hands-on experience with cloud platforms (e.g., AWS, GCP, Azure).
• Strong experience with data processing frameworks (e.g., Apache Spark, Flink).
• Expertise in SQL and NoSQL databases (e.g., MySQL, PostgreSQL, MongoDB, Cassandra).
• Experience with CI/CD pipelines and automation tools like GitHub Actions.
• Understanding of monitoring and logging tools (e.g., NewRelic, Grafana).

Desirable

• Certifications: AWS Certified Big Data Specialty, Google Professional Data Engineer, or equivalent.
• Strong analytical and problem-solving skills.
• Excellent communication skills, able to explain complex concepts to non‑technical stakeholders.
• Ability to work independently as well as collaboratively within cross-functional teams.

What you'll be doing

• Provide technical direction, set standards, and lead by example in engineering excellence.
• Facilitate Scrum ceremonies when required (stand‑ups, planning, grooming).
• Communicate clearly and transparently creating an inclusive environment where diverse opinions are encouraged.

Collaborative Attitude

• Strong team player with a collaborative approach to working with cross-functional teams within the Media Agency.
• Open to feedback and willing to provide constructive criticism to others.
• Be available for the team, responding within a reasonable time frame and if not possible clearly signposting alternative contacts who can guide.
• Building a community across Media Agency. Contribute to a positive and inclusive atmosphere within the team.

Knowledge Sharing and Empowerment

• Commitment to fostering a learning culture within the team and ensuring knowledge transfer across all levels.
• Support and mentor C3s and C4s engineers by providing them opportunities to lead initiatives and contribute to the technical roadmap.
• Share domain expertise proactively and help establish the engineering direction for the team.
• Support spikes, POCs and early investigative work.
• Encourage strong developer behaviours (e.g., cameras on for collaboration, documentation, active presence).
• Lead by example in communication, visibility, accountability and role-modelling Sainsbury’s values.

What’s in It for You

• Flexible working with a balanced approach to home and office.
• Colleague discounts across Sainsbury’s, Argos and Habitat.
• Private health cover.
• Generous holiday allowance.
• Bonus scheme.
• Pension plan.
• Discounts on gyms, restaurants, holidays, retail and more.