b Overview /b p In this role you will own and improve the data ingestion layer for Wayve’s autonomous driving platform. You will debug failing pipelines, handle corrupt data, and optimize Spark-based ETL workflows to scale with petabytes of real-world driving data. You’ll build resilient pipelines that isolate bad data and streamline multi-step workflows for annotation, data science, and model training. The role offers impact across data engineering, reliability, and performance, within a fast-moving AI platform focused on safe and scalable autonomous driving. /p b Pay / Benefits /b ul li equity participation /li li relocation support and visa sponsorship /li li hybrid working with core hours /li li learning and development budgets /li li health insurance and dental /li li maternity/paternity leave and wellbeing resources /li /ul b Responsibilities /b ul li Debug and unblock failing ingestion pipelines, stalled jobs and processing queues /li li Investigate corrupt, malformed or unexpected data from diverse sources /li li Optimize Spark jobs and batch-processing pipelines for throughput, reliability and compute efficiency /li li Design and implement more resilient pipelines to prevent a single bad segment from blocking broader workflows /li li Improve monitoring, retries and failure recovery across multi-step ingestion workflows /li li Prioritize and unblock datasets for downstream annotation, data science and model-training teams and contribute to technical direction of the ingestion platform /li li Work on high-volume pipelines for annotation, data science, model training and evaluation /li li Develop resilient systems that isolate problematic data and orchestrate complex dependencies, queues, retries and priority data flows /li li Make pragmatic improvements to performance, cost and operability of the Spark-based data platform /li li Ingest varied data formats from partners, suppliers and other third-party sources /li /ul b Key requirements /b ul li Strong production experience with Apache Spark and Python /li li Built, operated or debugged large-scale ingestion, ETL or distributed data-processing pipelines /li li Ability to diagnose production failures involving corrupt data, stalled queues, retries and complex dependencies /li li Understanding of optimizing high-throughput systems for reliability, performance and cost /li li Ability to learn an existing system quickly, work independently and deliver practical improvements in a fast-moving environment /li li Experience with Airflow, Flyte, Databricks, Delta Lake, Scala or Java would be an advantage /li /ul ul li Independent and proactive problem-solver /li li Good debugging mindset and pragmatic mindset /li li Ability to deliver practical improvements in a fast-moving environment /li li Apache Spark /li li Python /li li ETL pipelines /li /ul
Software Engineer, Data Ingestion employer: wayve
Wayve is an exceptional employer, offering a dynamic and inclusive work culture that fosters innovation and collaboration. With a focus on employee growth, you will have the opportunity to work alongside talented AI engineers in a hybrid environment, contributing to groundbreaking advancements in autonomous driving while enjoying the vibrant atmosphere of London. Join us to make a meaningful impact in the field of AI and data science.
