We’re looking for a Data Ingestion Engineer with experience working in the robotics industry. You will work with the Data team to improve the reliability, efficiency and throughput of the pipelines that move real-world driving data through the client.
Key Responsibilities:
- Debug and fix failing or blocked ingestion pipelines.
- Investigate issues caused by corrupt, malformed or unexpected data.
- Help make our pipelines more resilient so individual bad data segments do not block retries and queue management.
- Improve how we handle varied data formats from partners, suppliers and third-party sources.
- Optimise Spark jobs and data processing pipelines for throughput and compute
- Reduce operational toil around failed jobs, stalled pipelines and manual interventions.
- Work on high-volume batch processing systems where throughput, reliability and cost all matter.
- Help prioritise and unblock important datasets for downstream annotation, data science and model training teams.
- Partner with permanent engineers to keep critical ingestion work moving while longer-term platform improvements are developed.
- Contribute pragmatic improvements that make the system easier to operate, scale and trust.
- Strong production experience with Apache Spark.
- Experience building, debugging or operating large-scale data ingestion, ETL or data
- Experience with distributed data processing systems.
- Ability to optimise jobs for throughput, compute efficiency and reliability.
- Experience working in the robotics sector
#J-18808-Ljbffr
Data Ingestion Engineer in England employer: Cognitive Group | Part of the Focus Cloud Group
As a Senior D365 CE Functional Consultant in London, you'll join a dynamic Microsoft partner that values innovation and collaboration. The company fosters a supportive work culture where your insights and leadership are not only welcomed but essential, offering you the chance to influence project outcomes and mentor fellow consultants. With access to cutting-edge technologies and a commitment to professional development, this role provides a unique opportunity for meaningful growth in a vibrant hybrid work environment.
Contact Details:
Cognitive Group | Part of the Focus Cloud Group Recruitment Team
