Machine Learning Engineer | Neurobiology | Dr Albert Cardona | LMB 2611 in Cambridge

This is a fixed term position for 2 years due to time limited funding from an external grant.

Overall purpose: To lead the deployment, scaling and maintenance of machine learning systems for petabyte-scale connectomics data within the group of Dr. Albert Cardona at the MRC Laboratory of Molecular Biology. The role will focus on turning machine learning models for volume electron microscopy into robust, reproducible and observable production pipelines, supporting automated segmentation, synapse detection, proofreading and large-scale image analysis.

The post holder will design and maintain infrastructure for training, inference, monitoring, debugging and data processing across GPU/CPU clusters, on-prem storage systems and HPC environments. They will work closely with scientists, software engineers and collaborators to ensure that ML workflows are reliable, scalable, version-controlled and usable by the wider research group.

Inspire ideas for research in the context of the work of the Neurobiology Division. To play a major role in the work of the research group, through the conduct of research, introduction of new ideas, dissemination of research results, support and training of others. Contribute to the strategic development of the group and of the division overall. To be a focus of expertise on ML-based image processing, computational data analysis, systems administration and software engineering as relate to the day-to-day operations of a neuroscience research laboratory acquiring and analysing data from electron microscopes and other imaging modalities.

Main duties:

• To deploy, maintain and improve machine learning models for production-scale inference on large volume electron microscopy datasets.
• To build robust ML pipelines for training, batch inference, validation, monitoring and reprocessing across GPU and CPU compute environments.
• To debug failures across the full ML stack, including model execution, data loading, storage I/O, distributed jobs, container environments, cluster scheduling and database interactions.
• To keep up to date with developments in the field, proposing or implementing changes of direction as necessary.
• To identify, develop and apply a broad range of techniques to pursue the research objectives.
• To present your work at seminars within the laboratory and at external meetings.
• To contribute to laboratory-wide discussions on developments within the laboratory, particularly in the use of new techniques or new equipment.
• To disseminate research findings in the form of publications, presentations, and reports and thus support the QQR mission of the division.
• To train students, Postdoctoral Scientist and others and line management of group members where appropriate.
• To contribute to the MRC’s engagement with the public and in the translation of research findings into improvements in health care.

Key responsibilities: The key responsibility is to develop, deploy, maintain and improve production-scale machine learning systems for large-volume electron microscopy and connectomics data. The post holder will ensure that machine learning models can be run reliably, reproducibly and efficiently across large datasets, GPU/CPU clusters, storage systems and associated databases.

You will be expected to act independently in solving complex technical problems across the machine learning deployment stack, including data loading, model execution, distributed processing, storage I/O, containerised environments, cluster scheduling, monitoring and debugging.

To design and maintain robust infrastructure for training, inference, validation and reprocessing of machine learning models on multi-terabyte to petabyte-scale datasets. To translate prototype machine learning models and research code into reliable, version-controlled and documented production workflows that can be used by researchers and collaborators.

To build and maintain monitoring tools for pipeline performance, model output quality, job failures, data integrity, throughput, resource usage and storage bottlenecks. To work closely with scientists, software engineers, research support staff and external collaborators to ensure the smooth running of large-scale ML and data-processing workflows.

To contribute to the management and optimisation of compute and data infrastructure, including GPU servers, CPU clusters, large-scale storage systems, databases and cloud or HPC environments. To introduce and apply best practices in software engineering, MLOps, testing, documentation, reproducibility, version control and deployment.

To train and support group members in the use of deployed machine learning systems, data-processing pipelines and infrastructure tools. To ensure all software, data-processing and infrastructure work is conducted in accordance with good practice and in compliance with local policies, legal requirements and data protection standards.

To contribute to collaborations within the division, across the LMB and with external partners by providing expertise in ML deployment, scalable data processing and production infrastructure. Line management responsibilities may be undertaken where appropriate, including supporting staff development and performance management in accordance with MRC policies and procedures.

Working relationships: You will report to Dr. Albert Cardona and will liaise with other Research Support staff, Postdoctoral Scientists and students, not only in your group, but also across the LMB and with external collaborators.

Person Specification:

Academic qualifications and experience: PhD in computer science, engineering, data science, computational biology, physics, mathematics or a related quantitative discipline, or an MSc / MPhil plus significant equivalent professional experience in machine learning engineering, infrastructure engineering or large-scale data processing.

Significant experience deploying machine learning models in production or production-like environments, ideally involving computer vision, image analysis or large-scale scientific data. The candidate should have strong hands-on experience with Python, modern machine learning frameworks such as PyTorch, and robust software engineering practices including testing, documentation, version control and code review.

Extensive experience working with large-scale datasets, ideally multi-terabyte to petabyte-scale data, including data storage, data access patterns, chunked or distributed data formats, database-backed metadata systems and debugging performance bottlenecks. Experience with image data, electron microscopy, bioimage informatics or connectomics would be highly desirable.

The candidate should have experience building, deploying and maintaining ML pipelines for training, inference, validation and reprocessing across GPU and CPU infrastructure (e.g., HPC environments). They should be comfortable debugging failures across the full ML stack, including model execution, data loading, memory usage, storage I/O, distributed jobs, containers, cluster scheduling, dependency management and databases.

Experience with Linux systems administration, containers such as Docker or Singularity/Apptainer, workflow orchestration tools, SLURM or Kubernetes, and cloud platforms such as AWS or similar vendors would be desirable.

Experience with additional programming languages such as Rust, C++, Java, JavaScript, Bash or SQL would be beneficial, particularly where this has been applied to building reliable, scalable and maintainable infrastructure or data-processing systems.

Technical skills and expertise: Significant experience of and proven expertise in cutting edge research relevant to the project, and knowledge of the field. The ideal candidate will have a strong background in machine learning engineering and bioimage informatics, combining expertise in large-scale data processing and software engineering with specific experience deploying and maintaining machine learning systems in production or production-like environments.

Particularly useful would-be comprehensive experience in and knowledge of:

• Deploying machine learning models for large-scale batch inference, ideally for computer vision, image analysis or scientific imaging data.
• Developing and maintaining robust data-processing and ML pipelines for multi-terabyte to petabyte-scale datasets.
• Debugging production issues across data loading, model execution, storage I/O, memory usage, distributed jobs, containers, dependency management, databases and cluster scheduling.
• Working with GPU and CPU compute clusters, including job scheduling, resource monitoring, failure recovery and performance optimisation.
• Optimising data access patterns for large image volumes, including chunked or distributed formats such as Zarr, N5, HDF5 or similar.
• Knowledge of Drosophila larval neuroanatomy and the use of NBLAST to compare and identify neurons by morphology.
• CATMAID, Fiji, BiaPy and other open source software platforms for handling neuroscience imagery.
• Managing large datasets using system administrator skills in Linux and *BSD servers, and knowledge of databases such as Postgresql.
• The use of the Python programming language with evidence in the form of online version control repositories of software written by the applicant.
• Implementing algorithms into version-controlled software packages for use by others.
• Cloud computing with AWS or similar vendors.
• Programming languages including python, bash, R, Java, C++, SQL and JavaScript.

Additionally, the candidate will have the ability to and evidence of:

• Building reliable software, infrastructure, platforms or open-source tools.
• Write scientific manuscripts.
• Travel to and present at scientific conferences.
• Travel for work to visit other laboratories to work in collaboration.

Track record of research: This will include discoveries and significant achievements, and published papers preferably first author.

Other relevant evidence of:

• Independent working and decision making
• Excellent standards of research conduct
• Mentoring
• Originality
• Influencing skills
• Communication skills
• Leadership
• Intellectual Property

Additional information: Ability to travel internationally to scientific conferences and for extended visits to other laboratories. Applicants are required to submit a full CV and covering letter.

Corporate Responsibilities: You must at all times carry out your responsibilities with due regard to the UKRI:

• Code of Conduct
• Equality, Diversity and Inclusion policy
• Health and Safety policy
• Data Protection policy

Job descriptions should be reviewed on a regular basis and at the annual appraisal. Any changes should be made and agreed between you and your manager. The above lists are not exhaustive and you are required to undertake such duties as may reasonably be requested within the scope of the post. All employees are required to act professionally, co-operatively and flexibly in line with the requirements of the post, the MRC and UKRI. The role holder will be required to have the appropriate level of security screening/vetting required for the role. UKRI reserves the right to run or re-run security clearance as required during the course of employment.