Research Scientist - Multi-modal LLMs

At Encord, we are building the AI infrastructure of the future. One of the biggest challenges AI companies face today is data quality. The success of any AI application relies heavily on the quality of its training data, yet for most teams, this crucial step is both the most costly and time-consuming. We’re here to change that. As former computer scientists, physicists, and quants, we’ve experienced firsthand how a lack of tools to prepare quality training data impedes progress in building AI. We believe AI is at a stage similar to the early days of computing or the internet—where the potential is clear, but the surrounding tools and processes are still catching up. That’s why we started Encord. We are a talented and ambitious team of 70, working at the cutting edge of computer vision and deep learning. Backed by $30M in Series B funding from top investors like CRV and Y Combinator, we’re one of the fastest-growing companies in our space. Our platform is consistently rated the best by our customers, and we have big plans ahead.

As a Research Scientist focusing on multi-modal LLMs, you will allow all the data, metadata, and embeddings that live in our system to be explored, used, and analysed in ways no one thought possible. Although starting narrow with “smaller” multi-modal problems like improving similarity searches via metadata, we have high ambitions for this role. You will progressively work on harder problems that will improve user experience, surface the right (personalised) analytics to every customer, and put our users in the driver’s seat of a data development platform that can do things much beyond today’s standards. Tasks can include:

• Fine-tuning models to understand how our platform is used by customers.
• Employing LLM reasoning to assist customers in their data analysis tasks.
• Building tools for customers to interface naturally with our platform.

You will follow the latest research and accelerate state-of-the-art technologies to enrich customers’ data journeys. This role offers a great growth opportunity, with the potential to lead a bigger team of scientists over time in our efforts to build the ultimate data development platform.

What you will be doing:

• Building, fine-tuning, and experimenting with multi-modal LLMs to surface potential actions and analytical conclusions in a data-driven manner.
• Developing scalable and novel ways to personalise LLMs based on information from our data development platform.
• Building sophisticated RAG systems on other types of data than the usual text documents.
• Following the latest machine learning research to identify and apply new methods that improve our processes or the user experience.
• Ensuring our customers have the world’s most powerful AI-powered data development platform.

Skills for the job:

• A PhD or similarly strong academic background in machine learning, with 2+ years of hands-on experience in LLM fine-tuning, RAG systems, and prompt engineering.
• Proficiency with frameworks like PyTorch, Tensorflow, JAX, Pandas, and OpenCV.
• A solid understanding of transformer models and their common variants, loss functions, and pitfalls.
• A quick learner with a structured, organised approach to problem-solving.
• Excellent communication skills with an ability to uncover use cases and solve problems efficiently.
• Ambitious and self-motivated, with a proven track record of top performance in academic or professional settings.

Bonus skills:

• Experience working with data in the order of millions.
• Familiarity with using (and adapting) models like LLaMa and LLaVa.
• Experience with image-to-text embedding models like CLIP and SigLIP.
• Familiarity with cloud-based model training and inference.

We encourage people from all backgrounds, cultures, and skill levels to apply. It is okay to not meet all requirements listed as we are looking for individuals who are passionate and eager to learn.

What We Offer:

• Competitive salary, commission, and equity in a high-growth business.
• A collaborative, in-person culture with most of the team working in the office 3+ days a week (engineers typically work on-site Wednesdays).
• 25 days annual leave + public holidays.
• An annual learning and development budget to help you grow your skills.
• Company lunches twice a week and regular socials, including bi-annual off-sites.

At Encord, you’ll have the unique opportunity to be part of a fast-growing startup with a clear mission and vision. You’ll work on real-world AI use cases across a variety of industry verticals and get hands-on experience with cutting-edge computer vision and deep learning technologies. This is a role where you will grow quickly, take ownership of projects, and help shape the future of our company.