AI Backend Engineer in London

We are seeking a Backend Engineer to join our Group Data team, working as part of the AI Agent Hub delivery team. This role will help build, extend, and improve the backend services, APIs, integrations, and AI platform capabilities that support ITV's emerging internal AI platform.

The role has a primary focus on Python, FastAPI, Open WebUI backend extensions, LiteLLM integration, RAG pipelines, knowledge ingestion, model routing, secure internal system integration, evaluation workflows, and operationally robust AI services. You will work closely with frontend, platform, product, design, architecture, cyber security, and data colleagues to create secure, reliable, and maintainable backend capabilities that allow ITV teams to use foundational models, assistants, agents, multimodal tools, and internal system integrations through a single governed experience.

The AI Agent Hub is a secure, governed internal platform that enables ITV teams to explore, build, and use AI assistants, agents, multimodal tools, and workflow integrations through a unified user experience. It is intended to reduce fragmentation in AI tooling, improve governance and observability, and make it easier for teams to experiment safely and deliver value using shared platform capabilities.

As a Backend Engineer embedded into the Agent Hub delivery team, you will work day to day with software engineers, platform engineers, product, design, architecture, cyber security, and data colleagues to design and deliver the backend product and integration layer of the platform. You will help make complex capabilities such as LLM routing, knowledge ingestion, retrieval, tool calling, access control, evaluation feedback, usage analytics, and audit logging reliable, secure, observable, and useful to ITV employees.

This role is positioned initially as a stream-aligned team role embedded in delivery. As ITV's AI platform model evolves through 2026, there is a clear opportunity for the role to develop toward a more Forward Deployed Engineer style in 2027 and beyond, helping other teams adopt the platform, shaping integrations, and contributing to an enabling team capability within a broader core platform model.

What you will do:

• Build and improve backend services for the AI Agent Hub, primarily using Python, FastAPI, Open WebUI backend patterns, and OpenAI-compatible APIs.
• Develop APIs and integration services that support chat, assistant configuration, agent workflows, knowledge management, permissions, admin features, and usage visibility.
• Integrate backend services with LiteLLM or similar gateway capabilities for model routing, model aliases, virtual keys, user attribution, rate limits, budgets, retries, fallback behaviour, and usage reporting.
• Build and maintain RAG capabilities, including document ingestion, data cleaning, chunking, embeddings, retrieval, citation support, and vector store integration.
• Help improve the quality and reliability of knowledge pipelines by validating source data, detecting poor-quality inputs, and supporting evaluation-first approaches to retrieval and response quality.
• Build secure integration patterns for internal systems, APIs, knowledge sources, storage platforms, workflow automation tools, and MCP-based capabilities.
• Implement backend controls that support responsible AI use, including access control, audit logging, provenance capture, prompt and output guardrail integration (e.g. personal data masking), and human-in-the-loop review patterns.
• Work closely with frontend and platform engineers to align backend APIs with user experience needs, authentication flows, observability, deployment patterns, and operational support requirements.
• Run and debug the full application stack locally using Docker and Docker Compose, including reading logs, editing configuration, understanding image builds, and diagnosing environment issues.
• Contribute to automated testing, technical design, pull requests, runbooks, support documentation, and engineering standards.
• Over time, support wider AI platform adoption across ITV by helping teams connect data, expose tools safely, configure assistants and agents, and embed AI capabilities into existing workflows.

Skills you’ll need (minimum criteria):

• Experience building and supporting production backend services, ideally with 3+ years of applied backend engineering experience.
• Strong Python skills, including API development, testing, debugging, dependency management, and maintainable service design.
• Experience building REST APIs with FastAPI, Flask, Django, Node.js, or similar backend frameworks, with a willingness to work primarily in Python and FastAPI.
• Experience integrating backend services with third-party APIs, internal APIs, authentication providers, storage systems, and asynchronous or long-running workflows.
• Practical understanding of LLM-powered product experiences, including prompt and system context, model selection, context windows, token usage, streaming responses, error handling, and fallback states.
• Experience integrating with LLM APIs or OpenAI-compatible endpoints, or a strong interest in learning how these patterns work in a governed enterprise platform.
• Familiarity with RAG concepts such as document ingestion, chunking, embeddings, vector stores, retrieval quality, citations, and grounding responses in source material.
• Comfortable working with databases and storage technologies such as PostgreSQL, SQLite, Redis, object storage, or vector stores.
• Comfortable working with Git, pull requests, code review, automated testing, dependency updates, and local development using Docker or Docker Compose.
• Good understanding of security and privacy fundamentals for backend services, including authentication, authorisation, role-based access control, secrets management, audit logging, and safe handling of user data.
• Strong communication skills and the ability to explain trade-offs clearly, collaborate constructively, and contribute to technical discussions across product, design, engineering, architecture, cyber security, and data.

Other things we’re looking for (key criteria):

• Experience with Open WebUI backend customisation, Functions, Tools, Pipelines, or similar extension patterns in open-source AI platforms.
• Experience with LiteLLM or similar LLM gateways, including model routing, aliases, virtual keys, budgets, cost tracking, retries, fallback chains, and provider abstraction.
• Experience with asynchronous Python, background workers, job queues, event-driven processing, WebSockets, SSE, or streaming API patterns.
• Experience with data ingestion from unstructured sources such as PDFs, scripts, documents, transcripts, metadata exports, spreadsheets, or enterprise knowledge bases.
• Experience with vector stores or search technologies such as Postgres pgvector, Chroma, Qdrant, Pinecone, Weaviate, Amazon S3 Vector, OpenSearch, or hybrid keyword and vector retrieval.
• Familiarity with RAG evaluation approaches and tools such as RAGAS, DeepEval, LangSmith, or custom evaluation datasets and metrics.
• Experience with SSO and enterprise access patterns, ideally OIDC, OAuth2, SAML, Okta, Amazon Cognito, JWTs, or RBAC for internal tools.
• Experience with Kubernetes, Helm, GitHub Actions, CI/CD pipelines, environment management, or production deployment workflows.
• Familiarity with observability concepts, ideally including OpenTelemetry, Prometheus, Grafana, structured logs, traces, usage dashboards, and cost dashboards.
• Familiarity with AI security risks such as prompt injection, data leakage, unsafe tool use, over-permissive retrieval, and the need for guardrails and auditability.

Areas you may grow into:

• Building MCP servers or integrations that expose internal ITV systems, APIs, knowledge bases, and developer tools to AI clients.
• Designing more advanced RAG and retrieval strategies, including hybrid search, reranking, semantic caching, retrieval evaluation, and domain-specific chunking strategies for media and production data.
• Supporting agentic workflows that combine model calls, tools, permissions, state, memory, human approval, and audit trails.
• Working with agent frameworks such as LangGraph, LlamaIndex, CrewAI, Mastra, Bedrock Agents, Vertex AI Agent Engine, or similar technologies.
• Creating lightweight CLI tools, GitHub or GitLab integrations, code review agents, or CI/CD pipeline agents that support engineering software delivery workflows.
• Exploring model and agent lifecycle patterns, including onboarding, evaluation, release management, monitoring, rollback, cost control, and safe decommissioning.

Why this role is exciting:

This is an opportunity to help shape the backend and integration layer for a new internal AI platform at an early stage, with meaningful influence over how ITV employees securely use models, assistants, agents, internal knowledge, and business systems. You will work in a high-priority area at the intersection of backend engineering, AI, data integration, governance, observability, and internal platform delivery. The role offers the chance to turn complex AI platform capabilities into secure, reliable, and reusable services for teams across ITV. It is also a role with visible progression potential.