AI Backend Engineer

We are seeking a Backend Engineer to join our Group Data team, working as part of the AI Agent Hub delivery team. This role will help build, extend, and improve the backend services, APIs, integrations, and AI platform capabilities that support ITV's emerging internal AI platform. The role has a primary focus on Python, FastAPI, Open WebUI backend extensions, LiteLLM integration, RAG pipelines, knowledge ingestion, model routing, secure internal system integration, evaluation workflows, and operationally robust AI services.

You will work closely with frontend, platform, product, design, architecture, cyber security, and data colleagues to create secure, reliable, and maintainable backend capabilities that allow ITV teams to use foundational models, assistants, agents, multimodal tools, and internal system integrations through a single governed experience. The role sits initially in a small, hands‑on, stream‑aligned delivery team focused on shipping a high‑value internal product for ITV Studios and the wider business.

What you will do:

• Build and improve backend services for the AI Agent Hub, primarily using Python, FastAPI, Open WebUI backend patterns, and OpenAI-compatible APIs.
• Develop APIs and integration services that support chat, assistant configuration, agent workflows, knowledge management, permissions, admin features, and usage visibility.
• Integrate backend services with LiteLLM or similar gateway capabilities for model routing, model aliases, virtual keys, user attribution, rate limits, budgets, retries, fallback behaviour, and usage reporting.
• Build and maintain RAG capabilities, including document ingestion, data cleaning, chunking, embeddings, retrieval, citation support, and vector store integration.
• Help improve the quality and reliability of knowledge pipelines by validating source data, detecting poor‑quality inputs, and supporting evaluation‑first approaches to retrieval and response quality.
• Build secure integration patterns for internal systems, APIs, knowledge sources, storage platforms, workflow automation tools, and MCP‑based capabilities.
• Implement backend controls that support responsible AI use, including access control, audit logging, provenance capture, prompt and output guardrail integration (e.g. personal data masking), and human‑in‑the‑loop review patterns.
• Work closely with frontend and platform engineers to align backend APIs with user experience needs, authentication flows, observability, deployment patterns, and operational support requirements.
• Run and debug the full application stack locally using Docker and Docker Compose, including reading logs, editing configuration, understanding image builds, and diagnosing environment issues.
• Contribute to automated testing, technical design, pull requests, runbooks, support documentation, and engineering standards.
• Over time, support wider AI platform adoption across ITV by helping teams connect data, expose tools safely, configure assistants and agents, and embed AI capabilities into existing workflows.
• Design more advanced RAG and retrieval strategies, including hybrid search, reranking, semantic caching, retrieval evaluation, and domain‑specific chunking strategies for media and production data.
• Support agentic workflows that combine model calls, tools, permissions, state, memory, human approval, and audit trails.
• Work with agent frameworks such as LangGraph, LlamaIndex, CrewAI, Mastra, Bedrock Agents, Vertex AI Agent Engine, or similar technologies.
• Create lightweight CLI tools, GitHub or GitLab integrations, code review agents, or CI/CD pipeline agents that support engineering software delivery workflows.
• Explore model and agent lifecycle patterns, including onboarding, evaluation, release management, monitoring, rollback, cost control, and safe decommissioning.

Why this role is exciting:

This is an opportunity to help shape the backend and integration layer for a new internal AI platform at an early stage, with meaningful influence over how ITV employees securely use models, assistants, agents, internal knowledge, and business systems. You will work in a high‑priority area at the intersection of backend engineering, AI, data integration, governance, observability, and internal platform delivery.

Experience required:

• Experience building and supporting production backend services, ideally with 3+ years of applied backend engineering experience.
• Strong Python skills, including API development, testing, debugging, dependency management, and maintainable service design.
• Experience building REST APIs with FastAPI, Flask, Django, Node.js, or similar backend frameworks, with a willingness to work primarily in Python and FastAPI.
• Experience integrating backend services with third‑party APIs, internal APIs, authentication providers, storage systems, and asynchronous or long‑running workflows.
• Practical understanding of LLM‑powered product experiences, including prompt and system context, model selection, context windows, token usage, streaming responses, error handling, and fallback states.
• Experience integrating with LLM APIs or OpenAI‑compatible endpoints, or a strong interest in learning how these patterns work in a governed enterprise platform.
• Familiarity with RAG concepts such as document ingestion, chunking, embeddings, vector stores, retrieval quality, citations, and grounding responses in source material.
• Comfortable working with databases and storage technologies such as PostgreSQL, SQLite, Redis, object storage, or vector stores.
• Comfortable working with Git, pull requests, code review, automated testing, dependency updates, and local development using Docker or Docker Compose.
• Good understanding of security and privacy fundamentals for backend services, including authentication, authorisation, role‑based access control, secrets management, audit logging, and safe handling of user data.
• Strong communication skills and the ability to explain trade‑offs clearly, collaborate constructively, and contribute to technical discussions across product, design, engineering, architecture, cyber security, and data.

Other things we're looking for (key criteria):

• Experience with Open WebUI backend customisation, Functions, Tools, Pipelines, or similar extension patterns in open‑source AI platforms.
• Experience with LiteLLM or similar LLM gateways, including model routing, aliases, virtual keys, budgets, cost tracking, retries, fallback chains, and provider abstraction.
• Experience with asynchronous Python, background workers, job queues, event‑driven processing, WebSockets, SSE, or streaming API patterns.
• Experience with data ingestion from unstructured sources such as PDFs, scripts, documents, transcripts, metadata exports, spreadsheets, or enterprise knowledge bases.
• Experience with vector stores or search technologies such as Postgres pgvector, Chroma, Qdrant, Pinecone, Weaviate, Amazon S3 Vector, OpenSearch, or hybrid keyword and vector retrieval.
• Familiarity with RAG evaluation approaches and tools such as RAGAS, DeepEval, LangSmith, or custom evaluation datasets and metrics.
• Experience with SSO and enterprise access patterns, ideally OIDC, OAuth2, SAML, Okta, Amazon Cognito, JWTs, or RBAC for internal tools.
• Experience with Kubernetes, Helm, GitHub Actions, CI/CD pipelines, environment management, or production deployment workflows.
• Familiarity with observability concepts, ideally including OpenTelemetry, Prometheus, Grafana, structured logs, traces, usage dashboards, and cost dashboards.
• Familiarity with AI security risks such as prompt injection, data leakage, unsafe tool use, over‑permissive retrieval, and the need for guardrails and auditability.

Benefits:

• Flexible working with a range of options.
• Generous holiday allowance, plus you can buy more.
• Annual bonus opportunity.
• Competitive pension contribution.
• Save as you earn – with an opportunity to buy ITV shares.
• Wellbeing and volunteering days plus a wide range of opportunities to help you live a balanced and healthy life.

Diversity & Inclusion:

ITV is for everyone. We create and showcase content by, with and for everyone, connecting and reflecting modern audiences. We are committed to championing diversity, equity, and inclusion across our organisation and in everything we do. As part of this commitment, we actively encourage applications for this role from candidates who are Deaf, Disabled and Neurodivergent, People of Colour, and candidates from working class backgrounds. As a Disability Confident Leader, if you meet the minimum criteria for a role and you have declared that you are disabled, we’ll guarantee to take you to the next stage. We’re happy to discuss any support/personalisation you may need during our application and selection process as part of our reasonable adjustments.