Senior Data Engineer – Data Products & AI

Beacon Intelligence delivers data and insights that help R&D scientists develop better pharmaceuticals, faster than ever before.

Our focus is simple: we curate high-accuracy data enriched and transformed with AI. This is delivered through a range of platforms, increasingly via AI-native products and features that enable customers to extract insights faster than ever before.

This is a rare opportunity to make a meaningful impact on patients’ lives worldwide.

The Role

As a Senior Data/AI Engineer, you will play a central role in building and scaling our data products.

This is not a traditional back-end data engineering role. You will work closely with product, commercial and technical colleagues, transforming datasets into commercial, customer-facing solutions.

You will combine deep technical expertise with a strong understanding of how data can be structured, enriched and operationalised through AI systems.

What You’ll Be Doing

Data Product Development (Core Focus)

• Design and build scalable data products underpinning Beacon’s commercial offerings
• Transform raw and third-party data into structured, enriched, product-ready datasets
• Partner with product and commercial teams to define how data is packaged, accessed and monetised
• Enable delivery via APIs, internal tools and customer-facing platforms
• Apply AI and LLM capabilities to enrich and enhance data (e.g. classification, tagging, summarisation, insight generation)
• Design and build LLM-powered product features
• Integrate RAG into pipelines to improve data quality and unlock new features
• Support development of AI-driven products such as recommendation engines, search and insight tools
• Ensure systems are optimised, well-governed and resilient
• Support deployment and lifecycle management of data and AI systems
• Own and evolve the data platform architecture (Databricks, Azure, Airbyte) to support scalability
• Own performance and cost optimisation across Azure Databricks and supporting Azure services, including compute selection, autoscaling, workload monitoring and resource-efficiency improvements.
• Build and maintain robust data pipelines (batch and streaming) delivering reliable, production-ready datasets
• Ensure high standards in data quality, testing and observability
• Improve efficiency through automation, CI/CD and engineering best practice

Commercial & Stakeholder Impact

• Partner with senior stakeholders to identify high-value data product opportunities
• Translate business needs into practical, scalable data and AI solutions
• Act as a bridge between technical teams and commercial/product stakeholders
• Contribute to prioritisation based on business impact

Leadership & Growth

• Shape the evolution of our data product and AI strategy
• Mentor junior engineers and promote a strong engineering and product mindset
• Help build a high-performing, commercially aware data and AI engineering team

What We’re Looking For

Core Engineering Experience

• Deep, hands‑on experience designing, building and operating production data platforms using Azure Databricks and Azure Cloud environments
• Strong experience with Apache Spark, PySpark, Spark SQL and Delta Lake, including incremental processing, schema evolution, performance optimisation and batch and streaming workloads.
• Strong experience implementing Unity Catalog, including catalog and schema design, access control, data lineage, discovery and governance of production data assets.
• Proven experience designing lakehouse and Medallion architectures and converting raw and third‑party data into governed, reusable data products.
• Advanced Python and SQL, focused on production‑quality, scalable solutions
• Proven experience designing and building data pipelines and models
• Experience working with APIs and data delivery mechanisms (critical for productisation)
• Experience deploying Databricks workloads through automated CI/CD, preferably using Azure DevOps or GitHub Actions, automated testing and Databricks Declarative Automation Bundles.
• Experience integrating Azure Databricks with ADLS Gen2, Azure identity, Key Vault, APIs and external data sources.
• Experience building or supporting data products or customer‑facing data solutions
• Strong understanding of how to structure, expose and scale data for end users or clients
• Ability to think beyond pipelines and focus on commercial value
• Experience working closely with product or commercial teams
• Familiarity with generative AI / LLMs (Azure OpenAI, LangChain etc.) and experience applying them to real‑world use cases
• Experience evaluating, monitoring and optimising AI systems
• Experience working with unstructured data (text, documents, web data)
• Familiarity with embeddings, vector databases or semantic search
• Understanding of AI governance and data ethics
• Experience with Airbyte or modern data ingestion tools
• Experience with Terraform or infrastructure‑as‑code
• Exposure to LLM frameworks (e.g. LangChain, LlamaIndex)
• Experience with experimentation or A/B testing frameworks
• Background in building data‑driven / AI‑enabled products
• Domain experience in life sciences

How You Work

• Focused on shipping products, not just experimentation
• Think in terms of products, not just pipelines
• Commercially aware and motivated by impact
• Combine technical excellence with pragmatism and pace
• Collaborate effectively across disciplines and influence stakeholders
• Curious about AI and emerging technologies, with a focus on reliable, scalable delivery

Why Join Beacon Intelligence

• Build products, not just infrastructure – directly helping scientists treat and cure diseases
• Shape our AI future – define how we embed AI into products and workflows
• High‑impact role – influence product strategy and commercial performance
• Modern, evolving stack – work with cutting‑edge tools across data engineering and AI
• Growth and ownership – join early and grow as we scale our capabilities
• Collaborative environment – stakeholders who actively value data and insight

Location & Working Pattern

• London (hybrid working) – 2 days per week minimum in office
• Flexible working arrangements available

60% hands‑on technical

20% management and mentoring

Pay range and compensation package

Salary range: £80,000 to £90,000 + Bonus

Equal Opportunity Statement

Beacon Intelligence is committed to diversity and inclusivity in the workplace.

#J-18808-Ljbffr