Databricks Specialist Data, Analytics & AI | Senior Individual Contributor Location: Manchester(preferred) And London 3 days onsite Shape the Future of Enterprise Data & AI Perm role 60000-80000 annually We are looking for an experienced Databricks Specialist to become the technical owner of a large-scale Azure Databricks platform at the heart of a rapidly growing Data, Analytics & AI practice. This is a high-impact opportunity for a Databricks expert who enjoys combining platform engineering, data governance, cloud architecture, and AI innovation. You will play a pivotal role in driving the adoption of modern data engineering standards, enterprise governance frameworks, and next-generation AI capabilities across both internal platforms and client-facing solutions. Working closely with architects, data engineers, security teams, and business stakeholders, you will act as the Databricks Subject Matter Expert, ensuring the platform remains secure, scalable, cost-efficient, and aligned with strategic business objectives. What You'll Be Doing Databricks Platform Ownership Lead administration and optimisation of Azure Databricks environments. Manage workspace configuration, cluster policies, instance pools, and runtime upgrades. Define and implement platform governance, performance standards, and cost controls. Monitor platform health, performance, and operational stability. Unity Catalog & Data Governance Own Unity Catalog administration including metastores, catalogs, schemas, and access controls. Design enterprise-grade governance frameworks and RBAC models. Support audit logging, lineage tracking, compliance reporting, and data access reviews. Ensure governance standards align with security and regulatory requirements. Data Engineering Standards Champion Delta Lake best practices and Medallion Architecture principles. Define reusable engineering patterns, notebook standards, and deployment frameworks. Review solution designs and provide architectural guidance to delivery teams. Drive consistency and quality across all Databricks implementations. AI & Analytics Enablement Lead the implementation and governance of Databricks Genie for conversational analytics. Support AI orchestration capabilities including Model Serving, Agent Framework deployments, and integrations with external AI services. Establish guardrails and governance frameworks for responsible AI adoption. Infrastructure & DevOps Manage Databricks infrastructure using Terraform and Infrastructure-as-Code principles. Enhance CI/CD deployment pipelines through Azure DevOps. Collaborate with Azure infrastructure teams on networking, security, storage, and identity integration. Drive automation and operational excellence across the platform. Technical Leadership Act as the Databricks SME across the organisation. Mentor engineers and support architectural decision-making. Produce technical documentation, governance standards, and operational runbooks. Evaluate emerging Databricks and Azure capabilities for future adoption. Partner directly with Databricks to maximise platform value, training, and enablement opportunities. What We're Looking For Essential Experience 5 years of hands-on Databricks experience within enterprise environments. Strong expertise in Unity Catalog administration and governance. Deep knowledge of Delta Lake and Medallion Architecture. Experience with Terraform and Infrastructure-as-Code. Azure DevOps CI/CD pipeline implementation and management. Azure Databricks platform administration including networking, Entra ID, and ADLS Gen2. Strong understanding of data security, RBAC, audit logging, and compliance frameworks. Production-scale PySpark and SQL development experience. Excellent stakeholder engagement and communication skills. Desirable Experience Professional Services or Consulting background. Microsoft Fabric, Azure Data Factory, Synapse Analytics, or Event Hubs. dbt on Databricks. MLflow and Databricks Asset Bundles. Data Mesh architecture principles. Databricks Certified Data Engineer Professional certification. FinOps and cloud cost optimisation experience. Why Join? Own and shape a strategic enterprise Databricks platform. Work on cutting-edge Data, Analytics, AI, and Agentic AI initiatives. Influence platform strategy and architecture decisions. Collaborate with highly skilled engineers, architects, and AI specialists. Gain exposure to enterprise-scale client and internal transformation programmes. Continuous learning, certification, and career growth opportunities.
Locations
City of LondonLondon
