Senior Databricks Consultant

Location: Hybrid, preferably from London or Brighton with travel to client sites

Role type: Full-time, permanent

As a Senior Databricks Consultant at NETbuilder, you will lead the design and delivery of secure, governed Azure Databricks platforms for enterprise clients. The work runs from infrastructure-as-code and Unity Catalog governance through to AI agents and self-service data portals built natively on Databricks.

You will lead delivery teams and stay hands-on with the engineering. You will work alongside client security, platform and data teams, and colleagues across our practices, to turn complex requirements into platforms that are secure, resilient and cost-effective to run.

Clear, consistent communication with customers and internal teams is central to how you will succeed.

What You Will Do

• Design, provision and configure Azure Databricks platforms as code: workspaces, Unity Catalog, catalogs and schemas, ADLS Gen2 storage, networking and customer-managed keys.
• Automate environment setup, identity and access management and deployments with Terraform and Declarative Automation Bundles (formerly Databricks Asset Bundles).
• Design for secure, scalable and cost-effective operation as data volumes grow and new sources are onboarded at agreed latency.

Security, governance and resilience

• Implement role- and attribute-based access control, least privilege and auditability across data, platform resources and AI agents, using Unity Catalog, Microsoft Entra ID, service principals and managed identities.
• Build data retention, lifecycle management and automated hot, warm and cold storage tiering, with monitoring.
• Design and test backup, recovery and disaster recovery for pipelines, platform, data and automation, including Databricks Managed Disaster Recovery.
• Align solutions with client security standards and the Azure and Databricks Well-Architected Frameworks.

AI enablement

• Build, test and deploy repeatable Genie agents that answer questions about onboarded data: its structure, statistical profile, data quality and operational health.
• Put the supporting AI controls in place: access control, CI/CD, prompt and configuration management, monitoring and observability, using Unity AI Gateway and other native capabilities.
• Make sure agents query governed, trusted views rather than raw or base tables.
• Build frontend and backend self-service portals on Databricks Apps that surface data inventory, data contracts, lineage, data quality, pipeline status and operational health.
• Embed chat interfaces to Genie agents, working from client-provided UI designs.
• Use AI-assisted development, including Genie Code, within sound engineering, testing, security and deployment controls.

Assurance and handover

• Deliver automated unit, integration, functional, performance, scalability and resilience testing.
• Support external penetration testing and own remediation of findings in the solutions you deliver.
• Produce architecture and design documentation, deployment guides, operational runbooks and support procedures, and lead knowledge transfer.
• Lead project teams, manage client demands and expectations, and challenge where needed to get to the real business requirement.
• Provide technical support during pre-sales, including solution shaping and estimates.
• Help customers train their teams and adopt new technology, and share Databricks best practice across the team.

What You Will Bring

Core experience

• Ideally 5+ years in data or platform engineering, including at least 2-3 years delivering Databricks in production on Azure.
• A track record of managing client demands and expectations, with the confidence to challenge appropriately to understand the business need.
• Deep, hands‑on Azure Databricks platform experience: account and workspace administration, Unity Catalog design, private networking and customer‑managed keys.
• Delivery of governed platforms in security‑conscious enterprise environments, covering access control, audit, data retention and disaster recovery.
• Hands‑on experience with Databricks AI capabilities such as Genie, Model Serving, AI Gateway and MLflow, including how to evaluate, monitor and govern agents.
• Experience building production internal tools or data applications, ideally on Databricks Apps.
• Databricks certification at Professional level, or Associate level with the drive to progress (see Certifications).

Technical skills

• Python and SQL, including PySpark and Spark SQL; scripting in Bash.
• Infrastructure as code with Terraform (Databricks and AzureRM providers), plus Declarative Automation Bundles, the Databricks CLI, SDK and REST APIs.
• CI/CD with Azure DevOps or GitHub Actions, using Git‑based branching and review workflows.
• Azure services: ADLS Gen2 and storage lifecycle management, Key Vault, Microsoft Entra ID, managed identities, VNet injection and Private Link, Azure Monitor and Log Analytics.
• Databricks services: Unity Catalog (ABAC, row filters, column masks, lineage, system tables), Lakeflow Jobs and Declarative Pipelines, Delta Lake, serverless compute and SQL warehouses.
• Application development for Databricks Apps, using Python web frameworks or React.
• Automated testing, including pytest, data‑quality checks, and performance and load testing.
• PMO and ITSM tooling such as ServiceNow, Jira and Confluence.

Certifications

• Databricks Certified Data Engineer Professional, or Data Engineer Associate with a commitment to achieve Professional within six months of joining.

Highly desirable, or to be achieved within your first year:

• Microsoft Certified: Cloud and AI Security Engineer Associate (SC-500), or a current Azure Security Engineer Associate (AZ-500).
• HashiCorp Certified: Terraform Associate or above.

We will support you to add more, through internal and external training.

About You

• You care about technology and enjoy engineering practical, secure solutions.
• You solve problems proactively and bring strong analytical skills.
• You work independently and take ownership of your work.
• You communicate clearly, including with non‑technical colleagues and client security teams.
• You bring a professional, consultative approach to every client conversation.
• You work well as part of a team and enjoy helping others grow.
• You deliver to tight deadlines.
• You are willing to travel to client sites and to London as needed.

About NETbuilder

NETbuilder is a security services provider with a global presence and a clear ambition: to redefine how organisations consume managed security services in an AI‑driven threat landscape.

We specialise in AI managed security services and security engineering, combining differentiated offerings with deep engineering expertise and measurable customer outcomes.

Our mission is to secure our customers against modern threats, future adversaries, and emerging malicious AI activity through high‑quality services tailored to their platforms and environments.

Every investment a customer makes with NETbuilder should improve security posture and increase operational efficiency. We achieve this by applying the right expertise, services, and tools at every critical stage.

Why NETbuilder

• A tailored progression path, built around you and your career goals.
• Access to internal and external training that leads to professional certifications.
• A collaborative culture and a reward scheme that recognises your contribution.

#J-18808-Ljbffr