Senior Forward Deployed Engineer – Enterprise Data & Databricks (Trading Firm)
Location: London, UK (Onsite)
Position Type: Full-Time, Direct Hire
Compensation: £100,000 – £120,000 per annum
Work Permit Sponsorship: UK Skilled Worker Visa / Work Permit Sponsorship Available
Transform Cutting-Edge Data Platforms at Scale
Are you a deep-dive systems engineer who thrives at the intersection of hands-on production code, low-latency architecture, and high-impact technical consulting?
We are seeking a Senior Forward Deployed Engineer (FDE) for an embedded, high-visibility role based in London with a tier-one client environment. As an FDE, you will not simply write scripts or advise from the sidelines—you will own complex data platforms, ship production systems, and architect high-throughput Databricks ecosystems spanning multiple global locations.
If you excel in fast-paced, loosely defined environments where you can set your own direction, build frameworks from scratch, and deliver mission-critical solutions end-to-end, this challenge is for you.
What You Will Do
1. Core Architecture & Engineering Delivery
-
Streaming & Batch Performance: Design, tune, and operate high-throughput Apache Spark workloads across both batch and streaming pipelines, with a focus on shuffle optimization, cost efficiency, and zero-downtime reliability.
-
Databricks Ecosystem: Architect robust Delta Live Tables (DLT) pipelines, Databricks Workflows, and enterprise-grade Unity Catalog governance and access-control models across the global estate.
-
Automation & Infrastructure-as-Code: Build, provision, and maintain reproducible infrastructure utilizing Terraform, Terragrunt, and Databricks Asset Bundles (DABs), automated via GitHub Actions CI/CD workflows.
-
Advanced Data Modeling: Implement scalable data schemas, query optimization patterns, and fine-grained row-level security policies within Lakebase and lakehouse stores.
-
Platform & Full-Stack Interoperability: Develop backend services (Node.js) and frontend components (React) for internal platform tools, deploy applications via Databricks Apps, and bridge legacy/on-premise data estates with modern Databricks deployments.
2. Technical Leadership & Consulting
-
Lead as a De Facto Product Owner: Partner directly with cross-functional stakeholders (Finance, Risk, Compliance, and Operations) to extract functional needs, prioritize initiatives, and translate business requirements into technical design.
-
Strategic Advisory: Serve as an authoritative, opinionated technical voice, helping engineering peers navigate complex trade-offs, simplify architectures, and eliminate cost inefficiencies across offices.
-
Autonomous Workstream Execution: Navigate shifting technical priorities seamlessly; define your own scope, deliverables, and timelines when project blueprints are fluid or loosely structured.
-
Global Coordination: Collaborate smoothly across international time zones and teams, adapting your delivery and communication rhythm to varied engineering cultures.
3. Strategic Adoption & Engineering Excellence
-
Pioneer Modern DBX Features: Spearhead the roll-out and adoption of new platform capabilities (Lakebase, Databricks Apps, and metadata-driven constructs) through hands-on implementation and technical advocacy.
-
Framework Engineering: Build scalable, reusable, configuration-driven frameworks and standardized validation patterns.
-
Mentorship & Quality Standards: Support internal user groups with query debugging, pipeline troubleshooting, and technical documentation matching top-tier engineering standards.
What You Bring to the Team
Non-Negotiable Core Technical Skills
-
Advanced Apache Spark & Spark SQL: Demonstrated ability to inspect and rewrite execution plans, resolve partition skew, configure AQE, tune memory allocation, and optimize complex analytical queries at scale.
-
Spark Structured Streaming: Proven track record running mission-critical streaming pipelines in production (watermarking, checkpoint recovery, trigger strategies, state-store management, and exactly-once processing).
-
Deep Databricks Fluency: Hands-on mastery of Unity Catalog (governance, data discovery, row/column-level security), Delta Live Tables, and Databricks Workflows.
-
Production-Grade Python/PySpark: Deep expertise writing modular, testable, metadata-driven Python frameworks and configuration-based patterns.
-
Modern CI/CD & IaC: Hands-on experience automating infrastructure provisioning using Terraform and Databricks Asset Bundles (DABs).
-
Autonomous Mindset: Self-driven work ethic capable of scoping out solutions, prioritizing tasks, and driving projects to completion without needing rigid sprints or granular oversight.
Strongly Preferred Experience
-
12 years + Experience deploying and maintaining full-stack tools via Databricks Apps (Node.js/React).
-
Prior domain exposure to proprietary trading, quantitative finance, capital markets, or high-throughput financial environments.
-
Databricks Professional Certifications.
-
Experience integrating platform telemetry and monitoring frameworks (OTLP, VictoriaMetrics, Grafana).
Why Join Us?
-
UK Work Permit / Skilled Worker Visa Sponsorship Provided: We handle and sponsor work authorization for qualified candidates.
-
Direct Ownership & High Impact: Avoid corporate red tape—build, own, and deploy real systems directly into production with significant technical autonomy.
-
Cutting-Edge Stack: Work across the modern data stack—Databricks Asset Bundles, Delta Live Tables, Unity Catalog, and Lakebase.
-
Competitive Package: Attractive compensation package (£100,000 – £120,000 base) in the heart of London.
