Senior Data Engineer – (Databricks, DLT (Delta Live Tables), Qlik Replicate)
Certain Advantage are recruiting on behalf of our trading client for a Senior Data Engineer?with strong analytical capabilities and hands-on experience in building scalable data solutions using a programming language like Python or Scala.
We're looking for candidates who can offer experience in the below so please ensure your application includes your experience with:
* Databricks, Delta Live Tables (DLT)
* ADF, and Qlik Replicate?for data ingestion and replication.
* MS Copilot, AI Foundry , Databricks (MosiacAI, MLflow, Agentbricks, Genie).
You will be responsible for designing and implementing scalable data pipelines and enabling real-time and batch analytics across the organization.
Your Responsibilities:
Design and implement scalable data pipelines using?Databricks and Delta Live Tables (DLT).
Develop and manage?streaming data solutions?for real-time analytics and decision-making.
Ingest data from various sources—including?SAP systems using Qlik Replicate?and?ADF?to ingest data from various sources ensuring data quality and consistency.
Perform?data modelling?and optimize data structures for analytics and reporting.
Collaborate with cross-functional teams to understand data requirements and deliver robust solutions.
Monitor, troubleshoot, and enhance data workflows for performance and reliability.
Maintain technical documentation and contribute to data engineering best practices.
Required Skills & Qualifications:
Strong experience with?Databricks- DLT, Delta Lake concepts, UC governance.
Solid understanding of?streaming technologies?(e.g., Spark Structured Streaming, Autoloader).
Programming skills in?Python,?SQL
Proficiency in?data modelling, ETL/ELT processes, and data architecture.
Experience in MS Copilot, AI Foundry , Databricks (MosiacAI, MLflow, Agentbricks, Genie)
Strong analytical background with problem-solving skills.
Performance tuning concepts like watermarking, late data handling, parallelism & checkpointing.
Hands-on expertise in?ADF, and Qlik Replicate?for data ingestion and replication.
Experience working in?Azure cloud environments.
Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
...
Senior Data Engineer - Databricks & DLT in London employer: Certain Advantage
As a leading Defence Organisation, we pride ourselves on fostering a dynamic work culture that values innovation and collaboration. Our employees enjoy competitive rates, flexible hybrid working arrangements, and ample opportunities for professional growth within a supportive environment. Located in Stevenage, we offer a unique chance to contribute to impactful projects while maintaining a healthy work-life balance.
