ContractLondon, WaterlooPosted 14 hours agoCompetitive USD / Year This is an exciting role for a Senior Data Engineer with strong Databricks and Delta to join a world-leading Energy Trading company.Role Description: We are looking for a highly skilled Senior Data Engineer with strong analytical capabilities and hands-on experience in building scalable data solutions.The ideal candidate will have expertise in Databricks, streaming architectures, data ingestion frameworks, and dashboarding, along with experience working with SAP data sources.You will be responsible for designing and implementing scalable data pipelines and enabling real-time and batch analytics across the organization.Key Responsibilities:Design and implement scalable data pipelines using Databricks and Delta Live Tables (DLT)Develop and manage streaming data solutions for real-time analytics and decision-makingIngest data from various sourcesincluding SAP systems using Qlik Replicate and ADF to ingest data from various sources ensuring data quality and consistencyPerform data modelling and optimize data structures for analytics and reportingCollaborate with cross-functional teams to understand data requirements and deliver robust solutionsMonitor, troubleshoot, and enhance data workflows for performance and reliabilityMaintain technical documentation and contribute to data engineering best practicesRequired Skills & Qualifications:Strong experience with Databricks- DLT, Delta Lake concepts, UC governanceSolid understanding of streaming technologies (e.g., Spark Structured Streaming, Autoloader)Programming skills in Python, SQL, or ScalaProficiency in data modelling, ETL/ELT processes, and data architectureExperience in MS Copilot, AI Foundry , Databricks (MosiacAI, MLflow, Agentbricks, Genie)Strong analytical background with problem-solving skillsPerformance tuning concepts like watermarking, late data handling, parallelism & checkpointingHands-on expertise in ADF, and Qlik Replicate for data ingestion and replicationExperience working in Azure cloud environmentsWith a focus within Energy Trading, Oil & Gas, Financial Markets and Commodities, we offer a transparent Recruitment Service that has proven to be reliable and effective for over 40 years. We are ISO accredited and proud of our excellent TrustPilot Reviews. Your search for a New Contract Assignment or for a New Permanent Job will be in safe hands with Eaglecliff Recruitment. Please telephone for an immediate response or email your CV for a quick response. Eaglecliff Ltd is acting in the capacity of an employment agency for permanent recruitment and an employment business for contractor resourcing.
Senior Data Engineer : Databricks & Delta in London employer: Eaglecliff Recruitment
Eaglecliff Recruitment is an exceptional employer, offering a dynamic work environment in the heart of London, where innovation meets collaboration. With a strong focus on employee growth, we provide extensive training and mentoring opportunities for Agile practitioners, ensuring that our team members thrive in their careers while contributing to high-performing delivery teams in the energy trading sector. Our commitment to transparency and continuous improvement fosters a supportive culture that values every individual's contribution, making it a rewarding place to work.
