At a Glance
- Tasks: Design and optimise data pipelines using Azure Databricks for impactful economic analysis.
- Company: Join a leading tech firm with a focus on innovative data solutions.
- Benefits: Competitive salary, hybrid work model, and career development support.
- Other info: Collaborative environment with opportunities for professional growth.
- Why this job: Work with cutting-edge cloud technologies and tackle real-world data challenges.
- Qualifications: 8+ years in Data Engineering with strong Azure Databricks skills.
The predicted salary is between 75000 - 75000 £ per year.
Job Description
Senior Data Engineer (Azure Databricks) - SC Cleared
Location
London, UK (3 days per week onsite) \n
Salary
Up to £75,000 per annum \n
Job Type
Permanent, Full-Time \n
Security Clearance
Active SC Clearance Required \n
The Opportunity
\n We are seeking an experienced Senior Data Engineer to join a high-performing team responsible for developing and enhancing a modern Azure-based data platform supporting economic analysis, forecasting, and modelling activities. \n This role offers the opportunity to work with large-scale datasets, cloud-native technologies, and modern data engineering practices.
You will play a key role in designing and optimising data pipelines, implementing data governance processes, and delivering reliable data solutions that enable critical business insights. \n
Key Responsibilities
- \n Data Pipeline Development & Optimisation \n \n
- Design, develop, and maintain scalable and reliable data pipelines. \n
- Build ETL/ELT solutions to ingest, transform, and load data from multiple sources, including APIs, databases, and external data providers. \n
- Optimise pipeline performance, scalability, and cost efficiency. \n
- Implement data quality controls, validation processes, and monitoring solutions. \n \n Data Transformation & Processing \n \n
- Develop complex data transformation logic using Spark (Py Spark or Scala). \n
- Build data cleansing, enrichment, aggregation, and processing solutions. \n
- Ensure data accuracy, consistency, and reliability throughout the data life cycle. \n \n Azure Databricks Development \n \n
- Develop and support solutions within Azure Databricks. \n
- Work extensively with Unity Catalog, Delta Lake, Spark SQL, and related Databricks services. \n
- Apply Databricks best practices for development, deployment, and workload optimisation. \n
- Develop solutions using SQL, Python, R, YAML, and Java Script. \n \n Data Integration \n \n
- Integrate data from relational databases, APIs, and streaming platforms. \n
- Design and implement robust data integration patterns. \n
- Collaborate with development teams to support seamless data exchange across systems. \n \n Data Quality & Governance \n \n
- Utilise Azure Purview for data governance and data quality initiatives. \n
- Implement data lineage, metadata management, monitoring, and alerting capabilities. \n
- Ensure compliance with organisational data governance standards and best practices. \n \n Collaboration & Delivery \n \n
- Work closely with data scientists, analysts, economists, and technical stakeholders. \n
- Translate business requirements into scalable technical solutions. \n
- Participate in code reviews, technical discussions, and knowledge-sharing sessions. \n
- Contribute to Agile delivery processes and continuous improvement initiatives. \n \n Dev Ops & Automation \n \n
- Automate data engineering workflows and deployments. \n
- Develop and maintain CI/CD pipelines. \n
- Promote Dev Ops best practices across the wider engineering team. \n \n
- Required Skills & Experience
- \n \n
- 8+ years of Data Engineering experience (10+ years preferred). \n
- Minimum 3 years of hands-on Azure Databricks experience. \n
- Strong expertise in Python, Py Spark, and/or Scala. \n
- Extensive experience with: \n
- Azure Databricks \n
- Azure Data Factory \n
- Azure Blob Storage \n
- Azure SQL Database \n
- Delta Lake \n
- Spark SQL \n
- Strong understanding of: \n
- Data architecture \n
- Data modelling \n
- Data warehousing \n
- Data integration patterns \n
- Experience working with large-scale datasets and complex data pipelines. \n
- Knowledge of SQL and No SQL databases. \n
- Experience with data governance, data quality frameworks, and metadata management. \n
- Experience using Git and working within Agile delivery environments. \n
- Excellent communication, stakeholder management, and problem-solving skills. \n \n
Desirable Skills
- \n \n
- Azure Purview \n
- Kafka \n
- Azure Event Hubs \n
- Power BI \n
- Tableau \n
- Azure Dev Ops \n
- Jenkins \n
- Docker \n
- Kubernetes \n
- Experience within Financial Services, Banking, or Economic Data environments \n
- Microsoft Certified: Azure Data Engineer Associate \n
- Databricks Certifications \n \n
- What's On Offer
- \n \n
- Hybrid working model with 3 days per week in a London office. \n
- Opportunity to work on a large-scale Azure and Databricks data platform. \n
- Exposure to enterprise-level data engineering challenges and modern cloud technologies. \n
- Collaborative and technically focused environment. \n
- Career development and certification support opportunities. \n \n
Please note
Active SC Clearance is required for this position. Applicants without current SC Clearance will not be considered.
Data Engineer (SC Cleared, Databricks) in London employer: ConSol Partners
Join a forward-thinking company that values innovation and collaboration, offering a fully remote working environment for the Technical Author role. With a strong focus on employee growth, you will have the chance to work on high-profile defence programmes while enjoying the flexibility of remote work and occasional travel. Our supportive culture encourages continuous learning and development, making it an excellent place for those looking to make a meaningful impact in the engineering field.
StudySmarter Expert Advice🤫
We think this is how you could land Data Engineer (SC Cleared, Databricks) in London
✨Get Involved in Data Science Meetups
Tap into local data science meetups or workshops to connect with fellow enthusiasts and professionals. These events are goldmines for networking, and sometimes even lead directly to job openings at companies like ConSol Partners!
✨Show Off Your Projects
Start building a public portfolio showcasing your data science projects on platforms like GitHub or personal websites. Highlight unique analyses or models you've developed. This not only demonstrates your skills but also gets your name out there for roles like Data Engineer (SC Cleared, Databricks) at ConSol Partners.
✨Leverage Professional Networks
Join professional bodies related to data science, like the Data Science Society or similar organisations. Getting involved can lead to mentorship opportunities and insider knowledge about full-time positions at companies like ConSol Partners.
✨Apply Directly through Our Website
When you find a suitable opening like Data Engineer (SC Cleared, Databricks) at ConSol Partners, make sure to apply directly through our website. It gives you an edge and shows you're keen to join our team. Plus, who doesn’t love a direct application? It’s easier than navigating through job boards!
We think you need these skills to ace Data Engineer (SC Cleared, Databricks) in London
Some tips for your application 🫡
Show Off Your Projects:In the world of data science, your projects can speak volumes about your skills. Make sure to showcase a few key projects in your CV or portfolio, especially those that highlight your ability to work with data sets, build models, or use relevant tools like Python, R, or SQL. Don’t forget to include links to any GitHub repositories if applicable!
Quantify Your Achievements:Employers love numbers! When drafting your CV, highlight your achievements with quantifiable results. For instance, mention how your data analysis led to a certain percentage increase in efficiency or revenue at a previous job or project. These details can really make your application pop!
Craft a Tailored Cover Letter:For a full-time role at ConSol Partners, your cover letter should reflect your passion for data science and your excitement about the specific projects or values of the company. Dive into why you’re a good fit, how your skills align with their needs, and any unique perspectives you can bring to the team.
Stand Out with Relevant Courses and Certifications:Although experience talks, relevant courses or certifications can be your ticket to impressing hiring managers at ConSol Partners. Mention any standout courses you've completed that equipped you with essential skills, such as machine learning certifications or data visualisation courses. This shows your commitment to continuously developing your skills in the field!
How to prepare for a job interview at ConSol Partners
✨Brush Up on Your Statistics
For a data science role, we need to seriously sharpen our statistics skills. Get ready to tackle technical questions on probability distributions, hypothesis testing, and regression analysis. These are often the bread and butter of data science interviews, so don't just skim over them!
✨Showcase Your Projects
Prepare a killer portfolio showcasing your data science projects. We should include details about the datasets used, the tools and techniques applied, and the impact of your findings. If we can walk them through a particularly challenging project or a cool visualisation that had real-world implications, it’ll really make us stand out!
✨Get Comfortable with Python and R
Most data science positions require us to be proficient in programming languages like Python and R. We should practice common libraries like pandas, NumPy, and scikit-learn, and be ready for live coding exercises or algorithm questions. Showing off our coding chops can really impress the interviewers at ConSol Partners!
✨Prepare for Case Studies
Expect to encounter real-world case studies during the interview. We might be asked how we’d approach a data problem or analyse a dataset to extract insights. It's essential to think out loud and demonstrate our problem-solving process so that the interviewer can see our logical thinking in action.
