At a Glance
- Tasks: Design and optimise data pipelines using cutting-edge technologies like Databricks and PySpark.
- Company: Join a leading firm in data engineering with a focus on innovation.
- Benefits: Enjoy competitive pay, flexible remote work, and opportunities for professional growth.
- Other info: Collaborate in an Agile environment with great potential for career advancement.
- Why this job: Make a real impact by transforming data into actionable insights for businesses.
- Qualifications: 10-12 years of experience in data engineering and strong technical skills required.
The predicted salary is between 63000 - 77000 £ per year.
- Fixed term contract
- Contract Length: Initial 3-6 months with possible extensions
Experience Range - 10- 12 Years
Location Requirement: Onsite (3 days per week in the office and 2 days remote)
Applicants must be legally authorized to work in the United Kingdom without the need for current or future visa sponsorship
- Design, build and optimise robust data ingestion pipelines to acquire, transform and load data vendor platforms into client's data ecosystem.
- Develop scalable data engineering solutions using Databricks, Py Spark, Spark SQL and associated cloud technologies.
- Build and maintain secure, reliable and automated data ingestion processes from external vendor systems, including SFTP-based file transfers and other integration methods.
- Ensure data is landed, structured, governed and accessible to support reporting, analytics and business use cases.
- Work with Business Analysts, Product Owners, Architects and delivery squads to translate business requirements into technical data solutions.
- Support data discovery, profiling and validation activities to understand source data structures, data quality issues and data gaps.
- Develop and maintain data transformations, curated datasets and data models required to support reporting and analytical use cases.
- Monitor, troubleshoot and resolve data ingestion issues, defects and enhancements identified during development, testing, UAT and production support.
- Ensure solutions comply with data architecture standards, engineering best practices, security requirements and governance frameworks.
- Produce clear technical documentation for data ingestion processes, data flows and operational support requirements.
- Provide comprehensive handover documentation and knowledge transfer to the Data Support team following delivery of data ingestion pipelines.
- Collaborate within Agile delivery teams, actively contributing to sprint planning, stand-ups, retrospectives and continuous improvement activities.
- Identify opportunities to improve pipeline performance, automation, scalability and maintainability through process and technology enhancements.
- Support knowledge sharing and contribute to Engineering and Data Communities of Practice.
Essential
- Proven experience designing, building and supporting enterprise-scale data ingestion pipelines and ETL/ELT solutions.
- Strong hands-on experience with Databricks, Py Spark and Spark SQL.
- Experience developing and supporting secure data integrations using SFTP and other file-based or API-driven ingestion mechanisms.
- Experience ingesting and processing structured, semi-structured and unstructured data from internal and third-party source systems.
- Strong understanding of data modelling, transformation techniques and data warehousing principles.
- Experience working with cloud-based data lake and analytics platforms.
- Strong understanding of batch and near real-time data processing patterns.
- Experience conducting data profiling, discovery and validation activities to assess data quality, completeness and suitability for business requirements.
- Experience implementing data quality checks, reconciliations and monitoring processes.
- Ability to investigate and resolve ingestion, transformation and data quality issues identified during testing, UAT or production support.
- Understanding of data governance, security, data lineage and documentation standards.
- Experience producing technical documentation and operational handover materials.
- Strong stakeholder engagement skills with the ability to work effectively across business, architecture, engineering and analytics teams.
- Experience working within Agile delivery environments.
- Knowledge of source control, CI/CD practices and release management processes.
- Ability to work independently while collaborating effectively within cross-functional squads.
- Experience integrating data from retail technology platforms, Io T devices or third-party vendor systems.
- Experience working with AI Camera, Computer Vision or Electronic Shelf Edge Label (e SEL) technologies.
- Knowledge of Azure Data Lake, Azure Data Factory and related Azure data services.
- Experience supporting reporting, analytics or BI solutions through the creation of trusted and governed data assets.
- Experience working within large-scale retail or data transformation programmes.
- Role Description
- Role: Lead Data Engineer
- Fixed term contract
- Contract Length: Initial 3-6 months with possible extensions
Experience Range - 10- 12 Years
Location Requirement: Onsite (3 days per week in the office and 2 days remote)
Applicants must be legally authorized to work in the United Kingdom without the need for current or future visa sponsorship
Responsibilities
- Design, build and optimise robust data ingestion pipelines to acquire, transform and load data vendor platforms into client's data ecosystem.
- Develop scalable data engineering solutions using Databricks, Py Spark, Spark SQL and associated cloud technologies.
- Build and maintain secure, reliable and automated data ingestion processes from external vendor systems, including SFTP-based file transfers and other integration methods.
- Ensure data is landed, structured, governed and accessible to support reporting, analytics and business use cases.
- Work with Business Analysts, Product Owners, Architects and delivery squads to translate business requirements into technical data solutions.
- Support data discovery, profiling and validation activities to understand source data structures, data quality issues and data gaps.
- Develop and maintain data transformations, curated datasets and data models required to support reporting and analytical use cases.
- Monitor, troubleshoot and resolve data ingestion issues, defects and enhancements identified during development, testing, UAT and production support.
- Ensure solutions comply with data architecture standards, engineering best practices, security requirements and governance frameworks.
- Produce clear technical documentation for data ingestion processes, data flows and operational support requirements.
- Provide comprehensive handover documentation and knowledge transfer to the Data Support team following delivery of data ingestion pipelines.
- Collaborate within Agile delivery teams, actively contributing to sprint planning, stand-ups, retrospectives and continuous improvement activities.
- Identify opportunities to improve pipeline performance, automation, scalability and maintainability through process and technology enhancements.
- Support knowledge sharing and contribute to Engineering and Data Communities of Practice.
Essential
Skills & Experience
- Proven experience designing, building and supporting enterprise-scale data ingestion pipelines and ETL/ELT solutions.
- Strong hands-on experience with Databricks, Py Spark and Spark SQL.
- Experience developing and supporting secure data integrations using SFTP and other file-based or API-driven ingestion mechanisms.
- Experience ingesting and processing structured, semi-structured and unstructured data from internal and third-party source systems.
- Strong understanding of data modelling, transformation techniques and data warehousing principles.
- Experience working with cloud-based data lake and analytics platforms.
- Strong understanding of batch and near real-time data processing patterns.
- Experience conducting data profiling, discovery and validation activities to assess data quality, completeness and suitability for business requirements.
- Experience implementing data quality checks, reconciliations and monitoring processes.
- Ability to investigate and resolve ingestion, transformation and data quality issues identified during testing, UAT or production support.
- Understanding of data governance, security, data lineage and documentation standards.
- Experience producing technical documentation and operational handover materials.
- Strong stakeholder engagement skills with the ability to work effectively across business, architecture, engineering and analytics teams.
- Experience working within Agile delivery environments.
- Knowledge of source control, CI/CD practices and release management processes.
- Ability to work independently while collaborating effectively within cross-functional squads.
- Experience integrating data from retail technology platforms, Io T devices or third-party vendor systems.
- Experience working with AI Camera, Computer Vision or Electronic Shelf Edge Label (e SEL) technologies.
- Knowledge of Azure Data Lake, Azure Data Factory and related Azure data services.
- Experience supporting reporting, analytics or BI solutions through the creation of trusted and governed data assets.
- Experience working within large-scale retail or data transformation programmes.
Desirable
- Experience integrating data from retail technology platforms, Io T devices or third-party vendor systems.
- Experience working with AI Camera, Computer Vision or Electronic Shelf Edge Label (e SEL) technologies.
- Knowledge of Azure Data Lake, Azure Data Factory and related Azure data services.
- Experience supporting reporting, analytics or BI solutions through the creation of trusted and governed data assets.
- Experience working within large-scale retail or data transformation programmes.
Skills
- Py Spark
- Azure Data Factory
- Agile
- CI/CD
- #J-18808-Ljbffr
Lead Data engineer (Lead II - Data Engineering) in Leeds employer: UST
As a Principal Data Engineer at our company, you will thrive in a dynamic and innovative work culture that prioritises technical excellence and continuous learning. With opportunities for mentorship and collaboration across diverse teams, you will play a pivotal role in shaping the future of our cloud-scale data platform while enjoying the flexibility of a hybrid work environment in Nottingham or London. We are committed to fostering your professional growth and providing a supportive atmosphere where your contributions directly impact our success.
StudySmarter Expert Advice🤫
We think this is how you could land Lead Data engineer (Lead II - Data Engineering) in Leeds
✨Tap into Online Data Science Communities
Join online communities focused on data science like Kaggle, LinkedIn groups, or Reddit threads. These are goldmines for temporary gigs, as you can network with professionals and potentially hear about opportunities at companies like UST before they're even advertised!
✨Show Off Your Skills With Projects
Got some cool data science projects? Showcase them on platforms like GitHub or create a personal portfolio website. This visibility is crucial for landing temporary roles—let recruiters see your actual skills in action, which can set you apart from the crowd.
✨Check Out Specialist Job Boards
For temp roles, hit up job boards dedicated to tech and data science, like Stack Overflow Jobs or DataJobs. These platforms often feature openings that you won’t find on general job sites, including contracts with companies like UST.
✨Leverage University Resources
If you're currently at uni or recently graduated, tap into your school's career services. They often have connections with companies looking for temporary data science interns or contract workers, and they might even host job fairs with employers like UST.
We think you need these skills to ace Lead Data engineer (Lead II - Data Engineering) in Leeds
Some tips for your application 🫡
Highlight Your Data Projects:When applying for a temporary data science role at UST, make sure to showcase any relevant projects you've worked on. Whether it's a personal project, an academic undertaking, or contributions to an open-source initiative, detailing these experiences can really set you apart and demonstrate your practical skills.
Emphasise Your Analytical Skills:In your CV and cover letter, focus on the specific analytical skills that are key to data science. Mention any experience with statistical tools, programming languages like Python or R, and data visualisation software. Don't forget to include any certifications that may bolster your expertise!
Show Your Flexibility:Since this is a temporary role, it's important to convey your adaptability and willingness to learn. In your cover letter to UST, emphasise how quickly you can get up to speed with new tools or projects. Highlight any previous experiences where you've had to adjust to new environments or challenges.
Craft a Unique Data-Driven Cover Letter:Instead of the usual generic cover letter, spice it up with some data! Maybe you’ve improved a process by 20% in a past role or cleaned a dataset with over a million entries. Use these stats to your advantage to grab UST’s attention and show the tangible impact of your work.
How to prepare for a job interview at UST
✨Showcase Your Analytical Skills
For a data science gig, it's crucial to demonstrate your analytical abilities. Be ready to discuss previous projects and the methodologies you used. Think about how you can quantify your impact—did your analysis improve efficiency or save costs? These are the stories that will stick with interviewers at UST.
✨Brush Up on Technical Skills
You might face technical questions on tools relevant to data science, like Python, R, or SQL. Prepare to solve a problem live—perhaps they'll ask you to write a simple query or code snippet. It’s cool to talk about them, but we need to show we can do it in practice, especially in a temporary role where quick results matter.
✨Highlight Your Adaptability
Since this is a temporary position, emphasise your ability to learn quickly and adapt to new tools or workflows. Share examples of how you've thrived in fast-paced environments before, and how you can hit the ground running at UST.
✨Prepare a Portfolio of Your Work
Bring your portfolio to the table—showcase projects where you've leveraged data science techniques to solve problems. Whether it’s a GitHub repository or a set of case studies, having tangible examples of your work will help you stand out and show what you bring to the team at UST.
