- Design, develop, and maintain scalable batch and streaming data pipelines using Apache Spark, cloud-native technologies, and modern data platforms
- Build and optimize ETL/ELT frameworks to ingest, transform, validate, and curate data from diverse enterprise source systems
- Develop and implement data solutions leveraging Microsoft Fabric, OneLake, Lakehouse, Data Warehouse, and Semantic Model capabilities
- Design data models, storage strategies, and data processing architectures supporting enterprise analytics and reporting
- Partner with Analytics Engineers, BI teams, Data Scientists, Product Managers, Platform Engineers, Security teams, and business stakeholders
- Deliver trusted, reusable, scalable, and analytics-ready datasets and data products
- Drive engineering best practices including code quality, peer reviews, automated testing, CI/CD, and documentation
- Ensure data quality, governance, lineage, metadata management, security, privacy, and compliance requirements
- Monitor and optimize performance, scalability, reliability, and cost efficiency of distributed data processing workloads
- Develop reusable frameworks, libraries, templates, and engineering patterns
- Implement and support CI/CD pipelines, Infrastructure-as-Code principles, and deployment automation
- Troubleshoot complex data integration, performance, and operational issues
- Create and maintain technical documentation, architecture diagrams, operational runbooks, and implementation guides
- Participate in technical design sessions, architecture reviews, and engineering discussions
- Mentor and guide junior and mid-level data engineers through coaching, code reviews, design reviews, and knowledge sharing
- Evaluate and adopt emerging technologies, engineering practices, and AI-assisted development capabilities
- Support production operations through monitoring, root cause analysis, incident resolution, and continuous improvement
- Contribute to Agile delivery processes including sprint planning, estimation, backlog refinement, and execution
- Promote DataOps practices, automation, observability, and engineering excellence
- Serve as a technical subject matter expert for modern data platforms, Spark processing, Microsoft Fabric, and lakehouse architectures
Requirements
- Legal right to work in the UK for employment at Allstate
- Allstate is not providing sponsorship for this vacancy
- Minimum of 2 years of experience in Data Engineering and Big Data technologies
- Deep expertise in Apache Spark (PySpark/Spark SQL)
- Experience with Microsoft Fabric, OneLake, Lakehouse Architecture, Data Warehouse, and building Semantic Models
- Strong SQL and data modeling skills
- Experience building and optimizing ETL/ELT pipelines
- Strong proficiency in Java and experience building distributed applications and services
- Experience with Kafka or similar event-streaming platforms
- Understanding of Data Governance, Data Quality, Security, and Privacy controls
- Ability to mentor junior engineers and provide technical leadership without direct people management responsibilities
- Knowledge of streaming and batch data processing patterns
- Experience with observability solutions including logging, monitoring, tracing, and performance analysis
- Experience working in cloud-based analytics and data platform environments
- Experience with CI/CD, Git-based development, testing, and deployment automation
- Experience working in Agile delivery environments
Core Competencies
Demonstrates expertise in designing and developing scalable data pipelines using Apache Spark and Microsoft Fabric, with a strong focus on ETL/ELT processes, data governance, and cloud-based analytics. Proven ability to mentor junior engineers and drive engineering best practices in Agile environments.
Highest-signal resume keywords
- Apache Spark (PySpark/Spark SQL)
- Microsoft Fabric
- ETL/ELT Pipeline Optimization
- Data Governance and Quality
- CI/CD and Deployment Automation
Hard Skills
- Data Engineering
- Big Data Technologies
- SQL
- Data Modeling
- Java
- Kafka
- Streaming Data Processing
- Observability Solutions
- Cloud-Based Analytics
- Agile Delivery
Soft Skills
- Mentoring
- Technical Leadership
- Collaboration
Industry Keywords
- Data Governance
- Data Quality
- Security
- Privacy Controls
- DataOps
Tools & Technologies
- Apache Spark
- Microsoft Fabric
- OneLake
- Lakehouse Architecture
- Data Warehouse
- CI/CD
- Git
#J-18808-Ljbffr
Senior Consultant II – Software Engineer employer: Jobtailor
As a Client Services Coordinator at our dynamic company, you will thrive in a supportive work culture that prioritises employee growth and development. We offer comprehensive training, opportunities for advancement, and a collaborative environment where your contributions are valued. Located in a vibrant area, our team enjoys a healthy work-life balance and the chance to engage with diverse clients, making every day rewarding and meaningful.
