Senior Data Engineer

Responsibilities

• Manage the capture, storage and dissemination of the cloud infrastructure used by the business in the data hub and other data management systems.
• Assemble large, complex data sets that meet functional and non-functional business requirements.
• Implement data processes that deliver easy access to data domains for those who are enabled.
• Design data domain structures that enable self-service and more rigorous analytics.
• Optimise data structures to reduce costs associated with serving the data.
• Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
• Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and Azure 'big data' technologies (ETL & ELT).
• Design and build data models to enable analysts and data consumers to easily and quickly access needed data, both structured and unstructured.
• Build analytics technologies that utilise the data pipeline to provide actionable insights into customer acquisition, operational efficiency, and other key business performance metrics.
• Administer Azure databases, encompassing performance tuning, backup and recovery, and routine maintenance, ensuring optimal database functionality and reliability.
• Monitor access, audit logs, and ensure data masking and encryption at rest and in transit, adhering to strict security protocols.
• Understand data quality frameworks and testing methodologies to validate data accuracy and reliability, ensuring fit for purpose, trustworthy data for decision-making and reporting.
• Work with stakeholders including the Executive, Product, Data, and Design teams to assist with data-related technical issues and support their data infrastructure needs.
• Keep data separated and secure across global boundaries through multiple data centres and Azure regions.
• Design and develop Datalake and Deltalake technologies for cloud data ingestion, classification, storage, and dissemination.
• Collaborate with data and analytics experts to strive for greater functionality in data systems.
• Manage data domains with data stewards to generate data products for consumption.
• Work with metadata and metadata management tools, including reference data and master data management.
• Proactively build the latest Azure enterprise data warehouse through to PowerBI reporting.
• Backfill as the technical lead for the business intelligence/data management team.
• Manage the business intelligence team until a manager is brought on board to fill that function.

Qualifications, skills and experience

• 5+ years of experience in a Data Engineer role, with practical experience.
• Graduate degree in Computer Science, Statistics, Informatics, Information Systems or another quantitative field.
• Advanced working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as familiarity with a variety of databases.
• Experience building and optimizing 'big data' data pipelines, architectures, and data sets.
• Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
• Strong analytic skills related to working with unstructured datasets.
• Build processes supporting data transformation, data structures, metadata, dependency, and workload management.
• A successful history of manipulating, processing, and extracting value from large disconnected datasets.
• Working knowledge of message queuing, stream processing, and highly scalable 'big data' data stores.
• Strong project management and organisational skills.
• Experience supporting and working with cross-functional teams in a dynamic environment.
• Detailed experience in data modelling in cloud environments, structured and unstructured.
• Rostered on-call support for any issues with the Enterprise Data Warehouse (EDW cloud), Global Data Warehouse (GDW on premise), global ETL and data management (additional pay is provided for on-call activities).
• Certifications from Microsoft Azure would be highly considered, specifically Microsoft Certified: Azure Data Engineer Associate (DP-203), no later than 12 months old.
• Experience using big data tools: Hadoop, Spark, Kafka, etc., relational SQL and NoSQL databases with data pipeline and workflow management tools.
• Familiarity with Azure cloud services and the Microsoft Synapse technologies, as we are a Microsoft shop.
• Demonstrated experience in cloud technologies: PowerBI or similar data visualisation tool.
• An innovative mindset, curious about AI and emerging technologies.