Senior Data Engineer - £62,000 - Hybrid in Bridgend

The company is a not-for-profit housing association established in 2003 as the first large-scale voluntary transfer of council-owned homes to a social housing provider in Wales. The organisation is rooted in its mission to provide safe, high-quality and affordable homes across Bridgend and South Wales, creating thriving communities where people feel secure, supported, and proud to live. Today, the business manages over 6,000 homes, alongside nearly 700 leasehold flats, a portfolio of commercial properties, and more than 1,100 garages, supporting thousands of residents across the region. As a purpose-driven organisation, its core focus is to deliver homes and places where people feel safe and happy, while helping individuals live independently and confidently. With around 131 employees and a commitment to community development and sustainable living, they continue to play a vital role in local regeneration and social wellbeing. The organisation works to build strong partnerships, enhance resident engagement, and embed environmental sustainability into every part of its operations. The organisation is also dedicated to nurturing inclusive communities and has pledged to uphold strong equality, diversity and inclusion values across all aspects of its work. With a vision of helping build a better Bridgend and Wales, the organisation remains committed to improving lives, investing in homes, and supporting resilient, forward-looking communities.

The overall purpose of this role is to be the primary architect and builder of our modern data estate. This is a high impact, 'greenfield' role where you will design and implement the Azure Databricks infrastructure that forms the backbone of our future data, BI, and AI capabilities, and continue to mature our PowerBI architecture for ease and robustness of reporting and self-service. This role will champion a DataSecOps approach, ensuring that security, privacy and compliance are embedded throughout the data lifecycle from the outset. The nascent nature of our data capabilities means you will have the opportunity to help shape those capabilities to achieve lasting positive value for the organisation as a whole, our customers and our colleagues.

Key responsibilities and accountabilities:

• Lead the design and implementation of a scalable and secure data platform using Azure Databricks, enabling the ingestion, transformation and governance of data from key business systems while supporting a reliable 'single version of the truth' for reporting and analytics.
• Design and deploy secure, scalable ELT or ETL pipelines using Databricks (Unity Catalog), Delta Lake, and Azure Data Lake Storage (ADLS).
• Implement shift-left security principles within the data ingestion layer, including automated data masking, encryption at rest and in transit, and row- and column-level security controls.
• Establish and embed DataOps and DataSecOps practices, including CI/CD, version control (Git), and automated security scanning of data pipelines.
• Finalise a governed Power BI architecture, including standardised semantic models and dataflows.
• Ensure robust Identity and Access Management (IAM) across Power BI workspaces, enabling true self-service analytics without compromising data privacy.
• Proactively identify data and logic inconsistencies between systems, providing clear evidence and insight to support stakeholders in resolving operational issues.
• Act as a technical lead, coaching a Junior Data Engineer and Analyst on modern engineering standards and secure coding practices.
• Work with the Software Engineering Manager to establish naming conventions, metadata standards, and data residency/sovereignty protocols.
• Constructively challenge stakeholders and teams where necessary on scope and delivery to ensure deliverables remain on track.
• Support continuous improvement efforts through ongoing analysis and feedback, and implementation of best practices.
• Coach and mentor colleagues to improve the maturity of the capability.
• Work with housing and frontline teams to ensure system implementations improve service quality and accessibility for customers.
• Identify opportunities to leverage technology to enhance customer engagement, service delivery, and operational efficiencies.
• Support the effective management of people, processes, and performance, fostering a high-support, high-challenge environment to achieve organisational goals.
• Promote efficient, continuously improving ways of working, while role-modelling Valleys to Coast's values and behaviours at all times.
• Contribute to organisational planning and cross-team collaboration, with a clear focus on the Bridgend borough, social housing, and our customers.
• Deputise for your manager when required, and complete all mandatory training and professional development aligned to your role.
• Support the development of colleagues, ensuring excellence, integrity, compliance, and quality.
• Oversee health and safety responsibilities, including training, risk assessments, supervision, and monitoring, addressing any concerns and ensuring incidents are effectively managed.

What you'll need to succeed:

• Expert-level SQL and Python (ideally PySpark).
• A strong understanding of DataSecOps; experience with PII handling, data obfuscation, and RBAC (Role-Based Access Control).
• Proven ability to design data systems (preferably Medallion architectures).
• Advanced knowledge of Power BI security and administration.
• Familiarity with git and gitflow version control processes.
• Knowledge of cloud cost and resource optimisation (FinOps principles).
• Awareness of agile delivery methodologies (Scrum or Kanban).

Desirable:

• Experience with DBT (Data Build Tool) or Prefect.
• Awareness of AI/ML data preparation and the security implications of LLM training data.
• Knowledge of housing, asset management, or repairs systems (ideally Civica Cx and Total Mobile Service Connect).

Experience:

• Implemented and operated cloud-based data platforms, ideally on Azure (experience MUST include build, not just maintenance of existing).
• Delivered a Delta Lakehouse, Medallion Architecture, or similar curated data layering approach.
• Created end-to-end ELT/ETL pipelines using Databricks, Spark, Azure DataFactory, or equivalent tooling.
• Embedded security controls into pipelines (e.g., secure credential handling, encryption, RBAC, ACLs).
• Worked with CI/CD, version control (Git), code review processes, and automated testing for data.
• Designed data models (star schema, transformed delta tables, data marts) for analytics and BI.
• Ensured data quality through validation, profiling, reconciliation, and automated tests.
• Improved or restructured existing reporting environments to make them scalable, governed, and self-service friendly.
• Provided guidance, reviewed code, coached junior team members, and influenced team practices.

Desirable:

• Design of data architecture, as well as build.
• Azure Databricks experience.
• Experience managing multiple workstreams simultaneously, balancing major projects with smaller initiatives.

If you're interested in this role, click 'apply now' to forward an up-to-date copy of your CV, or call us now.