Developer (PySpark + Fabric) in London

Role: Developer (PySpark + Fabric)

Job Type: Permanent

Location: London, UK

Mode of Working: Hybrid

Working Hours: 40 hours per week (Full-time)

Are you looking to build innovative data engineering solutions using PySpark, Microsoft Fabric, and Azure technologies? We have an exciting opportunity for you.

Careers at TCS: It Means More

TCS is a purpose-led transformation company, built on belief. We go beyond helping businesses transform through technology—we enable them to make a meaningful difference to the people and communities they serve. Our clients include leading organisations across the UK and globally.

At TCS, it means more—more opportunity to make an impact that matters through challenging projects that demand innovation, collaboration, and thought leadership. Be part of a leading global financial markets infrastructure and data provider. Work within a team transforming and accelerating SDLC processes for critical application and data engineering platforms. Contribute to ambitious business transformation initiatives using modern cloud and data technologies.

The Role

As a Developer (PySpark + Fabric), you will play a key role in transforming critical data engineering capabilities within a leading financial services environment. You will design, build, and maintain data pipelines that support the end-to-end mastering lifecycle, from data ingestion and normalisation through to bi-temporal processing, publication, and analytics consumption. Working closely with data architects, domain experts, quality engineers, and business stakeholders, you will deliver scalable, high-performance solutions using PySpark, Microsoft Fabric, and Azure technologies.

Key Responsibilities

• Design, develop, and maintain PySpark data pipelines to ingest, transform, and master financial instrument and reference data.
• Implement and optimise bi-temporal data processing solutions to support complex mastering and historical data requirements.
• Develop and optimise Azure Cosmos DB data models, partitioning strategies, indexing policies, and Change Feed integrations.
• Build and maintain integrations with external APIs and data enrichment services, implementing robust authentication, retry, and error-handling mechanisms.
• Develop publication pipelines to distribute mastered data through Microsoft Fabric and downstream platforms.
• Implement data quality controls and validation frameworks to ensure data accuracy, consistency, and reliability.
• Develop and maintain unit, integration, and end-to-end test suites using PyTest and related frameworks.
• Support CI/CD processes for application deployment, infrastructure provisioning, and automated testing.
• Define and maintain configuration-driven solutions using YAML and JSON-based frameworks.
• Monitor, troubleshoot, and optimise production pipelines using observability, telemetry, and logging tools.
• Develop dataflows, ETL processes, semantic models, and analytics-ready datasets to support reporting and business requirements.
• Implement data governance controls, including role-based access, data masking, and compliance requirements.
• Optimise workload performance, scalability, and resource utilisation across distributed processing environments.
• Collaborate with business stakeholders, analysts, and engineering teams to translate requirements into technical solutions.
• Create and maintain technical documentation, standards, and knowledge repositories.

Your Profile:

Essential skills/knowledge/experience

• Strong proficiency in Python, PySpark, Spark SQL, DataFrame APIs, Structured Streaming, and Java.
• Experience developing and maintaining large-scale data processing and mastering pipelines using PySpark.
• Strong experience working with complex and nested data structures in distributed data environments.
• Experience implementing API integrations, including secure services, authentication, and retry mechanisms.
• Hands-on experience with Azure Cosmos DB, including data modelling, partitioning, indexing, throughput optimisation, and Change Feed processing.
• Experience with Azure Data Lake Storage (ADLS), OneLake, ABFS, and Azure Key Vault.
• Strong understanding of data modelling concepts, including temporal data patterns such as bi-temporal modelling, SCD Type 2, and effective dating.
• Experience with data quality and validation frameworks such as Great Expectations.
• Strong testing discipline using PyTest, test fixtures, mocking frameworks, and code coverage tools.
• Experience with Git-based CI/CD platforms such as GitLab CI, Azure DevOps, or similar.
• Experience working with YAML and JSON configuration files and Infrastructure-as-Code concepts.
• Strong experience with Spark for large-scale data processing using DataFrames, RDDs, and Spark SQL.
• Hands-on experience developing ETL workflows, lakehouse solutions, dataflows, pipelines, and semantic models.
• Familiarity with financial market data, time-series data, transactional datasets, or risk-related data domains.
• Strong communication and stakeholder engagement skills.

Desirable skills/knowledge/experience

• Experience with Microsoft Fabric, including Spark Job Definitions, Eventstream, Lakehouses, Data Pipelines, and Notebooks.
• Domain knowledge of financial instruments and reference data, including ISINs, CUSIPs, LEIs, PermIDs, and asset classifications.
• Experience with entity resolution, record matching platforms, and third-party enrichment services.
• Experience with Delta Lake and Change Data Feed (CDF) patterns.
• Knowledge of Cosmos DB SDK optimisation techniques, including bulk operations, optimistic concurrency, and RU optimisation.
• Experience using Jinja2 for configuration management or code generation.
• Familiarity with SonarQube for code quality and security analysis.
• Experience working within monorepo environments and Python package management frameworks.
• Experience preparing and optimising datasets for Power BI reporting and analytics.
• Knowledge of OneLake, Azure Data Lake, and distributed cloud data platforms.
• Understanding of regulatory and compliance requirements such as GDPR and SOX.

Rewards & Benefits

TCS is consistently voted a Top Employer in the UK and globally. Our competitive salary packages feature pension, health care, life assurance, laptop, phone, access to extensive training resources and discounts within the larger Tata network. We offer health & wellness initiatives and sports events; we are the proud sponsor of the London Marathon.

Diversity, Inclusion and Wellbeing

Tata Consultancy Services UK&I is committed to meeting the accessibility needs of all individuals in accordance with the UK Equality Act 2010 and the UK Human Rights Act 1998. We welcome and embrace diversity in race, nationality, ethnicity, disability, neurodiversity, gender identity, age, physical ability, gender reassignment, sexual orientation. We are a disability inclusive employer and encourage disabled people to apply for this role. As a Disability Confident Employer, we offer an interview to applicants with disabilities or long-term conditions who meet the minimum criteria for the role.

Next Steps: Application Process

• Online application: You can apply directly through LinkedIn/ by uploading your CV.
• Skill-Based discussion: This will be a level 1 interview with the project team, it can be via video or in-person.
• Managerial discussion: This discussion will focus on behavioural aspects and person-organisation fit.
• HR Discussion: This will be with one of the members of the HR team and will cover your career journey, aspirations for growth, compensation and any other questions you may have.

Join us and do more of what matters. Apply online now.