Developer (PySpark + Fabric)

Are you looking to build innovative data engineering solutions using PySpark, Microsoft Fabric, and Azure technologies? We have an exciting opportunity for you.

TCS is a purpose-led transformation company, built on belief. We go beyond helping businesses transform through technology—we enable them to make a meaningful difference to the people and communities they serve. Our clients include leading organisations across the UK and globally. At TCS, it means more—more opportunity to make an impact that matters through challenging projects that demand innovation, collaboration, and thought leadership.

Be part of a leading global financial markets infrastructure and data provider. Work within a team transforming and accelerating SDLC processes for critical application and data engineering platforms. Contribute to ambitious business transformation initiatives using modern cloud and data technologies.

The Role: As a Developer (PySpark + Fabric), you will play a key role in transforming critical data engineering capabilities within a leading financial services environment. You will design, build, and maintain data pipelines that support the end-to-end mastering lifecycle, from data ingestion and normalisation through to bi-temporal processing, publication, and analytics consumption. Working closely with data architects, domain experts, quality engineers, and business stakeholders, you will deliver scalable, high-performance solutions using PySpark, Microsoft Fabric, and Azure technologies.

Key Responsibilities:

• Design, develop, and maintain PySpark data pipelines to ingest, transform, and master financial instrument and reference data.
• Implement and optimise bi-temporal data processing solutions to support complex mastering and historical data requirements.
• Develop and optimise Azure Cosmos DB data models, partitioning strategies, indexing policies, and Change Feed integrations.
• Build and maintain integrations with external APIs and data enrichment services, implementing robust authentication, retry, and error-handling mechanisms.
• Develop publication pipelines to distribute mastered data through Microsoft Fabric and downstream platforms.
• Implement data quality controls and validation frameworks to ensure data accuracy, consistency, and reliability.
• Develop and maintain unit, integration, and end-to-end test suites using PyTest and related frameworks.
• Support CI/CD processes for application deployment, infrastructure provisioning, and automated testing.
• Define and maintain configuration-driven solutions using YAML and JSON-based frameworks.
• Monitor, troubleshoot, and optimise production pipelines using observability, telemetry, and logging tools.
• Develop dataflows, ETL processes, semantic models, and analytics-ready datasets to support reporting and business requirements.
• Implement data governance controls, including role-based access, data masking, and compliance requirements.
• Optimise workload performance, scalability, and resource utilisation across distributed processing environments.
• Collaborate with business stakeholders, analysts, and engineering teams to translate requirements into technical solutions.
• Create and maintain technical documentation, standards, and knowledge repositories.

Your Profile:

Essential skills/knowledge/experience:

• Strong proficiency in Python, PySpark, Spark SQL, DataFrame APIs, Structured Streaming, and Java.
• Experience developing and maintaining large-scale data processing and mastering pipelines using PySpark.
• Strong experience working with complex and nested data structures in distributed data environments.
• Experience implementing API integrations, including secure HTTPS services, authentication, and retry mechanisms.
• Hands-on experience with Azure Cosmos DB, including data modelling, partitioning, indexing, throughput optimisation, and Change Feed processing.
• Experience with Azure Data Lake Storage (ADLS), OneLake, ABFS, and Azure Key Vault.
• Strong understanding of data modelling concepts, including temporal data patterns such as bi-temporal modelling, SCD Type 2, and effective dating.
• Experience with data quality and validation frameworks such as Great Expectations.
• Strong testing discipline using PyTest, test fixtures, mocking frameworks, and code coverage tools.
• Experience with Git-based CI/CD platforms such as GitLab CI, Azure DevOps, or similar.
• Experience working with YAML and JSON configuration files and Infrastructure-as-Code concepts.
• Strong experience with Spark for large-scale data processing using DataFrames, RDDs, and Spark SQL.
• Hands-on experience developing ETL workflows, lakehouse solutions, dataflows, pipelines, and semantic models.
• Familiarity with financial market data, time-series data, transactional datasets, or risk-related data domains.
• Strong communication and stakeholder engagement skills.

Desirable skills/knowledge/experience:

• Experience with Microsoft Fabric, including Spark Job Definitions, Eventstream, Lakehouses, Data Pipelines, and Notebooks.
• Domain knowledge of financial instruments and reference data, including ISINs, CUSIPs, LEIs, PermIDs, and asset classifications.
• Experience with entity resolution, record matching platforms, and third-party enrichment services.
• Experience with Delta Lake and Change Data Feed (CDF) patterns.
• Knowledge of Cosmos DB SDK optimisation techniques, including bulk operations, optimistic concurrency, and RU optimisation.
• Experience using Jinja2 for configuration management or code generation.
• Familiarity with SonarQube for code quality and security analysis.
• Experience working within monorepo environments and Python package management frameworks.
• Experience preparing and optimising datasets for Power BI reporting and analytics.
• Knowledge of OneLake, Azure Data Lake, and distributed cloud data platforms.
• Understanding of regulatory and compliance requirements such as GDPR and SOX.

Rewards: we are the proud sponsor of the London Marathon. Diversity, Inclusion and Wellbeing Tata Consultancy Services UK.