Lead Data Engineer

What you'll get to do in the role:

• Design, build, and maintain resilient data pipelines that ingest data from Azure SQL, SaaS platforms, and event streams into BigQuery.
• Write Python code using DLT to define declarative, testable, version-controlled pipelines – no low‑code tools, just real engineering.
• Build and operate ML feature pipelines for low‑latency, real‑time data streams that feed ML models with accurate, fresh features.
• Own the operational health of systems you build – monitoring, alerting, error handling, and incident response.
• Collaborate with analytics engineers to understand data needs, validate schema design, and establish data quality standards that both analytics and ML rely on.
• Partner with the AI/ML platform team to design feature stores, streaming feature infrastructure, and model serving pipelines that power Liberis' decisioning engine.
• Identify and execute optimisation work – improving performance, reliability, and developer velocity without rearchitecting stable systems.
• Mentor junior engineers, helping them grow as engineers and supporting their career development.
• Participate in technical decisions about platform direction – infrastructure choices, tooling, architecture trade‑offs.
• Work cross‑functionally with product teams, analytics engineers, BI specialists, and the ML platform team to shape data requirements and platform capabilities.

What we think you'll need:

• Proven experience within data engineering roles – building and operating data pipelines at scale.
• Hands‑on experience building Modern Data Stack architectures – ingestion, warehouse, transformation, orchestration, reverse ETL.
• You’ve worked with tools like DLT, Fivetran, Airbyte (ingestion); BigQuery, Snowflake, Redshift (warehouse); DBT (transformation); Airflow or similar (orchestration).
• Strong Python programming – you write clean, testable, maintainable code with solid error handling and logging.
• Fluent SQL – you can write complex queries, understand execution plans, and optimise for performance and cost.
• Experience with cloud data platforms – building data warehouses in BigQuery, Redshift, Snowflake, or similar; you understand distributed processing, partitioning, cost optimisation, and data governance.
• Experience with infrastructure‑as‑code tools (Terraform, CloudFormation, Pulumi) or equivalent – you version‑control infrastructure and deploy it via CI/CD pipelines.
• Experience working in fast‑moving environments where requirements evolve and you adapt quickly without losing sight of reliability.
• Understanding of DevOps principles – you think in terms of observability, resilience, incident response, and operational excellence.
• You can set up monitoring and alerting that actually matters.

Bonus points if you have:

• Experience with DLT or similar declarative ELT frameworks; experience with Google Cloud Platform ecosystem (BigQuery, Cloud Run, Pub/Sub, Dataflow); experience with Kafka, Pub/Sub, or event streaming platforms; experience scaling data systems from 0 to 100M+ events/day; experience implementing data quality frameworks (Great Expectations, dbt tests, custom monitoring); background in fintech or high‑stakes data reliability environments where data quality directly impacts revenue.
• Experience working with distributed, asynchronous teams across timezones; experience in India tech ecosystem or building in resource‑constrained environments; experience migrating from legacy data infrastructure (Azure ADF, traditional ETL) to modern cloud‑native stacks.