- Ingestion and CDC: Implement pipelines using AWS Lambda , EMR for batch/streaming( future need); manage schema evolution and backfills.
- Orchestration: Build workflows with Step Functions and/or MWAA (Airflow), including retries, alerts, and SLAs.
- Lakehouse and warehouse: Model datasets on S3 (Parquet/Iceberg), Glue Data Catalog, Athena; partitioning, compaction, and performance tuning.
- Data quality and testing: Add automated tests (Great Expectations/Deequ, PySpark unit tests), data contracts, and CI gates.
- Security and governance: Implement IAM, KMS, VPC endpoints, Lake Formation policies, PII handling, audit trails (CloudTrail), and RBAC/RLS.
- Observability and FinOps: CloudWatch metrics/alerts, lineage, usage dashboards; cost optimisation (S3 lifecycle, compression, job sizing).
- CI/CD and IaC: Provision with Terraform/CloudFormation/CDK; build pipelines with GitHub Actions; environment promotion and rollbacks.
- Documentation and runbooks: Maintain pipeline diagrams, SLAs/SLOs, and incident playbooks.
- Adoption: be able to adopt capabilities from other teams instead of building duplicate capabilities and be able to contribute back to community.
Outcomes (first 60-90 days)
- Productionise one end to end pipeline with tests, monitoring, and alerting.
- Establish governance baseline (Lake Formation, tagging/classification, encryption) and data contracts for two key sources.
- Stand up CI/CD and IaC for data services; reduce at least one cost
Skills and experience
- 10+ years building data platforms on AWS; strong Python (incl. PySpark) and SQL.
- Hands on with DMS, Kinesis/MSK, Glue/EMR, Lambda, Step Functions/MWAA,
- IaC (Terraform) and CI/CD (GitHub Actions).
- Data quality frameworks (Great Expectations/Deequ), schema evolution, backfills.
- Security and compliance: IAM/KMS, private networking, Lake Formation,
- Performance and cost tuning across storage and compute.
Nice to have
- Snowflake, Apache Flink, Redshift Spectrum, OpenLineage, OPA/policy as
Senior Data Engineer in London employer: Intuition It Solutions
Join a forward-thinking company that values innovation and collaboration, where your expertise as a ServiceNow Lead/Architect will be instrumental in driving transformative projects. With a strong commitment to employee growth, we offer continuous learning opportunities and a supportive work culture that thrives on teamwork and excellence. Located in a vibrant area, our workplace fosters creativity and provides unique advantages, including access to cutting-edge technology and a diverse range of clients in regulated industries.
