Job Description
We are seeking an experienced Databricks Architect / Lead Data Engineer to lead the design, development, and delivery of enterprise-scale data platforms on Databricks and AWS.
The role combines architecture leadership with hands-on engineering responsibilities, including building scalable Lakehouse solutions, modern data pipelines, and AI/ML workloads.
Key Responsibilities
- Design and own Databricks Lakehouse architecture on AWS. Lead multiple Databricks initiatives and mentor data engineers.
- Build and optimize data pipelines using Python, PySpark, SQL, and Delta Lake. Implement Medallion Architecture (Bronze, Silver, Gold layers).
- Develop batch, streaming, and near real-time data solutions. Enable AI/ML and Generative AI use cases on Databricks.
- Establish governance, security, Unity Catalog, RBAC, and operational controls. Integrate Databricks with AWS services including S3, IAM, networking, and data platforms.
- Drive CI/CD, automated testing, deployment automation, and observability.
- Troubleshoot and optimize Spark workloads for performance, reliability, and cost. Required Experience In Data Engineering experience.
- And hands-on Databricks experience. as a Databricks Architect or equivalent architecture role.
- delivering AI/ML or Generative AI solutions on Databricks. and AWS experience.
#J-18808-Ljbffr
Lead Data Engineer employer: ELLIOTT MOSS CONSULTING PTE. LTD.
As a Test Analyst at our company, you will be part of a dynamic team dedicated to delivering high-quality software solutions in a collaborative and supportive environment. We prioritise employee growth through continuous learning opportunities and foster a culture of innovation and excellence, all while being located in a vibrant area that offers a great work-life balance. Join us to make a meaningful impact in the public sector and enjoy the unique advantages of working with cutting-edge technologies and a diverse team.
Contact Details:
ELLIOTT MOSS CONSULTING PTE. LTD. Recruitment Team
