ETL Developer (Prophecy )

Key Responsibilities:

• Build and optimize Prophecy data pipelines for large scale batch and streaming data workloads using Pyspark
• Define end-to-end data architecture leveraging prophecy integrated with databricks or Spark or other cloud-native compute engines
• Establish coding standards, reusable components, and naming conventions using Prophecy's visual designer and metdata-driven approach
• Implement scalable and efficient data models (e.g star schema, scd typ2) for data marts and analytics layer
• Integrate Prophecy pipelines with orchestration tools like Airflow, data catalog tools for lineage
• Implement version control, automated testing and deployment pipelines using Git and CI/CD (e.g GitHub and Jenkins)
• Monitor and tune performance of Spark jobs optimize data partitions and caching strategies
• Having experience and exposure to convert legacy etl tools like datastage, informatica into Prophecy pipelines using Transpiler component of Prophecy

Required skill & experience:

• 2+ years of hands-on experience with Prophecy (Using pyspark) approach
• 5+ years of experience in data engineering with tools such as Spark, Databricks,scala/Pyspark or SQL
• Strong understanding of ETL/ELT pipelines, distributed data processing and data lake architecture.
• Having exposure to ETL tools such as Informatica,
• Datastage or talend is added advantage
• Experience with Unity catalog, Delta lake and modern data lakehouse concepts
• Strong communication and stakeholder management skills.