At a Glance
- Tasks: Design and maintain data pipelines using Databricks for innovative projects.
- Company: Join a forward-thinking company focused on data engineering excellence.
- Benefits: Competitive daily rate, remote work, and potential contract extensions.
- Other info: Opportunity to work on exciting AI features and data privacy initiatives.
- Why this job: Make a real impact in the EdTech space with cutting-edge data solutions.
- Qualifications: 8+ years in software engineering, with 3+ years in senior data roles.
The predicted salary is between 182000 - 182000 £ per year.
Rate- £700/day Outside ir35
Work from Home
Duration- 12 months minimum (with possible extensions)
This is a Senior Databricks hands-on Architect who is a Databricks engineer by trade. This is a long-term contract engagement to provide dedicated data engineering depth within the Databricks and Shared IEP Services data layer, owning delivery across these workstreams and ensuring the resulting pipelines and schemas are production-grade, governed, and maintainable by the permanent team. The contractor will operate primarily within the Databricks and Shared IEP Services data layer, owning data engineering delivery across three areas:
- Product data ingestion - Building and maintaining daily pipelines from content and assessment products into the shared Back Office data layer.
- AI feature enablement - Designing and maintaining the schemas and data pipelines that power LLM-based features (GenAI Goals, Bulk Upload, Evaluation Document processing), including handling of parsed, normalized, and PII-sensitive document data.
- IEP & Medicaid integration - Building and maintaining pipelines for IEP, Medicaid, and related service data into Databricks and other IEP systems.
Essential Criteria
- Medallion architecture depth - designing and maintaining Bronze, Silver, and Gold schemas and pipelines using Databricks Delta Lake, including schema evolution and enforcement.
- Production pipeline engineering - orchestrating and operationally supporting complex pipelines; idempotency, robust error handling, and reconciliation.
- Unity Catalog governance - fine-grained access control, data lineage, and row/column-level security.
- Cross-system data integration - handling data cleanliness, integrity, normalization, backfilling, reconciliation, and other ETL challenges across disparate source systems.
- Data privacy & compliance - anonymizing, masking, or tokenizing PII; working understanding of FERPA, COPPA, or GDPR.
- Performance & cost optimization - Z-ordering, liquid clustering, data skipping, and shuffle-partition management to control cloud spend.
- Senior engineering experience - 8+ years overall software engineering, including 3+ years in a senior data engineering role.
Desirable Criteria
- Familiarity with Databricks AI development capabilities (eg vector search, model serving, LLM pipelines).
- EdTech engineering experience.
- Familiarity with EdTech specifications and standards such as 1EdTech, EdFi, CEDS, or OneRoster.
Senior Databricks Architect/Databricks Engineer- Contract- Remote employer: Careerwise
Join a forward-thinking company that values innovation and collaboration, offering a remote working environment that promotes work-life balance. As a Senior Databricks Architect, you'll have the opportunity to lead impactful projects while benefiting from competitive rates and a supportive culture that encourages professional growth and development in the rapidly evolving field of data engineering.
StudySmarter Expert Advice🤫
We think this is how you could land Senior Databricks Architect/Databricks Engineer- Contract- Remote
✨Tip Number 1
Network like a pro! Reach out to your connections in the data engineering field, especially those who work with Databricks. A friendly chat can lead to insider info about job openings that might not even be advertised yet.
✨Tip Number 2
Show off your skills! Create a portfolio showcasing your past projects, especially those involving Databricks and data pipelines. This gives potential employers a taste of what you can do and sets you apart from the crowd.
✨Tip Number 3
Prepare for interviews by brushing up on key concepts like medallion architecture and production pipeline engineering. We recommend practising common interview questions related to Databricks to boost your confidence.
✨Tip Number 4
Don’t forget to apply through our website! It’s the best way to ensure your application gets noticed. Plus, we love seeing candidates who are proactive about their job search!
We think you need these skills to ace Senior Databricks Architect/Databricks Engineer- Contract- Remote
Some tips for your application 🫡
Tailor Your CV:Make sure your CV is tailored to the Senior Databricks Architect role. Highlight your experience with Databricks, data pipelines, and any relevant projects that showcase your skills in production pipeline engineering and data governance.
Craft a Compelling Cover Letter:Your cover letter should tell us why you're the perfect fit for this contract. Share specific examples of how you've designed and maintained schemas or handled complex data integration challenges in previous roles.
Showcase Your Technical Skills:Don’t forget to include your technical skills related to Medallion architecture, Unity Catalog governance, and data privacy compliance. We want to see your depth of knowledge and how you’ve applied it in real-world scenarios.
Apply Through Our Website:For the best chance of getting noticed, apply directly through our website. It helps us keep track of applications and ensures you’re considered for this exciting opportunity!
How to prepare for a job interview at Careerwise
✨Know Your Databricks Inside Out
Make sure you brush up on your Databricks knowledge, especially around Medallion architecture and Delta Lake. Be ready to discuss how you've designed and maintained schemas and pipelines in the past, as this will show your hands-on experience.
✨Showcase Your Problem-Solving Skills
Prepare examples of complex pipeline engineering challenges you've faced and how you overcame them. Highlight your approach to error handling and reconciliation, as these are crucial for production-grade pipelines.
✨Understand Data Privacy Regulations
Familiarise yourself with data privacy laws like GDPR and FERPA. Be prepared to discuss how you've implemented data anonymisation or tokenisation in previous roles, as this is essential for the position.
✨Demonstrate Cross-System Integration Experience
Be ready to talk about your experience with ETL processes and how you've handled data integrity and cleanliness across different systems. This will show that you can manage the complexities of data integration effectively.
