Senior Data Engineer

Smarkets runs one of the world’s most sophisticated prediction markets, handling over £29 billion in volume since 2010 and engaging 200,000+ traders globally. We’re a technology company that revolutionises betting - from sports to political markets, delivering the sharpest prices and the fairest odds. Our stack is designed for scale, reliability, and speed: Linux, Kafka, Postgres, Kubernetes, with Python 3, C++, Rust and React powering our platform.

The Data Team is responsible for taking the wealth of data that Smarkets generates and using it to drive insights which improve the business. The team’s responsibilities currently span across three different areas:

• Data Engineering: development and maintenance of ETL pipelines, services and APIs, and data-related infrastructure like Redshift or BigQuery;
• Data Science and Machine Learning: data exploration, ML models training and ML Ops to extract new insights from data;
• Analytics and Reporting: creation of data models and dashboards as well as automation of reporting pipelines for different teams, stakeholders and third-parties.

In a typical week, a data engineer in the Data team would:

• Add a new Python ETL pipeline that segments users interested in specific sports through analysing behaviour which streamlines and tailors marketing communications to those users;
• Develop a new endpoint to a Flask API, add unit tests, and deploy the new version of the API into our production Kubernetes cluster;
• Train and evaluate an ML model to identify certain user patterns and provide it as a service to other engineering teams in a Flask API.

Our current technology stack primarily includes Linux, Docker, Kubernetes, Jenkins, Kafka, Python, Flask/FastAPI, Postgres, AWS Redshift, dbt, Google BigQuery, Prometheus, Grafana, Elastic Search, Kibana, Lightdash.

You will work closely with the Data team lead and the other team members who will assist you whenever needed, making your integration in the company as smooth as possible. The Data team works in an organised way using Agile methodologies and tools such as Jira and regular standups. You will find an environment where you have a clear engineering direction, can focus on your work and hone your skills as a data engineer through exciting projects.

As a member of the data team, your responsibilities will include contributions to:

• Developing and maintaining our Data ETL pipelines, some of which are real-time;
• Ensuring our data lake is kept in a healthy state, particularly our data warehouses: Redshift and BigQuery;
• Developing and maintaining Flask services and Postgres databases within the Data team;
• Assisting the different teams in the company with reporting, especially when it comes to automated reporting pipelines;
• Doing data exploration and training & deploying ML models used to perform different kinds of user segmentation, detect operational anomalies or estimate important business quantities.

What we are looking for in you:

• 3+ years of experience;
• Understanding of developing ETL pipelines using Python frameworks such as Luigi or Airflow;
• Experience with the development of Python-based REST APIs/services and their integration with databases (e.g. Postgres);
• Familiarity with the key tools of the Python data science stack, e.g. Pandas, NumPy and scikit-learn;
• Some experience with training and deploying ML models;
• Enjoy writing elegant, well-tested and maintainable code;
• Team player that enjoys contributing to the success of the team in a proactive and friendly environment;
• Bachelor’s in Computer Science, Math or equivalent field, or possess relevant experience.

We offer a competitive salary and benefits package, alongside a dynamic, collaborative environment where your work truly makes an impact and your voice is heard. Our benefits are designed around Health, Wealth and Development, supporting you both professionally and personally. These include:

• Stock Options: Vest over 4 years;
• Pension Scheme: Competitive plan via Aviva, with up to 6% matched contributions if you opt in;
• Health Insurance: Comprehensive coverage to keep you and your family healthy;
• Enhanced Parental Leave: Enhanced maternity and paternity leave;
• Stay Fuelled: Enjoy fresh, chef-made breakfast and lunch every day;
• Cycle-to-Work Scheme: Support for sustainable commuting;
• Learning & Development: £1,000 annual education budget;
• Holiday: 25 days paid leave + bank holidays, with the option to carry over 5 days;
• Flexible Working: Choose to work from home based on your needs and preferences;
• Team Energy: Regular socials, hackathons, and events.

We’re an equal opportunities employer and celebrate diversity in all its forms. If you need any adjustments during the recruitment process, please let us know — we’re happy to accommodate your needs.