At a Glance
- Tasks: Design and operate resilient web scraping systems to capture data at scale.
- Company: Join Adthena, a leading search intelligence platform with a vibrant culture.
- Benefits: Enjoy remote work, flexible hours, and a range of health and wellness perks.
- Other info: Collaborate with a passionate team and grow your career in an innovative setting.
- Why this job: Make a real impact in a dynamic tech environment while solving challenging problems.
- Qualifications: Deep Python skills and experience with web scraping technologies required.
The predicted salary is between 36000 - 60000 £ per year.
Our Mission
Join us in shaping the future of paid search intelligence. Adthena is an award-winning search intelligence platform powered by patented AI technology and a team of dedicated experts. We help brands, marketers, and agencies dominate their competitive landscapes with unparalleled insights into the paid search market. Our Whole Market View, Smart Monitor, and Local View solutions give clients the clarity to optimize spend, increase ROI, and stay ahead of the competition.
Trusted by global brands like Citibank, L’Oréal, and Volvo, and backed by Updata Partners, we’ve been redefining search intelligence since 2012. Ready to make an impact in an industry-leading company? Let’s do it together.
Why Join Us? We offer the best of both worlds: the maturity and stability of a well-established company combined with the energy, agility, and impact of a scaling tech organisation. If you’ve been hesitant about joining a typical tech company due to concerns about chaos, fragility, or volatility, Adthena is your answer.
- We’re smaller than large-scale enterprises, offering more autonomy, less red tape, and accelerated growth.
- Startup Engineering culture
- Strong Culture: company culture approval and engagement metrics
- Award-Winning Product and Stock Options
- Trust-Based Vacation, Remote-First with 3 Geohubs (London, Austin, Sydney)
- Flexible Work, Career Growth via bespoke training (Sherpa plans)
- Regular Hackdays, Training Days, Private Medical Insurance, Mental Health Support
- Family Care Package, Pension, Home Office Stipend, Swag, Birthday Day Off
- Volunteer Days and social events
About The Role
Reporting to: Vasanth John, VP of Engineering
Location: United Kingdom
Start Date: ASAP
More info: See FAQ below, our candidate hub, or reach out to recruitment@adthena.com
About Your Team
Adthena is a market leader in developing complex ETL and machine learning solutions. Our scraping team captures web and AI data at scale, including sources like Google’s AI Overviews, Perplexity, and ChatGPT, processing hundreds of millions of data points daily. We value curiosity, collaboration, and ownership, and engineers grow by solving challenging problems at scale.
What You’ll Do At Adthena
As a Web Scrape Engineer, you’ll design and operate resilient pipelines that can outsmart anti-bot defenses, adapt to fast-moving platforms, and deliver clean data into our core systems. You’ll collaborate with Data Engineering and DevOps to ensure scrapers are reliable, observable, and cost-efficient in the cloud. This role suits someone who thrives independently, enjoys solving hard problems at scale, and wants visibility of impact for global customers.
Responsibilities
- Lead design, build, deploy and maintain scalable, resilient web / AI / LLM scraping systems, ensuring optimal performance.
- Innovate anti-bot / evasion strategies: proxies, headless browsers, fingerprinting, CAPTCHAs.
- Integrate scrape outputs into Adthena’s pipeline: cleaning, deduplication, schema, delivery to our warehouse.
- Ensure resilience: monitoring, alerting, retries, failure modes, fallback strategies.
- Work with cloud infra (containers, orchestration), collaborating with DevOps for cost, reliability, deployment.
- Proactively adapt to upstream changes (site changes, new anti-bot defenses).
- Maintain strong documentation, best practices, code hygiene; communicate across time zones and teams.
Who We Look For
We encourage applicants from diverse backgrounds. If you have the skills but don’t meet every requirement, reach out – we’d love to explore how you could fit.
Preferred Experience
- Bachelor degree in Computer Science or equivalent practical experience.
- Minimum 5 years of commercial experience building and running production web scraping systems at scale.
- Deep Python skills; experience with Scrapy, Playwright, or Selenium.
- Practical anti-bot experience: proxies, CAPTCHAs, fingerprinting, JavaScript rendering.
- Experience in distributed systems and pipelines (Spark, Airflow) with integration into data warehouses (Redshift, PostgreSQL).
- Commercial experience scraping LLM/AI sources (Perplexity, ChatGPT, Claude, Gemini).
- Experience deploying scraping workloads in AWS/GCP using Docker and Kubernetes.
- Proven ability to build observable and resilient systems with Grafana, Prometheus, ELK.
- Strong debugging, performance optimization, reliability focus; independent self-starter with good communication.
Bonus Points If You Have
- Awareness of legal, ethical, and compliance considerations in scraping (GDPR, terms of service).
- Familiarity with Kafka or RabbitMQ.
- Infrastructure as code (Terraform, Ansible) and CI/CD pipelines (Jenkins, GitHub Actions).
- Cloud cost optimisation and scaling strategies for large-scale scraping.
Our Core Values
- Get Shit Done: taking action, owning challenges, and solving them.
- Freedom with Responsibility: delivering results and personal success.
- Deliver Remarkable Quality: strive to exceed expectations.
- Smart & Always Improving: continual learning and improvement.
Meet your teammates
You’ll work closely with:
- Paul Felby — Chief Product & Tech Officer
- Vasanth John — VP of Engineering
- Ivan Nikolov — Lead Technical Architect
Seniority level
- Mid-Senior level
Employment type
- Full-time
Job function
- Engineering and Information Technology
Industries
- Software Development
Web Scraping Engineer (Python) in City of London employer: Adthena
Adthena is an exceptional employer that combines the stability of an established company with the dynamic environment of a scaling tech organisation. With a strong focus on employee growth through bespoke training, trust-based vacation policies, and a remote-first culture, we empower our team to thrive while working on cutting-edge web scraping technologies. Our commitment to innovation, collaboration, and a supportive work culture makes Adthena a fantastic place for those looking to make a meaningful impact in the tech industry.
StudySmarter Expert Advice🤫
We think this is how you could land Web Scraping Engineer (Python) in City of London
✨Tip Number 1
Network like a pro! Reach out to people in the industry, attend meetups, and connect with potential colleagues on LinkedIn. You never know who might have the inside scoop on job openings or can refer you directly.
✨Tip Number 2
Show off your skills! Create a portfolio showcasing your web scraping projects. Whether it's a GitHub repo or a personal website, having tangible examples of your work can really set you apart from the crowd.
✨Tip Number 3
Prepare for interviews by practising common technical questions related to web scraping and Python. Brush up on anti-bot strategies and be ready to discuss how you've tackled challenges in past projects.
✨Tip Number 4
Don't forget to apply through our website! It’s the best way to ensure your application gets seen by the right people. Plus, it shows you're genuinely interested in joining our awesome team at Adthena!
We think you need these skills to ace Web Scraping Engineer (Python) in City of London
Some tips for your application 🫡
Tailor Your CV:Make sure your CV is tailored to the Web Scraping Engineer role. Highlight your Python skills and any experience with scraping systems, as well as your ability to tackle anti-bot challenges. We want to see how your background aligns with our mission!
Craft a Compelling Cover Letter:Your cover letter is your chance to shine! Share your passion for web scraping and how you can contribute to Adthena's innovative solutions. Don’t forget to mention specific projects or experiences that showcase your problem-solving skills.
Showcase Your Projects:If you've worked on relevant projects, whether personal or professional, make sure to include them in your application. We love seeing real-world examples of your work, especially if they demonstrate your ability to build resilient scraping systems.
Apply Through Our Website:We encourage you to apply directly through our website. It’s the best way to ensure your application gets into the right hands. Plus, it shows us you're serious about joining our team at Adthena!
How to prepare for a job interview at Adthena
✨Know Your Tech Stack
Make sure you’re well-versed in Python and the specific tools mentioned in the job description, like Scrapy, Playwright, or Selenium. Brush up on your anti-bot strategies and be ready to discuss how you've implemented them in past projects.
✨Showcase Problem-Solving Skills
Prepare examples of challenging problems you've solved in web scraping or data engineering. Be ready to explain your thought process and the impact of your solutions, especially in terms of performance optimisation and reliability.
✨Understand the Company Culture
Familiarise yourself with Adthena’s core values, such as 'Get Shit Done' and 'Freedom with Responsibility'. Think of ways you can demonstrate these values through your past experiences and how they align with your work ethic.
✨Ask Insightful Questions
Prepare thoughtful questions about the team dynamics, the technologies they use, and their approach to tackling anti-bot challenges. This shows your genuine interest in the role and helps you assess if it’s the right fit for you.
