Senior Python Scraping / Anti-Bot Engineer

Requirements

• The role is ideal for someone who believes in agentic coding and would like to improve their skills.
• Strong professional experience with Python (or another main language, e.g. GoLang, Rust, JavaScript) including production‑grade scraping systems.
• Deep hands‑on experience scraping Google or similarly protected platforms.
• Solid understanding of HTTP, TLS, cookies, headers, redirects, and browser networking behaviour.
• Experience with tools such as Playwright, Selenium, Puppeteer, or equivalent browser automation frameworks.
• Strong knowledge of HTML parsing, DOM traversal, and data extraction techniques.
• Experience handling rate limiting, CAPTCHA, IP rotation, and bot detection systems.
• Comfortable working with asynchronous and concurrent scraping architectures.
• Experience running scrapers at scale in cloud environments.
• Strong debugging skills and ability to reason about complex failure modes.
• Experience with headless browser fingerprinting and anti‑bot evasion techniques (Desirable).
• Familiarity with containerisation and orchestration, such as Docker and Kubernetes (Desirable).
• Experience with distributed task systems and job queues (Desirable).
• Background in data quality monitoring and anomaly detection (Desirable).
• Experience working with search, advertising, or competitive intelligence data.

What the job involves

• Reporting to: Michal Gondar, Head of Engineering.
• As a Senior Anti‑Bot Engineer, you will design, build, and operate large‑scale, resilient web scraping systems, with a strong focus on Google and Google‑like search environments.
• This role sits at the intersection of data engineering, reverse engineering, and systems reliability, and is critical to delivering accurate, timely, and trustworthy data at scale.
• You will work on complex scraping problems where naïve approaches do not work, including dynamic rendering, anti‑bot defences, frequent markup changes, and high‑volume data collection.
• Design and maintain large‑scale Python scraping systems that operate reliably against Google and other highly defended websites.
• Build scrapers for dynamic, JavaScript‑heavy pages using browser automation and hybrid approaches.
• Continuously adapt to changes in page structure, request flows, and anti‑automation mechanisms.
• Engineer robust extraction pipelines that prioritise data accuracy, consistency, and observability.
• Implement proxy, fingerprinting, and request‑routing strategies to maximise stability and throughput.
• Monitor scrape health, detect failures early, and rapidly debug production issues.
• Optimise performance, cost, and latency across scraping infrastructure.
• Work closely with data engineers, data scientists, and product teams to ensure scraped data is usable and trusted.
• Document scraping logic and operational runbooks for long‑term maintainability.

You’ll work closely with: Paul Felby, Chief Product & Tech Officer & Ivan Nikolov, Lead Technical Architect.