Requirements
- The role is ideal for someone who believes in agentic coding and would like to improve their skills
- Strong professional experience with Python (or another main language, e.g. GoLang, Rust, JavaScript) including production‑grade scraping systems
- Deep hands‑on experience scraping Google or similarly protected platforms
- Solid understanding of HTTP, TLS, cookies, headers, redirects, and browser networking behaviour
- Experience with tools such as Playwright, Selenium, Puppeteer, or equivalent browser automation frameworks
- Strong knowledge of HTML parsing, DOM traversal, and data extraction techniques
- Experience handling rate limiting, CAPTCHA, IP rotation, and bot detection systems
- Comfortable working with asynchronous and concurrent scraping architectures
- Experience running scrapers at scale in cloud environments
- Strong debugging skills and ability to reason about complex failure modes
- Experience with headless browser fingerprinting and anti‑bot evasion techniques
- (Desirable) Familiarity with containerisation and orchestration, such as Docker and Kubernetes
- (Desirable) Experience with distributed task systems and job queues
- (Desirable) Background in data quality monitoring and anomaly detection
- (Desirable) Experience working with search, advertising, or competitive intelligence data
What the job involves
- Reporting to: Michal Gondar, Head of Engineering
- As a Senior Anti‑Bot Engineer, you will design, build, and operate large‑scale, resilient web scraping systems, with a strong focus on Google and Google‑like search environments
- This role sits at the intersection of data engineering, reverse engineering, and systems reliability, and is critical to delivering accurate, timely, and trustworthy data at scale
- You will work on complex scraping problems where naïve approaches do not work, including dynamic rendering, anti‑bot defences, frequent markup changes, and high‑volume data collection
- Design and maintain large‑scale Python scraping systems that operate reliably against Google and other highly defended websites
- Build scrapers for dynamic, JavaScript‑heavy pages using browser automation and hybrid approaches
- Continuously adapt to changes in page structure, request flows, and anti‑automation mechanisms
- Engineer robust extraction pipelines that prioritise data accuracy, consistency, and observability
- Implement proxy, fingerprinting, and request‑routing strategies to maximise stability and throughput
- Monitor scrape health, detect failures early, and rapidly debug production issues
- Optimise performance, cost, and latency across scraping infrastructure
- Work closely with data engineers, data scientists, and product teams to ensure scraped data is usable and trusted
- Document scraping logic and operational runbooks for long‑term maintainability
- You’ll work closely with:
- Paul Felby, Chief Product & Tech Officer & Ivan Nikolov, Lead Technical Architect
