Web Scraping Engineer

Senior Python Scraping Engineer – Permanent

Location: Remote

What You\’ll Do

• Design and maintain large‑scale Python scraping systems that operate reliably against highly defended websites.
• Build scrapers for dynamic, JavaScript‑heavy pages using browser automation and hybrid techniques.
• Continuously adapt to changes in structure, request flows, and anti‑automation mechanisms.
• Engineer robust extraction pipelines focused on accuracy, consistency, and observability.
• Implement proxy, fingerprinting, and request‑routing strategies to increase stability and throughput.
• Monitor scrape health, detect failures early, and quickly resolve issues in production.
• Optimise performance, cost, and latency across scraping infrastructure.
• Collaborate with data engineering, data science, and product teams to ensure scraped data is reliable and usable.
• Maintain clear documentation and operational runbooks for long‑term maintainability.

Required Skills and Experience

• Strong professional experience with Python, including production‑grade scraping systems.
• Deep hands‑on experience scraping Google or similarly protected platforms.
• Solid understanding of HTTP, TLS, cookies, headers, redirects, and browser‑network behaviour.
• Experience with tools such as Playwright, Selenium, Puppeteer, or similar browser‑automation frameworks.
• Strong knowledge of HTML parsing, DOM traversal, and data‑extraction techniques.
• Experience handling rate limiting, CAPTCHAs, IP rotation, and bot‑detection systems.
• Comfortable working with asynchronous and concurrent scraping architectures.
• Experience running scrapers at scale in cloud environments.
• Strong debugging skills and the ability to reason about complex failure modes.
• Experience with headless browser fingerprinting and anti‑bot evasion techniques.

Nice to Have

• Familiarity with containerisation and orchestration technologies such as Docker and Kubernetes.
• Experience with distributed task systems and job queues.
• Background in data‑quality monitoring and anomaly detection.
• Experience working with search, advertising, or competitive‑intelligence data.

What We Value

• Pragmatic problem‑solving in hostile and fast‑changing technical environments.
• High ownership of systems that run continuously and at scale.
• A strong bias toward reliability, correctness, and observability.
• Ability to explain complex technical behaviour to non‑specialists.
• Curiosity about how large platforms evolve their defences and infrastructure.

Please send me a copy of your CV if you\’re interested and meet the above