Senior Web Scraping Engineer

At WebAutomation, we help businesses unlock the power of public web data at scale. Our products and custom scraping solutions empower organizations to access accurate, structured, and reliable datasets. We are an international, remote-first team passionate about cutting-edge web scraping technologies and large-scale data automation.

Role Overview

We are looking for a Senior Web Scraping Engineer with strong expertise in Python and Node.js to lead complex data extraction projects. You will design scalable scrapers, overcome advanced anti-bot systems, and build robust pipelines that power our products.

This role is ideal for someone who thrives on solving tough technical challenges, can effectively manage multiple projects in parallel, and quickly adapts to changing priorities in a fast-paced environment.

Key Responsibilities

• Build and maintain advanced scrapers using Python, Node.js, Scrapy, Puppeteer, Playwright, Selenium
• Optimize pipelines for scalability, efficiency, and minimal resource usage
• Debug, maintain, and enhance existing scraping systems under changing site structures
• Handle multiple scraping projects simultaneously while meeting deadlines and quality standardsAdapt quickly to daily priority changes and shifting project requirements
• Collaborate with product and data teams to ensure accuracy and reliability
• Mentor junior engineers and contribute to best practices in large-scale scraping
• Expertise in analyzing and extracting data from GraphQL endpoints, including introspection queries and schema discovery
• Experience with WebSocket reverse engineering for real-time data extraction
• Proficiency in analyzing API authentication flows (OAuth, JWT tokens, API key rotation, session management)
• Experience bypassing CloudFlare, PerimeterX, DataDome and other anti-bot solutions
• Skills in analyzing and replicating complex request signatures and encryption mechanisms

Must-Have Experience

• 5+ years in web scraping, backend engineering, or data automation
• Expert in Python (asyncio, threading) and Node.js (async/await)
• Hands-on with Scrapy, Puppeteer, Playwright, Selenium
• Strong debugging and problem-solving skills for complex scrapers
• Strong multi-tasking skills with the ability to prioritize work in a dynamic environment
• Database experience (SQL/NoSQL, Redis) and caching systems

Nice to Have

• Experience with Flask, Django, FastAPI
• Docker, Kubernetes, and CI/CD pipelines
• R&D or security research background
• Open-source contributions in scraping/data engineering

What We Offer

• Competitive salary based on expertise
• 100% remote & flexible work environment
• Opportunity to work on cutting-edge web & app scraping projects
• A collaborative international team that values innovation and autonomy
• Professional growth through challenging projects & mentorship

🚀 Ready to push the boundaries of web data extraction?

Apply now and join us at WebAutomation to shape the future of large-scale web scraping.