Urgently hiring 2 - Web Crawler Engineers

Salary: 30K-55K (excluding project bonuses, negotiable for exceptional candidates)

Working hours: 10 days on, 4 days off (12:00-24:00 with a 2-hour break for meals)

Probation period: 1 month

Company projects: IM development experience

Location: Initially remote, then on-site (Thailand/Bangkok, work from home)

1. Responsible for the development and maintenance of web data scraping for key business areas

2. Design and optimize stable and efficient crawler programs, addressing anti-scraping strategies (such as CAPTCHA, JS obfuscation, IP blocking, etc.)

3. Build and maintain distributed crawling systems to achieve large-scale concurrent data collection

4. Clean, deduplicate, and structure the collected data to ensure data quality

5. Monitor system operation status, promptly address anomalies, and improve data acquisition efficiency and stability

6. Assist the data team in completing data integration and tag extraction tasks

1. Bachelor’s degree or above in Computer Science, Software Engineering, or related fields, with 2-3 years or more of practical web scraping development experience.

2. Proficient in Python, familiar with mainstream web scraping frameworks such as Scrapy, Requests, Selenium, Playwright, etc.

3. Mastery of HTML / XPath / CSS Selector / Regular Expressions and other page parsing techniques.

4. Familiar with common anti-scraping mechanisms and their countermeasures, such as proxy pools, user-agent pools, CAPTCHA recognition, JS decryption, dynamic page loading, etc.

5. Experience with data storage technologies such as Redis, MongoDB, MySQL, etc.

6. Familiar with multithreading/asynchronous programming, possessing good code structure and development habits.

7. Some proficiency in Linux operations, able to independently deploy and maintain scraping scripts.

8. Responsible, logically clear, strong under pressure, and possess good teamwork spirit.

Preferred Qualifications:

1. Experience with Scrapy-Redis distributed web scraping projects.

2. Familiar with asynchronous task frameworks such as Kafka/Celery.

3. Experience handling image CAPTCHAs or sliding CAPTCHAs is preferred.

4. Familiar with common anti-cheating strategies and countermeasures.

5. Experience with web scraping projects related to adult content.