dvd888
2020-03-08 21:49:49 +08:00
Data source which need crawl: instagram, facebook, youtube, twitter, tiktok.
- The data crawlers need to capture includes content (text, pictures, videos) which be published by bloggers, as well as comments from readers.
- The list of bloggers can change at any time. crawlers don't need to care about specific lists.
- Crawlers need to be able to efficiently crawl the latest and hottest content, usually under a tag or category.
- Crawlers need effective solutions to resist the anti-crawler mechanism of data sources