时空索引>> Web-Spider>> 返回
项目作者： techguy-bhushan

项目描述 ：
multi-threaded webs crawler
高级语言： Python
项目地址: git://github.com/techguy-bhushan/Web-Spider.git
创建时间： 2017-09-22T04:34:37Z
项目社区：https://github.com/techguy-bhushan/Web-Spider
开源协议：
下载


Web-Spider

multi-threaded webs crawler

Required in System for run project

  • Python

Run trigger.py, before run this script please edit this script set HOME_PAGE and PROJECT_NAME parameter value
PROJECT_NAME value is name of directory where all data will store, HOME_PAGE is base url of web site which you want crawler e.x www.w2school.com

It will collect all links from that web site and store in crawled.txt file