# red-spider

**Repository Path**: CJP/red-spider

## Basic Information

- **Project Name**: red-spider
- **Description**: :spider:基于webmagic的可定制规则爬虫
- **Primary Language**: HTML
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2020-11-13
- **Last Updated**: 2020-12-19

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# red-spider
:spider:
[![HitCount](http://hits.dwyl.io/jpcui/red-spider.svg)](http://hits.dwyl.io/jpcui/red-spider)
[![Build Status](https://travis-ci.org/JPCui/red-spider.svg?branch=master)](https://travis-ci.org/JPCui/red-spider)


# TODO

- 去除

- 模块化

- 实现URL分发器

	- 发现URL交由分发器分发到不同队列 

- 重写 downloader，失败要归还到爬取队列

- 失败监听，记录失败URL，告警