Scrapy Scrapy / asset / free

Scrapy is a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.
3 favorites
submitted over 4 years ago, by Walkman
Scrapy popular asset



That is quite cool.. also what is web scraping vs screen scraping? I can probably remove one or the other.

pineapple, over 4 years ago


According to Wikipedia, they are two different things: "Screen scraping is normally associated with the programmatic collection of visual data from a source, instead of parsing data as in web scraping."
The Scrapy overview says it can do "screen scraping", however I don't really understand why (according to Wikipedia, it would be Web scraping, what Scrapy do), so I just made both tags.

Maybe web scraping is the subset of screen scraping, so Scrapy can do both :)

Walkman, over 4 years ago


This is really an incredible framework to work with! I was able to fully convert every bit of data from a shitty forum software to a better one with just a couple hundred lines of code with Scrapy. It has everything you need: built in cookie handling, automatic link following, data filtering, selecting data from HTML with XPath. It can even log you in to a website!
Highly recommended!

Walkman, over 4 years ago

Login or to comment.

Tutorials are any resources you learn from.

Examples: an intro to html5 screencast, a pdf about git, photoshop effects tutorials, meta-programming in ruby, lambda calculus, higher-order fixed-point combinators.

Tools are websites, apps or services used -on- your project (indirectly), to aid the process.

Examples: A color scheme generator, email marketing software, usability heat maps, css3 code generators, a downloadable png compressor.

Assets are downloadable files used -in- your projects, usually as code, textures, or images.

Examples: a jquery sticky menu, photoshop brushes, background textures, mvc frameworks, twitter bootstrap, 960 grid system.