tag: web crawlers

  • FandangoSEO FandangoSEO / both

    FandangoSEO is the brand new SEO Crawler & Log Analyzer on the cloud, born in Barcelona to help you rock any SEO strategy and increase traffic with visual and clear data

    0 comments | 0 favorites

    about 1 month ago, anonymously

    FandangoSEO
    popular tool
  • Rocket Validator Rocket Validator / both

    Rocket Validator is the fastest way to validate HTML5 on large sites. Just submit a starting URL, and our web crawler will find the internal pages of your site (up to 5,000 pages!), validate HTML on each of them using the latest validator from the W3C, and give you back the results in seconds!

    0 comments | 1 favorite

    12 months ago, by jaimeiniesta

    Rocket Validator
    popular tool
  • Apifier Apifier / free

    Apifier is a hosted web crawler that enables anyone with elementary coding skills to extract structured data from any website using a few simple lines of JavaScript code.

    0 comments | 0 favorites

    over 1 year ago, anonymously

    Apifier
    popular tool
  • Robots.txt Robots.txt / free

    Web site owners use the robots.txt file to give instructions about their site to web robots. This is called The Robots Exclusion Protocol.

    0 comments | 0 favorites

    almost 4 years ago, by Walkman

    Robots.txt
    popular tutorial
  • Scrapy Scrapy / free

    Scrapy is a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.

    3 comments | 3 favorites

    almost 4 years ago, by Walkman

    Scrapy
    popular asset

Tutorials are any resources you learn from.

Examples: an intro to html5 screencast, a pdf about git, photoshop effects tutorials, meta-programming in ruby, lambda calculus, higher-order fixed-point combinators.

Tools are websites, apps or services used -on- your project (indirectly), to aid the process.

Examples: A color scheme generator, email marketing software, usability heat maps, css3 code generators, a downloadable png compressor.

Assets are downloadable files used -in- your projects, usually as code, textures, or images.

Examples: a jquery sticky menu, photoshop brushes, background textures, mvc frameworks, twitter bootstrap, 960 grid system.