Projects / crawl


Crawl starts a depth-first traversal of the Web at the specified URLs. It stores all JPEG images that match the configured constraints. Crawl is fairly fast and allows for graceful termination. After terminating crawl, it is possible to restart it at exactly the same spot where it was terminated. It also keeps a persistent database that allows multiple crawls without revisiting sites.

Operating Systems

Recent releases

  •  18 May 2003 02:42

    Release Notes: Crawling is more reliable, and crawl performance has improved for large crawls.

    •  29 Jan 2002 03:47

      Release Notes: A complete rewrite for higher performance, including support for all media types, asynchronous DNS lookups, and optional wait time between hosts. The configuration file specifies permittable size of media objects depending on media-type, etc.

      •  13 Dec 2001 02:36

        Release Notes: This release has portability fixes, and supports downloading of media types other than just images.

        •  03 Jul 2001 20:18

          Release Notes: This release fixes a bug where crawl would stop early when encountering errors on subsequent connections. The verbosity level and number of concurrent connections are tunable now

          •  18 Jun 2001 20:09

            No changes have been submitted for this release.


            Project Spotlight


            A Fluent OpenStack client API for Java.


            Project Spotlight

            TurnKey TWiki Appliance

            A TWiki appliance that is easy to use and lightweight.