next up previous contents index
Next: 4.4.1 Default actions of Up: 4 The Gatherer Previous: Gatherer enumeration vs.

4.4 Extracting data for indexing: The Essence summarizing subsystem

   

After the Gatherer retrieves a document, it passes the document through a subsystem called Essence [14,13] to extract indexing information. Essence allows the Gatherer to collect indexing information easily from a wide variety of information, using different techniques depending on the type of data and the needs of the particular corpus being indexed. In a nutshell, Essence can determine the type of data pointed to by a URL (e.g., PostScript vs. HTML) gif, ``unravel'' presentation nesting formats (such as compressed ``tar'' files), select which types of data to index (e.g., don't index Audio files), and then apply a type-specific extraction algorithm (called a summarizer) to the data to generate a content summary. Users can customize each of these aspects, but often this is not necessary: Harvest is distributed with a ``stock'' set of type recognizers, presentation unnesters, candidate selectors, and summarizers that work well for many applications.

Starting with Harvest Version 1.2 we are also integrating support for summarizers based on outside ``component technologies'' of both a free and a commercial nature.

Below we describe the stock summarizer set, the current components distribution, and how users can customize summarizers to change how they operate and add summarizers for new types of data. If you develop a summarizer (or an interface to a commercial system) that is likely to be useful to other users, please notify us via email at harvest-dvl@cs.colorado.edu so we may include it in our components distribution.





next up previous contents index
Next: 4.4.1 Default actions of Up: 4 The Gatherer Previous: Gatherer enumeration vs.



Darren Hardy
Mon Apr 3 15:22:37 MDT 1995