Updates for first-in-field support + alwaysmathces searches. Refer to native

[idzebra-moved-to-github.git] / doc / introduction.xml

<chapter id="introduction">
 <!-- $Id: introduction.xml,v 1.40 2006-09-22 12:34:45 adam Exp $ -->
 <title>Introduction</title>
 
 <section id="overview">
  <title>Overview</title>
  
  <para>
   <ulink url="http://indexdata.dk/zebra/">Zebra</ulink>
   is a high-performance, general-purpose structured text
   indexing and retrieval engine. It reads records in a
   variety of input formats (eg. email, XML, MARC) and provides access
   to them through a powerful combination of boolean search
   expressions and relevance-ranked free-text queries.
  </para>

  <para>
   Zebra supports large databases (tens of millions of records,
   tens of gigabytes of data). It allows safe, incremental
   database updates on live systems. Because Zebra supports
   the industry-standard information retrieval protocol, Z39.50,
   you can search Zebra databases using an enormous variety of
   programs and toolkits, both commercial and free, which understand
   this protocol.  Application libraries are available to allow
   bespoke clients to be written in Perl, C, C++, Java, Tcl, Visual
   Basic, Python, PHP and more - see the
   <ulink url="&url.zoom;">ZOOM web site</ulink>
   for more information on some of these client toolkits.
  </para>

  <para>
   This document is an introduction to the Zebra system. It explains
   how to compile the software, how to prepare your first database,
   and how to configure the server to give you the
   functionality that you need.
  </para>
 </section>
 
 <section id="features">
  <title>Features</title>
  
  <para>
   This is an overview of some of Zebra's most important features:
  </para>
  
  <para>
   <itemizedlist>

    <listitem>
     <para>
      Very large databases: logical files can be
      automatically partitioned over multiple disks.
     </para>
    </listitem>

    <listitem>
     <para>
      Arbitrarily complex records.  The internal data format
      is a structured format conceptually similar to XML or GRS-1,
      which allows lists, nested structured data elements and
      variant forms of data.
     </para>
    </listitem>

    <listitem>
     <para>
      Robust updating - records can be added and deleted ``on the fly''
      without rebuilding the index from scratch.
      Records can be safely updated even while users are accessing
      the server.
      The update procedure is tolerant to crashes or hard interrupts
      during database updating - data can be reconstructed following
      a crash.
     </para>
    </listitem>

    <listitem>
     <para>
      Configurable to understand many input formats.
      A system of input filters driven by
      regular expressions allows most ASCII-based
      data formats to be easily processed.
      SGML, XML, ISO2709 (MARC), and raw text are also
      supported.
     </para>
    </listitem>

    <listitem>     
     <para>
      Searching supports a powerful combination of boolean queries as
      well as relevance-ranking (free-text) queries.  Truncation,
      masking, full regular expression matching and "approximate
      matching" (eg. spelling mistakes) are all handled.
     </para>
    </listitem>

    <listitem>
      <para>
        Index-only databases: data can be, and usually is, imported
        into Zebra's own storage, but Zebra can also refer to
        external files, building and maintaining indexes of "live"
        collections.
      </para>
    </listitem>

    <listitem>
     <para>
      Zebra is written in portable C, so it runs on most Unix-like systems 
      as well as Windows (NT/2000/2003).  A binary distribution for Windows 
      is available at
      <ulink url="&url.idzebra.download.win32;"/>,
      and pre-built packages are available for 
      <!--- some Linux
      distributions:
      Red Hat 7.x RPMs at
      <ulink url="http://ftp.indexdata.dk/pub/zebra/RedHat7.X/"/>
      and Debian packages at
      -->
      <literal>GNU/Debian Linux</literal> at 
      <ulink url="&url.idzebra.download.debian;"/>.
     </para>
    </listitem>
    
   </itemizedlist>
   
  </para>
  
  <para>
     <ulink url="&url.z39.50;">Z39.50</ulink> protocol support:
  </para>
  
  <para>   
   <itemizedlist>
    <listitem>
     <para>
      Protocol facilities: Init, Search, Present (retrieval),
      Segmentation (support for very large records), Delete, Scan
      (index browsing), Sort, Close and support for the ``update''
      Extended Service to add or replace an existing XML record.
     </para>
    </listitem>

    <listitem>
     <para>
      Piggy-backed presents are honored in the search request - that
      is, a subset of the found records can be returned directly with
      a search response, enabling search and retrieval to happen in a
      single round-trip.
     </para>
    </listitem>

    <listitem>
     <para>
      Named result sets are supported.
     </para>
    </listitem>

    <listitem>
     <para>
      Easily configured to support different application profiles, with
      tables for attribute sets, tag sets, and abstract syntaxes.
      Additional tables control facilities such as element mappings to
      different schema (eg., GILS-to-USMARC).
     </para>
    </listitem>

    <listitem>
     <para>
      Complex composition specifications using Espec-1 (partial support).
      Element sets are defined using the Espec-1 capability,
      and are specified in configuration files as simple element
      requests (and, optionally, variant requests).
     </para>
    </listitem>

    <listitem>
     <para>
      Multiple record syntaxes
      for data retrieval: GRS-1, SUTRS,
      XML, ISO2709 (MARC), etc. Records can be mapped between record syntaxes
      and schemas on the fly.      
     </para>
    </listitem>

   </itemizedlist>
   
  </para>

  
  <para>
    <ulink url="&url.sru;">SRU</ulink> Web Service support:
  </para>
  <para>   
   <itemizedlist>
    <listitem>
     <para>
       The protocol operations <literal>explain</literal>, 
       <literal>searchRetrieve</literal> and <literal>scan</literal>
       are supported. 
     </para>
    </listitem>
    <listitem>
     <para>
       <ulink url="&url.cql;">CQL</ulink> to internal query model RPN 
       conversion is supported. 
     </para>
    </listitem>
    <listitem>
     <para>
       Multiple XML record formats
      for data retrieval are supported, modelled over the  GRS-1, SUTRS,
      MARC record formats. Records can be mapped between record
       schemas on the fly. Arbitrarily complex XSLT transformations
      can be applied during record retrieval if one uses the 
       <literal>alvis</literal> filter module.
     </para>
    </listitem>
     <listitem>
      <para>
       Extended RPN queries for search/retrieve and scan are supported,
       for controlling approximate hit counts, etc.
     </para>
     </listitem>

   </itemizedlist>
   
  </para>

  
 </section>
 
  <section id="introduction-apps">
  <title>References and Zebra based Applications</title>
  <para>
   Zebra has been deployed in numerous applications, in both the
   academic and commercial worlds, in application domains as diverse
   as bibliographic catalogues, geospatial information, structured
   vocabulary browsing, government information locators, civic
   information systems, environmental observations, museum information
   and web indexes.
  </para>
  <para>
   Notable applications include the following:
  </para>


  <section id="koha-ils">
   <title>Koha free open-source ILS</title>
   <para>
     <ulink url="http://www.koha.org/">Koha</ulink> is a full-featured
     open-source ILS, initially developed  in 
     New Zealand by Katipo Communications Ltd, and first deployed in
     January of 2000 for Horowhenua Library Trust. It is currently
     maintained by a team of software providers and library technology
     staff from around the globe. 
    </para>
    <para>
     <ulink url="http://liblime.com/">LibLime</ulink>, 
     a company that is marketing and supporting Koha, adds in
     the new release of Koha 3.0 the Zebra
     database server to drive its bibliographic database.
    </para>
    <para>
     In early 2005, the Koha project development team began looking at
     ways to improve MARC support and overcome scalability limitations
     in the Koha 2.x series. After extensive evaluations of the best
     of the Open Source textual database engines - including MySQL
     full-text searching, PostgreSQL, Lucene and Plucene - the team
     selected Zebra. 
    </para>
    <para>
     "Zebra completely eliminates scalability limitations, because it
     can support tens of millions of records." explained Joshua
     Ferraro, LibLime's Technology President and Koha's Project
     Release Manager. "Our performance tests showed search results in
     under a second for databases with over 5 million records on a
     modest i386 900Mhz test server." 
    </para>
    <para>
     "Zebra also includes support for true boolean search expressions
     and relevance-ranked free-text queries, both of which the Koha
     2.x series lack. Zebra also supports incremental and safe
     database updates, which allow on-the-fly record
     management. Finally, since Zebra has at its heart the Z39.50
     protocol, it greatly improves Koha's support for that critical
     library standard." 
    </para>
    <para> 
     Although the bibliographic database will be moved to Zebra, Koha
     3.0 will continue to use a relational SQL-based database design
     for the 'factual' database. "Relational database managers have
     their strengths, in spite of their inability to handle large
     numbers of bibliographic records efficiently," summed up Ferraro,
     "We're taking the best from both worlds in our redesigned Koha
     3.0. 
     </para>
     <para>
     See also LibLime's newsletter article
      <ulink url="http://www.liblime.com/newsletter/2006/01/features/koha-earns-its-stripes/">
     Koha Earns its Stripes</ulink>.
     </para>
   </section>

  <section id="emilda-ils">
   <title>Emilda open source ILS</title>
   <para>
     <ulink url="http://www.emilda.org/">Emilda</ulink> 
     is a complete Integrated Library System, released under the 
     GNU General Public License. It has a
     full featured Web-OPAC, allowing comprehensive system management
     from virtually any computer with an Internet connection, has
     template based layout allowing anyone to alter the visual
     appearance of Emilda, and is
     XML based language for fast and easy portability to virtually any
     language.
     Currently, Emilda is used at three schools in Espoo, Finland.
    </para>
    <para>
     As a surplus, 100% MARC compatibility has been achieved using the
    Zebra Server from Index Data as backend server. 
    </para> 
   </section>

  <section id="reindex-ils">
   <title>ReIndex.Net web based ILS</title>
    <para>
     <ulink url="http://www.reindex.net/index.php?lang=en">Reindex.net</ulink>
     is a netbased library service offering all
     traditional functions on a very high level plus many new
     services. Reindex.net is a comprehensive and powerful WEB system
     based on standards such as XML and Z39.50.
     updates. Reindex supports MARC21, danMARC eller Dublin Core with
     UTF8-encoding.  
    </para>
    <para>
     Reindex.net runs on GNU/Debian Linux with Zebra and Simpleserver
     from Index 
     Data for bibliographic data. The relational database system
     Sybase 9 XML is used for
     administrative data. 
     Internally MARCXML is used for bibliographical records. Update
     utilizes Z39.50 extended services. 
    </para>
   </section>

   <section id="dads-article-database">
    <title>DADS - the DTV Article Database
     Service</title>
    <para>
    DADS is a huge database of more than ten million records, totalling
    over ten gigabytes of data.  The records are metadata about academic
    journal articles, primarily scientific; about 10% of these
    metadata records link to the full text of the articles they
    describe, a body of about a terabyte of information (although the
    full text is not indexed.)
   </para>
   <para>
    It allows students and researchers at DTU (Danmarks Tekniske
    Universitet, the Technical College of Denmark) to find and order
    articles from multiple databases in a single query.  The database
    contains literature on all engineering subjects.  It's available
    on-line through a web gateway, though currently only to registered
    users.
   </para>
   <para>
    More information can be found at
    <ulink url="http://www.dtv.dk/"/> and
    <ulink url="http://dads.dtv.dk"/>
   </para>
  </section>

  <section id="infonet-eprints">
   <title>Infonet Eprints</title>
   <para>
     The InfoNet Eprints service from the 
     <ulink url="http://www.dtv.dk/">
      Technical Knowledge Center of Denmark</ulink>
     provides access to documents stored in
     eprint/preprint servers and institutional research archives around
     the world. The service is based on Open Archives Initiative metadata
     harvesting of selected scientific archives around the world. These
     open archives offer free and unrestricted access to their contents.
    </para>
   <para>
    Infonet Eprints currently holds 1.4 million records from 16 archives.
    The online search facility is found at
    <ulink url="http://preprints.cvt.dk"/>.
   </para>
  </section>

  <section id="alvis-project">
   <title>Alvis</title>
   <para>
     The <ulink url="http://www.alvis.info/alvis/">Alvis</ulink> EU
     project run under the 6th Framework (IST-1-002068-STP)
     is building a semantic-based peer-to-peer search engine. A
     consortium of eleven partners from six different European
     Community countries plus Switzerland and China contribute
     with expertise in a broad range of specialties including network
     topologies, routing algorithms, linguistic analysis and
     bioinformatics. 
    </para>
    <para>
     The Zebra information retrieval indexing machine is used inside
     the Alvis framework to
     manage huge collections of natural language processed and
     enhanced XML data, coming from a topic relevant web crawl.
     In this application, Zebra swallows and manages 37GB of XML data
     in about 4 hours, resulting in search times of fractions of
     seconds. 
     </para>
   </section>


  <section id="uls">
   <title>ULS (Union List of Serials)</title>
   <para>
    The M25 Systems Team
    has created a union catalogue for the periodicals of the
    twenty-one constituent libraries of the University of London and
    the University of Westminster
    (<ulink url="http://www.m25lib.ac.uk/ULS/"/>).
    They have achieved this using an
    unusual architecture, which they describe as a
    ``non-distributed virtual union catalogue''.
   </para>
   <para>
    The member libraries send in data files representing their
    periodicals, including both brief bibliographic data and summary
    holdings.  Then 21 individual Z39.50 targets are created, each
    using Zebra, and all mounted on the single hardware server.
    The live service provides a web gateway allowing Z39.50 searching
    of all of the targets or a selection of them.  Zebra's small
    footprint allows a relatively modest system to comfortably host
    the 21 servers.
   </para>
   <para>
    More information can be found at
    <ulink url="http://www.m25lib.ac.uk/ULS/"/>
   </para>
  </section>

  <section id="nli">
   <title>NLI-Z39.50 - a Natural Language Interface for Libraries</title>
   <para>
    Fernuniversit&#x00E4;t Hagen in Germany have developed a natural
    language interface for access to library databases.
    <!-- <ulink
    url="http://ki212.fernuni-hagen.de/nli/NLIintro.html"/> -->
    In order to evaluate this interface for recall and precision, they
    chose Zebra as the basis for retrieval effectiveness.  The Zebra
    server contains a copy of the GIRT database, consisting of more
    than 76000 records in SGML format (bibliographic records from
    social science), which are mapped to MARC for presentation.
   </para>
   <para>
    (GIRT is the German Indexing and Retrieval Testdatabase.  It is a
    standard German-language test database for intelligent indexing
    and retrieval systems.  See
    <ulink url="http://www.gesis.org/forschung/informationstechnologie/clef-delos.htm"/>)
   </para>
   <para>
    Evaluation will take place as part of the TREC/CLEF campaign 2003 
    <ulink url="http://clef.iei.pi.cnr.it"/>.
    <!-- or <ulink url="http://www4.eurospider.ch/CLEF/"/> -->
   </para>
   <para>
    For more information, contact Johannes Leveling
    <email>Johannes.Leveling@FernUni-Hagen.De</email>
   </para>
  </section>

  <section id="various-web-indexes">
   <title>Various web indexes</title>
   <para>
    Zebra has been used by a variety of institutions to construct
    indexes of large web sites, typically in the region of tens of
    millions of pages.  In this role, it functions somewhat similarly
    to the engine of google or altavista, but for a selected intranet
    or a subset of the whole Web.
   </para>
   <para>
    For example, Liverpool University's web-search facility (see on
    the home page at
    <ulink url="http://www.liv.ac.uk/"/>
    and many sub-pages) works by relevance-searching a Zebra database
    which is populated by the Harvest-NG web-crawling software.
   </para>
   <para>
    For more information on Liverpool university's intranet search
    architecture, contact John Gilbertson
    <email>jgilbert@liverpool.ac.uk</email>
   </para>
   <para>
    Kang-Jin Lee
    has recently modified the Harvest web indexer to use Zebra as
    its native repository engine.  His comments on the switch over
    from the old engine are revealing:
    <blockquote>
     <para>
      The first results after some testing with Zebra are very
      promising.  The tests were done with around 220,000 SOIF files,
      which occupies 1.6GB of disk space.
     </para>
     <para>
      Building the index from scratch takes around one hour with Zebra
      where [old-engine] needs around five hours.  While [old-engine]
      blocks search requests when updating its index, Zebra can still
      answer search requests.
      [...]
      Zebra supports incremental indexing which will speed up indexing
      even further.
     </para>
     <para>
      While the search time of [old-engine] varies from some seconds
      to some minutes depending how expensive the query is, Zebra
      usually takes around one to three seconds, even for expensive
      queries.
      [...]
      Zebra can search more than 100 times faster than [old-engine]
      and can process multiple search requests simultaneously
     </para>
     <para>
      I am very happy to see such nice software available under GPL.
     </para>
    </blockquote>
   </para>
  </section>
 </section>
  
  
  <section id="introduction-support">
   <title>Support</title>
   <para>
    You can get support for Zebra from at least three sources.
   </para>
   <para>
    First, there's the Zebra web site at
    <ulink url="&url.idzebra;"/>,
     which always has the most recent version available for download.
     If you have a problem with Zebra, the first thing to do is see
     whether it's fixed in the current release.
   </para>
   <para>
    Second, there's the Zebra mailing list.  Its home page at
    <ulink url="&url.idzebra.mailinglist;"/>
     includes a complete archive of all messages that have ever been
     posted on the list.  The Zebra mailing list is used both for
     announcements from the authors (new
     releases, bug fixes, etc.) and general discussion.  You are welcome
     to seek support there.  Join by filling the form on the list home page.
   </para>
   <para>
    Third, it's possible to buy a commercial support contract, with
    well defined service levels and response times, from Index Data.
    See
    <ulink url="&url.indexdata.support;"/>
     for details.
   </para>
  </section>  


 <section id="future">
  <title>Future Directions</title>
  
  <para>
   These are some of the plans that we have for the software in the near
   and far future, ordered approximately as we expect to work on them.
  </para>
  
  <para>
   <itemizedlist>

    <listitem>
     <para>
       Improved support for XML in search and retrieval. Eventually,
       the goal is for Zebra to pull double duty as a flexible
       information retrieval engine and high-performance XML
       repository.  The recent addition of XPath searching is one
       example of the kind of enhancement we're working on.
     </para>
     <para>
       There is also the experimental <literal>ALVIS XSLT</literal>
       XML input filter, which unleashes the full power of DOM based
       XSLT transformations during indexing and record retrieval. Work
       on this filter has been sponsored by the ALVIS EU project
       <ulink url="http://www.alvis.info/alvis/"/>. We expect this filter to
       mature soon, as it is planned to be included in the version 2.0
       release of Zebra. 
     </para>
    </listitem>

    <listitem>
     <para>
       Finalisation and documentation of Zebra's C programming
       API, allowing updates, database management and other functions
       not readily expressed in Z39.50.  We will also consider
       exposing the API through SOAP.
     </para>
    </listitem>

    <listitem>
     <para>
       Improved free-text searching. We're first and foremost octet jockeys and
       we're actively looking for organisations or people who'd like
       to contribute experience in relevance ranking and text
       searching.
     </para>
    </listitem>

   </itemizedlist>
  </para>
  
  <para>
   Programmers thrive on user feedback. If you are interested in a
   facility that you don't see mentioned here, or if there's something
   you think we could do better, please drop us a mail.  Better still,
   implement it and send us the patches.
  </para>
  <para>
   If you think it's all really neat, you're welcome to drop us a line
   saying that, too. You can email us on
   <email>info@indexdata.dk</email>
   or check the contact info at the end of this manual.
  </para>
  
 </section>
</chapter>
 <!-- Keep this comment at the end of the file
 Local variables:
 mode: sgml
 sgml-omittag:t
 sgml-shorttag:t
 sgml-minimize-attributes:nil
 sgml-always-quote-attributes:t
 sgml-indent-step:1
 sgml-indent-data:t
 sgml-parent-document: "zebra.xml"
 sgml-local-catalogs: nil
 sgml-namecase-general:t
 End:
 -->