Change makefiles so that html files are automatically installed and

[idzebra-moved-to-github.git] / doc / architecture.xml

 <chapter id="architecture">
  <!-- $Id: architecture.xml,v 1.12 2006-09-03 21:37:26 adam Exp $ -->
  <title>Overview of Zebra Architecture</title>
  

  <section id="architecture-representation">
   <title>Local Representation</title>
   
   <para>
    As mentioned earlier, Zebra places few restrictions on the type of
    data that you can index and manage. Generally, whatever the form of
    the data, it is parsed by an input filter specific to that format, and
    turned into an internal structure that Zebra knows how to handle. This
    process takes place whenever the record is accessed - for indexing and
    retrieval.
   </para>

   <para>
    The RecordType parameter in the <literal>zebra.cfg</literal> file, or
    the <literal>-t</literal> option to the indexer tells Zebra how to
    process input records.
    Two basic types of processing are available - raw text and structured
    data. Raw text is just that, and it is selected by providing the
    argument <emphasis>text</emphasis> to Zebra. Structured records are
    all handled internally using the basic mechanisms described in the
    subsequent sections.
    Zebra can read structured records in many different formats.
    <!--
    How this is done is governed by additional parameters after the
    "grs" keyword, separated by "." characters.
    -->
   </para>
  </section>

  <section id="architecture-maincomponents">
   <title>Main Components</title>
   <para>
    The Zebra system is designed to support a wide range of data management
    applications. The system can be configured to handle virtually any
    kind of structured data. Each record in the system is associated with
    a <emphasis>record schema</emphasis> which lends context to the data
    elements of the record.
    Any number of record schemas can coexist in the system.
    Although it may be wise to use only a single schema within
    one database, the system poses no such restrictions.
   </para>
   <para>
    The Zebra indexer and information retrieval server consists of the
    following main applications: the <command>zebraidx</command>
    indexing maintenance utility, and the <command>zebrasrv</command>
    information query and retrieval server. Both are using some of the
    same main components, which are presented here.
   </para>    
   <para>    
    The virtual Debian package <literal>idzebra-2.0</literal>
    installs all the necessary packages to start
    working with Zebra - including utility programs, development libraries,
    documentation and modules. 
  </para>    
   
   <section id="componentcore">
    <title>Core Zebra Libraries Containing Common Functionality</title>
    <para>
     The core Zebra module is the meat of the <command>zebraidx</command>
    indexing maintenance utility, and the <command>zebrasrv</command>
    information query and retrieval server binaries. Shortly, the core
    libraries are responsible for  
     <variablelist>
      <varlistentry>
       <term>Dynamic Loading</term>
       <listitem>
        <para>of external filter modules, in case the application is
        not compiled statically. These filter modules define indexing,
        search and retrieval capabilities of the various input formats.  
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term>Index Maintenance</term>
       <listitem>
        <para> Zebra maintains Term Dictionaries and ISAM index
        entries in inverted index structures kept on disk. These are
        optimized for fast inset, update and delete, as well as good
        search performance.
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term>Search Evaluation</term>
       <listitem>
        <para>by execution of search requests expressed in PQF/RPN
         data structures, which are handed over from
         the YAZ server frontend API. Search evaluation includes
         construction of hit lists according to boolean combinations
         of simpler searches. Fast performance is achieved by careful
         use of index structures, and by evaluation specific index hit
         lists in correct order. 
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term>Ranking and Sorting</term>
       <listitem>
        <para>
         components call resorting/re-ranking algorithms on the hit
         sets. These might also be pre-sorted not only using the
         assigned document ID's, but also using assigned static rank
         information. 
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term>Record Presentation</term>
       <listitem>
        <para>returns - possibly ranked - result sets, hit
         numbers, and the like internal data to the YAZ server backend API
         for shipping to the client. Each individual filter module
         implements it's own specific presentation formats.
        </para>
       </listitem>
      </varlistentry>
     </variablelist>
     </para>
    <para> 
     The Debian package <literal>libidzebra-2.0</literal> 
     contains all run-time libraries for Zebra, the 
     documentation in PDF and HTML is found in 
     <literal>idzebra-2.0-doc</literal>, and
     <literal>idzebra-2.0-common</literal>
     includes common essential Zebra configuration files.
    </para>
   </section>
   

   <section id="componentindexer">
    <title>Zebra Indexer</title>
    <para>
     The  <command>zebraidx</command>
     indexing maintenance utility 
     loads external filter modules used for indexing data records of
     different type, and creates, updates and drops databases and
     indexes according to the rules defined in the filter modules.
    </para>    
    <para>    
     The Debian  package <literal>idzebra-2.0-utils</literal> contains
     the  <command>zebraidx</command> utility.
    </para>
   </section>

   <section id="componentsearcher">
    <title>Zebra Searcher/Retriever</title>
    <para>
     This is the executable which runs the Z39.50/SRU/SRW server and
     glues together the core libraries and the filter modules to one
     great Information Retrieval server application. 
    </para>    
    <para>    
     The Debian  package <literal>idzebra-2.0-utils</literal> contains
     the  <command>zebrasrv</command> utility.
    </para>
   </section>

   <section id="componentyazserver">
    <title>YAZ Server Frontend</title>
    <para>
     The YAZ server frontend is 
     a full fledged stateful Z39.50 server taking client
     connections, and forwarding search and scan requests to the 
     Zebra core indexer.
    </para>
    <para>
     In addition to Z39.50 requests, the YAZ server frontend acts
     as HTTP server, honoring
      <ulink url="http://www.loc.gov/standards/sru/srw/">SRW</ulink> 
     SOAP requests, and  
     <ulink url="&url.sru;">SRU</ulink> 
     REST requests. Moreover, it can
     translate incoming 
     <ulink url="&url.cql;">CQL</ulink>
     queries to
     <ulink url="http://indexdata.com/yaz/doc/tools.tkl#PQF">PQF</ulink>
      queries, if
     correctly configured. 
    </para>
    <para>
     <ulink url="http://www.indexdata.com/yaz">YAZ</ulink>
     is an Open Source  
     toolkit that allows you to develop software using the
     ANSI Z39.50/ISO23950 standard for information retrieval.
     It is packaged in the Debian packages     
     <literal>yaz</literal> and <literal>libyaz</literal>.
    </para>
   </section>
   
   <section id="componentmodules">
    <title>Record Models and Filter Modules</title>
    <para>
     The hard work of knowing <emphasis>what</emphasis> to index, 
     <emphasis>how</emphasis> to do it, and <emphasis>which</emphasis>
     part of the records to send in a search/retrieve response is
     implemented in 
     various filter modules. It is their responsibility to define the
     exact indexing and record display filtering rules.
     </para>
     <para>
     The virtual Debian package
     <literal>libidzebra-2.0-modules</literal> installs all base filter
     modules. 
    </para>

   
   <section id="componentmodulestext">
    <title>TEXT Record Model and Filter Module</title>
    <para>
      Plain ASCII text filter. TODO: add information here.
    </para>
   </section>

   <section id="componentmodulesgrs">
    <title>GRS Record Model and Filter Modules</title>
    <para>
    The GRS filter modules described in 
    <xref linkend="grs"/>
    are all based on the Z39.50 specifications, and it is absolutely
    mandatory to have the reference pages on BIB-1 attribute sets on
    you hand when configuring GRS filters. The GRS filters come in
    different flavors, and a short introduction is needed here.
    GRS filters of various kind have also been called ABS filters due
    to the <filename>*.abs</filename> configuration file suffix.
    </para>
    <para>
      The <emphasis>grs.marc</emphasis> and 
      <emphasis>grs.marcxml</emphasis> filters are suited to parse and
      index binary and XML versions of traditional library MARC records 
      based on the ISO2709 standard. The Debian package for both
      filters is 
     <literal>libidzebra-2.0-mod-grs-marc</literal>.
    </para>
    <para>
      GRS TCL scriptable filters for extensive user configuration come
     in two flavors: a regular expression filter 
     <emphasis>grs.regx</emphasis> using TCL regular expressions, and
     a general scriptable TCL filter called 
     <emphasis>grs.tcl</emphasis>        
     are both included in the 
     <literal>libidzebra-2.0-mod-grs-regx</literal> Debian package.
    </para>
    <para>
      A general purpose SGML filter is called
     <emphasis>grs.sgml</emphasis>. This filter is not yet packaged,
     but planned to be in the  
     <literal>libidzebra-2.0-mod-grs-sgml</literal> Debian package.
    </para>
    <para>
      The Debian  package 
      <literal>libidzebra-2.0-mod-grs-xml</literal> includes the 
      <emphasis>grs.xml</emphasis> filter which uses <ulink
      url="http://expat.sourceforge.net/">Expat</ulink> to 
      parse records in XML and turn them into IDZebra's internal GRS node
      trees. Have also a look at the Alvis XML/XSLT filter described in
      the next session.
    </para>
   </section>

   <section id="componentmodulesalvis">
    <title>ALVIS Record Model and Filter Module</title>
     <para>
      The Alvis filter for XML files is an XSLT based input
      filter. 
      It indexes element and attribute content of any thinkable XML format
      using full XPATH support, a feature which the standard Zebra
      GRS SGML and XML filters lacked. The indexed documents are
      parsed into a standard XML DOM tree, which restricts record size
      according to availability of memory.
    </para>
    <para>
      The Alvis filter 
      uses XSLT display stylesheets, which let
      the Zebra DB administrator associate multiple, different views on
      the same XML document type. These views are chosen on-the-fly in
      search time.
     </para>
    <para>
      In addition, the Alvis filter configuration is not bound to the
      arcane  BIB-1 Z39.50 library catalogue indexing traditions and
      folklore, and is therefore easier to understand.
    </para>
    <para>
      Finally, the Alvis  filter allows for static ranking at index
      time, and to to sort hit lists according to predefined
      static ranks. This imposes no overhead at all, both
      search and indexing perform still 
      <emphasis>O(1)</emphasis> irrespectively of document
      collection size. This feature resembles Googles pre-ranking using
      their Pagerank algorithm.
    </para>
    <para>
      Details on the experimental Alvis XSLT filter are found in 
      <xref linkend="record-model-alvisxslt"/>.
      </para>
     <para>
      The Debian package <literal>libidzebra-2.0-mod-alvis</literal>
      contains the Alvis filter module.
     </para>
    </section>

    <!--
   <section id="componentmodulessafari">
    <title>SAFARI Record Model and Filter Module</title>
    <para>
     SAFARI filter module TODO: add information here.
    </para>
   </section>
    -->

   </section>

  </section>


  <section id="architecture-workflow">
   <title>Indexing and Retrieval Workflow</title>

  <para>
   Records pass through three different states during processing in the
   system.
  </para>

  <para>

   <itemizedlist>
    <listitem>
     
     <para>
      When records are accessed by the system, they are represented
      in their local, or native format. This might be SGML or HTML files,
      News or Mail archives, MARC records. If the system doesn't already
      know how to read the type of data you need to store, you can set up an
      input filter by preparing conversion rules based on regular
      expressions and possibly augmented by a flexible scripting language
      (Tcl).
      The input filter produces as output an internal representation,
      a tree structure.

     </para>
    </listitem>
    <listitem>

     <para>
      When records are processed by the system, they are represented
      in a tree-structure, constructed by tagged data elements hanging off a
      root node. The tagged elements may contain data or yet more tagged
      elements in a recursive structure. The system performs various
      actions on this tree structure (indexing, element selection, schema
      mapping, etc.),

     </para>
    </listitem>
    <listitem>

     <para>
      Before transmitting records to the client, they are first
      converted from the internal structure to a form suitable for exchange
      over the network - according to the Z39.50 standard.
     </para>
    </listitem>

   </itemizedlist>

  </para>
  </section>

 </chapter> 

 <!-- Keep this comment at the end of the file
 Local variables:
 mode: sgml
 sgml-omittag:t
 sgml-shorttag:t
 sgml-minimize-attributes:nil
 sgml-always-quote-attributes:t
 sgml-indent-step:1
 sgml-indent-data:t
 sgml-parent-document: "zebra.xml"
 sgml-local-catalogs: nil
 sgml-namecase-general:t
 End:
 -->