X-Git-Url: http://git.indexdata.com/?p=idzebra-moved-to-github.git;a=blobdiff_plain;f=doc%2Farchitecture.xml;h=5b3a27ebd2b5cc89cc08f7225dca9864c90487be;hp=dfc83351b361aae2fcb3eb6ef30290b7b6e58931;hb=31b0ba1ecb737f9db4cf3340e93964c6679897fd;hpb=e70a548d193a5187b8074c439f2d7fa687a8e8c4

diff --git a/doc/architecture.xml b/doc/architecture.xml
index dfc8335..5b3a27e 100644
--- a/doc/architecture.xml
+++ b/doc/architecture.xml
@@ -1,29 +1,28 @@
  <chapter id="architecture">
-  <!-- $Id: architecture.xml,v 1.14 2006-11-24 13:05:11 adam Exp $ -->
-  <title>Overview of Zebra Architecture</title>
+  <title>Overview of &zebra; Architecture</title>
 
   <section id="architecture-representation">
    <title>Local Representation</title>
    
    <para>
-    As mentioned earlier, Zebra places few restrictions on the type of
+    As mentioned earlier, &zebra; places few restrictions on the type of
     data that you can index and manage. Generally, whatever the form of
     the data, it is parsed by an input filter specific to that format, and
-    turned into an internal structure that Zebra knows how to handle. This
+    turned into an internal structure that &zebra; knows how to handle. This
     process takes place whenever the record is accessed - for indexing and
     retrieval.
    </para>
 
    <para>
     The RecordType parameter in the <literal>zebra.cfg</literal> file, or
-    the <literal>-t</literal> option to the indexer tells Zebra how to
+    the <literal>-t</literal> option to the indexer tells &zebra; how to
     process input records.
     Two basic types of processing are available - raw text and structured
     data. Raw text is just that, and it is selected by providing the
-    argument <emphasis>text</emphasis> to Zebra. Structured records are
+    argument <emphasis>text</emphasis> to &zebra;. Structured records are
     all handled internally using the basic mechanisms described in the
     subsequent sections.
-    Zebra can read structured records in many different formats.
+    &zebra; can read structured records in many different formats.
     <!--
     How this is done is governed by additional parameters after the
     "grs" keyword, separated by "." characters.
@@ -34,7 +33,7 @@
   <section id="architecture-maincomponents">
    <title>Main Components</title>
    <para>
-    The Zebra system is designed to support a wide range of data management
+    The &zebra; system is designed to support a wide range of data management
     applications. The system can be configured to handle virtually any
     kind of structured data. Each record in the system is associated with
     a <emphasis>record schema</emphasis> which lends context to the data
@@ -44,7 +43,7 @@
     one database, the system poses no such restrictions.
    </para>
    <para>
-    The Zebra indexer and information retrieval server consists of the
+    The &zebra; indexer and information retrieval server consists of the
     following main applications: the <command>zebraidx</command>
     indexing maintenance utility, and the <command>zebrasrv</command>
     information query and retrieval server. Both are using some of the
@@ -53,14 +52,14 @@
    <para>    
     The virtual Debian package <literal>idzebra-2.0</literal>
     installs all the necessary packages to start
-    working with Zebra - including utility programs, development libraries,
+    working with &zebra; - including utility programs, development libraries,
     documentation and modules. 
   </para>    
    
    <section id="componentcore">
-    <title>Core Zebra Libraries Containing Common Functionality</title>
+    <title>Core &zebra; Libraries Containing Common Functionality</title>
     <para>
-     The core Zebra module is the meat of the <command>zebraidx</command>
+     The core &zebra; module is the meat of the <command>zebraidx</command>
     indexing maintenance utility, and the <command>zebrasrv</command>
     information query and retrieval server binaries. Shortly, the core
     libraries are responsible for  
@@ -77,7 +76,7 @@
       <varlistentry>
        <term>Index Maintenance</term>
        <listitem>
-        <para> Zebra maintains Term Dictionaries and ISAM index
+        <para> &zebra; maintains Term Dictionaries and ISAM index
         entries in inverted index structures kept on disk. These are
         optimized for fast inset, update and delete, as well as good
         search performance.
@@ -87,9 +86,9 @@
       <varlistentry>
        <term>Search Evaluation</term>
        <listitem>
-        <para>by execution of search requests expressed in PQF/RPN
+        <para>by execution of search requests expressed in &acro.pqf;/&acro.rpn;
          data structures, which are handed over from
-         the YAZ server frontend API. Search evaluation includes
+         the &yaz; server frontend &acro.api;. Search evaluation includes
          construction of hit lists according to boolean combinations
          of simpler searches. Fast performance is achieved by careful
          use of index structures, and by evaluation specific index hit
@@ -112,7 +111,7 @@
        <term>Record Presentation</term>
        <listitem>
         <para>returns - possibly ranked - result sets, hit
-         numbers, and the like internal data to the YAZ server backend API
+         numbers, and the like internal data to the &yaz; server backend &acro.api;
          for shipping to the client. Each individual filter module
          implements it's own specific presentation formats.
         </para>
@@ -122,17 +121,17 @@
      </para>
     <para> 
      The Debian package <literal>libidzebra-2.0</literal> 
-     contains all run-time libraries for Zebra, the 
+     contains all run-time libraries for &zebra;, the 
      documentation in PDF and HTML is found in 
      <literal>idzebra-2.0-doc</literal>, and
      <literal>idzebra-2.0-common</literal>
-     includes common essential Zebra configuration files.
+     includes common essential &zebra; configuration files.
     </para>
    </section>
    
 
    <section id="componentindexer">
-    <title>Zebra Indexer</title>
+    <title>&zebra; Indexer</title>
     <para>
      The  <command>zebraidx</command>
      indexing maintenance utility 
@@ -147,9 +146,9 @@
    </section>
 
    <section id="componentsearcher">
-    <title>Zebra Searcher/Retriever</title>
+    <title>&zebra; Searcher/Retriever</title>
     <para>
-     This is the executable which runs the Z39.50/SRU/SRW server and
+     This is the executable which runs the &acro.z3950;/&acro.sru;/&acro.srw; server and
      glues together the core libraries and the filter modules to one
      great Information Retrieval server application. 
     </para>    
@@ -160,32 +159,32 @@
    </section>
 
    <section id="componentyazserver">
-    <title>YAZ Server Frontend</title>
+    <title>&yaz; Server Frontend</title>
     <para>
-     The YAZ server frontend is 
-     a full fledged stateful Z39.50 server taking client
+     The &yaz; server frontend is 
+     a full fledged stateful &acro.z3950; server taking client
      connections, and forwarding search and scan requests to the 
-     Zebra core indexer.
+     &zebra; core indexer.
     </para>
     <para>
-     In addition to Z39.50 requests, the YAZ server frontend acts
+     In addition to &acro.z3950; requests, the &yaz; server frontend acts
      as HTTP server, honoring
-      <ulink url="&url.srw;">SRU SOAP</ulink> 
+      <ulink url="&url.sru;">&acro.sru; &acro.soap;</ulink> 
      requests, and  
-     <ulink url="&url.sru;">SRU REST</ulink> 
+     &acro.sru; &acro.rest;
      requests. Moreover, it can
      translate incoming 
-     <ulink url="&url.cql;">CQL</ulink>
+     <ulink url="&url.cql;">&acro.cql;</ulink>
      queries to
-     <ulink url="&url.yaz.pqf;">PQF</ulink>
+     <ulink url="&url.yaz.pqf;">&acro.pqf;</ulink>
       queries, if
      correctly configured. 
     </para>
     <para>
-     <ulink url="&url.yaz;">YAZ</ulink>
+     <ulink url="&url.yaz;">&yaz;</ulink>
      is an Open Source  
      toolkit that allows you to develop software using the
-     ANSI Z39.50/ISO23950 standard for information retrieval.
+     &acro.ansi; &acro.z3950;/ISO23950 standard for information retrieval.
      It is packaged in the Debian packages     
      <literal>yaz</literal> and <literal>libyaz</literal>.
     </para>
@@ -207,81 +206,83 @@
      modules. 
     </para>
 
-   
-   <section id="componentmodulestext">
-    <title>TEXT Record Model and Filter Module</title>
-    <para>
-      Plain ASCII text filter. TODO: add information here.
+   <section id="componentmodulesdom">
+    <title>&acro.dom; &acro.xml; Record Model and Filter Module</title>
+     <para>
+      The &acro.dom; &acro.xml; filter uses a standard &acro.dom; &acro.xml; structure as
+      internal data model, and can thus parse, index, and display 
+      any &acro.xml; document.
     </para>
-   </section>
-
-   <section id="componentmodulesgrs">
-    <title>GRS Record Model and Filter Modules</title>
     <para>
-    The GRS filter modules described in 
-    <xref linkend="grs"/>
-    are all based on the Z39.50 specifications, and it is absolutely
-    mandatory to have the reference pages on BIB-1 attribute sets on
-    you hand when configuring GRS filters. The GRS filters come in
-    different flavors, and a short introduction is needed here.
-    GRS filters of various kind have also been called ABS filters due
-    to the <filename>*.abs</filename> configuration file suffix.
+      A parser for binary &acro.marc; records based on the ISO2709 library
+      standard is provided, it transforms these to the internal
+      &acro.marcxml; &acro.dom; representation.  
     </para>
     <para>
-      The <emphasis>grs.marc</emphasis> and 
-      <emphasis>grs.marcxml</emphasis> filters are suited to parse and
-      index binary and XML versions of traditional library MARC records 
-      based on the ISO2709 standard. The Debian package for both
-      filters is 
-     <literal>libidzebra-2.0-mod-grs-marc</literal>.
+      The internal &acro.dom; &acro.xml; representation can be fed into four
+      different pipelines, consisting of arbitrarily many successive
+      &acro.xslt; transformations; these are for  
+     <itemizedlist>
+       <listitem><para>input parsing and initial
+          transformations,</para></listitem>
+       <listitem><para>indexing term extraction
+          transformations</para></listitem>
+       <listitem><para>transformations before internal document
+          storage, and </para></listitem>
+       <listitem><para>retrieve transformations from storage to output
+          format</para></listitem>
+      </itemizedlist>
     </para>
     <para>
-      GRS TCL scriptable filters for extensive user configuration come
-     in two flavors: a regular expression filter 
-     <emphasis>grs.regx</emphasis> using TCL regular expressions, and
-     a general scriptable TCL filter called 
-     <emphasis>grs.tcl</emphasis>        
-     are both included in the 
-     <literal>libidzebra-2.0-mod-grs-regx</literal> Debian package.
+      The &acro.dom; &acro.xml; filter pipelines use &acro.xslt; (and if  supported on
+      your platform, even &acro.exslt;), it brings thus full &acro.xpath;
+      support to the indexing, storage and display rules of not only
+      &acro.xml; documents, but also binary &acro.marc; records.
     </para>
     <para>
-      A general purpose SGML filter is called
-     <emphasis>grs.sgml</emphasis>. This filter is not yet packaged,
-     but planned to be in the  
-     <literal>libidzebra-2.0-mod-grs-sgml</literal> Debian package.
+      Finally, the &acro.dom; &acro.xml; filter allows for static ranking at index
+      time, and to to sort hit lists according to predefined
+      static ranks.
     </para>
     <para>
-      The Debian  package 
-      <literal>libidzebra-2.0-mod-grs-xml</literal> includes the 
-      <emphasis>grs.xml</emphasis> filter which uses <ulink
-      url="&url.expat;">Expat</ulink> to 
-      parse records in XML and turn them into IDZebra's internal GRS node
-      trees. Have also a look at the Alvis XML/XSLT filter described in
-      the next session.
-    </para>
-   </section>
+      Details on the experimental &acro.dom; &acro.xml; filter are found in 
+      <xref linkend="record-model-domxml"/>.
+      </para>
+     <para>
+      The Debian package <literal>libidzebra-2.0-mod-dom</literal>
+      contains the &acro.dom; filter module.
+     </para>
+    </section>
 
    <section id="componentmodulesalvis">
-    <title>ALVIS Record Model and Filter Module</title>
+    <title>ALVIS &acro.xml; Record Model and Filter Module</title>
+     <note>
+      <para>
+        The functionality of this record model has been improved and
+        replaced by the &acro.dom; &acro.xml; record model. See 
+        <xref linkend="componentmodulesdom"/>.
+      </para>
+     </note>
+
      <para>
-      The Alvis filter for XML files is an XSLT based input
+      The Alvis filter for &acro.xml; files is an &acro.xslt; based input
       filter. 
-      It indexes element and attribute content of any thinkable XML format
-      using full XPATH support, a feature which the standard Zebra
-      GRS SGML and XML filters lacked. The indexed documents are
-      parsed into a standard XML DOM tree, which restricts record size
+      It indexes element and attribute content of any thinkable &acro.xml; format
+      using full &acro.xpath; support, a feature which the standard &zebra;
+      &acro.grs1; &acro.sgml; and &acro.xml; filters lacked. The indexed documents are
+      parsed into a standard &acro.xml; &acro.dom; tree, which restricts record size
       according to availability of memory.
     </para>
     <para>
       The Alvis filter 
-      uses XSLT display stylesheets, which let
-      the Zebra DB administrator associate multiple, different views on
-      the same XML document type. These views are chosen on-the-fly in
+      uses &acro.xslt; display stylesheets, which let
+      the &zebra; DB administrator associate multiple, different views on
+      the same &acro.xml; document type. These views are chosen on-the-fly in
       search time.
      </para>
     <para>
       In addition, the Alvis filter configuration is not bound to the
-      arcane  BIB-1 Z39.50 library catalogue indexing traditions and
+      arcane  &acro.bib1; &acro.z3950; library catalogue indexing traditions and
       folklore, and is therefore easier to understand.
     </para>
     <para>
@@ -290,11 +291,11 @@
       static ranks. This imposes no overhead at all, both
       search and indexing perform still 
       <emphasis>O(1)</emphasis> irrespectively of document
-      collection size. This feature resembles Googles pre-ranking using
-      their Pagerank algorithm.
+      collection size. This feature resembles Google's pre-ranking using
+      their PageRank algorithm.
     </para>
     <para>
-      Details on the experimental Alvis XSLT filter are found in 
+      Details on the experimental Alvis &acro.xslt; filter are found in 
       <xref linkend="record-model-alvisxslt"/>.
       </para>
      <para>
@@ -303,6 +304,66 @@
      </para>
     </section>
 
+   <section id="componentmodulesgrs">
+    <title>&acro.grs1; Record Model and Filter Modules</title>
+     <note>
+      <para>
+        The functionality of this record model has been improved and
+        replaced by the &acro.dom; &acro.xml; record model. See 
+        <xref linkend="componentmodulesdom"/>.
+      </para>
+     </note>
+    <para>
+    The &acro.grs1; filter modules described in 
+    <xref linkend="grs"/>
+    are all based on the &acro.z3950; specifications, and it is absolutely
+    mandatory to have the reference pages on &acro.bib1; attribute sets on
+    you hand when configuring &acro.grs1; filters. The GRS filters come in
+    different flavors, and a short introduction is needed here.
+    &acro.grs1; filters of various kind have also been called ABS filters due
+    to the <filename>*.abs</filename> configuration file suffix.
+    </para>
+    <para>
+      The <emphasis>grs.marc</emphasis> and 
+      <emphasis>grs.marcxml</emphasis> filters are suited to parse and
+      index binary and &acro.xml; versions of traditional library &acro.marc; records 
+      based on the ISO2709 standard. The Debian package for both
+      filters is 
+     <literal>libidzebra-2.0-mod-grs-marc</literal>.
+    </para>
+    <para>
+      &acro.grs1; TCL scriptable filters for extensive user configuration come
+     in two flavors: a regular expression filter 
+     <emphasis>grs.regx</emphasis> using TCL regular expressions, and
+     a general scriptable TCL filter called 
+     <emphasis>grs.tcl</emphasis>        
+     are both included in the 
+     <literal>libidzebra-2.0-mod-grs-regx</literal> Debian package.
+    </para>
+    <para>
+      A general purpose &acro.sgml; filter is called
+     <emphasis>grs.sgml</emphasis>. This filter is not yet packaged,
+     but planned to be in the  
+     <literal>libidzebra-2.0-mod-grs-sgml</literal> Debian package.
+    </para>
+    <para>
+      The Debian  package 
+      <literal>libidzebra-2.0-mod-grs-xml</literal> includes the 
+      <emphasis>grs.xml</emphasis> filter which uses <ulink
+      url="&url.expat;">Expat</ulink> to 
+      parse records in &acro.xml; and turn them into ID&zebra;'s internal &acro.grs1; node
+      trees. Have also a look at the Alvis &acro.xml;/&acro.xslt; filter described in
+      the next session.
+    </para>
+   </section>
+   
+   <section id="componentmodulestext">
+    <title>TEXT Record Model and Filter Module</title>
+    <para>
+      Plain ASCII text filter. TODO: add information here.
+    </para>
+   </section>
+
     <!--
    <section id="componentmodulessafari">
     <title>SAFARI Record Model and Filter Module</title>
@@ -332,8 +393,8 @@
      
      <para>
       When records are accessed by the system, they are represented
-      in their local, or native format. This might be SGML or HTML files,
-      News or Mail archives, MARC records. If the system doesn't already
+      in their local, or native format. This might be &acro.sgml; or HTML files,
+      News or Mail archives, &acro.marc; records. If the system doesn't already
       know how to read the type of data you need to store, you can set up an
       input filter by preparing conversion rules based on regular
       expressions and possibly augmented by a flexible scripting language
@@ -360,7 +421,7 @@
      <para>
       Before transmitting records to the client, they are first
       converted from the internal structure to a form suitable for exchange
-      over the network - according to the Z39.50 standard.
+      over the network - according to the &acro.z3950; standard.
      </para>
     </listitem>
 
@@ -369,22 +430,22 @@
   </para>
   </section>
 
-  <section id="default-idx-zebra">
-   <title>Retrieval of Zebra internal record data</title>
+  <section id="special-retrieval">
+   <title>Retrieval of &zebra; internal record data</title>
    <para>
-    Starting with <literal>Zebra</literal> version 2.0.5 or newer, it is
+    Starting with <literal>&zebra;</literal> version 2.0.5 or newer, it is
     possible to use a special element set which has the prefix
     <literal>zebra::</literal>.
    </para>
    <para>
     Using this element will, regardless of record type, return
-    Zebra's internal index structure/data for a record.
+    &zebra;'s internal index structure/data for a record.
     In particular, the regular record filters are not invoked when
     these are in use.
-    This can in some cases make the retrival faster than regular
-    retrieval operations (for MARC, XML etc).
+    This can in some cases make the retrieval faster than regular
+    retrieval operations (for &acro.marc;, &acro.xml; etc).
    </para>
-   <table id="special-retrieval">
+   <table id="special-retrieval-types">
     <title>Special Retrieval Elements</title>
     <tgroup cols="2">
      <thead>
@@ -397,8 +458,8 @@
      <tbody>
       <row>
        <entry><literal>zebra::meta::sysno</literal></entry>
-       <entry>Get Zebra record system ID</entry>
-       <entry>XML and SUTRS</entry>
+       <entry>Get &zebra; record system ID</entry>
+       <entry>&acro.xml; and &acro.sutrs;</entry>
       </row>
       <row>
        <entry><literal>zebra::data</literal></entry>
@@ -407,32 +468,60 @@
       </row>
       <row>
        <entry><literal>zebra::meta</literal></entry>
-       <entry>Get Zebra record internal metadata</entry>
-       <entry>XML and SUTRS</entry>
+       <entry>Get &zebra; record internal metadata</entry>
+       <entry>&acro.xml; and &acro.sutrs;</entry>
       </row>
       <row>
        <entry><literal>zebra::index</literal></entry>
        <entry>Get all indexed keys for record</entry>
-       <entry>XML and SUTRS</entry>
+       <entry>&acro.xml; and &acro.sutrs;</entry>
       </row>
       <row>
        <entry>
 	<literal>zebra::index::</literal><replaceable>f</replaceable>
        </entry>
        <entry>
-	Get indexed keys for field <replaceable>f</replaceable>	of record
+	Get indexed keys for field <replaceable>f</replaceable>	for record
        </entry>
-       <entry>XML and SUTRS</entry>
+       <entry>&acro.xml; and &acro.sutrs;</entry>
       </row>
       <row>
        <entry>
-	<literal>zebra::index::</literal><replaceable>field</replaceable>:<replaceable>t</replaceable>
+	<literal>zebra::index::</literal><replaceable>f</replaceable>:<replaceable>t</replaceable>
        </entry>
        <entry>
 	Get indexed keys for field <replaceable>f</replaceable>
-	  and type <replaceable>t</replaceable> of record
+	  and type <replaceable>t</replaceable> for record
+       </entry>
+       <entry>&acro.xml; and &acro.sutrs;</entry>
+      </row>
+      <row>
+       <entry>
+	<literal>zebra::snippet</literal>
+       </entry>
+       <entry>
+	Get snippet for record for one or more indexes (f1,f2,..).
+	This includes a phrase from the original
+	record at the point where a match occurs (for a query). By default
+	give terms before - and after are included in the snippet. The
+	matching terms are enclosed within element
+	<literal>&lt;s&gt;</literal>. The snippet facility requires
+	Zebra 2.0.16 or later.
+       </entry>
+       <entry>&acro.xml; and &acro.sutrs;</entry>
+      </row>
+      <row>
+       <entry>
+	<literal>zebra::facet::</literal><replaceable>f1</replaceable>:<replaceable>t1</replaceable>,<replaceable>f2</replaceable>:<replaceable>t2</replaceable>,..
+       </entry>
+       <entry>
+	Get facet of a result set. The facet result is returned
+	as if it was a normal record, while in reality is a 
+	recap of most "important" terms in a result set for the fields 
+	given.
+	The facet facility first appeared in Zebra 2.0.20.
        </entry>
-       <entry>XML and SUTRS</entry>
+       <entry>&acro.xml;</entry>
       </row>
      </tbody>
     </tgroup>
@@ -460,14 +549,14 @@
     specific transformations will be applied to the raw record data. 
    </para>
    <para>
-    Also, Zebra internal metadata about the record can be accessed: 
+    Also, &zebra; internal metadata about the record can be accessed: 
     <screen>
       Z> f @attr 1=title my
       Z> format xml
       Z> elements zebra::meta::sysno
       Z> s 1+1
     </screen> 
-    displays in <literal>XML</literal> record syntax only internal
+    displays in <literal>&acro.xml;</literal> record syntax only internal
     record system number, whereas 
     <screen>
       Z> f @attr 1=title my
@@ -475,7 +564,7 @@
       Z> elements zebra::meta
       Z> s 1+1
     </screen> 
-    displays all available metadata on the record. These include sytem
+    displays all available metadata on the record. These include system
     number, database name,  indexed filename,  filter used for indexing,
     score and static ranking information and finally bytesize of record.
    </para>
@@ -498,7 +587,7 @@
       Z> s 1+1
     </screen>
     will display all indexed tokens from all indexed fields of the
-    first record, and it will display in <literal>SUTRS</literal>
+    first record, and it will display in <literal>&acro.sutrs;</literal>
     record syntax, whereas 
     <screen>
       Z> f @attr 1=title my
@@ -508,13 +597,13 @@
       Z> elements zebra::index::title:p
       Z> s 1+1
     </screen> 
-    displays in <literal>XML</literal> record syntax only the content
+    displays in <literal>&acro.xml;</literal> record syntax only the content
       of the zebra string index <literal>title</literal>, or
       even only the type <literal>p</literal> phrase indexed part of it.
    </para>
    <note>
     <para>
-     Trying to access numeric <literal>Bib-1</literal> use
+     Trying to access numeric <literal>&acro.bib1;</literal> use
      attributes or trying to access non-existent zebra intern string
      access points will result in a Diagnostic 25: Specified element set
      'name not valid for specified database.