Moved section about special retrieval zebra:: to Architecture chapter.

[idzebra-moved-to-github.git] / doc / architecture.xml

diff --git a/doc/architecture.xml b/doc/architecture.xml
index 7db023e..dfc8335 100644 (file)
--- a/doc/architecture.xml
+++ b/doc/architecture.xml
@@ -1,11 +1,10 @@
  <chapter id="architecture">
  <chapter id="architecture">

-  <!-- $Id: architecture.xml,v 1.10 2006-06-13 13:45:08 marc Exp $ -->
+  <!-- $Id: architecture.xml,v 1.14 2006-11-24 13:05:11 adam Exp $ -->

   <title>Overview of Zebra Architecture</title>
   <title>Overview of Zebra Architecture</title>

-  

 
 

-  <sect1 id="architecture-representation">
+  <section id="architecture-representation">

    <title>Local Representation</title>
    <title>Local Representation</title>

-
+   

    <para>
     As mentioned earlier, Zebra places few restrictions on the type of
     data that you can index and manage. Generally, whatever the form of
    <para>
     As mentioned earlier, Zebra places few restrictions on the type of
     data that you can index and manage. Generally, whatever the form of

@@ -30,9 +29,9 @@
     "grs" keyword, separated by "." characters.
     -->
    </para>
     "grs" keyword, separated by "." characters.
     -->
    </para>

-  </sect1>
+  </section>

 
 

-  <sect1 id="architecture-maincomponents">
+  <section id="architecture-maincomponents">

    <title>Main Components</title>
    <para>
     The Zebra system is designed to support a wide range of data management
    <title>Main Components</title>
    <para>
     The Zebra system is designed to support a wide range of data management

@@ -52,13 +51,13 @@
     same main components, which are presented here.
    </para>    
    <para>    
     same main components, which are presented here.
    </para>    
    <para>    

-    The virtual Debian package <literal>idzebra1.4</literal>
+    The virtual Debian package <literal>idzebra-2.0</literal>

     installs all the necessary packages to start
     working with Zebra - including utility programs, development libraries,
     documentation and modules. 
   </para>    
    
     installs all the necessary packages to start
     working with Zebra - including utility programs, development libraries,
     documentation and modules. 
   </para>    
    

-   <sect2 id="componentcore">
+   <section id="componentcore">

     <title>Core Zebra Libraries Containing Common Functionality</title>
     <para>
      The core Zebra module is the meat of the <command>zebraidx</command>
     <title>Core Zebra Libraries Containing Common Functionality</title>
     <para>
      The core Zebra module is the meat of the <command>zebraidx</command>

@@ -122,17 +121,17 @@
      </variablelist>
      </para>
     <para> 
      </variablelist>
      </para>
     <para> 

-     The Debian package <literal>libidzebra1.4</literal> 
+     The Debian package <literal>libidzebra-2.0</literal> 

      contains all run-time libraries for Zebra, the 
      documentation in PDF and HTML is found in 
      contains all run-time libraries for Zebra, the 
      documentation in PDF and HTML is found in 

-     <literal>idzebra1.4-doc</literal>, and
-     <literal>idzebra1.4-common</literal>
+     <literal>idzebra-2.0-doc</literal>, and
+     <literal>idzebra-2.0-common</literal>

      includes common essential Zebra configuration files.
     </para>
      includes common essential Zebra configuration files.
     </para>

-   </sect2>
+   </section>

    
 
    
 

-   <sect2 id="componentindexer">
+   <section id="componentindexer">

     <title>Zebra Indexer</title>
     <para>
      The  <command>zebraidx</command>
     <title>Zebra Indexer</title>
     <para>
      The  <command>zebraidx</command>

@@ -142,12 +141,12 @@
      indexes according to the rules defined in the filter modules.
     </para>    
     <para>    
      indexes according to the rules defined in the filter modules.
     </para>    
     <para>    

-     The Debian  package <literal>idzebra1.4-utils</literal> contains
+     The Debian  package <literal>idzebra-2.0-utils</literal> contains

      the  <command>zebraidx</command> utility.
     </para>
      the  <command>zebraidx</command> utility.
     </para>

-   </sect2>
+   </section>

 
 

-   <sect2 id="componentsearcher">
+   <section id="componentsearcher">

     <title>Zebra Searcher/Retriever</title>
     <para>
      This is the executable which runs the Z39.50/SRU/SRW server and
     <title>Zebra Searcher/Retriever</title>
     <para>
      This is the executable which runs the Z39.50/SRU/SRW server and

@@ -155,12 +154,12 @@
      great Information Retrieval server application. 
     </para>    
     <para>    
      great Information Retrieval server application. 
     </para>    
     <para>    

-     The Debian  package <literal>idzebra1.4-utils</literal> contains
+     The Debian  package <literal>idzebra-2.0-utils</literal> contains

      the  <command>zebrasrv</command> utility.
     </para>
      the  <command>zebrasrv</command> utility.
     </para>

-   </sect2>
+   </section>

 
 

-   <sect2 id="componentyazserver">
+   <section id="componentyazserver">

     <title>YAZ Server Frontend</title>
     <para>
      The YAZ server frontend is 
     <title>YAZ Server Frontend</title>
     <para>
      The YAZ server frontend is 

@@ -171,28 +170,28 @@
     <para>
      In addition to Z39.50 requests, the YAZ server frontend acts
      as HTTP server, honoring
     <para>
      In addition to Z39.50 requests, the YAZ server frontend acts
      as HTTP server, honoring

-      <ulink url="http://www.loc.gov/standards/sru/srw/">SRW</ulink> 
-     SOAP requests, and  
-     <ulink url="&url.sru;">SRU</ulink> 
-     REST requests. Moreover, it can
+      <ulink url="&url.srw;">SRU SOAP</ulink> 
+     requests, and  
+     <ulink url="&url.sru;">SRU REST</ulink> 
+     requests. Moreover, it can

      translate incoming 
      <ulink url="&url.cql;">CQL</ulink>
      queries to
      translate incoming 
      <ulink url="&url.cql;">CQL</ulink>
      queries to

-     <ulink url="http://indexdata.com/yaz/doc/tools.tkl#PQF">PQF</ulink>
+     <ulink url="&url.yaz.pqf;">PQF</ulink>

       queries, if
      correctly configured. 
     </para>
     <para>
       queries, if
      correctly configured. 
     </para>
     <para>

-     <ulink url="http://www.indexdata.com/yaz">YAZ</ulink>
+     <ulink url="&url.yaz;">YAZ</ulink>

      is an Open Source  
      toolkit that allows you to develop software using the
      ANSI Z39.50/ISO23950 standard for information retrieval.
      It is packaged in the Debian packages     
      <literal>yaz</literal> and <literal>libyaz</literal>.
     </para>
      is an Open Source  
      toolkit that allows you to develop software using the
      ANSI Z39.50/ISO23950 standard for information retrieval.
      It is packaged in the Debian packages     
      <literal>yaz</literal> and <literal>libyaz</literal>.
     </para>

-   </sect2>
+   </section>

    
    

-   <sect2 id="componentmodules">
+   <section id="componentmodules">

     <title>Record Models and Filter Modules</title>
     <para>
      The hard work of knowing <emphasis>what</emphasis> to index, 
     <title>Record Models and Filter Modules</title>
     <para>
      The hard work of knowing <emphasis>what</emphasis> to index, 

@@ -204,25 +203,23 @@
      </para>
      <para>
      The virtual Debian package
      </para>
      <para>
      The virtual Debian package

-     <literal>libidzebra1.4-modules</literal> installs all base filter
+     <literal>libidzebra-2.0-modules</literal> installs all base filter

      modules. 
     </para>
 
      modules. 
     </para>
 

-   <sect3 id="componentmodulestext">
+   
+   <section id="componentmodulestext">

     <title>TEXT Record Model and Filter Module</title>
     <para>
       Plain ASCII text filter. TODO: add information here.
     <title>TEXT Record Model and Filter Module</title>
     <para>
       Plain ASCII text filter. TODO: add information here.

-     <!--
-     <literal>text module missing as deb file<literal>
-     -->

     </para>
     </para>

-   </sect3>
+   </section>

 
 

-   <sect3 id="componentmodulesgrs">
+   <section id="componentmodulesgrs">

     <title>GRS Record Model and Filter Modules</title>
     <para>
     The GRS filter modules described in 
     <title>GRS Record Model and Filter Modules</title>
     <para>
     The GRS filter modules described in 

-    <xref linkend="record-model-grs"/>
+    <xref linkend="grs"/>

     are all based on the Z39.50 specifications, and it is absolutely
     mandatory to have the reference pages on BIB-1 attribute sets on
     you hand when configuring GRS filters. The GRS filters come in
     are all based on the Z39.50 specifications, and it is absolutely
     mandatory to have the reference pages on BIB-1 attribute sets on
     you hand when configuring GRS filters. The GRS filters come in

@@ -231,20 +228,12 @@
     to the <filename>*.abs</filename> configuration file suffix.
     </para>
     <para>
     to the <filename>*.abs</filename> configuration file suffix.
     </para>
     <para>

-     The <emphasis>grs.danbib</emphasis> filter is developed for 
-      DBC DanBib records.
-      DanBib is the Danish Union Catalogue hosted by DBC
-      (Danish Bibliographic Center). This filter is found in the
-      Debian package
-     <literal>libidzebra1.4-mod-grs-danbib</literal>.
-    </para>
-    <para>

       The <emphasis>grs.marc</emphasis> and 
       <emphasis>grs.marcxml</emphasis> filters are suited to parse and
       index binary and XML versions of traditional library MARC records 
       based on the ISO2709 standard. The Debian package for both
       filters is 
       The <emphasis>grs.marc</emphasis> and 
       <emphasis>grs.marcxml</emphasis> filters are suited to parse and
       index binary and XML versions of traditional library MARC records 
       based on the ISO2709 standard. The Debian package for both
       filters is 

-     <literal>libidzebra1.4-mod-grs-marc</literal>.
+     <literal>libidzebra-2.0-mod-grs-marc</literal>.

     </para>
     <para>
       GRS TCL scriptable filters for extensive user configuration come
     </para>
     <para>
       GRS TCL scriptable filters for extensive user configuration come

@@ -253,26 +242,26 @@
      a general scriptable TCL filter called 
      <emphasis>grs.tcl</emphasis>        
      are both included in the 
      a general scriptable TCL filter called 
      <emphasis>grs.tcl</emphasis>        
      are both included in the 

-     <literal>libidzebra1.4-mod-grs-regx</literal> Debian package.
+     <literal>libidzebra-2.0-mod-grs-regx</literal> Debian package.

     </para>
     <para>
       A general purpose SGML filter is called
      <emphasis>grs.sgml</emphasis>. This filter is not yet packaged,
      but planned to be in the  
     </para>
     <para>
       A general purpose SGML filter is called
      <emphasis>grs.sgml</emphasis>. This filter is not yet packaged,
      but planned to be in the  

-     <literal>libidzebra1.4-mod-grs-sgml</literal> Debian package.
+     <literal>libidzebra-2.0-mod-grs-sgml</literal> Debian package.

     </para>
     <para>
       The Debian  package 
     </para>
     <para>
       The Debian  package 

-      <literal>libidzebra1.4-mod-grs-xml</literal> includes the 
+      <literal>libidzebra-2.0-mod-grs-xml</literal> includes the 

       <emphasis>grs.xml</emphasis> filter which uses <ulink
       <emphasis>grs.xml</emphasis> filter which uses <ulink

-      url="http://expat.sourceforge.net/">Expat</ulink> to 
+      url="&url.expat;">Expat</ulink> to 

       parse records in XML and turn them into IDZebra's internal GRS node
       trees. Have also a look at the Alvis XML/XSLT filter described in
       the next session.
     </para>
       parse records in XML and turn them into IDZebra's internal GRS node
       trees. Have also a look at the Alvis XML/XSLT filter described in
       the next session.
     </para>

-   </sect3>
+   </section>

 
 

-   <sect3 id="componentmodulesalvis">
+   <section id="componentmodulesalvis">

     <title>ALVIS Record Model and Filter Module</title>
      <para>
       The Alvis filter for XML files is an XSLT based input
     <title>ALVIS Record Model and Filter Module</title>
      <para>
       The Alvis filter for XML files is an XSLT based input

@@ -309,27 +298,26 @@
       <xref linkend="record-model-alvisxslt"/>.
       </para>
      <para>
       <xref linkend="record-model-alvisxslt"/>.
       </para>
      <para>

-      The Debian package <literal>libidzebra1.4-mod-alvis</literal>
+      The Debian package <literal>libidzebra-2.0-mod-alvis</literal>

       contains the Alvis filter module.
      </para>
       contains the Alvis filter module.
      </para>

-    </sect3>
+    </section>

 
 

-   <sect3 id="componentmodulessafari">
+    <!--
+   <section id="componentmodulessafari">

     <title>SAFARI Record Model and Filter Module</title>
     <para>
      SAFARI filter module TODO: add information here.
     <title>SAFARI Record Model and Filter Module</title>
     <para>
      SAFARI filter module TODO: add information here.

-     <!--
-     <literal>safari module missing as deb file<literal>
-     -->

     </para>
     </para>

-   </sect3>
+   </section>
+    -->

 
 

-   </sect2>
+   </section>

 
 

-  </sect1>
+  </section>

 
 
 
 

-  <sect1 id="architecture-workflow">
+  <section id="architecture-workflow">

    <title>Indexing and Retrieval Workflow</title>
 
   <para>
    <title>Indexing and Retrieval Workflow</title>
 
   <para>

@@ -379,9 +367,160 @@
    </itemizedlist>
 
   </para>
    </itemizedlist>
 
   </para>

-  </sect1>
-
+  </section>

 
 

+  <section id="default-idx-zebra">
+   <title>Retrieval of Zebra internal record data</title>
+   <para>
+    Starting with <literal>Zebra</literal> version 2.0.5 or newer, it is
+    possible to use a special element set which has the prefix
+    <literal>zebra::</literal>.
+   </para>
+   <para>
+    Using this element will, regardless of record type, return
+    Zebra's internal index structure/data for a record.
+    In particular, the regular record filters are not invoked when
+    these are in use.
+    This can in some cases make the retrival faster than regular
+    retrieval operations (for MARC, XML etc).
+   </para>
+   <table id="special-retrieval">
+    <title>Special Retrieval Elements</title>
+    <tgroup cols="2">
+     <thead>
+      <row>
+       <entry>Element Set</entry>
+       <entry>Description</entry>
+       <entry>Syntax</entry>
+      </row>
+     </thead>
+     <tbody>
+      <row>
+       <entry><literal>zebra::meta::sysno</literal></entry>
+       <entry>Get Zebra record system ID</entry>
+       <entry>XML and SUTRS</entry>
+      </row>
+      <row>
+       <entry><literal>zebra::data</literal></entry>
+       <entry>Get raw record</entry>
+       <entry>all</entry>
+      </row>
+      <row>
+       <entry><literal>zebra::meta</literal></entry>
+       <entry>Get Zebra record internal metadata</entry>
+       <entry>XML and SUTRS</entry>
+      </row>
+      <row>
+       <entry><literal>zebra::index</literal></entry>
+       <entry>Get all indexed keys for record</entry>
+       <entry>XML and SUTRS</entry>
+      </row>
+      <row>
+       <entry>
+       <literal>zebra::index::</literal><replaceable>f</replaceable>
+       </entry>
+       <entry>
+       Get indexed keys for field <replaceable>f</replaceable> of record
+       </entry>
+       <entry>XML and SUTRS</entry>
+      </row>
+      <row>
+       <entry>
+       <literal>zebra::index::</literal><replaceable>field</replaceable>:<replaceable>t</replaceable>
+       </entry>
+       <entry>
+       Get indexed keys for field <replaceable>f</replaceable>
+         and type <replaceable>t</replaceable> of record
+       </entry>
+       <entry>XML and SUTRS</entry>
+      </row>
+     </tbody>
+    </tgroup>
+   </table>
+   <para>
+    For example, to fetch the raw binary record data stored in the
+    zebra internal storage, or on the filesystem, the following
+    commands can be issued:
+    <screen>
+      Z> f @attr 1=title my
+      Z> format xml
+      Z> elements zebra::data
+      Z> s 1+1
+      Z> format sutrs
+      Z> s 1+1
+      Z> format usmarc
+      Z> s 1+1
+    </screen>
+    </para>
+   <para>
+    The special 
+    <literal>zebra::data</literal> element set name is 
+    defined for any record syntax, but will always fetch  
+    the raw record data in exactly the original form. No record syntax
+    specific transformations will be applied to the raw record data. 
+   </para>
+   <para>
+    Also, Zebra internal metadata about the record can be accessed: 
+    <screen>
+      Z> f @attr 1=title my
+      Z> format xml
+      Z> elements zebra::meta::sysno
+      Z> s 1+1
+    </screen> 
+    displays in <literal>XML</literal> record syntax only internal
+    record system number, whereas 
+    <screen>
+      Z> f @attr 1=title my
+      Z> format xml
+      Z> elements zebra::meta
+      Z> s 1+1
+    </screen> 
+    displays all available metadata on the record. These include sytem
+    number, database name,  indexed filename,  filter used for indexing,
+    score and static ranking information and finally bytesize of record.
+   </para>
+   <para>
+    Sometimes, it is very hard to figure out what exactly has been
+    indexed how and in which indexes. Using the indexing stylesheet of
+    the Alvis filter, one can at least see which portion of the record
+    went into which index, but a similar aid does not exist for all
+    other indexing filters.  
+   </para>
+   <para>
+    The special
+    <literal>zebra::index</literal> element set names are provided to
+    access information on per record indexed fields. For example, the
+    queries 
+    <screen>
+      Z> f @attr 1=title my
+      Z> format sutrs
+      Z> elements zebra::index
+      Z> s 1+1
+    </screen>
+    will display all indexed tokens from all indexed fields of the
+    first record, and it will display in <literal>SUTRS</literal>
+    record syntax, whereas 
+    <screen>
+      Z> f @attr 1=title my
+      Z> format xml
+      Z> elements zebra::index::title
+      Z> s 1+1
+      Z> elements zebra::index::title:p
+      Z> s 1+1
+    </screen> 
+    displays in <literal>XML</literal> record syntax only the content
+      of the zebra string index <literal>title</literal>, or
+      even only the type <literal>p</literal> phrase indexed part of it.
+   </para>
+   <note>
+    <para>
+     Trying to access numeric <literal>Bib-1</literal> use
+     attributes or trying to access non-existent zebra intern string
+     access points will result in a Diagnostic 25: Specified element set
+     'name not valid for specified database.
+    </para>
+   </note>
+  </section>

 
  </chapter> 
 
 
  </chapter>