X-Git-Url: http://git.indexdata.com/?a=blobdiff_plain;f=doc%2Fadministration.xml;h=01e8f222f622578bf04c8ad78d747bee15b9b7a7;hb=296574b8e825da3594047d8d63c9639d90866047;hp=b29790cae9377a72093b054e27c701ef35dcea21;hpb=14a2dbce03d7802ab5b1e57b09d915339bb5fc54;p=idzebra-moved-to-github.git

diff --git a/doc/administration.xml b/doc/administration.xml
index b29790c..01e8f22 100644
--- a/doc/administration.xml
+++ b/doc/administration.xml
@@ -1,5 +1,5 @@
 <chapter id="administration">
- <!-- $Id: administration.xml,v 1.22 2006-02-15 11:07:47 marc Exp $ -->
+ <!-- $Id: administration.xml,v 1.33 2006-05-02 12:55:18 mike Exp $ -->
  <title>Administrating Zebra</title>
  <!-- ### It's a bit daft that this chapter (which describes half of
           the configuration-file formats) is separated from
@@ -843,7 +843,6 @@
     
     <screen>
      register: /d1:500M
-     
      shadow: /scratch1:100M /scratch2:200M
     </screen>
     
@@ -921,8 +920,530 @@
   </sect2>
   
  </sect1>
+
+
+ <sect1 id="administration-ranking">
+  <title>Relevance Ranking and Sorting of Result Sets</title>
+
+  <sect2>
+   <title>Overview</title>
+   <para>
+    The default ordering of a result set is left up to the server,
+    which inside Zebra means sorting in ascending document ID order. 
+    This is not always the order humans want to browse the sometimes
+    quite large hit sets. Ranking and sorting comes to the rescue.
+   </para>
+
+   <para> 
+    In cases where a good presentation ordering can be computed at
+    indexing time, we can use a fixed <literal>static ranking</literal>
+    scheme, which is provided for the <literal>alvis</literal>
+    indexing filter. This defines a fixed ordering of hit lists,
+    independently of the query issued. 
+   </para>
+
+   <para>
+    There are cases, however, where relevance of hit set documents is
+    highly dependent on the query processed.
+    Simply put, <literal>dynamic relevance ranking</literal> 
+    sorts a set of retrieved 
+    records such
+    that those most likely to be relevant to your request are
+    retrieved first. 
+    Internally, Zebra retrieves all documents that satisfy your
+    query, and re-orders the hit list to arrange them based on
+    a measurement of similarity between your query and the content of
+    each record. 
+   </para>
+
+   <para>
+    Finally, there are situations where hit sets of documents should be
+    <literal>sorted</literal> during query time according to the
+    lexicographical ordering of certain sort indexes created at
+    indexing time.
+   </para>
+  </sect2>
+
+
+ <sect2 id="administration-ranking-static">
+  <title>Static Ranking</title>
+  
+   <para>
+    Zebra uses internally inverted indexes to look up term occurencies
+    in documents. Multiple queries from different indexes can be
+    combined by the binary boolean operations <literal>AND</literal>, 
+    <literal>OR</literal> and/or <literal>NOT</literal> (which
+    is in fact a binary <literal>AND NOT</literal> operation). 
+    To ensure fast query execution
+    speed, all indexes have to be sorted in the same order.
+   </para>
+   <para>
+    The indexes are normally sorted according to document 
+    <literal>ID</literal> in
+    ascending order, and any query which does not invoke a special
+    re-ranking function will therefore retrieve the result set in
+    document 
+    <literal>ID</literal>
+    order.
+   </para>
+   <para>
+    If one defines the 
+    <screen>
+    staticrank: 1 
+    </screen> 
+    directive in the main core Zebra config file, the internal document
+    keys used for ordering are augmented by a preceeding integer, which
+    contains the static rank of a given document, and the index lists
+    are ordered 
+    first by ascending static rank,
+    then by ascending document <literal>ID</literal>.
+    Zero
+    is the ``best'' rank, as it occurs at the
+    beginning of the list; higher numbers represent worse scores.
+   </para>
+   <para>
+    The experimental <literal>alvis</literal> filter provides a
+    directive to fetch static rank information out of the indexed XML
+    records, thus making <emphasis>all</emphasis> hit sets orderd
+    after <emphasis>ascending</emphasis> static
+    rank, and for those doc's which have the same static rank, ordered
+    after <emphasis>ascending</emphasis> doc <literal>ID</literal>.
+    See <xref linkend="record-model-alvisxslt"/> for the gory details.
+   </para>
+    </sect2>
+
+
+ <sect2 id="administration-ranking-dynamic">
+  <title>Dynamic Ranking</title>
+   <para>
+    In order to fiddle with the static rank order, it is necessary to
+    invoke additional re-ranking/re-ordering using dynamic
+    ranking or score functions. These functions return positive
+    integer scores, where <emphasis>highest</emphasis> score is 
+    ``best'';
+    hit sets are sorted according to
+    <emphasis>decending</emphasis> 
+    scores (in contrary
+    to the index lists which are sorted according to
+    ascending rank number and document ID).
+   </para>
+   <para>
+    Dynamic ranking is enabled by a directive like one of the
+    following in the zebra config file (use only one of these a time!):
+    <screen> 
+    rank: rank-1        # default TDF-IDF like
+    rank: rank-static   # dummy do-nothing
+    rank: zvrank        # configurable, experimental TDF-IDF like
+    </screen>
+    Notice that the <literal>rank-1</literal> and
+    <literal>zvrank</literal> do not use the static rank 
+    information in the list keys, and will produce the same ordering
+    with or without static ranking enabled.
+   </para>
+   <para>
+    The dummy <literal>rank-static</literal> reranking/scoring
+    function returns just 
+    <literal>score = max int - staticrank</literal>
+    in order to preserve the static ordering of hit sets that would
+    have been produced had it not been invoked.
+    Obviously, to combine static and dynamic ranking usefully,
+    it is necessary
+    to make a new ranking 
+    function; this is left
+    as an exercise for the reader. 
+   </para>
+
+
+   <para>
+    Dynamic ranking is done at query time rather than
+    indexing time (this is why we
+    call it ``dynamic ranking'' in the first place ...)
+    It is invoked by adding
+    the Bib-1 relation attribute with
+    value ``relevance'' to the PQF query (that is,
+    <literal>@attr&nbsp;2=102</literal>, see also  
+    <ulink url="ftp://ftp.loc.gov/pub/z3950/defs/bib1.txt">
+     The BIB-1 Attribute Set Semantics</ulink>). 
+    To find all articles with the word <literal>Eoraptor</literal> in
+    the title, and present them relevance ranked, issue the PQF query:
+    <screen>
+     @attr 2=102 @attr 1=4 Eoraptor
+    </screen>
+   </para>
+ 
+   <para>
+     The default <literal>rank-1</literal> ranking module implements a 
+     TF-IDF (Term Frequecy over Inverse Document Frequency) like algorithm.
+   </para>
+
+   <warning>
+     <para>
+      Notice that <literal>dynamic ranking</literal> is not compatible
+      with <literal>estimated hit sizes</literal>, as all documents in
+      a hit set must be acessed to compute the correct placing in a
+      ranking sorted list. Therefore the use attribute setting
+      <literal>@attr&nbsp;2=102</literal> clashes with 
+      <literal>@attr&nbsp;9=integer</literal>. 
+     </para>
+   </warning>  
+
+   <para>
+     It is possible to apply dynamic ranking on only parts of the PQF query:
+     <screen>
+     @and @attr 2=102 @attr 1=1010 Utah @attr 1=1018 Springer
+     </screen>
+     searches for all documents which have the term 'Utah' on the
+     body of text, and which have the term 'Springer' in the publisher
+     field, and sort them in the order of the relvance ranking made on
+     the body-of-text index only. 
+   </para>
+    <para>
+     Ranking weights may be used to pass a value to a ranking
+     algorithm, using the non-standard BIB-1 attribute type 9.
+     This allows one branch of a query to use one value while
+     another branch uses a different one.  For example, we can search
+     for <literal>utah</literal> in the title index with weight 30, as
+     well as in the ``any'' index with weight 20:
+     <screen>
+     @attr 2=102 @or @attr 9=30 @attr 1=4 utah @attr 9=20 utah
+     </screen>
+    </para>
+    <warning>
+     <para>
+      The ranking-weight feature is experimental. It may change in future
+      releases of zebra, and is not production mature. 
+     </para>
+    </warning>
+    
+   <para>
+     Notice that dynamic ranking can be enabled in sever side CQL
+     query expansion by adding <literal>@attr&nbsp;2=102</literal> to
+     the CQL config file. For example
+     <screen>
+      relationModifier.relevant		= 2=102
+     </screen>
+     invokes dynamic ranking each time a CQL query of the form 
+    <screen>
+     Z> querytype cql
+     Z> f alvis.text =/relevant house
+    </screen>
+     is issued. Dynamic ranking can also be automatically used on
+     specific CQL indexes by (for example) setting
+     <screen>
+      index.alvis.text                        = 1=text 2=102
+     </screen>
+     which then invokes dynamic ranking each time a CQL query of the form 
+    <screen>
+     Z> querytype cql
+     Z> f alvis.text = house
+    </screen>
+     is issued.
+   </para>
+
+    </sect2>
+
+
+ <sect2 id="administration-ranking-sorting">
+  <title>Sorting</title>
+   <para>
+     Zebra sorts efficiently using special sorting indexes
+     (type=<literal>s</literal>; so each sortable index must be known
+     at indexing time, specified in the configuration of record
+     indexing.  For example, to enable sorting according to the BIB-1
+     <literal>Date/time-added-to-db</literal> field, one could add the line
+     <screen>
+        xelm /*/@created               Date/time-added-to-db:s
+     </screen>
+     to any <literal>.abs</literal> record-indexing configuration file.
+     Similarily, one could add an indexing element of the form
+     <screen><![CDATA[       
+      <z:index name="date-modified" type="s">
+       <xsl:value-of select="some/xpath"/>
+      </z:index>
+      ]]></screen>
+     to any <literal>alvis</literal>-filter indexing stylesheet.
+     </para>
+     <para>
+      Indexing can be specified at searching time using a query term
+      carrying the non-standard
+      BIB-1 attribute-type <literal>7</literal>.  This removes the
+      need to send a Z39.50 <literal>Sort Request</literal>
+      separately, and can dramatically improve latency when the client
+      and server are on separate networks.
+      The sorting part of the query is separate from the rest of the
+      query - the actual search specification - and must be combined
+      with it using OR.
+     </para>
+     <para>
+      A sorting subquery needs two attributes: an index (such as a
+      BIB-1 type-1 attribute) specifying which index to sort on, and a
+      type-7 attribute whose value is be <literal>1</literal> for
+      ascending sorting, or <literal>2</literal> for descending.  The
+      term associated with the sorting attribute is the priority of
+      the sort key, where <literal>0</literal> specifies the primary
+      sort key, <literal>1</literal> the secondary sort key, and so
+      on.
+     </para>
+    <para>For example, a search for water, sort by title (ascending),
+    is expressed by the PQF query
+     <screen>
+     @or @attr 1=1016 water @attr 7=1 @attr 1=4 0
+     </screen>
+      whereas a search for water, sort by title ascending, 
+     then date descending would be
+     <screen>
+     @or @or @attr 1=1016 water @attr 7=1 @attr 1=4 0 @attr 7=2 @attr 1=30 1
+     </screen>
+    </para>
+    <para>
+     Notice the fundamental differences between <literal>dynamic
+     ranking</literal> and <literal>sorting</literal>: there can be
+     only one ranking function defined and configured; but multiple
+     sorting indexes can be specified dynamically at search
+     time. Ranking does not need to use specific indexes, so
+     dynamic ranking can be enabled and disabled without
+     re-indexing; whereas, sorting indexes need to be
+     defined before indexing.
+     </para>
+
+ </sect2>
+
+
+ </sect1>
+
+ <sect1 id="administration-extended-services">
+  <title>Extended Services: Remote Insert, Update and Delete</title>
+  
+  <para>
+    The extended services are not enabled by default in zebra - due to the
+    fact that they modify the system.
+    In order to allow anybody to update, use
+    <screen>
+    perm.anonymous: rw
+    </screen>
+    in the main zebra configuration file <filename>zebra.cfg</filename>.
+    Or, even better, allow only updates for a particular admin user. For
+    user <literal>admin</literal>, you could use:
+    <screen>
+     perm.admin: rw
+     passwd: passwordfile
+    </screen>
+    And in <filename>passwordfile</filename>, specify users and
+    passwords as colon seperated strings:
+    <screen> 
+     admin:secret
+    </screen> 
+   </para>
+   <para>
+    We can now start a yaz-client admin session and create a database:
+   <screen>
+    <![CDATA[
+     $ yaz-client localhost:9999 -u admin/secret
+     Z> adm-create
+     ]]>
+   </screen>
+    Now the <literal>Default</literal> database was created,
+    we can insert an XML file (esdd0006.grs
+    from example/gils/records) and index it:
+   <screen>  
+    <![CDATA[
+     Z> update insert 1 esdd0006.grs
+     ]]>
+   </screen>
+    The 3rd parameter - <literal>1</literal> here -
+      is the opaque record ID from <literal>Ext update</literal>.
+      It a record ID that <emphasis>we</emphasis> assign to the record
+    in question. If we do not 
+    assign one, the usual rules for match apply (recordId: from zebra.cfg).
+   </para>
+   <para>
+    Actually, we should have a way to specify "no opaque record id" for
+    yaz-client's update command.. We'll fix that.
+   </para>
+   <para>
+    The newly inserted record can be searched as usual:
+    <screen>
+    <![CDATA[
+     Z> f utah
+     Sent searchRequest.
+     Received SearchResponse.
+     Search was a success.
+     Number of hits: 1, setno 1
+     SearchResult-1: term=utah cnt=1
+     records returned: 0
+     Elapsed: 0.014179
+     ]]>
+    </screen>
+   </para>
+   <para>
+    Let's delete the beast:
+    <screen>
+    <![CDATA[
+     Z> update delete 1
+     No last record (update ignored)
+     Z> update delete 1 esdd0006.grs
+     Got extended services response
+     Status: done
+     Elapsed: 0.072441
+     Z> f utah
+     Sent searchRequest.
+     Received SearchResponse.
+     Search was a success.
+     Number of hits: 0, setno 2
+     SearchResult-1: term=utah cnt=0
+     records returned: 0
+     Elapsed: 0.013610
+     ]]>
+     </screen>
+    </para>
+    <para>
+    If shadow register is enabled in your
+    <filename>zebra.cfg</filename>,
+    you must run the adm-commit command
+    <screen>
+    <![CDATA[
+     Z> adm-commit
+     ]]>
+    </screen>
+     after each update session in order write your changes from the
+     shadow to the life register space.
+   </para>
+   <para>
+    Extended services are also available from the YAZ client layer. An
+    example of an YAZ-PHP extended service transaction is given here:
+    <screen>
+    <![CDATA[
+     $record = '<record><title>A fine specimen of a record</title></record>';
+
+     $options = array('action' => 'recordInsert',
+                      'syntax' => 'xml',
+                      'record' => $record,
+                      'databaseName' => 'mydatabase'
+                     );
+
+     yaz_es($yaz, 'update', $options);
+     yaz_es($yaz, 'commit', array());
+     yaz_wait();
+
+     if ($error = yaz_error($yaz))
+       echo "$error";
+     ]]>
+    </screen>  
+    The <literal>action</literal> parameter can be any of 
+    <literal>recordInsert</literal> (will fail if the record already exists),
+    <literal>recordReplace</literal> (will fail if the record does not exist),
+    <literal>recordDelete</literal> (will fail if the record does not
+       exist), and
+    <literal>specialUpdate</literal> (will insert or update the record
+       as needed).
+   </para>
+   <para>
+    If a record is inserted
+    using the action  <literal>recordInsert</literal> 
+    one can specify the optional
+    <literal>recordIdOpaque</literal> parameter, which is a
+    client-supplied, opaque record identifier. This identifier will
+    replace zebra's own automagic identifier generation.  
+   </para>
+   <para>
+    When using the action <literal>recordReplace</literal> or
+    <literal>recordDelete</literal>, one must specify the additional 
+    <literal>recordIdNumber</literal> parameter, which must be an
+    existing Zebra internal system ID number. When retrieving existing
+    records, the ID number is returned in the field
+    <literal>/*/id:idzebra/localnumber</literal> in the namespace
+    <literal>xmlns:id="http://www.indexdata.dk/zebra/"</literal>,
+    where it can be picked up for later record updates or deletes. 
+   </para>
+ </sect1>
+
+
+  <sect1 id="gfs-config">
+   <title>YAZ Frontend Virtual Hosts</title>
+    <para>
+     <command>zebrasrv</command> uses the YAZ server frontend and does
+     support multiple virtual servers behind multiple listening sockets.
+    </para>
+    &zebrasrv-virtual;
+ 
+   <para>
+    Section "Virtual Hosts" in the YAZ manual.
+    <filename>http://www.indexdata.dk/yaz/doc/server.vhosts.tkl</filename>
+   </para>
+ </sect1>
+
+
+  <sect1 id="administration-cql-to-pqf">
+   <title>Server Side CQL to PQF Query Translation</title>
+   <para>
+    Using the
+    <literal>&lt;cql2rpn&gt;l2rpn.txt&lt;/cql2rpn&gt;</literal>
+      YAZ Frontend Virtual
+    Hosts option, one can configure
+    the YAZ Frontend CQL-to-PQF
+    converter, specifying the interpretation of various 
+    <ulink url="http://www.loc.gov/standards/sru/cql/">CQL</ulink>
+    indexes, relations, etc. in terms of Type-1 query attributes.
+    <!-- The  yaz-client config file -->  
+   </para>
+   <para>
+    For example, using server-side CQL-to-PQF conversion, one might
+    query a zebra server like this:
+    <screen>
+    <![CDATA[
+     yaz-client localhost:9999
+     Z> querytype cql
+     Z> find text=(plant and soil)
+     ]]>
+    </screen>
+     and - if properly configured - even static relevance ranking can
+     be performed using CQL query syntax:
+    <screen>
+    <![CDATA[
+     Z> find text = /relevant (plant and soil)
+     ]]>
+     </screen>
+   </para>
+
+   <para>
+    By the way, the same configuration can be used to 
+    search using client-side CQL-to-PQF conversion:
+    (the only difference is <literal>querytype cql2rpn</literal> 
+    instead of 
+    <literal>querytype cql</literal>, and the call specifying a local
+    conversion file)
+    <screen>
+    <![CDATA[
+     yaz-client -q local/cql2pqf.txt localhost:9999
+     Z> querytype cql2rpn
+     Z> find text=(plant and soil)
+     ]]>
+     </screen>
+   </para>
+
+   <para>
+    Exhaustive information can be found in the
+    Section "Specification of CQL to RPN mappings" in the YAZ manual.
+    <ulink url="http://www.indexdata.dk/yaz/doc/tools.tkl#tools.cql.map">
+     http://www.indexdata.dk/yaz/doc/tools.tkl#tools.cql.map</ulink>,
+   and shall therefore not be repeated here.
+   </para> 
+  <!-- 
+  <para>
+    See 
+      <ulink url="http://www.loc.gov/z3950/agency/zing/cql/dc-indexes.html">
+      http://www.loc.gov/z3950/agency/zing/cql/dc-indexes.html</ulink>
+    for the Maintenance Agency's work-in-progress mapping of Dublin Core
+    indexes to Attribute Architecture (util, XD and BIB-2)
+    attributes.
+   </para>
+   -->
+ </sect1>
+
+
  
 </chapter>
+
  <!-- Keep this comment at the end of the file
  Local variables:
  mode: sgml