<chapter id="administration">
- <!-- $Id: administration.xml,v 1.30 2006-05-01 12:59:33 marc Exp $ -->
+ <!-- $Id: administration.xml,v 1.33 2006-05-02 12:55:18 mike Exp $ -->
<title>Administrating Zebra</title>
<!-- ### It's a bit daft that this chapter (which describes half of
the configuration-file formats) is separated from
<sect1 id="administration-ranking">
<title>Relevance Ranking and Sorting of Result Sets</title>
+ <sect2>
+ <title>Overview</title>
<para>
The default ordering of a result set is left up to the server,
which inside Zebra means sorting in ascending document ID order.
</para>
<para>
- In case a good presentation ordering can be computed at
+ In cases where a good presentation ordering can be computed at
indexing time, we can use a fixed <literal>static ranking</literal>
scheme, which is provided for the <literal>alvis</literal>
indexing filter. This defines a fixed ordering of hit lists,
<para>
There are cases, however, where relevance of hit set documents is
- hghly dependent on the query processed.
+ highly dependent on the query processed.
Simply put, <literal>dynamic relevance ranking</literal>
- sortes a set of retrieved
+ sorts a set of retrieved
records such
that those most likely to be relevant to your request are
retrieved first.
- Internally, Zebra retrieves all documents ID's that satisfy your
- search query, and re-orders the hit list to arrange them based on
+ Internally, Zebra retrieves all documents that satisfy your
+ query, and re-orders the hit list to arrange them based on
a measurement of similarity between your query and the content of
each record.
</para>
lexicographical ordering of certain sort indexes created at
indexing time.
</para>
-
+ </sect2>
<sect2 id="administration-ranking-static">
are ordered
first by ascending static rank,
then by ascending document <literal>ID</literal>.
- </para>
- <para>
- This implies that the default rank <literal>0</literal>
- is the best rank at the
- beginning of the list, and <literal>max int</literal>
- is the worst static rank.
+ Zero
+ is the ``best'' rank, as it occurs at the
+ beginning of the list; higher numbers represent worse scores.
</para>
<para>
The experimental <literal>alvis</literal> filter provides a
after <emphasis>ascending</emphasis> static
rank, and for those doc's which have the same static rank, ordered
after <emphasis>ascending</emphasis> doc <literal>ID</literal>.
- See <xref linkend="record-model-alvisxslt"/> for the glory details.
+ See <xref linkend="record-model-alvisxslt"/> for the gory details.
</para>
</sect2>
<sect2 id="administration-ranking-dynamic">
<title>Dynamic Ranking</title>
<para>
- If one wants to do a little fiddeling with the static rank order,
- one has to invoke additional re-ranking/re-ordering using dynamic
- reranking or score functions. These functions return positive
- interger scores, where <emphasis>highest</emphasis> score is
- <emphasis>best</emphasis>, which means that the
- hit sets will be sorted according to
+ In order to fiddle with the static rank order, it is necessary to
+ invoke additional re-ranking/re-ordering using dynamic
+ ranking or score functions. These functions return positive
+ integer scores, where <emphasis>highest</emphasis> score is
+ ``best'';
+ hit sets are sorted according to
<emphasis>decending</emphasis>
scores (in contrary
to the index lists which are sorted according to
- <emphasis>ascending</emphasis> rank number and document ID).
+ ascending rank number and document ID).
</para>
<para>
- Those are in the zebra config file enabled by a directive like (use
- only one of these a time!):
+ Dynamic ranking is enabled by a directive like one of the
+ following in the zebra config file (use only one of these a time!):
<screen>
rank: rank-1 # default TDF-IDF like
rank: rank-static # dummy do-nothing
Notice that the <literal>rank-1</literal> and
<literal>zvrank</literal> do not use the static rank
information in the list keys, and will produce the same ordering
- with our without static ranking enabled.
+ with or without static ranking enabled.
</para>
<para>
The dummy <literal>rank-static</literal> reranking/scoring
function returns just
<literal>score = max int - staticrank</literal>
- in order to preserve the ordering of hit sets with and without it's
- call.
- Obviously, to combine static and dynamic ranking usefully, one wants
+ in order to preserve the static ordering of hit sets that would
+ have been produced had it not been invoked.
+ Obviously, to combine static and dynamic ranking usefully,
+ it is necessary
to make a new ranking
- function, which is left
+ function; this is left
as an exercise for the reader.
</para>
<para>
- Invoking dynamic ranking is done in query time (this is why we
- call it 'dynamic ranking' in the first place ..). One has to add
+ Dynamic ranking is done at query time rather than
+ indexing time (this is why we
+ call it ``dynamic ranking'' in the first place ...)
+ It is invoked by adding
the Bib-1 relation attribute with
- value "relevance" to the PQF query (that is, <literal>@attr
- 2=102</literal>, see also
+ value ``relevance'' to the PQF query (that is,
+ <literal>@attr 2=102</literal>, see also
<ulink url="ftp://ftp.loc.gov/pub/z3950/defs/bib1.txt">
The BIB-1 Attribute Set Semantics</ulink>).
- To find all articles with the word 'Eoraptor' in
- the title, and present them relevance ranked, one issues the PQF query:
+ To find all articles with the word <literal>Eoraptor</literal> in
+ the title, and present them relevance ranked, issue the PQF query:
<screen>
- Z> f @attr 2=102 @attr 1=4 Eoraptor
+ @attr 2=102 @attr 1=4 Eoraptor
</screen>
</para>
TF-IDF (Term Frequecy over Inverse Document Frequency) like algorithm.
</para>
+ <warning>
+ <para>
+ Notice that <literal>dynamic ranking</literal> is not compatible
+ with <literal>estimated hit sizes</literal>, as all documents in
+ a hit set must be acessed to compute the correct placing in a
+ ranking sorted list. Therefore the use attribute setting
+ <literal>@attr 2=102</literal> clashes with
+ <literal>@attr 9=integer</literal>.
+ </para>
+ </warning>
+
<para>
- It is possible to apply dynamic ranking on parts of the PQF query
- allone:
+ It is possible to apply dynamic ranking on only parts of the PQF query:
<screen>
- Z> f @and @attr 2=102 @attr 1=1010 Utah @attr 1=1018 Springer
+ @and @attr 2=102 @attr 1=1010 Utah @attr 1=1018 Springer
</screen>
searches for all documents which have the term 'Utah' on the
body of text, and which have the term 'Springer' in the publisher
field, and sort them in the order of the relvance ranking made on
- the body of text index only.
+ the body-of-text index only.
</para>
<para>
- Rank weight is a way to pass a value to a ranking algorithm - so that
- one APT has one value - while another as a different one. For
- example, we can
- search for 'utah' in use attribute set 'title' with weight 30, as
- well as in use attribute set 'any' with weight 20.
+ Ranking weights may be used to pass a value to a ranking
+ algorithm, using the non-standard BIB-1 attribute type 9.
+ This allows one branch of a query to use one value while
+ another branch uses a different one. For example, we can search
+ for <literal>utah</literal> in the title index with weight 30, as
+ well as in the ``any'' index with weight 20:
<screen>
- Z> f @attr 2=102 @or @attr 9=30 @attr 1=4 utah @attr 9=20 utah
+ @attr 2=102 @or @attr 9=30 @attr 1=4 utah @attr 9=20 utah
</screen>
</para>
<warning>
<para>
- The rank weight feature is experimental. It may change in future
+ The ranking-weight feature is experimental. It may change in future
releases of zebra, and is not production mature.
</para>
</warning>
<para>
- Notice that <literal>dynamic ranking</literal> can be enabled in
- sever side CQL query expansion by adding <literal>@attr
- 2=102</literal> to the CQL config file. For example
+ Notice that dynamic ranking can be enabled in sever side CQL
+ query expansion by adding <literal>@attr 2=102</literal> to
+ the CQL config file. For example
<screen>
relationModifier.relevant = 2=102
</screen>
- invokes dynamik ranking each time a CQL query of the form
+ invokes dynamic ranking each time a CQL query of the form
<screen>
Z> querytype cql
Z> f alvis.text =/relevant house
</screen>
- is issued. Dynamic ranking can be enabled on specific CQL indexes
- by (for example) setting
+ is issued. Dynamic ranking can also be automatically used on
+ specific CQL indexes by (for example) setting
<screen>
index.alvis.text = 1=text 2=102
</screen>
- which then invokes dynamik ranking each time a CQL query of the form
+ which then invokes dynamic ranking each time a CQL query of the form
<screen>
Z> querytype cql
Z> f alvis.text = house
<sect2 id="administration-ranking-sorting">
<title>Sorting</title>
<para>
- Sorting is enabled in the configuration of record indexing. For
- example, to enable sorting according to the BIB-1
+ Zebra sorts efficiently using special sorting indexes
+ (type=<literal>s</literal>; so each sortable index must be known
+ at indexing time, specified in the configuration of record
+ indexing. For example, to enable sorting according to the BIB-1
<literal>Date/time-added-to-db</literal> field, one could add the line
<screen>
xelm /*/@created Date/time-added-to-db:s
</screen>
- to any <literal>.abs</literal> record indexing config file, or
- similarily, one could add an indexing element of the form
+ to any <literal>.abs</literal> record-indexing configuration file.
+ Similarily, one could add an indexing element of the form
<screen><![CDATA[
<z:index name="date-modified" type="s">
<xsl:value-of select="some/xpath"/>
</z:index>
]]></screen>
- to any <literal>alvis</literal> indexing rule.
+ to any <literal>alvis</literal>-filter indexing stylesheet.
</para>
<para>
- To trigger a sorting on a pre-defined sorting index of type
- <literal>s</literal>, we can issue a sort with BIB-1
- embedded sort attribute set <literal>7</literal>.
- The embedded sort is a way to specify sort within a query - thus
- removing the need to send a Z39.50 <literal>Sort
- Request</literal> separately.
+ Indexing can be specified at searching time using a query term
+ carrying the non-standard
+ BIB-1 attribute-type <literal>7</literal>. This removes the
+ need to send a Z39.50 <literal>Sort Request</literal>
+ separately, and can dramatically improve latency when the client
+ and server are on separate networks.
+ The sorting part of the query is separate from the rest of the
+ query - the actual search specification - and must be combined
+ with it using OR.
</para>
<para>
- The value after attribute type <literal>7</literal> is
- <literal>1</literal> (=ascending), or <literal>2</literal>
- (=descending).
- The attributes+term (APT) node is separate from the rest of the
- PQF query, and must be <literal>@or</literal>'ed.
- The term associated with this attribute is the sorting level,
- where
- <literal>0</literal> specifies the primary sort key,
- <literal>1</literal> the secondary sort key, and so on.
+ A sorting subquery needs two attributes: an index (such as a
+ BIB-1 type-1 attribute) specifying which index to sort on, and a
+ type-7 attribute whose value is be <literal>1</literal> for
+ ascending sorting, or <literal>2</literal> for descending. The
+ term associated with the sorting attribute is the priority of
+ the sort key, where <literal>0</literal> specifies the primary
+ sort key, <literal>1</literal> the secondary sort key, and so
+ on.
</para>
<para>For example, a search for water, sort by title (ascending),
is expressed by the PQF query
<screen>
- Z> f @or @attr 1=1016 water @attr 7=1 @attr 1=4 0
+ @or @attr 1=1016 water @attr 7=1 @attr 1=4 0
</screen>
whereas a search for water, sort by title ascending,
then date descending would be
<screen>
- Z> f @or @or @attr 1=1016 water @attr 7=1 @attr 1=4 0 @attr 7=2 @attr 1=30 1
+ @or @or @attr 1=1016 water @attr 7=1 @attr 1=4 0 @attr 7=2 @attr 1=30 1
</screen>
</para>
<para>
Notice the fundamental differences between <literal>dynamic
- ranking</literal> and <literal>sorting</literal>: there can only
- be one ranking function defined and configured, but there can be
- specified multiple sorting indexes dynamically at search
- time. Ranking does not need to use specific indexes, which means,
+ ranking</literal> and <literal>sorting</literal>: there can be
+ only one ranking function defined and configured; but multiple
+ sorting indexes can be specified dynamically at search
+ time. Ranking does not need to use specific indexes, so
dynamic ranking can be enabled and disabled without
- re-indexing. On the other hand, sorting indexes need to be
+ re-indexing; whereas, sorting indexes need to be
defined before indexing.
</para>
projects / idzebra-moved-to-github.git / blobdiff