<chapter id="fields-and-charsets">
- <!-- $Id: field-structure.xml,v 1.12 2007-02-02 09:58:39 marc Exp $ -->
<title>Field Structure and Character Sets
</title>
special-purpose fields such as WWW-style linkages (URx).
</para>
+ <para>
+ Zebra 1.3 and Zebra versions 2.0.18 and earlier required that the field
+ type is a single character, e.g. <literal>w</literal> (for word), and
+ <literal>p</literal> for phrase. Zebra 2.0.20 and later allow field types
+ to be any string. This allows for greater flexibility - in particular
+ per-locale (language) fields can be defined.
+ </para>
+
+ <para>
+ Version 2.0.20 of Zebra can also be configured - per field - to use the
+ <ulink url="&url.icu;">ICU</ulink> library to perform tokenization and
+ normalization of strings. This is an alternative to the "charmap"
+ files which has been part of Zebra since its first release.
+ </para>
+
<section id="default-idx-file">
<title>The default.idx file</title>
<para>
The field types, and hence character sets, are associated with data
- elements by the .abs files (see above).
- The file <literal>default.idx</literal>
- provides the association between field type codes (as used in the .abs
- files) and the character map files (with the .chr suffix). The format
+ elements by the indexing rules (say <literal>title:w</literal>) in the
+ various filters. Fields are defined in a field definition file which,
+ by default, is called <filename>default.idx</filename>.
+ This file provides the association between field type codes
+ and the character map files (with the .chr suffix). The format
of the .idx file is as follows
</para>
-
+
<para>
<variablelist>
(non-space characters) separated by single space characters
(normalized to " " on display). When completeness is
disabled, each word is indexed as a separate entry. Complete subfield
- indexing is most useful for fields which are typically browsed (eg.
+ indexing is most useful for fields which are typically browsed (e.g.,
titles, authors, or subjects), or instances where a match on a
- complete subfield is essential (eg. exact title searching). For fields
+ complete subfield is essential (e.g., exact title searching). For fields
where completeness is disabled, the search engine will interpret a
search containing space characters as a word proximity search.
</para>
See <xref linkend="character-map-files"/> for details.
</para>
</listitem></varlistentry>
+
+ <varlistentry>
+ <term>icuchain <replaceable>filename</replaceable></term>
+ <listitem>
+ <para>
+ Specifies the filename with ICU tokenization and
+ normalization rules.
+ See <xref linkend="icuchain-files"/> for details.
+ Using icuchain for a field type is an alternative to
+ charmap. It does not make sense to define both
+ icuchain and charmap for the same field type.
+ </para>
+ </listitem></varlistentry>
</variablelist>
</para>
- <para>
- Following are three excerpts of the standard
- <filename>tab/default.idx</filename> configuration file. Notice
- that the <literal>index</literal> and <literal>sort</literal>
- are grouping directives, which bind all other following directives
- to them:
- <screen>
+ <example id="field-types">
+ <title>Field types</title>
+ <para>
+ Following are three excerpts of the standard
+ <filename>tab/default.idx</filename> configuration file. Notice
+ that the <literal>index</literal> and <literal>sort</literal>
+ are grouping directives, which bind all other following directives
+ to them:
+ <screen>
# Traditional word index
- # Used if completenss is 'incomplete field' (@attr 6=1) and
+ # Used if completeness is 'incomplete field' (@attr 6=1) and
# structure is word/phrase/word-list/free-form-text/document-text
index w
completeness 0
sort s
completeness 1
charmap string.chr
- </screen>
- </para>
+ </screen>
+ </para>
+ </example>
</section>
<section id="character-map-files">
- <title>The character map file format</title>
+ <title>Charmap Files</title>
<para>
The character map files are used to define the word tokenization
and character normalization performed before inserting text into
<para>
Curly braces {} may be used to enclose ranges of single
characters (possibly using the escape convention described in the
- preceding point), eg. {a-z} to introduce the
+ preceding point), e.g., {a-z} to introduce the
standard range of ASCII characters.
Note that the interpretation of such a range depends on
the concrete representation in your local, physical character set.
<listitem>
<para>
- paranthesises () may be used to enclose multi-byte characters -
- eg. diacritics or special national combinations (eg. Spanish
+ parentheses () may be used to enclose multi-byte characters -
+ e.g., diacritics or special national combinations (e.g., Spanish
"ll"). When found in the input stream (or a search term),
these characters are viewed and sorted as a single character, with a
sorting value depending on the position of the group in the value
character-representations to their natural form, etc. The
map directive can also be used to ignore leading articles in
searching and/or sorting, and to perform other special
- transformations. See section <xref
- linkend="leading-articles"/>.
+ transformations.
</para>
<para>
For example, <literal>scan.chr</literal> contains the following
map (ø) ø
map (å) å
]]></screen>
+ </para>
+ <para>
+ In addition to specifying sort orders, space (blank) handling,
+ and upper/lowercase folding, you can also use the character map
+ files to make &zebra; ignore leading articles in sorting records,
+ or when doing complete field searching.
+ </para>
+ <para>
+ This is done using the <literal>map</literal> directive in the
+ character map file. In a nutshell, what you do is map certain
+ sequences of characters, when they occur <emphasis> in the
+ beginning of a field</emphasis>, to a space. Assuming that the
+ character "@" is defined as a space character in your file, you
+ can do:
+ <screen>
+ map (^The\s) @
+ map (^the\s) @
+ </screen>
+ The effect of these directives is to map either 'the' or 'The',
+ followed by a space character, to a space. The hat ^ character
+ denotes beginning-of-field only when complete-subfield indexing
+ or sort indexing is taking place; otherwise, it is treated just
+ as any other character.
+ </para>
+ <para>
+ Because the <literal>default.idx</literal> file can be used to
+ associate different character maps with different indexing types
+ -- and you can create additional indexing types, should the need
+ arise -- it is possible to specify that leading articles should
+ be ignored either in sorting, in complete-field searching, or
+ both.
+ </para>
+ <para>
+ If you ignore certain prefixes in sorting, then these will be
+ eliminated from the index, and sorting will take place as if
+ they weren't there. However, if you set the system up to ignore
+ certain prefixes in <emphasis>searching</emphasis>, then these
+ are deleted both from the indexes and from query terms, when the
+ client specifies complete-field searching. This has the effect
+ that a search for 'the science journal' and 'science journal'
+ would both produce the same results.
</para>
</listitem></varlistentry>
<varlistentry>
<term>equivalent <replaceable>value-set</replaceable></term>
<listitem>
<para>
- This directive introduces equivalence classes of characters
- and/or strings for sorting purposes only. It resembles the map
- directive, but does not affect search and retrieval indexing,
- but only sorting order under present requests.
+ This directive introduces equivalence classes of strings for
+ searching purposes only. It's a one-to-many
+ conversion that takes place only during search before the map
+ directive kicks in.
</para>
<para>
- For example, <literal>scan.chr</literal> contains the following
- equivalent sorting instructions, which can be uncommented:
+ For example given:
<screen><![CDATA[
- # equivalent æä(ae)
- # equivalent øö(oe)
- # equivalent å(aa)
- # equivalent uü
+ equivalent æä(ae)
]]></screen>
</para>
+ <para>
+ a search for the <literal>äsel</literal> will be be match any of
+ <literal>æsel</literal>, <literal>äsel</literal> and
+ <literal>aesel</literal>.
+ </para>
</listitem></varlistentry>
</variablelist>
</para>
</section>
- <section id="leading-articles">
- <title>Ignoring leading articles</title>
+
+ <section id="icuchain-files">
+ <title>ICU Chain Files</title>
<para>
- In addition to specifying sort orders, space (blank) handling,
- and upper/lowercase folding, you can also use the character map
- files to make &zebra; ignore leading articles in sorting records,
- or when doing complete field searching.
+ The <ulink url="&url.icu;">ICU</ulink> chain files defines a
+ <emphasis>chain</emphasis> of rules
+ which specify the conversion process to be carried out for each
+ record string for indexing.
</para>
<para>
- This is done using the <literal>map</literal> directive in the
- character map file. In a nutshell, what you do is map certain
- sequences of characters, when they occur <emphasis> in the
- beginning of a field</emphasis>, to a space. Assuming that the
- character "@" is defined as a space character in your file, you
- can do:
- <screen>
- map (^The\s) @
- map (^the\s) @
- </screen>
- The effect of these directives is to map either 'the' or 'The',
- followed by a space character, to a space. The hat ^ character
- denotes beginning-of-field only when complete-subfield indexing
- or sort indexing is taking place; otherwise, it is treated just
- as any other character.
+ Both searching and sorting is based on the <emphasis>sort</emphasis>
+ normalization that ICU provides. This means that scan and sort will
+ return terms in the sort order given by ICU.
</para>
<para>
- Because the <literal>default.idx</literal> file can be used to
- associate different character maps with different indexing types
- -- and you can create additional indexing types, should the need
- arise -- it is possible to specify that leading articles should
- be ignored either in sorting, in complete-field searching, or
- both.
+ Zebra is using YAZ' ICU wrapper. Refer to the
+ <ulink url="&url.yaz.yaz-icu;">yaz-icu man page</ulink> for
+ documentation about the ICU chain rules.
</para>
+ <tip>
+ <para>
+ Use the yaz-icu program to test your icuchain rules.
+ </para>
+ </tip>
+ <example id="indexing-greek-example"><title>Indexing Greek text</title>
+ <para>
+ Consider a system where all "regular" text is to be indexed
+ using as Greek (locale: EL).
+ We would have to change our index type file - to read
+ <screen>
+ # Index greek words
+ index w
+ completeness 0
+ position 1
+ alwaysmatches 1
+ firstinfield 1
+ icuahain greek.xml
+ ..
+ </screen>
+ The ICU chain file <filename>greek.xml</filename> could look
+ as follows:
+ <screen><![CDATA[
+ <icu_chain locale="el">
+ <transform rule="[:Control:] Any-Remove"/>
+ <tokenize rule="l"/>
+ <transform rule="[[:WhiteSpace:][:Punctuation:]] Remove"/>
+ <display/>
+ <casemap rule="l"/>
+ </icu_chain>
+ ]]></screen>
+ </para>
+ </example>
<para>
- If you ignore certain prefixes in sorting, then these will be
- eliminated from the index, and sorting will take place as if
- they weren't there. However, if you set the system up to ignore
- certain prefixes in <emphasis>searching</emphasis>, then these
- are deleted both from the indexes and from query terms, when the
- client specifies complete-field searching. This has the effect
- that a search for 'the science journal' and 'science journal'
- would both produce the same results.
+ Zebra is shipped with a field types file <filename>icu.idx</filename>
+ which is an ICU chain version of <filename>default.idx</filename>.
</para>
+
+ <example id="indexing-marcxml-example"><title>MARCXML indexing using ICU</title>
+ <para>
+ The directory <filename>examples/marcxml</filename> includes
+ a complete sample with MARCXML records that are DOM XML indexed
+ using ICU chain rules. Study the
+ <filename>README</filename> in the <filename>marcxml</filename>
+ directory for details.
+ </para>
+ </example>
</section>
</chapter>
sgml-always-quote-attributes:t
sgml-indent-step:1
sgml-indent-data:t
- sgml-parent-document: "zebra.xml"
+ sgml-parent-document: "idzebra.xml"
sgml-local-catalogs: nil
sgml-namecase-general:t
End:
projects / idzebra-moved-to-github.git / blobdiff