<chapter id="querymodel">
- <!-- $Id: querymodel.xml,v 1.11 2006-06-22 14:01:55 marc Exp $ -->
+ <!-- $Id: querymodel.xml,v 1.15 2006-06-23 13:45:41 marc Exp $ -->
<title>Query Model</title>
<sect1 id="querymodel-overview">
<sect1 id="querymodel-pqf">
- <title>Prefix Query Format structure and syntax</title>
+ <title>Prefix Query Format syntax and semantics</title>
<para>
The <ulink url="&url.yaz.pqf;">PQF grammer</ulink>
is documented in the YAZ manual, and shall not be
</sect3>
<para>
- The use attributes (type 1) of the predefined attribute sets can
- be reconfigured by tweaking the files
- <filename>tab/*.att</filename>.
- New attribute sets can be defined by adding similar files in the
- configuration path of the server.
+ The <literal>use attributes (type 1)</literal> mappings the
+ predefined attribute sets are found in the
+ attribute set configuration files <filename>tab/*.att</filename>.
</para>
<note>
default index using the default attribite set, the server choice
of access point/index, and the default non-use attributes.
<screen>
- Z> find "information"
+ Z> find information
</screen>
</para>
<para>
Equivalent query fully specified including all default values:
<screen>
- Z> find @attrset bib-1 @attr 1=1017 @attr 2=3 @attr 3=3 @attr 4=1 @attr 5=100 @attr 6=1 "information"
+ Z> find @attrset bib-1 @attr 1=1017 @attr 2=3 @attr 3=3 @attr 4=1 @attr 5=100 @attr 6=1 information
</screen>
</para>
<para>
- Finding all documents which have empty titles. Notice that the
- empty term must be quoted, but is otherwise legal.
+ Finding all documents which have the term
+ <emphasis>debussy</emphasis> in the title field.
<screen>
- Z> find @attr 1=4 ""
+ Z> find @attr 1=4 debussy
</screen>
</para>
<sect3 id="querymodel-use-string">
- <title>Zebra's special use attribute type 1 of form 'string'</title>
+ <title>Zebra's special access point of type 'string'</title>
<para>
The numeric <literal>use (type 1)</literal> attribute is usually
refered to from a given
</screen>
</para>
<para>
- See also <xref linkend="querymodel-bib1-mapping"/> for details, and
+ See also <xref linkend="querymodel-pqf-apt-mapping"/> for details, and
<xref linkend="server-sru"/>
for the SRU PQF query extention using string names as a fast
debugging facility.
</sect3>
<sect3 id="querymodel-use-xpath">
- <title>Zebra's special use attribute type 1 of form 'XPath'
+ <title>Zebra's special access point of type 'XPath'
for GRS filters</title>
<para>
As we have seen above, it is possible (albeit seldom a great
<title>Explain Attribute Set</title>
<para>
The Z39.50 standard defines the
- <ulink url="&url.z39.50.explain;">Explain</ulink>attribute set
- <literal>exp-1</literal>, which is used to discover information
+ <ulink url="&url.z39.50.explain;">Explain</ulink> attribute set
+ <literal>Exp-1</literal>, which is used to discover information
about a server's search semantics and functional capabilities
Zebra exposes a "classic"
Explain database by base name <literal>IR-Explain-1</literal>, which
</para>
<para>
The attribute-set <literal>exp-1</literal> consists of a single
- <literal>Use (type 1)</literal> attribute.
+ <literal>use attribute (type 1)</literal>.
</para>
<para>
In addition, the non-Use
<tr>
<td>AlwaysMatches</td>
<td>103</td>
- <td>unsupported</td>
+ <td>supported</td>
</tr>
</tbody>
</table>
<para>
+ The relation attributes
+ <literal>1-5</literal> are supported and work exactly as
+ expected.
+ All ordering operations are based on a lexicographical ordering,
+ <emphasis>expect</emphasis> when the
+ <literal>structure attribute numeric (109)</literal> is used. In
+ this case, ordering is numerical. See
+ <xref linkend="querymodel-bib1-structure"/>.
+ <screen>
+ Z> find @attr 1=Title @attr 2=1 music
+ ...
+ Number of hits: 11745, setno 1
+ ...
+ Z> find @attr 1=Title @attr 2=2 music
+ ...
+ Number of hits: 11771, setno 2
+ ...
+ Z> find @attr 1=Title @attr 2=3 music
+ ...
+ Number of hits: 532, setno 3
+ ...
+ Z> find @attr 1=Title @attr 2=4 music
+ ...
+ Number of hits: 11463, setno 4
+ ...
+ Z> find @attr 1=Title @attr 2=5 music
+ ...
+ Number of hits: 11419, setno 5
+ </screen>
+ </para>
+
+ <para>
The relation attribute
- <literal>relevance (102)</literal> is supported, see
+ <literal>Relevance (102)</literal> is supported, see
<xref linkend="administration-ranking"/> for full information.
- <!-- always-matches (103) not supported for all indexes -->
</para>
- <para>
- All ordering operations are based on a lexicographical ordering,
- <emphasis>expect</emphasis> when the
- <literal>structure attribute numeric (109)</literal> is used. In
- this case, ordering is numerical. See
- <xref linkend="querymodel-bib1-structure"/>.
- </para>
+ <para>
+ Ranked search for <emphasis>information retrieval</emphasis> in
+ the title-register:
+ <screen>
+ Z> find @attr 1=4 @attr 2=102 "information retrieval"
+ </screen>
+ </para>
<para>
- Ranked search for <emphasis>information retrieval</emphasis> in
- the title-register:
- <screen>
- Z> find @attr 1=4 @attr 2=102 "information retrieval"
- </screen>
- </para>
+ The relation attribute
+ <literal>AlwaysMatches (103)</literal> is in the default
+ configuration
+ supported in conjecture with structure attribute
+ <literal>Phrase (1)</literal> (which may be omitted by
+ default).
+ It can be configured to work with other structure attributes,
+ see the configuration file
+ <filename>tab/default.idx</filename> and
+ <xref linkend="querymodel-pqf-apt-mapping"/>.
+ </para>
+ <para>
+ <literal>AlwaysMatches (103)</literal> is a
+ great way to discover how many documents have been indexed in a
+ given field. The search term is ignored, but needed for correct
+ PQF syntax. An empty search term may be supplied.
+ <screen>
+ Z> find @attr 1=Title @attr 2=103 ""
+ Z> find @attr 1=Title @attr 2=103 @attr 4=1 ""
+ </screen>
+ </para>
+
+
</sect3>
<sect3 id="querymodel-bib1-position">
<note>
The exact mapping between PQF queries and Zebra internal indexes
and index types is explained in
- <xref linkend="querymodel-bib1-mapping"/>.
+ <xref linkend="querymodel-pqf-apt-mapping"/>.
</note>
</sect3>
<note>
The exact mapping between PQF queries and Zebra internal indexes
and index types is explained in
- <xref linkend="querymodel-bib1-mapping"/>.
+ <xref linkend="querymodel-pqf-apt-mapping"/>.
</note>
</sect3>
</sect2>
<literal>idxpath</literal> attribute set.
</para>
+ <sect2 id="querymodel-zebra-attr-allrecords">
+ <title>Zebra specific retrieval of all records</title>
+ <para>
+ Zebra defines a hardwired <literal>string</literal> index name
+ called <literal>_ALLRECORDS</literal>. It matches any record
+ contained in the database, if used in conjunction with
+ the relation attribute
+ <literal>AlwaysMatches (103)</literal>.
+ </para>
+ <para>
+ The <literal>_ALLRECORDS</literal> index name is used for total database
+ export. The search term is ignored, it may be empty.
+ <screen>
+ Z> find @attr 1=_ALLRECORDS @attr 2=103 ""
+ </screen>
+ </para>
+ <para>
+ Combination with other index types can be made. For example, to
+ find all records which are <emphasis>not</emphasis> indexed in
+ the <literal>Title</literal> register, issue one of the two
+ equivalent queries:
+ <screen>
+ Z> find @not @attr 1=_ALLRECORDS @attr 2=103 "" @attr 1=Title @attr 2=103 ""
+ Z> find @not @attr 1=_ALLRECORDS @attr 2=103 "" @attr 1=4 @attr 2=103 ""
+ </screen>
+ </para>
+ <warning>
+ The special string index <literal>_ALLRECORDS</literal> is
+ experimental, and the provided functionality and syntax may very
+ well change in future releases of Zebra.
+ </warning>
+
+ </sect2>
<sect2 id="querymodel-zebra-attr-search">
<title>Zebra specific Search Extentions to all Attribute Sets</title>
faster and does not require clients to deal with the Sort
Facility.
</para>
+
+ <para>
+ All ordering operations are based on a lexicographical ordering,
+ <emphasis>expect</emphasis> when the
+ <literal>structure attribute numeric (109)</literal> is used. In
+ this case, ordering is numerical. See
+ <xref linkend="querymodel-bib1-structure"/>.
+ </para>
+
<para>
The possible values after attribute <literal>type 7</literal> are
<literal>1</literal> ascending and
<para>
This feature is enabled when defining the
<literal>xpath enable</literal> option in the GRS filter
- <literal>*.abs</literal> configuration files. If one wants to use
+ <filename>*.abs</filename> configuration files. If one wants to use
the special <literal>idxpath</literal> numeric attribute set, the
main Zebra configuraiton file <filename>zebra.cfg</filename>
directive <literal>attset: idxpath.att</literal> must be enabled.
</sect2>
- <sect2 id="querymodel-bib1-mapping">
- <title>Mapping from Bib1 Attributes to Zebra internal
+ <sect2 id="querymodel-pqf-apt-mapping">
+ <title>Mapping from PQF atomic APT queries to Zebra internal
register indexes</title>
<para>
- TO-DO
- </para>
-
-
- <!-- see in util/zebramap.c
- int zebra_maps_attr
-
- if (completeness_value == 2 || completeness_value == 3)
- *complete_flag = 1;
- else
- *complete_flag = 0;
- *reg_id = 0;
-
- *sort_flag =(sort_relation_value > 0) ? 1 : 0;
- *search_type = "phrase";
- strcpy(rank_type, "void");
- if (relation_value == 102)
- {
- if (weight_value == -1)
- weight_value = 34;
- sprintf(rank_type, "rank,w=%d,u=%d", weight_value, use_value);
- }
- if (relation_value == 103)
- {
- *search_type = "always";
- *reg_id = 'w';
- return 0;
- }
- if (*complete_flag)
- *reg_id = 'p';
- else
- *reg_id = 'w';
- switch (structure_value)
- {
- case 6: /* word list */
- *search_type = "and-list";
- break;
- case 105: /* free-form-text */
- *search_type = "or-list";
- break;
- case 106: /* document-text */
- *search_type = "or-list";
- break;
- case -1:
- case 1: /* phrase */
- case 2: /* word */
- case 108: /* string */
- *search_type = "phrase";
- break;
- case 107: /* local-number */
- *search_type = "local";
- *reg_id = 0;
- break;
- case 109: /* numeric string */
- *reg_id = 'n';
- *search_type = "numeric";
- break;
- case 104: /* urx */
- *reg_id = 'u';
- *search_type = "phrase";
- break;
- case 3: /* key */
- *reg_id = '0';
- *search_type = "phrase";
- break;
- case 4: /* year */
- *reg_id = 'y';
- *search_type = "phrase";
- break;
- case 5: /* date */
- *reg_id = 'd';
- *search_type = "phrase";
- break;
- default:
- return -1;
- }
- return 0;
-
- -->
+ The rules for PQF APT mapping are rather tricky to grasp in the
+ first place. We deal first with the rules for deciding which
+ internal register or string index to use, according to the use
+ attribute or access point specified in the query. Thereafter we
+ deal with the rules for determining the correct structure type of
+ the named register.
+ </para>
-
+ <sect3 id="querymodel-pqf-apt-mapping-accesspoint">
+ <title>Mapping of PQF APT access points</title>
<para>
- <emphasis>Use</emphasis> attributes are interpreted according to the
- attribute sets which have been loaded in the
- <literal>zebra.cfg</literal> file, and are matched against specific
- fields as specified in the <literal>.abs</literal> file which
- describes the profile of the records which have been loaded.
- If no Use attribute is provided, a default of Bib-1 Any is assumed.
+ Zebra understands four fundamental different types of access
+ points, of which only the
+ <emphasis>numeric use attribute</emphasis> type access points
+ are defined by the <ulink url="&url.z39.50;">Z39.50</ulink>
+ standard.
+ All other access point types are Zebra specific, and non-portable.
</para>
-
+
+ <table id="querymodel-zebra-mapping-accesspoint-types"
+ frame="all" rowsep="1" colsep="1" align="center">
+
+ <caption>Acces point name mapping</caption>
+ <thead>
+ <tr>
+ <td>Acess Point</td>
+ <td>Type</td>
+ <td>Grammar</td>
+ <td>Notes</td>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>Use attibute</td>
+ <td>numeric</td>
+ <td>[1-9][1-9]*</td>
+ <td>directly mapped to string index name</td>
+ </tr>
+ <tr>
+ <td>String index name</td>
+ <td>string</td>
+ <td>[a-zA-Z](\-?[a-zA-Z0-9])*</td>
+ <td>normalized name is used as internal string index name</td>
+ </tr>
+ <tr>
+ <td>Zebra internal index name</td>
+ <td>zebra</td>
+ <td>_[a-zA-Z](_?[a-zA-Z0-9])*</td>
+ <td>hardwired internal string index name</td>
+ </tr>
+ <tr>
+ <td>XPATH special index</td>
+ <td>XPath</td>
+ <td>/.*</td>
+ <td>special xpath search for GRS indexed records</td>
+ </tr>
+ </tbody>
+ </table>
+
+ <para>
+ <literal>Attribute set names</literal> and
+ <literal>string index names</literal> are normalizes
+ according to the following rules: all <emphasis>single</emphasis>
+ hyphens <literal>'-'</literal> are stripped, and all upper case
+ letters are folded to lower case.
+ </para>
+
+ <para>
+ <emphasis>Numeric use attributes</emphasis> are mapped
+ to the Zebra internal
+ string index according to the attribute set defintion in use.
+ The default attribute set is <literal>Bib-1</literal>, and may be
+ omitted in the PQF query.
+ </para>
+
+ <para>
+ According to normalization and numeric
+ use attribute mapping, it follows that the following
+ PQF queries are considered equivalent (assuming the default
+ configuration has not been altered):
+ <screen>
+ Z> find @attr 1=Body-of-text serenade
+ Z> find @attr 1=bodyoftext serenade
+ Z> find @attr 1=BodyOfText serenade
+ Z> find @attr 1=bO-d-Y-of-tE-x-t serenade
+ Z> find @attr 1=1010 serenade
+ Z> find @attrset Bib-1 @attr 1=1010 serenade
+ Z> find @attrset bib1 @attr 1=1010 serenade
+ Z> find @attrset Bib1 @attr 1=1010 serenade
+ Z> find @attrset b-I-b-1 @attr 1=1010 serenade
+ </screen>
+ </para>
+
+ <para>
+ The <emphasis>numerical</emphasis>
+ <literal>use attributes (type 1)</literal>
+ are interpreted according to the
+ attribute sets which have been loaded in the
+ <literal>zebra.cfg</literal> file, and are matched against specific
+ fields as specified in the <literal>.abs</literal> file which
+ describes the profile of the records which have been loaded.
+ If no use attribute is provided, a default of
+ <literal>Bib-1 Use Any (1016)</literal> is
+ assumed.
+ The predefined <literal>use attribute sets</literal>
+ can be reconfigured by tweaking the configuration files
+ <filename>tab/*.att</filename>, and
+ new attribute sets can be defined by adding similar files in the
+ configuration path <literal>profilePath</literal> of the server.
+ </para>
+
+ <para>
+ <literal>String indexes</literal> can be acessed directly,
+ independently which attribute set is in use. These are just
+ ignored. The above mentioned name normalization applies.
+ <literal>String index names</literal> are defined in the
+ used indexing filter configuration files, for example in the
+ <literal>GRS</literal>
+ <filename>*.abs</filename> configuration files, or in the
+ <literal>alvis</literal> filter XSLT indexing stylesheets.
+ </para>
+
+ <para>
+ <literal>Zebra internal indexes</literal> can be acessed directly,
+ according to the same rules as the user defined
+ <literal>string indexes</literal>. The only difference is that
+ <literal>Zebra internal indexe names</literal> are hardwired,
+ all uppercase and
+ must start with the character <literal>'_'</literal>.
+ </para>
+
+ <para>
+ Finally, <literal>XPATH</literal> access points are only
+ available using the <literal>GRS</literal> filter for indexing.
+ These acees point names must start with the character
+ <literal>'/'</literal>, they are <emphasis>not
+ normalized</emphasis>, but passed unaltered to the Zebra internal
+ XPATH engine. See <xref linkend="querymodel-use-xpath"/>.
+
+ </para>
+
+
+ </sect3>
+
+
+ <sect3 id="querymodel-pqf-apt-mapping-structuretype">
+ <title>Mapping of PQF APT structure and completeness to
+ register type</title>
+ <para>
+ Internally Zebra has in it's default configuration several
+ different types of registers or indexes, whose tokenization and
+ character normalization rules differ. This reflects the fact that
+ serching fundamental different tokens like dates, numbers,
+ bitfields and string based text needs different rulesets.
+ </para>
+
+ <table id="querymodel-zebra-mapping-structure-types"
+ frame="all" rowsep="1" colsep="1" align="center">
+
+ <caption>Structure and completeness mapping to register types</caption>
+ <thead>
+ <tr>
+ <td>Structure</td>
+ <td>Completeness</td>
+ <td>Register type</td>
+ <td>Notes</td>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>
+ phrase (@attr 4=1), word (@attr 4=2),
+ word-list (@attr 4=6),
+ free-form-text (@attr 4=105), or document-text (@attr 4=106)
+ </td>
+ <td>Incomplete field (@attr 6=1)</td>
+ <td>Word ('w')</td>
+ <td>Traditional tokenized and character normalized word index</td>
+ </tr>
+ <tr>
+ <td>
+ phrase (@attr 4=1), word (@attr 4=2),
+ word-list (@attr 4=6),
+ free-form-text (@attr 4=105), or document-text (@attr 4=106)
+ </td>
+ <td>complete field' (@attr 6=3)</td>
+ <td>Phrase ('p')</td>
+ <td>Character normalized, but not tokenized index for phrase
+ matches
+ </td>
+ </tr>
+ <tr>
+ <td>urx (@attr 4=104)</td>
+ <td>ignored</td>
+ <td>URX/URL ('u')</td>
+ <td>Special index for URL web adresses</td>
+ </tr>
+ <tr>
+ <td>numeric (@attr 4=109)</td>
+ <td>ignored</td>
+ <td>Numeric ('u')</td>
+ <td>Special index for digital numbers</td>
+ </tr>
+ <tr>
+ <td>key (@attr 4=3)</td>
+ <td>ignored</td>
+ <td>Null bitmap ('0')</td>
+ <td>Used for non-tokenizated and non-normalized bit sequences</td>
+ </tr>
+ <tr>
+ <td>year (@attr 4=4)</td>
+ <td>ignored</td>
+ <td>Year ('y')</td>
+ <td>Non-tokenizated and non-normalized 4 digit numbers</td>
+ </tr>
+ <tr>
+ <td>date (@attr 4=5)</td>
+ <td>ignored</td>
+ <td>Date ('d')</td>
+ <td>Non-tokenizated and non-normalized ISO date strings</td>
+ </tr>
+ <tr>
+ <td>ignored</td>
+ <td>ignored</td>
+ <td>Sort ('s')</td>
+ <td>Used with special sort attribute set (@attr 7=1, @attr 7=2)</td>
+ </tr>
+ <tr>
+ <td>overruled</td>
+ <td>overruled</td>
+ <td>special</td>
+ <td>Internal record ID register, used whenever
+ Relation Always Matches (@attr 2=103) is specified</td>
+ </tr>
+ </tbody>
+ </table>
+
+ <!-- see in util/zebramap.c -->
+
<para>
If a <emphasis>Structure</emphasis> attribute of
<emphasis>Phrase</emphasis> is used in conjunction with a
against the contents of the phrase (long word) register, if one
exists for the given <emphasis>Use</emphasis> attribute.
A phrase register is created for those fields in the
- <literal>.abs</literal> file that contains a
+ GRS <filename>*.abs</filename> file that contains a
<literal>p</literal>-specifier.
- <!-- ### whatever the hell _that_ is -->
+ <screen>
+ Z> scan @attr 1=Title @attr 4=1 @attr 6=3 beethoven
+ ...
+ bayreuther festspiele (1)
+ * beethoven bibliography database (1)
+ benny carter (1)
+ ...
+ Z> find @attr 1=Title @attr 4=1 @attr 6=3 "beethoven bibliography"
+ ...
+ Number of hits: 0, setno 5
+ ...
+ Z> find @attr 1=Title @attr 4=1 @attr 6=3 "beethoven bibliography database"
+ ...
+ Number of hits: 1, setno 6
+ </screen>
</para>
<para>
contains multiple words, the term will only match if all of the words
are found immediately adjacent, and in the given order.
The word search is performed on those fields that are indexed as
- type <literal>w</literal> in the <literal>.abs</literal> file.
+ type <literal>w</literal> in the GRS <filename>*.abs</filename> file.
+ <screen>
+ Z> scan @attr 1=Title @attr 4=1 @attr 6=1 beethoven
+ ...
+ beefheart (1)
+ * beethoven (18)
+ beethovens (7)
+ ...
+ Z> find @attr 1=Title @attr 4=1 @attr 6=1 beethoven
+ ...
+ Number of hits: 18, setno 1
+ ...
+ Z> find @attr 1=Title @attr 4=1 @attr 6=1 "beethoven bibliography"
+ ...
+ Number of hits: 2, setno 2
+ ...
+ </screen>
</para>
<para>
natural-language, relevance-ranked query.
This search type uses the word register, i.e. those fields
that are indexed as type <literal>w</literal> in the
- <literal>.abs</literal> file.
+ GRS <filename>*.abs</filename> file.
</para>
<para>
If the <emphasis>Structure</emphasis> attribute is
<emphasis>Numeric String</emphasis> the term is treated as an integer.
The search is performed on those fields that are indexed
- as type <literal>n</literal> in the <literal>.abs</literal> file.
+ as type <literal>n</literal> in the GRS
+ <filename>*.abs</filename> file.
</para>
<para>
If the <emphasis>Structure</emphasis> attribute is
<emphasis>URx</emphasis> the term is treated as a URX (URL) entity.
The search is performed on those fields that are indexed as type
- <literal>u</literal> in the <literal>.abs</literal> file.
+ <literal>u</literal> in the <filename>*.abs</filename> file.
</para>
<para>
replacement) is accepted when terms are matched against the register
contents.
</para>
+
+ </sect3>
</sect2>
<sect2 id="querymodel-regular">
projects / idzebra-moved-to-github.git / blobdiff