-<!-- $Id: tools.xml,v 1.28 2003-10-23 09:56:57 adam Exp $ -->
+<!-- $Id: tools.xml,v 1.54 2006-10-31 09:15:00 mike Exp $ -->
<chapter id="tools"><title>Supporting Tools</title>
<para>
top-set ::= [ '@attrset' string ]
- query-struct ::= attr-spec | simple | complex | '@term' term-type
+ query-struct ::= attr-spec | simple | complex | '@term' term-type query
attr-spec ::= '@attr' [ string ] string query-struct
<para>
The @attr operator is followed by an attribute specification
(<literal>attr-spec</literal> above). The specification consists
- of optional an attribute set, an attribute type-value pair and
- a sub query. The attribute type-value pair is packed in one string:
- an attribute type, a dash, followed by an attribute value.
+ of an optional attribute set, an attribute type-value pair and
+ a sub-query. The attribute type-value pair is packed in one string:
+ an attribute type, an equals sign, and an attribute value, like this:
+ <literal>@attr 1=1003</literal>.
The type is always an integer but the value may be either an
integer or a string (if it doesn't start with a digit character).
+ A string attribute-value is encoded as a Type-1 ``complex''
+ attribute with the list of values containing the single string
+ specified, and including no semantic indicators.
</para>
<para>
<literal>@and</literal>. Its semantics are described in
section 3.7.2 (Proximity) of Z39.50 the standard itself, which
can be read on-line at
- <ulink url="http://lcweb.loc.gov/z3950/agency/markup/09.html"/>
+ <ulink url="&url.z39.50.proximity;"/>
</para>
<para>
In PQF, the proximity operation is represented by a sequence
</itemizedlist>
(The numeric values of the relation and well-known unit-code
parameters are taken straight from
- <ulink url="http://lcweb.loc.gov/z3950/agency/asn1.html#ProximityOperator"
+ <ulink url="&url.z39.50.proximity.asn1;"
>the ASN.1</ulink> of the proximity structure in the standard.)
</para>
</sect3>
<sect3 id="pqf-examples"><title>PQF queries</title>
- <para>Queries using simple terms.
- <screen>
- dylan
- "bob dylan"
- </screen>
- </para>
- <para>Boolean operators.
- <screen>
- @or "dylan" "zimmerman"
- @and @or dylan zimmerman when
- @and when @or dylan zimmerman
- </screen>
- </para>
- <para>
- Reference to result sets.
- <screen>
- @set Result-1
- @and @set seta setb
- </screen>
- </para>
- <para>
- Attributes for terms.
- <screen>
- @attr 1=4 computer
- @attr 1=4 @attr 4=1 "self portrait"
- @attrset exp1 @attr 1=1 CategoryList
- @attr gils 1=2008 Copenhagen
- @attr 1=/book/title computer
- </screen>
- </para>
- <para>
- Proximity.
- <screen>
- @prox 0 3 1 2 k 2 dylan zimmerman
- </screen>
- <note><para>
- Here the parameters 0, 3, 1, 2, k and 2 represent exclusion,
- distance, ordered, relation, which-code and unit-code, in that
- order. So:
- <itemizedlist>
- <listitem><para>
- exclusion = 0: the proximity condition must hold
- </para></listitem>
- <listitem><para>
- distance = 3: the terms must be three units apart
- </para></listitem>
- <listitem><para>
- ordered = 1: they must occur in the order they are specified
- </para></listitem>
- <listitem><para>
- relation = 2: lessThanOrEqual (to the distance of 3 units)
- </para></listitem>
- <listitem><para>
- which-code is ``known'', so the standard unit-codes are used
- </para></listitem>
- <listitem><para>
- unit-code = 2: word.
- </para></listitem>
- </itemizedlist>
- So the whole proximity query means that the words
- <literal>dylan</literal> and <literal>zimmerman</literal> must
- both occur in the record, in that order, differing in position
- by three or fewer words (i.e. with two or fewer words between
- them.) The query would find ``Bob Dylan, aka. Robert
- Zimmerman'', but not ``Bob Dylan, born as Robert Zimmerman''
- since the distance in this case is four.
- </para></note>
- </para>
- <para>
- Specifying term type.
- <screen>
- @term string "a UTF-8 string, maybe?"
- </screen>
- </para>
- <para>Mixed queries
- <screen>
- @or @and bob dylan @set Result-1
-
- @attr 4=1 @and @attr 1=1 "bob dylan" @attr 1=4 "slow train coming"
-
- @and @attr 2=4 @attr gils 1=2038 -114 @attr 2=2 @attr gils 1=2039 -109
+ <example id="example.pqf.simple.terms">
+ <title>PQF queries using simple terms</title>
+ <para>
+ <screen>
+ dylan
+
+ "bob dylan"
+ </screen>
+ </para>
+ </example>
+ <example id="pqf.example.pqf.boolean.operators">
+ <title>PQF boolean operators</title>
+ <para>
+ <screen>
+ @or "dylan" "zimmerman"
+
+ @and @or dylan zimmerman when
+
+ @and when @or dylan zimmerman
+ </screen>
+ </para>
+ </example>
+ <example id="example.pqf.result.sets">
+ <title>PQF references to result sets</title>
+ <para>
+ <screen>
+ @set Result-1
+
+ @and @set seta @set setb
+ </screen>
+ </para>
+ </example>
+ <example id="example.pqf.attributes">
+ <title>Attributes for terms</title>
+ <para>
+ <screen>
+ @attr 1=4 computer
+
+ @attr 1=4 @attr 4=1 "self portrait"
+
+ @attrset exp1 @attr 1=1 CategoryList
+
+ @attr gils 1=2008 Copenhagen
+
+ @attr 1=/book/title computer
+ </screen>
+ </para>
+ </example>
+ <example id="example.pqf.proximity">
+ <title>PQF Proximity queries</title>
+ <para>
+ <screen>
+ @prox 0 3 1 2 k 2 dylan zimmerman
+ </screen>
+ <note><para>
+ Here the parameters 0, 3, 1, 2, k and 2 represent exclusion,
+ distance, ordered, relation, which-code and unit-code, in that
+ order. So:
+ <itemizedlist>
+ <listitem><para>
+ exclusion = 0: the proximity condition must hold
+ </para></listitem>
+ <listitem><para>
+ distance = 3: the terms must be three units apart
+ </para></listitem>
+ <listitem><para>
+ ordered = 1: they must occur in the order they are specified
+ </para></listitem>
+ <listitem><para>
+ relation = 2: lessThanOrEqual (to the distance of 3 units)
+ </para></listitem>
+ <listitem><para>
+ which-code is ``known'', so the standard unit-codes are used
+ </para></listitem>
+ <listitem><para>
+ unit-code = 2: word.
+ </para></listitem>
+ </itemizedlist>
+ So the whole proximity query means that the words
+ <literal>dylan</literal> and <literal>zimmerman</literal> must
+ both occur in the record, in that order, differing in position
+ by three or fewer words (i.e. with two or fewer words between
+ them.) The query would find ``Bob Dylan, aka. Robert
+ Zimmerman'', but not ``Bob Dylan, born as Robert Zimmerman''
+ since the distance in this case is four.
+ </para></note>
+ </para>
+ </example>
+ <example id="example.pqf.search.term.type">
+ <title>PQF specification of search term type</title>
+ <para>
+ <screen>
+ @term string "a UTF-8 string, maybe?"
+ </screen>
+ </para>
+ </example>
+ <example id="example.pqf.mixed.queries">
+ <title>PQF mixed queries</title>
+ <para>
+ <screen>
+ @or @and bob dylan @set Result-1
+
+ @attr 4=1 @and @attr 1=1 "bob dylan" @attr 1=4 "slow train coming"
+
+ @and @attr 2=4 @attr gils 1=2038 -114 @attr 2=2 @attr gils 1=2039 -109
</screen>
- <note>
+ <note>
<para>
- The last of these examples is a spatial search: in
- <ulink url="http://www.gils.net/prof_v2.html#sec_7_4"
+ The last of these examples is a spatial search: in
+ <ulink url="http://www.gils.net/prof_v2.html#sec_7_4"
>the GILS attribute set</ulink>,
- access point
- 2038 indicates West Bounding Coordinate and
- 2030 indicates East Bounding Coordinate,
- so the query is for areas extending from -114 degrees
- to no more than -109 degrees.
+ access point
+ 2038 indicates West Bounding Coordinate and
+ 2030 indicates East Bounding Coordinate,
+ so the query is for areas extending from -114 degrees
+ to no more than -109 degrees.
</para>
- </note>
- </para>
+ </note>
+ </para>
+ </example>
</sect3>
</sect2>
<sect2 id="CCL"><title>CCL</title>
</para>
<para>
- The <ulink url="http://europagate.dtv.dk/">EUROPAGATE</ulink>
- research project working under the Libraries programme
+ The EUROPAGATE research project working under the Libraries programme
of the European Commission's DG XIII has, amongst other useful tools,
implemented a general-purpose CCL parser which produces an output
structure that can be trivially converted to the internal RPN
license, it is included as a supplement to &yaz;.
</para>
- <sect3><title>CCL Syntax</title>
+ <sect3 id="ccl.syntax">
+ <title>CCL Syntax</title>
<para>
The CCL parser obeys the following grammar for the FIND argument.
</screen>
- <example><title>CCL queries</title>
+ <example id="example.ccl.queries">
+ <title>CCL queries</title>
<para>
The following queries are all valid:
</para>
</example>
</sect3>
- <sect3><title>CCL Qualifiers</title>
+ <sect3 id="ccl.qualifiers">
+ <title>CCL Qualifiers</title>
<para>
Qualifiers are used to direct the search to a particular searchable
lines in a CCL profile: qualifier specification,
qualifier alias, comments and directives.
</para>
- <sect4><title id="qualifier-specification">Qualifier specification</title>
+ <sect4 id="ccl.qualifier.specification">
+ <title>Qualifier specification</title>
<para>
A qualifier specification is of the form:
</para>
or <literal>c</literal> for completeness.
The attributes for the special qualifier name <literal>term</literal>
are used when no CCL qualifier is given in a query.
- <table><title>Common Bib-1 attributes</title>
+ <table id="ccl.common.bib1.attributes">
+ <title>Common Bib-1 attributes</title>
<tgroup cols="2">
<colspec colwidth="2*" colname="type"></colspec>
<colspec colwidth="9*" colname="description"></colspec>
<row>
<entry><literal>u=</literal><replaceable>value</replaceable></entry>
<entry>
- Use attribute. Common use attributes are
+ Use attribute (1). Common use attributes are
1 Personal-name, 4 Title, 7 ISBN, 8 ISSN, 30 Date,
62 Subject, 1003 Author), 1016 Any. Specify value
as an integer.
<row>
<entry><literal>r=</literal><replaceable>value</replaceable></entry>
<entry>
- Relation attribute. Common values are
+ Relation attribute (2). Common values are
1 <, 2 <=, 3 =, 4 >=, 5 >, 6 <>,
100 phonetic, 101 stem, 102 relevance, 103 always matches.
</entry>
<row>
<entry><literal>p=</literal><replaceable>value</replaceable></entry>
<entry>
- Position attribute. Values: 1 first in field, 2
+ Position attribute (3). Values: 1 first in field, 2
first in any subfield, 3 any position in field.
</entry>
</row>
<row>
<entry><literal>s=</literal><replaceable>value</replaceable></entry>
<entry>
- Structure attribute. Values: 1 phrase, 2 word,
+ Structure attribute (4). Values: 1 phrase, 2 word,
3 key, 4 year, 5 date, 6 word list, 100 date (un),
101 name (norm), 102 name (un), 103 structure, 104 urx,
105 free-form-text, 106 document-text, 107 local-number,
<row>
<entry><literal>t=</literal><replaceable>value</replaceable></entry>
<entry>
- Truncation attribute. Values: 1 right, 2 left,
+ Truncation attribute (5). Values: 1 right, 2 left,
3 left& right, 100 none, 101 process #, 102 regular-1,
103 regular-2, 104 CCL.
</entry>
<row>
<entry><literal>c=</literal><replaceable>value</replaceable></entry>
<entry>
- Completeness attribute. Values: 1 incomplete subfield,
+ Completeness attribute (6). Values: 1 incomplete subfield,
2 complete subfield, 3 complete field.
</entry>
</row>
</table>
</para>
<para>
- The complete list of Bib-1 attributes can be found
- <ulink url="http://lcweb.loc.gov/z3950/agency/defns/bib1.html">
- here
- </ulink>.
+ Refer to the complete
+ <ulink url="&url.z39.50.attset.bib1;">list of Bib-1 attributes</ulink>
</para>
<para>
It is also possible to specify non-numeric attribute values,
which are used in combination with certain types.
The special combinations are:
- <table><title>Special attribute combos</title>
+ <table id="ccl.special.attribute.combos">
+ <title>Special attribute combos</title>
<tgroup cols="2">
<colspec colwidth="2*" colname="name"></colspec>
<colspec colwidth="9*" colname="description"></colspec>
</row>
<row><entry><literal>r=o</literal></entry><entry>
- Allows operators greather-than, less-than, ... equals and
- sets relation attribute accordingly (relation ordered).
+ Allows ranges and the operators greather-than, less-than, ...
+ equals.
+ This sets Bib-1 relation attribute accordingly (relation
+ ordered). A query construct is only treated as a range if
+ dash is used and that is surrounded by white-space. So
+ <literal>-1980</literal> is treated as term
+ <literal>"-1980"</literal> not <literal><= 1980</literal>.
+ If <literal>- 1980</literal> is used, however, that is
+ treated as a range.
+ </entry>
+ </row>
+
+ <row><entry><literal>r=r</literal></entry><entry>
+ Similar to <literal>r=o</literal> but assumes that terms
+ are non-negative (not prefixed with <literal>-</literal>).
+ Thus, a dash will always be treated as a range.
+ The construct <literal>1980-1990</literal> is
+ treated as a range with <literal>r=r</literal> but as a
+ single term <literal>"1980-1990"</literal> with
+ <literal>r=o</literal>. The special attribute
+ <literal>r=r</literal> is available in YAZ 2.0.24 or later.
</entry>
</row>
</tgroup>
</table>
</para>
- <example><title>CCL profile</title>
+ <example id="example.ccl.profile"><title>CCL profile</title>
<para>
Consider the following definition:
</para>
date u=30 r=o
</screen>
<para>
- Four qualifiers are defined - <literal>ti</literal>,
- <literal>au</literal>, <literal>ranked</literal> and
- <literal>date</literal>.
- </para>
- <para>
<literal>ti</literal> and <literal>au</literal> both set
structure attribute to phrase (s=1).
<literal>ti</literal>
<para>
Query
<screen>
- year > 1980
+ date > 1980
</screen>
- is a valid query, while
+ is a valid query. But
<screen>
ti > 1980
</screen>
</para>
</example>
</sect4>
- <sect4><title>Qualifier alias</title>
+ <sect4 id="ccl.qualifier.alias">
+ <title>Qualifier alias</title>
<para>
A qualifier alias is of the form:
</para>
be an alias for <replaceable>q1</replaceable>,
<replaceable>q2</replaceable>... such that the CCL
query <replaceable>q=x</replaceable> is equivalent to
- <replaceable>q1=x or w2=x or ...</replaceable>.
+ <replaceable>q1=x or q2=x or ...</replaceable>.
</para>
</sect4>
- <sect4><title>Comments</title>
+ <sect4 id="ccl.comments">
+ <title>Comments</title>
<para>
Lines with white space or lines that begin with
character <literal>#</literal> are treated as comments.
</para>
</sect4>
- <sect4><title>Directives</title>
+ <sect4 id="ccl.directives">
+ <title>Directives</title>
<para>
Directive specifications takes the form
</para>
<para><literal>@</literal><replaceable>directive</replaceable> <replaceable>value</replaceable>
</para>
- <table><title>CCL directives</title>
+ <table id="ccl.directives.table">
+ <title>CCL directives</title>
<tgroup cols="3">
<colspec colwidth="2*" colname="name"></colspec>
<colspec colwidth="8*" colname="description"></colspec>
</table>
</sect4>
</sect3>
- <sect3><title>CCL API</title>
+ <sect3 id="ccl.api">
+ <title>CCL API</title>
<para>
All public definitions can be found in the header file
<filename>ccl.h</filename>. A profile identifier is of type
</para>
</sect3>
</sect2>
- <sect2 id="tools.cql"><title>CQL</title>
+ <sect2 id="cql"><title>CQL</title>
<para>
- <ulink url="http://www.loc.gov/z3950/agency/zing/cql/">CQL</ulink>
+ <ulink url="&url.cql;">CQL</ulink>
- Common Query Language - was defined for the
- <ulink url="http://www.loc.gov/z3950/agency/zing/srw/">SRW</ulink>
- protocol.
+ <ulink url="&url.sru;">SRU</ulink> protocol.
In many ways CQL has a similar syntax to CCL.
The objective of CQL is different. Where CCL aims to be
an end-user language, CQL is <emphasis>the</emphasis> protocol
- query language for SRW.
+ query language for SRU.
</para>
<tip>
<para>
If you are new to CQL, read the
- <ulink url="http://zing.z3950.org/cql/intro.html">Gentle
- Introduction</ulink>.
+ <ulink url="&url.cql.intro;">Gentle Introduction</ulink>.
</para>
</tip>
<para>
<listitem>
<para>
The parser converts a valid CQL query to PQF, thus providing a
- way to use CQL for both SRW/SRU servers and Z39.50 targets at the
+ way to use CQL for both SRU servers and Z39.50 targets at the
same time.
</para>
</listitem>
<listitem>
<para>
The parser converts CQL to
- <ulink url="http://www.loc.gov/z3950/agency/zing/cql/xcql.html">
- XCQL</ulink>.
+ <ulink url="&url.xcql;">XCQL</ulink>.
XCQL is an XML representation of CQL.
- XCQL is part of the SRW specification. However, since SRU
+ XCQL is part of the SRU specification. However, since SRU
supports CQL only, we don't expect XCQL to be widely used.
Furthermore, CQL has the advantage over XCQL that it is
easy to read.
</listitem>
</itemizedlist>
</para>
- <sect3 id="tools.cql.parsing"><title>CQL parsing</title>
+ <sect3 id="cql.parsing"><title>CQL parsing</title>
<para>
A CQL parser is represented by the <literal>CQL_parser</literal>
handle. Its contents should be considered &yaz; internal (private).
</para>
</sect3>
- <sect3 id="tools.cql.tree"><title>CQL tree</title>
+ <sect3 id="cql.tree"><title>CQL tree</title>
<para>
The the query string is valid, the CQL parser
generates a tree representing the structure of the
<synopsis>
#define CQL_NODE_ST 1
#define CQL_NODE_BOOL 2
-#define CQL_NODE_MOD 3
struct cql_node {
int which;
union {
struct {
char *index;
+ char *index_uri;
char *term;
char *relation;
+ char *relation_uri;
struct cql_node *modifiers;
- struct cql_node *prefixes;
} st;
struct {
char *value;
struct cql_node *left;
struct cql_node *right;
struct cql_node *modifiers;
- struct cql_node *prefixes;
} boolean;
- struct {
- char *name;
- char *value;
- struct cql_node *next;
- } mod;
} u;
};
</synopsis>
- There are three kinds of nodes, search term (ST), boolean (BOOL),
- and modifier (MOD).
+ There are two node types: search term (ST) and boolean (BOOL).
+ A modifier is treated as a search term too.
</para>
<para>
The search term node has five members:
</listitem>
<listitem>
<para>
+ <literal>index_uri</literal>: index URi for search term
+ or NULL if none could be resolved for the index.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
<literal>term</literal>: the search term itself.
</para>
</listitem>
</listitem>
<listitem>
<para>
- <literal>modifiers</literal>: relation modifiers for search
- term. The <literal>modifiers</literal> is a simple linked
- list (NULL for last entry). Each relation modifier node
- is of type <literal>MOD</literal>.
+ <literal>relation_uri</literal>: relation URI for search term.
</para>
</listitem>
<listitem>
<para>
- <literal>prefixes</literal>: index prefixes for search
- term. The <literal>prefixes</literal> is a simple linked
- list (NULL for last entry). Each prefix node
- is of type <literal>MOD</literal>.
+ <literal>modifiers</literal>: relation modifiers for search
+ term. The <literal>modifiers</literal> list itself of cql_nodes
+ each of type <literal>ST</literal>.
</para>
</listitem>
</itemizedlist>
<literal>modifiers</literal>: proximity arguments.
</para>
</listitem>
- <listitem>
- <para>
- <literal>prefixes</literal>: index prefixes.
- The <literal>prefixes</literal> is a simple linked
- list (NULL for last entry). Each prefix node
- is of type <literal>MOD</literal>.
- </para>
- </listitem>
- </itemizedlist>
- </para>
-
- <para>
- The modifier node is a "utility" node used for name-value pairs,
- such as prefixes, proximity arguements, etc.
- <itemizedlist>
- <listitem>
- <para>
- <literal>name</literal> name of mod node.
- </para>
- </listitem>
- <listitem>
- <para>
- <literal>value</literal> value of mod node.
- </para>
- </listitem>
- <listitem>
- <para>
- <literal>next</literal>: pointer to next node which is
- always a mod node (NULL for last entry).
- </para>
- </listitem>
</itemizedlist>
</para>
</sect3>
- <sect3 id="tools.cql.pqf"><title>CQL to PQF conversion</title>
+ <sect3 id="cql.to.pqf"><title>CQL to PQF conversion</title>
<para>
Conversion to PQF (and Z39.50 RPN) is tricky by the fact
that the resulting RPN depends on the Z39.50 target
capabilities (combinations of supported attributes).
- In addition, the CQL and SRW operates on index prefixes
+ In addition, the CQL and SRU operates on index prefixes
(URI or strings), whereas the RPN uses Object Identifiers
for attribute sets.
</para>
</para>
<para>
If conversion failed, <function>cql_transform_buf</function>
- returns a non-zero SRW error code; otherwise zero is returned
+ returns a non-zero SRU error code; otherwise zero is returned
(conversion successful). The meanings of the numeric error
- codes are listed in the SRW specifications at
- <ulink url="http://www.loc.gov/srw/diagnostic-list.html"/>
+ codes are listed in the SRU specifications at
+ <ulink url="&url.sru.diagnostics.list;"/>
</para>
<para>
If conversion fails, more information can be obtained by calling
error-code and sets the string-pointer at
<literal>*addinfop</literal> to point to a string containing
additional information about the error that occurred: for
- example, if the error code is 15 (``Illegal or unsupported index
+ example, if the error code is 15 (``Illegal or unsupported context
set''), the additional information is the name of the requested
- index set that was not recognised.
+ context set that was not recognised.
</para>
<para>
- The SRW error-codes may be translated into brief human-readable
+ The SRU error-codes may be translated into brief human-readable
error messages using
<synopsis>
const char *cql_strerror(int code);
open <literal>FILE</literal>.
</para>
</sect3>
- <sect3 id="tools.cql.map">
- <title>Specification of CQL to RPN mapping</title>
+ <sect3 id="cql.to.rpn">
+ <title>Specification of CQL to RPN mappings</title>
<para>
The file supplied to functions
<function>cql_transform_open_FILE</function>,
The following CQL patterns are recognized:
<variablelist>
<varlistentry><term>
- <literal>qualifier.</literal><replaceable>set</replaceable><literal>.</literal><replaceable>name</replaceable>
+ <literal>index.</literal><replaceable>set</replaceable><literal>.</literal><replaceable>name</replaceable>
</term>
<listitem>
<para>
- This pattern is invoked when a CQL qualifier, such as
+ This pattern is invoked when a CQL index, such as
dc.title is converted. <replaceable>set</replaceable>
- and <replaceable>name</replaceable> is the index set and qualifier
+ and <replaceable>name</replaceable> are the context set and index
name respectively.
Typically, the RPN specifies an equivalent use attribute.
</para>
<para>
- For terms not bound by a qualifier the pattern
- <literal>qualifier.srw.serverChoice</literal> is used.
- Here, the prefix <literal>srw</literal> is defined as
- <literal>http://www.loc.gov/zing/cql/srw-indexes/v1.0/</literal>.
+ For terms not bound by an index the pattern
+ <literal>index.cql.serverChoice</literal> is used.
+ Here, the prefix <literal>cql</literal> is defined as
+ <literal>http://www.loc.gov/zing/cql/cql-indexes/v1.0/</literal>.
If this pattern is not defined, the mapping will fail.
</para>
</listitem>
</varlistentry>
<varlistentry><term>
+ <literal>qualifier.</literal><replaceable>set</replaceable><literal>.</literal><replaceable>name</replaceable>
+ (DEPRECATED)
+ </term>
+ <listitem>
+ <para>
+ For backwards compatibility, this is recognised as a synonym of
+ <literal>index.</literal><replaceable>set</replaceable><literal>.</literal><replaceable>name</replaceable>
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry><term>
<literal>relation.</literal><replaceable>relation</replaceable>
</term>
<listitem>
</term>
<listitem>
<para>
- This specification defines a CQL index set for a given prefix.
+ This specification defines a CQL context set for a given prefix.
The value on the right hand side is the URI for the set -
<emphasis>not</emphasis> RPN. All prefixes used in
- qualifier patterns must be defined this way.
+ index patterns must be defined this way.
</para>
</listitem>
</varlistentry>
</variablelist>
</para>
- <example><title>CQL to RPN mapping file</title>
+ <example id="example.cql.to.rpn.mapping"><title>CQL to RPN mapping file</title>
<para>
- This simple file defines two index sets, three qualifiers and three
+ This simple file defines two context sets, three indexes and three
relations, a position pattern and a default structure.
</para>
<programlisting><![CDATA[
- set.srw = http://www.loc.gov/zing/cql/srw-indexes/v1.0/
+ set.cql = http://www.loc.gov/zing/cql/context-sets/cql/v1.1/
set.dc = http://www.loc.gov/zing/cql/dc-indexes/v1.0/
- qualifier.srw.serverChoice = 1=1016
- qualifier.dc.title = 1=4
- qualifier.dc.subject = 1=21
+ index.cql.serverChoice = 1=1016
+ index.dc.title = 1=4
+ index.dc.subject = 1=21
relation.< = 2=1
relation.eq = 2=3
<screen>
@attr 1=1016 @attr 2=3 @attr 4=1 @attr 3=3 @attr 6=1 "computer"
</screen>
- by rules <literal>qualifier.srw.serverChoice</literal>,
+ by rules <literal>index.cql.serverChoice</literal>,
<literal>relation.scr</literal>, <literal>structure.*</literal>,
<literal>position.any</literal>.
</para>
</screen>
</para>
</example>
+ <example id="example.cql.to.rpn.bathprofile">
+ <title>CQL to RPN using Bath Profile</title>
+ <para>
+ The file <filename>etc/pqf.properties</filename> has mappings from
+ the Bath Profile and Dublin Core to RPN.
+ If YAZ is installed as a package it's usually located
+ in <filename>/usr/share/yaz/etc</filename> and part of the
+ development package, such as <literal>libyaz-dev</literal>.
+ </para>
+ </example>
</sect3>
- <sect3 id="tools.cql.xcql"><title>CQL to XCQL conversion</title>
+ <sect3 id="cql.xcql"><title>CQL to XCQL conversion</title>
<para>
Conversion from CQL to XCQL is trivial and does not
require a mapping to be defined.
<para>
again, corresponding to the specific OIDs defined by the standard.
Refer to the
- <ulink url="http://lcweb.loc.gov/z3950/agency/defns/oids.html">
+ <ulink url="&url.z39.50.oids;">
Registry of Z39.50 Object Identifiers</ulink> for the
whole list.
</para>
release the associated memory again. For the structures describing the
Z39.50 PDUs and related structures, it is convenient to use the
memory-management system of the &odr; subsystem (see
- <link linkend="odr-use">Using ODR</link>). However, in some circumstances
+ <xref linkend="odr.use"/>). However, in some circumstances
where you might otherwise benefit from using a simple nibble memory
management system, it may be impractical to use
<function>odr_malloc()</function> and <function>odr_reset()</function>.
</para>
</sect1>
+
+ <sect1 id="tools.log"><title>Log</title>
+ <para>
+ &yaz; has evolved a fairly complex log system which should be useful both
+ for debugging &yaz; itself, debugging applications that use &yaz;, and for
+ production use of those applications.
+ </para>
+ <para>
+ The log functions are declared in header <filename>yaz/log.h</filename>
+ and implemented in <filename>src/log.c</filename>.
+ Due to name clash with syslog and some math utilities the logging
+ interface has been modified as of YAZ 2.0.29. The obsolete interface
+ is still available if in header file <filename>yaz/log.h</filename>.
+ The key points of the interface are:
+ </para>
+ <screen>
+ void yaz_log(int level, const char *fmt, ...)
+
+ void yaz_log_init(int level, const char *prefix, const char *name);
+ void yaz_log_init_file(const char *fname);
+ void yaz_log_init_level(int level);
+ void yaz_log_init_prefix(const char *prefix);
+ void yaz_log_time_format(const char *fmt);
+ void yaz_log_init_max_size(int mx);
+
+ int yaz_log_mask_str(const char *str);
+ int yaz_log_module_level(const char *name);
+ </screen>
+
+ <para>
+ The reason for the whole log module is the <function>yaz_log</function>
+ function. It takes a bitmask indicating the log levels, a
+ <literal>printf</literal>-like format string, and a variable number of
+ arguments to log.
+ </para>
+
+ <para>
+ The <literal>log level</literal> is a bit mask, that says on which level(s)
+ the log entry should be made, and optionally set some behaviour of the
+ logging. In the most simple cases, it can be one of <literal>YLOG_FATAL,
+ YLOG_DEBUG, YLOG_WARN, YLOG_LOG</literal>. Those can be combined with bits
+ that modify the way the log entry is written:<literal>YLOG_ERRNO,
+ YLOG_NOTIME, YLOG_FLUSH</literal>.
+ Most of the rest of the bits are deprecated, and should not be used. Use
+ the dynamic log levels instead.
+ </para>
+
+ <para>
+ Applications that use &yaz;, should not use the LOG_LOG for ordinary
+ messages, but should make use of the dynamic loglevel system. This consists
+ of two parts, defining the loglevel and checking it.
+ </para>
+
+ <para>
+ To define the log levels, the (main) program should pass a string to
+ <function>yaz_log_mask_str</function> to define which log levels are to be
+ logged. This string should be a comma-separated list of log level names,
+ and can contain both hard-coded names and dynamic ones. The log level
+ calculation starts with <literal>YLOG_DEFAULT_LEVEL</literal> and adds a bit
+ for each word it meets, unless the word starts with a '-', in which case it
+ clears the bit. If the string <literal>'none'</literal> is found,
+ all bits are cleared. Typically this string comes from the command-line,
+ often identified by <literal>-v</literal>. The
+ <function>yaz_log_mask_str</function> returns a log level that should be
+ passed to <function>yaz_log_init_level</function> for it to take effect.
+ </para>
+
+ <para>
+ Each module should check what log bits it should be used, by calling
+ <function>yaz_log_module_level</function> with a suitable name for the
+ module. The name is cleared from a preceding path and an extension, if any,
+ so it is quite possible to use <literal>__FILE__</literal> for it. If the
+ name has been passed to <function>yaz_log_mask_str</function>, the routine
+ returns a non-zero bitmask, which should then be used in consequent calls
+ to yaz_log. (It can also be tested, so as to avoid unnecessary calls to
+ yaz_log, in time-critical places, or when the log entry would take time
+ to construct.)
+ </para>
+
+ <para>
+ Yaz uses the following dynamic log levels:
+ <literal>server, session, request, requestdetail</literal> for the server
+ functionality.
+ <literal>zoom</literal> for the zoom client api.
+ <literal>ztest</literal> for the simple test server.
+ <literal>malloc, nmem, odr, eventl</literal> for internal debugging of yaz itself.
+ Of course, any program using yaz is welcome to define as many new ones, as
+ it needs.
+ </para>
+
+ <para>
+ By default the log is written to stderr, but this can be changed by a call
+ to <function>yaz_log_init_file</function> or
+ <function>yaz_log_init</function>. If the log is directed to a file, the
+ file size is checked at every write, and if it exceeds the limit given in
+ <function>yaz_log_init_max_size</function>, the log is rotated. The
+ rotation keeps one old version (with a <literal>.1</literal> appended to
+ the name). The size defaults to 1GB. Setting it to zero will disable the
+ rotation feature.
+ </para>
+
+ <screen>
+ A typical yaz-log looks like this
+ 13:23:14-23/11 yaz-ztest(1) [session] Starting session from tcp:127.0.0.1 (pid=30968)
+ 13:23:14-23/11 yaz-ztest(1) [request] Init from 'YAZ' (81) (ver 2.0.28) OK
+ 13:23:17-23/11 yaz-ztest(1) [request] Search Z: @attrset Bib-1 foo OK:7 hits
+ 13:23:22-23/11 yaz-ztest(1) [request] Present: [1] 2+2 OK 2 records returned
+ 13:24:13-23/11 yaz-ztest(1) [request] Close OK
+ </screen>
+
+ <para>
+ The log entries start with a time stamp. This can be omitted by setting the
+ <literal>YLOG_NOTIME</literal> bit in the loglevel. This way automatic tests
+ can be hoped to produce identical log files, that are easy to diff. The
+ format of the time stamp can be set with
+ <function>yaz_log_time_format</function>, which takes a format string just
+ like <function>strftime</function>.
+ </para>
+
+ <para>
+ Next in a log line comes the prefix, often the name of the program. For
+ yaz-based servers, it can also contain the session number. Then
+ comes one or more logbits in square brackets, depending on the logging
+ level set by <function>yaz_log_init_level</function> and the loglevel
+ passed to <function>yaz_log_init_level</function>. Finally comes the format
+ string and additional values passed to <function>yaz_log</function>
+ </para>
+
+ <para>
+ The log level <literal>YLOG_LOGLVL</literal>, enabled by the string
+ <literal>loglevel</literal>, will log all the log-level affecting
+ operations. This can come in handy if you need to know what other log
+ levels would be useful. Grep the logfile for <literal>[loglevel]</literal>.
+ </para>
+
+ <para>
+ The log system is almost independent of the rest of &yaz;, the only
+ important dependence is of <filename>nmem</filename>, and that only for
+ using the semaphore definition there.
+ </para>
+
+ <para>
+ The dynamic log levels and log rotation were introduced in &yaz; 2.0.28. At
+ the same time, the log bit names were changed from
+ <literal>LOG_something</literal> to <literal>YLOG_something</literal>,
+ to avoid collision with <filename>syslog.h</filename>.
+ </para>
+
+ </sect1>
+
+ <sect1 id="marc"><title>MARC</title>
+
+ <para>
+ YAZ provides a fast utility that decodes MARC records and
+ encodes to a varity of output formats. The MARC records must
+ be encoded in ISO2709.
+ </para>
+ <synopsis><![CDATA[
+ #include <yaz/marcdisp.h>
+
+ /* create handler */
+ yaz_marc_t yaz_marc_create(void);
+ /* destroy */
+ void yaz_marc_destroy(yaz_marc_t mt);
+
+ /* set XML mode YAZ_MARC_LINE, YAZ_MARC_SIMPLEXML, ... */
+ void yaz_marc_xml(yaz_marc_t mt, int xmlmode);
+ #define YAZ_MARC_LINE 0
+ #define YAZ_MARC_SIMPLEXML 1
+ #define YAZ_MARC_OAIMARC 2
+ #define YAZ_MARC_MARCXML 3
+ #define YAZ_MARC_ISO2709 4
+ #define YAZ_MARC_XCHANGE 5
+
+ /* supply iconv handle for character set conversion .. */
+ void yaz_marc_iconv(yaz_marc_t mt, yaz_iconv_t cd);
+
+ /* set debug level, 0=none, 1=more, 2=even more, .. */
+ void yaz_marc_debug(yaz_marc_t mt, int level);
+
+ /* decode MARC in buf of size bsize. Returns >0 on success; <=0 on failure.
+ On success, result in *result with size *rsize. */
+ int yaz_marc_decode_buf (yaz_marc_t mt, const char *buf, int bsize,
+ char **result, int *rsize);
+
+ /* decode MARC in buf of size bsize. Returns >0 on success; <=0 on failure.
+ On success, result in WRBUF */
+ int yaz_marc_decode_wrbuf (yaz_marc_t mt, const char *buf,
+ int bsize, WRBUF wrbuf);
+]]>
+ </synopsis>
+ <para>
+ A MARC conversion handle must be created by using
+ <function>yaz_marc_create</function> and destroyed
+ by calling <function>yaz_marc_destroy</function>.
+ </para>
+ <para>
+ All other function operate on a <literal>yaz_marc_t</literal> handle.
+ The output is specified by a call to <function>yaz_marc_xml</function>.
+ The <literal>xmlmode</literal> must be one of
+ <variablelist>
+ <varlistentry>
+ <term>YAZ_MARC_LINE</term>
+ <listitem>
+ <para>
+ A simple line-by-line format suitable for display but not
+ recommend for further (machine) processing.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>YAZ_MARC_MARCXML</term>
+ <listitem>
+ <para>
+ The resulting record is converted to MARCXML.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>YAZ_MARC_ISO2709</term>
+ <listitem>
+ <para>
+ The resulting record is converted to ISO2709 (MARC).
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ <para>
+ The actual conversion functions are
+ <function>yaz_marc_decode_buf</function> and
+ <function>yaz_marc_decode_wrbuf</function> which decodes and encodes
+ a MARC record. The former function operates on simple buffers, the
+ stores the resulting record in a WRBUF handle (WRBUF is a simple string
+ type).
+ </para>
+ <example id="example.marc.display">
+ <title>Display of MARC record</title>
+ <para>
+ The followint program snippet illustrates how the MARC API may
+ be used to convert a MARC record to the line-by-line format:
+ <programlisting><![CDATA[
+ void print_marc(const char *marc_buf, int marc_buf_size)
+ {
+ char *result; /* for result buf */
+ int result_len; /* for size of result */
+ yaz_marc_t mt = yaz_marc_create();
+ yaz_marc_xml(mt, YAZ_MARC_LINE);
+ yaz_marc_decode_buf(mt, marc_buf, marc_buf_size,
+ &result, &result_len);
+ fwrite(result, result_len, 1, stdout);
+ yaz_marc_destroy(mt); /* note that result is now freed... */
+ }
+]]>
+ </programlisting>
+ </para>
+ </example>
+ </sect1>
+
</chapter>
<!-- Keep this comment at the end of the file
projects / yaz-moved-to-github.git / blobdiff