<chapter id="tools"><title>Supporting Tools</title>
-
+
<para>
In support of the service API - primarily the ASN module, which
provides the pro-grammatic interface to the Z39.50 APDUs, &yaz; contains
</para>
</note>
<para>
- The PQF is defined by the pquery module in the YAZ library.
+ The PQF is defined by the pquery module in the YAZ library.
There are two sets of function that have similar behavior. First
set operates on a PQF parser handle, second set doesn't. First set
set of functions are more flexible than the second set. Second set
<synopsis>
#include <yaz/pquery.h>
- YAZ_PQF_Parser yaz_pqf_create (void);
+ YAZ_PQF_Parser yaz_pqf_create(void);
- void yaz_pqf_destroy (YAZ_PQF_Parser p);
+ void yaz_pqf_destroy(YAZ_PQF_Parser p);
- Z_RPNQuery *yaz_pqf_parse (YAZ_PQF_Parser p, ODR o, const char *qbuf);
+ Z_RPNQuery *yaz_pqf_parse(YAZ_PQF_Parser p, ODR o, const char *qbuf);
- Z_AttributesPlusTerm *yaz_pqf_scan (YAZ_PQF_Parser p, ODR o,
+ Z_AttributesPlusTerm *yaz_pqf_scan(YAZ_PQF_Parser p, ODR o,
Odr_oid **attributeSetId, const char *qbuf);
-
- int yaz_pqf_error (YAZ_PQF_Parser p, const char **msg, size_t *off);
+ int yaz_pqf_error(YAZ_PQF_Parser p, const char **msg, size_t *off);
</synopsis>
<para>
A PQF parser is created and destructed by functions
a Z39.50 RPN Query is returned which is created using ODR stream
<literal>o</literal>. If parsing failed, a NULL pointer is
returned.
- Function <function>yaz_pqf_scan</function> takes a scan query in
+ Function <function>yaz_pqf_scan</function> takes a scan query in
<literal>qbuf</literal>. If parsing was successful, the function
returns attributes plus term pointer and modifies
<literal>attributeSetId</literal> to hold attribute set for the
<synopsis>
#include <yaz/pquery.h>
- Z_RPNQuery *p_query_rpn (ODR o, oid_proto proto, const char *qbuf);
+ Z_RPNQuery *p_query_rpn(ODR o, oid_proto proto, const char *qbuf);
- Z_AttributesPlusTerm *p_query_scan (ODR o, oid_proto proto,
+ Z_AttributesPlusTerm *p_query_scan(ODR o, oid_proto proto,
Odr_oid **attributeSetP, const char *qbuf);
- int p_query_attset (const char *arg);
+ int p_query_attset(const char *arg);
</synopsis>
<para>
The function <function>p_query_rpn()</function> takes as arguments an
<para>
If the parse went well, <function>p_query_rpn()</function> returns a
pointer to a <literal>Z_RPNQuery</literal> structure which can be
- placed directly into a <literal>Z_SearchRequest</literal>.
+ placed directly into a <literal>Z_SearchRequest</literal>.
If parsing failed, due to syntax error, a NULL pointer is returned.
</para>
<para>
</para>
<para>
- The @attr operator is followed by an attribute specification
+ The @attr operator is followed by an attribute specification
(<literal>attr-spec</literal> above). The specification consists
of an optional attribute set, an attribute type-value pair and
a sub-query. The attribute type-value pair is packed in one string:
is used. This is the only encoding allowed in both versions 2 and 3
of the Z39.50 standard.
</para>
-
+
<sect3 id="PQF-prox">
<title>Using Proximity Operators with PQF</title>
<note>
<para>
The proximity operator <literal>@prox</literal> is a special
and more restrictive version of the conjunction operator
- <literal>@and</literal>. Its semantics are described in
+ <literal>@and</literal>. Its semantics are described in
section 3.7.2 (Proximity) of Z39.50 the standard itself, which
can be read on-line at
<ulink url="&url.z39.50.proximity;"/>
<para>
<screen>
@or @and bob dylan @set Result-1
-
+
@attr 4=1 @and @attr 1=1 "bob dylan" @attr 1=4 "slow train coming"
-
+
@and @attr 2=4 @attr gils 1=2038 -114 @attr 2=2 @attr gils 1=2039 -109
</screen>
<note>
-- Proximity operator
</screen>
-
+
<example id="example.ccl.queries">
<title>CCL queries</title>
<para>
The following queries are all valid:
</para>
-
+
<screen>
dylan
-
+
"bob dylan"
-
+
dylan or zimmerman
-
+
set=1
-
+
(dylan and bob) or set=1
-
+
+ righttrunc?
+
+ "notrunc?"
+
+ singlechar#mask
+
</screen>
<para>
Assuming that the qualifiers <literal>ti</literal>,
<literal>au</literal>
and <literal>date</literal> are defined we may use:
</para>
-
+
<screen>
ti=self portrait
-
+
au=(bob dylan and slow train coming)
date>1980 and (ti=((self portrait)))
-
+
</screen>
</example>
-
+
</sect3>
<sect3 id="ccl.qualifiers">
<title>CCL Qualifiers</title>
-
+
<para>
Qualifiers are used to direct the search to a particular searchable
index, such as title (ti) and author indexes (au). The CCL standard
<para>
A qualifier specification is of the form:
</para>
-
+
<para>
- <replaceable>qualifier-name</replaceable>
+ <replaceable>qualifier-name</replaceable>
[<replaceable>attributeset</replaceable><literal>,</literal>]<replaceable>type</replaceable><literal>=</literal><replaceable>val</replaceable>
- [<replaceable>attributeset</replaceable><literal>,</literal>]<replaceable>type</replaceable><literal>=</literal><replaceable>val</replaceable> ...
+ [<replaceable>attributeset</replaceable><literal>,</literal>]<replaceable>type</replaceable><literal>=</literal><replaceable>val</replaceable> ...
</para>
-
+
<para>
where <replaceable>qualifier-name</replaceable> is the name of the
qualifier to be used (eg. <literal>ti</literal>),
<replaceable>val</replaceable> is attribute value.
The <replaceable>type</replaceable> can be specified as an
integer or as it be specified either as a single-letter:
- <literal>u</literal> for use,
+ <literal>u</literal> for use,
<literal>r</literal> for relation,<literal>p</literal> for position,
<literal>s</literal> for structure,<literal>t</literal> for truncation
or <literal>c</literal> for completeness.
<ulink url="&url.z39.50.attset.bib1;">list of Bib-1 attributes</ulink>
</para>
<para>
- It is also possible to specify non-numeric attribute values,
+ It is also possible to specify non-numeric attribute values,
which are used in combination with certain types.
The special combinations are:
-
+
<table id="ccl.special.attribute.combos">
<title>Special attribute combos</title>
<tgroup cols="2">
This does not set the structure at all.
</entry>
</row>
-
+
<row><entry><literal>s=ol</literal></entry><entry>
Each token in the term is ORed. (or-list).
This does not set the structure at all.
</entry>
</row>
-
+
<row><entry><literal>s=ag</literal></entry><entry>
Tokens that appears as phrases (with blank in them) gets
structure phrase attached. Tokens that appers as words
This sets Bib-1 relation attribute accordingly (relation
ordered). A query construct is only treated as a range if
dash is used and that is surrounded by white-space. So
- <literal>-1980</literal> is treated as term
+ <literal>-1980</literal> is treated as term
<literal>"-1980"</literal> not <literal><= 1980</literal>.
If <literal>- 1980</literal> is used, however, that is
treated as a range.
</entry>
</row>
-
+
<row><entry><literal>r=r</literal></entry><entry>
Similar to <literal>r=o</literal> but assumes that terms
are non-negative (not prefixed with <literal>-</literal>).
<literal>r=r</literal> is available in YAZ 2.0.24 or later.
</entry>
</row>
-
+
<row><entry><literal>t=l</literal></entry><entry>
Allows term to be left-truncated.
If term is of the form <literal>?x</literal>, the resulting
Type-1 term is <literal>x</literal> and truncation is left.
</entry>
</row>
-
+
<row><entry><literal>t=r</literal></entry><entry>
Allows term to be right-truncated.
If term is of the form <literal>x?</literal>, the resulting
Type-1 term is <literal>x</literal> and truncation is right.
</entry>
</row>
-
+
<row><entry><literal>t=n</literal></entry><entry>
If term is does not include <literal>?</literal>, the
truncation attribute is set to none (100).
</entry>
</row>
-
+
<row><entry><literal>t=b</literal></entry><entry>
Allows term to be both left&right truncated.
If term is of the form <literal>?x?</literal>, the
<para>
Consider the following definition:
</para>
-
+
<screen>
ti u=4 s=1
au u=1 s=1
date u=30 r=o
</screen>
<para>
- <literal>ti</literal> and <literal>au</literal> both set
+ <literal>ti</literal> and <literal>au</literal> both set
structure attribute to phrase (s=1).
<literal>ti</literal>
sets the use-attribute to 4. <literal>au</literal> sets the
</para>
<para>
You can combine attributes. To Search for "ranked title" you
- can do
+ can do
<screen>
ti,ranked=knuth computer
</screen>
A qualifier alias is of the form:
</para>
<para>
- <replaceable>q</replaceable>
+ <replaceable>q</replaceable>
<replaceable>q1</replaceable> <replaceable>q2</replaceable> ..
</para>
<para>
which declares <replaceable>q</replaceable> to
- be an alias for <replaceable>q1</replaceable>,
+ be an alias for <replaceable>q1</replaceable>,
<replaceable>q2</replaceable>... such that the CCL
query <replaceable>q=x</replaceable> is equivalent to
<replaceable>q1=x or q2=x or ...</replaceable>.
<entry><literal>?</literal></entry>
</row>
<row>
+ <entry>mask</entry>
+ <entry>Masking character. Requires YAZ 4.2.58 or later</entry>
+ <entry><literal>#</literal></entry>
+ </row>
+ <row>
<entry>field</entry>
<entry>Specifies how multiple fields are to be
combined. There are two modes: <literal>or</literal>:
To parse a simple string with a FIND query use the function
</para>
<screen>
-struct ccl_rpn_node *ccl_find_str (CCL_bibset bibset, const char *str,
- int *error, int *pos);
+struct ccl_rpn_node *ccl_find_str(CCL_bibset bibset, const char *str,
+ int *error, int *pos);
</screen>
<para>
which takes the CCL profile (<literal>bibset</literal>) and query
</para>
<tip>
<para>
- If you are new to CQL, read the
+ If you are new to CQL, read the
<ulink url="&url.cql.intro;">Gentle Introduction</ulink>.
</para>
</tip>
uses a <literal>FILE</literal> handle which is opened for reading.
</para>
</sect3>
-
+
<sect3 id="cql.tree"><title>CQL tree</title>
<para>
The the query string is valid, the CQL parser
a pointer to the root node of the resulting tree.
</para>
<para>
- Each node in a CQL tree is represented by a
+ Each node in a CQL tree is represented by a
<literal>struct cql_node</literal>.
It is defined as follows:
<synopsis>
#define CQL_NODE_ST 1
#define CQL_NODE_BOOL 2
+#define CQL_NODE_SORT 3
struct cql_node {
int which;
union {
struct cql_node *right;
struct cql_node *modifiers;
} boolean;
+ struct {
+ char *index;
+ struct cql_node *next;
+ struct cql_node *modifiers;
+ struct cql_node *search;
+ } sort;
} u;
};
</synopsis>
- There are two node types: search term (ST) and boolean (BOOL).
+ There are three node types: search term (ST), boolean (BOOL)
+ and sortby (SORT).
A modifier is treated as a search term too.
</para>
<para>
</para>
<para>
- The boolean node represents both <literal>and</literal>,
- <literal>or</literal>, not as well as
+ The boolean node represents <literal>and</literal>,
+ <literal>or</literal>, <literal>not</literal> +
proximity.
<itemizedlist>
<listitem>
</itemizedlist>
</para>
+ <para>
+ The sort node represents both the SORTBY clause.
+ </para>
+
</sect3>
<sect3 id="cql.to.pqf"><title>CQL to PQF conversion</title>
<para>
Conversion to PQF (and Z39.50 RPN) is tricky by the fact
that the resulting RPN depends on the Z39.50 target
- capabilities (combinations of supported attributes).
+ capabilities (combinations of supported attributes).
In addition, the CQL and SRU operates on index prefixes
(URI or strings), whereas the RPN uses Object Identifiers
for attribute sets.
either an already open FILE or from a filename respectively.
</para>
<para>
- The handle is destroyed by <function>cql_transform_close</function>
+ The handle is destroyed by <function>cql_transform_close</function>
in which case no further reference of the handle is allowed.
</para>
<para>
int cql_transform_buf(cql_transform_t ct,
struct cql_node *cn, char *out, int max);
</synopsis>
- This function converts the CQL tree <literal>cn</literal>
+ This function converts the CQL tree <literal>cn</literal>
using handle <literal>ct</literal>.
For the resulting PQF, you supply a buffer <literal>out</literal>
which must be able to hold at at least <literal>max</literal>
<sect3 id="cql.to.rpn">
<title>Specification of CQL to RPN mappings</title>
<para>
- The file supplied to functions
+ The file supplied to functions
<function>cql_transform_open_FILE</function>,
<function>cql_transform_open_fname</function> follows
a structure found in many Unix utilities.
</term>
<listitem>
<para>
- This pattern is invoked when a CQL index, such as
+ This pattern is invoked when a CQL index, such as
dc.title is converted. <replaceable>set</replaceable>
and <replaceable>name</replaceable> are the context set and index
name respectively.
If this pattern is not defined, the mapping will fail.
</para>
<para>
- The pattern,
+ The pattern,
<literal>index.</literal><replaceable>set</replaceable><literal>.*</literal>
is used when no other index pattern is matched.
</para>
<para>
This pattern specifies how a CQL structure is mapped to RPN.
Note that this CQL pattern is somewhat to similar to
- CQL pattern <literal>relation</literal>.
+ CQL pattern <literal>relation</literal>.
The <replaceable>type</replaceable> is a CQL relation.
</para>
<para>
<listitem>
<para>
This specification defines a CQL context set for a given prefix.
- The value on the right hand side is the URI for the set -
+ The value on the right hand side is the URI for the set -
<emphasis>not</emphasis> RPN. All prefixes used in
index patterns must be defined this way.
</para>
index.cql.serverChoice = 1=1016
index.dc.title = 1=4
index.dc.subject = 1=21
-
+
relation.< = 2=1
relation.eq = 2=3
relation.scr = 2=3
containing XCQL).
<synopsis>
int cql_to_xml_buf(struct cql_node *cn, char *out, int max);
-void cql_to_xml(struct cql_node *cn,
+void cql_to_xml(struct cql_node *cn,
void (*pr)(const char *buf, void *client_data),
void *client_data);
void cql_to_xml_stdio(struct cql_node *cn, FILE *f);
a file.
</para>
</sect3>
+ <sect3 id="rpn.to.cql">
+ <title>PQF to CQL conversion</title>
+ <para>
+ Conversion from PQF to CQL is offered by the two functions shown
+ below. The former uses a generic stream for result. The latter
+ puts result in a WRBUF (string container).
+ <synopsis>
+#include <yaz/rpn2cql.h>
+
+int cql_transform_rpn2cql_stream(cql_transform_t ct,
+ void (*pr)(const char *buf, void *client_data),
+ void *client_data,
+ Z_RPNQuery *q);
+
+int cql_transform_rpn2cql_wrbuf(cql_transform_t ct,
+ WRBUF w,
+ Z_RPNQuery *q);
+ </synopsis>
+ The configuration is the same as used in CQL to PQF conversions.
+ </para>
+ </sect3>
</sect2>
</sect1>
<sect1 id="tools.oid"><title>Object Identifiers</title>
<para>
The basic YAZ representation of an OID is an array of integers,
- terminated with the value -1. This integer is of type
+ terminated with the value -1. This integer is of type
<literal>Odr_oid</literal>.
</para>
<para>
Odr_oid *odr_oiddup(ODR odr, const Odr_oid *o);
</screen>
</para>
-
+
<para>
OIDs can be compared with <function>oid_oidcmp</function> which returns
zero if the two OIDs provided are identical; non-zero otherwise.
</para>
-
+
<sect2 id="tools.oid.database"><title>OID database</title>
<para>
From YAZ version 3 and later, the oident system has been replaced
convert from string to OID or other way around.
</para>
<para>
- Unfortunately, whenever we supply a string we must also specify the
+ Unfortunately, whenever we supply a string we must also specify the
<emphasis>OID class</emphasis>. The class is necessary because some
strings correspond to multiple OIDs. An example of such a string is
- <literal>Bib-1</literal> which may either be an attribute-set
+ <literal>Bib-1</literal> which may either be an attribute-set
or a diagnostic-set.
</para>
<para>
- Applications using the YAZ database should include
+ Applications using the YAZ database should include
<filename>yaz/oid_db.h</filename>.
</para>
<para>
A YAZ database handle is of type <literal>yaz_oid_db_t</literal>.
Actually that's a pointer. You need not think deal with that.
YAZ has a built-in database which can be considered "constant" for
- most purposes.
+ most purposes.
We can get hold that by using function <function>yaz_oid_std</function>.
</para>
<para>
We can create an OID for the Bib-1 attribute set on the ODR stream
odr with:
<screen>
- Odr_oid *bib1 =
+ Odr_oid *bib1 =
yaz_string_to_oid_odr(yaz_oid_std(), CLASS_ATTSET, "Bib-1", odr);
</screen>
This is more complex than using <function>odr_getoidbystr</function>.
</sect2>
<sect2 id="tools.oid.std"><title>Standard OIDs</title>
-
+
<para>
All the object identifers in the standard OID database as returned
by <function>yaz_oid_std</function> can referenced directly in a
<emphasis>not</emphasis> call <function>nmem_init</function> or
<function>nmem_exit</function> unless you're absolute sure what
you're doing. Note that in previous &yaz; versions you'd have to call
- <function>nmem_init</function> yourself.
+ <function>nmem_init</function> yourself.
</para>
</sect1>
<sect1 id="tools.log"><title>Log</title>
<para>
- &yaz; has evolved a fairly complex log system which should be useful both
+ &yaz; has evolved a fairly complex log system which should be useful both
for debugging &yaz; itself, debugging applications that use &yaz;, and for
- production use of those applications.
+ production use of those applications.
</para>
<para>
The log functions are declared in header <filename>yaz/log.h</filename>
logged. This string should be a comma-separated list of log level names,
and can contain both hard-coded names and dynamic ones. The log level
calculation starts with <literal>YLOG_DEFAULT_LEVEL</literal> and adds a bit
- for each word it meets, unless the word starts with a '-', in which case it
+ for each word it meets, unless the word starts with a '-', in which case it
clears the bit. If the string <literal>'none'</literal> is found,
all bits are cleared. Typically this string comes from the command-line,
often identified by <literal>-v</literal>. The
</para>
<para>
- Each module should check what log bits it should be used, by calling
+ Each module should check what log bits it should be used, by calling
<function>yaz_log_module_level</function> with a suitable name for the
module. The name is cleared from a preceding path and an extension, if any,
so it is quite possible to use <literal>__FILE__</literal> for it. If the
name has been passed to <function>yaz_log_mask_str</function>, the routine
returns a non-zero bitmask, which should then be used in consequent calls
to yaz_log. (It can also be tested, so as to avoid unnecessary calls to
- yaz_log, in time-critical places, or when the log entry would take time
- to construct.)
+ yaz_log, in time-critical places, or when the log entry would take time
+ to construct.)
</para>
<para>
<para>
The log system is almost independent of the rest of &yaz;, the only
important dependence is of <filename>nmem</filename>, and that only for
- using the semaphore definition there.
+ using the semaphore definition there.
</para>
<para>
The dynamic log levels and log rotation were introduced in &yaz; 2.0.28. At
the same time, the log bit names were changed from
- <literal>LOG_something</literal> to <literal>YLOG_something</literal>,
+ <literal>LOG_something</literal> to <literal>YLOG_something</literal>,
to avoid collision with <filename>syslog.h</filename>.
</para>
</sect1>
-
+
<sect1 id="marc"><title>MARC</title>
-
+
<para>
YAZ provides a fast utility for working with MARC records.
Early versions of the MARC utility only allowed decoding of ISO2709.
</variablelist>
</para>
<para>
- The actual conversion functions are
+ The actual conversion functions are
<function>yaz_marc_decode_buf</function> and
<function>yaz_marc_decode_wrbuf</function> which decodes and encodes
a MARC record. The former function operates on simple buffers, the
<literal>collection</literal>.
</para></listitem>
<listitem><para>
- The leader is encoded as element <literal>l</literal> with the
+ The leader is encoded as element <literal>l</literal> with the
leader content as its (text) value.
</para></listitem>
<listitem><para>
YAZ version 2.1.20 or later includes a Retrieval facility tool
which allows a SRU/Z39.50 to describe itself and perform record
conversions. The idea is the following:
-
+
<itemizedlist>
<listitem>
<para>
<sect2 id="tools.retrieval.format">
<title>Retrieval XML format</title>
<para>
- All elements should be covered by namespace
+ All elements should be covered by namespace
<literal>http://indexdata.com/yaz</literal> .
The root element node must be <literal>retrievalinfo</literal>.
</para>
<para>
The <literal>retrievalinfo</literal> must include one or
- more <literal>retrieval</literal> elements. Each
+ more <literal>retrieval</literal> elements. Each
<literal>retrieval</literal> defines specific combination of
syntax, name and identifier supported by this retrieval service.
</para>
</variablelist>
</para>
<para>
- The <literal>retrieval</literal> may include one
+ The <literal>retrieval</literal> may include one
<literal>backend</literal> element. If a <literal>backend</literal>
element is given, it specifies how the records are retrieved by
some backend and how the records are converted from the backend to
<varlistentry><term><literal>marc</literal></term>
<listitem>
<para>
- The <literal>marc</literal> element specifies a conversion
- to - and from ISO2709 encoded MARC and
+ The <literal>marc</literal> element specifies a conversion
+ to - and from ISO2709 encoded MARC and
<ulink url="&url.marcxml;">&acro.marcxml;</ulink>/MarcXchange.
The following attributes may be specified:
<varlistentry><term><literal>inputformat</literal> (REQUIRED)</term>
<listitem>
<para>
- Format of input. Supported values are
+ Format of input. Supported values are
<literal>marc</literal> (for ISO2709); and <literal>xml</literal>
for MARCXML/MarcXchange.
</para>
<varlistentry><term><literal>outputformat</literal> (REQUIRED)</term>
<listitem>
<para>
- Format of output. Supported values are
- <literal>line</literal> (MARC line format);
+ Format of output. Supported values are
+ <literal>line</literal> (MARC line format);
<literal>marcxml</literal> (for MARCXML),
<literal>marc</literal> (ISO2709),
<literal>marcxhcange</literal> (for MarcXchange).
<para>
It should be easy to use the retrieval systems from applications. Refer
to the headers
- <filename>yaz/retrieval.h</filename> and
+ <filename>yaz/retrieval.h</filename> and
<filename>yaz/record_conv.h</filename>.
</para>
</sect2>
</sect1>
</chapter>
-
+
<!-- Keep this comment at the end of the file
Local variables:
mode: sgml
projects / yaz-moved-to-github.git / blobdiff