X-Git-Url: http://git.indexdata.com/?a=blobdiff_plain;f=doc%2Ftools.xml;h=9385c8a7df88f357bc6be0ff284df1ccd3b26869;hb=6f5c63a8b759040d31028a4f1437a9cbc7a21fd6;hp=3809efa1cbdc2dc554e465f148e5381ef816593e;hpb=087aa6548dbee815eed2e6bad4c5b105c35391cb;p=yaz-moved-to-github.git diff --git a/doc/tools.xml b/doc/tools.xml index 3809efa..9385c8a 100644 --- a/doc/tools.xml +++ b/doc/tools.xml @@ -1,4 +1,4 @@ - + Supporting Tools @@ -16,7 +16,7 @@ Z_RPNQuery structure. Some programmers will prefer to construct the query manually, perhaps using odr_malloc() to simplify memory management. - The &yaz; distribution includes two separate, query-generating tools + The &yaz; distribution includes three separate, query-generating tools that may be of use to you. @@ -131,7 +131,7 @@ top-set ::= [ '@attrset' string ] - query-struct ::= attr-spec | simple | complex | '@term' term-type + query-struct ::= attr-spec | simple | complex | '@term' term-type query attr-spec ::= '@attr' [ string ] string query-struct @@ -173,11 +173,15 @@ The @attr operator is followed by an attribute specification (attr-spec above). The specification consists - of optional an attribute set, an attribute type-value pair and - a sub query. The attribute type-value pair is packed in one string: - an attribute type, a dash, followed by an attribute value. + of an optional attribute set, an attribute type-value pair and + a sub-query. The attribute type-value pair is packed in one string: + an attribute type, an equals sign, and an attribute value, like this: + @attr 1=1003. The type is always an integer but the value may be either an integer or a string (if it doesn't start with a digit character). + A string attribute-value is encoded as a Type-1 ``complex'' + attribute with the list of values containing the single string + specified, and including no semantic indicators. @@ -297,118 +301,127 @@ PQF queries - Queries using simple terms. - - dylan - "bob dylan" - - - Boolean operators. - - @or "dylan" "zimmerman" - @and @or dylan zimmerman when - @and when @or dylan zimmerman - - - - Reference to result sets. - - @set Result-1 - @and @set seta setb - - - - Attributes for terms. - - @attr 1=4 computer - @attr 1=4 @attr 4=1 "self portrait" - @attr exp1 @attr 1=1 CategoryList - @attr gils 1=2008 Copenhagen - @attr 1=/book/title computer - - - - Proximity. - - @prox 0 3 1 2 k 2 dylan zimmerman - - - Here the parameters 0, 3, 1, 2, k and 2 represent exclusion, - distance, ordered, relation, which-code and unit-code, in that - order. So: - - - exclusion = 0: the proximity condition must hold - - - distance = 3: the terms must be three units apart - - - ordered = 1: they must occur in the order they are specified - - - relation = 2: lessThanOrEqual (to the distance of 3 units) - - - which-code is ``known'', so the standard unit-codes are used - - - unit-code = 2: word. - - - So the whole proximity query means that the words - dylan and zimmerman must - both occur in the record, in that order, differing in position - by three or fewer words (i.e. with two or fewer words between - them.) The query would find ``Bob Dylan, aka. Robert - Zimmerman'', but not ``Bob Dylan, born as Robert Zimmerman'' - since the distance in this case is four. - - - - Specifying term type. - - @term string "a UTF-8 string, maybe?" - - - Mixed queries - - @or @and bob dylan @set Result-1 - - @attr 4=1 @and @attr 1=1 "bob dylan" @attr 1=4 "slow train coming" - - @and @attr 2=4 @attr gils 1=2038 -114 @attr 2=2 @attr gils 1=2039 -109 + PQF queries using simple terms + + + dylan + "bob dylan" + + + + PQF boolean operators + + + @or "dylan" "zimmerman" + @and @or dylan zimmerman when + @and when @or dylan zimmerman + + + + PQF references to result sets + + + @set Result-1 + @and @set seta setb + + + + Attributes for terms + + + @attr 1=4 computer + @attr 1=4 @attr 4=1 "self portrait" + @attrset exp1 @attr 1=1 CategoryList + @attr gils 1=2008 Copenhagen + @attr 1=/book/title computer + + + + PQF Proximity queries + + + @prox 0 3 1 2 k 2 dylan zimmerman + + + Here the parameters 0, 3, 1, 2, k and 2 represent exclusion, + distance, ordered, relation, which-code and unit-code, in that + order. So: + + + exclusion = 0: the proximity condition must hold + + + distance = 3: the terms must be three units apart + + + ordered = 1: they must occur in the order they are specified + + + relation = 2: lessThanOrEqual (to the distance of 3 units) + + + which-code is ``known'', so the standard unit-codes are used + + + unit-code = 2: word. + + + So the whole proximity query means that the words + dylan and zimmerman must + both occur in the record, in that order, differing in position + by three or fewer words (i.e. with two or fewer words between + them.) The query would find ``Bob Dylan, aka. Robert + Zimmerman'', but not ``Bob Dylan, born as Robert Zimmerman'' + since the distance in this case is four. + + + + PQF specification of search term + + + @term string "a UTF-8 string, maybe?" + + + + PQF mixed queries + + + @or @and bob dylan @set Result-1 + + @attr 4=1 @and @attr 1=1 "bob dylan" @attr 1=4 "slow train coming" + + @and @attr 2=4 @attr gils 1=2038 -114 @attr 2=2 @attr gils 1=2039 -109 - + - The last of these examples is a spatial search: in - the GILS attribute set, - access point - 2038 indicates West Bounding Coordinate and - 2030 indicates East Bounding Coordinate, - so the query is for areas extending from -114 degrees - to no more than -109 degrees. + access point + 2038 indicates West Bounding Coordinate and + 2030 indicates East Bounding Coordinate, + so the query is for areas extending from -114 degrees + to no more than -109 degrees. - - + + + - Common Command Language + CCL Not all users enjoy typing in prefix query structures and numerical attribute values, even in a minimalistic test client. In the library - world, the more intuitive Common Command Language (or ISO 8777) has - enjoyed some popularity - especially before the widespread + world, the more intuitive Common Command Language - CCL (ISO 8777) + has enjoyed some popularity - especially before the widespread availability of graphical interfaces. It is still useful in applications where you for some reason or other need to provide a symbolic language for expressing boolean query structures. - The EUROPAGATE - research project working under the Libraries programme + The EUROPAGATE research project working under the Libraries programme of the European Commission's DG XIII has, amongst other useful tools, implemented a general-purpose CCL parser which produces an output structure that can be trivially converted to the internal RPN @@ -510,73 +523,349 @@ suggest a few short-hand notations. You can customize the CCL parser to support a particular set of qualifiers to reflect the current target profile. Traditionally, a qualifier would map to a particular - use-attribute within the BIB-1 attribute set. However, you could also - define qualifiers that would set, for example, the - structure-attribute. + use-attribute within the BIB-1 attribute set. It is also + possible to set other attributes, such as the structure + attribute. A CCL profile is a set of predefined CCL qualifiers that may be - read from a file. + read from a file or set in the CCL API. The YAZ client reads its CCL qualifiers from a file named - default.bib. Each line in the file has the form: - - - - qualifier-name - [attributeset,]type=val - [attributeset,]type=val ... - - - - where qualifier-name is the name of the - qualifier to be used (eg. ti), - type is attribute type in the attribute - set (Bib-1 is used if no attribute set is given) and - val is attribute value. - The type can be specified as an - integer or as it be specified either as a single-letter: - u for use, - r for relation,p for position, - s for structure,t for truncation - or c for completeness. - The attributes for the special qualifier name term - are used when no CCL qualifier is given in a query. + default.bib. There are four types of + lines in a CCL profile: qualifier specification, + qualifier alias, comments and directives. - - CCL profile + Qualifier specification - Consider the following definition: + A qualifier specification is of the form: - - ti u=4 s=1 - au u=1 s=1 - term s=105 - ranked r=102 - - Three qualifiers are defined, ti, - au and ranked. - ti and au both set - structure attribute to phrase (s=1). - ti - sets the use-attribute to 4. au sets the - use-attribute to 1. - When no qualifiers are used in the query the structure-attribute is - set to free-form-text (105). + qualifier-name + [attributeset,]type=val + [attributeset,]type=val ... + + + + where qualifier-name is the name of the + qualifier to be used (eg. ti), + type is attribute type in the attribute + set (Bib-1 is used if no attribute set is given) and + val is attribute value. + The type can be specified as an + integer or as it be specified either as a single-letter: + u for use, + r for relation,p for position, + s for structure,t for truncation + or c for completeness. + The attributes for the special qualifier name term + are used when no CCL qualifier is given in a query. + Common Bib-1 attributes + + + + + + Type + Description + + + + + u=value + + Use attribute (1). Common use attributes are + 1 Personal-name, 4 Title, 7 ISBN, 8 ISSN, 30 Date, + 62 Subject, 1003 Author), 1016 Any. Specify value + as an integer. + + + + + r=value + + Relation attribute (2). Common values are + 1 <, 2 <=, 3 =, 4 >=, 5 >, 6 <>, + 100 phonetic, 101 stem, 102 relevance, 103 always matches. + + + + + p=value + + Position attribute (3). Values: 1 first in field, 2 + first in any subfield, 3 any position in field. + + + + + s=value + + Structure attribute (4). Values: 1 phrase, 2 word, + 3 key, 4 year, 5 date, 6 word list, 100 date (un), + 101 name (norm), 102 name (un), 103 structure, 104 urx, + 105 free-form-text, 106 document-text, 107 local-number, + 108 string, 109 numeric string. + + + + + t=value + + Truncation attribute (5). Values: 1 right, 2 left, + 3 left& right, 100 none, 101 process #, 102 regular-1, + 103 regular-2, 104 CCL. + + + + + c=value + + Completeness attribute (6). Values: 1 incomplete subfield, + 2 complete subfield, 3 complete field. + + + + + +
+ + + The complete list of Bib-1 attributes can be found + + here + . - You can combine attributes. To Search for "ranked title" you - can do + It is also possible to specify non-numeric attribute values, + which are used in combination with certain types. + The special combinations are: + + Special attribute combos + + + + + + Name + Description + + + + + s=pw + The structure is set to either word or phrase depending + on the number of tokens in a term (phrase-word). + + + + s=al + Each token in the term is ANDed. (and-list). + This does not set the structure at all. + + + + s=ol + Each token in the term is ORed. (or-list). + This does not set the structure at all. + + + + r=o + Allows ranges and the operators greather-than, less-than, ... + equals. + This sets Bib-1 relation attribute accordingly (relation + ordered). A query construct is only treated as a range if + dash is used and that is surrounded by white-space. So + -1980 is treated as term + "-1980" not <= 1980. + If - 1980 is used, however, that is + treated as a range. + + + + r=r + Similar to r=o but assumes that terms + are non-negative (not prefixed with -). + Thus, a dash will always be treated as a range. + The construct 1980-1990 is + treated as a range with r=r but as a + single term "1980-1990" with + r=o. The special attribute + r=r is available in YAZ 2.0.24 or later. + + + + t=l + Allows term to be left-truncated. + If term is of the form ?x, the resulting + Type-1 term is x and truncation is left. + + + + t=r + Allows term to be right-truncated. + If term is of the form x?, the resulting + Type-1 term is x and truncation is right. + + + + t=n + If term is does not include ?, the + truncation attribute is set to none (100). + + + + t=b + Allows term to be both left&right truncated. + If term is of the form ?x?, the + resulting term is x and trunctation is + set to both left&right. + + + + +
+ + CCL profile + + Consider the following definition: + + - ti,ranked=knuth computer - - which will use "relation is ranked", "use is title", "structure is - phrase". + ti u=4 s=1 + au u=1 s=1 + term s=105 + ranked r=102 + date u=30 r=o + + + ti and au both set + structure attribute to phrase (s=1). + ti + sets the use-attribute to 4. au sets the + use-attribute to 1. + When no qualifiers are used in the query the structure-attribute is + set to free-form-text (105) (rule for term). + The date sets the relation attribute to + the relation used in the CCL query and sets the use attribute + to 30 (Bib-1 Date). + + + You can combine attributes. To Search for "ranked title" you + can do + + ti,ranked=knuth computer + + which will set relation=ranked, use=title, structure=phrase. + + + Query + + date > 1980 + + is a valid query. But + + ti > 1980 + + is invalid. + + + + Qualifier alias + + A qualifier alias is of the form: - - + + q + q1 q2 .. + + + which declares q to + be an alias for q1, + q2... such that the CCL + query q=x is equivalent to + q1=x or q2=x or .... + + + + Comments + + Lines with white space or lines that begin with + character # are treated as comments. + + + + Directives + + Directive specifications takes the form + + @directive value + + CCL directives + + + + + + + Name + Description + Default + + + + + truncation + Truncation character + ? + + + field + Specifies how multiple fields are to be + combined. There are two modes: or: + multiple qualifier fields are ORed, + merge: attributes for the qualifier + fields are merged and assigned to one term. + + merge + + + case + Specificies if CCL operatores and qualifiers should be + compared with case sensitivity or not. Specify 0 for + case sensitive; 1 for case insensitive. + 0 + + + + and + Specifies token for CCL operator AND. + and + + + + or + Specifies token for CCL operator OR. + or + + + + not + Specifies token for CCL operator NOT. + not + + + + set + Specifies token for CCL operator SET. + set + + + +
+ CCL API @@ -760,34 +1049,28 @@ struct cql_node *cql_parser_result(CQL_parser cp); #define CQL_NODE_ST 1 #define CQL_NODE_BOOL 2 -#define CQL_NODE_MOD 3 struct cql_node { int which; union { struct { char *index; + char *index_uri; char *term; char *relation; + char *relation_uri; struct cql_node *modifiers; - struct cql_node *prefixes; } st; struct { char *value; struct cql_node *left; struct cql_node *right; struct cql_node *modifiers; - struct cql_node *prefixes; } boolean; - struct { - char *name; - char *value; - struct cql_node *next; - } mod; } u; }; - There are three kinds of nodes, search term (ST), boolean (BOOL), - and modifier (MOD). + There are two node types: search term (ST) and boolean (BOOL). + A modifier is treated as a search term too. The search term node has five members: @@ -798,6 +1081,10 @@ struct cql_node { If an index is unspecified for a search term, index will be NULL. + + index_uri: index URi for search term + or NULL if none could be resolved for the index. + @@ -811,18 +1098,14 @@ struct cql_node { - modifiers: relation modifiers for search - term. The modifiers is a simple linked - list (NULL for last entry). Each relation modifier node - is of type MOD. + relation_uri: relation URI for search term. - prefixes: index prefixes for search - term. The prefixes is a simple linked - list (NULL for last entry). Each prefix node - is of type MOD. + modifiers: relation modifiers for search + term. The modifiers list itself of cql_nodes + each of type ST. @@ -844,37 +1127,6 @@ struct cql_node { modifiers: proximity arguments. - - - prefixes: index prefixes. - The prefixes is a simple linked - list (NULL for last entry). Each prefix node - is of type MOD. - - - - - - - The modifier node is a "utility" node used for name-value pairs, - such as prefixes, proximity arguements, etc. - - - - name name of mod node. - - - - - value value of mod node. - - - - - next: pointer to next node which is - always a mod node (NULL for last entry). - - @@ -933,9 +1185,9 @@ int cql_transform_error(cql_transform_t ct, char **addinfop); error-code and sets the string-pointer at *addinfop to point to a string containing additional information about the error that occurred: for - example, if the error code is 15 (``Illegal or unsupported index + example, if the error code is 15 (``Illegal or unsupported context set''), the additional information is the name of the requested - index set that was not recognised. + context set that was not recognised. The SRW error-codes may be translated into brief human-readable @@ -991,26 +1243,37 @@ int cql_transform_FILE(cql_transform_t ct, The following CQL patterns are recognized: - qualifier.set.name + index.set.name - This pattern is invoked when a CQL qualifier, such as + This pattern is invoked when a CQL index, such as dc.title is converted. set - and name is the index set and qualifier + and name are the context set and index name respectively. Typically, the RPN specifies an equivalent use attribute. - For terms not bound by a qualifier the pattern - qualifier.srw.serverChoice is used. - Here, the prefix srw is defined as - http://www.loc.gov/zing/cql/srw-indexes/v1.0/. + For terms not bound by an index the pattern + index.cql.serverChoice is used. + Here, the prefix cql is defined as + http://www.loc.gov/zing/cql/cql-indexes/v1.0/. If this pattern is not defined, the mapping will fail. + qualifier.set.name + (DEPRECATED) + + + + For backwards compatibility, this is recognised as a synonym of + index.set.name + + + + relation.relation @@ -1092,10 +1355,10 @@ int cql_transform_FILE(cql_transform_t ct, - This specification defines a CQL index set for a given prefix. + This specification defines a CQL context set for a given prefix. The value on the right hand side is the URI for the set - not RPN. All prefixes used in - qualifier patterns must be defined this way. + index patterns must be defined this way. @@ -1103,16 +1366,16 @@ int cql_transform_FILE(cql_transform_t ct, CQL to RPN mapping file - This simple file defines two index sets, three qualifiers and three + This simple file defines two context sets, three indexes and three relations, a position pattern and a default structure. @attr 1=1016 @attr 2=3 @attr 4=1 @attr 3=3 @attr 6=1 "computer" - by rules qualifier.srw.serverChoice, + by rules index.cql.serverChoice, relation.scr, structure.*, position.any. @@ -1241,15 +1504,13 @@ typedef struct oident PROTO_Z3950 - PROTO_SR + PROTO_GENERAL - If you don't care about talking to SR-based implementations (few - exist, and they may become fewer still if and when the ISO SR and ANSI - Z39.50 documents are merged into a single standard), you can ignore - this field on incoming packages, and always set it to PROTO_Z3950 - for outgoing packages. + Use PROTO_Z3950 for Z39.50 Object Identifers, + PROTO_GENERAL for other types (such as + those associated with ILL). @@ -1336,6 +1597,10 @@ typedef struct oident again, corresponding to the specific OIDs defined by the standard. + Refer to the + + Registry of Z39.50 Object Identifiers for the + whole list. @@ -1400,6 +1665,49 @@ typedef struct oident + Three utility functions are provided for translating OIDs' + symbolic names (e.g. Usmarc into OID structures + (int arrays) and strings containing the OID in dotted notation + (e.g. 1.2.840.10003.9.5.1). They are: + + + + int *oid_name_to_oid(oid_class oclass, const char *name, int *oid); + char *oid_to_dotstring(const int *oid, char *oidbuf); + char *oid_name_to_dotstring(oid_class oclass, const char *name, char *oidbuf); + + + + oid_name_to_oid() + translates the specified symbolic name, + interpreted as being of class oclass. (The + class must be specified as many symbolic names exist within + multiple classes - for example, Zthes is the + symbolic name of an attribute set, a schema and a tag-set.) The + sequence of integers representing the OID is written into the + area oid provided by the caller; it is the + caller's responsibility to ensure that this area is large enough + to contain the translated OID. As a convenience, the address of + the buffer (i.e. the value of oid) is + returned. + + + oid_to_dotstring() + Translates the int-array oid into a dotted + string which is written into the area oidbuf + supplied by the caller; it is the caller's responsibility to + ensure that this area is large enough. The address of the buffer + is returned. + + + oid_name_to_dotstring() + combines the previous two functions to derive a dotted string + representing the OID specified by oclass and + name, writing it into the buffer passed as + oidbuf and returning its address. + + + Finally, the module provides the following utility functions, whose meaning should be obvious: @@ -1437,7 +1745,7 @@ typedef struct oident release the associated memory again. For the structures describing the Z39.50 PDUs and related structures, it is convenient to use the memory-management system of the &odr; subsystem (see - Using ODR). However, in some circumstances + ). However, in some circumstances where you might otherwise benefit from using a simple nibble memory management system, it may be impractical to use odr_malloc() and odr_reset(). @@ -1487,6 +1795,117 @@ typedef struct oident + + MARC + + + YAZ provides a fast utility that decodes MARC records and + encodes to a varity of output formats. The MARC records must + be encoded in ISO2709. + + + + /* create handler */ + yaz_marc_t yaz_marc_create(void); + /* destroy */ + void yaz_marc_destroy(yaz_marc_t mt); + + /* set XML mode YAZ_MARC_LINE, YAZ_MARC_SIMPLEXML, ... */ + void yaz_marc_xml(yaz_marc_t mt, int xmlmode); + #define YAZ_MARC_LINE 0 + #define YAZ_MARC_SIMPLEXML 1 + #define YAZ_MARC_OAIMARC 2 + #define YAZ_MARC_MARCXML 3 + #define YAZ_MARC_ISO2709 4 + + /* supply iconv handle for character set conversion .. */ + void yaz_marc_iconv(yaz_marc_t mt, yaz_iconv_t cd); + + /* set debug level, 0=none, 1=more, 2=even more, .. */ + void yaz_marc_debug(yaz_marc_t mt, int level); + + /* decode MARC in buf of size bsize. Returns >0 on success; <=0 on failure. + On success, result in *result with size *rsize. */ + int yaz_marc_decode_buf (yaz_marc_t mt, const char *buf, int bsize, + char **result, int *rsize); + + /* decode MARC in buf of size bsize. Returns >0 on success; <=0 on failure. + On success, result in WRBUF */ + int yaz_marc_decode_wrbuf (yaz_marc_t mt, const char *buf, + int bsize, WRBUF wrbuf); +]]> + + + A MARC conversion handle must be created by using + yaz_marc_create and destroyed + by calling yaz_marc_destroy. + + + All other function operate on a yaz_marc_t handle. + The output is specified by a call to yaz_marc_xml. + The xmlmode must be one of + + + YAZ_MARC_LINE + + + A simple line-by-line format suitable for display but not + recommend for further (machine) processing. + + + + + + YAZ_MARC_MARXML + + + The resulting record is converted to MARCXML. + + + + + + YAZ_MARC_ISO2709 + + + The resulting record is converted to ISO2709 (MARC). + + + + + + + The actual conversion functions are + yaz_marc_decode_buf and + yaz_marc_decode_wrbuf which decodes and encodes + a MARC record. The former function operates on simple buffers, the + stores the resulting record in a WRBUF handle (WRBUF is a simple string + type). + + + Display of MARC record + + The followint program snippet illustrates how the MARC API may + be used to convert a MARC record to the line-by-line format: + + + + + +