X-Git-Url: http://git.indexdata.com/?a=blobdiff_plain;f=doc%2Fquerymodel.xml;h=6d41f895184156558c16f449c77b0a1b8e87c932;hb=1ae4d572640ac104c00f8306beed27b3a0bcfe3b;hp=bae113f2551991bc0c62332313c2d2500ede7aa2;hpb=a4d2f62568bcf788630502fc1cbcad1163d3f87a;p=idzebra-moved-to-github.git diff --git a/doc/querymodel.xml b/doc/querymodel.xml index bae113f..6d41f89 100644 --- a/doc/querymodel.xml +++ b/doc/querymodel.xml @@ -1,112 +1,686 @@ - - - Query Model - + + + Query Model + Query Model Overview + - - Zebra is born as a networking Information Retrieval engine adhering - to the international standards - Z39.50 and - SRU, - and implement the query model defined there. - Unfortunately, the Z39.50 query model has only defined a binary - encoded representation, which is used as transport packaging in - the Z39.50 protocol layer. This representation is not human - readable, nor defines any convenient way to specify queries. - - - Therefore, Index Data has defined a textual representaion in the - Prefix Query Format, short - PQF, which then has been adopted by other - parties developing Z39.50 software. It is also often referred to as - Prefix Query Notation, or in short - PQN, and is thoroughly explained in - . - + + Query Languages + + + Zebra is born as a networking Information Retrieval engine adhering + to the international standards + Z39.50 and + SRU, + and implement the + type-1 Reverse Polish Notation (RPN) query + model defined there. + Unfortunately, this model has only defined a binary + encoded representation, which is used as transport packaging in + the Z39.50 protocol layer. This representation is not human + readable, nor defines any convenient way to specify queries. + + + Since the type-1 (RPN) + query structure has no direct, useful string + representation, every origin application needs to provide some + form of mapping from a local query notation or representation to it. + + + + + Prefix Query Format (PQF) - In addition, Zebra can be configured to understand and map the - Common Query Language - (CQL) - to PQF. See an introduction on the mapping to the internal query - representation in - . - - + Index Data has defined a textual representaion in the + Prefix Query Format, short + PQF, which mappes + one-to-one to binary encoded + type-1 RPN query packages. + It has been adopted by other + parties developing Z39.50 software, and is often referred to as + Prefix Query Notation, or in short + PQN. See + for further explanaitions and + descriptions of Zebra's capabilities. + + + + + Common Query Language (CQL) + + The query model of the type-1 RPN, + expressed in PQF/PQN is natively supported. + On the other hand, the default SRU + webservices Common Query Language + CQL is not natively supported. + + + Zebra can be configured to understand and map CQL to PQF. See + . + + + + + + Operation types + + Zebra supports all of the three different + Z39.50/SRU operations defined in the + standards: explain, search, + and scan. A short description of the + functionality and purpose of each is quite in order here. + + + + Explain Operation + + The syntax of Z39.50/SRU queries is + well known to any client, but the specific + semantics - taking into account a + particular servers functionalities and abilities - must be + discovered from case to case. Enters the + explain operation, which provides the means + for learning which + fields (also called + indexes or access points + are provided, which default parameter the server uses, which + retrieve document formats are defined, and which specific parts + of the general query model are supported. + + + The Z39.50 embeddes the explain operation + by perfoming a + search in the magic + IR-Explain-1 database; + see . + + + In SRU, explain is an entirely seperate + operation, which returns an Zeerex + XML record according to the + structure defined by the protocol. + + + In both cases, the information gathered through + explain operations can be used to + auto-configure a client user interface to the servers + capabilities. + + + + + Search Operation + + Search and retrieve interactions are the raison d'être. + They are used to query the remote database and + return search result documents. Search queries span from + simple free text searches to nested complex boolean queries, + targeting specific indexes, and possibly enhanced with many + query semantic specifications. Search interactions are the heart + and soul of Z39.50/SRU servers. + + + + + Scan Operation + + The scan operation is a helper functionality, + which operates on one index or access point a time. + + + It provides + the means to investigate the content of specific indexes. + Scanning an index returns a handfull of terms actually fond in + the indexes, and in addition the scan + operation returns th enumber of documents indexed by each term. + A search client can use this information to propose proper + spelling of search terms, to auto-fill search boxes, or to + display controlled vocabularies. + + + + + + + + - Prefix Query Format structure and syntax + Prefix Query Format syntax and semantics - The - PQF - grammer is documented in the YAZ manual. - This textual PQF representation + The PQF grammer + is documented in the YAZ manual, and shall not be + repeated here. This textual PQF representation is always during search mapped to the equivalent Zebra internal query parse tree. + + + PQF tree structure + + The PQF parse tree - or the equivalent textual representation - + may start with one specification of the + attribute set used. Following is a query + tree, which + consists of atomic query parts (APT) or + named result sets, eventually + paired by boolean binary operators, and + finally recursively combined into + complex query trees. + + + + Attribute sets + + Attribute sets define the exact meaning and semantics of queries + issued. Zebra comes with some predefined attribute set + definitions, others can easily be defined and added to the + configuration. + - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Attribute sets predefined in Zebra
Attribute setShort handStatusNotes
Explainexp-1Special attribute set used on the special automagic + IR-Explain-1 database to gain information on + server capabilities, database names, and database + and semantics.predefined
Bib1bib-1Standard PQF query language attribute set which defines the + semantics of Z39.50 searching. In addition, all of the + non-use attributes (type 2-9) define the hard-wired + Zebra internal query + processing.default
GILSgilsExtention to the Bib1 attribute set.predefined
+ - - Explain Attribute Set - - The attribute-set exp-1 is defined for - searching an Explain IR-Explain-1 database. - It consists of a single Use (type 1) attribute. - + + The use attributes (type 1) mappings the + predefined attribute sets are found in the + attribute set configuration files tab/*.att. + + + + The Zebra internal query processing is modeled after + the Bib1 attribute set, and the non-use + attributes type 2-6 are hard-wired in. It is therefore essential + to be familiar with . + + + + + Boolean operators + + A pair of subquery trees, or of atomic queries, is combined + using the standard boolean operators into new query trees. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Boolean operators
KeywordOperatorDescription
@andbinary AND operatorSet intersection of two atomic queries hit sets
@orbinary OR operatorSet union of two atomic queries hit sets
@notbinary AND NOT operatorSet complement of two atomic queries hit sets
@proxbinary PROXIMY operatorSet intersection of two atomic queries hit sets. In + addition, the intersection set is purged for all + documents which do not satisfy the requested query + term proximity. Usually a proper subset of the AND + operation.
+ + + For example, we can combine the terms + information and retrieval + into different searches in the default index of the default + attribute set as follows. + Querying for the union of all documents containing the + terms information OR + retrieval: + + Z> find @or information retrieval + + + + Querying for the intersection of all documents containing the + terms information AND + retrieval: + The hit set is a subset of the coresponding + OR query. + + Z> find @and information retrieval + + + + Querying for the intersection of all documents containing the + terms information AND + retrieval, taking proximity into account: + The hit set is a subset of the coresponding + AND query. + + Z> find @prox 0 3 0 2 k 2 information retrieval + + See PQF grammer for details. + + + Querying for the intersection of all documents containing the + terms information AND + retrieval, in the same order and near each + other as described in the term list + The hit set is a subset of the coresponding + PROXIMY query. + + Z> find "information retrieval" + + + + + + + Atomic queries (APT) + + Atomic queries are the query parts which work on one acess point + only. These consist of an attribute list + followed by a single term or a + quoted term list, and are often called + Attributes-Plus-Terms (APT) queries. + + + Unsupplied non-use attributes type 2-9 are either inherited from + higher nodes in the query tree, or are set to Zebra's default values. + See for details. + + + + + + + + + + + + + + + + + + +
Atomic queries
attribute listList of orthogonal attributesAny of the orthogonal attribute types may be omitted, + these are inherited from higher query tree nodes, or if not + inherited, are set to the default Zebra configuration values. +
termsingle term + or quoted term list Here the search terms or list of search terms is added + to the query
+ + Querying for the term information in the + default index using the default attribite set, the server choice + of access point/index, and the default non-use attributes. + + Z> find information + + + + Equivalent query fully specified including all default values: + + Z> find @attrset bib-1 @attr 1=1017 @attr 2=3 @attr 3=3 @attr 4=1 @attr 5=100 @attr 6=1 information + + + + + Finding all documents which have the term + debussy in the title field. + + Z> find @attr 1=4 debussy + + + + + + + + Named Result Sets + + Named result sets are supported in Zebra, and result sets can be + used as operands without limitations. + + + After the execution of a search, the result set is available at + the server, such that the client can use it for subsequent + searches or retrieval requests. The Z30.50 standard actually + stresses the fact that result sets are voliatile. It may cease + to exist at any time point after search, and the server will + send a diagnostic to the effect that the requested + result set does not exist any more. + + + + Defining a named result set and re-using it in the next query, + using yaz-client. + + Z> f @attr 1=4 mozart + ... + Number of hits: 43, setno 1 + ... + Z> f @and @set 1 @attr 1=4 amadeus + ... + Number of hits: 14, setno 2 + ... + Z> f @attr 1=1016 beethoven + ... + Number of hits: 26, setno 3 + ... + + + + + Named result sets are only supported by the Z39.50 protocol. + The SRU web service is stateless, and therefore the notion of + named result sets does not exist when acessing a Zebra server by + the SRU protocol. + + + + + + Zebra's special access point of type 'string' + + The numeric use (type 1) attribute is usually + refered to from a given + attribute set. In addition, Zebra let you use + any internal index + name defined in your configuration + as use atribute value. This is a great feature for + debugging, and when you do + not need the complecity of defined use attribute values. It is + the preferred way of accessing Zebra indexes directly. + + + Finding all documents which have the term list "information + retrieval" in an Zebra index, using it's internal full string + name. Scanning the same index. + + Z> find @attr 1=sometext "information retrieval" + Z> scan @attr 1=sometext aterm + + + + Searching or scanning + the bib-1 use attribute 54 using it's string name: + + Z> find @attr 1=Code-language eng + Z> scan @attr 1=Code-language "" + + + + It is possible to search + in any silly string index - if it's defined in your + indexation rules and can be parsed by the PQF parser. + This is definitely not the recommended use of + this facility, as it might confuse your users with some very + unexpected results. + + Z> find @attr 1=silly/xpath/alike[@index]/name "information retrieval" + + + + See also for details, and + + for the SRU PQF query extention using string names as a fast + debugging facility. + + + + + Zebra's special access point of type 'XPath' + for GRS filters + + As we have seen above, it is possible (albeit seldom a great + idea) to emulate + XPath 1.0 based + search by defining use (type 1) + string attributes which in appearence + resemble XPath queries. There are two + problems with this approach: first, the XPath-look-alike has to + be defined at indexation time, no new undefined + XPath queries can entered at search time, and second, it might + confuse users very much that an XPath-alike index name in fact + gets populated from a possible entirely different XML element + than it pretends to access. + + + When using the GRS Record Model + (see ), we have the + possibility to embed life + XPath expressions + in the PQF queries, which are here called + use (type 1) xpath + attributes. You must enable the + xpath enable directive in your + .abs config files. + + + Only a very restricted subset of the + XPath 1.0 + standard is supported as the GRS record model is simpler than + a full XML DOM structure. See the following examples for + possibilities. + + + Finding all documents which have the term "content" + inside a text node found in a specific XML DOM + subtree, whose starting element is + adressed by XPath. + + Z> find @attr 1=/root content + Z> find @attr 1=/root/first content + + Notice that the + XPath must be absolute, i.e., must start with '/', and that the + XPath decendant-or-self axis followed by a + text node selection text() is implicitly + appended to the stated XPath. + + It follows that the above searches are interpreted as: + + Z> find @attr 1=/root//text() content + Z> find @attr 1=/root/first//text() content + + + + + Searching inside attribute strings is possible: + + Z> find @attr 1=/link/@creator morten + + + + + Filter the adressing XPath by a predicate working on exact + string values in + attributes (in the XML sense) can be done: return all those docs which + have the term "english" contained in one of all text subnodes of + the subtree defined by the XPath + /record/title[@lang='en']. And similar + predicate filtering. + + Z> find @attr 1=/record/title[@lang='en'] english + Z> find @attr 1=/link[@creator='sisse'] sibelius + Z> find @attr 1=/link[@creator='sisse']/description[@xml:lang='da'] sibelius + + + + + Combining numeric indexes, boolean expressions, + and xpath based searches is possible: + + Z> find @attr 1=/record/title @and foo bar + Z> find @and @attr 1=/record/title foo @attr 1=4 bar + + + + Escaping PQF keywords and other non-parseable XPath constructs + with '{ }' to prevent syntax errors: + + Z> find @attr {1=/root/first[@attr='danish']} content + Z> find @attr {1=/record/@set} oai + + + + It is worth mentioning that these dynamic performed XPath + queries are a performance bottelneck, as no optimized + specialized indexes can be used. Therefore, avoid the use of + this facility when speed is essential, and the database content + size is medium to large. + + + + + + + + Explain Attribute Set + + The Z39.50 standard defines the + Explain attribute set + Exp-1, which is used to discover information + about a server's search semantics and functional capabilities + Zebra exposes a "classic" + Explain database by base name IR-Explain-1, which + is populated with system internal information. + + The attribute-set exp-1 consists of a single + use attribute (type 1). + + In addition, the non-Use bib-1 attributes, that is, the types Relation, Position, Structure, Truncation, and Completeness are imported from - the bib-1 attrubute set, and may be used + the bib-1 attribute set, and may be used within any explain query. - + - + Use Attributes (type = 1) - - The following Explain search atributes are supported: - ExplainCategory (@attr 1=1), - DatabaseName (@attr 1=3), - DateAdded (@attr 1=9), - DateChanged(@attr 1=10). - - - A search in the use attribute ExplainCategory - supports only these predefined values: - CategoryList, TargetInfo, - DatabaseInfo, AttributeDetails. - + + The following Explain search atributes are supported: + ExplainCategory (@attr 1=1), + DatabaseName (@attr 1=3), + DateAdded (@attr 1=9), + DateChanged(@attr 1=10). + + + A search in the use attribute ExplainCategory + supports only these predefined values: + CategoryList, TargetInfo, + DatabaseInfo, AttributeDetails. + See tab/explain.att and the + Z39.50 standard for more information. - - + + Explain searches with yaz-client + Classic Explain only defines retrieval of Explain information + via ASN.1. Pratically no Z39.50 clients supports this. Fortunately + they don't have to - Zebra allows retrieval of this information + in other formats: + SUTRS, XML, + GRS-1 and ASN.1 Explain. + + + List supported categories to find out which explain commands are supported: Z> base IR-Explain-1 - Z> @attr exp1 1=1 categorylist + Z> find @attr exp1 1=1 categorylist Z> form sutrs Z> show 1+2 - + Get target info, that is, investigate which databases exist at this server endpoint: Z> base IR-Explain-1 - Z> @attr exp1 1=1 targetinfo + Z> find @attr exp1 1=1 targetinfo Z> form xml Z> show 1+1 Z> form grs-1 @@ -115,360 +689,1322 @@ Z> show 1+1 - + + + List all supported databases, the number of hits + is the number of databases found, which most commonly are the + following two: + the Default and the + IR-Explain-1 databases. + + Z> base IR-Explain-1 + Z> find @attr exp1 1=1 databaseinfo + Z> form sutrs + Z> show 1+2 + + + + + Get database info record for database Default. + + Z> base IR-Explain-1 + Z> find @and @attr exp1 1=1 databaseinfo @attr exp1 1=3 Default + + Identical query with explicitly specified attribute set: + + Z> base IR-Explain-1 + Z> find @attrset exp1 @and @attr 1=1 databaseinfo @attr 1=3 Default + + + + + Get attribute details record for database + Default. + This query is very useful to study the internal Zebra indexes. + If records have been indexed using the alvis + XSLT filter, the string representation names of the known indexes can be + found. + + Z> base IR-Explain-1 + Z> find @and @attr exp1 1=1 attributedetails @attr exp1 1=3 Default + + Identical query with explicitly specified attribute set: + + Z> base IR-Explain-1 + Z> find @attrset exp1 @and @attr 1=1 attributedetails @attr 1=3 Default + + + + + + + + Bib1 Attribute Set + + Most of the information contained in this section is an excerpt of + the ATTRIBUTE SET BIB-1 (Z39.50-1995) + SEMANTICS, + found at . The BIB-1 + Attribute Set Semantics from 1995, also in an updated + Bib-1 + Attribute Set + version from 2003. Index Data is not the copyright holder of this + information, except for the configuration details, the listing of + Zebra's capabilities, and the example queries. + + + + + Use Attributes (type 1) + + + A use attribute specifies an access point for any atomic query. + These acess points are highly dependent on the attribute set used + in the query, and are user configurable using the following + default configuration files: + tab/bib1.att, + tab/dan1.att, + tab/explain.att, and + tab/gils.att. + New attribute sets can be added by adding new + tab/*.att configuration files, which need to + be sourced in the main configuration zebra.cfg. + + + + In addition, Zebra allows the acess of + internal index names and dynamic + XPath as use attributes; see + and + . + + + + Phrase search for information retrieval in + the title-register, scanning the same register afterwards: + + Z> find @attr 1=4 "information retrieval" + Z> scan @attr 1=4 information + + + + + + + + + Zebra general Bib1 Non-Use Attributes (type 2-6) + + + Relation Attributes (type 2) + + + Relation attributes describe the relationship of the access + point (left side + of the relation) to the search term as qualified by the attributes (right + side of the relation), e.g., Date-publication <= 1975. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Relation Attributes (type 2)
RelationValueNotes
Less than1supported
Less than or equal2supported
Equal3default
Greater or equal4supported
Greater than5supported
Not equal6unsupported
Phonetic100unsupported
Stem101unsupported
Relevance102supported
AlwaysMatches103supported
+ + + The relation attributes + 1-5 are supported and work exactly as + expected. + All ordering operations are based on a lexicographical ordering, + expect when the + structure attribute numeric (109) is used. In + this case, ordering is numerical. See + . + + Z> find @attr 1=Title @attr 2=1 music + ... + Number of hits: 11745, setno 1 + ... + Z> find @attr 1=Title @attr 2=2 music + ... + Number of hits: 11771, setno 2 + ... + Z> find @attr 1=Title @attr 2=3 music + ... + Number of hits: 532, setno 3 + ... + Z> find @attr 1=Title @attr 2=4 music + ... + Number of hits: 11463, setno 4 + ... + Z> find @attr 1=Title @attr 2=5 music + ... + Number of hits: 11419, setno 5 + + + + + The relation attribute + Relevance (102) is supported, see + for full information. + + + + Ranked search for information retrieval in + the title-register: + + Z> find @attr 1=4 @attr 2=102 "information retrieval" + + + + + The relation attribute + AlwaysMatches (103) is in the default + configuration + supported in conjecture with structure attribute + Phrase (1) (which may be omitted by + default). + It can be configured to work with other structure attributes, + see the configuration file + tab/default.idx and + . + + + AlwaysMatches (103) is a + great way to discover how many documents have been indexed in a + given field. The search term is ignored, but needed for correct + PQF syntax. An empty search term may be supplied. + + Z> find @attr 1=Title @attr 2=103 "" + Z> find @attr 1=Title @attr 2=103 @attr 4=1 "" + + + + + + + + Position Attributes (type 3) + + + The position attribute specifies the location of the search term + within the field or subfield in which it appears. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Position Attributes (type 3)
PositionValueNotes
First in field 1unsupported
First in subfield2unsupported
Any position in field3default
+ + + The position attribute values first in field (1), + and first in subfield(2) are unsupported. + Using them does not trigger an error, but silent defaults to + any position in field (3). + + + + + + Structure Attributes (type 4) + + + The structure attribute specifies the type of search + term. This causes the search to be mapped on + different Zebra internal indexes, which must have been defined + at index time. + + + + The possible values of the + structure attribute (type 4) can be defined + using the configuration file + tab/default.idx. + The default configuration is summerized in this table. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Structure Attributes (type 4)
StructureValueNotes
Phrase 1default
Word2supported
Key3supported
Year4supported
Date (normalized)5supported
Word list6supported
Date (un-normalized)100unsupported
Name (normalized) 101unsupported
Name (un-normalized) 102unsupported
Structure103unsupported
Urx104supported
Free-form-text105supported
Document-text106supported
Local-number107supported
String108unsupported
Numeric string109supported
+ + + + The structure attribute values + Word list (6) + is supported, and maps to the boolean AND + combination of words supplied. The word list is useful when + google-like bag-of-word queries need to be translated from a GUI + query language to PQF. For example, the following queries + are equivalent: + + Z> find @attr 1=Title @attr 4=6 "mozart amadeus" + Z> find @attr 1=Title @and mozart amadeus + + + + + The structure attribute value + Free-form-text (105) and + Document-text (106) + are supported, and map both to the boolean OR + combination of words supplied. The following queries + are equivalent: + + Z> find @attr 1=Body-of-text @attr 4=105 "bach salieri teleman" + Z> find @attr 1=Body-of-text @attr 4=106 "bach salieri teleman" + Z> find @attr 1=Body-of-text @or bach @or salieri teleman + + This OR list of terms is very usefull in + combination with relevance ranking: + + Z> find @attr 1=Body-of-text @attr 2=102 @attr 4=105 "bach salieri teleman" + + + + + The structure attribute value + Local number (107) + is supported, and maps always to the Zebra internal document ID, + irrespectively which use attribute is specified. The following queries + have exactly the same unique record in the hit set: + + Z> find @attr 4=107 10 + Z> find @attr 1=4 @attr 4=107 10 + Z> find @attr 1=1010 @attr 4=107 10 + + + + + In + the GILS schema (gils.abs), the + west-bounding-coordinate is indexed as type n, + and is therefore searched by specifying + structure=Numeric String. + To match all those records with west-bounding-coordinate greater + than -114 we use the following query: + + Z> find @attr 4=109 @attr 2=5 @attr gils 1=2038 -114 + + + + The exact mapping between PQF queries and Zebra internal indexes + and index types is explained in + . + + + + + + Truncation Attributes (type = 5) + + + The truncation attribute specifies whether variations of one or + more characters are allowed between serch term and hit terms, or + not. Using non-default truncation attributes will broaden the + document hit set of a search query. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Truncation Attributes (type 5)
TruncationValueNotes
Right truncation 1supported
Left truncation2supported
Left and right truncation3supported
Do not truncate100default
Process # in search term101supported
RegExpr-1 102supported
RegExpr-2103supported
+ + + The truncation attribute values 1-3 perform the obvious way: + + Z> scan @attr 1=Body-of-text schnittke + ... + * schnittke (81) + schnittkes (31) + schnittstelle (1) + ... + Z> find @attr 1=Body-of-text @attr 5=1 schnittke + ... + Number of hits: 95, setno 7 + ... + Z> find @attr 1=Body-of-text @attr 5=2 schnittke + ... + Number of hits: 81, setno 6 + ... + Z> find @attr 1=Body-of-text @attr 5=3 schnittke + ... + Number of hits: 95, setno 8 + + + + + The truncation attribute value + Process # in search term (101) is a + poor-man's regular expression search. It maps + each # to .*, and + performes then a Regexp-1 (102) regular + expression search. The following two queries are equivalent: + + Z> find @attr 1=Body-of-text @attr 5=101 schnit#ke + Z> find @attr 1=Body-of-text @attr 5=102 schnit.*ke + ... + Number of hits: 89, setno 10 + + + + + The truncation attribute value + Regexp-1 (102) is a normal regular search, + see for details. + + Z> find @attr 1=Body-of-text @attr 5=102 schnit+ke + Z> find @attr 1=Body-of-text @attr 5=102 schni[a-t]+ke + + + + + The truncation attribute value + Regexp-2 (103) is a Zebra specific extention + which allows fuzzy matches. One single + error in spelling of search terms is allowed, i.e., a document + is hit if it includes a term which can be mapped to the used + search term by one character substitution, addition, deletion or + change of posiiton. + + Z> find @attr 1=Body-of-text @attr 5=100 schnittke + ... + Number of hits: 81, setno 14 + ... + Z> find @attr 1=Body-of-text @attr 5=103 schnittke + ... + Number of hits: 103, setno 15 + ... + + + + + + Completeness Attributes (type = 6) + + + + The Completeness Attributes (type = 6) + is used to specify that a given search term or term list is either + part of the terms of a given index/field + (Incomplete subfield (1)), or is + what literally is found in the entire field's index + (Complete field (3)). + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Completeness Attributes (type = 6)
CompletenessValueNotes
Incomplete subfield1default
Complete subfield2depreciated
Complete field3supported
+ + + The Completeness Attributes (type = 6) + is only partially and conditionally + supported in the sense that it is ignored if the hit index is + not of structure type="w" or + type="p". + + + Incomplete subfield (1) is the default, and + makes Zebra use + register type="w", whereas + Complete field (3) triggers + search and scan in index type="p". + + + The Complete subfield (2) is a reminiscens + from the happy MARC + binary format days. Zebra does not support it, but maps silently + to Complete field (3). + + + + The exact mapping between PQF queries and Zebra internal indexes + and index types is explained in + . + + + + + + + + + Advanced Zebra PQF Features + + The Zebra internal query engine has been extended to specific needs + not covered by the bib-1 attribute set query + model. These extentions are non-standard + and non-portable: most functional extentions + are modeled over the bib-1 attribute set, + defining type 7-9 attributes. + There are also the speciel + string type index names for the + idxpath attribute set. + + + + Zebra specific retrieval of all records + + Zebra defines a hardwired string index name + called _ALLRECORDS. It matches any record + contained in the database, if used in conjunction with + the relation attribute + AlwaysMatches (103). + + + The _ALLRECORDS index name is used for total database + export. The search term is ignored, it may be empty. + + Z> find @attr 1=_ALLRECORDS @attr 2=103 "" + + + + Combination with other index types can be made. For example, to + find all records which are not indexed in + the Title register, issue one of the two + equivalent queries: + + Z> find @not @attr 1=_ALLRECORDS @attr 2=103 "" @attr 1=Title @attr 2=103 "" + Z> find @not @attr 1=_ALLRECORDS @attr 2=103 "" @attr 1=4 @attr 2=103 "" + + + + The special string index _ALLRECORDS is + experimental, and the provided functionality and syntax may very + well change in future releases of Zebra. + + + + + + Zebra specific Search Extentions to all Attribute Sets + + Zebra extends the Bib1 attribute types, and these extentions are + recognized regardless of attribute + set used in a search operation query. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Zebra Search Attribute Extentions
NameValueOperationZebra version
Embedded Sort7search1.1
Term Set8search1.1
Rank Weight9search1.1
Approx Limit9search1.4
Term Reference10search1.4
+ + + Zebra Extention Embedded Sort Attribute (type 7) + + + The embedded sort is a way to specify sort within a query - thus + removing the need to send a Sort Request separately. It is both + faster and does not require clients to deal with the Sort + Facility. + + + + All ordering operations are based on a lexicographical ordering, + expect when the + structure attribute numeric (109) is used. In + this case, ordering is numerical. See + . + + + + The possible values after attribute type 7 are + 1 ascending and + 2 descending. + The attributes+term (APT) node is separate from the + rest and must be @or'ed. + The term associated with APT is the sorting level in integers, + where 0 means primary sort, + 1 means secondary sort, and so forth. + See also . + + + For example, searching for water, sort by title (ascending) + + Z> find @or @attr 1=1016 water @attr 7=1 @attr 1=4 0 + + + + Or, searching for water, sort by title ascending, then date descending + + Z> find @or @or @attr 1=1016 water @attr 7=1 @attr 1=4 0 @attr 7=2 @attr 1=30 1 + + + + + Zebra Extention Term Set Attribute (type 8) + + + The Term Set feature is a facility that allows a search to store + hitting terms in a "pseudo" resultset; thus a search (as usual) + + a scan-like facility. Requires a client that can do named result + sets since the search generates two result sets. The value for + attribute 8 is the name of a result set (string). The terms in + the named term set are returned as SUTRS records. + + + For example, searching for u in title, right truncated, and + storing the result in term set named 'aset' + + Z> find @attr 5=1 @attr 1=4 @attr 8=aset u + + + + The model has one serious flaw: we don't know the size of term + set. Experimental. Do not use in production code. + + + + Zebra Extention Rank Weight Attribute (type 9) + + + Rank weight is a way to pass a value to a ranking algorithm - so + that one APT has one value - while another as a different one. + See also . + + + For example, searching for utah in title with weight 30 as well + as any with weight 20: + + Z> find @attr 2=102 @or @attr 9=30 @attr 1=4 utah @attr 9=20 utah + + + + + Zebra Extention Approximative Limit Attribute (type 9) + + + Newer Zebra versions normally estemiates hit count for every APT + (leaf) in the query tree. These hit counts are returned as part of + the searchResult-1 facility in the binary encoded Z39.50 search + response packages. + + + By setting a limit for the APT we can make Zebra turn into + approximate hit count when a certain hit count limit is + reached. A value of zero means exact hit count. + + + For example, we might be intersted in exact hit count for a, but + for b we allow hit count estimates for 1000 and higher. + + Z> find @and a @attr 9=1000 b + + + + The estimated hit count fascility makes searches faster, as one + only needs to process large hit lists partially. + + + This facility clashes with rank weight, because there all + documents in the hit lists need to be examined for scoring and + re-sorting. + It is an experimental + extention. Do not use in production code. + + + + Zebra Extention Term Reference Attribute (type 10) + + + Zebra supports the searchResult-1 facility. + If the Term Reference Attribute (type 10) is + given, that specifies a subqueryId value returned as part of the + search result. It is a way for a client to name an APT part of a + query. + + + + Experimental. Do not use in production code. + + + + + + + + Zebra specific Scan Extentions to all Attribute Sets + + Zebra extends the Bib1 attribute types, and these extentions are + recognized regardless of attribute + set used in a scan operation query. + + + + + + + + + + + + + + + + + + + + + + + + + + +
Zebra Scan Attribute Extentions
NameTypeOperationZebra version
Result Set Narrow8scan1.3
Approximative Limit9scan1.4
+ + + Zebra Extention Result Set Narrow (type 8) + + + If attribute Result Set Narrow (type 8) + is given for scan, the value is the name of a + result set. Each hit count in scan is + @and'ed with the result set given. + + + Consider for example + the case of scanning all title fields around the + scanterm mozart, then refining the scan by + issuing a filtering query for amadeus to + restric the scan to the result set of the query: + + Z> scan @attr 1=4 mozart + ... + * mozart (43) + mozartforskningen (1) + mozartiana (1) + mozarts (16) + ... + Z> f @attr 1=4 amadeus + ... + Number of hits: 15, setno 2 + ... + Z> scan @attr 1=4 @attr 8=2 mozart + ... + * mozart (14) + mozartforskningen (0) + mozartiana (0) + mozarts (1) + ... + + + + + Experimental. Do not use in production code. + + + + Zebra Extention Approximative Limit (type 9) + + + The Zebra Extention Approximative Limit (type + 9) is a way to enable approx + hit counts for scan hit counts, in the same + way as for search hit counts. + + + + Experimental and buggy. Definitely not to be used in production code. + + + + + + + + Zebra special IDXPATH Attribute Set for GRS indexing + + The attribute-set idxpath consists of a single + Use (type 1) attribute. All non-use attributes + behave as normal. + + + This feature is enabled when defining the + xpath enable option in the GRS filter + *.abs configuration files. If one wants to use + the special idxpath numeric attribute set, the + main Zebra configuraiton file zebra.cfg + directive attset: idxpath.att must be enabled. + + The idxpath is depreciated, may not be + supported in future Zebra versions, and should definitely + not be used in production code. + + + + IDXPATH Use Attributes (type = 1) + + This attribute set allows one to search GRS filter indexed + records by XPATH like structured index names. + + + The idxpath option defines hard-coded + index names, which might clash with your own index names. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Zebra specific IDXPATH Use Attributes (type 1)
IDXPATHValueString IndexNotes
XPATH Begin1_XPATH_BEGINdepreciated
XPATH End2_XPATH_ENDdepreciated
XPATH CData1016_XPATH_CDATAdepreciated
XPATH Attribute Name3_XPATH_ATTR_NAMEdepreciated
XPATH Attribute CData1015_XPATH_ATTR_CDATAdepreciated
+ + + + See tab/idxpath.att for more information. + + + Search for all documents starting with root element + /root (either using the numeric or the string + use attributes): + + Z> find @attrset idxpath @attr 1=1 @attr 4=3 root/ + Z> find @attr idxpath 1=1 @attr 4=3 root/ + Z> find @attr 1=_XPATH_BEGIN @attr 4=3 root/ + + + + Search for all documents where specific nested XPATH + /c1/c2/../cn exists. Notice the very + counter-intuitive reverse notation! + + Z> find @attrset idxpath @attr 1=1 @attr 4=3 cn/cn-1/../c1/ + Z> find @attr 1=_XPATH_BEGIN @attr 4=3 cn/cn-1/../c1/ + + - List all supported databases, the number of hits - is the number of databases found, which most commonly are the - following two: - the Default and the - IR-Explain-1 databases. + Search for CDATA string text in any element - Z> base IR-Explain-1 - Z> f @attr exp1 1=1 databaseinfo - Z> form sutrs - Z> show 1+2 + Z> find @attrset idxpath @attr 1=1016 text + Z> find @attr 1=_XPATH_CDATA text - - Get database info record for database Default. - - Z> base IR-Explain-1 - Z> @and @attr exp1 1=1 databaseinfo @attr exp1 1=3 Default + Search for CDATA string anothertext in any + attribute: + + Z> find @attrset idxpath @attr 1=1015 anothertext + Z> find @attr 1=_XPATH_ATTR_CDATA anothertext - Identical query with explicitly specified attribute set: - - Z> base IR-Explain-1 - Z> @attrset exp1 @and @attr 1=1 databaseinfo @attr 1=3 Default + + + Search for all documents with have an XML element node + including an XML attribute named creator + + Z> find @attrset idxpath @attr 1=3 @attr 4=3 creator + Z> find @attr 1=_XPATH_ATTR_NAME @attr 4=3 creator - - Get attribute details record for database - Default. - This query is very useful to study the internal Zebra indexes. - If records have been indexed using the alvis - XSLT filter, the string representation names of the known indexes can be - found. + Combining usual bib-1 attribut set searches + with idxpath attribute set searches: - Z> base IR-Explain-1 - Z> @and @attr exp1 1=1 attributedetails @attr exp1 1=3 Default + Z> find @and @attr idxpath 1=1 @attr 4=3 link/ @attr 1=4 mozart + Z> find @and @attr 1=_XPATH_BEGIN @attr 4=3 link/ @attr 1=_XPATH_CDATA mozart - Identical query with explicitly specified attribute set: + + + Scanning is supportet on all idxpath + indexes, both specified as numeric use attributes, or as string + index names. - Z> base IR-Explain-1 - Z> @attrset exp1 @and @attr 1=1 attributedetails @attr 1=3 Default + Z> scan @attrset idxpath @attr 1=1016 text + Z> scan @attr 1=_XPATH_ATTR_CDATA anothertext + Z> scan @attrset idxpath @attr 1=3 @attr 4=3 '' - - - - - Bib1 Attribute Set - - Something about querying to be written .. - - - Most of the information contained in this section is an excerpt of - the ATTRIBUTE SET BIB-1 (Z39.50-1995) - SEMANTICS, found at The BIB-1 - Attribute Set Semantics from 1995, also in an updated - Bib-1 - Attribute Set - version from 2003. Index Data is not the copyright holder of this - information. - - - - - Use Attributes (type = 1) - - - - Relation Attributes (type = 2) - - - - - - Position Attributes (type = 3) - - - - Structure Attributes (type = 4) - - - - Truncation Attributes (type = 5) - - - - Completeness Attributes (type = 6) - - - - Zebra Extention Sorting Attributes (type = 7) - - - - Zebra Extention Search Estimation Attributes (type = 8) - + + - - Zebra Extention Weight Attributes (type = 9) - - - - - Mapping from Bib1 Attributes to Zebra internal + <sect2 id="querymodel-pqf-apt-mapping"> + <title>Mapping from PQF atomic APT queries to Zebra internal register indexes - - - - - Use attributes are interpreted according to the - attribute sets which have been loaded in the - zebra.cfg file, and are matched against specific - fields as specified in the .abs file which - describes the profile of the records which have been loaded. - If no Use attribute is provided, a default of Bib-1 Any is assumed. - - - - If a Structure attribute of - Phrase is used in conjunction with a - Completeness attribute of - Complete (Sub)field, the term is matched - against the contents of the phrase (long word) register, if one - exists for the given Use attribute. - A phrase register is created for those fields in the - .abs file that contains a - p-specifier. - - - - - If Structure=Phrase is - used in conjunction with Incomplete Field - the - default value for Completeness, the - search is directed against the normal word registers, but if the term - contains multiple words, the term will only match if all of the words - are found immediately adjacent, and in the given order. - The word search is performed on those fields that are indexed as - type w in the .abs file. - - - - If the Structure attribute is - Word List, - Free-form Text, or - Document Text, the term is treated as a - natural-language, relevance-ranked query. - This search type uses the word register, i.e. those fields - that are indexed as type w in the - .abs file. - - - - If the Structure attribute is - Numeric String the term is treated as an integer. - The search is performed on those fields that are indexed - as type n in the .abs file. - - - - If the Structure attribute is - URx the term is treated as a URX (URL) entity. - The search is performed on those fields that are indexed as type - u in the .abs file. - - - - If the Structure attribute is - Local Number the term is treated as - native Zebra Record Identifier. - - - - If the Relation attribute is - Equals (default), the term is matched - in a normal fashion (modulo truncation and processing of - individual words, if required). - If Relation is Less Than, - Less Than or Equal, - Greater than, or Greater than or - Equal, the term is assumed to be numerical, and a - standard regular expression is constructed to match the given - expression. - If Relation is Relevance, - the standard natural-language query processor is invoked. - - - - For the Truncation attribute, - No Truncation is the default. - Left Truncation is not supported. - Process # in search term is supported, as is - Regxp-1. - Regxp-2 enables the fault-tolerant (fuzzy) - search. As a default, a single error (deletion, insertion, - replacement) is accepted when terms are matched against the register - contents. - - - - - Regular expressions - - Each term in a query is interpreted as a regular expression if - the truncation value is either Regxp-1 (102) - or Regxp-2 (103). - Both query types follow the same syntax with the operands: - - - - x - - - Matches the character x. - - - - - . - - - Matches any character. - - - - - [..] - - - Matches the set of characters specified; - such as [abc] or [a-c]. - - - - - and the operators: - - - - x* - - - Matches x zero or more times. Priority: high. - - - - - x+ - - - Matches x one or more times. Priority: high. - - - - - x? - - - Matches x zero or once. Priority: high. - - - - - xy - - - Matches x, then y. - Priority: medium. - - - - - x|y - - - Matches either x or y. - Priority: low. - - - - - The order of evaluation may be changed by using parentheses. - - - - If the first character of the Regxp-2 query - is a plus character (+) it marks the - beginning of a section with non-standard specifiers. - The next plus character marks the end of the section. - Currently Zebra only supports one specifier, the error tolerance, - which consists one digit. + The rules for PQF APT mapping are rather tricky to grasp in the + first place. We deal first with the rules for deciding which + internal register or string index to use, according to the use + attribute or access point specified in the query. Thereafter we + deal with the rules for tetermining the correct structure type of + the named register. + + Mapping of PQF APT access points - Since the plus operator is normally a suffix operator the addition to - the query syntax doesn't violate the syntax for standard regular - expressions. + Zebra understands four fundamental different types of access + points, of which only the + numeric use attribute type access points + are defined by the Z39.50 + standard. + All other access point types are Zebra specific, and non-portable. - - - - Query examples + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Acces point name
Acess PointTypeGrammarNotes
Use attibutenumeric[1-9][1-9]*directly mapped to string index name
String index namestring[a-zA-Z](\-?[a-zA-Z0-9])*normalized name is used as internal string index name
Zebra internal index namezebra_[a-zA-Z](_?[a-zA-Z0-9])*hardwired internal string index name
XPATH special indexXPath/.*special xpath search for GRS indexed records
- Phrase search for information retrieval in - the title-register: - - @attr 1=4 "information retrieval" - - + Attribute set names and + string index names are normalizes + according to the following rules: all single + hyphens '-' are stripped, and all upper case + letters are folded to lower case. - Ranked search for the same thing: + Numeric use attributes are mapped + to the Zebra internal + string index according to the attribute set defintion in use. + The default attribute set is Bib-1, and may be + omitted in the PQF query. According to normalization and numeric + use attribute mapping, it follows that the following + PQF queries are considered equivalent (assuming the default + configuration has not been altered): - @attr 1=4 @attr 2=102 "Information retrieval" + Z> find @attr 1=Body-of-text serenade + Z> find @attr 1=bodyoftext serenade + Z> find @attr 1=BodyOfText serenade + Z> find @attr 1=bO-d-Y-of-tE-x-t serenade + Z> find @attr 1=1010 serenade + Z> find @attrset Bib-1 @attr 1=1010 serenade + Z> find @attrset bib1 @attr 1=1010 serenade + Z> find @attrset Bib1 @attr 1=1010 serenade + Z> find @attrset b-I-b-1 @attr 1=1010 serenade - Phrase search with a regular expression: - - @attr 1=4 @attr 5=102 "informat.* retrieval" - + The numerical + use attributes (type 1) + are interpreted according to the + attribute sets which have been loaded in the + zebra.cfg file, and are matched against specific + fields as specified in the .abs file which + describes the profile of the records which have been loaded. + If no use attribute is provided, a default of Bib-1 Any is + assumed. + The predefined use attribute sets + can be reconfigured by tweaking the configuration files + tab/*.att, and + new attribute sets can be defined by adding similar files in the + configuration path profilePath of the server. - - Ranked search with a regular expression: - - @attr 1=4 @attr 5=102 @attr 2=102 "informat.* retrieval" - - + + String indexes can be acessed directly, + independently which attribute set is in use. These are just + ignored. The above mentioned name normalization applies. + String index names are defined in the + used indexing filter configuration files, for example in the + GRS + *.abs configuration files, or in the + alvis filter XSLT indexing stylesheets. + - - In the GILS schema (gils.abs), the - west-bounding-coordinate is indexed as type n, - and is therefore searched by specifying - structure=Numeric String. - To match all those records with west-bounding-coordinate greater - than -114 we use the following query: - - @attr 4=109 @attr 2=5 @attr gils 1=2038 -114 - - - + + Zebra internal indexes can be acessed directly, + according to the same rules as the user defined + string indexes. The only difference is that + Zebra internal indexe names are hardwired, + all uppercase and + must start with the character '_'. + + + + Finally, XPATH access points are only + available using the GRS filter for indexing. + These acees point names must start with the character + '/', they are not + normalized, but passed unaltered to the Zebra internal + XPATH engine. See . + + + + + + + Mapping of PQF APT structure and type + + + + + + + + If a Structure attribute of + Phrase is used in conjunction with a + Completeness attribute of + Complete (Sub)field, the term is matched + against the contents of the phrase (long word) register, if one + exists for the given Use attribute. + A phrase register is created for those fields in the + .abs file that contains a + p-specifier. + + + + + If Structure=Phrase is + used in conjunction with Incomplete Field - the + default value for Completeness, the + search is directed against the normal word registers, but if the term + contains multiple words, the term will only match if all of the words + are found immediately adjacent, and in the given order. + The word search is performed on those fields that are indexed as + type w in the .abs file. + + + + If the Structure attribute is + Word List, + Free-form Text, or + Document Text, the term is treated as a + natural-language, relevance-ranked query. + This search type uses the word register, i.e. those fields + that are indexed as type w in the + .abs file. + + + + If the Structure attribute is + Numeric String the term is treated as an integer. + The search is performed on those fields that are indexed + as type n in the .abs file. + + + + If the Structure attribute is + URx the term is treated as a URX (URL) entity. + The search is performed on those fields that are indexed as type + u in the .abs file. + + + + If the Structure attribute is + Local Number the term is treated as + native Zebra Record Identifier. + + + + If the Relation attribute is + Equals (default), the term is matched + in a normal fashion (modulo truncation and processing of + individual words, if required). + If Relation is Less Than, + Less Than or Equal, + Greater than, or Greater than or + Equal, the term is assumed to be numerical, and a + standard regular expression is constructed to match the given + expression. + If Relation is Relevance, + the standard natural-language query processor is invoked. + + + + For the Truncation attribute, + No Truncation is the default. + Left Truncation is not supported. + Process # in search term is supported, as is + Regxp-1. + Regxp-2 enables the fault-tolerant (fuzzy) + search. As a default, a single error (deletion, insertion, + replacement) is accepted when terms are matched against the register + contents. + + + + + + + Zebra Regular Expressions in Truncation Attribute (type = 5) + + + Each term in a query is interpreted as a regular expression if + the truncation value is either Regxp-1 (@attr 5=102) + or Regxp-2 (@attr 5=103). + Both query types follow the same syntax with the operands: + + + + + + + + + + + + + + + + + + + + +
Regular Expression Operands
xMatches the character x.
.Matches any character.
[ .. ]Matches the set of characters specified; + such as [abc] or [a-c].
+ + + The above operands can be combined with the following operators: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Regular Expression Operators
x*Matches x zero or more times. + Priority: high.
x+Matches x one or more times. + Priority: high.
x? Matches x zero or once. + Priority: high.
xy Matches x, then y. + Priority: medium.
x|y Matches either x or y. + Priority: low.
( )The order of evaluation may be changed by using parentheses.
+ + + If the first character of the Regxp-2 query + is a plus character (+) it marks the + beginning of a section with non-standard specifiers. + The next plus character marks the end of the section. + Currently Zebra only supports one specifier, the error tolerance, + which consists one digit. + + + + Since the plus operator is normally a suffix operator the addition to + the query syntax doesn't violate the syntax for standard regular + expressions. + + + + For example, a phrase search with regular expressions in + the title-register is performed like this: + + Z> find @attr 1=4 @attr 5=102 "informat.* retrieval" + + + + + Combinations with other attributes are possible. For example, a + ranked search with a regular expression: + + Z> find @attr 1=4 @attr 5=102 @attr 2=102 "informat.* retrieval" + + + + @@ -630,108 +2367,6 @@ - -