-% Embedded metasearching with the MasterKey Widget Set
+% The MKWS manual: embedded metasearching with the MasterKey Widget Set
% Mike Taylor
-% 30 July 2014
+% October 2014
+
Introduction
-------------
+============
There are lots of practical problems in building resource discovery
solutions. One of the biggest, and most ubiquitous is incorporating
its metasearching middleware, trading off varying degrees of
flexibility against convenience:
-* pz2.js -- a low-level JavaScript library for interrogating the
- Service Proxy and Pazpar2. It allows the HTML/JavaScript programmer
- to create JavaScript applications display facets, records, etc. that
- are fetched from the metasearching middleware.
+* [pz2.js](http://www.indexdata.com/pazpar2/doc/ajaxdev.html) --
+ a low-level JavaScript library for interrogating the
+ [Service Proxy](http://www.indexdata.com/service-proxy/)
+ and
+ [Pazpar2](http://www.indexdata.com/pazpar2/).
+ It allows the HTML/JavaScript programmer
+ to create JavaScript applications to display facets, records,
+ etc. that are fetched from the metasearching middleware.
* masterkey-ui-core -- a higher-level, complex JavaScript library that
uses pz2.js to provide the pieces needed for building a
* MasterKey Demo UI -- an example of a searching application built on
top of masterkey-ui-core. Available as a public demo at
- http://mk2.indexdata.com/
+ <http://mk2.indexdata.com/>
-* MKDru -- a toolkit for embedding MasterKey-like searching into
- Drupal sites.
+* [MKDru](http://www.indexdata.com/masterkey-drupal) --
+ a toolkit for embedding MasterKey-like searching into
+ [Drupal](https://www.drupal.org/)
+ sites.
All of these approaches require programming to a greater or lesser
-extent. Against this backdrop, we introduced MKWS (the MasterKey
-Widget Set) -- a set of simple, very high-level HTML+CSS+JavaScript
+extent. Against this backdrop, we introduced
+[MKWS (the MasterKey Widget Set)](http://mkws.indexdata.com/)
+-- a set of simple, very high-level HTML+CSS+JavaScript
components that can be incorporated into any web-site to provide
MasterKey searching facilities. By placing `<div>`s with well-known
-identifiers in any HTML page, the various components of an application
+MKWS classes in any HTML page, the various components of an application
can be embedded: search-boxes, results areas, target information, etc.
Simple Example
---------------
+==============
The following is a complete MKWS-based searching application:
<link rel="stylesheet" href="http://mkws.indexdata.com/mkws.css" />
</head>
<body>
- <div id="mkwsSearch"></div>
- <div id="mkwsResults"></div>
+ <div class="mkwsSearch"></div>
+ <div class="mkwsResults"></div>
</body>
</html>
-Go ahead, try it! You don't even need a web-server. Just copy and
-paste this HTML into a file on your computer -- `/tmp/magic.html`,
-say -- and point your web-browser at it:
-`file:///tmp/magic.html`. Just like that, you have working
-metasearching.
-
+Go ahead, try it! Simply put the above in a file (e.g index.html),
+drop it into a folder accessible with an ordinary web-server (e.g Apache)
+and load it in your web browser (and no, usually, you can't just load the file
+directly from disk as some browsers, e.g Chrome, won't allow storing cookies).
+Just like that, you have working metasearching.
How the example works
---------------------
<div id="mkwsTargets"></div>
<div id="mkwsStat"></div>
+
Configuration
--------------
+=============
Many aspects of the behaviour of MKWS can be modified by setting
parameters into the `mkws_config` object. **This must be done *before*
Control over HTML and CSS
--------------------------
+=========================
More sophisticated applications will not simply place the `<div>`s
together, but position them carefully within an existing page
reference guide below.
+Customised display using Handlebars templates
+=============================================
+
+A lot can be done by styling widgets in CSS and changing basic MKWS config
+options. For further customisation, MKWS allows you to change the markup it
+outputs for any widget. This is done by overriding the
+[Handlebars](http://handlebarsjs.com/) template used to generate it. In general
+these consist of `{{things in double braces}}` that are replaced by values from
+the system. For details of Handlebars template syntax, see [the online
+documentation](http://handlebarsjs.com/).
+
+The templates used by the core widgets can be viewed in [our git
+repository](http://git.indexdata.com/?p=mkws.git;a=tree;f=src/mkws.templates;).
+Parameters are documented in a comment at the top of each template so
+you can see what's going where. If all you want to do is add a CSS class to
+something or change a `span` to a `div` it's easy to just copy the existing
+template and make your edits.
+
+Overriding templates
+--------------------
+
+To override the template for a widget, include it inline in the document
+as a `<script>` tag marked with a class of `mkwsTemplate_Foo` where Foo is the
+name of the template you want to override (typically the name of the widget).
+Inline Handlebars templates are distinguished from Javascript via a
+`type="text/x-handlebars-template"` attribute. For example, to override the
+Pager template you would include this in your document:
+
+ <script class="mkwsTemplate_Pager" type="text/x-handlebars-template">
+ ...new Pager template
+ </script>
+
+The Facet template has a special feature where you can override it on a
+per-facet basis by adding a dash and the facet name as a suffix eg.
+`Facet-Subjects` rather than `Facet`. (So `class="mkwsTemplate_Facet-Subjects"`)
+
+You can also explicitly specify a different template for a particular instance
+of a widget by providing the name of your alternative (eg. SpecialPager) as the
+value of the `template` key in the MKWS config object for that widget:
+for example, `<div class="mkwsPager" template="specialPager"/>`.
+
+Templates for MKWS can also be
+[precompiled](http://handlebarsjs.com/precompilation.html). If a precompiled
+template of the same name is found in the `Handlebars.templates` object, it
+will be used instead of the default.
+
+Inspecting metadata for templating
+----------------------------------
+
+MKWS makes requests to Service Proxy or Pazpar2 that perform the actual
+searching. Depending on how these are configured and what is available from the
+targets you are searching there may be more data available than what is
+presented by the default templates. In this case, you can redefine the
+`Record` template to include more fields in the full-record popup.
+
+Handlebars offers a convenient log helper that will output the contents of a
+variable for you to inspect. This lets you look at exactly what is being
+returned by the back end without needing to use a Javascript debugger. For
+example, you might prepend `{{log hits}}` to the Records template in order to
+see what is being returned with each search result in the list. In order for
+this to work you'll need to enable verbose output from Handlebars which is done
+by including this line or similar:
+
+ <script>Handlebars.logger.level = 1;</script>
+
+Internationalisation
+--------------------
+
+If you would like your template to use the built in translation functionality,
+output locale specific text via the mkws-translate helper like so:
+`{{{mkws-translate "a few words"}}}`.
+
+Example
+-------
+
+Rather than use the included AJAX helpers to render record details inline,
+here's a Records template that will link directly to the source via the address
+provided in the metadata as the first element of `md-electronic-url`:
+
+ <script class="mkwsTemplate_Records" type="text/x-handlebars-template">
+ {{#each hits}}
+ <div class="{{containerClass}}">
+ <a href="{{md-electronic-url.[0]}}">
+ <b>{{md-title}}</b>
+ </a>
+ {{#if md-title-remainder}}
+ <span>{{md-title-remainder}}</span>
+ {{/if}}
+ {{#if md-title-responsibility}}
+ <span><i>{{md-title-responsibility}}</i></span>
+ {{/if}}
+ </div>
+ {{/each}}
+ </script>
+
+For a more involved example where markup for multiple widgets is decorated with
+[Bootstrap](http://getbootstrap.com/) classes and a custom Handlebars helper is
+employed, take a look at the source of
+[topic.html](http://example.indexdata.com/topic.html?q=water).
+
+
Refinements
------------
+===========
-### Message of the day
+Message of the day
+------------------
Some applications might like to open with content in the area that
will subsequently be filled with result-records -- a message of the
search is made.
-### Customised display using Handlebars templates
-
-Certain aspects of the widget-set's display can be customised by
-providing Handlebars templates with well-known classes that begin with
-the string `mkwsTemplate_`. At present, the supported templates are:
-
-* `mkwsTemplate_Summary` -- used for each summary record in a list of
- results.
-
-* `mkwsTemplate_Record` -- used when displaying a full record.
-
-For both of these the metadata record is passed in, and its fields can
-be referenced in the template. As well as the metadata fields
-(`md-*`), two special fields are provided to the `mkwsTemplate_Summary`
-template, for creating popup links for full records. These are `_id`,
-which must be provided as the `id` attribute of a link tag, and
-`_onclick`, which must be provided as the `onclick` attribute.
-
-For example, an application can install a simple author+title summary
-record in place of the usual one providing the following template:
-
- <script class="mkwsTemplate_Summary" type="text/x-handlebars-template">
- {{#if md-author}}
- <span>{{md-author}}</span>
- {{/if}}
- <a href="#" id="{{_id}}" onclick="{{_onclick}}">
- <b>{{md-title}}</b>
- </a>
- </script>
-
-For details of Handlebars template syntax, see
-[the online documentation](http://handlebarsjs.com/).
-
-
-### Responsive design
+Responsive design
+-----------------
Metasearching applications may need to appear differently on
small-screened mobile devices, or change their appearance when
termlists should appear.
-### Popup results with jQuery UI
+Popup results with jQuery UI
+----------------------------
The [jQuery UI library](http://en.wikipedia.org/wiki/JQuery_UI)
can be used to construct MKWS applications in which the only component
http://example.indexdata.com/index-popup.html
-### Authentication and target configuration
+Authentication and target configuration
+---------------------------------------
By default, MKWS configures itself to use a demonstration account on a
service hosted by mkws.indexdata.com. This account (username `demo`,
subscription resources, it's necessary to create an account with
Index Data's hosted service proxy, and protect that account with
authentication tokens (to prevent unauthorised use of subscription
-resources). For information on how to do this, see
-[MKWS Target Selection](library-configuration.html)
+resources). For information on how to do this, see the next section.
+
+
+MKWS Target Selection
+=====================
+
+MKWS accesses targets using the Pazpar2 metasearching engine. Although
+Pazpar2 can be used directly, using a statically configured set of
+targets, this usage is unusual. More often, Pazpar2 is fronted by the
+Service Proxy (SP), which manages authentication, sessions, target
+selection, etc.
+
+This document assumes the SP is used, and explains how to go about
+making a set of targets (a "library") available, how to connect your
+MKWS application to that library, and how to choose which of the
+available targets to use.
+
+
+Maintaining the library
+-----------------------
+
+The service proxy accesses sets of targets that are known as
+"libraries". In general, each customer will have their own library,
+though some standard libraries may be shared between many customers --
+for example, a library containing all open-access academic journals.
+A library can also contain other configuration information, including
+the set of categories by which targets are classified for the library.
+
+Libraries are maintained using MKAdmin (MasterKey
+Admin). Specifically, those used by MKWS are generally maintained on
+the "MKX Admin" installation at
+<http://mkx-admin.indexdata.com/console/>
+
+In general, Index Data will create a library for each customer, then
+give the customer a username/password pair that they can use to enter
+MKAdmin and administrate that library.
+
+Once logged in, customers can select which targets to include (from
+the list of several thousand that MKAdmin knows about), and make
+customer-specific modifications -- e.g. overriding the titles of the
+targets.
+
+Most importantly, customers' administrators can add authentication
+credentials that the Service Proxy will used on their behalf when
+accessing subscription resources -- username/password pairs or proxies
+to use for IP-based authentication. Note that **it is then crucial to
+secure the library from use by unauthorised clients**, otherwise the
+customer's paid subscriptions will be exploited.
+
+Access to libraries is managed by creating one or more "User Access"
+records in MKAdmin, under the tab of that name. Each of these records
+provides a combination of credentials and other data that allow an
+incoming MKWS client to be identified as having legitimate access to
+the library. The authentication process, described below, works by
+searching for a matching User Access record.
+
+
+Authenticating your MWKS application onto the library
+-----------------------------------------------------
+
+Some MKWS applications will be content to use the default library with
+its selection of targets. Most, though, will want to define their own
+library providing a different range of available targets. An important
+case is that of applications that authenticate onto subscription
+resources by means of back-end site credentials stored in MKAdmin:
+precautions must be taken so that such library accounts do not allow
+unauthorised access.
+
+Setting up such a library is a process of several stages.
+
+### Create the User Access account
+
+Log in to MKAdmin to add a User Access account for your library:
+
+* Go to <http://mkx-admin.indexdata.com/console/>
+* Enter the adminstrative username/password
+* Go to the User Access tab
+* Create an end-user account
+* Depending on what authentication method it be used, set the
+ User Access account's username and password, or referring URL, or
+ Service Proxy hostname, or IP-address range.
+
+If your MWKS application runs at a well-known, permanent address --
+<http://yourname.com/app.html>, say -- you can set the User Access
+record so that this originating URL is recognised by setting it into
+the "Referring URL" field.
+
+If your application accesses the Service Proxy by a unique virtual
+hostname -- yourname.sp-mkws.indexdata.com, say -- you can tie the use
+of this hostname to your library by setting the User Access record's
+"Host Name" field to name of the host where the SP is accessed. **Note
+that this is not secure, as other applications can use this virtual
+hostname to gain access to your library.**
+
+Or if your application's users are coming from a well-known range of
+IP-address space, you can enter the range in the "IP Ranges"
+field. The format of this field is as follows: it can contain any
+number of ranges, separated by commas; each range is either a single
+IP address or two addresses separated by a hyphen; each IP address is
+four small integers separated by periods. For example,
+`80.229.143.255-80.229.143.255, 5.57.0.0-5.57.255.255, 127.0.0.1`.
+
+Alternatively, your application can authenticate by username and
+password credentials. This is a useful approach in several situations,
+including when you need to specify the use of a different library from
+usual one. To arrange for this, set the username and password as a
+single string separated by a slash -- e.g. "mike/swordfish" -- into
+the User Access record's Authentication field.
+
+You can set multiple fields into a single User Access record; or
+create multiple User Access records. For example, a single User Access
+record can specify both a Referring URL a username/password pair that
+can be used when running an application from a different URL. But if
+multiple Referring URLs are needed, then each must be specified in its
+own User Access record.
+
+### Tell the application to use the library
+
+In the HTML of the application, tell MKWS to authenticate on to the
+Service Proxy. When referer-based or IP-based authentication is used,
+this is very simple:
+
+ <script type="text/javascript">
+ var mkws_config = { service_proxy_auth:
+ "//sp-mkws.indexdata.com/service-proxy/?command=auth&action=perconfig" };
+ </script>
+
+> TODO This should be the default setting: see **MKWS-251**.
+
+And ensure that access to the MWKS application is from the correct
+Referrer URL or IP-range.
+
+### (Optional): access by a different virtual hostname
+
+When hostname-based authentication is in use, it's necessary to access
+the Service Proxy as the correctly named virtual host. This can be
+done by setting the `service_proxy_auth` configuration item to a
+URL containing that hostname, such as
+`//yourname.sp-mkws.indexdata.com/service-proxy/?command=auth&action=perconfig`
+
+> TODO It should be possible to change just the hostname without
+> needing to repeat the rest of the URL (protocol, path, query): see
+> **MKWS-252**.
+
+> TODO When changing the SP authentication URL, the Pazpar2 URL should
+> in general change along with it: see **MKWS-253**.
+
+### (Optional): embed credentials for access to the library
+
+When credential-based authentication is in use (username and
+password), it's necessary to pass these credentials into the Service
+Proxy when establishing the session. This can most simply be done just
+by setting the `service_proxy_auth` configuration item to a URL such as
+`//sp-mkws.indexdata.com/service-proxy/?command=auth&action=perconfig&username=mike&password=swordfish`
+
+> TODO It should be possible to add the username and password to the
+> configuration without needing to repeat the rest of the URL: see
+> **MKWS-254**.
+
+### (Optional): conceal credentials from HTML source
+
+Using a credential-based Service-Proxy authentication URL such as the
+one above reveals the the credentials to public view -- to anyone who
+does View Source on the MKWS application. This may be acceptable for
+some libraries, but is intolerable for those which provide
+authenticated access to subscription resources.
+
+In these circumstances, a more elaborate approach is necessary. The
+idea is to make a URL local to the customer that is used for
+authentication onto the Service Proxy, hiding the credentials in a
+local rewrite rule. Then local mechanisms can be used to limit access
+to that local authentication URL. Here is one way to do it when
+Apache2 is the application's web-server, which we will call
+yourname.com:
+
+Step 1: add a rewriting authentication alias to the configuration:
+
+ RewriteEngine on
+ RewriteRule /spauth/ http://sp-mkws.indexdata.com/service-proxy/?command=auth&action=check,login&username=U&password=PW [P]
+
+Step 2: set the MKWS configuration item `service_proxy_auth` to
+<http://yourname.com/spauth/>
+
+Step 3: protect access to the local path <http://yourname.com/spauth/>
+(e.g. using a `.htaccess` file).
+
+
+Choosing targets from the library
+---------------------------------
+
+MKWS applications can choose what subset of the library's targets to
+use, by means of several alternative settings on individual widgets or
+in the `mkws_config` structure:
+
+* `targets` -- contains a Pazpar2 targets string, typically of the form
+ "pz:id=" or "pz:id~" followed by a pipe-separated list of low-level
+ target IDs.
+ At present, these IDs can take one of two forms, depending on the
+ configuration of the Service Proxy being used: they may be based on
+ ZURLs (so a typical value would be something like
+ `pz:id=josiah.brown.edu:210/innopac|lui.indexdata.com:8080/solr4/select?fq=database:4902`)
+ or they may be UDBs (so a typical value would be something like
+ `pz:id=brown|artstor`)
+
+* `targetfilter` -- contains a CQL query which is used to find relevant
+ targets from the relvant library. For example,
+ `udb==Google_Images`
+ or
+ `categories=news`
+
+* `target` -- contains a single UDB, that of the sole target to be
+ used. For example,
+ `Google_Images`.
+ This is merely syntactic sugar for "targetfilter" with the query
+ `udb==NAME`
+
+For example, a `Records` widget can be limited to searching only in
+targets that have been categorised as news sources by providing an
+attribute as follows:
+
+ <div class="mkwsRecords" targetfilter='categories=news'/>
Reference Guide
----------------
+===============
-### Configuration object
+Configuration object
+--------------------
The configuration object `mkws_config` may be created before including
the MKWS JavaScript code to modify default behaviour. This structure
to lightly customise the display than my changing providing a full HTML
structure.
-#### Notes
+### Notes
1. ["sources", "subjects", "authors"]
3. [10, 20, 30, 50]
-4. http://mkws.indexdata.com/service-proxy-auth
+4. http://sp-mkws.indexdata.com/service-proxy-auth
-5. http://mkws.indexdata.com/service-proxy/
+5. http://sp-mkws.indexdata.com/service-proxy/
6. [["relevance"], ["title:1", "title"], ["date:0", "newest"], ["date:1", "oldest"]]
-### Language specification
+Language specification
+----------------------
Support for another UI language can be added by providing an entry in
the `mkws_config` object whose name is `language_` followed by the
-### jQuery UI popup invocation
+jQuery UI popup invocation
+--------------------------
The MasterKey Widget Set can be invoked in a popup window on top of the page.
</div>
----
-Element Type Default Description
--------- ----- --------- ------------
-popup_width string 880 Width of the popup window (if used), in
- pixels.
+Element Type Default Description
+-------- ----- ------- ------------
+popup_width string 880 Width of the popup window (if used), in
+ pixels.
-popup_height string 760 Height of the popup window (if used), in
- pixels.
+popup_height string 760 Height of the popup window (if used), in
+ pixels.
-popup_button string input.mkwsButton (Never change this.)
+popup_button string `input.mkwsButton` (Never change this.)
-popup_modal string 0 Modal confirmation mode. Valid values are 0 or 1
+popup_modal string 0 Modal confirmation mode. Valid values are 0 or 1
-popup_autoOpen string 1 Open popup window on load. Valid values are 0 or 1
+popup_autoOpen string 1 Open popup window on load. Valid values are 0 or 1
----
-### The structure of the HTML generated by the MKWS widgets
+The structure of the HTML generated by the MKWS widgets
+-------------------------------------------------------
In order to override the default CSS styles provided by the MasterKey Widget
Set, it's necessary to understand that structure of the HTML elements that are
- - -
-Copyright (C) 2013-2014 by IndexData ApS, <http://www.indexdata.com>
+Copyright (C) 2013-2014 by Index Data ApS, <http://www.indexdata.com>
projects / mkws-moved-to-github.git / blobdiff