<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:851920384;
mso-list-type:hybrid;
mso-list-template-ids:1785086330 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:.75in;
text-indent:-.25in;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:1.25in;
text-indent:-.25in;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:1.75in;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:2.25in;
text-indent:-.25in;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:2.75in;
text-indent:-.25in;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:3.25in;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:3.75in;
text-indent:-.25in;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:4.25in;
text-indent:-.25in;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:4.75in;
text-indent:-9.0pt;}
@list l1
{mso-list-id:1788500026;
mso-list-type:hybrid;
mso-list-template-ids:-382318124 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l1:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:.75in;
text-indent:-.25in;}
@list l1:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:1.25in;
text-indent:-.25in;}
@list l1:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:1.75in;
text-indent:-9.0pt;}
@list l1:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:2.25in;
text-indent:-.25in;}
@list l1:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:2.75in;
text-indent:-.25in;}
@list l1:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:3.25in;
text-indent:-9.0pt;}
@list l1:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:3.75in;
text-indent:-.25in;}
@list l1:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:4.25in;
text-indent:-.25in;}
@list l1:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:4.75in;
text-indent:-9.0pt;}
@list l2
{mso-list-id:1849521564;
mso-list-type:hybrid;
mso-list-template-ids:1481964502 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l2:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l2:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l2:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l2:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l2:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l2:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l2:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l2:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l2:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">Dear Johanna,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks for your question about this, and apologies for the delay in responding. I’ve been participating in work travel and conference activity the last two weeks and am just now catching up with listserv posts.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Improving the public interface search was something that we investigated a great deal over the spring and early summer based on feedback from a number of institutions using the PUI. Unfortunately, we determined that making the changes required
will necessitate a substantial change to the indexing for the application. We’re working to identify and obtain resources in order to do so while maintaining forward progress in other areas of the application.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">How the search on the public side currently works is documented only in technical terms. I’ve distilled what we know down for this purpose, but the explanation is still rather technical. If there are additional questions on the specifics,
I’m happy to try to answer them, but this is definitely something that I lean on Laney and others on the developer side for better understanding. (And any mistakes in interpretation in what’s below are mine.) Here is some information about how the PUI search
currently indexes and weights information in order to display results:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l2 level1 lfo1">ArchivesSpace has multiple indexers (one each essentially for staff side information, public side information, and a real-time indexer that updates the index as changes are made) but
all three put their information into one shared index. There is a field called fullrecord which takes nearly all the fields in ArchivesSpace and makes them a single field for the purposes of keyword search. PUI indexes fullrecord plus more for the collection
organization display. The code that creates the staff interface records is the same as what is used by the PUI indexer with some additions for the separate PUI records.
<b><o:p></o:p></b></li></ul>
<p class="MsoListParagraph">Because there is only one index currently there is only one fullrecord field rather than one for staff and one for public as you might expect. Everything pulling from one index that includes a field for almost everything in ArchivesSpace
is one of the reasons why information that is not displayed in the public interface affects public interface results.<b><o:p></o:p></b></p>
<p class="MsoListParagraph"><b><o:p> </o:p></b></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l2 level1 lfo1">Anything that appears in the fields included in fullrecord is included in the index and available to the public and staff sides, though what displays is determined by other settings
in the views. (This is why unpublished records rightly don’t appear in the PUI though they can affect search results.) On the public side, the most heavily weighted fields are identifier, title, and finding aid title, but the results in record types that are
resources and accessions are lifted highest, then agents and subjects.<o:p></o:p></li></ul>
<p class="MsoListParagraph">For more specifics, the values after the ^ show the magnitude of the weighting.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.25in;text-indent:.25in">Currently, these are hard-coded in the solrconfig.xml file and the solr model in the backend:
<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">From solrconfig.xml:<o:p></o:p></p>
<ol style="margin-top:0in" start="1" type="1">
<li class="MsoListParagraph" style="margin-left:.25in;mso-list:l0 level1 lfo2">pf = “four_part_id^50” (pf is for Phrase Fields which boosts the score of documents in cases where all of the terms in the q parameter appear in close proximity)<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:.25in;mso-list:l0 level1 lfo2">qf = “title^25 four_part_id^50 fullrecord” (qf is for Query Fields which specifies the fields in the index on which to perform the query)<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:.25in;mso-list:l0 level1 lfo2">bq = “primary_type:resource^100 primary_type:accession^100 primary_type:subject^50 primary_type:agent_person^50 primary_type:agent_corporate_entity^30 primary_type:agent_family^30”
(bq is for Boost Query which specifies a factor by which a term or phrase should be "boosted" in importance when considering a match)<o:p></o:p></li></ol>
<p class="MsoNormal" style="margin-left:.5in">Passed into the solr query from solr model in the backend:<o:p></o:p></p>
<ol style="margin-top:0in" start="1" type="1">
<li class="MsoListParagraph" style="margin-left:.25in;mso-list:l1 level1 lfo3">pf = "four_part_id^4"<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:.25in;mso-list:l1 level1 lfo3">qf = "four_part_id^3 title^2 finding_aid_filing_title^2 fullrecord"<o:p></o:p></li></ol>
<p class="MsoListParagraph"><b><o:p> </o:p></b></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l2 level1 lfo1">There were some changes made in some v2.3.x and v2.4.x releases of ArchivesSpace that made some parameters, such as whether the default operator is OR or AND, configurable, but they
only work on the staff side because of how the PUI works. Changing the operator does not work on the public side because the code for the public side overwrites some areas when the final solr query gets built before it is sent to solr for retrieval. Also,
there are some subqueries that are created in the PUI search that have AND and OR hardcoded so the final query contains a combination of ORs and ANDs. That is not configurable at all. Yale (and possibly Harvard as well, though Johanna would have a better sense
of this) has done some work to modify search for its own purposes but I believe their changes have been scaled back significantly as they saw what we saw in investigating this – as currently set up, making a change in one area negatively impacts search in
another area, including the staff interface.<o:p></o:p></li></ul>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We believe the only possibility for making substantial, lasting change to the PUI search is to refactor how search happens. This is a major undertaking, and it’s very important to us that doing so not negatively impact how people use the
PUI or the staff interface now or stop all progress on development in general for a significant period of time. Taking the time to identify ways to do this, determining the best path forward, and finding resources to pursue it is the reason we have not progressed
with PUI search the way we were hoping earlier in the year.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We are incredibly fortunate that ArchivesSpace has such an active and engaged user community and that the application has become so fundamental to people’s work. We take very seriously the degree to which making significant changes to it
would impact people’s work and want to pursue any such development in as thoughtful and responsible a way as we can. As plans progress we will involve the community in the discussions as they relate to PUI search specifically.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I hope knowing more about how the search currently works helps and please do reach out if you would like to discuss more before we reach that point.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Christine<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:#1F497D">Christine Di Bella<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:#1F497D">ArchivesSpace Program Manager<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:#1F497D"><a href="mailto:christine.dibella@lyrasis.org">christine.dibella@lyrasis.org</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:#1F497D">800.999.8558 x2905<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:#1F497D">678-235-2905<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:#1F497D">cdibella13 (Skype)</span><span style="color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><img border="0" width="267" height="61" style="width:2.7777in;height:.6388in" id="Picture_x0020_1" src="cid:image002.jpg@01D4596B.729E59A0" alt="ASpaceOrgHomeMedium"><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Carll, Johanna <jcarll@radcliffe.harvard.edu> <br>
<b>Sent:</b> Monday, September 17, 2018 9:33 AM<br>
<b>To:</b> Christine Di Bella <christine.dibella@lyrasis.org>; Archivesspace Users Group <archivesspace_users_group@lyralists.lyrasis.org><br>
<b>Subject:</b> PUI search functionality documentation <o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:black">Dear Christine,</span><span style="font-size:12.0pt;color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal">Now that we have had a few months of experience with the ArchivesSpace PUI here at Harvard, we are reviewing user feedback to help us prioritize post-launch development needs. One area of concern is the PUI search functionality, as we’ve
received multiple reports of unsatisfactory and unexpected search results.<o:p></o:p></p>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black">Can you direct us to – or share - </span><span style="color:red"> </span><span style="color:black">documentation on the PUI search functionality, including relevance ranking, weighting, and indexed fields? This
will help us evaluate what may be done locally to improve results, as well as participate in the discussion and planning for changes to the core code that would improve search results. <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:black">Thanks<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black">Johanna</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Johanna Carll<o:p></o:p></p>
<p class="MsoNormal">Archivist and Metadata Specialist<o:p></o:p></p>
<p class="MsoNormal">Schlesinger Library<o:p></o:p></p>
<p class="MsoNormal">Radcliffe Institute for Advanced Study Harvard University<o:p></o:p></p>
<p class="MsoNormal">10 Garden Street<o:p></o:p></p>
<p class="MsoNormal">Cambridge, MA 02138<o:p></o:p></p>
<p class="MsoNormal">617-495-8524<o:p></o:p></p>
<p class="MsoNormal"><a href="mailto:jcarll@radcliffe.harvard.edu"><span style="color:blue">jcarll@radcliffe.harvard.edu</span></a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>