<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.hoenzb
{mso-style-name:hoenzb;}
span.EmailStyle21
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle22
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle24
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Hello,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">We had the same problems with the index being corrupted after upgrading from version 1.5.1 to 2.0.1. Our server has a lot of RAM, but only one CPU. After, backing up our
mysql database, I stopped the application, blew away the entire </span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">/path/to/archivespace/data directory, and restarted the application, which rebuilt our indexes. Performance improved, but
the application is still running slower than it did before on version 1.5.1. We are experimenting now with moving the mysql database and/or solr index as well as moving the application to the Amazon AWS cloud. Hopefully in the future, as the ArchivesSpace
software is improved, it will run more efficiently and require fewer resources.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">-Ryan</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D">Ryan Edwards<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D">Digital Access and Systems Librarian<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D">Information Systems</span><span style="color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D">Getty Research Institute<br>
1200 Getty Center Drive, Suite 1100<br>
Los Angeles, CA 90049-1688<br>
<a href="mailto:reedwards@getty.edu"><span style="color:#0563C1">reedwards@getty.edu</span></a><br>
310.440.7398</span><span style="color:#1F497D"><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> archivesspace_users_group-bounces@lyralists.lyrasis.org [mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org]
<b>On Behalf Of </b>Christine Di Bella<br>
<b>Sent:</b> Tuesday, July 11, 2017 8:30 AM<br>
<b>To:</b> Archivesspace Users Group <archivesspace_users_group@lyralists.lyrasis.org><br>
<b>Subject:</b> Re: [Archivesspace_Users_Group] Indexing woes in v 2.0.1<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Hi Celia and Chelsea,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Since it sounds like this may be fairly specific to your setup and data, I’m going to put a tech support ticket in for you (you’ll see a separate message directly to you about
that in a minute). When it gets worked out, it’d be great if you’d post the solution back to the list.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Christine<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:#1F497D">Christine Di Bella<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:#1F497D">ArchivesSpace Program Manager<o:p></o:p></span></p>
<p class="MsoNormal"><a href="mailto:christine.dibella@lyrasis.org"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:#0563C1">christine.dibella@lyrasis.org</span></a><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:#1F497D">800.999.8558 x2905<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:#1F497D">678-235-2905<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:#1F497D">cdibella13 (Skype)</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><img border="0" width="267" height="61" id="Picture_x0020_1" src="cid:image002.jpg@01D2FA23.55A6CD80" alt="ASpaceOrgHomeMedium"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">
<a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org">archivesspace_users_group-bounces@lyralists.lyrasis.org</a> [<a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org">mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org</a>]
<b>On Behalf Of </b>Chelsea Lobdell<br>
<b>Sent:</b> Tuesday, July 11, 2017 10:33 AM<br>
<b>To:</b> Archivesspace Users Group <<a href="mailto:archivesspace_users_group@lyralists.lyrasis.org">archivesspace_users_group@lyralists.lyrasis.org</a>><br>
<b>Subject:</b> Re: [Archivesspace_Users_Group] Indexing woes in v 2.0.1<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">To add to Celia's post, we originally planned an upgrade from v1.5.3 to v2.0.1 because our Archivesspace instance was experience extreme periods of slowness with CPU usage going upwards of 95-98%. The log was
full of Java out of memory errors. Following the upgrade we increased the RAM on the server from 2G to 4G and increased the CPUs from 2 to 4 as well.
<o:p></o:p></p>
</div>
<p class="MsoNormal">When indexing issues were first reported I ran scripts/checkindex.sh data/solr_index/index which reported no issues with the index. However, it was evident that there was something a miss with the solr index. Which led to the deletion of
the following directories to for an entire re-index: <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.5pt">/path/to/archivespace<b>/data/solr_index/index/</b><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.5pt">/path/to/archivesspace<b>/data/indexer_pui_state</b><br>
/path/to/archivesspace<b>/data/indexer_state</b><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.5pt"><o:p> </o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.5pt">As Celia notes, this corrected issues for items that were added / modified previously but going forward addition / modification / deletions are not
behaving properly in search and browse. <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.5pt"><o:p> </o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.5pt">Also, the Solr log continue to be full of sever errors that all follow the pattern below:<o:p></o:p></span></p>
<pre>org.apache.solr.common.SolrException: Exception writing document id /repositories/6/resources/8766 to the index; possible analysis error.<o:p></o:p></pre>
<pre> at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:168)<o:p></o:p></pre>
<pre> at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)<o:p></o:p></pre>
<pre> ...<o:p></o:p></pre>
<pre>Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed<o:p></o:p></pre>
<pre> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:700)<o:p></o:p></pre>
<pre> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:714)<o:p></o:p></pre>
<pre> ... 40 more<o:p></o:p></pre>
<pre style="margin-bottom:12.0pt">Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded<o:p></o:p></pre>
<pre style="margin-bottom:12.0pt">I've been using JVM Top to monitor heap, GC, and CPU but so far I have not been able to draw any helpful conclusion from the data. <o:p></o:p></pre>
<pre style="margin-bottom:12.0pt">Any advice / suggestions on what to try from here would be greatly appreciated. <o:p></o:p></pre>
<pre>- Chelsea<o:p></o:p></pre>
</div>
</div>
<div>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><em><span style="color:gray">---------------</span></em><br>
<em><span style="color:gray">Chelsea Lobdell</span></em><br>
<em><span style="color:gray">Library Web Developer/ Swarthmore College</span></em><br>
<a href="mailto:clobdel1@swarthmore.edu" target="_blank"><i>clobdel1@swarthmore.edu</i></a><em><span style="color:gray"> / (610)690-6818</span></em><o:p></o:p></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Tue, Jul 11, 2017 at 9:37 AM, Celia Caust-Ellenbogen <<a href="mailto:ccauste1@swarthmore.edu" target="_blank">ccauste1@swarthmore.edu</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">Hello all,<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">We just upgraded to v 2.0.1 and now the indexer doesn't seem to be working properly - not in the staff interface nor in the PUI. We can create and change records, and when you are viewing the individual record it is correctly updated in
both Public and Staff. However, these changes are not reflected in the browse/search in either Public or Staff. Old deleted records are still showing up in browse/search while new records cannot be found in browse/search.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Any ideas for how to fix this?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Following a suggestion on this listserv, we already tried forcing a re-index by stopping the application, deleting the following directories, and restarting Archivesspace <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.5pt">/path/to/archivespace<b>/data/solr_index/index/</b><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.5pt">/path/to/archivesspace<b>/data/indexer_pui_state</b><br>
/path/to/archivesspace<b>/data/indexer_state</b><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">This updated the index in terms of old activity, but new activity is still not being captured.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">We would appreciate any suggestions for what to try next.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks!<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#888888">Celia<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#888888"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span class="hoenzb"><span style="color:#888888">-- </span><o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:7.5pt;color:#888888">Celia Caust-Ellenbogen </span><o:p></o:p></p>
<div>
<p class="MsoNormal"><a href="http://swarthmore.edu/friends-historical-library" target="_blank"><span style="font-size:7.5pt">Friends Historical Library of Swarthmore College</span></a><span style="color:#888888"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><a href="tel:(610)%20328-8498" target="_blank"><span style="font-size:7.5pt">610-328-8498</span></a><span style="color:#888888"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><a href="mailto:ccauste1@swarthmore.edu" target="_blank"><span style="font-size:7.5pt">ccauste1@swarthmore.edu</span></a><span style="color:#888888"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#888888"><o:p> </o:p></span></p>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
Archivesspace_Users_Group mailing list<br>
<a href="mailto:Archivesspace_Users_Group@lyralists.lyrasis.org">Archivesspace_Users_Group@lyralists.lyrasis.org</a><br>
<a href="http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group" target="_blank">http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>