[Archivesspace_Users_Group] Indexing woes in v 2.0.1

Tue Jul 11 12:02:39 EDT 2017

Hello,

We had the same problems with the index being corrupted after upgrading from version 1.5.1 to 2.0.1.  Our server has a lot of RAM, but only one CPU.  After, backing up our mysql database, I stopped the application, blew away the entire /path/to/archivespace/data directory, and restarted the application, which rebuilt our indexes.  Performance improved, but the application is still running slower than it did before on version 1.5.1.  We are experimenting now with moving the mysql database and/or solr index as well as moving the application to the Amazon AWS cloud.  Hopefully in the future, as the ArchivesSpace software is improved, it will run more efficiently and require fewer resources.

-Ryan

Ryan Edwards
Digital Access and Systems Librarian
Information Systems
Getty Research Institute
1200 Getty Center Drive, Suite 1100
Los Angeles, CA 90049-1688
reedwards at getty.edu<mailto:reedwards at getty.edu>
310.440.7398

From: archivesspace_users_group-bounces at lyralists.lyrasis.org [mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org] On Behalf Of Christine Di Bella
Sent: Tuesday, July 11, 2017 8:30 AM
To: Archivesspace Users Group <archivesspace_users_group at lyralists.lyrasis.org>
Subject: Re: [Archivesspace_Users_Group] Indexing woes in v 2.0.1

Hi Celia and Chelsea,

Since it sounds like this may be fairly specific to your setup and data, I’m going to put a tech support ticket in for you (you’ll see a separate message directly to you about that in a minute). When it gets worked out, it’d be great if you’d post the solution back to the list.

Christine

Christine Di Bella
ArchivesSpace Program Manager
christine.dibella at lyrasis.org<mailto:christine.dibella at lyrasis.org>
800.999.8558 x2905
678-235-2905
cdibella13 (Skype)

[ASpaceOrgHomeMedium]

From: archivesspace_users_group-bounces at lyralists.lyrasis.org<mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org> [mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org] On Behalf Of Chelsea Lobdell
Sent: Tuesday, July 11, 2017 10:33 AM
To: Archivesspace Users Group <archivesspace_users_group at lyralists.lyrasis.org<mailto:archivesspace_users_group at lyralists.lyrasis.org>>
Subject: Re: [Archivesspace_Users_Group] Indexing woes in v 2.0.1

To add to Celia's post, we originally planned an upgrade from v1.5.3 to v2.0.1 because our Archivesspace instance was experience extreme periods of slowness with CPU usage going upwards of 95-98%. The log was full of Java out of memory errors. Following the upgrade we increased the RAM on the server from 2G to 4G and increased the CPUs from 2 to 4 as well.
When indexing issues were first reported I ran scripts/checkindex.sh data/solr_index/index which reported no issues with the index. However, it was evident that there was something a miss with the solr index. Which led to the deletion of the following directories to for an entire re-index:
/path/to/archivespace/data/solr_index/index/
/path/to/archivesspace/data/indexer_pui_state
/path/to/archivesspace/data/indexer_state

As Celia notes, this corrected issues for items that were added / modified previously but going forward addition / modification / deletions are not behaving properly in search and browse.

Also, the Solr log continue to be full of sever errors that all follow the pattern below:

org.apache.solr.common.SolrException: Exception writing document id /repositories/6/resources/8766 to the index; possible analysis error.

        at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:168)

        at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)

        ...

Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed

        at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:700)

        at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:714)

        ... 40 more

Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

I've been using JVM Top to monitor heap, GC, and CPU but so far I have not been able to draw any helpful conclusion from the data.

Any advice / suggestions on what to try from here would be greatly appreciated.

- Chelsea

---------------
Chelsea Lobdell
Library Web Developer/ Swarthmore College
clobdel1 at swarthmore.edu<mailto:clobdel1 at swarthmore.edu> / (610)690-6818

On Tue, Jul 11, 2017 at 9:37 AM, Celia Caust-Ellenbogen <ccauste1 at swarthmore.edu<mailto:ccauste1 at swarthmore.edu>> wrote:
Hello all,

We just upgraded to v 2.0.1 and now the indexer doesn't seem to be working properly - not in the staff interface nor in the PUI. We can create and change records, and when you are viewing the individual record it is correctly updated in both Public and Staff. However, these changes are not reflected in the browse/search in either Public or Staff. Old deleted records are still showing up in browse/search while new records cannot be found in browse/search.

Any ideas for how to fix this?

Following a suggestion on this listserv, we already tried forcing a re-index by stopping the application, deleting the following directories, and restarting Archivesspace
/path/to/archivespace/data/solr_index/index/
/path/to/archivesspace/data/indexer_pui_state
/path/to/archivesspace/data/indexer_state

This updated the index in terms of old activity, but new activity is still not being captured.

We would appreciate any suggestions for what to try next.

Thanks!
Celia

--
Celia Caust-Ellenbogen
Friends Historical Library of Swarthmore College<http://swarthmore.edu/friends-historical-library>
610-328-8498<tel:(610)%20328-8498>
ccauste1 at swarthmore.edu<mailto:ccauste1 at swarthmore.edu>

_______________________________________________
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group at lyralists.lyrasis.org<mailto:Archivesspace_Users_Group at lyralists.lyrasis.org>
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lyralists.lyrasis.org/pipermail/archivesspace_users_group/attachments/20170711/cee9db51/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.jpg
Type: image/jpeg
Size: 4073 bytes
Desc: image002.jpg
URL: <http://lyralists.lyrasis.org/pipermail/archivesspace_users_group/attachments/20170711/cee9db51/attachment.jpg>