[Archivesspace_Users_Group] Understanding why ArchivesSpace seems to have died

[Tom Hanstra]

I never was able to complete my indexing of PUI and now it looks as if
ArchivesSpace has simply stopped working for me. I'm trying to understand
what happened.

I have tried to break out the various logs, though doing so has not worked
as expected.  All PUI activity was going to the indexer log and the end of
that log shows the staff log only:

I, [2021-03-12T16:46:07.137484 #2919]  INFO -- : Thread-2868: Staff Indexer
[2021-03-12 16:46:07 -0500] Index round complete
I, [2021-03-12T16:46:40.315017 #2919]  INFO -- : Thread-2868: Staff Indexer
[2021-03-12 16:46:40 -0500] Running index round

I have to go back farther to see the last PUI update:

I, [2021-03-12T15:37:18.971100 #2919]  INFO -- : Thread-2932: PUI Indexer
[2021-03-12 15:37:18 -0500] ~~~ Indexed 760150 of 763368 archival_object
records in repository UNDA
I, [2021-03-12T15:41:27.314764 #2919]  INFO -- : Thread-2932: PUI Indexer
[2021-03-12 15:41:27 -0500] ~~~ Indexed 760200 of 763368 archival_object
records in repository UNDA

At the end, that index had gone to a crawl, only indexing about 400
records/hour.

While the indexer log continued through the 16:46 time stamp, the pui and
archivespace.out logs both stop at 16:10. Here is the last thing in the PUI
log:

I, [2021-03-12T16:10:06.564227 #2919]  INFO -- :
[06eac683-aff7-4afe-9046-97e5551b94a9]   Rendered shared/_header.html.erb
(14.3ms)
I, [2021-03-12T16:10:06.570258 #2919]  INFO -- :
[06eac683-aff7-4afe-9046-97e5551b94a9]   Rendered
shared/_navigation.html.erb (4.6ms)
I, [2021-03-12T16:10:06.572299 #2919]  INFO -- :
[06eac683-aff7-4afe-9046-97e5551b94a9]   Rendered shared/_footer.html.erb
(0.4ms)
I, [2021-03-12T16:10:06.573882 #2919]  INFO -- :
[06eac683-aff7-4afe-9046-97e5551b94a9] Completed 404 Not Found in 2930ms
(Views: 79.4ms)

and in the archivesspace.out log:

INFO: [collection1] webapp= path=/update params={}
{add=[/repositories/2/archival_objects/1484673#pui,
/repositories/2/archival_objects/1484674#pui,
/repositories/2/archival_objects/1484675#pui,
/repositories/2/archival_objects/1484676#pui,
/repositories/2/archival_objects/1484677#pui,
/repositories/2/archival_objects/1484678#pui,
/repositories/2/archival_objects/1484679#pui,
/repositories/2/archival_objects/1484680#pui,
/repositories/2/archival_objects/1484681#pui,
/repositories/2/archival_objects/1484682#pui, ... (50 adds)]} 0 48
Mar 12, 2021 3:42:27 PM org.apache.solr.update.DirectUpdateHandler2 commit
INFO: start
commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
Mar 12, 2021 3:42:27 PM org.apache.solr.core.SolrDeletionPolicy onCommit
INFO: SolrDeletionPolicy.onCommit: commits: num=2
commit{dir=NRTCachingDirectory(MMapDirectory@/home/app/archivesspace/data/solr_index/index
lockFactory=NativeFSLockFactory@/home/app/archivesspace/data/solr_index/index;
maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_bx5,generation=15449}
commit{dir=NRTCachingDirectory(MMapDirectory@/home/app/archivesspace/data/solr_index/index
lockFactory=NativeFSLockFactory@/home/app/archivesspace/data/solr_index/index;
maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_bx6,generation=15450}
Mar 12, 2021 3:42:27 PM org.apache.solr.core.SolrDeletionPolicy
updateCommits
INFO: newest commit generation = 15450
Mar 12, 2021 3:42:27 PM org.apache.solr.search.SolrIndexSearcher <init>
INFO: Opening Searcher at 20bc5398[collection1] realtime
Mar 12, 2021 3:42:27 PM org.apache.solr.update.DirectUpdateHandler2 commit
INFO: end_commit_flush
Mar 12, 2021 4:10:06 PM org.apache.solr.core.SolrCore execute
INFO: [collection1] webapp= path=/select
params={mm=100%25&q=(uri:("\/repositories\/\-1\/actuator\/\-1"))&csv.escape=\&csv.header=true&start=0&csv.encapsulator="&fq=-exclude_by_default:true&fq=publish:true&fq=types:pui&rows=1&wt=json&facet=true}
hits=0 status=0 QTime=140

Neither the Staff nor Public interface are currently responding right now.

How can I tell if this fully died and, if it did, why it died?  Should I
restart?

Thanks for any help anyone might have. This is rather frustrating!

Tom



-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hanstra at nd.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lyralists.lyrasis.org/pipermail/archivesspace_users_group/attachments/20210312/aa27b964/attachment.html>