<div dir="ltr">Thanks again. <br><br>So, I currently break up my logs into four different individual logs with these settings:<div><br>## Possible Log levels: debug, info, warn, error, fatal (severe only)<br>#<br>AppConfig[:frontend_log] = "/var/log/archivesspace/frontend.log"<br>AppConfig[:frontend_log_level] = "debug"<br>#<br>AppConfig[:backend_log] = "/var/log/archivesspace/backend.log"<br>AppConfig[:backend_log_level] = "debug"<br>#<br>AppConfig[:pui_log] = "/var/log/archivesspace/pui.log"<br>AppConfig[:pui_log_level] = "debug"<br>#<br>AppConfig[:indexer_log] = "/var/log/archivesspace/indexer.log"<br>AppConfig[:indexer_log_level] = "debug"<br><br>In each of the other logs I see both INFO and DEBUG statements, sometimes a lot, other times fewer. But in the indexer.log file, I only see INFO statements, no DEBUG. <br><br>But your suggestion above that the line is like "indexed ..." got me searching other logs and I found a couple of lines like this in my backend log:<br><br>D, [2022-05-16T14:31:45.734159 #5273] DEBUG -- : Thread-4912: Responded with [200, {"Content-Type"=>"application/json", "Cache-Control"=>"private, must-revalidate, max-age=0", "Content-Length"=>"21295"}, ["{\"page_size\":1,\"first_page\":1,\"last_page\":1,\"this_page\":1,\"offset_first\":1,\"offset_last\":1,\"total_hits\":1,\"results\":[{\"id\":\"/repositories/2/archival_objects/1702566#pui\",\"uri\":\"/repositories/2/archival_objects/1702566\",\"title\":\"Brother Paul Hermit CSC not indexed.\",\"primary_type\":\"archival_object\",\"types\":[\"archival_object\",\"pui\",\"pui_archival... in 73ms<br><br>Is this the type of thing I should continue to look for?</div><div><br></div><div>Thanks,</div><div>Tom</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, May 18, 2022 at 4:12 PM Blake Carver <<a href="mailto:blake.carver@lyrasis.org">blake.carver@lyrasis.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
> <span style="font-size:14px;background-color:rgb(255,255,255);display:inline">- Even though the config setting
is for DEBUG, the indexer log is only showing INFO. Should there be more than that?</span><br style="font-size:14px;background-color:rgb(255,255,255)">
</div>
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Yep. Something must not be quite right in the config, you should see DEBUG showing up in the logs.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
If it's running 1x1 AND on DEBUG you should see something better by the crash, you should be able to spot the record. I can't remember what exactly it looks like, but something like "indexed some/path/with/an/id/in/it/here" and then you can see the Resource
or Archival Object ID in the line there. </div>
<div id="gmail-m_-9140078466833503249Signature">
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
</div>
<div></div>
<div></div>
<div id="gmail-m_-9140078466833503249divtagdefaultwrapper" dir="ltr" style="color:rgb(0,0,0);background-color:rgb(255,255,255)">
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;margin:0px">
<br>
</div>
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;margin:0px">
<br>
</div>
</div>
</div>
</div>
</div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_-9140078466833503249divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> <a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org" target="_blank">archivesspace_users_group-bounces@lyralists.lyrasis.org</a> <<a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org" target="_blank">archivesspace_users_group-bounces@lyralists.lyrasis.org</a>> on behalf of Tom Hanstra
<<a href="mailto:hanstra@nd.edu" target="_blank">hanstra@nd.edu</a>><br>
<b>Sent:</b> Wednesday, May 18, 2022 3:21 PM<br>
<b>To:</b> Archivesspace Users Group <<a href="mailto:archivesspace_users_group@lyralists.lyrasis.org" target="_blank">archivesspace_users_group@lyralists.lyrasis.org</a>><br>
<b>Subject:</b> Re: [Archivesspace_Users_Group] (re)indexing in 2.8.1</font>
<div> </div>
</div>
<div>
<div dir="ltr">Blake and others,
<div><br>
</div>
<div>Overall, we went for a week or so with no errors in terms of re-indexing. I'm looking into some database timeouts which are occurring periodically (AS and our database server are in two different locations) but with the connection
getting re-established, I can work to address that separately.<br>
<br>
But last night we did run into some errors which we are trying to track down. A couple of things:<br>
<br>
- Even though the config setting is for DEBUG, the indexer log is only showing INFO. Should there be more than that?<br>
<br>
- If the log says:<br>
<br>
I, [2022-05-17T22:27:11.756683 #5273] INFO -- : Thread-2890: Staff Indexer [2022-05-17 22:27:11 -0400] ~~~ Indexed 216<br>
of 20898 archival_object records in repository UNDA<br>
<br>
immediately followed by an error, how do we determine which record that is in ArchivesSpace? We are not sure how to find the record which is indicated in the log.<br>
<br>
Thanks again,</div>
<div>Tom<br>
<br>
<br>
</div>
</div>
<br>
<div>
<div dir="ltr">On Mon, May 16, 2022 at 5:19 PM Blake Carver <<a href="mailto:blake.carver@lyrasis.org" target="_blank">blake.carver@lyrasis.org</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
That could be it, looks like some kind of "funny" character ArchivesSpace doesn't like in there probably. Do the "Indexed x of x whatever " numbers start over right there in the log?<br>
You should see what record it was working on just before there. </div>
<div>
<div id="gmail-m_-9140078466833503249x_gmail-m_-4790138438609789856gmail-m_3871184477380241756Signature">
<div>
<div id="gmail-m_-9140078466833503249x_gmail-m_-4790138438609789856gmail-m_3871184477380241756divtagdefaultwrapper" dir="ltr" style="color:rgb(0,0,0);background-color:rgb(255,255,255)">
<div name="x_divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;margin:0px">
<font face="Calibri,Arial,Helvetica,sans-serif" size="3" color="black"><span dir="ltr" style="font-size:12pt;background-color:white"><font size="2"><span style="font-size:11pt"><br>
</span></font></span></font></div>
</div>
</div>
</div>
</div>
<div id="gmail-m_-9140078466833503249x_gmail-m_-4790138438609789856gmail-m_3871184477380241756appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_-9140078466833503249x_gmail-m_-4790138438609789856gmail-m_3871184477380241756divRplyFwdMsg" dir="ltr">
<font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b>
<a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org" target="_blank">
archivesspace_users_group-bounces@lyralists.lyrasis.org</a> <<a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org" target="_blank">archivesspace_users_group-bounces@lyralists.lyrasis.org</a>> on behalf of Tom Hanstra <<a href="mailto:hanstra@nd.edu" target="_blank">hanstra@nd.edu</a>><br>
<b>Sent:</b> Monday, May 16, 2022 10:43 AM<br>
<b>To:</b> Archivesspace Users Group <<a href="mailto:archivesspace_users_group@lyralists.lyrasis.org" target="_blank">archivesspace_users_group@lyralists.lyrasis.org</a>><br>
<b>Subject:</b> Re: [Archivesspace_Users_Group] (re)indexing in 2.8.1</font>
<div> </div>
</div>
<div>
<div dir="ltr">I've been monitoring logs and have run into a couple of instances where I receive this type of error:<br>
<br>
E, [2022-05-16T08:57:57.207392 #5273] ERROR -- : Thread-3962: Unhandled exception!<br>
E, [2022-05-16T08:57:57.207810 #5273] ERROR -- :<br>
invalid byte sequence in UTF-8<br>
<br>
<div>and then a lot of detail. Are these errors with data that should be addressed? I'm still in DEBUG mode in the backend log but it is not clear how to figure out which record(s) are being flagged. What am I looking for?</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Tom</div>
</div>
<br>
<div>
<div dir="ltr">On Thu, May 12, 2022 at 8:59 AM Blake Carver <<a href="mailto:blake.carver@lyrasis.org" target="_blank">blake.carver@lyrasis.org</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
> <span style="color:rgb(32,31,30);font-size:15px;background-color:rgb(255,255,255);display:inline">Could it be that we added a big record which is now having issues being extracted. Or a corrupted record which is causing such issues?</span><br style="color:rgb(32,31,30);font-size:15px;background-color:rgb(255,255,255)">
</div>
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div id="gmail-m_-9140078466833503249x_gmail-m_-4790138438609789856gmail-m_3871184477380241756x_gmail-m_6222110793183770756Signature">
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
</div>
<div></div>
<div></div>
<div id="gmail-m_-9140078466833503249x_gmail-m_-4790138438609789856gmail-m_3871184477380241756x_gmail-m_6222110793183770756divtagdefaultwrapper" dir="ltr" style="color:rgb(0,0,0);background-color:rgb(255,255,255)">
<div name="x_x_divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;margin:0px">
I don't think so? It could be, but those errors make me think there's something else going on with the DB, or maybe the network between the application and DB server? I'm not really sure what the problem is, but it seems more like a hardware/network/server
issue than an ArchivesSpace issue. I can't be sure, but those errors don't look like ArchivesSpace troubles to me. Those are pretty common errors, so I'd do some searching around to see what you can find to troubleshoot. </div>
</div>
</div>
</div>
</div>
<div id="gmail-m_-9140078466833503249x_gmail-m_-4790138438609789856gmail-m_3871184477380241756x_gmail-m_6222110793183770756appendonsend">
</div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_-9140078466833503249x_gmail-m_-4790138438609789856gmail-m_3871184477380241756x_gmail-m_6222110793183770756divRplyFwdMsg" dir="ltr">
<font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b>
<a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org" target="_blank">
archivesspace_users_group-bounces@lyralists.lyrasis.org</a> <<a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org" target="_blank">archivesspace_users_group-bounces@lyralists.lyrasis.org</a>> on behalf of Tom Hanstra <<a href="mailto:hanstra@nd.edu" target="_blank">hanstra@nd.edu</a>><br>
<b>Sent:</b> Wednesday, May 11, 2022 9:30 AM<br>
<b>To:</b> Archivesspace Users Group <<a href="mailto:archivesspace_users_group@lyralists.lyrasis.org" target="_blank">archivesspace_users_group@lyralists.lyrasis.org</a>><br>
<b>Subject:</b> Re: [Archivesspace_Users_Group] (re)indexing in 2.8.1</font>
<div> </div>
</div>
<div>
<div dir="ltr">Thanks again for your help.
<div><br>
</div>
<div>Late yesterday, both indexes indicated completion so I thought maybe things were going to be OK. Consequently, I did not do much in terms of testing.<br>
<br>
This morning, the logs again had errors, however. In the logs, I found this error in the indexer log:<br>
<br>
I, [2022-05-10T21:36:07.181427 #30003] INFO -- : Thread-2890: Staff Indexer [2022-05-10 21:36:07 -0400] Index round complete<br>
I, [2022-05-10T21:36:37.182006 #30003] INFO -- : Thread-2890: Staff Indexer [2022-05-10 21:36:37 -0400] Running index round<br>
E, [2022-05-10T21:36:37.283423 #30003] ERROR -- : Thread-2890: uri:classloader:/jsonmodel_client.rb:493:in `all'<br>
/home/app/archivesspace/data/tmp/jetty-0.0.0.0-8091-indexer.war-_aspace-indexer-any-/webapp/WEB-INF/app/lib/periodic_indexer.rb:154:in `run_index_round'<br>
/home/app/archivesspace/data/tmp/jetty-0.0.0.0-8091-indexer.war-_aspace-indexer-any-/webapp/WEB-INF/app/lib/periodic_indexer.rb:283:in `run'<br>
/home/app/archivesspace/data/tmp/jetty-0.0.0.0-8091-indexer.war-_aspace-indexer-any-/webapp/WEB-INF/app/main.rb:32:in `block in main'<br>
E, [2022-05-10T21:36:37.284431 #30003] ERROR -- : Thread-2890: #<RuntimeError: {"error":{"db_error":["Database integrity constraint conflict: Java::JavaSql::SQLNonTransientConnectionException: No operations allowed after connection closed."]}}<br>
<br>
and in the backup log there were issues with timeouts retrieving a record:<br>
<br>
Java::ComMysqlCjJdbcExceptions::CommunicationsException: The last packet successfully received from the server was 1,759 milliseconds ago. The last packet sent successfully to the server was 28,849,143 milliseconds ago. is longer than the server configured
value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to
avoid this problem.<br>
</div>
<div><br>
</div>
<div>Could it be that we added a big record which is now having issues being extracted. Or a corrupted record which is causing such issues?<br>
<br>
I've now restarted with the 1x1 and DEBUG on and only staff indexing and it is still thinking indexing is complete. I'll keep things going this way until we hit an error again and hopefully that will give additional information.<br>
<br>
I'll also look into the "autoReconnect=true" piece, since we seem to have a situation where, once this happens, nothing more works until a restart.</div>
<div><br>
</div>
<div>Thanks again for any thoughts on this,</div>
<div>Tom</div>
</div>
<br>
<div>
<div dir="ltr">On Wed, May 11, 2022 at 5:03 AM Andrew Morrison <<a href="mailto:andrew.morrison@bodleian.ox.ac.uk" target="_blank">andrew.morrison@bodleian.ox.ac.uk</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Indexing can also fail at the commit stage, not related to any one record. That is when ArchivesSpace tells Solr to transfer changes made in memory to storage. It does that at several points in the indexing process, but the longest one is at the end of the
PUI indexer's run. If, because you've got a lot of records, or slow storage on your Solr server, it takes longer it respond than the value of AppConfig[:indexer_solr_timeout_seconds], it will start all over again, and potentially go into a loop. The workaround
is to increase the timeout.</p>
<p><br>
</p>
<p>You might not notice you've got enough records to cause this until you do a full re-index, or someone edits something linked to most or all records (e.g. a repository, or a very widely-used subject), triggering the re-indexing of most of the system's records.<br>
</p>
<p><br>
</p>
<p>Andrew.</p>
<p><br>
</p>
<p><br>
</p>
<div>On 10/05/2022 22:06, Blake Carver wrote:<br>
</div>
<blockquote type="cite">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
1x1 would mean setting both records_per_thread and thread_count to 1. Having loglevel on debug and running at 1x1, you'll be able to see exactly which thing is being indexed as it happens, and when it crashes, you'll see what it was working through at the
time.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
PUI will always take longer, and a VERY long time 1x1, but unless you're sure which indexer is crashing, I'd switch them both up.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
You can just `grep Indexed archivesspace.out` after it's running and watch those numbers. As long as they're going up, all is well.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
It is also possible that it will finish without crashing running so slow as well. I've seen that happen with LARGE records. </div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_-9140078466833503249x_gmail-m_-4790138438609789856gmail-m_3871184477380241756x_gmail-m_6222110793183770756x_gmail-m_-7899171973022348328gmail-m_-4396951205564118884divRplyFwdMsg" dir="ltr">
<font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b>
<a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org" target="_blank">
archivesspace_users_group-bounces@lyralists.lyrasis.org</a> <a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org" target="_blank">
<archivesspace_users_group-bounces@lyralists.lyrasis.org></a> on behalf of Tom Hanstra
<a href="mailto:hanstra@nd.edu" target="_blank"><hanstra@nd.edu></a><br>
<b>Sent:</b> Tuesday, May 10, 2022 4:15 PM<br>
<b>To:</b> Archivesspace Users Group <a href="mailto:archivesspace_users_group@lyralists.lyrasis.org" target="_blank">
<archivesspace_users_group@lyralists.lyrasis.org></a><br>
<b>Subject:</b> Re: [Archivesspace_Users_Group] (re)indexing in 2.8.1</font>
<div> </div>
</div>
<div>
<div dir="ltr">Thanks, Blake.
<div><br>
</div>
<div>Turns out we did add quite a few records recently, so maybe there was something in there that it did not like all that much. </div>
<div><br>
</div>
<div>How can you tell which record it is choking on? Is that your "1x1" suggestion? Or does the DEBUG option make that more clear? I have my indexing set to:<br>
<br>
AppConfig[:indexer_records_per_thread] = 25<br>
AppConfig[:indexer_thread_count] = 2<br>
<br>
for both PUI and Staff records. I believe you are suggesting it would most easily be found using 1 and 1? I can see where that could take a long time. But it if is going to choke over and over on the same record, then that may be the best way to address it. <br>
<br>
Do you think if I just did staff indexing without PUI, that it would be identified faster? Or could it pass the staff side but then die on PUI later?</div>
<div><br>
</div>
<div>I hope to try some of these ideas after hours today, so if you can confirm that I've got the right idea, that would help.</div>
<div><br>
</div>
<div>Tom</div>
<div><br>
</div>
</div>
<br>
<div>
<div dir="ltr">On Tue, May 10, 2022 at 2:17 PM Blake Carver <<a href="mailto:blake.carver@lyrasis.org" target="_blank">blake.carver@lyrasis.org</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
> <span style="color:rgb(32,31,30);font-size:15px;background-color:rgb(255,255,255);display:inline">Is this possible?</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="color:rgb(32,31,30);font-size:15px;background-color:rgb(255,255,255);display:inline"><br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="color:rgb(32,31,30);font-size:15px;background-color:rgb(255,255,255);display:inline">Short answer, Yes, it's possible your indexer is starting over.</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="color:rgb(32,31,30);font-size:15px;background-color:rgb(255,255,255);display:inline"><br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="color:rgb(32,31,30);font-size:15px;background-color:rgb(255,255,255);display:inline">Long answer. This can be tricky to figure out. Something is wrong, the indexer never wants to do that. Sometimes "something" "bad" gets into ArchivesSpace and
the indexer will just crash and start over. The problem is the "something" can be anything and the "bad" can be hard to figure out. The more stuff you have in your DB, the harder it is to figure out.</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="color:rgb(32,31,30);font-size:15px;background-color:rgb(255,255,255);display:inline"><br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="color:rgb(32,31,30);font-size:15px;background-color:rgb(255,255,255);display:inline">First, I'd make sure this is happening. Your logs should make it obvious. You might see some FATAL errors just before it starts over. You MIGHT be able to
narrow it down from that. That is, what group of records had that error in the logs? Maybe that narrows it down enough. You just got lucky!</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="color:rgb(32,31,30);font-size:15px;background-color:rgb(255,255,255);display:inline"><br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="color:rgb(32,31,30);font-size:15px;background-color:rgb(255,255,255);display:inline">I don't think I've ever been so lucky. What I'd do next is set your loglevel to DEBUG and restart. If you're feeling lucky or just impatient or both, leave
the indexer speed as is. You'll get more details out of the logs and you should be able to narrow it down better. Ideally, you want to run the indexers at 1x1, which means it could take forrreeevverrrrr to get back around to the crash again. If you're lucky,
it'll crash on a record, you'll go look at that record, the problem will be obvious, and there will be much rejoicing. With it running 1x1 you should see exactly what's causing the fail. If it's not crashing on the same record every time.... ugh. That's an
even longer answer. </span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="color:rgb(32,31,30);font-size:15px;background-color:rgb(255,255,255);display:inline"><br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div>
<div id="gmail-m_-9140078466833503249x_gmail-m_-4790138438609789856gmail-m_3871184477380241756x_gmail-m_6222110793183770756x_gmail-m_-7899171973022348328gmail-m_-4396951205564118884x_gmail-m_2727772702640317328Signature">
<div>
<div id="gmail-m_-9140078466833503249x_gmail-m_-4790138438609789856gmail-m_3871184477380241756x_gmail-m_6222110793183770756x_gmail-m_-7899171973022348328gmail-m_-4396951205564118884x_gmail-m_2727772702640317328divtagdefaultwrapper" dir="ltr" style="color:rgb(0,0,0);background-color:rgb(255,255,255)">
<div name="x_x_x_x_divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;margin:0px">
<font size="3" face="Calibri,Arial,Helvetica,sans-serif" color="black"><span dir="ltr" style="font-size:12pt;background-color:white"><font size="2"><span style="font-size:11pt"><br>
</span></font></span></font></div>
</div>
</div>
</div>
</div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_-9140078466833503249x_gmail-m_-4790138438609789856gmail-m_3871184477380241756x_gmail-m_6222110793183770756x_gmail-m_-7899171973022348328gmail-m_-4396951205564118884x_gmail-m_2727772702640317328divRplyFwdMsg" dir="ltr">
<font face="Calibri,
sans-serif" color="#000000" style="font-size:11pt"><b>From:</b>
<a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org" target="_blank">
archivesspace_users_group-bounces@lyralists.lyrasis.org</a> <<a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org" target="_blank">archivesspace_users_group-bounces@lyralists.lyrasis.org</a>> on behalf of Tom Hanstra <<a href="mailto:hanstra@nd.edu" target="_blank">hanstra@nd.edu</a>><br>
<b>Sent:</b> Tuesday, May 10, 2022 10:23 AM<br>
<b>To:</b> Archivesspace Users Group <<a href="mailto:archivesspace_users_group@lyralists.lyrasis.org" target="_blank">archivesspace_users_group@lyralists.lyrasis.org</a>><br>
<b>Subject:</b> [Archivesspace_Users_Group] (re)indexing in 2.8.1</font>
<div> </div>
</div>
<div>
<div dir="ltr">I don't look at the logs a lot unless there are issues with ArchivesSpace, so maybe this is something normal. But, after a restart due to some complaints about database connectivity, it looks like our ArchivesSpace instance has decided to do
a full reindex. The index log sure looks as if it is starting from scratch and running through the indexing of both PUI and Staff indexes.
<div><br>
</div>
<div>
<div>Is this possible? Is it something that happens periodically and I just did not notice it? Nothing has changed in my data directory, so I don't see any reason for indexing to occur. Yet that is what the logs show.</div>
<div><br>
</div>
<div>If it is doing this for some reason, and knowing that we restart periodically, it seems like we will get into a loop where indexing just keeps happening all the time. Also, it would be helpful to understand what caused this to happen.</div>
<div><br>
</div>
<div>Any thoughts or experiences from those who have run this for longer would be appreciated. I'd like to understand if it would be a good idea to clear the data directory and perform a full index over the weekend rather than an unexpected and possibly never
ending round in the background.</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Tom</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr">
<div><b style="font-family:arial,helvetica,sans-serif;font-size:12.7273px;color:rgb(136,136,136)">Tom Hanstra</b><br>
</div>
<div style="color:rgb(136,136,136);font-size:12.8px">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div style="font-size:12.7273px">
<div>
<div><i style="font-size:12.7273px;font-family:arial,helvetica,sans-serif">Sr. Systems Administrator</i></div>
<div><a href="mailto:hanstra@nd.edu" style="color:rgb(17,85,204);font-size:12.7273px;font-family:arial,helvetica,sans-serif" target="_blank">hanstra@nd.edu</a><br>
</div>
</div>
<div><span style="font-family:arial,helvetica,sans-serif"><br>
</span></div>
</div>
<div style="font-size:12.7273px"><img src="https://ci3.googleusercontent.com/mail-sig/AIorK4wQjvBdM9TFi5bR5RBsq_1dY3HTxh-Kg_4W690bwTCSKeVGyazMoj0wdmkNgJ0kfjeRnparhiw"><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br>
Archivesspace_Users_Group mailing list<br>
<a href="mailto:Archivesspace_Users_Group@lyralists.lyrasis.org" target="_blank">Archivesspace_Users_Group@lyralists.lyrasis.org</a><br>
<a href="http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group" target="_blank">http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group</a><br>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr">
<div><b style="font-family:arial,helvetica,sans-serif;font-size:12.7273px;color:rgb(136,136,136)">Tom Hanstra</b><br>
</div>
<div style="color:rgb(136,136,136);font-size:12.8px">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div style="font-size:12.7273px">
<div>
<div><i style="font-size:12.7273px;font-family:arial,helvetica,sans-serif">Sr. Systems Administrator</i></div>
<div><a href="mailto:hanstra@nd.edu" style="color:rgb(17,85,204);font-size:12.7273px;font-family:arial,helvetica,sans-serif" target="_blank">hanstra@nd.edu</a><br>
</div>
</div>
<div><span style="font-family:arial,helvetica,sans-serif"><br>
</span></div>
</div>
<div style="font-size:12.7273px"><img src="https://ci3.googleusercontent.com/mail-sig/AIorK4wQjvBdM9TFi5bR5RBsq_1dY3HTxh-Kg_4W690bwTCSKeVGyazMoj0wdmkNgJ0kfjeRnparhiw"><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
Archivesspace_Users_Group mailing list
<a href="mailto:Archivesspace_Users_Group@lyralists.lyrasis.org" target="_blank">Archivesspace_Users_Group@lyralists.lyrasis.org</a>
<a href="http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group" target="_blank">http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group</a>
</pre>
</blockquote>
</div>
_______________________________________________<br>
Archivesspace_Users_Group mailing list<br>
<a href="mailto:Archivesspace_Users_Group@lyralists.lyrasis.org" target="_blank">Archivesspace_Users_Group@lyralists.lyrasis.org</a><br>
<a href="http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group" target="_blank">http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group</a><br>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr">
<div><b style="font-family:arial,helvetica,sans-serif;font-size:12.7273px;color:rgb(136,136,136)">Tom Hanstra</b><br>
</div>
<div style="color:rgb(136,136,136);font-size:12.8px">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div style="font-size:12.7273px">
<div>
<div><i style="font-size:12.7273px;font-family:arial,helvetica,sans-serif">Sr. Systems Administrator</i></div>
<div><a href="mailto:hanstra@nd.edu" style="color:rgb(17,85,204);font-size:12.7273px;font-family:arial,helvetica,sans-serif" target="_blank">hanstra@nd.edu</a><br>
</div>
</div>
<div><span style="font-family:arial,helvetica,sans-serif"><br>
</span></div>
</div>
<div style="font-size:12.7273px"><img src="https://ci3.googleusercontent.com/mail-sig/AIorK4wQjvBdM9TFi5bR5RBsq_1dY3HTxh-Kg_4W690bwTCSKeVGyazMoj0wdmkNgJ0kfjeRnparhiw"><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br>
Archivesspace_Users_Group mailing list<br>
<a href="mailto:Archivesspace_Users_Group@lyralists.lyrasis.org" target="_blank">Archivesspace_Users_Group@lyralists.lyrasis.org</a><br>
<a href="http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group" target="_blank">http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group</a><br>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr">
<div><b style="font-family:arial,helvetica,sans-serif;font-size:12.7273px;color:rgb(136,136,136)">Tom Hanstra</b><br>
</div>
<div style="color:rgb(136,136,136);font-size:12.8px">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div style="font-size:12.7273px">
<div>
<div><i style="font-size:12.7273px;font-family:arial,helvetica,sans-serif">Sr. Systems Administrator</i></div>
<div><a href="mailto:hanstra@nd.edu" style="color:rgb(17,85,204);font-size:12.7273px;font-family:arial,helvetica,sans-serif" target="_blank">hanstra@nd.edu</a><br>
</div>
</div>
<div><span style="font-family:arial,helvetica,sans-serif"><br>
</span></div>
</div>
<div style="font-size:12.7273px"><img src="https://ci3.googleusercontent.com/mail-sig/AIorK4wQjvBdM9TFi5bR5RBsq_1dY3HTxh-Kg_4W690bwTCSKeVGyazMoj0wdmkNgJ0kfjeRnparhiw"><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br>
Archivesspace_Users_Group mailing list<br>
<a href="mailto:Archivesspace_Users_Group@lyralists.lyrasis.org" target="_blank">Archivesspace_Users_Group@lyralists.lyrasis.org</a><br>
<a href="http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group" target="_blank">http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group</a><br>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr">
<div><b style="font-family:arial,helvetica,sans-serif;font-size:12.7273px;color:rgb(136,136,136)">Tom Hanstra</b><br>
</div>
<div style="color:rgb(136,136,136);font-size:12.8px">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div style="font-size:12.7273px">
<div>
<div><i style="font-size:12.7273px;font-family:arial,helvetica,sans-serif">Sr. Systems Administrator</i></div>
<div><a href="mailto:hanstra@nd.edu" style="color:rgb(17,85,204);font-size:12.7273px;font-family:arial,helvetica,sans-serif" target="_blank">hanstra@nd.edu</a><br>
</div>
</div>
<div><span style="font-family:arial,helvetica,sans-serif"><br>
</span></div>
</div>
<div style="font-size:12.7273px"><img src="https://ci3.googleusercontent.com/mail-sig/AIorK4wQjvBdM9TFi5bR5RBsq_1dY3HTxh-Kg_4W690bwTCSKeVGyazMoj0wdmkNgJ0kfjeRnparhiw"><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br>
Archivesspace_Users_Group mailing list<br>
<a href="mailto:Archivesspace_Users_Group@lyralists.lyrasis.org" target="_blank">Archivesspace_Users_Group@lyralists.lyrasis.org</a><br>
<a href="http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group" rel="noreferrer" target="_blank">http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div><b style="font-family:arial,helvetica,sans-serif;font-size:12.7273px;color:rgb(136,136,136)">Tom Hanstra</b><br></div><div style="color:rgb(136,136,136);font-size:12.8px"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div style="font-size:12.7273px"><div><div><i style="font-size:12.7273px;font-family:arial,helvetica,sans-serif">Sr. Systems Administrator</i></div><div><a href="mailto:hanstra@nd.edu" style="color:rgb(17,85,204);font-size:12.7273px;font-family:arial,helvetica,sans-serif" target="_blank">hanstra@nd.edu</a><br></div></div><div><span style="font-family:arial,helvetica,sans-serif"><br></span></div></div><div style="font-size:12.7273px"><img src="https://ci3.googleusercontent.com/mail-sig/AIorK4wQjvBdM9TFi5bR5RBsq_1dY3HTxh-Kg_4W690bwTCSKeVGyazMoj0wdmkNgJ0kfjeRnparhiw"><br></div></div></div></div></div></div></div></div></div></div></div>