<div dir="ltr">Thanks again for your help.<div><br></div><div>Late yesterday, both indexes indicated completion so I thought maybe things were going to be OK. Consequently, I did not do much in terms of testing.<br><br>This morning, the logs again had errors, however.  In the logs, I found this error in the indexer log:<br><br>I, [2022-05-10T21:36:07.181427 #30003]  INFO -- : Thread-2890: Staff Indexer [2022-05-10 21:36:07 -0400] Index round complete<br>I, [2022-05-10T21:36:37.182006 #30003]  INFO -- : Thread-2890: Staff Indexer [2022-05-10 21:36:37 -0400] Running index round<br>E, [2022-05-10T21:36:37.283423 #30003] ERROR -- : Thread-2890: uri:classloader:/jsonmodel_client.rb:493:in `all'<br>/home/app/archivesspace/data/tmp/jetty-0.0.0.0-8091-indexer.war-_aspace-indexer-any-/webapp/WEB-INF/app/lib/periodic_indexer.rb:154:in `run_index_round'<br>/home/app/archivesspace/data/tmp/jetty-0.0.0.0-8091-indexer.war-_aspace-indexer-any-/webapp/WEB-INF/app/lib/periodic_indexer.rb:283:in `run'<br>/home/app/archivesspace/data/tmp/jetty-0.0.0.0-8091-indexer.war-_aspace-indexer-any-/webapp/WEB-INF/app/main.rb:32:in `block in main'<br>E, [2022-05-10T21:36:37.284431 #30003] ERROR -- : Thread-2890: #<RuntimeError: {"error":{"db_error":["Database integrity constraint conflict: Java::JavaSql::SQLNonTransientConnectionException: No operations allowed after connection closed."]}}<br><br>and in the backup log there were issues with timeouts retrieving a record:<br><br>Java::ComMysqlCjJdbcExceptions::CommunicationsException: The last packet successfully received from the server was 1,759 milliseconds ago. The last packet sent successfully to the server was 28,849,143 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.<br></div><div><br></div><div>Could it be that we added a big record which is now having issues being extracted. Or a corrupted record which is causing such issues?<br><br>I've now restarted with the 1x1 and DEBUG on and only staff indexing and it is still thinking indexing is complete. I'll keep things going this way until we hit an error again and hopefully that will give additional information.<br><br>I'll also look into the "autoReconnect=true" piece, since we seem to have a situation where, once this happens, nothing more works until a restart.</div><div><br></div><div>Thanks again for any thoughts on this,</div><div>Tom</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, May 11, 2022 at 5:03 AM Andrew Morrison <<a href="mailto:andrew.morrison@bodleian.ox.ac.uk" target="_blank">andrew.morrison@bodleian.ox.ac.uk</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">

  
  <div>
    <p>Indexing can also fail at the commit stage, not related to any
      one record. That is when ArchivesSpace tells Solr to transfer
      changes made in memory to storage. It does that at several points
      in the indexing process, but the longest one is at the end of the
      PUI indexer's run. If, because you've got a lot of records, or
      slow storage on your Solr server, it takes longer it respond than
      the value of AppConfig[:indexer_solr_timeout_seconds], it will
      start all over again, and potentially go into a loop. The
      workaround is to increase the timeout.</p>
    <p><br>
    </p>
    <p>You might not notice you've got enough records to cause this
      until you do a full re-index, or someone edits something linked to
      most or all records (e.g. a repository, or a very widely-used
      subject), triggering the re-indexing of most of the system's
      records.<br>
    </p>
    <p><br>
    </p>
    <p>Andrew.</p>
    <p><br>
    </p>
    <p><br>
    </p>
    <div>On 10/05/2022 22:06, Blake Carver
      wrote:<br>
    </div>
    <blockquote type="cite">
      
      
      <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
         1x1 would mean setting both records_per_thread and thread_count
        to 1. Having loglevel on debug and running at 1x1, you'll be
        able to see exactly which thing is being indexed as it happens,
        and when it crashes, you'll see what it was working through at
        the time.</div>
      <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
        <br>
      </div>
      <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
        PUI will always take longer, and a VERY long time 1x1, but
        unless you're sure which indexer is crashing, I'd switch them
        both up.</div>
      <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
        <br>
      </div>
      <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
        You can just `grep Indexed archivesspace.out` after it's running
        and watch those numbers. As long as they're going up, all is
        well.</div>
      <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
        <br>
      </div>
      <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
        It is also possible that it will finish without crashing running
        so slow as well. I've seen that happen with LARGE records. </div>
      <hr style="display:inline-block;width:98%">
      <div id="gmail-m_-7899171973022348328gmail-m_-4396951205564118884divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b>
          <a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org" target="_blank">archivesspace_users_group-bounces@lyralists.lyrasis.org</a>
          <a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org" target="_blank"><archivesspace_users_group-bounces@lyralists.lyrasis.org></a>
          on behalf of Tom Hanstra <a href="mailto:hanstra@nd.edu" target="_blank"><hanstra@nd.edu></a><br>
          <b>Sent:</b> Tuesday, May 10, 2022 4:15 PM<br>
          <b>To:</b> Archivesspace Users Group
          <a href="mailto:archivesspace_users_group@lyralists.lyrasis.org" target="_blank"><archivesspace_users_group@lyralists.lyrasis.org></a><br>
          <b>Subject:</b> Re: [Archivesspace_Users_Group] (re)indexing
          in 2.8.1</font>
        <div> </div>
      </div>
      <div>
        <div dir="ltr">Thanks, Blake.
          <div><br>
          </div>
          <div>Turns out we did add quite a few records recently, so
            maybe there was something in there that it did not like all
            that much. </div>
          <div><br>
          </div>
          <div>How can you tell which record it is choking on?  Is that
            your "1x1" suggestion?  Or does the DEBUG option make that
            more clear?  I have my indexing set to:<br>
            <br>
            AppConfig[:indexer_records_per_thread]      = 25<br>
            AppConfig[:indexer_thread_count]            = 2<br>
            <br>
            for both PUI and Staff records. I believe you are suggesting
            it would most easily be found using 1 and 1?  I can see
            where that could take a long time. But it if is going to
            choke over and over on the same record, then that may be the
            best way to address it. <br>
            <br>
            Do you think if I just did staff indexing without PUI, that
            it would be identified faster?  Or could it pass the staff
            side but then die on PUI later?</div>
          <div><br>
          </div>
          <div>I hope to try some of these ideas after hours today, so
            if you can confirm that I've got the right idea, that would
            help.</div>
          <div><br>
          </div>
          <div>Tom</div>
          <div><br>
          </div>
        </div>
        <br>
        <div>
          <div dir="ltr">On Tue, May 10, 2022 at
            2:17 PM Blake Carver <<a href="mailto:blake.carver@lyrasis.org" target="_blank">blake.carver@lyrasis.org</a>>
            wrote:<br>
          </div>
          <blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
            <div dir="ltr">
              <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
                > <span style="color:rgb(32,31,30);font-size:15px;background-color:rgb(255,255,255);display:inline">Is
                  this possible?</span></div>
              <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
                <span style="color:rgb(32,31,30);font-size:15px;background-color:rgb(255,255,255);display:inline"><br>
                </span></div>
              <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
                <span style="color:rgb(32,31,30);font-size:15px;background-color:rgb(255,255,255);display:inline">Short
                  answer, Yes, it's possible your indexer is starting
                  over.</span></div>
              <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
                <span style="color:rgb(32,31,30);font-size:15px;background-color:rgb(255,255,255);display:inline"><br>
                </span></div>
              <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
                <span style="color:rgb(32,31,30);font-size:15px;background-color:rgb(255,255,255);display:inline">Long
                  answer. This can be tricky to figure out. Something is
                  wrong, the indexer never wants to do that. Sometimes
                  "something" "bad" gets into ArchivesSpace and the
                  indexer will just crash and start over. The problem is
                  the "something" can be anything and the "bad" can be
                  hard to figure out. The more stuff you have in your
                  DB, the harder it is to figure out.</span></div>
              <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
                <span style="color:rgb(32,31,30);font-size:15px;background-color:rgb(255,255,255);display:inline"><br>
                </span></div>
              <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
                <span style="color:rgb(32,31,30);font-size:15px;background-color:rgb(255,255,255);display:inline">First,
                  I'd make sure this is happening. Your logs should make
                  it obvious. You might see some FATAL errors just
                  before it starts over.  You MIGHT be able to narrow it
                  down from that. That is, what group of records had
                  that error in the logs? Maybe that narrows it down
                  enough. You just got lucky!</span></div>
              <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
                <span style="color:rgb(32,31,30);font-size:15px;background-color:rgb(255,255,255);display:inline"><br>
                </span></div>
              <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
                <span style="color:rgb(32,31,30);font-size:15px;background-color:rgb(255,255,255);display:inline">I
                  don't think I've ever been so lucky. What I'd do next
                  is set your loglevel to DEBUG and restart. If you're
                  feeling lucky or just impatient or both, leave the
                  indexer speed as is. You'll get more details out of
                  the logs and you should be able to narrow it down
                  better. Ideally, you want to run the indexers at 1x1,
                  which means it could take forrreeevverrrrr to get back
                  around to the crash again. If you're lucky, it'll
                  crash on a record, you'll go look at that record, the
                  problem will be obvious, and there will be much
                  rejoicing. With it running 1x1 you should see exactly
                  what's causing the fail. If it's not crashing on the
                  same record every time.... ugh. That's an even longer
                  answer. </span></div>
              <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
                <span style="color:rgb(32,31,30);font-size:15px;background-color:rgb(255,255,255);display:inline"><br>
                </span></div>
              <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
                <br>
              </div>
              <div>
                <div id="gmail-m_-7899171973022348328gmail-m_-4396951205564118884x_gmail-m_2727772702640317328Signature">
                  <div>
                    <div id="gmail-m_-7899171973022348328gmail-m_-4396951205564118884x_gmail-m_2727772702640317328divtagdefaultwrapper" dir="ltr" style="color:rgb(0,0,0);background-color:rgb(255,255,255)">
                      <div name="x_divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;margin:0px">
                        <font size="3" face="Calibri,Arial,Helvetica,sans-serif" color="black"><span dir="ltr" style="font-size:12pt;background-color:white"><font size="2"><span style="font-size:11pt"><br>
                              </span></font></span></font></div>
                    </div>
                  </div>
                </div>
              </div>
              <hr style="display:inline-block;width:98%">
              <div id="gmail-m_-7899171973022348328gmail-m_-4396951205564118884x_gmail-m_2727772702640317328divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri,
                  sans-serif" color="#000000"><b>From:</b>
                  <a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org" target="_blank">
archivesspace_users_group-bounces@lyralists.lyrasis.org</a> <<a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org" target="_blank">archivesspace_users_group-bounces@lyralists.lyrasis.org</a>>
                  on behalf of Tom Hanstra <<a href="mailto:hanstra@nd.edu" target="_blank">hanstra@nd.edu</a>><br>
                  <b>Sent:</b> Tuesday, May 10, 2022 10:23 AM<br>
                  <b>To:</b> Archivesspace Users Group <<a href="mailto:archivesspace_users_group@lyralists.lyrasis.org" target="_blank">archivesspace_users_group@lyralists.lyrasis.org</a>><br>
                  <b>Subject:</b> [Archivesspace_Users_Group]
                  (re)indexing in 2.8.1</font>
                <div> </div>
              </div>
              <div>
                <div dir="ltr">I don't look at the logs a lot unless
                  there are issues with ArchivesSpace, so maybe this is
                  something normal. But, after a restart due to some
                  complaints about database connectivity, it looks like
                  our ArchivesSpace instance has decided to do a full
                  reindex. The index log sure looks as if it is starting
                  from scratch and running through the indexing of both
                  PUI and Staff indexes.
                  <div><br>
                  </div>
                  <div>
                    <div>Is this possible?  Is it something that happens
                      periodically and I just did not notice it? Nothing
                      has changed in my data directory, so I don't see
                      any reason for indexing to occur. Yet that is what
                      the logs show.</div>
                    <div><br>
                    </div>
                    <div>If it is doing this for some reason, and
                      knowing that we restart periodically, it seems
                      like we will get into a loop where indexing just
                      keeps happening all the time. Also, it would be
                      helpful to understand what caused this to happen.</div>
                    <div><br>
                    </div>
                    <div>Any thoughts or experiences from those who have
                      run this for longer would be appreciated. I'd like
                      to understand if it would be a good idea to clear
                      the data directory and perform a full index over
                      the weekend rather than an unexpected and possibly
                      never ending round in the background.</div>
                    <div><br>
                    </div>
                    <div>Thanks,</div>
                    <div>Tom</div>
                    -- <br>
                    <div dir="ltr">
                      <div dir="ltr">
                        <div>
                          <div dir="ltr">
                            <div dir="ltr">
                              <div><b style="font-family:arial,helvetica,sans-serif;font-size:12.7273px;color:rgb(136,136,136)">Tom Hanstra</b><br>
                              </div>
                              <div style="color:rgb(136,136,136);font-size:12.8px">
                                <div dir="ltr">
                                  <div dir="ltr">
                                    <div dir="ltr">
                                      <div dir="ltr">
                                        <div style="font-size:12.7273px">
                                          <div>
                                            <div><i style="font-size:12.7273px;font-family:arial,helvetica,sans-serif">Sr. Systems Administrator</i></div>
                                            <div><a href="mailto:hanstra@nd.edu" style="color:rgb(17,85,204);font-size:12.7273px;font-family:arial,helvetica,sans-serif" target="_blank">hanstra@nd.edu</a><br>
                                            </div>
                                          </div>
                                          <div><span style="font-family:arial,helvetica,sans-serif"><br>
                                            </span></div>
                                        </div>
                                        <div style="font-size:12.7273px"><img src="https://ci3.googleusercontent.com/mail-sig/AIorK4wQjvBdM9TFi5bR5RBsq_1dY3HTxh-Kg_4W690bwTCSKeVGyazMoj0wdmkNgJ0kfjeRnparhiw"><br>
                                        </div>
                                      </div>
                                    </div>
                                  </div>
                                </div>
                              </div>
                            </div>
                          </div>
                        </div>
                      </div>
                    </div>
                  </div>
                </div>
              </div>
            </div>
            _______________________________________________<br>
            Archivesspace_Users_Group mailing list<br>
            <a href="mailto:Archivesspace_Users_Group@lyralists.lyrasis.org" target="_blank">Archivesspace_Users_Group@lyralists.lyrasis.org</a><br>
            <a href="http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group" rel="noreferrer" target="_blank">http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group</a><br>
          </blockquote>
        </div>
        <br clear="all">
        <div><br>
        </div>
        -- <br>
        <div dir="ltr">
          <div dir="ltr">
            <div>
              <div dir="ltr">
                <div dir="ltr">
                  <div><b style="font-family:arial,helvetica,sans-serif;font-size:12.7273px;color:rgb(136,136,136)">Tom
                      Hanstra</b><br>
                  </div>
                  <div style="color:rgb(136,136,136);font-size:12.8px">
                    <div dir="ltr">
                      <div dir="ltr">
                        <div dir="ltr">
                          <div dir="ltr">
                            <div style="font-size:12.7273px">
                              <div>
                                <div><i style="font-size:12.7273px;font-family:arial,helvetica,sans-serif">Sr.
                                    Systems Administrator</i></div>
                                <div><a href="mailto:hanstra@nd.edu" style="color:rgb(17,85,204);font-size:12.7273px;font-family:arial,helvetica,sans-serif" target="_blank">hanstra@nd.edu</a><br>
                                </div>
                              </div>
                              <div><span style="font-family:arial,helvetica,sans-serif"><br>
                                </span></div>
                            </div>
                            <div style="font-size:12.7273px"><img src="https://ci3.googleusercontent.com/mail-sig/AIorK4wQjvBdM9TFi5bR5RBsq_1dY3HTxh-Kg_4W690bwTCSKeVGyazMoj0wdmkNgJ0kfjeRnparhiw"><br>
                            </div>
                          </div>
                        </div>
                      </div>
                    </div>
                  </div>
                </div>
              </div>
            </div>
          </div>
        </div>
      </div>
      <br>
      <fieldset></fieldset>
      <pre>_______________________________________________
Archivesspace_Users_Group mailing list
<a href="mailto:Archivesspace_Users_Group@lyralists.lyrasis.org" target="_blank">Archivesspace_Users_Group@lyralists.lyrasis.org</a>
<a href="http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group" target="_blank">http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group</a>
</pre>
    </blockquote>
  </div>

_______________________________________________<br>
Archivesspace_Users_Group mailing list<br>
<a href="mailto:Archivesspace_Users_Group@lyralists.lyrasis.org" target="_blank">Archivesspace_Users_Group@lyralists.lyrasis.org</a><br>
<a href="http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group" rel="noreferrer" target="_blank">http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div><b style="font-family:arial,helvetica,sans-serif;font-size:12.7273px;color:rgb(136,136,136)">Tom Hanstra</b><br></div><div style="color:rgb(136,136,136);font-size:12.8px"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div style="font-size:12.7273px"><div><div><i style="font-size:12.7273px;font-family:arial,helvetica,sans-serif">Sr. Systems Administrator</i></div><div><a href="mailto:hanstra@nd.edu" style="color:rgb(17,85,204);font-size:12.7273px;font-family:arial,helvetica,sans-serif" target="_blank">hanstra@nd.edu</a><br></div></div><div><span style="font-family:arial,helvetica,sans-serif"><br></span></div></div><div style="font-size:12.7273px"><img src="https://ci3.googleusercontent.com/mail-sig/AIorK4wQjvBdM9TFi5bR5RBsq_1dY3HTxh-Kg_4W690bwTCSKeVGyazMoj0wdmkNgJ0kfjeRnparhiw"><br></div></div></div></div></div></div></div></div></div></div></div>