<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><br class=""></div><div class="">Many thanks Megan for the lead. I just poked around and found a few things out.</div><div class=""><br class=""></div>It turns out to be interesting in at least three bad ways.<div class=""><br class=""></div><div class="">1. User records are updated every time they authenticate. This is unnecessary because it is just updating their authentication source, which usually never changes. So this line could be made conditional on the source being different:</div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span><a href="https://github.com/archivesspace/archivesspace/blob/master/backend/app/model/authentication_manager.rb#L40" class="">https://github.com/archivesspace/archivesspace/blob/master/backend/app/model/authentication_manager.rb#L40</a></div><div class=""><br class=""></div><div class="">2. The search_indexer user is being re-authenticated a lot. This shouldn’t happen because it should be a non-expiring session. I haven’t looked into this one, but I did check the lock_version on the search_inxdexer user on a large-ish AS db I have lying around, that has been in production for less than a year and it is over 100,000. That’s nuts. Actually I just took a quick look and saw this:</div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span><a href="https://github.com/archivesspace/archivesspace/blob/master/indexer/app/lib/periodic_indexer.rb#L285" class="">https://github.com/archivesspace/archivesspace/blob/master/indexer/app/lib/periodic_indexer.rb#L285</a></div><div class="">The session is reset if there is an error during an indexing round. So that could be the culprit.</div><div class=""><br class=""></div><div class="">3. There is clearly a bug somewhere in the stack, probably in the Sequel layer, because it shouldn’t crash the system just because you have a whacky datetime somewhere. This bug may well have been fixed in a more recent version of that component.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Dealing with 1 will likely fix the problem, but 2 and 3 should also be investigated.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Cheers,</div><div class="">James</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div class=""><div class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Mar 16, 2021, at 1:24 AM, Tom Hanstra <<a href="mailto:hanstra@nd.edu" class="">hanstra@nd.edu</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Thanks, Megan. That update did fix things.<div class=""><br class=""></div><div class="">So, is this not something that can be addressed at the software fix level? Do other sites simply stop indexing or turn something else off when we hit DST in March?</div><div class=""><br class=""></div><div class="">Tom</div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Mar 15, 2021 at 10:12 AM Schanz, Megan <<a href="mailto:schanzme@msu.edu" class="">schanzme@msu.edu</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr" class="">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
I run across this every March for daylight savings. This is what I have in my notes to do each time if the indexer is running. Luckily it seems like I didn't have this scenario this year.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
<br class="">
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
<h3 dir="auto" class="">java.lang.IllegalArgumentException: HOUR_OF_DAY: 2 -> 3:Java::JavaLang::IllegalArgumentException:</h3><p dir="auto" class="">This error will appear in the logs when trying to start ArchivesSpace after daylight savings time. Reference:
<a href="http://lyralists.lyrasis.org/mailman/htdig/archivesspace_users_group/2019-March/006652.html" rel="nofollow noreferrer noopener" target="_blank" class="">
http://lyralists.lyrasis.org/mailman/htdig/archivesspace_users_group/2019-March/006652.html</a></p><p dir="auto" class="">Basically, the indexer user is being updated in the database with a time that does not exist due to daylight savings time.</p><p dir="auto" class="">To verify:</p>
<pre lang="plaintext" class=""><code class=""><span lang="plaintext" class="">SELECT * FROM user WHERE (user_mtime >= '2021-03-14 02:00:00' and user_mtime <= '2021-03-14 03:00:00') OR (system_mtime >= '2021-03-04 02:00:00' and system_mtime <= '2021-03-14 03:00:00');</span></code></pre><p dir="auto" class="">The record that should come back is the
<code class="">search_indexer</code> user.</p><p dir="auto" class="">To resolve:</p>
<pre lang="plaintext" class=""><code class=""><span lang="plaintext" class="">UPDATE user set user_mtime = NOW(), system_mtime=NOW() where username='search_indexer';</span></code></pre>
<br class="">
</div>
<div id="gmail-m_-3548360350817852693appendonsend" class=""></div>
<hr style="display:inline-block;width:98%" class="">
<div id="gmail-m_-3548360350817852693divRplyFwdMsg" dir="ltr" class=""><font face="Calibri, sans-serif" style="font-size:11pt" class=""><b class="">From:</b> <a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org" target="_blank" class="">archivesspace_users_group-bounces@lyralists.lyrasis.org</a> <<a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org" target="_blank" class="">archivesspace_users_group-bounces@lyralists.lyrasis.org</a>> on behalf of Tom Hanstra
<<a href="mailto:hanstra@nd.edu" target="_blank" class="">hanstra@nd.edu</a>><br class="">
<b class="">Sent:</b> Monday, March 15, 2021 9:10 AM<br class="">
<b class="">To:</b> Archivesspace Users Group <<a href="mailto:archivesspace_users_group@lyralists.lyrasis.org" target="_blank" class="">archivesspace_users_group@lyralists.lyrasis.org</a>><br class="">
<b class="">Subject:</b> Re: [Archivesspace_Users_Group] Java error - Java::JavaSql::SQLException: HOUR_OF_DAY: 2 -> 3</font>
<div class=""> </div>
</div>
<div class="">
<div dir="ltr" class="">Any suggestion on what ArchivesSpace might have changed? I had the server running but indexing was complete. What might it have been changing and in what database table would I look for that change?<br class="">
<br class="">
Alternately, since this is still test data, should I just overlay a backup copy of the database from earlier than the EDT cutover time? I don't know what affect that might have on other portions of ArchivesSpace.
<div class=""><br class="">
</div>
<div class="">Finally, what is the overall fix for this issue. If others have seen it, what can be done to make sure to avoid it in the future?</div>
<div class=""><br class="">
</div>
<div class="">Thanks,</div>
<div class="">Tom</div>
</div>
<br class="">
<div class="">
<div dir="ltr" class="">On Mon, Mar 15, 2021 at 9:03 AM Blake Carver <<a href="mailto:blake.carver@lyrasis.org" target="_blank" class="">blake.carver@lyrasis.org</a>> wrote:<br class="">
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="">
<div dir="ltr" class="">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
<a href="https://urldefense.com/v3/__https://gist.github.com/Blake-/d493da28be5554a49a3a3835bbd98f05__;!!HXCxUKc!lsrL41tXx-WGwpdlFWm60sfuIbz0DRfuyn4RvaCAka7R1eaKVNkWL7PbhWHEm8g$" target="_blank" class="">https://gist.github.com/Blake-/d493da28be5554a49a3a3835bbd98f05</a><br class="">
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
<br class="">
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
You'll want to find the date more like '2021-03-14 02:00%' or would it be <span style="background-color:rgb(255,255,255);display:inline" class="">03-13? Whatever the date was this year.</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
<span style="background-color:rgb(255,255,255);display:inline" class="">Find any date with a time between 2-3am and just change it to any real hour.</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">
<span style="background-color:rgb(255,255,255);display:inline" class="">ArchivesSpace did "something" (probably restarted?) at a bad time on Sunday morning and wrote a time that never happened.</span></div>
<div class="">
<div id="gmail-m_-3548360350817852693x_gmail-m_7814653410892901514Signature" class="">
<div class="">
<div id="gmail-m_-3548360350817852693x_gmail-m_7814653410892901514divtagdefaultwrapper" dir="ltr" style="background-color: rgb(255, 255, 255);" class="">
</div>
</div>
</div>
</div>
<div id="gmail-m_-3548360350817852693x_gmail-m_7814653410892901514appendonsend" class=""></div>
<hr style="display:inline-block;width:98%" class="">
<div id="gmail-m_-3548360350817852693x_gmail-m_7814653410892901514divRplyFwdMsg" dir="ltr" class=""><font face="Calibri, sans-serif" style="font-size:11pt" class=""><b class="">From:</b>
<a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org" target="_blank" class="">
archivesspace_users_group-bounces@lyralists.lyrasis.org</a> <<a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org" target="_blank" class="">archivesspace_users_group-bounces@lyralists.lyrasis.org</a>> on behalf of James Bullen <<a href="mailto:james@hudmol.com" target="_blank" class="">james@hudmol.com</a>><br class="">
<b class="">Sent:</b> Sunday, March 14, 2021 11:04 PM<br class="">
<b class="">To:</b> Archivesspace Users Group <<a href="mailto:archivesspace_users_group@lyralists.lyrasis.org" target="_blank" class="">archivesspace_users_group@lyralists.lyrasis.org</a>><br class="">
<b class="">Subject:</b> Re: [Archivesspace_Users_Group] Java error - Java::JavaSql::SQLException: HOUR_OF_DAY: 2 -> 3</font>
<div class=""> </div>
</div>
<div class="">
<div class=""><br class="">
</div>
Oh gee, I’ve seen that horror. And, yes, it is caused by records having times in that non-existent hour. From memory the only fix I could find was to delete the offending rows, or update their times.
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Cheers,</div>
<div class="">James</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Mar 15, 2021, at 1:59 PM, Tom Hanstra <<a href="mailto:hanstra@nd.edu" target="_blank" class="">hanstra@nd.edu</a>> wrote:</div>
<br class="">
<div class="">
<div dir="ltr" class="">I don't seem to be able to win with ArchivesSpace.
<div class=""><br class="">
</div>
<div class="">After having indexing again finally give up because of another Java heap error, I restarted archivesspace again with some different java settings. I believe it actually got through the indexing, at least of the first repository.</div>
<div class=""><br class="">
</div>
<div class="">But tonight, looking at the logs, there is another error that seems to have something to do with, perhaps, daylight savings time? I'm seeing this Java error in the logs, even after a restart:<br class="">
<br class="">
--------</div>
<div class="">INFO: An exception happened during JRuby-Rack startup<br class="">
Java::JavaSql::SQLException: HOUR_OF_DAY: 2 -> 3<br class="">
--- System<br class="">
jruby 9.2.12.0 (2.5.7) 2020-07-01 db01a49ba6 OpenJDK 64-Bit Server VM 25.272-b10 on 1.8.0_272-b10 +jit [linux-x86_64]<br class="">
Time: 2021-03-14 22:46:48 -0400<br class="">
Server: jetty/8.1.5.v20120716<br class="">
jruby.home: uri:<a class="">classloader://META-INF/jruby.home</a><br class="">
<br class="">
--- Context Init Parameters:<br class="">
jruby.max.runtimes = 1<br class="">
jruby.min.runtimes = 1<br class="">
public.root = /<br class="">
rack.env = production<br class="">
<br class="">
--- Backtrace<br class="">
Sequel::DatabaseError: Java::JavaSql::SQLException: HOUR_OF_DAY: 2 -> 3<br class="">
</div>
<div class=""><br class="">
</div>
<div class="">--------</div>
<div class=""><br class="">
</div>
<div class="">Did something get corrupted in the database? What might be happening to result in this error?</div>
<div class="">How do I fix it?</div>
<div class=""><br class="">
</div>
<div class="">Thanks,</div>
<div class="">Tom</div>
<div class="">
<div class=""><br class="">
</div>
-- <br class="">
<div dir="ltr" class="">
<div dir="ltr" class="">
<div class="">
<div dir="ltr" class="">
<div dir="ltr" class="">
<div class=""><b style="font-family:arial,helvetica,sans-serif;font-size:12.7273px;color:rgb(136,136,136)" class="">Tom Hanstra</b><br class="">
</div>
<div style="color:rgb(136,136,136);font-size:12.8px" class="">
<div dir="ltr" class="">
<div dir="ltr" class="">
<div dir="ltr" class="">
<div dir="ltr" class="">
<div style="font-size:12.7273px" class="">
<div class="">
<div class=""><i style="font-size:12.7273px;font-family:arial,helvetica,sans-serif" class="">Sr. Systems Administrator</i></div>
<div class=""><a href="mailto:hanstra@nd.edu" style="color:rgb(17,85,204);font-size:12.7273px;font-family:arial,helvetica,sans-serif" target="_blank" class="">hanstra@nd.edu</a><br class="">
</div>
</div>
<div class=""><span style="font-family:arial,helvetica,sans-serif" class=""><br class="">
</span></div>
</div>
<div style="font-size:12.7273px" class=""><img src="https://docs.google.com/uc?export=download&id=1GFX1KaaMTtQ2Kg2u8bMXt1YwBp96bvf0&revid=0B7APN9POn6xAQ244WWFYMFU3aVJwZ0lxbmVHK3FxNXlCd0RRPQ" class=""><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br class="">
Archivesspace_Users_Group mailing list<br class="">
<a href="mailto:Archivesspace_Users_Group@lyralists.lyrasis.org" target="_blank" class="">Archivesspace_Users_Group@lyralists.lyrasis.org</a><br class="">
<a href="https://urldefense.com/v3/__http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group__;!!HXCxUKc!lsrL41tXx-WGwpdlFWm60sfuIbz0DRfuyn4RvaCAka7R1eaKVNkWL7PbLTKdGT0$" target="_blank" class="">http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
_______________________________________________<br class="">
Archivesspace_Users_Group mailing list<br class="">
<a href="mailto:Archivesspace_Users_Group@lyralists.lyrasis.org" target="_blank" class="">Archivesspace_Users_Group@lyralists.lyrasis.org</a><br class="">
<a href="https://urldefense.com/v3/__http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group__;!!HXCxUKc!lsrL41tXx-WGwpdlFWm60sfuIbz0DRfuyn4RvaCAka7R1eaKVNkWL7PbLTKdGT0$" rel="noreferrer" target="_blank" class="">http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group</a><br class="">
</blockquote>
</div>
<br clear="all" class="">
<div class=""><br class="">
</div>
-- <br class="">
<div dir="ltr" class="">
<div dir="ltr" class="">
<div class="">
<div dir="ltr" class="">
<div dir="ltr" class="">
<div class=""><b style="font-family:arial,helvetica,sans-serif;font-size:12.7273px;color:rgb(136,136,136)" class="">Tom Hanstra</b><br class="">
</div>
<div style="color:rgb(136,136,136);font-size:12.8px" class="">
<div dir="ltr" class="">
<div dir="ltr" class="">
<div dir="ltr" class="">
<div dir="ltr" class="">
<div style="font-size:12.7273px" class="">
<div class="">
<div class=""><i style="font-size:12.7273px;font-family:arial,helvetica,sans-serif" class="">Sr. Systems Administrator</i></div>
<div class=""><a href="mailto:hanstra@nd.edu" style="color:rgb(17,85,204);font-size:12.7273px;font-family:arial,helvetica,sans-serif" target="_blank" class="">hanstra@nd.edu</a><br class="">
</div>
</div>
<div class=""><span style="font-family:arial,helvetica,sans-serif" class=""><br class="">
</span></div>
</div>
<div style="font-size:12.7273px" class=""><img src="https://docs.google.com/uc?export=download&id=1GFX1KaaMTtQ2Kg2u8bMXt1YwBp96bvf0&revid=0B7APN9POn6xAQ244WWFYMFU3aVJwZ0lxbmVHK3FxNXlCd0RRPQ" class=""><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br class="">
Archivesspace_Users_Group mailing list<br class="">
<a href="mailto:Archivesspace_Users_Group@lyralists.lyrasis.org" target="_blank" class="">Archivesspace_Users_Group@lyralists.lyrasis.org</a><br class="">
<a href="http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group" rel="noreferrer" target="_blank" class="">http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group</a><br class="">
</blockquote></div><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div dir="ltr" class="gmail_signature"><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><b style="font-family:arial,helvetica,sans-serif;font-size:12.7273px;color:rgb(136,136,136)" class="">Tom Hanstra</b><br class=""></div><div style="color:rgb(136,136,136);font-size:12.8px" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div style="font-size:12.7273px" class=""><div class=""><div class=""><i style="font-size:12.7273px;font-family:arial,helvetica,sans-serif" class="">Sr. Systems Administrator</i></div><div class=""><a href="mailto:hanstra@nd.edu" style="color:rgb(17,85,204);font-size:12.7273px;font-family:arial,helvetica,sans-serif" target="_blank" class="">hanstra@nd.edu</a><br class=""></div></div><div class=""><span style="font-family:arial,helvetica,sans-serif" class=""><br class=""></span></div></div><div style="font-size:12.7273px" class=""><img src="https://docs.google.com/uc?export=download&id=1GFX1KaaMTtQ2Kg2u8bMXt1YwBp96bvf0&revid=0B7APN9POn6xAQ244WWFYMFU3aVJwZ0lxbmVHK3FxNXlCd0RRPQ" class=""><br class=""></div></div></div></div></div></div></div></div></div></div></div>
_______________________________________________<br class="">Archivesspace_Users_Group mailing list<br class=""><a href="mailto:Archivesspace_Users_Group@lyralists.lyrasis.org" class="">Archivesspace_Users_Group@lyralists.lyrasis.org</a><br class="">http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group<br class=""></div></blockquote></div><br class=""></div></div></div></body></html>