[Archivesspace_Users_Group] Java error - Java::JavaSql::SQLException: HOUR_OF_DAY: 2 -> 3

Brian Hoffman brian.hoffman at lyrasis.org
Tue Mar 16 08:50:42 EDT 2021


I created an issue for this in JIRA with references to the information shared by Megan and James.

https://archivesspace.atlassian.net/browse/ANW-1229

Brian

From: <archivesspace_users_group-bounces at lyralists.lyrasis.org> on behalf of James Bullen <james at hudmol.com>
Reply-To: Archivesspace Users Group <archivesspace_users_group at lyralists.lyrasis.org>
Date: Monday, March 15, 2021 at 7:49 PM
To: Archivesspace Users Group <archivesspace_users_group at lyralists.lyrasis.org>
Subject: Re: [Archivesspace_Users_Group] Java error - Java::JavaSql::SQLException: HOUR_OF_DAY: 2 -> 3


Many thanks Megan for the lead. I just poked around and found a few things out.

It turns out to be interesting in at least three bad ways.

1.  User records are updated every time they authenticate. This is unnecessary because it is just updating their authentication source, which usually never changes. So this line could be made conditional on the source being different:
https://github.com/archivesspace/archivesspace/blob/master/backend/app/model/authentication_manager.rb#L40

2.  The search_indexer user is being re-authenticated a lot. This shouldn’t happen because it should be a non-expiring session. I haven’t looked into this one, but I did check the lock_version on the search_inxdexer user on a large-ish AS db I have lying around, that has been in production for less than a year and it is over 100,000. That’s nuts. Actually I just took a quick look and saw this:
https://github.com/archivesspace/archivesspace/blob/master/indexer/app/lib/periodic_indexer.rb#L285
The session is reset if there is an error during an indexing round. So that could be the culprit.

3.  There is clearly a bug somewhere in the stack, probably in the Sequel layer, because it shouldn’t crash the system just because you have a whacky datetime somewhere. This bug may well have been fixed in a more recent version of that component.


Dealing with 1 will likely fix the problem, but 2 and 3 should also be investigated.


Cheers,
James




On Mar 16, 2021, at 1:24 AM, Tom Hanstra <hanstra at nd.edu<mailto:hanstra at nd.edu>> wrote:

Thanks, Megan. That update did fix things.

So, is this not something that can be addressed at the software fix level? Do other sites simply stop indexing or turn something else off when we hit DST in March?

Tom

On Mon, Mar 15, 2021 at 10:12 AM Schanz, Megan <schanzme at msu.edu<mailto:schanzme at msu.edu>> wrote:
I run across this every March for daylight savings. This is what I have in my notes to do each time if the indexer is running. Luckily it seems like I didn't have this scenario this year.

java.lang.IllegalArgumentException: HOUR_OF_DAY: 2 -> 3:Java::JavaLang::IllegalArgumentException:
This error will appear in the logs when trying to start ArchivesSpace after daylight savings time. Reference: http://lyralists.lyrasis.org/mailman/htdig/archivesspace_users_group/2019-March/006652.html
Basically, the indexer user is being updated in the database with a time that does not exist due to daylight savings time.
To verify:

SELECT * FROM user WHERE (user_mtime >= '2021-03-14 02:00:00' and user_mtime <= '2021-03-14 03:00:00') OR (system_mtime >= '2021-03-04 02:00:00' and system_mtime <= '2021-03-14 03:00:00');
The record that should come back is the search_indexer user.
To resolve:

UPDATE user set user_mtime = NOW(), system_mtime=NOW() where username='search_indexer';

________________________________
From: archivesspace_users_group-bounces at lyralists.lyrasis.org<mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org> <archivesspace_users_group-bounces at lyralists.lyrasis.org<mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org>> on behalf of Tom Hanstra <hanstra at nd.edu<mailto:hanstra at nd.edu>>
Sent: Monday, March 15, 2021 9:10 AM
To: Archivesspace Users Group <archivesspace_users_group at lyralists.lyrasis.org<mailto:archivesspace_users_group at lyralists.lyrasis.org>>
Subject: Re: [Archivesspace_Users_Group] Java error - Java::JavaSql::SQLException: HOUR_OF_DAY: 2 -> 3

Any suggestion on what ArchivesSpace might have changed?  I had the server running but indexing was complete. What might it have been changing and in what database table would I look for that change?

Alternately, since this is still test data, should I just overlay a backup copy of the database from earlier than the EDT cutover time?  I don't know what affect that might have on other portions of ArchivesSpace.

Finally, what is the overall fix for this issue. If others have seen it, what can be done to make sure to avoid it in the future?

Thanks,
Tom

On Mon, Mar 15, 2021 at 9:03 AM Blake Carver <blake.carver at lyrasis.org<mailto:blake.carver at lyrasis.org>> wrote:
https://gist.github.com/Blake-/d493da28be5554a49a3a3835bbd98f05<https://urldefense.com/v3/__https:/gist.github.com/Blake-/d493da28be5554a49a3a3835bbd98f05__;!!HXCxUKc!lsrL41tXx-WGwpdlFWm60sfuIbz0DRfuyn4RvaCAka7R1eaKVNkWL7PbhWHEm8g$>

You'll want to find the date more like '2021-03-14 02:00%' or would it be 03-13? Whatever the date was this year.
Find any date with a time between 2-3am and just change it to any real hour.
ArchivesSpace did "something" (probably restarted?) at a bad time on Sunday morning and wrote a time that never happened.
________________________________
From: archivesspace_users_group-bounces at lyralists.lyrasis.org<mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org> <archivesspace_users_group-bounces at lyralists.lyrasis.org<mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org>> on behalf of James Bullen <james at hudmol.com<mailto:james at hudmol.com>>
Sent: Sunday, March 14, 2021 11:04 PM
To: Archivesspace Users Group <archivesspace_users_group at lyralists.lyrasis.org<mailto:archivesspace_users_group at lyralists.lyrasis.org>>
Subject: Re: [Archivesspace_Users_Group] Java error - Java::JavaSql::SQLException: HOUR_OF_DAY: 2 -> 3


Oh gee, I’ve seen that horror. And, yes, it is caused by records having times in that non-existent hour. From memory the only fix I could find was to delete the offending rows, or update their times.


Cheers,
James




On Mar 15, 2021, at 1:59 PM, Tom Hanstra <hanstra at nd.edu<mailto:hanstra at nd.edu>> wrote:

I don't seem to be able to win with ArchivesSpace.

After having indexing again finally give up because of another Java heap error, I restarted archivesspace again with some different java settings. I believe it actually got through the indexing, at least of the first repository.

But tonight, looking at the logs, there is another error that seems to have something to do with, perhaps, daylight savings time?  I'm seeing this Java error in the logs, even after a restart:

--------
INFO: An exception happened during JRuby-Rack startup
Java::JavaSql::SQLException: HOUR_OF_DAY: 2 -> 3
--- System
jruby 9.2.12.0 (2.5.7) 2020-07-01 db01a49ba6 OpenJDK 64-Bit Server VM 25.272-b10 on 1.8.0_272-b10 +jit [linux-x86_64]
Time: 2021-03-14 22:46:48 -0400
Server: jetty/8.1.5.v20120716
jruby.home: uri:classloader://META-INF/jruby.home

--- Context Init Parameters:
jruby.max.runtimes = 1
jruby.min.runtimes = 1
public.root = /
rack.env = production

--- Backtrace
Sequel::DatabaseError: Java::JavaSql::SQLException: HOUR_OF_DAY: 2 -> 3

--------

Did something get corrupted in the database?  What might be happening to result in this error?
How do I fix it?

Thanks,
Tom

--
Tom Hanstra
Sr. Systems Administrator
hanstra at nd.edu<mailto:hanstra at nd.edu>

[Image removed by sender.]
_______________________________________________
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group at lyralists.lyrasis.org<mailto:Archivesspace_Users_Group at lyralists.lyrasis.org>
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group<https://urldefense.com/v3/__http:/lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group__;!!HXCxUKc!lsrL41tXx-WGwpdlFWm60sfuIbz0DRfuyn4RvaCAka7R1eaKVNkWL7PbLTKdGT0$>

_______________________________________________
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group at lyralists.lyrasis.org<mailto:Archivesspace_Users_Group at lyralists.lyrasis.org>
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group<https://urldefense.com/v3/__http:/lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group__;!!HXCxUKc!lsrL41tXx-WGwpdlFWm60sfuIbz0DRfuyn4RvaCAka7R1eaKVNkWL7PbLTKdGT0$>


--
Tom Hanstra
Sr. Systems Administrator
hanstra at nd.edu<mailto:hanstra at nd.edu>

[Image removed by sender.]
_______________________________________________
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group at lyralists.lyrasis.org<mailto:Archivesspace_Users_Group at lyralists.lyrasis.org>
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group


--
Tom Hanstra
Sr. Systems Administrator
hanstra at nd.edu<mailto:hanstra at nd.edu>

[Image removed by sender.]
_______________________________________________
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group at lyralists.lyrasis.org<mailto:Archivesspace_Users_Group at lyralists.lyrasis.org>
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lyralists.lyrasis.org/pipermail/archivesspace_users_group/attachments/20210316/3b77e602/attachment.html>


More information about the Archivesspace_Users_Group mailing list