<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Hi all</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Just to confirm, Blake and Mark were able to resolve this issue, as we're not the first to hit it, so thanks. It looks as though it was down to a failed merging of top_containers, which left some orphaned sub_containers (with null top_container references in
the top_containers_link_rlshp table). We knew we had this issue but thought it was just affecting those records- however, it does seem to have had the knock on effect of causing login to fail.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
If anyone else hits it, the fix is here: <a href="https://gist.github.com/mark-cooper/bf8b9d5e86255b92b9576f683206ba79" id="LPlnk">https://gist.github.com/mark-cooper/bf8b9d5e86255b92b9576f683206ba79</a></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Cheers</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Scott</div>
<div>
<div id="Signature">
<div>
<div></div>
<div></div>
<div></div>
<div id="divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; font-family: Calibri, Helvetica, sans-serif; color: rgb(0, 0, 0);">
<p style="margin-top: 0px; margin-bottom: 0px;margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin-top:0; margin-bottom:0">
<br>
</p>
<p style="margin-top: 0px; margin-bottom: 0px;margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin-top:0; margin-bottom:0">
==========</p>
<p style="margin-top: 0px; margin-bottom: 0px;margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin-top:0; margin-bottom:0">
Scott Renton</p>
<p style="margin-top: 0px; margin-bottom: 0px;margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin-top:0; margin-bottom:0">
Digital Library Development & Systems</p>
<p style="margin-top: 0px; margin-bottom: 0px;margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin-top:0; margin-bottom:0">
Floor F East</p>
<p style="margin-top: 0px; margin-bottom: 0px;margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin-top:0; margin-bottom:0">
Argyle House</p>
<p style="margin-top: 0px; margin-bottom: 0px;margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin-top:0; margin-bottom:0">
515219</p>
</div>
</div>
</div>
</div>
<div id="appendonsend"></div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size: 11pt;"><b>From:</b> RENTON Scott<br>
<b>Sent:</b> 06 September 2021 14:29<br>
<b>To:</b> archivesspace_users_group@lyralists.lyrasis.org <archivesspace_users_group@lyralists.lyrasis.org><br>
<b>Subject:</b> Connection/login dropping</font>
<div> </div>
</div>
<div dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Hi all</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Our uptime has been a bit flaky the last few weeks- what we're finding is that, after a certain amount of service, users are unable to login, and if that's not noticed or acted on, the site generally stops responding some time later. A restart of the application
resolves it.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
We're running 2.8.1 against MariaDB 5.5-65, and Java 8. I have tried different versions of the mysqlconnector, but this does not seem to make a difference (first noticed on 5.1.39). The MySQL has been tuned to an extent. The VM has 16GB of memory, approximately
10GB of which is apportioned to AS. The MySQL process was restarted last week, and clearing that has still allowed the issue.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
It's pretty hard to spot any particular action which causes this. The logging we see once it's happened looks like this:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<span style="font-family:Consolas,Courier,monospace; font-size:9pt">E, [2021-08-18T06:21:01.548843 #90658] ERROR -- : Thread-28310: Unhandled exception!</span>
<div><span style="font-family:Consolas,Courier,monospace; font-size:9pt">E, [2021-08-18T06:21:01.552325 #90658] ERROR -- :
</span></div>
<div><br>
</div>
<div><span style="font-family:Consolas,Courier,monospace; font-size:9pt">apps.archivesspace.gems.gems.sequel_minus_5_dot_9_dot_0.lib.sequel.model.base.RUBY$method$parent_id$0(/apps/archivesspace/gems/gems/sequel-5.9.0/lib/sequel/model/base.rb:783)</span></div>
<div><span style="font-family:Consolas,Courier,monospace; font-size:9pt">apps.archivesspace.gems.gems.sequel_minus_5_dot_9_dot_0.lib.sequel.model.base.RUBY$method$parent_id$0$__VARARGS__(/apps/archivesspace/gems/gems/sequel-5.9.0/lib/sequel/model/base.rb)</span></div>
<div><span style="font-family:Consolas,Courier,monospace; font-size:9pt">org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:82)</span></div>
<span style="font-family:Consolas,Courier,monospace; font-size:9pt">org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)</span><br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
and will continue to spout "minus<i>_</i>5_dot_9_dot_0" type errors until we reboot.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
I have taken some statistics around the numbers of threads running pre and post bounce. I don't see anything obvious there, and I have left the config.rb value
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(64, 11, 217);">
<span style="font-variant-ligatures:no-common-ligatures">AppConfig[:db_max_c</span><span style="font-variant-ligatures: no-common-ligatures; color: rgb(0, 0, 163); background-color: rgb(224, 228, 9);">onnect</span><span style="font-variant-ligatures:no-common-ligatures">ions]
= proc { 20 + (AppConfig[:indexer_thread_count] * 2) }</span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(64, 11, 217);">
<br>
</p>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
as it is.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
I'm also aware that we have an issue some others have of a proliferation of FATAL errors since merging top/subcontainers (items cannot be found), but I do not know if it is related to this. </div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
However, if anyone recognises this behaviour, and can advise, I'd be grateful to hear about it.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Cheers</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Scott</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<pre style="color: rgb(0, 0, 0);"><br></pre>
</div>
<div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="x_Signature">
<div>
<div></div>
<div></div>
<div></div>
<div id="x_divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; font-family: Calibri, Helvetica, sans-serif; color: rgb(0, 0, 0);">
<p style="margin-top: 0px; margin-bottom: 0px;margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin-top:0px; margin-bottom:0px; margin-top:0; margin-bottom:0">
<br>
</p>
<p style="margin-top: 0px; margin-bottom: 0px;margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin-top:0px; margin-bottom:0px; margin-top:0; margin-bottom:0">
==========</p>
<p style="margin-top: 0px; margin-bottom: 0px;margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin-top:0px; margin-bottom:0px; margin-top:0; margin-bottom:0">
Scott Renton</p>
<p style="margin-top: 0px; margin-bottom: 0px;margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin-top:0px; margin-bottom:0px; margin-top:0; margin-bottom:0">
Digital Library Development & Systems</p>
<p style="margin-top: 0px; margin-bottom: 0px;margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin-top:0px; margin-bottom:0px; margin-top:0; margin-bottom:0">
Floor F East</p>
<p style="margin-top: 0px; margin-bottom: 0px;margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin-top:0px; margin-bottom:0px; margin-top:0; margin-bottom:0">
Argyle House</p>
<p style="margin-top: 0px; margin-bottom: 0px;margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin-top:0px; margin-bottom:0px; margin-top:0; margin-bottom:0">
515219</p>
</div>
</div>
</div>
</div>
</div>
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th’ ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.
</body>
</html>