<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Courier New";}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1841919935;
mso-list-template-ids:1892608454;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7 ;
mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:"Courier New";
mso-bidi-font-family:"Times New Roman";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7 ;
mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0A7 ;
mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:\F0A7 ;
mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7 ;
mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0A7 ;
mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:\F0A7 ;
mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7 ;
mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1
{mso-list-id:1880045278;
mso-list-template-ids:1441041546;}
@list l1:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7 ;
mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level2
{mso-level-number-format:bullet;
mso-level-text:\F0B7 ;
mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level3
{mso-level-number-format:bullet;
mso-level-text:\F0B7 ;
mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7 ;
mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level5
{mso-level-number-format:bullet;
mso-level-text:\F0B7 ;
mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level6
{mso-level-number-format:bullet;
mso-level-text:\F0B7 ;
mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7 ;
mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level8
{mso-level-number-format:bullet;
mso-level-text:\F0B7 ;
mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level9
{mso-level-number-format:bullet;
mso-level-text:\F0B7 ;
mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style>
</head>
<body lang="EN-AU" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">Hi Peter & Joshua,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">Thanks for getting back to me, here’s some replies to your posts.<br>
<br>
> memory allocation for app & solr<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">ArchivesSpace 3.20 with 35G<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">Solr 8.11.1 with 512M<br>
<br>
> The plugins that you are using probably aren't the culprit, but they can add/override index functionality, so listing those out may help as well.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US">These are the plugins that are enabled on our Dev / Test / Prod ArchivesSpace.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<ul style="margin-top:0cm" type="disc">
<li class="MsoNormal" style="mso-list:l0 level1 lfo3"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><a href="https://github.com/nla/nla_staff_spreadsheet_importer" target="_blank" title="Follow link">nla_staff_spreadsheet_importer </a>3.1 running
a fork of <a href="https://github.com/hudmol/nla_staff_spreadsheet_importer">https://github.com/hudmol/nla_staff_spreadsheet_importer</a><o:p></o:p></span></li><li class="MsoNormal" style="mso-list:l0 level1 lfo3"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><a href="https://github.com/nla/nla_accession_events" target="_blank" title="Follow link">nla_accession_events</a> 0.2 running a fork of
<a href="https://github.com/hudmol/accession_events">https://github.com/hudmol/accession_events</a> <o:p></o:p></span></li><li class="MsoNormal" style="mso-list:l0 level1 lfo3"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><a href="https://github.com/hudmol/as_reftracker" target="_blank" title="Follow link">as_reftracker</a> 1.0<o:p></o:p></span></li><li class="MsoNormal" style="mso-list:l0 level1 lfo3"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><a href="https://github.com/nla/nla_accessions_summary_reports" target="_blank" title="Follow link">nla_accessions_summary_reports</a> 3.0 running
a fork of <a href="https://github.com/hudmol/accessions_summary_reports">https://github.com/hudmol/accessions_summary_reports</a><o:p></o:p></span></li><li class="MsoNormal" style="mso-list:l0 level1 lfo3"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><a href="https://github.com/nla/archivesspace_local" target="_blank" title="Follow link">archivesspace_local</a> nla custom plugin only for various
AppConfig and other customisation (locales/en.yml)<o:p></o:p></span></li><li class="MsoNormal" style="mso-list:l0 level1 lfo3"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><a href="https://github.com/nla/nla_accession_reports" target="_blank" title="Follow link">nla_accession_reports</a> running a fork of
<a href="https://github.com/hudmol/nla_accession_reports/">https://github.com/hudmol/nla_accession_reports/</a><o:p></o:p></span></li><li class="MsoNormal" style="mso-list:l0 level1 lfo3"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><a href="https://github.com/hudmol/as_spreadsheet_bulk_updater">as_spreadsheet_bulk_updater</a> 1.5.2<o:p></o:p></span></li></ul>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<pre><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">> </span><span style="color:black">Couple of things that sprang to mind to check (if you haven't already). Have you noticed this same behavior in an instance that is not in use? IE have you set up a clone of your production instance, let it do its initial full index, and then just let it sit? Do you see errors in the app log that have any bearing on the problem or pop up around or just before the app goes unresponsive or OOM?<o:p></o:p></span></pre>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US">We were seeing identical issues in our dev and test instances of AS though a little less frequent (less concurrent editing users).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US">I have found the cause for our troubles, we upgraded to MySQL 8 in July/August last year, at the time our DBA / Systems Administrators added &serverTimezone=UTC to the connection
string in our config.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US">I’ve confirmed in our test instance that if edits to a record are started, then stopped (in this case around
</span><span style="font-size:11.0pt;mso-fareast-language:EN-US">2023-05-26 13:27:41 AEDT). While editing, the table correctly incremented the timestamp by 10 seconds (INTERVAL_PERIOD in frontend/app/assets/javascripts/update_monitor.js). However, after the
edits were stopped a new row with +10 hours was added to the table. As the frontend / backend continue to sync their copies of the active_edit table, the number of rows continued increasing by 1 with the 10-hr difference. After only a minute or so after exiting
the record the table had the following rows and would continue to add new rows with timestamps further into the future. Eventually the active_edits would expire when their timestamp was less than current time – 30 secs (EXPIRE_SECONDS in backend/app/model/active_edit.rb),
but it couldn’t keep pace with the rows being created.</span><span style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US">'17212459','/repositories/2/accessions/5133','mismith','2023-05-28 05:27:41'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US">'17212458','/repositories/2/accessions/5133','mismith','2023-05-27 19:27:41'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US">'17212457','/repositories/2/accessions/5133','mismith','2023-05-27 09:27:41'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US">'17212456','/repositories/2/accessions/5133','mismith','2023-05-26 23:27:41'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US">'17212455','/repositories/2/accessions/5133','mismith','2023-05-26 13:27:41'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US">'17212455','/repositories/2/accessions/5133','mismith','2023-05-26 03:27:41'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US">I’ve confirmed that our production instance has just over 15 million records in the active_edit table with timestamps as far in the future as 2150. We’re planning maintenance to
stop ArchivesSpace next week and clear the table, we’ve also updated the time zone in our connection string to the correct time zone &serverTimezone=Sydney/Australia (which matches our server timezone now). We’re also going to take ArchivesSpace up to v3.3.1
at the same time.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US">These queries helped when it came to working out system / global time zone settings.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US">SELECT @@GLOBAL.time_zone, @@SESSION.time_zone;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US">SELECT @@system_time_zone;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US">Hopefully, I’ve been able to describe that behaviour in a way that’s able to be understood, but feel free to ask me any further questions.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt;line-height:14.0pt"><b><span style="font-size:11.0pt;font-family:"Arial",sans-serif">Michael Smith</span></b><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:black"> |
</span><span style="font-size:11.0pt;font-family:"Arial",sans-serif">Software Developer<span style="color:black"><br>
02 6262 </span>1029<span style="color:black"> | </span></span><a href="mailto:mismith@nla.gov.au" title="mailto:Servicename@nla.gov.au"><span style="font-size:11.0pt;font-family:"Arial",sans-serif">mismith@nla.gov.au</span></a><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:black">
| National Library of Australia</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:0cm;margin-right:4.25pt;margin-bottom:25.0pt;margin-left:0cm;line-height:14.0pt">
<span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:black">The National Library of Australia acknowledges Australia’s First Nations Peoples – the First Australians – as the Traditional Owners and Custodians of this land and gives respect to the
Elders – past and present – and through them to all Australian Aboriginal and Torres Strait Islander people.</span><span style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p></o:p></span></p>
</div>
</div>
</div>
</body>
</html>