[Archivesspace_Users_Group] Diagnosing issues with ArchivesSpace

[Michael Smith]

Hi Peter & Joshua,

Thanks for getting back to me, here’s some replies to your posts.

> memory allocation for app & solr

ArchivesSpace 3.20 with 35G
Solr 8.11.1 with 512M

> The plugins that you are using probably aren't the culprit, but they can add/override index functionality, so listing those out may help as well.

These are the plugins that are enabled on our Dev / Test / Prod ArchivesSpace.


  *   nla_staff_spreadsheet_importer <https://github.com/nla/nla_staff_spreadsheet_importer> 3.1 running a fork of https://github.com/hudmol/nla_staff_spreadsheet_importer
  *   nla_accession_events<https://github.com/nla/nla_accession_events> 0.2 running a fork of https://github.com/hudmol/accession_events
  *   as_reftracker<https://github.com/hudmol/as_reftracker> 1.0
  *   nla_accessions_summary_reports<https://github.com/nla/nla_accessions_summary_reports> 3.0 running a fork of https://github.com/hudmol/accessions_summary_reports
  *   archivesspace_local<https://github.com/nla/archivesspace_local> nla custom plugin only for various AppConfig and other customisation (locales/en.yml)
  *   nla_accession_reports<https://github.com/nla/nla_accession_reports> running a fork of https://github.com/hudmol/nla_accession_reports/
  *   as_spreadsheet_bulk_updater<https://github.com/hudmol/as_spreadsheet_bulk_updater> 1.5.2


> Couple of things that sprang to mind to check (if you haven't already). Have you noticed this same behavior in an instance that is not in use? IE have you set up a clone of your production instance, let it do its initial full index, and then just let it sit? Do you see errors in the app log that have any bearing on the problem or pop up around or just before the app goes unresponsive or OOM?

We were seeing identical issues in our dev and test instances of AS though a little less frequent (less concurrent editing users).

I have found the cause for our troubles, we upgraded to MySQL 8 in July/August last year, at the time our DBA / Systems Administrators added &serverTimezone=UTC to the connection string in our config.

I’ve confirmed in our test instance that if edits to a record are started, then stopped (in this case around 2023-05-26 13:27:41 AEDT). While editing, the table correctly incremented the timestamp by 10 seconds (INTERVAL_PERIOD in frontend/app/assets/javascripts/update_monitor.js). However, after the edits were stopped a new row with +10 hours was added to the table. As the frontend / backend continue to sync their copies of the active_edit table, the number of rows continued increasing by 1 with the 10-hr difference. After only a minute or so after exiting the record the table had the following rows and would continue to add new rows with timestamps further into the future. Eventually the active_edits would expire when their timestamp was less than current time – 30 secs (EXPIRE_SECONDS in backend/app/model/active_edit.rb), but it couldn’t keep pace with the rows being created.

'17212459','/repositories/2/accessions/5133','mismith','2023-05-28 05:27:41'
'17212458','/repositories/2/accessions/5133','mismith','2023-05-27 19:27:41'
'17212457','/repositories/2/accessions/5133','mismith','2023-05-27 09:27:41'
'17212456','/repositories/2/accessions/5133','mismith','2023-05-26 23:27:41'
'17212455','/repositories/2/accessions/5133','mismith','2023-05-26 13:27:41'
'17212455','/repositories/2/accessions/5133','mismith','2023-05-26 03:27:41'

I’ve confirmed that our production instance has just over 15 million records in the active_edit table with timestamps as far in the future as 2150. We’re planning maintenance to stop ArchivesSpace next week and clear the table, we’ve also updated the time zone in our connection string to the correct time zone &serverTimezone=Sydney/Australia (which matches our server timezone now). We’re also going to take ArchivesSpace up to v3.3.1 at the same time.

These queries helped when it came to working out system / global time zone settings.

SELECT @@GLOBAL.time_zone, @@SESSION.time_zone;
SELECT @@system_time_zone;

Hopefully, I’ve been able to describe that behaviour in a way that’s able to be understood, but feel free to ask me any further questions.


Michael Smith  |  Software Developer
02 6262 1029  |  mismith at nla.gov.au<mailto:mismith at nla.gov.au>  |  National Library of Australia
The National Library of Australia acknowledges Australia’s First Nations Peoples – the First Australians – as the Traditional Owners and Custodians of this land and gives respect to the Elders – past and present – and through them to all Australian Aboriginal and Torres Strait Islander people.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lyralists.lyrasis.org/pipermail/archivesspace_users_group/attachments/20230526/ebe59dd2/attachment.html>