[Archivesspace_Users_Group] Slow searches in Staff webapp.

[Majewski, Steven Dennis (sdm7g)]

We’ve had some issues with certain searches in the staff interface ( v2.6.0 ) and I wonder if anyone else has observed this. 

We were getting proxy errors on some searches from the Staff web app. 
We are adjusting the proxy timeouts in apache to get around this, however, finding that some searches really do take too long. 
I’ve found a pattern in the failing searches and I think I’ve discovered two separate issues. 


First of all — the problem isn’t actually with the SOLR search, which returns quickly, 
But with formatting the results — specifically the ‘Found In’ column where it needs to collect the ancestors titles to show the items place in the hierarchy.
Searches that have only top level resources in the first page of results return quickly as there is no ancestors to display.
Searches with archival_objects in results take longer.  Top_container results appear to be the slowest, as there may be multiple resources in the list. 

Rendering _context.html.erb for top_containers maybe 5-10x slower worst case than rendering top level resources:

https://github.com/archivesspace/archivesspace/blob/master/frontend/app/views/search/_context.html.erb <https://github.com/archivesspace/archivesspace/blob/master/frontend/app/views/search/_context.html.erb>

I haven’t seen a similar issue with PUI searches which seem to use only SOLR, while the staff webapp makes multiple backend API calls that are probably doing SQL access. 
( Which leads me to think that this ancestor field ought to be cached in SOLR for the Staff search to avoid these worst case search events. )


The 2nd issue seems to be that our production server is significantly slower at this than test machines. 
This appears to be a greater factor than can be explained by the greater  load on production server.
I suspect, but haven’t yet proven, that the difference is due to using a central MySql server for production but using a local MySql server on test machines. 

This seems to add a similar order of magnitude slowdown to search results.   
So maybe worst case difference = 10 * 10 = 100x slower.  But not seeing that large a difference on average. 

These are mostly guesses from observing logs — I’m adding some instrumentation to make a more definitive diagnosis.  

— Steve Majewski

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lyralists.lyrasis.org/pipermail/archivesspace_users_group/attachments/20190920/f3bcb0d1/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4974 bytes
Desc: not available
URL: <http://lyralists.lyrasis.org/pipermail/archivesspace_users_group/attachments/20190920/f3bcb0d1/attachment.bin>