<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
FYI, after experimentation, here is what finally worked!:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<blockquote style="margin-top:0;margin-bottom:0">## By setting the next two options, you can control how many CPU cores are used,
<div>## and the amount of memory that will be consumed by the indexing process (more</div>
<div>## cores and/or more records per thread means more memory used).</div>
<div>AppConfig[:indexer_records_per_thread] = 100</div>
<div>AppConfig[:indexer_thread_count] = 2</div>
<div>AppConfig[:indexer_solr_timeout_seconds] = 999999</div>
<div>#</div>
<div>## PUI Indexer Settings</div>
<div>AppConfig[:pui_indexer_enabled] = true</div>
<div>AppConfig[:pui_indexing_frequency_seconds] = 30</div>
<div>AppConfig[:pui_indexer_records_per_thread] = 100</div>
<div>AppConfig[:pui_indexer_thread_count] = 2</div>
<div>#</div>
</blockquote>
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
This worked on my local VM with 4 GB RAM and on our Development server with 8 GB RAM.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
RAM was crucial here. On my 4 GB VM, I tried with 200 records per thread, 150 records per thread, 125 records per thread, and each time it started to swap out, then ran out of swap, then crashed. At 100 records per thread, it was swapping, but finished the
job in decent time (a couple hours).</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Just so there is a record of this!</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Mark<br>
</div>
<div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="Signature">
<div>
<div></div>
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
<div class="BodyFragment"><font size="2">
<div class="PlainText">-- <br>
<br>
<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><br>
Mark Cyzyk, M.A., M.L.S.<br>
Library Applications Group<br>
The Sheridan Libraries<br>
The Johns Hopkins University<br>
mcyzyk@jhu.edu<br>
<br>
<i>Verba volant, scripta manent.</i><br>
<br>
</div>
</font></div>
</div>
</div>
</div>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Mark Cyzyk<br>
<b>Sent:</b> Monday, July 4, 2022 11:56 AM<br>
<b>To:</b> archivesspace_users_group@lyralists.lyrasis.org <archivesspace_users_group@lyralists.lyrasis.org><br>
<b>Subject:</b> Solr indexing performance</font>
<div> </div>
</div>
<style type="text/css" style="display:none">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div dir="ltr">
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Dear ASpace User Group,</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I've got ASpace 3.2.0 running against external Solr in our Development VM, but Solr is taking a loooong time building the initial index.</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Like, it's been running now for a full week!</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I have tweaked the config.rb settings and restarted the ASpace service, but nothing seems to speed it up.</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
VM:</div>
<blockquote style="margin-top:0; margin-bottom:0">
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
2 cpus</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
8 GB RAM</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
looking at used resources, seems like there is a lot still FREE</div>
</blockquote>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
My settings in config.rb:</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<blockquote style="margin-top:0; margin-bottom:0">## By setting the next two options, you can control how many CPU cores are used,
<div>## and the amount of memory that will be consumed by the indexing process (more</div>
<div>## cores and/or more records per thread means more memory used).</div>
<div>AppConfig[:indexer_records_per_thread] = 250 <-- I bumped this up from 25<br>
</div>
<div>AppConfig[:indexer_thread_count] = 2</div>
<div>AppConfig[:indexer_solr_timeout_seconds] = 999999</div>
<div>#</div>
<div>## PUI Indexer Settings</div>
<div>AppConfig[:pui_indexer_enabled] = true</div>
<div>AppConfig[:pui_indexing_frequency_seconds] = 15 <-- I decreased to this, down from 30<br>
</div>
<div>AppConfig[:pui_indexer_records_per_thread] = 250 <-- I bumped this up from 25<br>
</div>
AppConfig[:pui_indexer_thread_count] = 2</blockquote>
</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Does anyone know how to speed up Solr indexing? I can't seem to find the bottleneck here.</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Advice appreciated,</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Mark<br>
</div>
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div id="x_Signature">
<div>
<div></div>
<div name="x_divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
<div class="x_BodyFragment"><font size="2">
<div class="x_PlainText">-- <br>
<br>
<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><br>
Mark Cyzyk, M.A., M.L.S.<br>
Library Applications Group<br>
The Sheridan Libraries<br>
The Johns Hopkins University<br>
mcyzyk@jhu.edu<br>
<br>
<i>Verba volant, scripta manent.</i><br>
<br>
</div>
</font></div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>