<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
Thanks, Blake! <br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
I'm running default config values for the AS log levels so they are all set to 'debug'. I took a closer look, and the timeout message happens exactly after the timeout amount I set (as you'd expect). Interestingly, Solr is in the middle of deleting documents
when it goes silent<br>
<br>
<span style="font-family: "Courier New", monospace;">I, [2023-01-26T09:18:40.357101 #78764] INFO -- : Thread-3384: Deleted 100 documents: #<Net::HTTPOK:0x72b3d9e></span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
.... 40 minutes pass with all the other AS log chatter ...</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted3">
<span style="font-family: "Courier New", monospace;">E, [2023-01-26T09:58:40.400971 #78764] ERROR -- : Thread-3384: SolrIndexerError when deleting records: Timeout error with POST {....}</span><br class="ContentPasted1">
<span style="font-family: "Courier New", monospace;">I, [2023-01-26T09:58:40.410522 #78764] INFO -- : Thread-3384: Deleted 100 documents: #<Net::HTTPOK:0x4ab44e31></span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted3">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted3">
This continuing delete phase goes on for a bit until it stops logging batch deletes.
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted3">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted3 ContentPasted5 ContentPasted6">
<span style="font-family: "Courier New", monospace;">I, [2023-01-26T09:59:11.734200 #78764] INFO -- : Thread-3384: Deleted 9 documents: #<Net::HTTPOK:0x1be6c3e9></span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted3 ContentPasted5 ContentPasted6">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted3 ContentPasted5 ContentPasted6 ContentPasted11">
.... 40 minutes pass with all the other AS log chatter ... And then the commit error pops up<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted3 ContentPasted5 ContentPasted6">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted3 ContentPasted4">
<span style="font-family: "Courier New", monospace;">E, [2023-01-26T10:39:11.746166 #78764] ERROR -- : Thread-3384: SolrIndexerError when committing:
</span>
<div class="ContentPasted4"><span style="font-family: "Courier New", monospace;">Timeout error with POST {"commit":{"softCommit":false}}.</span></div>
<div class="ContentPasted4"><br>
</div>
<div class="ContentPasted4">Then after some more time</div>
<div class="ContentPasted4"><br>
</div>
<div class="ContentPasted4 ContentPasted7"><span style="font-family: "Courier New", monospace;">I, [2023-01-26T11:06:35.678926 #78764] INFO -- : Thread-3384: Deleted 186992 documents: #<Net::HTTPOK:0x7e298af9></span><br class="ContentPasted7">
</div>
<div class="ContentPasted4 ContentPasted7"><br>
</div>
<div class="ContentPasted4 ContentPasted7">.... This all seems to indicate to me that the commit phase is taking an inordinate amount of time (almost 2 hours - maybe that's what I need to set the timeout to?). After that, the indexer starts the 2nd repo<br>
</div>
<div class="ContentPasted4 ContentPasted7"><br>
</div>
<div class="ContentPasted4 ContentPasted7 ContentPasted8"><span style="font-family: "Courier New", monospace;">I, [2023-01-26T11:06:35.765797 #78764] INFO -- : Thread-3384: PUI Indexer [2023-01-26 11:06:35 -0500] Indexed 2 additional PUI records in repository
Sherman</span><br class="ContentPasted8">
</div>
<div class="ContentPasted4 ContentPasted7 ContentPasted8"><br>
</div>
<div class="ContentPasted4 ContentPasted7 ContentPasted8">.... The indexer waits for a looong time with no timeout and no messaging - even though this is a tiny repo - and then starts the 3rd repo<br>
</div>
<div class="ContentPasted4 ContentPasted7 ContentPasted8"><br>
</div>
<div class="ContentPasted4 ContentPasted7 ContentPasted8 ContentPasted9"><span style="font-family: "Courier New", monospace;">I, [2023-01-26T11:31:32.795602 #78764] INFO -- : Thread-3384: PUI Indexer [2023-01-26 11:31:32 -0500] Indexed 188 additional PUI records
in repository Rauner-XO</span><br class="ContentPasted9">
</div>
<div class="ContentPasted4 ContentPasted7 ContentPasted8 ContentPasted9"><br>
</div>
<div class="ContentPasted4 ContentPasted7 ContentPasted8 ContentPasted9">And then the indexer starts the 4th repo soon after and seems to go on to complete normally<br>
</div>
<div class="ContentPasted4 ContentPasted7 ContentPasted8 ContentPasted9"><br>
</div>
<div class="ContentPasted4 ContentPasted7 ContentPasted8 ContentPasted9 ContentPasted10">
<span style="font-family: "Courier New", monospace;">I, [2023-01-26T11:31:33.369369 #78764] INFO -- : Thread-3384: PUI Indexer [2023-01-26 11:31:33 -0500] ~~~ Indexed 25 of 74785 archival_object records in repository thedartmouth</span><br class="ContentPasted10">
<br>
</div>
The Solr logs indicate that Solr is working this entire time doing adds and deletes. For example in one of the quiet phases:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted3 ContentPasted4">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted3 ContentPasted4 ContentPasted12">
<span style="font-family: "Courier New", monospace;">2023-01-26 10:23:35.928 INFO (qtp2101153819-523) [ x:archivesspace] o.a.s.u.p.LogUpdateProcessorFactory [archivesspace] webapp=/solr path=/update params={}{add=...</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted3 ContentPasted4 ContentPasted12 ContentPasted13">
<span style="font-family: "Courier New", monospace;">2023-01-26 10:23:38.195 INFO (qtp2101153819-468) [ x:archivesspace] o.a.s.u.p.LogUpdateProcessorFactory [archivesspace] webapp=/solr path=/update params={}{deleteByQuery=...</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted3 ContentPasted4 ContentPasted12 ContentPasted13">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted3 ContentPasted4 ContentPasted12 ContentPasted13">
So I'm falling back to this just being <b>super</b> slow for some reason. I do have some custom indexer addons, but I'm not sure why the PUI indexer would become
<b>so</b> much slower (21 hours) when the Staff indexer completes in a normal amount of time (a little under 6 hours). For previous versions this hasn't been quite that different (6hrs vs about 13hrs). We do have some collections that are quite large (10s of
thousands of AOs), so maybe that's part of the issue.<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted3 ContentPasted4 ContentPasted12 ContentPasted13">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted3 ContentPasted4 ContentPasted12 ContentPasted13">
I haven't checked to see if the PUI indexer is gathering that much more data (and traversing the tree more times - maybe?) than it was in 3.1.1, but that's on my 'to check' list.<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted3 ContentPasted4 ContentPasted12 ContentPasted13">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted3 ContentPasted4 ContentPasted12 ContentPasted13">
Joshua<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> archivesspace_users_group-bounces@lyralists.lyrasis.org <archivesspace_users_group-bounces@lyralists.lyrasis.org> on behalf of Blake Carver
<blake.carver@lyrasis.org><br>
<b>Sent:</b> Thursday, January 26, 2023 4:12 PM<br>
<b>To:</b> Archivesspace Users Group <archivesspace_users_group@lyralists.lyrasis.org><br>
<b>Subject:</b> Re: [Archivesspace_Users_Group] External Solr - Memory Allocation?</font>
<div> </div>
</div>
<div dir="ltr">
<div class="x_elementToProof">That's... interesting. </div>
<div class="x_elementToProof"><br>
</div>
<div class="x_elementToProof">That RAM allocation seems fine. That Solr timeout is way higher than I would think is needed.</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
Maybe set the loglevel to debug and see if it spits out something more useful? Maybe you'll be able to see what it's up to during that looooong time. I like your theory on that. </div>
<div class="x_elementToProof">
<div id="x_Signature">
<div>
<div id="x_divtagdefaultwrapper" dir="ltr" style="color:rgb(0,0,0); background-color:rgb(255,255,255)">
<div name="x_divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; margin:0px">
<font size="3" face="Calibri,Arial,Helvetica,sans-serif" color="black"><span dir="ltr" style="font-size:12pt; background-color:white"><font size="2"><span style="font-size:11pt"><br>
</span></font></span></font></div>
<div name="x_divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; margin:0px">
<font size="3" face="Calibri,Arial,Helvetica,sans-serif" color="black"><span dir="ltr" style="font-size:12pt; background-color:white"><font size="2"><span style="font-size:11pt"><br>
</span></font></span></font></div>
</div>
</div>
</div>
</div>
<div id="x_appendonsend"></div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> archivesspace_users_group-bounces@lyralists.lyrasis.org <archivesspace_users_group-bounces@lyralists.lyrasis.org> on behalf of Joshua D.
Shaw <Joshua.D.Shaw@dartmouth.edu><br>
<b>Sent:</b> Thursday, January 26, 2023 3:38 PM<br>
<b>To:</b> Archivesspace Users Group <archivesspace_users_group@lyralists.lyrasis.org><br>
<b>Subject:</b> Re: [Archivesspace_Users_Group] External Solr - Memory Allocation?</font>
<div> </div>
</div>
<div dir="ltr">
<div class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
Following up on this. And looking for some advice!<br>
</div>
<div class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<br>
</div>
<div class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
Even with the Solr timeout set to 40 minutes, I'm seeing some random Solr Timeout errors, though these do
<b>NOT</b> cause the indexer to restart. In the latest test run I see one Solr Timeout for delete and one for commit - both following the PUI indexer run for AOs for the first and largest repo (~630k AOs).<br>
</div>
<div class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<br>
</div>
<div class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
The indexer throws the delete timeout error, waits for a loooong time with seemingly no activity, throws the commit timeout error, waits again, and then picks back up as if nothing had gone wrong and continues with the initial index run. All of the index data
looks correct (ie correct number of objects in both the staff and PUI).</div>
<div class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<br>
</div>
<div class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
My theory is that the Solr update phase really is taking a super loooong time, but that the data has all been sent to Solr so the timeouts are really just ArchivesSpace waiting for Solr in between indexing one object type and the next and no index data is lost.<br>
</div>
<div class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<br>
</div>
<div class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
There are no corresponding log entries in the Solr logs that I can find.</div>
<div class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<br>
</div>
<div class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
I'm running solr 8.11.6 with 4GB and AS 3.3.1 with 4GB. Both bare metal on my laptop, so no container issues that might be at play. Solr memory use peaks at around 3.5GB.<br>
</div>
<div class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<br>
</div>
<div class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
I've kept the stock thread and records per thread settings and just upped the timeout (to 2400). I guess the next step is to set the timeout even higher - maybe an hour (3600)? I don't see a reason to run a lower thread or record count, but can certainly try
that as well, though I'm not looking forward to the time it will take (the current run takes 21 hours as it is - up from about 15 for 3.1.1)</div>
<div class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<br>
</div>
<div class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
Any advice appreciated!<br>
</div>
<div class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<br>
</div>
<div class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
Thanks!</div>
<div class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
Joshua<br>
</div>
<div id="x_x_appendonsend"></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_x_divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> archivesspace_users_group-bounces@lyralists.lyrasis.org <archivesspace_users_group-bounces@lyralists.lyrasis.org> on behalf of Joshua
D. Shaw <Joshua.D.Shaw@dartmouth.edu><br>
<b>Sent:</b> Tuesday, January 24, 2023 6:56 AM<br>
<b>To:</b> Archivesspace Users Group <archivesspace_users_group@lyralists.lyrasis.org><br>
<b>Subject:</b> [Archivesspace_Users_Group] External Solr - Memory Allocation?</font>
<div> </div>
</div>
<div dir="ltr">
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
Hey all</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<br>
</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
We're about to jump to v3.3.1 and I'm wondering if anyone has any suggestions for memory allocation for Solr?
<br>
</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<br>
</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
Currently we're running 6GB for the entire suite in v3.1.1 and are looking to keep the same overall memory footprint. Wondering if something like a 75/25 split (ie 4GB for AS and 2GB for Solr) would be a reasonable allocation? Or are people finding that Solr
is more demanding?</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<br>
</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
Thanks!</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
Joshua<br>
</div>
<div class="x_x_x_elementToProof">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div id="x_x_x_Signature">
<div>
<div></div>
<div></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
___________________</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Joshua Shaw (he, him)</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Library Web & Application Developer<br>
</div>
<span style="font-family:Calibri,Helvetica,sans-serif">Digital Library Technologies Group</span><br>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Dartmouth College<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
603.646.0405<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>