[Archivesspace_Users_Group] Solr checksum verification failed (was: Staging upgrade using Vagrant/Virtualbox, data in MySQL tables, Solr core present, reindexed, yet no data displayed in ASpace application)

Fri Jul 28 11:57:15 EDT 2023

Hi Peter,

"I believe that developers should utilise the version attribute of the<schema> tag better ... and the application should not go beyond comparing this schema version."

I certainly can't speak for the ArchivesSpace development team on what changes could or should be made but for context I think that's trying to solve a different problem from what the original intention was behind the checksum verification.

When ArchivesSpace transitioned from providing an embedded Solr, to requiring an external one, the primary concern was with ensuring that all users have a smooth transition particularly when following the documentation ArchivesSpace provides (and ArchivesSpace encompasses an extremely broad range of implementations: from very small archives with few staff [or volunteers] to archives with many staff and developer resources). We handle a lot of support and in most cases sites are using the exact Solr configurations provided by ArchivesSpace and the checksum verification is intended to "prove" that that is actually the case.

"A single-space difference would result in the exact same schema semantically and AS would still fail on startup"

If you're looking to guarantee that the Solr configurations are exactly those provided by ArchivesSpace this is what you want and has been very helpful for supporting self-hosted installations that have had issues setting up Solr to work with ArchivesSpace. A version attribute check wouldn't be fully equivalent for this purpose: support is scoped "for Solr deployments using the officially supported version with the standard configuration provided by the application. Everything else will be treated as best effort community-led support" (https://archivesspace.github.io/tech-docs/provisioning/solr.html).

"Fourth, most of 'solrconfig.xml' has nothing to do with AS and it is plain wrong to impose a particular version on the Solr administrator."

It's not imposed. As James mentions you can do whatever you want by setting: AppConfig[:solr_verify_checksums] = false

And this would apply in your case because you made changes to Solr configuration. But for many / most sites it's just a piece of background configuration that doesn't interfere with anything and just means that Solr was setup "correctly" for ArchivesSpace.

Best,
Mark

Mark Cooper
Hosting and Community Supported Programs
Lyrasis

________________________________
From: archivesspace_users_group-bounces at lyralists.lyrasis.org <archivesspace_users_group-bounces at lyralists.lyrasis.org> on behalf of James Bullen <james at hudmol.com>
Sent: Thursday, July 27, 2023 3:50 AM
To: Archivesspace Users Group <archivesspace_users_group at lyralists.lyrasis.org>
Subject: Re: [Archivesspace_Users_Group] Solr checksum verification failed (was: Staging upgrade using Vagrant/Virtualbox, data in MySQL tables, Solr core present, reindexed, yet no data displayed in ASpace application)

Hi Peter,

I feel your pain.

AppConfig[:solr_verify_checksums] = false

Cheers,
James

On Jul 27, 2023, at 8:14 PM, Peter Heiner <ph448 at cam.ac.uk> wrote:

Hi all,

I've just come across this error and I can see several problems that I'm sharing, hoping to be useful.

First off, my Solr converted the schema.xml I uploaded into a file named managed-schema, This is by no means unusual for recent versions, but the code that checks the checksums tries to download

AppConfig[':solr_url']/admin/file?file=schema.xml&contentType=text%2Fxml%3Bcharset%3Dutf-8

which trivially fails in this case.
Workaround: copy the file over to the name schema.xml

Second, the schema the downloaded one is being compared with is the one in the solr/ directory of the AS installation, which on the one hand makes sense, but also excludes any local schema modifications, like the ones we had in.
Workaround: copy your schema.xml and solrconfig.xml into your AS installation'ssolr/ directory

At this point you need to restart ArchivesSpace for it to recalculate the checksums for the local XML files, after that it should boot successfully.

Third, checksums are a horrible way of comparing XML documents. A single-space difference would result in the exact same schema semantically and AS would still fail on startup.
Solution: none locally. I believe that developers should utilise the version attribute of the<schema> tag better, coming up with a versioning scheme, updating the version accordingly whenever the schema is changed in development and the application should not go beyond comparing this schema version.

Fourth, most of 'solrconfig.xml' has nothing to do with AS and it is plain wrong to impose a particular version on the Solr administrator.
Solution: developers should create a configoverlay.json file administrators can drop in place that ensures that AS's requirements are satisfied. I believe this is supported by all currently supported versions of Solr.

Thanks,
p
________________________________
From: archivesspace_users_group-bounces at lyralists.lyrasis.org <archivesspace_users_group-bounces at lyralists.lyrasis.org> on behalf of Mark Cyzyk <mcyzyk at gmail.com>
Sent: 16 July 2023 21:09
To: mcyzyk at jhu.edu <mcyzyk at jhu.edu>; Connolly, Ashley <asconnolly at seattleschools.org>; Archivesspace Users Group <archivesspace_users_group at lyralists.lyrasis.org>
Subject: Re: [Archivesspace_Users_Group] Staging upgrade using Vagrant/Virtualbox, data in MySQL tables, Solr core present, reindexed, yet no data displayed in ASpace application

"Inconceivable!"

(I don't think "reload" means what I think it means -- at least not in a Solr context.)

Problem solved and moral of story:

  *   It is not enough to simply copy the Archivesspace Solr config files over to the corresponding directory on the Solr server.
  *   It is not enough to reload the Solr Archivesspace Core with these files in place.
  *   It is not enough to restart the Solr service with these files in place.
  *   You must delete/destroy the existing Archivesspace Core, you must create a new Archivesspace Core, you must copy the Archivesspace Solr config files over to the corresponding directory on the Solr server, and finally for good measure you must restart the Solr service with these files in place.
  *   With a Solr Archivesspace Core up and running, you now are ready to boot up the Archivesspace application and will not get a checksum mismatch error.

Last few lines of my Vagrant routine -- basically Delete, Create, Copy, Restart:

sudo su - solr -c "/opt/solr/bin/solr delete -c archivesspace -p 8983 -deleteConfig true"
sudo rm -Rv /opt/solr/server/solr/configsets/archivesspace
sudo mkdir -p /opt/solr/server/solr/configsets/archivesspace/conf
sudo cp -r /usr/share/archivesspace/solr/* /opt/solr/server/solr/configsets/archivesspace/conf/
sudo chown -R solr:solr /opt/solr/server/solr/configsets/archivesspace/conf/
sudo su - solr -c "/opt/solr/bin/solr create -c archivesspace -p 8983 -d archivesspace"
sudo systemctl restart solr
sudo /usr/share/archivesspace/scripts/setup-database.sh
sudo systemctl start archivesspace

It's all working, upgraded to 3.4.0 with plugins seemingly working.

Spinning in chair,
G'Day!

Mark

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
Mark Cyzyk, M.A., M.L.S.
Library Applications Group
The Sheridan Libraries
The Johns Hopkins University
mcyzyk at jhu.edu<mailto:mcyzyk at jhu.edu>

Verba volant, scripta manent.

On 7/16/23 11:12, Mark Cyzyk wrote:

More:

From what I'm reading, Solr maintains the old config file (schema.xml) in some kind of temp directory.  I'm thinking, therefore, that this old file is cached in this way and is the one being read and that that's how two different checksums are possible?

So my thinking is, maybe once the new Solr config files are copied over to the Solr Archivesspace core, that that core needs to be reloaded in order for Solr to get rid of the cache.

The last few lines from my Vagrant routine:

sudo cp -Rv /usr/share/archivesspace/solr/* /opt/solr/server/solr/configsets/archivesspace/conf/
sudo chown -R solr:solr /opt/solr/server/solr/configsets/archivesspace/conf/
curl "http://localhost:8983/solr/admin/cores?action=RELOAD&core=archivesspace"<http://localhost:8983/solr/admin/cores?action=RELOAD&core=archivesspace>
sudo systemctl restart solr
sudo /usr/share/archivesspace/scripts/setup-database.sh
sudo systemctl start archivesspace

But still no joy!  Still checksum mismatch.

...

As an experiment, in my config.rb I set:

AppConfig[:solr_verify_checksums] = false

Boot up ASpace, and error:

ArchivesSpaceThreadDump: Touch the file '/usr/share/archivesspace/thread_dump_indexer.txt' to trigger a thread dump
I, [2023-07-15T19:24:43.750508 #52921]  INFO -- : Thread-2930: Starting periodic indexer
I, [2023-07-15T19:24:43.752627 #52921]  INFO -- : Thread-2930: Starting PUI indexer
I, [2023-07-15T19:24:43.756222 #52921]  INFO -- : Thread-2932: Staff Indexer [2023-07-15 19:24:43 +0000] Running index round
E, [2023-07-15T19:24:44.997705 #52921] ERROR -- : Thread-2932: SolrIndexerError when indexing records: {
  "responseHeader":{
    "status":400,
    "QTime":38},
  "error":{
    "metadata":[
      "error-class","org.apache.solr.common.SolrException",
      "root-error-class","org.apache.solr.common.SolrException"],
    "msg":"ERROR: [doc=/repositories/3] unknown field 'position_int_sort'",
    "code":400}}

It's as if the new Solr config has not taken effect.  [It's not really a new config, both schemas are Version 1.5.  I'm just trying to overcome this mysterious checksum mismatch...]

Copying the ASpace Solr config files over to the corresponding Solr directory, reloading the Solr Archivesspace core with these new files in place, and restarting the Solr service is not enough.  I feel like I'm missing one final step.

Mark

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
Mark Cyzyk, M.A., M.L.S.
Library Applications Group
The Sheridan Libraries
The Johns Hopkins University
mcyzyk at jhu.edu<mailto:mcyzyk at jhu.edu>

Verba volant, scripta manent.

On 7/15/23 10:18, Mark Cyzyk wrote:
Thanks, Ashley,

I did not know about that Reindexer plugin. Useful!

I am able to reindex now, no prob.

My problem appears to be that the checksum on the schema.xml file from my Production server differs what the checksum on the schema.xml file in the 3.4.0 distribution, and this despite the fact that the both are Version 1.5 of the schema:

Outside the VM, on Disk:
mcyzyk at SPEEDY2022:~/VagrantBoxes/Archivesspace/archivesspace_CURRENT/solr$ sha256sum schema.xml
4d4849771a91d677f255b638cbefa8b8d67a236c56c645bb13c2842480614d78 schema.xml

mcyzyk at SPEEDY2022:~/VagrantBoxes/Archivesspace/archivesspace_NEW/solr$ sha256sum schema.xml
f066f17556bbe49b0526ff70ae885bdc9655666b503227667ef0c4315e4e5b0b  schema.xml

And so, with the Production version of 3.2.0 installed in my VM, when I go to upgrade to 3.4.0 everything initially works -- except this checksum mismatch.

In the VM, I copy the NEW (3.4.0) schema.xml file over into the Solr config and restart Solr.  The files at this point should, and do, have identical checksums.

vagrant at vagrant:~$ sha256sum /usr/share/archivesspace/solr/schema.xml
f066f17556bbe49b0526ff70ae885bdc9655666b503227667ef0c4315e4e5b0b

vagrant at vagrant:~$ sha256sum /opt/solr/server/solr/configsets/archivesspace/conf/schema.xml
f066f17556bbe49b0526ff70ae885bdc9655666b503227667ef0c4315e4e5b0b

And yet the Archivesspace space error persists upon bootup:  "Solr checksum verification failed"

With the two files being identical, I don't see how this is possible.  If ASpace is performing a checksum against schema.xml in the Solr config, and the schema.xml file in the ASpace directory, they are now equal.

Not sure why this error is still happening.

Mark

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
Mark Cyzyk, M.A., M.L.S.
Library Applications Group
The Sheridan Libraries
The Johns Hopkins University
mcyzyk at jhu.edu<mailto:mcyzyk at jhu.edu>

Verba volant, scripta manent.

On 7/14/23 21:27, Connolly, Ashley wrote:
Hey Mark,

Something that stuck out to me as I just did an upgrade to 3.4.0 from 3.3.1 today – I did not have either an indexer_state or indexer_pui_state directory in my Solr AS core, I only had an “index” directory. Running Solr 8.11.2.

I deleted the directory “index” from my Solr core data directory (for me, /home/solr/var-solr/data/archivesspace/data) and restarted Solr. I also installedhttps://github.com/lyrasis/aspace-reindexer and (temporarily) enabled the configuration option to auto-reindex on AS start. Ran the DB migrations per the upgrade docs and after that I had no issues starting ArchivesSpace.

Not sure if this helps or not but as I recall on some systems `rm` will return success if asked to delete a directory that doesn’t exist, so I thought it worth mentioning in case you missed deleting the Solr index. Good luck,

--
Mx. Ashley Rose Connolly (she/her), Senior Systems Engineer, SPS DoTS SysOps

From: archivesspace_users_group-bounces at lyralists.lyrasis.org<mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org><archivesspace_users_group-bounces at lyralists.lyrasis.org><mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org> on behalf of Mark Cyzyk <mcyzyk at gmail.com><mailto:mcyzyk at gmail.com>
Date: Friday, July 14, 2023 at 15:11
To: mcyzyk at jhu.edu<mailto:mcyzyk at jhu.edu> <mcyzyk at jhu.edu><mailto:mcyzyk at jhu.edu>, jesse.martinez at bc.edu<mailto:jesse.martinez at bc.edu><jesse.martinez at bc.edu><mailto:jesse.martinez at bc.edu>
Cc: Archivesspace Users Group <archivesspace_users_group at lyralists.lyrasis.org><mailto:archivesspace_users_group at lyralists.lyrasis.org>
Subject: Re: [Archivesspace_Users_Group] Staging upgrade using Vagrant/Virtualbox, data in MySQL tables, Solr core present, reindexed, yet no data displayed in ASpace application

Sorry to keep this thread going!

I returned to this project a little while ago and checked both schema.xml files and sure enough they have different checksums:

vagrant at vagrant:/usr/share/archivesspace/solr$ sha256sum schema.xml
f066f17556bbe49b0526ff70ae885bdc9655666b503227667ef0c4315e4e5b0b  schema.xml

vagrant at vagrant:/opt/solr/server/solr/configsets/archivesspace/conf$ sha256sum schema.xml
4d4849771a91d677f255b638cbefa8b8d67a236c56c645bb13c2842480614d78  schema.xml

Thinking maybe there was some kind of bit-flipping happening when copying between Vagrant/Virtualbox Host and Guest, I copied the Solr schema.xml file directly from within the VM itself.

cp /usr/share/archivesspace/solr/schema.xml /opt/solr/server/solr/configsets/archivesspace/conf/schema.xml

This results in two files with identical checksums:

vagrant at vagrant:~$ sha256sum /usr/share/archivesspace/solr/schema.xml
f066f17556bbe49b0526ff70ae885bdc9655666b503227667ef0c4315e4e5b0b

vagrant at vagrant:~$ sha256sum /opt/solr/server/solr/configsets/archivesspace/conf/schema.xml
f066f17556bbe49b0526ff70ae885bdc9655666b503227667ef0c4315e4e5b0b

Unfortunately, when I start up ASpace I still get the "Solr checksum verification failed."

I just am not sure how this is possible.  Both files have the same checksums!

Thinking...

Mark

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Mark Cyzyk, M.A., M.L.S.

Library Applications Group

The Sheridan Libraries

The Johns Hopkins University

mcyzyk at jhu.edu<mailto:mcyzyk at jhu.edu>

Verba volant, scripta manent.

On 7/14/23 11:58, Mark Cyzyk wrote:

After-Party:

Now that I have a copy of my production 3.2.0 running just fine via Vagrant, I run my Upgrade routines and ASpace DOES upgrade to 3.4.0!

But, when I do a search for anything I get a nice "Unable to Connect to Database" error message which, I have determined, is a misnomer.  What it means to say is:  "Solr checksum verification failed".

Full error message here:

ArchivesSpaceThreadDump: Touch the file '/usr/share/archivesspace/thread_dump_backend.txt' to trigger a thread dump
I, [2023-07-14T15:40:57.771226 #52824]  INFO -- : Thread-2002: Connecting to database: jdbc:mysql://localhost:3306/archivesspace?user=[REDACTED]&password=[REDACTED]&useUnicode=true&characterEncoding=UTF-8&useSS>
I, [2023-07-14T15:40:58.115235 #52824]  INFO -- : Thread-2002: All tables checked and confirmed set to UTF-8.  Nice job!
      ========================================================================
      A trace file has been written to the following location: /usr/share/archivesspace/data/tmp/aspace_diagnostic_1689349259.txt

      This file contains information that will assist developers in diagnosing
      problems with your ArchivesSpace installation.  Please review the file's
      contents for sensitive information (such as passwords) that you might not
      want to share.
      ========================================================================
Jul 14, 2023 3:40:59 PM org.eclipse.jetty.server.handler.ContextHandler$Context log
INFO: An exception happened during JRuby-Rack startup
Solr checksum verification failed (schema): expected [f066f17556bbe49b0526ff70ae885bdc9655666b503227667ef0c4315e4e5b0b] got [4d4849771a91d677f255b638cbefa8b8d67a236c56c645bb13c2842480614d78]

DB check, OK.
UTF-8 check, OK.
Solr Schema checksum check, Not OK.

And so, I then forced my Vagrant script to simply copy 3.4.0 Solr files* into /opt/solr/server/solr/configsets/archivesspace/conf    At this point, the files in these two directories should be exactly similar with identical checksums.

I restart Solr with these files in place.

I restart ASpace -- still checksum failure.

Trying to think how their checksums can differ.  Should be an exact copy.

Mark

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Mark Cyzyk, M.A., M.L.S.

Library Applications Group

The Sheridan Libraries

The Johns Hopkins University

mcyzyk at jhu.edu<mailto:mcyzyk at jhu.edu>

Verba volant, scripta manent.

On 7/13/23 16:19, Mark Cyzyk wrote:

Thanks Joshua, Jesse, Brian, Blake, Valerie for helping me this week on my VEXING Aspace/Vagrant/Solr issue.

Here is what's now working:  (!)

In my config.rb (for Vagrant version of this project):

# Vagrant ######################################################################
 AppConfig[:db_url] = "jdbc:mysql://localhost:3306/archivesspace?user=REDACTED&password=REDACTED&useUnicode=true&characterEncoding=UTF-8&useSSL=false&allowPublicKeyRetrieval=true"
 AppConfig[:backend_url] = "http://localhost:8089"<http://localhost:8089/>
 AppConfig[:frontend_url] = "http://localhost:8080"<http://localhost:8080/>
 AppConfig[:public_url] = "http://localhost:9999"<http://localhost:9999/>
 AppConfig[:solr_verify_checksums] = true
 AppConfig[:enable_solr] = false
 AppConfig[:solr_url] = "http://localhost:8983/solr/archivesspace"<http://localhost:8983/solr/archivesspace>
################################################################################

This and the following Force Reindex did the trick!

sudo service archivespace stop
sudo rm -rf /usr/share/archivesspace/data/indexer_state/
sudo rm -rf /usr/share/archivesspace/data/indexer_pui_state/
curl -X POST -H 'Content-Type: application/json' --data-binary '{"delete":{"query":"*:*" }}'http://localhost:8983/solr/archivesspace/update?commit=true
sudo service archivesspace start

Seriously appreciative.

Onward!

Mark

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Mark Cyzyk, M.A., M.L.S.

Library Applications Group

The Sheridan Libraries

The Johns Hopkins University

mcyzyk at jhu.edu<mailto:mcyzyk at jhu.edu>

On 7/13/23 11:24 AM, Jesse Martinez wrote:
Hi Mark,

It's normal for the archivesspace core not to have a Data import handler set up. And it's also normal to see that warning message about security in the solr logs.

Thank you for checking that the solr schema.xml file matches. The last thing I can recommend for troubleshooting solr is to bootstrap it with solr data from production, and see if that loads records into the ASpace interface.

Jesse

On Thu, Jul 13, 2023 at 10:49 AM Mark Cyzyk <mcyzyk at gmail.com<mailto:mcyzyk at gmail.com>> wrote:

In the Solr Web Gui --> Archivesspace Core --> Dataimport, there is this error message:

The solrconfig.xml file for this index does not have an operational DataImportHandler defined!

My solrconfig.xml, straight from 3.2.0 codebase:

<?xml version="1.0" encoding="utf-8"?>

<config>

  <lib dir="${solr.install.dir}/dist/" regex="solr-analysis-extras.*\.jar" />

  <lib dir="${solr.install.dir}/contrib/analysis-extras/lucene-libs" regex="lucene-analyzers-icu-.*\.jar" />

  <lib dir="${solr.install.dir}/contrib/analysis-extras/lib/" regex="icu4j-*.*\.jar" />

  <schemaFactory class="ClassicIndexSchemaFactory"/>

  <luceneMatchVersion>8.10</luceneMatchVersion>

  <dataDir>${solr.data.dir:}</dataDir>

  <directoryFactory name="DirectoryFactory" class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}" />

  <updateHandler class="solr.DirectUpdateHandler2">

    <maxPendingDeletes>1000</maxPendingDeletes>

    <autoCommit>

      <maxDocs>1000</maxDocs>

      <maxTime>60000</maxTime>

      <openSearcher>false</openSearcher>

    </autoCommit>

  </updateHandler>

  <query>

    <maxBooleanClauses>1024</maxBooleanClauses>

    <filterCache class="solr.FastLRUCache" size="512" initialSize="512" autowarmCount="0" />

    <queryResultCache class="solr.LRUCache" size="512" initialSize="512" autowarmCount="0" />

    <documentCache class="solr.LRUCache" size="512" initialSize="512" autowarmCount="0" />

    <enableLazyFieldLoading>true</enableLazyFieldLoading>

    <queryResultWindowSize>20</queryResultWindowSize>

    <queryResultMaxDocsCached>200</queryResultMaxDocsCached>

    <useColdSearcher>false</useColdSearcher>

    <maxWarmingSearchers>2</maxWarmingSearchers>

  </query>

  <requestDispatcher handleSelect="false">

    <requestParsers enableRemoteStreaming="true" multipartUploadLimitInKB="2048000" />

    <httpCaching never304="true" />

  </requestDispatcher>

  <requestHandler name="/select" class="solr.SearchHandler">

    <lst name="defaults">

      <str name="defType">edismax</str>

      <str name="echoParams">explicit</str>

      <int name="rows">10</int>

      <str name="df">fullrecord</str>

      <str name="pf">four_part_id^50</str>

      <str name="qf">title^25 four_part_id^50 fullrecord</str>

      <str name="bq">primary_type:resource^100</str>

      <str name="bq">primary_type:accession^100</str>

      <str name="bq">primary_type:subject^50</str>

      <str name="bq">primary_type:agent_person^50</str>

      <str name="bq">primary_type:agent_corporate_entity^30</str>

      <str name="bq">primary_type:agent_family^30</str>

    </lst>

  </requestHandler>

  <requestHandler name="/update" class="solr.UpdateRequestHandler"></requestHandler>

  <requestHandler name="/analysis/document" class="solr.DocumentAnalysisRequestHandler" startup="lazy" />

  <requestHandler name="/analysis/field" startup="lazy" class="solr.FieldAnalysisRequestHandler" />

  <requestHandler name="/replication" class="solr.ReplicationHandler" startup="lazy" />

  <queryResponseWriter name="json" class="solr.JSONResponseWriter">

    <str name="content-type">text/plain; charset=UTF-8</str>

  </queryResponseWriter>

  <admin>

    <defaultQuery>*:*</defaultQuery>

  </admin>

  <requestHandler name="/admin/luke" class="org.apache.solr.handler.admin.LukeRequestHandler" />

</config>

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Mark Cyzyk, M.A., M.L.S.

Library Applications Group

The Sheridan Libraries

The Johns Hopkins University

mcyzyk at jhu.edu<mailto:mcyzyk at jhu.edu>

Verba volant, scripta manent.

On 7/13/23 10:40, Mark Cyzyk wrote:
Hi Mark,

If you go to this URL

http://localhost:8983/solr/#/archivesspace/files?file=schema.xml

does the schema file shown there match the same one you copied over?

https://github.com/archivesspace/archivesspace/blob/v3.2.0/solr/schema.xml

Jesse

Yes, they match:

<schema name="archivesspace" version="1.5">

Thinking...

Mark

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Mark Cyzyk, M.A., M.L.S.

Library Applications Group

The Sheridan Libraries

The Johns Hopkins University

mcyzyk at jhu.edu<mailto:mcyzyk at jhu.edu>

Verba volant, scripta manent.

On 7/12/23 16:20, Jesse Martinez wrote:
Hi Mark,

If you go to this URL

http://localhost:8983/solr/#/archivesspace/files?file=schema.xml

does the schema file shown there match the same one you copied over?

https://github.com/archivesspace/archivesspace/blob/v3.2.0/solr/schema.xml

Jesse

On Wed, Jul 12, 2023 at 3:44 PM Mark Cyzyk <mcyzyk at gmail.com<mailto:mcyzyk at gmail.com>> wrote:
Another clue, from the Log in the Solr Web GUI:

7/12/2023, 3:19:27 PM
WARN false

CoreContainer
Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurityfor more info

Maybe ASpace simply does not have permission to add to Solr?  (And yet, this is just a warning.  And yet, for the sake of replicability, this very same Solr setup worked in Vagrant during our last ASpace upgrade.)

Here, setting up Solr in my Vagrant file (which I recently ran line-by-line, manually, without error):

  solr_provision = <<SCRIPT
cd /opt
sudo wget https://archive.apache.org/dist/lucene/solr/8.10.0/solr-8.10.0.tgz
sudo tar zxf solr-8.10.0.tgz
sudo bash /opt/solr-8.10.0/bin/install_solr_service.sh /opt/solr-8.10.0.tgz
sudo mkdir -p /opt/solr/server/solr/configsets/archivesspace/conf
sudo cp -r /vagrant/archivesspace_CURRENT/solr/* /opt/solr/server/solr/configsets/archivesspace/conf/
sudo chown -R solr:solr /opt/solr
sudo su - solr -c "/opt/solr/bin/solr create -c archivesspace -p 8983 -d archivesspace"
sudo systemctl restart solr
SCRIPT

Reading about Solr and Permissions....

And sorry for all the emails in this thread.  And yet, once this is resolved it may have value for others in future?

Mark

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Mark Cyzyk, M.A., M.L.S.

Library Applications Group

The Sheridan Libraries

The Johns Hopkins University

mcyzyk at jhu.edu<mailto:mcyzyk at jhu.edu>

On 7/12/23 11:46 AM, Joshua D. Shaw wrote:
Hey Mark

After the updates you've made to the config, try shutting down AS and clearing the indexer state directories and restarting. That should tell AS to try a new index.

Also, I think that +0000 is just a timestamp offset. Its not counting the objects indexed. That log line looks something like: PUI Indexer ... indexed x of y resources

Joshua

________________________________
From:archivesspace_users_group-bounces at lyralists.lyrasis.org<mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org><archivesspace_users_group-bounces at lyralists.lyrasis.org><mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org>on behalf of Mark Cyzyk<mcyzyk at gmail.com><mailto:mcyzyk at gmail.com>
Sent:Wednesday, July 12, 2023 11:32 AM
To:Brian Hoffman <brian.hoffman at lyrasis.org><mailto:brian.hoffman at lyrasis.org>; mcyzyk at jhu.edu<mailto:mcyzyk at jhu.edu><mcyzyk at jhu.edu><mailto:mcyzyk at jhu.edu>; Archivesspace Users Group<archivesspace_users_group at lyralists.lyrasis.org><mailto:archivesspace_users_group at lyralists.lyrasis.org>; jesse.martinez at bc.edu<mailto:jesse.martinez at bc.edu><jesse.martinez at bc.edu><mailto:jesse.martinez at bc.edu>
Subject:Re: [Archivesspace_Users_Group] Staging upgrade using Vagrant/Virtualbox, data in MySQL tables, Solr core present, reindexed, yet no data displayed in ASpace application

Brian,

I am trying to duplicate our Production ASpace 3.2.0 in a local Vagrant box so I can stage an upgrade to the latest codebase.

Try uncommenting that line and changing it to:

AppConfig[:solr_url] ="http://localhost:8983/solr/archivesspace"<http://localhost:8983/solr/archivesspace>

Done.  And yet:

************************************************************
  Welcome to ArchivesSpace!
  You can now point your browser to http://localhost:8080<http://localhost:8080/>
************************************************************
I, [2023-07-12T15:28:45.335247 #51936]  INFO -- : Thread-3306: PUI Indexer [2023-07-12 15:28:45 +0000] Running index round
I, [2023-07-12T15:28:47.533454 #51936]  INFO -- : Thread-3306: PUI Indexer [2023-07-12 15:28:47 +0000] Index round complete
I, [2023-07-12T15:28:50.634218 #51936]  INFO -- : Thread-2956: Staff Indexer [2023-07-12 15:28:50 +0000] Running index round
I, [2023-07-12T15:28:52.715507 #51936]  INFO -- : Thread-2956: Staff Indexer [2023-07-12 15:28:52 +0000] Index round complete

Still +0000!

Mark

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Mark Cyzyk, M.A., M.L.S.

Library Applications Group

The Sheridan Libraries

The Johns Hopkins University

mcyzyk at jhu.edu<mailto:mcyzyk at jhu.edu>

Verba volant, scripta manent.

On 7/12/23 11:10, Brian Hoffman wrote:
Hi Mark,

What version of ASpace are you using?

Try uncommenting that line and changing it to:

AppConfig[:solr_url] ="http://localhost:8983/solr/archivesspace"<http://localhost:8983/solr/archivesspace>

From:Mark Cyzyk <mcyzyk at gmail.com><mailto:mcyzyk at gmail.com>
Date: Wednesday, July 12, 2023 at 10:59 AM
To: Brian Hoffman <brian.hoffman at lyrasis.org><mailto:brian.hoffman at lyrasis.org>, mcyzyk at jhu.edu<mailto:mcyzyk at jhu.edu><mcyzyk at jhu.edu><mailto:mcyzyk at jhu.edu>, Archivesspace Users Group<archivesspace_users_group at lyralists.lyrasis.org><mailto:archivesspace_users_group at lyralists.lyrasis.org>, jesse.martinez at bc.edu<mailto:jesse.martinez at bc.edu><jesse.martinez at bc.edu><mailto:jesse.martinez at bc.edu>
Subject: Re: [Archivesspace_Users_Group] Staging upgrade using Vagrant/Virtualbox, data in MySQL tables, Solr core present, reindexed, yet no data displayed in ASpace application

Brian,

That value, commented out because it's the default, in my config is:

## The ArchivesSpace Solr index listens on port 8090 by default.  You can
## set it to something else below.
#AppConfig[:solr_url] = "http://localhost:8090"<http://localhost:8090/>

Mark

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Mark Cyzyk, M.A., M.L.S.

Library Applications Group

The Sheridan Libraries

The Johns Hopkins University

mcyzyk at jhu.edu<mailto:mcyzyk at jhu.edu>

Verba volant, scripta manent.

On 7/12/23 10:34, Brian Hoffman wrote:
Hi Mark,

I am referring to the value for :solr_url in the file ‘config/config.rb’

Brian

From:Mark Cyzyk <mcyzyk at gmail.com><mailto:mcyzyk at gmail.com>
Date: Wednesday, July 12, 2023 at 10:32 AM
To: Brian Hoffman <brian.hoffman at lyrasis.org><mailto:brian.hoffman at lyrasis.org>, mcyzyk at jhu.edu<mailto:mcyzyk at jhu.edu><mcyzyk at jhu.edu><mailto:mcyzyk at jhu.edu>, Archivesspace Users Group<archivesspace_users_group at lyralists.lyrasis.org><mailto:archivesspace_users_group at lyralists.lyrasis.org>, jesse.martinez at bc.edu<mailto:jesse.martinez at bc.edu><jesse.martinez at bc.edu><mailto:jesse.martinez at bc.edu>
Subject: Re: [Archivesspace_Users_Group] Staging upgrade using Vagrant/Virtualbox, data in MySQL tables, Solr core present, reindexed, yet no data displayed in ASpace application

Thanks, Brian,

Here is what I see on screen:

<snapshot1.png>

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Mark Cyzyk, M.A., M.L.S.

Library Applications Group

The Sheridan Libraries

The Johns Hopkins University

mcyzyk at jhu.edu<mailto:mcyzyk at jhu.edu>

Verba volant, scripta manent.

On 7/12/23 10:24, Brian Hoffman wrote:
Hi Mark,

I believe you have incorrectly set the solr url to a screen on the control panel.

Try replacing:

http://localhost:8983/solr/#/~cores/archivesspace

with

http://localhost:8983/solr/archivesspace<http://localhost:8983/solr/#/~cores/archivesspace>

From:archivesspace_users_group-bounces at lyralists.lyrasis.org<mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org><archivesspace_users_group-bounces at lyralists.lyrasis.org><mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org>on behalf of Mark Cyzyk<mcyzyk at gmail.com><mailto:mcyzyk at gmail.com>
Date: Wednesday, July 12, 2023 at 10:19 AM
To: jesse.martinez at bc.edu<mailto:jesse.martinez at bc.edu><jesse.martinez at bc.edu><mailto:jesse.martinez at bc.edu>,mcyzyk at jhu.edu<mailto:mcyzyk at jhu.edu><mcyzyk at jhu.edu><mailto:mcyzyk at jhu.edu>
Cc: Archivesspace Users Group <archivesspace_users_group at lyralists.lyrasis.org><mailto:archivesspace_users_group at lyralists.lyrasis.org>
Subject: Re: [Archivesspace_Users_Group] Staging upgrade using Vagrant/Virtualbox, data in MySQL tables, Solr core present, reindexed, yet no data displayed in ASpace application

Darn:

{

  "responseHeader":{

    "status":0,

    "QTime":187,

    "params":{

      "q":"*:*",

      "indent":"true",

      "q.op":"OR",

      "_":"1689171084647"}},

  "response":{"numFound":0,"start":0,"numFoundExact":true,"docs":[]

  }}

I will look more carefully at my Solr Aspace config (and yet the steps in my Vagrant file for setting this up are the same as last summer when I did this!).

Mark

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Mark Cyzyk, M.A., M.L.S.

Library Applications Group

The Sheridan Libraries

The Johns Hopkins University

mcyzyk at jhu.edu<mailto:mcyzyk at jhu.edu>

Verba volant, scripta manent.

On 7/12/23 10:09, Jesse Martinez wrote:
Thanks for sharing that, Mark!

I have another quick follow up question.

What do you see when you run a solr search like this one:

http://localhost:8983/solr/#/archivesspace/query?q=*:*&q.op=OR&indent=true

You should see all your indexed records. But if you don't see any results then it may indicate that your archivesspace configuration directory is not correct.

Jesse

On Wed, Jul 12, 2023 at 10:05 AM Mark Cyzyk <mcyzyk at gmail.com<mailto:mcyzyk at gmail.com>> wrote:

Thanks, Jesse,

Here is the output of those commands:

ps aux | grep solr | grep -v grep
solr         898  0.0  0.1  19096  9812 ?        Ss   Jul11   0:00 /lib/systemd/systemd --user
solr         910  0.0  0.0 169492  3368 ?        S    Jul11   0:00 (sd-pam)
solr        1223  0.5 12.5 4018692 742508 ?      Sl   Jul11   7:41 java -server -Xms512m -Xmx512m -XX:+UseG1GC -XX:+PerfDisableSharedMem -XX:+ParallelRefProcEnabled -XX:MaxGCPauseMillis=250 -XX:+UseLargePages -XX:+AlwaysPreTouch -XX:+ExplicitGCInvokesConcurrent -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/var/solr/logs/solr_gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=9 -XX:GCLogFileSize=20M -Dsolr.jetty.inetaccess.includes= -Dsolr.jetty.inetaccess.excludes= -Dsolr.log.dir=/var/solr/logs -Djetty.port=8983 -DSTOP.PORT=7983 -DSTOP.KEY=solrrocks -Duser.timezone=UTC -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 8983 /var/solr/logs -Djetty.home=/opt/solr/server -Dsolr.solr.home=/var/solr/data -Dsolr.data.home= -Dsolr.install.dir=/opt/solr -Dsolr.default.confdir=/opt/solr/server/solr/configsets/_default/conf -Dlog4j.configurationFile=/var/solr/log4j2.xml -Xss256k -Dsolr.log.muteconsole -jar start.jar --module=http --module=gzip

Puzzling!

Mark

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Mark Cyzyk, M.A., M.L.S.

Library Applications Group

The Sheridan Libraries

The Johns Hopkins University

mcyzyk at jhu.edu<mailto:mcyzyk at jhu.edu>

Verba volant, scripta manent.

On 7/12/23 09:52, Jesse Martinez wrote:
ps aux | grep solr | grep -v grep

--
Jesse Martinez
Senior Library Applications Developer
O'Neill Library, Boston College
jesse.martinez at bc.edu<mailto:jesse.martinez at bc.edu>
617-552-2509
he/him/his

_______________________________________________
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group at lyralists.lyrasis.org<mailto:Archivesspace_Users_Group at lyralists.lyrasis.org>
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

--
Jesse Martinez
Senior Library Applications Developer
O'Neill Library, Boston College
jesse.martinez at bc.edu<mailto:jesse.martinez at bc.edu>
617-552-2509
he/him/his

--
Jesse Martinez
Senior Library Applications Developer
O'Neill Library, Boston College
jesse.martinez at bc.edu<mailto:jesse.martinez at bc.edu>
617-552-2509
he/him/his

CAUTION: This email originated from outside of the organization. Please don't click links, open attachments, or reply with confidential details unless you are certain you know the sender and are expecting the content.

_______________________________________________
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group at lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lyralists.lyrasis.org/pipermail/archivesspace_users_group/attachments/20230728/32c5a563/attachment-0001.html>