[Archivesspace_Users_Group] Indexer erroring in 1.5.2
James Bullen
james at hudmol.com
Tue Mar 28 23:35:25 EDT 2017
Hi Brent,
Sorry for the slow reply. From memory this is caused by bad instance records left around after certain operations on digital objects. The bug that caused this has been fixed, but you may still have some of these bad records lying around.
The problem records are digital object type instances that don’t have a link to a digital object. This SQL will return any such records:
select i.* from instance i
left join instance_do_link_rlshp r on i.id = r.instance_id
left join enumeration_value e on i.instance_type_id = e.id
where instance_type_id = 349 and r.id is null
and e.value = 'digital_object';
If that query returns any rows, then just delete them. Hopefully that will fix the problem.
Cheers,
James
> On Mar 23, 2017, at 9:00 AM, Brent Ellingson <brent_ellingson at byu.edu> wrote:
>
> We are in the process of upgrading our site from 1.4.2 to 1.5.3 and we are seeing a similar error as has been described in this thread. We are seeing the error below and continued looping of the indexer trying to index the archival objects.
>
> ~~~ Indexed 1225 of 67886 archival_object records in repository 2 ( added 25 records in 6399.0ms ) ~~~
> ~~~ Indexed 1250 of 67886 archival_object records in repository 2 ( added 25 records in 5488.0ms ) ~~~
> E, [2017-03-16T16:53:51.425000 #779] ERROR -- : Thread-2392462: Unhandled exception!
> E, [2017-03-16T16:53:51.425000 #779] ERROR -- :
> undefined method `related_records' for nil:NilClass
> /opt/archivesspace-1.5.3/archivesspace/data/tmp/jetty-0.0.0.0-9089-backend.war-_-any-/webapp/WEB-INF/app/lib/subcontainer_to_aspace_json_mapper.rb:64:in `top_container'
> /opt/archivesspace-1.5.3/archivesspace/data/tmp/jetty-0.0.0.0-9089-backend.war-_-any-/webapp/WEB-INF/app/lib/subcontainer_to_aspace_json_mapper.rb:24:in `type_1'
> /opt/archivesspace-1.5.3/archivesspace/data/tmp/jetty-0.0.0.0-9089-backend.war-_-any-/webapp/WEB-INF/app/lib/subcontainer_to_aspace_json_mapper.rb:14:in `to_hash'
> org/jruby/RubyHash.java:1341:in `each'
> org/jruby/RubyEnumerable.java:757:in `map'
>
> Does a resolution to this issue exist?
> How do I find what the offending set of data is causing the problem?
> How do I proceed?
>
> Thank you for your help.
>
> Regards,
> __________________________
> Brent Ellingson
> Sr. Software Engineer
> BYU Library
> 801-422-6148
>
>
> Date: Mon, 23 Jan 2017 19:54:53 +0000
> From: "Majewski, Steven Dennis (sdm7g)" <sdm7g at eservices.virginia.edu <mailto:sdm7g at eservices.virginia.edu>>
> To: Archivesspace Users Group
> <archivesspace_users_group at lyralists.lyrasis.org <mailto:archivesspace_users_group at lyralists.lyrasis.org>>
> Subject: Re: [Archivesspace_Users_Group] Indexer erroring in 1.5.2
> Message-ID:
> <1B0EA75C-EE19-492A-B057-FBAEFBBFB880 at eservices.virginia.edu <mailto:1B0EA75C-EE19-492A-B057-FBAEFBBFB880 at eservices.virginia.edu>>
> Content-Type: text/plain; charset="utf-8"
>
>
> The first two issues appear to be due to trailing spaces in some of the container barcodes.
> Top containers have already been created for barcodes without trailing space, and so it appears that the barcodes don?t match.
>
>
>
> On Jan 20, 2017, at 3:49 PM, Majewski, Steven Dennis (sdm7g) <sdm7g at eservices.virginia.edu<mailto:sdm7g at eservices.virginia.edu <mailto:sdm7g at eservices.virginia.edu%3cmailto:sdm7g at eservices.virginia.edu>>> wrote:
>
>
> On Jan 13, 2017, at 6:15 PM, James Bullen <james at hudmol.com<mailto:james at hudmol.com <mailto:james at hudmol.com%3cmailto:james at hudmol.com>>> wrote:
>
>
> Hi Steve,
>
> That fits the pattern I?m seeing in John?s data. Have you checked the container conversion log for any errors relating to that record?
>
>
> Cheers,
> James
>
>
>
> Just re-did migration from 1.4.2 to 1.5.1 again, and I?m inspecting the logs.
>
> Yes, archival_objects/255 is on that list.
>
> error,message,url
> JSONModel::ValidationException,barcode_1 -- Mismatch when mapping between barcode and barcode_1,http://UL-sdm7g-mbp.local/resolve/readonly?uri=/repositories/3/archival_objects/331<http://ul-sdm7g-mbp.local/resolve/readonly?uri=/repositories/3/archival_objects/331 <http://ul-sdm7g-mbp.local/resolve/readonly?uri=/repositories/3/archival_objects/331>>
> JSONModel::ValidationException,barcode_1 -- Mismatch when mapping between barcode and barcode_1,http://UL-sdm7g-mbp.local/resolve/readonly?uri=/repositories/3/archival_objects/2059<http://ul-sdm7g-mbp.local/resolve/readonly?uri=/repositories/3/archival_objects/2059 <http://ul-sdm7g-mbp.local/resolve/readonly?uri=/repositories/3/archival_objects/2059>>
> JSONModel::ValidationException,indicator_1 -- Mismatch when mapping between indicator and indicator_1,http://UL-sdm7g-mbp.local/resolve/readonly?uri=/repositories/3/archival_objects/255<http://ul-sdm7g-mbp.local/resolve/readonly?uri=/repositories/3/archival_objects/255 <http://ul-sdm7g-mbp.local/resolve/readonly?uri=/repositories/3/archival_objects/255>>
>
>
>
> I need to inspect the other repo logs and see if all of the ones that break are getting caught, but it looks likely.
>
> I?m trying to interpret what exactly those messages are complaining about.
> Do those error messages and object dumps mean anything to you off the bat?
>
> ? Steve.
>
>
> On Jan 14, 2017, at 8:37 AM, Majewski, Steven Dennis (sdm7g) <sdm7g at eservices.virginia.edu<mailto:sdm7g at eservices.virginia.edu <mailto:sdm7g at eservices.virginia.edu%3cmailto:sdm7g at eservices.virginia.edu>>> wrote:
>
>
> It looks like the indexer gives and error because the backend is getting an error turning the SQL model into a JSON model.
>
> For some of the archival objects that are failing to index, the significant difference is that they do not have a sub_container in their instance. Trying to access the json from the backend in 1.5.2 just yields an error message, but going back to test 1.5.1 migration:
>
> [ This is json from the same screen capture in my previous message. ]
>
> archival_object 255 ( which causes error on index )
>
> {
> "lock_version": 0,
> "position": 17,
> "publish": true,
> "ref_id": "aspace_22a35d6a6ee59ac454ca8f35232b0b40",
> "title": "Buttons and uniform parts, souvenirs, and medals",
> "display_string": "Buttons and uniform parts, souvenirs, and medals, undated",
> "restrictions_apply": false,
> "created_by": "admin",
> "last_modified_by": "admin",
> "create_time": "2015-08-11T21:43:55Z",
> "system_mtime": "2017-01-06T22:25:38Z",
> "user_mtime": "2015-08-11T21:43:55Z",
> "suppressed": false,
> "level": "file",
> "jsonmodel_type": "archival_object",
> "external_ids": [],
> "subjects": [],
> "linked_events": [],
> "extents": [],
> "dates": [
> {
> "lock_version": 0,
> "expression": "undated",
> "created_by": "admin",
> "last_modified_by": "admin",
> "create_time": "2015-08-11T21:43:55Z",
> "system_mtime": "2015-08-11T21:43:55Z",
> "user_mtime": "2015-08-11T21:43:55Z",
> "date_type": "inclusive",
> "label": "creation",
> "jsonmodel_type": "date"
> }
> ],
> "external_documents": [],
> "rights_statements": [],
> "linked_agents": [],
> "instances": [
> {
> "lock_version": 0,
> "created_by": "admin",
> "last_modified_by": "admin",
> "create_time": "2015-08-11T21:43:55Z",
> "system_mtime": "2015-08-11T21:43:55Z",
> "user_mtime": "2015-08-11T21:43:55Z",
> "instance_type": "Realia",
> "jsonmodel_type": "instance",
> "is_representative": false,
> "container": {
> "lock_version": 0,
> "indicator_1": "Artifacts 1",
> "barcode_1": "X030898965",
> "created_by": "admin",
> "last_modified_by": "admin",
> "create_time": "2015-08-11T21:43:55Z",
> "system_mtime": "2016-11-22T16:58:54Z",
> "user_mtime": "2015-08-11T21:43:55Z",
> "type_1": "box",
> "jsonmodel_type": "container",
> "container_locations": []
> }
> }
> ],
> "notes": [],
> "uri": "/repositories/3/archival_objects/255",
> "repository": {
> "ref": "/repositories/3"
> },
> "resource": {
> "ref": "/repositories/3/resources/11"
> },
> "has_unpublished_ancestor": false
> }
>
> archival_object 256: ( indexes successfully, once 255 is deleted. )
>
> {
> "lock_version": 0,
> "position": 18,
> "publish": false,
> "ref_id": "aspace_15da7f2a9b0184a65b1280ae05a7598a",
> "title": "Buttons and uniform parts, souvenirs, and medals",
> "display_string": "Buttons and uniform parts, souvenirs, and medals, undated",
> "restrictions_apply": false,
> "created_by": "admin",
> "last_modified_by": "admin",
> "create_time": "2015-08-11T21:43:55Z",
> "system_mtime": "2017-01-06T22:25:38Z",
> "user_mtime": "2015-08-11T21:43:55Z",
> "suppressed": false,
> "level": "file",
> "jsonmodel_type": "archival_object",
> "external_ids": [],
> "subjects": [],
> "linked_events": [],
> "extents": [],
> "dates": [
> {
> "lock_version": 0,
> "expression": "undated",
> "created_by": "admin",
> "last_modified_by": "admin",
> "create_time": "2015-08-11T21:43:55Z",
> "system_mtime": "2015-08-11T21:43:55Z",
> "user_mtime": "2015-08-11T21:43:55Z",
> "date_type": "inclusive",
> "label": "creation",
> "jsonmodel_type": "date"
> }
> ],
> "external_documents": [],
> "rights_statements": [],
> "linked_agents": [],
> "instances": [
> {
> "lock_version": 0,
> "created_by": "admin",
> "last_modified_by": "admin",
> "create_time": "2015-08-11T21:43:55Z",
> "system_mtime": "2015-08-11T21:43:55Z",
> "user_mtime": "2015-08-11T21:43:55Z",
> "instance_type": "Realia",
> "jsonmodel_type": "instance",
> "is_representative": false,
> "sub_container": {
> "lock_version": 0,
> "created_by": "admin",
> "last_modified_by": "admin",
> "create_time": "2017-01-06T22:28:29Z",
> "system_mtime": "2017-01-06T22:28:29Z",
> "user_mtime": "2017-01-06T22:28:29Z",
> "jsonmodel_type": "sub_container",
> "top_container": {
> "ref": "/repositories/3/top_containers/1573"
> }
> },
> "container": {
> "type_1": "box",
> "indicator_1": "Artifacts 2",
> "barcode_1": "X030866782",
> "container_locations": [],
> "lock_version": 1
> }
> }
> ],
> "notes": [],
> "uri": "/repositories/3/archival_objects/256",
> "repository": {
> "ref": "/repositories/3"
> },
> "resource": {
> "ref": "/repositories/3/resources/11"
> },
> "has_unpublished_ancestor": false
> }
>
>
>
> But as I noted previously, I don?t see the significant difference in the data in 1.4.2 that was migrated that would account for one failing and the other succeeding.
>
>
> ( It appears that there are other records causing errors that don?t necessarily fit this same pattern. )
>
> But the fact that this info is missing in the 1.5.1 migration does suggest it?s not a 1.5.2 only bug, but a problem with the container migration in earlier versions that is just made more visible by some 1.5.2 changes.
>
>
> ? Steve.
>
>
>
> On Jan 12, 2017, at 11:39 PM, James Bullen <james at hudmol.com<mailto:james at hudmol.com <mailto:james at hudmol.com%3cmailto:james at hudmol.com>>> wrote:
>
>
> Hi,
>
> It looks like the problem John is seeing with the indexer failing after upgrading to 1.5.2 is caused by an issue with a record following the container conversion. I?m following up with John off-list on the specifics of his problem. Hopefully we?ll be able to confirm this when we get to a resolution.
>
> The container management upgrade has been the source of quite a few bugs and it is on our radar for review.
>
>
> Cheers,
> James
>
>
>
> On Jan 13, 2017, at 1:30 AM, Hambleton, John S <jhamblet at nmu.edu<mailto:jhamblet at nmu.edu <mailto:jhamblet at nmu.edu%3cmailto:jhamblet at nmu.edu>>> wrote:
>
> Yes, thank you,
> I sent my data to you off-list.
>
> John H
> NMU
>
> From: archivesspace_users_group-bounces at lyralists.lyrasis.org<mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org <mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org%3cmailto:archivesspace_users_group-bounces at lyralists.lyrasis.org>> [mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org <mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org>] On Behalf Of James Bullen
> Sent: Wednesday, January 11, 2017 6:56 PM
> To: Archivesspace Users Group <archivesspace_users_group at lyralists.lyrasis.org<mailto:archivesspace_users_group at lyralists.lyrasis.org <mailto:archivesspace_users_group at lyralists.lyrasis.org%3cmailto:archivesspace_users_group at lyralists.lyrasis.org>>>
> Subject: Re: [Archivesspace_Users_Group] Indexer erroring in 1.5.2
>
>
> Hi John and Steve,
>
> Can either of you send some sample data that surfaces this bug?
>
>
> Cheers,
> James
>
>
> On Jan 12, 2017, at 8:17 AM, Majewski, Steven Dennis (sdm7g) <sdm7g at eservices.virginia.edu<mailto:sdm7g at eservices.virginia.edu <mailto:sdm7g at eservices.virginia.edu%3cmailto:sdm7g at eservices.virginia.edu>>> wrote:
>
>
> After seeing this message, I tried 1.5.2 on my Mac (10.11.6) laptop and I?m also seeing the same undefined method `related_records' for nil:NilClass error, whether or not I do an intervening ( and successful, BTW ) update to 1.5.1.
> Reindexing never seems to progress and so browse resources never shows any records.
>
>
> ? Steve Majewski
>
>
>
> On Jan 11, 2017, at 1:58 PM, Hambleton, John S <jhamblet at nmu.edu<mailto:jhamblet at nmu.edu <mailto:jhamblet at nmu.edu%3cmailto:jhamblet at nmu.edu>>> wrote:
>
> Folks,
>
> I am testing archivesspace 1.5.2 on CentOS 6, using a copy of our 1.4.2 production database.
> Wondering why it seems like the indexer is taking so long to finish, I am seeing this in the archivesspace.out log, which makes me think the indexer is caught in an endless loop:
>
> Keep seeing the SAME indexer messages over and over again:
> ~~~ Indexed 1475 of 1941 accession records in repository 3 ( added 25 records in 8229.0ms ) ~~~ ~~~ Indexed 1500 of 1941 accession records in repository 3 ( added 25 records in 5251.0ms ) ~~~
>
> And then,
>
> E, [2017-01-11T13:41:10.740000 #32733] ERROR -- : Thread-6445852: Unhandled exception!
> E, [2017-01-11T13:41:10.741000 #32733] ERROR -- :
> undefined method `related_records' for nil:NilClass /usr/local/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/subcontainer_to_aspace_json_mapper.rb:64:in `top_container'
> /usr/local/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/subcontainer_to_aspace_json_mapper.rb:24:in `type_1'
> /usr/local/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/subcontainer_to_aspace_json_mapper.rb:14:in `to_hash'
> org/jruby/RubyHash.java:1341:in `each'
>
> And then seems to start indexing the same data over again:
> ~~~ Indexed 50 of 1941 accession records in repository 3 ( added 25 records in 11153.0ms ) ~~~ ~~~ Indexed 75 of 1941 accession records in repository 3 ( added 25 records in 10051.0ms ) ~~~ ~~~ Indexed 100 of 1941 accession records in repository 3 ( added 25 records in 9558.0ms ) ~~~
>
> Kind of looks like the indexer errors on ?something? then, tries to reindex the same accession records over and over again. I had followed the advice here:https://github.com/archivesspace/archivesspace/blob/master/UPGRADING_1.5.0.md#conversion <https://github.com/archivesspace/archivesspace/blob/master/UPGRADING_1.5.0.md#conversion>
> before starting archivesspace, that is, ?delete your Solr index files to start with a fresh index?.
>
> Any help is appreciated.
> Thanks,
> John H
> NMU
>
> _______________________________________________
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group at lyralists.lyrasis.org<mailto:Archivesspace_Users_Group at lyralists.lyrasis.org <mailto:Archivesspace_Users_Group at lyralists.lyrasis.org%3cmailto:Archivesspace_Users_Group at lyralists.lyrasis.org>>
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group <http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group>
>
> _______________________________________________
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group at lyralists.lyrasis.org<mailto:Archivesspace_Users_Group at lyralists.lyrasis.org <mailto:Archivesspace_Users_Group at lyralists.lyrasis.org%3cmailto:Archivesspace_Users_Group at lyralists.lyrasis.org>>
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group <http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group>
>
>
>
> _______________________________________________
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group at lyralists.lyrasis.org<mailto:Archivesspace_Users_Group at lyralists.lyrasis.org <mailto:Archivesspace_Users_Group at lyralists.lyrasis.org%3cmailto:Archivesspace_Users_Group at lyralists.lyrasis.org>>
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group <http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group>
>
>
> !DSPAM:587793334611909212977!
>
> _______________________________________________
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group at lyralists.lyrasis.org<mailto:Archivesspace_Users_Group at lyralists.lyrasis.org <mailto:Archivesspace_Users_Group at lyralists.lyrasis.org%3cmailto:Archivesspace_Users_Group at lyralists.lyrasis.org>>
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group <http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group>
>
> !DSPAM:587948af109451970717657!
> _______________________________________________
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group at lyralists.lyrasis.org<mailto:Archivesspace_Users_Group at lyralists.lyrasis.org <mailto:Archivesspace_Users_Group at lyralists.lyrasis.org%3cmailto:Archivesspace_Users_Group at lyralists.lyrasis.org>>
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group <http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group>
>
>
> !DSPAM:587948af109451970717657!
>
> _______________________________________________
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group at lyralists.lyrasis.org<mailto:Archivesspace_Users_Group at lyralists.lyrasis.org <mailto:Archivesspace_Users_Group at lyralists.lyrasis.org%3cmailto:Archivesspace_Users_Group at lyralists.lyrasis.org>>
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group <http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group>
>
> <7ebc58861f8c38997d24d1316e9276ec>_______________________________________________
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group at lyralists.lyrasis.org<mailto:Archivesspace_Users_Group at lyralists.lyrasis.org <mailto:Archivesspace_Users_Group at lyralists.lyrasis.org%3cmailto:Archivesspace_Users_Group at lyralists.lyrasis.org>>
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group <http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group>
>
> !DSPAM:58d2f405237347847715625! _______________________________________________
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group at lyralists.lyrasis.org <mailto:Archivesspace_Users_Group at lyralists.lyrasis.org>
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group <http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group>
>
>
> !DSPAM:58d2f405237347847715625!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lyralists.lyrasis.org/pipermail/archivesspace_users_group/attachments/20170329/0185f839/attachment.html>
More information about the Archivesspace_Users_Group
mailing list