[Archivesspace_Users_Group] Indexer erroring in 1.5.2

James Bullen james at hudmol.com
Mon Jan 23 18:07:46 EST 2017


Hi Steve,

Yes that’s how it looks to me too. It should really be able to handle that case so I would consider this to be a bug.


Cheers,
James


> On Jan 24, 2017, at 6:54 AM, Majewski, Steven Dennis (sdm7g) <sdm7g at eservices.virginia.edu> wrote:
> 
> 
> The first two issues appear to be due to trailing spaces in some of the container barcodes.
> Top containers have already been created for barcodes without trailing space, and so it appears that the barcodes don’t match. 
> 
> 
> 
>> On Jan 20, 2017, at 3:49 PM, Majewski, Steven Dennis (sdm7g) <sdm7g at eservices.virginia.edu <mailto:sdm7g at eservices.virginia.edu>> wrote:
>> 
>>> 
>>> On Jan 13, 2017, at 6:15 PM, James Bullen <james at hudmol.com <mailto:james at hudmol.com>> wrote:
>>> 
>>> 
>>> Hi Steve,
>>> 
>>> That fits the pattern I’m seeing in John’s data. Have you checked the container conversion log for any errors relating to that record?
>>> 
>>> 
>>> Cheers,
>>> James
>>> 
>> 
>> 
>> Just re-did migration from 1.4.2 to 1.5.1 again, and I’m inspecting the logs.
>> 
>> Yes, archival_objects/255 is on that list. 
>> 
>> error,message,url
>> JSONModel::ValidationException,barcode_1 -- Mismatch when mapping between barcode and barcode_1,http://UL-sdm7g-mbp.local/resolve/readonly?uri=/repositories/3/archival_objects/331 <http://ul-sdm7g-mbp.local/resolve/readonly?uri=/repositories/3/archival_objects/331>
>> JSONModel::ValidationException,barcode_1 -- Mismatch when mapping between barcode and barcode_1,http://UL-sdm7g-mbp.local/resolve/readonly?uri=/repositories/3/archival_objects/2059 <http://ul-sdm7g-mbp.local/resolve/readonly?uri=/repositories/3/archival_objects/2059>
>> JSONModel::ValidationException,indicator_1 -- Mismatch when mapping between indicator and indicator_1,http://UL-sdm7g-mbp.local/resolve/readonly?uri=/repositories/3/archival_objects/255 <http://ul-sdm7g-mbp.local/resolve/readonly?uri=/repositories/3/archival_objects/255>
>> 
>> 
>> 
>> I need to inspect the other repo logs and see if all of the ones that break are getting caught, but it looks likely.
>> 
>> I’m trying to interpret what exactly those messages are complaining about. 
>> Do those error messages and object dumps mean anything to you off the bat? 
>> 
>> — Steve. 
>> 
>> 
>>>> On Jan 14, 2017, at 8:37 AM, Majewski, Steven Dennis (sdm7g) <sdm7g at eservices.virginia.edu <mailto:sdm7g at eservices.virginia.edu>> wrote:
>>>> 
>>>> 
>>>> It looks like the indexer gives and error because the backend is getting an error turning the SQL model into a JSON model. 
>>>> 
>>>> For some of the archival objects that are failing to index, the significant difference is that they do not have a sub_container in their instance. Trying to access the json from the backend in 1.5.2 just yields an error message, but going back to test 1.5.1 migration:
>>>> 
>>>> [ This is json from the same screen capture in my previous message. ] 
>>>> 
>>>> archival_object 255 ( which causes error on index )
>>>> 
>>>> {
>>>>   "lock_version": 0,
>>>>   "position": 17,
>>>>   "publish": true,
>>>>   "ref_id": "aspace_22a35d6a6ee59ac454ca8f35232b0b40",
>>>>   "title": "Buttons and uniform parts, souvenirs, and medals",
>>>>   "display_string": "Buttons and uniform parts, souvenirs, and medals, undated",
>>>>   "restrictions_apply": false,
>>>>   "created_by": "admin",
>>>>   "last_modified_by": "admin",
>>>>   "create_time": "2015-08-11T21:43:55Z",
>>>>   "system_mtime": "2017-01-06T22:25:38Z",
>>>>   "user_mtime": "2015-08-11T21:43:55Z",
>>>>   "suppressed": false,
>>>>   "level": "file",
>>>>   "jsonmodel_type": "archival_object",
>>>>   "external_ids": [],
>>>>   "subjects": [],
>>>>   "linked_events": [],
>>>>   "extents": [],
>>>>   "dates": [
>>>>     {
>>>>       "lock_version": 0,
>>>>       "expression": "undated",
>>>>       "created_by": "admin",
>>>>       "last_modified_by": "admin",
>>>>       "create_time": "2015-08-11T21:43:55Z",
>>>>       "system_mtime": "2015-08-11T21:43:55Z",
>>>>       "user_mtime": "2015-08-11T21:43:55Z",
>>>>       "date_type": "inclusive",
>>>>       "label": "creation",
>>>>       "jsonmodel_type": "date"
>>>>     }
>>>>   ],
>>>>   "external_documents": [],
>>>>   "rights_statements": [],
>>>>   "linked_agents": [],
>>>>   "instances": [
>>>>     {
>>>>       "lock_version": 0,
>>>>       "created_by": "admin",
>>>>       "last_modified_by": "admin",
>>>>       "create_time": "2015-08-11T21:43:55Z",
>>>>       "system_mtime": "2015-08-11T21:43:55Z",
>>>>       "user_mtime": "2015-08-11T21:43:55Z",
>>>>       "instance_type": "Realia",
>>>>       "jsonmodel_type": "instance",
>>>>       "is_representative": false,
>>>>       "container": {
>>>>         "lock_version": 0,
>>>>         "indicator_1": "Artifacts 1",
>>>>         "barcode_1": "X030898965",
>>>>         "created_by": "admin",
>>>>         "last_modified_by": "admin",
>>>>         "create_time": "2015-08-11T21:43:55Z",
>>>>         "system_mtime": "2016-11-22T16:58:54Z",
>>>>         "user_mtime": "2015-08-11T21:43:55Z",
>>>>         "type_1": "box",
>>>>         "jsonmodel_type": "container",
>>>>         "container_locations": []
>>>>       }
>>>>     }
>>>>   ],
>>>>   "notes": [],
>>>>   "uri": "/repositories/3/archival_objects/255",
>>>>   "repository": {
>>>>     "ref": "/repositories/3"
>>>>   },
>>>>   "resource": {
>>>>     "ref": "/repositories/3/resources/11"
>>>>   },
>>>>   "has_unpublished_ancestor": false
>>>> }
>>>> 
>>>> archival_object 256: ( indexes successfully, once 255 is deleted. )
>>>> 
>>>> {
>>>>   "lock_version": 0,
>>>>   "position": 18,
>>>>   "publish": false,
>>>>   "ref_id": "aspace_15da7f2a9b0184a65b1280ae05a7598a",
>>>>   "title": "Buttons and uniform parts, souvenirs, and medals",
>>>>   "display_string": "Buttons and uniform parts, souvenirs, and medals, undated",
>>>>   "restrictions_apply": false,
>>>>   "created_by": "admin",
>>>>   "last_modified_by": "admin",
>>>>   "create_time": "2015-08-11T21:43:55Z",
>>>>   "system_mtime": "2017-01-06T22:25:38Z",
>>>>   "user_mtime": "2015-08-11T21:43:55Z",
>>>>   "suppressed": false,
>>>>   "level": "file",
>>>>   "jsonmodel_type": "archival_object",
>>>>   "external_ids": [],
>>>>   "subjects": [],
>>>>   "linked_events": [],
>>>>   "extents": [],
>>>>   "dates": [
>>>>     {
>>>>       "lock_version": 0,
>>>>       "expression": "undated",
>>>>       "created_by": "admin",
>>>>       "last_modified_by": "admin",
>>>>       "create_time": "2015-08-11T21:43:55Z",
>>>>       "system_mtime": "2015-08-11T21:43:55Z",
>>>>       "user_mtime": "2015-08-11T21:43:55Z",
>>>>       "date_type": "inclusive",
>>>>       "label": "creation",
>>>>       "jsonmodel_type": "date"
>>>>     }
>>>>   ],
>>>>   "external_documents": [],
>>>>   "rights_statements": [],
>>>>   "linked_agents": [],
>>>>   "instances": [
>>>>     {
>>>>       "lock_version": 0,
>>>>       "created_by": "admin",
>>>>       "last_modified_by": "admin",
>>>>       "create_time": "2015-08-11T21:43:55Z",
>>>>       "system_mtime": "2015-08-11T21:43:55Z",
>>>>       "user_mtime": "2015-08-11T21:43:55Z",
>>>>       "instance_type": "Realia",
>>>>       "jsonmodel_type": "instance",
>>>>       "is_representative": false,
>>>>       "sub_container": {
>>>>         "lock_version": 0,
>>>>         "created_by": "admin",
>>>>         "last_modified_by": "admin",
>>>>         "create_time": "2017-01-06T22:28:29Z",
>>>>         "system_mtime": "2017-01-06T22:28:29Z",
>>>>         "user_mtime": "2017-01-06T22:28:29Z",
>>>>         "jsonmodel_type": "sub_container",
>>>>         "top_container": {
>>>>           "ref": "/repositories/3/top_containers/1573"
>>>>         }
>>>>       },
>>>>       "container": {
>>>>         "type_1": "box",
>>>>         "indicator_1": "Artifacts 2",
>>>>         "barcode_1": "X030866782",
>>>>         "container_locations": [],
>>>>         "lock_version": 1
>>>>       }
>>>>     }
>>>>   ],
>>>>   "notes": [],
>>>>   "uri": "/repositories/3/archival_objects/256",
>>>>   "repository": {
>>>>     "ref": "/repositories/3"
>>>>   },
>>>>   "resource": {
>>>>     "ref": "/repositories/3/resources/11"
>>>>   },
>>>>   "has_unpublished_ancestor": false
>>>> }
>>>> 
>>>> 
>>>> 
>>>> But as I noted previously, I don’t see the significant difference in the data in 1.4.2 that was migrated that would account for one failing and the other succeeding. 
>>>> 
>>>> 
>>>> ( It appears that there are other records causing errors that don’t necessarily fit this same pattern. ) 
>>>> 
>>>> But the fact that this info is missing in the 1.5.1 migration does suggest it’s not a 1.5.2 only bug, but a problem with the container migration in earlier versions that is just made more visible by some 1.5.2 changes.  
>>>> 
>>>> 
>>>> — Steve. 
>>>> 
>>>> 
>>>> 
>>>>> On Jan 12, 2017, at 11:39 PM, James Bullen <james at hudmol.com <mailto:james at hudmol.com>> wrote:
>>>>> 
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> It looks like the problem John is seeing with the indexer failing after upgrading to 1.5.2 is caused by an issue with a record following the container conversion. I’m following up with John off-list on the specifics of his problem. Hopefully we’ll be able to confirm this when we get to a resolution.
>>>>> 
>>>>> The container management upgrade has been the source of quite a few bugs and it is on our radar for review.
>>>>> 
>>>>> 
>>>>> Cheers,
>>>>> James
>>>>> 
>>>>> 
>>>>> 
>>>>>> On Jan 13, 2017, at 1:30 AM, Hambleton, John S <jhamblet at nmu.edu <mailto:jhamblet at nmu.edu>> wrote:
>>>>>> 
>>>>>> Yes, thank you,
>>>>>> I sent my data to you off-list.
>>>>>>  
>>>>>> John H
>>>>>> NMU
>>>>>>  
>>>>>> From: archivesspace_users_group-bounces at lyralists.lyrasis.org <mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org> [mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org <mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org>] On Behalf Of James Bullen
>>>>>> Sent: Wednesday, January 11, 2017 6:56 PM
>>>>>> To: Archivesspace Users Group <archivesspace_users_group at lyralists.lyrasis.org <mailto:archivesspace_users_group at lyralists.lyrasis.org>>
>>>>>> Subject: Re: [Archivesspace_Users_Group] Indexer erroring in 1.5.2
>>>>>>  
>>>>>>  
>>>>>> Hi John and Steve,
>>>>>>  
>>>>>> Can either of you send some sample data that surfaces this bug?
>>>>>>  
>>>>>>  
>>>>>> Cheers,
>>>>>> James
>>>>>>  
>>>>>>  
>>>>>> On Jan 12, 2017, at 8:17 AM, Majewski, Steven Dennis (sdm7g) <sdm7g at eservices.virginia.edu <mailto:sdm7g at eservices.virginia.edu>> wrote:
>>>>>>  
>>>>>>  
>>>>>> After seeing this message, I tried 1.5.2 on my Mac (10.11.6) laptop and I’m also seeing the same 
>>>>>> undefined method `related_records' for nil:NilClass  
>>>>>> error, whether or not I do an intervening ( and successful, BTW ) update to 1.5.1. 
>>>>>> Reindexing never seems to progress and so browse resources never shows any records. 
>>>>>>  
>>>>>>  
>>>>>> — Steve Majewski
>>>>>>  
>>>>>>  
>>>>>>  
>>>>>> On Jan 11, 2017, at 1:58 PM, Hambleton, John S <jhamblet at nmu.edu <mailto:jhamblet at nmu.edu>> wrote:
>>>>>>  
>>>>>> Folks,
>>>>>>  
>>>>>> I am testing archivesspace 1.5.2 on CentOS 6, using a copy of our 1.4.2 production database.
>>>>>> Wondering why it seems like the indexer is taking so long to finish, I am seeing this in
>>>>>> the archivesspace.out log, which makes me think the indexer is caught in an endless loop:
>>>>>>  
>>>>>> Keep seeing the SAME indexer messages over and over again:
>>>>>> ~~~ Indexed 1475 of 1941 accession records in repository 3 ( added 25 records in  8229.0ms ) ~~~
>>>>>> ~~~ Indexed 1500 of 1941 accession records in repository 3 ( added 25 records in  5251.0ms ) ~~~
>>>>>>  
>>>>>> And then,
>>>>>>  
>>>>>> E, [2017-01-11T13:41:10.740000 #32733] ERROR -- : Thread-6445852: Unhandled exception!
>>>>>> E, [2017-01-11T13:41:10.741000 #32733] ERROR -- :
>>>>>> undefined method `related_records' for nil:NilClass
>>>>>> /usr/local/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/subcontainer_to_aspace_json_mapper.rb:64:in `top_container'
>>>>>> /usr/local/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/subcontainer_to_aspace_json_mapper.rb:24:in `type_1'
>>>>>> /usr/local/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/subcontainer_to_aspace_json_mapper.rb:14:in `to_hash'
>>>>>> org/jruby/RubyHash.java:1341:in `each'
>>>>>>  
>>>>>> And then seems to start indexing the same data over again:
>>>>>> ~~~ Indexed 50 of 1941 accession records in repository 3 ( added 25 records in  11153.0ms ) ~~~
>>>>>> ~~~ Indexed 75 of 1941 accession records in repository 3 ( added 25 records in  10051.0ms ) ~~~
>>>>>> ~~~ Indexed 100 of 1941 accession records in repository 3 ( added 25 records in  9558.0ms ) ~~~
>>>>>>  
>>>>>> Kind of looks like the indexer errors on “something” then, tries to reindex the same accession
>>>>>> records over and over again. I had followed the advice here:https://github.com/archivesspace/archivesspace/blob/master/UPGRADING_1.5.0.md#conversion <https://github.com/archivesspace/archivesspace/blob/master/UPGRADING_1.5.0.md#conversion>
>>>>>> before starting archivesspace, that is, “delete your Solr index files to start with a fresh index”.
>>>>>>  
>>>>>> Any help is appreciated.
>>>>>> Thanks,
>>>>>> John H
>>>>>> NMU
>>>>>>  
>>>>>> _______________________________________________
>>>>>> Archivesspace_Users_Group mailing list
>>>>>> Archivesspace_Users_Group at lyralists.lyrasis.org <mailto:Archivesspace_Users_Group at lyralists.lyrasis.org>
>>>>>> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group <http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group>
>>>>>> 
>>>>>> _______________________________________________
>>>>>> Archivesspace_Users_Group mailing list
>>>>>> Archivesspace_Users_Group at lyralists.lyrasis.org <mailto:Archivesspace_Users_Group at lyralists.lyrasis.org>
>>>>>> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group <http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group>
>>>>>> 
>>>>>> 
>>>>>>  
>>>>>>  _______________________________________________
>>>>>> Archivesspace_Users_Group mailing list
>>>>>> Archivesspace_Users_Group at lyralists.lyrasis.org <mailto:Archivesspace_Users_Group at lyralists.lyrasis.org>
>>>>>> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group <http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group>
>>>>>> 
>>>>>> 
>>>>>> !DSPAM:587793334611909212977!
>>>>> 
>>>>> _______________________________________________
>>>>> Archivesspace_Users_Group mailing list
>>>>> Archivesspace_Users_Group at lyralists.lyrasis.org <mailto:Archivesspace_Users_Group at lyralists.lyrasis.org>
>>>>> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group <http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group>
>>>> 
>>>> !DSPAM:587948af109451970717657! 
>>>> _______________________________________________
>>>> Archivesspace_Users_Group mailing list
>>>> Archivesspace_Users_Group at lyralists.lyrasis.org <mailto:Archivesspace_Users_Group at lyralists.lyrasis.org>
>>>> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group <http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group>
>>>> 
>>>> 
>>>> !DSPAM:587948af109451970717657!
>>> 
>>> _______________________________________________
>>> Archivesspace_Users_Group mailing list
>>> Archivesspace_Users_Group at lyralists.lyrasis.org <mailto:Archivesspace_Users_Group at lyralists.lyrasis.org>
>>> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group <http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group>
>> 
>> <7ebc58861f8c38997d24d1316e9276ec>_______________________________________________
>> Archivesspace_Users_Group mailing list
>> Archivesspace_Users_Group at lyralists.lyrasis.org <mailto:Archivesspace_Users_Group at lyralists.lyrasis.org>
>> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group <http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group>
> !DSPAM:58865fa6242613324871184!
> _______________________________________________
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group at lyralists.lyrasis.org
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
> 
> 
> !DSPAM:58865fa6242613324871184!

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lyralists.lyrasis.org/pipermail/archivesspace_users_group/attachments/20170124/bcf5057d/attachment-0001.html>


More information about the Archivesspace_Users_Group mailing list