Investigate EML semantic annotation indexing issues on the CNs

[amoeba]

@mbjones and @taojing2002 saw some errors reported in the CN indexing logs. @taojing2002 and I looked and couldn't find the errors that were reported but I decided to just go ahead and verify there weren't any issues.

I started by assuming the ADC indexing was working and queried the ADC Solr index for documents with semantic annotations (n=1351). I then checked that each object (1) existed on the CN and (2) was indexed on the CN and (3) if indexed, whether or not it had annotations.

Status	Count
Not found	1
Not indexed	58
Indexed, but missing annotations	11
Indexed, w/ annotations	1281

We should manually harvest that one "Not found" object, manually reindex the 58 and 11 above.

Full details, with PIDs

NOT FOUND

These need to get harvested

urn:uuid:a6bbe9d0-c281-4402-bf88-4f3c52c66fda

NOT INDEXED

These need to get indexed

doi:10.18739/A2VM42X50
doi:10.18739/A2P26Q37R
doi:10.18739/A2DN3ZW19
doi:10.18739/A2513TW18
doi:10.18739/A23B5W79P
doi:10.18739/A27S7HS4C
doi:10.18739/A2MC8RG5C
doi:10.18739/A2WH2DF2X
doi:10.18739/A2J960930
doi:10.18739/A25717N6T
doi:10.18739/A2416T00Z
doi:10.18739/A21834279
doi:10.18739/A2S46H60V
doi:10.18739/A2ZP3W08H
doi:10.18739/A2P55DG9K
doi:10.18739/A2CV4BR73
doi:10.18739/A20G3GZ2F
doi:10.18739/A24746R6K
doi:10.18739/A2VX06340
doi:10.18739/A2M61BQ0R
doi:10.18739/A2CJ87K8H
doi:10.18739/A2DJ58G97
doi:10.18739/A2086356K
doi:10.18739/A2930NV40
doi:10.18739/A24B2X49X
doi:10.18739/A2SX64931
doi:10.18739/A2599Z20Q
doi:10.18739/A2M32N96B
doi:10.18739/A2RX93D3S
doi:10.18739/A2FJ29C9G
doi:10.18739/A23775V6B
doi:10.18739/A2QZ22H33
doi:10.18739/A2QV3C418
doi:10.18739/A2WP9T67J
doi:10.18739/A2DV1CN8T
doi:10.18739/A21G0HV2P
doi:10.18739/A2BV79V7V
urn:uuid:69a40625-277a-4793-aa10-f148332d2456
doi:10.18739/A2N58CK9B
doi:10.18739/A2WS8HM1Z
doi:10.18739/A2707WP0Q
doi:10.18739/A21J9776B
doi:10.18739/A2CN6Z02Z
doi:10.18739/A2NC5SC63
doi:10.18739/A2HD7NS6P
doi:10.18739/A2GM81P1M
doi:10.18739/A24B2X59C
doi:10.18739/A2VQ2S97V
doi:10.1594/PANGAEA.779181
urn:uuid:29fbd2eb-3319-46ed-b416-3638ec020571
urn:uuid:40b5819c-a8d8-4f82-a9c4-ce2ec6cec1f0
urn:uuid:4cc06919-9562-4b4b-af99-fe524f118181
urn:uuid:d59a7b20-5704-4d37-9ee1-78e7a2e78982
urn:uuid:445deff9-b8cb-4023-8d7e-52802e429358
urn:uuid:b9b256da-0a15-459c-9b5a-36195e0dbb59
urn:uuid:b59de2d0-8531-456f-ab1a-dd009df9c844
urn:uuid:1c6521de-e47e-46ca-b9c8-d3910fe1fa9c
urn:uuid:02022a31-97b5-4178-b692-6d2a77c120eb

INDEXED, BUT NO ANNOTATIONS

These need reindexing and verification after they're reindexed

doi:10.18739/A28W3827B
doi:10.18739/A2319S30Q
doi:10.18739/A2N29P67H
doi:10.18739/A2DF6K36X
doi:10.18739/A2ZG6G71C
doi:10.18739/A20000081
doi:10.18739/A2VM42Z20
urn:uuid:72088082-251e-48f7-be83-9dd7508177e1
urn:uuid:09e1cd68-2209-4f42-ab41-291db507effa
doi:10.18739/A2T14TQ57
doi:10.18739/A2445HD27

[amoeba]

@taojing2002 can we work together to reharvest and reindex the PIDs above?

[amoeba]

We're still at:

Status	Count
Not indexed	58
Indexed, but missing annotations	11

@taojing2002 and I talked about this and this isn't a quick fix because we don't don't actually have a true reindex operation on the CNs in the sense that we can't trigger the CN to perform the usual index processing we do when an object is created or updated. We have separate tool (d1_index_build_tool.jar) that's run out of band that submits the update directly to Solr which just happens to use most of the same code and data as the actual CN index processor.

Unfortunately, d1_index_build_tool.jar is having an issue specifically with the semantic annotations so we'll need to track that down before we can move forward here. If we can't do that, a workaround would be to update sysmeta on the Member Node for each of the above objects, which would trigger a full sync->harvest->index cycle.

I think d1_index_build_tool is part of https://github.com/DataONEorg/cn-buildout though I'm not sure at this point.