Skip to content

new DataCite failure condition, Dataverse doesn't handle it gracefully. #12913

Open

Description

@donsizemore

In attempting to reproduce something else on demo.dataverse.org, I ran into intermittent publishing failures from the DataCite Test MDS API:

java.io.IOException: Response from postMetadata: 500, <!doctype html>
<h3>Error ID:</h3>
          <span class=“error”>f508a192</span>
          <h3>Details:</h3>
          <p>Web application could not be started by the Phusion Passenger(R) application server.</p>
          <p class=“bold”>Please read <a href=“https://www.phusionpassenger.com/library/admin/log_file/” class=“plain”>the Passenger log file</a> (search for the Error ID) to find the details of the error.</p>
          <p>You can also get a detailed report to appear directly on this page, but for security reasons it is only provided if Phusion Passenger(R) is run with <i>environment</i> set to <i>development</i> and/or with the <i>friendly error pages</i> option set to <i>on</i>.</p>
          <p>For more information about configuring environment and friendly error pages, see:</p>
          <ul>
            <li><a href=“https://www.phusionpassenger.com/library/config/nginx/reference/#passenger_friendly_error_pages”>Nginx integration mode</a></li>
            <li><a href=“https://www.phusionpassenger.com/library/config/apache/reference/#passengerfriendlyerrorpages”>Apache integration mode</a></li>
            <li><a href=“https://www.phusionpassenger.com/library/config/standalone/reference/#--friendly-error-pages---no-friendly-error-pages-friendly_error_pages”>Standalone mode</a></li>
          </ul>

Payara logs: Identifier not created: create failed but the database says identifierregistered = 't':

# select * from dvobject where identifier like '%FT4YBA%';
   id    |  dtype   |       createdate        |        indextime        |    modificationtime     |   permissionindextime   | permissionmodificationtime |     publicationdate     | creator_id | owner_id | releaseuser
_id | previewimageavailable |            storageidentifier            | authority |   globalidcreatetime    | identifierregistered |    identifier     | protocol | previewimagefail | separator
---------+----------+-------------------------+-------------------------+-------------------------+-------------------------+----------------------------+-------------------------+------------+----------+------------
----+-----------------------+-----------------------------------------+-----------+-------------------------+----------------------+-------------------+----------+------------------+-----------
 2737782 | Dataset  | 2026-11-11 13:40:28.852 | 2026-11-11 13:41:01.405 | 2026-11-11 13:41:01.199 | 2026-11-11 13:41:01.359 | 2026-11-11 13:40:28.852    | 2026-11-11 13:40:38.323 |        361 |  2737774 |
361 | f                     | s3dir://10.70122/FK2/FT4YBA             | 10.70122  | 2026-11-11 13:40:41.52  | t                    | FK2/FT4YBA        | doi      | f                | /

Dataverse also logs that it's unlocking the dataset, but 24+ hours later:
Screenshot 2026-11-12 at 17 37 19

As of yesterday evening there were 91 datasetlocks on demo.dataverse.org, all with reason Workflow

Leonid could produce the Phusion Passenger response from the DataCite Test MDS API with
$ curl -X POST "https://mds.datacite.org/metadata"

I'm only concerned with Dataverse thinking the identifier is registered when it isn't and leaving the dataset locked; each of those conditions will require corrective API calls from an archivist / administrator.

Activity

  1. wshahn commented on Nov 13, 2026

    @wshahn

    If I curl from api.test.datacite, I can see the dataset got registered. Because the state is findable below, it seems like identifierregistered = 't' is correct and something is wrong with the dataset lock workflow here.

    ~ $ curl -s https://api.test.datacite.org/dois/10.70122/FK2/FT4YBA |jq
    {
      "data": {
        "id": "10.70122/fk2/ft4yba",
        "type": "dois",
        "attributes": {
          "doi": "10.70122/fk2/ft4yba",
          "prefix": "10.70122",
          "suffix": "fk2/ft4yba",
          "identifiers": [],
          "alternateIdentifiers": [],
          "creators": [
            {
              "name": "Sizemore, Donald",
              "nameType": "Personal",
              "givenName": "Donald",
              "familyName": "Sizemore",
              "affiliation": [
                "University of North Carolina at Chapel Hill"
              ],
              "nameIdentifiers": []
            }
          ],
          "titles": [
            {
              "title": "guestbook test2"
            }
          ],
          "publisher": "Demo Dataverse",
          "container": {},
          "publicationYear": 2026,
          "subjects": [
            {
              "subject": "Demo Only"
            }
          ],
          "contributors": [
            {
              "name": "Sizemore, Donald",
              "nameType": "Personal",
              "givenName": "Donald",
              "familyName": "Sizemore",
              "affiliation": [
                "University of North Carolina at Chapel Hill"
              ],
              "contributorType": "ContactPerson",
              "nameIdentifiers": []
            }
          ],
          "dates": [
            {
              "date": "2026-11-11",
              "dateType": "Submitted"
            },
            {
              "date": "2026-11-11",
              "dateType": "Available"
            }
          ],
          "language": null,
          "types": {
            "ris": "DATA",
            "bibtex": "misc",
            "citeproc": "dataset",
            "schemaOrg": "Dataset",
            "resourceTypeGeneral": "Dataset"
          },
          "relatedIdentifiers": [
            {
              "relationType": "HasPart",
              "relatedIdentifier": "10.70122/fk2/ft4yba/qwt9xd",
              "relatedIdentifierType": "DOI"
            }
          ],
          "relatedItems": [],
          "sizes": [
            "46817"
          ],
          "formats": [
            "image/png"
          ],
          "version": "1.0",
          "rightsList": [
            {
              "rightsUri": "info:eu-repo/semantics/openAccess"
            },
            {
              "lang": "en",
              "rights": "Creative Commons Zero v1.0 Universal",
              "rightsUri": "https://creativecommons.org/publicdomain/zero/1.0/legalcode",
              "schemeUri": "https://spdx.org/licenses/",
              "rightsIdentifier": "cc0-1.0",
              "rightsIdentifierScheme": "SPDX"
            }
          ],
          "descriptions": [
            {
              "description": "guestbook test2",
              "descriptionType": "Abstract"
            }
          ],
          "geoLocations": [],
          "fundingReferences": [],
          "xml": "<snip>",
          "url": "https://demo.dataverse.org/citation?persistentId=doi:10.70122/FK2/FT4YBA",
          "contentUrl": null,
          "metadataVersion": 1,
          "schemaVersion": "http://datacite.org/schema/kernel-4",
          "source": "mds",
          "isActive": true,
          "state": "findable",
          "reason": null,
          "viewCount": 0,
          "viewsOverTime": [],
          "downloadCount": 0,
          "downloadsOverTime": [],
          "referenceCount": 0,
          "citationCount": 0,
          "citationsOverTime": [],
          "partCount": 0,
          "partOfCount": 0,
          "versionCount": 0,
          "versionOfCount": 0,
          "created": "2026-11-11T18:40:29.000Z",
          "registered": "2026-11-11T18:40:40.000Z",
          "published": "2026",
          "updated": "2026-11-11T18:40:40.000Z"
        },
        "relationships": {
          "client": {
            "data": {
              "id": "gdcc.harvard-test",
              "type": "clients"
            }
          },
          "provider": {
            "data": {
              "id": "harvard",
              "type": "providers"
            }
          },
          "media": {
            "data": {
              "id": "10.70122/fk2/ft4yba",
              "type": "media"
            }
          },
          "references": {
            "data": []
          },
          "citations": {
            "data": []
          },
          "parts": {
            "data": []
          },
          "partOf": {
            "data": []
          },
          "versions": {
            "data": []
          },
          "versionOf": {
            "data": []
          }
        }
      }
    }
    
  2. landreev commented on Nov 19, 2026

    @landreev
    Contributor

    @qqmyers That specific dataset may be registered, but the real issue is that DataCite's test/dev service (https://mds.test.datacite.org) was genuinely dead on that day (500 in response to all/any requests); and Dataverse does not appear to be able to handle that very well - each such failure appears to leave a lock behind.
    This keeps happening apparently - not seeing any of these errors in server.log today; but they kept popping up approx. every other day since Don noticed the above.

    As Don mentioned above, when the dev. mds service is in this state, a % curl -X POST "https://mds.datacite.org/metadata"
    results in a 500 and the html message quoted. When the service is working, this empty POST results in some meaningful message about missing or invalid arguments.

    Last time I observed this in r/t, I can't remember if I tried the above from outside of demo (like my own macbook); to eliminate the possibility that it was demo specifically being placed on the naughty list. But the main issue remains regardless - the app does not handle this error gracefully, whatever the reason.

  3. landreev commented on Nov 19, 2026

    @landreev
    Contributor

    I may know what it is. The key detail in Don's report is that all these locks are of type Workflow.
    If an exception is thrown during the DataCite exchange, this is how it's caught in the FinalizePublication command:

       } catch (Throwable e) {
          logger.warning("Failed to publicize the identifier " + dataset.getGlobalId().asString()
                  + ", or to publicize a file in the dataset; notifying the user(s), unlocking the dataset");
    
          // Send failure notification to the user:
          notifyUsersDatasetPublishStatus(ctxt, dataset, UserNotification.Type.PUBLISHFAILED_PIDREG);
    
          ctxt.datasets().removeDatasetLocks(dataset, DatasetLock.Reason.finalizePublication);
          ... 
       }
    

    ... but not DatasetLock.Reason.Workflow - ?

  4. qqmyers commented on Nov 19, 2026

    @qqmyers
    Member

    That makes sense - this line removes the workflow lock, but never called if you throw up top due to DataCIte:

    ctxt.datasets().removeDatasetLocks(ds, DatasetLock.Reason.Workflow);

    Maybe a try/finally since I don't think DataCite is the only thing that can throw a CommandException (e.g. validateFiles)

  5. landreev commented on Nov 19, 2026

    @landreev
    Contributor

    And also, early on in the command we attempt to register the identifier for the dataset (and possibly files) if it has not been registered yet; only the finalizePublication lock is removed there as well if that throws an exception.
    So yes a try/finally around the whole thing.

  6. donsizemore commented on Nov 20, 2026

    @donsizemore
    ContributorAuthor

    potentially related: #7395 and #10992

  7. landreev commented on Nov 20, 2026

    @landreev
    Contributor

    The older one is almost certainly about the dedicated finalizePublication lock proper, that we must have made Dataverse clean up properly since. This issue is specific to workflow locks (it became a problem on demo due to a combo of having a workflow added instance-wide and the test mds service instability). But we should use the opportunity to close it as well.

    Not sure about #10992, it looks like there's a lot going on there. At least some of it appears to be on account of the checksum validation of a giant file genuinely taking a long time.

    potentially related: #7395 and #10992

  8. landreev commented on Nov 20, 2026

    @landreev
    Contributor

    Something like this probably: cf15cf7
    That said, it took more changed lines than I expected to try and have these locks deleted in just one place, with try/finally.
    Wondering if it's actually going to be simpler and safer to try to remove that workflow lock in 3 separate catch() statements.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions