Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
38500c0
refactor(api): remove unnecessary `printStackTrace()` to make logging…
poikilotherm May 7, 2026
6cb935d
build(deps): update dataverse-spi to `2.2.0-SNAPSHOT` and enable Cent…
poikilotherm Sep 22, 2026
efa8dc0
refactor(export): extract details mapping logic in ExporterRegistryBe…
poikilotherm Sep 22, 2026
6320777
feat(export): add ExporterRegistryBean method requireExistsAndSupport…
poikilotherm Sep 22, 2026
25bf32b
feat(util): add StreamRegistry for tracking and auto-closing stream r…
poikilotherm Sep 22, 2026
a754ffa
feat(export): implement BulkExportPipeline for dataset version export…
poikilotherm Sep 22, 2026
fffbd82
test,style(export): consolidate argument validation tests in BulkExpo…
poikilotherm Sep 23, 2026
75c9bc3
feat(export): add transactional export retrieval method #11892
poikilotherm Sep 23, 2026
f79aae7
feat(export): implement bulk export retrieval method in ExportService…
poikilotherm Sep 23, 2026
1b7199b
feat(api): add violations support to JsonResponseBuilder and streamli…
poikilotherm Sep 24, 2026
51df03d
feat(api): add ProcessingExceptionHandler for improved JSON error han…
poikilotherm Sep 24, 2026
d6c87a5
feat(api): add MultiDatasetExportRequest for bulk dataset export API …
poikilotherm Sep 24, 2026
832704f
feat(export): implement bulk export in JSONExporter #11892
poikilotherm Sep 24, 2026
22e90ac
style(api): reorder parameters in exportDataset for consistency and r…
poikilotherm Sep 24, 2026
d0db50d
feat(api): introduce bulk dataset export endpoint
poikilotherm Sep 24, 2026
e4de049
feat(api): enforce request size limit for bulk dataset exports #11892
poikilotherm Sep 24, 2026
9d6f83b
style(api): rename EXPORT_CORRELATION_ID to EXPORT_CORRELATION_ID_HEA…
poikilotherm Sep 24, 2026
e2d7e8a
docs(api): enhance MultiDatasetExportRequest OpenAPI field documentat…
poikilotherm Sep 24, 2026
ad3c9da
feat(api): add MakeDataCount logging for bulk dataset exports #11892
poikilotherm Sep 24, 2026
ad4811d
docs(release-notes): add details for bulk dataset export feature #11892
poikilotherm Sep 24, 2026
8394fa6
docs: add documentation for bulk dataset export configuration and str…
poikilotherm Sep 24, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions doc/release-notes/11892-bulk-export.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
### Bulk Metadata Export via API

Metadata can now be retrieved for multiple dataset versions in a single request and delivered as **one combined export**.

A new API endpoint, `POST /api/datasets/export`, accepts a JSON body with the name of the metadata export format and a list of datasets, each identified by its persistent identifier and a version specification.
Each version may be given explicitly (for example `1.0`) or by one of the symbolic selectors `:latest`, `:latest-published`, or `:draft`; when omitted, the latest published version is used.

Please note: this is not a replacement for the OAI-PMH based harvesting protocol, which is still the recommended way to harvest metadata in larger quantities.
It reduces the number of round trips substantially for use cases around the Modern UI, where metadata for multiple user-selected datasets is requested.

**Note:** Bulk export relies on export formats that support it.
Only formats whose exporter plugin declares the new bulk export capability can be used to export multiple datasets at once.
The built-in Dataverse JSON format supports bulk export and combines the requested versions into a single JSON array.
Other formats (or bulk export for formats provided by external plugins) becomes available as plugins adopt the new capability.
Requests naming only a single dataset are handled exactly like the previous per-dataset export, so no bulk-capable format is required in that case.

#### Clear error reporting

- Request bodies that are missing or malformed are now rejected with precise, human-readable validation messages pointing at the offending field (for example an invalid version selector or a blank persistent identifier).
- If some of the requested datasets are unknown or cannot be seen by the requester, the response still lists **all** such problems in one place (an HTTP 400 response with a list of violations) instead of stopping at the first one - and deliberately without revealing which datasets exist on the instance.

#### Updates for System Administrators

- **New setting:** the maximum number of dataset versions per bulk export request is controlled by a new JVM setting `dataverse.api.export.bulk.max-request-size` (default: `32`).
- **Log tracing:** every bulk export is assigned a unique correlation identifier, returned to the client in the `X-Dataverse-Export-ID` response header and included in all log records of that batch.
Use the identifier from a failing request to find the related log entries.
- Downloads served through bulk export are counted and reported by the MakeDataCount service like regular metadata exports (for each released version in the batch).

#### Updates for Export Plugin Developers

The Exporter Plugin API (dataverse-spi) gains a bulk export capability: plugins implementing `io.gdcc.spi.export.caps.bulk.BulkDatasetExporter` receive a `BulkDatasetContext` over a batch of dataset versions and decide how the individual exports are framed and joined.
- Exports are lazy: each one materializes on demand (read from cache, generated on the fly for drafts), in a short transaction of its own, so memory and database connections stay independent of the batch size.
- Streams handed out by the context are tracked and closed automatically when the batch finishes, even if the plugin leaves them open.
- Dataverse implements this capability for its built-in JSON exporter, and ships extensive new unit tests covering the API request validation, the pipeline, and the transaction behavior.
44 changes: 44 additions & 0 deletions doc/sphinx-guides/source/api/native-api.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2416,6 +2416,50 @@ The standard has further evolved into a format called Croissant. For details, se

The ``schema.org`` format changed after Dataverse 6.4 as well. Previously its content type was "application/json" but now it is "application/ld+json".

.. _bulk-metadata-export-api:

Export Metadata of Multiple Datasets in One Request
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

|CORS| Export the metadata of several dataset versions in a single request, combined into a single output by the chosen format.
This is a ``POST`` to the same path as the single-endpoint export described above, with a JSON body naming the format and the list of dataset versions to export:

.. code-block:: bash

export SERVER_URL=https://demo.dataverse.org
export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
export METADATA_FORMAT=dataverse_json
export BODY="{\"exporter\":\"$METADATA_FORMAT\",\"datasets\":[{\"persistentId\":\"doi:10.5072/FK2/J8SJZB\",\"version\":\"1.0\"},{\"persistentId\":\"doi:10.5072/FK2/XY79KQ\"},{\"persistentId\":\"doi:10.5072/FK2/ABCDEF\",\"version\":\":draft\"}]}"

curl -X POST -H "Content-Type: application/json" -H "X-Dataverse-key: $API_TOKEN" -d "$BODY" "$SERVER_URL/api/datasets/export"

The fully expanded example above (without environment variables) looks like this:

.. code-block:: bash

curl -X POST -H "Content-Type: application/json" -H "X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" -d "{\"exporter\":\"dataverse_json\",\"datasets\":[{\"persistentId\":\"doi:10.5072/FK2/J8SJZB\",\"version\":\"1.0\"},{\"persistentId\":\"doi:10.5072/FK2/XY79KQ\"},{\"persistentId\":\"doi:10.5072/FK2/ABCDEF\",\"version\":\":draft\"}]}" "https://demo.dataverse.org/api/datasets/export"

Each entry in ``datasets`` couples a ``persistentId`` with an optional ``version``.
The version syntax is the same as for the :ref:`export-dataset-metadata-api` above (see :ref:`dataset-version-specifiers`):
``:latest``, ``:latest-published``, ``:draft``, or an explicit number such as ``1.0``.
When omitted, it defaults to ``:latest-published``, and an API token is not needed unless a version requires access.

Requests listing exactly one dataset is handled like :ref:`the single-endpoint export <export-dataset-metadata-api>`:
it returns the usual single-dataset output of the requested format.

A request listing two or more datasets combines all of the requested versions into a single output.
The format must support this, and using a format that does not results in an HTTP 400 (Bad Request) error.
What the combined output contains for a given format is decided by that format.

All requested datasets are resolved and their access checked before any export is produced.
If any requested dataset or version is not found or cannot be read by the requester, the *entire* request is rejected with an HTTP 400 response.
Each problematic entry is listed, so you get feedback on all entries in one round trip.
Similarly, a body that is missing or does not follow the schema above is rejected with a message naming the invalid field.

The maximum number of dataset versions per request is limited by the :ref:`dataverse.api.export.bulk.max-request-size` setting.
The successful response contains an ``X-Dataverse-Export-ID`` header carrying the identifier of the batch.
In case an error occurs during data streaming, use its value to correlate with the server's log of events.

List Files in a Dataset
~~~~~~~~~~~~~~~~~~~~~~~

Expand Down
15 changes: 15 additions & 0 deletions doc/sphinx-guides/source/developers/coding-style.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,21 @@ If you know of a way to easily share Netbeans configuration across a team, pleas
To avoid exposing potentially sensitive information in temporary files stored on the file system, use the
`edu.harvard.iq.dataverse.util.SecureTempFiles` utility instead of Java mechanics as `Files.createTempFile` directly.

### Tracking Streams Handed Out Beyond the Core's Control

When handing out streams to dataverse-spi based plugins, do not rely on them closing the streams.
The same is true for any library we use and hand streams to, as we cannot guarantee their diligence.
Use the `edu.harvard.iq.dataverse.util.StreamRegistry` utility to avoid leaking stream.
Hand out `registry.track(inputStream)` and close the registry in `try-with-resources` (or `finally`) when your operation is done.

With a registry in place, the caller may close the stream, close it more than once, or not close it at all, without leaking file descriptors or storage connections.
The registry closes whatever is left over when it closes, and logs a warning if more streams than the configured threshold are held open at once.
The registry's bookkeeping is thread-safe, but the individual tracked streams are *not* made thread-safe by tracking, so they must not be shared between threads.

The bulk metadata export pipeline (see `BulkExportPipeline`) is an example of this convention in action:
it lends each dataset version's export stream to exporter plugins through a `StreamRegistry`,
so plugins are free to process items in whatever order they like across threads.

## Bash

Generally, Google's Shell Style Guide at <https://google.github.io/styleguide/shell.xml> seems to have good advice.
Expand Down
15 changes: 15 additions & 0 deletions doc/sphinx-guides/source/installation/config.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3452,6 +3452,21 @@ Can also be set via any `supported MicroProfile Config API source`_, e.g. the en
This setting will be ignored unless the :ref:`dataverse.api.blocked.policy` is set to ``unblock-key``. Otherwise the deprecated :ref:`:BlockedApiKey` will be used


.. _dataverse.api.export.bulk.max-request-size:

dataverse.api.export.bulk.max-request-size
+++++++++++++++++++++++++++++++++++++++++++

Limits the maximum number of dataset versions that may be requested in a single bulk metadata export request (see :ref:`bulk-metadata-export-api` in the API Guide).
Requests naming more dataset versions than this limit are rejected with an HTTP 400 response.

Default: ``32``

``./asadmin create-jvm-options '-Ddataverse.api.export.bulk.max-request-size=16'``

Can also be set via any `supported MicroProfile Config API source`_, e.g. the environment variable ``DATAVERSE_API_EXPORT_BULK_MAX_REQUEST_SIZE``.


.. _dataverse.legacy.api-response-message-style:

dataverse.legacy.api-response-message-style
Expand Down
2 changes: 0 additions & 2 deletions modules/dataverse-parent/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -450,7 +450,6 @@
</snapshots>
</repository>
<!-- Uncomment when using snapshot releases from Maven Central -->
<!--
<repository>
<id>central-portal-snapshots</id>
<name>Central Portal Snapshots</name>
Expand All @@ -464,7 +463,6 @@
<enabled>true</enabled>
</snapshots>
</repository>
-->
<!--
<repository>
<id>s01-oss-sonatype</id>
Expand Down
2 changes: 1 addition & 1 deletion pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -701,7 +701,7 @@
<dependency>
<groupId>io.gdcc</groupId>
<artifactId>dataverse-spi</artifactId>
<version>2.1.0-M1</version>
<version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>javax.cache</groupId>
Expand Down
3 changes: 3 additions & 0 deletions src/main/java/ValidationMessages.properties
Original file line number Diff line number Diff line change
Expand Up @@ -45,3 +45,6 @@ guestbook.response.nameLength=Please limit response to 255 characters

email.invalid=is not a valid email address.
url.invalid=is not a valid URL.

api.request.body.null=Request body may not be absent
api.datasets.export.parameters.invalidVersion=Parameter "version" must be one of ":latest", ":latest-published", ":draft", or a version number "<major>.<minor>", e.g. "1.0"
7 changes: 7 additions & 0 deletions src/main/java/edu/harvard/iq/dataverse/api/ApiConstants.java
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
package edu.harvard.iq.dataverse.api;

import java.util.List;

public final class ApiConstants {

private ApiConstants() {
Expand Down Expand Up @@ -28,8 +30,13 @@ private ApiConstants() {
public static final String DS_VERSION_LATEST = ":latest";
public static final String DS_VERSION_DRAFT = ":draft";
public static final String DS_VERSION_LATEST_PUBLISHED = ":latest-published";
public static final String DS_VERSION_IDENTIFIER_REGEX = "^("+DS_VERSION_LATEST+"|"+DS_VERSION_DRAFT+"|"+DS_VERSION_LATEST_PUBLISHED+"|\\d+\\.\\d+)$";

// addFiles call
public static final String API_ADD_FILES_COUNT_PROCESSED = "Total number of files";
public static final String API_ADD_FILES_COUNT_SUCCESSFUL = "Number of files successfully added";

// Export
public static final String EXPORT_CORRELATION_ID_HEADER = "X-Dataverse-Export-ID";
public static final int DEFAULT_MAX_EXPORT_REQUEST_SIZE = 32;
}
Loading
Loading