Repository navigation
CSL Citations Exporter - #12777
Draft
poikilotherm wants to merge 146 commits into
Draft
CSL Citations Exporter#12777poikilotherm wants to merge 146 commits into
poikilotherm wants to merge 146 commits into
Conversation
…istryBean IQSS#12686 - Moved exporter management logic into a dedicated `ExporterRegistryBean` singleton for improved modularity and maintainability. - Simplified `ExportService` to delegate exporter logic to the new registry.
- Enable injectingthe registry and other components - The export process itself is stateless. State is involved in potential write locks, the loaded plugins, etc. - A stateless coordinator bean scales better for multiple export requests coming in.
…alidator, and storage abstraction IQSS#12686 The goal is removing the caching logic from the ExportService. At the same time, a distinct caching subsystem shall have policies about what gets cached, when it expires etc, all independent of a coordinating service like ExportService. This make cognitive loader smaller and allows extension without using more code branches.
…e into FileEmbargoExpiryInvalidator IQSS#12686
…eIOCache class IQSS#12686 - Reorganized export cache handling into a dedicated `StorageIOCache` service, improving modularity and reducing cognitive load in `ExportService`. - Streamlined caching operations with a unified approach across all storage drivers. - Deprecated legacy unversioned cache keys; introduced versioned aux tag schema for better cache qualification. - Enhanced write atomicity and cache eviction logic. - Remove stale code for size of exports
…OCache IQSS#12686 The legacy reading of cached exports is prone to produce bugs in production. When we rely on reading cached exports as prerequisites for other metadata formats, we might end up with stale data. Any export has no knowledge about whether and when an export of another format happened. We keep no provenance per format. Assuming there is a cached "latest" with the legacy file format, it would be read as a prerequisite format, but our invalidation mechanisms would not be able to tell if it's actually stale, because it was not yet re-exported. Any released version is immutable, thus if we rely in lookups on cached objects with the version present in the aux tag, we can be sure we get the latest data.
…constructor IQSS#12686 - Added null and blank checks for dataset, version, and formatName to ensure robust usage. - Introduced a convenience constructor for creating cache keys directly from a dataset version and format.
…rvice` package and rename `ExportService` to `ExportServiceBean` IQSS#12686 - "ExportServiceBean" is more aligned with the codebase style where EJBs mostly have a "Bean" name suffix. - Also move test classes into the same package (under the test source tree)
…2686 - Documented `tryRead`, `deleteQuietly`, and `storageFor` with proper Javadoc. - Clarified the stream-closing intent in `write` to make the leak-avoidance pattern explicit.
…to ExportServiceBean IQSS#12686 - Relocated the `invalidators` collection from the sealed interface to the service bean, where it logically belongs as a runtime dependency rather than a static on the contract. - Added a section marker for export data retrieval methods in `ExportServiceBean`. - Noted future plan to replace the static list with a registry pattern once plugins can supply their own invalidation logic.
…tServiceBean 12686 Making it simpler to read inline.
Added `ExportCache` as an CDI (not EJB) injected dependency in the service bean.
- Introduced `clearCachedFormats(DatasetVersion, List<String>)` as the version-specific clearing entry point, with the dataset-level overload delegating via a new `defaultVersion()` helper. - Added `clearCachedFormat(DatasetVersion, String)` to evict a single cache entry by key. - Added `requireExists` and `requireAllExist` validation methods to `ExporterRegistryBean` so format names are checked before eviction.
…QSS#12686 Align the related methods into one block, not divided by the cache handling stuff.
…istryBean IQSS#12686 - Added `buildFormatRequiredByMap` to build a read-only map of prerequisite format names to the exporters that depend on them. - Added `buildAndVerifyRequirements` to validate registry integrity: all prerequisite formats must have a registered exporter, and no cyclic prerequisite chains may exist. - Integrated the check into initialization as Step 4, failing fast with `ExportException` on any integrity violation (missing prerequisite or cycle).
…#12686 Added `formatRequiredBy` field to store the prerequisite format dependency map alongside the exporters map, populated during registry initialization. Will be reused during cascaded cache eviction or exporting of formats depending on a certain format.
…#12686 - Added `buildPrerequisitesChainDepth` to compute the prerequisite chain depth for each format (0 = no prerequisite, N = N levels deep). - Added `buildTopologicalComparator` to create an immutable comparator ordering exporters by depth, with format name as tiebreaker for deterministic results. - Exposed via `getTopologicalComparator()` so callers can sort the exporter list in a dependency-safe order. - Integrated as Step 5 in initialization, stored alongside the existing `formatRequiredBy` map.
…IQSS#12686 - Added `SecureTempFiles` utility that creates temp files with `0600` permissions on POSIX systems; on Windows it relies on the per-user `%TEMP%` ACLs. - Replaced raw `Files.createTempFile` in `StorageIOCache.write` with `SecureTempFiles.createOwnerOnlyTempFile` so other local users can no longer read or tamper with export temp files.
The cache key should not be responsible to carry the information about the "where" of an export, just about the "what". Changing dependent methods accordingly. Also, fixed ambiguity with the cache invalidator implementations: the invalidator should look for stale *versions* of dataset, not for the dataset as a whole being stale. The cache is treating versions individually, so they shall get stale individually, too. - Reduced `ExportCacheKey` to a single `auxTag` string, removing `Dataset`/`DatasetVersion` references for thread-safety and GC-friendliness. - Moved `TAG_PREFIX`/`TAG_SUFFIX` into `ExportCacheKey` as public constants. - Added explicit `Dataset` parameter to all `ExportCache` methods (`read`, `write`, `evict`) since the key no longer carries storage context. - Added explicit `DatasetVersion` parameter to `ExportCacheInvalidator.isStale`; updated `FileEmbargoExpiryInvalidator` with null-checks and released/archived status guard. - Updated `StorageIOCache` logging to use `dataset.getId()` instead of the version string.
…ndents set IQSS#12686 - Renamed `formatRequiredBy` to `transitiveDependents`, changing the value type from `List<String>` to `Set<String>` to capture all direct and transitive dependents per format. - Replaced `buildPrerequisitesChainDepth` with `buildTransitiveDependents`, which walks each exporter's prerequisite chain and registers it as a dependent of every ancestor format. - Updated `buildTopologicalComparator` to sort by new dependent-set - Merged `buildFormatRequiredByMap` into `verifyRequirements` as the former map is no longer stored for reuse - Moved `getFormatsDependingOn` to `getTransitiveDependents` to reflect the new semantics
…QSS#12686 - Introduced sealed `Details` interface exposing `localizedDisplayName`, `formatName`, `mediaType`, `isHarvestable`, and `isAvailableToUsers`, thus avoiding having to retrieve these details from the exporter, saving a roundtrip. - Made `ExporterDetails` record package-private to prevent external instantiation while allowing consumers to read via the interface. - Renamed `getLabels()` to `getDetails()`, returning `List<Details>` with the expanded field set. - Added `get(Details)` lookup method to resolve an exporter by its details object. These can only be created and handed out by the registry, thus we can be sure a matching exporter exists. - Removed unused `Collections` import.
…tCacheKey components IQSS#12686 - Replaced the single `auxTag` field with `formatName` and `friendlyVersion` so the key exposes its meaningful parts directly. - Moved `auxTag()` from a static factory into an instance method derived from the record's fields. - Split validation into `checkFormatName` and `checkVersion` private helpers for clearer intent (and compatibility with the constructor needing to be called first thing).
…ServiceBean IQSS#12686 These methods (`getExporter`, `isXMLFormat`, `getMediaType`) directly exposed the internal `exporterMap` and are no longer needed now that format details are resolved via the `Details` interface in the registry.
…IQSS#12686 Added null check in `get(String formatName)` to return `Optional.empty()` instead of throwing NPE when the underlying Map implementation does not permit null keys.
…ation - Tracks consecutive failures via an `AtomicInteger` streak; escalates from `FINE` to `WARNING` once the streak reaches the configured threshold. - A success resets the streak; a threshold of zero or negative deactivates escalation entirely. - Thread-safe and suitable for sharing across concurrent callers or use in `ConcurrentHashMap` contexts. - Warnings will not be flooding the log once threshold is reached via configurable repeat cycle. - To enable "all clear" messages once the threshold was met, the success recording may then return the number of failures. Using OptionalInt, the logging statement is a one-liner.
…vadoc IQSS#12686 - Clarified that the legacy unqualified name is ignored for read/write cycles and only purged via `evictAll`, rather than being a read fallback. - Fix typos
…SS#12686 - Replace static `FINE`-level logging in `tryRead` and `deleteQuietly` with threshold-based escalation via `FailureEscalation` instances (threshold: 256). - Log a recovery warning once consecutive failures drop below the threshold after previously exceeding it. - Include the current failure streak in the read-path log message for operational context.
…EJB IQSS#12686 - Introduces a `@Stateless` EJB that funnels all export data production (draft, cached, bulk) through a single path for uniform staleness validation, prerequisite resolution, and error wrapping. - Cached reads consult registered `ExportCacheInvalidator` instances; stale entries are evicted and reported as a miss. - Prerequisite formats are resolved recursively with circular-chain detection via an in-flight `LinkedHashSet`. - Non-cacheable (draft) versions are produced to `SecureTempFiles` with `DELETE_ON_CLOSE` to avoid in-memory retention of large exports. - `IllegalStateException` from exporters is wrapped in `ExportException` with dataset context for consistent reporting across all production paths.
…QSS#12686 - Injecting `ExportPipelineBean` as an `@EJB` - Removed the static `invalidators` list and its associated Javadoc from `ExportServiceBean` - they are now owned by the pipeline.
…eam handling IQSS#11892 Documented new settings for bulk metadata export request size limits and introduced guidelines for plugin developers on safe stream handling using StreamRegistry. Updated API guide with details for the bulk metadata export endpoint, including usage, error handling, and CORS considerations.
Now allowing to use this immutable class as a DTO, too. - Added Jakarta JSON-B creator/property as well as `@JsonbTransient` annotations for de/serialization - Added OpenAPI `@Schema` descriptions/examples - Added `@NotBlank` and other Bean Validation metadata annotations. - Implemented value-based `equals` and `hashCode`. - Refactored and extended the unit tests accordingly
…ying and return values IQSS#12747 Replaced mutable collections with `List.of()` and `List.copyOf()` to ensure immutability and improve safety for internal state handling in `DataCitation` and `DatasetVersion`. Simplified collection initialization and streamlined methods accordingly. Increasing immutability to ensure serialization to JSON works out nicely. Also, making sure to never return null values, but empty lists avoids NPEs.
…er in DataCitation IQSS#12747 Instead of saving a JPA entity who's only attribute "name" will be used, immediately use an immutable, thread-safe value. Fix unit test to provide a valid dataset type, which was untested before.
…tion` for clarity IQSS#12747 Better align with modern Java style (enums are allowed to have methods). Update the only usage in Access API class.
…rd for immutability and clarity IQSS#12747 Instead of saving DatasetField JPA entities which are neither immutable nor thread-safe, transfer to immutable value objects.
…ving redundant emptiness validations IQSS#12747 If the collections are empty, the foreach loop is a noop, no if branch necessary.
… field scope IQSS#12747 Do not store or risk bleeding a mutable and non-thread-safe entity into serialization.
… null check IQSS#12747 Included the `UR` field in RIS exports for DOI-based persistent identifiers in the same null check already being present. Updated serialization logic and unit tests accordingly.
…DataCitation` IQSS#12747 Replaced less efficient string concatenation with already present `StringBuilder`. Updated exception handling with logging for better error tracking.
…pe.fromString` in `DataCitation` IQSS#12747 Instead of always adding more if branches, use the CLS provided lookup function first. All our usual types are covered. Any non-matching type is not simply declared a dataset when it isn't, but mapped to a generic type.
…n `DataCitation` IQSS#12747 Improved JSON serialization by using a Map provided by the CLS lib `MapJsonBuilder` instead of converting to String and reparsing with JSON-P. This should be a bit faster than the alternative.
Reordering makes it easier to read. Also ditching the null checks, as no longer necessary. Instead, make sure the addition of series titles does not throw a NSE exception.
…d improve error handling IQSS#12747 Use `Optional` to handle potential `null` values when retrieving `DatasetType`. Expanded exception handling to include `NullPointerException` for safer CSL type assignment.
This is a partial reverse of e534005. The MapJsonBuilder uses internal citeproc types, which are not serializable using Jakarta JSON-P (they work fine with JSON-B). As long as citeproc doesn't provide a builder that uses JSON-P types, using the String workaround is the only option.
By annotating crucial parts of the class, we now can fully serialize and deserialize DataCitations using JSON-B, which is much easier than handcrafting using JSON-P. Again, the idea is to hand over a JSON serialized DataCitation object inside the Dataverse JSON format, so intree exporters have direct access to it by deserializing.
…d test coverage IQSS#12747 Now also testing a lot of expected behavior. In addition, added JSON-B testing for the new serializability.
…for exports IQSS#12747 Adding a new property "datacitation" to the Dataverse JSON export format, which can be consumed by exporters.
Introduced `CslExporter` to support Citation Style Language (CSL) format for dataset exports. Includes support for single and bulk dataset exports, leveraging JSON-B for serialization and deserialization.
…dTest IQSS#12747 Performance optimization and ensure code style compliance
…simplify media type resolution IQSS#12747 By moving the details about a format (identity and MIME type) instead of in methods we can reuse the information inside API and Exporters we want to migrate to while sunsetting DataCitation.
…ion in CslExporter IQSS#12747
…pping in endpoints IQSS#12747
…rmat validation and exception handling IQSS#12747 - Prioritze retrieval of citation from the (cached) export subsystem instead of on-the-fly generation if an exporter for the requested format is present - Unify lookup logic for internal citation format endpoint and parameterized endpoint - Remove unnecessary try-catch blocks to unwrap exceptions (rely on JAX-RS exception handlers instead)
poikilotherm
force-pushed
the
12747-csl-exporter
branch
from
October 9, 2026 00:08
69fec99 to
3da2337
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
First steps to refactoring the citation system to become part of the metadata exporting subsystem.
Open TODOs:
Which issue(s) this PR closes:
This PR relies on #12688 and #12737 to be merged first.
Special notes for your reviewer:
Suggestions on how to test this:
Does this PR introduce a user interface change? If mockups are available, please link/include them here:
Is there a release notes update needed for this change?:
Additional documentation: