Data Lab¶

New in this release¶

The Data Lab is new in Release 2.0.0. Its version, 0.1.0, is its own and is released separately from the Release 2 pin (ADR-0071). It is the dataspace metadata catalogue of OpenOP: it describes datasets, their distributions and their data quality, and does not store the data itself.

• Metadata catalogue. A DCAT-AP 3.0 catalogue on piveau-hub, using the 6G-DALI Metadata Application Profile (MAP): dataset, distribution and dqv: quality-measurement records, with faceted search.
• Data Lab UI. One web app with public catalogue and dataset pages (no sign-in) and a management console at #/admin to create catalogues, manage members, and create, edit, delete and promote datasets through a MAP form.
• Data Lab API. A headless FastAPI service for catalogue lifecycle, membership and dataset management with MAP validation. It is also the intended integration point for the future DataOps orchestrator.
• Two catalogues per deployment. oop-org (production) and oop-org-staging (submissions awaiting review). Stewards create further public or private catalogues on demand, and promote datasets from staging to production by hand.
• Access control by piveau. Members join a catalogue as reader, contributor or publisher. Private catalogues are refused by piveau itself to non-members, including on direct calls to the hub and the search service.
• Registration API for scripts. PUT /data-lab/hub/datasets/{id}?catalogue={id} with a Keycloak client-credentials token. Scripts never receive the hub API key.
• Deployment. Docker Compose profile behind the platform Nginx (/data-lab/*), and the Helm subchart datalab for Kubernetes (UI on NodePort 30083, realm dataspace created by the Keycloak chart).

Upgrade and enablement¶

Nothing to upgrade from 1.0.0: the Data Lab is off by default and existing installations are unaffected until it is enabled. Enabling needs both settings, then restarts Keycloak so it imports realm dataspace:

helm upgrade oop-platform ./oop-platform-chart -n oop --reuse-values \
  --set datalab.enabled=true --set keycloak.realms.dataspace.enabled=true

Requirements: Keycloak 26.4 or newer, and global.auth.enabled=true (the Data Lab is not deployed without login). Access control must be enabled when the deployment is first created, because piveau cannot switch it on over existing data. See the Helm README, section Data Lab.

Known limitations¶

The full list is in the spec's Implementation & Delivery › Known Gaps. The main ones:

• Direct registration is not validated (T1, T2). Records created or edited through the Data Lab UI or API are checked against the MAP. Records PUT straight to the hub Registration API only get an RDF syntax check and carry whatever access level the caller passes, because piveau's profile mode breaks indexing in hub-search.
• Not in this release: the dataspace connector (EDC / Dataspace Protocol), S3-compatible object storage, DataOps/MLOps registration flows, automated data-quality runs, governance and policy enforcement, and GAIA-X self-description signing.
• No public SPARQL endpoint, because the triple store holds private and public metadata together.
• Promotion is manual. A steward promotes staging to production in the UI. Promotion does not check the production id and can replace an existing record (T29).
• Helm has no platform Nginx (T15). The UI is the only public entry, on NodePort 30083. Hub and search APIs are reachable from outside the cluster only through kubectl port-forward, and datalab.baseUri defaults to the NodePort address. Set the base URI before the first install. Helm also runs one Virtuoso where Compose runs two (ADR-0069).
• Editing through the console can lose fields (T18, T24, T27). Edits drop fields the form lacks, including provenance, checksums and distribution licence and description, and lose the stable URIs of distributions and quality measurements. Keep the original record before editing.
• Limits. The API lists at most 100 catalogues (T14), a catalogue's visibility cannot change after creation (T9), and access control cannot be switched on over existing data (T11).
• Open questions. Who is put in the Operator group and who administers realm dataspace (F2), and how resource URIs resolve (F9), are undecided.
• Observability. Metrics, JSON logs and dependency-aware readiness are planned, not implemented (T3 to T5).