I14Y Harvester - OGD Statistical Federal Office (ABN + PROD)

The harvesting of the OGD Statistical Federal Office catalogue is done via Github actions in ABN and PROD. Synchronisation takes place automatically every day at midnight. Synchronisation can also be triggered manually if necessary. A log file is saved after each synchronisation.

Features

• Import DCAT datasets from xml/rdf or ttl files to I14Y API
• Supported properties for dcat.Dataset:

Property	Requirement level
dct:title	mandatory
dct:description	mandatory
dct:accessRight (chosen from: PUBLIC, NON_PUBLIC, CONFIDENTIAL, RESTRICTED)	mandatory
dct:publisher (stated in config.py)	mandatory
dct:identifier	mandatory
dct:issued	optional
dct:modified	optional
dcat:landingPage	optional
dcat:keyword	optional
dct:language	optional
dcat:contactPoint	optional
documentation (foaf:page)	optional
schema:image	optional
dct:temporal	optional
dcat:temporalResolution	optional
frequency (dct:accrualPeriodicity)	optional
dct:isReferencedBy	optional
dct:relation	optional
spatial/geographical coverage (dct:spatial)	optional
dct:conformsTo	optional
dcat:theme	optional
dcat:version	optional
adms:versionNotes	optional

prov.qualifiedAttribution and prov.qualifiedRelation are not supported automatically, you can add those informations manually on I14Y.

• Supported properties for dcat.Distribution:

Property	Requirement level
dct:title (if not stated, set automatically to 'Datenexport')	mandatory
dct:description (if not stated, set automatically to 'Export der Daten')	mandatory
dcat:accessURL	mandatory
dcat:downloadURL	optional
dct:license	optional
dct:issued	optional
dct:modified	optional
dct:rights	optional
dct:language	optional
schema:image	optional
dcat:spatialResolutionInMeters	optional
dcat:temporalResolution	optional
dct:conformsTo	optional
dcat:mediaType	optional
dct:format	optional
dct:packageFormat	optional
spdx:checksum	optional
dcat:byteSize	optional

Process Overview

In detail, the process works as follows:

1. Authentication Process

• Obtains access token using client credentials
• Uses secrets CLIENT_ID and CLIENT_SECRET stored securely

2. Data Harvesting Process

• Fetches current datasets: Retrieves OGD from DAM API (https://dam-api.bfs.admin.ch/hub/api/ogd/harvest)
• Fetches already existing datasets from I14Y: Gets all datasets for organisation CH1 with identifier xxx@bundesamt-fur-statistik-bfs
• Processes each dataset:
  • Compares with previous version
  • Identifies new, updated, unchanged or deleted datasets
  • Creates appropriate payload for I14Y API

3. Dataset Processing Logic

• For new/updated datasets:
  • Checks if the dataset is valid:
    • At least one description for the dataset
    • PDF distribution is discarded
    • At least one distribution per dataset
  • Creates proper API payload
  • Submits to I14Y API (api-a.i14y.admin.ch for ABN, api-a.i14y.admin.ch for PROD)
  • For new datasets, automatically sets:
    • Status: "Recorded"
    • Level: "Public"

4. Structures Import

• Once the datasets are imported, we also import the Structures for the datasets that were created or updated by the Harvester
• datasets.json is used by StructureImporter to know which datasets were created or updated by the harvester
• For now in the workflows, we import the structures only for ABN, not prod

5. Output and Logging

(Logs retention: 2 days, configurable)

• Artifacts:

  • datasets.json: State of processed datasets by harvester
  • harvest_log.txt: Detailed operation log for import of datasets
  • structure_import_log.txt: Detailed operation log for import of structures

• Log includes:

  • Timestamp of operation
  • List of created datasets
  • List of updated datasets
  • List of unchanged datasets
  • List of deleted datasets