DCAT Harvester Template

An harvester template for importing automatically and periodically datasets into I14Y using GitHub Actions.

Features

• Automated harvesting via scheduled GitHub Actions. - Pre-configured workflows for staging (ABN) and production (PROD) environments.
• template scripts to harvest DCAT datasets in rdf format or json format

Supported properties for dcat.Dataset:

Property	Requirement level
dct:title	mandatory
dct:description	mandatory
dct:accessRight (chosen from: PUBLIC, NON_PUBLIC, CONFIDENTIAL, RESTRICTED)	mandatory
dct:publisher (stated in config.py)	mandatory
dct:identifier	mandatory
dct:issued	optional
dct:modified	optional
dcat:landingPage	optional
dcat:keyword	optional
dct:language	optional
dcat:contactPoint	optional
documentation (foaf:page)	optional
schema:image	optional
dct:temporal	optional
dcat:temporalResolution	optional
frequency (dct:accrualPeriodicity)	optional
dct:isReferencedBy	optional
dct:relation	optional
spatial/geographical coverage (dct:spatial)	optional
dct:conformsTo	optional
dcat:theme	optional
dcat:version	optional
adms:versionNotes	optional

NOTE: prov.qualifiedAttribution and prov.qualifiedRelation are not supported automatically, you can add those informations manually on I14Y.

Supported properties for dcat.Distribution:

Property	Requirement level
dct:title (if not stated, set automatically to 'Datenexport')	mandatory
dct:description (if not stated, set automatically to 'Export der Daten')	mandatory
dcat:accessURL	mandatory
dcat:downloadURL	optional
dct:license	optional
dct:issued	optional
dct:modified	optional
dct:rights	optional
dct:language	optional
schema:image	optional
dcat:spatialResolutionInMeters	optional
dcat:temporalResolution	optional
dct:conformsTo	optional
dcat:mediaType	optional
dct:format	optional
dct:packageFormat	optional
spdx:checksum	optional
dcat:byteSize	optional

Prerequisites

• Python 3.8+
• pip package manager
• GitHub repository secrets configured (see Installation).

Installation

1. Clone and Setup

git clone [repository-url]
cd harvester_template
pip install -r requirements.txt  # Install dependencies

2. Configuration

Environment Variables

1. Navigate to your GitHub repository:
   • Settings → Secrets → Actions
   • Add the following secrets (provided via encrypted email):
     • CLIENT_ID_PROD, CLIENT_SECRET_PROD (Production)
     • CLIENT_ID_ABN, CLIENT_SECRET_ABN (Staging)

File Configuration Edit src/config.py:

PUBLISHER_ID = "your-publisher-id"  # Provided by BFS
HARVESTER_API_URL = ""  # API endpoint (if you want to harvest using an API endpoint)
HARVESTER_FILE_PATH = "data/datasets.json"  # Local file path (if you want to harvest form a file)

3. Select and configure Harvester

You can either use your own harvester.py script, or use a template harvester script. In the latter case, choose either the JSON or RDF harvester:

1. Copy the harvester template

# For JSON datasets:
cp src/harvesters_template/json/harvester.py src/harvester.py

# For RDF datasets:
cp src/harvesters_template/rdf/harvester.py src/harvester.py

2. Adapt the template:
   • Edit the copied src/harvester.py to match your dataset structure (e.g., field mappings).
   • No need to modify the GitHub Actions workflows (.yml files) — they’re preconfigured to use src/harvester.py.

Usage

1. Run Locally (Testing):

python src/harvester.py  # Tests the active harvester (JSON/RDF)

Before deploying, verify your src/harvester.py:

• For JSON: Check field_mapping.py matches your dataset fields.
• For RDF: Validate dcat_properties_importer.py aligns with your RDF schema.

2. **Automate via Github Actions:

To set up the github action workflow, you just need to uncomment the first line of the script:

• For pruduction: use .github/workflows/harvester_prod.yml.
• For staging: Use .github/workflows/harvester_abn.yml.

To trigger manually:

• Go to GitHub → Actions → [Workflow] → Run Workflow.

Repository Structure

harvester_template/
├── .github/workflows                # GitHub Actions workflows
│   ├── harvester_abn.yml
│   └── harvester_prod.yml
├── data/                            # Sample datasets (optional)
│   ├── datasets.rdf
│   └── datasets.json
├── src/
│    ├── harvesters_template/        # Harvester templates to import DCAT datasets
│    │   ├── json/  
│    │   │   ├── field_mapping.py  
│    │   │   └── harvester.py  
│    │   └── rdf/  
│    │        ├── dcat_properties_importer.py  
│    │        ├── utils.py
│    │        └── harvester.py  
│    ├── config.py                   # Global settings 
│    ├── harvester.py                # Active harvester (copy from harvesters_template/json/ or harvesters_template/rdf/)
│    └── mappings.py                 # Some useful properties codes mapping, e.g. themes (optional and adjustable for specific use case)
├── requirements.txt                 # Python dependencies 
└── README.md                        # This file