Cube Creator

Cube Creator is a tool to create RDF data Cubes (based on rdf-cube-schema) out of CSV files.

Technologies

The tool is built with Typescript and is currently composed of two main parts: a Hydra API (built with hydra-box) and a Vue.js user interface. See the Architecture document for a high-level overview.

Running locally

Preparing the environment

To be able to run transform/publish pipelines locally, an OIDC secret must be added to cli/.env as

AUTH_RUNNER_CLIENT_SECRET=foo-bar

It is obtained from keycloak

Alternatively, you can bypass authentication altogether by setting the environment variables in .env similar to the following example:

VUE_APP_E2E=true
VUE_APP_X_USER=john-doe
VUE_APP_X_PERMISSION=pipelines:read,pipelines:write

If you have already started the application, make sure to run lando rebuild -y to apply the changes.

Publishing to an RDF Store

Cube publishing uses separate graph store protocol and query endpoints.

PUBLISH_GRAPH_STORE_ENDPOINT=
PUBLISH_GRAPH_QUERY_ENDPOINT=
PUBLISH_GRAPH_STORE_USER=...
PUBLISH_GRAPH_STORE_PASSWORD=...

If you pass publishStore in job configuration, use: storeEndpoint and queryEndpoint.

Starting

The easiest way it to start a local dockerized environment which will run the database, API and UI, and provide set up local HTTPS endpoints for them.

1. Download and install lando
   • it will install docker desktop if necessary
2. Run yarn to install packages
3. Run lando start inside the repo
   • Docker daemon is also started automatically
4. (Optional) Run yarn seed-data to add sample projects to the database

Docker containers will start and the services will be available under the these URLs:

Service	URL
API	https://cube-creator.lndo.site/
UI	https://app.cube-creator.lndo.site/
Fuseki	https://db.cube-creator.lndo.site/
Minio	https://s3.cube-creator.lndo.site/
Jaeger	http://jaeger.cube-creator.lndo.site
Prometheus	http://prometheus.cube-creator.lndo.site

Lando uses its own Certificate Authority and it won't be trusted by your system. To trust the CA, follow the steps on https://docs.lando.dev/config/security.html#trusting-the-ca

Workflow

This repository is using the GitHub-Flow. Hence all changes should be integrated using pull requests.

Commit messages usually follow the guidelines from Conventional Commits with the types in lower case.

To submit a bug or a feature request please create an issue in this repository.

Release process

All notable changes to the project(s) should be documented using @atlassian/changesets. When preparing a PR, run yarn changeset in the repository which will ask what changed, the affected packages, and create a small markdown file which must be committed to the repository.

Alternatively, that file can be created directly in the browser, courtesy of Changesets bot which creates PR comment summarising the changes included in the PR in question.

Releases are managed automatically by GitHub actions. As PRs get merged to master, the first job creates or updates a pull request which bumps package versions according to the combined information from all accumulated changeset files. Once merged, the second job kicks in which tags the repository and triggers INT deployment.

The package versions can also be bumped manually by running yarn changeset version in the repository and committing the result. When pushed, the new versions will be picked up by the aforementioned GitHub workflow job to tag the respository.

E2E tests

There are two types of e2e tests:

• API (Hydra): e2e tests that take the API as entrypoint
• UI (Cypress): browser-based e2e tests

API e2e tests

Running the E2E tests can be done using: docker compose run --rm e2e-tests, and docker compose run --rm e2e-tests -- --grep pattern lets you select which tests to run.

For brevity, use npm script npm run test:e2e --grep pattern

UI e2e tests

We use Cypress to run UI e2e tests.

To simplify the tests, we circumvent authentication in the app. For that, the following variables need to be set in .env before running the UI:

VUE_APP_E2E=true
VUE_APP_X_USER=john-doe
VUE_APP_X_PERMISSION=pipelines:read,pipelines:write

We need a running instance of the app to test. The easiest way is to start lando: lando start

Then the following command can be used to run the tests interactively:

yarn --cwd ui test:e2e --url https://app.cube-creator.lndo.site

The --headless option allows running the tests without seeing the browser.