- Java 65.2%
- TypeScript 17.9%
- HTML 5.9%
- PLpgSQL 4.7%
- SCSS 3.3%
- Other 2.9%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
|
|
||
| .github | ||
| .mvn | ||
| docs | ||
| hello-data-commons | ||
| hello-data-deployment | ||
| hello-data-portal | ||
| hello-data-sidecars | ||
| hello-data-subsystems | ||
| .gitignore | ||
| AGENTS.md | ||
| CLAUDE.md | ||
| CODE-OF-CONDUCT.md | ||
| CONTRIBUTING.md | ||
| LICENSE | ||
| mvnw | ||
| mvnw.cmd | ||
| pom.xml | ||
| README.de.md | ||
| README.md | ||
| renovate.json | ||
| THIRD-PARTY-LICENSES.md | ||
HelloDATA BE: Open-Source Enterprise Data Platform
The Open Enterprise Data Platform in a Single Portal
HelloDATA BE integrates the prowess of open-source tools into a unified, enterprise-grade data platform. It simplifies end-to-end data engineering by converging tools like dbt, Airflow, and Superset, anchored on a robust Postgres database. Our platform encapsulates the agility of open source with the dependability of enterprise solutions, streamlining your data workflow in one accessible portal.
Vision and Goal
Agile and transparent data platforms are vital in a rapidly evolving digital landscape. HelloDATA BE tries to help by offering an open-source solution tailored to enterprise needs. Our mission is to democratize data and provide end-to-end- innovation-driven data handling. Learn More.
Quick Start
Essential Setup
- Docker Desktop Settings: For Mac users, adjust Docker Desktop settings for multi-platform support (especially for arm64-chip users). Details.
- Hosts File Entry: Ensure
127.0.0.1 host.docker.internalis added to/etc/hosts. For Windows enable in Docker Desktop WSL settings How to.
Start-Up Instructions
Change directory to hello-data-deployment/docker-compose and run:
docker-compose up -d
Pulling all images and starting up the containers will take a while. Once completed, access the web portal at localhost:8080 (default: admin/admin).
Note:
- Detailed start-up instructions, troubleshooting, and FAQs are in the docker-compose README.
- Specific setup information for Windows users.
Hint
If you don't have enough resources, you can try to start the portal and basic services only with:
docker compose up -d hello-data-portal-ui
If you see any service unhealthy, try to repeat the command. If this doesn't help, please check the logs of the unhealthy services and create an issue on GitHub.
The above command will start only the Portal, Keycloak, dbt and the Postgres database. However, other services like Airflow and the Superset will not be available. You can later start the other services by running:
docker compose up -d hello-data-airflow-sidecar hello-data-superset-sidecar-default-data-domain hello-data-sftpgo-sidecar
Or just pick one of required services by analyzing the docker compose yaml files:
docker compose up -d <service-name>
Key Features
HelloDATA BE is more than a platform; it's a holistic data solution:
- Unified enterprise data platform with a seamless open-source toolkit.
- Domain-centric architecture for effective data governance.
- Robust data modeling with lineage tracking.
- Real-time insights into runs and orchestrations.
- Company-wide dashboard creation and sharing.
Architecture and Components
HelloDATA BE is built with modularity and extensibility architecture in mind, supported by NATS, Keycloak, and more, ensuring secure, efficient, and adaptable data handling. Explore the Architecture.
Domain View: Business and Data Domains
The different views are a vital aspect of the HelloDATA BE. Adding multiple business domains with n data domains makes it enterprise-ready out of the box.
- Business Domain: Core services, including portal, orchestration, and monitoring.
- Data Domain: The heart of data storage, encompassing tables/schemas, dbt data models, data marts, DWH environments, and more.
Multiple Data Domains coexist in each business domain, each with dedicated storage and specialized data models. Dive Deeper. Also, check the User Manual for a detailed functional overview.
Docs Website: All the reference and developer documentation here on GitHub is also found as a website for more comfort.
Contributing to HelloDATA BE
Join our development journey:
- Developer Guidelines
- HelloDATA BE Architecture
Development Status
Stay updated with our progress and plans on our Roadmap, or get in contact with Discussion or PR/Issue.
Contributors
Current Main Contributors
- Andreea Hansel (Tester)
- Dario Bagatto (Developer)
- Dmitry Grigoriev (Developer)
- Felix Neidhart (Developer)
- Micha Eichmann (Developer)
- Michael Disteli (Product-Manager)
- Rajib Mitra (Developer)
- Slawomir Wieczorek (Developer)
(in alphabetical order)
Former Contributors
- Adrian Berger (Developer)
- Lorin Reber (Product-Owner, Value-Lead)
- Nicolas Schmid (Initial developer & Architect)
- Simon Späti (Developer)
- Thomas Amlang (Developer)
(in alphabetical order)
License
HelloDATA BE is released under the BSD 3-Clause License. Copyright (c) 2024 Kanton Bern.
Get in Touch
We recommend you start a discussion here on GitHub or create an issue. Alternatively, you can reach us at info-hd-be@bedag.ch.