Open Data Lakehouse Stack

Project to get data and store it in a modern data lakehouse and show it in a data viz platform.

Table of Contents

1. About The Project
   • Built With
2. Getting Started
   • Prerequisites
   • Installation
3. Usage
4. Roadmap
5. Contact

About The Project

This project aims to implement a complete stack, from data extraction to presentation, using free tools, with an open datalakehouse concept and state-of-the-art technologies.

(back to top)

Built With

(back to top)

Getting Started

Some software and libraries are needed to run the project.

Prerequisites

1. Docker
2. Python > 3.8

Installation

1. Python - https://www.python.org/downloads/
2. Docker - https://docs.docker.com/get-docker/
3. Airbyte - https://docs.airbyte.io/
4. Minio - https://min.io/docs/minio/container/index.html

   cd config/containers/minio
   docker-compose up -d

5. MySQL

   cd config/containers/mysql
   docker-compose up -d

6. Airflow - https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html

(back to top)

Contact

Danton Bertuol - [email protected]

(back to top)

Fonts

https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html

https://docs.airbyte.com/deploying-airbyte/local-deployment

https://min.io/docs/minio/container/index.html

(back to top)