Certificate reader for travel industry

This is an API resporstory for certificate reader project. To read more about the whole project and frontend: https://github.com/Dataatti/sertifikaattilukija/blob/main/README.md

Features

Periodically fetch data from certificate issuers and serve the data to frontend.

Data endpoint provides a open endpoint where you can search for comany sertificates by their name, vat-number, municipality or by the certificates they have. Documentation

Tech Stack

The data is fetched by scraping certificate provider websites with Node.js functions and saved to a PostgreSQL database.

The data is fetched periodicly every sunday using Breejs to help with scheduling.

Docker is used to run project in production documentation

How to run

Installation

Clone the repository

  git clone https://github.com/Dataatti/sertifikaattilukija-api.git

Change to the project directory

  cd sertifikaattilukija-api

Environment variables

Environment variables have to be set before running the application.

Either copy .env.example to local .env file and update values

  cp .env.example .env

or set them in a environment specific way (if you are running the application in a cloud provider for example).

Variable list

Variable	Value
DATABASE_CONNECTION_URL	PostgreSQL connection url
PORT	(Optional) Start server in specific port, default: 4242

Application

Install project dependencies based on lockfile

  npm install

Run the development server

See Starting server with vscode debugger

Starting server with vscode debugger

• Create .vscode/launch.json
• Add lines below to the file

{
  "version": "0.2.0",
  "configurations": [
    {
      "runtimeVersion": "16.14.0",
      "type": "node",
      "request": "launch",
      "name": "Launch api server",
      "program": "${workspaceFolder}/server.ts",
      "preLaunchTask": "tsc: build - tsconfig.json",
      "outFiles": ["${workspaceFolder}/dist/**/*.js"],
      "outputCapture": "std",
      "cwd": "${workspaceFolder}/dist",
      "envFile": "${workspaceFolder}/.env"
      // optionally
      // "env": {
      //   DATABASE_CONNECTION_URL: <CONNECTION-URL>
      //   PORT: <PORT>
      // }
    }
  ]
}

• Run vscode debugger

Docker

Application can be run with docker. Docker container compiles the typescript and then removes all typescript references from directory and then runs the application in production mode.

Building docker:

docker build -t sertifikaattilukija .

Docker uses .env file to set enviroment variables. To generate enviroment variables see Environment variables

Starting docker container

docker run -it -d -p 4242 --env-file .env sertifikaattilukija

Accessing docker logs

docker exec -it <container-id> cat logs.log

Deplying to production

The api needs PostgreSQL to store data, the database can be hosted in any cloud provider, all we need is the ability to connect to it via PostgreSQL connection url.

The api can be deployed to any cloud provider using Docker see: Docker section

• Clone the repostory and go to the directory
• Copy and fill in the enviroment variables
• Build and run the docker container see: docker section
  • Database seeds will be run to the databse automatically when the container starts
• Confirm that the api is running by going to the public url and you should then see Cannot GET / message

Deplying can be automated using Github Actions

Run tests with jest

Jest tests are run against all pull and push requests by a quality check Github action.

To run tests locally run

npm run test

Data scraping and database

Database

When server is initialized the database client is created and the needed tables are created if they don't exist yet in database.

Database schemas can be found at: https://github.com/Dataatti/sertifikaattilukija-api/blob/main/utils/database.ts#L29

Notes

If in table company blacklisted value is set to truethe /data endpoint will ignore it

Data scraping

Company information is fetched from PRH Open Data

The list of scraped certificate websites is based on the certificates that are part of Business Finland's Sustainable Travel Finland program.

The websites and api's where data is fetched can be found in scrapers.json