Skip to content
This repository has been archived by the owner on Feb 10, 2025. It is now read-only.

RabbitMQ forwader to Amazon services

License

Notifications You must be signed in to change notification settings

AirHelp/rabbit-amazon-forwarder

Folders and files

NameName
Last commit message
Last commit date
Nov 27, 2018
Apr 5, 2022
Feb 22, 2017
Nov 27, 2018
Mar 16, 2017
Apr 4, 2017
Sep 29, 2017
Nov 27, 2018
Apr 5, 2022
Sep 29, 2017
Sep 29, 2017
Nov 27, 2018
Mar 3, 2017
Oct 31, 2017
Jun 19, 2018
Nov 14, 2017
Apr 5, 2022
Apr 5, 2022
Apr 4, 2017
Oct 31, 2017
Sep 4, 2019
Nov 27, 2018
Feb 10, 2025
Feb 10, 2025
Nov 27, 2018

Repository files navigation

RabbitMQ -> Amazon forwarder

Go Report Card

Application to forward messages from RabbitMQ to different Amazon services.

Key features:

  • forwarding RabbitMQ message to AWS SNS topic
  • forwarding RabbitMQ message to AWS SNS queue
  • triggering AWS lambda function directly from RabbitMQ message
  • automatic RabbitMQ reconnect
  • message delivery assurance based on RabbitMQ persistency and AWS error handling
  • dedicated dead-letter exchange and queue creation
  • http health checks and restart functionality

Architecture

Alt text

Configuration

The list of RabbitMQ sources and corresponding AWS target resources are stored in mapping file.

Mapping file

Sample of RabbitMQ -> SNS mapping file. All fields are required. Samples are located in examples directory.

[
  {
    "source" : {
      "type" : "RabbitMQ",
      "name" : "test-rabbit",
      "connection" : "amqp://guest:guest@localhost:5672/",
      "topic" : "amq.topic",
      "queue" : "test-queue",
      "routingKeys" : ["#"]
    },
    "destination" : {
      "type" : "SNS",
      "name" : "test-sns",
      "target" : "arn:aws:sns:eu-west-1:XXXXXXXX:test-forwarder"
    }
  }
]

Environment variables

Forwarder uses the following environment variables:

export MAPPING_FILE=/config/mapping.json
export AWS_REGION=region
export AWS_ACCESS_KEY_ID=access_key
export AWS_SECRET_ACCESS_KEY=secret_key

Using TLS with rabbit

Specify amqps for the rabbit connection ub the mapping file:

 "connection" : "amqps://guest:guest@localhost:5671/",

Additional environment variables for working with TLS and rabbit:

export CA_CERT=/certs/ca_certificate.pem
export CERT_FILE=/certs/client_certificate.pem
export KEY_FILE=/certs/client_key.pem

Amazon configuration

When making subscription to SNS -> SQS/HTTP/HTTPS set Raw message delivery to ensure that json messages are not escaped.

Build docker image

make build

Run

Using docker:

docker run \
-e AWS_REGION=$AWS_REGION \
-e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
-e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
-e MAPPING_FILE=/config/mapping.json \
-v $MAPPING_FILE:/config/mapping.json \
-p 8080:8080 \
airhelp/rabbit-amazon-forwarder

Using docker-compose:

docker-compose up

Test

docker-compose build --pull
docker-compose run --rm tests

Release

make push
docker tag airhelp/rabbit-amazon-forwarder airhelp/rabbit-amazon-forwarder:$VERSION
docker push airhelp/rabbit-amazon-forwarder:$VERSION

Supervisor

Supervisor is a module which starts the consumer->forwarder pairs. Exposed endpoints:

  • APP_URL/health - returns status if all consumers are running
  • APP_URL/restart - restarts all consumer->forwarder pairs