Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: eheydrick/aws-cleaner
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: 2.0.0
Choose a base ref
...
head repository: eheydrick/aws-cleaner
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: master
Choose a head ref
Loading
Showing with 567 additions and 302 deletions.
  1. +4 −1 .gitignore
  2. +115 −0 .rubocop.yml
  3. +15 −0 .travis.yml
  4. +53 −19 CHANGELOG.md
  5. +1 −5 Gemfile
  6. +0 −39 Gemfile.lock
  7. +51 −13 README.md
  8. +12 −0 Rakefile
  9. +12 −7 aws-cleaner.gemspec
  10. BIN aws-cleaner.png
  11. +111 −212 bin/aws_cleaner.rb
  12. +4 −6 config.yml.sample
  13. +189 −0 lib/aws-cleaner.rb
5 changes: 4 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -79,9 +79,12 @@ build/

# for a library or gem, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in:
# Gemfile.lock
Gemfile.lock
# .ruby-version
# .ruby-gemset

# unless supporting rvm < 1.11.0 or doing something fancy, ignore this:
.rvmrc

### aws_cleaner config ###
config.yml
115 changes: 115 additions & 0 deletions .rubocop.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
# Rubocop, we're buddies and all, but we're going to have to disagree on the following -

# Allow compact class definitions
Style/ClassAndModuleChildren:
Enabled: false

# Allow if statements!
Style/GuardClause:
Enabled: false

# Allow more complex ruby methods
Metrics/AbcSize:
Max: 100

# you cant determine complexity!
PerceivedComplexity:
Enabled: false

# Disable requirement of "encoding" headers on files
Encoding:
Enabled: false

# Increase line length, we're not on VT220s anymore
LineLength:
Max: 255
# To make it possible to copy or click on URIs in the code, we allow lines
# containing a URI to be longer than Max.
URISchemes:
- http
- https

# Longer classes aren't _so_ bad
Metrics/ClassLength:
Max: 125

# Increase allowed lines in a method. Short methods are good, but 10 lines
# is a bit too low.
MethodLength:
CountComments: false # count full line comments?
Max: 40

# Favor explicit over implicit code: don't complain of "redundant returns"
RedundantReturn:
Enabled: false

# Don't complain about if/unless modifiers. The merit of this is debatable
# and it will likely require building of over-length lines.
IfUnlessModifier:
Enabled: false

# Raise allowed CyclomaticComplexity & Perceivedto 10.
CyclomaticComplexity:
Max: 15

Metrics/PerceivedComplexity:
Max: 15

# Don't force a word array unless 5 elements
WordArray:
MinSize: 5

# Don't complain about unused block args
UnusedBlockArgument:
Enabled: false

# allow both hash syntaxes
Style/HashSyntax:
Enabled: false

# allow final rescue
Style/RescueModifier:
Enabled: false

Naming/AccessorMethodName:
Enabled: false

# allow larger modules
Metrics/ModuleLength:
Max: 150

# disable opinionated doc requirements such as a top-level class comments
Documentation:
Enabled: false

# allow using parenthases to group an expresion
Lint/ParenthesesAsGroupedExpression:
Enabled: false

# NumericLiterals:
# Enabled: false

# allow multiple spaces between methofs and first arg
Layout/SpaceBeforeFirstArg:
Enabled: false

# There are too many non-ruby files that run up against rubocop rules in a cookbook
AllCops:
Include:
- '**/*.rb'

# allow longer blocks
Metrics/BlockLength:
Max: 50

# allow longer argument lists
Metrics/ParameterLists:
Max: 6

# allow using $stderr
Style/StderrPuts:
Enabled: false

# disable naming check
Naming/FileName:
Enabled: false
15 changes: 15 additions & 0 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
language: ruby
dist: xenial
sudo: false
cache: bundler

before_install:
- gem install bundler

rvm:
- 2.4.6
- 2.5.5
- 2.6.3

script:
- bundle exec rake
72 changes: 53 additions & 19 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -2,32 +2,66 @@

### Unreleased

### 2.0.0
- add support for sending notifications to Slack. Note: the config settings for chat notifications has changed to add support for multiple chat providers.
### 2.4.2 - 2019-12-01
- Remove unused hipchat support

### 1.0.0
### 2.4.1 - 2019-09-19
- Replace chef-api with chef-infra-api

### 2.4.0 - 2019-07-31
- Use aws-sdk v3. (Thanks [@genehand](https://github.com/genehand))
- Replace trollop with optimist
- Test on latest ruby versions, drop 2.3
- Update to Rubocop 0.73

### 2.3.0 - 2018-03-28
- Make sensu removal optional. Defaults to enabled but can be disabled by setting `:enable: false` in the config. (Thanks [@majormoses](https://github.com/majormoses))

### 2.2.1 - 2018-02-22
- Fix webhook templating
- Handle webhook exceptions better

### 2.2.0 - 2018-02-21
- Improve logging. You can now log to a file in addition to stdout. [#10](https://github.com/eheydrick/aws-cleaner/issues/10) (Thanks [@huynt1979](https://github.com/huynt1979))
- Update to latest Rubocop to address security issue
- Test on latest rubies

### 2.1.1 - 2017-05-07
- Rename lib file to be more easily consumed. Previously had to `gem install aws-cleaner` and then within any code use `require 'aws_cleaner/aws_cleaner.rb'`. Now you only need to `require aws-cleaner`. ([@majormoses](https://github.com/majormoses))

### 2.1.0 - 2017-03-28
- Refactor logic into a library (Thanks [@majormoses](https://github.com/majormoses))
- Remove the `argument` parameter from the webhook config. We now always use the instance ID when templating webhooks.

### 2.0.1 - 2016-06-30
- Actually add `slack-poster` dependency

### 2.0.0 - 2016-06-30
- Add support for sending notifications to Slack. Note: the config settings for chat notifications has changed to add support for multiple chat providers.

### 1.0.0 - 2016-04-26
- AWS Cleaner now uses [CloudWatch Events](http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/WhatIsCloudWatchEvents.html) instead of
AWS Config to receive EC2 instance termination events. CloudWatch Events delivers events in seconds while AWS Config can take several minutes.

### 0.3.1
- add better error handling
### 0.3.1 - 2016-04-21
- Add better error handling

### 0.3.0
- add hipchat notifications when webhooks fire
### 0.3.0 - 2016-01-06
- Add hipchat notifications when webhooks fire

### 0.2.1
- fix options
### 0.2.1 - 2015-12-22
- Fix options

### 0.2.0
- add webhooks
### 0.2.0 - 2015-12-22
- Add webhooks

### 0.1.3
- look for chef-provisioning attributes when searching for chef nodes
### 0.1.3 - 2015-07-30
- Look for chef-provisioning attributes when searching for chef nodes

### 0.1.2
- improve exception handling
- fix typos
- notify hipchat only when enabled
### 0.1.2 - 2015-07-30
- Improve exception handling
- Fix typos
- Notify hipchat only when enabled

### 0.1.1
- initial release
### 0.1.1 - 2015-07-29
- Initial release
6 changes: 1 addition & 5 deletions Gemfile
Original file line number Diff line number Diff line change
@@ -1,7 +1,3 @@
source 'https://rubygems.org'

gem 'aws-sdk-core'
gem 'chef-api'
gem 'hipchat'
gem 'rest-client'
gem 'trollop'
gemspec
39 changes: 0 additions & 39 deletions Gemfile.lock

This file was deleted.

64 changes: 51 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,23 +1,39 @@
## AWS Cleaner

[![Build Status](https://travis-ci.org/eheydrick/aws-cleaner.svg?branch=master)](https://travis-ci.org/eheydrick/aws-cleaner)
[![Gem Version](https://badge.fury.io/rb/aws-cleaner.svg)](http://badge.fury.io/rb/aws-cleaner)
[![Dependency Status](https://gemnasium.com/badges/github.com/eheydrick/aws-cleaner.svg)](https://gemnasium.com/github.com/eheydrick/aws-cleaner)

AWS Cleaner listens for EC2 termination events produced by AWS [CloudWatch Events](http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/WhatIsCloudWatchEvents.html)
and removes the instances from Chef and Sensu monitoring. Optionally
sends messages to Hipchat or Slack when actions occur.
and removes the instances from Chef. It can optionally remove the node from Sensu Monitoring (defaults true), fire off webhooks, and send notifications via Slack when actions occur.

![aws-cleaner](https://raw.github.com/eheydrick/aws-cleaner/master/aws-cleaner.png)

### Prerequisites

You will need to create a CloudWatch Events rule that's configured to send termination event messages to SQS.

1. Create an SQS Queue for cloudwatch-events
2. Goto CloudWatch Events in the AWS Console
3. Click *Create rule*
4. Select event source of *EC2 instance state change notification*
5. Select specific state of *Terminated*
6. Add a target of *SQS Queue* and set queue to the cloudwatch-events queue created in step one
7. Give the rule a name/description and click *Create rule*
1. Goto CloudWatch Events in the AWS Console
1. Click *Create rule*
1. Select event source of *EC2 instance state change notification*
1. Select specific state of *Terminated*
1. Add a target of *SQS Queue* and set queue to the cloudwatch-events queue created in step one
1. Give the rule a name/description and click *Create rule*

You will also need to create a user with the required permissions. I recommend creating a 'aws-cleaner' user in chef and add it to its own group. The minimum permissions we found that works is read and delete nodes/clients.

Steps:

1. on chef server: `chef-server-ctl user-create aws-cleaner AWS Cleaner`
1. on chef server: `address@domain.tld "$SOMEREALLYLONGRANDOMPASSWORD" -f aws-cleaner.pem`
1. on chef server: `chef-server-ctl org-user-add $MYORG aws-cleaner`
1. on workstation: `gem install knife-acl`
1. on workstation: `knife group create aws-cleaner`
1. on workstation: `knife group add user aws-cleaner aws-cleaner`
1. on workstation: `knife acl bulk add group aws-cleaner clients '.*' read,delete -y`
1. on workstation: `knife acl bulk add group aws-cleaner nodes '.*' read,delete -y`

An astute reader might notice that this wont work for new nodes that come up as they have not had their ACL updated. I recommend that you add the who bulk acl knife commands (modified for just self as opposed to bulk) as part of your normal bootstrap process before deleting your validation key.

### Installation

@@ -34,7 +50,8 @@ Options:
Copy the example config file ``config.yml.sample`` to ``config.yml``
and fill in the configuration details. You will need AWS Credentials
and are strongly encouraged to use an IAM user with access limited to
the AWS CloudWatch Events SQS queue.
the AWS CloudWatch Events SQS queue.You will need to specify the region
in the config even if you are using IAM Credentials.

The app takes one arg '-c' that points at the config file. If -c is
omitted it will look for the config file in the current directory.
@@ -43,6 +60,11 @@ The app is started by running aws_config.rb and it will run until
terminated. A production install would start it with upstart or
similar.

### Logging

By default aws-cleaner will log to STDOUT. If you wish to log to a specific file
add a `log` section to the config. See [`config.yml.sample`](config.yml.sample) for an example.

### Webhooks

AWS Cleaner can optionally make an HTTP request to a specified endpoint. You can
@@ -59,13 +81,29 @@ To enable webhooks, add a `:webhooks:` section to the config:
:template_variables:
:variable: 'fqdn'
:method: 'get_chef_fqdn' (or 'get_chef_node_name')
:argument: '@instance_id'
```

Chat notifications can be sent when the webhook successfully executes. See
config.yml.sample for an example of the config.

### Limitations
### Sensu

Currently only supports a single AWS region.
By default aws-cleaner assumes that removing from sensu is desired as this was one of its core intentions. To allow people to leverage this without sensu you can disable it via config:
```
:sensu:
:enable: false
```

When wanting to use sensu you will want the following config:
```
:sensu:
:url: 'http://sensu.example.com:4567'
:enable: true
```

While we currently assume sensu removal being desired is considered the default it may not always be so you should set `enable` to true to avoid a breaking change later.

### Limitations

- Currently only supports a single AWS region.
- Only support chef and sensu with non self signed certificates. Look at Aws Certificate Manager or Let's Encrypt for free SSL certificates.
12 changes: 12 additions & 0 deletions Rakefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
require 'bundler/gem_tasks'
require 'rubocop/rake_task'
require 'rspec/core/rake_task'

desc 'Run Rubocop'
RuboCop::RakeTask.new(:rubocop) do |task|
task.patterns = ['bin/*.rb', 'lib/**/*.rb']
end

RSpec::Core::RakeTask.new(:spec)

task default: [:rubocop, :spec]
Loading