Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: eheydrick/aws-cleaner
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: 2.0.0
Choose a base ref
...
head repository: eheydrick/aws-cleaner
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: master
Choose a head ref

Commits on Jun 30, 2016

  1. add slack-poster dependency

    eheydrick committed Jun 30, 2016
    Copy the full SHA
    4de7730 View commit details
  2. don't include Gemfile.lock

    eheydrick committed Jun 30, 2016
    Copy the full SHA
    2977a0a View commit details
  3. Release 2.0.1

    eheydrick committed Jun 30, 2016
    Copy the full SHA
    dd00df6 View commit details
  4. Copy the full SHA
    0816b13 View commit details

Commits on Oct 5, 2016

  1. closes #1

    Additional doc on using IAM as opposed to access keys.
    majormoses committed Oct 5, 2016
    Copy the full SHA
    fe66699 View commit details
  2. Merge pull request #2 from majormoses/master

    closes #1
    eheydrick authored Oct 5, 2016
    Copy the full SHA
    0564e0f View commit details
  3. Copy the full SHA
    f062b15 View commit details
  4. Merge pull request #4 from majormoses/master

    closes #2 adding limitiation of self signed certificates
    eheydrick authored Oct 5, 2016
    Copy the full SHA
    1712970 View commit details

Commits on Oct 6, 2016

  1. add image

    eheydrick committed Oct 6, 2016
    Copy the full SHA
    46db76e View commit details
  2. move image down

    eheydrick committed Oct 6, 2016
    Copy the full SHA
    af7b727 View commit details

Commits on Oct 7, 2016

  1. closes #7:

    added some doc on the min required permissions and setup.
    majormoses committed Oct 7, 2016
    Copy the full SHA
    c57127c View commit details
  2. Merge pull request #8 from majormoses/master

    closes #7:
    eheydrick authored Oct 7, 2016
    Copy the full SHA
    0d441bc View commit details

Commits on Dec 23, 2016

  1. first pass at refactor logic into lib

    added .rubocop.yml
    updated rest-client
    removed unsafe eval
    majormoses committed Dec 23, 2016
    Copy the full SHA
    0e73979 View commit details

Commits on Dec 27, 2016

  1. Merge pull request #9 from majormoses/feature/refactor-lib

    refactor logic into lib
    eheydrick authored Dec 27, 2016
    Copy the full SHA
    4db68a2 View commit details

Commits on Dec 28, 2016

  1. Copy the full SHA
    f1f74f4 View commit details
  2. rename to .rubocop.yml

    eheydrick committed Dec 28, 2016
    Copy the full SHA
    3cc62b5 View commit details
  3. add travis config

    eheydrick committed Dec 28, 2016
    Copy the full SHA
    07aa861 View commit details
  4. address rubocop issues

    eheydrick committed Dec 28, 2016
    Copy the full SHA
    ebf251c View commit details
  5. add travis badge

    eheydrick committed Dec 28, 2016
    Copy the full SHA
    213209a View commit details
  6. fixing readme formatting

    majormoses committed Dec 28, 2016
    Copy the full SHA
    8b9526f View commit details
  7. Merge pull request #13 from majormoses/master

    fixing readme formatting
    eheydrick authored Dec 28, 2016
    Copy the full SHA
    039e25a View commit details

Commits on Mar 28, 2017

  1. Copy the full SHA
    fc6b60a View commit details
  2. Copy the full SHA
    459fdb7 View commit details
  3. Remove unused variable

    eheydrick committed Mar 28, 2017
    Copy the full SHA
    b85b99a View commit details
  4. Copy the full SHA
    f17b8ef View commit details
  5. update CHANGELOG

    eheydrick committed Mar 28, 2017
    Copy the full SHA
    ccc2057 View commit details
  6. Capitalize

    eheydrick committed Mar 28, 2017
    Copy the full SHA
    210861f View commit details
  7. update CHANGELOG

    eheydrick committed Mar 28, 2017
    Copy the full SHA
    574061a View commit details
  8. Copy the full SHA
    8a95335 View commit details
  9. Release 2.1.0

    eheydrick committed Mar 28, 2017
    Copy the full SHA
    657feec View commit details

Commits on Apr 4, 2017

  1. Copy the full SHA
    6ac1cb5 View commit details

Commits on May 8, 2017

  1. Merge pull request #14 from majormoses/master

    rename files to make external use more intuitive.
    eheydrick authored May 8, 2017
    Copy the full SHA
    3c8a8c5 View commit details
  2. Release 2.1.1

    eheydrick committed May 8, 2017
    Copy the full SHA
    d8d75d3 View commit details

Commits on Feb 21, 2018

  1. Add simple logging to main program (#15)

    * Add simple logging to main program
    
    Use YAML.safe_load
    
    Refactor main program
    
    * Use local variables to address order of removal
    huynt1979 authored and eheydrick committed Feb 21, 2018
    Copy the full SHA
    d600a5d View commit details
  2. Update rubocop (#16)

    * Update to rubocop 0.52 to address security issue
    
    * Test on latest rubies
    
    * Update for new rubocop
    eheydrick authored Feb 21, 2018
    Copy the full SHA
    3526ef3 View commit details
  3. Add a note about logging

    eheydrick committed Feb 21, 2018
    Copy the full SHA
    c7d3830 View commit details
  4. Release 2.2.0

    eheydrick committed Feb 21, 2018
    Copy the full SHA
    d9cd1f6 View commit details

Commits on Feb 23, 2018

  1. Fix webhook templating (#17)

    The string replacement was overwriting part of the hook config and it
    broke URL templating when more than 1 node was being terminated
    
    Also handle webhook exceptions better
    eheydrick authored Feb 23, 2018
    Copy the full SHA
    63e7d77 View commit details
  2. Release 2.2.1

    eheydrick committed Feb 23, 2018
    Copy the full SHA
    cf617ba View commit details
  3. Remove useless assignment

    eheydrick committed Feb 23, 2018
    Copy the full SHA
    35648d9 View commit details

Commits on Mar 28, 2018

  1. allow disabling removing from sensu (#19)

    It defaults to the old behavior (not set will equal true) while allowing it to be disabled. At some point in the future we can change that default if that makes sense.
    
    Signed-off-by: Ben Abrams <me@benabrams.it>
    majormoses authored and eheydrick committed Mar 28, 2018
    Copy the full SHA
    c67ad3e View commit details
  2. Release 2.3.0

    eheydrick committed Mar 28, 2018
    Copy the full SHA
    4a16201 View commit details

Commits on Jul 31, 2019

  1. Use v3 of the aws-sdk with the service-specific SQS gem (#23)

    * Use v3 of the aws-sdk with the service-specific SQS gem
    
    * update to bundler v2 for travis
    genehand authored and eheydrick committed Jul 31, 2019
    Copy the full SHA
    5f72412 View commit details
  2. Copy the full SHA
    bf48bda View commit details
  3. Copy the full SHA
    f9aebb4 View commit details
  4. Copy the full SHA
    14b84fc View commit details

Commits on Aug 1, 2019

  1. Release 2.4.0

    eheydrick committed Aug 1, 2019
    Copy the full SHA
    3f112f4 View commit details
  2. Copy the full SHA
    67bb575 View commit details

Commits on Sep 19, 2019

  1. Copy the full SHA
    5566245 View commit details
  2. Release 2.4.1

    eheydrick committed Sep 19, 2019
    Copy the full SHA
    6b376f7 View commit details
Showing with 567 additions and 302 deletions.
  1. +4 −1 .gitignore
  2. +115 −0 .rubocop.yml
  3. +15 −0 .travis.yml
  4. +53 −19 CHANGELOG.md
  5. +1 −5 Gemfile
  6. +0 −39 Gemfile.lock
  7. +51 −13 README.md
  8. +12 −0 Rakefile
  9. +12 −7 aws-cleaner.gemspec
  10. BIN aws-cleaner.png
  11. +111 −212 bin/aws_cleaner.rb
  12. +4 −6 config.yml.sample
  13. +189 −0 lib/aws-cleaner.rb
5 changes: 4 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -79,9 +79,12 @@ build/

# for a library or gem, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in:
# Gemfile.lock
Gemfile.lock
# .ruby-version
# .ruby-gemset

# unless supporting rvm < 1.11.0 or doing something fancy, ignore this:
.rvmrc

### aws_cleaner config ###
config.yml
115 changes: 115 additions & 0 deletions .rubocop.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
# Rubocop, we're buddies and all, but we're going to have to disagree on the following -

# Allow compact class definitions
Style/ClassAndModuleChildren:
Enabled: false

# Allow if statements!
Style/GuardClause:
Enabled: false

# Allow more complex ruby methods
Metrics/AbcSize:
Max: 100

# you cant determine complexity!
PerceivedComplexity:
Enabled: false

# Disable requirement of "encoding" headers on files
Encoding:
Enabled: false

# Increase line length, we're not on VT220s anymore
LineLength:
Max: 255
# To make it possible to copy or click on URIs in the code, we allow lines
# containing a URI to be longer than Max.
URISchemes:
- http
- https

# Longer classes aren't _so_ bad
Metrics/ClassLength:
Max: 125

# Increase allowed lines in a method. Short methods are good, but 10 lines
# is a bit too low.
MethodLength:
CountComments: false # count full line comments?
Max: 40

# Favor explicit over implicit code: don't complain of "redundant returns"
RedundantReturn:
Enabled: false

# Don't complain about if/unless modifiers. The merit of this is debatable
# and it will likely require building of over-length lines.
IfUnlessModifier:
Enabled: false

# Raise allowed CyclomaticComplexity & Perceivedto 10.
CyclomaticComplexity:
Max: 15

Metrics/PerceivedComplexity:
Max: 15

# Don't force a word array unless 5 elements
WordArray:
MinSize: 5

# Don't complain about unused block args
UnusedBlockArgument:
Enabled: false

# allow both hash syntaxes
Style/HashSyntax:
Enabled: false

# allow final rescue
Style/RescueModifier:
Enabled: false

Naming/AccessorMethodName:
Enabled: false

# allow larger modules
Metrics/ModuleLength:
Max: 150

# disable opinionated doc requirements such as a top-level class comments
Documentation:
Enabled: false

# allow using parenthases to group an expresion
Lint/ParenthesesAsGroupedExpression:
Enabled: false

# NumericLiterals:
# Enabled: false

# allow multiple spaces between methofs and first arg
Layout/SpaceBeforeFirstArg:
Enabled: false

# There are too many non-ruby files that run up against rubocop rules in a cookbook
AllCops:
Include:
- '**/*.rb'

# allow longer blocks
Metrics/BlockLength:
Max: 50

# allow longer argument lists
Metrics/ParameterLists:
Max: 6

# allow using $stderr
Style/StderrPuts:
Enabled: false

# disable naming check
Naming/FileName:
Enabled: false
15 changes: 15 additions & 0 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
language: ruby
dist: xenial
sudo: false
cache: bundler

before_install:
- gem install bundler

rvm:
- 2.4.6
- 2.5.5
- 2.6.3

script:
- bundle exec rake
72 changes: 53 additions & 19 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -2,32 +2,66 @@

### Unreleased

### 2.0.0
- add support for sending notifications to Slack. Note: the config settings for chat notifications has changed to add support for multiple chat providers.
### 2.4.2 - 2019-12-01
- Remove unused hipchat support

### 1.0.0
### 2.4.1 - 2019-09-19
- Replace chef-api with chef-infra-api

### 2.4.0 - 2019-07-31
- Use aws-sdk v3. (Thanks [@genehand](https://github.com/genehand))
- Replace trollop with optimist
- Test on latest ruby versions, drop 2.3
- Update to Rubocop 0.73

### 2.3.0 - 2018-03-28
- Make sensu removal optional. Defaults to enabled but can be disabled by setting `:enable: false` in the config. (Thanks [@majormoses](https://github.com/majormoses))

### 2.2.1 - 2018-02-22
- Fix webhook templating
- Handle webhook exceptions better

### 2.2.0 - 2018-02-21
- Improve logging. You can now log to a file in addition to stdout. [#10](https://github.com/eheydrick/aws-cleaner/issues/10) (Thanks [@huynt1979](https://github.com/huynt1979))
- Update to latest Rubocop to address security issue
- Test on latest rubies

### 2.1.1 - 2017-05-07
- Rename lib file to be more easily consumed. Previously had to `gem install aws-cleaner` and then within any code use `require 'aws_cleaner/aws_cleaner.rb'`. Now you only need to `require aws-cleaner`. ([@majormoses](https://github.com/majormoses))

### 2.1.0 - 2017-03-28
- Refactor logic into a library (Thanks [@majormoses](https://github.com/majormoses))
- Remove the `argument` parameter from the webhook config. We now always use the instance ID when templating webhooks.

### 2.0.1 - 2016-06-30
- Actually add `slack-poster` dependency

### 2.0.0 - 2016-06-30
- Add support for sending notifications to Slack. Note: the config settings for chat notifications has changed to add support for multiple chat providers.

### 1.0.0 - 2016-04-26
- AWS Cleaner now uses [CloudWatch Events](http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/WhatIsCloudWatchEvents.html) instead of
AWS Config to receive EC2 instance termination events. CloudWatch Events delivers events in seconds while AWS Config can take several minutes.

### 0.3.1
- add better error handling
### 0.3.1 - 2016-04-21
- Add better error handling

### 0.3.0
- add hipchat notifications when webhooks fire
### 0.3.0 - 2016-01-06
- Add hipchat notifications when webhooks fire

### 0.2.1
- fix options
### 0.2.1 - 2015-12-22
- Fix options

### 0.2.0
- add webhooks
### 0.2.0 - 2015-12-22
- Add webhooks

### 0.1.3
- look for chef-provisioning attributes when searching for chef nodes
### 0.1.3 - 2015-07-30
- Look for chef-provisioning attributes when searching for chef nodes

### 0.1.2
- improve exception handling
- fix typos
- notify hipchat only when enabled
### 0.1.2 - 2015-07-30
- Improve exception handling
- Fix typos
- Notify hipchat only when enabled

### 0.1.1
- initial release
### 0.1.1 - 2015-07-29
- Initial release
6 changes: 1 addition & 5 deletions Gemfile
Original file line number Diff line number Diff line change
@@ -1,7 +1,3 @@
source 'https://rubygems.org'

gem 'aws-sdk-core'
gem 'chef-api'
gem 'hipchat'
gem 'rest-client'
gem 'trollop'
gemspec
39 changes: 0 additions & 39 deletions Gemfile.lock

This file was deleted.

64 changes: 51 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,23 +1,39 @@
## AWS Cleaner

[![Build Status](https://travis-ci.org/eheydrick/aws-cleaner.svg?branch=master)](https://travis-ci.org/eheydrick/aws-cleaner)
[![Gem Version](https://badge.fury.io/rb/aws-cleaner.svg)](http://badge.fury.io/rb/aws-cleaner)
[![Dependency Status](https://gemnasium.com/badges/github.com/eheydrick/aws-cleaner.svg)](https://gemnasium.com/github.com/eheydrick/aws-cleaner)

AWS Cleaner listens for EC2 termination events produced by AWS [CloudWatch Events](http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/WhatIsCloudWatchEvents.html)
and removes the instances from Chef and Sensu monitoring. Optionally
sends messages to Hipchat or Slack when actions occur.
and removes the instances from Chef. It can optionally remove the node from Sensu Monitoring (defaults true), fire off webhooks, and send notifications via Slack when actions occur.

![aws-cleaner](https://raw.github.com/eheydrick/aws-cleaner/master/aws-cleaner.png)

### Prerequisites

You will need to create a CloudWatch Events rule that's configured to send termination event messages to SQS.

1. Create an SQS Queue for cloudwatch-events
2. Goto CloudWatch Events in the AWS Console
3. Click *Create rule*
4. Select event source of *EC2 instance state change notification*
5. Select specific state of *Terminated*
6. Add a target of *SQS Queue* and set queue to the cloudwatch-events queue created in step one
7. Give the rule a name/description and click *Create rule*
1. Goto CloudWatch Events in the AWS Console
1. Click *Create rule*
1. Select event source of *EC2 instance state change notification*
1. Select specific state of *Terminated*
1. Add a target of *SQS Queue* and set queue to the cloudwatch-events queue created in step one
1. Give the rule a name/description and click *Create rule*

You will also need to create a user with the required permissions. I recommend creating a 'aws-cleaner' user in chef and add it to its own group. The minimum permissions we found that works is read and delete nodes/clients.

Steps:

1. on chef server: `chef-server-ctl user-create aws-cleaner AWS Cleaner`
1. on chef server: `address@domain.tld "$SOMEREALLYLONGRANDOMPASSWORD" -f aws-cleaner.pem`
1. on chef server: `chef-server-ctl org-user-add $MYORG aws-cleaner`
1. on workstation: `gem install knife-acl`
1. on workstation: `knife group create aws-cleaner`
1. on workstation: `knife group add user aws-cleaner aws-cleaner`
1. on workstation: `knife acl bulk add group aws-cleaner clients '.*' read,delete -y`
1. on workstation: `knife acl bulk add group aws-cleaner nodes '.*' read,delete -y`

An astute reader might notice that this wont work for new nodes that come up as they have not had their ACL updated. I recommend that you add the who bulk acl knife commands (modified for just self as opposed to bulk) as part of your normal bootstrap process before deleting your validation key.

### Installation

@@ -34,7 +50,8 @@ Options:
Copy the example config file ``config.yml.sample`` to ``config.yml``
and fill in the configuration details. You will need AWS Credentials
and are strongly encouraged to use an IAM user with access limited to
the AWS CloudWatch Events SQS queue.
the AWS CloudWatch Events SQS queue.You will need to specify the region
in the config even if you are using IAM Credentials.

The app takes one arg '-c' that points at the config file. If -c is
omitted it will look for the config file in the current directory.
@@ -43,6 +60,11 @@ The app is started by running aws_config.rb and it will run until
terminated. A production install would start it with upstart or
similar.

### Logging

By default aws-cleaner will log to STDOUT. If you wish to log to a specific file
add a `log` section to the config. See [`config.yml.sample`](config.yml.sample) for an example.

### Webhooks

AWS Cleaner can optionally make an HTTP request to a specified endpoint. You can
@@ -59,13 +81,29 @@ To enable webhooks, add a `:webhooks:` section to the config:
:template_variables:
:variable: 'fqdn'
:method: 'get_chef_fqdn' (or 'get_chef_node_name')
:argument: '@instance_id'
```

Chat notifications can be sent when the webhook successfully executes. See
config.yml.sample for an example of the config.

### Limitations
### Sensu

Currently only supports a single AWS region.
By default aws-cleaner assumes that removing from sensu is desired as this was one of its core intentions. To allow people to leverage this without sensu you can disable it via config:
```
:sensu:
:enable: false
```

When wanting to use sensu you will want the following config:
```
:sensu:
:url: 'http://sensu.example.com:4567'
:enable: true
```

While we currently assume sensu removal being desired is considered the default it may not always be so you should set `enable` to true to avoid a breaking change later.

### Limitations

- Currently only supports a single AWS region.
- Only support chef and sensu with non self signed certificates. Look at Aws Certificate Manager or Let's Encrypt for free SSL certificates.
12 changes: 12 additions & 0 deletions Rakefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
require 'bundler/gem_tasks'
require 'rubocop/rake_task'
require 'rspec/core/rake_task'

desc 'Run Rubocop'
RuboCop::RakeTask.new(:rubocop) do |task|
task.patterns = ['bin/*.rb', 'lib/**/*.rb']
end

RSpec::Core::RakeTask.new(:spec)

task default: [:rubocop, :spec]
Loading