Skip to content

Commit

Permalink
Simplify the format support radically to only "feed_specific"
Browse files Browse the repository at this point in the history 
For now we only need the feed_specific format and in this step only for
the two feeds for which formats had already been defined in
intelmq-mailgen. Making this very simple hopefully will make it easier
to implement the more complex logic we actually need later on.

This solves several defects:

 - The logic that selects the feed specific CSV format now actually
   works (fixes #20)

 - The CSV format for Sinkhole-HTTP-Drone is now the same as that for
   Botnet-Drone-Hadoop. The former was apparently never tested because
   it had a typo in one of the IntelMQ identifiers (destination.id
   instead of destination.ip)

 - The template is chosen by intelmq-mailgen now. With the way the
   contact DB currently works, this is the only way to get the right
   template. The template depends on the feed-name which the contact DB
   does not know anything about at the moment.

   This will have to change in some way, but for now we get something
   that should actually work.

   How the template name is constructed is described in README.md

Other consequences:

 - There are no fallback formats anymore. It's not clear how to handle
   cases where no message can be generated because the format is
   unknown. We need a better way than what we get with this commit
   (mailgen produces an exception and the notification remains in the
   database as unsent) but simply sending a very generic mail as
   previously is probably not the right solution either. That solution
   was an ad-hoc fix/workaround for a defect in earlier versions.

   See #23
  • Loading branch information
@bernhard-herzog
bernhard-herzog committed Jul 6, 2016
1 parent 23b07eb commit a055bd5
Show file tree
Hide file tree
Showing 2 changed files with 57 additions and 183 deletions.
15 changes: 11 additions & 4 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -106,10 +106,17 @@ The body text may allow some substitutions, depending on the format. For
instance, the CSV based formats replace `${events_as_csv}` with the CSV
formatted event data.

Specific Templates
------------------
A template which will be elaborated for more specific templates is called
`specific.txt` this template has to exist in the template directory.

Feed-Specific Templates
-----------------------

For the format `feed_specific`, `intelmq-mailgen` ignores the template
that may have been set by the contact-DB bot and chooses a template
based on the feed-name, but only for those feeds it supports. The
template name is of the form `template-FEEDNAME.txt` where `FEEDNAME` is
replaced by the event's 'feed.name' attribute. The template is looked up
in the template directory as defined above.



Security considerations
Expand Down
225 changes: 46 additions & 179 deletions intelmq-mailgen
View file
Original file line number Diff line number Diff line change
Expand Up @@ -232,119 +232,6 @@ def create_mail(sender, recipient, subject, body, attachments):
return msg


malware_csv_columns = [
"source.asn",
"source.ip",
"time.source",
"malware.name",
"source.port",
"destination.ip",
"destination.port",
"protocol.transport",
"destination.fqdn"
]

# A dictionary defining the ids of the columns which
# have to be used for this classification type and
# their Names
botnet_drone_csv_columns = [
("source.asn", "ASN"),
("source.ip", "IP"),
("time.source", "Timestamp (UTC)"),
("classification.identifier", "Malware"),
("source.port", "Source Port"),
("destination.ip", "Target IP"),
("destination.port", "Target Port"),
("protocol.transport", "Protocol"),
("destination.fqdn", "Target Hostname"),
]

http_sinkhole_columns = [
('time.source', 'Timestamp (UTC)'),
('source.ip', 'IP'),
('source.port', 'Source Port'),
('source.asn', 'Source ASN'),
('source.geolocation.cc', 'Source Country Code'),
('malware.name', 'Malware Name'),
('source.tor_node', 'Tor Node'),
('source.reverse_dns', 'Hostname'),
('destination.port', 'Destination Port'),
('destination.id', 'Destination IP'),
('destination.asn', 'Destination ASN'),
('destination.geolocation.cc', 'Destination Country Code'),
('classification.type', 'Classification Type'),
]

most_csv_columns = [
"classification.identifier",
"classification.taxonomy",
"classification.type",
"comment",
"destination.abuse_contact",
"destination.account",
"destination.allocated",
"destination.as_name",
"destination.asn",
"destination.fqdn",
"destination.geolocation.cc",
"destination.geolocation.city",
"destination.geolocation.country",
"destination.geolocation.latitude",
"destination.geolocation.longitude",
"destination.geolocation.region",
"destination.geolocation.state",
"destination.ip",
"destination.local_hostname",
"destination.local_ip",
"destination.network",
"destination.port",
"destination.registry",
"destination.reverse_dns",
"destination.tor_node",
"destination.url",
"event_description.target",
"event_description.text",
"event_description.url",
"event_hash",
"malware.hash",
"malware.hash.md5",
"malware.hash.sha1",
"malware.name",
"malware.version",
"misp_uuid",
"protocol.application",
"protocol.transport",
"rtir_id",
"screenshot_url",
"source.abuse_contact",
"source.account",
"source.allocated",
"source.as_name",
"source.asn",
"source.fqdn",
"source.geolocation.cc",
"source.geolocation.city",
"source.geolocation.country",
"source.geolocation.cymru_cc",
"source.geolocation.geoip_cc",
"source.geolocation.latitude",
"source.geolocation.longitude",
"source.geolocation.region",
"source.geolocation.state",
"source.ip",
"source.local_hostname",
"source.local_ip",
"source.network",
"source.port",
"source.registry",
"source.reverse_dns",
"source.tor_node",
"source.url",
"status",
"time.source"
]


def clearsign(gpgme_ctx, text):
plaintext = io.BytesIO(text.encode())
signature = io.BytesIO()
Expand Down Expand Up @@ -426,62 +313,52 @@ def mail_format_malware_as_csv(cur, notification, config, gpgme_ctx):


def mail_format_feed_specific_as_csv(cur, notification, config, gpgme_ctx):
"""Creates one email with csv attachment for botnet_drone
:returns: list of email objects
:rtype: list
"""
notification["template"] = "specific.txt"

return mail_format_as_csv(cur, notification,
config, gpgme_ctx, botnet_drone_csv_columns)


def mail_format_botnet_drone_as_csv(cur, notification, config, gpgme_ctx):
"""Creates one email with csv attachment for botnet_drone
:returns: list of email objects
:rtype: list
"""
return mail_format_as_csv(cur, notification,
config, gpgme_ctx, botnet_drone_csv_columns)


def mail_format_http_sinkhole_as_csv(cur, notification, config, gpgme_ctx):
"""Creates one email with csv attachment for botnet_drone
:returns: list of email objects
:rtype: list
"""
return mail_format_as_csv(cur, notification,
config, gpgme_ctx, http_sinkhole_columns)

"""Creates one email with csv attachment based on feed-name.
def mail_format_GENERIC_as_csv(cur, notification, config, gpgme_ctx):
"""Creates one email with csv attachment for malware.
This function assumes that notification["feed_name"] is actually one
of the feed names used as key in feed_specific_formats.
:returns: list of email objects
:rtype: list
"""
return mail_format_as_csv(cur, notification,
config, gpgme_ctx, most_csv_columns)


known_formatters = {
"feed_specific":
{
("csv", "Botnet-Drone-Hadoop"): mail_format_botnet_drone_as_csv,
("csv", "Sinkhole-HTTP-Drone"): mail_format_http_sinkhole_as_csv,
("csv", "DEFAULT"): mail_format_feed_specific_as_csv,
},
'generic':
{
("csv", "botnet drone"): mail_format_botnet_drone_as_csv,
("csv", "GENERIC"): mail_format_GENERIC_as_csv,
}
feed_name = notification["feed_name"]
columns = feed_specific_formats.get(feed_name)
if columns is None:
# TODO
raise RuntimeError
notification["template"] = "template-" + feed_name + ".txt"

return mail_format_as_csv(cur, notification, config, gpgme_ctx, columns)


# Map feed names to CSV column specifications
feed_specific_formats = {
"Botnet-Drone-Hadoop": [
("source.asn", "ASN"),
("source.ip", "IP"),
("time.source", "Timestamp (UTC)"),
("classification.identifier", "Malware"),
("source.port", "Source Port"),
("destination.ip", "Target IP"),
("destination.port", "Target Port"),
("protocol.transport", "Protocol"),
("destination.fqdn", "Target Hostname"),
],
"Sinkhole-HTTP-Drone": [
("source.asn", "ASN"),
("source.ip", "IP"),
("time.source", "Timestamp (UTC)"),
("classification.identifier", "Malware"),
("source.port", "Source Port"),
("destination.ip", "Target IP"),
("destination.port", "Target Port"),
("protocol.transport", "Protocol"),
("destination.fqdn", "Target Hostname"),
],
}



def create_mails(cur, notification, config, gpgme_ctx):
"""Create one or several email objects for a notification.
Expand All @@ -498,28 +375,18 @@ def create_mails(cur, notification, config, gpgme_ctx):
"""

emails = []
formatter = known_formatters.get((notification["format"],
notification["classification_type"]))
if formatter is None:
# Try Fallback on "GENERIC" type, that is use a standard set
# of fields to export.

if notification["format"] == "feed_specific":
formatter = known_formatters['feed_specific'].get(('csv',
notification["feed_name"]))
if formatter is not None:
formatter = known_formatters['feed_specific'].get(('csv',
'DEFAULT'))

else:
formatter = known_formatters['generic'].get((notification["format"],
"GENERIC"))
formatter = None

if (notification["format"] == "feed_specific"
and notification["feed_name"] in feed_specific_formats):
formatter = mail_format_feed_specific_as_csv

if formatter is not None:
emails = formatter(cur, notification, config, gpgme_ctx)
else:
msg = ("Cannot generate emails for combination (%r, %r)" %
(notification["format"], notification["classification_type"]))
msg = ("Cannot generate emails for combination (%r, %r)"
% (notification["format"], notification["feed_name"]))
print(msg, file=sys.stderr)
raise NotImplementedError(msg)

Expand Down Expand Up @@ -623,7 +490,7 @@ def generate_notifications_interactively(config, cur, notifications):
i["email"],
i["template"],
i["format"],
i["classification_type"],
i["feed_name"],
len(i["event_ids"]))
)
valid_answers = ("c", "s", "a", "q")
Expand Down

0 comments on commit a055bd5

Please sign in to comment.