Skip to content

Latest commit

 

History

History
170 lines (97 loc) · 11.4 KB

manage_eu_logs.md

File metadata and controls

170 lines (97 loc) · 11.4 KB
copyright lastupdated keywords subcollection
years
2018, 2020
2020-06-22
LogDNA, IBM, Log Analysis, logging, overview
Log-Analysis-with-LogDNA

{:new_window: target="_blank"} {:shortdesc: .shortdesc} {:screen: .screen} {:pre: .pre} {:table: .aria-labeledby="caption"} {:codeblock: .codeblock} {:tip: .tip} {:download: .download} {:important: .important} {:note: .note} {:external: target="_blank" .external}

Managing logs for EU-supported resources

{: #manage_eu_logs}

Across every industry, organizations require tighter controls and visibility into where their data is stored and processed in the {{site.data.keyword.cloud}}. To manage logs that are generated in your EU-supported account by using the {{site.data.keyword.la_full_notm}} service, consider the following information: {:shortdesc}

  1. You must provision 1 {{site.data.keyword.la_full_notm}} instance in the EU-DE (Frankfurt) location.

  2. You must enable your account to be EU-supported, so support is handled by team members in the European Union.

  3. To manage logs that are generated by EU-Supported {{site.data.keyword.cloud_notm}} services and Cloud Foundry (CF) resources by using the {{site.data.keyword.la_full_notm}} service, you must provision 1 logging instance in the EU-DE (Frankfurt) location. Then, you must enable this logging instance to collect service platform logs.

    • To manage logs from {{site.data.keyword.cloud_notm}} services, you must provision services in the Frankfurt location.

    • To manage logs from Cloud Foundry (CF), you must provision CF applications in the Frankfurt location.

  4. To manage logs from Kubernetes clusters and VSIs that run in {{site.data.keyword.cloud_notm}} or outside the {{site.data.keyword.cloud_notm}}, you must configure the LogDNA agent to forward logs to instances in Frankfurt.

  5. You must restrict access to users to see and manage logs from logging instances in Frankfurt.

  6. You must ensure that you archive to an EU-Supported {{site.data.keyword.cos_full_notm}} (COS) bucket.

Step 1. Set on the EU-Supported flag in your account

{: #manage_eu_logs_step1}

You must enable your account to be EU-Supported so that logging instances in Frankfurt are supported by EU members.

Consider the following information when you turn on the EU-supported flag in your account:

  • Support is handled by team members in the European Union (EU).
  • In the event that your issue requires non-EU expert assistance, it will be reviewed and approval given prior to any non-EU intervention.
  • You can filter and identify the {{site.data.keyword.cloud_notm}} Catalog services that are EU-Supported.

Step 2. Provision {{site.data.keyword.la_full_notm}} instances in Frankfurt

{: #manage_eu_logs_step2}

You can provision {{site.data.keyword.la_full_notm}} instances in multiple locations. However, only instances that are provisioned in the EU-DE (Frankfurt) location are EU-Supported. {: important}

For more information on how to provision an instance, see Provisioning an instance.

Step 3. Configure 1 instance to manage service platform logs

{: #manage_eu_logs_step3}

You can provision 1 or more instances in the Frankfurt location. However, only 1 of those instances can be configured to collect platform service logs for resources that run in the Frankfurt location. For more information on how to configure an instance to collect platform service logs, see Configuring service logs.

Check out the list of {{site.data.keyword.cloud_notm}} resources that automatically collect logs and forward them to your logging instance. See platform service logs.

Step 4. Configure your LogDNA agents

{: #manage_eu_logs_step4}

When you configure a LogDNA agent for a resource such as a Kubernetes cluster, you specify to which instance that agent connects. You can only connect 1 agent per resource to a LogDNA instance.

If you have agents that collect logs from resources, apps and services that run in the {{site.data.keyword.cloud_notm}} or outside the {{site.data.keyword.cloud_notm}}, and that must be EU-supported, you need to configure your agents to forward data to a logging instance in Frankfurt. Learn more:

Learn more about how to detach an agent, see Disconnecting a LogDNA agent.

Step 5. Restrict user access to view and manage logs

{: #manage_eu_logs_step5}

{{site.data.keyword.iamlong}} (IAM) enables you to securely authenticate users and control access to all cloud resources consistently in the {{site.data.keyword.cloud_notm}}.

Every user that accesses the {{site.data.keyword.la_full_notm}} service in your account must be assigned an access policy with an IAM user role defined. The policy determines what actions the user can perform within the context of the service or instance you select. The allowable actions are customized and defined as operations that are allowed to be performed on the service. The actions are then mapped to IAM user roles. Learn more about the IAM user roles for the {{site.data.keyword.la_full_notm}}.

You might have users across different geographies. However, to comply with EU law, only EU personel can see and access log data from your EU-supported infrastructure, apps, and services. To restrict access to users, you can configure an access group, and define policies that restrict access to those users to the instances that you have configured in Frankfurt.

Grant permissions to users to manage the logging service

{: #manage_eu_logs_step5-1}

To grant administrator permissions to users, complete the following steps:

  1. Create an access group, then add users to it. For example, create an access group named logdna-eu-admins. Learn more.

  2. Assign access to a group by configuring policies.

    For example, you can add a policy to the access group for each instance of the {{site.data.keyword.la_full_notm}} service in Frankfurt. For each policy, select the platform role administrator if you want administrators of the logging service to be able to grant other users permissions to work with logging instances in Frankfurt. If you want to remove permissions to manage users to administrators of the logging service in your account, choose the platform role editor. Select the service role manager.

Grant permissions to users to view logs

{: #manage_eu_logs_step5-2}

To grant viewer permissions to users, complete the following steps:

  1. Create an access group, then add users to it. For example, create an access group named logdna-eu-users. Learn more.

  2. Assign access to a group by configuring policies.

    For example, you can add a policy to the access group for each instance of the {{site.data.keyword.la_full_notm}} service in Frankfurt. For each policy, select the platform role viewer to grant users permissions to view logs. Select the service role reader.

Step 6. Exporting logs

{: #manage_eu_logs_step6}

To make the EU-DE (Frankfurt) location EU-Supported, the web UI export functionality is not available for instances that are provisioned in Frankfurt. In addition, you cannot use the API to export data to an email address.

You can export data to a local file or to a terminal by using the LogDNA export API and a service key. Only administrators can create service keys. Users can view them. Service keys are only used to export data from your instance by using the API.

Notice that users in the account that have permissions to view logs through the LogDNA web UI can export data by using the API if they have an active service key. A user, with a policy in IAM to view logs, can also view service keys that an administrator has created. If you do not want users to be able to export data locally, service keys must be deleted.

Step 7. Archiving logs

{: #manage_eu_logs_step7}

When you archive logs from a LogDNA instance to a COS bucket, consider the following information:

  • When you provision an instance of the COS service, this instance is a global one in your account. It is not region bound.

  • You must configure a bucket that complies with the EU-Supported and GDPR regulations. For the list of COS EU-supported endpoints, see EU-supported endpoints.

    For example, consider the following scenarios:

    • For a bucket with single site resiliency, you can create the bucket in Amsterdam or Milan.

    • For a bucket with regional resiliency, you can create the bucket in the EU-DE location to keep the data in Frankfurt.

    • For a bucket with cross region resiliency, you can create the bucket in the eu-geo location. Data is kept within the EU geography across datacenters that are located in Milan, Amsterdam, and Frankfurt.

  • You must restrict user access to manage archived log files in these buckets.

  • Users are responsible for downloading files to EU-supported locations.

To learn how to configure archiving for your LogDNA instance, see Archiving logs.

Step 8. Querying archived logs with SQL Query service

{: #manage_eu_logs_step8}

{{site.data.keyword.sqlquery_short}} provides a serverless, no-ETL solution to easily query data stored in COS. Learn more.

You can use this service to analyze data from logging archived files in COS.

Once you have SQL Query running on IBM Cloud, you can immediately start querying your data using the SQL Query user interface, programmatically by using either the REST API or the Python ibmcloudsql library, or write a serverless function by using {{site.data.keyword.openwhisk_short}}.

When you query logs, consider the following information:

  • You must provision an instance of the {{site.data.keyword.sqlquery_short}} service in Frankfurt.
  • You must restrict user access to work with that instance. Users need the platform viewer role to launch the UI, and the service writer role to run queries.
  • When you open the UI, the {{site.data.keyword.sqlquery_short}} service automatically generates a unique COS bucket that will store all of the results as CSV files from your SQL queries. To make sure that you are using an EU-Supported bucket, create one. You can specify your custom bucket to store results in.