Merge pull request #414 from Icinga:fix/rewrite_service_check_daemon

Fix: Rewrite Icinga for Windows service check daemon

Rewrite Icinga for Windows service check daemon for collecting metrics over time, to improve performance, decrease required resources and fix memory leaks.

doc/100-General/01-Upgrading.md

@@ -20,6 +20,24 @@ After upgrading to Icinga for Windows v1.8.0, you will require to open a new Ici
 
 **NOTE:** In some cases the changes for the EventLog will only apply, **after** the system has been rebooted. Afterwards every Icinga for Windows EventLog entry is written in a newly created `Icinga for Windows` log.
 
+### Custom Daemon Handling
+
+With Icinga for Windows v1.8.0 we removed the entire list of currently available `$Global` variables:
+
+* `$Global:IcingaThreads`
+* `$Global:IcingaThreadContent`
+* `$Global:IcingaThreadPool`
+* `$Global:IcingaTimers`
+* `$Global:IcingaDaemonData`
+
+All of these have been centralized inside one, new variable called `$Global:Icinga`. You can read more about the structure of this `hashtable` object on the [Developer Guide](../900-Developer-Guide/00-General.md/#Data-Management).
+
+The important change is, that in case you created custom daemons or API endpoints using on of the above globals, you will have to migrate your code to properly make use of `$Global:Icinga`, otherwise your daemons will not work anymore once you upgrade to Icinga for Windows v1.8.0.
+
+The benefit of this change is that you no longer require to take care of synchronising global data between newly created threads, as Icinga for Windows will make the public part of `$Global:Icinga.Public` shared for every single instance automatically.
+
+Please [contact us](https://icinga.com/company/contact/) in case you require assistance with migrating your current code to Icinga for Windows v1.8.0.
+
 ## Upgrading to v1.7.0 (2021-11-09)
 
 ### REST-Api and Api-Checks

doc/100-General/10-Changelog.md

@@ -23,6 +23,7 @@ Released closed milestones can be found on [GitHub](https://github.com/Icinga/ic
 * [#409](https://github.com/Icinga/icinga-powershell-framework/issues/409) Fixes URL builder for `Sync-IcingaRepository` which will now properly test the JSON file and try a secondary fallback by pointing to the `ifw.repo.json` in case a URL is returning the directory listing instead
 * [#411](https://github.com/Icinga/icinga-powershell-framework/pull/411) Fixes Icinga Director error message output because of missing `[string]::Format()`
 * [#412](https://github.com/Icinga/icinga-powershell-framework/issues/412) Fixes possible defective state of the Icinga Agent by using a custom service user for JEA profiles which is larger than 20 digits
+* [#414](https://github.com/Icinga/icinga-powershell-framework/pull/414) Fixes Service Check Daemon with a total rewrite to improve performance, decrease required resources and fix memory leaks
 * [#418](https://github.com/Icinga/icinga-powershell-framework/pull/418) Fixes crash on wrong variable usage introduced by [#411](https://github.com/Icinga/icinga-powershell-framework/pull/411)
 * [#421](https://github.com/Icinga/icinga-powershell-framework/issues/421) Fixes experimental state of `API Check` feature by removing that term and removing the requirement to install `icinga-powershell-restapi` and `icinga-powershell-apichecks`
 * [#436](https://github.com/Icinga/icinga-powershell-framework/pull/436) Fixes a lookup error for existing plugin documentation files, which caused files not being generated properly in case a similar name was already present on the system

doc/900-Developer-Guide/00-General.md

@@ -45,6 +45,81 @@ In this case, our `NestedModules` variable within our `.psd1` file requires the
     )
 ```
 
+### Data Management
+
+Icinga for Windows is using one global variable `$Global:Icinga`, to store information for daemons and other tasks. This variable is split into three different categories, which you can read more on below. The general architecture of this construct is a simple `hashtable`.
+You can interact with this variable and sub-entries like you would with normal `hashtables`, making data stored a lot easier to access and maintain.
+
+#### Private
+
+Everything which should be stored while a daemon is running internally or within a PowerShell session and **not** being shared with other daemons, is stored within the `$Global:Icinga.Private` space.
+
+The following entries are set by default within the `Private` space:
+
+| Category  | Description |
+| ---       | ---         |
+| Timers    | All created timers by using `Start-IcingaTimer` are stored under this environment variable |
+| Scheduler | Once plugins are executed, performance data, check results and exit codes are stored in this section, in case the PowerShell instance is set to run as daemon |
+| Daemons   | This is a place where all daemon data should be added and stored, separated by a namespace for each module as entry. This data is **not** shared between other daemons |
+
+#### Example Data
+
+```powershell
+$Global:Icinga.Private.Timers.DefaultTimer
+```
+
+```powershell
+$Global:Icinga.Private.Scheduler.CheckResults
+```
+
+```powershell
+$Global:Icinga.Private.Daemons.ServiceCheck.PerformanceCache
+```
+
+#### Public
+
+Everything stored within the `Public` space of `$Global:Icinga` is automatically shared between all threads of the current PowerShell instance. If you run the `ServiceCheckDaemon` in addition with the `RestAPI` for example, metrics over time will be read from the public shared space from the `RestApi` and used during check execution.
+
+There is no manual configuration required to share the information, as Icinga for Windows will deal with this for you, once a new thread instance is created.
+
+The following entries are set by default within the `Public` space:
+
+| Category           | Description |
+| ---                | ---         |
+| ThreadPools        | A list of all thread pools available to create new thread limits for certain background daemons |
+| Daemons            | A place to store shared information for each single daemon within a namespace, making data accessible to other threads |
+| Threads            | A list of all started and available threads running by Icinga for Windows |
+| PerformanceCounter | A space to share all PerformanceCounter information between threads, which counters are already created for internal usage |
+
+##### Example Data
+
+```powershell
+$Global:Icinga.Public.ThreadPools.MainPool
+```
+
+```powershell
+$Global:Icinga.Public.Daemons.RESTApi.ClientBlacklist
+```
+
+```powershell
+$Global:Icinga.Public.Threads.'Start-IcingaForWindowsDaemon::Add-IcingaForWindowsDaemon::Main::0'
+```
+
+#### Protected
+
+This is a section reserved for Icinga for Windows and Icinga developers in general. This space will store general information for Icinga for Windows, determining on how the PowerShell instance is handling internal requests and procedures.
+
+As custom module developer, you can **read** from this space but are in genetal **not** allowed to store information there. Please use the `Private` and `Public` space for this.
+
+The following entries are set by default within the `Protected` space:
+
+| Category    | Description |
+| ---         | ---         |
+| JEAContext  | Tells Icinga for Windows that the current environment is running within a JEA context |
+| RunAsDaemon | Tells Icinga for Windows that the current PowerShell instance is running as daemon, changing behaviors on error and plugin execution handling |
+| DebugMode   | Enables the debug mode of Icinga for Windows, printing additional details during operations or tasks |
+| Minimal     | Changes certain behavior regarding check execution and internal error handling |
+
 ## Using Icinga for Windows Dev Tools
 
 Maintaining the entire structure above seems to be complicated at the beginning, especially when considering to update the `NestedModules` section whenever you make changes. To mitigate this, Icinga for Windows provides a bunch of Cmdlets to help with the process

doc/900-Developer-Guide/10-Custom-Daemons.md

@@ -94,18 +94,13 @@ function Add-IcingaAgentServiceTest()
 }
 ```
 
-Depending on our daemon, later usage and possible sharing of data between all loaded daemons might be required. In addition we might want to spawn child threads as single tasks being executed. To do so, we will parse the frameworks `global` data to the thread.
+Depending on our daemon, later usage and possible sharing of data between all loaded daemons might be required. In addition we might want to spawn child threads as single tasks being executed. To access the global, shared data, use can use the `Global:Icinga.Public` variable. This content is shared automatically between each single created thread.
 
 Our recommendation is to always do this for every daemon, as later changes might be more complicated and time consuming.
 
 ```powershell
 function Add-IcingaAgentServiceTest()
 {
-    # Allow us to parse the framework global data to this thread
-    param (
-        $IcingaDaemonData
-    );
-
     # Everything which will be executed inside the thread
     # belongs here
 }
@@ -116,11 +111,6 @@ Now as the basic part is finished, we will require to make our framework librari
 ```powershell
 function Add-IcingaAgentServiceTest()
 {
-    # Allow us to parse the framework global data to this thread
-    param (
-        $IcingaDaemonData
-    );
-
     # Import the framework library components and initialise it
     # as daemon
     Use-Icinga -LibOnly -Daemon;
@@ -132,29 +122,19 @@ As we will parse the `global` framework data anyways, we should already make use
 ```powershell
 function Add-IcingaAgentServiceTest()
 {
-    # Allow us to parse the framework global data to this thread
-    param (
-        $IcingaDaemonData
-    );
-
     # Import the framework library components and initialise it
     # as daemon
     Use-Icinga -LibOnly -Daemon;
 
-    # Add a synchronized hashtable to the global data background
+    # Add a hashtable to the public data background
     # daemon hashtable to write data to. In addition it will
     # allow to share data collected from this daemon with others
-    $IcingaDaemonData.BackgroundDaemon.Add(
-        'TestIcingaAgentService',
-        [hashtable]::Synchronized(@{ })
-    );
+    $Global:Icinga.Public.Daemons.Add('TestIcingaAgentService', @{ });
+
     # This will add another hashtable to our previous
     # TestIcingaAgentService hashtable to store actual service
     # information
-    $IcingaDaemonData.BackgroundDaemon.TestIcingaAgentService.Add(
-        'ServiceState',
-        [hashtable]::Synchronized(@{ })
-    );
+    $Global:Icinga.Public.Daemons.TestIcingaAgentService.Add('ServiceState', @{ });
 }
 ```
 
@@ -165,29 +145,19 @@ Because the code is executed as separate thread, we will have to ensure it will
 ```powershell
 function Add-IcingaAgentServiceTest()
 {
-    # Allow us to parse the framework global data to this thread
-    param (
-        $IcingaDaemonData
-    );
-
     # Import the framework library components and initialise it
     # as daemon
     Use-Icinga -LibOnly -Daemon;
 
-    # Add a synchronized hashtable to the global data background
+    # Add a hashtable to the public data background
     # daemon hashtable to write data to. In addition it will
     # allow to share data collected from this daemon with others
-    $IcingaDaemonData.BackgroundDaemon.Add(
-        'TestIcingaAgentService',
-        [hashtable]::Synchronized(@{ })
-    );
+    $Global:Icinga.Public.Daemons.Add('TestIcingaAgentService', @{ });
+
     # This will add another hashtable to our previous
     # TestIcingaAgentService hashtable to store actual service
     # information
-    $IcingaDaemonData.BackgroundDaemon.TestIcingaAgentService.Add(
-        'ServiceState',
-        [hashtable]::Synchronized(@{ })
-    );
+    $Global:Icinga.Public.Daemons.TestIcingaAgentService.Add('ServiceState', @{ });
 
     # Keep our code executed as long as the PowerShell service is
     # being executed. This is required to ensure we will execute
@@ -197,34 +167,24 @@ function Add-IcingaAgentServiceTest()
 }
 ```
 
-*ALWAYS* ensure you add some sort for `sleep` at the end of the `while` loop to allow your CPU some breaks. If you do not do this, you might suffer from high CPU loads. The `sleep duration` interval can depend either on a simple CPU cycle break or by telling the daemon to execute tasks only in certain interval. In our case we wish to execute the daemon every `5 seconds`.
+*ALWAYS* ensure you add some sort for `sleep` at the end of the `while` loop to allow your CPU some breaks. If you do not do this, you might suffer from high CPU loads. The `sleep duration` interval can depend either on a simple CPU cycle break or by telling the daemon to execute tasks only in certain Intervalls. In our case we wish to execute the daemon every `5 seconds`.
 
 ```powershell
 function Add-IcingaAgentServiceTest()
 {
-    # Allow us to parse the framework global data to this thread
-    param (
-        $IcingaDaemonData
-    );
-
     # Import the framework library components and initialise it
     # as daemon
     Use-Icinga -LibOnly -Daemon;
 
-    # Add a synchronized hashtable to the global data background
+    # Add a hashtable to the public data background
     # daemon hashtable to write data to. In addition it will
     # allow to share data collected from this daemon with others
-    $IcingaDaemonData.BackgroundDaemon.Add(
-        'TestIcingaAgentService',
-        [hashtable]::Synchronized(@{ })
-    );
+    $Global:Icinga.Public.Daemons.Add('TestIcingaAgentService', @{ });
+
     # This will add another hashtable to our previous
     # TestIcingaAgentService hashtable to store actual service
     # information
-    $IcingaDaemonData.BackgroundDaemon.TestIcingaAgentService.Add(
-        'ServiceState',
-        [hashtable]::Synchronized(@{ })
-    );
+    $Global:Icinga.Public.Daemons.TestIcingaAgentService.Add('ServiceState', @{ });
 
     # Keep our code executed as long as the PowerShell service is
     # being executed. This is required to ensure we will execute
@@ -243,29 +203,19 @@ This is basically the foundation of every single daemon you will write. Now we w
 ```powershell
 function Add-IcingaAgentServiceTest()
 {
-    # Allow us to parse the framework global data to this thread
-    param (
-        $IcingaDaemonData
-    );
-
     # Import the framework library components and initialise it
     # as daemon
     Use-Icinga -LibOnly -Daemon;
 
-    # Add a synchronized hashtable to the global data background
+    # Add a hashtable to the public data background
     # daemon hashtable to write data to. In addition it will
     # allow to share data collected from this daemon with others
-    $IcingaDaemonData.BackgroundDaemon.Add(
-        'TestIcingaAgentService',
-        [hashtable]::Synchronized(@{ })
-    );
+    $Global:Icinga.Public.Daemons.Add('TestIcingaAgentService', @{ });
+
     # This will add another hashtable to our previous
     # TestIcingaAgentService hashtable to store actual service
     # information
-    $IcingaDaemonData.BackgroundDaemon.TestIcingaAgentService.Add(
-        'ServiceState',
-        [hashtable]::Synchronized(@{ })
-    );
+    $Global:Icinga.Public.Daemons.TestIcingaAgentService.Add('ServiceState', @{ });
 
     # Initialise our error counter variable
     [int]$RestartErrors = 0;
@@ -283,7 +233,7 @@ function Add-IcingaAgentServiceTest()
         if ($null -ne $ServiceState) {
             # Add the current service state to our hashtable.
             Add-IcingaHashtableItem `
-                -Hashtable $IcingaDaemonData.BackgroundDaemon.TestIcingaAgentService.ServiceState `
+                -Hashtable $Global:Icinga.Public.Daemons.TestIcingaAgentService.ServiceState `
                 -Key 'value' `
                 -Value $ServiceState.Status `
                 -Override | Out-Null;
@@ -295,15 +245,15 @@ function Add-IcingaAgentServiceTest()
                     Restart-Service 'icinga2' -ErrorAction Stop;
 
                     Add-IcingaHashtableItem `
-                        -Hashtable $IcingaDaemonData.BackgroundDaemon.TestIcingaAgentService.ServiceState `
+                        -Hashtable $Global:Icinga.Public.Daemons.TestIcingaAgentService.ServiceState `
                         -Key 'restart_error' `
                         -Value 0 `
                         -Override | Out-Null;
                 } catch {
                     # Add an error counter in case we failed
                     $RestartErrors += 1;
                     Add-IcingaHashtableItem `
-                        -Hashtable $IcingaDaemonData.BackgroundDaemon.TestIcingaAgentService.ServiceState `
+                        -Hashtable $Global:Icinga.Public.Daemons.TestIcingaAgentService.ServiceState `
                         -Key 'restart_error' `
                         -Value $RestartErrors `
                         -Override | Out-Null;
@@ -322,7 +272,7 @@ function Add-IcingaAgentServiceTest()
 
 Once our function is completed we only require to call it once our daemon is registered. Do to so, we will use the Cmdlet `New-IcingaThreadInstance`.
 
-As arguments we will have to add a unique `name` to use for this thread as well as a `thread pool`, on which the function will be added to. In our case we will use the Frameworks default pool. Last but not least we require to parse possible `Arguments` to our function and tell the thread to `Start` right after being created. For the arguments we will parse the frameworks `global` `IcingaDaemonData` we also use inside our function to store data in.
+As arguments we will have to add a unique `name` to use for this thread as well as a `thread pool`, on which the function will be added to. The name of the thread is automatically constructed based on the caller function, the function being executed and the given name. If you create a thread which is used as a `main` thread for several sub-threads, you could call it `Main`. For our example, the constructed thread name will then internally be `Start-IcingaAgentServiceTest::IcingaAgentServiceTest::Main::0`. The last added digit will auto increment, in case you are adding the identical thread multiple times, allowing you to access certain threads again by their index id. For the thread pool, we will use the default thread pool `MainPool` of Icinga for Windows. To customize this, you can use `Add-IcingaThreadPool` with a unique name and the amount if allowed instances. Last but not least we require to parse possible `Arguments` to our function and tell the thread to `Start` right after being created.
 
 This call will be added inside the `Start-IcingaAgentServiceTest` we created earlier and didn't touch so far yet.
 
@@ -333,12 +283,10 @@ function Start-IcingaAgentServiceTest()
     # function as command to call it and parse all our
     # arguments to it
     New-IcingaThreadInstance `
-        -Name 'Icinga_PowerShell_IcingaAgent_StateCheck' `
-        -ThreadPool $global:IcingaDaemonData.IcingaThreadPool.BackgroundPool `
+        -Name 'Main' `
+        -ThreadPool (Get-IcingaThreadPool -Name 'MainPool') `
         -Command 'Add-IcingaAgentServiceTest' `
-        -CmdParameters @{
-            'IcingaDaemonData' = $global:IcingaDaemonData;
-        } `
+        -CmdParameters @{ } `
         -Start;
 ```
 
@@ -368,13 +316,13 @@ Once the service is stopped and your `administrative PowerShell` is open, we wil
 
 ```powershell
 Use-Icinga;
-Start-IcingaPowerShellDaemon;
+Start-IcingaForWindowsDaemon;
 ```
 
 Once done you will receive back your prompt, however all registered background daemons are running. To access the collected data from daemons, you can print the content of the `global` framework data. If you wish to check if your daemon was loaded properly and data is actually written, we can access our created hashtable and get the current service state of it
 
 ```powershell
-$global:IcingaDaemonData.BackgroundDaemon.TestIcingaAgentService.ServiceState['value'];
+$Global:Icinga.Public.Daemons.TestIcingaAgentService.ServiceState['value'];
 ```
 
 In case your Icinga Agent service is installed and your daemon is running properly, this should print the current state of the service.