Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GEP-1762: In Cluster Gateway Deployments #1757

Merged
merged 6 commits into from
Sep 27, 2023
+248 −0
Merged

GEP-1762: In Cluster Gateway Deployments #1757

Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
138222d
GEP-1762: In Cluster Gateway Deployments
howardjohn Mar 22, 2023
f9ddd75
Address Nick's comments
howardjohn Aug 25, 2023
5cd70cb
Drop attachTo
howardjohn Sep 15, 2023
4c2ae88
Address Rob's comments
howardjohn Sep 15, 2023
1c01615
Remove GC and clarify address
howardjohn Sep 18, 2023
c6a7dcb
Address comments
howardjohn Sep 18, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
245 changes: 245 additions & 0 deletions geps/gep-1762.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,245 @@
# GEP-1762: In Cluster Gateway Deployments

* Status: Provisional

## Overview

Gateway API provides a common abstraction over different implementations, whether they are implemented by cloud load balancers, in-cluster deployments, or other mechanisms. However, many in-cluster implementations have solved some of the same problems in different ways.

Related discussions:

* [Support cluster-local Gateways](https://github.com/kubernetes-sigs/gateway-api/discussions/1247)
* [Scaling Gateway Resources](https://github.com/kubernetes-sigs/gateway-api/discussions/1355)
* [Manual deployments](https://github.com/kubernetes-sigs/gateway-api/issues/1687)
* [Merging Gateways](https://github.com/kubernetes-sigs/gateway-api/pull/1863/)
* [Per-Gateway Infrastructure](https://github.com/kubernetes-sigs/gateway-api/pull/1757)

## Goals

* Provide prescriptive guidance for how in-cluster implementations should behave.
* Provide requirements (tested by conformance) for how in-cluster implementations should behave.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like most if not all of this GEP would be pretty difficult to write conformance tests for, do you have any ideas for how this would work?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is a good point. Not really... any ideas?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is unfortunately going to be a feature that doesn't/can't have conformance tests.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* Provide requirements (tested by conformance) for how in-cluster implementations should behave.
* Provide requirements for how in-cluster implementations should behave.


Note that some changes will be suggestions, while others will be requirements.

## Non-Goals

* Provide guidance to how out-of-cluster implementations should behave. Rather, this document aims to bring consistency between these types.

## Terminology

This document uses a few terms throughout. To ensure consistency, they are defined below:

* In-cluster deployment: refers to an implementation that actuates a `Gateway` by running a data plane in the cluster.
This is *often*, but not necessarily, by deploying a `Deployment`/`DaemonSet` and `Service`.
* Automated deployment: refers to an implementation that automatically deploys the data plane based on a `Gateway`.
That is, the user simply creates a `Gateway` resource and the rest is handled behind the scenes by the implementation.

## Design

This GEP both introduces new API fields, and standardizes how implementations should behave when implementing the existing API.

### Automated Deployments

A simple `Gateway`, as is configured below is assumed to be an automated deployment:

```yaml
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
name: my-gateway
spec:
gatewayClassName: example
listeners:
- name: default
port: 80
protocol: HTTP
```

With this configuration, an implementation:

* MUST mark the Gateway as `Programmed` and provide an address in `Status.Addresses` where the Gateway can be reached on each configured port.
* MUST label all generated resources (Service, Deployment, etc) with `gateway.networking.k8s.io/gateway-name: my-gateway` (where `my-gateway` is the name of the Gateway resource).
* MUST provision generated resources in the same namespace as the Gateway if they are namespace scoped resources.
* Cluster scoped resources are not recommended.
* SHOULD name all generated resources `my-gateway-example` (`<NAME>-<GATEWAY CLASS>`).
This is not simply `NAME` to reduce the chance of conflicts with existing resources.
Where required, this can also serve as the prefix for the object.

### Customizations
robscott marked this conversation as resolved.
Show resolved Hide resolved

With any in-cluster deployment, customization requirements will arise.

Some common requirements would be:

* `Service.spec.type`, to control whether a service is a `ClusterIP` or `LoadBalancer`.
* IP in the Service to assign to it.
* Arbitrary labels and annotations on generated resources.
* Any other arbitrary fields; the list is unbounded. Some examples would be:
* CPU and memory requests
* Service `externalTrafficPolicy`
* Affinity rules.

This GEP currently only aims to solve a subset of these concerns. Additional concerns may be addressed in future revisions or other GEPs.

#### Gateway Type

This is handled by [GEP-1651](https://github.com/kubernetes-sigs/gateway-api/pull/1653), so won't be described here.

#### Gateway IP

This section just clarifies and existing part of the spec, how to handle `.spec.addresses` for in-cluster implementations.
Like all other Gateway types, this should impact the address the `Gateway` is reachable at.

For implementations using a `Service`, this means the `clusterIP` or `loadBalancerIP` (depending on the `Service` type).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ExternalIP may also be valid here for NodePort case

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Don't want to block this merging but would be great to cover this in a follow up.


For example:

```yaml
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
name: my-gateway
spec:
addresses:
- type: IPAddress
value: 1.1.1.1
gatewayClassName: example
listeners:
- name: default
port: 80
protocol: HTTP
```

This would generate a `Service` with `clusterIP` or `loadBalancerIP`, depending on the Service type.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't really tell if this is trying to tell implementations what they should do, or just provide an example of how some implementations handle this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The intent here got muddled through the revisions. Its meant to be "If they provide an IP, you better use it" and whether you use it as the clusterIP vs the loadBalancerIP is based on the type of service, which was determined by routability previously

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is already covered reasonably well in the spec:

gateway-api/apis/v1beta1/gateway_types.go

Lines 134 to 163 in cd223ea

// Addresses requested for this Gateway. This is optional and behavior can
// depend on the implementation. If a value is set in the spec and the
// requested address is invalid or unavailable, the implementation MUST
// indicate this in the associated entry in GatewayStatus.Addresses.
//
// The Addresses field represents a request for the address(es) on the
// "outside of the Gateway", that traffic bound for this Gateway will use.
// This could be the IP address or hostname of an external load balancer or
// other networking infrastructure, or some other address that traffic will
// be sent to.
//
// The .listener.hostname field is used to route traffic that has already
// arrived at the Gateway to the correct in-cluster destination.
//
// If no Addresses are specified, the implementation MAY schedule the
// Gateway in an implementation-specific manner, assigning an appropriate
// set of Addresses.
//
// The implementation MUST bind all Listeners to every GatewayAddress that
// it assigns to the Gateway and add a corresponding entry in
// GatewayStatus.Addresses.
//
// Support: Extended
//
// +optional
// <gateway:validateIPAddress>
// +kubebuilder:validation:MaxItems=16
// +kubebuilder:validation:XValidation:message="IPAddress values must be unique",rule="self.all(a1, a1.type == 'IPAddress' ? self.exists_one(a2, a2.type == a1.type && a2.value == a1.value) : true )"
// +kubebuilder:validation:XValidation:message="Hostname values must be unique",rule="self.all(a1, a1.type == 'Hostname' ? self.exists_one(a2, a2.type == a1.type && a2.value == a1.value) : true )"
Addresses []GatewayAddress `json:"addresses,omitempty"`

It may be worth describing how in-cluster implementations can accomplish this in this GEP, but I don't think we need to add any new requirements here. Just trying to avoid introducing too many unique sources of requirements in case they need to change in the future.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made it more clear, I think. LMK what you think



#### Labels and Annotations

Labels and annotations for generated resources are specified in `infrastructure`:

```yaml
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
name: my-gateway
spec:
infrastructure:
labels:
foo: bar
annotations:
name: my-annotation
```

These are both `map[string]string` types, just like in `ObjectMeta`.

Any labels or annotations here are added to all generated resources.
Note this may mean an annotation intended for a `Service` may end up on a `Deployment` (for example).
This is typically not a concern; however, if an implementation is aware of specific meanings of certain labels or annotations, they MAY
exclude these from irrelevant resources.

This is intended to clearly identify resources associatd with a specific application, environment, or Gateway.
howardjohn marked this conversation as resolved.
Show resolved Hide resolved
Additionally, it can be used support integration with the kitchen-sync of Kubernetes extensions which rely on labels and annotations.

Validation will be added to prevent any usage with `gateway.networking.k8s.io/` prefix, to avoid conflicts with `gateway.networking.k8s.io/gateway-name` or other future additions.

#### Arbitrary Customization

GEP-1867 introduces a new `infrastructure` field, which allows customization of some common configurations (version, size, etc)
and allows a per-Gateway generic `parametersRef`.
This can be utilized for the remainder of customizations.

### Resource Attachment

Resources Generated in response to the `Gateway` will have two attributes:

* A `gateway.networking.k8s.io/gateway-name: <NAME>` label.
howardjohn marked this conversation as resolved.
Show resolved Hide resolved
* A name `<NAME>-<GATEWAY CLASS>`. This format is not strictly required for implementations, but strongly recommended for consistency in attachment.

The generated resources MUST be in the same namespaces as the `Gateway`.

Implementations MAY set `ownerReferences` to the `Gateway` in most cases, as well, but this is not required
as some implementations may have different cleanup mechanisms.


These can be relied on to attach resources to.
While "Policy attachment" in Gateway API would use attachment to the actual `Gateway` resource itself,
howardjohn marked this conversation as resolved.
Show resolved Hide resolved
many existing resources attach only to resources like `Deployment` or `Service`.

An example using these:
```yaml
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
name: gateway
spec:
gatewayClassName: example
listeners:
- name: default
hostname: "example.com"
port: 80
protocol: HTTP
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: gateway
spec:
# Match the generated Deployment by reference
# Note: Do not use `kind: Gateway`.
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: gateway-example
minReplicas: 2
maxReplicas: 5
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 50
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: gateway
spec:
minAvailable: 1
selector:
# Match the generated Deployment by label
matchLabels:
gateway.networking.k8s.io/metadata.name: gateway
```

Note: there is [discussion](https://github.com/kubernetes-sigs/gateway-api/discussions/1355) around a way to attach a HPA to a Gateway directly.
howardjohn marked this conversation as resolved.
Show resolved Hide resolved

## API

This GEP extends the `infrastructure` API introduced in [GEP-1867](https://gateway-api.sigs.k8s.io/geps/gep-1867).

```go
type GatewayInfrastructure struct {
// Labels that should be applied to any resources created in response to this Gateway.
howardjohn marked this conversation as resolved.
Show resolved Hide resolved
//
// For implementations creating other Kubernetes objects, this should be the `metadata.labels` field on resources.
// For other implementations, this refers to any relevant (implementation specific) "labels" concepts.
//
// Support: Extended
// +kubebuilder:validation:MaxItems=16
Labels map[string]string `json:"labels,omitempty"`
// Annotations that should be applied to any resources created in response to this Gateway.
//
// For implementations creating other Kubernetes objects, this should be the `metadata.annotations` field on resources.
// For other implementations, this refers to any relevant (implementation specific) "annotations" concepts.
//
// Support: Extended
// +kubebuilder:validation:MaxItems=16
Annotations map[string]string `json:"annotations,omitempty"`
...
}
```

## Future Work

* Allow various policies, [such as HPA](https://github.com/kubernetes-sigs/gateway-api/discussions/1355), to attach directly to `Gateway` rather than just `Deployment`.