Separate Polaris-native API from Iceberg Rest Catalog API spec

[HonahX]

Not ready for review

This PR fixes: #553 by separating Polaris-native catalog APIs from Iceberg REST Catalog spec. Dev list discussion: https://lists.apache.org/thread/1fqocs00pno0xfr4ss2p69d6dv5h8qzf

• Move notifications API to polaris-apis/notifications-api.yaml. The new yaml contains necessary paths and components def for the notifications API
• Create polaris-catalog-service.yaml which will be the root spec file that groups IRC Apis from rest-catalog-open-api.yaml and polaris-apis/notifications-api.yaml. This one will be used in the builder to generate open api service.
• Add generated bundle generated/bundled-polaris-catalog-service.yaml for website to render the preview of APIs

There will be 2 follow-ups of this PR

1. Update hugo site's rest-catalog-open-api.md to point to the generated bundle generated/bundled-polaris-catalog-service.yaml
2. Make rest-catalog-open-api.yaml match the released version (currently 1.7.1)

[snazy]

I thought we do not want to have two PRs #906 and #856 for the same thing (see #884 and https://lists.apache.org/thread/bcnh1dwgoxd2dvtqk6z935gfzmh4q0jq)

[HonahX]

@snazy Thanks for bringing this up. Sorry for the confusion, the 2 PR should for the different changes but I forgot to update the title. I will pay attention to this in the future to better follow the guideline

[jackye1995]

do we need to also commit the generated file? If possible I think we should have that in .gitignore

[HonahX]

Yeah, it is a weird but it is necessary for Polaris's hugo site to render preview.
https://github.com/apache/polaris/blob/main/site/content/in-dev/unreleased/rest-catalog-open-api.md?plain=1#L27

polaris/site/layouts/shortcodes/redoc-polaris.html

Line 39 in 569bf02

{{- $url = printf "https://raw.githubusercontent.com/apache/polaris/refs/heads/main/spec/%s" $spec }}

The site need a url of the yaml to render the preview: https://polaris.apache.org/in-dev/unreleased/rest-catalog-open-api/, so we have to push it to the github

We can remove the generated one from github if our site can render a local yaml, but that to my current understanding will require non-trivial change to our site so we may explore later.

[jackye1995]

should this also be moved to polaris-apis/polaris-management-api.yaml?

[HonahX]

I’m thinking of renaming the folder to polaris-catalog-apis to better reflect its purpose. I'd like to keep polaris-management-api.yaml in the root of spec, ensuring that every YAML file at the root is a complete OpenAPI spec, while those in subfolders serve as components or parts of specific endpoint groups. WDYT?

[jackye1995]

we might want to rename this as iceberg-rest-catalog-open-api.yaml just to be clear.

[HonahX]

Good point! I plan to do the rename in a follow-up PR because

1. The current rest-catalog-open-api.yaml is referenced in our hugo site setting
2. We need to extract additional stuff from rest-catalog-open-api.yaml to make it match a released version of API.

I think we can do the above 2 together in one separate PR so the current one won't contain too many changes

[jackye1995]

this does not include the management APIs?

[HonahX]

The Management APIs and Catalog APIs have different scopes and are typically referenced separately for service/client generation, as seen here:
https://github.com/HonahX/polaris/blob/4c7ae00b542806928dc05d1e4f56d51014dedac5/api/iceberg-service/build.gradle.kts#L51
https://github.com/HonahX/polaris/blob/4c7ae00b542806928dc05d1e4f56d51014dedac5/api/management-service/build.gradle.kts#L48

They also have different URL prefixes, security configurations, etc. Given these differences, I think it makes sense to keep them in separate YAML files for better flexibility.

The goal of this PR is to clearly separate Polaris-specific catalog APIs from Iceberg catalog APIs.

[jackye1995]

I think we should directly write this out instead of pointing to the IRC spec, since from what was previously discussed, the IRC spec token endpoint is being deprecated.

[HonahX]

That’s a great point! Just to confirm my understanding: The OAuth2 scheme needs to be updated because it references the removed token endpoint.

I’ll make this explicit and add a note to ensure that if we remove the token endpoint from Polaris or change it to a test-only endpoint: #12, we remember to update the scheme as well.