Skip to content

Commit

Permalink
docs: Add MaxCompute Sink to Glossary and link to MaxCompute config (#59
Browse files Browse the repository at this point in the history
)

* chore: add glossary and link to official depot config for maxcompute sink

* chore: update guide
  • Loading branch information
ekawinataa authored Dec 23, 2024
1 parent 7d465ea commit ab9fc34
Show file tree
Hide file tree
Showing 4 changed files with 19 additions and 3 deletions.
11 changes: 9 additions & 2 deletions docs/docs/guides/create_firehose.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,9 +133,9 @@ _**Note:**_ [_**DATABASE**_](../sinks/influxdb-sink.md#sink_influx_db_name) _**a

- it requires the following [variables](../sinks/bigquery-sink.md) to be set.
- For INPUT_SCHEMA_DATA_TYPE = protobuf, this sink will generate bigquery schema from protobuf message schema and update bigquery table with the latest generated schema.
- The protobuf message of a `google.protobuf.Timestamp` field might be needed when table partitioning is enabled.
- The protobuf message of a `google.protobuf.Timestamp` field might be needed when table partitioning is enabled.
- For INPUT_SCHEMA_DATA_TYPE = json, this sink will generate bigquery schema by infering incoming json. In future we will add support for json schema as well coming from stencil.
- The timestamp column is needed incase of partition table. It can be generated at the time of ingestion by setting the config. Please refer to config `SINK_BIGQUERY_ADD_EVENT_TIMESTAMP_ENABLE` in [depot bigquery sink config section](https://github.com/goto/depot/blob/main/docs/reference/configuration/bigquery-sink.md#sink_bigquery_add_event_timestamp_enable)
- The timestamp column is needed incase of partition table. It can be generated at the time of ingestion by setting the config. Please refer to config `SINK_BIGQUERY_ADD_EVENT_TIMESTAMP_ENABLE` in [depot bigquery sink config section](https://github.com/goto/depot/blob/main/docs/reference/configuration/bigquery-sink.md#sink_bigquery_add_event_timestamp_enable)
- Google cloud credential with some bigquery permission is required to run this sink.

## Create a Bigtable sink
Expand All @@ -149,4 +149,11 @@ _**Note:**_ [_**DATABASE**_](../sinks/influxdb-sink.md#sink_influx_db_name) _**a

If you'd like to connect to a sink which is not yet supported, you can create a new sink by following the [contribution guidelines](../contribute/contribution.md)

## Create a MaxCompute sink

- it requires the following [variables](../sinks/maxcompute-sink.md) to be set. Please follow the Configuration section in the MaxCompute Sink documentation for more details.
- As of now it only supports INPUT_SCHEMA_DATA_TYPE = protobuf. Schema creation and update is inferred from protobuf schema.
- The protobuf message of a `google.protobuf.Timestamp` field might be needed when table partitioning is enabled.
- INPUT_SCHEMA_DATA_TYPE = json will be supported in future.
- Schema/Dataset need to be created in advance in MaxCompute.
- Service account requires ODPS and Tunnel Service permissions.
1 change: 1 addition & 0 deletions docs/docs/reference/glossary.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@
[Log sink configs](../advance/generic.md)

## M
[MaxCompute Sink](../sinks/maxcompute-sink.md)

[metrics](metrics.md)

Expand Down
9 changes: 8 additions & 1 deletion docs/docs/sinks/maxcompute-sink.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,11 @@
# MaxCompute sink
# MaxCompute

MaxCompute Sink is a sink connector that allows you to write data from Kafka to Alibaba's MaxCompute Table. It supports consuming data in Protobuf and JSON(ToDo) format.

## Configuration

MaxCompute Sink requires the following variables to be set along with Generic ones.
Please refer to the [Depot's MaxCompute Sink Configuration](https://github.com/goto/depot/blob/main/docs/reference/configuration/maxcompute.md) for more.

### Datatype Protobuf

Expand Down
1 change: 1 addition & 0 deletions docs/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ module.exports = {
"sinks/redis-sink",
"sinks/elasticsearch-sink",
"sinks/blob-sink",
"sinks/maxcompute-sink",
],
},
{
Expand Down

0 comments on commit ab9fc34

Please sign in to comment.