Cubestore Router space is growing continuously due to temp-uploads

[viktordebulat]

Problem

We're using self-hosted cubestore 1 router + 1 worker for preaggregations in production mode. There is also Cubejs API and refresh worker.
Space on router is growing due to temp-uploads directory which seems never purges. I see csv.gz files from day 0 and even files for preaggregations which don't exist anymore.

For storage we're using self-managed S3 compatible storage (similar to Minio) in the same k8s cluster as Cube.

Last versions tried - v1.1.0 and v1.1.15

Config or router is simple and uses standard env variables approach: CUBESTORE_SERVER_NAME, CUBESTORE_WORKERS, CUBESTORE_META_PORT, CUBESTORE_MINIO_SERVER_ENDPOINT, CUBESTORE_MINIO_BUCKET, CUBESTORE_MINIO_ACCESS_KEY_ID, CUBESTORE_MINIO_SECRET_ACCESS_KEY

My questions are:

• Any clue, why old and even non-existing preaggregations files are not deleted from temp-uploads on cubestore router? Maybe there is some config missed?
• How does cubestore local storage housekeeping works, could it be configured somehow or should be managed by some external cronjobs?
• Could it be an issue when from time too time we get in cubestore router logs something like CubeError { message: "File sizes for cachestore-current doesn't match after upload. Expected to be 24 but 0 uploaded", backtrace: "", cause: Internal } for metastore-* and cachestore-current? Files are present on S3 storage, it's just might be a short time lag after upload to make that file availabe due to metadata for files not added yet. Does it retry on checking file size in case of error or not?

[allekai]

We are facing similar issues with growing space consumption in our router container.

Some more insight into how cachestore, metastore and temp-files are handled would be much appreciated. In what ways can we configure cube in this regard?

[viktordebulat]

Okay, we stopped changing schema and pre-aggregations for a while and it router space has stopped growing:

At the same time worker space decreased:

Some housekeeping under the hood, I assume. As S3 volume space also decrease symmetrically with worker:

So, need to understand why temp files are not deleted from router (again, there are couple related to non existing preaggregations).

[viktordebulat]

Made manual cleanup for old temp-uploads dir files. Will observe if it will grow up so fast again. But looks like housekeeping is working only for worker.