[destination-S3]Can S3 Destination Support Column Sorting Before Writing?

[bambangraharjo]

Is it possible to sort the order of columns when writing data to S3 using the S3 destination connector? Alternatively, can we create a custom connector to handle this?

The idea is that if column sorting is supported, the files written to S3 can have the same column order as the target table in Redshift. This would allow us to use the files directly with Redshift's COPY command or Redshift Spectrum without additional preprocessing.

I prefer not to write directly to Redshift due to potential issues with SUPER columns exceeding 65k bytes or VARCHAR columns larger than 65k being automatically converted to NULL, which happens with the Redshift destination connector.

Would love to hear thoughts or suggestions! 😊

[marcosmarxm]

Currently, you can't change this behavior. I couldn't find any existing feature request for this, @bambangraharjo, so feel free to open a new request to discuss a future implementation.

[bambangraharjo]

Hi @marcosmarxm , I just discovered that the columns are already sorted alphabetically, and the Airbyte columns always follow the same order. Based on this, it's possible to use the copy command to Redshift with column mentions without needing additional sorting. Thank you.