Allow Transloadit-hosted Companion with other uploaders

[Murderlon]

Closes #5515

We want to allow people to only use Transloadit for Companion, even if they use another uploader. This makes using Transloadit less all-or-nothing.

Unfortunately there is no pretty way of doing this. Instead of separating Companion logic from uploaders, every uploader implements remote uploads themselves. It works like this:

1. Install a remote plugin (such as @uppy/google-drive)
2. The remote plugin creates a companion request client (@uppy/companion-client) and @uppy/provider-views
3. Inside @uppy/provider-views we call uppy.registerRequestClient() to put the request client on core so it can be shared.
   • We need to store request clients by a unique ID, so we can share RequestClient instances across files
     this allows us to do rate limiting and synchronous operations like refreshing provider tokens
     example: refreshing tokens: if each file has their own requestclient,
     we don't have any way to synchronize all requests in order to block all requests, refresh the token, and unblock all requests and allow them to run with a the new access token.
4. When the upload starts, uploader plugins (such as @uppy/aws-s3) filters the remote files and gets the instructions to tell Companion what uploader to use:

uppy/packages/@uppy/aws-s3/src/index.ts

Lines 925 to 972 in d6156bb

	#getCompanionClientArgs(file: UppyFile<M, B>) {
	return {
	...file.remote?.body,
	protocol: 's3-multipart',
	size: file.data.size,
	metadata: file.meta,
	}
	}
	#upload = async (fileIDs: string[]) => {
	if (fileIDs.length === 0) return undefined
	const files = this.uppy.getFilesByIds(fileIDs)
	const filesFiltered = filterNonFailedFiles(files)
	const filesToEmit = filterFilesToEmitUploadStarted(filesFiltered)
	this.uppy.emit('upload-start', filesToEmit)
	const promises = filesFiltered.map((file) => {
	if (file.isRemote) {
	const getQueue = () => this.requests
	this.#setResumableUploadsCapability(false)
	const controller = new AbortController()
	const removedHandler = (removedFile: UppyFile<M, B>) => {
	if (removedFile.id === file.id) controller.abort()
	}
	this.uppy.on('file-removed', removedHandler)
	const uploadPromise = this.uppy
	.getRequestClientForFile<RequestClient<M, B>>(file)
	.uploadRemoteFile(file, this.#getCompanionClientArgs(file), {
	signal: controller.signal,
	getQueue,
	})
	this.requests.wrapSyncFunction(
	() => {
	this.uppy.off('file-removed', removedHandler)
	},
	{ priority: -1 },
	)()
	return uploadPromise
	}
	return this.#uploadLocalFile(file)
	})

---

With this PR you can do this:

uppy.use(AwsS3, { /* ... */ })
uppy.use(Transloadit, { companionOnly: true /* ... */ })

This is the least ugly way I could come up with.

This is possible due to a new value on Uppy's state, remoteUploader, which is set by the Transloadit plugin when companionOnly is true.

You can also (ab)use this to allow users to change the remote uploader in other cases: uppy.setState({ remoteUploader: 'multipart' }).

[github-actions]

Diff output files

diff --git a/packages/@uppy/aws-s3/lib/index.js b/packages/@uppy/aws-s3/lib/index.js
index 9fd9876..9b2e182 100644
--- a/packages/@uppy/aws-s3/lib/index.js
+++ b/packages/@uppy/aws-s3/lib/index.js
@@ -694,10 +694,18 @@ function _uploadLocalFile2(file) {
 }
 function _getCompanionClientArgs2(file) {
   var _file$remote;
+  const opts = {
+    ...this.opts,
+  };
+  if (file.tus) {
+    Object.assign(opts, file.tus);
+  }
   return {
     ...((_file$remote = file.remote) == null ? void 0 : _file$remote.body),
-    protocol: "s3-multipart",
+    endpoint: opts.endpoint,
+    protocol: this.uppy.getState().remoteUploader || "s3-multipart",
     size: file.data.size,
+    headers: opts.headers,
     metadata: file.meta,
   };
 }
diff --git a/packages/@uppy/transloadit/lib/index.js b/packages/@uppy/transloadit/lib/index.js
index 26d1b01..17a027a 100644
--- a/packages/@uppy/transloadit/lib/index.js
+++ b/packages/@uppy/transloadit/lib/index.js
@@ -25,6 +25,7 @@ const defaultOptions = {
   waitForMetadata: false,
   alwaysRunAssembly: false,
   importFromUploadURLs: false,
+  companionOnly: false,
   limit: 20,
   retryDelays: [7000, 10000, 15000, 20000],
   clientName: null,
@@ -248,25 +249,26 @@ export default class Transloadit extends BasePlugin {
           : this.opts.assemblyOptions;
         (_assemblyOptions$fiel = assemblyOptions.fields) != null ? _assemblyOptions$fiel : assemblyOptions.fields = {};
         validateParams(assemblyOptions.params);
+        const ids = this.opts.companionOnly ? fileIDs.filter(id => this.uppy.getFile(id).isRemote) : fileIDs;
         try {
           var _this$assembly2;
           const assembly = (_this$assembly2 = this.assembly) != null
             ? _this$assembly2
-            : await _classPrivateFieldLooseBase(this, _createAssembly)[_createAssembly](fileIDs, assemblyOptions);
+            : await _classPrivateFieldLooseBase(this, _createAssembly)[_createAssembly](ids, assemblyOptions);
           if (assembly == null) throw new Error("All files were canceled after assembly was created");
           if (this.opts.importFromUploadURLs) {
-            await _classPrivateFieldLooseBase(this, _reserveFiles)[_reserveFiles](assembly, fileIDs);
+            await _classPrivateFieldLooseBase(this, _reserveFiles)[_reserveFiles](assembly, ids);
           }
-          fileIDs.forEach(fileID => {
+          ids.forEach(fileID => {
             const file = this.uppy.getFile(fileID);
             this.uppy.emit("preprocess-complete", file);
           });
           _classPrivateFieldLooseBase(this, _createAssemblyWatcher)[_createAssemblyWatcher](
             assembly.status.assembly_id,
           );
-          _classPrivateFieldLooseBase(this, _connectAssembly)[_connectAssembly](assembly, fileIDs);
+          _classPrivateFieldLooseBase(this, _connectAssembly)[_connectAssembly](assembly, ids);
         } catch (err) {
-          fileIDs.forEach(fileID => {
+          ids.forEach(fileID => {
             const file = this.uppy.getFile(fileID);
             this.uppy.emit("preprocess-complete", file);
             this.uppy.emit("upload-error", file, err);
@@ -361,6 +363,11 @@ export default class Transloadit extends BasePlugin {
     this.defaultLocale = locale;
     _classPrivateFieldLooseBase(this, _rateLimitedQueue)[_rateLimitedQueue] = new RateLimitedQueue(this.opts.limit);
     this.i18nInit();
+    if (this.opts.companionOnly) {
+      this.uppy.setState({
+        remoteUploader: "tus",
+      });
+    }
     this.client = new Client({
       service: this.opts.service,
       client: _classPrivateFieldLooseBase(this, _getClientVersion)[_getClientVersion](),
@@ -380,7 +387,7 @@ export default class Transloadit extends BasePlugin {
         "upload-success",
         _classPrivateFieldLooseBase(this, _onFileUploadURLAvailable)[_onFileUploadURLAvailable],
       );
-    } else {
+    } else if (!this.opts.companionOnly) {
       this.uppy.use(Tus, {
         storeFingerprintForResuming: false,
         allowedMetaFields: true,
diff --git a/packages/@uppy/xhr-upload/lib/index.js b/packages/@uppy/xhr-upload/lib/index.js
index 7c2e2d1..a5f0b86 100644
--- a/packages/@uppy/xhr-upload/lib/index.js
+++ b/packages/@uppy/xhr-upload/lib/index.js
@@ -366,9 +366,12 @@ function _getCompanionClientArgs2(file) {
   var _file$remote;
   const opts = this.getOptions(file);
   const allowedMetaFields = getAllowedMetaFields(opts.allowedMetaFields, file.meta);
+  if (file.tus) {
+    Object.assign(opts, file.tus);
+  }
   return {
     ...((_file$remote = file.remote) == null ? void 0 : _file$remote.body),
-    protocol: "multipart",
+    protocol: this.uppy.getState().remoteUploader || "multipart",
     endpoint: opts.endpoint,
     size: file.data.size,
     fieldname: opts.fieldName,

[mifi]

I think this is a very clever way to allow people to use Transloadit hosted Companion to upload local and remote files all the way to their destination. I'm afraid that this will further complicate our already very complex uploading/plugin logic, making it more error prone and even harder to understand, but I do see the great benefits so I think we have to do it even though it's hacky.

[mifi]

I don't think this is a very clean way of doing it. We should pick the options that we need OR use typescript to verify that things are consistent. Or maybe better:

return {
         ...file.remote?.body,
      protocol: this.uppy.getState().remoteUploader || 's3-multipart',
      size: file.data.size,
      metadata: file.meta,
      ...(file.tus && {
            endpoint: file.tus.endpoint,
           headers: file.tus.headers,
      })
}

[mifi]

ditto

[mifi]

I really don't like using state for this. Can't we instead make it a pure option on each uploader, instead of passing the transloadit plugin option through state like this? Yes it's more verbose but it's so much easier to grok when reading/understanding how the code works, compared to analysing state mutations. If the verbosity is a problem, that could be solved with good documation, or something like a helper method or wrapper plugin that will initialise the correct plugins with the correct options

[mifi]

Suggested change

	const ids =
	// with the `companionOnly` option, we should not post-process local files, because they will be uploaded directly to the final destination and not processed by Transloadit assemblies.
	const ids =

[mifi]

Suggested change

	/**
	/**
	todo document in uppy.io instead

[mifi]

I'm not sure if companionOnly is the right name for this. It makes it sound like we are only using Companion, but in fact we are also using Transloadit+assemblies. Maybe a better name would be something like remoteOnly? considering that what the option does is to make it only process remote files. (as opposed to processing both remote and local files without the option).