Support more file types

[jovandeginste]

I'm not a python dev, but was working on a similar project (personal) in Golang. My project was standalone (no central web server).

I would really like to see more file type support in paperless. I spun up a separate Tika server for generic document content & metadata retrieval.

Since this is a web app, the browser can not show many file types natively. To get around this issue, we might convert anything "foreign" to a PDF using some separate process. I found the gotenberg project to be interesting.

So we might have a hook on incoming "foreign" files, that sends it to Tika for content+metadata; then, when we know the file's type, we can send it to the gotenberg project to have a PDF version for web display. Of course, when the user retrieves the file, he can have it in its original form (eg. docx).

What do you think?

I can help with the Tika and Gotenberg part, I think, but again, I'm not a Python dev...

[jonaswinkler]

On the original project, there is a pull request the-paperless-project/paperless#600 done by @Tooa that was never merged. I'd like to see more file type support in paperless.

Here are a couple notes.

• Paperless has native support for storing both an original document, which can be any type, and an associated archived document, which must be PDF.
• I don't like pulling in lots of dependencies into the project to support this, and these projects seem pretty heavy. Most users will be happy with paperless just processing PDF documents and scanned images and these users shouldn't do any more configuration / dependency installation for additional mime types they don't need paperless to support. This is especially true for bare metal installations. This means that we have to make this entirely optional / configurable.

Regarding the python part:

• It's not pretty hard. Most of the hard work is integrating these external dependencies in python. Getting paperless to process new file types is pretty straight forward. Have a look at src/paperless_text/ to see a very simple parser implementation for plain text files. You simply announce to paperless what mime types your parser supports and write methods to generate content / archived PDF / thumbnail and metadata from a file.
• In order to make this optional, we need some configuration option (maybe the URL of the tika server) to enable this feature. In apps.py, you'll find a line that says document_consumer_declaration.connect(text_consumer_declaration). This is what tells paperless to use the parser. If you make this optional, paperless will not use the parser at all.

I've also looked into using some native python libraries to support individual file types such as docx and odf, but these usually only go as far as extracting text from the file, and we'd still need to generate a PDF document and thumbnail for display in the web app. For odf documents, this usually means having a complete install of LibreOffice available, which is bad.

[jonaswinkler]

There's also some very basic documentation on how to write custom parsers available.

https://paperless-ng.readthedocs.io/en/latest/extending.html#making-custom-parsers

[jovandeginste]

Hi, I saw the PR, but didn't look into the code yet (not sure how applicable it would be for the ng part).

I didn't see anywhere how to store the "native" document. I always get a parse error when I try to upload a doc file. I didn't see anywhere a list of "allowed file types" that I could edit... What did I miss?

I totally agree with the optional part. I would actually not mind it if I could eg. pre-parse any other documents, and upload a native+pdf version to paperless. That should work for most of my use cases...

[jonaswinkler]

The only thing you need to change is the INSTALLED_APPS list in src/paperless/settings.py. This will make your extension available to paperless, regardless of deployment method.

Edit: I should probably add this to the documentation as well.

[jovandeginste]

Could it be possible to have this list extended using an Env var, so that the Docker image is easily extended?

[jonaswinkler]

No. Once the docker image is built, there's no way to permanently add additional apps to the src directory. Therefore, making this configurable doesn't really make sense right now.

[jovandeginste]

I would add my plugin (while testing) like this (Docker CLI parameters):

-v /path/to/dev/paperless_tika:/usr/src/paperless/src/paperless_tika:ro

This would literally add the paperless_tika directory as a subdirectory under /usr/src/paperless/src, and thus accessible...

[jonaswinkler]

For extending paperless, you should really set up a development environment. This won't work with the docker image.

[jovandeginste]

First part is ready: Tika

Todo: Gotenberg

[jovandeginste]

Gotenberg working too, now

[jovandeginste]

Current version: https://github.com/jovandeginste/paperless_tika

[jovandeginste]

For some reason, it only picks a limited number of filetypes. When eg. I add a docx file to my consume folder, it throws:

Not consuming file myflie.docx: Unknown file extension.

Any idea what I'm missing? 😕

[jonaswinkler]

Not sure. Need to debug later. application/vnd.openxmlformats-officedocument.wordprocessingml.document is def. the correct mime type and python recognizes docx as a valid file extension for that. The only reason I can think of is that the parser isn't actually registered.

Btw.

We can totally add this to the main repo. The python dependencies aren't that big of a deal. It's just that having both web servers available should be optional and configurable. If unavailable / not configured, we can simply not announce the parser to paperless. Provide some documentation on how to adjust the docker-compose file to enable this feature, done. So if you want to prepare a pull request into a new feature branch, lets say feature-tika, that would make it much more easier for me to debug.

[jovandeginste]

As a matter of completeness, to debug:

import mimetypes
mimetypes.init()
print(mimetypes.knownfiles)

inside the container now says:

[
  '/etc/mime.types', '/etc/httpd/mime.types', '/etc/httpd/conf/mime.types', '/etc/apache/mime.types',
  '/etc/apache2/mime.types', '/usr/local/etc/httpd/conf/mime.types', '/usr/local/lib/netscape/mime.types',
  '/usr/local/etc/httpd/conf/mime.types', '/usr/local/etc/mime.types'
]

Edit: so not sure where defaults come from...

[jovandeginste]

Regarding the PR: I'm still working out some kinks, as you see 😉

Feel free to provide feedback on the code as is! Eg. is there a nicer way to list a large number of extensions?

[jonaswinkler]

As in, nicer than in the "mime_types": [] array? No. That should be nice enough, except if you intent to list a couple hundred. (do you?)

[jonaswinkler]

Two obvious points are as follows:

• Don't re-use RasterisedDocumentParser. No dependencies between parsers. Rather, we should put the pdf thumbnail generation in the documents/parsers.py, probably. As a function that many parsers may reuse as they see fit.
• Catch HTTP errors / exceptions and raise ParseErrors when something goes wrong.

[jovandeginste]

I updated my code to reflect this (I hope). Again, I'm not a Python dev 😄

[jovandeginste]

Any other filetypes on your wish list?