Some code confused me.

[run1999]

Please check your code.

domains/s2orc/extract_papers.py

8 from joblib import Parallel, delayed
Does this Parallel module work in your code?

29 if len(papers) > 512:
I think it should be ‘==’ but not ‘>‘ , cause your shell script batch size is 512 , but your code create 513 files instead.

52 return metadata
Is this 'metadata' declared in the function?

By the way, I don't see your "get_metadata" used in the main function.