Data integrity error: More than allowed put data [20971482 > 20971479]

[tpo]

Below I have evidently replaced stuff with "foo". Please forgive me and ask in case my search and replace was too stupid.

Problem

$ ls -lh /tmp/foo
-rw-rw-r-- 1 foo foo 1.8M Jan 22 14:40 /tmp/foo

$ cat /tmp/foo | borg create --compression none -v --stats ssh://[email protected]:22/mnt/backup/borg/foo::foo-2025-01-22_13:07:20 -
Creating archive at "ssh://[email protected]:22/mnt/backup/borg/foo::foo-2025-01-22_13:07:20"
Data integrity error: More than allowed put data [20971482 > 20971479]
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 5401, in main
    exit_code = archiver.run(args)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 5321, in run
    return set_ec(func(args))
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 190, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 682, in do_create
    create_inner(archive, cache, fso)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 636, in create_inner
    archive.save(comment=args.comment, timestamp=args.timestamp)
  File "/usr/lib/python3/dist-packages/borg/archive.py", line 666, in save
    self.manifest.write()
  File "/usr/lib/python3/dist-packages/borg/helpers/manifest.py", line 265, in write
    self.repository.put(self.MANIFEST_ID, self.key.encrypt(data))
  File "/usr/lib/python3/dist-packages/borg/remote.py", line 476, in do_rpc
    return self.call(f.__name__, named, **extra)
  File "/usr/lib/python3/dist-packages/borg/remote.py", line 711, in call
    for resp in self.call_many(cmd, [args], **kw):
  File "/usr/lib/python3/dist-packages/borg/remote.py", line 794, in call_many
    handle_error(unpacked)
  File "/usr/lib/python3/dist-packages/borg/remote.py", line 752, in handle_error
    raise IntegrityError(args[0].decode())
borg.helpers.errors.IntegrityError: Data integrity error: More than allowed put data [20971482 > 20971479]

Platform: Linux foo 5.4.0-200-generic #220-Ubuntu SMP Fri Sep 27 13:19:16 UTC 2024 x86_64
Linux: Unknown Linux
Borg: 1.2.8  Python: CPython 3.8.10 msgpack: 0.6.2 fuse: None [pyfuse3,llfuse]
PID: 2148951  CWD: /var/lib/postgresql
sys.argv: ['/usr/bin/borg', 'create', '--compression', 'none', '-v', '--stats', 'ssh://[email protected]:22/mnt/backup/borg/foo::foo-2025-01-22_13:07:20', '-']
SSH_ORIGINAL_COMMAND: None

Info about the last successful backup:

$ borg info ::foo-2024-10-11_23:30:02
Archive name: foo-2024-10-11_23:30:02
Archive fingerprint: 7fd3ab95e153801fd5bbbc018acf96376c7c5c62e41d32f9f559d66f8f9b3ae3
Comment:
Hostname: foo
Username: foo
Time (start): Fri, 2024-10-11 23:30:05
Time (end): Fri, 2024-10-11 23:30:05
Duration: 0.18 seconds
Number of files: 1
Command line: /usr/bin/borg create --compression none -v --stats ssh://[email protected]:22/mnt/backup/borg/foo::foo-2024-10-11_23:30:02 -
Utilization of maximum supported archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:                    0 B                  0 B                817 B
All archives:                1.67 TB              1.67 TB            347.29 GB

                       Unique chunks         Total chunks
Chunk index:                  533422              1071811

$ borg info
Repository ID: 795f8929cbf39df9414806abd5781e602c1dc55730b04fd5969fa22c507671a9
Location: ssh://[email protected]:22/mnt/backup/borg/foo
Encrypted: Yes (repokey)
Cache: /home/foo/.cache/borg/795f8929cbf39df9414806abd5781e602c1dc55730b04fd5969fa22c507671a9
Security dir: /home/foo/.config/borg/security/795f8929cbf39df9414806abd5781e602c1dc55730b04fd5969fa22c507671a9
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
All archives:                1.67 TB              1.67 TB            347.29 GB

                       Unique chunks         Total chunks
Chunk index:                  533420              1071811

$ borg --version
borg 1.2.8

This bugs looks similar to:

• add exception handler for "Data integrity error: More than allowed put data" #5307
• per archive total item count limit #1473
• per archive total item count limit #1452

However I couldn't find anything there that could help me. I do not remember where, but I've seen somewhere that borg implemented a mechanism to prevent bumping against the MAX_DATA_SIZE [1] so I've compiled and installed borg 1.28 on the client. However that didn't help me either.

The server is running:

$ ssh [email protected]
$ borg --version
borg 1.1.16

Is there anything I can do about this problem?

Thanks a lot!
*t

[1] https://fossies.org/dox/borgbackup-1.4.0/namespaceborg_1_1constants.html#a8e0d20acb689bdd8672e85e9c005e859

[ThomasWaldmann]

How many archives are in that repo?

[tpo]

How many archives are in that repo?

$ borg list | wc -l
161315

[ThomasWaldmann]

Having 161 thousand archives is not a good idea:

• it seems your manifest (which stores the list of archives) grew beyond its maximum allowed size - that is why you get that exception.
• if you would need to run a full borg check on the repo, the archives part of the check likely would take longer than you would like to wait.

So, you need to carefully use borg prune (use --dry-run --list to make sure it does what you want) to slowly reduce the count of archives and borg compact to free the related repo space.

• https://borgbackup.readthedocs.io/en/1.2.8/usage/prune.html
• https://borgbackup.readthedocs.io/en/1.2.8/usage/compact.html

[ThomasWaldmann]

Side note: that is a borg 1.x limitation, recent code in master branch (borg2, beta and not production-ready) does not have that limitation.

[tpo]

So, you need to carefully use borg prune (use --dry-run to make sure it does what you want) to slowly reduce the count of archives and borg compact to free the related repo space.

After you asked me how many archives there are in the repo I sensed a hint and started to purge old archives. The process is ongoing... I'll let you know about the results. Thanks a lot!

[tpo]

After you asked me how many archives there are in the repo I sensed a hint and started to purge old archives. The process is ongoing... I'll let you know about the results. Thanks a lot!

Seems to work now. Thanks a lot!!!