Skip to content

Add a configurable parameter to the number of backup attempts that are made.#1

Merged
tglman merged 4 commits intoorientechnologies:3.2.xfrom
odysseaspenta:3.2.x
Feb 10, 2026
Merged

Add a configurable parameter to the number of backup attempts that are made.#1
tglman merged 4 commits intoorientechnologies:3.2.xfrom
odysseaspenta:3.2.x

Conversation

@odysseaspenta
Copy link
Copy Markdown

@odysseaspenta odysseaspenta commented Jan 4, 2026

Currently, if the backup fails for some reason, the server can get into a situation where it continuously schedules another backup to take place that can potentially fill up the backup log with entries. Here is example output from a server that is configured to perform backups every so often. When the backup fails, the server schedules another backup event to take place immediately. This behavior was present in 3.2.47.

2026-01-04T16:54:59 [INFO] Backup FULL_BACKUP in (ms):823 (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:00 [INFO] Scheduled [FULL_BACKUP] task : a18c2be8-ab4e-485d-b0f8-bfe01bb94c54. Next execution will be Sun Jan 04 15:20:00 EST 2026 (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:00 [INFO] Backup started FULL_BACKUP (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:00 [INFO] Backup FULL_BACKUP in (ms):794 (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:00 [INFO] Scheduled [FULL_BACKUP] task : a18c2be8-ab4e-485d-b0f8-bfe01bb94c54. Next execution will be Sun Jan 04 15:20:00 EST 2026 (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:00 [INFO] Backup started FULL_BACKUP (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:00 [INFO] Backup FULL_BACKUP in (ms):697 (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:01 [INFO] Scheduled [FULL_BACKUP] task : a18c2be8-ab4e-485d-b0f8-bfe01bb94c54. Next execution will be Sun Jan 04 15:20:00 EST 2026 (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:01 [INFO] Backup started FULL_BACKUP (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:01 [INFO] Backup FULL_BACKUP in (ms):698 (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:01 [INFO] Scheduled [FULL_BACKUP] task : a18c2be8-ab4e-485d-b0f8-bfe01bb94c54. Next execution will be Sun Jan 04 15:20:00 EST 2026 (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:01 [INFO] Backup started FULL_BACKUP (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:01 [INFO] Backup FULL_BACKUP in (ms):725 (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:02 [INFO] Scheduled [FULL_BACKUP] task : a18c2be8-ab4e-485d-b0f8-bfe01bb94c54. Next execution will be Sun Jan 04 15:20:00 EST 2026 (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:02 [INFO] Backup started FULL_BACKUP (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:02 [INFO] Backup FULL_BACKUP in (ms):717 (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:02 [INFO] Scheduled [FULL_BACKUP] task : a18c2be8-ab4e-485d-b0f8-bfe01bb94c54. Next execution will be Sun Jan 04 15:20:00 EST 2026 (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:02 [INFO] Backup started FULL_BACKUP (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:02 [INFO] Backup FULL_BACKUP in (ms):726 (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:03 [INFO] Scheduled [FULL_BACKUP] task : a18c2be8-ab4e-485d-b0f8-bfe01bb94c54. Next execution will be Sun Jan 04 15:20:00 EST 2026 (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:03 [INFO] Backup started FULL_BACKUP (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:03 [INFO] Backup FULL_BACKUP in (ms):725 (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:03 [INFO] Scheduled [FULL_BACKUP] task : a18c2be8-ab4e-485d-b0f8-bfe01bb94c54. Next execution will be Sun Jan 04 15:20:00 EST 2026 (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:03 [INFO] Backup started FULL_BACKUP (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:03 [INFO] Backup FULL_BACKUP in (ms):680 (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:04 [INFO] Scheduled [FULL_BACKUP] task : a18c2be8-ab4e-485d-b0f8-bfe01bb94c54. Next execution will be Sun Jan 04 15:20:00 EST 2026 (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:04 [INFO] Backup started FULL_BACKUP (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:04 [INFO] Backup FULL_BACKUP in (ms):704 (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:04 [INFO] Scheduled [FULL_BACKUP] task : a18c2be8-ab4e-485d-b0f8-bfe01bb94c54. Next execution will be Sun Jan 04 15:20:00 EST 2026 (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:04 [INFO] Backup started FULL_BACKUP (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:04 [INFO] Backup FULL_BACKUP in (ms):693 (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:05 [INFO] Scheduled [FULL_BACKUP] task : a18c2be8-ab4e-485d-b0f8-bfe01bb94c54. Next execution will be Sun Jan 04 15:20:00 EST 2026 (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:05 [INFO] Backup started FULL_BACKUP (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:05 [INFO] Backup FULL_BACKUP in (ms):679 (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:05 [INFO] Scheduled [FULL_BACKUP] task : a18c2be8-ab4e-485d-b0f8-bfe01bb94c54. Next execution will be Sun Jan 04 15:20:00 EST 2026 (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:05 [INFO] Backup started FULL_BACKUP (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:05 [INFO] Backup FULL_BACKUP in (ms):760 (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:06 [INFO] Scheduled [FULL_BACKUP] task : a18c2be8-ab4e-485d-b0f8-bfe01bb94c54. Next execution will be Sun Jan 04 15:20:00 EST 2026 (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:06 [INFO] Backup started FULL_BACKUP (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:06 [INFO] Backup FULL_BACKUP in (ms):675 (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:06 [INFO] Scheduled [FULL_BACKUP] task : a18c2be8-ab4e-485d-b0f8-bfe01bb94c54. Next execution will be Sun Jan 04 15:20:00 EST 2026 (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:06 [INFO] Backup started FULL_BACKUP (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:06 [INFO] Backup FULL_BACKUP in (ms):723 (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T16:55:07 [INFO] Scheduled [FULL_BACKUP] task : a18c2be8-ab4e-485d-b0f8-bfe01bb94c54. Next execution will be Sun Jan 04 15:20:00 EST 2026 (com.orientechnologies.agent.services.backup.OBackupTask)

Added support for limiting the number of times that a failed backup is attempted using a configurable parameter. This it to prevent the situation where the server gets in an endless loop of retrying to succesful make a backup that fails, filling up the backup log with entries. Here is the output of the same situation using the fixes suggested with this pull request. After 3 attempts, the server quits trying to make the backup.

2026-01-04T15:18:22 [INFO] Scheduled [FULL_BACKUP] task : a18c2be8-ab4e-485d-b0f8-bfe01bb94c54. Next execution will be Sun Jan 04 15:20:00 EST 2026 (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T15:20:00 [INFO] Backup started FULL_BACKUP (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T15:20:00 [WARNING] Backup failed for [oConfig-db]. Retry attempt 1/3 scheduled in 60 seconds (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T15:20:00 [INFO] Backup FULL_BACKUP in (ms):373 (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T15:21:00 [INFO] Backup started FULL_BACKUP (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T15:21:00 [WARNING] Backup failed for [oConfig-db]. Retry attempt 2/3 scheduled in 300 seconds (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T15:21:00 [INFO] Backup FULL_BACKUP in (ms):384 (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T15:26:00 [INFO] Backup started FULL_BACKUP (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T15:26:01 [WARNING] Backup failed for [oConfig-db]. Retry attempt 3/3 scheduled in 900 seconds (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T15:26:01 [INFO] Backup FULL_BACKUP in (ms):272 (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T15:41:01 [INFO] Backup started FULL_BACKUP (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T15:41:01 [WARNING] Backup failed for [oConfig-db] after 3 retries. Manual intervention required. Backup scheduling stopped. (com.orientechnologies.agent.services.backup.OBackupTask)
2026-01-04T15:41:01 [INFO] Backup FULL_BACKUP in (ms):758 (com.orientechnologies.agent.services.backup.OBackupTask)

…s attempted using a configurable parameter. This it to prevent the situation where the server gets in an endless loop of retrying to succesful make a backup that fails, filling up the backup log with entries.
Copy link
Copy Markdown
Member

@tglman tglman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good, there are few small changes, but done this can be merged

</parent>
<groupId>com.orientechnologies</groupId>
<artifactId>agent</artifactId>
<version>3.2.50-SNAPSHOT</version>
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here for the snapshot

<parent>
<groupId>com.orientechnologies</groupId>
<artifactId>orientdb-parent</artifactId>
<version>3.2.50-SNAPSHOT</version>
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here for the snapshot

</parent>
<groupId>com.orientechnologies</groupId>
<artifactId>orientdb-enterprise-parent</artifactId>
<version>3.2.50-SNAPSHOT</version>
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here for the snapshot

error.setStackTrace(sw.toString());

// Add retry count if listener is OBackupTask
if (listener instanceof OBackupTask) {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of specific casting here, maybe is better to refactor the doBackup method and add the retry count for logging

() -> {
try {
final long start = tickStart();
strategy.doBackup(OBackupTask.this);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this case should be possible to pass the retry as parameter, and modify also the schedule method for pass the 0 as retry count

@odysseaspenta
Copy link
Copy Markdown
Author

I have applied the requested changes. Please take another look when you get a chance.

@odysseaspenta odysseaspenta requested a review from tglman February 10, 2026 13:52
@tglman tglman merged commit 1a9d5ba into orientechnologies:3.2.x Feb 10, 2026
3 checks passed
@tglman
Copy link
Copy Markdown
Member

tglman commented Feb 24, 2026

Hi,

just shipped 3.2.50 with this fix

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants