Make Nextcloud NAS backups independent, verified and retryable
This commit is contained in:
admin committed
1 parent
ffe3c354be
commit
7e94915442
15 files changed
+889
-504
No files matched your search
@@ -2,3 +2,6 @@
|
|||||||
*.log
|
*.log
|
||||||
/.cache/
|
/.cache/
|
||||||
/tmp/
|
/tmp/
|
||||||
|
|
||||||
|
__pycache__/
|
||||||
|
*.pyc
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
# Änderungen
|
||||||
|
|
||||||
|
## 2.0.0 — 2026-10-26
|
||||||
|
|
||||||
|
- Sicherung direkt aus dem rclone-Remote statt aus dem verzögert gestarteten Desktop-FUSE-Mount.
|
||||||
|
- Privater lokaler Zwischenstand, Cloud-Inventuren und Hash-/Downloadvergleich vor der Archivierung.
|
||||||
|
- Prüfung des tatsächlichen NAS-Dateisystems einschließlich gestapelter autofs-/NFS-Mounts; Repository-ID gebunden, keine automatische Initialisierung.
|
||||||
|
- Prüfung jedes neuen Archivs mit Datenverifikation und vollständigem Inventurvergleich vor Bereinigung.
|
||||||
|
- 15-Minuten-Prüftimer mit 24-Stunden-Sicherungsintervall, unabhängiger stündlicher Alterswächter und Desktop-Warnungen.
|
||||||
|
- Schutz bei massenhaftem Datenverlust, knappen Datenträgern, fremdem Staging, konkurrierenden Läufen und Abbrüchen.
|
||||||
|
- Atomare Laufzustände, Wiederholung fehlgeschlagener Wartung, mindestens sieben Archivstände zusätzlich zur Vierwochenfrist.
|
||||||
|
- Update erhält Schlüssel, Konfiguration und bestehende Archive; Restore-CLI weiterhin verfügbar.
|
||||||
|
- Automatisierte Fehler- und Integrationstests mit echtem rclone/Borg und Wiederherstellung.
|
||||||
+94
-249
@@ -1,283 +1,128 @@
|
|||||||
# Dokumentation: nextcloud-borg-backup
|
# Betriebsdokumentation
|
||||||
|
|
||||||
## Ziel
|
## Architektur und Befund
|
||||||
|
|
||||||
Diese Anwendung sichert den via rclone gemounteten Ordner `/home/johannes/Nextcloud` regelmäßig auf den via autofs erreichbaren Synology-Pfad `/net/thor/volume1/NetBackup/Nextcloud`.
|
Das vorhandene Projekt liegt unter `~/localdev/nextcloud-borg-backup`; Gitea: https://gitea.jr-it-services.de/johannes/nextcloud-borg-backup.
|
||||||
|
|
||||||
Die Sicherung verwendet Borg, weil Borg inkrementelle, deduplizierte und verschlüsselte Archive erstellt. Dadurch bleiben tägliche Backups platzsparend und einzelne Dateien können aus älteren Ständen wiederhergestellt werden.
|
Am 26.10.2026 wurde das Ausbleiben täglicher Sicherungen untersucht. Der persistente User-Timer startete nach dem Login teilweise vor dem Desktop-rclone-Mount: am 26.10. Abbruch wegen leerer Quelle um 08:48:38, rclone-Start erst 08:48:54. Es gab keine Wiederholung. Zusätzlich waren ein FUSE-Abbruch am 15.10. und ein Verzeichnis-Cache von 72 Stunden konfiguriert. Das NAS war erreichbar; das letzte erfolgreiche Archiv stammte vom 23.10., nicht vom 13.10.
|
||||||
|
|
||||||
## Architektur
|
Version 2 entkoppelt die Sicherung vollständig vom Desktop-Mount:
|
||||||
|
|
||||||
Bestandteile:
|
```text
|
||||||
|
Nextcloud WebDAV (rclone Nextcloud:)
|
||||||
- `scripts/nextcloud-borg-backup`: Haupt-CLI für Backup, Restore, Suche, Mount, Check und Retention.
|
-> vollständige Inventur, Plausibilitätsprüfung
|
||||||
- `etc/config.example`: Beispielkonfiguration mit den produktiven Standardpfaden.
|
-> rclone sync in privaten lokalen Zwischenstand
|
||||||
- `systemd/user/nextcloud-borg-backup.service`: systemd User Service für einen Backup-Lauf.
|
-> Hash-/Downloadvergleich, unveränderte zweite Cloud-Inventur
|
||||||
- `systemd/user/nextcloud-borg-backup.timer`: täglicher Timer.
|
-> Borg create auf geprüftem NFS-Share
|
||||||
- `scripts/install.sh`: Installation in das Home-Verzeichnis des Users.
|
-> Dateizahl/Bytezahl vergleichen, Archivdaten verifizieren, Repository prüfen
|
||||||
- `scripts/uninstall.sh`: Entfernt Service, Timer und Executable, lässt Daten bewusst bestehen.
|
-> atomare Erfolgsmarkierung
|
||||||
|
-> Aufbewahrung und Kompaktierung
|
||||||
Die Anwendung läuft als User `johannes`. Das ist für diesen Fall sinnvoll, weil die Quelle ein rclone/FUSE-Mount unterhalb des Home-Verzeichnisses ist. Ein root-Service könnte je nach FUSE-Konfiguration keine stabilen Leserechte auf diesen Mount haben.
|
|
||||||
|
|
||||||
## Pfade
|
|
||||||
|
|
||||||
Standardkonfiguration:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
SOURCE_DIR="${HOME}/Nextcloud"
|
|
||||||
REPO_DIR="/net/thor/volume1/NetBackup/Nextcloud/borg-repo"
|
|
||||||
ARCHIVE_PREFIX="nextcloud"
|
|
||||||
RETENTION_WITHIN="4w"
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Lokale Laufzeitdaten:
|
Die Cloud ist immer die Quelle. Keine Uploads oder Löschungen in Nextcloud. Nur der eigene lokale Zwischenstand wird mit `rclone sync --delete-after` an Cloud-Löschungen angepasst. Bei Fehlern entsteht kein als erfolgreich markiertes Archiv und es wird keine Aufbewahrungsbereinigung gestartet. Vorhandene Borg-Archive bleiben nutzbar.
|
||||||
|
|
||||||
|
## Dateien und Zustände
|
||||||
|
|
||||||
|
| Pfad | Inhalt |
|
||||||
|
| --- | --- |
|
||||||
|
| `~/.local/bin/nextcloud-borg-backup` | Bash-CLI für Backup und vorhandene Restore-Befehle |
|
||||||
|
| `~/.local/libexec/nextcloud-borg-runner.py` | Python-Steuerung, keine externen Python-Pakete |
|
||||||
|
| `~/.config/nextcloud-borg-backup/config` | Vertrauenswürdige Bash-Konfiguration |
|
||||||
|
| `~/.config/nextcloud-borg-backup/passphrase` | Bestehende Borg-Passphrase, niemals regenerieren |
|
||||||
|
| `~/.local/share/nextcloud-borg-backup/staging/` | Unverschlüsselter, privater Cloud-Zwischenstand |
|
||||||
|
| `~/.local/share/nextcloud-borg-backup/staging.owner.json` | Bindung des Zwischenstands an die Cloud-Quelle |
|
||||||
|
| `~/.local/state/nextcloud-borg-backup/last-success.json` | Zeitpunkt, Archivname, Dateizahl und Bytezahl der bestätigten Sicherung |
|
||||||
|
| `~/.local/state/nextcloud-borg-backup/last-attempt.json` | Laufstatus und Fehlerphase |
|
||||||
|
| `~/.local/state/nextcloud-borg-backup/repository.json` | Bindung an die Borg-Repository-ID |
|
||||||
|
| `~/.local/state/nextcloud-borg-backup/verified-inventory.json` | Zuletzt geprüfte Datei-Inventur mit Größen, Zeiten und Serverhashes |
|
||||||
|
| `~/.local/state/nextcloud-borg-backup/logs/` | Ablaufprotokolle, neue Protokolle nach 90 Tagen bereinigt |
|
||||||
|
|
||||||
|
Vollständige Kindprozessausgaben stehen im systemd-Journal; die zusätzlichen Ablaufdateien enthalten Phasen und Status. Die Marker werden über temporäre Dateien, fsync und atomare Umbenennung geschrieben. Ein laufender Zwischenstand ist kein Wiederherstellungspunkt. Der letzte bestätigte Stand liegt im Borg-Repository.
|
||||||
|
|
||||||
|
## Schutzmaßnahmen
|
||||||
|
|
||||||
|
- Ausschließlich ein Lauf dank `flock`; auch manuelle Wartungsbefehle verwenden denselben Lock. Borg besitzt zusätzlich seinen eigenen Repository-Lock. Fremde Borg-Prozesse werden nicht durch automatisches `break-lock` übergangen.
|
||||||
|
- Der NFS-Share muss exakt `EXPECTED_NAS_SOURCE` mit `EXPECTED_NAS_FSTYPE` entsprechen. Für die Workstation: `thor:/volume1/NetBackup`, `nfs4`. Autofs wird vor der Prüfung durch einen begrenzten Zugriff ausgelöst. Backup legt keine Verzeichnisse unter `/net` und kein Repository an.
|
||||||
|
- Erste Quell-Plausibilisierung gegen das jüngste vorhandene Borg-Archiv; danach gegen den letzten bestätigten Stand. Rückgang von Dateizahl **oder** Bytezahl um mehr als `MAX_SOURCE_DROP_PERCENT=20` führt zum Abbruch vor der Synchronisierung. Leere Quelle wird immer abgewiesen.
|
||||||
|
- Staging muss lokal sein, darf keine Symlinks/Mounts enthalten und darf sich nicht mit State, Repository oder `~/Nextcloud` überschneiden. Ein bereits gefülltes, fremdes Verzeichnis wird nicht übernommen.
|
||||||
|
- Hashvergleich, wenn jede Quelldatei MD5/SHA1 anbietet. Dateien ohne Serverhash werden vor dem Sync mit `rclone copy --ignore-times` erneut gelesen und anschließend zusätzlich per `rclone check --download` verglichen. Leere Dateien sind bereits durch Größe null eindeutig. Bei Zeilenumbrüchen in ungehashten Dateinamen erfolgt ein vollständiger Bytevergleich aller Dateien, weil rclones Dateiliste solche Namen nicht darstellen kann. Das kostet zusätzlich Übertragungszeit, vermeidet aber den unsicheren Rückfall auf reine Größenvergleiche.
|
||||||
|
- Zusätzlich berechnet Python die lokalen SHA1-/MD5-Werte unabhängig von rclone und vergleicht jede verfügbare Cloud-Prüfsumme sowie die Dateigrößen.
|
||||||
|
- Inventur vor und nach Sync/Check muss exakt gleich sein, einschließlich Dateinamen, Größen, Zeitstempeln und vorhandenen Hashes. Änderungen während des Laufs führen zu einem späteren neuen Versuch.
|
||||||
|
- Borg-Warnungen zählen als Fehler. Jeder neue Archivstand durchläuft `borg check --archives-only --verify-data --glob-archives <Archiv>` und einen Repository-Check. Ältere Optionen `FULL_CHECK_VERIFY_DATA`, `FULL_CHECK_INTERVAL_DAYS`, `REPOSITORY_CHECK_AFTER_BACKUP`, `SOURCE_DIR`, `ONE_FILE_SYSTEM` und `EXCLUDE_CACHES` beeinflussen den neuen Backup-Pfad nicht mehr: jedes neue Archiv wird geprüft und alle Cloud-Dateien werden erfasst. Aktive alte Exclude-Regeln führen zu einem erklärenden Abbruch, statt unbemerkt Dateien auszulassen.
|
||||||
|
- Erst nach dieser Prüfung wird der Erfolg gespeichert. Ein anschließender Wartungsfehler lässt das bestätigte Archiv gültig; der nächste Timerlauf versucht die Wartung erneut. `--keep-within 4w --keep-last 7` schützt zusätzlich die letzten sieben Archivstände.
|
||||||
|
- Einzelne Kindprozesse sind auf sechs Stunden begrenzt; systemd begrenzt den Gesamtlauf auf zwölf Stunden und beendet die gesamte Prozessgruppe. Keine endlose Wiederholung innerhalb eines Backup-Prozesses. Das nächste 15-Minuten-Fenster versucht erneut.
|
||||||
|
|
||||||
|
## Zeitplanung und Warnungen
|
||||||
|
|
||||||
|
`nextcloud-borg-backup.timer` prüft alle 15 Minuten plus maximal 60 Sekunden Zufallsverzögerung, erstmals zwei Minuten nach Start des User Managers. Eine bestätigte Sicherung ist nach 24 Stunden wieder fällig. `Persistent=true` holt verpasste Kalendertermine nach. Bei einem aktiven Lauf startet systemd keinen zweiten gleichnamigen Service.
|
||||||
|
|
||||||
|
`nextcloud-borg-health.timer` prüft unabhängig stündlich sowie nach Start des User Managers das Alter. Nach 36 Stunden ohne bestätigten Stand meldet `health` Exitcode 2 und der Wächter versucht eine kritische Desktop-Benachrichtigung. Fehler beim Backup werden ebenfalls gemeldet; Benachrichtigungen werden sechs Stunden gedrosselt. Ohne Desktop-Benachrichtigungsdienst bleiben Journal und Exitstatus die Überwachung. Es wird keine E-Mail versandt.
|
||||||
|
|
||||||
|
Auf der Workstation ist `loginctl show-user johannes -p Linger` bereits `yes`. Der User Manager kann damit ohne Desktop-Anmeldung laufen. Voraussetzung bleiben eingeschaltete Workstation, erreichbare Cloud/NAS und ohne Dialog lesbare rclone-/Borg-Zugangsdaten. Bei ausgeschaltetem Rechner erfolgen weder Sicherung noch Warnung; beim nächsten Start wird nachgeholt.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
~/.config/nextcloud-borg-backup/config
|
nextcloud-borg-backup health
|
||||||
~/.config/nextcloud-borg-backup/passphrase
|
systemctl --user list-timers 'nextcloud-borg-*'
|
||||||
~/.config/nextcloud-borg-backup/excludes
|
journalctl --user -u nextcloud-borg-backup.service --since today
|
||||||
~/.local/state/nextcloud-borg-backup/logs/
|
journalctl --user -u nextcloud-borg-health.service --since today
|
||||||
~/Nextcloud-Restore/
|
cat ~/.local/state/nextcloud-borg-backup/last-attempt.json
|
||||||
```
|
```
|
||||||
|
|
||||||
## Installation
|
## Fehlerbehebung
|
||||||
|
|
||||||
|
**NAS nicht erreichbar / falscher Mount:** `findmnt -T /net/thor/volume1/NetBackup/Nextcloud/borg-repo` prüfen, autofs/NAS-Verbindung wiederherstellen. Niemals ein neues Repository an einem Ersatzpfad initialisieren. NFS-Hardmounts können bei Serverausfall im Kernel hängen; auch Prozess-Timeouts können einen ununterbrechbaren Kernelzugriff nicht sofort beenden.
|
||||||
|
|
||||||
|
**Cloud ändert sich während des Laufs:** Der nächste Timerlauf wiederholt. Bei dauernder hoher Änderungsrate ist eine ruhigere Zeit oder eine serverseitige Snapshot-Sicherung erforderlich. WebDAV liefert keinen atomaren Snapshot der gesamten Nextcloud. Lokale, noch nicht zur Cloud hochgeladene Desktop-Änderungen sind erst nach dem Upload Teil der Quelle.
|
||||||
|
|
||||||
|
**Quellrückgang / Löschgrenze:** Zuerst absichtliche Löschungen oder Umstrukturierungen prüfen und ältere Archive aufbewahren. Nur nach Prüfung `MAX_SOURCE_DROP_PERCENT` bzw. `MAX_STAGE_DELETES` in der privaten Konfiguration gezielt erhöhen, manuell sichern, anschließend Schutzgrenzen zurücksetzen. Diese Grenzen werden nicht automatisch aufgeweicht.
|
||||||
|
|
||||||
|
**Staging zu groß / Speicher knapp:** Keine Archive oder Staging-Daten während eines laufenden Backups löschen. Nach Stoppen des Services kann ausschließlich der Zwischenstand zusammen mit seiner Owner-Datei entfernt und später neu aufgebaut werden. Die Borg-Historie und Passphrase bleiben bestehen. Im Zweifel freien Speicher schaffen oder `STAGING_DIR` auf einen privaten lokalen Datenträger verschieben; die konservative Platzprüfung bleibt aktiv.
|
||||||
|
|
||||||
|
**Repository-ID geändert:** Erst klären, ob das richtige NAS und Repository vorliegen. `repository.json` nicht automatisch löschen. Bei bewusstem Repository-Wechsel State und Baseline nach Prüfung neu aufsetzen. Ein fehlendes oder beschädigtes State-JSON soll nicht still als Erfolg gelten.
|
||||||
|
|
||||||
|
**Abbruch / Stromausfall:** Ein unvollständiger Zwischenstand wird beim nächsten Lauf erneut synchronisiert. Ein Archiv vor der Erfolgsmarkierung gilt nicht als bestätigt und wird nicht als Anlass für Bereinigung verwendet. Borg-Checkpoint-Archive sind keine bestätigten Sicherungen.
|
||||||
|
|
||||||
|
## Wiederherstellung und Notfall
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cd /home/johannes/localdev/nextcloud-borg-backup
|
|
||||||
./scripts/install.sh
|
|
||||||
```
|
|
||||||
|
|
||||||
Der Installer:
|
|
||||||
|
|
||||||
1. kopiert die CLI nach `~/.local/bin/nextcloud-borg-backup`,
|
|
||||||
2. legt die Konfiguration an, falls sie noch nicht existiert,
|
|
||||||
3. erzeugt eine Borg-Passphrase, falls noch keine existiert,
|
|
||||||
4. installiert systemd-User-Service und Timer,
|
|
||||||
5. aktiviert den Timer.
|
|
||||||
|
|
||||||
Für automatische Läufe ohne aktive Anmeldung:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
sudo loginctl enable-linger johannes
|
|
||||||
```
|
|
||||||
|
|
||||||
## Erster Backup-Lauf
|
|
||||||
|
|
||||||
```bash
|
|
||||||
systemctl --user start nextcloud-borg-backup.service
|
|
||||||
```
|
|
||||||
|
|
||||||
Logs:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
journalctl --user -u nextcloud-borg-backup.service -f
|
|
||||||
ls -lah ~/.local/state/nextcloud-borg-backup/logs/
|
|
||||||
```
|
|
||||||
|
|
||||||
Status:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
nextcloud-borg-backup status
|
|
||||||
nextcloud-borg-backup list
|
nextcloud-borg-backup list
|
||||||
```
|
nextcloud-borg-backup restore "" "Dokumente/Beispiel.pdf" ~/Restore-Test
|
||||||
|
nextcloud-borg-backup restore nextcloud-2026-10-23_08-45-12 "Dokumente" ~/Restore-Test
|
||||||
## Zeitplanung
|
|
||||||
|
|
||||||
Der Timer läuft täglich:
|
|
||||||
|
|
||||||
```ini
|
|
||||||
OnCalendar=*-*-* 03:15:00
|
|
||||||
Persistent=true
|
|
||||||
RandomizedDelaySec=30m
|
|
||||||
```
|
|
||||||
|
|
||||||
`Persistent=true` sorgt dafür, dass ein verpasster Lauf nachgeholt wird, sobald der User-systemd-Manager wieder aktiv ist.
|
|
||||||
|
|
||||||
## Retention
|
|
||||||
|
|
||||||
Die Aufbewahrung ist auf vier Wochen ausgelegt:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
RETENTION_WITHIN="4w"
|
|
||||||
```
|
|
||||||
|
|
||||||
Nach jedem Backup läuft:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
borg prune --keep-within 4w
|
|
||||||
borg compact
|
|
||||||
borg check --repository-only
|
|
||||||
```
|
|
||||||
|
|
||||||
Damit bleiben alle Archive der letzten vier Wochen erhalten. Bei einem täglichen Timer sind dadurch mindestens tägliche Wiederherstellungspunkte über rund einen Monat verfügbar. Manuelle Zusatzläufe innerhalb der vier Wochen werden ebenfalls behalten.
|
|
||||||
|
|
||||||
## Resilienzmaßnahmen
|
|
||||||
|
|
||||||
Die Anwendung bricht bewusst ab, wenn eine Sicherheitsannahme nicht erfüllt ist:
|
|
||||||
|
|
||||||
- Quelle ist nicht erreichbar oder nicht lesbar.
|
|
||||||
- Quelle wirkt leer und `ALLOW_EMPTY_SOURCE=false`.
|
|
||||||
- Synology-Ziel ist nicht erreichbar oder nicht beschreibbar.
|
|
||||||
- Freier Platz am Ziel liegt unter `MIN_FREE_REPO_GB`.
|
|
||||||
- Borg-Passphrase fehlt bei verschlüsseltem Repository.
|
|
||||||
- Ein anderer Backup-Lauf hält bereits den Lock.
|
|
||||||
|
|
||||||
Weitere Maßnahmen:
|
|
||||||
|
|
||||||
- Borg-Repo wird bei Bedarf einmalig initialisiert.
|
|
||||||
- Borg verwendet deduplizierte Archive statt Vollkopien.
|
|
||||||
- Nach dem Backup läuft Retention und Kompaktierung.
|
|
||||||
- Nach dem Backup läuft ein Repository-Check.
|
|
||||||
- Optional kann regelmäßig ein vollständiger Datencheck aktiviert werden:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
FULL_CHECK_VERIFY_DATA="true"
|
|
||||||
FULL_CHECK_INTERVAL_DAYS="7"
|
|
||||||
```
|
|
||||||
|
|
||||||
Ein vollständiger `borg check --verify-data` kann bei großen Repositories lange dauern.
|
|
||||||
|
|
||||||
## Dateien suchen
|
|
||||||
|
|
||||||
Über alle Archive suchen:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
nextcloud-borg-backup search "Steuer"
|
|
||||||
```
|
|
||||||
|
|
||||||
Nur in einem bestimmten Archiv suchen:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
nextcloud-borg-backup search "Steuer" nextcloud-2026-07-10_03-15-00
|
|
||||||
```
|
|
||||||
|
|
||||||
Die Ausgabe enthält bei archivübergreifender Suche den Archivnamen und den Pfad.
|
|
||||||
|
|
||||||
## Archive mounten
|
|
||||||
|
|
||||||
Alle Archive als lesbare Struktur mounten:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
nextcloud-borg-backup mount
|
nextcloud-borg-backup mount
|
||||||
```
|
|
||||||
|
|
||||||
Standard-Mountpunkt:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
${XDG_RUNTIME_DIR}/nextcloud-borg-backup-mount
|
|
||||||
```
|
|
||||||
|
|
||||||
Ein einzelnes Archiv mounten:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
nextcloud-borg-backup mount nextcloud-2026-07-10_03-15-00
|
|
||||||
```
|
|
||||||
|
|
||||||
Unmount:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
nextcloud-borg-backup umount
|
nextcloud-borg-backup umount
|
||||||
```
|
```
|
||||||
|
|
||||||
Falls `borg mount` nicht verfügbar ist, muss auf Arch Linux das passende FUSE-Paket installiert sein.
|
Ziel ist standardmäßig `~/Nextcloud-Restore/<Archivname>`, niemals automatisch die produktive Cloud. Wiederhergestellte Dateien prüfen und erst danach gezielt zurückkopieren.
|
||||||
|
|
||||||
## Restore
|
Für ein neues System braucht man Borg 1.x, Repository und die originale Passphrase bzw. den exportierten Schlüssel. Sicherer zusätzlicher Export:
|
||||||
|
|
||||||
Neuestes Archiv komplett wiederherstellen:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
nextcloud-borg-backup restore
|
BORG_PASSCOMMAND='cat /sicherer/pfad/passphrase' borg key export /net/thor/volume1/NetBackup/Nextcloud/borg-repo /sicherer/pfad/borg-key.txt
|
||||||
```
|
```
|
||||||
|
|
||||||
Ziel:
|
Passphrase und exportierten Schlüssel unabhängig vom NAS und der Workstation sicher verwahren. Keine Zugangsdaten in Git, Logs oder Chat kopieren.
|
||||||
|
|
||||||
```bash
|
## Tests und Installation
|
||||||
~/Nextcloud-Restore/<Archivname>/
|
|
||||||
```
|
|
||||||
|
|
||||||
Bestimmten Pfad aus dem neuesten Archiv wiederherstellen:
|
`make check` prüft Bash-Syntax, Python, alle vier systemd-Units und automatisierte Ausfallszenarien. Integrationstests verwenden echtes rclone mit lokalem Test-Remote sowie echtes Borg in temporären Verzeichnissen: Archiv erstellen, Daten prüfen, extrahieren und Bytevergleich. Sie verändern weder echte Cloud noch produktives Repository.
|
||||||
|
|
||||||
```bash
|
15 Tests prüfen außerdem leere/geschrumpfte Quelle, falsches NAS, fremdes/symlinkendes Staging, Netzfehler in Sync und Check, geänderte Cloud-Inventur, Repository-Austausch, konkurrierende Läufe, Timeout, Zeitrücksprung und Wartungswiederholung. Gleiche Dateigröße und gleicher Zeitstempel bei geändertem Inhalt werden im Integrationstest ausdrücklich abgedeckt.
|
||||||
nextcloud-borg-backup restore "" "Dokumente/Projekt"
|
|
||||||
```
|
|
||||||
|
|
||||||
Bestimmten Pfad aus einem bestimmten Archiv wiederherstellen:
|
Der Installer aktualisiert Programme und Units, erhält private Konfiguration und Schlüssel und startet die Timer neu. Eine fehlende Passphrase bei vorhandenem Repository führt zum Abbruch. Die Deinstallation stoppt Backup/Timer und entfernt Programme/Units; Daten, Schlüssel, State und Archive bleiben erhalten.
|
||||||
|
|
||||||
```bash
|
## Grenzen
|
||||||
nextcloud-borg-backup restore nextcloud-2026-07-10_03-15-00 "Dokumente/Projekt" ~/Restore-Test
|
|
||||||
```
|
|
||||||
|
|
||||||
Wichtig: Die Anwendung extrahiert standardmäßig nicht direkt zurück in den produktiven Nextcloud-Ordner. Das reduziert das Risiko, versehentlich aktuelle Dateien zu überschreiben. Nach Prüfung können die wiederhergestellten Dateien gezielt zurückkopiert werden.
|
Die Sicherung schützt die durch das konfigurierte Nextcloud-Konto erreichbaren Dateien. Nextcloud-Datenbank, App-Konfiguration, Freigaberechte und serverseitige Versions-/Papierkorb-Metadaten sind kein Bestandteil dieser Datei-Sicherung. Ein vollständiges Nextcloud-Server-Backup bleibt separat erforderlich. Aktuelle Schäden oder Ransomware können als neue Dateiversion übernommen werden; die vierwöchige Historie und Mindestanzahl ersetzen kein unabhängiges unveränderbares Offsite-Backup.
|
||||||
|
|
||||||
## Disaster Recovery
|
## Offizielle Referenzen
|
||||||
|
|
||||||
Für eine Wiederherstellung auf einem neuen System werden benötigt:
|
- [rclone sync: Zielrichtung und Löschverhalten](https://rclone.org/commands/rclone_sync/)
|
||||||
|
- [rclone check: Hash- und Downloadvergleich](https://rclone.org/commands/rclone_check/)
|
||||||
|
- [Borg 1.4 create: Warnungen und Fehler](https://borgbackup.readthedocs.io/en/1.4.5/usage/create.html)
|
||||||
|
- [Borg check](https://borgbackup.readthedocs.io/en/stable/usage/check.html)
|
||||||
|
|
||||||
1. das Borg-Repository unter `/net/thor/volume1/NetBackup/Nextcloud/borg-repo`,
|
## Einrichtung auf der Workstation am 26.10.2026
|
||||||
2. die Borg-Passphrase aus `~/.config/nextcloud-borg-backup/passphrase`,
|
|
||||||
3. Borg selbst.
|
|
||||||
|
|
||||||
Minimaler Restore:
|
Bestehende Konfiguration und Passphrase wurden beibehalten. Die neuen Einstellungen wurden ausdrücklich ergänzt; die vorherige Konfiguration liegt privat unter `~/.config/nextcloud-borg-backup/config.pre-v2-20261026`. Beide Timer sind aktiviert. Das vorhandene Linger blieb aktiv. Der Desktop-Mount und dessen Autostart wurden nicht umgebaut.
|
||||||
|
|
||||||
```bash
|
Für die Erstabnahme wurde der private Zwischenstand aus dem vorhandenen Archiv vom 23.10. vorgefüllt. Der anschließende direkte Cloud-Abgleich und sämtliche Prüfungen bleiben dadurch unverändert erforderlich; ein aus dem Altarchiv gelesener Stand gilt nicht automatisch als aktuelle Sicherung. Der beobachtete aktuelle Cloud-Bestand umfasst 8.218 Dateien mit 38.274.641.627 Bytes.
|
||||||
export BORG_PASSCOMMAND='cat /pfad/zur/passphrase'
|
|
||||||
borg list /net/thor/volume1/NetBackup/Nextcloud/borg-repo
|
|
||||||
borg extract /net/thor/volume1/NetBackup/Nextcloud/borg-repo::ARCHIVNAME
|
|
||||||
```
|
|
||||||
|
|
||||||
Die Passphrase sollte separat sicher abgelegt werden. Ohne Passphrase ist ein verschlüsseltes Repository nicht nutzbar.
|
|
||||||
|
|
||||||
## Wartung
|
|
||||||
|
|
||||||
Manueller Repository-Check:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
nextcloud-borg-backup check
|
|
||||||
```
|
|
||||||
|
|
||||||
Vollständiger Datencheck:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
nextcloud-borg-backup check --verify-data
|
|
||||||
```
|
|
||||||
|
|
||||||
Borg-Lock lösen, falls ein Lauf hart abgebrochen wurde und sicher kein Backup mehr läuft:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
nextcloud-borg-backup break-lock
|
|
||||||
```
|
|
||||||
|
|
||||||
Timer deaktivieren:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
systemctl --user disable --now nextcloud-borg-backup.timer
|
|
||||||
```
|
|
||||||
|
|
||||||
Deinstallation der lokalen Anwendung:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
./scripts/uninstall.sh
|
|
||||||
```
|
|
||||||
|
|
||||||
Die Deinstallation löscht bewusst nicht:
|
|
||||||
|
|
||||||
- Borg-Repository,
|
|
||||||
- Passphrase,
|
|
||||||
- Konfiguration,
|
|
||||||
- Logs,
|
|
||||||
- Restore-Verzeichnis.
|
|
||||||
|
|
||||||
## Gitea-Upload
|
|
||||||
|
|
||||||
Das Verzeichnis `/home/johannes/localdev/nextcloud-borg-backup` ist als eigenständiges Projekt vorbereitet. Vor dem Push:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cd /home/johannes/localdev/nextcloud-borg-backup
|
|
||||||
git init
|
|
||||||
git add .
|
|
||||||
git commit -m "Initial nextcloud borg backup application"
|
|
||||||
git remote add origin <gitea-url>
|
|
||||||
git push -u origin main
|
|
||||||
```
|
|
||||||
|
|
||||||
Keine lokalen Geheimnisse werden im Projekt abgelegt. Die produktive Passphrase entsteht erst bei der Installation unter `~/.config/nextcloud-borg-backup/passphrase`.
|
|
||||||
@@ -1,8 +1,13 @@
|
|||||||
.PHONY: check install uninstall
|
.PHONY: check test install uninstall
|
||||||
|
|
||||||
check:
|
check:
|
||||||
bash -n scripts/nextcloud-borg-backup scripts/install.sh scripts/uninstall.sh
|
bash -n scripts/nextcloud-borg-backup scripts/install.sh scripts/uninstall.sh
|
||||||
systemd-analyze --user verify systemd/user/nextcloud-borg-backup.service systemd/user/nextcloud-borg-backup.timer
|
python3 -m py_compile scripts/nextcloud-borg-runner.py
|
||||||
|
systemd-analyze --user verify systemd/user/nextcloud-borg-backup.service systemd/user/nextcloud-borg-backup.timer systemd/user/nextcloud-borg-health.service systemd/user/nextcloud-borg-health.timer
|
||||||
|
python3 -m unittest discover -s tests -v
|
||||||
|
|
||||||
|
test:
|
||||||
|
python3 -m unittest discover -s tests -v
|
||||||
|
|
||||||
install:
|
install:
|
||||||
./scripts/install.sh
|
./scripts/install.sh
|
||||||
|
|||||||
@@ -1,94 +1,52 @@
|
|||||||
# nextcloud-borg-backup
|
# nextcloud-borg-backup
|
||||||
|
|
||||||
Tägliches Borg-Backup des lokalen Nextcloud-Mounts nach Synology:
|
Verifizierte, versionierte Sicherung von Nextcloud auf das Synology-NAS. Die Workstation liest `Nextcloud:` direkt mit rclone, prüft einen privaten lokalen Zwischenstand und erstellt daraus verschlüsselte Borg-Archive auf `/net/thor/volume1/NetBackup/Nextcloud/borg-repo`. Der Desktop-Mount `~/Nextcloud` ist unabhängig davon.
|
||||||
|
|
||||||
- Quelle: `/home/johannes/Nextcloud`
|
Die Übertragung geht ausschließlich **von Nextcloud zum NAS**. Dateien werden in Nextcloud weder geändert noch gelöscht. Das NAS enthält eine Borg-Historie, keinen gewöhnlichen Dateispiegel; gelöschte oder überschriebene Dateien bleiben aus älteren Archiven wiederherstellbar.
|
||||||
- Ziel: `/net/thor/volume1/NetBackup/Nextcloud/borg-repo`
|
|
||||||
- Historie: `4w`, also vier Wochen wiederherstellbare Archive
|
|
||||||
- Betrieb: systemd User Timer, täglich um 03:15 Uhr mit Zufallsverzögerung
|
|
||||||
- Restore: über CLI, Suche, Borg-Mount oder gezieltes Extrahieren
|
|
||||||
|
|
||||||
Das Projekt ist bewusst klein gehalten: eine Bash-CLI, Konfiguration, systemd-User-Units und Dokumentation. Es eignet sich direkt für ein Gitea-Repository.
|
## Schutz und Zeitplanung
|
||||||
|
|
||||||
## Installation
|
- Alle 15 Minuten wird geprüft, ob eine Sicherung fällig ist; nach Erfolg frühestens nach 24 Stunden. Fehler werden beim nächsten Timerlauf erneut versucht, auch nach Neustart oder Aufwachen.
|
||||||
|
- Eigene direkte Cloud-Verbindung ohne FUSE-Verzeichnis-Cache. Vollständige Inventur vor und nach der Übertragung; Abbruch, wenn sich die Quelle währenddessen ändert.
|
||||||
|
- SHA1/MD5-Prüfung des Zwischenstands gegen Nextcloud. Dateien ohne Serverhash werden vollständig neu gelesen und per Downloadvergleich geprüft.
|
||||||
|
- Vor und nach der Cloud-Übertragung: Prüfung des tatsächlichen NFS-Mounts, des vorhandenen Repositorys und des freien Speicherplatzes. Keine automatische Repository-Neuanlage im Backup.
|
||||||
|
- Schutz vor leerer Quelle, Rückgang von Dateizahl oder Datenmenge um mehr als 20 Prozent und mehr als 1.000 Löschungen im lokalen Zwischenstand.
|
||||||
|
- Jeder neue Borg-Stand wird vollständig auf Lesbarkeit und Datenintegrität geprüft; Dateizahl und Bytezahl müssen zur Cloud-Inventur passen. Erst dann gilt der Lauf als bestätigt.
|
||||||
|
- Vier Wochen Historie, zusätzlich mindestens sieben Archive; Bereinigung ausschließlich nach erfolgreicher Sicherung. Fehlgeschlagene Wartung wird separat erneut versucht.
|
||||||
|
- Unabhängiger stündlicher Alterswächter: Desktop-Warnung bei mehr als 36 Stunden ohne bestätigte Sicherung. Warnungen maximal alle sechs Stunden, Fehler und Details im Journal.
|
||||||
|
|
||||||
|
## Installation / Update
|
||||||
|
|
||||||
|
Voraussetzungen: Linux, systemd User Manager, Python 3, Borg **1.x**, rclone, util-linux (`findmnt`, `flock`), GNU coreutils und optional `notify-send`. rclone muss `Nextcloud:` bereits authentifiziert kennen; das NAS wird durch das vorhandene autofs eingebunden.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cd /home/johannes/localdev/nextcloud-borg-backup
|
cd ~/localdev/nextcloud-borg-backup
|
||||||
|
make check
|
||||||
./scripts/install.sh
|
./scripts/install.sh
|
||||||
```
|
systemctl --user start --no-block nextcloud-borg-backup.service
|
||||||
|
|
||||||
Der Installer legt an:
|
|
||||||
|
|
||||||
- `~/.local/bin/nextcloud-borg-backup`
|
|
||||||
- `~/.config/nextcloud-borg-backup/config`
|
|
||||||
- `~/.config/nextcloud-borg-backup/passphrase`
|
|
||||||
- `~/.config/systemd/user/nextcloud-borg-backup.service`
|
|
||||||
- `~/.config/systemd/user/nextcloud-borg-backup.timer`
|
|
||||||
|
|
||||||
Der Installer überschreibt eine vorhandene Konfiguration nicht.
|
|
||||||
|
|
||||||
Danach den ersten Lauf manuell starten:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
systemctl --user start nextcloud-borg-backup.service
|
|
||||||
journalctl --user -u nextcloud-borg-backup.service -f
|
journalctl --user -u nextcloud-borg-backup.service -f
|
||||||
```
|
```
|
||||||
|
|
||||||
Timer prüfen:
|
Bestehende Konfiguration, Passphrase und Historie bleiben erhalten. Neue Einstellungen haben kompatible Standardwerte; siehe [etc/config.example](etc/config.example). Für neue Installationen: Konfiguration und NAS prüfen, das Repository einmal ausdrücklich mit `nextcloud-borg-backup init` anlegen. Das Backup selbst legt niemals ein Ersatzrepository an.
|
||||||
|
|
||||||
|
Der lokale Zwischenstand benötigt ungefähr die Datenmenge der Cloud. Zusätzlich fordert der konservative Speichercheck vor jedem Lauf noch einmal diese Datenmenge plus 5 GiB freien Speicher. Ein Zwischenstand gehört auf einen privaten lokalen Datenträger, niemals auf den Desktop-Mount.
|
||||||
|
|
||||||
|
## Betrieb und Wiederherstellung
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
systemctl --user status nextcloud-borg-backup.timer
|
nextcloud-borg-backup health # JSON, Exit 0: aktuell, Exit 2: überfällig
|
||||||
systemctl --user list-timers nextcloud-borg-backup.timer
|
nextcloud-borg-backup backup # sofortiger manueller Lauf
|
||||||
```
|
nextcloud-borg-backup backup --if-due # wie der Timer
|
||||||
|
|
||||||
Damit der User-Timer auch ohne aktive Anmeldung läuft:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
sudo loginctl enable-linger johannes
|
|
||||||
```
|
|
||||||
|
|
||||||
## Bedienung
|
|
||||||
|
|
||||||
```bash
|
|
||||||
nextcloud-borg-backup backup
|
|
||||||
nextcloud-borg-backup list
|
nextcloud-borg-backup list
|
||||||
nextcloud-borg-backup status
|
nextcloud-borg-backup info
|
||||||
nextcloud-borg-backup search Rechnung
|
systemctl --user list-timers 'nextcloud-borg-*'
|
||||||
|
nextcloud-borg-backup restore "" "Dokumente/Beispiel.pdf" ~/Restore-Test
|
||||||
nextcloud-borg-backup mount
|
nextcloud-borg-backup mount
|
||||||
nextcloud-borg-backup restore
|
|
||||||
nextcloud-borg-backup check --verify-data
|
|
||||||
```
|
|
||||||
|
|
||||||
## Restore-Beispiele
|
|
||||||
|
|
||||||
Neuestes Archiv komplett nach `~/Nextcloud-Restore/<Archivname>` wiederherstellen:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
nextcloud-borg-backup restore
|
|
||||||
```
|
|
||||||
|
|
||||||
Bestimmte Datei im neuesten Archiv wiederherstellen:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
nextcloud-borg-backup restore "" "Dokumente/Beispiel.pdf"
|
|
||||||
```
|
|
||||||
|
|
||||||
Bestimmtes Archiv und bestimmten Pfad wiederherstellen:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
nextcloud-borg-backup restore nextcloud-2026-07-10_03-15-00 "Dokumente"
|
|
||||||
```
|
|
||||||
|
|
||||||
Archive bequem lesbar mounten:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
nextcloud-borg-backup mount
|
|
||||||
xdg-open "${XDG_RUNTIME_DIR}/nextcloud-borg-backup-mount"
|
|
||||||
nextcloud-borg-backup umount
|
nextcloud-borg-backup umount
|
||||||
```
|
```
|
||||||
|
|
||||||
## Wichtige Hinweise
|
Eine manuelle Sofortsicherung kann lange dauern. Das Archiv wird erst nach erfolgreicher Datenprüfung als bestätigt markiert. `health` zeigt auch die letzte fehlgeschlagene Phase; `status` zeigt die herkömmlichen Borg-Informationen.
|
||||||
|
|
||||||
Die Borg-Passphrase liegt lokal in `~/.config/nextcloud-borg-backup/passphrase`. Ohne diese Datei kann ein verschlüsseltes Borg-Repository im Notfall nicht wiederhergestellt werden. Eine Kopie gehört daher separat und sicher abgelegt.
|
Die Passphrase in `~/.config/nextcloud-borg-backup/passphrase` und ein Borg-Schlüsselexport gehören zusätzlich an einen sicheren unabhängigen Ort. Niemals ins Repository einchecken. Der lokale Zwischenstand ist unverschlüsselt; seine Dateirechte sind privat, eine Datenträgerverschlüsselung richtet dieses Projekt nicht ein.
|
||||||
|
|
||||||
Weitere Details stehen in [Dokumentation.md](Dokumentation.md).
|
Details, Fehlerbehandlung, Grenzen und Abnahme: [Dokumentation.md](Dokumentation.md).
|
||||||
+15
-32
@@ -1,44 +1,27 @@
|
|||||||
# nextcloud-borg-backup configuration
|
# Trusted Bash configuration; never commit credentials or passphrases.
|
||||||
#
|
REMOTE="Nextcloud:"
|
||||||
# Copy to:
|
STAGING_DIR="${HOME}/.local/share/nextcloud-borg-backup/staging"
|
||||||
# ~/.config/nextcloud-borg-backup/config
|
|
||||||
|
|
||||||
SOURCE_DIR="${HOME}/Nextcloud"
|
|
||||||
REPO_DIR="/net/thor/volume1/NetBackup/Nextcloud/borg-repo"
|
REPO_DIR="/net/thor/volume1/NetBackup/Nextcloud/borg-repo"
|
||||||
|
EXPECTED_NAS_SOURCE="thor:/volume1/NetBackup"
|
||||||
|
EXPECTED_NAS_FSTYPE="nfs4"
|
||||||
ARCHIVE_PREFIX="nextcloud"
|
ARCHIVE_PREFIX="nextcloud"
|
||||||
|
|
||||||
# Borg repository encryption. repokey-blake2 needs a passphrase file for
|
|
||||||
# unattended systemd timer runs. The installer creates this file if missing.
|
|
||||||
BORG_ENCRYPTION="repokey-blake2"
|
BORG_ENCRYPTION="repokey-blake2"
|
||||||
BORG_PASSPHRASE_FILE="${HOME}/.config/nextcloud-borg-backup/passphrase"
|
BORG_PASSPHRASE_FILE="${HOME}/.config/nextcloud-borg-backup/passphrase"
|
||||||
BORG_COMPRESSION="zstd,6"
|
BORG_COMPRESSION="zstd,6"
|
||||||
|
|
||||||
# Four weeks of restorable history. With the daily timer this keeps at least
|
|
||||||
# one month of archives; manual extra runs inside the window are also retained.
|
|
||||||
RETENTION_WITHIN="4w"
|
RETENTION_WITHIN="4w"
|
||||||
|
PRUNE_AFTER_BACKUP="true"
|
||||||
# Safety checks.
|
COMPACT_AFTER_PRUNE="true"
|
||||||
ALLOW_EMPTY_SOURCE="false"
|
BACKUP_INTERVAL_HOURS="24"
|
||||||
|
MAX_BACKUP_AGE_HOURS="36"
|
||||||
|
MIN_SOURCE_FILES="1"
|
||||||
|
MAX_SOURCE_DROP_PERCENT="20"
|
||||||
|
MAX_STAGE_DELETES="1000"
|
||||||
|
MIN_FREE_STAGE_GB="5"
|
||||||
MIN_FREE_REPO_GB="5"
|
MIN_FREE_REPO_GB="5"
|
||||||
PRECHECK_TIMEOUT_SECONDS="60"
|
COMMAND_TIMEOUT_SECONDS="21600"
|
||||||
|
|
||||||
# Runtime paths.
|
|
||||||
LOG_DIR="${HOME}/.local/state/nextcloud-borg-backup/logs"
|
LOG_DIR="${HOME}/.local/state/nextcloud-borg-backup/logs"
|
||||||
STATE_DIR="${HOME}/.local/state/nextcloud-borg-backup"
|
STATE_DIR="${HOME}/.local/state/nextcloud-borg-backup"
|
||||||
|
LOCK_FILE="${STATE_DIR}/nextcloud-borg-backup.lock"
|
||||||
RESTORE_DIR="${HOME}/Nextcloud-Restore"
|
RESTORE_DIR="${HOME}/Nextcloud-Restore"
|
||||||
MOUNT_DIR="${XDG_RUNTIME_DIR:-/tmp}/nextcloud-borg-backup-mount"
|
MOUNT_DIR="${XDG_RUNTIME_DIR:-/tmp}/nextcloud-borg-backup-mount"
|
||||||
EXCLUDE_FILE="${HOME}/.config/nextcloud-borg-backup/excludes"
|
EXCLUDE_FILE="${HOME}/.config/nextcloud-borg-backup/excludes"
|
||||||
LOCK_FILE="${HOME}/.local/state/nextcloud-borg-backup/nextcloud-borg-backup.lock"
|
|
||||||
|
|
||||||
# Borg behavior.
|
|
||||||
PRUNE_AFTER_BACKUP="true"
|
|
||||||
COMPACT_AFTER_PRUNE="true"
|
|
||||||
REPOSITORY_CHECK_AFTER_BACKUP="true"
|
|
||||||
FULL_CHECK_INTERVAL_DAYS="7"
|
|
||||||
FULL_CHECK_VERIFY_DATA="false"
|
|
||||||
EXCLUDE_CACHES="true"
|
|
||||||
ONE_FILE_SYSTEM="true"
|
|
||||||
|
|
||||||
# Leave empty unless the rclone mount exposes unstable metadata and Borg rereads
|
|
||||||
# too much. Conservative alternative: BORG_FILES_CACHE="mtime,size"
|
|
||||||
BORG_FILES_CACHE=""
|
|
||||||
+37
-13
@@ -10,8 +10,17 @@ CONFIG_DIR="${XDG_CONFIG_HOME:-${HOME}/.config}/${APP_NAME}"
|
|||||||
STATE_DIR="${XDG_STATE_HOME:-${HOME}/.local/state}/${APP_NAME}"
|
STATE_DIR="${XDG_STATE_HOME:-${HOME}/.local/state}/${APP_NAME}"
|
||||||
SYSTEMD_USER_DIR="${XDG_CONFIG_HOME:-${HOME}/.config}/systemd/user"
|
SYSTEMD_USER_DIR="${XDG_CONFIG_HOME:-${HOME}/.config}/systemd/user"
|
||||||
|
|
||||||
mkdir -p "$BIN_DIR" "$CONFIG_DIR" "$STATE_DIR/logs" "$SYSTEMD_USER_DIR"
|
for required in python3 borg rclone findmnt flock timeout systemctl; do
|
||||||
|
command -v "$required" >/dev/null || { printf 'Missing dependency: %s\n' "$required" >&2; exit 1; }
|
||||||
|
done
|
||||||
|
[[ "$(borg --version)" == 'borg 1.'* ]] || { printf 'Borg 1.x required\n' >&2; exit 1; }
|
||||||
|
|
||||||
|
EXISTING_CONFIG=false
|
||||||
|
[[ ! -f "$CONFIG_DIR/config" ]] || EXISTING_CONFIG=true
|
||||||
|
|
||||||
|
mkdir -p "${HOME}/.local/libexec" "$BIN_DIR" "$CONFIG_DIR" "$STATE_DIR/logs" "$SYSTEMD_USER_DIR"
|
||||||
|
|
||||||
|
install -m 0750 "$ROOT_DIR/scripts/nextcloud-borg-runner.py" "${HOME}/.local/libexec/nextcloud-borg-runner.py"
|
||||||
install -m 0750 "$ROOT_DIR/scripts/nextcloud-borg-backup" "$BIN_DIR/nextcloud-borg-backup"
|
install -m 0750 "$ROOT_DIR/scripts/nextcloud-borg-backup" "$BIN_DIR/nextcloud-borg-backup"
|
||||||
|
|
||||||
if [[ ! -f "$CONFIG_DIR/config" ]]; then
|
if [[ ! -f "$CONFIG_DIR/config" ]]; then
|
||||||
@@ -25,23 +34,38 @@ if [[ ! -f "$CONFIG_DIR/excludes" ]]; then
|
|||||||
install -m 0600 "$ROOT_DIR/etc/excludes.example" "$CONFIG_DIR/excludes"
|
install -m 0600 "$ROOT_DIR/etc/excludes.example" "$CONFIG_DIR/excludes"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
if [[ ! -f "$CONFIG_DIR/passphrase" ]]; then
|
# Respect installations that store the original key at a custom path.
|
||||||
if command -v openssl >/dev/null 2>&1; then
|
KEY_PATH="$(source "$CONFIG_DIR/config"; printf '%s' "${BORG_PASSPHRASE_FILE:-$CONFIG_DIR/passphrase}")"
|
||||||
openssl rand -base64 48 > "$CONFIG_DIR/passphrase"
|
if [[ ! -f "$KEY_PATH" ]]; then
|
||||||
else
|
if [[ "$EXISTING_CONFIG" == true ]]; then
|
||||||
od -An -tx1 -N48 /dev/urandom | tr -d ' \n' > "$CONFIG_DIR/passphrase"
|
printf 'Existing installation but missing default passphrase. Restore the key or check the configured key path before updating.\n' >&2
|
||||||
printf '\n' >> "$CONFIG_DIR/passphrase"
|
exit 1
|
||||||
fi
|
fi
|
||||||
chmod 0600 "$CONFIG_DIR/passphrase"
|
# Never replace a missing key for an existing encrypted repository with random data.
|
||||||
printf 'Created Borg passphrase file: %s\n' "$CONFIG_DIR/passphrase"
|
if [[ -f "$CONFIG_DIR/config" ]]; then
|
||||||
|
source "$CONFIG_DIR/config"
|
||||||
|
if timeout 30 test -f "${REPO_DIR:-/net/thor/volume1/NetBackup/Nextcloud/borg-repo}/config"; then
|
||||||
|
printf 'Existing repository but missing passphrase; restore the original key first.\n' >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
if command -v openssl >/dev/null 2>&1; then
|
||||||
|
openssl rand -base64 48 > "$KEY_PATH"
|
||||||
|
else
|
||||||
|
od -An -tx1 -N48 /dev/urandom | tr -d ' \n' > "$KEY_PATH"
|
||||||
|
printf '\n' >> "$KEY_PATH"
|
||||||
|
fi
|
||||||
|
chmod 0600 "$KEY_PATH"
|
||||||
|
printf 'Created Borg passphrase file: %s\n' "$KEY_PATH"
|
||||||
printf 'Store a copy of this passphrase somewhere safe for disaster recovery.\n'
|
printf 'Store a copy of this passphrase somewhere safe for disaster recovery.\n'
|
||||||
fi
|
fi
|
||||||
|
|
||||||
install -m 0644 "$ROOT_DIR/systemd/user/nextcloud-borg-backup.service" "$SYSTEMD_USER_DIR/nextcloud-borg-backup.service"
|
for unit in nextcloud-borg-backup.service nextcloud-borg-backup.timer nextcloud-borg-health.service nextcloud-borg-health.timer; do
|
||||||
install -m 0644 "$ROOT_DIR/systemd/user/nextcloud-borg-backup.timer" "$SYSTEMD_USER_DIR/nextcloud-borg-backup.timer"
|
install -m 0644 "$ROOT_DIR/systemd/user/$unit" "$SYSTEMD_USER_DIR/$unit"
|
||||||
|
done
|
||||||
systemctl --user daemon-reload
|
systemctl --user daemon-reload
|
||||||
systemctl --user enable --now nextcloud-borg-backup.timer
|
systemctl --user enable nextcloud-borg-backup.timer nextcloud-borg-health.timer
|
||||||
|
systemctl --user restart nextcloud-borg-backup.timer nextcloud-borg-health.timer
|
||||||
|
|
||||||
printf '\nInstalled %s.\n' "$APP_NAME"
|
printf '\nInstalled %s.\n' "$APP_NAME"
|
||||||
printf 'Next steps:\n'
|
printf 'Next steps:\n'
|
||||||
|
|||||||
+36
-127
@@ -3,7 +3,7 @@ set -Eeuo pipefail
|
|||||||
umask 077
|
umask 077
|
||||||
|
|
||||||
APP_NAME="nextcloud-borg-backup"
|
APP_NAME="nextcloud-borg-backup"
|
||||||
APP_VERSION="1.0.0"
|
APP_VERSION="2.0.0"
|
||||||
|
|
||||||
CONFIG_FILE="${NCB_CONFIG:-${XDG_CONFIG_HOME:-${HOME}/.config}/${APP_NAME}/config}"
|
CONFIG_FILE="${NCB_CONFIG:-${XDG_CONFIG_HOME:-${HOME}/.config}/${APP_NAME}/config}"
|
||||||
|
|
||||||
@@ -30,10 +30,11 @@ shell_quote() {
|
|||||||
|
|
||||||
usage() {
|
usage() {
|
||||||
cat <<'EOF'
|
cat <<'EOF'
|
||||||
nextcloud-borg-backup - Borg backup helper for a mounted Nextcloud folder
|
nextcloud-borg-backup - Borg backup verified cloud-to-NAS backup and restore helper
|
||||||
|
|
||||||
Usage:
|
Usage:
|
||||||
nextcloud-borg-backup backup
|
nextcloud-borg-backup backup [--if-due]
|
||||||
|
nextcloud-borg-backup health [--notify]
|
||||||
nextcloud-borg-backup init
|
nextcloud-borg-backup init
|
||||||
nextcloud-borg-backup list
|
nextcloud-borg-backup list
|
||||||
nextcloud-borg-backup info [ARCHIVE]
|
nextcloud-borg-backup info [ARCHIVE]
|
||||||
@@ -101,23 +102,6 @@ ensure_dirs() {
|
|||||||
mkdir -p "$LOG_DIR" "$STATE_DIR"
|
mkdir -p "$LOG_DIR" "$STATE_DIR"
|
||||||
}
|
}
|
||||||
|
|
||||||
log_file_for_run() {
|
|
||||||
local timestamp
|
|
||||||
timestamp="$(date '+%Y-%m-%d_%H-%M-%S')"
|
|
||||||
printf '%s/%s_%s.log' "$LOG_DIR" "$APP_NAME" "$timestamp"
|
|
||||||
}
|
|
||||||
|
|
||||||
start_logging() {
|
|
||||||
ensure_dirs
|
|
||||||
local log_file
|
|
||||||
log_file="$(log_file_for_run)"
|
|
||||||
exec > >(tee -a "$log_file") 2>&1
|
|
||||||
info "== ${APP_NAME} ${APP_VERSION} =="
|
|
||||||
info "-- Time: $(date -Is)"
|
|
||||||
info "-- Config: $CONFIG_FILE"
|
|
||||||
info "-- Log: $log_file"
|
|
||||||
}
|
|
||||||
|
|
||||||
with_lock() {
|
with_lock() {
|
||||||
exec 9>"$LOCK_FILE"
|
exec 9>"$LOCK_FILE"
|
||||||
flock -n 9 || die "Another ${APP_NAME} run is already active: $LOCK_FILE"
|
flock -n 9 || die "Another ${APP_NAME} run is already active: $LOCK_FILE"
|
||||||
@@ -141,7 +125,10 @@ borg_repo_exists() {
|
|||||||
init_repo() {
|
init_repo() {
|
||||||
require_command "$BORG_BIN"
|
require_command "$BORG_BIN"
|
||||||
check_passphrase_for_encryption
|
check_passphrase_for_encryption
|
||||||
mkdir -p "$(repo_parent)"
|
timeout 60 test -d "$(repo_parent)" || die "NAS parent unavailable; mount NAS before init"
|
||||||
|
local actual_source
|
||||||
|
actual_source="$(findmnt --direction backward --first-only -n -o SOURCE -T "$(repo_parent)")"
|
||||||
|
[[ "$actual_source" == "${EXPECTED_NAS_SOURCE:-thor:/volume1/NetBackup}" ]] || die "Wrong NAS backing mount: $actual_source"
|
||||||
if borg_repo_exists; then
|
if borg_repo_exists; then
|
||||||
info "-- Borg repository already exists: $REPO_DIR"
|
info "-- Borg repository already exists: $REPO_DIR"
|
||||||
return 0
|
return 0
|
||||||
@@ -150,77 +137,6 @@ init_repo() {
|
|||||||
"$BORG_BIN" init --encryption="$BORG_ENCRYPTION" "$REPO_DIR"
|
"$BORG_BIN" init --encryption="$BORG_ENCRYPTION" "$REPO_DIR"
|
||||||
}
|
}
|
||||||
|
|
||||||
timeout_cmd() {
|
|
||||||
if have timeout; then
|
|
||||||
timeout "$PRECHECK_TIMEOUT_SECONDS" "$@"
|
|
||||||
else
|
|
||||||
"$@"
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
count_source_entries() {
|
|
||||||
timeout_cmd find "$SOURCE_DIR" -mindepth 1 -maxdepth 1 -print -quit | wc -l
|
|
||||||
}
|
|
||||||
|
|
||||||
free_repo_gb() {
|
|
||||||
df -PBG "$(repo_parent)" | awk 'NR==2 { gsub(/G/, "", $4); print $4 }'
|
|
||||||
}
|
|
||||||
|
|
||||||
preflight() {
|
|
||||||
require_command "$BORG_BIN"
|
|
||||||
require_command flock
|
|
||||||
require_command df
|
|
||||||
require_command find
|
|
||||||
|
|
||||||
info "-- Source: $SOURCE_DIR"
|
|
||||||
info "-- Repository: $REPO_DIR"
|
|
||||||
|
|
||||||
timeout_cmd test -d "$SOURCE_DIR" || die "Source directory is not reachable: $SOURCE_DIR"
|
|
||||||
timeout_cmd test -r "$SOURCE_DIR" || die "Source directory is not readable: $SOURCE_DIR"
|
|
||||||
|
|
||||||
local entry_count
|
|
||||||
entry_count="$(count_source_entries | tr -d '[:space:]')"
|
|
||||||
if [[ "$ALLOW_EMPTY_SOURCE" != "true" && "${entry_count:-0}" -eq 0 ]]; then
|
|
||||||
die "Source directory looks empty. Refusing to create an empty backup."
|
|
||||||
fi
|
|
||||||
|
|
||||||
mkdir -p "$(repo_parent)"
|
|
||||||
timeout_cmd test -w "$(repo_parent)" || die "Repository parent is not writable/reachable: $(repo_parent)"
|
|
||||||
|
|
||||||
local free_gb
|
|
||||||
free_gb="$(free_repo_gb)"
|
|
||||||
info "-- Free space at repository parent: ${free_gb}G"
|
|
||||||
[[ "${free_gb:-0}" -ge "$MIN_FREE_REPO_GB" ]] || die "Not enough free space at repository parent; need ${MIN_FREE_REPO_GB}G"
|
|
||||||
|
|
||||||
if ! borg_repo_exists; then
|
|
||||||
init_repo
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
create_archive() {
|
|
||||||
local timestamp archive
|
|
||||||
timestamp="$(date '+%Y-%m-%d_%H-%M-%S')"
|
|
||||||
archive="${ARCHIVE_PREFIX}-${timestamp}"
|
|
||||||
|
|
||||||
local borg_args
|
|
||||||
borg_args=(
|
|
||||||
create
|
|
||||||
--stats
|
|
||||||
--show-rc
|
|
||||||
--compression "$BORG_COMPRESSION"
|
|
||||||
)
|
|
||||||
|
|
||||||
[[ "$EXCLUDE_CACHES" == "true" ]] && borg_args+=(--exclude-caches)
|
|
||||||
[[ "$ONE_FILE_SYSTEM" == "true" ]] && borg_args+=(--one-file-system)
|
|
||||||
[[ -f "$EXCLUDE_FILE" ]] && borg_args+=(--exclude-from "$EXCLUDE_FILE")
|
|
||||||
|
|
||||||
info "-- Creating archive: $archive"
|
|
||||||
(
|
|
||||||
cd "$SOURCE_DIR"
|
|
||||||
"$BORG_BIN" "${borg_args[@]}" "${REPO_DIR}::${archive}" .
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
archive_names() {
|
archive_names() {
|
||||||
"$BORG_BIN" list --short --glob-archives "${ARCHIVE_PREFIX}-*" "$REPO_DIR"
|
"$BORG_BIN" list --short --glob-archives "${ARCHIVE_PREFIX}-*" "$REPO_DIR"
|
||||||
}
|
}
|
||||||
@@ -231,7 +147,7 @@ latest_archive() {
|
|||||||
|
|
||||||
prune_repo() {
|
prune_repo() {
|
||||||
info "-- Pruning archives with retention: keep-within $RETENTION_WITHIN"
|
info "-- Pruning archives with retention: keep-within $RETENTION_WITHIN"
|
||||||
"$BORG_BIN" prune --list --show-rc --glob-archives "${ARCHIVE_PREFIX}-*" --keep-within "$RETENTION_WITHIN" "$REPO_DIR"
|
"$BORG_BIN" prune --list --show-rc --glob-archives "${ARCHIVE_PREFIX}-*" --keep-within "$RETENTION_WITHIN" --keep-last 7 "$REPO_DIR"
|
||||||
}
|
}
|
||||||
|
|
||||||
compact_repo() {
|
compact_repo() {
|
||||||
@@ -239,15 +155,6 @@ compact_repo() {
|
|||||||
"$BORG_BIN" compact "$REPO_DIR"
|
"$BORG_BIN" compact "$REPO_DIR"
|
||||||
}
|
}
|
||||||
|
|
||||||
days_since_stamp() {
|
|
||||||
local stamp="$1"
|
|
||||||
[[ -f "$stamp" ]] || { printf '999999\n'; return 0; }
|
|
||||||
local now then
|
|
||||||
now="$(date +%s)"
|
|
||||||
then="$(stat -c %Y "$stamp")"
|
|
||||||
printf '%s\n' "$(((now - then) / 86400))"
|
|
||||||
}
|
|
||||||
|
|
||||||
check_repo() {
|
check_repo() {
|
||||||
local verify_data="${1:-false}"
|
local verify_data="${1:-false}"
|
||||||
if [[ "$verify_data" == "true" ]]; then
|
if [[ "$verify_data" == "true" ]]; then
|
||||||
@@ -259,30 +166,29 @@ check_repo() {
|
|||||||
fi
|
fi
|
||||||
}
|
}
|
||||||
|
|
||||||
maybe_periodic_full_check() {
|
cmd_runner() {
|
||||||
[[ "$FULL_CHECK_VERIFY_DATA" == "true" ]] || return 0
|
load_config
|
||||||
local stamp="${STATE_DIR}/last-full-check"
|
export REPO_DIR ARCHIVE_PREFIX BORG_BIN BORG_COMPRESSION BORG_PASSPHRASE_FILE
|
||||||
local age_days
|
export RETENTION_WITHIN PRUNE_AFTER_BACKUP COMPACT_AFTER_PRUNE
|
||||||
age_days="$(days_since_stamp "$stamp")"
|
export STATE_DIR LOG_DIR MIN_FREE_REPO_GB EXCLUDE_FILE LOCK_FILE
|
||||||
[[ "$age_days" -ge "$FULL_CHECK_INTERVAL_DAYS" ]] || return 0
|
export FULL_CHECK_INTERVAL_DAYS FULL_CHECK_VERIFY_DATA
|
||||||
check_repo true
|
# New settings are optional in existing configs, and inherited by the runner.
|
||||||
date -Is > "$stamp"
|
export REMOTE="${REMOTE:-Nextcloud:}"
|
||||||
|
export STAGING_DIR="${STAGING_DIR:-${HOME}/.local/share/nextcloud-borg-backup/staging}"
|
||||||
|
export EXPECTED_NAS_SOURCE="${EXPECTED_NAS_SOURCE:-thor:/volume1/NetBackup}"
|
||||||
|
export EXPECTED_NAS_FSTYPE="${EXPECTED_NAS_FSTYPE:-nfs4}"
|
||||||
|
export BACKUP_INTERVAL_HOURS="${BACKUP_INTERVAL_HOURS:-24}"
|
||||||
|
export MAX_BACKUP_AGE_HOURS="${MAX_BACKUP_AGE_HOURS:-36}"
|
||||||
|
export MIN_SOURCE_FILES="${MIN_SOURCE_FILES:-1}"
|
||||||
|
export MAX_SOURCE_DROP_PERCENT="${MAX_SOURCE_DROP_PERCENT:-20}"
|
||||||
|
export MAX_STAGE_DELETES="${MAX_STAGE_DELETES:-1000}"
|
||||||
|
export MIN_FREE_STAGE_GB="${MIN_FREE_STAGE_GB:-5}"
|
||||||
|
export COMMAND_TIMEOUT_SECONDS="${COMMAND_TIMEOUT_SECONDS:-21600}"
|
||||||
|
export NCB_RUNNER="${NCB_RUNNER:-${HOME}/.local/libexec/nextcloud-borg-runner.py}"
|
||||||
|
exec python3 "$NCB_RUNNER" "$@"
|
||||||
}
|
}
|
||||||
|
|
||||||
cmd_backup() {
|
cmd_backup() { cmd_runner backup "$@"; }
|
||||||
load_config
|
|
||||||
start_logging
|
|
||||||
with_lock
|
|
||||||
preflight
|
|
||||||
create_archive
|
|
||||||
if [[ "$PRUNE_AFTER_BACKUP" == "true" ]]; then
|
|
||||||
prune_repo
|
|
||||||
[[ "$COMPACT_AFTER_PRUNE" == "true" ]] && compact_repo
|
|
||||||
fi
|
|
||||||
[[ "$REPOSITORY_CHECK_AFTER_BACKUP" == "true" ]] && check_repo false
|
|
||||||
maybe_periodic_full_check
|
|
||||||
info "-- Backup finished successfully"
|
|
||||||
}
|
|
||||||
|
|
||||||
cmd_list() {
|
cmd_list() {
|
||||||
load_config
|
load_config
|
||||||
@@ -388,6 +294,8 @@ cmd_restore() {
|
|||||||
|
|
||||||
cmd_check() {
|
cmd_check() {
|
||||||
load_config
|
load_config
|
||||||
|
ensure_dirs
|
||||||
|
with_lock
|
||||||
require_command "$BORG_BIN"
|
require_command "$BORG_BIN"
|
||||||
local verify_data=false
|
local verify_data=false
|
||||||
[[ "${1:-}" == "--verify-data" ]] && verify_data=true
|
[[ "${1:-}" == "--verify-data" ]] && verify_data=true
|
||||||
@@ -418,7 +326,8 @@ main() {
|
|||||||
|
|
||||||
case "$cmd" in
|
case "$cmd" in
|
||||||
backup) cmd_backup "$@" ;;
|
backup) cmd_backup "$@" ;;
|
||||||
init) load_config; init_repo "$@" ;;
|
health) cmd_runner health "$@" ;;
|
||||||
|
init) load_config; ensure_dirs; with_lock; init_repo "$@" ;;
|
||||||
list) cmd_list "$@" ;;
|
list) cmd_list "$@" ;;
|
||||||
info) cmd_info "$@" ;;
|
info) cmd_info "$@" ;;
|
||||||
search) cmd_search "$@" ;;
|
search) cmd_search "$@" ;;
|
||||||
@@ -426,9 +335,9 @@ main() {
|
|||||||
umount|unmount) cmd_umount "$@" ;;
|
umount|unmount) cmd_umount "$@" ;;
|
||||||
restore) cmd_restore "$@" ;;
|
restore) cmd_restore "$@" ;;
|
||||||
check) cmd_check "$@" ;;
|
check) cmd_check "$@" ;;
|
||||||
prune) load_config; prune_repo "$@"; compact_repo ;;
|
prune) load_config; ensure_dirs; with_lock; prune_repo "$@"; compact_repo ;;
|
||||||
status) cmd_status "$@" ;;
|
status) cmd_status "$@" ;;
|
||||||
break-lock|unlock) cmd_break_lock "$@" ;;
|
break-lock|unlock) load_config; ensure_dirs; with_lock; cmd_break_lock "$@" ;;
|
||||||
-h|--help|help|"") usage ;;
|
-h|--help|help|"") usage ;;
|
||||||
--version|version) printf '%s %s\n' "$APP_NAME" "$APP_VERSION" ;;
|
--version|version) printf '%s %s\n' "$APP_NAME" "$APP_VERSION" ;;
|
||||||
*) usage >&2; die "Unknown command: $cmd" ;;
|
*) usage >&2; die "Unknown command: $cmd" ;;
|
||||||
|
|||||||
Executable
+392
@@ -0,0 +1,392 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Verified cloud staging and fail-closed Borg backup. Standard library only."""
|
||||||
|
import datetime as dt
|
||||||
|
import fcntl
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
from pathlib import Path, PurePosixPath
|
||||||
|
import shutil
|
||||||
|
import signal
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import time
|
||||||
|
|
||||||
|
|
||||||
|
class BackupError(RuntimeError):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def log(message):
|
||||||
|
print(f'{dt.datetime.now().astimezone().isoformat(timespec="seconds")} {message}', flush=True)
|
||||||
|
|
||||||
|
|
||||||
|
def atomic_json(path, data):
|
||||||
|
descriptor, name = tempfile.mkstemp(prefix=path.name + '.', dir=path.parent)
|
||||||
|
tmp = Path(name)
|
||||||
|
try:
|
||||||
|
with os.fdopen(descriptor, 'w') as stream:
|
||||||
|
json.dump(data, stream, ensure_ascii=False, indent=2)
|
||||||
|
stream.flush()
|
||||||
|
os.fsync(stream.fileno())
|
||||||
|
tmp.replace(path)
|
||||||
|
directory = os.open(path.parent, os.O_RDONLY | os.O_DIRECTORY)
|
||||||
|
try:
|
||||||
|
os.fsync(directory)
|
||||||
|
finally:
|
||||||
|
os.close(directory)
|
||||||
|
finally:
|
||||||
|
tmp.unlink(missing_ok=True)
|
||||||
|
|
||||||
|
|
||||||
|
def read_json(path):
|
||||||
|
if not path.exists():
|
||||||
|
return {}
|
||||||
|
return json.loads(path.read_text())
|
||||||
|
|
||||||
|
|
||||||
|
class Runner:
|
||||||
|
def __init__(self):
|
||||||
|
e = os.environ
|
||||||
|
self.state = Path(e['STATE_DIR']).expanduser().absolute()
|
||||||
|
self.repo = Path(e['REPO_DIR']).absolute()
|
||||||
|
self.stage = Path(e['STAGING_DIR']).expanduser().absolute()
|
||||||
|
self.remote = e['REMOTE']
|
||||||
|
if not re.fullmatch(r'[A-Za-z0-9_.-]+', e['ARCHIVE_PREFIX']):
|
||||||
|
raise BackupError('Unsafe ARCHIVE_PREFIX')
|
||||||
|
self.borg = e.get('BORG_BIN', 'borg')
|
||||||
|
self.timeout = int(e.get('COMMAND_TIMEOUT_SECONDS', '21600'))
|
||||||
|
self.success_file = self.state / 'last-success.json'
|
||||||
|
self.attempt_file = self.state / 'last-attempt.json'
|
||||||
|
self.child = None
|
||||||
|
self.phase = 'preflight'
|
||||||
|
self.state.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||||
|
self.success = read_json(self.success_file)
|
||||||
|
# An old timestamp from a different repository/source must never suppress a backup.
|
||||||
|
if self.success and (self.success.get('repo') != str(self.repo) or
|
||||||
|
self.success.get('remote') != self.remote):
|
||||||
|
self.success = {}
|
||||||
|
|
||||||
|
def run(self, args, *, capture=False, cwd=None, timeout=None):
|
||||||
|
log('Running: ' + ' '.join(map(str, args)))
|
||||||
|
self.child = subprocess.Popen(list(map(str, args)), cwd=cwd,
|
||||||
|
stdout=subprocess.PIPE if capture else None,
|
||||||
|
start_new_session=True, text=True)
|
||||||
|
try:
|
||||||
|
out, _ = self.child.communicate(timeout=timeout or self.timeout)
|
||||||
|
if self.child.returncode:
|
||||||
|
raise BackupError(f'{args[0]} failed (exit {self.child.returncode}); phase={self.phase}')
|
||||||
|
return out
|
||||||
|
except BaseException:
|
||||||
|
self.stop_child()
|
||||||
|
raise
|
||||||
|
finally:
|
||||||
|
self.child = None
|
||||||
|
|
||||||
|
def stop_child(self):
|
||||||
|
if self.child and self.child.poll() is None:
|
||||||
|
os.killpg(self.child.pid, signal.SIGTERM)
|
||||||
|
try:
|
||||||
|
self.child.wait(timeout=10)
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
os.killpg(self.child.pid, signal.SIGKILL)
|
||||||
|
self.child.wait()
|
||||||
|
|
||||||
|
def cloud(self, *args, capture=False):
|
||||||
|
return self.run(['rclone', *args, '--contimeout', '20s', '--timeout', '2m',
|
||||||
|
'--retries', '3', '--low-level-retries', '10',
|
||||||
|
'--log-level', 'INFO', '--stats', '1m'], capture=capture)
|
||||||
|
|
||||||
|
def nas(self):
|
||||||
|
# Trigger autofs, then verify the backing mount. Never mkdir or init on /net.
|
||||||
|
self.run(['timeout', '60', 'test', '-f', self.repo / 'config'], timeout=70)
|
||||||
|
mounts = json.loads(self.run(['findmnt', '--json', '--target', self.repo,
|
||||||
|
'--output', 'SOURCE,FSTYPE,MAJ:MIN'], capture=True, timeout=60))['filesystems']
|
||||||
|
device = self.repo.stat().st_dev
|
||||||
|
backing_device = f'{os.major(device)}:{os.minor(device)}'
|
||||||
|
mounts = [mount for mount in mounts if mount.get('maj:min') == backing_device]
|
||||||
|
expected = os.environ.get('EXPECTED_NAS_SOURCE', 'thor:/volume1/NetBackup')
|
||||||
|
fstype = os.environ.get('EXPECTED_NAS_FSTYPE', 'nfs4')
|
||||||
|
if len(mounts) != 1 or mounts[0]['source'] != expected or mounts[0]['fstype'] != fstype:
|
||||||
|
raise BackupError(f'Wrong NAS mount; expected {expected} ({fstype}), got {mounts}')
|
||||||
|
free = shutil.disk_usage(self.repo).free
|
||||||
|
if free < int(os.environ.get('MIN_FREE_REPO_GB', '5')) * 1024**3:
|
||||||
|
raise BackupError('Insufficient free NAS space')
|
||||||
|
|
||||||
|
def manifest(self):
|
||||||
|
raw = json.loads(self.cloud('lsjson', self.remote, '--recursive', '--files-only',
|
||||||
|
'--hash', '--no-mimetype', capture=True))
|
||||||
|
result = {}
|
||||||
|
for item in raw:
|
||||||
|
name = item['Path']
|
||||||
|
if (not name or name.startswith('/') or '..' in PurePosixPath(name).parts or
|
||||||
|
'\x00' in name or item.get('IsDir') or item['Size'] < 0 or name in result):
|
||||||
|
raise BackupError('Unsafe, duplicate or unsupported cloud entry')
|
||||||
|
result[name] = {'size': item['Size'], 'mtime': item.get('ModTime'),
|
||||||
|
'hashes': {k: v for k, v in item.get('Hashes', {}).items() if v}}
|
||||||
|
return result
|
||||||
|
|
||||||
|
def verify_local_hashes(self, manifest):
|
||||||
|
# Independent validation avoids any size-only fallback chosen inside rclone.
|
||||||
|
for name, item in manifest.items():
|
||||||
|
path = self.stage / name
|
||||||
|
if path.is_symlink() or not path.is_file() or path.stat().st_size != item['size']:
|
||||||
|
raise BackupError(f'Staging file missing, unsafe or wrong size: {name}')
|
||||||
|
hashes = {key.lower().replace('-', ''): value for key, value in item['hashes'].items()}
|
||||||
|
algorithm = next((key for key in ('sha1', 'md5') if hashes.get(key)), None)
|
||||||
|
if algorithm:
|
||||||
|
digest = hashlib.new(algorithm, usedforsecurity=False)
|
||||||
|
with path.open('rb') as stream:
|
||||||
|
while chunk := stream.read(8 * 1024 * 1024):
|
||||||
|
digest.update(chunk)
|
||||||
|
if digest.hexdigest().lower() != hashes[algorithm].lower():
|
||||||
|
raise BackupError(f'Independent staging hash mismatch: {name}')
|
||||||
|
log('Independent local size/hash verification passed')
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def totals(manifest):
|
||||||
|
return len(manifest), sum(x['size'] for x in manifest.values())
|
||||||
|
|
||||||
|
def guard_source(self, manifest, baseline):
|
||||||
|
count, size = self.totals(manifest)
|
||||||
|
if count < int(os.environ.get('MIN_SOURCE_FILES', '1')) or size <= 0:
|
||||||
|
raise BackupError('Empty or implausibly small cloud source; refusing backup')
|
||||||
|
percent = float(os.environ.get('MAX_SOURCE_DROP_PERCENT', '20'))
|
||||||
|
if not 0 <= percent <= 100:
|
||||||
|
raise BackupError('MAX_SOURCE_DROP_PERCENT must be between 0 and 100')
|
||||||
|
for key, current in [('files', count), ('bytes', size)]:
|
||||||
|
previous = baseline.get(key, 0)
|
||||||
|
if previous and current < previous * (1 - percent / 100):
|
||||||
|
raise BackupError(f'Cloud {key} dropped by more than {percent}%: {previous} -> {current}')
|
||||||
|
log(f'Cloud inventory: {count} files, {size} bytes')
|
||||||
|
return count, size
|
||||||
|
|
||||||
|
def staging_preflight(self, manifest):
|
||||||
|
# Refuse symlinked, overlapping, mounted or foreign staging paths before rclone sync.
|
||||||
|
for other in (self.repo, self.state, Path.home() / 'Nextcloud'):
|
||||||
|
if self.stage == other or self.stage in other.parents or other in self.stage.parents:
|
||||||
|
raise BackupError(f'Unsafe staging overlap: {other}')
|
||||||
|
if self.stage.resolve() != self.stage:
|
||||||
|
raise BackupError('Staging path contains a symlink')
|
||||||
|
self.stage.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||||
|
os.chmod(self.stage, 0o700)
|
||||||
|
for base, dirs, files in os.walk(self.stage):
|
||||||
|
for name in dirs + files:
|
||||||
|
path = Path(base) / name
|
||||||
|
if path.is_symlink() or path.is_mount():
|
||||||
|
raise BackupError(f'Unexpected symlink/mount in staging: {path}')
|
||||||
|
if self.stage.is_mount():
|
||||||
|
raise BackupError('Staging must be an ordinary local directory')
|
||||||
|
owner_file = self.stage.parent / (self.stage.name + '.owner.json')
|
||||||
|
expected = {'remote': self.remote, 'stage': str(self.stage)}
|
||||||
|
if owner_file.exists():
|
||||||
|
if read_json(owner_file) != expected:
|
||||||
|
raise BackupError('Staging belongs to another cloud source')
|
||||||
|
elif any(self.stage.iterdir()):
|
||||||
|
raise BackupError('Nonempty unowned staging directory; refusing sync')
|
||||||
|
else:
|
||||||
|
atomic_json(owner_file, expected)
|
||||||
|
mount = json.loads(self.run(['findmnt', '--json', '--target', self.stage,
|
||||||
|
'--output', 'FSTYPE'], capture=True))['filesystems'][0]
|
||||||
|
if mount['fstype'] not in ('ext4', 'btrfs', 'xfs', 'f2fs', 'zfs', 'tmpfs'):
|
||||||
|
raise BackupError(f'Staging must be local; found {mount["fstype"]}')
|
||||||
|
total = self.totals(manifest)[1]
|
||||||
|
# Conservative full-copy headroom also covers temporary transfer files.
|
||||||
|
need = total + int(os.environ.get('MIN_FREE_STAGE_GB', '5')) * 1024**3
|
||||||
|
if shutil.disk_usage(self.stage).free < need:
|
||||||
|
raise BackupError(f'Insufficient staging space: need {need} bytes free')
|
||||||
|
|
||||||
|
def baseline(self):
|
||||||
|
if self.success:
|
||||||
|
return self.success
|
||||||
|
data = json.loads(self.run([self.borg, 'info', '--json', '--last', '1',
|
||||||
|
'--glob-archives', os.environ['ARCHIVE_PREFIX'] + '-*',
|
||||||
|
self.repo], capture=True))
|
||||||
|
archives = data.get('archives', [])
|
||||||
|
if not archives:
|
||||||
|
return {}
|
||||||
|
stats = archives[0]['stats']
|
||||||
|
return {'files': stats['nfiles'], 'bytes': stats['original_size']}
|
||||||
|
|
||||||
|
def stale(self):
|
||||||
|
age = time.time() - self.success.get('completed_at', 0)
|
||||||
|
return age < 0 or age > float(os.environ.get('MAX_BACKUP_AGE_HOURS', '36')) * 3600
|
||||||
|
|
||||||
|
def notify(self, message):
|
||||||
|
log('WARNING: ' + message)
|
||||||
|
stamp = self.state / 'last-notification.json'
|
||||||
|
if time.time() - read_json(stamp).get('at', 0) < 6 * 3600:
|
||||||
|
return
|
||||||
|
if shutil.which('notify-send'):
|
||||||
|
result = subprocess.run(['notify-send', '--urgency=critical',
|
||||||
|
'Nextcloud NAS-Sicherung', message], timeout=15,
|
||||||
|
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||||
|
if result.returncode == 0:
|
||||||
|
atomic_json(stamp, {'at': time.time()})
|
||||||
|
|
||||||
|
def backup(self, if_due=False):
|
||||||
|
with Path(os.environ['LOCK_FILE']).open('a') as lock:
|
||||||
|
try:
|
||||||
|
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||||
|
except BlockingIOError:
|
||||||
|
log('Another backup/maintenance command holds the lock; skipping')
|
||||||
|
return
|
||||||
|
age = time.time() - self.success.get('completed_at', 0)
|
||||||
|
if if_due and 0 <= age < float(os.environ.get('BACKUP_INTERVAL_HOURS', '24')) * 3600:
|
||||||
|
if self.success.get('maintenance_pending') or read_json(self.attempt_file).get('phase') == 'retention':
|
||||||
|
self.nas()
|
||||||
|
self.maintenance()
|
||||||
|
atomic_json(self.attempt_file, {'status': 'success', 'finished_at': time.time()})
|
||||||
|
log('Verified backup is still current; no transfer needed')
|
||||||
|
return
|
||||||
|
attempt = {'started_at': time.time(), 'status': 'running'}
|
||||||
|
atomic_json(self.attempt_file, attempt)
|
||||||
|
try:
|
||||||
|
self.perform()
|
||||||
|
except BaseException as exc:
|
||||||
|
atomic_json(self.attempt_file, {**attempt, 'finished_at': time.time(),
|
||||||
|
'status': 'failed', 'phase': self.phase, 'error': str(exc)})
|
||||||
|
try:
|
||||||
|
self.notify('Sicherung fehlgeschlagen: ' + str(exc) +
|
||||||
|
(' — Altersgrenze ohne bestätigtes Backup überschritten.' if self.stale() else ''))
|
||||||
|
except Exception as notify_error:
|
||||||
|
log(f'Notification unavailable: {notify_error}')
|
||||||
|
raise
|
||||||
|
atomic_json(self.attempt_file, {**attempt, 'finished_at': time.time(), 'status': 'success'})
|
||||||
|
|
||||||
|
def check_identity(self):
|
||||||
|
identity = json.loads(self.run([self.borg, 'info', '--json', self.repo], capture=True))['repository']['id']
|
||||||
|
bound = read_json(self.state / 'repository.json')
|
||||||
|
if bound and bound != {'repo': str(self.repo), 'id': identity}:
|
||||||
|
raise BackupError('Repository identity changed; refusing backup/retention')
|
||||||
|
atomic_json(self.state / 'repository.json', {'repo': str(self.repo), 'id': identity})
|
||||||
|
|
||||||
|
def perform(self):
|
||||||
|
self.nas()
|
||||||
|
self.check_identity()
|
||||||
|
excludes = Path(os.environ['EXCLUDE_FILE'])
|
||||||
|
if excludes.exists() and any(x.strip() and not x.lstrip().startswith('#')
|
||||||
|
for x in excludes.read_text().splitlines()):
|
||||||
|
raise BackupError('Active legacy Borg excludes found; resolve before full cloud backup')
|
||||||
|
baseline = self.baseline()
|
||||||
|
self.phase = 'cloud inventory'
|
||||||
|
before = self.manifest()
|
||||||
|
count, size = self.guard_source(before, baseline)
|
||||||
|
self.staging_preflight(before)
|
||||||
|
self.phase = 'cloud sync'
|
||||||
|
common = {'md5', 'sha1'}
|
||||||
|
unhashed = sorted(name for name, item in before.items() if item['size'] and not any(
|
||||||
|
key.lower().replace('-', '') in common for key in item['hashes']))
|
||||||
|
# rclone's raw file lists cannot encode newline names. Fall back to a full byte read.
|
||||||
|
full_download = any('\n' in name or '\r' in name for name in unhashed)
|
||||||
|
file_list = self.state / 'unhashed-files.txt'
|
||||||
|
if unhashed and not full_download:
|
||||||
|
file_list.write_text('\n'.join(unhashed) + '\n')
|
||||||
|
log(f'Reading {len(unhashed)} files without server hashes in full')
|
||||||
|
self.cloud('copy', self.remote, str(self.stage), '--ignore-times',
|
||||||
|
'--files-from-raw', str(file_list), '--transfers', '4')
|
||||||
|
sync_flags = ['--ignore-times'] if full_download else ['--checksum']
|
||||||
|
self.cloud('sync', self.remote, str(self.stage), '--check-first', '--delete-after',
|
||||||
|
'--max-delete', os.environ.get('MAX_STAGE_DELETES', '1000'),
|
||||||
|
'--create-empty-src-dirs', '--transfers', '4', '--checkers', '8', *sync_flags)
|
||||||
|
self.phase = 'cloud verification'
|
||||||
|
self.cloud('check', self.remote, str(self.stage), *(['--download'] if full_download else []))
|
||||||
|
if unhashed and not full_download:
|
||||||
|
self.cloud('check', self.remote, str(self.stage), '--download',
|
||||||
|
'--files-from-raw', str(file_list))
|
||||||
|
self.verify_local_hashes(before)
|
||||||
|
after = self.manifest()
|
||||||
|
if before != after:
|
||||||
|
raise BackupError('Cloud changed during sync/verification; retry with a stable inventory')
|
||||||
|
atomic_json(self.state / 'verified-inventory.json', after)
|
||||||
|
self.nas()
|
||||||
|
self.phase = 'borg create'
|
||||||
|
archive = os.environ['ARCHIVE_PREFIX'] + '-' + dt.datetime.now().strftime('%Y-%m-%d_%H-%M-%S')
|
||||||
|
target = f'{self.repo}::{archive}'
|
||||||
|
self.run([self.borg, 'create', '--stats', '--show-rc', '--compression',
|
||||||
|
os.environ['BORG_COMPRESSION'], target, '.'], cwd=self.stage)
|
||||||
|
data = json.loads(self.run([self.borg, 'info', '--json', target], capture=True))
|
||||||
|
stats = data['archives'][0]['stats']
|
||||||
|
if stats['nfiles'] != count or stats['original_size'] != size:
|
||||||
|
raise BackupError(f'Archive inventory mismatch: {stats}; expected {count} files/{size} bytes')
|
||||||
|
self.phase = 'archive verification'
|
||||||
|
self.run([self.borg, 'check', '--archives-only', '--verify-data',
|
||||||
|
'--glob-archives', archive, self.repo])
|
||||||
|
self.run([self.borg, 'check', '--repository-only', self.repo])
|
||||||
|
# Success means a complete verified archive. Retention failure must not lose this fact.
|
||||||
|
self.success = {'completed_at': time.time(), 'archive': archive, 'files': count,
|
||||||
|
'bytes': size, 'repo': str(self.repo), 'remote': self.remote,
|
||||||
|
'archive_id': data['archives'][0]['id'], 'maintenance_pending': True}
|
||||||
|
atomic_json(self.success_file, self.success)
|
||||||
|
log(f'BACKUP VERIFIED: {archive} ({count} files, {size} bytes)')
|
||||||
|
self.maintenance()
|
||||||
|
log('Backup and maintenance finished successfully')
|
||||||
|
|
||||||
|
def maintenance(self):
|
||||||
|
self.phase = 'retention'
|
||||||
|
self.check_identity()
|
||||||
|
if self.success.get('archive'):
|
||||||
|
archived = json.loads(self.run([self.borg, 'info', '--json',
|
||||||
|
f'{self.repo}::{self.success["archive"]}'], capture=True))['archives'][0]
|
||||||
|
if archived['id'] != self.success['archive_id']:
|
||||||
|
raise BackupError('Verified archive identity changed; refusing retention')
|
||||||
|
if os.environ.get('PRUNE_AFTER_BACKUP', 'true') == 'true':
|
||||||
|
self.run([self.borg, 'prune', '--list', '--show-rc', '--glob-archives',
|
||||||
|
os.environ['ARCHIVE_PREFIX'] + '-*', '--keep-within',
|
||||||
|
os.environ['RETENTION_WITHIN'], '--keep-last', '7', self.repo])
|
||||||
|
if os.environ.get('COMPACT_AFTER_PRUNE', 'true') == 'true':
|
||||||
|
self.run([self.borg, 'compact', self.repo])
|
||||||
|
if self.success:
|
||||||
|
self.success['maintenance_pending'] = False
|
||||||
|
atomic_json(self.success_file, self.success)
|
||||||
|
log('Retention finished successfully')
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
os.umask(0o077)
|
||||||
|
runner = Runner()
|
||||||
|
def interrupted(signum, _frame):
|
||||||
|
runner.stop_child()
|
||||||
|
raise BackupError(f'Interrupted by signal {signum}')
|
||||||
|
signal.signal(signal.SIGTERM, interrupted)
|
||||||
|
signal.signal(signal.SIGINT, interrupted)
|
||||||
|
if sys.argv[1:] in (['health'], ['health', '--notify']):
|
||||||
|
print(json.dumps({'healthy': not runner.stale(), 'last_success': runner.success,
|
||||||
|
'last_attempt': read_json(runner.attempt_file)}, ensure_ascii=False, indent=2))
|
||||||
|
if runner.stale() and '--notify' in sys.argv:
|
||||||
|
runner.notify('NAS-Sicherung überfällig: kein bestätigtes Backup innerhalb der Altersgrenze. Protokolle und health prüfen.')
|
||||||
|
return 2 if runner.stale() else 0
|
||||||
|
if sys.argv[1:] not in (['backup'], ['backup', '--if-due']):
|
||||||
|
raise BackupError('Usage: runner backup [--if-due] | health')
|
||||||
|
logs = Path(os.environ['LOG_DIR'])
|
||||||
|
logs.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||||
|
path = logs / ('backup-' + dt.datetime.now().strftime('%Y%m%d-%H%M%S') + '.log')
|
||||||
|
# Journal + file, without an orphaned tee process on timeout/interruption.
|
||||||
|
for old in logs.glob('backup-*.log'):
|
||||||
|
if old.is_file() and not old.is_symlink() and old.stat().st_mtime < time.time() - 90 * 86400:
|
||||||
|
old.unlink()
|
||||||
|
with path.open('a', buffering=1) as stream:
|
||||||
|
class Tee:
|
||||||
|
def write(self, data):
|
||||||
|
sys.__stdout__.write(data)
|
||||||
|
stream.write(data)
|
||||||
|
def flush(self):
|
||||||
|
sys.__stdout__.flush()
|
||||||
|
stream.flush()
|
||||||
|
sys.stdout = Tee()
|
||||||
|
# Child output goes to journal; lifecycle/phase/status additionally go to the run log.
|
||||||
|
try:
|
||||||
|
runner.backup('--if-due' in sys.argv)
|
||||||
|
finally:
|
||||||
|
sys.stdout = sys.__stdout__
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
try:
|
||||||
|
sys.exit(main())
|
||||||
|
except (Exception, KeyboardInterrupt) as error:
|
||||||
|
log(f'ERROR: {error}')
|
||||||
|
sys.exit(1)
|
||||||
@@ -5,11 +5,13 @@ APP_NAME="nextcloud-borg-backup"
|
|||||||
BIN_DIR="${HOME}/.local/bin"
|
BIN_DIR="${HOME}/.local/bin"
|
||||||
SYSTEMD_USER_DIR="${XDG_CONFIG_HOME:-${HOME}/.config}/systemd/user"
|
SYSTEMD_USER_DIR="${XDG_CONFIG_HOME:-${HOME}/.config}/systemd/user"
|
||||||
|
|
||||||
systemctl --user disable --now "${APP_NAME}.timer" 2>/dev/null || true
|
systemctl --user stop "${APP_NAME}.service"
|
||||||
|
systemctl --user disable --now "${APP_NAME}.timer" nextcloud-borg-health.timer 2>/dev/null || true
|
||||||
systemctl --user daemon-reload
|
systemctl --user daemon-reload
|
||||||
|
|
||||||
rm -f "${SYSTEMD_USER_DIR}/${APP_NAME}.service" "${SYSTEMD_USER_DIR}/${APP_NAME}.timer"
|
rm -f "${SYSTEMD_USER_DIR}/${APP_NAME}.service" "${SYSTEMD_USER_DIR}/${APP_NAME}.timer"
|
||||||
rm -f "${BIN_DIR}/${APP_NAME}"
|
rm -f "${SYSTEMD_USER_DIR}/nextcloud-borg-health.service" "${SYSTEMD_USER_DIR}/nextcloud-borg-health.timer"
|
||||||
|
rm -f "${BIN_DIR}/${APP_NAME}" "${HOME}/.local/libexec/nextcloud-borg-runner.py"
|
||||||
systemctl --user daemon-reload
|
systemctl --user daemon-reload
|
||||||
|
|
||||||
printf 'Removed user service, timer, and executable.\n'
|
printf 'Removed user service, timer, and executable.\n'
|
||||||
|
|||||||
@@ -1,13 +1,20 @@
|
|||||||
[Unit]
|
[Unit]
|
||||||
Description=Back up mounted Nextcloud folder to Synology with Borg
|
Description=Verified Nextcloud backup to NAS (independent of desktop mount)
|
||||||
Documentation=man:borg(1)
|
Documentation=man:borg(1)
|
||||||
|
StartLimitIntervalSec=0
|
||||||
|
|
||||||
[Service]
|
[Service]
|
||||||
Type=oneshot
|
Type=oneshot
|
||||||
ExecStart=%h/.local/bin/nextcloud-borg-backup backup
|
ExecStart=%h/.local/bin/nextcloud-borg-backup backup --if-due
|
||||||
TimeoutStartSec=12h
|
TimeoutStartSec=12h
|
||||||
|
TimeoutStopSec=30s
|
||||||
|
KillMode=control-group
|
||||||
Nice=10
|
Nice=10
|
||||||
IOSchedulingClass=best-effort
|
IOSchedulingClass=best-effort
|
||||||
IOSchedulingPriority=6
|
IOSchedulingPriority=6
|
||||||
|
UMask=0077
|
||||||
|
NoNewPrivileges=true
|
||||||
Environment=LANG=C.UTF-8
|
Environment=LANG=C.UTF-8
|
||||||
Environment=LC_ALL=C.UTF-8
|
Environment=LC_ALL=C.UTF-8
|
||||||
|
Environment=BORG_LOCK_WAIT=60
|
||||||
|
Environment=BORG_NON_INTERACTIVE=1
|
||||||
@@ -1,10 +1,12 @@
|
|||||||
[Unit]
|
[Unit]
|
||||||
Description=Daily Nextcloud Borg backup timer
|
Description=Check every 15 minutes for due or failed Nextcloud NAS backup
|
||||||
|
|
||||||
[Timer]
|
[Timer]
|
||||||
OnCalendar=*-*-* 03:15:00
|
OnStartupSec=2min
|
||||||
|
OnCalendar=*:0/15
|
||||||
Persistent=true
|
Persistent=true
|
||||||
RandomizedDelaySec=30m
|
RandomizedDelaySec=60s
|
||||||
|
AccuracySec=30s
|
||||||
Unit=nextcloud-borg-backup.service
|
Unit=nextcloud-borg-backup.service
|
||||||
|
|
||||||
[Install]
|
[Install]
|
||||||
|
|||||||
@@ -0,0 +1,9 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Warn if verified Nextcloud NAS backup is overdue
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=oneshot
|
||||||
|
ExecStart=%h/.local/bin/nextcloud-borg-backup health --notify
|
||||||
|
TimeoutStartSec=45s
|
||||||
|
UMask=0077
|
||||||
|
NoNewPrivileges=true
|
||||||
@@ -0,0 +1,11 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Hourly Nextcloud backup age watchdog
|
||||||
|
|
||||||
|
[Timer]
|
||||||
|
OnStartupSec=5min
|
||||||
|
OnCalendar=hourly
|
||||||
|
Persistent=true
|
||||||
|
RandomizedDelaySec=2min
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=timers.target
|
||||||
@@ -0,0 +1,222 @@
|
|||||||
|
import importlib.util
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
from pathlib import Path
|
||||||
|
import subprocess
|
||||||
|
import tempfile
|
||||||
|
import unittest
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
spec = importlib.util.spec_from_file_location('runner', ROOT / 'scripts/nextcloud-borg-runner.py')
|
||||||
|
m = importlib.util.module_from_spec(spec)
|
||||||
|
spec.loader.exec_module(m)
|
||||||
|
|
||||||
|
|
||||||
|
class BackupTests(unittest.TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
self.tmp = tempfile.TemporaryDirectory()
|
||||||
|
self.root = Path(self.tmp.name)
|
||||||
|
self.env = patch.dict(os.environ, {
|
||||||
|
'STATE_DIR': str(self.root/'state'), 'REPO_DIR': str(self.root/'repo'),
|
||||||
|
'STAGING_DIR': str(self.root/'stage'), 'REMOTE': ':local:'+str(self.root/'source'),
|
||||||
|
'ARCHIVE_PREFIX': 'nextcloud', 'BORG_COMPRESSION': 'zstd,1',
|
||||||
|
'EXCLUDE_FILE': str(self.root/'excludes'), 'LOCK_FILE': str(self.root/'state/lock'),
|
||||||
|
'MIN_FREE_STAGE_GB': '0', 'MIN_FREE_REPO_GB': '0', 'MAX_STAGE_DELETES': '1000',
|
||||||
|
'PRUNE_AFTER_BACKUP': 'false', 'BORG_PASSPHRASE': '',
|
||||||
|
'BORG_PASSCOMMAND': '', 'BORG_NON_INTERACTIVE': '1',
|
||||||
|
})
|
||||||
|
self.env.start()
|
||||||
|
self.runner = m.Runner()
|
||||||
|
|
||||||
|
def tearDown(self):
|
||||||
|
self.env.stop()
|
||||||
|
self.tmp.cleanup()
|
||||||
|
|
||||||
|
def manifest(self, count=10, size=100):
|
||||||
|
return {f'f{i}': {'size': size, 'mtime': 'now', 'hashes': {}} for i in range(count)}
|
||||||
|
|
||||||
|
def test_empty_and_shrinking_sources_rejected(self):
|
||||||
|
for manifest in ({}, self.manifest(7), self.manifest(10, 70)):
|
||||||
|
with self.subTest(manifest=manifest), self.assertRaises(m.BackupError):
|
||||||
|
self.runner.guard_source(manifest, {'files': 10, 'bytes': 1000})
|
||||||
|
self.runner.guard_source(self.manifest(8), {'files': 10, 'bytes': 1000})
|
||||||
|
|
||||||
|
def test_wrong_nas_rejected_before_writes(self):
|
||||||
|
self.runner.repo.mkdir()
|
||||||
|
with patch.object(self.runner, 'run', side_effect=['', json.dumps({'filesystems': [
|
||||||
|
{'source': '/dev/sda1', 'fstype': 'ext4'}]})]):
|
||||||
|
with self.assertRaisesRegex(m.BackupError, 'Wrong NAS'):
|
||||||
|
self.runner.nas()
|
||||||
|
self.assertEqual(list(self.runner.repo.iterdir()), [])
|
||||||
|
|
||||||
|
def test_stacked_autofs_uses_actual_backing_device(self):
|
||||||
|
self.runner.repo.mkdir()
|
||||||
|
device = self.runner.repo.stat().st_dev
|
||||||
|
dev = f'{os.major(device)}:{os.minor(device)}'
|
||||||
|
with patch.object(self.runner, 'run', side_effect=['', json.dumps({'filesystems': [
|
||||||
|
{'source': '-hosts', 'fstype': 'autofs', 'maj:min': '0:99999'},
|
||||||
|
{'source': 'thor:/volume1/NetBackup', 'fstype': 'nfs4', 'maj:min': dev}]})]):
|
||||||
|
self.runner.nas()
|
||||||
|
|
||||||
|
def test_symlink_and_unowned_stage_rejected(self):
|
||||||
|
(self.root/'source').mkdir()
|
||||||
|
(self.root/'stage').symlink_to(self.root/'source', target_is_directory=True)
|
||||||
|
with self.assertRaisesRegex(m.BackupError, 'symlink'):
|
||||||
|
self.runner.staging_preflight(self.manifest())
|
||||||
|
(self.root/'stage').unlink()
|
||||||
|
(self.root/'stage').mkdir()
|
||||||
|
(self.root/'stage/file').write_text('private')
|
||||||
|
with self.assertRaisesRegex(m.BackupError, 'unowned'):
|
||||||
|
self.runner.staging_preflight(self.manifest())
|
||||||
|
|
||||||
|
def test_subprocess_failure_never_becomes_success(self):
|
||||||
|
with patch.object(self.runner, 'perform', side_effect=m.BackupError('network failure')), \
|
||||||
|
patch.object(self.runner, 'notify') as notify:
|
||||||
|
with self.assertRaises(m.BackupError):
|
||||||
|
self.runner.backup()
|
||||||
|
notify.assert_called_once()
|
||||||
|
self.assertFalse(self.runner.success_file.exists())
|
||||||
|
self.assertEqual(m.read_json(self.runner.attempt_file)['status'], 'failed')
|
||||||
|
|
||||||
|
def test_due_skip_and_clock_rollback(self):
|
||||||
|
import time
|
||||||
|
self.runner.success = {'completed_at': time.time()}
|
||||||
|
with patch.object(self.runner, 'perform') as perform:
|
||||||
|
self.runner.backup(True)
|
||||||
|
perform.assert_not_called()
|
||||||
|
self.runner.success = {'completed_at': time.time()+86400}
|
||||||
|
with patch.object(self.runner, 'perform') as perform:
|
||||||
|
self.runner.backup(True)
|
||||||
|
perform.assert_called_once()
|
||||||
|
|
||||||
|
def test_changed_cloud_never_reaches_borg_create(self):
|
||||||
|
calls = []
|
||||||
|
def run(args, **kwargs):
|
||||||
|
calls.append(list(map(str, args)))
|
||||||
|
return json.dumps({'repository': {'id': 'test'}})
|
||||||
|
with patch.object(self.runner, 'nas'), patch.object(self.runner, 'run', side_effect=run), \
|
||||||
|
patch.object(self.runner, 'baseline', return_value={}), \
|
||||||
|
patch.object(self.runner, 'manifest', side_effect=[self.manifest(), self.manifest(11)]), \
|
||||||
|
patch.object(self.runner, 'staging_preflight'), patch.object(self.runner, 'cloud'), patch.object(self.runner, 'verify_local_hashes'):
|
||||||
|
with self.assertRaisesRegex(m.BackupError, 'Cloud changed'):
|
||||||
|
self.runner.perform()
|
||||||
|
self.assertFalse(any('create' in x for x in calls))
|
||||||
|
self.assertFalse(self.runner.success_file.exists())
|
||||||
|
|
||||||
|
def test_prune_failure_retried_even_when_backup_current(self):
|
||||||
|
import time
|
||||||
|
self.runner.success = {'completed_at': time.time()}
|
||||||
|
m.atomic_json(self.runner.attempt_file, {'phase': 'retention', 'status': 'failed'})
|
||||||
|
with patch.object(self.runner, 'nas'), patch.object(self.runner, 'maintenance') as maintenance:
|
||||||
|
self.runner.backup(True)
|
||||||
|
maintenance.assert_called_once()
|
||||||
|
self.assertEqual(m.read_json(self.runner.attempt_file)['status'], 'success')
|
||||||
|
|
||||||
|
def test_repository_replacement_rejected(self):
|
||||||
|
m.atomic_json(self.runner.state/'repository.json', {'repo': str(self.runner.repo), 'id': 'old'})
|
||||||
|
with patch.object(self.runner, 'nas'), patch.object(self.runner, 'run', return_value=json.dumps(
|
||||||
|
{'repository': {'id': 'new'}})):
|
||||||
|
with self.assertRaisesRegex(m.BackupError, 'identity changed'):
|
||||||
|
self.runner.perform()
|
||||||
|
|
||||||
|
def test_failed_sync_or_check_never_creates_or_prunes(self):
|
||||||
|
for failing in ('sync', 'check'):
|
||||||
|
calls = []
|
||||||
|
def cloud(command, *args, **kwargs):
|
||||||
|
if command == failing:
|
||||||
|
raise m.BackupError('simulated disconnect')
|
||||||
|
with self.subTest(command=failing), patch.object(self.runner, 'nas'), \
|
||||||
|
patch.object(self.runner, 'run', side_effect=lambda args, **kw: (
|
||||||
|
calls.append(list(map(str, args))) or json.dumps({'repository': {'id': 'test'}}))), \
|
||||||
|
patch.object(self.runner, 'baseline', return_value={}), \
|
||||||
|
patch.object(self.runner, 'manifest', return_value=self.manifest()), \
|
||||||
|
patch.object(self.runner, 'staging_preflight'), \
|
||||||
|
patch.object(self.runner, 'cloud', side_effect=cloud):
|
||||||
|
with self.assertRaises(m.BackupError):
|
||||||
|
self.runner.perform()
|
||||||
|
self.assertFalse(any('create' in x or 'prune' in x for x in calls))
|
||||||
|
|
||||||
|
def test_timeout_stops_child(self):
|
||||||
|
import sys
|
||||||
|
with self.assertRaises(subprocess.TimeoutExpired):
|
||||||
|
self.runner.run([sys.executable, '-c', 'import time; time.sleep(30)'], timeout=0.1)
|
||||||
|
self.assertIsNone(self.runner.child)
|
||||||
|
|
||||||
|
def test_overlapping_backup_is_skipped(self):
|
||||||
|
import fcntl
|
||||||
|
with (self.runner.state/'lock').open('a') as lock:
|
||||||
|
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||||
|
with patch.object(self.runner, 'perform') as perform:
|
||||||
|
self.runner.backup()
|
||||||
|
perform.assert_not_called()
|
||||||
|
|
||||||
|
def test_independent_hash_check_detects_same_size_corruption(self):
|
||||||
|
import hashlib
|
||||||
|
self.runner.stage.mkdir()
|
||||||
|
(self.runner.stage/'file').write_bytes(b'wrong')
|
||||||
|
manifest = {'file': {'size': 5, 'hashes': {'SHA-1': hashlib.sha1(b'right').hexdigest()}}}
|
||||||
|
with self.assertRaisesRegex(m.BackupError, 'hash mismatch'):
|
||||||
|
self.runner.verify_local_hashes(manifest)
|
||||||
|
|
||||||
|
def test_pending_maintenance_survives_interrupted_attempt_record(self):
|
||||||
|
import time
|
||||||
|
self.runner.success = {'completed_at': time.time(), 'maintenance_pending': True}
|
||||||
|
m.atomic_json(self.runner.attempt_file, {'status': 'running'})
|
||||||
|
with patch.object(self.runner, 'nas'), patch.object(self.runner, 'maintenance') as maintenance:
|
||||||
|
self.runner.backup(True)
|
||||||
|
maintenance.assert_called_once()
|
||||||
|
|
||||||
|
def test_real_rclone_borg_backup_and_restore(self):
|
||||||
|
source = self.root/'source'
|
||||||
|
source.mkdir()
|
||||||
|
(source/'hello.txt').write_text('recoverable cloud data\n')
|
||||||
|
(source/'sub').mkdir()
|
||||||
|
(source/'sub/unicode ä.txt').write_bytes(b'\x00\x01'*2048)
|
||||||
|
subprocess.run(['borg', 'init', '--encryption=none', str(self.runner.repo)], check=True,
|
||||||
|
stdout=subprocess.DEVNULL)
|
||||||
|
# Only the host mount probe is mocked; real rclone, Borg create/check and restore run.
|
||||||
|
with patch.object(self.runner, 'nas'), patch.object(self.runner, 'staging_preflight',
|
||||||
|
side_effect=lambda _: self.runner.stage.mkdir()):
|
||||||
|
self.runner.backup()
|
||||||
|
success = m.read_json(self.runner.success_file)
|
||||||
|
self.assertEqual(success['files'], 2)
|
||||||
|
dest = self.root/'restore'
|
||||||
|
dest.mkdir()
|
||||||
|
subprocess.run(['borg', 'extract', f'{self.runner.repo}::{success["archive"]}'],
|
||||||
|
cwd=dest, check=True)
|
||||||
|
for path in source.rglob('*'):
|
||||||
|
if path.is_file():
|
||||||
|
self.assertEqual(path.read_bytes(), (dest/path.relative_to(source)).read_bytes())
|
||||||
|
# Same-size, same-time edits must work with hashes AND without server hashes.
|
||||||
|
original_manifest = self.runner.manifest
|
||||||
|
for day, no_hashes in [(1, False), (2, True)]:
|
||||||
|
old = (source/'hello.txt').stat()
|
||||||
|
(source/'hello.txt').write_text(f'{day}ECOVERABLE cloud data\n')
|
||||||
|
os.utime(source/'hello.txt', ns=(old.st_atime_ns, old.st_mtime_ns))
|
||||||
|
def manifest():
|
||||||
|
result = original_manifest()
|
||||||
|
if no_hashes:
|
||||||
|
for value in result.values():
|
||||||
|
value['hashes'] = {}
|
||||||
|
return result
|
||||||
|
with patch.object(self.runner, 'nas'), patch.object(self.runner, 'staging_preflight'), \
|
||||||
|
patch.object(self.runner, 'manifest', side_effect=manifest), \
|
||||||
|
patch.object(m.dt, 'datetime', wraps=m.dt.datetime) as clock:
|
||||||
|
clock.now.return_value = m.dt.datetime(2030, 1, day)
|
||||||
|
self.runner.backup()
|
||||||
|
self.assertEqual((source/'hello.txt').read_bytes(), (self.runner.stage/'hello.txt').read_bytes())
|
||||||
|
# A refused staging deletion must preserve the last verified success/archive.
|
||||||
|
previous = m.read_json(self.runner.success_file)
|
||||||
|
(source/'hello.txt').unlink()
|
||||||
|
with patch.dict(os.environ, {'MAX_STAGE_DELETES': '0', 'MAX_SOURCE_DROP_PERCENT': '100'}), \
|
||||||
|
patch.object(self.runner, 'nas'), patch.object(self.runner, 'staging_preflight'), \
|
||||||
|
patch.object(self.runner, 'notify'):
|
||||||
|
with self.assertRaises(m.BackupError):
|
||||||
|
self.runner.backup()
|
||||||
|
self.assertEqual(m.read_json(self.runner.success_file), previous)
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
unittest.main()
|
||||||
Reference in new issue
Block a user