Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(database): Physical Backup & Restore #2419

Draft
wants to merge 29 commits into
base: master
Choose a base branch
from
Draft

feat(database): Physical Backup & Restore #2419

wants to merge 29 commits into from

Conversation

tanmoysrt
Copy link
Member

@tanmoysrt tanmoysrt commented Jan 14, 2025 •
edited
Loading

Resolve #1093

  • Physical Backup Flow [Tested]
  • Physical Restoration Flow via Physical Backup Restoration doctype [Not Tested]
  • Integrate in site migration
  • UI Update
  • Testcases (if any)

Agent PR - frappe/agent#154

tanmoysrt added 27 commits January 8, 2025 17:13
@tanmoysrt
refactor: Remove unused fields from Site Backup doctype
03782a5
@tanmoysrt
refactor(snapshot): Allow to exclude boot volume while snapshot
d2bcf69 
For physical backup, we don't need boot volume,
  Just need data volume which holds /var/lib/mysql
@tanmoysrt
feat(site-backup): Added required fields for physical backup
d4099a2
@tanmoysrt
feat: Added Agent job for physical backup of database
4ae28dd
@tanmoysrt
feat(site-backup): Added support for physical backup of database
204cca6
@tanmoysrt
feat(site-backup): Store the required data for physical backup restore
c378d8e 
On agent callback, store the table schema
 Innodb tables and MyISAM table names of the database
 From Agent Job data to Site Backup doctype
@tanmoysrt
feat(snapshot): Added function to create volume from snapshot
14d27da
@tanmoysrt
feat(physical-backup-restoration): Added doctype and util functions
a56dda1
@tanmoysrt
feat(virtual-machine): Add support for temporary volumes
385fe0f 
For physical restore, we will attach other disk to the server
That can mess up with logic of disk size and other things
So, put those kind of volumes in temporary volume list
To prevent any miscalculation
@tanmoysrt
refactor(virtual-machine): Split attach_new_volume fn to reuse parts
01c07bf
@tanmoysrt
feat(agent): Added function for physical backup restore
b652b0e
@tanmoysrt
chore(cspell): Add missing word
f4820a9
@tanmoysrt
feat(physical-backup-restore): Implemented complete flow of restoration
046b5e6
@tanmoysrt
chore(cspell): Fix conflict
9e8734a
@tanmoysrt
Merge branch 'master' into physical_backup_restore
9115898
@tanmoysrt
fix(physical-backup-restore): Send agent requests to database server
93e1659
@tanmoysrt
Merge branch 'master' into physical_backup_restore
4eca9b0
@tanmoysrt
refactor(site-backup): Generate snapshot request key on insert
d271912
@tanmoysrt
fix(site-backup): Send agent request to Database Server
06cd205
@tanmoysrt
refactor(site-backup): Send private ip of database server
f1ce6e5
@tanmoysrt
refactor(site-backup): In snapshot create api set admin user temporarily
c699c65 
We are validating the requests by provided request key
Instead of adding ignore_permissions, temporarily set user to Admin
Then, reset it back
@tanmoysrt
chore(site-backup): Site Backup DocType layout updated
ecc5a00
@tanmoysrt
chore(site-backup): Add Physical Backup function and button to Site
d7bdb66
@tanmoysrt
feat(site-backup): Modify permission of db directory before backup
35bc8fa
@tanmoysrt
feat(site-backup): On disk snapshot status update sync that to backup
c26c7ec
@tanmoysrt
Merge branch 'master' into physical_backup_restore
cbae905
@tanmoysrt
feat(disk-snapshot): Record the snapshot completion duration
4709021
@frappe frappe deleted a comment from codecov bot Jan 17, 2025
@codecov Codecov
Copy link

codecov bot commented Jan 17, 2025

Codecov Report

Attention: Patch coverage is 27.63975% with 233 lines in your changes missing coverage. Please review.

Project coverage is 37.98%. Comparing base (227c62a) to head (4709021).

Files with missing lines Patch % Lines
..._backup_restoration/physical_backup_restoration.py 20.13% 115 Missing ⚠️
press/press/doctype/site_backup/site_backup.py 40.65% 54 Missing ⚠️
...s/press/doctype/virtual_machine/virtual_machine.py 25.00% 18 Missing ⚠️
press/api/site_backup.py 0.00% 17 Missing ⚠️
...ype/virtual_disk_snapshot/virtual_disk_snapshot.py 16.66% 15 Missing ⚠️
press/agent.py 45.45% 6 Missing ⚠️
...storation_step/physical_backup_restoration_step.py 0.00% 5 Missing ⚠️
press/press/doctype/agent_job/agent_job.py 50.00% 2 Missing ⚠️
press/press/doctype/site/site.py 75.00% 1 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #2419      +/-   ##
==========================================
- Coverage   38.34%   37.98%   -0.36%     
==========================================
  Files         378      382       +4     
  Lines       29741    30035     +294     
==========================================
+ Hits        11404    11409       +5     
- Misses      18337    18626     +289

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@tanmoysrt
refactor(physical-restore): Reimplement the process flow
87b4050 
- Make the step by step method execution like
Virtual Machine Migration
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@balamurali27 balamurali27 Awaiting requested review from balamurali27 balamurali27 will be requested when the pull request is marked ready for review balamurali27 is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Physical backups and restore during migrations
1 participant
@tanmoysrt