Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enhance file cache to encrypt data written to the cache file #1192

Merged
merged 5 commits into from
Apr 17, 2023
Merged

Enhance file cache to encrypt data written to the cache file #1192

merged 5 commits into from
Apr 17, 2023

Conversation

jiangliu
Copy link
Collaborator

Enhance the FileCache subsystem to optionally encrypt data written to the local cache file.

@jiangliu jiangliu requested a review from a team as a code owner March 31, 2023 16:43
@jiangliu jiangliu requested review from liubin, changweige and adamqqqplay and removed request for a team March 31, 2023 16:43
@anolis-bot
Copy link
Collaborator

@jiangliu , a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/63312

@anolis-bot
Copy link
Collaborator

@jiangliu , The CI test is completed, please check result:

Test CaseTest Result
build rust golang image✅ SUCCESS
compile nydusd❌ FAIL

Sorry, your test job failed. Please get the details in the link.

@jiangliu
Copy link
Collaborator Author

A sample nydusd.conf to test the encryption. You may tweek with enable_convergent_encryption and enable_encryption.

version = 2
id = "id1"
backend.type = "localfs"
backend.localfs.dir = "/root/ws/image-service.git/images"
cache.type = "filecache"
cache.compressed = false
cache.validate = true
cache.filecache.work_dir = "/root/ws/image-service.git/cache"
cache.filecache.enable_encryption = true
cache.filecache.enable_convergent_encryption = false
cache.filecache.encryption_key = "fc4a7db5614afc2f400e9478bebed1aefdbc9d7cd03210b84f144683a7a6fd1a"
rafs.mode = "direct"

@anolis-bot
Copy link
Collaborator

@jiangliu , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/63313

@anolis-bot
Copy link
Collaborator

@jiangliu , The CI test is completed, please check result:

Test CaseTest Result
build rust golang image✅ SUCCESS
compile nydusd❌ FAIL

Sorry, your test job failed. Please get the details in the link.

@anolis-bot
Copy link
Collaborator

@jiangliu , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/63314

@anolis-bot
Copy link
Collaborator

@jiangliu , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/63315

@anolis-bot
Copy link
Collaborator

@jiangliu , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/63316

@anolis-bot
Copy link
Collaborator

@jiangliu , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/63317

@anolis-bot
Copy link
Collaborator

@jiangliu , The CI test is completed, please check result:

Test CaseTest Result
build rust golang image✅ SUCCESS
compile nydusd✅ SUCCESS
compile ctr remote✅ SUCCESS
compile nydus snapshotter✅ SUCCESS
run container with rafs✅ SUCCESS
run container with zran✅ SUCCESS
run container with rafs and compile linux✅ SUCCESS

Congratulations, your test job passed!

@anolis-bot
Copy link
Collaborator

@jiangliu , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/63333

@anolis-bot
Copy link
Collaborator

@jiangliu , The CI test is completed, please check result:

Test CaseTest Result
build rust golang image✅ SUCCESS
compile nydusd✅ SUCCESS
compile ctr remote✅ SUCCESS
compile nydus snapshotter✅ SUCCESS
run container with rafs✅ SUCCESS
run container with zran✅ SUCCESS
run container with rafs and compile linux✅ SUCCESS

Congratulations, your test job passed!

@anolis-bot
Copy link
Collaborator

@jiangliu , The CI test is completed, please check result:

Test CaseTest Result
build rust golang image✅ SUCCESS
compile nydusd✅ SUCCESS
compile ctr remote✅ SUCCESS
compile nydus snapshotter✅ SUCCESS
run container with rafs✅ SUCCESS
run container with zran✅ SUCCESS
run container with rafs and compile linux✅ SUCCESS

Congratulations, your test job passed!

@anolis-bot
Copy link
Collaborator

@jiangliu , The CI test is completed, please check result:

Test CaseTest Result
build rust golang image✅ SUCCESS
compile nydusd✅ SUCCESS
compile ctr remote✅ SUCCESS
compile nydus snapshotter✅ SUCCESS
run container with rafs✅ SUCCESS
run container with zran✅ SUCCESS
run container with rafs and compile linux✅ SUCCESS

Congratulations, your test job passed!

@ccx1024cc
Copy link
Contributor

ccx1024cc commented Apr 4, 2023 •
edited
Loading

A sample nydusd.conf to test the encryption. You may tweek with enable_convergent_encryption and enable_encryption.

version = 2
id = "id1"
backend.type = "localfs"
backend.localfs.dir = "/root/ws/image-service.git/images"
cache.type = "filecache"
cache.compressed = false
cache.validate = true
cache.filecache.work_dir = "/root/ws/image-service.git/cache"
cache.filecache.enable_encryption = true
cache.filecache.enable_convergent_encryption = false
cache.filecache.encryption_key = "fc4a7db5614afc2f400e9478bebed1aefdbc9d7cd03210b84f144683a7a6fd1a"
rafs.mode = "direct"

Please add e2e tests, instead of in manual way~~~~

@jiangliu
Copy link
Collaborator Author

A sample nydusd.conf to test the encryption. You may tweek with enable_convergent_encryption and enable_encryption.

version = 2
id = "id1"
backend.type = "localfs"
backend.localfs.dir = "/root/ws/image-service.git/images"
cache.type = "filecache"
cache.compressed = false
cache.validate = true
cache.filecache.work_dir = "/root/ws/image-service.git/cache"
cache.filecache.enable_encryption = true
cache.filecache.enable_convergent_encryption = false
cache.filecache.encryption_key = "fc4a7db5614afc2f400e9478bebed1aefdbc9d7cd03210b84f144683a7a6fd1a"
rafs.mode = "direct"

Please add e2e tests, instead of in manual way~~~~

It has long dependency chain to enable unit test for this feature. We need to enhance nydus/snapsotter/converter, nydusify and smoke:)
So it would be great to merge this first so we can do it step by step.

@anolis-bot
Copy link
Collaborator

@jiangliu , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/67367

@anolis-bot
Copy link
Collaborator

@jiangliu , The CI test is completed, please check result:

Test CaseTest Result
build rust golang image✅ SUCCESS
compile nydusd✅ SUCCESS
compile ctr remote✅ SUCCESS
compile nydus snapshotter✅ SUCCESS
run container with rafs✅ SUCCESS
run container with zran✅ SUCCESS
run container with rafs and compile linux✅ SUCCESS

Congratulations, your test job passed!

jiangliu added 5 commits April 15, 2023 16:33
@jiangliu
storage: add flag to indicate encrypted data chunk
d31c3b3 
Add method and flag to indicate that a data chunk is encrypted or not.

Signed-off-by: Jiang Liu <gerry@linux.alibaba.com>
@jiangliu
storage: add fields for chunk encryption
5f1fc40 
Add data fields to BlobInfo and CacheFile for chunk encryption.

Signed-off-by: Jiang Liu <gerry@linux.alibaba.com>
@jiangliu
storage: introduce struct CipherContext
f82cf6d 
Introduce struct CipherContext for data encryption/decryption.

Signed-off-by: Jiang Liu <gerry@linux.alibaba.com>
@jiangliu
api: add encryption configuration to file cache
37273bf 
Add encryption configuration to file cache, so we can encrypt data
written to the local cache file.

Signed-off-by: Jiang Liu <gerry@linux.alibaba.com>
@jiangliu
storage: encrypt data in local cache file
2a23e99 
Encrypt data before writing data to local cache file.

Signed-off-by: Jiang Liu <gerry@linux.alibaba.com>
@anolis-bot
Copy link
Collaborator

@jiangliu , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/67719

@anolis-bot
Copy link
Collaborator

@jiangliu , The CI test is completed, please check result:

Test CaseTest Result
build rust golang image✅ SUCCESS
compile nydusd✅ SUCCESS
compile ctr remote✅ SUCCESS
compile nydus snapshotter✅ SUCCESS
run container with rafs✅ SUCCESS
run container with zran✅ SUCCESS
run container with rafs and compile linux✅ SUCCESS

Congratulations, your test job passed!

imeoer
imeoer reviewed Apr 17, 2023
View reviewed changes
api/src/config.rs
pub enable_convergent_encryption: bool,
/// Key for data encryption, a heximal representation of [u8; 32].
#[serde(default)]
pub encryption_key: String,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this need to be persisted in local disk?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hope we can pass it by API next.

@imeoer imeoer merged commit 0dc95f8 into dragonflyoss:master Apr 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@imeoer imeoer imeoer approved these changes

@liubin liubin Awaiting requested review from liubin liubin is a code owner automatically assigned from dragonflyoss/nydus-reviewers

@changweige changweige Awaiting requested review from changweige changweige is a code owner automatically assigned from dragonflyoss/nydus-reviewers

@adamqqqplay adamqqqplay Awaiting requested review from adamqqqplay adamqqqplay is a code owner automatically assigned from dragonflyoss/nydus-reviewers

Assignees
No one assigned
Labels
anolis_test_pass
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@jiangliu @anolis-bot @ccx1024cc @imeoer