Saving an image with the same file name causes the image to be deleted

[syndbg]

I have a Paperclip model Product with paperclip attachment column :image and I use a S3 bucket.
I upload the images through a Paperclip file input field.
When I upload 2 times (in less than 1) minute different images with same file name, the second one causes the image to be delete altogether.

Scenario:

• Upload image box.jpg for Product instance
• In less than 1 minute, upload a different image, but also named box.jpg
• The second upload causes the image to be deleted altogether, falling back to the default image.

Edit: configuration

• Rails 4
• Paperclip 4.3.0
• ActiveAdmin (1.0.0. pre1)

In the log what actually happens is 2 calls to S3 are issued:

[paperclip] deleting products/99/original.jpeg
[AWS S3 204 0.228651 0 retries] delete_object(:bucket_name=>"my_bucket",:key=>"products/99/original.jpeg")

[paperclip] deleting products/99/original.jpeg
[AWS S3 204 0.229262 0 retries] delete_object(:bucket_name=>"my_bucket",:key=>"products/99/original.jpeg")

[jyurek]

What does your attachment definition look like? It does delete, but only because it will them upload a new copy. Hmm, now that I think about this, it might be that S3 can occasionally do things out of order, and a delete and a write on the same key might confuse it.

[syndbg]

I was able to produce the above mentioned issue at work.
It'll take me some time to get the same setup at home.

I'll come back to this when I have replicated the setup and investigated the s3 logic.

I haven't made myself familiar with Paperclip but from the easy to navigate codebase, I assume that /~https://github.com/thoughtbot/paperclip/blob/master/lib/paperclip/storage/s3.rb#L382 has the picture twice for deletion instead of once. So somewhere where queued_for_delete gets appended, it happens more often than it should.

[EddieOne]

I'm having images deleted randomly, how was this fixed?

[caboteria]

I checked with Amazon support, and they confirmed that S3 operations are processed asynchronously, so it's quite possible for them to finish in a different order than they start.

To answer your question, S3 does process your requests as it receives them, but there's no guarantee which request will complete first. S3 is a highly distributed system, so these requests are being processed asynchronously.

It looks like we have at least a couple of possible fixes:

• don't delete at all, the update will overwrite the old file
• find the file version before updating, and delete that specific version, so we don't delete the new version if the operations are handled out of order

The second approach might depend on S3 bucket versioning so I don't know if it's a general approach. The first approach should always work, I think. Is there a reason that we're sending a delete and then an upload?

[caboteria]

Evidently this problem has been seen before because attachment#reprocess! explicitly preserves old files, and the note in the source indicates that this is "due to inconsistencies in timing of S3 commands":

paperclip/lib/paperclip/attachment.rb

Line 337 in 82cd378

# NOTE: Calling reprocess WILL NOT delete existing files. This is due to

As a workaround I think I'll enable :preserve_files. It's better to have some orphan objects in S3 than to lose objects.

[caboteria]

I added a note about this bug to the s3 wiki page.

/~https://github.com/thoughtbot/paperclip/wiki/Paperclip-with-Amazon-S3

[sidraval]

Thank you for reporting this. Unfortunately, we will be deprecating Paperclip and therefore will not have the bandwidth to address this issue.