Polymorphic Destroy

Is there a clever way to destroy just the link in Polymorphic
association? For instance, in the example they have folders containing
images. If I want to destroy the link from the folder to the image (not
the image and not the folder), do I have to search through the join
models to find the correct join model that links the folder and the
image, or is there a more clever way? That’s what I’m doing now, and it
feels a little dumb.

Thanks.