Destroy options on join table

Suppose I have the join as below. Is there a way to make it only
destroy the field_map only when there is no longer a reference to
it ?

has_many :field_map_joins, :dependent => :destroy
has_many :field_maps, :through => :field_map_joins,
:dependent => :destroy