Weird counter_cache behaviour, went unnoticed, is untested?

Hello,

in the current project i’m working on i have a simple relation model
setup

Gallery and Image
now i have the counter_cache set up correctly and it worked wonders
while i had only a simple crud.
Hovewer i am now implementing a manager for images and galleries, say
i always select one gallery and select images i want associated with
that gallery. After i select the images i do an ajax request with two
params, ID of the gallery and array of image ID’s which are to be
associated with the said gallery. Here to counter_cache fails my
expectations. I decided to let active_record handle all the changes so
i simply find my gallery via g=Gallery.find(id) and then set
g.image_ids=[1,2,3,4,5]

This does not increment the counter_cache correctly.

example:
say i have counter set to 2. I assign 5 more images via the manager,
counter cache is still 2, then i assign only 3 ids (means remove 4).
the counter cache ends up on -2.

So removing the images from gallery still updates the cache fine but
when i add them, the counter cache fail to do what it’s supposed to

Say do i use the counter_cache incorectly or should i assign the image
ids some other way for the cache to pick them up?