Memcache - any understand how this would work?

Ok, our question is this. We have memcache running on each of our
application servers. (Lets say 4). When one of those servers goes
down (say server 1), the other 3 servers have no way to retrieve a
needed cached item from server 1, so it goes to the DB - etc to
construct what is needed. Now, can that item be recached on servers
2-4? When server 1 goes back up, can that cache be automatically
cleaned out so that there isn’t any stale data now being served?

Afaik, this hapens automatically. The memcached design is such that
servers are not aware of each other. The client maintains a list of
servers which it queries in order for data. The servers themselves
just store and discard data on a FIFO basis, and on expiry values. So,
as soon as one server goes down, the others will pick up the slack,
and as soon as it comes back up (I assume it’s a h/w failure) its
cache will be flushed and it’ll restart from scratch.

I’m not sure what’ll happen if there’s a network read error and I
suspect stale data can be served. However, counter this by setting
good expiry values for cached data.

Vish