Fixes crashes, deadlocks and assertions failure with #1650.
One cannot remove object references one does not own.

Reopens #1637.