Note that we cannot hold the playlist lock while removing the callback,
since the callback function takes the playlist lock.