Avoid client_set_state on destroyed client