In case of no timeout this timer is not removed, leading to the re-addition in a later call. This fix removes it properly.