Numerical study of the effect of instability on detonation dynamics and critical phenomena