On a conjecture concerning character degrees of some $p$-groups