Measurement of cos 2β in B0→D(*)h0 decays with a time-dependent dalitz plot analysis of D→Ks0π+π

We study the time-dependent Dalitz plot of D→KS0π+π- in B0→D(*)h0 decays, where h0 is a π0, η, η′, or ω meson and D*→Dπ0, using a data sample of 383×106 Υ(4S)→BB̄ decays collected with the BABAR detector. We determine cos 2β=0.42±0.49±0.09±0.13, sin 2β=0.29±0. 34±0.03±0.05, and |λ|=1.01±0.08±0.02, where the first error is statistical, the second is the experimental systematic uncertainty, and the third, where given, is the Dalitz model uncertainty. Assuming the world average value for sin 2β and |λ|=1, cos 2β>0 is preferred over cos 2β<0 at 86% confidence level.