Penalized likelihood estimation for semiparametric mixed models, with application to alcohol treatment research