I don't want a man that already has kids..2 years too late?

When I met my husband he already had two children from his ex girlfriend. I was all ready to be a loving stepmom and have kids of our own. After dealing with his ex which i hate, the other children(2 and 4), and all the drama with his family wanting him to be with her(I did not steal him, they were long broken up) , i don't want to do it anymore. Not to mention he cheated with her once. Although I love my husband and he loves me very much, i am depressed all the time beccause of this. We have a 15 month old so it would be very hypocritical of me to go looking for a man without kids. Its like a loselose situation. What to do?