Too cold to plant?

It's been unseasonably cold outside, which brings me to this question: is it too cold to plant our peppers?
We bought four plants at Home Depot in Yonkers when we visited Mitch's parents for Passover. The Yonkers store has a big outdoor department that kick's butt over Manhattan's, so we go each spring to stock up on soil, mulch and some herbs or veggies like peppers. (We've tried to grow peppers from seed, but we've had no luck in getting them to sprout.)
So, now we have four bell pepper plants (green, yellow, orange and red) but Mitch won't let me stick 'em in the pots. He says it's too cool. He wants me to wait.
But I'm like, so, are they gonna catch cold?
All of our herbs from last year (mint, oregano, thyme and parsley)have come back and are growing nicely. Our spring flowers (columbine, pansies) are blooming. The roses are coming along.
But he's worried about the peppers? Of course, it won't kill them to hang out in the apartment for another week or so. But I think he's being a bit like an overprotective parent.
I'm thinking that if we plant them a little early this year, before Memorial Day, they'll get a head start and they'll yield their fruit sooner too.
So, if you're a planter, let me know what you think.
Thanks!