Jim Gaffigan Jokes Raising Five Kids in New York City Was a ‘Huge Mistake’ (Exclusive)

Jim Gaffigan is giving a glimpse into raising his five kids in the Big Apple.

On Wednesday, the comedian, 57, attended the Garden of Laughs comedy benefit held at The Theater at Madison Square Garden and quipped with PEOPLE about the difficulty of raising his five kids he shares with wife Jeannie Gaffigan — Marre, 19, Jack, 17, Katie, 14, Michael 12, and Patrick, 11 — in New York City.

“It was a huge mistake,” Gaffigan joked about living in N.Y.C. with his family of seven.

However, the dad of five thinks that his children have enjoyed growing up in such a big city, while acknowledging how much it has changed. “I mean, also, it’s different, right? Because now there’s — I’d love to walk a block in New York City and not smell weed. You know what I mean?” Gaffigan said.

“When you’re in your twenties, you’re like, ‘Yeah, there should be no rules.’ And then when you’re a parent, you’re like, ‘Hey, maybe not every block.’ And of course, I live in the East Village, so I’m kind of asking for it,” he continued.

Wednesday’s comedy show was a sold-out event and raised more than $2 million for the Garden of Dreams foundation — the biggest fundraiser in the organization’s history.

Last March, Gaffigan shared that raising five kids can be “terrifying” during an appearance on Today. “It’s wonderful and really, really hard,” he shared. “What no one tells you is that when they get older, it gets so much harder. It’s just a different type of hard.”

Never miss a story — sign up for PEOPLE’s free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories. 

The star added that one of the most “frustrating” things as a parent is “being able to communicate some of the things that I might have learned in a way where they might hear it.”

However, Gaffigan also said that one of the best parts of parenting is watching his young ones find what they’re best at. “I know that it sounds corny, but as an adult you recognize how valuable it is to have something to be passionate about, to pour your energy into,” he shared, adding, “That’s the guaranteed layup for me, knowing that they have a passion so they’re not going to be lost.”