Duffy and his wife Rachel have nine children. The youngest, a girl, has Down syndrome. In September 2019, at age 47, Duffy resigned from Congress around the time of her birth, citing the need to spend more time with his family and to help care for his youngest daughter.