Harriet E. "Hattie" Adams Wilson was born in Milford, New Hampshire, the daughter of Joshua Green, a Black "hooper of barrels," and Margaret Ann (or Adams) Smith, a washerwoman of white-Irish ancestry. Her father died when she was very young, and her mother abandoned her at the farm of Nehemiah Hayward Jr., a well-to-do Milford farmer.