Joseph added the profession of physician to that of clothier. Mary Ayer's father was a merchant in Boston, and a large landowner in Haverhill. In 1691-2 in Ipswich, "Mr. Joseph Calef had a grant of land with Liberty to set up a Fulling Mill in any Convenient place not allready given or granted to another . . . provided it be done and finished within twelve months after this day upon these conditions that he full cloth for ye Inhabitants for their pay sooner than for other townsmen." A fulling mill and saw mill on Mill River were finished just in time for the wedding, and the young people set up housekeeping with an uncommonly good start in life for those pioneer days.
In 1699 he is one of the subscribers for the bell of the New Meetingg House and his seat in the church is one of position. In Ipswich the woman of the leading families sat against the wall and "ye fifth pew on ye south east side of ye great door" was Mary's. Joseph sat with the men in the fourth seat, one occupied only by those bearing titles of respect. His stone in the First Parish graveyard reads:
Here Lyes What Was Mortal of Mr. Joseph Calef who Died December ye 28, 1707 in ye 36 year of His age.
Mary Ayer was an able woman and brought up the six children and a grandson well. Not until all her children were grown up did she marry Thomas Choate and so happy were the relations with this stepfather that he left a legacy to her daughter Mary Calef White, while his son by a former marriage, Col. John Choate, was the guardian of her orphan grandson, John Calef, and left him a legacy. Her grave is in the old Essex yard which once was a part of Ipswich, then called Chebacco, where was Thomas Choate's home.