I have actually found an Ezra Abel matching your description in Berrien Michigan in the 1880 census. He is married to Charlotte and has a daughter M Lydia, 29, born in NY and a daughter Louisa, 26, born in Pennsylvania, and a grandson, Larue, 4, son of Louisa.
In 1860, in Berrien, Michigan, we have Ezra and Charlotte, along with Edward, who is 19, George who is 13, Mary L, who is 8 (M. Lydia), and Frances L, who is 7 (Louisa?).
In the 1850 census, I have found him in Claverack, NY with Charlotte, Edward and George, but no other siblings or parents. I will search for them separately.
Kath