In 1870 census there is a William Storrs/Starrs age 60, clergyman, born NY, is living in Wirt, Allegany, New York, with Lydia age 64, born NY, keeping house. Marital status not indicated.
Not seeing an Irish clergyman yet.
Added: In 1880 He is described as a farmer, age 70, living with Lydia, 75, all of their parents born NY, so not your man?
In 1855 he is with Lydia, his wife, and is a preacher.
In 1865 with Lydia, baptist clergyman, no children recorded for marriage.