In 1881, as you say, James is in Gretton with his large family.
In 1871, in Barrowden there's
James Coleman Head 30 Ag Lab Barrowden
Sarah " Wife 29 Seaton
John " Son 8 Scholar "
Josiah C. " Son 6 " Barrowden
William " Son 4 " "
James " Son 2 " "
RG10/3306/49/15
and in 1861, James is a carter at John Balls' farm in Barrowden. He's listed as unmarried then.
There's a possible marriage with James Coleman and Sarah Cobley listed in 1st quarter 1862 Uppingham Vol 7a page 377. This might give you his father.
Good luck
Isobel