Is this your James? From the 1891 census:-
James Davis, Head, 24, Coal Miner, Born East Dean
George Davis, Brother, 19, Coal Miner, Born East Dean
William Davis, Brother, 17, Coal Miner, Born East Dean
Mary Ann Turner, Aunt, 67, House Keeper, Born East Dean
George Fowler?, Lodger, Widower, Coal Miner, Born West Dean
Beatrice Ryeland, Visitor, 6, Scholar, Born Painswick
Living at Blakeney Hill, East Dean RG12/2006 Folio 6 Page 8
Sue