Have you traced his parents in the census? Just a few thoughts below on possible parents
1861 living in Syresham
James James 62 Ag Lab Biddlesdon, Buckinghamshire,
Hannah James 57 Ladborough, Warwickshire,
Charles James 19 Biddlesdon, Buckinghamshire
1851 Living in Biddlesdon, Buckinghamshire
James James 53 Farmer 12 Acres Shalstone, Buckinghamshire
Anna James 47 Farmer Wife Warwickshire
Joseph James 22 Farmers Son Syresham, Northamptonshire
Charles James 6 Farmer Son Syresham, Northamptonshire
1841 Living in Biddlesden
James James 35 Ag Lab Buckinghamshire
Anna James 35
Joseph James 13 Buckinghamshire
Clementina James 6 Buckinghamshi
Whilst there is a John James age 10 living in Syresham and working as a servant
Baptism for Clementina
https://familysearch.org/ark:/61903/1:1:NPY9-W4D and I think this is the birth of the son Charles with mother's maiden name Thacker from the GRO Birth Index
CHARLES JAMES Mother's maiden name THACKER
GRO Ref Sept Qtr 1841 BRACKLEY Volume 15 Page 210
Baptism of Hannah Thacker that ties into the birth place in 1861
https://familysearch.org/ark:/61903/1:1:JMNH-LKSI can't see a marriage to James James to Hannah Thacker at the moment -Maybe why the family used the Thacker surname on occasions
Kay