I agree wholeheartedly with Hugatree.
I believe your Charles James Baines was born 28/3/1845 Irlam, Lancashire, son of Thomas Baines and Fanny Higgins (per online tree and aforementioned census). He had 3 brothers and 3 sisters - can account for all 6 of them, and the deaths of Thomas and Fanny but there is no further sign of Charles James Baines after the 1861 census. No marriage, no death, or appearance in census in the UK after 1861. It all fits that he could have ended up in South Africa but appears to have been a natural son and not adopted. I can't find any trace of him in military service records unfortunately.
The family were quite well to do and his siblings all seemed to marry well too. The family were in London in 1861 and Fanny died there - death Dec.qtr.1861 Wandsworth. Thomas still in London 1871 but was back in Lancashire in 1881. 1881 Census shows him as a boarder, late Parliamentary Agent, born Leeds (ref.RG10 - 3695 - 6 - 6). His death is reg'd Dec.qtr.1881 West Derby, Lancashire.
A Public Tree on A...y has a picture of his father Thomas (and his father Edward.
Thomas b.8/4/1806 Leeds (familysearch), son of Edward Baines, MP and Charlotte Talbot. You can glean more by looking at the tree.
There is certainly no family that fit in Norfolk and the Charles James Baines b.1845, son of Thomas and Fanny (pet form of Frances), effectively 'disappears' between 1861 and 1871 never to be seen in English records again.
Annette