Dear Helen
Don't know if you are still pursuing this (quite old) thread, but I have only just found it while Googling for other WALFORD ancestors.
I believe that the Charles Henry WALFORD (b1836?) who starts this thread was the second son of Gainsford Gibling WALFORD (GGW) who was the miller found in the 1841 Little Wilbraham census by Bryant. I have him born in Stratford St Mary, Suffolk, as was his brother Walter Gainsford WALFORD, my great grandfather. His is the family, including my grandmother, which Bryant found in the 1881 census living at Brundon Hall near Sudbury.
GGW died of severe bronchitis at 8 Napier Street, Shoreditch on 04 Jun 1859 with Jane WALFORD, probably his wife, present at the death. In case you didn't find them in the 1851 census (not helped by Ancetsry's incorrect transcription to read 'Cawford Santford!!) they were living at 11 Hulls Terrace, Old Luke Street, Middlesex.
I have GGW's wife as Jane HARDY, born in Bures St Mary, Suffolk, about 1806. Parents = Abraham HARDY and Jemima GOOD(E) both born about 1780.
But I have got no further back than these two on Jane's side and no further that mother, Ann WALFORD, on Gainsford's side. i.e. no father for Gainsford, husband for Jane or children after Gainsford. (Hadn't seen the 'bastard son' reference by the way. Is this from parish records?)
I'll follow up on the other 3 Giblin(g) WALFORDs that Bryant found and the George that you found and let you know if I find anything - assuming you're still interested in this line.
Kind regards, Bob Marshall