Charles and Ann Rolfe had several children :
Charles 20 Apr 1802( died infant)
James 24 Nov 1803
George 13 Jun 1805
Henry William 31 Oct 1808
There is a will for a Charles Rolfe of Lydd dated 1842 in which he leaves estate to wife Ann and eldest son James. In the will Charles describes himself as a grazier but I don't think this should rule him out as your Charles. ( The will is available on Ancestry or from TNA by download for £3.33 or thereabouts )
I also believe that Charles Rolfe may be older than you think. This family in 1841 look possible :
Charles Rolfe b 1759 Kent Grazier
Ann Rolfe 70
James37
address Lydd
1851 census in Lydd
James47 farmer b Lydd
Hester
Wightwick 58 Aunt b New Romney ( This would confirm the Charles Rolfe / Ann Wightwick marriage)
There is also a will for Charles Rolfe the Elder , Gentleman of New Romney , dated 1811 which leaves estate to his son Charles and there is a baptism for Charles Rolfe 11 Sep 1759 in New Romney , parents Charles and Elizabeth. I think this Charles Rolfe , 1759 , is probably the correct person.
Both wills mentioned are on TNA or available from Ancestry. Transcripts of baptisms and marriages are here :
http://woodchurchancestry.org.uk/romneymarshbaptisms/index.phphttp://woodchurchancestry.org.uk/midkentmarriages/index.phpHope this helps !