Are you sure he was born 1829?
Baptism;
30 Aug 1834, St John the Baptist, Frome
CHARLES Millgrove, Son of CHARLES/FANNY
Birth 31 July 1832***
Fathers Occp Labourer
Charles MILGROVE/Fanny KNIGHT married 19 April 1829, St Leonard, Marston Bigot
Witnesses, Eliza Milgrove/Thomas Yates
Other childrens Baptisms to CHARLES or CHAS/FANNY same Church;
ELIZA ANN 14 Nov 1829
THOMAS 6 Feb 1836
WM 22 Dec 1838
All on FreeREG
I've looked at 1841/1851 Census for Charles/Fanny and their Daug Louisa 1849 is with her Uncle/Aunt Moses/Mary Carver in 1861 Census.
I presume you have their marriage 1832 Mary MILLGROVE to Moses CARVER;
Mary Carver is born c 1806-1811 Frome on Census.
Death Reg has her born c 1806
FreeREG has 2
13 Feb 1808 to THOMAS/GRACE
28 Sept 1811 to WILLIAM/ELIZABETH (age 3 years)
1841 living near to Moses/Mary are;
GRACE Millgrove, 70
James, 30
Henry, 25
Henry married 1851 age 32, Father THOMAS, occp Collier
Ambrose/Maria Millgrove are witnesses
Thomas M married Grace COX 10 Aug 1802 Frome
(It does not say he is Widowed)
I was trying to prove Mary Millgove/Carver may be a half sister to Charles c 1794 if his Father Thomas was married twice.
I cant locate Ambrose M, but there is a Burial for a MARIA Milgrove 1881 Buckland Dinham age 74
(cant find either on Census)