I agree that it would have been helpful to have been told about the previous thread, as we've been going over a lot of old ground. Still, I think I might have made a bit of progress...
I've been concentrating on Alice Ulysse nee Pullen, whose parents were George Pullen and Caroline Street. In 1861 Caroline said she was born in Islington, and I've been looking for her in 1851.
A strong match is in Brighton (HO107/1645 fo665 p18) where she appears with her parents William and Martha Streete and 3 siblings. All the children were said to have been born in Shoreditch, and there was a mass baptism at St Leonard's Shoreditch on 21 Jun 1835 of 5 children of William and Martha Streete, the 4 youngest of whom match the ones in the 1851 census. So I'm pretty certain that's the right family.
What I'm wondering now is whether Caroline (who we know is Alice's mother) may have had Henry before she married George Pullen, in which case his birth is likely to be found as Street(e). I haven't found anything yet to indicate whether his father was a Budd, or whether the Budd family might simply have adopted him.
If I'm right, Henry's life post-army and his account of his parents might be a kind of conflation of bits of his background: in 1851 Caroline's father was an eating house keeper - as was Henry later. And for whatever reason, maybe he was taken in by Robert Budd and his family, and viewed them as his parents. (They ended up in Sheffield, where Henry said his mother lived in his army papers.)
The 1891 census indicates that even if all this did happen, Henry still knew about his sister Alice. In fact since there were lots of Budds and Pullens around the Petworth area, and there are intermarriages, it's by no means impossible that George Pullen was related to the Budd family in some way.
Can anyone recall, please, if we have found Henry in 1861, as this might shed considerable light on this (he's not with Caroline). Other things that might help would be the wills of Robert Budd and his wife, and, if they exist, those of William and Martha Streete. (I haven't found their deaths.) Also deaths and wills for Caroline Pullen nee Streete, and possibly her husband George.