Have you seen the marriage entry of Francis and Charlotte? Do the witnesses provide any clues?
I think the two Rebecca Sabey burials in 1851, two days apart, are one and the same Rebecca. When the workhouse found where she was from they shipped her body back to Bolnhurst for burial.
I think Joseph was probably born in 1792, the son of William and Sarah (Throssell) - on baptism in Bolnhurst on 20 Oct 1795 he was age 3. But there's also a Bedford Gaol entry
Name: Joseph Sabey Age: 21
Date of Offence: 21st December 1814 Offense: Bastardy
Height: 5 ft 9 inches Hair colour: Brown Complexion: Fresh
Number of children: 0
Residence town: Colmworth
This was only a week before he married Rebecca. Whilst I can't see a baptism of a child I suspect that the Overseers leaned on him, obtained a marriage licence, and marched him down the aisle.
Thanks to John who got me heavily involved in Boaz Sabey, with whom I have no blood relationship, or more specifically the Lugsdens, I now have a tree on Ancestry with 837 people on it, not one of whom is connected to me. I'm now the world's leading expert on Lugsdens from Beds, London, Canada, US (but not Australia who has its own Lugsden expert).
It's possible that Lt Staughton Baptist Church book might include details of when Francis was admitted into membership of the church and where he was from.