I thought I had a theory, but unfortunately I think I'm only confusing things even more!
A Joseph Palfrey Rowlands married in Carnarvon in 1843, and on the same page is Ellin Hughes. This appears to be them in 1851 at Mount Pleasant, Llanbeblig:
Joseph P Rowland 39 Mariner b Caernarvon
Ellin Rowland 25
George P Rowland 7
James Rowland 2
Ellin Rowland 3 M
ROWLANDS, ELLIN MMN: HUGHES, 1851 M Quarter in CARNARVON Volume 27 Page 325
Strangely I could only find one James Rowland/s MMN Hughes, and that was James Porter Rowlands who, according to the 1851 census you posted, is the son of Palfrey & Catherine, although the age doesn't fit with the birth reg, which was Sept Q 1848. Then to confuse things even more, Familysearch has an 1849 baptism for a James Porter Rowlands, son of Joseph Palfrey Rowlands & Ellen
https://familysearch.org/ark:/61903/1:1:FM98-9BDI had thought that George P Rowland might have been your missing Palfrey, but there are two birth entries, both Carnarvon, both MMN Hughes:
ROWLANDS, GEORGE 1843 D Quarter Volume 27 Page 318
ROWLANDS, PALFREY 1844 J Quarter Volume 27 Page 337
I lose track of Joseph & Ellin after 1851, but George (Palfrey?) becomes a baker and by 1871 is in the workhouse.
Also, although Joseph Palfrey Rowland/s gives his birth as c 1812, Familysearch has this baptism:
7 Oct 1821 Llanbeblig, Caernarvon
Joseph Palphrey Griffith Or Rowlands
Parents: Palphrey Rowlands & Jane Griffith
https://familysearch.org/ark:/61903/1:1:FM98-JPXI'm completely intrigued now as to what the birth certificate for Palfrey has to say!