It does look as though William was Sarah's brother. There are a lot of William Perry's around. I did find one in a later census with Thomas and Ann parents but he was down as unmarried maybe he was widowed.
I will look into it some more. The family did move around a lot!