Having now looked at the 1861 census I may have to eat my words about fashionability. There are some very large and apparently well-to-do households in Westbourne Square, some with 5, 6 or 7 servants - so it does look as though they must have been large and imposing residences.
You may be interested to know that 22 Westbourne Square was occupied in 1861 by John J Wynne, a 57 yr old Irish-born barrister, with his wife Marcia, four children and an Irish-born female servant.