We have the Mitre Inn which was one of about 20 listed in the 1861 census. In 1851 there seemed to be only about four specifically named. Certainly no other licenced premises that come close to your 'Milners'.
However, in the 1841/51/61 period there was a family called MILNER in Kington who were wine merchants (probably originally from Shropshire), a son later (1861) being an Estate Agent & Auctioneer.
Sorry, can't find any references to Assembly Rooms. Hope someone else can help with that.
Chilis