Well, although his age is out by 5 years I believe your Joseph was loging at the Luther E Clarke Victoria Home, Whitechapel, London in 1891.
Joseph Ivey, lodger, single, 35, brushmaker, employed, birthplace; On the High Seas.
In 1881 his birthdate is placed at 1859, so still out a bit, and there aren't any other Joseph Ivey's who were brushmakers, except this one!