Do you think this might be your James at Fitzroy? The details are not exactly right, but pretty close.
Go to
http://www.rootschat.com/links/0195j/ Put the number 515 in the VPRS box
Put HARVEY in the 'Any words' box and Search
There are four pages so use the arrow to go to page 4 and then look for:
Harvey, James: No. 14202Click on the link below the name then on the next page you will see a link for the PDF.
On this 1876 record he is a Shoemaker, born England c1812, and the story that goes with this conviction is here
http://trove.nla.gov.au/ndp/del/article/58154358The son who is the 'gentleman' is the right age for James Jnr.
Now go to page 3 and there is another entry:
Harvey, James: No. 14938On this one for the following year he is a Shoemaker born London c1809 and the notes say he arrived on the
Waterloo in 1839 (just a few years out). I can't see an article that goes with this one.
It is a shame they were minor offences because you would have gotten photos if he had been gaoled for longer, usually 6 months or more.
You can search Trove using this term including the inverted commas
"Harvey Fitzroy"~20 and it will find all articles where those words are within 20 words of each other. You can change the number to whatever you like to narrow or broaden your results.
March 1878
http://trove.nla.gov.au/ndp/del/article/59574441James Harvey is charged, on warrant frorn the Fitzroy Bench, with disobeying a summons for using obscene language on the 11th ultimo. Description :- English, a shoemaker, 67 years of age, 5 feet 10 or 11 inches high, rather stout, dark complexion, hair, beard, whiskers, and moustache; dressed in old grey tweed trousers and vest, old dark coat, and black soft felt hat; very dirty and slovenly looking.So he hasn't turned up to Court on this occasion but they did find him because this was a week later:
James Harvey, a disreputable dirty looking old man was charged with using obscene language. Con. Walch described the language used as being very obscene. A Mr. John Smith was called to prove the language used by Harvey. He stated for the language Harvey had used to his wife and family he ought to be hanged. Harvey pleaded that the language he had used was caused by his son keeping very late hours. James was fined 40s. or a month's imprisonment, for which he thanked the bench.
http://trove.nla.gov.au/ndp/del/article/59574487I can't see anything else apart from the incident you mentioned in 1866.
Debra