Brown served as Horatio Hornblower's coxswain on the HMS Lydia, and the HMS Sutherland. On the HMS Nonsuch, he served as Captain Hornblower's coxswain and steward. Faithful, burly, Brown also served as Hornblower's butler at Smallbridge. He was admired by Hornblower for his cheerfulness and self-confidence.
Brown had served in the Royal Navy since he was only eleven years old. In 1810, when Brown was 28, the HMS Sutherland was captured by the French. Hornblower and Bush were to be taken to Paris for a mock trial and execution. Hornblower, allowed to take one person with him, asked for Brown to accompany them. Brown escaped with Hornblower and Bush, and wintered at the Chateau de Graçay. He escaped France alongside his captain and first lieutenant. He went on to serve Hornblower in later years as his butler in Smallbridge.