Brown served as Horatio Hornblower's coxswain on HMS Lydia, and the HMS Sutherland. On HMS Nonsuch, he served as Captain Hornblower's coxswain and steward. Faithful, burly, Brown also served as Hornblower's butler at Smallbridge. He was admired by Hornblower for his cheerfulness and self-confidence.
Brown had served in the Royal Navy since he was eleven years old.
In 1810, when Brown was 28, the HMS Sutherland was captured by the French. Hornblower and Bush were to be taken to Paris for a mock trial and execution. Hornblower, allowed to take one servant with him, asked for Brown to accompany them. Brown escaped with Hornblower and Bush, and wintered at the Chateau de Graçay. He escaped France alongside his captain and first lieutenant. He married the French girl he met that winter when they returned to the chateau after the Treaty of Fontainebleau.
He went on to serve Hornblower in later years as his butler in Smallbridge.