He spoke for the last time in the House of Lords on 25 June 1804, during the debate on the second reading of the Additional Force Bill, which he condemned as a feeble and inadequate measure (Parl. Debates, 1st ser. ii. 832, 833). He died at Goodwood, Sussex, on 29 Dec. 1806, in the seventy-second year of his age, and was buried in Chichester Cathedral, his body having been first opened and filled with slack lime, according to his directions.
Richmond was a remarkably handsome man, with a dignified bearing and graceful and courteous manner. As a politician he was both hasty and ambitious. Though an indifferent speaker, ‘at the East India House, in his quality of a proprietor, no less than as a peer of parliament at Westminster, he was ever active, vigilant in detecting and exposing abuses, real or imaginary, perpetually harassing every department with inquiries, and attacking in turn the army, the admiralty, and the treasury’ (Hist. and Posth. Memoirs of Sir N. Wraxall, ii. 60). Horace Walpole, who never tired singing Richmond's praises, worshipped ‘his thousand virtues beyond any man's,’ and declared that he was ‘intrepid and tender, inflexible and humane beyond example’ (Letters, vii. 379). But Burke, while drawing a long and flattering picture of Richmond, expresses his opinion that ‘your grace dissipates your mind into too great a variety of minute pursuits, all of which, from the natural vehemence of your temper, you follow with almost equal passion’ (Correspondence, i. 376).
Richmond married, on 1 April 1757, Lady Mary Bruce, the only child of Charles, third earl of Ailesbury and fourth earl of Elgin, by his third wife, Lady Caroline Campbell, only daughter of John, fourth duke of Argyll. ‘The perfectest match,’ says Walpole, ‘in the world—youth, beauty, riches, alliances, and all the blood of the kings from Bruce to Charles II. They are the prettiest couple in England, except the father-in-law and mother’ (Letters, iii. 67). The duchess died at Goodwood on 5 Nov. 1796, without issue, and was buried in Chichester Cathedral on the 14th of the same month. Richmond left four illegitimate daughters, and was succeeded in his honours by his nephew, Charles, the only son of his younger brother, Lord George Henry Lennox.
Richmond was gazetted a major-general on 9 March 1761, lieutenant-general on 30 April 1770, general on 20 Nov. 1782, colonel of the royal regiment of horse guards on 15 July 1795, and field-marshal on 30 July 1796. He was elected F.R.S. on 11 Dec. 1755, and F.S.A. on 6 June 1793. He was a patron of literature and of the fine arts, and in March 1758 opened a gratuitous school for the study of painting and sculpture in a gallery in his garden at Whitehall, engaging Giovanni Battista Cipriani the painter and Joseph Wilton the sculptor to direct the instruction of the students (Walpole, Anecdotes of Painting, 1849, i. xiii.; Edward Edwards, Anecdotes of Painters, 1808, pp. xvi–xix). The collection of casts from the antique formed by Richmond for this purpose was the first of the kind in England. Some of them eventually came into the possession of the Royal Academy (Leslie and Taylor, Life of Sir Joshua Reynolds, i. 158–9, 316). Horace Walpole dedicated to Richmond the fourth volume of his ‘Anecdotes of Painting,’ printed at Strawberry Hill in 1771. Several of Richmond's letters will be found in the ‘Correspondence’ of Burke and Chatham respectively, and also in Lord Albemarle's ‘Memoirs of the Marquis of Rockingham,’ where some extracts from his ‘Journal’ kept during the last days of the first Rockingham administration are printed. The authorship of ‘An Answer to a short Essay [by James Glenie] on the Modes of Defence best adapted to the situation and circumstances of this Island,’ London, 1785, 8vo (anon.), is attributed to Richmond in the catalogue of the Advocates' Library.
Richmond sat twice to Sir Joshua Reynolds, who also executed a copy of one of these portraits for his wife's stepfather, Henry Seymour Conway [q. v.] A portrait of Richmond, painted at Rome by Pompeo Battoni, and another by Gainsborough, are in the possession of the present Duke of Richmond. The half-length portrait of Richmond by Romney, which belonged to Baroness Burdett Coutts, was engraved by James Watson in 1778. The duchess sat to Sir Joshua Reynolds no less than seven times. Richmond House, Whitehall, was destroyed by fire on 21 Dec. 1791 (Ann. Reg. 1791, Chron. pp. 52*–4*).