Born Virginia Fortune Ryan on 9 February 1933, she was the daughter of John Barry Ryan Jr. and Margaret (Nin) Kahn. Her maternal grandfather was the renowned German-American financier and philanthropist Otto Kahn, a prominent figure in New York society and a leading patron of the arts. Virginia spent her childhood between New York City and Newport, Rhode Island, and was educated at the Brearley School in Manhattan, later attending the James Franklyn School of Professional Arts.
From an early age, Virginia developed a fondness for England. Each year, her mother brought her to London, where they stayed in a flat overlooking St James’s Park. During one such visit, at the age of 16, she met David Ogilvy at a dance at the Savoy Hotel. Three years later, he proposed, and in October 1952 they were married at St Margaret’s, Westminster, in a grand ceremony attended by Queen Elizabeth the Queen Mother and Princess Margaret.