In 1955 he began freelancing as a fashion artist for Harper's Bazaar. It was here that he first came into contact with photography. Inspired by the photographic contact sheets he saw passing through the art director's desk he decided to find a job as a photographer’s assistant. Whilst working for Flowers that he received his first photographic commission from Ernestine Carter, the then fashion editor of The Sunday Times.
In 1957 he was hired by British Vogue where he remained working until 1963. During this period he worked closely with top models of the period, including Joy Weston, Jennifer Hocking, Paulene Stone and Jean Shrimpton.
Through his inspirational fashion photography, Duffy encapsulated the mood of the 1960s and alongside the likes of Terrance Donovan and David Bailey, became a symbol of this progressive and alternative era.
Apart from Vogue, Duffy also worked for publications including Glamour, Esquire, Town Magazine, Queen Magazine as well as The Observer, The Times and The Daily Telegraph. He also worked on contract for French Elle for two periods the first between 1963 and 1968, and the second between 1971 and 1979.
As well as fashion photography, Duffy was the creative force behind record album sleeve art for three David Bowie album covers, most notably the iconic Aladdin Sane. He was also successful within the advertising world shooting award-winning campaigns for both Benson & Hedges and Smirnoff in the 1970s.
In 1965 Duffy was asked to create a Pirelli calendar, which he shot in Monaco. He was commissioned to shoot a second calendar in 1973, which he created in collaboration with British pop artist Allen Jones and air brush specialist Phillip Castle.