Growing up in a devout Baptist family of five children, Oliver was instilled with a strong sense of community and a desire to assist those around him. He attended Acadia University, majoring in philosophy, and later Dalhousie University, where he earned a degree in law.
Called to the Bar in 1965, he began practising law in Nova Scotia and became active in the professional community, serving on the boards of several legal committees, in a career that spanned 36 years. Oliver maintained distinguished tenures both as a civil litigator and as an educator, teaching law at the Technical University of Nova Scotia, Saint Mary's University and Dalhousie University Law School. He has served also on the executive of several private companies and has lectured on human rights, the Canadian constitution and election law.
Oliver's community involvement led to a career in politics, and he was particularly interested in promoting equality for Blacks, First Nations and other minorities in Canadian society. Inspired by former Nova Scotia premier Robert Stanfield, Oliver began working with the Progressive Conservative Party in 1972, and remained involved with the party over the next 30 years. During that time he served as the director of legal affairs in six general elections between 1972 and 1988. After his appointment to the senate, he served as a member of standing Senate committees on banking, trade and commerce; agriculture and forestry; and was the chairman of the standing committee on transport and communications, as well as other Senate-House of Commons committees. Oliver has worked on several Private Member's Bills, including one to amend the section of the criminal code regarding stalking, and another to address the increasing problem of computer SPAM.
Oliver continued to be active in community service throughout his career, serving in positions that have included President and Chairman of the Halifax Children's Aid Society; Chairman, President and Director of the Neptune Theatre Foundation; Director of the Halifax-Dartmouth Welfare Council; Founding Director of the Black United Front; and Founding President and First Chairman of the Society for the Protection and Preservation of Black Culture in Nova Scotia. In 2003, Dalhousie University awarded him with an honorary doctorate in recognition of his lifetime of achievement, both in the public and private sectors.