On November 17, 1982, Stafford was appointed the second Bishop of Memphis, Tennessee, by Pope John Paul II. He was installed on January 18, 1983.
During his tenure, he revised the structure of the Pastoral Office, improved the fiscal conditions of the diocese, and concentrated on the evangelization of African Americans. On May 30, 1986, he was named Archbishop of Denver, Colorado.
He was chairman of the USCCB Commission for Ecumenical and Interreligious Affairs (1984–1991) and co-president of the Dialogue between Roman Catholics and Lutherans (1984–1997). He was also President of the Catholic Conference of the United States’ Adhoc Committee for the Evaluation of the Catholic Telecommunications Network (1987-1989).
As President of the Pontifical Council for the Laity beginning August 20, 1996, he organized the twelfth World Youth Day which took place in Paris, August 1997.
Pope St. John Paul II proclaimed him Cardinal February 21, 1998.