Frederick Avery, ULC Minister