Ye has been travelling to Australia for years as it is where his wife of three years, Bianca Censori, was born