Starting out as the home of the Abbot of St Mary's Abbey in the 13th century the Manor has grown to the house we see today. Following the Dissolution of the monasteries, the Manor was retained by the Crown and was used as the base for the Council of the North. The building grew significantly during the reign of Elizabeth I often reusing stone reclaimed from St Mary's Abbey. The Council of the North was abolished in 1641 and the building gradually declined thereafter.
The Manor became the home of the Yorkshire School for the Blind in the 1830's and on the departure of that organisation in 1958 ownership passed to the City Council. They leased the building to the University and it is currently occupied by the Department of Archaeology