Varsity House construction photos are available, including several depicting the dismantling of part of the East Stands at Memorial Field.
Maps now label the old Davis Varsity House as “Davis Field House.” A dedicatory plaque inside the building also seems to use that name, so perhaps the “Varsity” part has been inaccurate all along.