The Crown Estate, which owns the building housing the restaurant, has declined to renew its lease, citing plans to renovate the property and convert parts of it into office spaces