Raja vacates PM House for caretaker