He may move out of Number Ten altogether for work and into the Cabinet Office at 70 Whitehall.