Handlers are the lightest objects to operate Request/Response, so it will only reference/load as much data as you describe for it ...
For this specific case, all you need to do is implement the
IRequireSessionState
in the handler.
As the example right here:
public class Handler : IHttpHandler, System.Web.SessionState.IRequiresSessionState
Cheers.