No, there are no drawbacks UNLESS you are using a 32-bit OS with about 4GB of memory. Your program can use up to 3GB of memory as a 32-bit app with the LAA flag turned on. In that situation you could find the OS starved for memory and constantly paging so performance will slow to a crawl. This is a fairly unlikely scenario today but I saw it happen about twenty years ago. If that is not your situation then the only real drawback will be that it masks a potential problem. If your program must remain as 32-bit and it must address more than 2GB of memory then that's will you will have to do.
If I were you, I would look at tracking your allocations so you can find where you are leaking memory. MFC has this built in using the DEBUG_NEW macro. If you are not using MFC then take a look at this
Memory Allocation Tracking for C++ Code[
^] and follow its guidelines.
This is a particularly important issue for me because I write automation applications that must run non-stop for months and months. Memory leaks are an obstacle to program stability and I make sure that my programs have none.