First of all, if you haven't already, do a Google search for "out of core solver". That's the name of the class of general solutions to your problem.
Having said that, you may be able to give yourself a little breathing room if you're only a little bit too big. In Windows, there is a setting you can set in your app that bumps the maximum from 2GB up to 3GB, as long as the OS itself is configured to allow for that. In Microsoft VC++, the command line option is /LARGEADDRESSAWARE, and is in the System section of the Linker Project options in the IDE. My understanding is that an app compiled with this option run on a 32-bit system gets 3GB max, and run on a 64-bit system gets 4GB max, while without this option, 2GB is the max.
Note that there is another problem with 32-bit apps, which is that they also have a 32-bit address space. This means that memory fragmentation inside the app can also have a big impact, because in order to allocate a single array that is 250MB in size, your app needs 250GB of contiguous space inside its address space. If you perform a lot of allocations and deallocations of big things, you'll quickly get to a position where even though the app isn't using anywhere close to 2GB (or 3GB if the option is set), a large memory allocation will still fail. You can mitigate this problem to some degree by allocating your large arrays up front and never deallocating them, but instead reusing them.
Having said that, you may be able to give yourself a little breathing room if you're only a little bit too big. In Windows, there is a setting you can set in your app that bumps the maximum from 2GB up to 3GB, as long as the OS itself is configured to allow for that. In Microsoft VC++, the command line option is /LARGEADDRESSAWARE, and is in the System section of the Linker Project options in the IDE. My understanding is that an app compiled with this option run on a 32-bit system gets 3GB max, and run on a 64-bit system gets 4GB max, while without this option, 2GB is the max.
Note that there is another problem with 32-bit apps, which is that they also have a 32-bit address space. This means that memory fragmentation inside the app can also have a big impact, because in order to allocate a single array that is 250MB in size, your app needs 250GB of contiguous space inside its address space. If you perform a lot of allocations and deallocations of big things, you'll quickly get to a position where even though the app isn't using anywhere close to 2GB (or 3GB if the option is set), a large memory allocation will still fail. You can mitigate this problem to some degree by allocating your large arrays up front and never deallocating them, but instead reusing them.