Whenever we try to get space in heap(using malloc or new) we can get out of heap space error if it exceeds the limit. But, we have virtual memory so why can't OS allocate that much space and then do swapping when I need some part of it or not?
Firstly, even the virtual memory does not imply an infinite amount of memory. I have run out virtual memory quite a few times.
Secondly and more commonly, there is a limit on the amount of memory each process can own by itself and a limit on the amount of memory each process can share with other processes, which are usually defined before the start of the process, especially on a multi-user environment. Those limits will not be increased automatically even if the process is running out of the limits, because that is basic meaning of the limits! That is why the OS cannot (or, you may say, will not) allocate the requested space even if there are plenty of freely available virtual memory left.
Thirdly, it may also happen because of memory fragmentation.
Fourthly, Java virtual machines (as I last heard of them) have its memory usage limit specified when it starts. Whether it uses of virtual memory or not, its usage of memory will be bounded by that limit.
Lastly, which may or may not the least, there are other situations.