A computer typically has vastly more hard disk space that RAM. To compensate for limitations in the size of RAM, the operating system reserves part of the disk to use as additional RAM. This is called virtual memory. The RAM and the virtual memory are then divided into even chunks called pages. From the programmer's perspective, there is a large amount of memory available. Under the hood, the operating system keeps some of the pages of memory in RAM and leaves the remainder in virtual memory, on the disk. The operating system tries to ensure that the more commonly used pages are in RAM, as this is more efficient. If a program needs to access memory that it stored on disk, the operating system swaps out a page from RAM and swaps in a page from the disk. This is called paging. Paging is expensive, so the operating system tries to reduce how often it occurs.
If the amount of memory demanded by a program (or the collection of running programs) exceeds the amount of virtual memory, it is possible that the operating system increases the amount of virtual memory. As virtual memory is essentially a file, this explains what is going on when "Windows is increasing the size of your virtual memory paging file."
The "process" you refer to could be called "increasing virtual memory". I'm not sure whether it has a special name.