Increasing the speed of memory typically involves increasing the size of the bit cell. For example increasing the capacitor size in a DRAM (which using one transistor and one capacitor to store a bit) allows faster reading of the bit but obviously also reduces the number of bit cells that can be fit onto a chip. (Similarly, SRAM is less dense but faster than DRAM.)
From an economic perspective, since speed is generally valued, techniques to increase speed are allowed spend more effort (i.e., cost) in research, design, and manufacture to increase speed.
Aside from the cost factor (users seeking higher capacity will seek lower cost per bit), higher capacity memories are generally slower from a combination of lesser integration and greater distance between a bit cell and the consumer of the data. Greater integration (on the same chip, in the same package, on the same board, in the same system/enclosure) not only reduces physical communication overhead (crossing physical boundaries has overhead) but also interface overhead as less integrated interfaces are typically more generic and less optimized for one specific use.
Indexing overhead is an additional factor making larger memories slower. Hardware effectively traverses a binary tree to select the desired data. (This factor is typically minor, but it seemed worth mentioning.)
Another significant tradeoff in memory system design is access granularity/randomness. Larger access granules and less random access correlate with increased density/lower cost per bit. This is also a result of economic and physical factors.