Wikipedia has great information on this topic, but as a brief, simplified explanation, know that it boils down mainly to the involvement of mechanical moving parts in a traditional Hard Disk Drive (HDD).
The size of the addressing space in itself doesn't affect the access times. Otherwise you could theoretically build a very small hard-disk drive with only 4GB and expect it to run just as fast as your RAM chip. In reality that is not possible.
Hard Disk Drives
All the data is read and written by the read and write heads, laying on the round disks that physically store all the data. Whenever data in a specific address is specified, the heads must move to the correct track (the time it takes is the Seek Time) and the disk must rotate until the requested sector is under the head (Rotational Latency). The time until both of these tasks are completed is the Access Time, and only then the data transfer can actually start.
A - Track ; C - (Track) Sector
In contrast, RAM cards have no such mechanical moving parts and therefore the time constraints are much lower. They are mainly there to allow proper synchronization of the electronic components that comprise the memory card and ensure that it works reliably.
You probably also heard of the term Solid State Drive (SSD). Seagate recently manufactured such a drive with 60TB capacity, and its access times are still much shorter than those of hard-disk drives. How? Like RAM chips, they have no moving parts.