What governs the "speed" of a programming language?
There is no such thing as the "speed" of a programming language. There is only the speed of a particular program written by a particular progammer executed by a particular version of a particular implementation of a particular execution engine running within a particular environment.
There can be huge performance differences in running the same code written in the same language on the same machine using different implementations. Or even using different versions of the same implementation. For example, running the exact same ECMAScript benchmark on the exact same machine using a version of SpiderMonkey from 10 years ago vs a version from this year will probably yield a performance increase anywhere between 2×–5×, maybe even 10×. Does that then mean that ECMAScript is 2× faster than ECMAScript, because running the same program on the same machine is 2× faster with the newer implementation? That doesn't make sense.
Has this anything to do with memory management?
Why does this happen?
Resources. Money. Microsoft probably employs more people making coffee for their compiler programmers than the entire PHP, Ruby, and Python community combined has people working on their VMs.
For more or less any feature of a programming language that impacts performance in some way, there is also a solution. For example, C (I'm using C here as a stand-in for a class of similar languages, some of which even existed before C) is not memory-safe, so that multiple C programs running at the same time can trample on each other's memory. So, we invent virtual memory, and make all C programs go through a layer of indirection so that they can pretend they are the only ones running on the machine. However, that is slow, and so, we invent the MMU, and implement virtual memory in hardware to speed it up.
But! Memory-safe languages don't need all that! Having virtual memory doesn't help them one bit. Actually, it's worse: not only does virtual memory not help memory-safe languages, virtual memory, even when implemented in hardware, still impacts performance. It can be especially harmful to the performance of garbage collectors (which is what a significant number of implementations of memory-safe languages use).
Another example: modern mainstream general purpose CPUs employ sophisticated tricks to reduce the frequency of cache misses. A lot of those tricks amount to trying to predict what code is going to be executed and what memory is going to be needed in the future. However, for languages with a high degree of runtime polymorphism (e.g. OO languages) it is really, really hard to predict those access patterns.
But, there is another way: the total cost of cache misses is the number of cache misses multiplied by the cost of an individual cache miss. Mainstream CPUs try to reduce the number of misses, but what if you could reduce the cost of an individual miss?
The Azul Vega-3 CPU was specifically designed for running virtualized JVMs, and it had a very powerful MMU with some specialized instructions for helping garbage collection and escape detection (the dynamic equivalent to static escape analysis) and powerful memory controllers, and the entire system could still make progress with over 20000 outstanding cache misses in flight. Unfortunately, like most language-specific CPUs, its design was simply out-spent and out-brute-forced by the "giants" Intel, AMD, IBM, and the likes.
The CPU architecture is just one example that has an impact on how easy or how hard it is to have a high-performance implementation of a language. A language like C, C++, D, Rust that is a good fit for the modern mainstream CPU programming model will be easier to make fast than a language that has to "fight" and circumvent the CPU, like Java, ECMAScript, Python, Ruby, PHP.
Really, it's all a question of money. If you spend equal amounts of money to develop a high-performance algorithm in ECMAScript, a high-performance implementation of ECMAScript, a high-performance operating system designed for ECMAScript, a high-performance CPU designed for ECMAScript as has been spent over the last decades to make C-like languages go fast, then you will likely see equal performance. It's just that, at this time, much more money has been spent making C-like languages fast than making ECMAScript-like languages fast, and the assumptions of C-like languages are baked into the entire stack from MMUs and CPUs to operating systems and virtual memory systems up to libraries and frameworks.
Personally, I am most familiar with Ruby (which is generally considered to be a "slow language"), so I will give two examples: the
Hash class (one of the central data structures in Ruby, a key-value dictionary) in the Rubinius Ruby implementation is written in 100% pure Ruby, and it has about the same performance as the
Hash class in YARV (the most widely-used implementation), which is written in C. And there is an image manipulation library written as a C extension for YARV, that also has a (slow) pure Ruby "fallback version" for implementations that don't support C which uses a ton of highly-dynamic and reflective Ruby tricks; an experimental branch of JRuby, utilizing the Truffle AST interpreter framework and Graal JIT compilation framework by Oracle Labs, can execute that pure Ruby "fallback version" as fast as the YARV can execute the original highly-optimized C version. This is simply (well, anything but) achieved by some really clever people doing really clever stuff with dynamic runtime optimizations, JIT compilation, and partial evaluation.