When I click "view certificate," under Public Key Algorithm it usually says "RSA Encryption" or "Elliptic Curve Public Key." I assume this is the algorithm used to agree on a premaster secret (symmetric keys) between the client and server, or steps 4-6 in the image below. Why not use Diffie-Hellman then? Doesn't ephemeral Diffie-Hellman provide forward secrecy?
The typical process to establish a secure channel uses asymmetric cryptography for two purposes: to allow the parties to authenticate each other, and to establish a shared symmetric key (in TLS, that's the premaster secret). At least one of the sides needs to authenticate the other, otherwise the two sides could each establish a secure channel with a man-in-the-middle. The shared symmetric key must be established in such a way that a man-in-the-middle cannot find it out. There are many different ways to do this, and I'll focus on ways that are actually used in TLS.
Most TLS connections do authentication and shared key establishment independently. This is the only way in TLS 1.3 (except with pre-shared symmetric keys), and it's the most common way in previous TLS versions.
To establish a shared key, each side creates an ephemeral Diffie-Hellman key (usually elliptic-curve DH, but it can also be finite-field DH) and sends the public key to the other side. Each sides receives the other side's public key, calculates the shared secret, and discards the ephemeral DH key. Thanks to the security properties of Diffie-Hellman, it's impossible to reconstruct the shared secret by observing wire traffic: an adversary would also need to have one of the private DH keys. Hence this method guarantees perfect forward secrecy.
For authentication, the client authenticates the server. (The server can optionally authenticate the client, but having one side do authentication is enough to prevent active man-in-the-middle attacks.) The way this works is that the server sends a digital signature (RSA, DSA or ECDSA) of the handshake, i.e. of the packets that both sides have either read or written so far, and the client verifies this signature. The client either already knows the server's public key (public key pinning or raw public keys, both rare settings), or far more commonly verifies a certificate chain from a root certificate authority trusted by the client to the server's public key.
The authentication process has no impact on the shared key establishment. That's why you don't find anything about Diffie-Hellman in the certificate information: it isn't involved in that part of the protocol. The algorithms mentioned in the server certificate are the one used to sign the handshake (subject public key algorithm) and the one used by the CA to sign the certificate (certificate signature algorithm).
By the way, “PKCS#1 v1.5 RSA Encryption” in X.509 certificates is called this for historical reasons, but it's actually a PKCS#1 v1.5 RSA signature.
TLS ≤1.2 also has cipher suites that use static Diffie-Hellman (also called non-ephemeral Diffie-Hellman). With these cipher suites, DH also provides the authentication part of the process. The server always uses the same DH key. The client verifies that the DH public key that it receives from the server is the one that it expects. If an adversary tries to impersonate the server and sends the legitimate server's public key, they won't be able to calculate the shared secret because they don't have the private key. If an adversary tries to impersonate the server and sends their own public key, the client will abort the connection because it sees a public key that is not the expected one. Static Diffie-Hellman does not provide forward secrecy, since obtaining the server's static private key is enough to decrypt all past recorded traffic.
Just like with signature-based cipher suites, the client doesn't actually need to know the server's public key in advance. The client can use a public key infrastructure to validate the server's public key: it checks that the server's public key comes in a certificate that has been signed by a (chain of) CA that the client trusts. An X.509 certificate can contain a signature of an (EC)DH public key. It's rarely done in practice, because it only has a marginal performance benefit, it's less secure (no forward secrecy), and there's a self-reinforcing loop that almost nobody does it (because there's no reason to) so almost nobody implements support for it (because there's no demand) so almost nobody does it (because there's no software support).
Diffie-Hellman does not directly provide signing and so cannot be used in Certificates. El Gamal (which is related to Diffie-Hellman) can, but it not used because this would require an extra verification step.
RSA could be used to sign the Diffie-Hellman key, and that key actually used to set up the session key, but it doesn't work that way. See Diffie-Hellman and its TLS/SSL usage on https://security.stackexchange.com for practical details.