DNS Caching Server
A DNS caching server typically sits between client systems and authoritative DNS servers. The cache figures out where to send a request for a particular hostname. This in turn may require numerous lookups to a root server, the server that handles "org", the server that handles "example", the server that handles "foo", and so on, assuming a hostname of "foo.example.org", though the lookup may stop before there if the "example.org" zone hosts the foo record instead of delegating that to a "foo" subdomain, or, more likely in this example, that "foo" does not exist in the zone "example.org". "NS" is short for nameserver, if you want to know who to ask about "org" or so forth.
This recursive process and due to networks in the olden times being both slow and of limited bandwidth means that one really should cache lookups, though records do need to be timed out, eventually. Modern sites often set very low time-to-live values, which has both positives and negatives, one negative being that there may not be much, if any time given to cache anything.
A "recursing" server such as unbound(8) is another name for a caching DNS server, as opposed to authoritative servers, nsd(8) being an example, which typically do not support recursion (or "must not" support recursion for reasons not delved into here). Client systems talk to recursive servers; recursive servers talk to various other servers and eventually an authoritative server for the query.
The address of a caching DNS server is typically returned by a DHCP server or otherwise made known to client systems, such as the ISP telling the customer what addresses should be used, if there is no DHCP or similar protocol in use. There are also various public caching DNS servers, 9.9.9.9 and 1.1.1.1 are two examples, so client systems might ignore what the ISP tells them to use. However, there are naughty ISP out there that block or redirect DNS traffic, so it may not be so simple as to use 9.9.9.9 as a DNS server to avoid the ISP spying on (and reselling for profit) your DNS requests.
Be sure to check who owns your DNS server and whether that organization is problematic, e.g. how much spam or tracking are they involved in, which governments could compel them to log and share what data, etc.
DNS Interception
DNS is traditionally unencrypted; there is now DNSSEC for some but hardly all zones. DNS requests should be assumed to travel in the clear. In this case it is a trivial matter for a router or firewall to transparently send DNS requests to some other IP address, so instead of 9.9.9.9:53 your computer is instead talking to some address operated by the ISP or some other entity. The DNS request can be logged and the requests made available to anyone with a pocketbook (advertising a.k.a. spam companies, in particular), or anyone who has hacked the ISP, or any rogue employee at the ISP who is up to no good, or political powers nosing around for who knows what. Most ISP are probably not this bad most of the time, though it may be best to assume the worst, especially in nations with negligent levels of consumer protection or political unrest.
Modern systems can send DNS over TLS or even HTTPS, assuming the remote DNS server supports that. There may be canary DNS settings that can prevent the use of DNS over HTTPS. Prudence suggests using tcpdump or similar to log that DNS requests are actually going to the correct port and are encrypted, and to check whether canary DNS is in effect: a negative response from "use-application-dns.net." indicates that DNS over HTTPS should not be used, though one may be able to force an application to use DNS over HTTPS regardless. (DoH also assumes you trust the DoH server more than your local DNS server.)
It may be good to check the settings now and then, as some software vendors like to make random changes, or maybe something in the configuration management failed to deploy the host correctly. I have a lovely story of Ansible and selinux getting into a fight that resulted in some very insecure systems being deployed—thanks, selinux!—and other lovely stories about the Microsofts or Firefoxen of the world changing user preferences time and again. Eternal vigilance, etc.
Another option is to run your own caching DNS server outside the reach of the ISP, and to send your DNS requests to that server using a VPN. This assumes the ISP that hosts your remote server is not also doing DNS interception, though presumably there is less profit motive for a random hosting provider to log and sell DNS requests. Using a DNS server used by many other people may also help prevent an individual from standing out from the crowd, and thus makes them less of a target for targeted advertising, or less of a nail to hammer down should they belong to the wrong political faction at the wrong time. Paranoid? Maybe, though some have a stronger sense of privacy and would like to see regulations around data handling, especially given the many and varied data leaks that keep springing up in some countries.
Forwarding DNS
For unbound(8) on OpenBSD and to use TLS connections to Quad9, with a fallback to making the queries ourselves if Quad9 is down, try the following configuration. There are tradeoffs here; with "forward-first: yes" an attacker could cause Quad9 to be unavailable, for example if your ISP or someone upstream of them is malicious, they could block your requests to 9.9.9.9, and then your DNS queries will be visible to the ISP or upstream attacker. On the other hand if Quad9 does go down with "forward-first: no" set then all your DNS requests will fail. This will let you know when someone is blocking Quad9 (probably malicious) or when there is a network outage that renders Quad9 offline (probably not malicious), at the cost of breaking any service that relies heavily on DNS being available. This is why I like larger time-to-live values, so that services can better ride out brief outages. Longer TTL do mean that it will take longer to move a service to a new address, so here too there are tradeoffs.
All your client systems should use the server running unbound as their DNS server. This may require that one disable various random daemons or scripts that like to mangle the DNS settings; systemd on RedHat Linux was particularly bad here, and I eventually gave up and marked the /etc/resolv.conf file as immutable after making the required changes.
Without forwarding setup, tcpdump(8) or similar traffic sniffing tools should show DNS requests. The following has been cleaned up a bit, though does show some of the packets resulting from the command `host example.org` on the local system. There can be a lot of noise depending on the tcpdump flags, but the gist of the records below is a query about example.org (in plaintext!) from 104.207.156.138 to 199.43.135.53, and some responses back.
There may be no packets if your DNS server has cached the lookup, so for a clean test first restart the DNS server, or clear its cache. Using tcpdump may also show that other daemons (rspamd) make their own DNS requests independent of the system resolver. If these requests need to go through a particular server then setting up a transparent firewall rule to redirect them will be necessary. This is why it is good to use tcpdump now and then, to see what is going on on a network, and to find software or devices that need to be configured differently, or firewall rules adjust to account for something discovered.
Bad Domains Block
Caching DNS servers can generally be configured to block various problematic domains. There are blacklists that one can subscribe to that collect spammers, trackers, and other such malicious domains, or for example to block the given site X:
There are devices such as the pi-hole that provide network-wide blocking, assuming you can get all your client systems pointed at it (now there might be a use for a transparent firewall). The pi-hole in turn may need to use DNS over TLS or HTTPS to reach another DNS server (say, 9.9.9.9), or may need to talk to a different DNS server over a VPN to escape your local ISP, if you have one of those naughty ISP who intercept DNS traffic.
Fuss and Bother
Isn't this all very wasteful, to have a DNS proxy and blacklists and a VPN to avoid the ISP and another encrypted tunnel to some other DNS provider? Sure, but some countries are pretty slack on the regulatory front, or may have malicious ISP so good choices can be limited or impossible. That is, for the user with a dislike of the holy trinity of ads, malware, and tracking, to say less of vile political pressure against minority groups. Hence the less bad options of pi-holes and blacklists and forwarding DNS elsewhere, even if this creates an arms race of complexity as the spammers spam more and more and the users block more and more.
Shire Erehwesle
DNS-over-TLS (DoT) makes it possible to encrypt DNS messages and gives a DNS client the possibility to authenticate a resolver. As implied by the name, this is done by sending DNS messages over TLS. Unbound can handle TLS encrypted DNS messages since 2011, long before the IETF DPRIVE working group started its work on the DoT specification.
There are, however, DNS clients that do not support DoT but are able to use DNS-over-HTTPS (DoH) instead. Where DoT sends a DNS message directly over TLS, DoH has an HTTP layer in between. Where DoT uses its own TCP port (853), DoH uses the standard HTTPS port (443).
https://unbound.docs.nlnetlabs.nl/en/latest/topics/privacy/dns-over-https.html