The purpose of this article is to delve into the intricacies of DNS resolution within Kubernetes clusters and address common challenges and misconceptions related to DNS configuration and troubleshooting.
Readers can expect to gain insights into:
Understanding the role of DNS in Kubernetes service discovery and communication.
Identifying common DNS-related issues in Kubernetes workloads and clusters.
Learning effective strategies for troubleshooting DNS lookup problems.
Acquiring practical knowledge and best practices for configuring and managing DNS in Kubernetes environments.
Kubernetes is an open-source container orchestration platform that automates the deployment, scaling, and management of containerized applications. It abstracts away infrastructure complexities and provides a platform-agnostic environment for deploying applications.
Networking Model:
Kubernetes employs a flat, virtual network where every pod gets its own IP address.
Pods can communicate with each other directly, regardless of which node they are running on.
Services act as stable endpoints for pods, providing load balancing and service discovery.
Kubernetes uses DNS to resolve service names to their corresponding IP addresses within the cluster.
Understanding DNS in Kubernetes
DNS, or Domain Name System, is a fundamental protocol used on the internet to translate human-readable domain names (like google.com) into IP addresses (like 172.217.7.206). It acts as a distributed database, organizing domain names into a hierarchical structure and providing a mechanism for querying and resolving these names to their corresponding IP addresses.
How DNS works in Kubernetes
In Kubernetes, DNS plays a critical role in facilitating communication between different components within the cluster. When a pod needs to communicate with another pod or service, it typically refers to them by their domain names. Kubernetes employs a DNS-based service discovery mechanism, allowing pods to resolve these domain names to the appropriate IP addresses within the cluster.
DNS resolution in Kubernetes
In Kubernetes, DNS resolution refers to the process by which domain names are translated into IP addresses within the Kubernetes cluster. When a pod or service needs to communicate with another pod or service by its domain name, the Kubernetes DNS service (kube-dns or CoreDNS) resolves that domain name to the corresponding IP address.
This enables seamless communication between different components within the Kubernetes environment, facilitating service discovery and inter-pod communication.
DNS resolution in Kubernetes is critical for maintaining the functionality and connectivity of applications running within the cluster.
Components of DNS resolution in Kubernetes
DNS Pods: In Kubernetes, DNS functionality is typically provided by a set of DNS pods deployed within the cluster. These pods host DNS server software, such as CoreDNS, which handles DNS resolution requests from other pods and services within the cluster.
CoreDNS: CoreDNS is a flexible, extensible DNS server that is commonly used in Kubernetes clusters. It supports various plugins and configurations, allowing it to serve as a DNS server for Kubernetes services, pods, and external domains.
Cluster DNS Service: The Cluster DNS Service acts as a front-end for DNS resolution requests within the Kubernetes cluster. It exposes a DNS interface that pods and services can query to resolve domain names to IP addresses. This service typically runs on a well-known IP address (e.g., 10.96.0.10) and listens for DNS queries from other components within the cluster.
DNS in Kubernetes comprises DNS pods hosting server software like CoreDNS, which provides DNS resolution services to other components within the cluster. The Cluster DNS Service acts as a centralized entry point for DNS queries, enabling seamless communication between pods and services using domain names.
Importance of DNS in Kubernetes for Service Discovery:
In Kubernetes, service discovery is crucial for facilitating communication between microservices within a cluster. DNS plays a vital role in this process by providing a dynamic mapping between service names and their corresponding IP addresses.
Significance of DNS in Kubernetes:
Allows services to be accessed by their logical names rather than specific IP addresses.
Enables dynamic scaling and reconfiguration of services without impacting client applications.
Facilitates seamless fai lover and load balancing across multiple instances of a service.
Simplifies application development and deployment by abstracting away network complexities.
TL;DR:
To ensure efficient access to resources from containers within a Kubernetes pod, it's best to use absolute domain names that end with a dot. This is especially important for accessing resources outside the cluster. If absolute domain names aren't feasible, modifying the container's resolver behavior using the pod specification field dnsConfig can make accessing off-cluster resources with relative domain names equally efficient.
Let's explore why this is the case by understanding what happens behind the scenes when we visit a website like github.com in our web browser.
Simulating DNS resolution using nslookup on both workstations and Kubernetes containers.
The nslookup command is a useful tool for simulating DNS resolution, allowing us to understand how domain names are translated into IP addresses. We can use it both on a workstation and within a Kubernetes container to observe DNS resolution behavior.
Here's how to use nslookup to simulate DNS resolution on both platforms:
On Workstations
Within Kubernetes Containers
On Workstations
To simulate DNS resolution on a workstation, execute the nslookup command with the -debug flag followed by the domain name you wish to query. This command queries the configured nameserver for the IP address of the specified domain and displays the resolution process along with the final IP address received.
$ nslookup -debug github.com
Server: 2603:8001:cb01:8fd9:c641:1eff:fee7:d45f
Address: 2603:8001:cb01:8fd9:c641:1eff:fee7:d45f#53
QUESTIONS:
github.com, type = A, class = IN
ANSWERS:
-> github.com
internet address = 192.30.255.112
ttl = 33
AUTHORITY RECORDS:
ADDITIONAL RECORDS:
Non-authoritative answer:
Name: github.com
Address: 192.30.255.112Observations:
The resolver queries the configured nameserver for github.com.
It receives an IP address (192.30.255.112) as the answer.
Using nslookup in a Kubernetes Container:
Similarly, we repeat the nslookup test within an Ubuntu container running on a Kubernetes cluster. First, ensure the dnsutils package is installed to enable nslookup functionality within the container.
# nslookup -debug github.com
Server: 10.8.0.10
Address: 10.8.0.10#53
[OBMITTED]
QUESTIONS:
github.com.default.svc.cluster.local, type = A, class = IN
ANSWERS:
[OBMITTED]
** server can't find github.com.default.svc.cluster.local: NXDOMAIN
[OBMITTED]
QUESTIONS:
github.com.svc.cluster.local, type = A, class = IN
ANSWERS:
[OBMITTED]
** server can't find github.com.svc.cluster.local: NXDOMAIN
[OBMITTED]
QUESTIONS:
github.com.cluster.local, type = A, class = IN
ANSWERS:
[OBMITTED]
** server can't find github.com.cluster.local: NXDOMAIN
[OBMITTED]
QUESTIONS:
github.com.us-central1-c.c.red-forklift-301112.internal, type = A, class = IN
ANSWERS:
[OBMITTED]
** server can't find github.com.us-central1-c.c.red-forklift-301112.internal: NXDOMAIN
[OBMITTED]
QUESTIONS:
github.com.c.red-forklift-301112.internal, type = A, class = IN
ANSWERS:
[OBMITTED]
** server can't find github.com.c.red-forklift-301112.internal: NXDOMAIN
[OBMITTED]
QUESTIONS:
github.com.google.internal, type = A, class = IN
ANSWERS:
[OBMITTED]
** server can't find github.com.google.internal: NXDOMAIN
[OBMITTED]
QUESTIONS:
github.com, type = A, class = IN
ANSWERS:
-> github.com
internet address = 140.82.114.3
ttl = 42
AUTHORITY RECORDS:
ADDITIONAL RECORDS:
------------
Non-authoritative answer:
Name: github.com
Address: 140.82.114.3
------------
QUESTIONS:
github.com, type = AAAA, class = IN
ANSWERS:
AUTHORITY RECORDS:
-> github.com
origin = dns1.p08.nsone.net
mail addr = hostmaster.nsone.net
serial = 1611586688
refresh = 43200
retry = 7200
expire = 1209600
minimum = 3600
ttl = 1799
ADDITIONAL RECORDS:
------------Observations:
The resolver within the Kubernetes container queries the configured nameserver.
Initially, it sends multiple queries with various domain name suffixes but receives NXDOMAIN responses.
Finally, it queries for github.com directly and receives the correct IP address (140.82.114.3).
We observe that the resolver in the container sends multiple queries with domain name suffixes before querying for the correct domain.
Further Testing:
Additional testing involves appending an extra dot at the end of the domain name (github.com.) to understand its impact on DNS resolution.
# nslookup -debug github.com.
Server: 10.8.0.10
Address: 10.8.0.10#53------------
QUESTIONS:
github.com, type = A, class = IN
ANSWERS:
-> github.com
internet address = 140.82.112.3
ttl = 59
AUTHORITY RECORDS:
ADDITIONAL RECORDS:
------------
Non-authoritative answer:
Name: github.com
Address: 140.82.112.3
------------
QUESTIONS:
github.com, type = AAAA, class = IN
ANSWERS:
AUTHORITY RECORDS:
-> github.com
origin = dns1.p08.nsone.net
mail addr = hostmaster.nsone.net
serial = 1611586688
refresh = 43200
retry = 7200
expire = 1209600
minimum = 3600
ttl = 1799
ADDITIONAL RECORDS:
------------Observations:
By appending an extra dot at the end of the domain name, the resolver queries the nameserver directly and receives the correct IP address.
This highlights the distinction between absolute and relative domain names in DNS resolution.
Executing nslookup commands on both workstations and Kubernetes containers enables us to comprehend DNS resolution behavior across different environments and aids in troubleshooting DNS-related issues effectively.
Configuration of resolv.conf Files:
The resolv.conf file is a crucial component that holds DNS resolver configuration settings. These settings dictate how DNS resolution is performed on a system. In the article, the configurations of the resolv.conf files on both a workstation and a Kubernetes container are compared to assess their influence on DNS resolution.
Workstation's Resolver Configuration
(/etc/resolv.conf):
The resolv.conf file on a workstation typically resides in the /etc directory.
It contains directives that specify the DNS servers to use for name resolution and may include additional settings such as search domains.
Let see also look at our workstation’s resolver configuration.
$ cat /etc/resolv.conf
#
# macOS Notice
#
# This file is not consulted for DNS hostname resolution, address
# resolution, or the DNS query routing mechanism used by most
# processes on this system.
#
# To view the DNS configuration used by this system, use:
# scutil --dns
#
# SEE ALSO
# dns-sd(1), scutil(8)
#
# This file is automatically generated.
#
search natmtn.rr.com
nameserver 2603:8001:cb01:8fd9:c641:1eff:fee7:d45f
nameserver 192.168.1.1Observations:
The search directive specifies the domain suffix to append to queries that do not contain a dot. In this case, it's set to natmtn.rr.com.
Two nameserver directives are provided, indicating the IP addresses of the DNS servers (2603:8001:cb01:8fd9:c641:1eff:fee7:d45f and 192.168.1.1) used by the workstation.
Kubernetes Container's resolv.conf Configuration:
Similarly, Kubernetes containers also have their own resolv.conf files, but they might be dynamically generated based on the cluster's DNS configuration.
The location of the resolv.conf file inside a container can vary depending on the container runtime and operating system.
# cat /etc/resolv.conf
nameserver 10.8.0.10
search default.svc.cluster.local svc.cluster.local cluster.local us-central1-c.c.red-forklift-301112.internal c.red-forklift-301112.internal google.internal
options ndots:5Observations:
The nameserver directive specifies the IP address of the DNS server (10.8.0.10) used by the Ubuntu container.
The search directive lists domain suffixes to be appended to queries. If a query does not contain a dot, these suffixes will be appended in sequence until a resolution is found.
The options ndots:5 directive sets the threshold for the number of dots in a name before an initial absolute query is made. In this case, if a name contains five or more dots, it will be treated as an absolute query. The default value for ndots is 1.
Explanation of ndots Option:
The ndots option determines how many dots must appear in a domain name before it is considered a candidate for absolute querying.
If a name contains fewer dots than the specified threshold, the resolver appends the domain suffixes listed in the search directive before attempting resolution.
In this configuration, any domain name with five or more dots will be treated as an absolute query, bypassing the domain suffix search list.
This configuration ensures efficient DNS resolution by optimizing query behavior based on the structure of domain names.
Test Results and Performance Metrics:
In a recent study, Kubernetes workloads were configured to evaluate the performance impact of different domain name constructions. Various workloads, including deployments (nginx, app-1, app-2) and jobs (load-1, load-2), were orchestrated to stress-test DNS resolution mechanisms. The performance metrics, comprising response times and query rates, were meticulously logged and analyzed to compare the efficiency of accessing resources using absolute and relative domain names.
The findings unveiled that utilizing absolute domain names consistently yielded superior performance compared to their relative counterparts, particularly noticeable when accessing off-cluster resources.
Workstation with github.com:
When accessing github.com from a workstation, the domain name utilized is a relative one with a single dot. With an ndots value of one, the domain name is forwarded to the nameserver for resolution as an absolute domain name.
$ cat /etc/resolv.conf s
earch natmtn.rr.com
nameserver 2603:8001:cb01:8fd9:c641:1eff:fee7:d45f
nameserver 192.168.1.1Ubuntu container with github.com:
Within an Ubuntu container, the domain name for github.com remains relative, also with a single dot.
However, with an ndots value set to five, the domain name undergoes iteration through each of the six search list elements before ultimately being resolved as an absolute domain name.
This iterative process results in all queries returning the NXDOMAIN message initially.
# cat /etc/resolv.conf
nameserver 10.8.0.10
search default.svc.cluster.local svc.cluster.local cluster.local us-central1-c.c.red-forklift-301112.internal c.red-forklift-301112.internal google.internal
options ndots:5Ubuntu container with github.com.:
In contrast, when accessing github.com from an Ubuntu container, the domain name utilized is an absolute one.
Consequently, the domain name is directly forwarded to the nameserver for resolution.
# nslookup -debug github.com.
Server: 10.8.0.10
Address: 10.8.0.10#53
------------
QUESTIONS:
github.com, type = A, class = IN
ANSWERS: ->
github.com internet address = 140.82.112.3
ttl = 59
AUTHORITY RECORDS:
ADDITIONAL RECORDS:
------------
Non-authoritative answer:
Name: github.com
Address: 140.82.112.3
------------
QUESTIONS:
github.com, type = AAAA, class = IN
ANSWERS:
AUTHORITY RECORDS:
-> github.com
origin = dns1.p08.nsone.net
mail addr = hostmaster.nsone.net
serial = 1611586688
refresh = 43200
retry = 7200
expire = 1209600
minimum = 3600
ttl = 1799
ADDITIONAL RECORDS:
------------Observation:
The ndots option of five is a Kubernetes feature designed to optimize relative name lookups of cluster resources.
The impact of using relative names depends on how deep the resolver needs to iterate through the search list elements to get a resolution.
Performance Analysis:
Two workloads, app-1 and app-2, were deployed to examine the impact of setting ndots to five. app-1 accessed the nginx service using a relative domain name (nginx.default.srv.cluster.local), whereas app-2 employed an absolute domain name (nginx.default.svc.cluster.local). Load was applied to both workloads at a rate of 10 queries per second (qps) for 30 seconds.
$ kubectl logs load-app-1-92gxz
[OBMITTED]
Code 200 : 300 (100.0 %)
Response Header Sizes : count 300 avg 227 +/- 0 min 227 max 227 sum 68100
Response Body/Total Sizes : count 300 avg 239 +/- 0 min 239 max 239 sum 71700
All done 300 calls (plus 4 warmup) 13.309 ms avg, 10.0 qpsThen the job putting a load on app-2; the one using an absolute domain name.
$ kubectl logs load-app-2-vzf8f
[OBMITTED]
Code 200 : 300 (100.0 %)
Response Header Sizes : count 300 avg 227 +/- 0 min 227 max 227 sum 68100
Response Body/Total Sizes : count 300 avg 239 +/- 0 min 239 max 239 sum 71700
All done 300 calls (plus 4 warmup) 6.843 ms avg, 10.0 qpsObservations:
app-1, using the relative domain name, had about a 6 ms performance penalty compared to app-2, which used the absolute domain name.
The penalty increased linearly by 6 ms for each relative domain name (with less than five dots) for off-cluster resource accessed by the application.
Note:
It is best to use absolute domain names when accessing off-cluster resources from a container of a Kubernetes pod.
While it is also most efficient to use absolute domain names for on-cluster resources, the impact of using relative names depends on how deep the resolver needs to iterate through the search list element to get a resolution.
Controlling ndots with dnsConfig:
In cases where controlling domain names accessed by a workload is not possible (e.g., using pre-built container images or third-party libraries), the dnsConfig pod specification field can be used to control the value of ndots in the container.
Setting ndots to 1 ensures that domain name resolution within the container mimics that of the workstation, with any domain name containing a dot being resolved as an absolute domain initially.
dnsConfig:
options:
- name: ndots
value: "1"In such cases, domain name resolution within the container aligns with that of the workstation, where any domain name containing a dot is resolved as an absolute domain initially.
Additional Workloads:
Two additional workloads, app-3 and load-3, were established to illustrate the impact of dnsConfig on relative domain name resolution. app-3 employed a relative domain name (nginx.default.svc.cluster.local) with dnsConfig specified. Load was applied to app-3 to assess its performance.
$ kubectl logs load-app-3-lgfvw
[OBMITTED]
Code 200 : 300 (100.0 %)
Response Header Sizes : count 300 avg 227 +/- 0 min 227 max 227 sum 68100
Response Body/Total Sizes : count 300 avg 239 +/- 0 min 239 max 239 sum 71700
All done 300 calls (plus 4 warmup) 7.121 ms avg, 10.0 qpsObservation:
Even though app-3 used a relative domain name, it had essentially the same performance as app-2, which used an absolute domain name.
These results highlight the importance of efficient DNS configuration, especially in Kubernetes environments, and the impact it can have on workload performance.
Common DNS Lookup Issues and Troubleshooting
DNS Misconfiguration:
Incorrect DNS configuration can lead to resolution failures or delays.
Misconfigured DNS servers or search domains can cause unexpected behavior.
Pod DNS Resolution Issues:
Pods may fail to resolve domain names, leading to connectivity problems with other services.
Incorrect or incomplete DNS records can prevent proper resolution.
Cluster DNS Service Problems:
Issues with the cluster's DNS service can affect all pods within the cluster.
Service downtime or misconfigurations can disrupt DNS resolution for all workloads.
Network Connectivity Problems:
Network issues such as packet loss or firewall rules can interfere with DNS resolution.
DNS requests may fail to reach the DNS server or receive responses.
Strategies for Troubleshooting DNS Lookup Problems:
Checking DNS Configuration:
Review the DNS configuration of the affected pod or node.
Ensure that the correct DNS server addresses and search domains are configured.
Verify that DNS settings align with Kubernetes best practices.
Analyzing DNS Logs:
Check DNS server logs for errors or warnings related to the failed DNS queries.
Look for patterns or trends in DNS resolution failures.
Monitor DNS server performance metrics to identify bottlenecks.
Debugging DNS Resolution within Kubernetes Clusters:
Use tools like nslookup, dig, or kubectl to perform DNS queries from within the cluster.
Verify that DNS requests are reaching the DNS server and receiving responses.
Check for DNS-related events or errors in Kubernetes control plane components like kube-dns or coredns.
Validate DNS pod configurations and ensure that DNS pods are healthy and running.
Conclusion
The comparison of DNS resolution behaviors between workstations and Kubernetes containers sheds light on the intricacies of domain name translation. While workstations typically exhibit straightforward resolution processes, Kubernetes containers, with their dynamic environments, introduce additional complexities. Through simulations using nslookup, we've observed the importance of understanding domain suffixes, nameserver configurations, and the impact of absolute versus relative domain names on resolution outcomes. This understanding is crucial for optimizing DNS configurations and troubleshooting issues effectively in both workstation and containerized environments.