1. Core concepts

  • Pod: A Pod is the smallest deployable unit in Kubernetes. A Pod usually runs one main application container. In some cases, a Pod can include additional sidecar containers (for example, with a service mesh) to handle concerns such as metrics, encryption, and traffic routing. Even without a service mesh, keeping one main container per Pod is a common best practice for independent scaling and simpler operations.

    Networking-wise, each Pod gets its own IP address inside the cluster.

  • Deployment: A Deployment manages stateless applications. It declaratively defines the desired state (for example, replica count, Pod template, and update strategy) and ensures Kubernetes continuously reconciles actual state with desired state.

    Pods are ephemeral, so if one dies, the Deployment (through ReplicaSets) creates a replacement. The scheduler selects a suitable node, and the kubelet on that node pulls images and starts containers.

  • StatefulSet: A StatefulSet manages stateful workloads such as databases. Unlike Deployments, Pods in a StatefulSet have stable identities and stable storage.

    Each Pod gets its own PersistentVolumeClaim (PVC), and StatefulSets are commonly paired with a headless Service to give each Pod a stable network identity.

  • DaemonSet: A DaemonSet ensures one Pod (or more, depending on configuration) runs on each eligible node. Typical use cases include log collection, node monitoring, and security agents.

  • Service: A Service provides stable network access to a group of Pods. While Deployments/StatefulSets manage Pod lifecycle and scaling, Services manage connectivity and traffic routing.

    A Service typically has:

    • a stable virtual IP (ClusterIP), and
    • a DNS name for discovery.

    Kubernetes maintains Endpoint/EndpointSlice objects to track which Pod IPs are currently backing a Service.

For StatefulSets, a headless Service (clusterIP: None) is often used when clients should connect directly to individual Pods rather than through one shared virtual IP.

  • Secret and ConfigMap:
    • ConfigMap stores non-sensitive configuration.
    • Secret stores sensitive data such as API keys, passwords, or tokens.

    Both are namespaced resources.

2. Taints/Tolerations and nodeSelector/labels

  • Labels + nodeSelector: Labels are key-value metadata on nodes (and other resources). You can use nodeSelector in a Pod spec to constrain scheduling to nodes with matching labels.

  • Taints + tolerations: Taints are applied to nodes to repel Pods by default. A Pod must declare matching tolerations to be scheduled onto tainted nodes. This is useful for reserving special nodes (for example, high-memory nodes) for specific workloads.

3. Connection flow

  • Internal communication: Most communication is Pod-to-Service. The caller resolves the target via cluster DNS, then traffic is routed to one of the backing Pods.

    In practice, kube-proxy (or an equivalent data plane implementation) programs routing rules so traffic sent to the Service VIP is forwarded to healthy Pod endpoints.

    If a service mesh (for example, Istio) is enabled, the sidecar proxy (for example, Envoy) can intercept traffic and apply policies such as mTLS, telemetry collection, and advanced routing.

  • External communication:

    Case A: Without service mesh

    1. NodePort: Exposes a Service on a static port on every node.
    2. LoadBalancer: Provisions an external load balancer (typically cloud-managed) that forwards traffic to the Service (often via NodePort under the hood).
    3. Ingress: Provides Layer-7 HTTP(S) routing across multiple Services based on host/path rules.

    Case B: With Istio service mesh

    • Traffic usually enters through an Istio Gateway.
    • VirtualService resources define routing rules.
    • Sidecar proxies handle policy enforcement and traffic management between services.

4. RBAC vs. AuthorizationPolicy in Istio

  • Kubernetes RBAC controls who can do what to Kubernetes API resources (for example, list Pods, create Deployments, or delete Services).

  • Istio AuthorizationPolicy controls which workloads can communicate with other workloads at runtime (data-plane access control).

These two mechanisms are complementary:

  • RBAC secures the Kubernetes control plane.
  • AuthorizationPolicy secures service-to-service traffic.

TL;DR: RBAC governs access to Kubernetes resources. Istio AuthorizationPolicy governs access between running services.

5. Liveness and readiness probes

Kubernetes can self-heal applications using health probes.

  • Liveness probe: Checks whether a container is still alive. If it fails repeatedly, Kubernetes restarts the container.

  • Readiness probe: Checks whether a container is ready to serve traffic. If it fails, Kubernetes temporarily removes that Pod from Service endpoints until it becomes ready again.

This distinction helps avoid sending traffic to Pods that are running but not yet able (or no longer able) to handle requests.

6. Certificate request flow (cert-manager + ACME)

Related resources: CertificateRequest, Certificate, Issuer, ClusterIssuer, Order, Challenge.

At a high level, cert-manager automates certificate issuance and renewal. In the most common workflow, you apply a Certificate resource and cert-manager creates a CertificateRequest for you. (You can also create a CertificateRequest directly if you need lower-level control.)

When using an ACME provider such as Let’s Encrypt:

  1. Request is created: cert-manager creates (or observes) a CertificateRequest for a given domain / set of DNS names.
  2. Order is created: cert-manager creates an Order to request a certificate from the ACME server. The Order contains details such as the requested domain(s) and the list of challenges that must be completed.
  3. Challenges are solved: cert-manager creates one or more Challenge resources (for example HTTP-01 or DNS-01) to prove domain ownership.
  4. Certificate is issued: once the ACME server validates the challenges, it issues the certificate.
  5. Stored as a Secret: cert-manager stores the resulting private key and certificate chain in a Kubernetes Secret (as referenced by the Certificate / CertificateRequest).

Reference: https://medium.com/@gregoire.waymel/istio-cert-manager-lets-encrypt-demystified-c1cbed011d67

7. Scaling

Two common approaches to autoscaling in Kubernetes are:

  • HPA (Horizontal Pod Autoscaler): Kubernetes scales the number of Pod replicas based on metrics. The most common signals are CPU and memory utilization, but custom/external metrics are also possible (depending on your metrics pipeline).

  • KEDA (Kubernetes Event-driven Autoscaling): KEDA adds event-driven autoscaling using custom resources such as ScaledObject, ScaledJob, and TriggerAuthentication. It can scale workloads based on event sources (for example, queue depth). Under the hood, KEDA typically creates/manages an HPA for you.

8. Monitoring and logging

  • Monitoring (Prometheus + Grafana + Alertmanager):
    • Your application exposes metrics (commonly at a /metrics HTTP endpoint).
    • Prometheus scrapes those time-series metrics and stores them in its TSDB. If you use the Prometheus Operator, resources like ServiceMonitor/PodMonitor define what gets scraped.
    • Prometheus supports PromQL for querying metrics and can evaluate alerting rules.
    • Alertmanager routes firing alerts to notification channels (for example Slack, email, PagerDuty).
    • Grafana visualizes metrics and dashboards by querying Prometheus.
  • Logging (Loki + Promtail):
    • In Kubernetes, the standard pattern is for applications to write logs to stdout/stderr.
    • Node-level agents (for example Promtail) collect container logs from the node and ship them to a central system.
    • Loki stores logs (typically with label-based indexing) so you can query and explore them later (often from Grafana, using LogQL).

The key idea is that logs stored only on a node are ephemeral: if a Pod is evicted/drained and rescheduled elsewhere, you’ll lose node-local log files unless they’ve already been shipped to a centralized logging backend.