Introduction

Kubernetes excels in container orchestration and infrastructure management, providing a solution for service discovery, scaling, and resource management. However, when it comes to the needs of developers Kubernetes raises many questions such as:

• How useful is Kubernetes' service discovery without built-in resiliency or access control?
• StatefulSet is a powerful primitive, but what is it truly useful for? While great for creating Kubernetes operators, it doesn’t help developers implement stateful business processes.
• Kubernetes can manage storage volumes, but modern applications often need state management solutions such as key/value stores, databases and more. How to use one consistently across languages and clouds, the cloud native way?
• Kubernetes Jobs can handle scheduled tasks, but they often involve unnecessary complexity for developers. Do you need to schedule a whole pod to run a task every 2 minutes?
• Kubernetes can manage distributed locks, but the process can be cumbersome. Similarly, do developers need to write an operator to handle an application lock?
• Why isn't there a Kubernetes-like project specifically for developers? There is—it's Dapr, and here’s why. Inspired by the cloud-native movement and built on the same principles of polyglotism, heterogeneous applications, and multi-cloud, the Dapr project complements Kubernetes and improves developers’ productivity the same way Kubernetes improves operations teams’ productivity.

We will look at these areas where Kubernetes and Dapr differ, showcasing how Dapr complements Kubernetes by addressing the specific needs of developers. If you know Kubernetes, this article will show what you might be missing and how Dapr can enhance the work of both operations teams and developers. If you are already developing with Dapr, sign up to Diagrid Conductor (it is free) and find out how it helps organizations operate Dapr in production with confidence. If you prefer a video version of this post, check out my webinar recording and slides on this topic.

‍

Let’s start with application lifecycle management and address the distributed application pain points from the ground up.

1. Application Lifecycle

Kubernetes excels in application lifecycle management, supporting a diverse range of workloads including stateless applications, stateful applications, cron jobs, singletons, and even rapidly scaling serverless workloads through Knative. Its primary role is orchestrating containerized applications, irrespective of their language, framework, or architecture.

Kubernetes abstracts the underlying infrastructure, enabling developers to combine containers into pods in various configurations through init-containers and sidecars. This feature is beneficial in cloud environments, ensuring applications are decoupled from the underlying compute and can be managed uniformly at a large scale through automation.

Dapr, in contrast, does not dictate how applications are run; instead, it adapts to various application lifecycles. It seamlessly integrates with different Kubernetes constructs like Deployments and StatefulSets and acts as a dedicated sidecar for long-running applications. Dapr can also run in DaemonSets (or Deployments) to act as a shared sidecar—an option particularly useful with function-based models requiring rapid scalability. Additionally, Dapr can be injected into short-lived Jobs, providing flexibility to support different deployment models in Kubernetes environments and serverless setups.

2. Health Checks

On Kubernetes, applications rely on health checks to ensure reliability and resilience. There are three types: Startup Probes verify if the application has started correctly, Readiness Probes check if the application is ready to serve incoming traffic, and Liveness Probes determine if an application needs to be restarted.

As a good cloud-native citizen, the Dapr sidecar container also provides sidecar health endpoints for Kubernetes to probe, ensuring its components and sidecars are functioning correctly. This integration allows Kubernetes to monitor the health of Dapr, ensuring that the application can interact with the sidecar effectively.

Dapr offers an additional layer of health checks for the application itself, which doesn't directly affect the application's lifecycle but improves its interactions with other applications and infrastructure. Unlike Kubernetes health checks, which might restart the application or stop incoming traffic to an unhealthy container, Dapr's application health checks operate differently. For example, if an application becomes temporarily unhealthy, Dapr can proactively stop accepting new work on behalf of the application by:

• Unsubscribing from all pub/sub subscriptions
• Stopping all input bindings
• Short-circuiting all service-invocation requests, which terminate in the Dapr runtime and are not forwarded to the application

These changes are temporary, and Dapr resumes normal operations once it detects that the application is responsive again. Application health checks ensure smoother operation and reduce potential strain on the application during recovery periods. Dapr App health checks are meant to be complementary to, and not replace, any Kubernetes health checks. Kubernetes health checks ensure that the application is running, and cause the platform to restart the application in case of failures. Dapr’s app health checks focus on pausing work to an application that is currently unable to accept it, but is expected to be able to resume accepting work eventually.

3. Synchronous Communication

Kubernetes has the Service Resource, which is a method for exposing a network endpoint for an application running as one or more Pods in your cluster. Depending on the type, kind, and spec fields in the Service definition, there are 7-8 types of Services in Kubernetes. Among these, NodePort, LoadBalancer, and Gateway help to get traffic into the Kubernetes cluster. Additionally, there is a service that helps discover endpoints outside of the Kubernetes cluster.

The most popular type is ClusterIP, which helps with in-cluster service calls, allowing applications to discover the right pod when performing service-to-service interactions. This enables an application to use the service name only to call another application Pod. While this approach works well within Kubernetes, it becomes problematic when running the same application locally, outside of Kubernetes, as service names require manual configuration in environment variables. In addition, Kubernetes' internal service discovery lacks certain resilience features such as circuit breakers, retries, and timeouts, and doesn't natively support tracing, metrics, or traffic encryption. To integrate these capabilities, one typically needs to introduce an additional service mesh layer into the Kubernetes environment. Dapr's Service Invocation API addresses these limitations.

With Dapr Service Invocation, developers get the following benefits:

• Location transparency: When deployed on Kubernetes, Dapr leverages Kubernetes service discovery to locate services within the cluster, but it can also integrate with other service discovery systems like Consul, making it universal for various deployment environments, including local.
• Resiliency: Dapr provides built-in resiliency features for retries, circuit breakers, and timeouts, ensuring that services can handle transient networking failures gracefully.
• Observability: Dapr offers comprehensive tracing, network metrics, and logging capabilities, enabling developers to monitor and debug service interactions effectively.
• Security: Achieved through access control mechanisms, ensuring that only authorized services can communicate with each other, mutual TLS (mTLS) and token authentication between the application and the sidecar

Kubernetes' service discovery is limited to simply locating services, necessitating an additional layer such as a service mesh to address resilience, observability, and security needs. Dapr addresses these needs directly through its Service Invocation API, providing a comprehensive solution for service-to-service communication.

4. Access Control

Namespaces are a crucial part of Kubernetes, allowing you to group your workloads together. However, they only provide a grouping concept. In Kubernetes, every Pod can talk to every other Pod, regardless of their namespace. This default behavior has security implications, particularly when multiple independent applications operated by different teams run in the same cluster. To prevent this, Kubernetes has the NetworkPolicy resource which acts like an application firewall. By defining resources of the type NetworkPolicy, developers can create ingress and egress firewall rules for workload Pods.

Network policies allow:

• Pod-level network isolation: Ensures that Pods can only communicate with authorized Pods.
• Support for IP and namespace-based rules: Provides flexibility in defining broader network rules.
• Ability to define inbound and outbound traffic: Controls both incoming and outgoing traffic to/from Pods.
• Protocol and port-based filtering: Supports filtering based on protocols (TCP/UDP/SCTP) and ports.

The main limitation of this approach is that NetworkPolicy operates on the L3/L4 networking layers and doesn’t understand application protocols. This means it cannot enforce granular access control to limit access to called applications from specific operations and HTTP verbs from the calling applications. To offer such features, Kubernetes requires a service mesh such as Istio, or you can use Dapr.

Dapr complements Kubernetes by providing access control features that extend beyond standard HTTP protocol-level security. It incorporates similar functionalities to service meshes but also includes access control for a broader range of APIs. Dapr's Access Control Lists (ACLs) allow fine-grained control over service-to-service invocation, enabling users to define detailed policies governing which services can invoke others.

Furthermore, Dapr extends its granular security model to other APIs like Pub/Sub messaging, enabling Zero-Trust architecture. The Pub/Sub scoping feature enables topic access control, allowing administrators to define which applications can publish or subscribe to specific topics.

These capabilities often raise the question, is Dapr then a service mesh? In depth, Dapr, when utilized alongside Kubernetes, effectively assumes the role of a service mesh, as detailed in its documentation Dapr as a Service Mesh. It provides essential service mesh capabilities like secure traffic encryption, access control, network resiliency, network metrics, and tracing.

However, unlike traditional service meshes that operate as a transparent proxy, used by operations teams without any involvement from developers, Dapr offers a set of APIs for explicit interaction through a standardized, polyglot API for applications to call. This approach gives developers greater visibility and control over communication patterns, not limited to synchronous interaction only.

In summary, Kubernetes provides fundamental L3/L4 network access control via NetworkPolicy but requires service meshes like Istio or eBPF-based Cilium for more granular control. Dapr, similarly to these technologies, brings a comprehensive security model that not only includes HTTP access control but also extends to other API interactions.

5. Event-driven Communication

Kubernetes alone does not provide abstractions for implementing event-driven applications, but the wider CNCF projects offers several projects to fill this gap. Strimzi is used for operating Apache Kafka, Knative Eventing for distributing events, KEDA for scaling out event-driven consumers, and CloudEvents serves as a standard messaging envelope.

In this context, Dapr simplifies event-driven interactions with its pub/sub API by integrating with various message brokers using CloudEvents. This API allows for asynchronous communication between services, decoupling the event producers from consumers. Some of the key benefits of Dapr's pub/sub API include:

• Message Broker Abstraction: Dapr abstracts the complexity of accessing underlying messaging systems, providing a uniform and straightforward way to implement publish and subscribe mechanisms in applications.
• Push and Pull Model for Subscriptions: Dapr supports both push and pull modes for subscriptions, allowing developers to choose the mode that best fits their application's needs.
• Messaging Patterns: Dapr supports messaging patterns like content-based message routing and filtering, and aggregator/splitter through message batches, making it flexible for various use cases.
• Dead-letter Topics and Resiliency: Dapr includes features like dead-letter topics for handling message failures and resiliency policies for retrying and managing message delivery failures.
• Message Expiration: Dapr allows for message expiration, ensuring that stale messages do not reach the consumer, irrespective of message broker implementation.
• Cross-Language Support: Dapr's pub/sub API is language-agnostic, making it easy to integrate into applications written in different programming languages.

These capabilities simplify the development of event-driven applications by abstracting message brokers implementation and unifying access across all languages.

6. State Management

Stateless services are stateless because they do not maintain internal state, requiring external state storage such as databases, filesystems, or other data stores to offload the state. Kubernetes aids in this aspect by providing persistent volume (PV) support for Pods. A Pod can declare and use file storage through PVCs. This indirection between the Pod and the PV allows for the decoupling of their lifecycle and ensures data outlives ephemeral Pods. Kubernetes offers several types of volumes, ranging from cloud-provider-specific storage to network storage or even node-shared file systems. An important aspect to consider is the PVC’s accessModes field with the following values:

• ReadWriteOnce: A volume that can be mounted to a single node at a time.
• ReadOnlyMany: The volume can be mounted to multiple nodes.
• ReadWriteMany: The volume can be mounted by many nodes.

These modes dictate how many nodes a volume can be mounted on, not how many pods can access the volume. To address this, a new ReadWriteOncePod accessMode was added, which guarantees that only a single Pod has access to a volume. Even with these modes, once a storage is mounted to a Pod, there is no guarantee of isolation or concurrency control.

Dapr StateStore API, on the other hand, does not constrain storage access to the file system only, but provides a unified Key/Value-like API for accessing a variety of data stores, which could be running on the same Kubernetes cluster (like PostgreSQL, Redis, MongoDB) or as cloud services (AWS DynamoDB, Azure Cosmos, GCP Firestore, etc.).

Key features of Dapr’s state API include:

• State Management: It abstracts state management, allowing applications to interact with state stores through a consistent model regardless of the underlying state store.
• Concurrency: Provides first-write/last-write and consistency behaviors such as strong or eventual consistency.
• Transactions: Supports transactions for multi-item operations on the state, and bulk operations.
• Data Encryption and TTL: Can encrypt data at rest and enforce TTL (time-to-live) regardless of the backing data store.
• Access Control: Allows defining which applications can access the state store.

In summary, Kubernetes offers file system-based volumes to your application with limited concurrency and isolation. In contrast, Dapr offers API-level access to state stores, providing a more flexible and feature-rich solution for managing application state.

7. Stateful Workloads

In Kubernetes, the concept of stateful workloads is defined through StatefulSets. A StatefulSet in Kubernetes is designed to manage stateful applications from a running process point of view and provides several guarantees. These include:

• Storage: Contrary to stateless workloads (such as those managed by Deployment) where all instances have access to a shared persistent volume, in a StatefulSet, every instance can have dedicated persistent storage.
• Networking: Similar to the storage requirements, a distributed stateful application requires a stable network identity. Every instance should be reachable at a predictable address that should not change dynamically, as is the case with Pod IP addresses in a ReplicaSet.
• Identity: Clustered stateful applications depend heavily on each instance having a stable and unique identity. Each instance knows its own identity and can maintain long-lived storage and network connections though it.
• Ordinality: In addition to a unique and long-lived identity, instances of clustered stateful applications have a fixed position in the collection of instances. This ordering typically impacts the sequence in which the instances are scaled up and down.

While the StatefulSet guarantees provide a powerful foundation for writing custom Kubernetes Operators for managing databases, message brokers, and key/value stores, they are too low-level and not useful for application developers. If you ask a developer what a stateful application is, they will point to stateful business processes or service interactions using the SAGA pattern as an example.

This is why the concept of stateful applications in Dapr is centered around stateful network interactions, not the stateful process lifecycle. Dapr introduces the Workflow API, where a stateful application is defined by the sequence and durability of its interactions. This API allows developers to define a series of steps that an application should execute, even if the application is restarted, ensuring that each step is completed before moving on to the next. Dapr workflows provide a durable way to handle complex interactions within the applications by persisting the state of the workflow into any transactional state store. Using the Dapr SDK for the different languages, the developers can define various workflow patterns such as Sequence, External Trigger, Split/Syncrhonize, Timer, and more.

To sum up, stateful workloads in Kubernetes refer to the application instances, focusing on the order in which application instances should start and stop, a predictable network endpoint per instance, and storage for each instance. Dapr's view of statefulness is more about the service interactions defined within the application itself, focusing on the sequence and durability of these interactions.

In the realm of distributed applications, developers often leverage both synchronous and asynchronous interactions. The optimal approach frequently involves a blend of interaction styles, tailored to specific business requirements. Unlike other application frameworks, Dapr not only caters to both synchronous and asynchronous patterns but also enables advanced choreography and orchestration patterns, all within a single programming model available for multiple languages.

8. Finite Tasks

The smallest programmable unit in Kubernetes is the container. If you want to execute any kind of logic, you have to put it into a container and run it as a Pod at least once as a Job. Kubernetes will find a node to schedule the Pod and run it there the desired number of times, and at the desired parallelism. This allows the offloading of the placement of containers onto the nodes and scheduling the tasks to run in Kubernetes, but has the overhead of packaging and the full lifecycle of any kind of logic into a container of its own.

In Dapr, on the other hand, the smallest computational unit is the Actor. Actors allow you to write your code in a self-contained unit (called an actor) that receives messages and processes them one at a time, without any kind of concurrency or threading concerns. While your code processes a message, it can send one or more messages to other actors or create new actors. Just as Kubernetes manages container placement and lifecycle on multiple nodes, Dapr manages actor instances among the application workloads. The Dapr actor runtime automatically activates an actor when a request is received, and if unused for a certain period, it performs garbage collection. Actor instances are distributed across the cluster, with Dapr migrating them from failed nodes to healthy ones as needed.

What about tasks that need to run periodically? Kubernetes has CronJob, which allows the execution of a unit of work defined by the Job resource to be triggered by a temporal event, for example, through a cron expression. A recurring task in Dapr is not a complete application that starts and executes until shutdown, but an invocation to your application endpoint. A recurring call can be durable and survive process restarts, or be in-memory. It can be implemented as Actor timers and reminders, as the Cron Binding, or through the brand new Dapr Job API coming in Dapr 1.14. The end result is the same: your application receives a call at the specified time intervals to perform the desired action.

In summary, Kubernetes Jobs or CronJobs are suitable for implementing recurring tasks that execute complex tasks, run less frequently, and consume a large amount of resources. Kubernetes will schedule your job on a node that has capacity, and once completed, it will be shut down and the resources on the node will be freed up for another job. Dapr Jobs, on the other hand, are executed on already running sidecars and application instances. As a result, they are more suitable for tasks that need to be executed more frequently, faster, and lighter. Once your application receives the trigger from Dapr, it performs the task and keeps running, ready for other tasks.

9. Distributed Locks

Locks are used to provide mutually exclusive access to a resource such as a database, file system, or message broker. Locks are usually applied to operations that mutate state, and any resource that is shared where updates occur may require a lock to prevent conflicts, not on reads.

In Kubernetes, one way to implement this concept is by ensuring that only one instance of an application is active at a time while maintaining high availability. The concept of running a single Pod replica, especially in failure scenarios, is more nuanced. Kubernetes primitives like ReplicaSets prioritize availability over consistency, which means a ReplicaSet with replicas: 1 can result in temporary situations where more than one instance is active (for example, during Pod node relocation or if a node gets disconnected from the cluster). StatefulSets, on the other hand, favor consistency over availability and are tailored for stronger guarantees over the number of instances.

Kubernetes also has the Leases resource, which provides a mechanism to lock shared resources and coordinate activity between members of a set. The Lease object is used by Kubernetes itself for node heartbeats and control plane component leader election. It is possible to use this resource in a custom Kubernetes Operator to implement distributed locks, but this is not a generic solution.

Dapr offers a more generic approach with its Distributed Lock API, allowing applications to use a named lock for exclusive access to shared resources. This API works by ensuring that at any given time, only one instance of an application holds the lock. This lease-based mechanism avoids deadlocks in case of application failures by ensuring automatic release of locks after a set period. Dapr's approach provides flexibility, allowing the use of various storage and lock implementations, and ensures that even with multiple healthy Pod replicas, only one service instance actively performs the business functionality.

In summary, Kubernetes allows control over the number of running application instances, which can be used in managing access to shared resources. However, acquiring an exclusive lock within the application, either via Kubernetes' Lease resource or Dapr's Distributed Lock API, offers more granular control. Dapr's approach allows for different backing stores for the lock and a simple API interaction for the distributed lock problem.

10. Configurations and Secrets

Kubernetes offers configuration management through ConfigMap and Secret objects, designed for general-purpose and sensitive data, respectively. Both ConfigMaps and Secrets store and manage key-value pairs, offering more flexibility than simple environment variables. ConfigMaps can be utilized in two ways: as references for environment variables (where keys become environment variable names) or as files within a volume mounted to a Pod, with keys as filenames. Changes in ConfigMaps can be dynamically reflected in mounted volumes, aiding applications that support hot reloading of configuration files. However, environment variables do not update dynamically once a process has started. Despite their utility, ConfigMaps and Secrets have limitations, such as a 1 MB size cap for Secrets and possible quotas on the number of ConfigMaps and Secrets per namespace.

Dapr's approach to configuration and secrets management offers additional flexibility and is not tightly coupled to Kubernetes. The Dapr Configuration API allows for dynamic updates to application configurations as changes occur. It can integrate with various configuration systems, pushing updates directly to applications, a feature particularly useful for environments where configurations need to be frequently updated or varied across different stages or deployments.

Key features of Dapr's Configuration and Secrets Management include:

• Integration with Various Systems: Dapr can pull configurations and secrets from a variety of sources, including cloud services, local stores, and external systems, providing a unified interface for configuration management.
• Dynamic Updates: Dapr can push configuration changes to applications in real-time, allowing for immediate updates without requiring restarts.
• Scoped Secrets: Dapr allows scoping of Secrets to specific applications, ensuring that each application gets only the configuration data it needs, enhancing security.

In summary, Kubernetes' ConfigMaps and Secrets provide a structured way to handle configuration data, decoupling configuration definition from usage and enabling independent management of configuration-consuming objects. However, they have their limitations in terms of data size and scope of use. Dapr's Configuration API, extends this flexibility, allowing for dynamic, real-time configuration updates and integration with a broader range of systems, making it a more suitable choice for complex or portable scenarios.

11. Resiliency

Resiliency, crucial in cloud-native applications, is tackled differently by Kubernetes and Dapr. Kubernetes ensures the continuous operation of application processes, while Dapr focuses on the resilience of network interactions, particularly when dealing with application or infrastructure failures.

Kubernetes’s approach to resiliency

Kubernetes is primarily responsible for ensuring a consistent number of replicas of the container processes by:

• Node failure handling: In Kubernetes, node failures disrupt all Pods on the affected node, risking application downtime. To address this, Kubernetes monitors node health via heartbeats and, upon detecting a failure, marks the node as "unreachable." It then reschedules the pods to other available nodes, ensuring service continuity and minimizing disruption.
• Container failure handling: Container failures, whether due to code issues or system problems, can lead to service interruptions. Kubernetes mitigates this by constantly monitoring container health and automatically restarting failed containers. Additionally, it can maintain multiple pod replicas, providing high availability and load distribution against individual pod failures.
• Rolling updates & rollbacks: Application updates carry the risk of downtime or new bugs. Kubernetes uses rolling updates, sequentially updating pods to maintain uninterrupted service. It also provides rollback options, allowing quick reversion to stable versions in case of issues, thus ensuring ongoing service stability.

Dapr's approach to resiliency

Dapr has no say on the application process resiliency, rather it is focused on network communications among applications and infrastructure services.

1. Timeouts: Operations that hang indefinitely can cause system delays and inefficiencies. Dapr combats this by setting maximum durations for operations, thus preventing indefinite delays and enhancing overall system responsiveness.
2. Retries/Back-offs: Transient failures in network or service connectivity can disrupt application workflows. Dapr's solution involves automatic retries with a back-off strategy, improving reliability in unstable network environments.
3. Circuit Breakers: Continuous operation failures can overload systems. Dapr uses circuit breakers to temporarily halt failing operations, reducing the risk of repeated failures.

It is important to highlight that Dapr’s resilience policies are not only for application interactions, but also interactions with backing infrastructure services such as message brokers, databases, configuration stores, etc.

In summary, Kubernetes focuses on maintaining the resilience of application processes, managing container and node lifecycles to ensure high availability. Dapr, on the other hand, specializes in the resilience of service interactions, addressing challenges like timeouts, retries, and circuit breakers. Together, they complement each other for ensuring both process execution and network communication resilience in cloud-native environments.

Extending Kubernetes with Dapr

Dapr and Kubernetes are two distinct technologies. Comparing them directly is akin to comparing apples to oranges; while Kubernetes excels in container orchestration, Dapr specializes in connecting applications and infrastructure. Kubernetes' worldview is limited to container instances, file volumes, and network connections. Dapr is all about providing applications APIs, and addressing cross-cutting concerns for developers.

Kubernetes is the tool for operations teams, providing a robust platform for deploying, scaling, and managing applications in cloud environments. It's a universal platform for container management, but it leaves a gap in application-level developer concerns—an "empty space" that Dapr fills in.

To fill that space, developers have to complement Kubernetes with a service mesh for secure and reliable application connectivity, a pub/sub system for event-driven interactions, workflow engines, cron schedulers, and a myriad of libraries for connecting to third-party configuration, secret, and state stores. Or, developers can use Dapr, which provides all of these capabilities uniformly for all languages to implement microservices patterns the cloud-native way, ensuring consistency and efficiency across diverse development environments.

In summary, Kubernetes and Dapr are complementary technologies. Kubernetes excels in container orchestration and infrastructure management, making it an essential tool for operations teams. Dapr, on the other hand, focuses on the application layer, providing a comprehensive set of APIs and tools that address common development challenges in building microservices and distributed applications. Together, they create a powerful duo for building, deploying, and managing cloud-native applications.

Interested in continuing this discussion? Tag me on twitter with your questions, or dive deeper straight into Dapr. Join the Dapr Discord server to connect with a vibrant community of developers, and connect with us to find out how Diagrid Conductor helps organizations operate Dapr in production with confidence.