Understanding Dapr Pub/Sub Subscription Types: Declarative, Programmatic & Streaming

Dapr 1.14 introduces a new method for delivering asynchronous messages to pub/sub subscribers, unlocking new use cases when creating event-driven distributed applications. This post demonstrates how to use all three types of subscriptions across different programming languages and analyzes their pros and cons. To illustrate this we will use a publicly available sample application and monitor the subscriptions using Diagrid Conductor. If you don’t have a Conductor Free account, sign up to follow along.

The Dapr pub/sub API is the most popular API of the Building Block APIs. It enables applications written in various languages to communicate asynchronously and removes the complexities of the underlying messaging infrastructure. There are always two applications involved - the publisher app, that writes messages to an input channel and sends them to a topic, and the subscriber app that subscribes to the topic and receives messages from an output channel. An intermediary message broker copies each message from the publisher’s input channel to an output channel for all subscribers to listen on. Until recently, the only way messages could be delivered to subscribers was by the Dapr sidecar pushing each message to the application endpoints. This method is common in pub/sub systems such as Knative Eventing, GCP Pub/Sub, and AWS EventBridge. However, it is less common for traditional messaging and streaming applications, where the subscriber opens a connection to the message broker to consume messages, as with Apache Kafka and RabbitMQ. With the release of Dapr 1.14, the pub/sub API enables this with streaming subscriptions, allowing the subscriber to open a connection to the sidecar to receive messages. Let’s dive into each subscription type in Dapr to analyze the pros and cons.

Declarative Subscriptions

Dapr Declarative subscriptions specify the topics that applications subscribe to, and the application endpoint where the messages should be routed. Here's a simple declarative subscription in YAML, configuring Dapr to deliver messages to a subscriber application:

apiVersion: dapr.io/v2alpha1
kind: Subscription
spec:
 pubsubname: pubsub
 topic: orders
 routes:
   default: /orders
scopes:
 - declarative-subscriber

In the subscriber, the incoming requests are handled by implementing the `/orders` endpoint in the application. See the sample app for additional details.

When to use Declarative Subscriptions?

Pros:

• Declarative subscriptions are easy to configure and understand via YAML files. This is ideal for teams that prefer a configuration-based approach rather than writing application code.
• Keeping subscription configuration separate from application code allows defining subscription parameters (such as broker, topic, dead letter topic, bulk message consumption) uniformly and independently. This is particularly useful when you don't want to or cannot change the application code and is useful when integrating with legacy applications.
• It is easy to reuse these subscriptions by copying and pasting the subscription definition for other applications or by “scoping” a single subscription to multiple applications.

Cons:

• This approach is less flexible for complex scenarios where subscription behavior can change at runtime. Any configuration updates require editing the configuration files and restarting the Dapr sidecar to pick up the new configuration (with Dapr 1.14 restarting will not be required using Dapr Component hot-reloading).
• It is not preferred by teams that prefer everything to be defined as part of the application code, where it can be validated in the IDE and tested with the application.

Overall, even if not very flexible, declarative subscriptions are a good starting point aligned with the Kubernetes mindset of YAML files to reduce application code.

Programmatic Subscriptions

Dapr Programmatic subscriptions allow developers to define subscriptions directly in their application code. Rather than specifying the details in a YAML file, subscribing applications implement the /dapr/subscribe endpoint and return a JSON object containing the subscription details. On startup, the Dapr sidecar reaches out to this endpoint checking for topic subscriptions. Here's a simple example of a programmatic subscription in Python:

# Subscribe to a topic
@app.route('/dapr/subscribe', methods=['GET'])
def subscribe():
   subscriptions = [{
       'pubsubname': 'pubsub',
       'topic': 'orders',
       'route': 'orders'
   }]
   return jsonify(subscriptions)

In this subscriber, the incoming requests are still handled by implementing the /orders endpoint in the application, whereas the /dapr/subscribe endpoint just creates the subscription. See the sample app for additional details.

When to use Programmatic Subscriptions?

Pros:

• Programmatic subscriptions provide greater flexibility, allowing developers to define and manage subscriptions directly through their application code, which is useful for dynamic scenarios.
• This approach offers better integration with the application code, enabling easier validation and testing within the IDE.

Cons:

• Managing subscriptions in application code leads to tighter coupling between the subscription options and the application implementation.
• For teams that prefer configuration-based approaches, programmatic subscriptions might be less appealing as they require more code and less separation of concerns between development and platform/infrastructure teams.

Both declarative and programmatic subscriptions are similar in the way subscriptions are defined and message delivery mechanism. In both scenarios, the application is required to have an endpoint for consuming the messages, where the sidecar is pushing every message. Streaming Subscriptions address some of these limitations and challenges in a fundamentally different way.

Streaming Subscriptions

Push-based subscriptions rely on the Dapr sidecar to push messages to the application via HTTP POST or gRPC calls. While this method is simple and offloads the responsibility of routing, retries, and dead letter queues to Dapr, it falls short in situations where applications need to subscribe or unsubscribe from topics at runtime based on specific conditions or events. Additionally, when applications require lower latency and higher throughput, maintaining a persistent connection is more effective than establishing a new one for each message.

Streaming subscriptions address these needs by allowing applications to open a persistent connection to the sidecar and pull messages from it. Here's an example of a streaming subscription using the Dapr SDK in Go:

client, err := daprd.NewClient()
deadLetterTopic := "deadletter"


sub, err := client.Subscribe(
	context.Background(), daprd.SubscriptionOptions{
  PubsubName:      "pubsub",
  Topic:           "orders",
  DeadLetterTopic: &deadLetterTopic,
})

fmt.Printf(">>Created subscription\n")

for {
  msg, err := sub.Receive()
  log.Printf("Received: : %s\n", msg.RawData)
}

In this example, the application creates and maintains a persistent gRPC connection to the Dapr sidecar and processes any incoming messages as they come in. Notice how there is no endpoint defined on the subscribing application at all. See the sample app for additional details.

When to use Streaming Subscriptions?

Pros:

• Streaming subscriptions provide greater flexibility by allowing applications to subscribe and unsubscribe at runtime. This is particularly useful in systems that need to respond to changing conditions or events.
• Increased performance by maintaining a persistent gRPC connection between the application and sidecar, reducing the overhead associated with establishing new connections for each message that is delivered.
• The subscriber no longer needs to create an endpoint and open an application port to listen for sidecar calls. Rather the connection is opened outbound from the subscriber to the sidecar. 

Cons:

• For teams that prefer configuration-based approaches, streaming subscriptions might be less appealing as they require more application code with additional error handling, testing, and maintenance overhead.
• At the time of writing, streaming subscriptions are an Alpha feature supported only by the Go SDK, with others to come in upcoming Dapr versions (1.14+).

Discovering Dapr Subscriptions & More

Large-scale distributed applications use multiple interaction styles, from synchronous service calls to asynchronous event-driven interactions, workflows, actors, resiliency policies, and more. When there is an incident, gaining visibility into such a system and understanding the configurations, application interactions, networking metrics, and alerts can be a daunting task. Diagrid Conductor provides unified visibility into the Dapr runtime that is unmatched by other generic monitoring tools. Subscription visibility is not an exception. Conductor centralizes the visibility of different subscription types — declarative, programmatic, and streaming — allowing teams to monitor subscriptions without switching between disparate tools or interfaces.

Regardless of the subscription type, Conductor lists them in a single view and generates a unified representation as a YAML file for quick assessment of the configuration. When more detailed information is required, Conductor provides networking metrics, including those related to subscriptions of all types. This includes data on message delivery latencies and failure rates, as well as default alerts on critical issues. Conductor allows all parties to easily track which services are subscribed to which topics and how messages are routed across the system without knowing the intricacies of the application code.

To experience unified visibility of your Dapr resources, sign up for Diagrid Conductor today (it's free). Try out the sample subscriptions application to see how Conductor can enhance your operational efficiency and give you deeper insights into your Dapr deployments.