In today's fast-paced development world, knowing best practices can mean the difference between missing a deadline or delivering on time, and between a security breach or no breach. Best practices encapsulate the distilled knowledge of practitioners and subject matter experts, allowing developers to avoid common pitfalls and build solutions without reinventing the wheel. However, staying current with these evolving practices can be challenging due to constant time constraints and the rapid evolution of software technologies. Diagrid was founded by the creators of the Dapr project, and has an engineering team working full-time on developing Dapr, with the Conductor SaaS product helping users operate thousands of Dapr Kubernetes clusters. This extensive involvement provides the team with deep insights into the best practices for effectively developing and operating Dapr. We have encapsulated and codified this knowledge as part of Conductor’s Advisor feature and made it freely available.

In this article, we share key pitfalls to avoid and best practices to follow for developing secure and reliable distributed applications with Dapr. We will also demonstrate how Conductor's Advisor feature automates the process of detecting and implementing these recommendations in your projects. This automation not only saves time and money but also ensures that your applications continuously adhere to the latest standards, giving you a peace of mind. Check out the demo of Advisor:

Security Best Practices

Ensuring your application is secure and reliable starts in development, and shouldn’t be an afterthought of the operations team. Zero Trust Security is at the heart of application development and addressing these concerns early can save considerable time and resources later. For a distributed application using Dapr, this involves enforcing security best practices at multiple levels.

A good starting point is to implement coarse-grained security by scoping Dapr resource specifications to applications. This controls which apps can access resources like infrastructure components and configuration policies. Next, configure authorization rules to define the actions each application can perform, ensuring fine-grained access control. Securing sensitive connection information is crucial, so use secrets to manage data like passwords and certificates, preventing exposure in plain text. In some cases, stricter security measures, such as locking Dapr APIs and enabling token authentication for app-sidecar interactions, may be necessary. By navigating these steps, you add more fine-grained security controls, ensuring your Dapr applications are secure and reliable from the development phase onward.

But how do you identify which Dapr configurations are not following these best practices and which applications are affected? This is where Conductor Advisor comes in. It continuously checks your Dapr installation for misconfigurations and highlights how to resolve them. Below is an overview of the current list of security checks performed by Conductor.

For each advisory, you can see whether it is resolved or unresolved, its potential impact, how many resources are affected, and how to resolve it. Since we don’t have time to go through all of these in this post, we will focus on developer-focused advisories. Operational concerns, such as the Dapr control plane and deployment options, will be covered in a follow-up article.

Scoping Dapr Resources to Apps

Dapr offers several resource types: Component Spec, Subscription Spec, Resiliency Spec, HTTPEndpoint Spec, and Configuration Spec. Except for Configuration, all these can be (and should be) scoped to one or more apps. The first step is to ensure that only the applications needing access to these resources can use them. Creating a resource without adding scopes means that any app in the Kubernetes namespace where the resource is created loads and accesses it through the Dapr sidecar. This could lead to an exposed application accessing all unscoped resources and the backing infrastructure services such as message brokers, databases, and configuration stores. Bellow is how Conductor Advisor highlights this best practice:

Conductor provides a description of the potential impact if the best practice is not applied, along with brief steps to resolve it, a sample code snippet, and a link to Dapr documentation for more details. Additionally, it lists the affected Dapr resources, allowing you to access and review them with a single click.

Enforcing Access Control

Scoping a resource is a good first step, but it is not granular enough. Without further constraints, an application with scoped Dapr resources can have full access to the backing service, such as a Kafka broker, represented by the resource, like a pub/sub component definition. Therefore, we must further restrict access by specifying what actions an application can perform. The allowed operations and the way they are restricted vary among Dapr APIs.

PubSub Topic and Access Control

Scoping a pub/sub component spec to an app allows only that application to access the pub/sub broker, but it grants full access to all topics and operations, regardless of whether it is producing or consuming. You can further restrict access by specifying which topics are available for publishing and subscriptions to particular applications. Here is what Conductor recommends for Pub/Sub Topic Scoping:

For example, to lock down PubSub broker access, you can configure it so that App1 can only publish to topic1, and App2 can only consume from the same topic. No other app can access any other topics from this component.

Service Invocation Access Control

Dapr's Service Invocation API enables synchronous interactions between applications. By default, all apps can discover and call each other on any method, operation, and protocol (HTTP or gRPC). To ensure security, you need granular access control. This limits which apps can call each other, over which protocol, operation, and path through service invocation APIs.

Here's what Conductor’s Access Control advisory for Service Invocations recommends:

This setup allows only the method /op1 from app1 to be called from app 2. All other requests from all other apps, including other methods on app1, are denied. This allows extremely granular control over which methods can be called on an application and from which apps. This means that if an application is compromised its blast radius of what else it could access is restricted, crucial in highly secure environments.

Setting Binding Direction

Dapr Binding components should be limited by indicating whether they can be used as input or output bindings, thereby restricting the types of operations they expose. Imagine you have a Kafka input binding for consuming messages from a topic. If the app is exposed, the same app can use the scoped binding to produce messages as well. To prevent this, limit the binding to the direction of its interaction:

This snippet restricts the binding to only consume messages, ensuring it cannot produce messages. Conductor detects the lack of direction specification in bindings but doesn't dictate what its value should be. Notice that in some cases, it is also valid to have both input and output as valid directions. Specifying it explicitly makes it clear it is a deliberate decision.

Avoiding Sensitive Data

Many Dapr component specs require sensitive data, such as passwords and certificates, to connect to external systems and backing infrastructure. It's better security practice to retrieve these values from external secret stores rather than including them as plain text in resource definitions. This is a common bad practice found in many production deployment scenarios, and addressing it in lower environments is ideal. Here is an example of using `secretKeyRef` for the `redisPassword` field in a Redis component provided by Conductor:

In this example, the Redis password is stored in a Kubernetes Secret named `redis-secret` with a key `redis-password`. This approach ensures that sensitive information is not hard-coded into your component specs, enhancing the security of your application.

Securing App and Sidecar Interactions

In production, you typically enable mTLS to encrypt interactions between sidecars, allowing the pod to act as the security boundary (one of Conductor Enterprise’s operational advisories we will not cover in this post). But what about securing the interactions within the Kubernetes Pod between the app and the sidecar containers? This can be essential in highly regulated and secure environments. There are two types of communication here: from the application to the Dapr sidecar and vice-versa, from the Dapr sidecar to the application. We'll show the application reaching outbound first.

By default, Dapr relies on the network boundary to limit access to its public API. If you plan on exposing the Dapr API outside of that boundary, consider enabling token authentication for Dapr APIs. This ensures only requests with valid tokens are received by the Dapr sidecar. Enable token-based authentication for app-to-sidecar interactions:

Then, modify your application code to include the specified token in requests to the Dapr sidecar.

For some building blocks such as pub/sub, service invocation, and input bindings, Dapr communicates with an app over HTTP or gRPC. To enable the application to authenticate requests from the Dapr sidecar, configure Dapr to send an API token as a header (in HTTP requests) or metadata (in gRPC requests). To instruct Dapr to use the token in the secret when sending requests to the app, add an annotation to your Deployment template spec:

This ensures that only authenticated requests are processed, enhancing your application's overall security. Conductor detects the lack of such configurations and lists these as advisories too.

If the pod-level network boundary is sufficient for your application and these recommendations don’t apply, you can dismiss these (and any other) advisories so they are not listed (but still accessible if needed).

Locking Down Unused Dapr APIs

For Daprized applications deployed at the edge of your architecture, such as those called from an external application via an API gateway, you can further secure them by enabling only the Dapr APIs that are used. For example, to enable only the state API v1 over HTTP, set the following configuration:

This configuration disables all other APIs, including those still in alpha and management APIs such as the health API, metadata API, and shutdown API. This is an example best practice that is not in Conductor yet but is on the roadmap. Conductor checks for many more issues. Now, let's look at a few reliability recommendations.

Reliability Best Practices

Improving application reliability is crucial for minimizing downtime and should be configured during development too. Here are key techniques to enhance the resiliency of your applications interactions on cloud-native infrastructure like Kubernetes.

Enabling Application Health Checks

On Kubernetes, applications rely on health checks to ensure the reliability and availability of application processes. These include Startup Probes, Readiness Probes, and Liveness Probes. Dapr adds another layer of health checks to enhance interactions with other applications and infrastructure. Unlike Kubernetes health checks, which might restart the application or stop incoming traffic to an unhealthy container, Dapr app health checks take different actions when they fail:

• Unsubscribing from All Pub/Sub Subscriptions: This stops the application from receiving new messages.
• Stopping All Input Bindings: Halts all incoming data flows.
• Short-Circuiting All Service-Innovation Requests: Terminates requests within the Dapr sidecar, preventing them from reaching the application.

These measures are temporary, and once Dapr detects the application is responsive again, it resumes normal operations. This approach ensures smoother operations and reduces strain on the application during recovery periods, complementing Kubernetes health checks with a more adaptable view of system health. Here is an example of how to configure Dapr health checks in your application configuration as shown by Conductor Advisor:

By enabling this best practice, your applications will become more resilient in the event of transient instability.

Using Resiliency Policies

Resiliency is crucial in cloud-native applications and is handled differently by Kubernetes and Dapr. Kubernetes ensures continuous operation by maintaining the desired number of Pod replicas and rolling back failing deployments. Dapr focuses on the resilience of network interactions, especially during application or infrastructure failures and has a set of built-in resiliency policies for API interactions . These policies allow developers to remove the burden of developing resiliency into their application code and can instead standardize resiliency across their entire application portfolio via configuration files. Dapr resiliency policies include timeouts, retries, and circuit breakers. Here is an example of configuring a Dapr resiliency policy:

Here are a few tips to use resiliency policies more efficiently

• Scope Policies Appropriately: Ensure resiliency policies are scoped to the correct applications as described earlier in this article. In addition to scoping to specific apps, resiliency policies are also Applied to targets which could be apps, components and actors. Ensure your policies have both kinds of scopes.
• Evaluate Default Policies: Confirm that default resiliency policies provide the desired behavior for your application. Be aware that some client libraries also have retry behaviors that may conflict with the resiliency policy.
• Use Conductor for Insights: To identify issues that may go undetected, Conductor exposes resiliency policy metrics on graphs.

By implementing these resiliency best practices, you can enhance the robustness of your applications against network and service disruptions. Dapr's app health checks combined with resiliency policies provide a solid foundation to improving application reliability in cloud-native environments.

Continuous Best Practices Detection

Learning best practices for a technology and staying updated with its evolution requires continuous effort. For Dapr, some practices are documented, others are discussed in the Dapr Discord, answered on Stack Overflow, or in the heads of engineers who assist customers daily. To democratize this knowledge, the Conductor Advisor feature automates issue detection and provides easy-to-follow resolution steps, allowing developers to focus on implementing business solutions.

There are two versions of Conductor: Free and Enterprise:

• Conductor Free helps developers identify development-time best practices. It includes advisories that can enhance the security and reliability of your applications during the development phase (which is the focus of this blog post)
• Conductor Enterprise (with 30 day trial) includes both development and operational advisories. It offers comprehensive insights for operations teams into securing, optimizing, and maintaining your applications in production environments.

Applying best practices and securing an application is not a one-time activity either. Every time there is a change in your application configurations or a new Dapr release, there can be a new best practice. The Conductor Advisor feature continuously analyzes your application configurations and runtime behavior, offering recommendations. At Diagrid, we will keep enhancing our advisories as Dapr evolves. Sign up for Conductor today and perform a one-click scan of your Dapr applications to discover which best practices you might have missed.