Welcome to the third part of the "Mastering Dapr Inner Development Loop" series. If you missed the first or second parts, be sure to check them out here (Part 1) and here (Part 2).

One of the key principles of 12-factor apps and cloud-native development is dev/prod parity, which emphasizes keeping development, staging, and production environments as similar as possible. While this doesn't mean you need to develop directly on Kubernetes, it does mean discovering discrepancies, misconfigurations, and security gaps as early as possible—ideally on the development machine where iterating and debugging issues is fastest. This principle is also central to the shift-left movement.

When running your application locally as a standalone process, Dapr is configured with CLI flags or the multi-app run configuration file. As a next step, you want to ensure that the applications can be built into containers, deployed on Kubernetes with the right configurations, complies with best practices and functions as intended on Kubernetes too. To run your application on Kubernetes, you need to create Deployment files for your application Pods with the Dapr annotations (instead of CLI flags), update Dapr CRDs for Kubernetes (which might refer Kubernetes Secrets instead of hardcoded passwords, and other Kubernetes resources such as CondifMaps, Secrets, URLs, and more). In this phase, by ensuring your applications are containerized and have the right Kubernetes configurations, you bring your development environment closer to smooth deployment and operation on production.

Installing Dapr and Conductor on Kubernetes

The first thing you would need is a local Kubernetes distribution such as Minikube or Kind, or any Kubernetes cluster running on the cloud. The local options are very mature at this point and can be used with reasonable resource consumption. Once you get a hold of the Kubernetes cluster and authenticate with kubectl, deploying Dapr is straightforward. You could use Dapr CLI for that purpose, or do it with Diagrid Conductor Free. In this post we will use Conductor Free as that is the tool that will help us with next steps too. Follow this quickstart guide for the fastest way of running Kubernetes locally and installing Conductor Free and Dapr on it. The guide also provides steps for installing a single instance of Redis, which can be used as a backing infrastructure service for the sample applications.

‍

These are one-time initial setup steps that do not need to be repeated later. Completing these steps will ensure you have Kubernetes with a Dapr Control plane installed and Conductor configured to monitor and offer you insights and best practices.

Continuously Applying Changes to Kubernetes

The next phase involves a series of steps executed in rapid succession. Typically, this includes building a container image, deploying it into Kubernetes with the necessary configurations, interacting with the application, observing the results in the logs, and making necessary alterations to the code or configurations. These steps should be performed multiple times until the desired configurations are found and the application behaves as expected on Kubernetes. For this purpose, we will use Skaffold, which effectively addresses all of these needs and plays nicely with Dapr too. Skaffold is a standalone binary that can be installed with a single command. For example, to install Skaffold on macOS, you can use:

There are other tools that perform similar tasks, such as Garden, Tilt, Draft, Ko, Okteto, DevSpace, Forge,  Velocity,  and you could use these, or even perform the steps in less-automated fashion with pure kubectl. For this example, we will use Skaffold to streamline the development workflow and quickly deploy applications to a local Kubernetes environment. With Skaffold installed, we can proceed into building container images.

Building Container Images

Dapr imposes no requirements on how container images are built, nor does it offer any aids in building them. This step can be time-consuming in local development, but fortunately, many tools have optimized this process. Skaffold supports several options for building images, including Dockerfile, Jib, Ko, and Buildpacks. In this post, we will use Dockerfile as we already have Docker installed on the local machine to run Kubernetes, and with Dockerfiles we can make sure the containers are built the same way locally as they will be in the production build pipeline. Here is the Dockerfile for the Publisher application.

FROM openjdk:11
EXPOSE 5001
ENV PORT 5001
VOLUME /tmp
COPY target/*.jar app.jar
ENTRYPOINT ["java","-jar","/app.jar"]

Publisher application Dockerfile

Once the container is built and available in a container registry, it can be redeployed to the container runtime for testing and verification. The advantage of using a local Kubernetes with Skaffold is that you don’t need to wait for your container image to be uploaded to a remote repository. Skaffold can build the container image and load it into Minikube or Kind w/o an internet roundtrip.

Applying Changes into Kubernetes

The simplest way to deploy Daprized applications into Kubernetes is through the Dapr CLI using a multi-app run feature and pointing Kuberntes as the deployment target. It will deploy your containerized applications into Kubernetes, make sure Dapr sidecars are injected, and stream container logs too. However, there are a few limitations to this approach:

• It doesn’t build the container image, so you must remember to do this after every application code change.
• It doesn’t push or load the image into the local Kubernetes registry, requiring you to push the image to an internet-hosted container registry, every time to pull from.
• It doesn’t watch for code and configuration changes and resync them into Kuberntes, necessitating manual redeployment after every change.
• It doesn’t automatically perform port-forwarding, leaving it to developers to do.
• The multi-app run file gets overpopulated with standalone and Kubernetes specific options that then map Dapr options from the  multi-app run template into Kubernetes annotations which can get messy.

These are some reasons why we use Skaffold in this post, as it is designed for rapid development on Kubernetes and it addresses all of the limitations. One downside of Skaffold is that it needs its own configuration file to perform these steps. As this is not a full Skaffold tutorial, so here’s an extract from the final shape of the Skaffold file that runs the Publisher application:

apiVersion: skaffold/v4beta11
kind: Config
metadata:
  name: publisher

build:
  tagPolicy:
    inputDigest: { }
  artifacts:
    - image: bibryam/publisher
      docker:
        dockerfile: Dockerfile
  local: {}

manifests:
  rawYaml:
    - ../common/kubernetes/*
    - deploy/*

deploy:
  kubectl: {}
  logs:
    prefix: auto
    jsonParse:
      fields: ["time", "@timestamp", "level", "msg", "message"]

portForward:
  - resourceType: deployment
    resourceName: publisher
    port: 5001
    localPort: 5001
  - resourceType: deployment
    resourceName: publisher
    port: 9090
    localPort: 5011
  - resourceType: deployment
    resourceName: publisher
    port: 3500
    localPort: 3501

Skaffold config file for Publisher application

With this file in place, run the following command to deploy both applications into Kubernetes:

skaffold dev

This single command will continuously perform the following actions:

• Build the container image with the defined method (Dockerfile in our case).
• Load the image into Minikube without pushing to a public registry such as Docker Hub.
• Deploy the application with the container image into Kubernetes.
• Apply any additional files such as component definitions or other CRDs.
• Start port-forwarding to interact with the application or sidecar.
• Start streaming the logs from all containers.
• Watch for changes in any files that are synced or included in the container image.

For example, if you make any changes to the Java code that uses Dapr SDKs, save the changes and perform Maven build, this action will trigger container image build and redeployment, so that changes appear on Kubernetes within seconds. If you update the pubsub component file, it will be reapplied. If you update the Dapr annotations in the application Deployment file, it will be re-applied to Kubernetes. After any change, the application Pod will be restarted, port-forwarding will be re-established and log streaming resumed automatically. This sequence of actions will allow you to iterate rapidly on altering any Dapr configuration specific for Kubernetes and validate the result.

Validating Dapr Resources on Kubernetes

While Skaffold will help you continuously deploy your containerized application into Kubernetes, it will not show you what is the resulting Daprized application in Kubernetes. That is the job of Conductor. Conductor will continuously watch for Daprized applications in the Kubernetes cluster and visualize these to you. By logging into the Conductor console, you can discover everything about the Dapr control plane and the Publisher and Consumer applications:

Conductor will show you the overall health of the Dapr control plane and alert you if it degrades as that can impact the applications too. With Conductor you can see detailed information about Dapr Components, Resiliency policies, Actors, Subscriptions (not only declarative, but also programmatic subscriptions as is the case in the Subscriber application), and more. A key differentiator of Conductor from other generic graphical interfaces for Kubernetes such as Lens, Monokle, Kubernetes Dashboard, K9s, etc., is that Conductor understands Dapr. Conductor can validate component connections through initialization checks and show you the semantic connection between a Dapr resource and the applications it applies. All of that gives you a holistic view of the Dapr runtime and not only static YAML views. By combining Skaffold and Conductor, you can alter local Dapr YAML files, and observe the changes in Kubernetes from Conductor.

Invoking Applications on Kubernetes

Skaffold has built-in support for port-forwarding from exposed Kubernetes resources on your cluster to your local machine. For demonstration purposes, some of the applications and Dapr sidecar ports are explicitly declared here for easier interaction and to prevent the default random port allocation.

portForward:
  - resourceType: deployment
    resourceName: publisher
    port: 5001
    localPort: 5001
  - resourceType: deployment
    resourceName: publisher
    port: 9090
    localPort: 5011
  - resourceType: deployment
    resourceName: publisher
    port: 3500
    localPort: 3501

Skaffold configuration for Publisher application port-forwarding

With port-forwarding in place, you can interact with the application or the sidecar running in Kubernetes as if it were running locally, using the same curl commands we used earlier. While this is effective for basic functionality testing, it is not ideal for performance and resiliency validation. An option here would be to use the Apache Bench tool to make requests. This command-line tool can test the performance of an HTTP request. For example, to send 100 requests (-n 100) from a single user (-c 1) to the service:

ab -n 100 -c 100 -p order.json -T "application/json" http://localhost:5001/pubsub/orders 

Although the goal here is not to perform a full performance test, this tool can quickly test the concurrency configuration of your application, observe how competing consumers behave, trigger Dapr resiliency policies, help identify code bottlenecks, and more. And again, observe the result of these interactions from Conductor’s metrics graphs.

Conductor provides detailed metrics breakdowns, application interaction graphs, component metrics, resilience metrics, actor metrics, and more. To see this in action, you can scale Redis instance to 0 replicas and re-run the 100 requests with Apache Bench. Shortly after, you will see in Conductor Apps Graph the Publisher application failing to publish messages. Further breakdown of these metrics can be seen in the application and component metrics graphs.

Streaming Logs from Pods

While interacting with the application, we want to check the logs and metrics and validate that the set configurations have the desired effect. Skaffold and Conductor are a great duo for this purpose too. Skaffold automates observing logs by streaming logs from all containers in real time, making it easier to monitor and debug your applications without manually locating and accessing individual Pod. To tune this feature for Dapr, you can configure Skaffold:

deploy:
  kubectl: {}
  logs:
    prefix: auto
    jsonParse:
      fields: ["time", "@timestamp", "level", "msg", "message"]

Skaffold JSON log configuration for Dapr and Spring Boot log formats

This configuration will parse JSON formatted logs that come from Dapr sidecar and Spring Boot application, and present it as an easy to read text message with different colors for each Pod and leaving out any other field. While Skaffold streams real time logs from the application Pods, Conductor also aggregates important sidecar logs in the background too.

Conductor will raise alerts from sidecar logs (not application logs) at warning, error, and fatal levels and keep these for 12 hours. During this time window, if you need to search for any issues that you may have missed from the terminal—which can happen during longer runs—they are stored and can be easily matched to the source application.

Following Dapr Best Practices

The primary reason for deploying to a local Kubernetes cluster is to ensure your application and Dapr are properly integrated into Kubernetes. Depending on the Dapr features used, multiple checks should be performed before proceeding to the outer developer loop:

• The application sidecar has the right configuration file attached from a ConfigMap.
• Template metadata is propagated as expected.
• Application and sidecar health checks are working as expected.
• Application and Sidecar shutdown behavior is functioning as expected.
• File volumes are mounted for the Dapr sidecar when needed.
• Kubernetes secrets are accessed from Dapr as expected.
• Application and API tokens are passed as expected
• The correct log level and log format are enabled.

Once you have validated the application’s functionality, the next step before sharing your application with the rest of the company is to apply Dapr best practices. Deployment tools and Kubernetes health checks can detect obvious syntactic mistakes, but they won’t help you discover Dapr-specific best practices and optimizations, as such tools don't understand Dapr. Conductor, on the other hand, was created by the creators and maintainers of Dapr and includes all that knowledge in its Advisor feature. Conductor Advisor continuously scans your application configuration files and observes signals to offer best practice suggestions. Applying these best practices leads to creating more resilient and secure applications aligned with zero-trust principles, making them reliable and performant in production environments.

Even for this minimalistic sample project, Conductor Advisor offers multiple suggestions:

• The pub/sub component has no scopes defined, which means it will be accessible to all future apps. This can be addressed by adding the following snippet to the pubsub.yaml

• The pub/sub component has no topic access defined, which means all topics are accessible for all operations. We can address this by allowing Publisher application publish access to orders topic, and Subscriber application, subscription access by adding the following snippet:

• Applications have no access control policy specified, meaning any exploited application can access all other applications via the service invocation API. Fixing this one requires adding a new Dapr Configuration CRD.
• Dapr application health checks are not enabled. These help Dapr detect a failure in the app’s health and to stop accepting new work on behalf of the application. Fixing it requires implementing health endpoints in the applications and  updating both Publisher and Subscriber applications’ deployment.yaml files. Luckily Spring Boot already offers such endpoints and all we have to is add the following annotations:

• The application and sidecar interactions can be further secured with a token, and so forth.

These are only a few examples of how Conductor Free helps with creating more secure and reliable applications. Conductor Enterprise on the other hand, performs around 50 checks, ensuring the applications are secure, reliable, and optimized for cost saving in production. If any of these settings are intentional and the advisor doesn’t apply, it can be dismissed. Applying these advisors on the local machine with Skaffold is fast and re-running the advisor is instant. By fixing these issues, our application becomes ready for the outer developer loop.

Towards the Outer Developer Loop

Building and testing distributed applications is hard. Dapr helps during development and testing, while Skaffold and Conductor help with validating Dapr best practices before deploying to other environments. By deploying into Kubernetes, you can use Conductor to continuously monitor your Kubernetes cluster, providing detailed insights into Dapr components, resiliency policies, and application interactions. It offers best practices and recommendations to ensure your applications are secure and performant. This step is vital for discovering and fixing any configuration or dependency issues early, making it easier to manage applications in other environments.

Once validated locally, you can confidently push the code to source control and deploy it to shared environments. In these environments, Conductor Enterprise can be used to fine-tune the applications, ensuring they are secure, resilient, and optimized for production use. This comprehensive approach ensures your applications are robust and reliable across all deployment stages.

‍