Hugo Smitter from FICO covers how they harnessed Dapr for seamless, efficient, and scalable Pulsar integration.
Explore:
• An overview of the FICO® Platform and strategic business objectives.
• Key contributions to the Dapr community, enhancing its support for Apache Pulsar.
• How Dapr abstracts the Pulsar client API, making it easier to build, deploy, and manage event-driven applications.
• FICO’s experience, including benchmarks, performance metrics, and practical migration tips.
Unlock the potential of Apache Pulsar with Dapr and transform your event management and processing capabilities. This session provides practical knowledge, real-world examples, and technical details on building high scale event driven services.
Would you prefer to read than watch? Here is a write up of his talk:
Beyond DevOps: The Rise of Platform Engineering as a Strategic Discipline
FICO has embarked on a transformative journey in platform engineering—a strategic shift from its roots in credit scoring to a platform-first approach. This evolution is underpinned by the adoption of StreamNative Pulsar, enabling a decoupled architecture that enhances flexibility and scalability. By integrating Apache Pulsar with Dapr, FICO has created a sophisticated ecosystem that supports modularity, secure communication, and operational efficiency. Here’s a breakdown of how FICO has achieved this transformation.
From Credit Scoring to a Platform Ecosystem
FICO’s origins in credit scoring laid the foundation for its transition into a broader ecosystem of analytical tools. Recognizing the need for agility in its services, FICO adopted a platform-first mindset to deliver hyper-personalized insights across the customer lifecycle. This required breaking monolithic processes into modular components, an endeavor fraught with challenges in maintaining robust communication among these parts. The company’s Northstar platform exemplifies this shift, combining bespoke products to optimize customer decision-making, such as loan portfolio strategies.
Building the Foundation: Decoupled Architecture and Event Services
To support this modularity, FICO developed an event service guided by architectural principles like API-first design, separation of concerns, and declarative CI/CD models. The selection of an event broker—a core element for enabling inter-component communication—was critical. FICO’s stringent criteria included:
- Security: Features like pluggable authentication and encrypted payloads.
- Scalability: Multi-cloud support and low-latency, high-throughput messaging.
- Flexibility: Support for advanced features such as schema management and multi-tenancy.
Ultimately, Apache Pulsar emerged as the preferred choice over Kafka due to its operational efficiency, community support, and superior performance benchmarks.
The Role of Dapr in Maintaining Technology Agnosticism
To ensure that the platform remains agnostic to specific technologies, FICO integrated Dapr as an abstraction layer. Dapr’s distributed application runtime simplifies the construction of event-driven distributed systems by offering lightweight building blocks compatible with Kubernetes. These include APIs for gRPC and HTTP REST, enabling seamless workflow management, communication, and state management.
However, integrating Dapr with Pulsar wasn’t without challenges. Issues such as schema enforcement, OIDC authentication, and diverse subscription types required extensive collaboration with the Dapr community and support from Diagrid. In just six months, the team overcame these obstacles while maintaining API consistency, reinforcing the robustness of the combined architecture.
Overcoming Technical Hurdles
Resource consumption and system performance are critical considerations for high-scale messaging. The speaker clarified that Dapr’s memory usage remains minimal, even under heavy loads, thanks to its design as an application-level abstraction rather than a service mesh proxy. Despite early limitations, Dapr’s extensibility and transparency have enabled FICO to tune the system effectively for their needs.
A persistent challenge has been addressing message duplication in Apache Pulsar, which impacts performance and reliability. Collaborative efforts within the Pulsar and Dapr communities aim to devise custom solutions for these issues, ensuring continuous improvement and system optimization.
Strategic Lessons from FICO’s Journey
FICO’s journey highlights several strategic lessons for platform engineers and application architects:
- Architectural Agility: A decoupled architecture provides the foundation for scalability and flexibility, enabling organizations to pivot quickly in response to evolving requirements.
- Technology Agnosticism: Leveraging abstraction layers like Dapr ensures that organizations remain adaptable to future technology shifts without overhauling their underlying architecture.
- Community Collaboration: Active engagement with open-source communities accelerates the resolution of technical challenges and fosters innovation.
The Future of Platform Engineering
FICO’s innovative use of Apache Pulsar and Dapr underscores the strategic role of platform engineering in achieving operational excellence and delivering superior customer experiences. By embracing decoupled architectures and fostering collaboration within technology ecosystems, organizations can navigate the complexities of modern application development with confidence.
FICO’s approach serves as an inspiring blueprint for enterprises aiming to harness the power of platform engineering to drive digital transformation.
