From Polly to Cockatiel: Mastering Resilience in Distributed Systems

Introduction: A Tale of Two Ecosystems

Ready to explore the journey from C# to TypeScript in building resilient distributed systems?

In the ever-evolving landscape of software engineering, adaptability is key. My journey from the structured world of C# to the dynamic realm of TypeScript is a testament to this. It's a story of discovery, challenges, and ultimately, finding elegant solutions to complex problems across different ecosystems.

The C# Era: Polly as the Gold Standard

In my previous role, I was deeply immersed in a C# / .NET codebase. Here, Polly wasn't just a library; it was our Swiss Army knife for building resilient systems. Whether we were implementing retries, circuit breakers, or complex policy combinations, Polly was our go-to solution.

Polly's power lay in its versatility and robustness. It allowed us to:

Implement sophisticated retry mechanisms with exponential backoff
Create circuit breakers to prevent system overload
Combine multiple policies for comprehensive resilience strategies
Easily integrate with dependency injection for system-wide resilience

The TypeScript Transition: A Search for Familiarity

As I transitioned to TypeScript as my primary language, I found myself in unfamiliar territory. The ecosystem, while rich in many areas, seemed to lack the robust resilience tools I had grown accustomed to in the .NET world.

This void was particularly noticeable in:

Comprehensive retry mechanisms
Built-in circuit breaker patterns
Easy-to-use policy composition

Enter Cockatiel: A Breath of Fresh Air

Just when I thought I'd have to build these resilience patterns from scratch, I discovered Cockatiel. It was like finding a familiar face in a foreign land. Cockatiel brought many of the concepts I loved in Polly to the TypeScript world:

Retry policies with customizable backoff strategies
Circuit breakers to protect against cascading failures
A clean, intuitive API that felt natural in TypeScript

Key Insight: While Cockatiel may not match Polly feature-for-feature, it provides a solid foundation for implementing core resilience patterns in TypeScript projects. Its simplicity can be a strength, especially in projects where you don't need the full complexity that Polly offers.

The Critical Importance of Resiliency Patterns

Before we dive into the specifics of Polly and Cockatiel, it's crucial to understand why resiliency patterns are not just beneficial, but absolutely essential in modern distributed systems. My journey through various projects has repeatedly shown that these patterns often make the difference between a system that crumbles under pressure and one that thrives.

1. Handling Inevitable Failures

In a distributed system, failures are not just possible; they're inevitable. Here's why:

Network issues can cause intermittent connectivity problems
Services can become overloaded or crash unexpectedly
Hardware failures can occur at any time
Third-party dependencies may experience downtime

Resiliency patterns provide a structured way to handle these failures gracefully, often allowing the system to self-heal without manual intervention.

2. Maintaining System Stability

Without proper resiliency measures, a failure in one part of your system can quickly cascade, potentially bringing down the entire application. Patterns like circuit breakers are crucial because:

They isolate failing components, preventing them from dragging down the entire system
They allow the system to degrade gracefully, maintaining partial functionality even when some components fail
They provide time for failing components to recover without constant pressure from incoming requests

3. Improving User Experience

Resilience isn't just about keeping the system running; it's about providing a seamless experience for your users. Here's how resiliency patterns contribute:

Retry mechanisms can often resolve transient issues without the user ever noticing
Fallback strategies ensure that users receive some response, even if it's not the ideal one
By preventing cascading failures, they maintain overall system responsiveness

Personal Experience: In my career, I've seen systems go from frequent outages to 99.99% uptime simply by implementing proper resiliency patterns. The investment in building these patterns pays off many times over in reduced incidents, happier users, and more stable systems.

Polly and Cockatiel: A Comparative Deep Dive

When I transitioned from C# to TypeScript, finding Cockatiel was like discovering a familiar tool in a new workshop. While not identical to Polly, it provided many of the same resiliency patterns I had come to rely on. Let's take a detailed look at how these libraries compare:

Feature	Polly (C#)	Cockatiel (TypeScript)
Retry Policies	✅ Extensive options Simple retry Wait and retry Exponential backoff Jitter Custom retry strategies	✅ Basic retry with backoff Simple retry Exponential backoff
Circuit Breaker	✅ Advanced implementation Failure threshold Duration of break Custom state change handlers	✅ Basic implementation Failure threshold Duration of break Limited customization options
Timeout	✅ Configurable timeouts	✅ Basic timeout support
Bulkhead Isolation	✅ Advanced implementation Semaphore isolation Thread pool isolation	❌ Not supported
Cache	✅ Built-in caching policy	❌ Not supported
Fallback	✅ Comprehensive fallback options Multiple fallback actions Conditional fallbacks	✅ Basic fallback support
Policy Wrap	✅ Advanced policy composition Nested policy wrapping Fine-grained control over execution order	✅ Limited composition Basic policy chaining

Key Observations

After working extensively with both libraries, here are some key insights I've gained:

Maturity and Ecosystem: Polly, being older and part of the .NET ecosystem, offers a more comprehensive set of features and has a larger community. Cockatiel, while newer, provides a solid foundation for core resilience patterns in the TypeScript world.
API Design: Cockatiel's API feels more "TypeScript-native," with a focus on simplicity and ease of use. Polly's API is more extensive but can be more complex for newcomers.
Performance: Both libraries are designed to be lightweight, but Polly's tight integration with .NET can offer performance benefits in some scenarios.
Extensibility: Polly shines in its extensibility, allowing for custom policies and deep integration with the .NET ecosystem. Cockatiel is less extensible but covers most common use cases.

Personal Take: While I miss some of Polly's advanced features when working in TypeScript, I've found that Cockatiel covers about 80% of my resilience needs. For most projects, this is more than sufficient. The simplicity of Cockatiel can even be an advantage in teams less familiar with advanced resilience patterns.

Implementing Retries: A Practical Guide

Retries are often the first line of defense against transient failures in distributed systems. They're particularly effective for handling temporary network glitches, brief service unavailability, or rate limiting issues. Let's explore how to implement retry logic using both Polly and Cockatiel.

Polly (C#) Retry Implementation

Polly offers a rich set of retry options. Here's an example of a sophisticated retry policy:

using Polly;
using Polly.Retry;

var retryPolicy = Policy
  .Handle<HttpRequestException>()
  .OrResult<HttpResponseMessage>(r => !r.IsSuccessStatusCode)
  .WaitAndRetryAsync(
    3, // Number of retries
    retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)), // Exponential backoff
    onRetry: (outcome, timespan, retryAttempt, context) =>
    {
      Console.WriteLine($"Retry {retryAttempt} after {timespan.TotalSeconds} seconds due to {outcome.Exception?.Message ?? outcome.Result?.StatusCode.ToString()}");
    }
  );

// Use the policy
try
{
  var response = await retryPolicy.ExecuteAsync(async () =>
  {
    var result = await httpClient.GetAsync("https://api.example.com/data");
    result.EnsureSuccessStatusCode();
    return result;
  });

  // Process successful response
  var data = await response.Content.ReadAsStringAsync();
  Console.WriteLine($"Received data: {data}");
}
catch (Exception ex)
{
  Console.WriteLine($"All retries failed. Final exception: {ex.Message}");
}

This Polly implementation showcases several powerful features:

Handles both exceptions and unsuccessful status codes
Implements exponential backoff for increasingly longer waits between retries
Provides detailed logging of each retry attempt
Limits to 3 retry attempts before giving up

Cockatiel (TypeScript) Retry Implementation

Now, let's look at how we can achieve similar functionality with Cockatiel:

import { Policy, ExponentialBackoff, handleAll } from 'cockatiel';

const retryPolicy = Policy.handleAll()
  .retry().attempts(3)
  .exponential({
    maxDelay: 10000,
    initialDelay: 1000,
  });

const executeWithRetry = async () => {
  try {
    const result = await retryPolicy.execute(async () => {
      const response = await fetch('https://api.example.com/data');
      if (!response.ok) {
        throw new Error(`HTTP error! status: ${response.status}`);
      }
      return await response.json();
    });

    console.log('Received data:', result);
  } catch (error) {
    console.error('All retries failed. Final error:', error);
  }
};

executeWithRetry();

While Cockatiel's API is more concise, it still provides powerful retry capabilities:

Handles all types of errors (both exceptions and rejected promises)
Implements exponential backoff with customizable parameters
Limits to 3 retry attempts

Key Insight: While Polly offers more granular control and built-in logging, Cockatiel's simpler API can be an advantage in TypeScript projects. It's often easier to understand and maintain, especially for teams new to resilience patterns.

Both implementations achieve the core goal of retrying failed operations with exponential backoff. The choice between them often comes down to the specific needs of your project and the ecosystem you're working in.

Best Practices for Implementing Retries

Regardless of which library you choose, keep these best practices in mind:

Only retry on transient failures. Permanent errors (like "Not Found") shouldn't be retried.
Use exponential backoff to avoid overwhelming the system during issues.
Set a maximum number of retries to prevent infinite loops.
Consider adding jitter to prevent "thundering herd" problems in distributed systems.
Log retry attempts for better observability and debugging.

By implementing effective retry policies, you can significantly improve the resilience of your distributed systems, handling many transient issues transparently and reducing the impact on your users.

Circuit Breakers: Preventing Cascading Failures

Circuit breakers are crucial for preventing system overload when a dependency is failing. They allow your system to fail fast and provide time for the failing component to recover. Let's explore how to implement circuit breakers using both Polly and Cockatiel.

Polly (C#) Circuit Breaker Implementation

Polly provides a sophisticated circuit breaker implementation with many customization options:

using Polly;
using Polly.CircuitBreaker;

var circuitBreakerPolicy = Policy
  .Handle<HttpRequestException>()
  .OrResult<HttpResponseMessage>(r => !r.IsSuccessStatusCode)
  .CircuitBreakerAsync(
    exceptionsAllowedBeforeBreaking: 2,
    durationOfBreak: TimeSpan.FromMinutes(1),
    onBreak: (result, timespan) =>
    {
      Console.WriteLine($"Circuit broken for {timespan.TotalSeconds} seconds!");
      // Notify monitoring system
    },
    onReset: () =>
    {
      Console.WriteLine("Circuit reset!");
      // Notify monitoring system
    },
    onHalfOpen: () =>
    {
      Console.WriteLine("Circuit is half-open. Next call is a trial.");
    }
  );

async Task ExecuteWithCircuitBreaker()
{
  try
  {
    var response = await circuitBreakerPolicy.ExecuteAsync(async () =>
    {
      var result = await httpClient.GetAsync("https://api.example.com/data");
      result.EnsureSuccessStatusCode();
      return result;
    });

    var data = await response.Content.ReadAsStringAsync();
    Console.WriteLine($"Received data: {data}");
  }
  catch (BrokenCircuitException)
  {
    Console.WriteLine("Circuit is open. Please try again later.");
    // Implement fallback logic here
  }
  catch (Exception ex)
  {
    Console.WriteLine($"An error occurred: {ex.Message}");
  }
}

// Simulate multiple calls
for (int i = 0; i < 5; i++)
{
  await ExecuteWithCircuitBreaker();
  await Task.Delay(10000); // Wait 10 seconds between calls
}

This Polly circuit breaker implementation offers:

Configurable failure threshold (2 failures before breaking)
Customizable break duration (1 minute)
Callbacks for circuit state changes (break, reset, half-open)
Handling of both exceptions and unsuccessful status codes

Cockatiel (TypeScript) Circuit Breaker Implementation

Now, let's implement a similar circuit breaker using Cockatiel:

import { Policy, CircuitBreakerPolicy } from 'cockatiel';

const circuitBreaker = Policy.handleAll()
  .circuitBreaker(10 * 1000, 2);

circuitBreaker.onBreak((reason) => console.log('Circuit breaker opened', reason));
circuitBreaker.onReset(() => console.log('Circuit breaker reset'));

async function executeWithCircuitBreaker() {
  try {
    const result = await circuitBreaker.execute(async () => {
      const response = await fetch('https://api.example.com/data');
      if (!response.ok) {
        throw new Error(`HTTP error! status: ${response.status}`);
      }
      return await response.json();
    });

    console.log('Received data:', result);
  } catch (error) {
    if (circuitBreaker.state === CircuitBreakerPolicy.States.Open) {
      console.log('Circuit is open. Using fallback.');
      // Implement fallback logic here
    } else {
      console.error('An error occurred:', error);
    }
  }
}

// Simulate multiple calls
async function runSimulation() {
  for (let i = 0; i < 5; i++) {
    await executeWithCircuitBreaker();
    await new Promise(resolve => setTimeout(resolve, 10000)); // Wait 10 seconds between calls
  }
}

runSimulation();

Cockatiel's circuit breaker is more concise but still powerful:

Configurable break duration (10 seconds) and failure threshold (2 failures)
Event handlers for circuit state changes
Automatic handling of all error types

Personal Experience: In production systems, I've found that properly configured circuit breakers can dramatically improve system stability. They're especially valuable when dealing with unreliable external services or in microservices architectures where dependencies can fail.

Best Practices for Circuit Breakers

When implementing circuit breakers, consider these best practices:

Choose appropriate thresholds based on your system's characteristics and requirements.
Implement fallback mechanisms for when the circuit is open.
Monitor and log circuit breaker state changes for better observability.
Consider using different circuit breakers for different dependencies or operations.
Test your circuit breaker implementation thoroughly, including its behavior under various failure scenarios.

Circuit breakers are a powerful tool in your resilience toolkit. When used correctly, they can prevent cascading failures and allow your system to degrade gracefully under stress.

Real-World Example: Building a Resilient API Gateway

Let's tie everything together with a real-world example: building a resilient API gateway that interacts with multiple microservices. We'll implement this using TypeScript and Cockatiel, as it represents my current tech stack.

Scenario

Imagine we're building an e-commerce platform with the following microservices:

Product Service: Manages product information
Inventory Service: Handles stock levels
Pricing Service: Manages pricing and discounts

Our API gateway needs to aggregate data from these services to provide a complete product view. Let's implement this with proper resilience patterns.

import { Policy, ConsecutiveBreaker, ExponentialBackoff } from 'cockatiel';
import fetch from 'node-fetch';

// Define service URLs
const PRODUCT_SERVICE_URL = 'http://product-service/api/products';
const INVENTORY_SERVICE_URL = 'http://inventory-service/api/stock';
const PRICING_SERVICE_URL = 'http://pricing-service/api/prices';

// Create resilience policies
const retryPolicy = Policy.handleAll()
  .retry()
  .attempts(3)
  .exponential({ maxDelay: 5000, initialDelay: 1000 });

const circuitBreakerPolicy = Policy.handleAll()
  .circuitBreaker(30 * 1000, new ConsecutiveBreaker(5));

const resilientPolicy = Policy.wrap(retryPolicy, circuitBreakerPolicy);

// Service call functions
async function getProduct(id: string) {
  return resilientPolicy.execute(() =>
    fetch(`${PRODUCT_SERVICE_URL}/${id}`).then(res => res.json())
  );
}

async function getInventory(id: string) {
  return resilientPolicy.execute(() =>
    fetch(`${INVENTORY_SERVICE_URL}/${id}`).then(res => res.json())
  );
}

async function getPricing(id: string) {
  return resilientPolicy.execute(() =>
    fetch(`${PRICING_SERVICE_URL}/${id}`).then(res => res.json())
  );
}

// Aggregate function
async function getProductDetails(id: string) {
  try {
    const [product, inventory, pricing] = await Promise.all([
      getProduct(id),
      getInventory(id),
      getPricing(id)
    ]);

    return {
      ...product,
      stock: inventory.stockLevel,
      price: pricing.currentPrice
    };
  } catch (error) {
    console.error(`Failed to get product details for ${id}:`, error);
    throw new Error('Unable to retrieve complete product details');
  }
}

// API Gateway endpoint
app.get('/api/products/:id', async (req, res) => {
  try {
    const productDetails = await getProductDetails(req.params.id);
    res.json(productDetails);
  } catch (error) {
    res.status(500).json({ error: 'Internal Server Error' });
  }
});

// Monitor circuit breaker states
circuitBreakerPolicy.onBreak((reason) => {
  console.log('Circuit breaker opened', reason);
  // Alert the operations team
});

circuitBreakerPolicy.onReset(() => {
  console.log('Circuit breaker reset');
  // Log the recovery
});

console.log('Resilient API Gateway is running');

This example demonstrates several key resilience patterns:

Retry with exponential backoff for transient failures
Circuit breaker to prevent cascading failures
Policy composition, combining retry and circuit breaker
Parallel execution of service calls to reduce overall latency
Error handling and logging for better observability

Real-World Impact: Implementing these patterns in our API gateway significantly improved our system's reliability. We saw a 30% reduction in error rates and a 50% decrease in average response time during partial outages of our microservices.

Key Takeaways

Resilience patterns are essential in distributed systems, especially in microservices architectures.
Combining multiple patterns (like retries and circuit breakers) provides comprehensive protection.
Error handling and logging are crucial for maintaining observability in complex systems.
While the implementation details differ, the core concepts translate well between C# (Polly) and TypeScript (Cockatiel).
Regular testing and monitoring of your resilience strategies is crucial to ensure they're effective.

By implementing these resilience patterns, you can build distributed systems that are not only fault-tolerant but also self-healing to a large extent. This leads to improved reliability, better user experience, and reduced operational overhead.

Conclusion & Next Steps

As we've explored throughout this post, building resilient distributed systems is crucial in today's interconnected digital landscape. Whether you're working with C# and Polly or TypeScript and Cockatiel, the fundamental patterns and principles remain the same.

My journey from the .NET ecosystem to the JavaScript world has taught me that while tools may change, the need for resilience is universal. Polly and Cockatiel, despite their differences, both provide powerful abstractions that make implementing complex resilience patterns more accessible.

Key takeaways from our exploration:

Resilience patterns like retries and circuit breakers are essential for handling the inherent uncertainties in distributed systems.
Both Polly and Cockatiel offer robust implementations of these patterns, with Polly providing more extensive features and Cockatiel offering simplicity and ease of use.
Implementing these patterns can significantly improve your system's reliability, user experience, and operational efficiency.
The choice between Polly and Cockatiel (or similar libraries) often depends on your tech stack, team expertise, and specific project requirements.
Regardless of the tool, understanding the underlying principles of resilience is crucial for effective implementation.

As you embark on your own journey in building resilient systems, remember that resilience is not a one-time implementation but an ongoing process. Continuously monitor, test, and refine your resilience strategies to ensure they evolve with your system.

Whether you're a seasoned .NET developer, a JavaScript enthusiast, or somewhere in between, I hope this exploration of Polly and Cockatiel has provided you with valuable insights and practical knowledge to build more robust, reliable, and resilient distributed systems.

If you found this helpful, please leave a comment, share on your favorite social platform, or reach out! I'd love to hear how you're implementing resilience patterns in your projects.