Work · 2018–2025

From polling to pub/sub

Moving a global set of services from legacy polling to event-driven messaging on AWS SQS and SNS, and onto Kubernetes.

Work case study · 2 min read · event #000033

  • AWS SQS
  • AWS SNS
  • RabbitMQ
  • Kubernetes
  • PHP
  • Laravel
  • Redis
  • Varnish
  • Infor M3
  • Akeneo
  • Shopify

The problem

Colart’s systems had grown by adding polling jobs. One service asked another for new data on a schedule, over and over, whether or not anything had changed. That is wasteful, it makes the data only as fresh as the polling interval, and each new integration added another schedule to keep an eye on. The business also ran across several countries, with an ERP, a product information system, an online shop and warehouses that all needed the same facts.

What I did

I designed and grew a global event-driven service-oriented architecture. Services stopped asking each other for changes and started publishing events instead: something happens, a message goes out, and the services that care subscribe. The messaging layer moved to AWS SQS and SNS, using publish/subscribe patterns.

We had started on RabbitMQ. As volumes grew I moved us to AWS-native messaging, which reduced the operational overhead of running a broker ourselves.

Around the messaging I wrote the middleware and integration libraries that synchronised data between our internal microservices, Infor M3 (the ERP), Akeneo (the PIM) and Shopify. The same libraries supported automated end-to-end order management for warehouses in different countries.

Related infrastructure work

Working with DevOps, I led the move from manually provisioned EC2 instances to a highly available Kubernetes cluster on AWS. To take pressure off the legacy SQL database, I introduced a tiered caching strategy: Varnish for page caching, Redis for fast queries, and NoSQL stores for high-throughput reads that did not need to be authoritative.

What I took from it

Moving from polling to events changes how you think about failure. A poller that misses a run just catches up next time; a subscriber has to deal with duplicates, ordering and retries on purpose. Choosing managed messaging meant we spent that effort on our business logic rather than on keeping a broker alive.

What I’d do next

If I were doing it again I would write down the event contracts earlier, as a shared schema registry with versioning, so that teams adding a subscriber do not have to read another service’s code to find out what an event contains.

← Back to work