Ruby, the value of software solutions is not only in raw throughput. There are many other valuable attributes including: speed and cost of development, speed and cost of enhancement, cost of deployment, cost of operational management, reliability and robustness, etc. All software architecture and design results from choices in the compromises between these different attributes. There are never perfect solutions, only successful compromises.
I’ve used Kafka with Protocol Buffers to achieve ultra-high throughput database replication systems. After a thorough evaluation of real-world performance, we determined that the superior compression was not a particular advantage, but that the tight coupling made necessary by Protobuf schemas added unneeded complexity and created versioning issues that slowed the deployment of enhancements and fixes. Replacing it with self-describing JSON messages and GZIP compression had negligible impact on performance.
The lightweight and implicit contract between requesters and responders, that is the hallmark of REST, leads us into effective component design patterns. Unfortunately, much of what I’ve seen labelled as REST APIs are in fact procedural APIs that use HTTP transport — and completely miss the benefits of REST.
What our actual experience has taught us is that the most significant improvement in microservices performance comes from implementing them as reactive systems with asynchronous actor-model components. The only time we have found gRPC to be especially useful was for some high-throughput trading applications where its impact upon design could be compartmentalized. Otherwise, we have found that its complexity and the design models it pushes us into are rarely cost-effective. The models we found most effective are described in the following post:
Thank you and keep up the good work!
