Todd, I will try to express my opinions in a more concrete manner.
First, I agree almost entirely with Mr. Koutanov. Your remark, "It would appear that Koutanov is a seasoned expert in distributed systems, and a big fan of Kafka, a fantastic and powerful tool. When it comes to databases though, he doesn't seem to have an intuition for the best way to use a database to replace Kafka.", tells me that you do not have comparable real-world experience. He is not using intuition, he is applying knowledge and software engineering skills and has explained why replacing Kafka with a DBMS is a very sub-optimal solution.
To give you a little more perspective on where I'm coming from, I have been a professional software developer since 1972. I've made a very good living by keeping up with technology trends while avoiding falling for hype at the expense of engineering (founded and sold my own company). Though you have read the PostgreSQL manual, you do not seem to understand Postgres internals or architecture. I was a senior architect for database replication products at a company that distributes Postgres and sells products that add functionality and tools to open source PostgreSQL. I'm a big fan of Postgres, but like any good software system, it embodies the compromises and choices that best suit it to the usage patterns of the majority of its adopters.
In general, relational database management systems are optimized to reliably implement relational algebra, maintain referential and data integrity, and to execute SQL queries. To do that, RDBMS are limited in their ability to optimize data writes. For example, Postgres has a superb query engine which is the most-used capability of any RDBMS. To do that it must organize and write data, not for optimum write speed, but ultimately for optimum query performance.
Kafka is architected and implemented for an entirely different purpose. It publishes messages (records) to multiple subscribers. In practical terms a Kafka message is a byte array and the only "key" that concerns Kafka it the topic to which a message is published. Kafka’s FIFO queues are read sequentially, not queried by content. That preserves event order and enables blazingly fast reads.
Kafka message queues are not permanent data stores. Each message has a finite lifetime and is then purged, regardless if it has been read or processed. Subscribers are totally independent of each other and read messages by position (offset) from a topic queue. Each subscriber needs to keep track of its own last processed message position. A subscriber that needs a permanent record of messages must write them to a persistent store.
Kafka is amazingly fast at both writing and reading message queues. Its speed can be orders of magnitude faster than an RDBMS. It is, however, very sensitive to message size and network latency. Having used Kafka to move CDC messages within a set of replicated databases, I have weeks of running benchmarks comparing the relative data handling speeds of Kafka and Postgres from which to form an opinion.
I am having difficulty understanding your use case. You seem unconcerned with requirements for distributed computing and data — a critical part of modern computing. The solution that you present appears to assume a single Postgres server that is used by all participants. Feel free to correct me if I've misunderstood. I'll go through some of your points and try to respond to them:
Multiple Tables: Kafka is often used to convey events and each event type can have its own data content. Creating separate tables for each event type is a pointless creation of additional complexity.
Signalling, not Polling, not Messaging/Query after Signal: You do not seem to understand the overhead (performance and maintenance) that triggers represent. Using Postgres, if I want to propagate changes in database state, the most economical way to convey those changes is to capture the them from the Postgres write ahead log (WAL) for which Postgres provides a command, put them into a message, publish the message to a Kafka topic, and any process that is interested can subscribe to the topic(s). No additional triggers, queries, or signals - and if the schema gets changed, the message will automatically change too.
Security: Your remark about granularity is apples and oranges. Kafka granularity is equivalent to Linux/UNIX file permissions (actually looks pretty much the same). It effectively treats a queue topic as a file and can be applied to both users and groups. It works pretty well in a distributed environment. Postgres security is not just for tables and views, it can be applied to almost any database object: triggers, procedures, etc. Security is pretty much a wash between Kafka and Postgres.
Scaling: From my perspective, if our use case is propagating messages/events in a distributed world, Kafka scales relatively linearly and your solution would scale with exponential complexity. I agree that one should not over-engineer, but neither should one try to build on a badly flawed foundation.
Don't make decisions based on synthetic benchmarks: Your comment, "how much of that bench-marked speed translates to software success?", is at best naive. If an application's functions cannot be performed at an acceptable speed, that's a pretty costly and serious issue. If scaling vertically is the only solution, one will always hit physical limits. If the application is not architected to scale horizontally, that is costly and serious issue too. I/O is a poor place to ignore a fundamental performance issue because it appears "good enough" today. Any application that is useful, always gets used far more than its creators assumes.
Multiple Consumers: Postgres does an excellent job of managing concurrent access by multiple consumers, but any locking strategy has its performance limits. That is why modern reliable, scalable, multi-user applications are moving to more reactive solutions, like the actor model, that do not use locking - and limit the amount of time that persistent data stores, like Postgres, need to hold locks.
I'm sorry, but I do not think that you understand the purpose for which Kafka was implemented and I do not believe that your Postgres-based design can in any meaningful way replace Kafka. All serious computing is rapidly moving to distributed process and data. In my original response, I included a link that you responded to with: "I think that's getting a bit off-topic. When someone is using Kafka, they are typically looking for a "portable" solution that isn't locked into a cloud provider." You obviously didn't read the article. My team does not use technologies that are proprietary to any specific cloud vendor. Everything we build can also be run on-premises using Docker and Kubernetes. The goal of cloud-based computing is reliability and automatic failover, high performance and auto-scaling, and near real-time data consistency. Doing it the old fashioned way is too hard and too expensive.
