I visualize Palantir as a mycorrhizal fungal network and I’m also a disciple of the ‘Just use Postgres‘ philosophy.

So naturally, when I saw that you could CDC stream into Palantir I wanted to take it for a spin.
First things first, if you are on Aurura Postgres you need to do some configuration to enable rds.logical_replication

In a nut shell
GRANT rds_replication TO cdc_user;
Set rds.logical_replication = 1
Reboot postgres

Create a slot
SELECT pg_create_logical_replication_slot('example_slot', 'pgoutput');

Make a demo table

CREATE TABLE public.demo_cdc_orders (
order_id BIGSERIAL PRIMARY KEY,
order_status TEXT NOT NULL,
quantity INT NOT NULL,
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

Create a publication for the table

CREATE PUBLICATION example_publication FOR TABLE public.demo_cdc_orders;

Head to Palantir

Notes: You can only connect to a publication on the ‘writer’ endpoint.

If there is nothing on the other end to receive the CDC data it will spool to disk… and it could crash your DB if you run out of disk.

Also, depending on your slots, multiple tables run with in it. (And will show under the same build job in Palantir)

Save and hit start on the stream

Do a test insert

NSERT INTO public.demo_cdc_orders (order_status, quantity)
VALUES ('NEW', 5);

And if all goes well, you should see a record.

Now, do an update

UPDATE public.demo_cdc_orders
SET order_status = 'SHIPPED',
    quantity = 6,
    updated_at = now()
WHERE order_id = 1;

So now, you can create an object from this streaming dataset

Now if we were to add this to a workshop app,

Set workshop to auto update

UPDATE public.demo_cdc_orders
SET order_status = 'INVOICED',
    quantity = 6,
    updated_at = now()
WHERE order_id = 1;

Make an update, and watch it magically replicate a second later on the workshop! Pretty nifty software that domintates!