I thought Dagger had/has a lot of potential to be "AWS-CDK for CI pipelines".
I.e. declaratively setup a web of CI / deployment tasks, based on docker, with a code-first DSL, instead of the morass of copy-pasted (and yes orbs) CircleCI yaml files we have strewn about our internals repos.
But their DSL for defining your pipelines is ... golang? Like who would pick golang as "a friendly language for setting up configs".
The underlying tech is technically language-agnostic, just as aws-cdk's is (you can share cdk constructs across TypeScript/Python), but it's rooted in golang as the originating/first-class language, so imo will never hit aws-cdk levels of ergonomics.
That technical nit aside, I love the idea; ran a few examples of it a year or so ago and was really impressed with the speed; just couldn't wrap my around "how can I make this look like cdk".
Still looks like "a circa-2000s Java builder API" and doesn't look like pleasant / declarative / idiomatic TypeScript, which is what aws-cdk pulled off.
Genuinely impressively (imo), aws-cdk intermixes "it's declarative" (you're setting up your desired state) but also "it's code" (you can use all the usual abstractions) in a way that is pretty great & unique.
Could you share an example of aws-cdk code that you think Dagger should take inspiration from? Dagger and aws-cdk work very differently under the hood, so it's difficult to make an apples-to-apples comparison. If there's a way to make Dagger more TS-native without sacrificing other important properties of Dagger, I'm interested. Thanks.
Hello! Yeah, I totally get Dagger is more "hey client please create a DAG via RPC calls", but just making something up in 30 seconds, like this is what I had in mind:
Like I'm still building a DAG, but by creating objects with "kinda POJOs" (doesn't have to be literally POJOs) and then stitching them together, like the outputs of 1 construct (the build) can be used as inputs to the other constructs (tests & container).
Can you description the deployment setup, somewhere in the docs/maybe with a diagram?
I get this is a backend library, which is great, but like does it use postgres replication slots? Per the inherited queries, do they all live on 1 machine, and we just assume that machine needs to be sufficiently beefy to serve all currently-live queries?
Do all of my (backend) live-queries live/run on that one beefy machine? What's the life cycle for live-queries? Like how can I deploy new ones / kill old ones / as I'm making deployments / business logic changes that might change the queries?
This is all really hard ofc, so apologies for all the questions, just trying to understand -- thanks!
Great questions — happy to clarify how deployment and lifecycle work today.
Let me begin by answering: what exactly is this engine? It's simply a computation + cache layer that lives in the same process as the calling code, not a server on its own.
Think of a LinkedQL instance (new PGClient()) and its concept of a "Live Query" engine as simply a query client (e.g. new pg.Client()) with an in-memory compute + cache layer.
---
1. Deployment model (current state)
The Live Query engine runs as part of your application process — the same place you’d normally run a Postgres/MySQL client.
For Postgres, yes: it uses one logical replication slot per LinkedQL engine instance. The live query engine instantiates on top of that slot and uses internal "windows" to dedupe overlapping queries, so 500 queries that are only variations of "SELECT * FROM users" still map to one main window; and 500 of such "windows" still run over the same replication slot.
As hinted at above, yes; each LinkedQL instance (new PGClient()) runs on the same machine as the running app (just as you'd have it with new pg.Client()) – and maps to a single Live Query engine under the hood.
That engine uses a single replication slot. You specify the slot name like:
new PGClient({ ..., walSlotName: 'custom_slot_name' }); // default is: "linkedql_default_slot" – as per https://linked-ql.netlify.app/docs/setup#postgresql
A second LinkedQL instance would require another slot name:
new PGClient({ ..., walSlotName: 'custom_slot_name_2' });
We’re working toward multi-instance coordination (multiple engines sharing the same replication stream + load balancing live queries). That’s planned, but not started yet.
---
3. Lifecycle of live queries
The Live Query engine runs on-demand and not indefinitely. It begins to exist when at least one client subscribes ({ live: true }) and effectively cleans up and disappears the moment the last subscriber disconnects (result.abort()). Calling client.disconnect() also ends all subscriptions and does clean up.
---
4. Deployments / code changes
Deploying new code doesn’t require “migrating” live queries.
When you restart the application:
• the Live Query starts on a clean slate with the first subscribing query (client.query('...', { live: true })).
• if you have provided a persistent replication slot name (the default being ephemeral), LinkedQL moves the position to the slot's current position and runs from there.
In other words: nothing persists across deploys; everything starts clean as your app starts.
---
5. Diagram / docs
A deployment diagram is a good idea — I’ll add one to the docs.
---
Well, I hope that helps — and no worries about the questions. This space is hard, and happy to explain anything in more detail.
Would you clarify what a "transaction" in this instance would mean?
LinkedQL definitely optimizes at multiple levels between a change happening on your database and the live result your application sees. The most significant of these being its concept of query windows and query inheritance which ensure multiple overlapping queries converge on a single "actual" query window under the hood.
Database transactions. Sometimes, when you require exceptionally high throughout and performance, it can be a viable strategy to batch multiple operations into the same transactions in order to reduce roundtrips, io and network latency.
Of course, it comes at the cost of some stability. However I was just curious if such an abstraction could support such use cases. Thank you for the link to the paper!
And of course achieving that "exceptionally high throughput and performance" is the ultimate goal for a system of this nature.
Now, yes — LinkedQL reasons explicitly in terms of transactions, end-to-end, as covered in the paper.
The key structural distinction is that LinkedQL does not have the concept of its own transactions, "since it doesn’t initiate writes". Instead, it acts as an event-processing pipeline that sits downstream of your database — with a strict "transaction-through rule" enforced across the pipeline.
What that transactional guarantee means in practice is this:
Incoming database transactions (via WAL/binlog) are treated as "atomic" units. All events produced by a single database transaction are received, processed, and propagated through the pipeline with their transactional grouping preserved, all the way to the output stream.
→ LinkedQL receives the resulting batch of mutation events from that transaction
→ processes that batch as "one" atomic unit
→ emits it downstream as "one" atomic unit
→ observers bound to the view see a "single" state transition composed of many changes, rather than "a flurry" of intermediate transitions.
Effectively, a systems that thinks in terms of batching and other throughput-oriented write patterns. LinkedQL just doesn’t initiate its own transactions — it preserves yours, end-to-end.
As a potential user, I'd probably be thinking through things like: if I have a ~small-fleet of 10 ECS tasks serving my REST/API endpoints, would I run `client.query`s on these same machines, or would it be better to have a dedicated pool of "live query" machines that are separate from most API serving, so that maybe I get more overlap of inherited queries.
...also I think there is a limit on WAL slots? Or at least I'd probably want not each of my API servers to be consuming their own WAL slots.
Totally makes sense this is all "things you worry about later" (where later might be now-/soon-ish) given the infra/core concepts you've got working now -- looking really amazing!
Thanks — this is a really good scenario to walk through, and I’m happy to extend the conversation.
First, I’m implicitly assuming your 10 ECS tasks are talking to the same Postgres instance and may issue overlapping queries. Once that’s the case, WAL slots and backend orchestration naturally enter the story — not just querying.
A few concrete facts first.
PostgreSQL caps logical replication slots via `max_replication_slots`.
Each LinkedQL Live Query engine instance uses one slot.
Whether “10 instances” is a problem depends entirely on your Postgres config and workload specifics. I’d expect 10 to be fine in many setups — but not universally. It really does depend.
---
That said, if you want strong deduplication across services, the pattern I’d recommend is centralizing queries in a separate service.
One service owns the LinkedQL engine and the replication slot. Other backend services query that service instead of Postgres directly.
Conceptually:
[API services] → [Live Query service (LinkedQL)] → Postgres
From the caller’s point of view this works like a REST API server (e.g. `GET /users?...`), but it doesn’t have to be "just" REST.
If your technology stack requirements allow, the orchestration can get more interesting. We built a backend framework called Webflo that’s designed specifically for long-lived request connections and cross-runtime reactivity — and it fits this use case very naturally.
In the query-hosting service, you install Webflo as your backend framework, define routes by exposing request-handling functions, and have these functions simply return LinkedQL's live result rows as-is:
// the root "/" route
export default async function(event, next) {
if (next.stepname) return next();
const q = event.url.q;
const liveResult = await client.query(q, {
live: true,
signal: event.signal
});
// Send the initial rows and keep the request open
event.respondWith(liveResult.rows, { done: false });
}
Here, the handler starts a live query and returns the live result rows issued by LinkedQL as "live" response.
* The client immediately receives the initial query result
* The HTTP connection stays open
* Mutations to the sent object are synced automatically over the wire and the client-side copy continues to behave as a live object
* If the client disconnects, event.signal is aborted and the live query shuts down
On the client side, you'd do:
const response = await fetch('db-service/users?q=...');
const liveResponse = await LiveResponse.from(response);
// A normal JS array — but a live one
console.log(liveResponse.body);
Observer.observe(liveResponse.body, mutations => {
console.log(mutations);
});
// Closing the connection tears down the live query upstream
liveResponse.background.close();
There’s no separate realtime API to plumb manually, no explicit WebSocket setup, and no subscription lifecycle to manage. The lifetime of the live query is simply the lifetime of the request connection.
---
In this setup:
* WAL consumption stays bounded
* live queries are deduped centrally
* API services remain stateless
* lifecycle is automatic, not manually managed
I haven’t personally run this exact topology at scale yet, but it fits the model cleanly and is very much the direction the architecture is designed to support.
Once you use Webflo, this stops feeling like “realtime plumbing” and starts feeling like normal request/response — just with live mode.
These two suggestions are fine, but I don't think they make fixtures really that much better--they're still a morass of technical debt & should be avoided at all costs.
The article doesn't mention what I hate most about fixtures: the noise of all the other crap in the fixture that doesn't matter to the current test scenario.
I.e. I want to test "merge these two books" -- great -- but now when stepping through the code, I have 30, 40, 100 other books floating around the code/database b/c "they were added by the fixture" that I need to ignore / step through / etc. Gah.
Author here. I didn't mention it because I wasn't writing an evaluation of fixtures. Just writing about how to make better use of fixtures. I actually use both fixtures and factories depending on the project specifics and also whether it is even my decision to make. :)
I still use my index finger; I've just gotten used to moving my hand ~slightly over from j to the nub.
I would definitely prefer their trackpoint module be "flipped upside down" so the nub was on top, directly next to the H key, so I could move "just the index finger", and not my palm, but it's really not a big deal now that I'm used to it.
They seem to get this feedback a lot, b/c they have an FAQ entry about (nub location), which asserts the current thumb location is due to space/engineering constraints. But, dunno, I kinda wonder if that was for the smaller UHK60? B/c just looking at my UHK80, it really seems like the nub could be by the H if they wanted it to. :-)
So not "perfect perfect" but still really amazing imo, and so glad I switched over -- I'm like 10 years late to split keyboards, custom layers for movement / programming binds, everything the cool kids have been doing forever, but I couldn't give up a trackpoint. But here we are, finally! :-)
(Also fwiw I held off on the UHK80 for about a year b/c they were having firmware issues on initial release, repeated/missed keys, that sort of thing, but its been rock solid for me; literally zero issues.)
Everyone's definition of "production quality" is different :-), but Joist is a "mikro-ish" (more so ActiveRecord-ish) ORM that has a few killer features:
I really want to use pipelining for our "em.flush" of sending all INSERTs & UPDATEs to the db as part of a transaction, b/c my initial prototyping showed a 3-6x increase:
If you're not in a transaction, afaiu pipelining is not as applicable/useful b/c any SQL statement failing in the pipeline fails all other queries after it, and imo it would suck for separate/unrelated web requests that "share a pipeline" to have one request fail the others -- but for a single txn/single request, these semantics are what you expect anyway.
Unfortunately in the TypeScript ecosystem, the node-pg package/driver doesn't support pipelining yet, instead this "didn't quite hit mainstream adoption and now the author is AWOL" driver does: https://github.com/porsager/postgres
I've got a branch to convert our TypeScript ORM to postgres.js solely for this "send all our INSERTs/UPDATEs/DELETEs in parallel" perf benefit, and have some great stats so far:
Oh hello! Very happy to hear from you, and even happier to be wrong about your "AWOL-ness" (since I want to ship postgres.js to prod). :-)
My assumption was just from, afaict, the general lack of triage on GitHub issues, i.e. for a few needs we have like tracing/APM, and then also admittedly esoteric topics like this stack trace fixing:
Fwiw I definitely sympathize with issue triage being time-consuming/sometimes a pita, i.e. where a nontrivial/majority of issues are from well-meaning but maybe naive users asking for free support/filing incorrect/distracting issues.
I don't have an answer, but just saying that's where my impression came from.
Thanks a lot. You're spot on about issue triage etc. I haven't had the time to keep up, but I read all issues when they're created and deal with anything critical. I'm using Postgres.js myself in big deployments and know others are too. The metrics branch should be usable, and I could probably find time to get that part released. It's been ready for a while. I do have some important changes in the pipeline for v4, but won't be able to focus on it until December.
Great to hear you're using postgres.js in prod/large deployments! That sort of real-world-driven usage/improvements/roadmap imo leads to the best results for open source projects.
Also interesting about a potential v4! I'll keep lurking on the github project and hope to see what it brings!
That was a pretty nasty assumption you made about them though: That they're MIA because they're upset that their pet project isn't as popular as they'd like.
Jeez.
That said, I hope node-postgres can support this soon. As it stands, every single query you add to a transaction adds a serial network roundtrip which is devastating not just in execution time but how long you're holding any locks inside the transaction.
Imo Hyprland should merge this hyprscrolling plugin into the main project, and just ship it as the default (only?) layout option -- it just scales to "more than 4 windows" so much better than either of Hyprland's master/dwindle layouts.
[1] I tried vanilla arch + archinstall + sway/niri/etc but really couldn't make it work from scratch, vs. the contrast of Omarchy which was "wow this all works" :shrug:
I've been using Ghostty, and other GPU-based apps like Alacritty / WezTerm / Zed, because they're ofc better/faster...
Ironically they've all made my DX worse, by highlighting how terrible the nvidia drivers actually worked on both my old Regolith i3wm/compositor-less or new sway/wayland setup.
Like it's ridiculously terrible.
I've tried every magical env flag that Claude can think of, and 4 of the various 550/560/575/580 driver versions--they all suck at screensharing, or resume from sleep, or just out-right buginess/crashes/segfaults in the nvidia drivers.
It must have always been this bad, but I just didn't notice, b/c I never actually used my GPU? lol
I had a similar experience with Wayland. If I turned compositing effects of in X11, everything worked fine on my two machines (one with a 1050Ti, the other with an AMD card old enough to need the "radeon" driver). Wayland would lag, or crash or just show garbled output.
I was wondering the same, about their backend domain model (or lack of it).
Fwiw in the TypeScript space, we built Joist (https://joist-orm.io/) to do exactly this.
Granted, we went with a Rails/ActiveRecord minimalist take on DDD instead of some of the more elaborate (overkill imo) implementations that are common i.e. in the .NET space.
> Microservices are a design pattern for organisations as opposed
> to technology ... breakout into multiple teams
I agree, but just saying "multiple teams" has led many eng directors to think "I have two squads now --> omg they cannot both be in the same monolith".
When both squads are 5 people each.
And the squads re-org (or "right size") every 9 months to re-prioritize on the latest features.
Five years go by, 7 team/re-org changes, all of which made sense, but thank god we didn't microservice on the 2nd/3rd/4th/5th/6th team boundaries. :grimmacing:
We should stay "stable, long-lived teams" -- like you need to have a team that exists with the same ownership and mandate for ~18 months to prove its a stable entity worth forming your architecture around.
I.e. declaratively setup a web of CI / deployment tasks, based on docker, with a code-first DSL, instead of the morass of copy-pasted (and yes orbs) CircleCI yaml files we have strewn about our internals repos.
But their DSL for defining your pipelines is ... golang? Like who would pick golang as "a friendly language for setting up configs".
The underlying tech is technically language-agnostic, just as aws-cdk's is (you can share cdk constructs across TypeScript/Python), but it's rooted in golang as the originating/first-class language, so imo will never hit aws-cdk levels of ergonomics.
That technical nit aside, I love the idea; ran a few examples of it a year or so ago and was really impressed with the speed; just couldn't wrap my around "how can I make this look like cdk".