I’m a big fan of tmducken. We use it heavily in our prod systems. That said, we’ve recently started exploring ducktape [1] in our new projects and have been really impressed with the performance. It also support more complex types on insertions and queries which has been helpful for us. Not affiliated with the project, but just wanted to show it some love since it’s a bit newer. It was created by an active contributor to tmducken.
It really is - we use duckdb as our main workhorse in our entire stack.
As a side note, thank you for all the work you have done in the Clojure ecosystem! Techascent’s data science packages was a major tipping point in my company’s decision to build our data science ecosystem in Clojure and has been wonderful to work with.
At Cronitor we use ClickHouse, but we're leaving it behind for our next product and building directly on Parquet and DuckDB.
We think the future of observability in the AI age is self-hosted directly on NVMe backed by cheap and limitless object storage. I don't want to send customer conversations and agent thoughts to a giant multi-tenant borg SaaS database like Sentry or BetterStack.
> self-hosted directly on NVMe backed by cheap and limitless object storage.
How is that?! NVMe is a protocol for fast PCIE based local storage or NVMe fabric which is PCIe over network. Object storage(in the sense of S3, R2 etc) are usually networked horizontally scaling non posix bucketed storage. They are much much slower because of the network calls..
How would NVMe map to something like S3? And what for?
I'm just guessing but observability is often looking at small slices of hot data and then there's a vast set of cold data that is occasionally needed. Sounds like NVMe is for the hot data cache and object storage is for cold data.
Yes, you nailed it. Old data is important sometimes, like when a problem has been identified and investigated, but most workloads are looking at the current state of the system. So we keep the hot data cached locally (not nas/ebs) and s3 is always the source of truth. DuckDB over parquet files on a local ssd is fast enough you don’t need a traditional database.
We are using DuckLake with a “lakehouse” architecture for the observability agent.
Duckdb CLI is a powerhouse, it can load files as diverse as gzipped json lines, so you can stuff compressed logs straight into a directory yet still easily query them with SQL when you need to.
> Developing such a high quality power tool in such an open manner is honorable.
Credit where credit is due, I would say their efforts are more than just "honorable", I could easily prefix that with an "extremely" and maybe add in a ", most excellent" afterwards.
You understand this is more than that right? This is saying, next time you need to analyze a big dataset, tell your AI to load it in DuckDB, and if it needs to run a query that SQL cannot do, tell it to do what they did in this blog, and it'll have the query run using Clojure super fast.
It's not really about querying a DB from Clojure, well not in the boring sense of how do I run some SQL over data in an existing DB.
What this is supposed to replace is say a big Spark Cluster for processing/querying large datasets.
Impressive, you can really do a lot on a single node when it comes to big-data queries nowadays, I agree too many jump straight to a Spark cluster or something similar when you can just write a small script on a single node.
DuckDB, for those who don't know it, has a great extension system, for example it can directly use OLTP databases such as PostgreSQL, MySQL, SQLite, SQL Server; cloud data warehouses/data lakes and big data formats (Iceberg, Delta, Snowflake, Hive, ORC, Parquet, AVRO), other data sources (ODBC), storage (S3), and much much more.
kianN | 20 hours ago
[1] https://github.com/dynamic-alpha/ducktape
hrrld | 18 hours ago
kianN | 13 hours ago
As a side note, thank you for all the work you have done in the Clojure ecosystem! Techascent’s data science packages was a major tipping point in my company’s decision to build our data science ecosystem in Clojure and has been wonderful to work with.
encoderer | 20 hours ago
We think the future of observability in the AI age is self-hosted directly on NVMe backed by cheap and limitless object storage. I don't want to send customer conversations and agent thoughts to a giant multi-tenant borg SaaS database like Sentry or BetterStack.
reacharavindh | 11 hours ago
> self-hosted directly on NVMe backed by cheap and limitless object storage.
How is that?! NVMe is a protocol for fast PCIE based local storage or NVMe fabric which is PCIe over network. Object storage(in the sense of S3, R2 etc) are usually networked horizontally scaling non posix bucketed storage. They are much much slower because of the network calls..
How would NVMe map to something like S3? And what for?
sethev | 8 hours ago
encoderer | 7 hours ago
We are using DuckLake with a “lakehouse” architecture for the observability agent.
eterm | 20 hours ago
ambicapter | 20 hours ago
Credit where credit is due, I would say their efforts are more than just "honorable", I could easily prefix that with an "extremely" and maybe add in a ", most excellent" afterwards.
arikrahman | 16 hours ago
solarized | 19 hours ago
most of my query now are llm generated. i created custom api to connect llm to any databases im currently using.
sakjur | 16 hours ago
didibus | 16 hours ago
It's not really about querying a DB from Clojure, well not in the boring sense of how do I run some SQL over data in an existing DB.
What this is supposed to replace is say a big Spark Cluster for processing/querying large datasets.
c0_0p_ | 14 hours ago
davidpapermill | 14 hours ago
If we had developed our tech in Python we’d be a world of hurt now as we scale.
didibus | 16 hours ago
mnming | 14 hours ago
I did consider tmducken in the beginning.
(Disclaimer: I maintain o11ylite)
alex_smart | 13 hours ago
HackerThemAll | 12 hours ago
https://duckdb.org/docs/current/core_extensions/overview
https://duckdb.org/community_extensions/list_of_extensions
walthamstow | 7 hours ago