Optimize Trends Queries: First pass using DuckDB #11

Open

taf wants to merge 0 commits from trends__query_optimization into main

pull from: trends__query_optimization

merge into: taf:main

taf:main

taf:numa_scan_sidecar__elixir-side-perf

taf:trends__query_optimization__duckdb

taf:trends__query_optimization__duckdb__2

Conversation 0 Commits - Files changed - +2263 -70

taf commented 2026-06-19 23:57:44 +00:00

Owner

Copy link

No description provided.

taf added 48 commits 2026-06-19 23:57:44 +00:00

dev compose file f4b807edfa

start with trends fc5beed8d1

add trends view, controller 81fb3b1efd

adapt the html/chart stuff to this app 77938834a6

fix hover card 740354e266

ui to help show what is happening 5caf5f39df

trends starting to display decently 43e4ce0819

use sigmoid relevance weighting rather than thresholds 5c241c6c6b

style 5887a39b90

new chart, and fix controller to remove unused threshold 3fed001d70

default to trends, not search c3d88032cf

overall index will now point towards trends, 'home' is now 'search_home' 048cd66c79

clarify names 24ec1890e6

update examples, routes ae3e01f1d3

start reorganizing templates for trends 75d5229356

wiring 46fd7d1fb9

update trends controller to pass sigmoid params 1a491f636f

add compute by feed 71f60abd80

make a reusable component out of the sigmoid viz 25eb9a0cb2

unset the url param after it is consumed e849d11680

refine trends UI further 7c03b96ae5

improve CI accuracy 1ba81843a0

break out sigmoid_weight 453b5197de

use macros to clean up queries 84b77011b3

clean up the charts ecc7dda1a4

style low relevance results to de-emphasize dc28d3cffb

move for clarity 81fdb93a11

fix shading d3645114a1

2-column view to highlight the feeds farthest from the median in both directions simultaneously 5bb7aee62e

fix tooltips e44c83a651

add weekly by-feed comparison, as that could be useful 672fbcfe91

break out the trends analysis queries for interactive explain/analyze 3e7afc7ae7

long timeouts to cope for now a00a79d0dd

some stuff to try, assisted by claude on this one d651a9bfb0

bump shm size and fix migration for experimental indices a5e2501828

start ripping stuff out a05a0930f5

remove unused b2c8a46132

cumulative histogram, relevance distribution, not keeping 4201cad5cc

Not keeping "cumulative histogram, relevance distribution, not keeping" ... 8cbd705226

This reverts commit 4201cad5cc.

WIP: add dump_sample.sh for exporting dev data subsets ... 3cbbb903a3

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

fix: use POSTGRES_HOST consistently in dump_sample.sh ... 2e48d948ed

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

add clone_test_db.sh and document testing strategy ... 2fa371bf14

Tests run against podsearch_test, a TEMPLATE clone of podsearch_dev.
This copies data + HNSW indexes instantly, avoiding the ~15 min CSV
import + index rebuild. The Ecto Sandbox rolls back per-test writes
so the cloned data stays intact across mix test runs.

Also expands README with export/import and testing documentation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

add dev data sampling docs and gitignore for .env and sample data ... 974aa1d60e

Expanded README with export/import instructions for dump_sample.sh,
including copy-pasteable docker exec commands for loading CSVs.
Added .env and podsearch_sample/ to .gitignore.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

trends via duckdb, main module 72026b665d

duckdb trends, wire up 0f00e89c55

attempt fix timeout fd72e64c9e

add timing to Req.post for DI Embeddings API, log latency f4ba860e9b

TrendsDuck: Fix window timestamp format, and stop silently failing, log errors usefully instead dd4e7284d7

Merge conflict checking is in progress. Try again in few moments.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin trends__query_optimization:trends__query_optimization

git switch trends__query_optimization

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main

git merge --no-ff trends__query_optimization

git switch trends__query_optimization

git rebase main

git switch main

git merge --ff-only trends__query_optimization

git switch trends__query_optimization

git rebase main

git switch main

git merge --no-ff trends__query_optimization

git switch main

git merge --squash trends__query_optimization

git switch main

git merge --ff-only trends__query_optimization

git switch main

git merge trends__query_optimization

git push origin main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

taf/podcast_search!11

No description provided.