\n<\/p>\n
A federated search engine \u2014 the search engine an AI agent or LLM reaches for when it needs to find something true to reason over.<\/p>\n
Ever had your computer reboot on you, or a power outage hit mid-session? Every
\nthread your agent was holding \u2014 gone. Now you have the tool to get it back.
\nNever forget what your agent lost again. Magpie indexes everything your AI
\nhas ever worked through, locally, so a crash is a hiccup instead of amnesia.<\/p>\n
A normal search engine looks in one place. Magpie takes one question and fans it
\nacross everything that matters at once \u2014 the AI’s entire conversation history,
\nthe files on the machine, a structured knowledge graph, a vector store, and the live
\nweb \u2014 and pulls the answer back from wherever it actually lives.
\nFive sources, one call.
\nAnd it searches each one the right way. It can grep for an exact string or regex
\nwhen you know the precise token \u2014 a file path, an error, a line of code. It can
\nsearch by keyword. It can search by meaning, so it finds the thing even when the
\nwords don’t match. It can do all of that at once.
\nThen it does the part that makes it trustworthy: it fuses everything into a
\nsingle ranked answer, and every result carries a trust tier \u2014 fact > reference > lead > stale. The solid sources rise, the loose ones are marked as
\nleads to verify, duplicates collapse, and it’s all trimmed to fit so it never
\nfloods the AI’s context. Ask it to go deep and it expands one question into many,
\nreads the pages, and tells you how many independent sources agree \u2014 a full
\nresearch sweep without an army of agents.
\nIt runs entirely on the machine. No server, no account, and no telemetry
\nunless you turn it on. The AI’s transcripts and files never leave. It plugs into
\nwhatever AI is running over MCP, so the agent can reach all six sources the
\ninstant it needs them.
\nIt is a tool for an AI \u2014 an agent or an LLM.<\/p>\n
At its core is a local index of the AI’s transcripts: a SQLite database with two
\nstructures built side by side \u2014<\/p>\n
an FTS5 full-text index (BM25 keyword ranking), and
\na vector index (sqlite-vec) of 384-dim embeddings produced locally by a
\nsmall all-MiniLM-L6-v2 model.<\/p>\n
Everything is redacted at ingest \u2014 a scrubber strips ~30 classes of secrets
\n(keys, tokens, private keys, connection strings) before a single byte hits the
\nindex.
\nOn top of that index sit the five search modes:<\/p>\n
Mode
\nWhat it does<\/p>\n
grep
\nliteral \/ regex match (exact tokens: paths, errors, code)<\/p>\n
lexical
\nFTS5 \/ BM25 keyword<\/p>\n
semantic
\nembedding K-NN, cosine distance in the vector index<\/p>\n
hybrid
\nlexical + semantic fused by RRF<\/p>\n
rerank
\nhybrid, then a cross-encoder (jina-reranker) re-scores each candidate<\/p>\n
Around that sits the federation layer \u2014 the part that makes it federated:<\/p>\n
A provider plugin system. Five backends (transcripts, files, knowledge
\ngraph, vector, web), each returns Hit objects tagged with a trust
\ntier.
\nA fan-out: one query goes to all providers concurrently (\u22648 workers), each
\nwith a 5-second timeout that fails open \u2014 a slow source contributes nothing
\nrather than blocking the call.
\nTrust-weighted RRF fusion \u2014 Reciprocal Rank Fusion where each source’s rank
\nis multiplied by its trust weight (fact \u00d73, reference \u00d72, lead \u00d71, stale \u00d70.3), damping constant 60. This is the math that merges six heterogeneous
\nsources into one honest ranking.
\nCross-source dedup by content hash \u2014 the same fact found in three places
\ncollapses to one hit, tagged with where else it appeared (corroboration).
\nA token-budget trim, so the merged set never overflows the calling AI’s
\ncontext.<\/p>\n
And it exposes all of this to an AI over an MCP server \u2014 the tools it hands
\nan agent are exactly: search, recent, session, list_sessions, stats,
\nreindex. Note what’s not in that list: nothing that writes an answer.<\/p>\n
RAG = Retrieval-Augmented Generation. It’s a two-stage pipeline, and the
\ndefining stage is the second one: a retriever finds chunks \u2192 they’re stuffed into
\na prompt \u2192 a language model generates the prose answer. The “G” is the whole
\npoint of the name; without a generator writing the answer, it isn’t RAG.
\nMagpie has no G:<\/p>\n
There is no generator anywhere in the search path. Nothing in Magpie
\ncomposes a natural-language answer. The closest thing to a model \u2014 the
\ncross-encoder reranker \u2014 outputs a relevance number per result and reorders
\nthe list. It scores; it never writes a sentence.
\nIt stops at “here are the ranked hits.” A RAG owns the prompt assembly and
\nthe model call. Magpie returns the fused, trust-ranked results and hands them
\nback through MCP. What the AI does next \u2014 whether it even generates anything \u2014
\nis the AI’s job, outside Magpie.
\nIts retriever is more than a RAG’s retriever, not less. A textbook RAG
\nretriever is one vector store: embed the query, top-k by cosine, done.
\nMagpie’s retrieval is six sources, five modes, trust-weighted fusion,
\ncross-source dedup. It’s a far more capable “R” \u2014 but it’s still only the R.<\/p>\n
Plug Magpie into an AI and the pair can form a RAG \u2014 Magpie is the R, the AI you
\nbring is the G. But Magpie by itself ships only the R, and a stronger R than
\nusual. It finds and ranks the truth; it never generates the answer.
\nDeep web search \u2014 research breadth without the token bill
\nThe expensive part of “deep research” is reasoning, and the multi-agent
\napproach pays for it N times over \u2014 one full LLM context per agent, often
\nmillions of tokens for a single question. But reasoning doesn’t need to fan
\nout; one capable model already in context can synthesize. Only the searching
\nneeds breadth \u2014 and searching the web is pure retrieval, zero LLM tokens.
\nmagpie-search deepweb is built on that asymmetry. It fires several sub-queries
\nat the web in parallel, fuses them by trust-weighted RRF + dedup-by-URL into one
\ncompact, token-budget-trimmed source set, optionally reads the top pages’ text
\n(still token-free), and reports how many independent domains corroborate the
\nresult \u2014 an agent-free version of the verification a research swarm pays agents
\nto do.
\nSo you get the breadth, page-reading, and corroboration of a multi-agent deep
\nsearch, but your model only pays for a single synthesis pass over a trimmed
\nresult set.
\nToken cost, measured \u2014 one deep question:<\/p>\n
Approach
\nTokens the model pays<\/p>\n
Multi-agent deep-research swarm (N agents each read pages into their own context)
\n~2,000,000<\/p>\n
magpie-search deepweb –thorough (6 angles \u2192 12 sources, 12 full pages read)
\n~1,050<\/p>\n
That’s ~2,000\u00d7 fewer tokens \u2014 about 1\/2000th the cost \u2014 because the searching
\nand page-reading are pure retrieval (zero model tokens); your model only does
\nthe final synthesis pass over the trimmed, corroborated set.
\n# one question, several angles, read the top pages \u2014 all token-free retrieval
\nmagpie-search deepweb “the question” –q “another angle” –q “a third angle” –thorough
\nThe model in your loop then does one synthesis pass over the merged, corroborated
\nset. That’s the whole saving: the breadth is free, you pay only for the answer.<\/p>\n
pip install magpie-search
\nOr install the latest straight from source (pulls all dependencies):
\npip install “git+https:\/\/github.com\/xfloukiex-lab\/magpie-search.git”
\nOptional \u2014 add the local-LLM features (the cross-encoder reranker runs on the
\nbase install; the session summarizer needs Ollama):
\n# 1. Install Ollama (free, runs entirely locally) \u2014 https:\/\/ollama.com\/download
\n# 2. Pull the model magpie-search uses
\nollama pull phi3.5
\nPython 3.10+ on Windows, macOS, and Linux.<\/p>\n
magpie-search index # build the index (incremental)
\nmagpie-search search “that retry backoff thing” # keyword search
\nmagpie-search search –mode hybrid “…” # keyword + semantic, fused
\nmagpie-search search –mode rerank “…” # + cross-encoder rerank
\nmagpie-search stats # sanity-check the index
\nConnect it to your AI (MCP)
\nMagpie speaks the Model Context Protocol, so any MCP-capable agent can call it.
\nPoint your client at the bundled server:<\/p>\n
The agent then has search, recent, session, list_sessions, stats, and
\nreindex available \u2014 federated, trust-ranked, context-budgeted.<\/p>\n
Command
\nWhat<\/p>\n
magpie-search index
\nIncremental indexing pass over ~\/.claude\/projects\/<\/p>\n
magpie-search search “q”
\nSearch \u2014 –mode grep|lexical|semantic|hybrid|rerank<\/p>\n
magpie-search recent –n 30
\nLatest 30 messages of the newest session<\/p>\n
magpie-search session SESSION-ID
\nFull transcript of one session<\/p>\n
magpie-search list
\nRecent sessions<\/p>\n
magpie-search stats
\nIndex size, last-indexed time, row counts<\/p>\n
magpie-search backup
\nBack up ~\/.claude\/projects\/ to a configurable destination<\/p>\n
Add –help to any command for full options.<\/p>\n
import magpie_search<\/p>\n
results = magpie_search.search(“retry backoff”, mode=”hybrid”, k=5)
\nfor h in results(“hits”):
\n print(h(“trust”), h(“source”), h(“snippet”))<\/p>\n
# LLM features (needs Ollama + phi3.5)
\nimport magpie_search.llm
\nranked = magpie_search.llm.search_rerank(query=”retry backoff”, k=3, pool=10)
\nsummary = magpie_search.llm.summarize(session_id=”abc-123″, n_messages=80)<\/p>\n
magpie-search backup copies your transcript tree to a destination of your
\nchoice \u2014 a local folder (default, zero config), a remote SSH target (NAS \/ home
\nserver), or a remote SSH target with VM boot\/suspend. Configure it in
\n~\/.magpie-search\/backup.env:
\nMAGPIE_SEARCH_BACKUP_SSH_HOST=user@nas.local
\nMAGPIE_SEARCH_BACKUP_SSH_DEST=~\/claude-transcripts\/
\nUseful flags: –dry-run, –no-suspend, –show-config. Backup copies; it
\nnever deletes originals.<\/p>\n
Everything is environment-variable driven with sensible defaults.<\/p>\n
Var
\nDefault
\nWhat<\/p>\n
MAGPIE_SEARCH_HOME
\n~\/.magpie-search
\nData directory (DB, models, logs)<\/p>\n
MAGPIE_SEARCH_MODELS_DIR
\n$MAGPIE_SEARCH_HOME\/models
\nfastembed model cache<\/p>\n
MAGPIE_SEARCH_OLLAMA_HOST
\nhttp:\/\/localhost:11434
\nOllama server URL<\/p>\n
MAGPIE_SEARCH_TOKENIZER
\nheuristic
\nSet to tiktoken for precise budget counting<\/p>\n
MAGPIE_SEARCH_AUDIT_LOG
\n$MAGPIE_SEARCH_HOME\/llm-audit.jsonl
\nPer-call audit log<\/p>\n
The summarizer passes through a 6-probe guardrail stack (length,
\nproper-noun-safety, identifier-safety, refusal-drift, semantic-grounding,
\nself-verify); all six must pass for trust: clean. Any failure suppresses the
\nsummary and returns trust: degraded \u2014 quiet over wrong. Raw messages stay
\naccessible via magpie-search session SESSION-ID.<\/p>\n
Magpie Search is a local tool. No server, no account, no auto-update, no crash
\nreporter, and no telemetry unless you explicitly opt in (see below). Your
\ntranscripts, the index, the audit log, the model cache, and the backups all live
\non your machine.
\nOpt-in telemetry. Telemetry is off by default \u2014 magpie sends nothing
\nuntil you run magpie-search telemetry enable (or set
\nMAGPIE_SEARCH_TELEMETRY=1). When on, it sends only anonymous usage: which
\ncommand ran, search mode, result\/hit counts, latency, error class, and your
\nmagpie\/python\/OS versions, tagged with a random install id. It never sends
\nyour queries, file paths, results, transcript content, username, or IP \u2014 a
\nhard content firewall in telemetry.py drops anything that isn’t a number or a
\nshort enum token. Disable anytime with magpie-search telemetry disable; check
\nstate with magpie-search telemetry status. The only
\nnetwork calls it ever makes are: your local Ollama server (LLM features), your
\nown backup target (only when you run backup), and a one-time model download
\nfrom Hugging Face on first run. Verify it yourself with tcpdump, Wireshark, or
\na network-blocked sandbox.<\/p>\n
Run magpie-search index (and optionally backup) on a schedule. Ready-made
\nunits live in installers\/ for systemd (Linux), launchd (macOS),
\nand Task Scheduler (Windows).<\/p>\n
“rsync not on PATH” \u2014 falls back to scp -r. On Windows, install
\nGit for Windows, which ships rsync.
\nSearch returns nothing \u2014 run magpie-search stats; if last_indexed_at is
\nnull, run magpie-search index.
\nSummarizer always degraded \u2014 that’s the false-positive guard working as
\ndesigned. Raw transcripts remain available via session SESSION-ID.<\/p>\n
Magpie Search is built by VektorGeist LLC.
\nWe build local-first tools for people who run their own AI. Magpie is the search
\ncore; our agent platform is at vektorgeist.com.<\/p>\n
Licensed under the Apache License 2.0 \u2014 see LICENSE. A federated search engine \u2014 the search engine an AI agent or LLM reaches for when it needs to find something true to reason over. Ever had your computer reboot on you, or a power outage hit mid-session? Every thread your agent was holding \u2014 gone. Now you have the tool to get it back. […]<\/p>\n","protected":false},"author":1,"featured_media":5925,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[676],"tags":[],"class_list":["post-5924","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-tech-ai"],"_links":{"self":[{"href":"https:\/\/daiilynews.cu.ma\/index.php?rest_route=\/wp\/v2\/posts\/5924","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/daiilynews.cu.ma\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/daiilynews.cu.ma\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/daiilynews.cu.ma\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/daiilynews.cu.ma\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=5924"}],"version-history":[{"count":0,"href":"https:\/\/daiilynews.cu.ma\/index.php?rest_route=\/wp\/v2\/posts\/5924\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/daiilynews.cu.ma\/index.php?rest_route=\/wp\/v2\/media\/5925"}],"wp:attachment":[{"href":"https:\/\/daiilynews.cu.ma\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=5924"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/daiilynews.cu.ma\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=5924"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/daiilynews.cu.ma\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=5924"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
\nCopyright \u00a9 2026 VektorGeist LLC.
\n“Magpie Search” and the magpie mark are trademarks of VektorGeist LLC. The code
\nis open under Apache-2.0; the brand and name are reserved.
\n
\n
Source link <\/a><\/p>\n","protected":false},"excerpt":{"rendered":"