\n
\nLet Claude \u2014 or any LLM \u2014 actually watch a video.
\nMost AI tools don’t really see a video. Paste a YouTube link into ChatGPT and it
\nreads the transcript, not the picture. Claude won’t take a video file at all.
\nEven Gemini, which can read video natively, has to send it up to Google and
\nsamples frames at a fixed interval (1 fps by default), so fast cuts slip past.
\nclaude-real-video does it differently, and locally: point it at a URL or a
\nfile, and it pulls the frames that actually matter (every scene change, not a
\nfixed quota), throws away the near-duplicates, transcribes the audio, and hands
\nyou a clean folder any LLM can read \u2014 on your own machine, nothing uploaded.
\ncrv “https:\/\/www.youtube.com\/watch?v=…”
\n# \u2192 crv-out\/frames\/*.jpg + crv-out\/transcript.txt + crv-out\/MANIFEST.txt
\nThen drop the frames + MANIFEST.txt into Claude \/ ChatGPT \/ Gemini and ask away.<\/p>\n
Why not just sample frames?
\nMost “let an LLM watch a video” scripts (and Gemini’s own pipeline) grab frames
\nat a fixed interval \u2014 e.g. one per second. That over-samples a static
\nscreencast and under-samples a fast-cut reel. claude-real-video is smarter:<\/p>\n
fixed-interval sampling
\nclaude-real-video<\/p>\n
Frame selection
\nevery N seconds
\nscene-change detection + density floor<\/p>\n
Repeated shots (A-B-A cuts)
\nsent again every time
\nsliding-window dedup sends each shot once<\/p>\n
Static slide (10 min)
\n~600 near-identical frames
\ncollapses to 1 (dedup)<\/p>\n
Fast-cut reel
\nmisses frames between samples
\ncatches each visual change<\/p>\n
Audio
\noften ignored
\nWhisper transcript w\/ language detect<\/p>\n
Where the video goes
\noften uploaded to a cloud
\nstays on your machine<\/p>\n
Input
\nusually local file only
\nURL (yt-dlp) or local file<\/p>\n
You feed the model fewer, more meaningful frames \u2014 cheaper context, better
\nunderstanding.<\/p>\n
pip install claude-real-video # core (frames + dedup)
\npip install “claude-real-video(whisper)” # + audio transcription
\nSystem requirement: ffmpeg
\nffmpeg \/ ffprobe are used for frame extraction and audio, and aren’t
\npip-installable. Install them once:<\/p>\n
OS
\ncommand<\/p>\n
macOS
\nbrew install ffmpeg<\/p>\n
Linux
\nsudo apt install ffmpeg (or your distro’s package manager)<\/p>\n
Windows
\nwinget install Gyan.FFmpeg \u2014 or choco install ffmpeg \u2014 or download a build and add its bin\\ folder to your PATH<\/p>\n
Verify it’s on your PATH:<\/p>\n
Transcription uses the whisper CLI (installed by the (whisper) extra, or
\npip install openai-whisper). Whisper also relies on ffmpeg.
\nWorks on macOS, Windows, and Linux \u2014 Python 3.10+.<\/p>\n
# A YouTube \/ Instagram \/ TikTok \/ … link
\ncrv “https:\/\/www.instagram.com\/reel\/XXXX\/”<\/p>\n
# A local file, English transcript, output to .\/out
\ncrv lecture.mp4 -o out –lang en<\/p>\n
# Frames only, no transcription
\ncrv clip.mp4 –no-transcribe<\/p>\n
# A login-gated video (your own \/ authorised use): pass a Netscape cookie file
\ncrv “https:\/\/…” –cookies cookies.txt
\npython -m claude_real_video … works as an alias for crv too.<\/p>\n
flag
\ndefault
\nmeaning<\/p>\n
-o, –out
\ncrv-out
\noutput directory<\/p>\n
–scene
\n0.30
\nscene-change sensitivity (lower = more frames)<\/p>\n
–fps-floor
\n1.0
\nat least one frame every N seconds<\/p>\n
–max-frames
\n150
\nhard cap on total frames<\/p>\n
–lang
\nauto
\nWhisper language (en, zh, auto, …)<\/p>\n
–dedup-threshold
\n8
\n% of pixels that must change for a frame to count as new; higher = fewer frames<\/p>\n
–dedup-window
\n4
\ncompare against the last N kept frames \u2014 a shot the model already saw doesn’t come back after a cutaway (1 = consecutive-only)<\/p>\n
–report
\noff
\nkeep dropped frames in .\/dropped + write report.html visualising every keep\/drop decision<\/p>\n
–no-transcribe
\noff
\nskip audio<\/p>\n
–keep-audio
\noff
\nalso save the full soundtrack (audio.m4a) so audio models can hear it<\/p>\n
–cookies
\n\u2013
\nNetscape cookie file for login-gated sources<\/p>\n
from claude_real_video import process<\/p>\n
r = process(“https:\/\/youtu.be\/…”, “out”, lang=”en”)
\nprint(r.frame_count, r.transcript_path)<\/p>\n
Fetch \u2014 yt-dlp for URLs (optional cookies), or copy a local file.
\nExtract \u2014 one chronological ffmpeg select pass grabs every scene change
\nplus a density floor (at least one frame every –fps-floor seconds), so
\nfast cuts and slow screencasts are both covered.
\nDedup \u2014 real pixel difference (downscaled RGB, not a perceptual hash \u2014 hashes
\ngo blind on flat colours and equal-luma hue changes) against a sliding window
\nof the last –dedup-window kept frames, so an A-B-A cutaway doesn’t re-send a
\nshot the model has already seen. –report writes report.html showing every
\nkeep\/drop decision with its diff %, for tuning.
\nText \u2014 if the video already has subtitles (a sidecar .srt\/.vtt next to a
\nlocal file, or an embedded subtitle track), those are used as the transcript \u2014
\nfaster and more accurate than re-transcribing. Only when there are no subtitles
\ndoes it fall back to Whisper on the audio (skipped cleanly if there’s no audio).
\nAudio (optional, –keep-audio) \u2014 save the full original soundtrack
\n(audio.m4a: music + speech + effects, copied losslessly when possible). The
\ntranscript only has the words; the audio file lets a model that can listen
\n(Gemini, GPT-4o, \u2026) actually hear the music and tone.
\nManifest \u2014 MANIFEST.txt summarises everything for the model.<\/p>\n
So the model can see (key frames), read (transcript) and \u2014 with –keep-audio \u2014
\nhear (full soundtrack) the video. The transcript is plain text any model can read;
\nthe tool doesn’t burn subtitles into the video \u2014 burning is a presentation choice,
\nnot something needed to make a video AI-readable.<\/p>\n
Only download content you have the right to. The –cookies option is for
\nyour own, authorised access \u2014 don’t ship credentials in a repo.
\nRe-running overwrites the output directory.<\/p>\n
MIT
\n