alexion/skills
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Implement YouTube ingest skill #23

New Issue
Closed
opened 2026-08-26 22:18:28 -04:00 by alexion · 0 comments
alexion commented 2026-08-26 22:18:28 -04:00
Owner
Copy Link

Implement youtube-ingest skill

Goal

Create a higher-level youtube-ingest skill for turning a YouTube URL into useful Obsidian-ready knowledge artifacts.

This should build on the existing youtube-transcript skill rather than replacing it.
youtube-transcript should remain the low-level primitive for fetching a video's title and transcript.
youtube-ingest should orchestrate the full ingestion workflow: transcript analysis, structured source-note creation, optional visual/frame analysis, and follow-up note suggestions.

Background

The motivating video is Eero Alvar's "Obsidian and AI were made for each other".
URL: https://www.youtube.com/watch?v=hhtExS5UQBI

The video demonstrates a dedicated Obsidian/Pi workflow for YouTube summaries.
It also demonstrates that transcript-only ingestion can miss important information that is visible on screen, such as folder structure, agent prompts, and generated note structure.

In our exploratory workflow, we used yt-dlp to fetch the transcript and then downloaded timestamped captions plus the video to inspect targeted frames.
This allowed us to recover details of the creator's YouTube summary agent and output format that were not obvious from the plain transcript alone.

Relationship to existing youtube-transcript skill

Keep youtube-transcript focused and stable.
It should continue to fetch the title and transcript as JSON.

Add a new skill named youtube-ingest or ingest-youtube.
Prefer youtube-ingest if following noun-first naming for YouTube workflows.

The new skill may call the existing transcript tool internally, but it should not require callers to know about that implementation detail.

User-facing behavior

When the user provides a YouTube URL and asks to ingest it, the skill should create or propose a structured source note.
The note should be suitable for an Obsidian vault.
The note should capture what the video says, how the video is structured, which concepts are worth linking, and what follow-up notes may be useful.

The default mode should be transcript-only and fast.
A deeper visual mode should be available when the video visibly demonstrates workflows, UIs, prompts, file structures, diagrams, slides, or generated documents.

Modes

Quick ingest

Quick ingest should avoid downloading the full video.
It should fetch metadata and transcript, analyze the transcript, and generate a structured source note.

Use quick ingest for lectures, podcasts, essays, and normal talking-head videos where the spoken content is the main artifact.

Deep ingest

Deep ingest should use timestamped captions and selective frame extraction.
It should avoid blindly extracting hundreds of frames.
It should locate candidate moments from captions, extract a small set of representative screenshots around those moments, inspect them, and use them as evidence.

Use deep ingest when the video shows something important on screen.
Examples include an Obsidian vault layout, an AGENTS.md file, a generated note, a terminal command, a diagram, or a UI flow.

Phase-based workflow

The skill should be documented and implemented around explicit phases.
This makes partial progress easy to resume and makes failures easier to diagnose.

Phase 1: Capture source assets

Goal: get stable source materials outside the vault.

Recommended temporary working directory shape:

/tmp/youtube-ingest/<video-id>/
  metadata.json
  <video-id>.en.json3
  <video-id>.webm

Required outputs for quick ingest:

  • metadata JSON from yt-dlp --dump-json --skip-download
  • transcript text or transcript JSON from youtube-transcript

Additional outputs for deep ingest:

  • timestamped captions in json3 format
  • downloaded video file, capped at a reasonable height such as 720p

Use yt-dlp directly for timestamped captions.
The existing youtube-transcript script currently discards timestamps, while json3 preserves tStartMs and dDurationMs for caption events.

Example commands from the exploration:

yt-dlp --dump-json --no-warnings --skip-download "$url" > metadata.json

yt-dlp --skip-download --no-warnings --write-auto-subs --sub-langs en --sub-format json3 -o '%(id)s.%(ext)s' "$url"

yt-dlp --no-warnings -f 'bv*[height<=720]+ba/b[height<=720]/b' -o '%(id)s.%(ext)s' "$url"

Prefer manual English captions when present.
Fall back to English automatic captions.
Use the same language preference order as youtube-transcript: en, en-US, en-GB, then any en* caption.

Phase 2: Locate candidate moments

Goal: identify timestamps where screenshots may answer questions that transcript text cannot answer.

Parse the json3 captions into timestamped events.
Search for terms that signal visible artifacts or important workflow transitions.
The search terms should be configurable or generated from the transcript and the user's stated goal.

For the motivating video, useful terms included:

  • AI inbox
  • vault boundary
  • agents.md
  • YouTube summaries
  • summary of the whole video
  • argument structures
  • link finder
  • missing links
  • semantic embedding
  • sub-agents
  • outlier finder
  • integrated terminal

The phase should produce both machine-readable and human-readable outputs.

Example outputs:

candidate-moments.json
candidate-moments.md
candidate-moments-shortlist.md

Each candidate should include:

  • topic label
  • timestamp
  • start time in milliseconds
  • matched terms
  • exact caption event text
  • surrounding caption context
  • reason the moment may matter

The shortlist should deduplicate nearby hits and keep only the most likely timestamps for visual inspection.

Phase 3: Extract representative frames

Goal: extract a small number of useful screenshots from shortlisted timestamps.

For each shortlisted timestamp, extract the exact frame and nearby frames.
A good default is timestamp - 1s, timestamp, and timestamp + 1s.

Example output shape:

frames/
  03-24_vault-boundary_exact.jpg
  03-23_vault-boundary_minus1.jpg
  03-25_vault-boundary_plus1.jpg
extracted-frames.md
contact-sheet-exact.jpg

Use ffmpeg for extraction.
Generate a contact sheet so the agent or user can quickly decide which frames matter.

Example command pattern:

ffmpeg -hide_banner -loglevel error -ss "$timestamp_seconds" -i "$video" -frames:v 1 -q:v 2 "$output"

Phase 4: Answer visual questions from frames

Goal: use screenshots as evidence rather than automatically preserving them as vault attachments.

This phase is intentionally different from dumping images into the vault.
By default, inspect the frames and extract the information they reveal.
Only copy images into the vault if they add durable value as visual references.

For the motivating video, frame inspection answered these questions:

  • The YouTube workflow lives under INBOX (AI)/pi/youtube-summary/.
  • The folder contains an AGENTS.md file and a fetch_transcript.py script.
  • Pi is launched from the youtube-summary directory.
  • The visible agent prompt heading is YouTube Summarizer Guidelines.
  • The agent prompt instructs the agent to run python3 fetch_transcript.py "<url>".
  • The transcript script outputs JSON containing title and transcript.
  • The agent prompt instructs the agent to read the JSON output and analyze the transcript carefully.
  • The agent prompt instructs the agent to create a summary document formatted exactly with specified sections.
  • The agent prompt instructs the agent to include the video URL at the very top of the document.

The recovered generated-note structure was:

# <Video title>

Video URL: <url>

## Summary

## Video structure

## Key concepts

## Key points

## Argument structures

The visible details of each section were:

  • Summary is a short abstract highlighting the creator's main purpose and context.
  • Video structure is a numbered list describing the step-by-step progression of the narrative.
  • Key concepts is a curated list of critical concepts needed to understand the video's core arguments.
  • Key concepts formats each concept as a Markdown subheading.
  • Key concepts includes brief definitions.
  • Key concepts prioritizes the creator's specific interpretation when it differs from the strict formal real-world definition.
  • Key points is a flat list of the most crucial insights, main thesis, and takeaways.
  • Argument structures extracts logical reasoning patterns behind key points.
  • Argument structures maps arguments into readable premise and conclusion structures.
  • Argument structures should preserve important nuance and avoid forcing every argument into a rigid template.

Phase 5: Generate or update Obsidian notes

Goal: write useful vault artifacts, not just a generic summary.

The source note should distinguish video content from vault-specific interpretation.
Recommended sections:

---
date-created: <date>
source: youtube
url: <url>
caption-source: <manual|automatic english captions>
---
Tags: [[youtube]], ...

# <Video title>

## Summary

## Video structure

## Key concepts

## Key points

## Argument structures

## Relevance to this vault

## Possible notes

## Open questions

The first five content sections are based on the creator's demonstrated format.
Relevance to this vault, Possible notes, and Open questions are additions for second-brain ingestion.
They make the output useful for the local vault rather than merely summarizing the video.

Possible notes should be concrete internal-link candidates.
They may be proposed rather than created automatically.
If the skill creates notes, it should prefer small claim-like notes over generic encyclopedia pages.

For example, from the motivating video, possible note candidates included:

  • [[obsidian as ai interface]]
  • [[ai agent garage]]
  • [[ai inbox boundary]]
  • [[youtube ingest workflow]]

Phase 6: Cleanup and commit guidance

Goal: avoid polluting the vault with raw ingestion artifacts.

Raw video, metadata JSON, timestamped caption JSON, and bulk frame exports should remain outside the vault by default.
The skill should report where temporary assets were placed.
The skill should copy only selected durable assets into the vault when explicitly requested.

If the repository is under Git, the skill may suggest separate commits.
It should not commit automatically unless explicitly asked.

Suggested commit split:

Add YouTube source note for <title>
Add visual references for <title>

Output format requirements

The skill should be explicit about whether it writes files or only proposes content.
If it writes files, it should report every path created or modified.
If it creates temporary files, it should report the temp directory.
If visual analysis was skipped, it should say why.

Errors should name the failing phase and the next corrective action.

Implementation notes

Use Nix-packaged tools where available.
For this repository's packaged skills, command wrappers may use mkSkill.commands env patching to point scripts at Nix-provided executables.
The existing youtube-transcript skill already demonstrates this pattern for Python and yt-dlp.

The deep-ingest implementation likely needs:

  • Python for parsing json3 captions and producing candidate moments
  • yt-dlp for metadata, captions, and video download
  • ffmpeg for frame extraction and contact sheets

Consider implementing helper scripts inside the skill directory:

youtube-ingest/
  SKILL.md
  scripts/capture_assets.py
  scripts/find_candidate_moments.py
  scripts/extract_frames.py
  references/source-note-format.md

The initial implementation can be a documented skill workflow without full automation.
A later implementation can automate the helper scripts.

Acceptance criteria

  • A new skill exists for YouTube ingestion.
  • The existing youtube-transcript skill remains available as a lower-level primitive.
  • Quick ingest can produce a structured source note from a YouTube URL and transcript.
  • The source note includes summary, video structure, key concepts, key points, and argument structures.
  • The source note includes vault-specific relevance, possible notes, and open questions.
  • Deep ingest can download timestamped captions without relying on the current transcript script's plain-text output.
  • Deep ingest can locate candidate visual moments from timestamped captions.
  • Deep ingest can extract a small number of frames around shortlisted moments.
  • Deep ingest can use frames to answer visible-structure questions without copying every frame into the vault.
  • Temporary assets are kept outside the vault by default.
  • The skill reports all created or modified paths.
  • The skill's SKILL.md gives enough guidance for an agent to perform the workflow manually even before every helper script is automated.
# Implement `youtube-ingest` skill ## Goal Create a higher-level `youtube-ingest` skill for turning a YouTube URL into useful Obsidian-ready knowledge artifacts. This should build on the existing `youtube-transcript` skill rather than replacing it. `youtube-transcript` should remain the low-level primitive for fetching a video's title and transcript. `youtube-ingest` should orchestrate the full ingestion workflow: transcript analysis, structured source-note creation, optional visual/frame analysis, and follow-up note suggestions. ## Background The motivating video is Eero Alvar's "Obsidian and AI were made for each other". URL: https://www.youtube.com/watch?v=hhtExS5UQBI The video demonstrates a dedicated Obsidian/Pi workflow for YouTube summaries. It also demonstrates that transcript-only ingestion can miss important information that is visible on screen, such as folder structure, agent prompts, and generated note structure. In our exploratory workflow, we used `yt-dlp` to fetch the transcript and then downloaded timestamped captions plus the video to inspect targeted frames. This allowed us to recover details of the creator's YouTube summary agent and output format that were not obvious from the plain transcript alone. ## Relationship to existing `youtube-transcript` skill Keep `youtube-transcript` focused and stable. It should continue to fetch the title and transcript as JSON. Add a new skill named `youtube-ingest` or `ingest-youtube`. Prefer `youtube-ingest` if following noun-first naming for YouTube workflows. The new skill may call the existing transcript tool internally, but it should not require callers to know about that implementation detail. ## User-facing behavior When the user provides a YouTube URL and asks to ingest it, the skill should create or propose a structured source note. The note should be suitable for an Obsidian vault. The note should capture what the video says, how the video is structured, which concepts are worth linking, and what follow-up notes may be useful. The default mode should be transcript-only and fast. A deeper visual mode should be available when the video visibly demonstrates workflows, UIs, prompts, file structures, diagrams, slides, or generated documents. ## Modes ### Quick ingest Quick ingest should avoid downloading the full video. It should fetch metadata and transcript, analyze the transcript, and generate a structured source note. Use quick ingest for lectures, podcasts, essays, and normal talking-head videos where the spoken content is the main artifact. ### Deep ingest Deep ingest should use timestamped captions and selective frame extraction. It should avoid blindly extracting hundreds of frames. It should locate candidate moments from captions, extract a small set of representative screenshots around those moments, inspect them, and use them as evidence. Use deep ingest when the video shows something important on screen. Examples include an Obsidian vault layout, an `AGENTS.md` file, a generated note, a terminal command, a diagram, or a UI flow. ## Phase-based workflow The skill should be documented and implemented around explicit phases. This makes partial progress easy to resume and makes failures easier to diagnose. ### Phase 1: Capture source assets Goal: get stable source materials outside the vault. Recommended temporary working directory shape: ```text /tmp/youtube-ingest/<video-id>/ metadata.json <video-id>.en.json3 <video-id>.webm ``` Required outputs for quick ingest: - metadata JSON from `yt-dlp --dump-json --skip-download` - transcript text or transcript JSON from `youtube-transcript` Additional outputs for deep ingest: - timestamped captions in `json3` format - downloaded video file, capped at a reasonable height such as 720p Use `yt-dlp` directly for timestamped captions. The existing `youtube-transcript` script currently discards timestamps, while `json3` preserves `tStartMs` and `dDurationMs` for caption events. Example commands from the exploration: ```bash yt-dlp --dump-json --no-warnings --skip-download "$url" > metadata.json yt-dlp --skip-download --no-warnings --write-auto-subs --sub-langs en --sub-format json3 -o '%(id)s.%(ext)s' "$url" yt-dlp --no-warnings -f 'bv*[height<=720]+ba/b[height<=720]/b' -o '%(id)s.%(ext)s' "$url" ``` Prefer manual English captions when present. Fall back to English automatic captions. Use the same language preference order as `youtube-transcript`: `en`, `en-US`, `en-GB`, then any `en*` caption. ### Phase 2: Locate candidate moments Goal: identify timestamps where screenshots may answer questions that transcript text cannot answer. Parse the `json3` captions into timestamped events. Search for terms that signal visible artifacts or important workflow transitions. The search terms should be configurable or generated from the transcript and the user's stated goal. For the motivating video, useful terms included: - `AI inbox` - `vault boundary` - `agents.md` - `YouTube summaries` - `summary of the whole video` - `argument structures` - `link finder` - `missing links` - `semantic embedding` - `sub-agents` - `outlier finder` - `integrated terminal` The phase should produce both machine-readable and human-readable outputs. Example outputs: ```text candidate-moments.json candidate-moments.md candidate-moments-shortlist.md ``` Each candidate should include: - topic label - timestamp - start time in milliseconds - matched terms - exact caption event text - surrounding caption context - reason the moment may matter The shortlist should deduplicate nearby hits and keep only the most likely timestamps for visual inspection. ### Phase 3: Extract representative frames Goal: extract a small number of useful screenshots from shortlisted timestamps. For each shortlisted timestamp, extract the exact frame and nearby frames. A good default is `timestamp - 1s`, `timestamp`, and `timestamp + 1s`. Example output shape: ```text frames/ 03-24_vault-boundary_exact.jpg 03-23_vault-boundary_minus1.jpg 03-25_vault-boundary_plus1.jpg extracted-frames.md contact-sheet-exact.jpg ``` Use `ffmpeg` for extraction. Generate a contact sheet so the agent or user can quickly decide which frames matter. Example command pattern: ```bash ffmpeg -hide_banner -loglevel error -ss "$timestamp_seconds" -i "$video" -frames:v 1 -q:v 2 "$output" ``` ### Phase 4: Answer visual questions from frames Goal: use screenshots as evidence rather than automatically preserving them as vault attachments. This phase is intentionally different from dumping images into the vault. By default, inspect the frames and extract the information they reveal. Only copy images into the vault if they add durable value as visual references. For the motivating video, frame inspection answered these questions: - The YouTube workflow lives under `INBOX (AI)/pi/youtube-summary/`. - The folder contains an `AGENTS.md` file and a `fetch_transcript.py` script. - Pi is launched from the `youtube-summary` directory. - The visible agent prompt heading is `YouTube Summarizer Guidelines`. - The agent prompt instructs the agent to run `python3 fetch_transcript.py "<url>"`. - The transcript script outputs JSON containing `title` and `transcript`. - The agent prompt instructs the agent to read the JSON output and analyze the transcript carefully. - The agent prompt instructs the agent to create a summary document formatted exactly with specified sections. - The agent prompt instructs the agent to include the video URL at the very top of the document. The recovered generated-note structure was: ```markdown # <Video title> Video URL: <url> ## Summary ## Video structure ## Key concepts ## Key points ## Argument structures ``` The visible details of each section were: - `Summary` is a short abstract highlighting the creator's main purpose and context. - `Video structure` is a numbered list describing the step-by-step progression of the narrative. - `Key concepts` is a curated list of critical concepts needed to understand the video's core arguments. - `Key concepts` formats each concept as a Markdown subheading. - `Key concepts` includes brief definitions. - `Key concepts` prioritizes the creator's specific interpretation when it differs from the strict formal real-world definition. - `Key points` is a flat list of the most crucial insights, main thesis, and takeaways. - `Argument structures` extracts logical reasoning patterns behind key points. - `Argument structures` maps arguments into readable premise and conclusion structures. - `Argument structures` should preserve important nuance and avoid forcing every argument into a rigid template. ### Phase 5: Generate or update Obsidian notes Goal: write useful vault artifacts, not just a generic summary. The source note should distinguish video content from vault-specific interpretation. Recommended sections: ```markdown --- date-created: <date> source: youtube url: <url> caption-source: <manual|automatic english captions> --- Tags: [[youtube]], ... # <Video title> ## Summary ## Video structure ## Key concepts ## Key points ## Argument structures ## Relevance to this vault ## Possible notes ## Open questions ``` The first five content sections are based on the creator's demonstrated format. `Relevance to this vault`, `Possible notes`, and `Open questions` are additions for second-brain ingestion. They make the output useful for the local vault rather than merely summarizing the video. Possible notes should be concrete internal-link candidates. They may be proposed rather than created automatically. If the skill creates notes, it should prefer small claim-like notes over generic encyclopedia pages. For example, from the motivating video, possible note candidates included: - `[[obsidian as ai interface]]` - `[[ai agent garage]]` - `[[ai inbox boundary]]` - `[[youtube ingest workflow]]` ### Phase 6: Cleanup and commit guidance Goal: avoid polluting the vault with raw ingestion artifacts. Raw video, metadata JSON, timestamped caption JSON, and bulk frame exports should remain outside the vault by default. The skill should report where temporary assets were placed. The skill should copy only selected durable assets into the vault when explicitly requested. If the repository is under Git, the skill may suggest separate commits. It should not commit automatically unless explicitly asked. Suggested commit split: ```text Add YouTube source note for <title> Add visual references for <title> ``` ## Output format requirements The skill should be explicit about whether it writes files or only proposes content. If it writes files, it should report every path created or modified. If it creates temporary files, it should report the temp directory. If visual analysis was skipped, it should say why. Errors should name the failing phase and the next corrective action. ## Implementation notes Use Nix-packaged tools where available. For this repository's packaged skills, command wrappers may use `mkSkill.commands` env patching to point scripts at Nix-provided executables. The existing `youtube-transcript` skill already demonstrates this pattern for Python and `yt-dlp`. The deep-ingest implementation likely needs: - Python for parsing `json3` captions and producing candidate moments - `yt-dlp` for metadata, captions, and video download - `ffmpeg` for frame extraction and contact sheets Consider implementing helper scripts inside the skill directory: ```text youtube-ingest/ SKILL.md scripts/capture_assets.py scripts/find_candidate_moments.py scripts/extract_frames.py references/source-note-format.md ``` The initial implementation can be a documented skill workflow without full automation. A later implementation can automate the helper scripts. ## Acceptance criteria - A new skill exists for YouTube ingestion. - The existing `youtube-transcript` skill remains available as a lower-level primitive. - Quick ingest can produce a structured source note from a YouTube URL and transcript. - The source note includes summary, video structure, key concepts, key points, and argument structures. - The source note includes vault-specific relevance, possible notes, and open questions. - Deep ingest can download timestamped captions without relying on the current transcript script's plain-text output. - Deep ingest can locate candidate visual moments from timestamped captions. - Deep ingest can extract a small number of frames around shortlisted moments. - Deep ingest can use frames to answer visible-structure questions without copying every frame into the vault. - Temporary assets are kept outside the vault by default. - The skill reports all created or modified paths. - The skill's `SKILL.md` gives enough guidance for an agent to perform the workflow manually even before every helper script is automated.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
alexion
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: alexion/skills#23
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?