Skip to content

Appendix D. Building the first library

The order of work for filling VeeJay's database and library from a collection that already exists: the playlist, the files already on disk, and what the web knows. The same order applies to any large import of existing likes.

The order

  1. Read the playlist. Read the playlist and the details of every video in one batch (title, description, channel, views, length, publish date), and write one row for each video, only with what YouTube returns, exactly as it returns it. A raw snapshot of the answers is kept, and every write goes in a change log that can undo it. A video that is deleted, private or blocked gets no row and is listed in a report. A video that appears twice in the playlist gets one row.
  2. Identify the songs, one song at a time. Read the evidence for every video of one song together, then decide and write the whole song before the next: the artist and the song, each video's disambiguation, its origin, performance, version, type and content, whether a video is an MTV piece (which moves to its own table), and the Primary when it is clear. Where it is not clear, the song is marked Primary pending. A small script shows the evidence and writes the decisions, each recorded with its reason. A value that a later step replaces is kept in the change log. Fields that are not known start at their plain values.
  3. Research (chapter 7). It starts with a search. The MusicBrainz identifiers, the album, year, disc and track, the Wikipedia link and lead, the IMVDb identifiers through Wikidata, and the Primary are found and verified. Metadata can be reused from a matching file already on disk, matched by title and by length against YouTube's. A row that cannot be completed stays unverified and goes on a review list.
  4. Download the videos, one at a time, into a new folder beside the old library. Each is named and tagged only from its row, with the YouTube link, the ID, the update time and the view count in the tags. An unverified row is not downloaded. The resolution of the files settles any song that was Primary pending.
  5. Lyrics.
  6. Audio last. The audio is copied from the existing library or found on SoulSeek, and named exactly like its Primary video.
  7. Switch over. The new folder replaces the old library after the checks pass. The old library is kept until then.

A pilot first

The first run is a pilot of about 100 videos chosen for the hard cases: artists with a common name, backing bands, covers, songs from films and games, and videos with no clear Primary. The full run follows once the pilot's results are checked against answers that are known.

The order of research work

  1. Artists that are not yet identified.
  2. Songs that block a video from being verified.
  3. Songs whose Primary is still pending.
  4. The website and social fields of the remaining artists.

What may be reused from an earlier library

Nothing that says which song, album, year, disc, track, artist or title a video is: those are decided again from YouTube and MusicBrainz, and the old values are only compared with the results, with any difference going to the review list. Once a song is identified and verified, its lyrics and its Wikipedia link and lead may be copied, but only after each passes a check that it still belongs to that song. The Wikipedia page's title must name the song or the artist, the lead must match that page, and synchronized lyrics are checked against the audio recording, never against a video, because a video can be an edit: the last timestamp must not run past the end of the matching audio, and must reach at least half of it. A song whose audio is not available yet keeps its lyrics marked as not timing-checked until it is. A file's path is reused only when the file exists and its length matches YouTube's. Every copied value is recorded with its source.