07. Research and verification¶

How VeeJay finds out who an artist and a song really are, and how much it trusts what it finds.

7.1 Search first¶

Research on every artist and song begins with a web search, and this holds for a quick survey or a design check as well as for the real pipeline. VeeJay searches Google and Bing, one engine in each of two browser tabs, at the pace of a person: one search every 15 to 30 seconds. Before each batch it runs a self-test on artists whose answers are known. On any CAPTCHA or unusual-traffic page it stops, and it never solves or bypasses one.

A search finds. It never proves. Every link on a results page is collected and sorted by the formats of the outside identifiers VeeJay knows (a Spotify artist, a Wikidata item, a YouTube channel, and so on). The page's layout is never relied on, and the description, songs and members shown beside the results are hints only.

Each thing found is written at once as an unverified data point, recording the artist, the field it is for, the value and the engine that found it. Wikidata, MusicBrainz, Lidarr and the other interfaces are used after the search, to verify what it found.

Search supplies identity and spelling, including for lyrics. It never supplies lyric text. It never sets a Primary: it only supplies evidence of an official video.

7.2 The chain of trust¶

A data point is verified by examining its source, and the examination starts from something certain.

• An anchor is an identity established by hard evidence: a MusicBrainz match made by recording credit, a channel of one of our own videos, or your word.
• A data point is verified when a verified page links to it. Trust flows from an anchor through the links that verified pages publish.
• It is confirmed when the target links back, or when two verified pages list it.
• A data point that fails is rejected. It stays on record as history, so a bad candidate is never checked twice.

Verification is done by scripts and interfaces wherever possible (MusicBrainz and Wikidata links, the YouTube API, the HTML of official sites), and in a browser only where a page cannot be read otherwise. A verified value is then promoted into the artist's own field. The live fields hold verified values only.

Verification is a point-in-time record with its date. It is not rechecked on a schedule, and drift is accepted. A verified video is set back to unverified only for a strong contradiction: both engines and VeeJay's own verification agree.

7.3 The identity score¶

Each artist has a score from 0 to 100 that says how well their identity is established by verified evidence. Every verified data point counts toward it, and having every data point adds up to 100. Reaching 100 is hard by design: only the biggest artists, with the widest presence on the internet, can reach it.

• Only verified data points score. A candidate that has not been verified adds nothing.
• Each data point has a weight in the blueprint. The weights are stored with the field they belong to, and the score is the share of the weight that has been earned.
• The weights are tuned against a set of artists whose answers are known: popular artists such as Taylor Swift and Michael Jackson set the ceiling, and regional bands with real presence set the floor that decides the passing cutoff. The set is rerun after any change to the scoring.
• An artist is trusted when the score reaches the cutoff, the evidence is confirmed in at least two groups of sources, at least one of our own songs is confirmed, and nothing conflicts.

7.4 Sources and authority¶

When two sources disagree about a value, the order of authority decides which is kept:

1. You. A fact that you show or state is established. It is never changed, shortened or replaced to suit a rule, another source or another row. Anything that looks odd is reported to you and left alone.
2. Wikipedia. It is authoritative on its own and does not need a second source. A title, year, disc or track that Wikipedia states is written exactly as Wikipedia states it, parentheses and subtitles included.
3. Lidarr and MusicBrainz. They supply what Wikipedia does not state. Lidarr is asked first for album track lists, then MusicBrainz.
4. Wikidata.
5. Search results, which are the lowest and only ever candidates.

This order decides which verified value is kept. It does not decide where research begins, which is always a search.

How a field is filled.

• A blank field fills only from a verified data point.
• A higher-ranked verified source replaces a lower-ranked value when the new one passes verification. The replacement is logged and shown in the run summary.
• A value that two sources agree on is never replaced, and a value that you supplied is never replaced.
• Ties and conflicts go to review.
• A value that cannot be found is left blank. Nothing is ever guessed, and the reason it could not be found is logged.

Where a fact came from. Every researched value records its source, so bad data can be traced to where it came from.

Safety. Before any run that writes, the database is backed up. Every write is recorded in a change log with the old value, the new value, the rule and the source, so any change can be undone by a script. A source that has a second route is switched at once when the first stops answering, and a run never skips a song because a provider failed. A page reached through a redirect that lands on a different title is never taken.

7.5 Needs attention¶

When VeeJay cannot decide, it asks you. Each question is a review item with the artist, song or video it is about (exactly one), the field, the current and the proposed value, the evidence, a recommendation and a status of Open, Decided or Applied. Your answer is one of three: accept the proposed value, keep the current one, or use a value you give.

A video is held while any review item for it, for its song or for its artist is still Open, and it is not marked verified. You answer inside the app, and each question can be answered in a click.