Use wikidata to complete seeds

[sebastian-nagel]

Initially, the news crawler was seeded with URLs from news sites from DMOZ, see #8 for the procedure. DMOZ isn't updated anymore, but Wikidata could be a replacement to complete the seed list:

• select all instances of newspaper (news media, or similar) having an official website:

  SELECT DISTINCT ?item ?itemLabel ?lang ?url
  WHERE
  { 
    ?item wdt:P31/wdt:P279* wd:Q11032.
    ?item wdt:P856 ?url.  # with official website
    SERVICE wikibase:label { bd:serviceParam wikibase:language "en,de,ru,fr,es,it,ja,zh,*" }
    OPTIONAL {
       ?item wdt:P407 ?language.
       ?language wdt:P220 ?lang.
     }
  }
  LIMIT 50

  (execute query on Wikidata query service)

[tfmorris]

Wikidata based seed URLs will probably require some significant deduplication, filtering, reranking, etc, but here's a version of the query which adds the language of the URL to account for sites which have different base URLs for different languages, like Blick. It also expands the language list (because * doesn't work), but it could be generalized more. As an example of the type of filtering needed, the Hubei Daily item has three URLs - a corporate site, an e-paper, and a 404.

SELECT DISTINCT ?item ?itemLabel ?lang ?worklang ?url WHERE {
  ?item (wdt:P31/(wdt:P279*)) wd:Q11032;
    p:P856 ?statement.
  ?statement ps:P856 ?url.
  OPTIONAL {
    ?statement pq:P407 ?worklanguage.
    ?worklanguage wdt:P220 ?worklang.
  }
  OPTIONAL {
    ?item wdt:P407 ?language.
    ?language wdt:P220 ?lang.
  }
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en,de,uk,ru,fr,es,it,ja,zh,ar,hu,pt,be,rus,ce,br,cs,sv,dk,da,he,fi,nb,id,eu,pl,nl,az,mar,lv,hr,am,ba,r". }
}
LIMIT 100

Query
As of today, there are 11,177 results. There are more than 200 languages represented, plus a couple of thousand sites with no language tag, and that distribution looks like about what you'd expect (the two letter codes represent TLDs, not language codes, eg. hk, ru, uk, de, au, cn, etc):

eng	3562
fra	826
spa	586
rus	467
deu	316
ita	177
ara	168
ukr	166
fin	152
zho	146
jpn	145
swe	140
nor	122
hk	112
ru	112
por	108
hun	103
nld	93
uk	90
de	86
kor	86
au	78
cn	78
pol	66
hin	60
bel	59

[tfmorris]

The query above from a year ago now returns 156K URLs vs 11K before, but a more specific query for just news websites (Q17232649) returns a very tractable 3553 URLs for ~2500 unique entities covering 90+ languages. 571 of the sites don't include any language info, but that should be pretty easy to figure out just be fetching the home page.

[wumpus]

Tom, is there an easy way to explain how to look up what Wikidata's 2 or 3 letter codes are? I'd like to present lists of potential news urls to web-languages editors. We use 3-letter iso-639-3 codes. I do have a mapping of ISO 639-3 to 2-letter wikipedia names.

Web-languages: https://github.com/commoncrawl/web-languages

[tfmorris]

Greg @wumpus this paragraph describes the general scheme:

Each Wikipedia project has a code, which is used as a subdomain of wikipedia.org. The codes mostly conform to ISO 639-1 two-letter codes or ISO 639-3 three-letter codes, with preference given to a two-letter code if available.[14] For example, en stands for English in ISO 639-1, so the English Wikipedia is at en.wikipedia.org.

but with the various exceptions, the easiest approach is probably to look things up in the table that follows. I'm not sure if that Wikipedia article/table has an equivalent in all languages.

Since I'm not 100% sure what you're trying to map to what and I mentioned a couple of other sets of two/three letter codes, let me also provide a couple of additional answers:

• The list of counts for two and three letter codes that I provided in my earlier comment, Use wikidata to complete seeds #50 (comment), doesn't have very good provenance, but I think it's a mix of ISO 639-3 language codes for those URLs which were tagged with them, plus the TLDs for those URLs which had no language specified.
• The list of mostly two letter codes given to the label service in the SPARQL query only influences the language of the Wikidata item label which is returned and can probably be mostly ignored. Arguably, it's wrong to prefer English over, say the native language, so instead of Ukrainska Pravda, it should be Українська правда to be most easily recognizable to native speakers.

Note that there's a huge difference in scale here. Your list has over 6,000 languages in the "living" category, while there are less than 350 Wikipedias (and some of those are tiny), and the query I provided (which can probably be tuned) covers news websites for fewer than 100 languages.

[wumpus]

Thanks Tom, that's the mapping that I already have. Interesting that your example queries "en" but the return value is "eng".

Huh, and "r" and "rus" came back as "ru".

[tfmorris]

Those codes in the label service clause can be ignored, per my last bullet point above. The only thing they influence is the search order for the label to be used in the itemLabel column. They don't restrict the query itself in any way. The code r is probably a typo.

If you want to parse this list out into chunks to present to different language speakers, I'd recommend filtering on workLang which should be the language represented by the URL, but the data is fairly messy, so probably needs cleaning first. The query could also be tweaked to work better. The intent was to have lang available as a fallback for workLang in case it was missing, but it's actually generating all combinations of the two, which isn't useful. The data is crowdsourced, so is also inherently messy and represented non-uniformly.