Baixar dados da Wikidata usando o serviço WDQS e o pacote wdq

GitHub -> https://github.com/nichtich/wdq#readme 

Passos

(pgm python de exemplo pr_list.py)

import os
import datetime
import time

Gera a query base (e testa no WDQS)

query_base = """PREFIX prov: <http://www.w3.org/ns/prov#>
PREFIX wikibase: <http://wikiba.se/ontology#>

SELECT (?statement as ?node1) (?pr_pred as ?label) (?ref as ?node2)
WHERE {
   ?ref ?pr_pred ?pr_obj .
   ?statement prov:wasDerivedFrom ?ref .
}"""

f1 = open("/home/cloud-di/pr_list.txt", 'r', encoding="utf8")
pr_list = f1.readlines()
f2 = open("/home/cloud-di/pr_list_v2.sh", mode="w", encoding="utf-8")

for pr_item in pr_list:

Substitui variável por constante para evitar problema de timeout

    pr_pred = pr_item.replace("http://www.wikidata.org/prop/reference/", "pr:").strip('"\n' + '\n')

    query_exec = query_base.replace ("?pr_pred", pr_pred)
#    print(query_exec)

    query_file = "/app/wdq/sparql/"+pr_pred+".sparql"       
    f3 = open(query_file, mode="w", encoding="utf-8")
    f3.write(query_exec)

Gera o script com os comando do wdq e a query final

    output_file = "/app/wdq/data/references.tsv"
    cmd = "/app/wdq/wdq --format tsv --ignore --query " + query_file + " >> " + output_file
    f2.write(cmd+"\n")
       
    f3.close()
       
f1.close()
f2.close()

Executa o script com os comando do wdq e a query final