corpusparser

A tool for parsing ssj500k and Kres into a unified .json format.

Quickstart

Run make. You will get a container with python3 and this package installed.

Input:

ssj500k

To parse ssj500k, point to the monolythic ssj500k-sl.body.xml file (tested on ssj500k 2.1).

Kres

To parse Kres, point to folders:

Kres folder, containig several (around 20K) .xml files (F00XXXXX.xml.parsed.xml).
Kres SRL folder, containing SRL links for the corresponding F00...xml files (F00XXXXX.srl.json).

Internal data format

This is the internal python dict data format. It can be stored to file as .json or stored into a database for application usage.

{
	'sid': 'F0034713.5.0',
	'text': 'Mednarodni denarni sklad je odobril 30 milijard evrov vredno posojilo Grčiji. ',
	'tokens': [
		{'text': 'Mednarodni', 'lemma': 'mednaroden', 'msd': 'Ppnmeid', 'word': True, 'tid': 1},
		{'text': 'denarni', 'lemma': 'denaren', 'msd': 'Ppnmeid', 'word': True, 'tid': 2},
		{'text': 'sklad', 'lemma': 'sklad', 'msd': 'Somei', 'word': True, 'tid': 3},
		{'text': 'je', 'lemma': 'biti', 'msd': 'Gp-ste-n', 'word': True, 'tid': 4},
		{'text': 'odobril', 'lemma': 'odobriti', 'msd': 'Ggdd-em', 'word': True, 'tid': 5},
		{'text': '30', 'lemma': '30', 'msd': 'Kag', 'word': True, 'tid': 6},
		{'text': 'milijard', 'lemma': 'milijarda', 'msd': 'Sozmr', 'word': True, 'tid': 7}, # ... 
	]
	'jos_links': [
		{'to': 1, 'from': 3, 'afun': 'dol'},
		{'to': 2, 'from': 3, 'afun': 'dol'},
		{'to': 3, 'from': 5, 'afun': 'ena'}, # ...
	]
	'srl_links': [
		{'to': 3, 'from': 5, 'afun': 'ACT'},
		{'to': 7, 'from': 5, 'afun': 'PAT'}
	]
}

1.6 KiB Raw Blame History