42 lines
1.6 KiB
Markdown
42 lines
1.6 KiB
Markdown
|
# corpusparser
|
||
|
|
A tool for parsing ssj500k and Kres into a unified .json format.
|
||
|
|
|
||
|
|
## Quickstart
|
||
|
|
Run `make`. You will get a container with python3 and this package installed.
|
||
|
|
|
||
|
|
## Input:
|
||
|
|
### ssj500k
|
||
|
|
To parse ssj500k, point to the monolythic `ssj500k-sl.body.xml` file (tested on ssj500k 2.1).
|
||
|
|
|
||
|
|
### Kres
|
||
|
|
To parse Kres, point to folders:
|
||
|
|
* Kres folder, containig several (around 20K) .xml files (`F00XXXXX.xml.parsed.xml`).
|
||
|
|
* Kres SRL folder, containing SRL links for the corresponding F00...xml files (`F00XXXXX.srl.json`).
|
||
|
|
|
||
|
|
## Internal data format
|
||
|
|
This is the internal python dict data format. It can be stored to file as `.json` or stored into a database for application usage.
|
||
|
|
```python
|
||
|
|
{
|
||
|
|
'sid': 'F0034713.5.0',
|
||
|
|
'text': 'Mednarodni denarni sklad je odobril 30 milijard evrov vredno posojilo Grčiji. ',
|
||
|
|
'tokens': [
|
||
|
|
{'text': 'Mednarodni', 'lemma': 'mednaroden', 'msd': 'Ppnmeid', 'word': True, 'tid': 1},
|
||
|
|
{'text': 'denarni', 'lemma': 'denaren', 'msd': 'Ppnmeid', 'word': True, 'tid': 2},
|
||
|
|
{'text': 'sklad', 'lemma': 'sklad', 'msd': 'Somei', 'word': True, 'tid': 3},
|
||
|
|
{'text': 'je', 'lemma': 'biti', 'msd': 'Gp-ste-n', 'word': True, 'tid': 4},
|
||
|
|
{'text': 'odobril', 'lemma': 'odobriti', 'msd': 'Ggdd-em', 'word': True, 'tid': 5},
|
||
|
|
{'text': '30', 'lemma': '30', 'msd': 'Kag', 'word': True, 'tid': 6},
|
||
|
|
{'text': 'milijard', 'lemma': 'milijarda', 'msd': 'Sozmr', 'word': True, 'tid': 7}, # ...
|
||
|
|
]
|
||
|
|
'jos_links': [
|
||
|
|
{'to': 1, 'from': 3, 'afun': 'dol'},
|
||
|
|
{'to': 2, 'from': 3, 'afun': 'dol'},
|
||
|
|
{'to': 3, 'from': 5, 'afun': 'ena'}, # ...
|
||
|
|
]
|
||
|
|
'srl_links': [
|
||
|
|
{'to': 3, 'from': 5, 'afun': 'ACT'},
|
||
|
|
{'to': 7, 'from': 5, 'afun': 'PAT'}
|
||
|
|
]
|
||
|
|
}
|
||
|
|
```
|