Structure assignment pipeline

Pipeline for parsing a list of arbitrary Slovene strings and assigning each to a syntactic structure in the DDD database, generating provisional new structures if necessary.

Setup

Most of the scripts come from other repositories and python libraries. Run the set-up script:

$ scripts/setup.sh

Usage

The main script is scripts/process.py. There are several modes ( identified via the -part parameter), depending on whether you want to run the whole pipeline from start to finish (daring!), or with manual intervention (of the parse) in between. XML validation is also provided separately.

strings_to_parse

The input is a file of Slovene strings (one string per line). The script runs the python obeliks tokeniser on the input, tweaks the conllu output a little, and runs JOS-configured classla to parse the output. It then translates the JOS tags (msds and dependencies) from English to Slovene and converts the output to TEI XML. Example:

$ python scripts/process.py -mode strings_to_parse -infile /tmp/strings.txt -outfile /tmp/parsed.xml

parse_to_dictionary

The input should be a TEI XML file (in the same particular format as the output of strings_to_parse) and an xml file of structure specifications. It first splits the TEI file into two files, one with the single-component units and the other with the multiple-component units. For each, it then assigns each unit to a syntactic structure from the DDD database and converts the output into CJVT dictionary XML format. For the single-component units, this is pretty trivial, but for multiple-component units it is more involved, and includes two runs of the MWE extraction script wani.py, generating missing structures in between. At the end, the single-component and multiple-component dictionary files are merged into one dictionary file. Example:

$ python scripts/process.py -mode parse_to_dictionary -infile /tmp/parsed.xml -instructs /tmp/structures_old.xml -outfile /tmp/dictionary.xml -structures /tmp/structures_new.xml

strings_to_dictionary

Combines strings_to_parse in parse_to_dictionary into one call (whereby you forfeit the chance to fix potential parse errors in between). Example:

$ python scripts/process.py -mode strings_to_dictionary -infile /tmp/strings.txt -instructs /tmp/structures_old.xml -outfile /tmp/dictionary.xml -outstructs /tmp/structures_new.xml

all

Same as strings_to_dictionary, but also validates the dictionary and structures outputs, just in case.

$ python scripts/process.py -mode all -infile /tmp/strings.txt -instructs /tmp/structures_old.xml -outfile /tmp/dictionary.xml -outstructs /tmp/structures_new.xml

REST API

The package provides a REST API with endpoints roughly mirroring the process.py modes. For most calls, POST is needed, so that input structures can be easily provided. If processing resulted in temporary new structures, their number is recorded in @new_structures.

Example curl calls:

$ curl -k https://proc1.cjvt.si/structures/strings_to_parse?string=velika%20miza
$ curl -k -X POST -F strings=@/tmp/strings.txt https://proc1.cjvt.si/structures/strings_to_parse
$ curl -k -X POST -F parsed=@/tmp/parse.xml -F structures=@/tmp/structures.xml https://proc1.cjvt.si/structures/parse_to_dictionary
$ curl -k -X POST -F strings=@/tmp/strings.txt -F structures=@/tmp/structures.xml https://proc1.cjvt.si/structures/strings_to_dictionary

Note

Note that any new structures generated are given temporary ids (@tempId), because they only get real ids once they are approved and added to the DDD database. That is normally done via the django import_structures.py script in the ddd_core repository, as part of a batch of DDD updates. That script replaces the temporary ids in the structure specifications and updates the ids in the dictionary file.