cjvt-valency/README.md

cjvt-valency

Required submodules:

• https://gitea.cjvt.si/kristjan/cjvt-corpusparser.git

$ git submodule init
$ git submodule update

Components

Credentials

Copy env.default to env.local (gitignored).
Modify database credentials in env.local.
The file is used by make.

Database (2 containers)

Set db admin, user, pass, etc in 'Makefile'.
Spin up the database service and create users:
Make sure you create a folder for the data on host machine (see mongodb-stack.yml volumes.

$ mkdir -p ${HOME}/mongo_container/data/  # default one
# $ make database-clean  # opt, removes docker services, not data
$ make database-service
$ make database-users  # only first time; user data persists too

Populate the database with data form files:

• ssj500k.xml
• kres.xml
• kres_SRL.json

Set path to files in Makefile.

# spin up a container with python env
$ make python-env

# install our packages
$ make python-env-install

# run the code
# beforehand, set the data files in Makefile
# instead of mounting directories into the container, you can
# create a link inside ./data, that points to the desired location 

# I've separated the processes for better memory management
$ make fill-database-ssj

$ make fill-database-kres
# You can detach from the running process using Ctrl-p + Ctrl-q

# this is a long operation
# if running on a remote server, use nohup:
$ nohup $(make fill-database > fill-database.log) &

If all goes well, we should be able to inspect the database, filled with corpora, on 0.0.0.0:8087.

Flask backend (1 container)

Relies heavily on the database. Set that up first.

# spin up container
$ make python-env

# install our packages
$ make python-env-install

# needs to be ran once to modify a new database
$ make backend-prepare-db

# if you have the file prepared (sskj_senses.json), you can
# fill the database with some senses
$ make sskj-senses

# with debugger
$ make backend-dev

# production
$ make backend-prod

API endpoints:

• GET word list (pre-cached)
• GET reduced frames (pre-cached)
• POST senses
• User auth logic

Vue frontend (1 container)

Relies on Flask backend.
Before running make, you might need to set the correct api address.
Check ./src/frontend_vue/config/config_prod.json.
bash

# $ make frontend-dev  # development
$ make frontend-prod

App available on: http://0.0.0.0:8080.

Production deployment

Prerequisite: machine with free ports 80 and 8084.

Database

Either build the database from scratch (lenghty process) using above instructions or just migrate the database from the faculty server (recommended).

Build container my-mongo:

# run once and destroy containers
$ make database-service

Backend

Set database connection details in /src/backend_flask/db_config.py.
Change 'valuser' and 'valuserpass' to the database user.

mongodb://valuser:valuserpass@my_mongo/valdb

In the above line, replace valuser with the username and valuserpass with the password that was used to create the database tables (the values were set in the root Makefile).

You can also set the number of workers in /src/backend_flask/entrypoint.sh.
In line with gunicorn -t 4 -b 127.0.0.1:8084 app:app, edit the -t parameter.
Rule of thumb is 2x number of available CPU cores.

Build the backend container:

# From git root
$ make build-backend-flask

Frontend

Set the server address (where backend will be runnig) in src/frontend_vue/config/config_prod.json.
Build the /dist folder that contains the static app (we will be using Nginx to serve it).

# From git root
$ make build-frontend-prod

All set, now run the stack.
Stack configuration in production.yaml.

# From git root
$ make deploy-prod-stack

Uploading a mongo dump

There's a 15GB mongo dump containing the fully processed kres and ssj data.
We can use that file to deploy our aplication.
With this database, we will need a minimum of 8GB ram to serve the app.
If the server is struggling, frontend will throw "Network errors".

Check 0.0.0.0:8081 and remove (or backup) the current example database valdb.

Run the stack with mongo port mapped: (uncomment the lines in production.yaml)

ports:
  - 27017:27017

Run a separate my-mongo container with the mounted data:

$ mongo run -it --net host -v <local_dump_path>/dumps my-mongo /bin/bash

Inside the container (edit the uesrname, password):

$ mongorestore /dumps/valdb --db valdb --uri=mongodb://valuser:valuserpass@0.0.0.0:27017

After uploading, restart the stack with 27017 commented out.

Script running

Environment setup

pip install -r requirements.txt
pip install git+https://gitea.cjvt.si/ozbolt/luscenje_struktur.git
pip install git+https://gitea.cjvt.si/kristjan/cjvt-corpusparser.git

Running on already setup environment

make database-service

Setting up environment for running on proc1 - ramdisk

# create ramdisk
sudo mount -t tmpfs tmpfs /mnt/tmp
sudo mount -o remount,size=120G,noexec,nosuid,nodev,noatime /mnt/tmp

# change volumes to /mnt/tmp:/data/db
vim dockerfiles/database/mongodb-stack.yml

# change Makefile -runStack to mkdir -p /mnt/tmp
vim dockerfiles/database/mongodb-stack.yml

docker swarm init
make database-service
make database-users

docker exec -it ef0a /bin/bash

# following steps in docker bash:
    mongorestore --gzip --archive=dump.gz --db valdb --uri=mongodb://<REGULAR USERNAME>:<REGULAR PASSWORD>@0.0.0.0:27017

    # add privilegies for user to write into other databases like extvaldb
    mongo --username <ADMIN USER> --password --authenticationDatabase admin
    use valdb
    db.grantRolesToUser(<REGULAR USER>, [{ role: "readWrite", db: "extvaldb"}])

    # check if it worked by
    mongo --username <REGULAR USER> --password --authenticationDatabase valdb

# make mongodb visible only privately
docker stack rm dbstack
cd dockerfiles/database/
docker-compose up