cjvt-valency/README.md

# cjvt-valency
Required submodules:

* `https://gitea.cjvt.si/kristjan/cjvt-corpusparser.git`

```bash
$ git submodule init
$ git submodule update
```

## Components

### Credentials
Copy `env.default` to `env.local` (gitignored).  
Modify database credentials in `env.local`.  
The file is used by `make`.  


### Database (2 containers)
Set db admin, user, pass, etc in 'Makefile'.  
Spin up the database service and create users:  
Make sure you create a folder for the data on host machine (see `mongodb-stack.yml` `volumes`.  
```bash
$ mkdir -p ${HOME}/mongo_container/data/  # default one
# $ make database-clean  # opt, removes docker services, not data
$ make database-service
$ make database-users  # only first time; user data persists too
```

Populate the database with data form files:  

* ssj500k.xml
* kres.xml
* kres_SRL.json

Set path to files in `Makefile`.  
```bash
# spin up a container with python env
$ make python-env

# install our packages
$ make python-env-install

# run the code
# beforehand, set the data files in Makefile
# instead of mounting directories into the container, you can
# create a link inside ./data, that points to the desired location 

# I've separated the processes for better memory management
$ make fill-database-ssj

$ make fill-database-kres
# You can detach from the running process using Ctrl-p + Ctrl-q

# this is a long operation
# if running on a remote server, use nohup:
$ nohup $(make fill-database > fill-database.log) &
```

If all goes well, we should be able to inspect the database, filled with corpora, on `0.0.0.0:8087`.  


### Flask backend (1 container)
Relies heavily on the database. Set that up first. 
```bash
# spin up container
$ make python-env

# install our packages
$ make python-env-install

# needs to be ran once to modify a new database
$ make backend-prepare-db

# if you have the file prepared (sskj_senses.json), you can
# fill the database with some senses
$ make sskj-senses

# with debugger
$ make backend-dev

# production
$ make backend-prod
```

API endpoints:

* GET word list (pre-cached)
* GET reduced frames (pre-cached)
* POST senses
* User auth logic


### Vue frontend (1 container)
Relies on Flask backend.  
Before running `make`, you might need to set the correct api address.  
Check `./src/frontend_vue/config/config_prod.json`.  
bash
```
# $ make frontend-dev  # development
$ make frontend-prod
```

App available on: `http://0.0.0.0:8080`.  


## Production deployment
Prerequisite: machine with free ports 80 and 8084.  


### Database
Either build the database from scratch (lenghty process) using above instructions or just migrate the database from the faculty server (recommended).  

Build container my-mongo:
```bash
# run once and destroy containers
$ make database-service
```

### Backend
Set database connection details in `/src/backend_flask/db_config.py`.  
Change 'valuser' and 'valuserpass' to the database user.
```bash
mongodb://valuser:valuserpass@my_mongo/valdb
```
In the above line, replace `valuser` with the username and `valuserpass` with the password that was used to create the database tables (the values were set in the root Makefile).  

You can also set the number of workers in `/src/backend_flask/entrypoint.sh`.  
In line with `gunicorn -t 4 -b 127.0.0.1:8084 app:app`, edit the `-t` parameter.  
Rule of thumb is 2x number of available CPU cores.  

Build the backend container:
```bash
# From git root
$ make build-backend-flask
```

### Frontend
Set the server address (where backend will be runnig) in `src/frontend_vue/config/config_prod.json`.  
Build the `/dist` folder that contains the static app (we will be using Nginx to serve it).  
```bash
# From git root
$ make build-frontend-prod
```

All set, now run the stack.  
Stack configuration in `production.yaml`.  

```bash
# From git root
$ make deploy-prod-stack
```



## Uploading a mongo dump
There's a 15GB mongo dump containing the fully processed kres and ssj data.  
We can use that file to deploy our aplication.  
With this database, we will need a minimum of 8GB ram to serve the app.  
If the server is struggling, frontend will throw "Network errors".  

Check `0.0.0.0:8081` and remove (or backup) the current example database `valdb`.  

Run the stack with mongo port mapped:
(uncomment the lines in `production.yaml`)
```yml
ports:
  - 27017:27017
```

Run a separate my-mongo container with the mounted data:
```bash
$ mongo run -it --net host -v <local_dump_path>/dumps my-mongo /bin/bash
```

Inside the container (edit the uesrname, password):
```bash
$ mongorestore /dumps/valdb --db valdb --uri=mongodb://valuser:valuserpass@0.0.0.0:27017
```

After uploading, restart the stack with `27017` commented out.  

## Script running

### Environment setup
```bash
pip install -r requirements.txt
pip install git+https://gitea.cjvt.si/ozbolt/luscenje_struktur.git
pip install git+https://gitea.cjvt.si/kristjan/cjvt-corpusparser.git
```

### Running on already setup environment
```bash
make database-service
```

### Setting up environment for running on proc1 - ramdisk

```bash
# create ramdisk
sudo mount -t tmpfs tmpfs /mnt/tmp
sudo mount -o remount,size=120G,noexec,nosuid,nodev,noatime /mnt/tmp

# change volumes to /mnt/tmp:/data/db
vim dockerfiles/database/mongodb-stack.yml

# change Makefile -runStack to mkdir -p /mnt/tmp
vim dockerfiles/database/mongodb-stack.yml

docker swarm init
make database-service
make database-users

docker exec -it ef0a /bin/bash

# following steps in docker bash:
    mongorestore --gzip --archive=dump.gz --db valdb --uri=mongodb://<REGULAR USERNAME>:<REGULAR PASSWORD>@0.0.0.0:27017

    # add privilegies for user to write into other databases like extvaldb
    mongo --username <ADMIN USER> --password --authenticationDatabase admin
    use valdb
    db.grantRolesToUser(<REGULAR USER>, [{ role: "readWrite", db: "extvaldb"}])

    # check if it worked by
    mongo --username <REGULAR USER> --password --authenticationDatabase valdb

# make mongodb visible only privately
docker stack rm dbstack
cd dockerfiles/database/
docker-compose up
```