Contains the tool for viewing valency frames and adding user-defined senses to underlying sentences.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

181 lines
4.6 KiB

2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
  1. # cjvt-valency
  2. Required submodules:
  3. * `https://gitea.cjvt.si/kristjan/cjvt-corpusparser.git`
  4. ```bash
  5. $ git submodule init
  6. $ git submodule update
  7. ```
  8. ## Components
  9. ### Credentials
  10. Copy `env.default` to `env.local` (gitignored).
  11. Modify database credentials in `env.local`.
  12. The file is used by `make`.
  13. ### Database (2 containers)
  14. Set db admin, user, pass, etc in 'Makefile'.
  15. Spin up the database service and create users:
  16. Make sure you create a folder for the data on host machine (see `mongodb-stack.yml` `volumes`.
  17. ```bash
  18. $ mkdir -p ${HOME}/mongo_container/data/ # default one
  19. # $ make database-clean # opt, removes docker services, not data
  20. $ make database-service
  21. $ make database-users # only first time; user data persists too
  22. ```
  23. Populate the database with data form files:
  24. * ssj500k.xml
  25. * kres.xml
  26. * kres_SRL.json
  27. Set path to files in `Makefile`.
  28. ```bash
  29. # spin up a container with python env
  30. $ make python-env
  31. # install our packages
  32. $ make python-env-install
  33. # run the code
  34. # beforehand, set the data files in Makefile
  35. # instead of mounting directories into the container, you can
  36. # create a link inside ./data, that points to the desired location
  37. # I've separated the processes for better memory management
  38. $ make fill-database-ssj
  39. $ make fill-database-kres
  40. # You can detach from the running process using Ctrl-p + Ctrl-q
  41. # this is a long operation
  42. # if running on a remote server, use nohup:
  43. $ nohup $(make fill-database > fill-database.log) &
  44. ```
  45. If all goes well, we should be able to inspect the database, filled with corpora, on `0.0.0.0:8087`.
  46. ### Flask backend (1 container)
  47. Relies heavily on the database. Set that up first.
  48. ```bash
  49. # spin up container
  50. $ make python-env
  51. # install our packages
  52. $ make python-env-install
  53. # needs to be ran once to modify a new database
  54. $ make backend-prepare-db
  55. # if you have the file prepared (sskj_senses.json), you can
  56. # fill the database with some senses
  57. $ make sskj-senses
  58. # with debugger
  59. $ make backend-dev
  60. # production
  61. $ make backend-prod
  62. ```
  63. API endpoints:
  64. * GET word list (pre-cached)
  65. * GET reduced frames (pre-cached)
  66. * POST senses
  67. * User auth logic
  68. ### Vue frontend (1 container)
  69. Relies on Flask backend.
  70. Before running `make`, you might need to set the correct api address.
  71. Check `./src/frontend_vue/config/config_prod.json`.
  72. bash
  73. ```
  74. # $ make frontend-dev # development
  75. $ make frontend-prod
  76. ```
  77. App available on: `http://0.0.0.0:8080`.
  78. ## Production deployment
  79. Prerequisite: machine with free ports 80 and 8084.
  80. ### Database
  81. Either build the database from scratch (lenghty process) using above instructions or just migrate the database from the faculty server (recommended).
  82. Build container my-mongo:
  83. ```bash
  84. # run once and destroy containers
  85. $ make database-service
  86. ```
  87. ### Backend
  88. Set database connection details in `/src/backend_flask/db_config.py`.
  89. Change 'valuser' and 'valuserpass' to the database user.
  90. ```bash
  91. mongodb://valuser:valuserpass@my_mongo/valdb
  92. ```
  93. In the above line, replace `valuser` with the username and `valuserpass` with the password that was used to create the database tables (the values were set in the root Makefile).
  94. You can also set the number of workers in `/src/backend_flask/entrypoint.sh`.
  95. In line with `gunicorn -t 4 -b 127.0.0.1:8084 app:app`, edit the `-t` parameter.
  96. Rule of thumb is 2x number of available CPU cores.
  97. Build the backend container:
  98. ```bash
  99. # From git root
  100. $ make build-backend-flask
  101. ```
  102. ### Frontend
  103. Set the server address (where backend will be runnig) in `src/frontend_vue/config/config_prod.json`.
  104. Build the `/dist` folder that contains the static app (we will be using Nginx to serve it).
  105. ```bash
  106. # From git root
  107. $ make build-frontend-prod
  108. ```
  109. All set, now run the stack.
  110. Stack configuration in `production.yaml`.
  111. ```bash
  112. # From git root
  113. $ make deploy-prod-stack
  114. ```
  115. ## Uploading a mongo dump
  116. There's a 15GB mongo dump containing the fully processed kres and ssj data.
  117. We can use that file to deploy our aplication.
  118. With this database, we will need a minimum of 8GB ram to serve the app.
  119. If the server is struggling, frontend will throw "Network errors".
  120. Check `0.0.0.0:8081` and remove (or backup) the current example database `valdb`.
  121. Run the stack with mongo port mapped:
  122. (uncomment the lines in `production.yaml`)
  123. ```yml
  124. ports:
  125. - 27017:27017
  126. ```
  127. Run a separate my-mongo container with the mounted data:
  128. ```bash
  129. $ mongo run -it --net host -v <local_dump_path>/dumps my-mongo /bin/bash
  130. ```
  131. Inside the container (edit the uesrname, password):
  132. ```bash
  133. $ mongorestore /dumps/valdb --db valdb --uri=mongodb://valuser:valuserpass@0.0.0.0:27017
  134. ```
  135. After uploading, restart the stack with `27017` commented out.