Fix (pogostostpojavljanja is now array) for importing term lists, new version.

README.md

@@ -1 +1,53 @@
-# RSDO (Razvoj Slovenščine v Digitalnem Okolju) - Concordancer
+# RSDO (Razvoj Slovenščine v Digitalnem Okolju) - Concordancer
+This repository contains the source code of the concordancer which is a part of the [Terminology Portal](https://github.com/clarinsi/rsdo_term_portal).
+
+There are two main parts:
+- Web application with backend (API interface) written in ASP.NET Core 6 and frontend written in React
+- System Manager which is written in .NET 6 and is used to initially set up the system
+
+## Prerequisites
+Web application requires following external dependencies
+- OpenSearch v2.3.0 (use docker image: `opensearchproject/opensearch:2.3.0`)
+- PostgreSQL (use docker image: `postgres`)
+
+## Building source code
+### Frontend (React application)
+To build a frontend application successfully, first, [Node.js](https://nodejs.org/en/) must be installed (at least version 16).
+Use `npm install` to install all required dependencies, and `npm run build` to build the application.
+For more info see the [Readme](src/apps/Rsdo.Concordancer.Api/WebApp/README.md) inside the frontend [root](src/apps/Rsdo.Concordancer.Api/WebApp) folder.
+
+### Backend (ASP.NET Core 6) and System Manager (.NET 6)
+Open [Rsdo.Concordancer.sln](src/Rsdo.Concordancer.sln) inside of your IDE and build the solution, or use `dotnet build` command to build the solution. All required dependencies will be installed via NuGet packages.
+
+## Creating docker images
+There are two Dockerfile's in the source repository for each part. Both images should be compiled within the context of the project root folder.
+
+### Web application
+[Dockerfile](src/apps/Rsdo.Concordancer.Api/Dockerfile) for Web Application accepts additional optional argument (WEBAPP_PUBLIC_URL) which is mapped to React [PUBLIC_URL](https://create-react-app.dev/docs/using-the-public-folder/) variable. If the variable is omited, then application should be deployed on a domain root.
+
+`docker build -t rsdo-concordancer-api -f src/apps/Rsdo.Concordancer.Api/Dockerfile .`
+
+`docker build --build-arg WEBAPP_PUBLIC_URL=/corpus -t rsdo-concordancer-api -f src/apps/Rsdo.Concordancer.Api/Dockerfile .`
+
+### System Manager
+Use following [Dockerfile](src/apps/Rsdo.Concordancer.SystemManager/Dockerfile) to build System Manager.
+
+`docker build -t rsdo-concordancer-systemmanager -f src/apps/Rsdo.Concordancer.SystemManager/Dockerfile .`
+
+## Environment variables
+
+### Web application
+Web application must be configured with the following Environment variables and values:
+- `RSDO:Database:MasterConnectionString` holds connection string to the main database; e.g.: `rsdo_db;Username=rsdo;Password=strongpassword;Database=Concordancer`
+- `RSDO:Elastic:ConnectionString` holds connection string to OpenSearch server; e.g.: `http://rsdo_opensearch:9200`
+- `RSDO:Tokenizer:ClasslaUrl` holds and url to external service which is used for tokenizing search strings; e.g.: `https://orodja.cjvt.si/oznacevalnik/ajax_api/v1/slv/process`
+
+### System Manager
+System manager must be configured with the following Environment variables and values:
+- `RSDO:Database:MasterConnectionString` holds connection string to the main database; e.g.: `rsdo_db;Username=rsdo;Password=strongpassword;Database=Concordancer`
+- `RSDO:Elastic:ConnectionString` holds connection string to OpenSearch server; e.g.: `http://rsdo_opensearch:9200`
+
+## Using System Manager
+To initialy setup the system, you have to use System Manager and run following commands:
+1. `dotnet Rsdo.Concordancer.SystemManager.dll createMasterDb --connectionString="Host=rsdo_db;Username=rsdo;Password=strongpassword;Database=postgres"` which will create master database with the name Concordancer. You can select different name with the `--name` option
+2. `dotnet Rsdo.Concordancer.SystemManager.dll importSloleks --sourceFile=/data/Sloleks2.0.LMF/sloleks_clarin_2.0.xml` which will import Sloleks (which is used for better searching). XML file is not part of this repository, but it can be downloaded from the following [URL](https://www.clarin.si/repository/xmlui/bitstream/handle/11356/1230/Sloleks2.0.LMF.zip?sequence=3&isAllowed=y)

src/apps/Rsdo.Concordancer.Api/Rsdo.Concordancer.Api.csproj

@@ -4,7 +4,7 @@
         <TargetFramework>net6.0</TargetFramework>
         <GenerateDocumentationFile>true</GenerateDocumentationFile>
         <EnableNETAnalyzers>false</EnableNETAnalyzers>
-        <Version>1.0.1</Version>
+        <Version>1.0.2</Version>
     </PropertyGroup>
 
     <ItemGroup>

src/apps/Rsdo.Concordancer.SystemManager/Rsdo.Concordancer.SystemManager.csproj

@@ -5,7 +5,7 @@
         <TargetFramework>net6.0</TargetFramework>
         <GenerateDocumentationFile>true</GenerateDocumentationFile>
         <EnableNETAnalyzers>false</EnableNETAnalyzers>
-        <Version>1.0.1</Version>
+        <Version>1.0.2</Version>
     </PropertyGroup>
 
     <ItemGroup>

src/core/Rsdo.Concordancer.Core/Rsdo.Concordancer.Core.csproj

@@ -4,7 +4,7 @@
         <TargetFramework>net6.0</TargetFramework>
         <GenerateDocumentationFile>true</GenerateDocumentationFile>
         <EnableNETAnalyzers>false</EnableNETAnalyzers>
-        <Version>1.0.1</Version>
+        <Version>1.0.2</Version>
     </PropertyGroup>
 
     <ItemGroup>

src/core/Rsdo.Concordancer.Data/Rsdo.Concordancer.Data.csproj

@@ -4,7 +4,7 @@
         <TargetFramework>net6.0</TargetFramework>
         <GenerateDocumentationFile>true</GenerateDocumentationFile>
         <EnableNETAnalyzers>false</EnableNETAnalyzers>
-        <Version>1.0.1</Version>
+        <Version>1.0.2</Version>
     </PropertyGroup>
 
     <ItemGroup>

src/core/Rsdo.Concordancer.Infrastructure/Rsdo.Concordancer.Infrastructure.csproj

@@ -4,7 +4,7 @@
         <TargetFramework>net6.0</TargetFramework>
         <GenerateDocumentationFile>true</GenerateDocumentationFile>
         <EnableNETAnalyzers>false</EnableNETAnalyzers>
-        <Version>1.0.1</Version>
+        <Version>1.0.2</Version>
     </PropertyGroup>
 
     <ItemGroup>

src/core/Rsdo.Concordancer.Infrastructure/Search/AddRecordsHandlers/BaseAddRecordsHandler.cs

@@ -1,5 +1,6 @@
 using System;
 using System.Collections.Generic;
+using System.Linq;
 using System.Threading.Tasks;
 using OpenSearch.Client;
 using Rsdo.Concordancer.Infrastructure.Search.Extensions;
@@ -27,23 +28,27 @@ public abstract class BaseAddRecordsHandler<TEntity, TElasticEntity> : IAddRecor
         // Get index name
         var indexName = indexProvider.IndexName;
 
-        // Create request
+        // Batch entities by 1000
-        var request = new BulkRequest(indexName)
+        foreach (var batch in entities.Chunk(1000))
         {
-            Operations = new List<IBulkOperation>(),
+            // Create request
-            Timeout = TimeSpan.FromMinutes(5),
+            var request = new BulkRequest(indexName)
-        };
+            {
+                Operations = new List<IBulkOperation>(),
+                Timeout = TimeSpan.FromMinutes(5),
+            };
 
-        // Convert entities to elastic entities
+            // Convert entities to elastic entities
-        foreach (var entity in entities)
+            foreach (var entity in batch)
-        {
+            {
-            var elasticEntity = ConvertEntity(entity);
+                var elasticEntity = ConvertEntity(entity);
-            request.Operations.Add(new BulkIndexOperation<TElasticEntity>(elasticEntity));
+                request.Operations.Add(new BulkIndexOperation<TElasticEntity>(elasticEntity));
+            }
+
+            // Get response
+            var response = await client.BulkAsync(request);
+            response.ThrowIfInvalid();
         }
-
-        // Get response
-        var response = await client.BulkAsync(request);
-        response.ThrowIfInvalid();
     }
 
     protected abstract TElasticEntity ConvertEntity(TEntity entity);

src/core/Rsdo.Concordancer.ServiceModel/Rsdo.Concordancer.ServiceModel.csproj

@@ -4,7 +4,7 @@
         <TargetFramework>net6.0</TargetFramework>
         <GenerateDocumentationFile>true</GenerateDocumentationFile>
         <EnableNETAnalyzers>false</EnableNETAnalyzers>
-        <Version>1.0.1</Version>
+        <Version>1.0.2</Version>
     </PropertyGroup>
 
     <ItemGroup>

src/core/Rsdo.Concordancer.Services/RequestHandlers/TermLists/ImportTermListHandler.cs

@@ -109,7 +109,7 @@ public class ImportTermListHandler : IRequestHandler<ImportTermList, ExecutionRe
                         Lemma = termObj["kandidat"].Value<string>(),
                         Msd = termObj["POSoznake"].Value<string>(),
                         Weight = termObj["ranking"].Value<decimal>(),
-                        Frequency = termObj["pogostostpojavljanja"].Value<int>(),
+                        Frequency = termObj["pogostostpojavljanja"].Values<int>().First(),
                         TermListAutoId = termList.AutoId,
                     }.ApplyCreateValues();
 

src/core/Rsdo.Concordancer.Services/RequestHandlers/Texts/IndexTextHandler.cs

@@ -90,39 +90,35 @@ public class IndexTextHandler : IRequestHandler<IndexText, ExecutionResult>
 
     private static List<Concordance> GetConcordances(Text text, Paragraph paragraph, List<Token> tokens)
     {
-        // Get window size (max 10)
-        var windowSize = Math.Min(tokens.Count - 1, 10);
-
-        // Create list of tokens with positions
-        var tokensIdx = tokens.Select((t, i) => new KeyValuePair<int, Token>(i, t)).ToList();
-
-        // Get tokens which will appear in current concordance
-        List<KeyValuePair<int, Token>> concordanceTokens;
-
         var concordances = new List<Concordance>();
-        while ((concordanceTokens = tokensIdx.Where(x => x.Key >= -windowSize && x.Key <= windowSize).ToList().OrderBy(x => x.Key).ToList()).Any())
+        for (var i = 0; i < tokens.Count; i++)
         {
-            if (concordanceTokens.Last().Key < 0)
-            {
-                break;
-            }
-
             var concordance = new Concordance()
             {
                 ParagraphId = paragraph.Id,
                 TextId = text.Id,
             };
 
-            // Loop through tokens and set it in the position
+            // set center token
-            foreach (var concordanceToken in concordanceTokens)
+            concordance.SetToken(tokens[i], 0);
+
+            // set left and right context tokens
+            for (var c = 1; c <= 10; c++)
             {
-                concordance.SetToken(concordanceToken.Value, concordanceToken.Key);
+                // left context
+                if (i - c >= 0)
+                {
+                    concordance.SetToken(tokens[i - c], -c);
+                }
+
+                // right context
+                if (i + c < tokens.Count)
+                {
+                    concordance.SetToken(tokens[i + c], c);
+                }
             }
 
             concordances.Add(concordance);
-
-            // Decrease indexes of tokens (shift tokens to left, relative to window size)
-            tokensIdx = tokensIdx.Select(x => new KeyValuePair<int, Token>(x.Key - 1, x.Value)).ToList();
         }
 
         return concordances;

src/core/Rsdo.Concordancer.Services/Rsdo.Concordancer.Services.csproj

@@ -4,7 +4,7 @@
         <TargetFramework>net6.0</TargetFramework>
         <GenerateDocumentationFile>true</GenerateDocumentationFile>
         <EnableNETAnalyzers>false</EnableNETAnalyzers>
-        <Version>1.0.1</Version>
+        <Version>1.0.2</Version>
     </PropertyGroup>
 
     <ItemGroup>