 Ozbolt Menegatti
|
9e8cd2a2ec
|
Issue #1000
|
4 years ago |
|
Ozbolt Menegatti
|
1d4c0238a6
|
fixing how min_freq is used and more verbose writer
|
5 years ago |
|
Ozbolt Menegatti
|
8fee3f8a8e
|
Testing delayed insertions of representations
|
5 years ago |
|
Ozbolt Menegatti
|
6bb3586051
|
Attempt at speed optimization with sql-join
|
5 years ago |
|
Ozbolt Menegatti
|
4124036474
|
match_num now loaded from database
and --keep-db deprecated in favour of --new-db (harder for me to fu*k up)
|
5 years ago |
|
Ozbolt Menegatti
|
07242f74c8
|
Also remember representations step.
|
5 years ago |
|
Ozbolt Menegatti
|
33528f1495
|
step_done now implemented in database.py
|
5 years ago |
|
Ozbolt Menegatti
|
3ea62ed242
|
dispersions now loaded into database and stored/loaded.
|
5 years ago |
|
Ozbolt Menegatti
|
dedc031696
|
Step recorded: generate_renders
|
5 years ago |
|
Ozbolt Menegatti
|
046aef031f
|
adding timeinfo
|
5 years ago |
|
Ozbolt Menegatti
|
2018745d52
|
files loaded now in database
|
5 years ago |
|
Ozbolt Menegatti
|
8cca761b91
|
min frequecy now part of writer
|
5 years ago |
|
Ozbolt Menegatti
|
3f1c154705
|
can now load csv files
|
5 years ago |
|
Ozbolt Menegatti
|
d497749c78
|
better database commiting
|
5 years ago |
|
Ozbolt Menegatti
|
b25e3de76b
|
adding total keyword to progress and total time spent
|
5 years ago |
|
Ozbolt Menegatti
|
771547b7e4
|
progress for dispersions
|
5 years ago |
|
Ozbolt Menegatti
|
f9bfac6430
|
If no output, then just commit stuff to database and exit.
|
5 years ago |
|
Ozbolt Menegatti
|
ec02242f47
|
num-words now part of database
|
5 years ago |
|
Ozbolt Menegatti
|
ea92b44d71
|
Removing parallel stuff
|
5 years ago |
|
Ozbolt Menegatti
|
d771137dc7
|
removing pickled structures
|
5 years ago |
|
Ozbolt Menegatti
|
a07d14011d
|
simplifying progress, because I will remove the parallel stuff
|
5 years ago |
|
Ozbolt Menegatti
|
577983427e
|
Better error reporting in parsing syntactic structures
|
5 years ago |
|
Ozbolt Menegatti
|
48795c6227
|
common msd now calculated per colocation id and not for whole corpus
|
5 years ago |
|
Ozbolt Menegatti
|
2f789e6550
|
last agreement now confirms some matches even if not all matches are ok
|
5 years ago |
|
Ozbolt Menegatti
|
1401b82324
|
Adding msd to out formatter
|
5 years ago |
|
Ozbolt Menegatti
|
47340fe80c
|
common msd now based on (lemma,msd0) not only lemma #757-127
|
5 years ago |
|
Ozbolt Menegatti
|
8c20295adf
|
Adding dispersions to sqlite, finished moving to it.
|
5 years ago |
|
Ozbolt Menegatti
|
b5e281bdf4
|
adding indexes for speed and set_representations via database
|
5 years ago |
|
Ozbolt Menegatti
|
188763c06a
|
Incorporating database also in MatchStore
|
5 years ago |
|
Ozbolt Menegatti
|
c25844a335
|
adding separate database class
|
5 years ago |
|
Ozbolt Menegatti
|
fa8a5e55f8
|
Merge branch 'sqlite'
|
5 years ago |
|
Ozbolt Menegatti
|
c2c2ce7ff8
|
making sorted words sorted a bit more non-randomly.
|
5 years ago |
|
Ozbolt Menegatti
|
8b06c4ec38
|
Skipping already used abailable words, stupid refactoring bug
|
5 years ago |
|
Ozbolt Menegatti
|
11706b6f81
|
word stats on sqlite now, not yet really working.
|
5 years ago |
|
Ozbolt Menegatti
|
1256a4de40
|
Fixing loading bad gz files and progress showing
|
5 years ago |
|
Ozbolt Menegatti
|
049f5ca3dc
|
Adding new N* msds
|
5 years ago |
|
Ozbolt Menegatti
|
cfdb36b894
|
Adding ability to load gz files.
|
5 years ago |
|
Ozbolt Menegatti
|
d2f6f8dac8
|
adding new Nw msd
|
5 years ago |
|
Ozbolt Menegatti
|
70b05e8637
|
New progress bar
|
5 years ago |
|
Ozbolt Menegatti
|
3552f14b81
|
Loader to its own module
|
5 years ago |
|
Ozbolt Menegatti
|
51cf3e7064
|
Improving debugging ouptut
|
5 years ago |
|
Ozbolt Menegatti
|
dc285ce265
|
Saving memory in word-stats
|
5 years ago |
|
Ozbolt Menegatti
|
37acabc076
|
able to load pickled structures
|
5 years ago |
|
Ozbolt Menegatti
|
f0109771aa
|
chunk size now handled in file-sentence-generator
|
5 years ago |
|
Ozbolt Menegatti
|
0d8aeb2282
|
load_files now returns a generator of senteces, not a generator of the whole file
This makes it much slower, but more adaptable for huge files.
|
5 years ago |
|
Ozbolt Menegatti
|
a8183cf507
|
word stats now collected more memory-efficient
|
5 years ago |
|
Ozbolt Menegatti
|
90dbbca5d5
|
HUGE refactor, creating lots of modules, no code changes though!
|
5 years ago |
|
Ozbolt Menegatti
|
43c6c9151b
|
Simplifying and also improving the speed (less regex comparisons!)
|
5 years ago |
|
Ozbolt Menegatti
|
09bdd0fe3f
|
Adding gitignore
|
5 years ago |
|
Ozbolt Menegatti
|
c0939fbbd4
|
fixed performance bug for representations
No more creating millions of namedtuple classes. Works about 15x faster
|
5 years ago |
