4 Commits

Author	SHA1	Message	Date
Ozbolt Menegatti	f0109771aa	chunk size now handled in file-sentence-generator	2019-06-16 00:59:44 +02:00
Ozbolt Menegatti	0d8aeb2282	load_files now returns a generator of senteces, not a generator of the whole file ... This makes it much slower, but more adaptable for huge files.	2019-06-15 22:30:43 +02:00
Ozbolt Menegatti	a8183cf507	word stats now collected more memory-efficient	2019-06-15 22:20:20 +02:00
Ozbolt Menegatti	90dbbca5d5	HUGE refactor, creating lots of modules, no code changes though!	2019-06-15 18:55:35 +02:00