Gradle task localizationUpdate does not work

[tobiasdiez]

localizationUpdate leads to resorting and adding of many language keys due to the recent changes. Thus, at the moment it is not possible to remove a pair from the english file and propagate this change to the other languages.

[koppor]

I think, I can work on that in January. Meanwhile, you should be able to log in at crowdin and remove the key there. Need to make you maintainer there, though.

[koppor]

Please DO NOT remove obsolete keys in other language files. This is done by crowdin automatically: #3585

I will remove the task completely or have it output that there is no need for a check, because of crowdin!

[tobiasdiez]

@koppor Right now we have tests in place that require to remove obsolete language keys from other languages. It would be nice if we could remove the localizationUpdate step completely (less things to do manually), but then the tests should be adopted accordingly.

[koppor]

I thought, I adapted the tests. Seems, I missed something. Have to check... Am 05.01.2018 12:01 schrieb "Tobias Diez" <notifications@github.com>:

…

@koppor </~https://github.com/koppor> Right now we have tests in place that require to remove obsolete language keys from other languages. It would be nice if we could remove the localizationUpdate step completely (less things to do manually), but then the tests should be adopted accordingly. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3573 (comment)>, or mute the thread </~https://github.com/notifications/unsubscribe-auth/ABTaftbvUCzp6PiQbYoCZPnhgV_ph9zcks5tHgECgaJpZM4RLq3L> .

[tobiasdiez]

This test fails if you remove language keys from the english file but not from the others:

jabref/src/test/java/org/jabref/logic/l10n/LocalizationConsistencyTest.java

Lines 50 to 51 in 3b2d35f

	@Test
	public void nonEnglishFilesMustHaveSubsetOfKeys() {

(If crowdin handles this case automatically, then there is no need for this test, right?)

A few other tests still refer to the localizationUpdate (which should not be invoked, right?), e.g.,

jabref/src/test/java/org/jabref/logic/l10n/LocalizationConsistencyTest.java

Lines 140 to 142 in 3b2d35f

	assertEquals("DETECTED LANGUAGE KEYS WHICH ARE NOT IN THE ENGLISH LANGUAGE FILE\n" +
	"1. PASTE THESE INTO THE ENGLISH LANGUAGE FILE\n" +
	"2. EXECUTE: gradlew localizationUpdate\n" +

[koppor]

It is not as straight-forward as I thought.

English escapes more characters than non-English:

Executing\ command\ \"%0\"...=Executing command \"%0\"...

German:

Executing\ command\ "%0"...=Ausführung des Kommandos "%0"...

[Siedlerchr]

Any updates on this? I think we don't need the script anylonger or?

[koppor]

An admin might still need it. Quoting: #3635 (comment)

That task won't be necessary for all contributors, only for maintainers wanting to cleanup the files (like sorting, ...). However, this is not absolutely necessary anymore since Crowdin basically takes care.

I think, I just close it and put an "on-hold" label on it.

[koppor]

Maybe we need to rework the quotes in the English file.

[Brainsucker92]

Finally I found some time to take a closer look at the syncLang.py script. To be honest, it's pretty difficult to determine the exact origin of this problem. My very first thought was that there might be a problem with different character encodings. The script always assumes the content of the property files is written in UTF-8 encoding. But Eclipse is using ISO-8859-1 encoding by default which is determined by the content type. After some more research within the script I found some really nasty little mistake, if you have a closer look at lines 52 and 53:

jabref/scripts/syncLang.py

Lines 52 to 53 in 01d3d4b

	duplicates.append(u"{key}={value}".format(key=key, value=value))
	translation_in_list = "u{key}={value}".format(key=key, value=keys_checked[key])

I'm not 100% certain if this will fix the whole problem but we're going to find out real soon, I guess.

I pushed the fix for this issue here:
/~https://github.com/Brainsucker92/jabref/commit/ea5d56e4a7a8bcb25c2bb00dc3651c807bc9739e

Additionally I added a new class that allows the script to locate it's position within the repository.
This might be helpful at some point.

UPDATE:

I DID IT! I found a logical issue deep within the merging process of the files. Have a look here:

jabref/scripts/syncLang.py

Lines 307 to 308 in 01d3d4b

	for missing_key in keys_missing:
	keys[missing_key] = ""

Missing keys are added as empty Strings to the dictionary... So far so good. But then:

jabref/scripts/syncLang.py

Lines 314 to 321 in 01d3d4b

	for line in main_lines:
	key = main_keys.key_from_line(line)
	if key is not None:
	# Do not write empty keys
	if keys[key] != "":
	other_lines_to_write.append(u"{key}={value}\n".format(key=key, value=keys[key]))
	else:
	other_lines_to_write.append(line)

Keys that contain empty Strings are not written to the new properties file which leads to missing keys over and over again. I simply removed the if and everything worked perfectly fine.

New commit:
/~https://github.com/Brainsucker92/jabref/commit/ba7d87d2415c54992952b89cdc4673194a5ee929