HDDS-10744. Standardize byte array conversion to String for LiveFileMetaData in RocksDB.

[fapifta]

What changes were proposed in this pull request?

LiveFileMetaData class in RocksDB has three methods that are returning a byte[] which we convert to String after any call.
These methods are:

• columnFamilyName()
• smallestKey()
• largestKey()

We use 3 different conversion to String for the returned byte arrays.
For largestKey and smallestKey we use FixedLengthStringCodec.bytes2String and new String(byte[], UTF_8)
For columnFamilyName we use org.apache.hadoop.hdds.StringUtils.bytes2String, new String(byte[], UTF_8), and org.bouncycastle.util.Strings.fromByteArray.

From these methods, FixedLengthStringCodec throws an exception if the conversion can not be done, and it uses ISO_8859_1 as the charset for the conversion, while the rest uses UTF_8 charset for the conversion, and replaces the characters that UTF-8 can not represent.

Based on how and where we use these it seems to be safe to settle on UTF-8 as the target charset, and use StringUtils.bytes2String from our own utilities which uses the String constructor as of now by the way.

Removing org.bouncycastle.util.Strings usage is also beneficial for crypto compliance related development.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-10744

How was this patch tested?

CI

[adoroszlai]

@fapifta the same conversion appears in another class, should we remove that, too?

ozone/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/metadata/DatanodeStoreSchemaThreeImpl.java

Line 33 in dfe1ea5

import org.bouncycastle.util.Strings;

ozone/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/metadata/DatanodeStoreSchemaThreeImpl.java

Line 154 in dfe1ea5

String cf = Strings.fromByteArray(file.columnFamilyName());

Could use Ozone's StringUtils, too:

ozone/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/StringUtils.java

Lines 99 to 101 in dfe1ea5

	public static String bytes2String(byte[] bytes) {
	return bytes2String(bytes, 0, bytes.length);
	}

ozone/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/StringUtils.java

Lines 61 to 63 in dfe1ea5

	public static String bytes2String(byte[] bytes, int offset, int length) {
	return new String(bytes, offset, length, UTF8);
	}

[fapifta]

Hmm.. thank you for spotting this @adoroszlai, my IDE made fun of me... I searched for all occurance of bouncycastle and it seems it does not show just the first n results...

On the other hand, I spotted a few other things that I might fix together to standardize on one method for this conversion. I will post a second version shortly.

[adoroszlai]

Thanks @fapifta for updating the patch.

[Galsza]

@fapifta Thanks for the change it's looking good to me after Attila's recommendations.

[szetszwo]

Removing org.bouncycastle.util.Strings usage is also beneficial for crypto compliance related development.

Agree.

... FixedLengthStringCodec throws an exception if the conversion can not be done, and it uses ISO_8859_1 as the charset for the conversion, while the rest uses UTF_8 charset for the conversion ...

If we change the charset, how the new code read the existing DB?

[adoroszlai]

Thanks @fapifta for updating the patch, LGTM.

[fapifta]

Thank you @adoroszlai for the review, let's see if @szetszwo is also ok with this patch, or at least give him a chance to react for some time.
So what I did, I just reverted the unrelated formatting change, and the change in compactionIfNeeded, so that it is using the old way with ISO-8859-1 conversion.

In the meantime I have added HDDS-10762 as I have some performance concerns, also I kind of have technical concerns around this, as in an other place the same smallestKey and largetKey result is converted using UTF-8 conversion, and this was the original state also, so we have two code parts that handles this same data differently... But it is out of my knowledge area, and I can not spend more time on to understand this as of now.

With that this patch should preserve the old behaviour, using the same formats and charsets for the conversion, but via the same method call instead of using different utilities at different places.

[adoroszlai]

Thanks @fapifta for the patch, @Galsza, @szetszwo for the review.