HDFS with one NameNode and no JournalNodes (no HA) for Integration Tests

[fhennig]

Druid uses HDFS for DeepStorage in its unit tests. A new HDFS instance needs to be spun up every time a test is run, and HDFS is a major contributor to the overall test duration, and also causes tests to fail occasionally because of its long startup times.

As I have learned today (thanks to Lars!), HDFS only uses a second NameNode and the JournalNodes for high availability (HA) but can also run without them. I think it would be great to have this. For an intance that only lives a few minutes and might not even see data written to it, the most bare-bones setup should be used.

[lfrancke]

It is a great idea but I'm not sure if our HDFS even supports a non-HA setup.
So we'd probably have at least one JN but only a single NN.

[maltesander]

As far as i remember:

• When creating a namenode we check if there is already an active namenode
• When creating datanodes we check that all namenode are ready or in standby mode

Ill run a quick test today.

[maltesander]

Running with nightly:
ERROR hdfs_controller: stackable_operator::logging::controller: Failed to reconcile object controller.name="hdfsclusters.hdfs.stackable.tech" error=reconciler for object HdfsCluster.v1alpha1.hdfs.stackable.tech/simple-hdfs.default failed error.sources=[Missing node role journalnode]

I removed the check here

hdfs-operator/rust/crd/src/lib.rs

Line 507 in ecacb76

} else {

But the namenode is not coming up. So this would require more investigation.

[lfrancke]

Yes, it requires a different configuration if running without HA.
I'm not sure if we even want to support that though....

[maltesander]

Yeah but it could significantly lower the test duration / resources if we can strip it down. Or switch to S3?

[lfrancke]

We should either support it or if we don't support it then we shouldn't allow invalid configurations...

[maltesander]

Yeah, i was more talking about optimizing the tests. I think we could switch to minio for deep storage in most cases?

[fhennig]

At least for druid we could, yes.

[maltesander]

At least for druid we could, yes.

Ok did not see this is the hdfs-operator repo...the deep storage stuff pointed me to druid.. :D