Demultiplexed FASTQ support

[mtomko]

No description provided.

[tmgreen]

The length limit for a unix command line is 4096 chars, which is starting to look achievable with this new demuxed input mode. I guess that is all the more reason to switch to a JSON config file--was that just for pq-launcher or for pq as well?

[mtomko]

I wasn't planning to implement it in PoolQ in this PR. I think that is how PoolQ4 will work. In our case, the poolq launcher constructs the config object directly from its command line, but I'm planning to change the poolq launcher command line to take only 2 parameters: --db and --job-id, which it will use to read everything it needs from the database, obviating the need for command lines that might exceed the limit.

[tmgreen]

Looks great. I have one performance-related comment/question, and then 🧼 🧇 🦕

[tmgreen]

PQ4 might be a good time to make the switch from Array[Char] to Array[Byte] for all sequence stuff... With COMPACT_STRINGS under the hood it might be a lot faster.

[mtomko]

I don't know about COMPACT_STRINGS. I had just been agonizing over the decision to between String and Array[Char] in PQ4 and had arrived at the conclusion that using Array[Char] was going to be a huge hassle due to how Scala/Java treat arrays (reference equality, no hash codes - to gain those, I'd basically have to re-implement String, I think). I may be able traffick in Array[Byte] and convert to String when it comes time to look up things in the Reference though.

[mtomko]

Another PQ4 thought, I'm currently getting FASTQ records as Array[String] (4 elements in the array, 1 per line). It would be interesting to read inte Array[Array[Byte]] but I'd have to implement a lot of the low level processing on my own then. It might be a worthwhile exercise but it's definitely dipping into I/O stuff that I don't have a lot of experience with.