How to use Vector without knowing the length

[wypb]

I am using Velox internally to implement TEXTFILE reader. I have implemented a TextFileRowReader, which inherits from dwio::common::RowReader. The approximate code is as follows:

class TextFileRowReader : public dwio::common::RowReader {
 public:
  TextFileRowReader(
      const std::shared_ptr<ReaderBase>& readerBase,
      const dwio::common::RowReaderOptions& options);
  ~TextFileRowReader() override = default;

  uint64_t next(
      uint64_t size,
      velox::VectorPtr& result,
      const dwio::common::Mutation* = nullptr) override;
}

I encountered a problem when implementing the TextFileRowReader#next method. When reading TextFile, I didn’t know how many rows of data there were in the file, so I was a little unclear on how to allocate space to result. My current implementation is as follows:

uint64_t TextFileRowReader::next(
    uint64_t size,
    velox::VectorPtr& result,
    const dwio::common::Mutation* mutation) {
  auto rowsToRead = size;

  auto rowVector = result->as<RowVector>();
  rowVector->resize(rowsToRead);

  while (rowsToRead > 0) {
    auto results = lineReader_->readLine(lineBuffer_);

    // no more rows to read
    if (!results) {
      break;
    }

    lineDeserializer_->deserialize(result, lineBuffer_, size - rowsToRead);
    rowsToRead--;
  }

  rowVector->resize(size - rowsToRead);
  return size - rowsToRead;
}

In the above implementation, I use rowVector->resize(rowsToRead); to reserve space for the vector, but the actual number of rows read is size - rowsToRead, which may be less than size.

Later, I called rowVector->resize(size - rowsToRead); to resize the vector again, so that the output result would be correct.

My question is, I called resize twice on rowVector above. Is this recommended? Or is there any other better implementation?

[aditi-pandit]

@wypb : Is size parameter to TextFileRowReader::next the max number of rows we could output ?

If yes, then your code above is fine. Though I would make it a bit more readable by naming rowsToRead to numRowsLeft.

[wypb]

Yes, size is the maximum number of lines that can be output per batch. Thank you for your reply.

[majetideepak]

@wypb It will be great if you can contribute your TextFile reader to Velox :).

[wypb]

Hi @majetideepak Deepak, I am very happy to contribute this part of the code to the community, but I think the community will not merge it, because Meta has already implemented the TEXTFILE reader internally, and they plan to open source it. I think it would be more reasonable to push meta to open source the TEXTFILE reader. @Yuhta FYI.