error tensorflow on ubuntu

[sarahmermet]

Hi

Running the latest version of aradeepopsis with the latest version of nextflow (version 22.10.6), I get the following tensorflow error.
Can you help me?

Error executing process > 'ARADEEPOPSIS:SEGMENT:RECORDS (1)'

Caused by:
  Process `ARADEEPOPSIS:SEGMENT:RECORDS (1)` terminated with an error exit status (1)

Command executed:

  #!/usr/bin/env python

  import logging
  import os
  import pickle

  import tensorflow as tf

  from data_record import create_record

  logger = tf.get_logger()
  logger.propagate = False
  logger.setLevel('INFO')

  images = tf.io.gfile.glob('images/*')

  count = len(images)
  invalid = 0
  scale_factors = {}

  with tf.io.TFRecordWriter('chunk.tfrecord') as writer:
      for i in range(count):
          filename = os.path.basename(images[i])
          image_data = tf.io.gfile.GFile(images[i], 'rb').read()
          try:
              image = tf.io.decode_image(image_data, channels=3)
          except tf.errors.InvalidArgumentError:
              logger.info("%s is either corrupted or not a supported image format" % filename)
              invalid += 1
              with open("invalid.txt", "a") as broken:
                  broken.write(f'{filename}\n')
              continue

          height, width = image.shape[:2]
          ratio = 1.0

          if max(height,width) > 602:
              logger.info('%s: dimensions %d x %d are too large,' % (filename, height, width))
              ratio = max(height,width)/602
              new_height, new_width = int(height/ratio), int(width/ratio)
              logger.info('%s: resized to %d x %d (scale factor:%f)' % (filename, new_height, new_width, ratio))
              image = tf.image.resize(image, size=[new_height,new_width], preserve_aspect_ratio=False, antialias=True)
              image_data = tf.image.encode_png(tf.cast(image, tf.uint8)).numpy()
              tf.io.write_file(os.path.join(f'images/{filename}'), image_data)

          scale_factors[filename] = ratio
          record = create_record(image_data=image_data,
                              filename=filename,
executor >  local (9)
[fc/63fa6d] process > ARADEEPOPSIS:SEGMENT:RECORDS (10) [ 60%] 12 of 20, failed: 12
[-        ] process > ARADEEPOPSIS:SEGMENT:MODEL        -
[-        ] process > ARADEEPOPSIS:TRAITS:MEASURE       -
[-        ] process > ARADEEPOPSIS:QC:SUMMARY           -
[-        ] process > ARADEEPOPSIS:QC:SHINY             -
Error executing process > 'ARADEEPOPSIS:SEGMENT:RECORDS (1)'

Caused by:
  Process `ARADEEPOPSIS:SEGMENT:RECORDS (1)` terminated with an error exit status (1)

Command executed:

  #!/usr/bin/env python

  import logging
  import os
  import pickle

  import tensorflow as tf

  from data_record import create_record

  logger = tf.get_logger()
  logger.propagate = False
  logger.setLevel('INFO')

  images = tf.io.gfile.glob('images/*')

  count = len(images)
  invalid = 0
  scale_factors = {}

  with tf.io.TFRecordWriter('chunk.tfrecord') as writer:
      for i in range(count):
          filename = os.path.basename(images[i])
          image_data = tf.io.gfile.GFile(images[i], 'rb').read()
          try:
              image = tf.io.decode_image(image_data, channels=3)
          except tf.errors.InvalidArgumentError:
              logger.info("%s is either corrupted or not a supported image format" % filename)
              invalid += 1
              with open("invalid.txt", "a") as broken:
                  broken.write(f'{filename}\n')
              continue

          height, width = image.shape[:2]
          ratio = 1.0

          if max(height,width) > 602:
              logger.info('%s: dimensions %d x %d are too large,' % (filename, height, width))
              ratio = max(height,width)/602
              new_height, new_width = int(height/ratio), int(width/ratio)
              logger.info('%s: resized to %d x %d (scale factor:%f)' % (filename, new_height, new_width, ratio))
              image = tf.image.resize(image, size=[new_height,new_width], preserve_aspect_ratio=False, antialias=True)
              image_data = tf.image.encode_png(tf.cast(image, tf.uint8)).numpy()
              tf.io.write_file(os.path.join(f'images/{filename}'), image_data)

          scale_factors[filename] = ratio
          record = create_record(image_data=image_data,
                              filename=filename,
                              height=height,
                              width=width,
                              ratio=ratio)

          writer.write(record.SerializeToString())

  pickle.dump(scale_factors, open("ratios.p", "wb"))

Command exit status:
  1

Command output:
  (empty)

Command error:
  Traceback (most recent call last):
    File ".command.sh", line 7, in <module>
      import tensorflow as tf
  ModuleNotFoundError: No module named 'tensorflow'

Work dir:
  /root/work/8a/b5ca9564a0919906763a4579ab128a

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

[phue]

HI @sarahmermet,

it appears that it cannot find the tensorflow installation. Which profile did you use to run it? I recommend using docker.

[sarahmermet]

Hi

I used conda and I tried it on different computers.

[phue]

Could you try with docker, to see if it works? Otherwise, please attach the .nextflow.log file so I can take a look what is happening

[sarahmermet]

Sorry for the delay, I tried to use docker but the problem is that I am under windows and so it is not possible to use both docker desktop and nextflow since nextflow requires to use linux, unless I am wrong.
Here is the .nextflow.log file

nextflow.log.txt

[phue]

It should be possible to run Docker within the WSL on Windows, see here: https://docs.docker.com/desktop/windows/wsl/

But back to your original problem: I think there is also no conda installation in your WSL environment where you are trying to run the pipeline. Could you try installing conda first?