Multimodal sound monitoring: hydrophones, vessel tracking via AIS, radar, and camera systems

[valentina-s]

The real-time interpretation of underwater sounds can be improved by applying OSS machine learning and time-series techniques to streams of audio and, where available, the real-time output of other environmental sensors.

Quick intro to a list of steps or objectives involved:

• Watch the ML on the Edge webinar for motivation and problem articulation
• Build bioacoustic source separator detecting biological sounds within the underwater soundscape leveraging existing OSS tools, spleeter, for example
  • Train spleeter on sample musical instruments
  • Isolate orca and/or humpback sounds from the rest of the audio
  • Train spleeter on biological vs not biological
  • Repeat until we get a degree of separation with which we’re satisfied.
• Build vessel noise monitoring system
  • Expand on Val’s temporal averaging (2s, 30s, 30min) and orca, ship, boat, misc model training schemes
  • Detect (bearing, range, vessel type)
    • Via AIS - usually larger commercial vessel and some private yachts
    • Via RADAR & image – usually smaller speedboats and cruisers
  • Associate targets with underwater noise characteristics
  • Store detections in database.
• Deploy detectors and test in the cloud, federating these data sources
  • Orcasound HLS or FLAC streams
  • AIS data from AIShub and/or M2 system
    non-AIS RADAR targets from M2 system (AWS-based)
  • Pan-tilt camera visible light images of RADAR targets
• Deploy and test on the edge, e.g. via a Jetson Nano
  • Utilize local underwater audio and AIS vhf radio transmissions

Expected outcomes: Improve automated analysis of Orcasound audio data, including integration with data related to vessel noise, both in the cloud and through edge-computing.

Required skills:
Python, Tensorflow/Keras/Ketos - Machine Learning

Bonus skills:
Database, Computer vision, Cloud computing

Mentors:
Praful, Val, Scott

Difficulty level: Hard

Project Size: 350 h

Resources:

• Bioacoustic Source Separation
• Orca training data
• Orca autoencoder experiments
• https://github.com/deezer/spleeter
• https://protectedseas.net/marine-monitor-m2
• http://www.shipnoise.net/
• ShipsEar: ship noise database

Learning model architecture ideas from Praful:

1. fine-tuning the separation library to focus on marine mammal vocalization (starting with orcas) + ship noises
2. pass the ship noises to an open source version of Shazam trained with the ShipEars or other ship noise DB
   3a. pass the whale vocalizations to a species classifier similar to the one released by Google for humpback whales, but trained on orca vocalizations
   3b. pass the whale vocals to an open source version of Shazam
   3c. compare Shazam (3b) results to Google AI (3a)
3. Pass the classified whale vocal to a pod classifier
4. Work with Earth Species Project on their algorithms to tease out individuals once we know the pod (they're working on a solution for the cocktail party problem)

• for step 1 we can train Spleeter
• for step 2, 3b, 4 we can use a modified version of Shazam
• for step 3a we can train Google's CNN
• for step 4 we can also try a CNN
• for step 5 we we can see how to work with ESP as they worked on a project for cocktail party problem (edited)

The advantage is these are all open source software projects with peer-reviewed papers, so we have resources to leverage.

[TranHuuNhatHuy]

Hello @prafulfillment, @valentina-s and @scottveirs ,
I am very interested in this project and currently training the biosource separator model as well as preparing a contributor proposal. My first and foremost question is that, until now I still cannot access your Slack channel here as it keeps saying I don't have a working account. I noticed one of mentor's comments, this, regarding a Slack channel about hydrophone-nodes. Are we contributors supposed to communicate here on Github comments, or better on those Slack channels?

[TGAmbra]

Hi @valentina-s ,
I am interested in this project and I was wondering if we could discuss about it.
My name is Ambra Jin. I am currently studying at Artificial Intelligence at Queen Mary University London and in September I am pursuing my doctoral degree in the CDT AI for Health in UCL.
My first master's group project required us to segment a catheter in MRI interventional surgeries.The goal of the study was to use different CNNs to construct a 3D model of the catheter in realtime. To segment a medical image, we compared the performance of various convolutional neural networks. I handled statistical data across platforms in this project, using Matlab and Python code to simulate and train data. I additionally preprocessed all of the training data to produce false artifacts, increasing the variability of the data, and then post-processed it to get a parameterised model. In addition, in my second master's, I improved my skills by using machine learning in the context of signal processing in the Machine Learning and Computer Vision modules.
I believe my previous experience will help me ahead for this project, and I hope we can have a chat about it.
I can send my CV
Best wishes
Ambra

[valentina-s]

@TranHuuNhatHuy Can you try this link: https://orcasound.slack.com/signin? You are welcome to ask questions here, but it would be good to also join slack.

[Praful]

The @Praful copied into this thread is not the person assigned above and has nothing to do with this project! Please use the correct github handle for that person. Thanks

[TranHuuNhatHuy]

@TranHuuNhatHuy Can you try this link: https://orcasound.slack.com/signin? You are welcome to ask questions here, but it would be good to also join slack.

Thank you @valentina-s for the link, however I still cannot sign in to the Slack channel. I'm afraid that your Slack channel currently might not accept or recognize new emails as accepted collaborators. Could you please review this?

My apologies if this issue bothers you. As this is a huge project, I would like to establish a consistent communication channel so we can exchange ideas and coordinate our works efficiently, and Slack seems to be a near-optimal solution for that. For now I will use Github here until the Slack channel is available.

[TranHuuNhatHuy]

The @Praful copied into this thread is not the person assigned above and has nothing to do with this project! Please use the correct github handle for that person. Thanks

My apologies for this unexpected reference. I have corrected the handle. Thank you for pointing this out.

[Benjamintdk]

@TranHuuNhatHuy Can you try this link: https://orcasound.slack.com/signin? You are welcome to ask questions here, but it would be good to also join slack.

Thank you @valentina-s for the link, however I still cannot sign in to the Slack channel. I'm afraid that your Slack channel currently might not accept or recognize new emails as accepted collaborators. Could you please review this? My apologies if this issue bothers you. As this is a huge project, I would like to establish a consistent communication channel so we can exchange ideas and coordinate our works efficiently, and Slack seems to be a near-optimal solution for that. For now I will use Github here until the Slack channel is available.

Seems like you don't yet have a slack account. If you don't have one, you must create it first before signing in.

[TGAmbra]

Dear @Benjamintdk , I have been trying to sign in to the slack channel too. I have a slacks account, but it shows me the same message that I don't have an account in this workspace. On the other hand my other workspaces are listed below.

[scottveirs]

Hey @TranHuuNhatHuy, @TGAmbra, and @Benjamintdk here is a direct invitation to the Orcasound Slack workspace. Let me know if it doesn't work for you. If it does, please introduce yourself in the #gsoc channel if you're planning to contribute via Google Summer of Code (GSoC) in 2022, or in #general if you're not aiming at GSoC.

[TranHuuNhatHuy]

Hey @TranHuuNhatHuy, @TGAmbra, and @Benjamintdk here is a direct invitation to the Orcasound Slack workspace. Let me know if it doesn't work for you. If it does, please introduce yourself in the #gsoc channel if you're planning to contribute via Google Summer of Code (GSoC) in 2022, or in #general if you're not aiming at GSoC.

Thank you @scottveirs for this. I'm surprised that I'm not the only one experiencing difficulty in joining Slack. Maybe it's better to change the Slack channel's visibility so everyone can join freely? It's in GSOC application period so there will be tons of traffic anyway.

[TGAmbra]

Hi @prafulfillment @veirs @valentina-s @scottveirs @pmdnhd , I checked all the resources attached to the github project page of "Multimodal sound monitoring"

1. In the webinar "Edge computing for hydrophones", in the Keto software presentation, I found it very fascinating that even the river otter crunches sound are picked up, as I did not know that they share the same habitat
2. Is the bioacustic source separation document a template from which we add on for our application proposal?
3. Unfortunately I was unable to access the Orca training data because it requires the owner's email address.
4. I cloned the orca autoencoder library. I couldn't run orca autoencoder MLP.py since I didn't have the data, but I did download all of the necessary libraries and have CUDA and CUDnn operating on my laptop
5. I used the spleeter notebook from the deezer github project. It was fascinating to watch how the orchestra and the singing voice are separated. They put "!spleeter separate -h" in line 9, which showed error as they meant "--help" (?)
6. I'm pleased the marine monitor resource described what an M2 is because I was having trouble understanding it from the Keto software presentation.

[scottveirs]

I've replied to 2 of these questions (1 & 6) in Slack #gsoc channel, @TGAmbra.