Alice Potts
Minwoo Kim
Filippo Sanzeni
Davin Browner Conaty
John Innes Centre:
Eavesdropping on Nature: DIY Bioacoustics is a project focussed on the fruitful entanglement of design, science, sound and the public sphere. Our goals are to advance both design and science by “thinking about the future of science in the context of design–as well as design in the context of science” (Ito, 2016) and to prototype the process in a way which is in accordance with open source and DIY methodologies.
We are developing an open source and DIY sensor/service for biologists, using sound recordings to identify and track different species of leafhoppers through the different calls they make in order to monitor crop health. The sensor could also be utilised by citizen scientists, farmers and visual artists/computational designers.
We will be developing an open source and DIY sensor/service for biologists, using sound recordings to identify and track different species of leafhoppers, to monitor crop health remotely. The sensor/service could also be utilised by citizen scientists, farmers and visual artists/computational designers.
Non-invasive bioacoustic monitoring has become an increasingly effective way of monitoring ecosystem diversity and health. Bioacoustics paired with machine learning has been cited as an effective way of automatically identifying animals such as frogs (Xie, 2017), birds (Zhao et al, 2017) and fish (Sattar et al, 2016) amongst other animals. Bioacoustics is an area of scientific research which would benefit from (i) continued expansion of machine learning and automated identification of insect species (ii) creation of open source hardware for conducting research. Our aim is to contribute to (i) by applying bioacoustics and machine learning to insect recognition and to (ii) by creating an open source, diy and hackable acoustic sensor for identification of various insect species.
Recent work on insect recognition and intelligent traps is seen in (Silva et al, 2014) and on methods of creating low cost sensors for insect recognition in (Silva et al, 2015). Problems with ambient noise in traditional acoustic recording are identified in (Chen et al, 2014) and they show how low cost optical sensors provide more accuracy and high data capacity than previous methods. We hope to build on this work by creating an innovative open source model and associated hardware for conducting research into insect ecosystems.
Insects are vectors of diseases while also pollinating a large proportion of the world’s food production. Further to this, they also constitute a growing food market which is expected to be worth 55 billion dollars by 2023. (Global Market Insight, 2016).
Our aim is to contribute to ongoing research into insect recognition and ecosystems by applying bioacoustics and machine learning to insect recognition and to the democratisation of scientific research by creating an open source, diy and hackable acoustic sensor for identification of various insect species. The final sensor would be non-invasive, weatherproof and wireless and would link to a dashboard, displaying visually the patterns and statistics gathered which could eventually be linked to and open data structure such as wiki data for sharing information about various environments throughout the world. In the first phase of development we will be using already available kits on the market for rapid prototyping and proof of concept. The second step will be creating a custom PCB which would fit all of the necessary components and circuitry. On the software side, we will be relying on open source projects, such as SuperCollider. We hope that our research will contribute to the development of the use of bioacoustics in the study of insect ecosystems while also furthering the democratisation of science.
Human interpretations of insects are subject to what (Whaley, 2006) calls the deception of dissimulation (the hiding of the real). They are interpreted as swarm-like templates for technologies or as having a kind of language for example. This is the product of a constant human impulse to interpret animal behaviour on terms which are already intelligible to us. We cannot move outside of this intelligibility and as a result, have a tendency to create approximations of identity through myth or compromise. These compromises are a kind of useful fake.
We are interested in a particular myth that persisted for many years about the Homoptera, Cicadellidae or “Leafhopper” which was dispelled by the research of Frej Ossiannilsson in the 1940s (Ossiannilsson, 1946 and 1949). Leafhoppers were consistently classified as being “Muettes” meaning the “dumb ones” or “Silentia” meaning “silent, stillness, quiet, noiseless”. In contrast, their larger cousins the Cicada were “Chanteuses” meaning “Singers” or “Stridulantia”. Leafhoppers were, then, the supposedly silent yet ubiquitous scion of the more commendable aesthetic qualities of the Cicada. However, they can be found on all continents, in nearly every habitat that supports vascular plant life and have an intimate relationship with those plants choosing to feed on their above-ground stems or leaves. Their feeding distorts and reshapes plants through puncturing and suction of plant juices and leaves traces through curling, stunting, white blotches, drying, yellowing and distortion depending on the specific activity and sub species. Their proscribed silence was ended when Ossiannilsson laid the groundwork for studies of how the Leafhoppers interaction with plants in another way: through vibration and sound.
Imms (1951) states that sound making was unique to the cicadas and that other members of “Auchenorrhyncha” did not produce sounds. What Ossiannilsson shows is a kind of failure of listening on the part of the previous studies. Instead, as shown in recent studies building on Ossiannilsson, Leafhoppers transmit signals through plant material substrates. These signals are still faint to the human ear so special amplifying equipment is needed to tune into this mediated acoustic behaviour. Leafhoppers do this because their small size means that to communicate through open air would require “singing” at an extremely high frequency and these high frequency transmissions are not suitable in their plant rich environments due to scatter, reflection and interference. To avoid this they communicate vital information, between sexes, through the plants. What is apparent in the communicative behaviour of the leafhopper and its plants is a kind of “ecological exformation” from the perspective of human perception. This is to say that the acoustic interaction between the plant and the leafhopper is an unknown in the tuneable reality of the human ear. It is a communicative interaction which is crucial for the planet (for example: revealing crop health) yet it is only realised following Ossiannilsson’s research and through technological innovations such as amplifying equipment. New acoustic information is discovered or revealed as a result of treating the Cidacellidae’s communication channels as radically non human; as vital information transmitted through the vibration of plants. This exchange creates a micro interaction where a kind of bio-semantic meaning can be decoded through different technological tunings into reality. For example, through bio-acoustic information and sound. This information forms a point of communicative anastomosis between human, machine, plant and animal where bio-exformation turns into bio-information which can be decoded and utilised by the person, farmer, citizen, observer or scientist. It becomes vital information, shared between everything that attends to it.
Our question is, then, how can this acoustic and neuromuscular information be organised and attended to by biologists, farmers, artists, designers or citizen scientists? How can sonic technology and machine learning aid this process? Our aim is to build the potential for a media ecological system surrounding the leafhopper through open source bioacoustics and machine learning. The outcome will be a DIY sensor/service for biologists, using sound recordings to identify and track different species of leafhoppers in order facilitate the translation and exchange of vital information. We hope that aspects of this media system can be used by farmers to monitor plant or animal health remotely, in visual art or computational design projects or by citizen scientists.
Setup:
Breadboard:
Sensor PCB:
BOM:
R1 - 5M R2 - 150k R3 - 100k R4, R5 - 1k R6 - 1M
C1 - 10u
U1 - TLC2902
MISC: T1 - 10M trimmer J1 - piezoelectric disc J2 - 3.5mm jack
MIT License
Copyright (c) [2017] [Davin Browner Conaty, Minwoo Kim, Filippo Sanzeni, Alice Potts]
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
The project is being approached through an antidisciplinary design framework. The production of the sensor will rely on entangling bioacoustics, citizen science, open source and diy technological development and public science.
Interdisciplinarity is vital to achieving breakthrough work across disciplines. Interdisciplinary work is when a group of people from different disciplines work together, while anti-disciplinary work is a process which temporarily or permanently suspends existing knowledge structures in order to facilitate the creation of something innovative and new. Anti-Disciplinarity is “about working in spaces that simply do not fit into any existing academic discipline” (Ito, 2016).
Bioacoustics, or acoustic ecology, is the mapping of biological ecosystems through sound. It presents the opportunity to listen to but also eavesdrop on nature and one of its central ideas is:
Diversity in the soundscape = diversity in the landscape.
Our collaborators at the John Innes Centre for Plant and Microbial Sciences are studying how leafhoppers communicate information between one another, by vibrating plants. More specifically, they are interested in how leafhoppers change their acoustic behaviour when residing on plants adversely affected by the bacteria Phytoplasma? In other words, how it might be possible for leafhoppers to communicate the health of plant ecosystems to humans. What is apparent in the communicative behaviour of the leafhopper and the plant substrates it communicates through is a kind of “ecological exformation” from the perspective of human perception. This is to say that the acoustic interaction between the plant and the leafhopper is an unknown in the tuneable reality of the human ear. Using technology, we can eavesdrop on this communication in order to uncover vital information about the health of ecosystems, which might be utilised by the farmer, citizen, observer, scientist or custodians of local parks and wildlife. It becomes vital information, shared between anything that attends to it.
Our aim is to build the potential for a DIY sensor for biologists which will use sound to identify and track different species of leafhoppers in order to facilitate the translation and exchange of vital information. We hope that aspects of this media system can ultimately be used by farmers to monitor plant or animal health remotely, in visual art or computational design projects or by citizen scientists.
Equally important to our project is the notion of citizen science (the practice of the scientific method by non-professional scientists) and DIY. Both are focussed on how science can become actionable within the public sphere and how the methods of, for example, biology, might be extended beyond the institutional research lab by making legitimate the claim that:
Science = A routine activity that is open to anyone.
A central part of our vision of citizen science and DIY is the idea that scientific technologies should be made publicly available and actionable. We think that this is done by developing biological technologies in a way so that they are approachable, co-designed and linked to a continuing process of knowledge sharing and exchange. The DIY movement, through its principle that the basis of decision making is always better when it's made by the people who are directly impacted by that decision making, provides an interesting practical framework for citizen science. In this sense, both the DIY movement and citizen science, are radically anti-disciplinary; consistently asking questions like: What constitutes knowledge? Who’s authorised to validate knowledge? As well as: how to make, build things and conduct experiments?
Our aim is to fuse these great traditions and make scientific technology actionable for as diverse a set of people as is possible. The goal is to open the “black box” of research by promoting scientific literacy, relocating the site of research away from institutions and opening up the scientific method to the multiple cultures of exchange that exist in society.
By the measure that imitation is the most sincere form of flattery it's clear that humans have an, often undervalued, love for the sonic qualities of insects. Our aural and emotional response to insects is evident in the onomatopoeia of the word “buzz” to the influence of the cricket in compositions like Béla Bartók’s solo piano piece: “The Night’s Music”. Bartók was an amateur entomologist, owning a large collection of beetles and flies, and his marrying of scientific curiosity and entertainment is something that we want to replicate.
A combination of hard science, serious play, sound and the empowering force of DIY and citizen science means that both design and science can be understood and absorbed by people external to the design process. We think that through sonifying and DIY-ing the study of insect behaviour as a form of entertainment, provided by nature and contextualised by culture, we can create an experience that can be re-utilised and expanded. In effect, we will achieve our goal of increasing engagement with ecological systems by opening the door to everyday sonic experience of ecological sound.
The aim then, returning to the leafhopper, is to enable people to whistle a tune that has never been whistled before; to understand, in a qualitative sense, the acoustic structure and polyphony of ecological systems that surround us.
Whaley, S. “Detecting Deception: A Bibliography of Counterdeception Across Time, "Cultures, and Discipline”. (2006).
Chen, Yanping, et al. "Flying insect classification with inexpensive sensors." Journal of insect behavior 27.5 (2014): 657-677.
Bardeli, Rolf, et al. "Detecting bird sounds in a complex acoustic environment and application to bioacoustic monitoring." Pattern Recognition Letters 31.12 (2010): 1524-1534.
Ito, Joichi. (2016). Can design advance science, and can science advance design? url: https://www.pubpub.org/pub/designandscience
Ossiannilsson, F. On sound-production and the sound-producing organ in Swedish Auchenorrhyncha (A preliminary note). Opusc. Entomol. (1946). 11:82-84.
Ossiannilsson, F. Insect drummers. A study on the morphology and function of the sound-producing organ of Swedish Homoptera Auchenorrhyncha with notes on their sound-production. Opusc. (1949): Entomol. 11:82-84.
Sattar, F, et al. "Identification of fish vocalizations from ocean acoustic data." Applied Acoustics 110 (2016): 248-255.
Silva, Diego F., et al. "Exploring low cost laser sensors to identify flying insect species." Journal of Intelligent & Robotic Systems 80 (2015): 313.
Silva, Diego F., et al. "Applying machine learning and audio analysis techniques to insect recognition in intelligent traps." Machine Learning and Applications (ICMLA), 2013 12th International Conference on. Vol. 1. IEEE, 2013.
Zhao, Zhao, et al. "Automated bird acoustic event detection and robust species classification." Ecological Informatics 39 (2017): 99-108.
Xie, Jie. "Multi-label classification of frog species via deep learning." PeerJ Preprints 5 (2017): e3007v1.
Global Market Insight "edible Insects Market Size, Share - Global Industry Report 2023." [online] Available at: https://www.gminsights.com/industry-analysis/edible-insects-market.