Botanical time machines: AI is unlocking a treasure trove of data held in herbarium collections

Botanical time machines: AI is unlocking a treasure trove of data held in herbarium collections

A herbarium specimen of _Cheiranthera linearis_ (commonly known as finger-flower), collected in 1912 by Edwin James Semmens, former principal of the Victorian School of Forestry. University of Melbourne

In 1770, after Captain Cook’s Endeavour struck the Great Barrier Reef and was held up for repairs, botanists Joseph Banks and Daniel Solander collected hundreds of plants.

One of those pressed plants is among 170,000 specimens in the herbarium at the University of Melbourne.

Worldwide, more than 395 million specimens are housed in herbaria. Together they comprise an unparalleled record of Earth’s plant and fungal life over time.

We wanted to find a better, faster way to tap into this wealth of information. Our new research describes the development and testing of a new AI-driven tool Hespi (short for “herbarium specimen sheet pipeline”). It has the potential to revolutionise access to biodiversity data and open up new avenues for research.

A composite image showing a pressed plant specimen collected by Joseph Banks and Daniel Solander in 1770 together with a scale and colour chart, alongside a closeup of the handwritten label
The specimen sheet for spreading nut-heads (Epaltes australis), collected by Joseph Banks and Daniel Solander in 1770. (Note, the collection date was historically incorrectly written as 1776 on the specimen label). University of Melbourne Herbarium Collection

The digitisation challenge

To unlock the full potential of herbaria, institutions worldwide are striving to digitise them. This means photographing each specimen at high resolution and converting the information on its label into searchable digital data.

Once digitised, specimen records can be made available to the public through online databases such as the University of Melbourne Herbarium Collection Online. They are also fed into large biodiversity portals such as the Australasian Virtual Herbarium, the Atlas of Living Australia, or the Global Biodiversity Information Facility. These platforms make centuries of botanical knowledge accessible to researchers everywhere.

But digitisation is a monumental task. Large herbaria, such as the National Herbarium of New South Wales and the Australian National Herbarium have used high-capacity conveyor belt systems to rapidly image millions of specimens. Even with this level of automation, digitising the 1.15 million specimens at the National Herbarium of NSW took more than three years.

For smaller institutions without industrial-scale setups, the process is far slower. Staff, volunteers and citizen scientists photograph specimens and painstakingly transcribe their labels by hand.

At the current pace, many collections won’t be fully digitised for decades. This delay keeps vast amounts of biodiversity data locked away. Researchers in ecology, evolution, climate science and conservation urgently need access to large-scale, accurate biodiversity datasets. A faster approach is essential.

A composite image showing a photo of a yam daisy, image of the specimen in the collection and map showing specimen collection locations across Australia.
Map of specimen collection locations for Yam daisy (Microseris lanceolata) from records in The Australasian Virtual Herbarium. Neville Walsh, VicFlora

How AI is speeding things up

To address this challenge, we created Hespi – open-source software for automatically extracting information from herbarium specimens.

Hespi combines advanced computer vision techniques with AI tools such as object detection, image classification and large language models.

First, it takes an image of the specimen sheet which comprises the pressed plant and identifying text. Then it recognises and extracts text, using a combination of optical character recognition and handwritten text recognition.

Deciphering handwriting is challenging for people and computers alike. So Hespi passes the extracted text through OpenAI’s GPT-4o Large Language Model to correct any errors. This substantially improves the results.

So in seconds, Hespi locates the main specimen label on a herbarium sheet and reads the information it contains. This includes taxonomic names, collector details, location, latitude and longitude, and collection dates. It captures the data and converts it into a digital format, ready for use in research.

For example, Hespi correctly detected and extracted all relevant components from the herbarium sheet below. This large brown algae specimen was collected in 1883 at St Kilda.

An image showing how Hespi reads the plant specimen sheet and tags information such as the genus, species, locality and year of collection.
Results from Hespi on a sample of large brown algae (MELUA002557a) from the University of Melbourne, identifying important details such as the genus, species, locality and year of collection. University of Melbourne Herbarium

We tested Hespi on thousands of specimen images from the University of Melbourne Herbarium and other collections worldwide. We created test datasets for different stages in the pipeline and assessed the various components.

It achieved a high degree of accuracy. So it has the potential to save a lot of time, compared to manual data extraction.

We are developing a graphical user interface for the software so herbarium curators will be able to manually check and correct the results.

Just the beginning

Herbaria already contribute to society in many ways: from species identification and taxonomy to ecological monitoring, conservation, education, and even forensic investigations.

By mobilising large volumes of specimen-associated data, AI systems such as Hespi are enabling new and innovative applications at a scale never before possible.

AI has been used to automatically extract detailed leaf measurements and other traits from digitised specimens, unlocking centuries of historical collections for rapid research into plant evolution and ecology.

And this is just the beginning — computer vision and AI could soon be applied in many other ways, further accelerating and expanding botanical research in the years ahead.

Photo of a well-lit pressed plant specimen sheet on black table with camera mounted above, looking down.
The digitisation pipeline at the University of Melbourne Herbarium begins with the generation of a high-resolution specimen image. University of Melbourne Herbarium

Beyond herbaria

AI pipelines such as Hespi have the potential to extract text from labels in any museum or archival collection where high-quality digital images exist.

Our next step is a collaboration with Museums Victoria to adapt Hespi to create an AI digitisation pipeline suitable for museum collections. The AI pipeline will mobilise biodiversity data for about 12,500 specimens in the museum’s globally-significant fossil graptolite collection.

An image showing a dark grey fossil graptolite specimen with numbers attached alongside handwritten labels with annotations from Hespi.
A fossil graptolite specimen from Museums Victoria annotated by Hespi during data digitsition. Museums Victoria

We are also starting a new project with the Australian Research Data Commons (ARDC) to make the software more flexible. This will allow curators in museums and other institutions to customise Hespi to extract data from all kinds of collections — not just plant specimens.

Tranformational technology

Just as AI is reshaping many aspects of daily life, these technologies can transform access to biodiversity data. Human-AI collaborations could help overcome one of the biggest bottlenecks in collection digitisation — the slow, manual transcription of label data.

Mobilising the information already locked in herbaria, museums, and archives worldwide is essential to make it available for the cross-disciplinary research needed to understand and address the biodiversity crisis.

We wish to acknowledge our colleagues at the Melbourne Data Analytics Platform, including Karen Thompson and Emily Fitzgerald, who contributed to this research.

The Conversation

The authors do not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and have disclosed no relevant affiliations beyond their academic appointment.

US pipeline protester’s obstruction conviction overturned by appeals court

New trial for Mylene Vialard after Minnesota judges find ‘pervasive’ prosecutorial misconduct in Line 3 protest case

The controversial felony conviction of a peaceful climate activist has been overturned by an appeals court due to “pervasive” prosecutorial misconduct.

Mylene Vialard, 56, was found guilty of felony obstruction in 2023 for her role in trying to halt construction of a fossil-fuel pipeline through Indigenous territory in Minnesota, in a trial beset by irregularities.

Continue reading…

How could we clean up the algal bloom?

How could we clean up the algal bloom?

The author inspects a restored native oyster reef at Coffin Bay, South Australia. Stefan Andrews

South Australia’s catastrophic harmful algal bloom now affects almost 30% of the state’s coastline, stretching from the Coorong in the state’s southeast to the seafood-rich Spencer Gulf to the west.

With no end in sight, many South Australians are searching for solutions.

Trials of physical, chemical and biological solutions have been run around the world at much smaller scale, usually in aquaculture fish pens and leases.

Unfortunately, the enormous Karenia mikimotoi bloom in SA is far too big for any of these technologies. But they could provide temporary relief at high-risk sites, such as vital breeding grounds for vulnerable species such as the giant Australian cuttlefish.

How can algae be managed at smaller scales?

This disaster arose from a triple threat: marine heatwaves linked to climate change, floodwaters and cold upwellings carrying nutrients algae need, and marine habitat loss leaving coasts extra vulnerable. Prevention is the best cure – so we need to tackle each of these threats.

In the meantime, the bloom’s persistence presents an opportunity to experiment with potential solutions.

Chemical

Anyone who has managed a pool knows how effective chemicals are in removing unwanted algae. But in the sea, chemicals can have unintended impacts. There’s more than one type of algae out there and we don’t want to kill the good guys.

In response to algal blooms at sea, researchers have even tried chemical crop-dusting by planes.

Clay has been used against marine algal blooms for more than 50 years in China and Japan. The clay particles bind to algal cells in surface waters, causing them to settle to the bottom where they stop growing or die. But to date, it has required a huge amount of clay to be effective.

Modified clay technology is a promising environmentally friendly solution. Released as a slurry, the clay binds and removes phosphorus, essentially removing the bloom’s fuel source.

Physical

Physical solutions often involve mixing the water column to break up the warm surface layers suitable for algal growth.

Other solutions physically block and disturb algae. Pumping air through tubes to create “bubble curtains” can stop algae from passing through.

This technology is useful in aquaculture pens. But bubble curtains can do little against intense algal blooms, forcing fish farmers to simply drag their pens out of harms’ way. So they’re never going to work at scale.

Biological

Natural systems have their own checks and balances. The oceans are full of microorganisms that naturally prey on algae – such as single-celled organisms (ciliates and flagellates) – or suppress it, including bacteria and viruses.

Such natural microbial warfare may help solve the SA bloom, with promising signs of bioluminescent “sea sparkle” algae dining on Karenia.

Scientists in the United States are working to extract natural algicides produced by algae-killing marine bacteria to combat Red Tide blooms in Florida.

Some bacterial algicides only kill specific harmful algae species, offering promise for low-risk application. To date, these solutions have largely been used at small, experimental scale.

By contrast, algicidal bacteria are found in abundance on common seagrasses around the world, providing healthy seagrass meadows with natural immunity. Conserving and restoring seagrass offers one way to tackle the problem longer term.

Underwater evidence of SA’s harmful algal bloom (Great Southern Reef)

Helping nature to help itself

SA was once home to 1,500km of shellfish reefs, formed by billions of native oysters. These ecosystems served as the natural kidneys of our coastline. A single oyster is capable of filtering a bathtub of water a day, removing excess nutrients and algae.

Within a century of European settlement, these natural reefs were all but destroyed by overharvesting. The good news: they can be restored. For example, restoring two hectares of shellfish reef at Adelaide’s Glenelg Beach saw oysters filtering over 12 million litres of water a day within 1.5 years of reef construction.

These figures suggest SA’s lost shellfish reefs would have been filtering over half a trillion litres of seawater daily. This natural wastewater treatment system could have removed nutrients washed into our coastal seas before they could feed a bloom.

Diving on the restored shellfish reef during the algal bloom has given us some hope. Despite the soupy green water, the native oysters appear to be doing well. Similarly, oysters on leases in the affected areas are thriving as they feed on the highly nutritious Karenia algae.

By contrast, many filter-feeding bivalves have been devastated by the bloom, notably the habitat-forming razorfish (pen shells) and cockles.

Earlier this month, Premier Peter Malinauskas announced plans to restore a total of 15 hectares of shellfish reef alongside community groups across 15 locations. This provides an opportunity to learn whether larger-scale oyster restoration can help future-proof SA seas against harmful algae.

Efforts to restore Coffin Bay’s lost oyster reefs through community-based restoration (eyrelab)

Rebuildling resilient ecosystems

This devastating algal bloom will eventually dissipate. Once the harmful algae have fallen back to normal levels, nature can rebuild.

But future blooms are likely as the climate warms and strengthens heavy downpours that flush nutrients out to sea. To prepare, we should explore ways of restoring ecosystems able to hasten recovery and rebuild natural resilience.

Investing in scaling up conservation and restoration of filter-feeding shellfish reefs and bacteria-harbouring seagrass meadows – the sea’s kidneys and immune system – will be necessary alongside long-term underwater monitoring.

If we continue with a reactive, fragmented approach to climate, nutrient pollution, and biodiversity loss, we’re guaranteed to face more costly catastrophes. We need to act to build long-term ecological and socio-economic resilience.

The Conversation

Dominic McAfee receives funding from the South Australian Government. He is affiliated with conservation non-profit EyreLab (Educating Youth in Restoration Ecology Lab).

Sean Connell does not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.