GnCrossmap

This gem crossmaps a checklist of scientific names to names from a data source in GN Resolver.

Checklist has to be in a CSV format.

Compatibility

This gem is compatible with Ruby versions higher or equal to 2.1.0

Installation

Add this line to your application's Gemfile:

gem 'gn_crossmap'

And then execute:

bundle

Or install it yourself as:

gem install gn_crossmap

Usage

Usage as a Web Application

see [gn_crossmap_web] project

Usage from command line

# to see help
crossmap --help

# to compare with default source (Catalogue of Life)
crossmap -i my_list.csv -o my_list_col.csv

# to compare with other source (Index Fungorum in this example)
crossmap -i my_list.csv -o my_list_if.csv -d 5

# to use standard intput and/or output
cat my_list.csv | crossmap -i - -o - > output

# to keep only taxonID (if given) from original input
# no original fields will be kept without taxonID
cat my_list.csv | crossmap -i my_list.csv -s

# to show classification from the source
cat my_list.csv | crossmap -i my_list.csv -w

Usage as Ruby Library (API description)

GnCrossmap.run

Compares an input list to a data source from GN Resolver and writes result into an output file.

opts = { input: input, output: output, data_source_id: 1 ,
         skip_original: true, alt_headers: [] }
GnCrossmap.run(opts)

input : (string) Either a path to a csv file with list of names, or "-" which designates STDIN

output : (string) Either a path to the output file, or "-" which designates STDOUT

data_source_id : (integer) id of a data source from GN resolver

skip_original : (boolean) if true only taxonID (if given) is preserved from original data. Otherwise all original data is preserved. If there is no taxonID, no original data will be preserved.

alt_headers : (array) empty array by default. If alt_headers are not empty they are used instead of the headers supplied with the file

resolver_url : URL to globalnames' resolver. Default is http://resolver.globalnames.org

with_classification : (boolean) if true, adds classification path to the output

GnCrossmap.logger=

Allows to set logger to a custom logger (default is STDERR)

Usage Example

require "gn_crossmap"

# If you want to change logger -- default Logging is to standard error

GnCrossmap.logger = MyCustomLogger.new

opts = { input: "path/to/input.csv", output: "path/to/output.csv,
         data_source_id: 5 , skip_original: true }
GnCrossmap.run("path/to/input.csv", "path/to/output.csv", 5, true)

# if you want to use alternative headers instead of ones supplied in a file

opts = { input: "path/to/input.csv", output: "path/to/output.csv,
         data_source_id: 5 , skip_original: true,
         alt_headers: %w(taxonId, scientificName, rank) }
GnCrossmap.run(opts)

If you want to get intermediate statistics for each resolution cycle use a block:

GnCrossmap.run(opts) do |stats|
  puts stats
  puts "Matches:"
  stats[:matches].each do |key, value|
    puts "#{GnCrossmap::MATCH_TYPES[key]}: #{value}"
  end
end

To trigger termination of the resolution before it is completed

GnCrossmap.run(opts) do
  # do something and then return STOP string from the block
  "STOP"
end

Intermediate stats format

Field	Description
status	current phase: (init, ingested
total_records	total number of names in original list
ingestion_start	time when the reading from csv started
ingestion_span	time of intermediate checkpoint of reading csv
ingested_records	number of ingested records at an intermediate checkpoint
resolution_start	time when resolution of names started
resolution_stop	time when resolution of names stopped
resolution_span	time of intermediate checkpoint of resolving names
resolved_records	number of names already processed
speed	weighted speed average
current_speed	speed of resolution for the last batch
matches	Distribution of processed data by match type (see below)
errors	First 0-10 errors found during the csv file processing

Match types

Match types dictionary can be accessed with GnCrossmap::MATCH_TYPES constant

Match code	Match type
0	No match
1	Exact string match
2	Canonical form exact match
3	Canonical form fuzzy match
4	Partial canonical form match
5	Partial canonical form fuzzy match
6	Genus part match
7	Error in matching

Input file format

• Comma Separated File with names of fields in first row.
• Columns can be separated by tab, comma or semicolon
• At least some columns should have recognizable fields

taxonID kingdom phylum class order family genus species subspecies variety form scientificNameAuthorship scientificName taxonRank

simplest Example -- only scientificName

scientificName
Animalia
Macrobiotus echinogenitus subsp. areolatus Murray, 1907

taxonID and scientificName Example

taxonID;scientificName
1;Macrobiotus echinogenitus subsp. areolatus Murray, 1907
...

taxonID	scientificName
1	Animalia
2	Macrobiotus echinogenitus subsp. areolatus Murray, 1907

Rank Example

taxonID;scientificName;taxonRank
1;Macrobiotus echinogenitus f. areolatus Murray, 1907;form
...

taxonID	scientificName	taxonRank
1	Animalia	kingdom
2	Macrobiotus echinogenitus subsp. areolatus Murray, 1907	subspecies

Family and Authorship Example

taxonID;family;scientificName;scientificNameAuthorship
1;Macrobiotidae;Macrobiotus echinogenitus subsp. areolatus;Murray, 1907
...

taxonID	family	scientificName	scientificNameAuthorship
1		Animalia
2	Macrobiotidae	Macrobiotus echinogenitus	Murray

Fine-grained Example

TaxonId;kingdom;subkingdom;phylum;subphylum;superclass;class;subclass;cohort;superorder;order;suborder;infraorder;superfamily;family;subfamily;tribe;subtribe;genus;subgenus;section;species;subspecies;variety;form;ScientificNameAuthorship
1;Animalia;;Tardigrada;;;Eutardigrada;;;;Parachela;;;Macrobiotoidea;Macrobiotidae;;;;Macrobiotus;;;harmsworthi;obscurus;;;Dastych, 1985

TaxonId	kingdom	subkingdom	phylum	subphylum	superclass	class	subclass	cohort	superorder	order	suborder	infraorder	superfamily	family	subfamily	tribe	subtribe	genus	subgenus	section	species	subspecies	variety	form	ScientificNameAuthorship
136021	Animalia		Pogonophora
136022	Animalia		Pogonophora			Frenulata																			Webb, 1969
565443	Animalia		Tardigrada			Eutardigrada				Parachela			Macrobiotoidea	Macrobiotidae				Macrobiotus			harmsworthi	obscurus			Dastych, 1985

More examples can be found in spec/files directory

Output file format

Output includes following fields:

Field	Description
classification	classification path of the data source (if available)
editDistance	for fuzzy-matching -- how many characters differ between checklist and data source name
matchSize	number of returned matches for a name
matchType	what kind of match it is
matchedCanonicalForm	canonical form of the matched name
matchedRank	corresponding rank from the data source
matchedScientificName	name matched from the GN Reolver data source
rank	rank from the source (if it was given/inferred)
scientificName	name from the checklist
score	heuristic score from 0 to 1 where 1 is a good match, 0.5 match requires further human investigation
taxonID	original ID attached to a name in the checklist

Types of Matches

The output fomat returns 7 possible types of matches:

1. Exact string match - The exact name was matched (but ignoring non-ascii characters)
2. Exact match by canonical form of a name - The canonical form of the name (a version of a scientific name that contains complete versions of the latin words, but lacks insertions of subtaxa, annotations, or authority information) was matched
3. Fuzzy match by canonical form - The canonical form gave a fuzzy (detecting lexical or spelling variations of a name using Tony Rees' algorithm TAXAMATCH) match
4. Partial exact match by species part of canonical form - The canonical form returned a partial but exact match
5. Partial fuzzy match by species part of canonical form - The canonical form returned a partial, fuzzy match
6. Exact match by genus part of a canonical form - The genus part of the canonical form of the species name returned an exact match
7. [Blank] - No match

Development

After checking out the repo, run bin/setup to install dependencies. Then, run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release to create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org

Contributing

1. Fork it ( https://github.com/[my-github-username]/gn_crossmap/fork )
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create a new Pull Request

Copyright

Author -- Dmitry Mozzherin

Copyright (c) 2015-2016 Marine Biological Laboratory. See LICENSE for details.