Autocomplete compatibility

docs/autocomplete_compatibility.md

@@ -0,0 +1,41 @@
+# Autocomplete Compatability
+
+Many lists are used to drive autocompleting user interface components. In order to support this, a standardised concept of an "autocomplete compatible" list has been defined. A list which is autocomplete compatible has the information required to drive an autocomplete-based selection mechanism, and has that information in well-known fields so no additional configuration is required to use that list in an autocomplete mechanism that understands this standard.
+
+# Specification
+
+An autocomplete-compatible list must have:
+
+1. A `name` field, which is a string value in every record. This should be the "best" name for the identified object, which should usually be the one that regular users are most likely to use - this may not always be the official or legal name. For instance, an organisation may have a well-known trading name that is different from their name as registered with Companies House.
+2. (Optionally) an `match_synonyms` field which, in every record in which it is present, is an array of strings. These should be alternative names by which the object may be known, but which can only refer to this object.
+2. (Optionally) a `suggestion_synonyms` field which, in every record in which it is present, is an array of strings. These should be alternative names by which the object may possibly be known, but which might not always refer to this object.
+2. (Optionally) an `hint` field which, in every record in which it is present, is a string. It should be a message to be shown to users to help them decide if this is the object they mean to select.
+
+In addition, for any given list, the combination of `name` and values inside `match_synonyms` must not contain duplicates - there can be no string that matches more than one record in the `name` field or a `match_synonym`. Duplication within `suggestion_synonyms` is fine.
+
+## Matching
+
+A user-inputted string is compared to names from `name`, `match_synonyms` or `suggestion_synonyms` fields to see if they "match" any records. The nature of this comparison is deliberately weakly specified, as autocomplete logic may use arbitrarily clever fuzzy matches. However, it will always include case-insensitive string matching, so there is no need to list multiple names in a record that differ only in case.
+
+See https://bat-design-history.netlify.app/register-trainee-teachers/autocomplete-improvements/ for a discussion of the development of fuzzy matching in autocompletion engines.
+
+# Documentation
+
+A reference data list may declare that it is autocomplete compatible with the following wording:
+
+```
+This list is [autocomplete compatible](autocomplete_compatability.md).
+```
+
+# Validation
+
+The RSpec tests for a list can confirm it's autocomplete compatible with the following pattern:
+
+```ruby
+  describe 'qualifications' do
+    it 'is a valid autocomplete-capable list' do
+      DfE::ReferenceData::Qualifications::QUALIFICATIONS.validate_autocomplete_compatibility!
+    end
+  end
+
+```

docs/lists_degrees.md

@@ -18,6 +18,8 @@ Source: https://github.com/DFE-Digital/apply-for-teacher-training-prototype/blob
 
 Quality: Manually updated on an ad-hoc basis. Please submit a pull request if inaccuracies or omissions are found.
 
+This list is [autocomplete compatible](autocomplete_compatability.md).
+
 | Field | Type | Purpose |
 |---|---|---|
 | `id` | UUID | A unique identifier. The same as `dttp_id` if that field is non-`nil`, otherwise a new UUID was minted at import time. |
@@ -49,6 +51,8 @@ Source: https://github.com/DFE-Digital/apply-for-teacher-training-prototype/blob
 
 Quality: Manually updated on an ad-hoc basis. Please submit a pull request if inaccuracies or omissions are found.
 
+This list is [autocomplete compatible](autocomplete_compatability.md).
+
 | Field | Type | Purpose |
 |---|---|---|
 | `id` | UUID | A unique identifier. The same as `dttp_id` if that field is non-`nil`, otherwise a new UUID was minted at import time. |
@@ -78,6 +82,8 @@ Source: Automatically derived from joining the `TYPES` and `GENERIC_TYPES` lists
 
 Quality: Automatically derived from the source data, so only as correct as they are.
 
+This list is [autocomplete compatible](autocomplete_compatability.md).
+
 | Field | Type | Purpose |
 |---|---|---|
 | `id` | UUID | A unique identifier. The same as `dttp_id` if that field is non-`nil`, otherwise a new UUID was minted at import time. |
@@ -101,6 +107,8 @@ Users: Apply team.
 
 Quality: Manually updated on an ad-hoc basis. Please submit a pull request if inaccuracies or omissions are found.
 
+This list is [autocomplete compatible](autocomplete_compatability.md).
+
 | Field | Type | Purpose |
 |---|---|---|
 | `id` | UUID | A unique identifier. |
@@ -126,6 +134,8 @@ Source: https://github.com/DFE-Digital/apply-for-teacher-training-prototype/blob
 
 Quality: Manually updated on an ad-hoc basis. Please submit a pull request if inaccuracies or omissions are found.
 
+This list is [autocomplete compatible](autocomplete_compatability.md).
+
 | Field | Type | Purpose |
 |---|---|---|
 | `id` | UUID | A unique identifier. The same as `dttp_id` if that field is non-`nil`, otherwise a new UUID was minted at import time. |
@@ -151,6 +161,8 @@ Source: https://github.com/DFE-Digital/apply-for-teacher-training-prototype/blob
 
 Quality: Manually updated on an ad-hoc basis. Please submit a pull request if inaccuracies or omissions are found.
 
+This list is [autocomplete compatible](autocomplete_compatability.md).
+
 | Field | Type | Purpose |
 |---|---|---|
 | `id` | UUID | A unique identifier. The same as `dttp_id` if that field is non-`nil`, otherwise a new UUID was minted at import time. |
@@ -175,6 +187,8 @@ Source: Automatically derived from joining the `SINGLE_SUBJECTS` and `COMBINED_S
 
 Quality: Automatically derived from the source data, so only as correct as they are.
 
+This list is [autocomplete compatible](autocomplete_compatability.md).
+
 | Field | Type | Purpose |
 |---|---|---|
 | `id` | UUID | A unique identifier. The same as `dttp_id` if that field is non-`nil`, otherwise a new UUID was minted at import time. |
@@ -201,6 +215,8 @@ Source: https://github.com/DFE-Digital/apply-for-teacher-training-prototype/blob
 
 Quality: Manually updated on an ad-hoc basis. Please submit a pull request if inaccuracies or omissions are found.
 
+This list is [autocomplete compatible](autocomplete_compatability.md).
+
 | Field | Type | Purpose |
 |---|---|---|
 | `id` | UUID | A unique identifier. The same as `dttp_id` if that field is non-`nil`, otherwise a new UUID was minted at import time. |

docs/lists_qualifications.md

@@ -18,6 +18,8 @@ Source: https://www.gov.uk/what-different-qualification-levels-mean/list-of-qual
 
 Quality: Manually updated on an ad-hoc basis. Please submit a pull request if inaccuracies or omissions are found.
 
+This list is [autocomplete compatible](autocomplete_compatability.md).
+
 | Field | Type | Purpose |
 |---|---|---|
 | `id` | UUID | A unique identifier |

lib/dfe/reference_data/degrees/institutions.rb

@@ -71,8 +71,7 @@ module Degrees
             suggestion_synonyms: [],
             match_synonyms:
             ['University of St Mark and St John',
-             'University of Saint Mark and Saint John',
-             'University of St Mark and St John'],
+             'University of Saint Mark and Saint John'],
             hesa_itt_code: '14',
             dttp_id: '3a71f34a-2887-e711-80d8-005056ac45bb',
             ukprn: '10037449' },

spec/lib/dfe/reference_data/degrees/autocomplete_spec.rb

@@ -0,0 +1,5 @@
+RSpec.describe DfE::ReferenceData::Degrees::SINGLE_SUBJECTS do
+  describe 'single subjects' do
+    let(:single_subjects) { DfE::ReferenceData::Degrees::SINGLE_SUBJECTS.all }
+  end
+end

spec/lib/dfe/reference_data/degrees/combined_subjects_spec.rb

@@ -1,3 +1,5 @@
+require 'support/autocomplete'
+
 RSpec.describe DfE::ReferenceData::Degrees::COMBINED_SUBJECTS do
   describe 'subject IDs are correct' do
     let(:records) { described_class.all }
@@ -35,4 +37,6 @@
       end
     end
   end
+
+  it_should_behave_like 'a valid autocomplete-capable list'
 end

spec/lib/dfe/reference_data/degrees/generic_types.rb

@@ -0,0 +1,7 @@
+RSpec.describe DfE::ReferenceData::Degrees::GENERIC_TYPES do
+  describe 'generic types' do
+    it 'is a valid autocomplete-capable list' do
+      DfE::ReferenceData::GENERIC_TYPES.validate_autocomplete_compatibility!
+    end
+  end
+end

spec/lib/dfe/reference_data/degrees/grades_spec.rb

@@ -0,0 +1,5 @@
+require 'support/autocomplete'
+
+RSpec.describe DfE::ReferenceData::Degrees::GRADES do
+  it_should_behave_like 'a valid autocomplete-capable list'
+end

spec/lib/dfe/reference_data/degrees/institutions_spec.rb

@@ -0,0 +1,5 @@
+require 'support/autocomplete'
+
+RSpec.describe DfE::ReferenceData::Degrees::INSTITUTIONS do
+  it_should_behave_like 'a valid autocomplete-capable list'
+end

spec/lib/dfe/reference_data/degrees/single_subjects_spec.rb

@@ -1,3 +1,5 @@
+require 'support/autocomplete'
+
 RSpec.describe DfE::ReferenceData::Degrees::SINGLE_SUBJECTS do
   describe 'single subjects' do
     let(:single_subjects) { DfE::ReferenceData::Degrees::SINGLE_SUBJECTS.all }
@@ -22,4 +24,6 @@
       end
     end
   end
+
+  it_should_behave_like 'a valid autocomplete-capable list'
 end

spec/lib/dfe/reference_data/degrees/subjects_spec.rb

@@ -0,0 +1,5 @@
+require 'support/autocomplete'
+
+RSpec.describe DfE::ReferenceData::Degrees::SUBJECTS do
+  it_should_behave_like 'a valid autocomplete-capable list'
+end

spec/lib/dfe/reference_data/degrees/types.rb

@@ -0,0 +1,7 @@
+RSpec.describe DfE::ReferenceData::Degrees::TYPES do
+  describe 'types' do
+    it 'is a valid autocomplete-capable list' do
+      DfE::ReferenceData::TYPES.validate_autocomplete_compatibility!
+    end
+  end
+end

spec/lib/dfe/reference_data/degrees/types_including_generics.rb

@@ -0,0 +1,7 @@
+RSpec.describe DfE::ReferenceData::Degrees::TYPES_INCLUDING_GENERICS do
+  describe 'types including generics' do
+    it 'is a valid autocomplete-capable list' do
+      DfE::ReferenceData::TYPES_INCLUDING_GENERICS.validate_autocomplete_compatibility!
+    end
+  end
+end

spec/lib/dfe/reference_data/qualifications_spec.rb

@@ -0,0 +1,5 @@
+require 'support/autocomplete'
+
+RSpec.describe DfE::ReferenceData::Qualifications::QUALIFICATIONS do
+  it_should_behave_like 'a valid autocomplete-capable list'
+end

spec/support/autocomplete.rb

@@ -0,0 +1,51 @@
+# Seriously, rubocop, this is not a complicated method
+# rubocop:disable Metrics/AbcSize
+# rubocop:disable Metrics/CyclomaticComplexity
+# rubocop:disable Metrics/PerceivedComplexity
+
+def validate_autocomplete_compatible_record!(record)
+  data = record.data
+
+  # Basic validation: has name, match_synonyms, suggestion_synonyms, hint of appropriate types
+  # TODO: Rewrite this into a one-off schema check when schemas are merged in
+  throw "Record #{record.id} lacks a name string" unless !data[:name].nil? && data[:name].is_a?(String)
+  throw "Record #{record.id} has a non-string hint" unless data[:hint].nil? || data[:hint].is_a?(String)
+  ms = data[:match_synonyms]
+  throw "Record #{record.id} has non-array-of-strings match synonyms" unless ms.nil? || ms.is_a?(Array) || ms.all? { |e| e.is_a?(String) }
+  ss = data[:suggestion_synonyms]
+  throw "Record #{record.id} has non-array-of-strings suggestion synonyms" unless ss.nil? || ss.is_a?(Array) || ss.all? { |e| e.is_a?(String) }
+end
+
+# rubocop:enable Metrics/AbcSize
+# rubocop:enable Metrics/CyclomaticComplexity
+# rubocop:enable Metrics/PerceivedComplexity
+
+RSpec::Matchers.define :be_valid_autocomplete_compatible do
+  match do |actual|
+    # Keep track of names and match synonyms that have been used in previous
+    # records, to ensure uniqueness
+    used_names = {}
+
+    actual.all.each do |record|
+      validate_autocomplete_compatible_record!(record)
+
+      # Unique name check
+      names = Array.new(record.match_synonyms || [])
+      names << record.name
+
+      names.each do |name|
+        throw "Record #{record.id} has name/match synonym #{name} which is already used by record #{used_names[name]}" if used_names[name]
+        used_names[name] = record.id
+      end
+    end
+
+    # Ok, I guess we're good then!
+    true
+  end
+end
+
+RSpec.shared_examples 'a valid autocomplete-capable list' do
+  it 'is a valid autocomplete-capable list' do
+    expect(described_class).to be_valid_autocomplete_compatible
+  end
+end