Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix ROR Service #639

Merged
merged 2 commits into from
Jul 31, 2024
Merged

Fix ROR Service #639

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
79a9878
updated ROR downloader code
briri Jul 31, 2024
58690ee
removed debug line
briri Jul 31, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 16 additions & 18 deletions app/services/external_apis/ror_service.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,6 @@ def fetch(force: false)

# Fetch the Zenodo metadata for ROR to see if we have the latest data dump
metadata = fetch_zenodo_metadata

if metadata.present?
FileUtils.mkdir_p(file_dir)

Expand All @@ -67,29 +66,25 @@ def fetch(force: false)
if old_checksum_val == metadata[:checksum]
log_message(method: method, message: 'There is no new ROR file to process.')
else
download_file = metadata.fetch(:links, {})[:download]
download_file = download_file = metadata['key']
download_url = metadata.fetch('links', {}).fetch('download', metadata.fetch('links', {})['self'])
log_message(method: method, message: "New ROR file detected - checksum #{metadata[:checksum]}")
log_message(method: method, message: "Downloading #{download_file}")
log_message(method: method, message: "From #{download_url}")

payload = download_ror_file(url: metadata.fetch(:links, {})[:download])
payload = download_ror_file(url: download_url)
if payload.present?
file = File.open(zip_file, 'wb')
file.write(payload)

# rubocop:disable Metrics/BlockNesting
if validate_downloaded_file(file_path: zip_file, checksum: metadata[:checksum])
json_file = download_file.split('/').last.gsub('.zip', '')
json_file = "#{json_file}.json" unless json_file.end_with?('.json')

# Process the ROR JSON
if process_ror_file(zip_file: zip_file, file: json_file)
checksum = File.open(checksum_file, 'w')
checksum.write(metadata[:checksum])
end
else
log_error(method: method, error: StandardError.new('Downloaded ROR zip does not match checksum!'))
json_file = download_file.split('/').last.gsub('.zip', '')
json_file = "#{json_file}.json" unless json_file.end_with?('.json')

# Process the ROR JSON
if process_ror_file(zip_file: zip_file, file: json_file)
checksum = File.open(checksum_file, 'w')
checksum.write(metadata[:checksum])
end
# rubocop:enable Metrics/BlockNesting
else
log_error(method: method, error: StandardError.new('Unable to download ROR file!'))
end
Expand Down Expand Up @@ -121,7 +116,7 @@ def fetch_zenodo_metadata

# Extract the most recent file's metadata
file_metadata = json.fetch('hits', {}).fetch('hits', []).first&.fetch('files', [])&.last&.with_indifferent_access
unless file_metadata.present? && file_metadata.fetch(:links, {})[:download].present?
unless file_metadata.present?
handle_http_failure(method: 'No file found in ROR metadata from Zenodo', http_response: resp)
notify_administrators(obj: 'RorService', response: resp)
return nil
Expand All @@ -140,9 +135,12 @@ def download_ror_file(url:)

headers = {
host: 'zenodo.org',
Accept: 'application/zip'
Accept: 'application/json',
'Content-Type': 'application/json',
'User-Agent': "California Digital Library - dmptool.org (mailto:[email protected])"
}
resp = http_get(uri: url, additional_headers: headers, debug: false)

unless resp.present? && resp.code == 200
handle_http_failure(method: "Fetching ROR file from Zenodo - #{url}", http_response: resp)
notify_administrators(obj: 'RorService', response: resp)
Expand Down
Loading