Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test workflow #40

Open
wants to merge 132 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
132 commits
Select commit Hold shift + click to select a range
7ec40fb
Update VMSdatacall_proposedWorkflow.r
roimartins Apr 27, 2021
99f6216
Create global.R
roimartins Dec 3, 2021
c88acfa
Create 1_eflalo_tacsat_preprocessing.R
roimartins Dec 3, 2021
8ac80e5
Rename global.R to 0_global.R
roimartins Dec 3, 2021
51185b6
Create 2_eflalo_tacsat_analysis.R
roimartins Dec 3, 2021
4d3758d
Create 3_data_submission.R
roimartins Dec 3, 2021
c5e9231
Create 1_eflalo_tacsat_preprocessing.R
mrska Dec 13, 2021
b55bc12
add functions which will become icesDATSU
colinpmillar Dec 13, 2021
e4b40fa
* add function to get dataset version IDs and add dependencies
colinpmillar Dec 13, 2021
816aaf2
Add main web services, and a worksheet to test things
colinpmillar Dec 14, 2021
56fefbd
copied the four files comprising the workflow over to my own folder, …
Dec 14, 2021
4aac492
Changed the order of tho checks 'Remove trip with overlap with anothe…
mrska Dec 14, 2021
36c3538
Merge branch 'test-workflow' of https://github.com/ices-eg/wg_WGSFD i…
mrska Dec 14, 2021
440f77b
Merge branch 'test-workflow' of https://github.com/ices-eg/wg_WGSFD i…
Dec 14, 2021
bf1b2b9
added necessary shapfiles, functions, and script to assign metier to …
Dec 14, 2021
51f7873
add feild types and code to create dummy data
colinpmillar Dec 14, 2021
6752e4b
Merge branch 'test-workflow' of https://github.com/ices-eg/wg_WGSFD i…
Dec 14, 2021
2e48791
added Einar's QC markdown
Dec 14, 2021
3ca01b3
merge by area added to assign metier script
Dec 14, 2021
8bd3ce1
Merge branch 'test-workflow' of https://github.com/ices-eg/wg_WGSFD i…
Dec 14, 2021
3db2a52
added a function that extracts FAO code from a spatial polygon layer
Jan 25, 2022
df38dd8
wrote a wrapper for these few lines
Jan 25, 2022
6ab71de
corrected an object name and removed the unneeded columns from input …
Jan 25, 2022
fcba90b
added the rest of blob/master/Metiers/Scripts/script_metiers.R
Jan 25, 2022
feae58e
commit latest change in workflow development
roimartins Feb 2, 2022
6fd6bf7
commit all changes in workflow for datacall
roimartins Feb 2, 2022
eb8f865
Merge branch 'test-workflow' of https://github.com/ices-eg/wg_WGSFD i…
roimartins Feb 2, 2022
828a592
changes from Marta in the preprocessing stage
roimartins Feb 2, 2022
7c08ea3
changes to file structure in the folder
roimartins Feb 2, 2022
cbb2ce5
change when table 1 and table 2 are loaded
roimartins Feb 2, 2022
1def84e
add lines for WEB API submission
colinpmillar Feb 7, 2022
242da65
further work on DATSU functions
colinpmillar Feb 7, 2022
7843222
small edit: remove argument name from call to screen_vms_file
colinpmillar Feb 7, 2022
6c7f48c
year changed to test with one year only
TorstenSchulze Feb 7, 2022
6e1fbc5
years changed back to original version
TorstenSchulze Feb 7, 2022
db33b12
Merge branch 'test-workflow' of https://github.com/ices-eg/wg_WGSFD i…
TorstenSchulze Feb 7, 2022
b7dfd01
remove trailing spaces
colinpmillar Feb 7, 2022
ea446e3
update with colin's changes
colinpmillar Feb 7, 2022
a1481ed
Merge branch 'test-workflow' of https://github.com/ices-eg/wg_WGSFD i…
TorstenSchulze Feb 7, 2022
9e0ecf4
add icesConnect to packages to install
colinpmillar Feb 7, 2022
34e5158
Update the vessellength category and the submission api part
roimartins Feb 7, 2022
390a7ef
Merge branch 'test-workflow' of https://github.com/ices-eg/wg_WGSFD i…
TorstenSchulze Feb 8, 2022
206bb63
settings changed to fit Thuenen folder structure
TorstenSchulze Feb 8, 2022
b5c86ab
Codes changed 2022-02-07 by Christian and Torsten
TorstenSchulze Feb 8, 2022
d770c79
review the submission via API lines
roimartins Feb 8, 2022
d64e4db
Merge branch 'test-workflow' of https://github.com/ices-eg/wg_WGSFD i…
roimartins Feb 8, 2022
383786e
minor typos and comments added
roimartins Feb 8, 2022
eafc2bc
comments with instructions added
roimartins Feb 8, 2022
484849c
testing with German data ongoing
TorstenSchulze Feb 8, 2022
c039bf8
wrong folder
TorstenSchulze Feb 8, 2022
6675281
mimor change only
TorstenSchulze Feb 9, 2022
feed5b6
code is running on German data
TorstenSchulze Feb 9, 2022
c3f79d3
replace teh sp funciton in VMS points in ICES region by sf lirbary pr…
roimartins Feb 11, 2022
5cebbd7
fix vessel length vocabulary references
roimartins Feb 11, 2022
1dd4d50
remove assignation of loaded files ino variables.
roimartins Feb 11, 2022
89bf292
add filter the acgive pings only
roimartins Feb 14, 2022
b016a92
replace empty space by "no_required"
roimartins Feb 14, 2022
4406797
changes in German data to run test proposed workflow
TorstenSchulze Feb 14, 2022
da25ead
changes in German data to comply with ices datsu rules
TorstenSchulze Feb 15, 2022
f3e1f2d
Merge branch 'test-workflow' of https://github.com/ices-eg/wg_WGSFD i…
TorstenSchulze Feb 15, 2022
920fe9e
last corrections, code is running for DEU_2020
TorstenSchulze Feb 15, 2022
159b986
Update 3_data_submission.R
roimartins Feb 15, 2022
4eacc28
Create Data preparation and submission.md
LaraSalvany Feb 15, 2022
7ce4a5c
Update Data preparation and submission.md
roimartins Feb 15, 2022
e48c56f
Update Data preparation and submission.md
roimartins Feb 15, 2022
00fb1e3
Update 1_eflalo_tacsat_preprocessing.R
roimartins Feb 15, 2022
4f04688
Update Data preparation and submission.md
roimartins Feb 15, 2022
c0179b4
ann. VE-ID fixed
TorstenSchulze Feb 15, 2022
6a0297a
Merge branch 'test-workflow' of https://github.com/ices-eg/wg_WGSFD i…
TorstenSchulze Feb 15, 2022
90b33a9
Update Data preparation and submission.md
LaraSalvany Feb 15, 2022
f75e515
Update Data preparation and submission.md
LaraSalvany Feb 16, 2022
9bd3589
Update Data preparation and submission.md
LaraSalvany Feb 16, 2022
15962c0
Update Data preparation and submission.md
LaraSalvany Feb 16, 2022
9ee57ff
Update Data preparation and submission.md
LaraSalvany Feb 16, 2022
53f9973
runs severas years
TorstenSchulze Feb 16, 2022
c67e7b7
Merge branch 'test-workflow' of https://github.com/ices-eg/wg_WGSFD i…
TorstenSchulze Feb 16, 2022
f8bfbad
fix the VE_LEN field in table 1 and table 2
roimartins Feb 16, 2022
fb93b95
Merge branch 'test-workflow' of https://github.com/ices-eg/wg_WGSFD i…
roimartins Feb 16, 2022
956ceae
add rm.na
roimartins Feb 16, 2022
4ef9fa8
change spaces
roimartins Feb 16, 2022
7277f5a
update the main folder files with suggested changes
roimartins Feb 16, 2022
1bc7943
add country code check and na.rm TRUE
roimartins Feb 16, 2022
10b9d45
Update Data preparation and submission.md
roimartins Feb 16, 2022
686ab64
Update Data preparation and submission.md
roimartins Feb 16, 2022
3fe0858
QC-markdown code working on 2022 datsu format
TorstenSchulze Feb 17, 2022
1865d3b
Merge branch 'test-workflow' of https://github.com/ices-eg/wg_WGSFD i…
TorstenSchulze Feb 17, 2022
1ba3057
QC-markdown code working on 2022 datsu format
TorstenSchulze Feb 17, 2022
e9c1230
QC-codes without national pathes to data
TorstenSchulze Feb 17, 2022
68687e9
Update Data preparation and submission.md
LaraSalvany Feb 18, 2022
9f197b1
Update Data preparation and submission.md
LaraSalvany Feb 18, 2022
e13b79a
Update Data preparation and submission.md
LaraSalvany Feb 18, 2022
743fbde
Update Data preparation and submission.md
LaraSalvany Feb 18, 2022
bb202ad
Update Data preparation and submission.md
LaraSalvany Feb 18, 2022
25a05c3
Update Data preparation and submission.md
LaraSalvany Feb 18, 2022
d1a63ab
Update Data preparation and submission.md
LaraSalvany Feb 18, 2022
79fd53b
Update Data preparation and submission.md
LaraSalvany Feb 18, 2022
07c3e5e
Update Data preparation and submission.md
LaraSalvany Feb 18, 2022
510b8e1
Update Data preparation and submission.md
LaraSalvany Feb 18, 2022
b8af356
Update Data preparation and submission.md
LaraSalvany Feb 18, 2022
3d6c18d
Update Data preparation and submission.md
LaraSalvany Feb 18, 2022
95ea5e4
Update Data preparation and submission.md
LaraSalvany Feb 18, 2022
ae0298c
Update Data preparation and submission.md
LaraSalvany Feb 18, 2022
6611666
Update Data preparation and submission.md
LaraSalvany Feb 18, 2022
185a3e2
Update Data preparation and submission.md
LaraSalvany Feb 18, 2022
37252a1
Update Data preparation and submission.md
LaraSalvany Feb 18, 2022
85262f4
Update Data preparation and submission.md
LaraSalvany Feb 18, 2022
40be545
Update Data preparation and submission.md
LaraSalvany Feb 18, 2022
afdc11c
Update Data preparation and submission.md
LaraSalvany Feb 18, 2022
50ce856
Update Data preparation and submission.md
roimartins Feb 18, 2022
385253f
Update Data preparation and submission.md
roimartins Feb 18, 2022
a35faa6
Update Data preparation and submission.md
roimartins Feb 18, 2022
002f7b5
Update Data preparation and submission.md
roimartins Feb 18, 2022
f7f4577
Update Data preparation and submission.md
roimartins Feb 18, 2022
16df873
Update Data preparation and submission.md
roimartins Feb 18, 2022
aeeb125
Update Data preparation and submission.md
roimartins Feb 18, 2022
360a938
headers updated
roimartins Feb 18, 2022
38b5ab1
Merge branch 'test-workflow' of https://github.com/ices-eg/wg_WGSFD i…
roimartins Feb 18, 2022
678280e
Update Data preparation and submission.md
LaraSalvany Feb 18, 2022
4e1e22d
Update Data preparation and submission.md
roimartins Feb 18, 2022
162c57b
Update Data preparation and submission.md
LaraSalvany Feb 18, 2022
3545f73
Update Data preparation and submission.md
LaraSalvany Feb 19, 2022
d994ac5
Update Data preparation and submission.md
roimartins Feb 21, 2022
d3712ce
Update Data preparation and submission.md
roimartins Feb 21, 2022
11802d5
Update README.md
roimartins Feb 22, 2022
402428c
Update README.md
roimartins Feb 22, 2022
6d66a1f
Update README.md
roimartins Feb 22, 2022
8751802
Update README.md
roimartins Feb 22, 2022
b7634c3
Update README.md
roimartins Feb 22, 2022
59f8414
Update README.md
roimartins Feb 22, 2022
8695802
Update README.md
roimartins Feb 22, 2022
aecc174
Update Data preparation and submission.md
LaraSalvany Feb 22, 2022
48b452b
Update Data preparation and submission.md
LaraSalvany Feb 22, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 67 additions & 0 deletions VMS-datacall/0_global.R
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
#-------------------------------------------------------------------------------
#
# Script to extract and process VMS and logbook data for ICES VMS data call
#
# By: Niels Hintzen, Katell Hamon, Marcel Machiels#
# Code by: Niels Hintzen
# Contact: [email protected]
#
# Date: 25-Jan-2017
# Update Date: 29-Jan-2019 ; Updated by: Roi Martinez
# Update Date: 04-Feb-2020 ; Updated by: Colin Millar
# Update Date: 07-Feb 2020 ; Updated by: Neil Campbell
# Client: ICES
#-------------------------------------------------------------------------------

#--------------------READ ME----------------------------------------------------
# The following script is a proposed workflow example to processes the ICES
# VMS datacall request. It is not an exact template to be applied to data from
# every member state and needs to be adjusted according to the data availability
# and needs of every member state.
#-------------------------------------------------------------------------------


#- Clear workspace
rm(list=ls())

library(vmstools) #- download from www.vmstools.org
library(Matrix) #- available on CRAN
library(ggplot2) #- available on CRAN
library(dplyr) #- available on CRAN
library(sp)
library(doBy)
library(mixtools)
library(tidyr)
library(glue)
library(gt)


#- Settings paths
codePath <- "VMSdatacall/R" #Location where you store R scripts
dataPath <- "VMSdatacall/Data" #Location where you store tacsat (VMS) and eflalo (logbook) data
outPath <- "VMSdatacall/Results" #Location where you want to store the results

#- Setting specific thresholds
spThres <- 20 #Maximum speed threshold in analyses in nm
intThres <- 5 #Minimum difference in time interval in minutes to prevent pseudo duplicates
intvThres <- 240 #Maximum difference in time interval in minutes to prevent intervals being too large to be realistic
lanThres <- 1.5 #Maximum difference in log10-transformed sorted weights

#- Re-run all years as we have new field for no. vessels
yearsToSubmit <- sort(2009:2020)

#- Set the gear names for which automatic fishing activity is wanted
# It is important to fill out the gears you want to apply auto detection for
autoDetectionGears <- c("TBB","OTB","OTT","SSC","SDN","DRB","PTB","HMD")

#- Decide if you want to visualy analyse speed-histograms to identify fishing activity
# peaks or have prior knowledge and use the template provided around lines 380 below
visualInspection <- FALSE

#- Specify how landings should be distributed over the VMS pings: By day, ICES rectangle, trip basis or otherwise
linkEflaloTacsat <- c("day","ICESrectangle","trip")
# other options
# linkEflaloTacsat <- c("day","ICESrectangle","trip")
# linkEflaloTacsat <- c("ICESrectangle","trip")
# linkEflaloTacsat <- c("day","trip")
# linkEflaloTacsat <- c("trip")
Loading