Changes to ZSYSTEM_FILTER/PIPE_FILTER in restrict.txt are ignored if they happen within 1 second #318

nars1 · 2018-07-24T21:22:24Z

Final Release Note

Description

With GT.M V6.3-005, one can specify M code to run a filter that decides whether to allow a restriction or not and to control what commands actually get run when a user specifies ZSYSTEM or a PIPE OPEN command (see http://tinco.pair.com/bhaskar/gtm/doc/articles/GTM_V6.3-005_Release_Notes.html#GTM-8877 for details). While testing this feature out, we noticed that if the $ydb_dist/restrict.txt or $gtm_dist/restrict.txt file is updated multiple times within the same second, where the first update included one M labelref for ZSYSTEM_FILTER/PIPE_FILTER and the second update included a different M labelref, the second (and later) M labelref did not take effect when a ZSYSTEM or PIPE OPEN command ran afterwards. The first M labelref still took effect incorrectly.

Draft Release Note

Changes to ZSYSTEM_FILTER/PIPE_FILTER in restrict.txt (which resides in $ydb_dist/$gtm_dist) correctly take effect for future ZSYSTEM/PIPE OPEN commands. In GT.M V6.3-005, where this feature was introduced, if restrict.txt is changed multiple times within the same second for ZSYSTEM_FILTER/PIPE_FILTER, the changes did not take effect. A workaround was to "touch" the restrict.txt file afterwards to make its timestamp newer (relative to an auto-generated file filter_commands.tab).

…re ignored if they happen within 1 second M labelrefs specified in ZSYSTEM_FILTER/PIPE_FILTER in $ydb_dist/restrict.txt were stored in a table format suitable for a call-in invocation in the file $ydb_dist/filter_commands.tab. Any newer changes to restrict.txt relative to filter_commands.tab required recreating filter_commands.tab. This check was done using 1-second level timestamp granularity which is too coarse. This is now fixed to do nanosecond-level timestamp checking. That ensures multiple changes to restrict.txt within the same second result in recreation of filter_commands.tab. In addition, if the timestamp matches at the nanosecond level, a recreation is done (whereas previously if the timestamp at the second level matched, no recreation was done). This is a just-to-be-safe change.

…red if they happen within 1 second (#319) M labelrefs specified in ZSYSTEM_FILTER/PIPE_FILTER in $ydb_dist/restrict.txt were stored in a table format suitable for a call-in invocation in the file $ydb_dist/filter_commands.tab. Any newer changes to restrict.txt relative to filter_commands.tab required recreating filter_commands.tab. This check was done using 1-second level timestamp granularity which is too coarse. This is now fixed to do nanosecond-level timestamp checking. That ensures multiple changes to restrict.txt within the same second result in recreation of filter_commands.tab. In addition, if the timestamp matches at the nanosecond level, a recreation is done (whereas previously if the timestamp at the second level matched, no recreation was done). This is a just-to-be-safe change.

) (#339) Tests the following release note: The GT.M restriction mechanism recognizes the following lines: ZSYSTEM_FILTER[:M labelref] PIPE_FILTER[:M labelref] The labelref must include a routine name. If a process is restricted by a ZSYSTEM or PIPE_OPEN line in the restrictions file that restriction takes precedence over the corresponding filter restriction. Otherwise when a process is subject to these restrictions, GT.M inserts an invocation of the labelref prior to the restricted command, passing a string containing the argument to the ZSYSTEM command or the command deviceparameter of the PIPE OPEN. The path to the filter routine must be included in $zroutines. FIS recommends that the filter routine is placed in a location with restricted access such as $gtm_dist. If the filter invocation return is -1,GT.M produces a RESTRICTEDOP error, otherwise it executes the command using the returned string via output parameters as a, possibly identical, replacement for the original string. Since GT.M uses the call-ins mechanism to execute the filters, a filter invocation inside a TP transaction in call-ins produces a CITPNESTED error. Note that because ZSYSTEM and OPEN are not Isolated actions FIS recommends against their use within a TP transaction. Filters also increment the nested level of call-ins. A recursive filter invocation produces a NOFILTERNEST error. GT.M reports all filter errors to the operator log accompanied by a COMMFILTERERR. An example restrict file for this ZSYSTEM_FILTER:^filterzsy PIPE_FILTER:^filterzsy The actual filter routine: ``` filterzsy(inarg,outarg); if ""=inarg set outarg="-1;must provide a command" quit for i=1:1 set arg=$piece(inarg,";",i) quit:""=arg do quit:$data(outarg) . for quit:$zchar(9,32)'[$extract(arg) set arg=$extract(arg,2,9999) . set cmd=$piece(arg," ") . for restrict="sudo","cd" if cmd=restrict set outarg="-1;command "_restrict_" not permitted" quit . quit:$data(outarg) . if "echo"=cmd set $piece(arg," ")="echo #",$piece(inarg,";",i)=arg ;example of modification set:'$data(outarg) outarg=inarg quit +outarg ``` Filter execution starts with $STACK=1 ($ZLEVEL=2). Following are the GT.M commands, Intrinsic Special Variables, and functions whose behavior changes in the context of a filter invocation. ZGOTO 0 (zero) returns to the processing of the restricted command as does ZGOTO 1 (one) with no entryref, while ZGOTO 1:entryref replaces the originally invoked filter and continues filter execution. $ZTRAP/$ETRAP NEW'd at level 1. $ZLEVEL initializes to one (1) in GTM$CI, and increments for every new stack level. $STACK initializes to zero (0) in GTM$CI frame, and increments for every new stack level. $ESTACK NEW'd at level one (1) in GTM$CI frame. $ECODE/$STACK() initialized to the empty string at level one (1) in GTM$CI frame. After the filter completes, GT.M restores the above to their values at the invocation of the filter. (GTM-8877) Also tests YottaDB/YottaDB318: Changes to ZSYSTEM_FILTER/PIPE_FILTER in restrict.txt (which resides in $ydb_dist/$gtm_dist) correctly take effect for future ZSYSTEM/PIPE OPEN commands. In GT.M V6.3-005, where this feature was introduced, if restrict.txt is changed multiple times within the same second for ZSYSTEM_FILTER/PIPE_FILTER, the changes did not take effect. A workaround was to "touch" the restrict.txt file afterwards to make its timestamp newer (relative to an auto-generated file filter_commands.tab).

…trict.c The check for whether one file is newer than the other was done as follows in a prior commit. (rmtime.tv_sec > fmtime.tv_sec) || (rmtime.tv_nsec >= fmtime.tv_nsec) But this will result in a false evaluation in the case rmtime.tv_sec < fmtime.tv_sec but rmtime.tv_nsec > fmtime.tv_nsec. That is what happened in an in-house test on the Raspberry Pi3 boxes which caused the filter_commands.tab file to be recreated in the middle of adding lines from a multi-line restrict.txt file. The check is now corrected to read as follows (rmtime.tv_sec > fmtime.tv_sec) || (rmtime.tv_sec == fmtime.tv_sec) && (rmtime.tv_nsec >= fmtime.tv_nsec) With this check, the test reliably passes on the Pi3 boxes.

…#327) The check for whether one file is newer than the other was done as follows in a prior commit. (rmtime.tv_sec > fmtime.tv_sec) || (rmtime.tv_nsec >= fmtime.tv_nsec) But this will result in a false evaluation in the case rmtime.tv_sec < fmtime.tv_sec but rmtime.tv_nsec > fmtime.tv_nsec. That is what happened in an in-house test on the Raspberry Pi boxes which caused the filter_commands.tab file to be recreated in the middle of adding lines from a multi-line restrict.txt file. The check is now corrected to read as follows (rmtime.tv_sec > fmtime.tv_sec) || (rmtime.tv_sec == fmtime.tv_sec) && (rmtime.tv_nsec >= fmtime.tv_nsec) With this check, the test reliably passes on the Raspberry Pi and continues to pass on Linux x86_64.

…multiple lines of restrict.txt We had a v63005/gtm8877 subtest failure where line 50 below issued a COMMFILTERERR error. gtm8877.csh ----------- 42 # FILTER RETURNING A STRING 43 rm $ydb_dist/restrict.txt 44 cat > $ydb_dist/restrict.txt << EOF 45 ZSYSTEM_FILTER:filterfn2^gtm8877 46 PIPE_FILTER:filterfn2^gtm8877 47 EOF 48 chmod -w $ydb_dist/restrict.txt 49 echo "# FILTER RETURNING A STRING" 50 $ydb_dist/mumps -run zsystemfn^gtm8877 51 $ydb_dist/mumps -run pipeopenfn^gtm8877 52 echo "" Expected output --------------- filtered output filtered output Actual output ------------- %YDB-E-COMMFILTERERR, Error executing the command filter for gtmzsystem. 150379666,(Call-In),%YDB-E-CINOENTRY, No entry specified for gtmzsystem in the call-in table filtered output We clearly added ZSYSTEM_FILTER just before PIPE_FILTER in line 45 of gtm8877.csh above but that is found missing when we actually run it. This test passed almost always so this was some edge case. Looking at the code, for each line in restrict.txt, restrict_init() checks if ZSYSTEM_FILTER or PIPE_FILTER is defined and if so invokes the following macro. restrict.c ---------- 69 #define PUT_FLNAME_IN_MAPPING_FILE(RPATH, FPATH, FP, C_CALL_NAME, M_REF_NAME, STAT_RM, SAVE_ERRNO, ERR_STR) \ 70 { \ 71 if (!ACCESS(FPATH, F_OK)) /* Filter file exists, now check modified time */ \ 72 { \ 73 Stat(RPATH, &rTime); \ 74 rmtime = rTime.st_mtim; \ 75 Stat(FPATH, &fTime); \ 76 fmtime = fTime.st_mtim; \ 77 /* Check if restrict.txt file modification time (rmtime) is newer than \ 78 * filter_commands.tab file modification time (fmtime). If so, recreate filter_commands.tab. \ 79 */ \ 80 if ((rmtime.tv_sec > fmtime.tv_sec) \ 81 || ((rmtime.tv_sec == fmtime.tv_sec) && (rmtime.tv_nsec >= fmtime.tv_nsec))) \ 82 { /* Delete the older mapping file and recreate new if required */ \ 83 created_now = TRUE; \ 84 gtm_file_remove(FPATH, strlen(FPATH), &STAT_RM); \ 85 append_filter(FPATH, FP, C_CALL_NAME, M_REF_NAME, SAVE_ERRNO, ERR_STR); \ 86 } else if (created_now) /* This process created a new file,append to it */ \ 87 { \ 88 append_filter(FPATH, FP, C_CALL_NAME, M_REF_NAME, SAVE_ERRNO, ERR_STR); \ 89 } \ The macro checks if the file filter_commands.tab exists and if so compares the file modification time of that against restrict.txt. If the comparison shows restrict.txt is newer, it recreates filter_commands.tab. Otherwise, it checks if we had created the file now ("created_now" variable is TRUE) and if so appends the current filter line (ZSYSTEM_FILTER or PIPE_FILTER) onto filter_commands.tab. And otherwise, it does nothing because the two are supposedly in sync. And that makes the following scenario (and the test failure) possible. 1) PUT_FLNAME_IN_MAPPING_FILE macro is called for ZSYSTEM_FILTER line in restrict.txt. It decides filter_commands.tab needs to be created and adds a line for ZSYSTEM_FILTER. 2) PUT_FLNAME_IN_MAPPING_FILE macro is called for PIPE_FILTER line in restrict.txt. And it finds that the nanosecond-level timestamp matches between restrict.txt and filter_commands.tab. Therefore decides to recreate the file again while still processing the same restrict.txt. This means the line added to filter_commands.tab in Step 1 is lost. The check at line 86 is done after the check at line 80 whereas it should be done before. If we created this file because of a prior call to PUT_FLNAME_IN_MAPPING_FILE macro, we should append to it unconditionally. Otherwise we should do the file modification check. This is now fixed. In addition, once the PUT_FLNAME_IN_MAPPING_FILE macro is called once in restrict_init(), there is no need to do any more system calls (ACCESS/access or Stat/stat calls) as we know whether we need to append to a newly created file ("created_now" variable is TRUE) OR we need to do nothing (because the first invocation of the macro found filter_commands.tab is newer than restrict.txt. Whereas the current code makes unnecessary access() and stat() calls. That is also fixed. While at this, the macro has been enhanced to pass the "created_now" variable across instead of silently using/modifying it. A new variable "created_now_initialized" indicates whether "created_now" is usable or not (i.e. the PUT_FLNAME_IN_MAPPING_FILE macro has been called once). And that is also passed to the macro.

…e lines of restrict.txt (#332) We had a v63005/gtm8877 subtest failure where line 50 below issued a COMMFILTERERR error. gtm8877.csh ----------- 42 # FILTER RETURNING A STRING 43 rm $ydb_dist/restrict.txt 44 cat > $ydb_dist/restrict.txt << EOF 45 ZSYSTEM_FILTER:filterfn2^gtm8877 46 PIPE_FILTER:filterfn2^gtm8877 47 EOF 48 chmod -w $ydb_dist/restrict.txt 49 echo "# FILTER RETURNING A STRING" 50 $ydb_dist/mumps -run zsystemfn^gtm8877 51 $ydb_dist/mumps -run pipeopenfn^gtm8877 52 echo "" Expected output --------------- filtered output filtered output Actual output ------------- %YDB-E-COMMFILTERERR, Error executing the command filter for gtmzsystem. 150379666,(Call-In),%YDB-E-CINOENTRY, No entry specified for gtmzsystem in the call-in table filtered output We clearly added ZSYSTEM_FILTER just before PIPE_FILTER in line 45 of gtm8877.csh above but that is found missing when we actually run it. This test passed almost always so this was some edge case. Looking at the code, for each line in restrict.txt, restrict_init() checks if ZSYSTEM_FILTER or PIPE_FILTER is defined and if so invokes the following macro. restrict.c ---------- 69 #define PUT_FLNAME_IN_MAPPING_FILE(RPATH, FPATH, FP, C_CALL_NAME, M_REF_NAME, STAT_RM, SAVE_ERRNO, ERR_STR) \ 70 { \ 71 if (!ACCESS(FPATH, F_OK)) /* Filter file exists, now check modified time */ \ 72 { \ 73 Stat(RPATH, &rTime); \ 74 rmtime = rTime.st_mtim; \ 75 Stat(FPATH, &fTime); \ 76 fmtime = fTime.st_mtim; \ 77 /* Check if restrict.txt file modification time (rmtime) is newer than \ 78 * filter_commands.tab file modification time (fmtime). If so, recreate filter_commands.tab. \ 79 */ \ 80 if ((rmtime.tv_sec > fmtime.tv_sec) \ 81 || ((rmtime.tv_sec == fmtime.tv_sec) && (rmtime.tv_nsec >= fmtime.tv_nsec))) \ 82 { /* Delete the older mapping file and recreate new if required */ \ 83 created_now = TRUE; \ 84 gtm_file_remove(FPATH, strlen(FPATH), &STAT_RM); \ 85 append_filter(FPATH, FP, C_CALL_NAME, M_REF_NAME, SAVE_ERRNO, ERR_STR); \ 86 } else if (created_now) /* This process created a new file,append to it */ \ 87 { \ 88 append_filter(FPATH, FP, C_CALL_NAME, M_REF_NAME, SAVE_ERRNO, ERR_STR); \ 89 } \ The macro checks if the file filter_commands.tab exists and if so compares the file modification time of that against restrict.txt. If the comparison shows restrict.txt is newer, it recreates filter_commands.tab. Otherwise, it checks if we had created the file now ("created_now" variable is TRUE) and if so appends the current filter line (ZSYSTEM_FILTER or PIPE_FILTER) onto filter_commands.tab. And otherwise, it does nothing because the two are supposedly in sync. And that makes the following scenario (and the test failure) possible. 1) PUT_FLNAME_IN_MAPPING_FILE macro is called for ZSYSTEM_FILTER line in restrict.txt. It decides filter_commands.tab needs to be created and adds a line for ZSYSTEM_FILTER. 2) PUT_FLNAME_IN_MAPPING_FILE macro is called for PIPE_FILTER line in restrict.txt. And it finds that the nanosecond-level timestamp matches between restrict.txt and filter_commands.tab. Therefore decides to recreate the file again while still processing the same restrict.txt. This means the line added to filter_commands.tab in Step 1 is lost. The check at line 86 is done after the check at line 80 whereas it should be done before. If we created this file because of a prior call to PUT_FLNAME_IN_MAPPING_FILE macro, we should append to it unconditionally. Otherwise we should do the file modification check. This is now fixed. In addition, once the PUT_FLNAME_IN_MAPPING_FILE macro is called once in restrict_init(), there is no need to do any more system calls (ACCESS/access or Stat/stat calls) as we know whether we need to append to a newly created file ("created_now" variable is TRUE) OR we need to do nothing (because the first invocation of the macro found filter_commands.tab is newer than restrict.txt. Whereas the current code makes unnecessary access() and stat() calls. That is also fixed. While at this, the macro has been enhanced to pass the "created_now" variable across instead of silently using/modifying it. A new variable "created_now_initialized" indicates whether "created_now" is usable or not (i.e. the PUT_FLNAME_IN_MAPPING_FILE macro has been called once). And that is also passed to the macro.

nars1 added the upstream issue label Jul 24, 2018

nars1 added this to the r124 milestone Jul 24, 2018

nars1 self-assigned this Jul 24, 2018

nars1 closed this as completed Jul 24, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Changes to ZSYSTEM_FILTER/PIPE_FILTER in restrict.txt are ignored if they happen within 1 second #318

Changes to ZSYSTEM_FILTER/PIPE_FILTER in restrict.txt are ignored if they happen within 1 second #318

nars1 commented Jul 24, 2018

Changes to ZSYSTEM_FILTER/PIPE_FILTER in restrict.txt are ignored if they happen within 1 second #318

Changes to ZSYSTEM_FILTER/PIPE_FILTER in restrict.txt are ignored if they happen within 1 second #318

Comments

nars1 commented Jul 24, 2018

Final Release Note

Description

Draft Release Note