SARIF OM changes for nullable int batch 1 #2650

shaopeng-gh · 2023-03-28T06:02:05Z

NEW: Update Microsoft.Json.Schema.* packages from 2.1.0.0 to 2.3.0.0. #2650
BRK: Update Artifact.Offset, Invocation.ExitCode, Invocation.ExitSignalNumber, Invocation.ProcessId, Result.OccurrenceCount, EdgeTraversal.StepOverEdgeCount, Notification.ThreadId, StackFrame.ThreadId, ThreadFlowLocation.NestingLevel, and WebResponse.StatusCode properties from int to int? type. #2650

The reason of this change is, the upgrade of the Microsoft.Json.Schema.* packages exposed an issue that a few int properties should be nullable but it is current not nullable in our C# code.

Due to impact of the 4 line/columns numbers for Region, proposed for a separated PR and will need a confirmation for the details before start.

shaopeng-gh · 2023-03-28T06:46:48Z

src/Samples/SarifToCsv/SarifCsvColumnWriters.cs

@@ -106,7 +106,7 @@ public static Action<WriteContext> GetWriter(string columnName)
            writers["Kind"] = (c) => { c.Writer.Write(c.Result.Kind.ToString()); };
            writers["Level"] = (c) => { c.Writer.Write(c.Result.Level.ToString()); };
            writers["Message.Text"] = (c) => { c.Writer.Write(c.Result.Message?.Text ?? ""); };
-            writers["OccurrenceCount"] = (c) => { c.Writer.Write(c.Result.OccurrenceCount); };
+            writers["OccurrenceCount"] = (c) => { c.Writer.Write(c.Result.OccurrenceCount?.ToString() ?? ""); };


writers["OccurrenceCount"] = (c) => { c.Writer.Write(c.Result.OccurrenceCount?.ToString() ?? ""); };

note, this file is from sample solution. #Closed

shaopeng-gh · 2023-03-28T06:48:49Z

src/Sarif/Schemata/sarif-2.1.0-rtm.6.json

@@ -1740,6 +1744,7 @@
        "charLength": {


charLength

for charLength and byteLength, in the spec they indeed have the default value 0. I think we missed it. #Closed

shaopeng-gh · 2023-03-28T06:53:03Z

src/Sarif/Schemata/sarif-2.1.0-rtm.6.json

@@ -1709,24 +1709,28 @@
        "startLine": {
          "description": "The line number of the first character in the region.",
          "type": "integer",
+          "default": 0,


"default": 0,

for the 4 number properties, we will do in a separated PR.
This change is temporary just for skip them so they will not be generated as nullable, and the PR is against the dev branch, we will get to it before to main, so this change will not show up in main at all. #Closed

so what's odd about this is we have a default of -1 for other things, why not this?

how can a default be lower than the minimum value? seems odd.

shaopeng-gh · 2023-03-28T06:55:56Z

src/Sarif/Autogenerated/Region.cs

@@ -39,24 +39,32 @@ public virtual SarifNodeKind SarifNodeKind
        /// The line number of the first character in the region.
        /// </summary>
        [DataMember(Name = "startLine", IsRequired = false, EmitDefaultValue = false)]
+        [DefaultValue(0)]


[DefaultValue(0)]

this file is generated from the RTM6 json schema, please see my comment on that file in this PR. #WontFix

Right, so we're making the same mistake though! '0' is a default value that effectively means it isn't there, so we should have a nullable type here, again.

This is more painful from an API use standpoint, maybe too painful! But everything would be consistent.

I do see from the spec that the region class in particular has complex/unusual default value semantics...

michaelcfanning · 2023-03-30T23:04:46Z

src/Sarif/CodeGenHints.json

@@ -221,6 +221,16 @@
      }
    }
  ],
+  "ExternalProperties.Schema": [


Schema

how did we miss this? #Closed

replied in the other comment.

michaelcfanning · 2023-03-30T23:05:49Z

src/Sarif/CodeGenHints.json

@@ -1075,6 +1095,26 @@
      }
    }
  ],
+  "TranslationMetadata.DownloadUri": [


DownloadUri

Good changes, but hm. how'd you find them? #Resolved

not sure how we missed them.
I found them by test not eye.

michaelcfanning · 2023-03-30T23:06:34Z

    [DefaultValue(-1)]

why is this still here? :) when will it go away?

In reply to: 1491078338

Refers to: src/Sarif/Autogenerated/Address.cs:43 in fefed9a. [](commit_id = fefed9a, deletion_comment = False)

michaelcfanning · 2023-03-30T23:09:03Z

src/Sarif/Autogenerated/ExternalProperties.cs

@@ -38,6 +38,7 @@ public virtual SarifNodeKind SarifNodeKind
        /// The URI of the JSON schema corresponding to the version of the external property file format.
        /// </summary>
        [DataMember(Name = "schema", IsRequired = false, EmitDefaultValue = false)]
+        [JsonConverter(typeof(Microsoft.CodeAnalysis.Sarif.Readers.UriConverter))]
        public virtual Uri Schema { get; set; }


Uri

There's a 1x1 mapping between this type, URI and the JsonConverter attribute. That could have been the way we handled this in the codegen hints + jschema (i.e., provide a way to emit a specific attribute for every instance of a property with a specific return type). Just an idea, can you confirm this feature does not already exist? #Resolved

The UriConverter class is from SDK not JSchama so we can not directly always generate this attribute when Uri type in JSchema.

We could have support hint that base on type, not just name, then we can have all this declared once. I checked the hint is only based on name.

An other way is we could have still use base on name but support partial wild card e.g. *.*.Uri* then we can have it declared declared once, except this kind of special one Schema that does no have Uri in the name.
But I checked we only support * at the first part only: *.Uri.

private static string MakeHintDictionaryKey(string typeName, string propertyName) { return typeName + "." + propertyName.ToPascalCase(); }

Let me know which direction, I can create a work item and work on it.

michaelcfanning · 2023-03-30T23:15:44Z

src/Sarif/Autogenerated/Region.cs

@@ -39,24 +39,32 @@ public virtual SarifNodeKind SarifNodeKind
        /// The line number of the first character in the region.
        /// </summary>
        [DataMember(Name = "startLine", IsRequired = false, EmitDefaultValue = false)]
+        [DefaultValue(0)]


DefaultValue

have you proven this is required?

the value of 0 is a default for start line, so is the ignore and populate attribute sufficient here? i.e., no need to specify the zero value? #WontFix

michaelcfanning · 2023-03-30T23:16:16Z

src/Sarif/Autogenerated/Region.cs

@@ -116,8 +128,14 @@ public virtual SarifNodeKind SarifNodeKind
        /// </summary>
        public Region()
        {
+            StartLine = 0;


StartLine

this is all unnecessary from the .NET standpoint, it's just wasted CPU cycles. #WontFix

correct, there are 2 thing here:

the line column numbers will be gone after we work on it and regenerates, for this PR we can ignore all changes related to them for now,

CharLength and ByteLength is a real change here. It is autogenerated.
I have created an issue in JSchema:
Improvement: remove unnecessary initialization of property default value in ToDotNet jschema#172

michaelcfanning · 2023-03-30T23:27:01Z

src/Sarif/NotYetAutoGenerated/Address.cs

@@ -15,7 +15,7 @@ namespace Microsoft.CodeAnalysis.Sarif
    /// A physical or virtual address, or a range of addresses, in an 'addressable region' (memory or a binary file).
    /// </summary>
    [DataContract]
-    [GeneratedCode("Microsoft.Json.Schema.ToDotNet", "2.1.0.0")]
+    [GeneratedCode("Microsoft.Json.Schema.ToDotNet", "2.3.0.0")]
    public partial class Address : PropertyBagHolder, ISarifNode


partial

so do you know for sure not that every file in notyetautogenerated contains other code changes that need to be preserved? or can we delete any of these? #WontFix

Yes I did have compared that every file in the notyetautogenerated folder have difference between the raw auto generated one, when work on these I believe some tests are needed, I think for the separate change.

michaelcfanning · 2023-03-30T23:28:02Z

src/Sarif/Query/Evaluators/SarifEvaluators.cs

@@ -29,7 +29,7 @@ public static IExpressionEvaluator<Result> ResultEvaluator(TermExpression term)
                case "message.text":
                    return new StringEvaluator<Result>(r => r.Message?.Text, term, StringComparison.OrdinalIgnoreCase);
                case "occurrencecount":
-                    return new LongEvaluator<Result>(r => r.OccurrenceCount, term);
+                    return new LongEvaluator<Result>(r => r.OccurrenceCount ?? 1, term);


OccurrenceCount

gee look at this, this hadn't occurred to me! occurrence count's default is 1!!

wonder if we should include this in a spec errata #Resolved

wonder if we should include this in a spec errata ---- I agree with this and will need to talk to you if we consider "the spec is missing this default as 1 by mistake". If so we can keep the OccurrenceCount as not nullable and have default as 1.

All types with a default must be nullable!! We can discuss in stand-up today. :)

All types with a default must be nullable!! ---- @michaelcfanning I have question about this because it is the opposite of what my understanding, my understanding and also my PR that lead to this was that was reviewed and checked in.
I created an email to describe details.
Also please review that change that was checked in: microsoft/jschema#167

And because integers without default we HAVE to make them nullable, if with a default you also want them to be nullable that would be basically everything should be nullable.

michaelcfanning · 2023-03-30T23:30:56Z

      "default": -1,

observe that the minimum value here matches the default value!

In reply to: 1491093798

Refers to: src/Sarif/Schemata/sarif-2.1.0-rtm.6.json:1740 in fefed9a. [](commit_id = fefed9a, deletion_comment = False)

michaelcfanning · 2023-03-30T23:31:50Z

src/Sarif/Schemata/sarif-2.1.0-rtm.6.json

@@ -1709,24 +1709,28 @@
        "startLine": {
          "description": "The line number of the first character in the region.",
          "type": "integer",
+          "default": 0,


default

something is wrong here. if this is the default, then minimum should be set as 0 as well. #WontFix

michaelcfanning · 2023-03-30T23:34:07Z

src/Sarif.Converters/StaticDriverVerifierConverter.cs

@@ -225,6 +225,11 @@ private void ProcessLine(string logFileLine, ref int nestingLevel, Result result
                    }
                }

+                if (threadFlowLocation.NestingLevel == 0)


NestingLevel

why is this correct?

how do we distinguish between not present vs. an actual nesting level of 0? #Resolved

you are correct, this is code for converting StaticDriverVerifier result to SARIF, the NestingLevel here is converted so if it is 0 then it means 0, not unknown. I have reverted this change, and fixed the test expected outputs to have this property.

michaelcfanning · 2023-03-30T23:35:04Z

...lTests.Sarif/TestData/Multitool/ValidateCommand/Inputs/SARIF1002.UrisMustBeValid_Valid.sarif

@@ -43,7 +43,7 @@
        },
        {
          "location": {
-            "uri": "file:/c:/src/file.c",
+            "uri": "file:///c:/src/file.c",


file

i see! #Closed

michaelcfanning · 2023-03-30T23:35:38Z

src/Test.UnitTests.Sarif/Writers/SarifLoggerTests.cs

@@ -680,7 +680,7 @@ public void SarifLogger_LogsStartAndEndTimesByDefault()
            // Other properties should be empty.
            invocation.CommandLine.Should().BeNull();
            invocation.WorkingDirectory.Should().BeNull();
-            invocation.ProcessId.Should().Be(0);
+            invocation.ProcessId.Should().Be(null);


Be(null)

Just use .BeNull() here? #Resolved

correct, batch fixed all existing ones.

michaelcfanning · 2023-03-31T14:23:32Z

src/Sarif/Schemata/sarif-2.1.0-rtm.6.json

@@ -1709,24 +1709,28 @@
        "startLine": {
          "description": "The line number of the first character in the region.",
          "type": "integer",
+          "default": 0,
          "minimum": 1


"minimum": 1

set all minimums to default and then let's please make every int value that has a default in the schema be a nullable item.

of course, we are skipping region for now. I wonder if we should build a special region deserializer! :) #WontFix

github-advanced-security · 2023-04-04T03:02:19Z

You have successfully added a new CodeQL configuration .github/workflows/codeql-analysis.yml:analyze. As part of the setup process, we have scanned this repository and found 6 existing alerts. Please check the repository Security tab to see all alerts.

src/Sarif/Autogenerated/Region.cs

            CharOffset = -1;
+            CharLength = 0;


src/Sarif/Autogenerated/Region.cs

            ByteOffset = -1;
+            ByteLength = 0;


src/Sarif/Autogenerated/Region.cs

@@ -116,8 +128,14 @@
        /// </summary>
        public Region()
        {
+            StartLine = 0;


src/Sarif/Autogenerated/Region.cs

@@ -116,8 +128,14 @@
        /// </summary>
        public Region()
        {
+            StartLine = 0;
+            StartColumn = 0;


src/Sarif/Autogenerated/Region.cs

@@ -116,8 +128,14 @@
        /// </summary>
        public Region()
        {
+            StartLine = 0;
+            StartColumn = 0;
+            EndLine = 0;


src/Sarif/Autogenerated/Region.cs

+            StartLine = 0;
+            StartColumn = 0;
+            EndLine = 0;
+            EndColumn = 0;


shaopeng-gh · 2023-04-04T22:27:07Z

Thanks, I see your comments about the changes about the 4 line/column numbers.
You are correct that all changes related to them are wrong.
The reason is I want to skip these 4 fields and keep them not nullable now, and because they are auto generated, the way I skip them is to add a default value to them. Can be any integer but why I choose 0 which is less than the minimum 1? Because it is actually the effective default, if not most of the tests will fail.
Those temporary changes related to them are auto generated which will be auto generated again once we work on these 4 fields.
So we can ignore any changes related to them now.
Also this will go to DEV branch first so will not get merged to main before we get to these 4 fields.
I put the comment where and will close those ones, to avoid duplicated comments to look.

shaopeng-gh · 2023-04-04T22:31:47Z

    [DefaultValue(-1)]

Will need to fix this in the future changes, this change set is only those triggered by the new version of JSchema, must act or else will not build.

In reply to: 1491078338

Refers to: src/Sarif/Autogenerated/Address.cs:43 in fefed9a. [](commit_id = fefed9a, deletion_comment = False)

shaopeng-gh and others added 11 commits March 20, 2023 10:32

Update JSchema from 2.1.0 to 2.3.0

b2755ce

default auto generated

9e01d59

Fix default for Region.StartColumn, Region.CharLength, Region.ByteLength

c7a4e9e

Merge branch 'main' into users/shaopeng-gh/updatejschema

8191636

Skip 4

5db7146

Fix tests

aff3990

Release History

22e2a6d

Fix release history

9ad7bec

Update release history

6301d31

update release history

1e24696

update release history

fefed9a

shaopeng-gh marked this pull request as ready for review March 28, 2023 06:39

shaopeng-gh requested review from michaelcfanning, EasyRhinoMSFT, marmegh and cfaucon as code owners March 28, 2023 06:39

shaopeng-gh commented Mar 28, 2023

View reviewed changes

michaelcfanning reviewed Mar 30, 2023

View reviewed changes

michaelcfanning reviewed Mar 31, 2023

View reviewed changes

shaopeng-gh marked this pull request as draft April 4, 2023 02:19

Batch fix BeNull()

4853307

github-advanced-security bot found potential problems Apr 4, 2023

View reviewed changes

Fix StaticDriverVerifierConverter NestingLevel

96fd780

shaopeng-gh marked this pull request as ready for review April 4, 2023 06:26

shaopeng-gh changed the title ~~SARIF OM changes for nullable int~~ SARIF OM changes for nullable int batch 1 Apr 5, 2023

michaelcfanning merged commit 3ca7c4c into dev/SARIF_OM_Changes Apr 5, 2023

michaelcfanning deleted the users/shaopeng-gh/updatejschema_skip4 branch April 5, 2023 19:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SARIF OM changes for nullable int batch 1 #2650

SARIF OM changes for nullable int batch 1 #2650

shaopeng-gh commented Mar 28, 2023 •

edited

Loading

shaopeng-gh Mar 28, 2023 •

edited

Loading

shaopeng-gh Mar 28, 2023 •

edited

Loading

shaopeng-gh Mar 28, 2023 •

edited

Loading

michaelcfanning Mar 30, 2023

shaopeng-gh Mar 28, 2023 •

edited

Loading

michaelcfanning Mar 30, 2023

michaelcfanning Mar 30, 2023 •

edited by shaopeng-gh

Loading

shaopeng-gh Apr 4, 2023

michaelcfanning Mar 30, 2023 •

edited by shaopeng-gh

Loading

shaopeng-gh Apr 4, 2023

michaelcfanning commented Mar 30, 2023 •

edited by shaopeng-gh

Loading

michaelcfanning Mar 30, 2023 •

edited by shaopeng-gh

Loading

shaopeng-gh Apr 4, 2023 •

edited

Loading

michaelcfanning Mar 30, 2023 •

edited by shaopeng-gh

Loading

michaelcfanning Mar 30, 2023 •

edited by shaopeng-gh

Loading

shaopeng-gh Apr 4, 2023 •

edited

Loading

michaelcfanning Mar 30, 2023 •

edited by shaopeng-gh

Loading

shaopeng-gh Apr 4, 2023

michaelcfanning Mar 30, 2023 •

edited by shaopeng-gh

Loading

shaopeng-gh Apr 4, 2023 •

edited

Loading

michaelcfanning Apr 4, 2023

shaopeng-gh Apr 4, 2023 •

edited

Loading

michaelcfanning commented Mar 30, 2023 •

edited by shaopeng-gh

Loading

michaelcfanning Mar 30, 2023 •

edited by shaopeng-gh

Loading

michaelcfanning Mar 30, 2023 •

edited by shaopeng-gh

Loading

shaopeng-gh Apr 4, 2023

michaelcfanning Mar 30, 2023 •

edited by shaopeng-gh

Loading

michaelcfanning Mar 30, 2023 •

edited by shaopeng-gh

Loading

shaopeng-gh Apr 4, 2023

michaelcfanning Mar 31, 2023 •

edited by shaopeng-gh

Loading

github-advanced-security bot commented Apr 4, 2023

shaopeng-gh commented Apr 4, 2023

shaopeng-gh commented Apr 4, 2023

SARIF OM changes for nullable int batch 1 #2650

SARIF OM changes for nullable int batch 1 #2650

Conversation

shaopeng-gh commented Mar 28, 2023 • edited Loading

shaopeng-gh Mar 28, 2023 • edited Loading

Choose a reason for hiding this comment

shaopeng-gh Mar 28, 2023 • edited Loading

Choose a reason for hiding this comment

shaopeng-gh Mar 28, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shaopeng-gh Mar 28, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michaelcfanning Mar 30, 2023 • edited by shaopeng-gh Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michaelcfanning Mar 30, 2023 • edited by shaopeng-gh Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michaelcfanning commented Mar 30, 2023 • edited by shaopeng-gh Loading

michaelcfanning Mar 30, 2023 • edited by shaopeng-gh Loading

Choose a reason for hiding this comment

shaopeng-gh Apr 4, 2023 • edited Loading

Choose a reason for hiding this comment

michaelcfanning Mar 30, 2023 • edited by shaopeng-gh Loading

Choose a reason for hiding this comment

michaelcfanning Mar 30, 2023 • edited by shaopeng-gh Loading

Choose a reason for hiding this comment

shaopeng-gh Apr 4, 2023 • edited Loading

Choose a reason for hiding this comment

michaelcfanning Mar 30, 2023 • edited by shaopeng-gh Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michaelcfanning Mar 30, 2023 • edited by shaopeng-gh Loading

Choose a reason for hiding this comment

shaopeng-gh Apr 4, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shaopeng-gh Apr 4, 2023 • edited Loading

Choose a reason for hiding this comment

michaelcfanning commented Mar 30, 2023 • edited by shaopeng-gh Loading

michaelcfanning Mar 30, 2023 • edited by shaopeng-gh Loading

Choose a reason for hiding this comment

michaelcfanning Mar 30, 2023 • edited by shaopeng-gh Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michaelcfanning Mar 30, 2023 • edited by shaopeng-gh Loading

Choose a reason for hiding this comment

michaelcfanning Mar 30, 2023 • edited by shaopeng-gh Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michaelcfanning Mar 31, 2023 • edited by shaopeng-gh Loading

Choose a reason for hiding this comment

github-advanced-security bot commented Apr 4, 2023

shaopeng-gh commented Apr 4, 2023

shaopeng-gh commented Apr 4, 2023

shaopeng-gh commented Mar 28, 2023 •

edited

Loading

shaopeng-gh Mar 28, 2023 •

edited

Loading

shaopeng-gh Mar 28, 2023 •

edited

Loading

shaopeng-gh Mar 28, 2023 •

edited

Loading

shaopeng-gh Mar 28, 2023 •

edited

Loading

michaelcfanning Mar 30, 2023 •

edited by shaopeng-gh

Loading

michaelcfanning Mar 30, 2023 •

edited by shaopeng-gh

Loading

michaelcfanning commented Mar 30, 2023 •

edited by shaopeng-gh

Loading

michaelcfanning Mar 30, 2023 •

edited by shaopeng-gh

Loading

shaopeng-gh Apr 4, 2023 •

edited

Loading

michaelcfanning Mar 30, 2023 •

edited by shaopeng-gh

Loading

michaelcfanning Mar 30, 2023 •

edited by shaopeng-gh

Loading

shaopeng-gh Apr 4, 2023 •

edited

Loading

michaelcfanning Mar 30, 2023 •

edited by shaopeng-gh

Loading

michaelcfanning Mar 30, 2023 •

edited by shaopeng-gh

Loading

shaopeng-gh Apr 4, 2023 •

edited

Loading

shaopeng-gh Apr 4, 2023 •

edited

Loading

michaelcfanning commented Mar 30, 2023 •

edited by shaopeng-gh

Loading

michaelcfanning Mar 30, 2023 •

edited by shaopeng-gh

Loading

michaelcfanning Mar 30, 2023 •

edited by shaopeng-gh

Loading

michaelcfanning Mar 30, 2023 •

edited by shaopeng-gh

Loading

michaelcfanning Mar 30, 2023 •

edited by shaopeng-gh

Loading

michaelcfanning Mar 31, 2023 •

edited by shaopeng-gh

Loading