internal/encoding/yaml: decode empty inputs as "null"

The YAML 1.2.2 spec says:

    YAML allows the node content to be omitted in many cases. Nodes with
    empty content are interpreted as if they were plain scalars with an
    empty value. Such nodes are commonly resolved to a “null” value.

And indeed, multiple of their examples show empty inputs decoding
as null, such as

    ---
    ---
    # Empty
    ...

resulting in two "null" values.

The old YAML decoder seems to have wanted this behavior for years,
but as my TODO on the new decoder explains, it returned a "null" literal
alongside an io.EOF error, and the consumer discarded the literal.

This resolves the mishandling of empty inputs, which reached io.EOF
directly without providing any decoded CUE expression,
and also resolves all panics that this caused in cmd/cue.

Fixes #1790.
Fixes #2714.
Fixes #3337.
Closes #1807.

Signed-off-by: Daniel Martí <[email protected]>
Change-Id: Iaa3356c7eee5e84dc51386a057a470138e1ac3ce
Reviewed-on: https://review.gerrithub.io/c/cue-lang/cue/+/1198876
Unity-Result: CUE porcuepine <[email protected]>
Reviewed-by: Roger Peppe <[email protected]>
TryBot-Result: CUEcueckoo <[email protected]>

cmd/cue/cmd/testdata/script/encoding_empty.txtar

@@ -14,8 +14,8 @@ cmp stdout as-cue.stdout
 stderr 'unexpected end of JSON input'
 ! exec cue export jsonl: empty
 stderr '^panic: '
-! exec cue export yaml: empty
-stderr '^panic: '
+exec cue export yaml: empty
+cmp stdout as-yaml.stdout
 exec cue export toml: empty
 cmp stdout as-toml.stdout
 
@@ -27,8 +27,8 @@ cmp stdout as-cue.stdout
 stderr 'unexpected end of JSON input'
 ! exec cue export jsonl: newlines
 stderr '^panic: '
-! exec cue export yaml: newlines
-stderr '^panic: '
+exec cue export yaml: newlines
+cmp stdout as-yaml.stdout
 exec cue export toml: newlines
 cmp stdout as-toml.stdout
 
@@ -41,13 +41,15 @@ cmp stdout as-cue.stdout
 stderr 'invalid character ./. looking for beginning of value'
 ! exec cue export jsonl: slash-comments
 stderr 'invalid character ./. looking for beginning of value'
-! exec cue export yaml: hash-comments
-stderr '^panic: '
+exec cue export yaml: hash-comments
+cmp stdout as-yaml.stdout
 exec cue export toml: hash-comments
 cmp stdout as-toml.stdout
 
 -- as-cue.stdout --
 {}
+-- as-yaml.stdout --
+null
 -- as-toml.stdout --
 {}
 -- empty --

cmd/cue/cmd/testdata/script/import_list.txtar

@@ -58,4 +58,4 @@ module: "test.example"
 language: version: "v0.9.0"
 -- issue2721/empty.yaml --
 -- issue2721/empty.cue.golden --
-[]
+[null]

internal/encoding/yaml/decode.go

@@ -120,10 +120,7 @@ func (d *decoder) Decode() (ast.Expr, error) {
 			// If the input is empty, we produce a single null literal with EOF.
 			// Note that when the input contains "---", we get an empty document
 			// with a null scalar value inside instead.
-			//
-			// TODO(mvdan): the old decoder seemingly intended to do this,
-			// but returned a "null" literal with io.EOF, which consumers ignored.
-			if false && !d.yamlNonEmpty {
+			if !d.yamlNonEmpty {
 				return &ast.BasicLit{
 					Kind:  token.NULL,
 					Value: "null",

internal/encoding/yaml/decode_test.go

@@ -47,7 +47,7 @@ var unmarshalTests = []struct {
 }{
 	{
 		"",
-		"",
+		"null",
 	},
 	{
 		"{}",
@@ -862,7 +862,7 @@ var decoderTests = []struct {
 	want string
 }{{
 	"",
-	"",
+	"null",
 }, {
 	"a: b",
 	`a: "b"`,

pkg/encoding/yaml/testdata/gen.txtar

@@ -79,7 +79,7 @@ unmarshalStream: {
 	}, {
 		b: 2
 	}]
-	empty: []
+	empty: [null]
 	nums: [1, 2]
 	null1: [1, null, 2]
 	null2: [1, null, 2]