feat: K8S KYAML output format support (#2560)

* feat: K8S KYAML output format support

Reference: https://github.com/kubernetes/enhancements/blob/master/keps/sig-cli/5295-kyaml/README.md
Co-authored-by: Codex <codex@openai.com>
Generated-with: OpenAI Codex CLI (partial)
Signed-off-by: Robin H. Johnson <rjohnson@coreweave.com>

* build: gomodcache/gocache should not be committed

Signed-off-by: Robin H. Johnson <rjohnson@coreweave.com>

* chore: fix spelling of behaviour

Signed-off-by: Robin H. Johnson <robbat2@gentoo.org>

* build: pass GOFLAGS to docker to support buildvcs=false

In trying to develop the KYAML support, various tests gave false
positive results because they made assumptions about Git functionality
Make it possible to avoid that by passing GOFLAGS='-buildvcs=false' to
to Makefile.

Signed-off-by: Robin H. Johnson <robbat2@gentoo.org>

* doc: cover documentScenarios for tests

Signed-off-by: Robin H. Johnson <rjohnson@coreweave.com>

* build: exclude go caches from gosec

Without tuning, gosec scans all of the vendor/gocache/gomodcache, taking
several minutes (3m35 here), whereas the core of the yq takes only 15
seconds to scan.

If we intend to remediate upstream issues in future; add a seperate
target to scan those.

Signed-off-by: Robin H. Johnson <rjohnson@coreweave.com>

---------

Signed-off-by: Robin H. Johnson <rjohnson@coreweave.com>
Signed-off-by: Robin H. Johnson <robbat2@gentoo.org>
Co-authored-by: Codex <codex@openai.com>
This commit is contained in:
Robin H. Johnson 2025-12-31 20:14:53 -08:00 committed by GitHub
parent 23abf50fef
commit c6029376a5
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
26 changed files with 1388 additions and 74 deletions

View File

@ -34,13 +34,13 @@ The command you ran:
yq eval-all 'select(fileIndex==0) | .a.b.c' data1.yml data2.yml
```
**Actual behavior**
**Actual behaviour**
```yaml
cat: meow
```
**Expected behavior**
**Expected behaviour**
```yaml
this: should really work

4
.gitignore vendored
View File

@ -69,3 +69,7 @@ debian/files
.vscode
yq3
# Golang
.gomodcache/
.gocache/

View File

@ -197,6 +197,21 @@ Note: PRs with small changes (e.g. minor typos) may not be merged (see https://j
make [local] test # Run in Docker container
```
- **Problem**: Tests fail with a VCS error:
```bash
error obtaining VCS status: exit status 128
Use -buildvcs=false to disable VCS stamping.
```
- **Solution**:
Git security mechanisms prevent Golang from detecting the Git details inside
the container; either build with the `local` option, or pass GOFLAGS to
disable Golang buildvcs behaviour.
```bash
make local test
# OR
make test GOFLAGS='-buildvcs=true'
```
### Documentation Generation Issues
- **Problem**: Generated docs don't update after test changes
- **Solution**:

View File

@ -35,6 +35,7 @@ clean:
## prefix before other make targets to run in your local dev environment
local: | quiet
@$(eval ENGINERUN= )
@$(eval GOFLAGS="$(GOFLAGS)" )
@mkdir -p tmp
@touch tmp/dev_image_id
quiet: # this is silly but shuts up 'Nothing to be done for `local`'

View File

@ -4,6 +4,7 @@ IMPORT_PATH := github.com/mikefarah/${PROJECT}
export GIT_COMMIT = $(shell git rev-parse --short HEAD)
export GIT_DIRTY = $(shell test -n "$$(git status --porcelain)" && echo "+CHANGES" || true)
export GIT_DESCRIBE = $(shell git describe --tags --always)
GOFLAGS :=
LDFLAGS :=
LDFLAGS += -X main.GitCommit=${GIT_COMMIT}${GIT_DIRTY}
LDFLAGS += -X main.GitDescribe=${GIT_DESCRIBE}
@ -33,6 +34,7 @@ DEV_IMAGE := ${PROJECT}_dev
ENGINERUN := ${ENGINE} run --rm \
-e LDFLAGS="${LDFLAGS}" \
-e GOFLAGS="${GOFLAGS}" \
-e GITHUB_TOKEN="${GITHUB_TOKEN}" \
-v ${ROOT}/vendor:/go/src${SELINUX} \
-v ${ROOT}:/${PROJECT}/src/${IMPORT_PATH}${SELINUX} \

View File

@ -3,7 +3,7 @@
![Build](https://github.com/mikefarah/yq/workflows/Build/badge.svg) ![Docker Pulls](https://img.shields.io/docker/pulls/mikefarah/yq.svg) ![Github Releases (by Release)](https://img.shields.io/github/downloads/mikefarah/yq/total.svg) ![Go Report](https://goreportcard.com/badge/github.com/mikefarah/yq) ![CodeQL](https://github.com/mikefarah/yq/workflows/CodeQL/badge.svg)
A lightweight and portable command-line YAML, JSON, INI and XML processor. `yq` uses [jq](https://github.com/stedolan/jq) (a popular JSON processor) like syntax but works with yaml files as well as json, xml, ini, properties, csv and tsv. It doesn't yet support everything `jq` does - but it does support the most common operations and functions, and more is being added continuously.
A lightweight and portable command-line YAML, JSON, INI and XML processor. `yq` uses [jq](https://github.com/stedolan/jq) (a popular JSON processor) like syntax but works with yaml files as well as json, kyaml, xml, ini, properties, csv and tsv. It doesn't yet support everything `jq` does - but it does support the most common operations and functions, and more is being added continuously.
yq is written in Go - so you can download a dependency free binary for your platform and you are good to go! If you prefer there are a variety of package managers that can be used as well as Docker and Podman, all listed below.
@ -415,7 +415,7 @@ Flags:
-h, --help help for yq
-I, --indent int sets indent level for output (default 2)
-i, --inplace update the file in place of first file given.
-p, --input-format string [auto|a|yaml|y|json|j|props|p|csv|c|tsv|t|xml|x|base64|uri|toml|hcl|h|lua|l|ini|i] parse format for input. (default "auto")
-p, --input-format string [auto|a|yaml|y|json|j|kyaml|ky|props|p|csv|c|tsv|t|xml|x|base64|uri|toml|hcl|h|lua|l|ini|i] parse format for input. (default "auto")
--lua-globals output keys as top-level global variables
--lua-prefix string prefix (default "return ")
--lua-suffix string suffix (default ";\n")
@ -424,7 +424,7 @@ Flags:
-N, --no-doc Don't print document separators (---)
-0, --nul-output Use NUL char to separate values. If unwrap scalar is also set, fail if unwrapped scalar contains NUL char.
-n, --null-input Don't read input, simply evaluate the expression given. Useful for creating docs from scratch.
-o, --output-format string [auto|a|yaml|y|json|j|props|p|csv|c|tsv|t|xml|x|base64|uri|toml|hcl|h|shell|s|lua|l|ini|i] output format type. (default "auto")
-o, --output-format string [auto|a|yaml|y|json|j|kyaml|ky|props|p|csv|c|tsv|t|xml|x|base64|uri|toml|hcl|h|shell|s|lua|l|ini|i] output format type. (default "auto")
-P, --prettyPrint pretty print, shorthand for '... style = ""'
--properties-array-brackets use [x] in array paths (e.g. for SpringBoot)
--properties-separator string separator to use between keys and values (default " = ")

View File

@ -154,6 +154,37 @@ EOM
assertEquals "$expected" "$X"
}
testInputKYaml() {
cat >test.kyaml <<'EOL'
# leading
{
a: 1, # a line
# head b
b: 2,
c: [
# head d
"d", # d line
],
}
EOL
read -r -d '' expected <<'EOM'
# leading
a: 1 # a line
# head b
b: 2
c:
# head d
- d # d line
EOM
X=$(./yq e -p=kyaml -P test.kyaml)
assertEquals "$expected" "$X"
X=$(./yq ea -p=kyaml -P test.kyaml)
assertEquals "$expected" "$X"
}
@ -313,4 +344,4 @@ EOM
assertEquals "$expected" "$X"
}
source ./scripts/shunit2
source ./scripts/shunit2

View File

@ -280,6 +280,55 @@ EOM
assertEquals "$expected" "$X"
}
testOutputKYaml() {
cat >test.yml <<'EOL'
# leading
a: 1 # a line
# head b
b: 2
c:
# head d
- d # d line
EOL
read -r -d '' expected <<'EOM'
# leading
{
a: 1, # a line
# head b
b: 2,
c: [
# head d
"d", # d line
],
}
EOM
X=$(./yq e --output-format=kyaml test.yml)
assertEquals "$expected" "$X"
X=$(./yq ea --output-format=kyaml test.yml)
assertEquals "$expected" "$X"
}
testOutputKYamlShort() {
cat >test.yml <<EOL
a: b
EOL
read -r -d '' expected <<'EOM'
{
a: "b",
}
EOM
X=$(./yq e -o=ky test.yml)
assertEquals "$expected" "$X"
X=$(./yq ea -o=ky test.yml)
assertEquals "$expected" "$X"
}
testOutputXmComplex() {
cat >test.yml <<EOL
a: {b: {c: ["cat", "dog"], +@f: meow}}

View File

@ -83,6 +83,7 @@ Create a test file `pkg/yqlib/<format>_test.go` using the `formatScenario` patte
- `scenarioType` can be `"decode"` (test decoding to YAML) or `"roundtrip"` (encode/decode preservation)
- Create a helper function `test<Format>Scenario()` that switches on `scenarioType`
- Create main test function `Test<Format>FormatScenarios()` that iterates over scenarios
- The main test function should use `documentScenarios` to ensure testcase documentation is generated.
Test coverage must include:
- Basic data types (scalars, arrays, objects/maps)
@ -338,6 +339,7 @@ Create `pkg/yqlib/operator_<type>_test.go` using the `expressionScenario` patter
- Include `subdescription` for longer test names
- Set `expectedError` if testing error cases
- Create main test function that iterates over scenarios
- The main test function should use `documentScenarios` to ensure testcase documentation is generated.
Test coverage must include:
- Basic data types and nested structures

View File

@ -166,6 +166,9 @@ func configureDecoder(evaluateTogether bool) (yqlib.Decoder, error) {
}
yqlib.ConfiguredYamlPreferences.EvaluateTogether = evaluateTogether
if format.DecoderFactory == nil {
return nil, fmt.Errorf("no support for %s input format", inputFormat)
}
yqlibDecoder := format.DecoderFactory()
if yqlibDecoder == nil {
return nil, fmt.Errorf("no support for %s input format", inputFormat)
@ -197,18 +200,22 @@ func configureEncoder() (yqlib.Encoder, error) {
}
yqlib.ConfiguredXMLPreferences.Indent = indent
yqlib.ConfiguredYamlPreferences.Indent = indent
yqlib.ConfiguredKYamlPreferences.Indent = indent
yqlib.ConfiguredJSONPreferences.Indent = indent
yqlib.ConfiguredYamlPreferences.UnwrapScalar = unwrapScalar
yqlib.ConfiguredKYamlPreferences.UnwrapScalar = unwrapScalar
yqlib.ConfiguredPropertiesPreferences.UnwrapScalar = unwrapScalar
yqlib.ConfiguredJSONPreferences.UnwrapScalar = unwrapScalar
yqlib.ConfiguredYamlPreferences.ColorsEnabled = colorsEnabled
yqlib.ConfiguredKYamlPreferences.ColorsEnabled = colorsEnabled
yqlib.ConfiguredJSONPreferences.ColorsEnabled = colorsEnabled
yqlib.ConfiguredHclPreferences.ColorsEnabled = colorsEnabled
yqlib.ConfiguredTomlPreferences.ColorsEnabled = colorsEnabled
yqlib.ConfiguredYamlPreferences.PrintDocSeparators = !noDocSeparators
yqlib.ConfiguredKYamlPreferences.PrintDocSeparators = !noDocSeparators
encoder := yqlibOutputFormat.EncoderFactory()

10
examples/kyaml.kyaml Normal file
View File

@ -0,0 +1,10 @@
# leading
{
a: 1, # a line
# head b
b: 2,
c: [
# head d
"d", # d line
],
}

7
examples/kyaml.yml Normal file
View File

@ -0,0 +1,7 @@
# leading
a: 1 # a line
# head b
b: 2
c:
# head d
- d # d line

View File

@ -0,0 +1,9 @@
# KYaml
Encode and decode to and from KYaml (a restricted subset of YAML that uses flow-style collections).
KYaml is useful when you want YAML data rendered in a compact, JSON-like form while still supporting YAML features like comments.
Notes:
- Strings are always double-quoted in KYaml output.
- Anchors and aliases are expanded (KYaml output does not emit them).

View File

@ -0,0 +1,253 @@
# KYaml
Encode and decode to and from KYaml (a restricted subset of YAML that uses flow-style collections).
KYaml is useful when you want YAML data rendered in a compact, JSON-like form while still supporting YAML features like comments.
Notes:
- Strings are always double-quoted in KYaml output.
- Anchors and aliases are expanded (KYaml output does not emit them).
## Encode kyaml: plain string scalar
Strings are always double-quoted in KYaml output.
Given a sample.yml file of:
```yaml
cat
```
then
```bash
yq -o=kyaml '.' sample.yml
```
will output
```yaml
"cat"
```
## encode flow mapping and sequence
Given a sample.yml file of:
```yaml
a: b
c:
- d
```
then
```bash
yq -o=kyaml '.' sample.yml
```
will output
```yaml
{
a: "b",
c: [
"d",
],
}
```
## encode non-string scalars
Given a sample.yml file of:
```yaml
a: 12
b: true
c: null
d: "true"
```
then
```bash
yq -o=kyaml '.' sample.yml
```
will output
```yaml
{
a: 12,
b: true,
c: null,
d: "true",
}
```
## quote non-identifier keys
Given a sample.yml file of:
```yaml
"1a": b
"has space": c
```
then
```bash
yq -o=kyaml '.' sample.yml
```
will output
```yaml
{
"1a": "b",
"has space": "c",
}
```
## escape quoted strings
Given a sample.yml file of:
```yaml
a: "line1\nline2\t\"q\""
```
then
```bash
yq -o=kyaml '.' sample.yml
```
will output
```yaml
{
a: "line1\nline2\t\"q\"",
}
```
## preserve comments when encoding
Given a sample.yml file of:
```yaml
# leading
a: 1 # a line
# head b
b: 2
c:
# head d
- d # d line
- e
# trailing
```
then
```bash
yq -o=kyaml '.' sample.yml
```
will output
```yaml
# leading
{
a: 1, # a line
# head b
b: 2,
c: [
# head d
"d", # d line
"e",
],
# trailing
}
```
## Encode kyaml: anchors and aliases
KYaml output does not support anchors/aliases; they are expanded to concrete values.
Given a sample.yml file of:
```yaml
base: &base
a: b
copy: *base
```
then
```bash
yq -o=kyaml '.' sample.yml
```
will output
```yaml
{
base: {
a: "b",
},
copy: {
a: "b",
},
}
```
## Encode kyaml: yaml to kyaml shows formatting differences
KYaml uses flow-style collections (braces/brackets) and explicit commas.
Given a sample.yml file of:
```yaml
person:
name: John
pets:
- cat
- dog
```
then
```bash
yq -o=kyaml '.' sample.yml
```
will output
```yaml
{
person: {
name: "John",
pets: [
"cat",
"dog",
],
},
}
```
## Encode kyaml: nested lists of objects
Lists and objects can be nested arbitrarily; KYaml always uses flow-style collections.
Given a sample.yml file of:
```yaml
- name: a
items:
- id: 1
tags:
- k: x
v: y
- k: x2
v: y2
- id: 2
tags:
- k: z
v: w
```
then
```bash
yq -o=kyaml '.' sample.yml
```
will output
```yaml
[
{
name: "a",
items: [
{
id: 1,
tags: [
{
k: "x",
v: "y",
},
{
k: "x2",
v: "y2",
},
],
},
{
id: 2,
tags: [
{
k: "z",
v: "w",
},
],
},
],
},
]
```

View File

@ -1,7 +1,12 @@
package yqlib
import (
"bufio"
"errors"
"io"
"strings"
"github.com/fatih/color"
)
type Encoder interface {
@ -25,3 +30,63 @@ func mapKeysToStrings(node *CandidateNode) {
mapKeysToStrings(child)
}
}
// Some funcs are shared between encoder_yaml and encoder_kyaml
func PrintYAMLDocumentSeparator(writer io.Writer, PrintDocSeparators bool) error {
if PrintDocSeparators {
log.Debug("writing doc sep")
if err := writeString(writer, "---\n"); err != nil {
return err
}
}
return nil
}
func PrintYAMLLeadingContent(writer io.Writer, content string, PrintDocSeparators bool, ColorsEnabled bool) error {
reader := bufio.NewReader(strings.NewReader(content))
// reuse precompiled package-level regex
// (declared in decoder_yaml.go)
for {
readline, errReading := reader.ReadString('\n')
if errReading != nil && !errors.Is(errReading, io.EOF) {
return errReading
}
if strings.Contains(readline, "$yqDocSeparator$") {
// Preserve the original line ending (CRLF or LF)
lineEnding := "\n"
if strings.HasSuffix(readline, "\r\n") {
lineEnding = "\r\n"
}
if PrintDocSeparators {
if err := writeString(writer, "---"+lineEnding); err != nil {
return err
}
}
} else {
if len(readline) > 0 && readline != "\n" && readline[0] != '%' && !commentLineRe.MatchString(readline) {
readline = "# " + readline
}
if ColorsEnabled && strings.TrimSpace(readline) != "" {
readline = format(color.FgHiBlack) + readline + format(color.Reset)
}
if err := writeString(writer, readline); err != nil {
return err
}
}
if errors.Is(errReading, io.EOF) {
if readline != "" {
// the last comment we read didn't have a newline, put one in
if err := writeString(writer, "\n"); err != nil {
return err
}
}
break
}
}
return nil
}

318
pkg/yqlib/encoder_kyaml.go Normal file
View File

@ -0,0 +1,318 @@
//go:build !yq_nokyaml
package yqlib
import (
"bytes"
"io"
"regexp"
"strconv"
"strings"
)
type kyamlEncoder struct {
prefs KYamlPreferences
}
func NewKYamlEncoder(prefs KYamlPreferences) Encoder {
return &kyamlEncoder{prefs: prefs}
}
func (ke *kyamlEncoder) CanHandleAliases() bool {
// KYAML is a restricted subset; avoid emitting anchors/aliases.
return false
}
func (ke *kyamlEncoder) PrintDocumentSeparator(writer io.Writer) error {
return PrintYAMLDocumentSeparator(writer, ke.prefs.PrintDocSeparators)
}
func (ke *kyamlEncoder) PrintLeadingContent(writer io.Writer, content string) error {
return PrintYAMLLeadingContent(writer, content, ke.prefs.PrintDocSeparators, ke.prefs.ColorsEnabled)
}
func (ke *kyamlEncoder) Encode(writer io.Writer, node *CandidateNode) error {
log.Debug("encoderKYaml - going to print %v", NodeToString(node))
if node.Kind == ScalarNode && ke.prefs.UnwrapScalar {
return writeString(writer, node.Value+"\n")
}
destination := writer
tempBuffer := bytes.NewBuffer(nil)
if ke.prefs.ColorsEnabled {
destination = tempBuffer
}
// Mirror the YAML encoder behaviour: trailing comments on the document root
// are stored in FootComment and need to be printed after the document.
trailingContent := node.FootComment
if err := ke.writeCommentBlock(destination, node.HeadComment, 0); err != nil {
return err
}
if err := ke.writeNode(destination, node, 0); err != nil {
return err
}
if err := ke.writeInlineComment(destination, node.LineComment); err != nil {
return err
}
if err := writeString(destination, "\n"); err != nil {
return err
}
if err := ke.PrintLeadingContent(destination, trailingContent); err != nil {
return err
}
if ke.prefs.ColorsEnabled {
return colorizeAndPrint(tempBuffer.Bytes(), writer)
}
return nil
}
func (ke *kyamlEncoder) writeNode(writer io.Writer, node *CandidateNode, indent int) error {
switch node.Kind {
case MappingNode:
return ke.writeMapping(writer, node, indent)
case SequenceNode:
return ke.writeSequence(writer, node, indent)
case ScalarNode:
return writeString(writer, ke.formatScalar(node))
case AliasNode:
// Should have been exploded by the printer, but handle defensively.
if node.Alias == nil {
return writeString(writer, "null")
}
return ke.writeNode(writer, node.Alias, indent)
default:
return writeString(writer, "null")
}
}
func (ke *kyamlEncoder) writeMapping(writer io.Writer, node *CandidateNode, indent int) error {
if len(node.Content) == 0 {
return writeString(writer, "{}")
}
if err := writeString(writer, "{\n"); err != nil {
return err
}
for i := 0; i+1 < len(node.Content); i += 2 {
keyNode := node.Content[i]
valueNode := node.Content[i+1]
entryIndent := indent + ke.prefs.Indent
if err := ke.writeCommentBlock(writer, keyNode.HeadComment, entryIndent); err != nil {
return err
}
if valueNode.HeadComment != "" && valueNode.HeadComment != keyNode.HeadComment {
if err := ke.writeCommentBlock(writer, valueNode.HeadComment, entryIndent); err != nil {
return err
}
}
if err := ke.writeIndent(writer, entryIndent); err != nil {
return err
}
if err := writeString(writer, ke.formatKey(keyNode)); err != nil {
return err
}
if err := writeString(writer, ": "); err != nil {
return err
}
if err := ke.writeNode(writer, valueNode, entryIndent); err != nil {
return err
}
// Always emit a trailing comma; KYAML encourages explicit separators,
// and this ensures all quoted strings have a trailing `",` as requested.
if err := writeString(writer, ","); err != nil {
return err
}
inline := valueNode.LineComment
if inline == "" {
inline = keyNode.LineComment
}
if err := ke.writeInlineComment(writer, inline); err != nil {
return err
}
if err := writeString(writer, "\n"); err != nil {
return err
}
foot := valueNode.FootComment
if foot == "" {
foot = keyNode.FootComment
}
if err := ke.writeCommentBlock(writer, foot, entryIndent); err != nil {
return err
}
}
if err := ke.writeIndent(writer, indent); err != nil {
return err
}
return writeString(writer, "}")
}
func (ke *kyamlEncoder) writeSequence(writer io.Writer, node *CandidateNode, indent int) error {
if len(node.Content) == 0 {
return writeString(writer, "[]")
}
if err := writeString(writer, "[\n"); err != nil {
return err
}
for _, child := range node.Content {
itemIndent := indent + ke.prefs.Indent
if err := ke.writeCommentBlock(writer, child.HeadComment, itemIndent); err != nil {
return err
}
if err := ke.writeIndent(writer, itemIndent); err != nil {
return err
}
if err := ke.writeNode(writer, child, itemIndent); err != nil {
return err
}
if err := writeString(writer, ","); err != nil {
return err
}
if err := ke.writeInlineComment(writer, child.LineComment); err != nil {
return err
}
if err := writeString(writer, "\n"); err != nil {
return err
}
if err := ke.writeCommentBlock(writer, child.FootComment, itemIndent); err != nil {
return err
}
}
if err := ke.writeIndent(writer, indent); err != nil {
return err
}
return writeString(writer, "]")
}
func (ke *kyamlEncoder) writeIndent(writer io.Writer, indent int) error {
if indent <= 0 {
return nil
}
return writeString(writer, strings.Repeat(" ", indent))
}
func (ke *kyamlEncoder) formatKey(keyNode *CandidateNode) string {
// KYAML examples use bare keys. Quote keys only when needed.
key := keyNode.Value
if isValidKYamlBareKey(key) {
return key
}
return `"` + escapeDoubleQuotedString(key) + `"`
}
func (ke *kyamlEncoder) formatScalar(node *CandidateNode) string {
switch node.Tag {
case "!!null":
return "null"
case "!!bool":
return strings.ToLower(node.Value)
case "!!int", "!!float":
return node.Value
case "!!str":
return `"` + escapeDoubleQuotedString(node.Value) + `"`
default:
// Fall back to a string representation to avoid implicit typing surprises.
return `"` + escapeDoubleQuotedString(node.Value) + `"`
}
}
var kyamlBareKeyRe = regexp.MustCompile(`^[A-Za-z_][A-Za-z0-9_-]*$`)
func isValidKYamlBareKey(s string) bool {
// Conservative: require an identifier-like key; otherwise quote.
if s == "" {
return false
}
return kyamlBareKeyRe.MatchString(s)
}
func escapeDoubleQuotedString(s string) string {
var b strings.Builder
b.Grow(len(s) + 2)
for _, r := range s {
switch r {
case '\\':
b.WriteString(`\\`)
case '"':
b.WriteString(`\"`)
case '\n':
b.WriteString(`\n`)
case '\r':
b.WriteString(`\r`)
case '\t':
b.WriteString(`\t`)
default:
if r < 0x20 {
// YAML double-quoted strings support \uXXXX escapes.
b.WriteString(`\u`)
hex := "0000" + strings.ToUpper(strconv.FormatInt(int64(r), 16))
b.WriteString(hex[len(hex)-4:])
} else {
b.WriteRune(r)
}
}
}
return b.String()
}
func (ke *kyamlEncoder) writeCommentBlock(writer io.Writer, comment string, indent int) error {
if strings.TrimSpace(comment) == "" {
return nil
}
lines := strings.Split(strings.ReplaceAll(comment, "\r\n", "\n"), "\n")
for _, line := range lines {
trimmed := strings.TrimSpace(line)
if trimmed == "" {
continue
}
if err := ke.writeIndent(writer, indent); err != nil {
return err
}
toWrite := line
if !commentLineRe.MatchString(toWrite) {
toWrite = "# " + toWrite
}
if err := writeString(writer, toWrite); err != nil {
return err
}
if err := writeString(writer, "\n"); err != nil {
return err
}
}
return nil
}
func (ke *kyamlEncoder) writeInlineComment(writer io.Writer, comment string) error {
comment = strings.TrimSpace(strings.ReplaceAll(comment, "\r\n", "\n"))
if comment == "" {
return nil
}
lines := strings.Split(comment, "\n")
first := strings.TrimSpace(lines[0])
if first == "" {
return nil
}
if !strings.HasPrefix(first, "#") {
first = "# " + first
}
if err := writeString(writer, " "); err != nil {
return err
}
return writeString(writer, first)
}

View File

@ -1,13 +1,10 @@
package yqlib
import (
"bufio"
"bytes"
"errors"
"io"
"strings"
"github.com/fatih/color"
"go.yaml.in/yaml/v4"
)
@ -24,63 +21,11 @@ func (ye *yamlEncoder) CanHandleAliases() bool {
}
func (ye *yamlEncoder) PrintDocumentSeparator(writer io.Writer) error {
if ye.prefs.PrintDocSeparators {
log.Debug("writing doc sep")
if err := writeString(writer, "---\n"); err != nil {
return err
}
}
return nil
return PrintYAMLDocumentSeparator(writer, ye.prefs.PrintDocSeparators)
}
func (ye *yamlEncoder) PrintLeadingContent(writer io.Writer, content string) error {
reader := bufio.NewReader(strings.NewReader(content))
// reuse precompiled package-level regex
// (declared in decoder_yaml.go)
for {
readline, errReading := reader.ReadString('\n')
if errReading != nil && !errors.Is(errReading, io.EOF) {
return errReading
}
if strings.Contains(readline, "$yqDocSeparator$") {
// Preserve the original line ending (CRLF or LF)
lineEnding := "\n"
if strings.HasSuffix(readline, "\r\n") {
lineEnding = "\r\n"
}
if ye.prefs.PrintDocSeparators {
if err := writeString(writer, "---"+lineEnding); err != nil {
return err
}
}
} else {
if len(readline) > 0 && readline != "\n" && readline[0] != '%' && !commentLineRe.MatchString(readline) {
readline = "# " + readline
}
if ye.prefs.ColorsEnabled && strings.TrimSpace(readline) != "" {
readline = format(color.FgHiBlack) + readline + format(color.Reset)
}
if err := writeString(writer, readline); err != nil {
return err
}
}
if errors.Is(errReading, io.EOF) {
if readline != "" {
// the last comment we read didn't have a newline, put one in
if err := writeString(writer, "\n"); err != nil {
return err
}
}
break
}
}
return nil
return PrintYAMLLeadingContent(writer, content, ye.prefs.PrintDocSeparators, ye.prefs.ColorsEnabled)
}
func (ye *yamlEncoder) Encode(writer io.Writer, node *CandidateNode) error {

View File

@ -22,6 +22,12 @@ var YamlFormat = &Format{"yaml", []string{"y", "yml"},
func() Decoder { return NewYamlDecoder(ConfiguredYamlPreferences) },
}
var KYamlFormat = &Format{"kyaml", []string{"ky"},
func() Encoder { return NewKYamlEncoder(ConfiguredKYamlPreferences) },
// KYaml is stricter YAML
func() Decoder { return NewYamlDecoder(ConfiguredYamlPreferences) },
}
var JSONFormat = &Format{"json", []string{"j"},
func() Encoder { return NewJSONEncoder(ConfiguredJSONPreferences) },
func() Decoder { return NewJSONDecoder() },
@ -89,6 +95,7 @@ var INIFormat = &Format{"ini", []string{"i"},
var Formats = []*Format{
YamlFormat,
KYamlFormat,
JSONFormat,
PropertiesFormat,
CSVFormat,

30
pkg/yqlib/kyaml.go Normal file
View File

@ -0,0 +1,30 @@
//go:build !yq_nokyaml
package yqlib
type KYamlPreferences struct {
Indent int
ColorsEnabled bool
PrintDocSeparators bool
UnwrapScalar bool
}
func NewDefaultKYamlPreferences() KYamlPreferences {
return KYamlPreferences{
Indent: 2,
ColorsEnabled: false,
PrintDocSeparators: true,
UnwrapScalar: true,
}
}
func (p *KYamlPreferences) Copy() KYamlPreferences {
return KYamlPreferences{
Indent: p.Indent,
ColorsEnabled: p.ColorsEnabled,
PrintDocSeparators: p.PrintDocSeparators,
UnwrapScalar: p.UnwrapScalar,
}
}
var ConfiguredKYamlPreferences = NewDefaultKYamlPreferences()

542
pkg/yqlib/kyaml_test.go Normal file
View File

@ -0,0 +1,542 @@
//go:build !yq_nokyaml
package yqlib
import (
"bufio"
"bytes"
"fmt"
"regexp"
"strings"
"testing"
"github.com/mikefarah/yq/v4/test"
)
var ansiRe = regexp.MustCompile(`\x1b\[[0-9;]*m`)
func stripANSI(s string) string {
return ansiRe.ReplaceAllString(s, "")
}
var kyamlFormatScenarios = []formatScenario{
{
description: "Encode kyaml: plain string scalar",
subdescription: "Strings are always double-quoted in KYaml output.",
scenarioType: "encode",
indent: 2,
input: "cat\n",
expected: "\"cat\"\n",
},
{
description: "encode plain int scalar",
scenarioType: "encode",
indent: 2,
input: "12\n",
expected: "12\n",
skipDoc: true,
},
{
description: "encode plain bool scalar",
scenarioType: "encode",
indent: 2,
input: "true\n",
expected: "true\n",
skipDoc: true,
},
{
description: "encode plain null scalar",
scenarioType: "encode",
indent: 2,
input: "null\n",
expected: "null\n",
skipDoc: true,
},
{
description: "encode flow mapping and sequence",
scenarioType: "encode",
indent: 2,
input: "a: b\nc:\n - d\n",
expected: "{\n" +
" a: \"b\",\n" +
" c: [\n" +
" \"d\",\n" +
" ],\n" +
"}\n",
},
{
description: "encode non-string scalars",
scenarioType: "encode",
indent: 2,
input: "a: 12\n" +
"b: true\n" +
"c: null\n" +
"d: \"true\"\n",
expected: "{\n" +
" a: 12,\n" +
" b: true,\n" +
" c: null,\n" +
" d: \"true\",\n" +
"}\n",
},
{
description: "quote non-identifier keys",
scenarioType: "encode",
indent: 2,
input: "\"1a\": b\n\"has space\": c\n",
expected: "{\n" +
" \"1a\": \"b\",\n" +
" \"has space\": \"c\",\n" +
"}\n",
},
{
description: "escape quoted strings",
scenarioType: "encode",
indent: 2,
input: "a: \"line1\\nline2\\t\\\"q\\\"\"\n",
expected: "{\n" +
" a: \"line1\\nline2\\t\\\"q\\\"\",\n" +
"}\n",
},
{
description: "preserve comments when encoding",
scenarioType: "encode",
indent: 2,
input: "# leading\n" +
"a: 1 # a line\n" +
"# head b\n" +
"b: 2\n" +
"c:\n" +
" # head d\n" +
" - d # d line\n" +
" - e\n" +
"# trailing\n",
expected: "# leading\n" +
"{\n" +
" a: 1, # a line\n" +
" # head b\n" +
" b: 2,\n" +
" c: [\n" +
" # head d\n" +
" \"d\", # d line\n" +
" \"e\",\n" +
" ],\n" +
" # trailing\n" +
"}\n",
},
{
description: "Encode kyaml: anchors and aliases",
subdescription: "KYaml output does not support anchors/aliases; they are expanded to concrete values.",
scenarioType: "encode",
indent: 2,
input: "base: &base\n" +
" a: b\n" +
"copy: *base\n",
expected: "{\n" +
" base: {\n" +
" a: \"b\",\n" +
" },\n" +
" copy: {\n" +
" a: \"b\",\n" +
" },\n" +
"}\n",
},
{
description: "Encode kyaml: yaml to kyaml shows formatting differences",
subdescription: "KYaml uses flow-style collections (braces/brackets) and explicit commas.",
scenarioType: "encode",
indent: 2,
input: "person:\n" +
" name: John\n" +
" pets:\n" +
" - cat\n" +
" - dog\n",
expected: "{\n" +
" person: {\n" +
" name: \"John\",\n" +
" pets: [\n" +
" \"cat\",\n" +
" \"dog\",\n" +
" ],\n" +
" },\n" +
"}\n",
},
{
description: "Encode kyaml: nested lists of objects",
subdescription: "Lists and objects can be nested arbitrarily; KYaml always uses flow-style collections.",
scenarioType: "encode",
indent: 2,
input: "- name: a\n" +
" items:\n" +
" - id: 1\n" +
" tags:\n" +
" - k: x\n" +
" v: y\n" +
" - k: x2\n" +
" v: y2\n" +
" - id: 2\n" +
" tags:\n" +
" - k: z\n" +
" v: w\n",
expected: "[\n" +
" {\n" +
" name: \"a\",\n" +
" items: [\n" +
" {\n" +
" id: 1,\n" +
" tags: [\n" +
" {\n" +
" k: \"x\",\n" +
" v: \"y\",\n" +
" },\n" +
" {\n" +
" k: \"x2\",\n" +
" v: \"y2\",\n" +
" },\n" +
" ],\n" +
" },\n" +
" {\n" +
" id: 2,\n" +
" tags: [\n" +
" {\n" +
" k: \"z\",\n" +
" v: \"w\",\n" +
" },\n" +
" ],\n" +
" },\n" +
" ],\n" +
" },\n" +
"]\n",
},
}
func testKYamlScenario(t *testing.T, s formatScenario) {
prefs := ConfiguredKYamlPreferences.Copy()
prefs.Indent = s.indent
prefs.UnwrapScalar = false
switch s.scenarioType {
case "encode":
test.AssertResultWithContext(
t,
s.expected,
mustProcessFormatScenario(s, NewYamlDecoder(ConfiguredYamlPreferences), NewKYamlEncoder(prefs)),
s.description,
)
default:
panic(fmt.Sprintf("unhandled scenario type %q", s.scenarioType))
}
}
func documentKYamlScenario(_ *testing.T, w *bufio.Writer, i interface{}) {
s := i.(formatScenario)
if s.skipDoc {
return
}
switch s.scenarioType {
case "encode":
documentKYamlEncodeScenario(w, s)
default:
panic(fmt.Sprintf("unhandled scenario type %q", s.scenarioType))
}
}
func documentKYamlEncodeScenario(w *bufio.Writer, s formatScenario) {
writeOrPanic(w, fmt.Sprintf("## %v\n", s.description))
if s.subdescription != "" {
writeOrPanic(w, s.subdescription)
writeOrPanic(w, "\n\n")
}
writeOrPanic(w, "Given a sample.yml file of:\n")
writeOrPanic(w, fmt.Sprintf("```yaml\n%v\n```\n", s.input))
writeOrPanic(w, "then\n")
expression := s.expression
if expression == "" {
expression = "."
}
if s.indent == 2 {
writeOrPanic(w, fmt.Sprintf("```bash\nyq -o=kyaml '%v' sample.yml\n```\n", expression))
} else {
writeOrPanic(w, fmt.Sprintf("```bash\nyq -o=kyaml -I=%v '%v' sample.yml\n```\n", s.indent, expression))
}
writeOrPanic(w, "will output\n")
prefs := ConfiguredKYamlPreferences.Copy()
prefs.Indent = s.indent
prefs.UnwrapScalar = false
writeOrPanic(w, fmt.Sprintf("```yaml\n%v```\n\n", mustProcessFormatScenario(s, NewYamlDecoder(ConfiguredYamlPreferences), NewKYamlEncoder(prefs))))
}
func TestKYamlFormatScenarios(t *testing.T) {
for _, s := range kyamlFormatScenarios {
testKYamlScenario(t, s)
}
genericScenarios := make([]interface{}, len(kyamlFormatScenarios))
for i, s := range kyamlFormatScenarios {
genericScenarios[i] = s
}
documentScenarios(t, "usage", "kyaml", genericScenarios, documentKYamlScenario)
}
func TestKYamlEncoderPrintDocumentSeparator(t *testing.T) {
t.Run("enabled", func(t *testing.T) {
prefs := NewDefaultKYamlPreferences()
prefs.PrintDocSeparators = true
var buf bytes.Buffer
err := NewKYamlEncoder(prefs).PrintDocumentSeparator(&buf)
if err != nil {
t.Fatal(err)
}
if buf.String() != "---\n" {
t.Fatalf("expected doc separator, got %q", buf.String())
}
})
t.Run("disabled", func(t *testing.T) {
prefs := NewDefaultKYamlPreferences()
prefs.PrintDocSeparators = false
var buf bytes.Buffer
err := NewKYamlEncoder(prefs).PrintDocumentSeparator(&buf)
if err != nil {
t.Fatal(err)
}
if buf.String() != "" {
t.Fatalf("expected no output, got %q", buf.String())
}
})
}
func TestKYamlEncoderEncodeUnwrapScalar(t *testing.T) {
prefs := NewDefaultKYamlPreferences()
prefs.UnwrapScalar = true
var buf bytes.Buffer
err := NewKYamlEncoder(prefs).Encode(&buf, &CandidateNode{
Kind: ScalarNode,
Tag: "!!str",
Value: "cat",
})
if err != nil {
t.Fatal(err)
}
if buf.String() != "cat\n" {
t.Fatalf("expected unwrapped scalar, got %q", buf.String())
}
}
func TestKYamlEncoderEncodeColorsEnabled(t *testing.T) {
prefs := NewDefaultKYamlPreferences()
prefs.UnwrapScalar = false
prefs.ColorsEnabled = true
var buf bytes.Buffer
err := NewKYamlEncoder(prefs).Encode(&buf, &CandidateNode{
Kind: MappingNode,
Content: []*CandidateNode{
{Kind: ScalarNode, Tag: "!!str", Value: "a"},
{Kind: ScalarNode, Tag: "!!str", Value: "b"},
},
})
if err != nil {
t.Fatal(err)
}
out := stripANSI(buf.String())
if !strings.Contains(out, "a:") || !strings.Contains(out, "\"b\"") {
t.Fatalf("expected colourised output to contain rendered tokens, got %q", out)
}
}
func TestKYamlEncoderWriteNodeAliasAndUnknown(t *testing.T) {
ke := NewKYamlEncoder(NewDefaultKYamlPreferences()).(*kyamlEncoder)
t.Run("alias_nil", func(t *testing.T) {
var buf bytes.Buffer
err := ke.writeNode(&buf, &CandidateNode{Kind: AliasNode}, 0)
if err != nil {
t.Fatal(err)
}
if buf.String() != "null" {
t.Fatalf("expected null for nil alias, got %q", buf.String())
}
})
t.Run("alias_value", func(t *testing.T) {
var buf bytes.Buffer
err := ke.writeNode(&buf, &CandidateNode{
Kind: AliasNode,
Alias: &CandidateNode{Kind: ScalarNode, Tag: "!!int", Value: "12"},
}, 0)
if err != nil {
t.Fatal(err)
}
if buf.String() != "12" {
t.Fatalf("expected dereferenced alias value, got %q", buf.String())
}
})
t.Run("unknown_kind", func(t *testing.T) {
var buf bytes.Buffer
err := ke.writeNode(&buf, &CandidateNode{Kind: Kind(12345)}, 0)
if err != nil {
t.Fatal(err)
}
if buf.String() != "null" {
t.Fatalf("expected null for unknown kind, got %q", buf.String())
}
})
}
func TestKYamlEncoderEmptyCollections(t *testing.T) {
ke := NewKYamlEncoder(NewDefaultKYamlPreferences()).(*kyamlEncoder)
t.Run("empty_mapping", func(t *testing.T) {
var buf bytes.Buffer
err := ke.writeNode(&buf, &CandidateNode{Kind: MappingNode}, 0)
if err != nil {
t.Fatal(err)
}
if buf.String() != "{}" {
t.Fatalf("expected empty mapping, got %q", buf.String())
}
})
t.Run("empty_sequence", func(t *testing.T) {
var buf bytes.Buffer
err := ke.writeNode(&buf, &CandidateNode{Kind: SequenceNode}, 0)
if err != nil {
t.Fatal(err)
}
if buf.String() != "[]" {
t.Fatalf("expected empty sequence, got %q", buf.String())
}
})
}
func TestKYamlEncoderScalarFallbackAndEscaping(t *testing.T) {
ke := NewKYamlEncoder(NewDefaultKYamlPreferences()).(*kyamlEncoder)
t.Run("unknown_tag_falls_back_to_string", func(t *testing.T) {
var buf bytes.Buffer
err := ke.writeNode(&buf, &CandidateNode{Kind: ScalarNode, Tag: "!!timestamp", Value: "2020-01-01T00:00:00Z"}, 0)
if err != nil {
t.Fatal(err)
}
if buf.String() != "\"2020-01-01T00:00:00Z\"" {
t.Fatalf("expected quoted fallback, got %q", buf.String())
}
})
t.Run("escape_double_quoted", func(t *testing.T) {
got := escapeDoubleQuotedString("a\\b\"c\n\r\t" + string(rune(0x01)))
want := "a\\\\b\\\"c\\n\\r\\t\\u0001"
if got != want {
t.Fatalf("expected %q, got %q", want, got)
}
})
t.Run("valid_bare_key", func(t *testing.T) {
if isValidKYamlBareKey("") {
t.Fatalf("expected empty string to be invalid")
}
if isValidKYamlBareKey("1a") {
t.Fatalf("expected leading digit to be invalid")
}
if !isValidKYamlBareKey("a_b-2") {
t.Fatalf("expected identifier-like key to be valid")
}
})
}
func TestKYamlEncoderCommentsInMapping(t *testing.T) {
prefs := NewDefaultKYamlPreferences()
prefs.UnwrapScalar = false
ke := NewKYamlEncoder(prefs).(*kyamlEncoder)
var buf bytes.Buffer
err := ke.writeNode(&buf, &CandidateNode{
Kind: MappingNode,
Content: []*CandidateNode{
{
Kind: ScalarNode,
Tag: "!!str",
Value: "a",
HeadComment: "key head",
LineComment: "key line",
FootComment: "key foot",
},
{
Kind: ScalarNode,
Tag: "!!str",
Value: "b",
HeadComment: "value head",
},
},
}, 0)
if err != nil {
t.Fatal(err)
}
out := buf.String()
if !strings.Contains(out, "# key head\n") {
t.Fatalf("expected key head comment, got %q", out)
}
if !strings.Contains(out, "# value head\n") {
t.Fatalf("expected value head comment, got %q", out)
}
if !strings.Contains(out, ", # key line\n") {
t.Fatalf("expected inline key comment fallback, got %q", out)
}
if !strings.Contains(out, "# key foot\n") {
t.Fatalf("expected foot comment fallback, got %q", out)
}
}
func TestKYamlEncoderCommentBlockAndInlineComment(t *testing.T) {
ke := NewKYamlEncoder(NewDefaultKYamlPreferences()).(*kyamlEncoder)
t.Run("comment_block_prefixing_and_crlf", func(t *testing.T) {
var buf bytes.Buffer
err := ke.writeCommentBlock(&buf, "line1\r\n\r\n# already\r\nline2", 2)
if err != nil {
t.Fatal(err)
}
want := " # line1\n # already\n # line2\n"
if buf.String() != want {
t.Fatalf("expected %q, got %q", want, buf.String())
}
})
t.Run("inline_comment_prefix_and_first_line_only", func(t *testing.T) {
var buf bytes.Buffer
err := ke.writeInlineComment(&buf, "hello\r\nsecond line")
if err != nil {
t.Fatal(err)
}
if buf.String() != " # hello" {
t.Fatalf("expected %q, got %q", " # hello", buf.String())
}
})
t.Run("inline_comment_already_prefixed", func(t *testing.T) {
var buf bytes.Buffer
err := ke.writeInlineComment(&buf, "# hello")
if err != nil {
t.Fatal(err)
}
if buf.String() != " # hello" {
t.Fatalf("expected %q, got %q", " # hello", buf.String())
}
})
}

7
pkg/yqlib/no_kyaml.go Normal file
View File

@ -0,0 +1,7 @@
//go:build yq_nokyaml
package yqlib
func NewKYamlEncoder(_ KYamlPreferences) Encoder {
return nil
}

View File

@ -284,4 +284,12 @@ RDBMS
expeñded
bananabananabananabanana
edwinjhlee
flox
flox
unlabelled
kyaml
KYAML
nokyaml
buildvcs
behaviour
GOFLAGS
gocache

View File

@ -1,2 +1,2 @@
#!/bin/bash
go build -tags "yq_nolua yq_noini yq_notoml yq_noxml yq_nojson yq_nohcl" -ldflags "-s -w" .
go build -tags "yq_nolua yq_noini yq_notoml yq_noxml yq_nojson yq_nohcl yq_nokyaml" -ldflags "-s -w" .

View File

@ -1,4 +1,4 @@
#!/bin/bash
# Currently, the `yq_nojson` feature must be enabled when using TinyGo.
tinygo build -no-debug -tags "yq_nolua yq_noini yq_notoml yq_noxml yq_nojson yq_nocsv yq_nobase64 yq_nouri yq_noprops yq_nosh yq_noshell yq_nohcl" .
tinygo build -no-debug -tags "yq_nolua yq_noini yq_notoml yq_noxml yq_nojson yq_nocsv yq_nobase64 yq_nouri yq_noprops yq_nosh yq_noshell yq_nohcl yq_nokyaml" .

View File

@ -3,9 +3,11 @@
set -o errexit
set -o pipefail
if command -v gosec &> /dev/null
then
gosec "${PWD}" ./...
else
./bin/gosec "${PWD}" ./...
fi
OPTS=(
-exclude-dir=vendor
-exclude-dir=.gomodcache
-exclude-dir=.gocache
)
command -v gosec &> /dev/null && BIN=gosec || BIN=./bin/gosec
"${BIN}" "${OPTS[@]}" "${PWD}" ./...

View File

@ -783,7 +783,7 @@ _FAIL_NOT_SAME_='eval failNotSame --lineno "${LINENO:-}"'
# None
startSkipping() { __shunit_skip=${SHUNIT_TRUE}; }
# Resume the normal recording behavior of assert and fail calls.
# Resume the normal recording behaviour of assert and fail calls.
#
# Args:
# None
@ -1293,7 +1293,7 @@ if command [ "${__shunit_mode}" = "${__SHUNIT_MODE_STANDALONE}" ]; then
command . "`_shunit_prepForSourcing \"${__shunit_script}\"`"
fi
# Configure default output coloring behavior.
# Configure default output coloring behaviour.
_shunit_configureColor "${SHUNIT_COLOR}"
# Execute the oneTimeSetUp function (if it exists).