From d51f1fad3153a673a244ea981d9b40c8313a9b9d Mon Sep 17 00:00:00 2001 From: Mike Farah Date: Fri, 17 Apr 2026 16:31:31 +1000 Subject: [PATCH] v4.53.2 --- SUMMARY.md | 2 + operators/anchor-and-alias-operators.md | 16 +- operators/datetime.md | 2 +- operators/parent.md | 59 ++++++ operators/slice-array.md | 84 +++++++- operators/system-operators.md | 76 +++++++ operators/traverse-read.md | 2 +- usage/hcl.md | 2 +- usage/kyaml.md | 253 +++++++++++++++++++++++ usage/toml.md | 260 ++++++++++++++++++++++++ 10 files changed, 742 insertions(+), 14 deletions(-) create mode 100644 operators/system-operators.md create mode 100644 usage/kyaml.md diff --git a/SUMMARY.md b/SUMMARY.md index 8a5f7797..88918a62 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -69,6 +69,7 @@ * [String Operators](operators/string-operators.md) * [Style](operators/style.md) * [Subtract](operators/subtract.md) + * [System](operators/system-operators.md) * [Tag](operators/tag.md) * [To Number](operators/to_number.md) * [Traverse (Read)](operators/traverse-read.md) @@ -88,6 +89,7 @@ * [Working with HCL](usage/hcl.md) * [Working with LUA](usage/lua.md) * [Working with TOML](usage/toml.md) +* [Working with KYAML](usage/kyaml.md) * [Working with Shell Output](usage/shellvariables.md) * [Front Matter](usage/front-matter.md) * [Split into multiple files](usage/split-into-multiple-files.md) diff --git a/operators/anchor-and-alias-operators.md b/operators/anchor-and-alias-operators.md index b5dc62eb..97b4ec13 100644 --- a/operators/anchor-and-alias-operators.md +++ b/operators/anchor-and-alias-operators.md @@ -22,7 +22,7 @@ see https://yaml.org/type/merge.html Given a sample.yml file of: ```yaml -- &CENTER +- &CENTRE x: 1 y: 2 - &LEFT @@ -32,7 +32,7 @@ Given a sample.yml file of: r: 10 - &SMALL r: 1 -- !!merge <<: *CENTER +- !!merge <<: *CENTRE r: 10 ``` then @@ -288,7 +288,7 @@ see https://yaml.org/type/merge.html. This has the correct data, but the wrong k Given a sample.yml file of: ```yaml -- &CENTER +- &CENTRE x: 1 y: 2 - &LEFT @@ -299,7 +299,7 @@ Given a sample.yml file of: - &SMALL r: 1 - !!merge <<: - - *CENTER + - *CENTRE - *BIG ``` then @@ -318,7 +318,7 @@ see https://yaml.org/type/merge.html. This has the correct data, but the wrong k Given a sample.yml file of: ```yaml -- &CENTER +- &CENTRE x: 1 y: 2 - &LEFT @@ -401,7 +401,7 @@ Taken from https://yaml.org/type/merge.html. Same values as legacy, but with the Given a sample.yml file of: ```yaml -- &CENTER +- &CENTRE x: 1 y: 2 - &LEFT @@ -412,7 +412,7 @@ Given a sample.yml file of: - &SMALL r: 1 - !!merge <<: - - *CENTER + - *CENTRE - *BIG ``` then @@ -432,7 +432,7 @@ Taken from https://yaml.org/type/merge.html. Same values as legacy, but with the Given a sample.yml file of: ```yaml -- &CENTER +- &CENTRE x: 1 y: 2 - &LEFT diff --git a/operators/datetime.md b/operators/datetime.md index 685618fc..559998ab 100644 --- a/operators/datetime.md +++ b/operators/datetime.md @@ -2,7 +2,7 @@ Various operators for parsing and manipulating dates. -## Date time formattings +## Date time formatting This uses Golang's built in time library for parsing and formatting date times. When not specified, the RFC3339 standard is assumed `2006-01-02T15:04:05Z07:00` for parsing. diff --git a/operators/parent.md b/operators/parent.md index 02375f87..fc474404 100644 --- a/operators/parent.md +++ b/operators/parent.md @@ -79,6 +79,46 @@ will output c: cat ``` +## Get the top (root) parent +Use negative numbers to get the top parents. You can think of this as indexing into the 'parents' array above + +Given a sample.yml file of: +```yaml +a: + b: + c: cat +``` +then +```bash +yq '.a.b.c | parent(-1)' sample.yml +``` +will output +```yaml +a: + b: + c: cat +``` + +## Root +Alias for parent(-1), returns the top level parent. This is usually the document node. + +Given a sample.yml file of: +```yaml +a: + b: + c: cat +``` +then +```bash +yq '.a.b.c | root' sample.yml +``` +will output +```yaml +a: + b: + c: cat +``` + ## N-th parent You can optionally supply the number of levels to go up for the parent, the default being 1. @@ -116,6 +156,25 @@ a: c: cat ``` +## N-th negative +Similarly, use negative numbers to index backwards from the parents array + +Given a sample.yml file of: +```yaml +a: + b: + c: cat +``` +then +```bash +yq '.a.b.c | parent(-2)' sample.yml +``` +will output +```yaml +b: + c: cat +``` + ## No parent Given a sample.yml file of: ```yaml diff --git a/operators/slice-array.md b/operators/slice-array.md index 9b89210b..8daf340f 100644 --- a/operators/slice-array.md +++ b/operators/slice-array.md @@ -1,8 +1,8 @@ -# Slice/Splice Array +# Slice Array or String -The slice array operator takes an array as input and returns a subarray. Like the `jq` equivalent, `.[10:15]` will return an array of length 5, starting from index 10 inclusive, up to index 15 exclusive. Negative numbers count backwards from the end of the array. +The slice operator works on both arrays and strings. Like the `jq` equivalent, `.[10:15]` will return a subarray (or substring) of length 5, starting from index 10 inclusive, up to index 15 exclusive. Negative numbers count backwards from the end of the array or string. -You may leave out the first or second number, which will refer to the start or end of the array respectively. +You may leave out the first or second number, which will refer to the start or end of the array or string respectively. ## Slicing arrays Given a sample.yml file of: @@ -103,3 +103,81 @@ will output - cow ``` +## Slicing strings +Given a sample.yml file of: +```yaml +country: Australia +``` +then +```bash +yq '.country[0:5]' sample.yml +``` +will output +```yaml +Austr +``` + +## Slicing strings - without the second number +Finishes at the end of the string + +Given a sample.yml file of: +```yaml +country: Australia +``` +then +```bash +yq '.country[5:]' sample.yml +``` +will output +```yaml +alia +``` + +## Slicing strings - without the first number +Starts from the start of the string + +Given a sample.yml file of: +```yaml +country: Australia +``` +then +```bash +yq '.country[:5]' sample.yml +``` +will output +```yaml +Austr +``` + +## Slicing strings - use negative numbers to count backwards from the end +Negative indices count from the end of the string + +Given a sample.yml file of: +```yaml +country: Australia +``` +then +```bash +yq '.country[-5:]' sample.yml +``` +will output +```yaml +ralia +``` + +## Slicing strings - Unicode +Indices are rune-based, so multi-byte characters are handled correctly + +Given a sample.yml file of: +```yaml +greeting: héllo +``` +then +```bash +yq '.greeting[1:3]' sample.yml +``` +will output +```yaml +él +``` + diff --git a/operators/system-operators.md b/operators/system-operators.md new file mode 100644 index 00000000..df1a76cd --- /dev/null +++ b/operators/system-operators.md @@ -0,0 +1,76 @@ +# System Operators + +The `system` operator allows you to run an external command and use its output as a value in your expression. + +**Security warning**: The system operator is disabled by default. You must explicitly pass `--security-enable-system-operator` to use it. + +**Note:** When enabled, the system operator can replicate the functionality of `env` and `load` +operators via external commands. Enabling it effectively overrides `--security-disable-env-ops` +and `--security-disable-file-ops`. + +## Usage + +```bash +yq --security-enable-system-operator --null-input '.field = system("command"; "arg1")' +``` + +The operator takes: +- A command string (required) +- An argument (or an array of arguments), separated from the command by `;` (optional) + +The current matched node's value is serialised and piped to the command via stdin. The command's stdout (with trailing newline stripped) is returned as a string. + +## Disabling the system operator + +The system operator is disabled by default. When disabled, an error is returned instead of running the command, consistent with `--security-disable-env-ops` and `--security-disable-file-ops`. + +Use `--security-enable-system-operator` flag to enable it. + +## system operator returns error when disabled +Use `--security-enable-system-operator` to enable the system operator. + +Given a sample.yml file of: +```yaml +country: Australia +``` +then +```bash +yq '.country = system("/usr/bin/echo"; "test")' sample.yml +``` +will output +```bash +Error: system operations are disabled, use --security-enable-system-operator to enable +``` + +## Run a command with an argument +Use `--security-enable-system-operator` to enable the system operator. + +Given a sample.yml file of: +```yaml +country: Australia +``` +then +```bash +yq --security-enable-system-operator '.country = system("/usr/bin/echo"; "test")' sample.yml +``` +will output +```yaml +country: test +``` + +## Run a command without arguments +Omit the semicolon and args to run the command with no extra arguments. + +Given a sample.yml file of: +```yaml +a: hello +``` +then +```bash +yq --security-enable-system-operator '.a = system("/usr/bin/echo")' sample.yml +``` +will output +```yaml +a: "" +``` + diff --git a/operators/traverse-read.md b/operators/traverse-read.md index 1c1a85ae..d418bc00 100644 --- a/operators/traverse-read.md +++ b/operators/traverse-read.md @@ -3,7 +3,7 @@ This is the simplest (and perhaps most used) operator. It is used to navigate deeply into yaml structures. -## NOTE --yaml-fix-merge-anchor-to-spec flag (v4.47.1 [#2110](https://github.com/mikefarah/yq/issues/2110)) +## NOTE --yaml-fix-merge-anchor-to-spec flag `yq` doesn't merge anchors `<<:` to spec, in some circumstances it incorrectly overrides existing keys when the spec documents not to do that. To minimise disruption while still fixing the issue, a flag has been added to toggle this behaviour. This will first default to false; and log warnings to users. Then it will default to true (and still allow users to specify false if needed) diff --git a/usage/hcl.md b/usage/hcl.md index 4d636a70..7704ca40 100644 --- a/usage/hcl.md +++ b/usage/hcl.md @@ -7,7 +7,7 @@ HCL is commonly used in HashiCorp tools like Terraform for configuration files. - String interpolation and expressions (preserved without quotes) - Comments (leading, head, and line comments) - Nested structures (maps and lists) -- Syntax colorization when enabled +- Syntax colorisation when enabled ## Parse HCL diff --git a/usage/kyaml.md b/usage/kyaml.md new file mode 100644 index 00000000..56a3463c --- /dev/null +++ b/usage/kyaml.md @@ -0,0 +1,253 @@ +# KYaml + +Encode and decode to and from KYaml (a restricted subset of YAML that uses flow-style collections). + +KYaml is useful when you want YAML data rendered in a compact, JSON-like form while still supporting YAML features like comments. + +Notes: +- Strings are always double-quoted in KYaml output. +- Anchors and aliases are expanded (KYaml output does not emit them). + +## Encode kyaml: plain string scalar +Strings are always double-quoted in KYaml output. + +Given a sample.yml file of: +```yaml +cat + +``` +then +```bash +yq -o=kyaml '.' sample.yml +``` +will output +```yaml +"cat" +``` + +## encode flow mapping and sequence +Given a sample.yml file of: +```yaml +a: b +c: + - d + +``` +then +```bash +yq -o=kyaml '.' sample.yml +``` +will output +```yaml +{ + a: "b", + c: [ + "d", + ], +} +``` + +## encode non-string scalars +Given a sample.yml file of: +```yaml +a: 12 +b: true +c: null +d: "true" + +``` +then +```bash +yq -o=kyaml '.' sample.yml +``` +will output +```yaml +{ + a: 12, + b: true, + c: null, + d: "true", +} +``` + +## quote non-identifier keys +Given a sample.yml file of: +```yaml +"1a": b +"has space": c + +``` +then +```bash +yq -o=kyaml '.' sample.yml +``` +will output +```yaml +{ + "1a": "b", + "has space": "c", +} +``` + +## escape quoted strings +Given a sample.yml file of: +```yaml +a: "line1\nline2\t\"q\"" + +``` +then +```bash +yq -o=kyaml '.' sample.yml +``` +will output +```yaml +{ + a: "line1\nline2\t\"q\"", +} +``` + +## preserve comments when encoding +Given a sample.yml file of: +```yaml +# leading +a: 1 # a line +# head b +b: 2 +c: + # head d + - d # d line + - e +# trailing + +``` +then +```bash +yq -o=kyaml '.' sample.yml +``` +will output +```yaml +# leading +{ + a: 1, # a line + # head b + b: 2, + c: [ + # head d + "d", # d line + "e", + ], + # trailing +} +``` + +## Encode kyaml: anchors and aliases +KYaml output does not support anchors/aliases; they are expanded to concrete values. + +Given a sample.yml file of: +```yaml +base: &base + a: b +copy: *base + +``` +then +```bash +yq -o=kyaml '.' sample.yml +``` +will output +```yaml +{ + base: { + a: "b", + }, + copy: { + a: "b", + }, +} +``` + +## Encode kyaml: yaml to kyaml shows formatting differences +KYaml uses flow-style collections (braces/brackets) and explicit commas. + +Given a sample.yml file of: +```yaml +person: + name: John + pets: + - cat + - dog + +``` +then +```bash +yq -o=kyaml '.' sample.yml +``` +will output +```yaml +{ + person: { + name: "John", + pets: [ + "cat", + "dog", + ], + }, +} +``` + +## Encode kyaml: nested lists of objects +Lists and objects can be nested arbitrarily; KYaml always uses flow-style collections. + +Given a sample.yml file of: +```yaml +- name: a + items: + - id: 1 + tags: + - k: x + v: y + - k: x2 + v: y2 + - id: 2 + tags: + - k: z + v: w + +``` +then +```bash +yq -o=kyaml '.' sample.yml +``` +will output +```yaml +[ + { + name: "a", + items: [ + { + id: 1, + tags: [ + { + k: "x", + v: "y", + }, + { + k: "x2", + v: "y2", + }, + ], + }, + { + id: 2, + tags: [ + { + k: "z", + v: "w", + }, + ], + }, + ], + }, +] +``` + diff --git a/usage/toml.md b/usage/toml.md index 7cc4c375..dab4abc6 100644 --- a/usage/toml.md +++ b/usage/toml.md @@ -141,3 +141,263 @@ will output dependencies: {} ``` +## Roundtrip: inline table attribute +Given a sample.toml file of: +```toml +name = { first = "Tom", last = "Preston-Werner" } + +``` +then +```bash +yq '.' sample.toml +``` +will output +```yaml +name = { first = "Tom", last = "Preston-Werner" } +``` + +## Roundtrip: table section +Given a sample.toml file of: +```toml +[owner.contact] +name = "Tom" +age = 36 + +``` +then +```bash +yq '.' sample.toml +``` +will output +```yaml +[owner.contact] +name = "Tom" +age = 36 +``` + +## Roundtrip: array of tables +Given a sample.toml file of: +```toml +[[fruits]] +name = "apple" +[[fruits.varieties]] +name = "red delicious" + +``` +then +```bash +yq '.' sample.toml +``` +will output +```yaml +[[fruits]] +name = "apple" +[[fruits.varieties]] +name = "red delicious" +``` + +## Roundtrip: arrays and scalars +Given a sample.toml file of: +```toml +A = ["hello", ["world", "again"]] +B = 12 + +``` +then +```bash +yq '.' sample.toml +``` +will output +```yaml +A = ["hello", ["world", "again"]] +B = 12 +``` + +## Roundtrip: simple +Given a sample.toml file of: +```toml +A = "hello" +B = 12 + +``` +then +```bash +yq '.' sample.toml +``` +will output +```yaml +A = "hello" +B = 12 +``` + +## Roundtrip: deep paths +Given a sample.toml file of: +```toml +[person] +name = "hello" +address = "12 cat st" + +``` +then +```bash +yq '.' sample.toml +``` +will output +```yaml +[person] +name = "hello" +address = "12 cat st" +``` + +## Roundtrip: empty array +Given a sample.toml file of: +```toml +A = [] + +``` +then +```bash +yq '.' sample.toml +``` +will output +```yaml +A = [] +``` + +## Roundtrip: sample table +Given a sample.toml file of: +```toml +var = "x" + +[owner.contact] +name = "Tom Preston-Werner" +age = 36 + +``` +then +```bash +yq '.' sample.toml +``` +will output +```yaml +var = "x" + +[owner.contact] +name = "Tom Preston-Werner" +age = 36 +``` + +## Roundtrip: empty table +Given a sample.toml file of: +```toml +[dependencies] + +``` +then +```bash +yq '.' sample.toml +``` +will output +```yaml +[dependencies] +``` + +## Roundtrip: comments +Given a sample.toml file of: +```toml +# This is a comment +A = "hello" # inline comment +B = 12 + +# Table comment +[person] +name = "Tom" # name comment + +``` +then +```bash +yq '.' sample.toml +``` +will output +```yaml +# This is a comment +A = "hello" # inline comment +B = 12 + +# Table comment +[person] +name = "Tom" # name comment +``` + +## Roundtrip: sample from web +Given a sample.toml file of: +```toml +# This is a TOML document +title = "TOML Example" + +[owner] +name = "Tom Preston-Werner" +dob = 1979-05-27T07:32:00-08:00 + +[database] +enabled = true +ports = [8000, 8001, 8002] +data = [["delta", "phi"], [3.14]] +temp_targets = { cpu = 79.5, case = 72.0 } + +# [servers] yq can't do this one yet +[servers.alpha] +ip = "10.0.0.1" +role = "frontend" + +[servers.beta] +ip = "10.0.0.2" +role = "backend" + +``` +then +```bash +yq '.' sample.toml +``` +will output +```yaml +# This is a TOML document +title = "TOML Example" + +[owner] +name = "Tom Preston-Werner" +dob = 1979-05-27T07:32:00-08:00 + +[database] +enabled = true +ports = [8000, 8001, 8002] +data = [["delta", "phi"], [3.14]] +temp_targets = { cpu = 79.5, case = 72.0 } + +# [servers] yq can't do this one yet +[servers.alpha] +ip = "10.0.0.1" +role = "frontend" + +[servers.beta] +ip = "10.0.0.2" +role = "backend" +``` + +## Encode: Simple mapping produces table section +Given a sample.yml file of: +```yaml +arg: + hello: foo + +``` +then +```bash +yq -o toml '.' sample.yml +``` +will output +```toml +[arg] +hello = "foo" +``` +