diff --git a/operators/anchor-and-alias-operators.md b/operators/anchor-and-alias-operators.md index 14b0bef2..cfdc8579 100644 --- a/operators/anchor-and-alias-operators.md +++ b/operators/anchor-and-alias-operators.md @@ -5,6 +5,18 @@ Use the `alias` and `anchor` operators to read and write yaml aliases and anchor `yq` supports merge aliases (like `<<: *blah`) however this is no longer in the standard yaml spec (1.2) and so `yq` will automatically add the `!!merge` tag to these nodes as it is effectively a custom tag. +## NOTE --yaml-fix-merge-anchor-to-spec flag +`yq` doesn't merge anchors `<<:` to spec, in some circumstances it incorrectly overrides existing keys when the spec documents not to do that. + +To minimise disruption while still fixing the issue, a flag has been added to toggle this behaviour. This will first default to false; and log warnings to users. Then it will default to true (and still allow users to specify false if needed). + +This flag also enables advanced merging, like inline maps, as well as fixes to ensure when exploding a particular path, neighbours are not affect ed. + +Long story short, you should be setting this flag to true. + +See examples of the flag differences below, where LEGACY is with the flag off; and FIXED is with the flag on. + + ## Merge one map see https://yaml.org/type/merge.html @@ -34,68 +46,6 @@ y: 2 r: 10 ``` -## Merge multiple maps -see https://yaml.org/type/merge.html - -Given a sample.yml file of: -```yaml -- &CENTER - x: 1 - y: 2 -- &LEFT - x: 0 - y: 2 -- &BIG - r: 10 -- &SMALL - r: 1 -- !!merge <<: - - *CENTER - - *BIG -``` -then -```bash -yq '.[4] | explode(.)' sample.yml -``` -will output -```yaml -r: 10 -x: 1 -y: 2 -``` - -## Override -see https://yaml.org/type/merge.html - -Given a sample.yml file of: -```yaml -- &CENTER - x: 1 - y: 2 -- &LEFT - x: 0 - y: 2 -- &BIG - r: 10 -- &SMALL - r: 1 -- !!merge <<: - - *BIG - - *LEFT - - *SMALL - x: 1 -``` -then -```bash -yq '.[4] | explode(.)' sample.yml -``` -will output -```yaml -r: 10 -x: 1 -y: 2 -``` - ## Get anchor Given a sample.yml file of: ```yaml @@ -254,7 +204,39 @@ f: cat: b ``` -## Explode with merge anchors +## Dereference and update a field +Use explode with multiply to dereference an object + +Given a sample.yml file of: +```yaml +item_value: &item_value + value: true +thingOne: + name: item_1 + !!merge <<: *item_value +thingTwo: + name: item_2 + !!merge <<: *item_value +``` +then +```bash +yq '.thingOne |= (explode(.) | sort_keys(.)) * {"value": false}' sample.yml +``` +will output +```yaml +item_value: &item_value + value: true +thingOne: + name: item_1 + value: false +thingTwo: + name: item_2 + !!merge <<: *item_value +``` + +## LEGACY: Explode with merge anchors +Caution: this is for when --yaml-fix-merge-anchor-to-spec=false; it's not to YAML spec because the merge anchors incorrectly override the object values (foobarList.b is set to bar_b when it should still be foobarList_b). Flag will default to true in late 2025 + Given a sample.yml file of: ```yaml foo: &foo @@ -301,33 +283,201 @@ foobar: thing: foobar_thing ``` -## Dereference and update a field -Use explode with multiply to dereference an object +## LEGACY: Merge multiple maps +see https://yaml.org/type/merge.html. This has the correct data, but the wrong key order; set --yaml-fix-merge-anchor-to-spec=true to fix the key order. Given a sample.yml file of: ```yaml -item_value: &item_value - value: true -thingOne: - name: item_1 - !!merge <<: *item_value -thingTwo: - name: item_2 - !!merge <<: *item_value +- &CENTER + x: 1 + y: 2 +- &LEFT + x: 0 + y: 2 +- &BIG + r: 10 +- &SMALL + r: 1 +- !!merge <<: + - *CENTER + - *BIG ``` then ```bash -yq '.thingOne |= explode(.) * {"value": false}' sample.yml +yq '.[4] | explode(.)' sample.yml ``` will output ```yaml -item_value: &item_value - value: true -thingOne: - name: item_1 - value: false -thingTwo: - name: item_2 - !!merge <<: *item_value +r: 10 +x: 1 +y: 2 +``` + +## LEGACY: Override +see https://yaml.org/type/merge.html. This has the correct data, but the wrong key order; set --yaml-fix-merge-anchor-to-spec=true to fix the key order. + +Given a sample.yml file of: +```yaml +- &CENTER + x: 1 + y: 2 +- &LEFT + x: 0 + y: 2 +- &BIG + r: 10 +- &SMALL + r: 1 +- !!merge <<: + - *BIG + - *LEFT + - *SMALL + x: 1 +``` +then +```bash +yq '.[4] | explode(.)' sample.yml +``` +will output +```yaml +r: 10 +x: 1 +y: 2 +``` + +## FIXED: Explode with merge anchors +Set `--yaml-fix-merge-anchor-to-spec=true` to get this correct merge behaviour (flag will default to true in late 2025). +Observe that foobarList.b property is still foobarList_b. + +Given a sample.yml file of: +```yaml +foo: &foo + a: foo_a + thing: foo_thing + c: foo_c +bar: &bar + b: bar_b + thing: bar_thing + c: bar_c +foobarList: + b: foobarList_b + !!merge <<: + - *foo + - *bar + c: foobarList_c +foobar: + c: foobar_c + !!merge <<: *foo + thing: foobar_thing +``` +then +```bash +yq 'explode(.)' sample.yml +``` +will output +```yaml +foo: + a: foo_a + thing: foo_thing + c: foo_c +bar: + b: bar_b + thing: bar_thing + c: bar_c +foobarList: + b: foobarList_b + a: foo_a + thing: foo_thing + c: foobarList_c +foobar: + c: foobar_c + a: foo_a + thing: foobar_thing +``` + +## FIXED: Merge multiple maps +Set `--yaml-fix-merge-anchor-to-spec=true` to get this correct merge behaviour (flag will default to true in late 2025). +Taken from https://yaml.org/type/merge.html. Same values as legacy, but with the correct key order. + +Given a sample.yml file of: +```yaml +- &CENTER + x: 1 + y: 2 +- &LEFT + x: 0 + y: 2 +- &BIG + r: 10 +- &SMALL + r: 1 +- !!merge <<: + - *CENTER + - *BIG +``` +then +```bash +yq '.[4] | explode(.)' sample.yml +``` +will output +```yaml +x: 1 +y: 2 +r: 10 +``` + +## FIXED: Override +Set `--yaml-fix-merge-anchor-to-spec=true` to get this correct merge behaviour (flag will default to true in late 2025). +Taken from https://yaml.org/type/merge.html. Same values as legacy, but with the correct key order. + +Given a sample.yml file of: +```yaml +- &CENTER + x: 1 + y: 2 +- &LEFT + x: 0 + y: 2 +- &BIG + r: 10 +- &SMALL + r: 1 +- !!merge <<: + - *BIG + - *LEFT + - *SMALL + x: 1 +``` +then +```bash +yq '.[4] | explode(.)' sample.yml +``` +will output +```yaml +r: 10 +y: 2 +x: 1 +``` + +## Exploding inline merge anchor +Set `--yaml-fix-merge-anchor-to-spec=true` to get this correct merge behaviour (flag will default to true in late 2025). + + +Given a sample.yml file of: +```yaml +a: + b: &b 42 +!!merge <<: + c: *b +``` +then +```bash +yq 'explode(.) | sort_keys(.)' sample.yml +``` +will output +```yaml +a: + b: 42 +c: 42 ``` diff --git a/operators/entries.md b/operators/entries.md index 232ff4df..4e2ce2dd 100644 --- a/operators/entries.md +++ b/operators/entries.md @@ -2,7 +2,7 @@ Similar to the same named functions in `jq` these functions convert to/from an object and an array of key-value pairs. This is most useful for performing operations on keys of maps. -Use `with_entries(op)` as a syntatic sugar for doing `to_entries | op | from_entries`. +Use `with_entries(op)` as a syntactic sugar for doing `to_entries | op | from_entries`. ## to_entries Map Given a sample.yml file of: diff --git a/operators/traverse-read.md b/operators/traverse-read.md index fefe24df..d418bc00 100644 --- a/operators/traverse-read.md +++ b/operators/traverse-read.md @@ -2,6 +2,15 @@ This is the simplest (and perhaps most used) operator. It is used to navigate deeply into yaml structures. + +## NOTE --yaml-fix-merge-anchor-to-spec flag +`yq` doesn't merge anchors `<<:` to spec, in some circumstances it incorrectly overrides existing keys when the spec documents not to do that. + +To minimise disruption while still fixing the issue, a flag has been added to toggle this behaviour. This will first default to false; and log warnings to users. Then it will default to true (and still allow users to specify false if needed) + +See examples of the flag differences below, where LEGACY is the flag off; and FIXED is with the flag on. + + ## Simple map navigation Given a sample.yml file of: ```yaml @@ -303,37 +312,6 @@ will output foo_a ``` -## Traversing merge anchors with override -Given a sample.yml file of: -```yaml -foo: &foo - a: foo_a - thing: foo_thing - c: foo_c -bar: &bar - b: bar_b - thing: bar_thing - c: bar_c -foobarList: - b: foobarList_b - !!merge <<: - - *foo - - *bar - c: foobarList_c -foobar: - c: foobar_c - !!merge <<: *foo - thing: foobar_thing -``` -then -```bash -yq '.foobar.c' sample.yml -``` -will output -```yaml -foo_c -``` - ## Traversing merge anchors with local override Given a sample.yml file of: ```yaml @@ -365,7 +343,93 @@ will output foobar_thing ``` -## Splatting merge anchors +## Select multiple indices +Given a sample.yml file of: +```yaml +a: + - a + - b + - c +``` +then +```bash +yq '.a[0, 2]' sample.yml +``` +will output +```yaml +a +c +``` + +## LEGACY: Traversing merge anchors with override +This is legacy behaviour, see --yaml-fix-merge-anchor-to-spec + +Given a sample.yml file of: +```yaml +foo: &foo + a: foo_a + thing: foo_thing + c: foo_c +bar: &bar + b: bar_b + thing: bar_thing + c: bar_c +foobarList: + b: foobarList_b + !!merge <<: + - *foo + - *bar + c: foobarList_c +foobar: + c: foobar_c + !!merge <<: *foo + thing: foobar_thing +``` +then +```bash +yq '.foobar.c' sample.yml +``` +will output +```yaml +foo_c +``` + +## LEGACY: Traversing merge anchor lists +Note that the later merge anchors override previous, but this is legacy behaviour, see --yaml-fix-merge-anchor-to-spec + +Given a sample.yml file of: +```yaml +foo: &foo + a: foo_a + thing: foo_thing + c: foo_c +bar: &bar + b: bar_b + thing: bar_thing + c: bar_c +foobarList: + b: foobarList_b + !!merge <<: + - *foo + - *bar + c: foobarList_c +foobar: + c: foobar_c + !!merge <<: *foo + thing: foobar_thing +``` +then +```bash +yq '.foobarList.thing' sample.yml +``` +will output +```yaml +bar_thing +``` + +## LEGACY: Splatting merge anchors +With legacy override behaviour, see --yaml-fix-merge-anchor-to-spec + Given a sample.yml file of: ```yaml foo: &foo @@ -398,40 +462,9 @@ foo_a foobar_thing ``` -## Traversing merge anchor lists -Note that the later merge anchors override previous +## LEGACY: Splatting merge anchor lists +With legacy override behaviour, see --yaml-fix-merge-anchor-to-spec -Given a sample.yml file of: -```yaml -foo: &foo - a: foo_a - thing: foo_thing - c: foo_c -bar: &bar - b: bar_b - thing: bar_thing - c: bar_c -foobarList: - b: foobarList_b - !!merge <<: - - *foo - - *bar - c: foobarList_c -foobar: - c: foobar_c - !!merge <<: *foo - thing: foobar_thing -``` -then -```bash -yq '.foobarList.thing' sample.yml -``` -will output -```yaml -bar_thing -``` - -## Splatting merge anchor lists Given a sample.yml file of: ```yaml foo: &foo @@ -465,21 +498,140 @@ bar_thing foobarList_c ``` -## Select multiple indices +## FIXED: Traversing merge anchors with override +Set `--yaml-fix-merge-anchor-to-spec=true` to get this correct merge behaviour. + Given a sample.yml file of: ```yaml -a: - - a - - b - - c +foo: &foo + a: foo_a + thing: foo_thing + c: foo_c +bar: &bar + b: bar_b + thing: bar_thing + c: bar_c +foobarList: + b: foobarList_b + !!merge <<: + - *foo + - *bar + c: foobarList_c +foobar: + c: foobar_c + !!merge <<: *foo + thing: foobar_thing ``` then ```bash -yq '.a[0, 2]' sample.yml +yq '.foobar.c' sample.yml ``` will output ```yaml -a -c +foobar_c +``` + +## FIXED: Traversing merge anchor lists +Set `--yaml-fix-merge-anchor-to-spec=true` to get this correct merge behaviour. Note that the keys earlier in the merge anchors sequence override later ones + +Given a sample.yml file of: +```yaml +foo: &foo + a: foo_a + thing: foo_thing + c: foo_c +bar: &bar + b: bar_b + thing: bar_thing + c: bar_c +foobarList: + b: foobarList_b + !!merge <<: + - *foo + - *bar + c: foobarList_c +foobar: + c: foobar_c + !!merge <<: *foo + thing: foobar_thing +``` +then +```bash +yq '.foobarList.thing' sample.yml +``` +will output +```yaml +foo_thing +``` + +## FIXED: Splatting merge anchors +Set `--yaml-fix-merge-anchor-to-spec=true` to get this correct merge behaviour. Note that the keys earlier in the merge anchors sequence override later ones + +Given a sample.yml file of: +```yaml +foo: &foo + a: foo_a + thing: foo_thing + c: foo_c +bar: &bar + b: bar_b + thing: bar_thing + c: bar_c +foobarList: + b: foobarList_b + !!merge <<: + - *foo + - *bar + c: foobarList_c +foobar: + c: foobar_c + !!merge <<: *foo + thing: foobar_thing +``` +then +```bash +yq '.foobar[]' sample.yml +``` +will output +```yaml +foo_a +foobar_thing +foobar_c +``` + +## FIXED: Splatting merge anchor lists +Set `--yaml-fix-merge-anchor-to-spec=true` to get this correct merge behaviour. Note that the keys earlier in the merge anchors sequence override later ones + +Given a sample.yml file of: +```yaml +foo: &foo + a: foo_a + thing: foo_thing + c: foo_c +bar: &bar + b: bar_b + thing: bar_thing + c: bar_c +foobarList: + b: foobarList_b + !!merge <<: + - *foo + - *bar + c: foobarList_c +foobar: + c: foobar_c + !!merge <<: *foo + thing: foobar_thing +``` +then +```bash +yq '.foobarList[]' sample.yml +``` +will output +```yaml +foobarList_b +foo_thing +foobarList_c +foo_a ```