Updating readme

README.md

@@ -1,8 +1,8 @@
 # yq
 
-a lightweight and portable command-line YAML processor. `yq` uses [jq](https://github.com/stedolan/jq) like syntax but works with yaml files as well as json. It doesn't yet support everything `jq` does - but it does support the most common operations and functions, and more is being added continuously.
+aa lightweight and portable command-line YAML, JSON, INI and XML processor. `yq` uses [jq](https://github.com/stedolan/jq) like syntax but works with yaml files as well as json, xml, ini, properties, csv and tsv. It doesn't yet support everything `jq` does - but it does support the most common operations and functions, and more is being added continuously.
 
-yq is written in go - so you can download a dependency free binary for your platform and you are good to go! If you prefer there are a variety of package managers that can be used as well as docker, all listed below.
+yq is written in go - so you can download a dependency free binary for your platform and you are good to go! If you prefer there are a variety of package managers that can be used as well as Docker and Podman, all listed below.
 
 ## Quick Usage Guide
 
@@ -13,10 +13,10 @@ yq '.a.b[0].c' file.yaml
 
 Pipe from STDIN:
 ```bash
-cat file.yaml | yq '.a.b[0].c'
+yq '.a.b[0].c' < file.yaml
 ```
 
-Update a yaml file, inplace
+Update a yaml file, in place
 ```bash
 yq -i '.a.b[0].c = "cool"' file.yaml
 ```
@@ -27,7 +27,13 @@ NAME=mike yq -i '.a.b[0].c = strenv(NAME)' file.yaml
 ```
 
 Merge multiple files
-```
+```bash
+# merge two files
+yq -n 'load("file1.yaml") * load("file2.yaml")'
+
+# merge using globs:
+# note the use of `ea` to evaluate all the files at once
+# instead of in sequence
 yq ea '. as $item ireduce ({}; . * $item )' path/to/*.yml
 ```
 
@@ -40,6 +46,18 @@ yq -i '
 ' file.yaml
 ```
 
+Find and update an item in an array:
+```bash
+yq '(.[] | select(.name == "foo") | .address) = "12 cat st"'
+```
+
+Convert JSON to YAML
+```bash
+yq -Poy sample.json
+```
+
+See [recipes](https://mikefarah.gitbook.io/yq/recipes) for more examples and the [documentation](https://mikefarah.gitbook.io/yq/) for more information.
+
 Take a look at the discussions for [common questions](https://github.com/mikefarah/yq/discussions/categories/q-a), and [cool ideas](https://github.com/mikefarah/yq/discussions/categories/show-and-tell)
 
 ## Install

operators/anchor-and-alias-operators.md

@@ -5,6 +5,18 @@ Use the `alias` and `anchor` operators to read and write yaml aliases and anchor
 `yq` supports merge aliases (like `<<: *blah`) however this is no longer in the standard yaml spec (1.2) and so `yq` will automatically add the `!!merge` tag to these nodes as it is effectively a custom tag.
 
 
+## NOTE --yaml-fix-merge-anchor-to-spec flag
+`yq` doesn't merge anchors `<<:` to spec, in some circumstances it incorrectly overrides existing keys when the spec documents not to do that.
+
+To minimise disruption while still fixing the issue, a flag has been added to toggle this behaviour. This will first default to false; and log warnings to users. Then it will default to true (and still allow users to specify false if needed).
+
+This flag also enables advanced merging, like inline maps, as well as fixes to ensure when exploding a particular path, neighbours are not affect ed.
+
+Long story short, you should be setting this flag to true.
+
+See examples of the flag differences below, where LEGACY is with the flag off; and FIXED is with the flag on.
+
+
 ## Merge one map
 see https://yaml.org/type/merge.html
 
@@ -34,68 +46,6 @@ y: 2
 r: 10
 ```
 
-## Merge multiple maps
-see https://yaml.org/type/merge.html
-
-Given a sample.yml file of:
-```yaml
-- &CENTER
-  x: 1
-  y: 2
-- &LEFT
-  x: 0
-  y: 2
-- &BIG
-  r: 10
-- &SMALL
-  r: 1
-- !!merge <<:
-    - *CENTER
-    - *BIG
-```
-then
-```bash
-yq '.[4] | explode(.)' sample.yml
-```
-will output
-```yaml
-r: 10
-x: 1
-y: 2
-```
-
-## Override
-see https://yaml.org/type/merge.html
-
-Given a sample.yml file of:
-```yaml
-- &CENTER
-  x: 1
-  y: 2
-- &LEFT
-  x: 0
-  y: 2
-- &BIG
-  r: 10
-- &SMALL
-  r: 1
-- !!merge <<:
-    - *BIG
-    - *LEFT
-    - *SMALL
-  x: 1
-```
-then
-```bash
-yq '.[4] | explode(.)' sample.yml
-```
-will output
-```yaml
-r: 10
-x: 1
-y: 2
-```
-
 ## Get anchor
 Given a sample.yml file of:
 ```yaml
@@ -254,7 +204,39 @@ f:
   cat: b
 ```
 
-## Explode with merge anchors
+## Dereference and update a field
+Use explode with multiply to dereference an object
+
+Given a sample.yml file of:
+```yaml
+item_value: &item_value
+  value: true
+thingOne:
+  name: item_1
+  !!merge <<: *item_value
+thingTwo:
+  name: item_2
+  !!merge <<: *item_value
+```
+then
+```bash
+yq '.thingOne |= (explode(.) | sort_keys(.)) * {"value": false}' sample.yml
+```
+will output
+```yaml
+item_value: &item_value
+  value: true
+thingOne:
+  name: item_1
+  value: false
+thingTwo:
+  name: item_2
+  !!merge <<: *item_value
+```
+
+## LEGACY: Explode with merge anchors
+Caution: this is for when --yaml-fix-merge-anchor-to-spec=false; it's not to YAML spec because the merge anchors incorrectly override the object values (foobarList.b is set to bar_b when it should still be foobarList_b). Flag will default to true in late 2025
+
 Given a sample.yml file of:
 ```yaml
 foo: &foo
@@ -301,33 +283,201 @@ foobar:
   thing: foobar_thing
 ```
 
-## Dereference and update a field
-Use explode with multiply to dereference an object
+## LEGACY: Merge multiple maps
+see https://yaml.org/type/merge.html. This has the correct data, but the wrong key order; set --yaml-fix-merge-anchor-to-spec=true to fix the key order.
 
 Given a sample.yml file of:
 ```yaml
-item_value: &item_value
-  value: true
-thingOne:
-  name: item_1
-  !!merge <<: *item_value
-thingTwo:
-  name: item_2
-  !!merge <<: *item_value
+- &CENTER
+  x: 1
+  y: 2
+- &LEFT
+  x: 0
+  y: 2
+- &BIG
+  r: 10
+- &SMALL
+  r: 1
+- !!merge <<:
+    - *CENTER
+    - *BIG
 ```
 then
 ```bash
-yq '.thingOne |= explode(.) * {"value": false}' sample.yml
+yq '.[4] | explode(.)' sample.yml
 ```
 will output
 ```yaml
-item_value: &item_value
-  value: true
-thingOne:
-  name: item_1
-  value: false
-thingTwo:
-  name: item_2
-  !!merge <<: *item_value
+r: 10
+x: 1
+y: 2
+```
+
+## LEGACY: Override
+see https://yaml.org/type/merge.html. This has the correct data, but the wrong key order; set --yaml-fix-merge-anchor-to-spec=true to fix the key order.
+
+Given a sample.yml file of:
+```yaml
+- &CENTER
+  x: 1
+  y: 2
+- &LEFT
+  x: 0
+  y: 2
+- &BIG
+  r: 10
+- &SMALL
+  r: 1
+- !!merge <<:
+    - *BIG
+    - *LEFT
+    - *SMALL
+  x: 1
+```
+then
+```bash
+yq '.[4] | explode(.)' sample.yml
+```
+will output
+```yaml
+r: 10
+x: 1
+y: 2
+```
+
+## FIXED: Explode with merge anchors
+Set `--yaml-fix-merge-anchor-to-spec=true` to get this correct merge behaviour (flag will default to true in late 2025).
+Observe that foobarList.b property is still foobarList_b.
+
+Given a sample.yml file of:
+```yaml
+foo: &foo
+  a: foo_a
+  thing: foo_thing
+  c: foo_c
+bar: &bar
+  b: bar_b
+  thing: bar_thing
+  c: bar_c
+foobarList:
+  b: foobarList_b
+  !!merge <<:
+    - *foo
+    - *bar
+  c: foobarList_c
+foobar:
+  c: foobar_c
+  !!merge <<: *foo
+  thing: foobar_thing
+```
+then
+```bash
+yq 'explode(.)' sample.yml
+```
+will output
+```yaml
+foo:
+  a: foo_a
+  thing: foo_thing
+  c: foo_c
+bar:
+  b: bar_b
+  thing: bar_thing
+  c: bar_c
+foobarList:
+  b: foobarList_b
+  a: foo_a
+  thing: foo_thing
+  c: foobarList_c
+foobar:
+  c: foobar_c
+  a: foo_a
+  thing: foobar_thing
+```
+
+## FIXED: Merge multiple maps
+Set `--yaml-fix-merge-anchor-to-spec=true` to get this correct merge behaviour (flag will default to true in late 2025).
+Taken from https://yaml.org/type/merge.html. Same values as legacy, but with the correct key order.
+
+Given a sample.yml file of:
+```yaml
+- &CENTER
+  x: 1
+  y: 2
+- &LEFT
+  x: 0
+  y: 2
+- &BIG
+  r: 10
+- &SMALL
+  r: 1
+- !!merge <<:
+    - *CENTER
+    - *BIG
+```
+then
+```bash
+yq '.[4] | explode(.)' sample.yml
+```
+will output
+```yaml
+x: 1
+y: 2
+r: 10
+```
+
+## FIXED: Override
+Set `--yaml-fix-merge-anchor-to-spec=true` to get this correct merge behaviour (flag will default to true in late 2025).
+Taken from https://yaml.org/type/merge.html. Same values as legacy, but with the correct key order.
+
+Given a sample.yml file of:
+```yaml
+- &CENTER
+  x: 1
+  y: 2
+- &LEFT
+  x: 0
+  y: 2
+- &BIG
+  r: 10
+- &SMALL
+  r: 1
+- !!merge <<:
+    - *BIG
+    - *LEFT
+    - *SMALL
+  x: 1
+```
+then
+```bash
+yq '.[4] | explode(.)' sample.yml
+```
+will output
+```yaml
+r: 10
+y: 2
+x: 1
+```
+
+## Exploding inline merge anchor
+Set `--yaml-fix-merge-anchor-to-spec=true` to get this correct merge behaviour (flag will default to true in late 2025).
+
+
+Given a sample.yml file of:
+```yaml
+a:
+  b: &b 42
+!!merge <<:
+  c: *b
+```
+then
+```bash
+yq 'explode(.) | sort_keys(.)' sample.yml
+```
+will output
+```yaml
+a:
+  b: 42
+c: 42
 ```
 

operators/entries.md

@@ -2,7 +2,7 @@
 
 Similar to the same named functions in `jq` these functions convert to/from an object and an array of key-value pairs. This is most useful for performing operations on keys of maps.
 
-Use `with_entries(op)` as a syntatic sugar for doing `to_entries | op | from_entries`.
+Use `with_entries(op)` as a syntactic sugar for doing `to_entries | op | from_entries`.
 
 ## to_entries Map
 Given a sample.yml file of:

operators/string-operators.md

@@ -3,7 +3,7 @@
 ## RegEx
 This uses Golang's native regex functions under the hood - See their [docs](https://github.com/google/re2/wiki/Syntax) for the supported syntax.
 
-Case insensitive tip: prefix the regex with `(?i)` - e.g. `test("(?i)cats)"`.
+Case insensitive tip: prefix the regex with `(?i)` - e.g. `test("(?i)cats")`.
 
 ### match(regEx)
 This operator returns the substring match details of the given regEx.

operators/traverse-read.md

@@ -2,6 +2,15 @@
 
 This is the simplest (and perhaps most used) operator. It is used to navigate deeply into yaml structures.
 
+
+## NOTE --yaml-fix-merge-anchor-to-spec flag
+`yq` doesn't merge anchors `<<:` to spec, in some circumstances it incorrectly overrides existing keys when the spec documents not to do that.
+
+To minimise disruption while still fixing the issue, a flag has been added to toggle this behaviour. This will first default to false; and log warnings to users. Then it will default to true (and still allow users to specify false if needed)
+
+See examples of the flag differences below, where LEGACY is the flag off; and FIXED is with the flag on.
+
+
 ## Simple map navigation
 Given a sample.yml file of:
 ```yaml
@@ -303,37 +312,6 @@ will output
 foo_a
 ```
 
-## Traversing merge anchors with override
-Given a sample.yml file of:
-```yaml
-foo: &foo
-  a: foo_a
-  thing: foo_thing
-  c: foo_c
-bar: &bar
-  b: bar_b
-  thing: bar_thing
-  c: bar_c
-foobarList:
-  b: foobarList_b
-  !!merge <<:
-    - *foo
-    - *bar
-  c: foobarList_c
-foobar:
-  c: foobar_c
-  !!merge <<: *foo
-  thing: foobar_thing
-```
-then
-```bash
-yq '.foobar.c' sample.yml
-```
-will output
-```yaml
-foo_c
-```
-
 ## Traversing merge anchors with local override
 Given a sample.yml file of:
 ```yaml
@@ -365,7 +343,93 @@ will output
 foobar_thing
 ```
 
-## Splatting merge anchors
+## Select multiple indices
+Given a sample.yml file of:
+```yaml
+a:
+  - a
+  - b
+  - c
+```
+then
+```bash
+yq '.a[0, 2]' sample.yml
+```
+will output
+```yaml
+a
+c
+```
+
+## LEGACY: Traversing merge anchors with override
+This is legacy behaviour, see --yaml-fix-merge-anchor-to-spec
+
+Given a sample.yml file of:
+```yaml
+foo: &foo
+  a: foo_a
+  thing: foo_thing
+  c: foo_c
+bar: &bar
+  b: bar_b
+  thing: bar_thing
+  c: bar_c
+foobarList:
+  b: foobarList_b
+  !!merge <<:
+    - *foo
+    - *bar
+  c: foobarList_c
+foobar:
+  c: foobar_c
+  !!merge <<: *foo
+  thing: foobar_thing
+```
+then
+```bash
+yq '.foobar.c' sample.yml
+```
+will output
+```yaml
+foo_c
+```
+
+## LEGACY: Traversing merge anchor lists
+Note that the later merge anchors override previous, but this is legacy behaviour, see --yaml-fix-merge-anchor-to-spec
+
+Given a sample.yml file of:
+```yaml
+foo: &foo
+  a: foo_a
+  thing: foo_thing
+  c: foo_c
+bar: &bar
+  b: bar_b
+  thing: bar_thing
+  c: bar_c
+foobarList:
+  b: foobarList_b
+  !!merge <<:
+    - *foo
+    - *bar
+  c: foobarList_c
+foobar:
+  c: foobar_c
+  !!merge <<: *foo
+  thing: foobar_thing
+```
+then
+```bash
+yq '.foobarList.thing' sample.yml
+```
+will output
+```yaml
+bar_thing
+```
+
+## LEGACY: Splatting merge anchors
+With legacy override behaviour, see --yaml-fix-merge-anchor-to-spec
+
 Given a sample.yml file of:
 ```yaml
 foo: &foo
@@ -398,40 +462,9 @@ foo_a
 foobar_thing
 ```
 
-## Traversing merge anchor lists
-Note that the later merge anchors override previous
+## LEGACY: Splatting merge anchor lists
+With legacy override behaviour, see --yaml-fix-merge-anchor-to-spec
 
-Given a sample.yml file of:
-```yaml
-foo: &foo
-  a: foo_a
-  thing: foo_thing
-  c: foo_c
-bar: &bar
-  b: bar_b
-  thing: bar_thing
-  c: bar_c
-foobarList:
-  b: foobarList_b
-  !!merge <<:
-    - *foo
-    - *bar
-  c: foobarList_c
-foobar:
-  c: foobar_c
-  !!merge <<: *foo
-  thing: foobar_thing
-```
-then
-```bash
-yq '.foobarList.thing' sample.yml
-```
-will output
-```yaml
-bar_thing
-```
-
-## Splatting merge anchor lists
 Given a sample.yml file of:
 ```yaml
 foo: &foo
@@ -465,21 +498,140 @@ bar_thing
 foobarList_c
 ```
 
-## Select multiple indices
+## FIXED: Traversing merge anchors with override
+Set `--yaml-fix-merge-anchor-to-spec=true` to get this correct merge behaviour.
+
 Given a sample.yml file of:
 ```yaml
-a:
-  - a
-  - b
-  - c
+foo: &foo
+  a: foo_a
+  thing: foo_thing
+  c: foo_c
+bar: &bar
+  b: bar_b
+  thing: bar_thing
+  c: bar_c
+foobarList:
+  b: foobarList_b
+  !!merge <<:
+    - *foo
+    - *bar
+  c: foobarList_c
+foobar:
+  c: foobar_c
+  !!merge <<: *foo
+  thing: foobar_thing
 ```
 then
 ```bash
-yq '.a[0, 2]' sample.yml
+yq '.foobar.c' sample.yml
 ```
 will output
 ```yaml
-a
-c
+foobar_c
+```
+
+## FIXED: Traversing merge anchor lists
+Set `--yaml-fix-merge-anchor-to-spec=true` to get this correct merge behaviour. Note that the keys earlier in the merge anchors sequence override later ones
+
+Given a sample.yml file of:
+```yaml
+foo: &foo
+  a: foo_a
+  thing: foo_thing
+  c: foo_c
+bar: &bar
+  b: bar_b
+  thing: bar_thing
+  c: bar_c
+foobarList:
+  b: foobarList_b
+  !!merge <<:
+    - *foo
+    - *bar
+  c: foobarList_c
+foobar:
+  c: foobar_c
+  !!merge <<: *foo
+  thing: foobar_thing
+```
+then
+```bash
+yq '.foobarList.thing' sample.yml
+```
+will output
+```yaml
+foo_thing
+```
+
+## FIXED: Splatting merge anchors
+Set `--yaml-fix-merge-anchor-to-spec=true` to get this correct merge behaviour. Note that the keys earlier in the merge anchors sequence override later ones
+
+Given a sample.yml file of:
+```yaml
+foo: &foo
+  a: foo_a
+  thing: foo_thing
+  c: foo_c
+bar: &bar
+  b: bar_b
+  thing: bar_thing
+  c: bar_c
+foobarList:
+  b: foobarList_b
+  !!merge <<:
+    - *foo
+    - *bar
+  c: foobarList_c
+foobar:
+  c: foobar_c
+  !!merge <<: *foo
+  thing: foobar_thing
+```
+then
+```bash
+yq '.foobar[]' sample.yml
+```
+will output
+```yaml
+foo_a
+foobar_thing
+foobar_c
+```
+
+## FIXED: Splatting merge anchor lists
+Set `--yaml-fix-merge-anchor-to-spec=true` to get this correct merge behaviour. Note that the keys earlier in the merge anchors sequence override later ones
+
+Given a sample.yml file of:
+```yaml
+foo: &foo
+  a: foo_a
+  thing: foo_thing
+  c: foo_c
+bar: &bar
+  b: bar_b
+  thing: bar_thing
+  c: bar_c
+foobarList:
+  b: foobarList_b
+  !!merge <<:
+    - *foo
+    - *bar
+  c: foobarList_c
+foobar:
+  c: foobar_c
+  !!merge <<: *foo
+  thing: foobar_thing
+```
+then
+```bash
+yq '.foobarList[]' sample.yml
+```
+will output
+```yaml
+foobarList_b
+foo_thing
+foobarList_c
+foo_a
 ```