Merging from master

pkg/yqlib/doc/operators/group-by.md

@@ -26,7 +26,7 @@ will output
     bar: 100
 ```
 
-## Group by field, with nuls
+## Group by field, with nulls
 Given a sample.yml file of:
 ```yaml
 - cat: dog

pkg/yqlib/doc/operators/headers/Main.md

@@ -26,7 +26,7 @@ yq '.a.b[0].c' file.yaml
 cat file.yaml | yq '.a.b[0].c'
 ```
 
-## Update a yaml file, inplace
+## Update a yaml file, in place
 ```bash
 yq -i '.a.b[0].c = "cool"' file.yaml
 ```

pkg/yqlib/doc/operators/headers/to_number.md

@@ -0,0 +1,2 @@
+# To Number
+Parses the input as a number. yq will try to parse values as an int first, failing that it will try float. Values that already ints or floats will be left alone.

pkg/yqlib/doc/operators/kind.md

@@ -0,0 +1,77 @@
+
+## Get kind
+Given a sample.yml file of:
+```yaml
+a: cat
+b: 5
+c: 3.2
+e: true
+f: []
+g: {}
+h: null
+```
+then
+```bash
+yq '.. | kind' sample.yml
+```
+will output
+```yaml
+map
+scalar
+scalar
+scalar
+scalar
+seq
+map
+scalar
+```
+
+## Get kind, ignores custom tags
+Unlike tag, kind is not affected by custom tags.
+
+Given a sample.yml file of:
+```yaml
+a: !!thing cat
+b: !!foo {}
+c: !!bar []
+```
+then
+```bash
+yq '.. | kind' sample.yml
+```
+will output
+```yaml
+map
+scalar
+map
+seq
+```
+
+## Add comments only to scalars
+An example of how you can use kind
+
+Given a sample.yml file of:
+```yaml
+a:
+  b: 5
+  c: 3.2
+e: true
+f: []
+g: {}
+h: null
+```
+then
+```bash
+yq '(.. | select(kind == "scalar")) line_comment = "this is a scalar"' sample.yml
+```
+will output
+```yaml
+a:
+  b: 5 # this is a scalar
+  c: 3.2 # this is a scalar
+e: true # this is a scalar
+f: []
+g: {}
+h: null # this is a scalar
+```
+

pkg/yqlib/doc/operators/pick.md

@@ -13,7 +13,7 @@ myMap:
   cat: meow
   dog: bark
   thing: hamster
-  hamster: squeek
+  hamster: squeak
 ```
 then
 ```bash
@@ -22,7 +22,7 @@ yq '.myMap |= pick(["hamster", "cat", "goat"])' sample.yml
 will output
 ```yaml
 myMap:
-  hamster: squeek
+  hamster: squeak
   cat: meow
 ```
 

pkg/yqlib/doc/operators/to_number.md

@@ -0,0 +1,49 @@
+# To Number
+Parses the input as a number. yq will try to parse values as an int first, failing that it will try float. Values that already ints or floats will be left alone.
+
+## Converts strings to numbers
+Given a sample.yml file of:
+```yaml
+- "3"
+- "3.1"
+- "-1e3"
+```
+then
+```bash
+yq '.[] | to_number' sample.yml
+```
+will output
+```yaml
+3
+3.1
+-1e3
+```
+
+## Doesn't change numbers
+Given a sample.yml file of:
+```yaml
+- 3
+- 3.1
+- -1e3
+```
+then
+```bash
+yq '.[] | to_number' sample.yml
+```
+will output
+```yaml
+3
+3.1
+-1e3
+```
+
+## Cannot convert null
+Running
+```bash
+yq --null-input '.a.b | to_number'
+```
+will output
+```bash
+Error: cannot convert node value [null] at path a.b of tag !!null to number
+```
+

pkg/yqlib/doc/usage/csv-tsv.md

@@ -2,7 +2,7 @@
 Encode/Decode/Roundtrip CSV and TSV files.
 
 ## Encode 
-Currently supports arrays of homogenous flat objects, that is: no nesting and it assumes the _first_ object has all the keys required:
+Currently supports arrays of homogeneous flat objects, that is: no nesting and it assumes the _first_ object has all the keys required:
 
 ```yaml
 - name: Bobo

pkg/yqlib/doc/usage/headers/csv-tsv.md

@@ -2,7 +2,7 @@
 Encode/Decode/Roundtrip CSV and TSV files.
 
 ## Encode 
-Currently supports arrays of homogenous flat objects, that is: no nesting and it assumes the _first_ object has all the keys required:
+Currently supports arrays of homogeneous flat objects, that is: no nesting and it assumes the _first_ object has all the keys required:
 
 ```yaml
 - name: Bobo

pkg/yqlib/doc/usage/headers/recipes.md

@@ -0,0 +1,5 @@
+# Recipes
+
+These examples are intended to show how you can use multiple operators together so you get an idea of how you can perform complex data manipulation.
+
+Please see the details [operator docs](https://mikefarah.gitbook.io/yq/operators) for details on each individual operator.

pkg/yqlib/doc/usage/lua.md

@@ -0,0 +1 @@
+

pkg/yqlib/doc/usage/recipes.md

@@ -0,0 +1,155 @@
+# Recipes
+
+These examples are intended to show how you can use multiple operators together so you get an idea of how you can perform complex data manipulation.
+
+Please see the details [operator docs](https://mikefarah.gitbook.io/yq/operators) for details on each individual operator.
+
+## Find items in an array
+We have an array and we want to find the elements with a particular name.
+
+Given a sample.yml file of:
+```yaml
+- name: Foo
+  numBuckets: 0
+- name: Bar
+  numBuckets: 0
+```
+then
+```bash
+yq '.[] | select(.name == "Foo")' sample.yml
+```
+will output
+```yaml
+name: Foo
+numBuckets: 0
+```
+
+### Explanation:
+- `.[]` splats the array, and puts all the items in the context.
+- These items are then piped (`|`) into `select(.name == "Foo")` which will select all the nodes that have a name property set to 'Foo'.
+- See the [select](https://mikefarah.gitbook.io/yq/operators/select) operator for more information.
+
+## Find and update items in an array
+We have an array and we want to _update_ the elements with a particular name.
+
+Given a sample.yml file of:
+```yaml
+- name: Foo
+  numBuckets: 0
+- name: Bar
+  numBuckets: 0
+```
+then
+```bash
+yq '(.[] | select(.name == "Foo") | .numBuckets) |= . + 1' sample.yml
+```
+will output
+```yaml
+- name: Foo
+  numBuckets: 1
+- name: Bar
+  numBuckets: 0
+```
+
+### Explanation:
+- Following from the example above`.[]` splats the array, selects filters the items.
+- We then pipe (`|`) that into `.numBuckets`, which will select that field from all the matching items
+- Splat, select and the field are all in brackets, that whole expression is passed to the `|=` operator as the left hand side expression, with `. + 1` as the right hand side expression.
+- `|=` is the operator that updates fields relative to their own value, which is referenced as dot (`.`).
+- The expression `. + 1` increments the numBuckets counter.
+- See the [assign](https://mikefarah.gitbook.io/yq/operators/assign-update) and [add](https://mikefarah.gitbook.io/yq/operators/add) operators for more information.
+
+## Multiple or complex updates to items in an array
+We have an array and we want to _update_ the elements with a particular name in reference to its type.
+
+Given a sample.yml file of:
+```yaml
+myArray:
+  - name: Foo
+    type: cat
+  - name: Bar
+    type: dog
+```
+then
+```bash
+yq 'with(.myArray[]; .name = .name + " - " + .type)' sample.yml
+```
+will output
+```yaml
+myArray:
+  - name: Foo - cat
+    type: cat
+  - name: Bar - dog
+    type: dog
+```
+
+### Explanation:
+- The with operator will effectively loop through each given item in the first given expression, and run the second expression against it.
+- `.myArray[]` splats the array in `myArray`. So `with` will run against each item in that array
+- `.name = .name + " - " + .type` this expression is run against every item, updating the name to be a concatenation of the original name as well as the type.
+- See the [with](https://mikefarah.gitbook.io/yq/operators/with) operator for more information and examples.
+
+## Sort an array by a field
+Given a sample.yml file of:
+```yaml
+myArray:
+  - name: Foo
+    numBuckets: 1
+  - name: Bar
+    numBuckets: 0
+```
+then
+```bash
+yq '.myArray |= sort_by(.numBuckets)' sample.yml
+```
+will output
+```yaml
+myArray:
+  - name: Bar
+    numBuckets: 0
+  - name: Foo
+    numBuckets: 1
+```
+
+### Explanation:
+- We want to resort `.myArray`.
+- `sort_by` works by piping an array into it, and it pipes out a sorted array.
+- So, we use `|=` to update `.myArray`. This is the same as doing `.myArray = (.myArray | sort_by(.numBuckets))`
+
+## Filter, flatten, sort and unique
+Lets
+
+Given a sample.yml file of:
+```yaml
+- type: foo
+  names:
+    - Fred
+    - Catherine
+- type: bar
+  names:
+    - Zelda
+- type: foo
+  names: Fred
+- type: foo
+  names: Ava
+```
+then
+```bash
+yq '[.[] | select(.type == "foo") | .names] | flatten | sort | unique' sample.yml
+```
+will output
+```yaml
+- Ava
+- Catherine
+- Fred
+```
+
+### Explanation:
+- `.[] | select(.type == "foo") | .names` will select the array elements of type "foo"
+- Splat `.[]` will unwrap the array and match all the items. We need to do this so we can work on the child items, for instance, filter items out using the `select` operator.
+- But we still want the final results back into an array. So after we're doing working on the children, we wrap everything back into an array using square brackets around the expression. `[.[] | select(.type == "foo") | .names]`
+- Now have have an array of all the 'names' values. Which includes arrays of strings as well as strings on their own.
+- Pipe `|` this array through `flatten`. This will flatten nested arrays. So now we have a flat list of all the name value strings
+- Next we pipe `|` that through `sort` and then `unique` to get a sorted, unique list of the names!
+- See the [flatten](https://mikefarah.gitbook.io/yq/operators/flatten), [sort](https://mikefarah.gitbook.io/yq/operators/sort) and [unique](https://mikefarah.gitbook.io/yq/operators/unique) for more information and examples.
+

pkg/yqlib/doc/usage/shellvariables.md

@@ -34,7 +34,7 @@ Given a sample.yml file of:
 ascii_=_symbols: replaced with _
 "ascii_	_controls": dropped (this example uses \t)
 nonascii_א_characters: dropped
-effrot_expeñded_tò_preserve_accented_latin_letters: moderate (via unicode NFKD)
+effort_expeñded_tò_preserve_accented_latin_letters: moderate (via unicode NFKD)
 
 ```
 then
@@ -46,7 +46,7 @@ will output
 ascii___symbols='replaced with _'
 ascii__controls='dropped (this example uses \t)'
 nonascii__characters=dropped
-effrot_expended_to_preserve_accented_latin_letters='moderate (via unicode NFKD)'
+effort_expended_to_preserve_accented_latin_letters='moderate (via unicode NFKD)'
 ```
 
 ## Encode shell variables: empty values, arrays and maps

pkg/yqlib/doc/usage/xml.md

@@ -53,7 +53,7 @@ Given a sample.xml file of:
 ```
 then
 ```bash
-yq -p=xml '.' sample.xml
+yq -oy '.' sample.xml
 ```
 will output
 ```yaml
@@ -78,7 +78,7 @@ Given a sample.xml file of:
 ```
 then
 ```bash
-yq -p=xml ' (.. | select(tag == "!!str")) |= from_yaml' sample.xml
+yq -oy ' (.. | select(tag == "!!str")) |= from_yaml' sample.xml
 ```
 will output
 ```yaml
@@ -100,7 +100,7 @@ Given a sample.xml file of:
 ```
 then
 ```bash
-yq -p=xml '.' sample.xml
+yq -oy '.' sample.xml
 ```
 will output
 ```yaml
@@ -110,6 +110,42 @@ animal:
   - goat
 ```
 
+## Parse xml: force as an array
+In XML, if your array has a single item, then yq doesn't know its an array. This is how you can consistently force it to be an array. This handles the 3 scenarios of having nothing in the array, having a single item and having multiple.
+
+Given a sample.xml file of:
+```xml
+<zoo><animal>cat</animal></zoo>
+```
+then
+```bash
+yq -oy '.zoo.animal |= ([] + .)' sample.xml
+```
+will output
+```yaml
+zoo:
+  animal:
+    - cat
+```
+
+## Parse xml: force all as an array
+Because of the way yq works, when updating everything you need to update the children before the parents. By default `..` will match parents first, so we reverse that before updating.
+
+Given a sample.xml file of:
+```xml
+<zoo><thing><frog>boing</frog></thing></zoo>
+```
+then
+```bash
+yq -oy '([..] | reverse | .[]) |= [] + .' sample.xml
+```
+will output
+```yaml
+zoo:
+  thing:
+    frog: boing
+```
+
 ## Parse xml: attributes
 Attributes are converted to fields, with the default attribute prefix '+'. Use '--xml-attribute-prefix` to set your own.
 
@@ -122,7 +158,7 @@ Given a sample.xml file of:
 ```
 then
 ```bash
-yq -p=xml '.' sample.xml
+yq -oy '.' sample.xml
 ```
 will output
 ```yaml
@@ -142,7 +178,7 @@ Given a sample.xml file of:
 ```
 then
 ```bash
-yq -p=xml '.' sample.xml
+yq -oy '.' sample.xml
 ```
 will output
 ```yaml
@@ -161,7 +197,7 @@ Given a sample.xml file of:
 ```
 then
 ```bash
-yq -p=xml '.' sample.xml
+yq -oy '.' sample.xml
 ```
 will output
 ```yaml
@@ -190,7 +226,7 @@ Given a sample.xml file of:
 ```
 then
 ```bash
-yq -p=xml -o=xml '.' sample.xml
+yq '.' sample.xml
 ```
 will output
 ```xml
@@ -221,7 +257,7 @@ Given a sample.xml file of:
 ```
 then
 ```bash
-yq -p=xml -o=xml --xml-skip-directives '.' sample.xml
+yq --xml-skip-directives '.' sample.xml
 ```
 will output
 ```xml
@@ -257,7 +293,7 @@ for x --></x>
 ```
 then
 ```bash
-yq -p=xml '.' sample.xml
+yq -oy '.' sample.xml
 ```
 will output
 ```yaml
@@ -289,7 +325,7 @@ Given a sample.xml file of:
 ```
 then
 ```bash
-yq -p=xml -o=xml --xml-keep-namespace=false '.' sample.xml
+yq --xml-keep-namespace=false '.' sample.xml
 ```
 will output
 ```xml
@@ -314,7 +350,7 @@ Given a sample.xml file of:
 ```
 then
 ```bash
-yq -p=xml -o=xml --xml-raw-token=false '.' sample.xml
+yq --xml-raw-token=false '.' sample.xml
 ```
 will output
 ```xml
@@ -489,7 +525,7 @@ for x --></x>
 ```
 then
 ```bash
-yq -p=xml -o=xml '.' sample.xml
+yq '.' sample.xml
 ```
 will output
 ```xml
@@ -522,7 +558,7 @@ Given a sample.xml file of:
 ```
 then
 ```bash
-yq -p=xml -o=xml '.' sample.xml
+yq '.' sample.xml
 ```
 will output
 ```xml