refactoring doc gen

pkg/yqlib/doc/operators/headers/Main.md

@@ -0,0 +1,62 @@
+# NAME
+  *yq* is a portable command-line YAML processor
+
+# SYNOPSIS 
+
+yq [eval/eval-all] [expression] files..
+
+eval/e  - Apply the expression to each document in each yaml file in sequence
+
+eval-all/ea - Loads all yaml documents of all yaml files and runs expression once
+
+# DESCRIPTION
+
+a lightweight and portable command-line YAML processor. `yq` uses [jq](https://github.com/stedolan/jq) like syntax but works with yaml files as well as json. It doesn't yet support everything `jq` does - but it does support the most common operations and functions, and more is being added continuously.
+
+This documentation is also available at https://mikefarah.gitbook.io/yq/
+# QUICK GUIDE 
+
+## Read a value:
+```bash
+yq e '.a.b[0].c' file.yaml
+```
+
+## Pipe from STDIN:
+```bash
+cat file.yaml | yq e '.a.b[0].c' -
+```
+
+## Update a yaml file, inplace
+```bash
+yq e -i '.a.b[0].c = "cool"' file.yaml
+```
+
+## Update using environment variables
+```bash
+NAME=mike yq e -i '.a.b[0].c = strenv(NAME)' file.yaml
+```
+
+## Merge multiple files
+```
+yq ea '. as $item ireduce ({}; . * $item )' path/to/*.yml
+```
+
+## Multiple updates to a yaml file
+```bash
+yq e -i '
+  .a.b[0].c = "cool" |
+  .x.y.z = "foobar" |
+  .person.name = strenv(NAME)
+' file.yaml
+```
+
+See the [documentation](https://mikefarah.gitbook.io/yq/) for more.
+
+# KNOWN ISSUES / MISSING FEATURES
+- `yq` attempts to preserve comment positions and whitespace as much as possible, but it does not handle all scenarios (see https://github.com/go-yaml/yaml/tree/v3 for details)
+- Powershell has its own...opinions: https://mikefarah.gitbook.io/yq/usage/tips-and-tricks#quotes-in-windows-powershell
+
+# BUGS / ISSUES / FEATURE REQUESTS
+
+Please visit the GitHub page https://github.com/mikefarah/yq/.
+

pkg/yqlib/doc/operators/headers/add.md

@@ -0,0 +1,8 @@
+# Add
+
+Add behaves differently according to the type of the LHS:
+* arrays: concatenate
+* number scalars: arithmetic addition
+* string scalars: concatenate
+
+Use `+=` as append assign for things like increment. Note that `.a += .x` is equivalent to running `.a = .a + .x`.

pkg/yqlib/doc/operators/headers/alternative-default-value.md

@@ -0,0 +1,3 @@
+# Alternative (Default value)
+
+This operator is used to provide alternative (or default) values when a particular expression is either null or false.

pkg/yqlib/doc/operators/headers/anchor-and-alias-operators.md

@@ -0,0 +1,6 @@
+# Anchor and Alias Operators
+
+Use the `alias` and `anchor` operators to read and write yaml aliases and anchors. The `explode` operator normalises a yaml file (dereference (or expands) aliases and remove anchor names).
+
+`yq` supports merge aliases (like `<<: *blah`) however this is no longer in the standard yaml spec (1.2) and so `yq` will automatically add the `!!merge` tag to these nodes as it is effectively a custom tag.
+

pkg/yqlib/doc/operators/headers/assign-update.md

@@ -0,0 +1,9 @@
+# Assign (Update)
+
+This operator is used to update node values. It can be used in either the:
+
+### plain form: `=`
+Which will assign the LHS node values to the RHS node values. The RHS expression is run against the matching nodes in the pipeline.
+
+### relative form: `|=`
+This will do a similar thing to the plain form, however, the RHS expression is run against _the LHS nodes_. This is useful for updating values based on old values, e.g. increment.

pkg/yqlib/doc/operators/headers/boolean-operators.md

@@ -0,0 +1,11 @@
+# Boolean Operators
+
+The `or` and `and` operators take two parameters and return a boolean result. 
+
+`not` flips a boolean from true to false, or vice versa. 
+
+`any` will return `true` if there are any `true` values in a array sequence, and `all` will return true if _all_ elements in an array are true.
+
+`any_c(condition)` and `all_c(condition)` are like `any` and `all` but they take a condition expression that is used against each element to determine if it's `true`. Note: in `jq` you can simply pass a condition to `any` or `all` and it simply works - `yq` isn't that clever..yet
+
+These are most commonly used with the `select` operator to filter particular nodes.

pkg/yqlib/doc/operators/headers/collect-into-array.md

@@ -0,0 +1,4 @@
+# Collect into Array
+
+This creates an array using the expression between the square brackets.
+

pkg/yqlib/doc/operators/headers/comment-operators.md

@@ -0,0 +1,11 @@
+# Comment Operators
+
+Use these comment operators to set or retrieve comments.
+
+Like the `=` and `|=` assign operators, the same syntax applies when updating comments:
+
+### plain form: `=`
+This will assign the LHS nodes comments to the expression on the RHS. The RHS is run against the matching nodes in the pipeline
+
+### relative form: `|=` 
+Similar to the plain form, however the RHS evaluates against each matching LHS node! This is useful if you want to set the comments as a relative expression of the node, for instance its value or path.

pkg/yqlib/doc/operators/headers/contains.md

@@ -0,0 +1,3 @@
+# Contains
+
+This returns `true` if the context contains the passed in parameter, and false otherwise.

pkg/yqlib/doc/operators/headers/create-collect-into-object.md

@@ -0,0 +1,3 @@
+# Create, Collect into Object
+
+This is used to construct objects (or maps). This can be used against existing yaml, or to create fresh yaml documents.

pkg/yqlib/doc/operators/headers/delete.md

@@ -0,0 +1,3 @@
+# Delete
+
+Deletes matching entries in maps or arrays.

pkg/yqlib/doc/operators/headers/document-index.md

@@ -0,0 +1,3 @@
+# Document Index
+
+Use the `documentIndex` operator (or the `di` shorthand) to select nodes of a particular document.

pkg/yqlib/doc/operators/headers/encode-decode.md

@@ -0,0 +1,20 @@
+# Encoder / Decoder
+
+Encode operators will take the piped in object structure and encode it as a string in the desired format. The decode operators do the opposite, they take a formatted string and decode it into the relevant object structure.
+
+Note that you can optionally pass an indent value to the encode functions (see below).
+
+These operators are useful to process yaml documents that have stringified embeded yaml/json/props in them.
+
+
+| Format | Decode (from string) | Encode (to string) |
+| --- | -- | --|
+| Yaml | from_yaml | to_yaml(i)/@yaml |
+| JSON | from_json | to_json(i)/@json |
+| Properties |  | to_props/@props |
+| CSV |  | to_csv/@csv |
+| TSV |  | to_tsv/@tsv |
+
+
+CSV and TSV format both accept either a single array or scalars (representing a single row), or an array of array of scalars (representing multiple rows). 
+

pkg/yqlib/doc/operators/headers/entries.md

@@ -0,0 +1,3 @@
+# Entries
+
+Similar to the same named functions in `jq` these functions convert to/from an object and an array of key-value pairs. This is most useful for performing operations on keys of maps.

pkg/yqlib/doc/operators/headers/env-variable-operators.md

@@ -0,0 +1,3 @@
+# Env Variable Operators
+
+This operator is used to handle environment variables usage in path expressions. While environment variables can, of course, be passed in via your CLI with string interpolation, this often comes with complex quote escaping and can be tricky to write and read. Note that there are two forms, `env` which will parse the environment variable as a yaml (be it a map, array, string, number of boolean) and `strenv` which will always parse the argument as a string.

pkg/yqlib/doc/operators/headers/equals.md

@@ -0,0 +1,13 @@
+# Equals
+
+This is a boolean operator that will return `true` if the LHS is equal to the RHS and `false` otherwise.
+
+```
+.a == .b
+```
+
+It is most often used with the select operator to find particular nodes:
+
+```
+select(.a == .b)
+```

pkg/yqlib/doc/operators/headers/file-operators.md

@@ -0,0 +1,11 @@
+# File Operators
+
+File operators are most often used with merge when needing to merge specific files together. Note that when doing this, you will need to use `eval-all` to ensure all yaml documents are loaded into memory before performing the merge (as opposed to `eval` which runs the expression once per document).
+
+Note that the `fileIndex` operator has a short alias of `fi`.
+
+## Merging files
+Note the use of eval-all to ensure all documents are loaded into memory.
+```bash
+yq eval-all 'select(fi == 0) * select(filename == "file2.yaml")' file1.yaml file2.yaml
+```

pkg/yqlib/doc/operators/headers/flatten.md

@@ -0,0 +1,2 @@
+# Flatten
+This recursively flattens arrays.

pkg/yqlib/doc/operators/headers/group-by.md

@@ -0,0 +1,3 @@
+# Group By
+
+This is used to group items in an array by an expression.

pkg/yqlib/doc/operators/headers/has.md

@@ -0,0 +1,3 @@
+# Has
+
+This is operation that returns true if the key exists in a map (or index in an array), false otherwise.

pkg/yqlib/doc/operators/headers/keys.md

@@ -0,0 +1,3 @@
+# Keys
+
+Use the `keys` operator to return map keys or array indices. 

pkg/yqlib/doc/operators/headers/length.md

@@ -0,0 +1,3 @@
+# Length
+
+Returns the lengths of the nodes. Length is defined according to the type of the node.

pkg/yqlib/doc/operators/headers/load.md

@@ -0,0 +1,14 @@
+# Load
+
+The `load`/`strload` operator allows you to load in content from another file referenced in your yaml document.
+
+Note that you can use string operators like `+` and `sub` to modify the value in the yaml file to a path that exists in your system.
+
+Use `strload` to load text based content as a string block, and `load` to interpret the file as yaml.
+
+Lets say there is a file `../../examples/thing.yml`:
+
+```yaml
+a: apple is included
+b: cool
+```

pkg/yqlib/doc/operators/headers/map.md

@@ -0,0 +1,3 @@
+# Map
+
+Maps values of an array. Use `map_values` to map values of an object.

pkg/yqlib/doc/operators/headers/multiply-merge.md

@@ -0,0 +1,22 @@
+# Multiply (Merge)
+
+Like the multiple operator in jq, depending on the operands, this multiply operator will do different things. Currently numbers, arrays and objects are supported.
+
+## Objects and arrays - merging
+Objects are merged deeply matching on matching keys. By default, array values override and are not deeply merged.
+
+Note that when merging objects, this operator returns the merged object (not the parent). This will be clearer in the examples below.
+
+### Merge Flags
+You can control how objects are merged by using one or more of the following flags. Multiple flags can be used together, e.g. `.a *+? .b`.  See examples below
+
+- `+` to append arrays
+- `?` to only merge existing fields
+- `d` to deeply merge arrays
+
+### Merging files
+Note the use of `eval-all` to ensure all documents are loaded into memory.
+
+```bash
+yq eval-all 'select(fileIndex == 0) * select(fileIndex == 1)' file1.yaml file2.yaml
+```

pkg/yqlib/doc/operators/headers/parent.md

@@ -0,0 +1,3 @@
+# Parent
+
+Parent simply returns the parent nodes of the matching nodes.

pkg/yqlib/doc/operators/headers/path.md

@@ -0,0 +1,5 @@
+# Path
+
+The path operator can be used to get the traversal paths of matching nodes in an expression. The path is returned as an array, which if traversed in order will lead to the matching node.
+
+You can get the key/index of matching nodes by using the `path` operator to return the path array then piping that through `.[-1]` to get the last element of that array, the key.

pkg/yqlib/doc/operators/headers/pipe.md

@@ -0,0 +1,3 @@
+# Pipe
+
+Pipe the results of an expression into another. Like the bash operator.

pkg/yqlib/doc/operators/headers/recursive-descent-glob.md

@@ -0,0 +1,21 @@
+# Recursive Descent (Glob)
+
+This operator recursively matches (or globs) all children nodes given of a particular element, including that node itself. This is most often used to apply a filter recursively against all matches. It can be used in either the 
+
+## match values form `..`
+This will, like the `jq` equivalent, recursively match all _value_ nodes. Use it to find/manipulate particular values.
+
+For instance to set the `style` of all _value_ nodes in a yaml doc, excluding map keys:
+
+```bash
+yq eval '.. style= "flow"' file.yaml
+```
+
+## match values and map keys form `...`
+The also includes map keys in the results set. This is particularly useful in YAML as unlike JSON, map keys can have their own styling, tags and use anchors and aliases.
+
+For instance to set the `style` of all nodes in a yaml doc, including the map keys:
+
+```bash
+yq eval '... style= "flow"' file.yaml
+```

pkg/yqlib/doc/operators/headers/reduce.md

@@ -0,0 +1,22 @@
+# Reduce
+
+Reduce is a powerful way to process a collection of data into a new form.
+
+```
+<exp> as $<name> ireduce (<init>; <block>)
+```
+
+e.g.
+
+```
+.[] as $item ireduce (0; . + $item)
+```
+
+On the LHS we are configuring the collection of items that will be reduced `<exp>` as well as what each element will be called `$<name>`. Note that the array has been splatted into its individual elements.
+
+On the RHS there is `<init>`, the starting value of the accumulator and `<block>`, the expression that will update the accumulator for each element in the collection. Note that within the block expression, `.` will evaluate to the current value of the accumulator. 
+
+## yq vs jq syntax
+Reduce syntax in `yq` is a little different from `jq` - as `yq` (currently) isn't as sophisticated as `jq` and its only supports infix notation (e.g. a + b, where the operator is in the middle of the two parameters) - where as `jq` uses a mix of infix notation with _prefix_ notation (e.g. `reduce a b` is like writing `+ a b`).
+
+To that end, the reduce operator is called `ireduce` for backwards compatability if a `jq` like prefix version of `reduce` is ever added.

pkg/yqlib/doc/operators/headers/select.md

@@ -0,0 +1,3 @@
+# Select
+
+Select is used to filter arrays and maps by a boolean expression.

pkg/yqlib/doc/operators/headers/sort-keys.md

@@ -0,0 +1,11 @@
+# Sort Keys
+
+The Sort Keys operator sorts maps by their keys (based on their string value). This operator does not do anything to arrays or scalars (so you can easily recursively apply it to all maps).
+
+Sort is particularly useful for diffing two different yaml documents:
+
+```bash
+yq eval -i -P 'sort_keys(..)' file1.yml
+yq eval -i -P 'sort_keys(..)' file2.yml
+diff file1.yml file2.yml
+```

pkg/yqlib/doc/operators/headers/sort.md

@@ -0,0 +1,5 @@
+# Sort
+
+Sorts an array. Use `sort` to sort an array as is, or `sort_by(exp)` to sort by a particular expression (e.g. subfield).
+
+Note that at this stage, `yq` only sorts scalar fields.

pkg/yqlib/doc/operators/headers/split-into-documents.md

@@ -0,0 +1,3 @@
+# Split into Documents
+
+This operator splits all matches into separate documents

pkg/yqlib/doc/operators/headers/string-operators.md

@@ -0,0 +1,44 @@
+# String Operators
+
+## RegEx
+This uses golangs native regex functions under the hood - See https://github.com/google/re2/wiki/Syntax for the supported syntax.
+
+
+## String blocks, bash and newlines
+Bash is notorious for chomping on precious trailing newline characters, making it tricky to set strings with newlines properly. In particular, the `$( exp )` _will trim trailing newlines_.
+
+For instance to get this yaml:
+
+```
+a: |
+  cat
+```
+
+Using `$( exp )` wont work, as it will trim the trailing new line.
+
+```
+m=$(echo "cat\n") yq e -n '.a = strenv(m)'
+a: cat
+```
+
+However, using printf works:
+```
+printf -v m "cat\n" ; m="$m" yq e -n '.a = strenv(m)'
+a: |
+  cat
+```
+
+As well as having multiline expressions:
+```
+m="cat
+"  yq e -n '.a = strenv(m)'
+a: |
+  cat
+```
+
+Similarly, if you're trying to set the content from a file, and want a trailing new line:
+
+```
+IFS= read -rd '' output < <(cat my_file)
+output=$output ./yq e '.data.values = strenv(output)' first.yml
+```

pkg/yqlib/doc/operators/headers/style.md

@@ -0,0 +1,3 @@
+# Style
+
+The style operator can be used to get or set the style of nodes (e.g. string style, yaml style)

pkg/yqlib/doc/operators/headers/subtract.md

@@ -0,0 +1,3 @@
+# Subtract
+
+You can use subtract to subtract numbers, as well as removing elements from an array.

pkg/yqlib/doc/operators/headers/tag.md

@@ -0,0 +1,3 @@
+# Tag
+
+The tag operator can be used to get or set the tag of nodes (e.g. `!!str`, `!!int`, `!!bool`).

pkg/yqlib/doc/operators/headers/traverse-read.md

@@ -0,0 +1,3 @@
+# Traverse (Read)
+
+This is the simplest (and perhaps most used) operator, it is used to navigate deeply into yaml structures.

pkg/yqlib/doc/operators/headers/union.md

@@ -0,0 +1,3 @@
+# Union
+
+This operator is used to combine different results together.

pkg/yqlib/doc/operators/headers/unique.md

@@ -0,0 +1,3 @@
+# Unique
+
+This is used to filter out duplicated items in an array.

pkg/yqlib/doc/operators/headers/variable-operators.md

@@ -0,0 +1,5 @@
+# Variable Operators
+
+Like the `jq` equivalents, variables are sometimes required for the more complex expressions (or swapping values between fields).
+
+Note that there is also an additional `ref` operator that holds a reference (instead of a copy) of the path, allowing you to make multiple changes to the same path.

pkg/yqlib/doc/operators/headers/with.md

@@ -0,0 +1,3 @@
+# With
+
+Use the `with` operator to conveniently make multiple updates to a deeply nested path, or to update array elements relatively to each other. The first argument expression sets the root context, and the second expression runs against that root context.