From 2dbcb51b821134c33f69fec0c507ea6ce9474e42 Mon Sep 17 00:00:00 2001 From: Mike Farah Date: Sun, 27 Feb 2022 12:10:01 +1100 Subject: [PATCH] v4.21.1 --- SUMMARY.md | 5 +- operators/column.md | 66 +++++++++++++++++++++ operators/encode-decode.md | 66 +++++++++++++++++++++ operators/line.md | 68 +++++++++++++++++++++ operators/load.md | 108 ++++++++++++++++++++++++++++++++-- operators/reverse.md | 48 +++++++++++++++ operators/sort.md | 23 ++++++++ operators/string-operators.md | 68 +++++++++++++++++++-- 8 files changed, 441 insertions(+), 11 deletions(-) create mode 100644 operators/column.md create mode 100644 operators/line.md create mode 100644 operators/reverse.md diff --git a/SUMMARY.md b/SUMMARY.md index 6b5e758b..d78f4dd3 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -19,6 +19,7 @@ * [Assign (Update)](operators/assign-update.md) * [Boolean Operators](operators/boolean-operators.md) * [Collect into Array](operators/collect-into-array.md) + * [Column](operators/column.md) * [Comment Operators](operators/comment-operators.md) * [Contains](operators/contains.md) * [Create, Collect into Object](operators/create-collect-into-object.md) @@ -36,6 +37,7 @@ * [Has](operators/has.md) * [Keys](operators/keys.md) * [Length](operators/length.md) + * [Line](operators/line.md) * [Load](operators/load.md) * [Map](operators/map.md) * [Multiply (Merge)](operators/multiply-merge.md) @@ -44,6 +46,7 @@ * [Pipe](operators/pipe.md) * [Recursive Descent (Glob)](operators/recursive-descent-glob.md) * [Reduce](operators/reduce.md) + * [Reverse](operators/reverse.md) * [Select](operators/select.md) * [Sort](operators/sort.md) * [Sort Keys](operators/sort-keys.md) @@ -61,9 +64,9 @@ ## Usage * [Output format](usage/output-format.md) -* [Working with Properties](usage/properties.md) * [Working with CSV, TSV](usage/csv-tsv.md) * [Working with JSON](usage/convert.md) +* [Working with Properties](usage/properties.md) * [Working with XML](usage/xml.md) * [Front Matter](usage/front-matter.md) * [Split into multiple files](usage/split-into-multiple-files.md) diff --git a/operators/column.md b/operators/column.md new file mode 100644 index 00000000..3902933a --- /dev/null +++ b/operators/column.md @@ -0,0 +1,66 @@ +# Column + +Returns the column of the matching node. Starts from 1, 0 indicates there was no column data. + +{% hint style="warning" %} +Note that versions prior to 4.18 require the 'eval/e' command to be specified. + +`yq e ` +{% endhint %} + +## Returns column of _value_ node +Given a sample.yml file of: +```yaml +a: cat +b: bob +``` +then +```bash +yq '.b | column' sample.yml +``` +will output +```yaml +4 +``` + +## Returns column of _key_ node +Pipe through the key operator to get the column of the key + +Given a sample.yml file of: +```yaml +a: cat +b: bob +``` +then +```bash +yq '.b | key | column' sample.yml +``` +will output +```yaml +1 +``` + +## First column is 1 +Given a sample.yml file of: +```yaml +a: cat +``` +then +```bash +yq '.a | key | column' sample.yml +``` +will output +```yaml +1 +``` + +## No column data is 0 +Running +```bash +yq --null-input '{"a": "new entry"} | column' +``` +will output +```yaml +0 +``` + diff --git a/operators/encode-decode.md b/operators/encode-decode.md index 191a1c56..cf73bcd3 100644 --- a/operators/encode-decode.md +++ b/operators/encode-decode.md @@ -15,6 +15,7 @@ These operators are useful to process yaml documents that have stringified embed | CSV | | to_csv/@csv | | TSV | | to_tsv/@tsv | | XML | from_xml | to_xml(i)/@xml | +| Base64 | @base64d | @base64 | CSV and TSV format both accept either a single array or scalars (representing a single row), or an array of array of scalars (representing multiple rows). @@ -22,6 +23,8 @@ CSV and TSV format both accept either a single array or scalars (representing a XML uses the `--xml-attribute-prefix` and `xml-content-name` flags to identify attributes and content fields. +Base64 assumes [rfc4648](https://rfc-editor.org/rfc/rfc4648.html) encoding. Encoding and decoding both assume that the content is a string. + {% hint style="warning" %} Note that versions prior to 4.18 require the 'eval/e' command to be specified. @@ -354,3 +357,66 @@ b: foo: bar ``` +## Encode a string to base64 +Given a sample.yml file of: +```yaml +coolData: a special string +``` +then +```bash +yq '.coolData | @base64' sample.yml +``` +will output +```yaml +YSBzcGVjaWFsIHN0cmluZw== +``` + +## Encode a yaml document to base64 +Pipe through @yaml first to convert to a string, then use @base64 to encode it. + +Given a sample.yml file of: +```yaml +a: apple +``` +then +```bash +yq '@yaml | @base64' sample.yml +``` +will output +```yaml +YTogYXBwbGUK +``` + +## Decode a base64 encoded string +Decoded data is assumed to be a string. + +Given a sample.yml file of: +```yaml +coolData: V29ya3Mgd2l0aCBVVEYtMTYg8J+Yig== +``` +then +```bash +yq '.coolData | @base64d' sample.yml +``` +will output +```yaml +Works with UTF-16 😊 +``` + +## Decode a base64 encoded yaml document +Pipe through `from_yaml` to parse the decoded base64 string as a yaml document. + +Given a sample.yml file of: +```yaml +coolData: YTogYXBwbGUK +``` +then +```bash +yq '.coolData |= (@base64d | from_yaml)' sample.yml +``` +will output +```yaml +coolData: + a: apple +``` + diff --git a/operators/line.md b/operators/line.md new file mode 100644 index 00000000..dcbf6b6e --- /dev/null +++ b/operators/line.md @@ -0,0 +1,68 @@ +# Line + +Returns the line of the matching node. Starts from 1, 0 indicates there was no line data. + +{% hint style="warning" %} +Note that versions prior to 4.18 require the 'eval/e' command to be specified. + +`yq e ` +{% endhint %} + +## Returns line of _value_ node +Given a sample.yml file of: +```yaml +a: cat +b: + c: cat +``` +then +```bash +yq '.b | line' sample.yml +``` +will output +```yaml +3 +``` + +## Returns line of _key_ node +Pipe through the key operator to get the line of the key + +Given a sample.yml file of: +```yaml +a: cat +b: + c: cat +``` +then +```bash +yq '.b | key| line' sample.yml +``` +will output +```yaml +2 +``` + +## First line is 1 +Given a sample.yml file of: +```yaml +a: cat +``` +then +```bash +yq '.a | line' sample.yml +``` +will output +```yaml +1 +``` + +## No line data is 0 +Running +```bash +yq --null-input '{"a": "new entry"} | line' +``` +will output +```yaml +0 +``` + diff --git a/operators/load.md b/operators/load.md index 8f37566a..0b2afb73 100644 --- a/operators/load.md +++ b/operators/load.md @@ -1,18 +1,50 @@ # Load -The `load`/`strload` operator allows you to load in content from another file referenced in your yaml document. +The load operators allows you to load in content from another file. Note that you can use string operators like `+` and `sub` to modify the value in the yaml file to a path that exists in your system. -Use `strload` to load text based content as a string block, and `load` to interpret the file as yaml. +You can load files of the following supported types: -Lets say there is a file `../../examples/thing.yml`: +|Format | Load Operator | +| --- | --- | +| Yaml | load | +| XML | load_xml | +| Properties | load_props | +| Plain String | load_str | +| Base64 | load_base64 | + +## Samples files for tests: + +### yaml + +`../../examples/thing.yml`: ```yaml a: apple is included b: cool ``` +### xml +`small.xml`: + +```xml +is some xml +``` + +### properties +`small.properties`: + +```properties +this.is = a properties file +``` + +### base64 +`base64.txt`: +``` +bXkgc2VjcmV0IGNoaWxsaSByZWNpcGUgaXMuLi4u +``` + {% hint style="warning" %} Note that versions prior to 4.18 require the 'eval/e' command to be specified. @@ -89,7 +121,7 @@ something: ``` then ```bash -yq '.something |= strload("../../examples/" + .file)' sample.yml +yq '.something |= load_str("../../examples/" + .file)' sample.yml ``` will output ```yaml @@ -98,3 +130,71 @@ something: |- b: cool. ``` +## Load from XML +Given a sample.yml file of: +```yaml +cool: things +``` +then +```bash +yq '.more_stuff = load_xml("../../examples/small.xml")' sample.yml +``` +will output +```yaml +cool: things +more_stuff: + this: is some xml +``` + +## Load from Properties +Given a sample.yml file of: +```yaml +cool: things +``` +then +```bash +yq '.more_stuff = load_props("../../examples/small.properties")' sample.yml +``` +will output +```yaml +cool: things +more_stuff: + this: + is: a properties file +``` + +## Merge from properties +This can be used as a convenient way to update a yaml document + +Given a sample.yml file of: +```yaml +this: + is: from yaml + cool: ay +``` +then +```bash +yq '. *= load_props("../../examples/small.properties")' sample.yml +``` +will output +```yaml +this: + is: a properties file + cool: ay +``` + +## Load from base64 encoded file +Given a sample.yml file of: +```yaml +cool: things +``` +then +```bash +yq '.more_stuff = load_base64("../../examples/base64.txt")' sample.yml +``` +will output +```yaml +cool: things +more_stuff: my secret chilli recipe is.... +``` + diff --git a/operators/reverse.md b/operators/reverse.md new file mode 100644 index 00000000..eb9cafc3 --- /dev/null +++ b/operators/reverse.md @@ -0,0 +1,48 @@ +# Reverse + +Reverses the order of the items in an array + +{% hint style="warning" %} +Note that versions prior to 4.18 require the 'eval/e' command to be specified. + +`yq e ` +{% endhint %} + +## Reverse +Given a sample.yml file of: +```yaml +- 1 +- 2 +- 3 +``` +then +```bash +yq 'reverse' sample.yml +``` +will output +```yaml +- 3 +- 2 +- 1 +``` + +## Sort descending by string field +Use sort with reverse to sort in descending order. + +Given a sample.yml file of: +```yaml +- a: banana +- a: cat +- a: apple +``` +then +```bash +yq 'sort_by(.a) | reverse' sample.yml +``` +will output +```yaml +- a: cat +- a: banana +- a: apple +``` + diff --git a/operators/sort.md b/operators/sort.md index 9885d092..b7b810d3 100644 --- a/operators/sort.md +++ b/operators/sort.md @@ -2,8 +2,11 @@ Sorts an array. Use `sort` to sort an array as is, or `sort_by(exp)` to sort by a particular expression (e.g. subfield). +To sort by descending order, pipe the results through the `reverse` operator after sorting. + Note that at this stage, `yq` only sorts scalar fields. + {% hint style="warning" %} Note that versions prior to 4.18 require the 'eval/e' command to be specified. @@ -28,6 +31,26 @@ will output - a: cat ``` +## Sort descending by string field +Use sort with reverse to sort in descending order. + +Given a sample.yml file of: +```yaml +- a: banana +- a: cat +- a: apple +``` +then +```bash +yq 'sort_by(.a) | reverse' sample.yml +``` +will output +```yaml +- a: cat +- a: banana +- a: apple +``` + ## Sort array in place Given a sample.yml file of: ```yaml diff --git a/operators/string-operators.md b/operators/string-operators.md index a13288fa..47c3e320 100644 --- a/operators/string-operators.md +++ b/operators/string-operators.md @@ -49,6 +49,38 @@ Note that versions prior to 4.18 require the 'eval/e' command to be specified.&# `yq e ` {% endhint %} +## To up (upper) case +Works with unicode characters + +Given a sample.yml file of: +```yaml +água +``` +then +```bash +yq 'upcase' sample.yml +``` +will output +```yaml +ÁGUA +``` + +## To down (lower) case +Works with unicode characters + +Given a sample.yml file of: +```yaml +ÁgUA +``` +then +```bash +yq 'downcase' sample.yml +``` +will output +```yaml +água +``` + ## Join strings Given a sample.yml file of: ```yaml @@ -105,14 +137,14 @@ will output captures: [] ``` -## Match with capture groups +## Match with global capture group Given a sample.yml file of: ```yaml abc abc ``` then ```bash -yq '[match("(abc)+"; "g")]' sample.yml +yq '[match("(ab)(c)"; "g")]' sample.yml ``` will output ```yaml @@ -120,16 +152,22 @@ will output offset: 0 length: 3 captures: - - string: abc + - string: ab offset: 0 - length: 3 + length: 2 + - string: c + offset: 2 + length: 1 - string: abc offset: 4 length: 3 captures: - - string: abc + - string: ab offset: 4 - length: 3 + length: 2 + - string: c + offset: 6 + length: 1 ``` ## Match with named capture groups @@ -268,6 +306,24 @@ a: cart b: heart ``` +## Custom types: that are really strings +When custom tags are encountered, yq will try to decode the underlying type. + +Given a sample.yml file of: +```yaml +a: !horse cat +b: !goat heat +``` +then +```bash +yq '.[] |= sub("(a)", "${1}r")' sample.yml +``` +will output +```yaml +a: !horse cart +b: !goat heart +``` + ## Split strings Given a sample.yml file of: ```yaml