Bumping version

Bumps [github.com/elliotchance/orderedmap](https://github.com/elliotchance/orderedmap) from 1.4.0 to 1.5.0.
- [Release notes](https://github.com/elliotchance/orderedmap/releases)
- [Commits](https://github.com/elliotchance/orderedmap/compare/v1.4.0...v1.5.0)

---
updated-dependencies:
- dependency-name: github.com/elliotchance/orderedmap
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

.github/workflows/go.yml

@@ -10,10 +10,10 @@ jobs:
     runs-on: ubuntu-latest
     steps:
 
-    - name: Set up Go 1.18
+    - name: Set up Go 1.19
       uses: actions/setup-go@v3
       with:
-        go-version: 1.18
+        go-version: 1.19
       id: go
 
     - name: Check out code into the Go module directory

.github/workflows/release.yml

@@ -12,7 +12,7 @@ jobs:
       - uses: actions/checkout@v3
       - uses: actions/setup-go@v3
         with:
-          go-version: '^1.18'
+          go-version: '^1.19'
       - name: Compile man page markup
         id: gen-man-page-md
         run: |

.gitignore

@@ -56,4 +56,9 @@ _build
 debian/files
 
 # intellij
-/.idea
+/.idea
+
+# vscode
+.vscode
+
+yq3

Dockerfile

@@ -1,4 +1,4 @@
-FROM golang:1.18.4 as builder
+FROM golang:1.19.1 as builder
 
 WORKDIR /go/src/mikefarah/yq
 

Dockerfile.dev

@@ -1,4 +1,4 @@
-FROM golang:1.18.4
+FROM golang:1.19.1
 
 COPY scripts/devtools.sh /opt/devtools.sh
 

README.md

@@ -3,7 +3,7 @@
 ![Build](https://github.com/mikefarah/yq/workflows/Build/badge.svg)  ![Docker Pulls](https://img.shields.io/docker/pulls/mikefarah/yq.svg) ![Github Releases (by Release)](https://img.shields.io/github/downloads/mikefarah/yq/total.svg) ![Go Report](https://goreportcard.com/badge/github.com/mikefarah/yq) ![CodeQL](https://github.com/mikefarah/yq/workflows/CodeQL/badge.svg)
 
 
-a lightweight and portable command-line YAML, JSON and XML processor. `yq` uses [jq](https://github.com/stedolan/jq) like syntax but works with yaml files as well as json and xml. It doesn't yet support everything `jq` does - but it does support the most common operations and functions, and more is being added continuously.
+a lightweight and portable command-line YAML, JSON and XML processor. `yq` uses [jq](https://github.com/stedolan/jq) like syntax but works with yaml files as well as json, xml, properties, csv and tsv. It doesn't yet support everything `jq` does - but it does support the most common operations and functions, and more is being added continuously.
 
 yq is written in go - so you can download a dependency free binary for your platform and you are good to go! If you prefer there are a variety of package managers that can be used as well as Docker and Podman, all listed below.
 
@@ -73,6 +73,8 @@ yq -P sample.json
 
 See the [documentation](https://mikefarah.gitbook.io/yq/) for more examples.
 
+Take a look at the discussions for [common questions](https://github.com/mikefarah/yq/discussions/categories/q-a), and [cool ideas](https://github.com/mikefarah/yq/discussions/categories/show-and-tell)
+
 ## Install
 
 ### [Download the latest binary](https://github.com/mikefarah/yq/releases/latest)
@@ -207,12 +209,27 @@ RUN apk add --no-cache tzdata
 USER yq
 ```
 
+#### Podman with SELinux
+
+If you are using podman with SELinux, you will need to set the shared volume flag `:z` on the volume mount:
+
+```
+-v "${PWD}":/workdir:z
+```
+
 ### GitHub Action
 ```
   - name: Set foobar to cool
     uses: mikefarah/yq@master
     with:
       cmd: yq -i '.foo.bar = "cool"' 'config.yml'
+  - name: Get an entry with a variable that might contain dots or spaces
+    id: get_username
+    uses: mikefarah/yq@master
+    with:
+      cmd: yq '.all.children.["${{ matrix.ip_address }}"].username' ops/inventories/production.yml
+  - name: Reuse a variable obtained in another step
+    run: echo ${{ steps.get_username.outputs.result }}
 ```
 
 See https://mikefarah.gitbook.io/yq/usage/github-action for more.
@@ -290,10 +307,10 @@ Supported by @rmescandon (https://launchpad.net/~rmescandon/+archive/ubuntu/yq)
 - Keeps yaml formatting and comments when updating (though there are issues with whitespace)
 - [Decode/Encode base64 data](https://mikefarah.gitbook.io/yq/operators/encode-decode)
 - [Load content from other files](https://mikefarah.gitbook.io/yq/operators/load)
-- [Convert to/from json](https://mikefarah.gitbook.io/yq/v/v4.x/usage/convert)
+- [Convert to/from json/ndjson](https://mikefarah.gitbook.io/yq/v/v4.x/usage/convert)
 - [Convert to/from xml](https://mikefarah.gitbook.io/yq/v/v4.x/usage/xml)
 - [Convert to/from properties](https://mikefarah.gitbook.io/yq/v/v4.x/usage/properties)
-- [Convert to csv/tsv](https://mikefarah.gitbook.io/yq/usage/csv-tsv)
+- [Convert to/from csv/tsv](https://mikefarah.gitbook.io/yq/usage/csv-tsv)
 - [General shell completion scripts (bash/zsh/fish/powershell)](https://mikefarah.gitbook.io/yq/v/v4.x/commands/shell-completion)
 - [Reduce](https://mikefarah.gitbook.io/yq/operators/reduce) to merge multiple files or sum an array or other fancy things.
 - [Github Action](https://mikefarah.gitbook.io/yq/usage/github-action) to use in your automated pipeline (thanks @devorbitus)

cmd/version.go

@@ -11,7 +11,7 @@ var (
 	GitDescribe string
 
 	// Version is main version number that is being run at the moment.
-	Version = "4.26.1"
+	Version = "4.27.5"
 
 	// VersionPrerelease is a pre-release marker for the version. If this is "" (empty string)
 	// then it means that it is a final release. Otherwise, this is a pre-release

github-action/Dockerfile

@@ -2,7 +2,8 @@ FROM mikefarah/yq:4
 
 COPY entrypoint.sh /entrypoint.sh
 
-# this seems to be the default user in github actions
-USER 1001
+# github action recommendation is to run as root.
+# https://docs.github.com/en/actions/creating-actions/dockerfile-support-for-github-actions#user
+USER root
 
 ENTRYPOINT ["/entrypoint.sh"]

go.mod

@@ -2,11 +2,11 @@ module github.com/mikefarah/yq/v4
 
 require (
 	github.com/a8m/envsubst v1.3.0
-	github.com/alecthomas/participle/v2 v2.0.0-beta.4
+	github.com/alecthomas/participle/v2 v2.0.0-beta.5
 	github.com/alecthomas/repr v0.1.0
-	github.com/elliotchance/orderedmap v1.4.0
+	github.com/elliotchance/orderedmap v1.5.0
 	github.com/fatih/color v1.13.0
-	github.com/goccy/go-json v0.9.10
+	github.com/goccy/go-json v0.9.11
 	github.com/goccy/go-yaml v1.9.5
 	github.com/jinzhu/copier v0.3.5
 	github.com/magiconair/properties v1.8.6
@@ -27,4 +27,4 @@ require (
 	golang.org/x/xerrors v0.0.0-20220609144429-65e65417b02f // indirect
 )
 
-go 1.18
+go 1.19

go.sum

@@ -1,15 +1,15 @@
 github.com/a8m/envsubst v1.3.0 h1:GmXKmVssap0YtlU3E230W98RWtWCyIZzjtf1apWWyAg=
 github.com/a8m/envsubst v1.3.0/go.mod h1:MVUTQNGQ3tsjOOtKCNd+fl8RzhsXcDvvAEzkhGtlsbY=
 github.com/alecthomas/assert/v2 v2.0.3 h1:WKqJODfOiQG0nEJKFKzDIG3E29CN2/4zR9XGJzKIkbg=
-github.com/alecthomas/participle/v2 v2.0.0-beta.4 h1:ublfGBm+x+p2j7KotHhrUMbKtejT7M0Gv1Mt1u3absw=
-github.com/alecthomas/participle/v2 v2.0.0-beta.4/go.mod h1:RC764t6n4L8D8ITAJv0qdokritYSNR3wV5cVwmIEaMM=
+github.com/alecthomas/participle/v2 v2.0.0-beta.5 h1:y6dsSYVb1G5eK6mgmy+BgI3Mw35a3WghArZ/Hbebrjo=
+github.com/alecthomas/participle/v2 v2.0.0-beta.5/go.mod h1:RC764t6n4L8D8ITAJv0qdokritYSNR3wV5cVwmIEaMM=
 github.com/alecthomas/repr v0.1.0 h1:ENn2e1+J3k09gyj2shc0dHr/yjaWSHRlrJ4DPMevDqE=
 github.com/alecthomas/repr v0.1.0/go.mod h1:2kn6fqh/zIyPLmm3ugklbEi5hg5wS435eygvNfaDQL8=
 github.com/cpuguy83/go-md2man/v2 v2.0.2/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
 github.com/davecgh/go-spew v1.1.0 h1:ZDRjVQ15GmhC3fiQ8ni8+OwkZQO4DARzQgrnXU1Liz8=
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
-github.com/elliotchance/orderedmap v1.4.0 h1:wZtfeEONCbx6in1CZyE6bELEt/vFayMvsxqI5SgsR+A=
-github.com/elliotchance/orderedmap v1.4.0/go.mod h1:wsDwEaX5jEoyhbs7x93zk2H/qv0zwuhg4inXhDkYqys=
+github.com/elliotchance/orderedmap v1.5.0 h1:1IsExUsjv5XNBD3ZdC7jkAAqLWOOKdbPTmkHx63OsBg=
+github.com/elliotchance/orderedmap v1.5.0/go.mod h1:wsDwEaX5jEoyhbs7x93zk2H/qv0zwuhg4inXhDkYqys=
 github.com/fatih/color v1.10.0/go.mod h1:ELkj/draVOlAH/xkhN6mQ50Qd0MPOk5AAr3maGEBuJM=
 github.com/fatih/color v1.13.0 h1:8LOYc1KYPPmyKMuN8QV2DNRWNbLo6LZ0iLs8+mlH53w=
 github.com/fatih/color v1.13.0/go.mod h1:kLAiJbzzSOZDVNGyDpeOxJ47H46qBXwg5ILebYFFOfk=
@@ -17,8 +17,8 @@ github.com/go-playground/assert/v2 v2.0.1/go.mod h1:VDjEfimB/XKnb+ZQfWdccd7VUvSc
 github.com/go-playground/locales v0.13.0/go.mod h1:taPMhCMXrRLJO55olJkUXHZBHCxTMfnGwq/HNwmWNS8=
 github.com/go-playground/universal-translator v0.17.0/go.mod h1:UkSxE5sNxxRwHyU+Scu5vgOQjsIJAF8j9muTVoKLVtA=
 github.com/go-playground/validator/v10 v10.4.1/go.mod h1:nlOn6nFhuKACm19sB/8EGNn9GlaMV7XkbRSipzJ0Ii4=
-github.com/goccy/go-json v0.9.10 h1:hCeNmprSNLB8B8vQKWl6DpuH0t60oEs+TAk9a7CScKc=
-github.com/goccy/go-json v0.9.10/go.mod h1:6MelG93GURQebXPDq3khkgXZkazVtN9CRI+MGFi0w8I=
+github.com/goccy/go-json v0.9.11 h1:/pAaQDLHEoCq/5FFmSKBswWmK6H0e8g4159Kc/X/nqk=
+github.com/goccy/go-json v0.9.11/go.mod h1:6MelG93GURQebXPDq3khkgXZkazVtN9CRI+MGFi0w8I=
 github.com/goccy/go-yaml v1.9.5 h1:Eh/+3uk9kLxG4koCX6lRMAPS1OaMSAi+FJcya0INdB0=
 github.com/goccy/go-yaml v1.9.5/go.mod h1:U/jl18uSupI5rdI2jmuCswEA2htH9eXfferR3KfscvA=
 github.com/hexops/gotextdiff v1.0.3 h1:gitA9+qJrrTCsiCl7+kh75nPqQt1cx4ZkudSTLoUqJM=

how-it-works.md

@@ -37,7 +37,7 @@ The `=` operator then pipes the 'root' context through the `rhs` expression of `
 dog
 ```
 
-Both sides have now been evaluated, so now the operator copies across the value from the RHS (`.b`) to the the LHS (`.a`), and it returns the now updated context:
+Both sides have now been evaluated, so now the operator copies across the value from the RHS (`.b`) to the LHS (`.a`), and it returns the now updated context:
 
 ```yaml
 a: dog

pkg/yqlib/candidate_node.go

@@ -157,7 +157,7 @@ func (n *CandidateNode) UpdateAttributesFrom(other *CandidateNode, prefs assignP
 	n.Node.Kind = other.Node.Kind
 
 	// don't clobber custom tags...
-	if strings.HasPrefix(n.Node.Tag, "!!") || n.Node.Tag == "" {
+	if prefs.ClobberCustomTags || strings.HasPrefix(n.Node.Tag, "!!") || n.Node.Tag == "" {
 		n.Node.Tag = other.Node.Tag
 	}
 

pkg/yqlib/csv_test.go

@@ -150,7 +150,7 @@ func testCSVScenario(t *testing.T, s formatScenario) {
 	}
 }
 
-func documentCSVDecodeObjectScenario(t *testing.T, w *bufio.Writer, s formatScenario, formatType string) {
+func documentCSVDecodeObjectScenario(w *bufio.Writer, s formatScenario, formatType string) {
 	writeOrPanic(w, fmt.Sprintf("## %v\n", s.description))
 
 	if s.subdescription != "" {
@@ -250,9 +250,9 @@ func documentCSVScenario(t *testing.T, w *bufio.Writer, i interface{}) {
 	case "encode-tsv":
 		documentCSVEncodeScenario(w, s, "tsv")
 	case "decode-csv-object":
-		documentCSVDecodeObjectScenario(t, w, s, "csv")
+		documentCSVDecodeObjectScenario(w, s, "csv")
 	case "decode-tsv-object":
-		documentCSVDecodeObjectScenario(t, w, s, "tsv")
+		documentCSVDecodeObjectScenario(w, s, "tsv")
 	case "roundtrip-csv":
 		documentCSVRoundTripScenario(w, s, "csv")
 

pkg/yqlib/decoder_json.go

@@ -22,13 +22,13 @@ func (dec *jsonDecoder) Init(reader io.Reader) {
 
 func (dec *jsonDecoder) Decode(rootYamlNode *yaml.Node) error {
 
-	var dataBucket interface{}
+	var dataBucket orderedMap
 	log.Debug("going to decode")
 	err := dec.decoder.Decode(&dataBucket)
 	if err != nil {
 		return err
 	}
-	node, err := dec.convertToYamlNode(dataBucket)
+	node, err := dec.convertToYamlNode(&dataBucket)
 
 	if err != nil {
 		return err
@@ -38,39 +38,36 @@ func (dec *jsonDecoder) Decode(rootYamlNode *yaml.Node) error {
 	return nil
 }
 
-func (dec *jsonDecoder) convertToYamlNode(data interface{}) (*yaml.Node, error) {
-	switch data := data.(type) {
-	case nil:
-		return createScalarNode(nil, "null"), nil
-	case float64, float32:
-		// json decoder returns ints as float.
-		return parseSnippet(fmt.Sprintf("%v", data))
-	case int, int64, int32, string, bool:
-		return createScalarNode(data, fmt.Sprintf("%v", data)), nil
-	case map[string]interface{}:
-		return dec.parseMap(data)
-	case []interface{}:
-		return dec.parseArray(data)
-	default:
-		return nil, fmt.Errorf("unrecognised type :(")
+func (dec *jsonDecoder) convertToYamlNode(data *orderedMap) (*yaml.Node, error) {
+	if data.kv == nil {
+		switch rawData := data.altVal.(type) {
+		case nil:
+			return createScalarNode(nil, "null"), nil
+		case float64, float32:
+			// json decoder returns ints as float.
+			return parseSnippet(fmt.Sprintf("%v", rawData))
+		case int, int64, int32, string, bool:
+			return createScalarNode(rawData, fmt.Sprintf("%v", rawData)), nil
+		case []*orderedMap:
+			return dec.parseArray(rawData)
+		default:
+			return nil, fmt.Errorf("unrecognised type :( %v", rawData)
+		}
 	}
-}
-
-func (dec *jsonDecoder) parseMap(dataMap map[string]interface{}) (*yaml.Node, error) {
 
 	var yamlMap = &yaml.Node{Kind: yaml.MappingNode}
-
-	for key, value := range dataMap {
-		yamlValue, err := dec.convertToYamlNode(value)
+	for _, keyValuePair := range data.kv {
+		yamlValue, err := dec.convertToYamlNode(&keyValuePair.V)
 		if err != nil {
 			return nil, err
 		}
-		yamlMap.Content = append(yamlMap.Content, createScalarNode(key, fmt.Sprintf("%v", key)), yamlValue)
+		yamlMap.Content = append(yamlMap.Content, createScalarNode(keyValuePair.K, keyValuePair.K), yamlValue)
 	}
 	return yamlMap, nil
+
 }
 
-func (dec *jsonDecoder) parseArray(dataArray []interface{}) (*yaml.Node, error) {
+func (dec *jsonDecoder) parseArray(dataArray []*orderedMap) (*yaml.Node, error) {
 
 	var yamlMap = &yaml.Node{Kind: yaml.SequenceNode}
 

pkg/yqlib/doc/operators/alternative-default-value.md

@@ -79,3 +79,36 @@ will output
 cat
 ```
 
+## Update or create - entity exists
+This initialises `a` if it's not present
+
+Given a sample.yml file of:
+```yaml
+a: 1
+```
+then
+```bash
+yq '(.a // (.a = 0)) += 1' sample.yml
+```
+will output
+```yaml
+a: 2
+```
+
+## Update or create - entity does not exist
+This initialises `a` if it's not present
+
+Given a sample.yml file of:
+```yaml
+b: camel
+```
+then
+```bash
+yq '(.a // (.a = 0)) += 1' sample.yml
+```
+will output
+```yaml
+b: camel
+a: 1
+```
+

pkg/yqlib/doc/operators/assign-update.md

@@ -7,6 +7,11 @@ Which will assign the LHS node values to the RHS node values. The RHS expression
 
 ### relative form: `|=`
 This will do a similar thing to the plain form, however, the RHS expression is run against _the LHS nodes_. This is useful for updating values based on old values, e.g. increment.
+
+
+### Flags
+- `c` clobber custom tags
+
 {% hint style="warning" %}
 Note that versions prior to 4.18 require the 'eval/e' command to be specified. 
 
@@ -235,3 +240,37 @@ a:
     - bogs
 ```
 
+## Custom types are maintained by default
+Given a sample.yml file of:
+```yaml
+a: !cat meow
+b: !dog woof
+```
+then
+```bash
+yq '.a = .b' sample.yml
+```
+will output
+```yaml
+a: !cat woof
+b: !dog woof
+```
+
+## Custom types: clovver
+Use the `c` option to clobber custom tags
+
+Given a sample.yml file of:
+```yaml
+a: !cat meow
+b: !dog woof
+```
+then
+```bash
+yq '.a =c .b' sample.yml
+```
+will output
+```yaml
+a: !dog woof
+b: !dog woof
+```
+

pkg/yqlib/doc/operators/encode-decode.md

@@ -9,12 +9,12 @@ These operators are useful to process yaml documents that have stringified embed
 
 | Format | Decode (from string) | Encode (to string) |
 | --- | -- | --|
-| Yaml | from_yaml | to_yaml(i)/@yaml |
-| JSON | from_json | to_json(i)/@json |
+| Yaml | from_yaml/@yamld | to_yaml(i)/@yaml |
+| JSON | from_json/@jsond | to_json(i)/@json |
 | Properties | from_props/@propsd  | to_props/@props |
 | CSV | from_csv/@csvd | to_csv/@csv |
 | TSV | from_tsv/@tsvd | to_tsv/@tsv |
-| XML | from_xml | to_xml(i)/@xml |
+| XML | from_xml/@xmld | to_xml(i)/@xml |
 | Base64 | @base64d | @base64 |
 
 

pkg/yqlib/doc/operators/headers/assign-update.md

@@ -6,4 +6,8 @@ This operator is used to update node values. It can be used in either the:
 Which will assign the LHS node values to the RHS node values. The RHS expression is run against the matching nodes in the pipeline.
 
 ### relative form: `|=`
-This will do a similar thing to the plain form, however, the RHS expression is run against _the LHS nodes_. This is useful for updating values based on old values, e.g. increment.
+This will do a similar thing to the plain form, however, the RHS expression is run against _the LHS nodes_. This is useful for updating values based on old values, e.g. increment.
+
+
+### Flags
+- `c` clobber custom tags

pkg/yqlib/doc/operators/headers/encode-decode.md

@@ -9,12 +9,12 @@ These operators are useful to process yaml documents that have stringified embed
 
 | Format | Decode (from string) | Encode (to string) |
 | --- | -- | --|
-| Yaml | from_yaml | to_yaml(i)/@yaml |
-| JSON | from_json | to_json(i)/@json |
+| Yaml | from_yaml/@yamld | to_yaml(i)/@yaml |
+| JSON | from_json/@jsond | to_json(i)/@json |
 | Properties | from_props/@propsd  | to_props/@props |
 | CSV | from_csv/@csvd | to_csv/@csv |
 | TSV | from_tsv/@tsvd | to_tsv/@tsv |
-| XML | from_xml | to_xml(i)/@xml |
+| XML | from_xml/@xmld | to_xml(i)/@xml |
 | Base64 | @base64d | @base64 |
 
 

pkg/yqlib/doc/operators/headers/multiply-merge.md

@@ -14,6 +14,7 @@ You can control how objects are merged by using one or more of the following fla
 - `d` deeply merge arrays
 - `?` only merge _existing_ fields
 - `n` only merge _new_ fields
+- `c` clobber custom tags
 
 
 ### Merge two files together

pkg/yqlib/doc/operators/multiply-merge.md

@@ -14,6 +14,7 @@ You can control how objects are merged by using one or more of the following fla
 - `d` deeply merge arrays
 - `?` only merge _existing_ fields
 - `n` only merge _new_ fields
+- `c` clobber custom tags
 
 
 ### Merge two files together
@@ -481,3 +482,26 @@ b: !goat
   dog: woof
 ```
 
+## Custom types: clobber tags
+Use the `c` option to clobber custom tags. Note that the second tag is now used
+
+Given a sample.yml file of:
+```yaml
+a: !horse
+  cat: meow
+b: !goat
+  dog: woof
+```
+then
+```bash
+yq '.a *=c .b' sample.yml
+```
+will output
+```yaml
+a: !goat
+  cat: meow
+  dog: woof
+b: !goat
+  dog: woof
+```
+

pkg/yqlib/doc/operators/string-operators.md

@@ -112,6 +112,26 @@ will output
 cat; meow; 1; ; true
 ```
 
+## Trim strings
+Given a sample.yml file of:
+```yaml
+- ' cat'
+- 'dog '
+- ' cow cow '
+- horse
+```
+then
+```bash
+yq '.[] | trim' sample.yml
+```
+will output
+```yaml
+cat
+dog
+cow cow
+horse
+```
+
 ## Match string
 Given a sample.yml file of:
 ```yaml

pkg/yqlib/doc/usage/convert.md

@@ -1,10 +1,9 @@
 # JSON
 
-Encode and decode to and from JSON. Note that, unless you have multiple JSON documents in a single file, YAML is a _superset_ of JSON - so `yq` can read any json file without doing anything special.
+Encode and decode to and from JSON. Supports multiple JSON documents in a single file (e.g. NDJSON).
 
-If you do have mulitple JSON documents in a single file (e.g. NDJSON) then you will need to explicity use the json parser `-p=json`.
+Note that YAML is a superset of (single document) JSON - so you don't have to use the JSON parser to read JSON when there is only one JSON document in the input. You will probably want to pretty print the result in this case, to get idiomatic YAML styling.
 
-This means you don't need to 'convert' a JSON file to YAML - however if you want idiomatic YAML styling, then you can use the `-P/--prettyPrint` flag, see examples below.
 
 {% hint style="warning" %}
 Note that versions prior to 4.18 require the 'eval/e' command to be specified. 

pkg/yqlib/doc/usage/headers/convert.md

@@ -1,7 +1,6 @@
 # JSON
 
-Encode and decode to and from JSON. Note that, unless you have multiple JSON documents in a single file, YAML is a _superset_ of JSON - so `yq` can read any json file without doing anything special.
+Encode and decode to and from JSON. Supports multiple JSON documents in a single file (e.g. NDJSON).
 
-If you do have mulitple JSON documents in a single file (e.g. NDJSON) then you will need to explicity use the json parser `-p=json`.
+Note that YAML is a superset of (single document) JSON - so you don't have to use the JSON parser to read JSON when there is only one JSON document in the input. You will probably want to pretty print the result in this case, to get idiomatic YAML styling.
 
-This means you don't need to 'convert' a JSON file to YAML - however if you want idiomatic YAML styling, then you can use the `-P/--prettyPrint` flag, see examples below.

pkg/yqlib/encoder.go

@@ -66,8 +66,13 @@ func (o *orderedMap) UnmarshalJSON(data []byte) error {
 		}
 		return nil
 	case '[':
-		var arr []orderedMap
-		return json.Unmarshal(data, &arr)
+		var res []*orderedMap
+		if err := json.Unmarshal(data, &res); err != nil {
+			return err
+		}
+		o.altVal = res
+		o.kv = nil
+		return nil
 	}
 
 	return json.Unmarshal(data, &o.altVal)

pkg/yqlib/json_test.go

@@ -22,6 +22,13 @@ const sampleNdJson = `{"this": "is a multidoc json file"}
 {"a number": 4}
 `
 
+const sampleNdJsonKey = `{"a": "first", "b": "next", "ab": "last"}`
+
+const expectedJsonKeysInOrder = `a: first
+b: next
+ab: last
+`
+
 const expectedNdJsonYaml = `this: is a multidoc json file
 ---
 each:
@@ -157,6 +164,13 @@ var jsonScenarios = []formatScenario{
 		expected:     expectedNdJsonYaml,
 		scenarioType: "decode-ndjson",
 	},
+	{
+		description:  "Decode NDJSON, maintain key order",
+		skipDoc:      true,
+		input:        sampleNdJsonKey,
+		expected:     expectedJsonKeysInOrder,
+		scenarioType: "decode-ndjson",
+	},
 	{
 		description:  "numbers",
 		skipDoc:      true,
@@ -164,6 +178,20 @@ var jsonScenarios = []formatScenario{
 		expected:     "- 3\n- 3\n- 3.1\n- -1\n",
 		scenarioType: "decode-ndjson",
 	},
+	{
+		description:  "number single",
+		skipDoc:      true,
+		input:        "3",
+		expected:     "3\n",
+		scenarioType: "decode-ndjson",
+	},
+	{
+		description:  "empty string",
+		skipDoc:      true,
+		input:        `""`,
+		expected:     "\"\"\n",
+		scenarioType: "decode-ndjson",
+	},
 	{
 		description:  "strings",
 		skipDoc:      true,

pkg/yqlib/lexer_participle.go

@@ -152,6 +152,7 @@ var participleYqRules = []*participleYqRule{
 
 	{"Uppercase", `upcase|ascii_?upcase`, opTokenWithPrefs(changeCaseOpType, nil, changeCasePrefs{ToUpperCase: true}), 0},
 	{"Downcase", `downcase|ascii_?downcase`, opTokenWithPrefs(changeCaseOpType, nil, changeCasePrefs{ToUpperCase: false}), 0},
+	simpleOp("trim", trimOpType),
 
 	{"HexValue", `0[xX][0-9A-Fa-f]+`, hexValue(), 0},
 	{"FloatValueScientific", `-?[1-9](\.\d+)?[Ee][-+]?\d+`, floatValue(), 0},
@@ -181,8 +182,8 @@ var participleYqRules = []*participleYqRule{
 	{"GreaterThan", `\s*>\s*`, opTokenWithPrefs(compareOpType, nil, compareTypePref{OrEqual: false, Greater: true}), 0},
 	{"LessThan", `\s*<\s*`, opTokenWithPrefs(compareOpType, nil, compareTypePref{OrEqual: false, Greater: false}), 0},
 
-	{"AssignRelative", `\|=`, assignOpToken(true), 0},
-	{"Assign", `=`, assignOpToken(false), 0},
+	{"AssignRelative", `\|=[c]*`, assignOpToken(true), 0},
+	{"Assign", `=[c]*`, assignOpToken(false), 0},
 
 	{`whitespace`, `[ \t\n]+`, nil, 0},
 
@@ -193,8 +194,8 @@ var participleYqRules = []*participleYqRule{
 
 	{"Union", `,`, opToken(unionOpType), 0},
 
-	{"MultiplyAssign", `\*=[\+|\?dn]*`, multiplyWithPrefs(multiplyAssignOpType), 0},
-	{"Multiply", `\*[\+|\?dn]*`, multiplyWithPrefs(multiplyOpType), 0},
+	{"MultiplyAssign", `\*=[\+|\?cdn]*`, multiplyWithPrefs(multiplyAssignOpType), 0},
+	{"Multiply", `\*[\+|\?cdn]*`, multiplyWithPrefs(multiplyOpType), 0},
 
 	{"AddAssign", `\+=`, opToken(addAssignOpType), 0},
 	{"Add", `\+`, opToken(addOpType), 0},
@@ -316,6 +317,9 @@ func assignOpToken(updateAssign bool) yqAction {
 		log.Debug("assignOpToken %v", rawToken.Value)
 		value := rawToken.Value
 		prefs := assignPreferences{DontOverWriteAnchor: true}
+		if strings.Contains(value, "c") {
+			prefs.ClobberCustomTags = true
+		}
 		op := &Operation{OperationType: assignOpType, Value: assignOpType.Type, StringValue: value, UpdateAssign: updateAssign, Preferences: prefs}
 		return &token{TokenType: operationToken, Operation: op}, nil
 	}
@@ -386,6 +390,7 @@ func envSubstWithOptions() yqAction {
 func multiplyWithPrefs(op *operationType) yqAction {
 	return func(rawToken lexer.Token) (*token, error) {
 		prefs := multiplyPreferences{}
+		prefs.AssignPrefs = assignPreferences{}
 		options := rawToken.Value
 		if strings.Contains(options, "+") {
 			prefs.AppendArrays = true
@@ -394,11 +399,14 @@ func multiplyWithPrefs(op *operationType) yqAction {
 			prefs.TraversePrefs = traversePreferences{DontAutoCreate: true}
 		}
 		if strings.Contains(options, "n") {
-			prefs.AssignPrefs = assignPreferences{OnlyWriteNull: true}
+			prefs.AssignPrefs.OnlyWriteNull = true
 		}
 		if strings.Contains(options, "d") {
 			prefs.DeepMergeArrays = true
 		}
+		if strings.Contains(options, "c") {
+			prefs.AssignPrefs.ClobberCustomTags = true
+		}
 		prefs.TraversePrefs.DontFollowAlias = true
 		op := &Operation{OperationType: op, Value: multiplyOpType.Type, StringValue: options, Preferences: prefs}
 		return &token{TokenType: operationToken, Operation: op}, nil

pkg/yqlib/lib.go

@@ -1,5 +1,4 @@
 // Use the top level Evaluator or StreamEvaluator to evaluate expressions and return matches.
-//
 package yqlib
 
 import (
@@ -80,7 +79,7 @@ var notEqualsOpType = &operationType{Type: "NOT_EQUALS", NumArgs: 2, Precedence:
 
 var compareOpType = &operationType{Type: "COMPARE", NumArgs: 2, Precedence: 40, Handler: compareOperator}
 
-//createmap needs to be above union, as we use union to build the components of the objects
+// createmap needs to be above union, as we use union to build the components of the objects
 var createMapOpType = &operationType{Type: "CREATE_MAP", NumArgs: 2, Precedence: 15, Handler: createMapOperator}
 
 var shortPipeOpType = &operationType{Type: "SHORT_PIPE", NumArgs: 2, Precedence: 45, Handler: pipeOperator}
@@ -146,6 +145,7 @@ var captureOpType = &operationType{Type: "CAPTURE", NumArgs: 1, Precedence: 50,
 var testOpType = &operationType{Type: "TEST", NumArgs: 1, Precedence: 50, Handler: testOperator}
 var splitStringOpType = &operationType{Type: "SPLIT", NumArgs: 1, Precedence: 50, Handler: splitStringOperator}
 var changeCaseOpType = &operationType{Type: "CHANGE_CASE", NumArgs: 0, Precedence: 50, Handler: changeCaseOperator}
+var trimOpType = &operationType{Type: "TRIM", NumArgs: 0, Precedence: 50, Handler: trimSpaceOperator}
 
 var loadOpType = &operationType{Type: "LOAD", NumArgs: 1, Precedence: 50, Handler: loadYamlOperator}
 
@@ -430,7 +430,7 @@ func (p *Operation) toString() string {
 	}
 }
 
-//use for debugging only
+// use for debugging only
 func NodesToString(collection *list.List) string {
 	if !log.IsEnabledFor(logging.DEBUG) {
 		return ""

pkg/yqlib/operator_alternative.go

@@ -2,7 +2,24 @@ package yqlib
 
 func alternativeOperator(d *dataTreeNavigator, context Context, expressionNode *ExpressionNode) (Context, error) {
 	log.Debugf("-- alternative")
-	return crossFunction(d, context, expressionNode, alternativeFunc, true)
+	prefs := crossFunctionPreferences{
+		CalcWhenEmpty: true,
+		Calculation:   alternativeFunc,
+		LhsResultValue: func(lhs *CandidateNode) (*CandidateNode, error) {
+			if lhs == nil {
+				return nil, nil
+			}
+			truthy, err := isTruthy(lhs)
+			if err != nil {
+				return nil, err
+			}
+			if truthy {
+				return lhs, nil
+			}
+			return nil, nil
+		},
+	}
+	return crossFunctionWithPrefs(d, context, expressionNode, prefs)
 }
 
 func alternativeFunc(d *dataTreeNavigator, context Context, lhs *CandidateNode, rhs *CandidateNode) (*CandidateNode, error) {

pkg/yqlib/operator_alternative_test.go

@@ -85,6 +85,24 @@ var alternativeOperatorScenarios = []expressionScenario{
 			"D0, P[], (!!bool)::true\n",
 		},
 	},
+	{
+		description:    "Update or create - entity exists",
+		subdescription: "This initialises `a` if it's not present",
+		expression:     "(.a // (.a = 0)) += 1",
+		document:       `a: 1`,
+		expected: []string{
+			"D0, P[], (doc)::a: 2\n",
+		},
+	},
+	{
+		description:    "Update or create - entity does not exist",
+		subdescription: "This initialises `a` if it's not present",
+		expression:     "(.a // (.a = 0)) += 1",
+		document:       `b: camel`,
+		expected: []string{
+			"D0, P[], (!!map)::b: camel\na: 1\n",
+		},
+	},
 }
 
 func TestAlternativeOperatorScenarios(t *testing.T) {

pkg/yqlib/operator_assign.go

@@ -3,6 +3,7 @@ package yqlib
 type assignPreferences struct {
 	DontOverWriteAnchor bool
 	OnlyWriteNull       bool
+	ClobberCustomTags   bool
 }
 
 func assignUpdateFunc(prefs assignPreferences) crossFunctionCalculation {
@@ -15,18 +16,29 @@ func assignUpdateFunc(prefs assignPreferences) crossFunctionCalculation {
 	}
 }
 
+// they way *= (multipleAssign) is handled, we set the multiplePrefs
+// on the node, not assignPrefs. Long story.
+func getAssignPreferences(preferences interface{}) assignPreferences {
+	prefs := assignPreferences{}
+
+	switch typedPref := preferences.(type) {
+	case assignPreferences:
+		prefs = typedPref
+	case multiplyPreferences:
+		prefs = typedPref.AssignPrefs
+	}
+	return prefs
+}
+
 func assignUpdateOperator(d *dataTreeNavigator, context Context, expressionNode *ExpressionNode) (Context, error) {
 	lhs, err := d.GetMatchingNodes(context, expressionNode.LHS)
 	if err != nil {
 		return Context{}, err
 	}
 
-	prefs := assignPreferences{}
-	// they way *= (multipleAssign) is handled, we set the multiplePrefs
-	// on the node, not assignPrefs. Long story.
-	if p, ok := expressionNode.Operation.Preferences.(assignPreferences); ok {
-		prefs = p
-	}
+	prefs := getAssignPreferences(expressionNode.Operation.Preferences)
+
+	log.Debug("assignUpdateOperator prefs: %v", prefs)
 
 	if !expressionNode.Operation.UpdateAssign {
 		// this works because we already ran against LHS with an editable context.

pkg/yqlib/operator_assign_test.go

@@ -208,6 +208,23 @@ var assignOperatorScenarios = []expressionScenario{
 			"D0, P[], (doc)::a:\n    b:\n        - null\n        - c: bogs\n",
 		},
 	},
+	{
+		description: "Custom types are maintained by default",
+		document:    "a: !cat meow\nb: !dog woof",
+		expression:  `.a = .b`,
+		expected: []string{
+			"D0, P[], (doc)::a: !cat woof\nb: !dog woof\n",
+		},
+	},
+	{
+		description:    "Custom types: clovver",
+		subdescription: "Use the `c` option to clobber custom tags",
+		document:       "a: !cat meow\nb: !dog woof",
+		expression:     `.a =c .b`,
+		expected: []string{
+			"D0, P[], (doc)::a: !dog woof\nb: !dog woof\n",
+		},
+	},
 }
 
 func TestAssignOperatorScenarios(t *testing.T) {

pkg/yqlib/operator_comments.go

@@ -63,8 +63,12 @@ func assignCommentsOperator(d *dataTreeNavigator, context Context, expressionNod
 		}
 		if preferences.FootComment && candidate.Node.Kind == yaml.DocumentNode && comment != "" {
 			candidate.TrailingContent = "# " + comment
+		} else if preferences.FootComment && candidate.Node.Kind == yaml.DocumentNode {
+			candidate.TrailingContent = comment
+
 		} else if preferences.FootComment && candidate.Node.Kind != yaml.DocumentNode {
 			candidate.Node.FootComment = comment
+			candidate.TrailingContent = ""
 		}
 
 	}

pkg/yqlib/operator_comments_test.go

@@ -62,6 +62,15 @@ var commentOperatorScenarios = []expressionScenario{
 			"D0, P[], (doc)::a: cat\n# cat\n",
 		},
 	},
+	{
+		skipDoc:     true,
+		description: "Set foot comment, using an expression",
+		document:    "a: cat\n\n# hi",
+		expression:  `. foot_comment=""`,
+		expected: []string{
+			"D0, P[], (doc)::a: cat\n",
+		},
+	},
 	{
 		skipDoc:    true,
 		document:   `a: cat`,
@@ -112,6 +121,24 @@ var commentOperatorScenarios = []expressionScenario{
 			"D0, P[], (!!str)::welcome!\n",
 		},
 	},
+	{
+		skipDoc:     true,
+		description: "strip trailing comment recurse all",
+		document:    "a: cat\n\n# haha",
+		expression:  `... comments= ""`,
+		expected: []string{
+			"D0, P[], (!!map)::a: cat\n",
+		},
+	},
+	{
+		skipDoc:     true,
+		description: "strip trailing comment recurse values",
+		document:    "a: cat\n\n# haha",
+		expression:  `.. comments= ""`,
+		expected: []string{
+			"D0, P[], (!!map)::a: cat\n",
+		},
+	},
 	{
 		description:           "Head comment with document split",
 		dontFormatInputForDoc: true,

pkg/yqlib/operator_create_map.go

@@ -72,7 +72,7 @@ func sequenceFor(d *dataTreeNavigator, context Context, matchingNode *CandidateN
 	return &CandidateNode{Node: innerList, Document: document, Path: path}, nil
 }
 
-//NOTE: here the document index gets dropped so we
+// NOTE: here the document index gets dropped so we
 // no longer know where the node originates from.
 func listToNodeSeq(list *list.List) *yaml.Node {
 	node := yaml.Node{Kind: yaml.SequenceNode, Tag: "!!seq"}

pkg/yqlib/operator_encoder_decoder_test.go

@@ -81,8 +81,8 @@ var encoderDecoderOperatorScenarios = []expressionScenario{
 	},
 	{
 		description: "Decode tsv encoded string",
-		document: `a: "cats	dogs\ngreat	cool as well"`,
-		expression: `.a |= @tsvd`,
+		document:    `a: "cats	dogs\ngreat	cool as well"`,
+		expression:  `.a |= @tsvd`,
 		expected: []string{
 			"D0, P[], (doc)::a:\n    - cats: great\n      dogs: cool as well\n",
 		},

pkg/yqlib/operator_multiply_test.go

@@ -523,6 +523,35 @@ var multiplyOperatorScenarios = []expressionScenario{
 			"D0, P[], (doc)::a: !horse {cat: meow, dog: woof}\nb: !goat {dog: woof}\n",
 		},
 	},
+	{
+		description:    "Custom types: clobber tags",
+		subdescription: "Use the `c` option to clobber custom tags. Note that the second tag is now used",
+		document:       "a: !horse {cat: meow}\nb: !goat {dog: woof}",
+		expression:     ".a *=c .b",
+		expected: []string{
+			"D0, P[], (doc)::a: !goat {cat: meow, dog: woof}\nb: !goat {dog: woof}\n",
+		},
+	},
+	{
+		skipDoc:        true,
+		description:    "Custom types: clobber tags - *=",
+		subdescription: "Use the `c` option to clobber custom tags - on both the `=` and `*` operator. Note that the second tag is now used",
+		document:       "a: !horse {cat: meow}\nb: !goat {dog: woof}",
+		expression:     ".a =c .a *c .b",
+		expected: []string{
+			"D0, P[], (doc)::a: !goat {cat: meow, dog: woof}\nb: !goat {dog: woof}\n",
+		},
+	},
+	{
+		skipDoc:        true,
+		description:    "Custom types: dont clobber tags - *=",
+		subdescription: "Use the `c` option to clobber custom tags - on both the `=` and `*` operator. Note that the second tag is now used",
+		document:       "a: !horse {cat: meow}\nb: !goat {dog: woof}",
+		expression:     ".a *= .b",
+		expected: []string{
+			"D0, P[], (doc)::a: !horse {cat: meow, dog: woof}\nb: !goat {dog: woof}\n",
+		},
+	},
 	{
 		skipDoc:     true,
 		description: "Custom types: that are really maps",
@@ -532,6 +561,24 @@ var multiplyOperatorScenarios = []expressionScenario{
 			"D0, P[], (doc)::a: {cat: !horse 5}\nb: {cat: 5}\n",
 		},
 	},
+	{
+		skipDoc:     true,
+		description: "Relative merge, new fields only",
+		document:    "a: {a: original}\n",
+		expression:  `.a *=n load("../../examples/thing.yml")`,
+		expected: []string{
+			"D0, P[], (doc)::a: {a: original, b: cool.}\n",
+		},
+	},
+	{
+		skipDoc:     true,
+		description: "Relative merge",
+		document:    "a: {a: original}\n",
+		expression:  `.a *= load("../../examples/thing.yml")`,
+		expected: []string{
+			"D0, P[], (doc)::a: {a: apple is included, b: cool.}\n",
+		},
+	},
 }
 
 func TestMultiplyOperatorScenarios(t *testing.T) {

pkg/yqlib/operator_sort.go

@@ -17,7 +17,7 @@ func sortOperator(d *dataTreeNavigator, context Context, expressionNode *Express
 }
 
 // context represents the current matching nodes in the expression pipeline
-//expressionNode is your current expression (sort_by)
+// expressionNode is your current expression (sort_by)
 func sortByOperator(d *dataTreeNavigator, context Context, expressionNode *ExpressionNode) (Context, error) {
 
 	results := list.New()

pkg/yqlib/operator_strings.go

@@ -13,6 +13,26 @@ type changeCasePrefs struct {
 	ToUpperCase bool
 }
 
+func trimSpaceOperator(d *dataTreeNavigator, context Context, expressionNode *ExpressionNode) (Context, error) {
+	results := list.New()
+	for el := context.MatchingNodes.Front(); el != nil; el = el.Next() {
+		candidate := el.Value.(*CandidateNode)
+
+		node := unwrapDoc(candidate.Node)
+
+		if guessTagFromCustomType(node) != "!!str" {
+			return Context{}, fmt.Errorf("cannot trim %v, can only operate on strings. ", node.Tag)
+		}
+
+		newStringNode := &yaml.Node{Kind: yaml.ScalarNode, Tag: node.Tag, Style: node.Style}
+		newStringNode.Value = strings.TrimSpace(node.Value)
+		results.PushBack(candidate.CreateReplacement(newStringNode))
+
+	}
+
+	return context.ChildContext(results), nil
+}
+
 func changeCaseOperator(d *dataTreeNavigator, context Context, expressionNode *ExpressionNode) (Context, error) {
 	results := list.New()
 	prefs := expressionNode.Operation.Preferences.(changeCasePrefs)

pkg/yqlib/operator_strings_test.go

@@ -47,6 +47,17 @@ var stringsOperatorScenarios = []expressionScenario{
 			"D0, P[], (!!str)::cat; meow; 1; ; true\n",
 		},
 	},
+	{
+		description: "Trim strings",
+		document:    `[" cat", "dog ", " cow cow ", horse]`,
+		expression:  `.[] | trim`,
+		expected: []string{
+			"D0, P[0], (!!str)::cat\n",
+			"D0, P[1], (!!str)::dog\n",
+			"D0, P[2], (!!str)::cow cow\n",
+			"D0, P[3], (!!str)::horse\n",
+		},
+	},
 	{
 		skipDoc:    true,
 		document:   `[!horse cat, !goat meow, !frog 1, null, true]`,

pkg/yqlib/operators.go

@@ -5,6 +5,7 @@ import (
 	"fmt"
 
 	"github.com/jinzhu/copier"
+	logging "gopkg.in/op/go-logging.v1"
 	"gopkg.in/yaml.v3"
 )
 
@@ -18,7 +19,21 @@ func compoundAssignFunction(d *dataTreeNavigator, context Context, expressionNod
 		return Context{}, err
 	}
 
-	assignmentOp := &Operation{OperationType: assignOpType, Preferences: expressionNode.Operation.Preferences}
+	// tricky logic when we are running *= with flags.
+	// we have an op like: .a *=nc .b
+	// which should roughly translate to .a =c .a *nc .b
+	// note that the 'n' flag only applies to the multiple op, not the assignment
+	// but the clobber flag applies to both!
+
+	prefs := assignPreferences{}
+	switch typedPref := expressionNode.Operation.Preferences.(type) {
+	case assignPreferences:
+		prefs = typedPref
+	case multiplyPreferences:
+		prefs.ClobberCustomTags = typedPref.AssignPrefs.ClobberCustomTags
+	}
+
+	assignmentOp := &Operation{OperationType: assignOpType, Preferences: prefs}
 
 	for el := lhs.MatchingNodes.Front(); el != nil; el = el.Next() {
 		candidate := el.Value.(*CandidateNode)
@@ -83,8 +98,10 @@ func resultsForRHS(d *dataTreeNavigator, context Context, lhsCandidate *Candidat
 	}
 
 	for rightEl := rhs.MatchingNodes.Front(); rightEl != nil; rightEl = rightEl.Next() {
-		log.Debugf("Applying calc")
 		rhsCandidate := rightEl.Value.(*CandidateNode)
+		if !log.IsEnabledFor(logging.DEBUG) {
+			log.Debugf("Applying lhs: %v, rhsCandidate, %v", NodeToString(lhsCandidate), NodeToString(rhsCandidate))
+		}
 		resultCandidate, err := prefs.Calculation(d, context, lhsCandidate, rhsCandidate)
 		if err != nil {
 			return err

pkg/yqlib/stream_evaluator.go

@@ -110,7 +110,7 @@ func (s *streamEvaluator) Evaluate(filename string, reader io.Reader, node *Expr
 			Node:      &dataBucket,
 			FileIndex: s.fileIndex,
 		}
-		//move document comments into candidate node
+		// move document comments into candidate node
 		// otherwise unwrap drops them.
 		candidateNode.TrailingContent = dataBucket.FootComment
 		dataBucket.FootComment = ""

pkg/yqlib/string_evaluator.go

@@ -0,0 +1,85 @@
+package yqlib
+
+import (
+	"bytes"
+	"container/list"
+	"errors"
+	"fmt"
+	"io"
+
+	yaml "gopkg.in/yaml.v3"
+)
+
+type StringEvaluator interface {
+	Evaluate(expression string, input string, encoder Encoder, leadingContentPreProcessing bool, decoder Decoder) (string, error)
+}
+
+type stringEvaluator struct {
+	treeNavigator DataTreeNavigator
+	fileIndex     int
+}
+
+func NewStringEvaluator() StringEvaluator {
+	return &stringEvaluator{
+		treeNavigator: NewDataTreeNavigator(),
+	}
+}
+
+func (s *stringEvaluator) Evaluate(expression string, input string, encoder Encoder, leadingContentPreProcessing bool, decoder Decoder) (string, error) {
+
+	// Use bytes.Buffer for output of string
+	out := new(bytes.Buffer)
+	printer := NewPrinter(encoder, NewSinglePrinterWriter(out))
+
+	InitExpressionParser()
+	node, err := ExpressionParser.ParseExpression(expression)
+	if err != nil {
+		return "", err
+	}
+
+	reader, leadingContent, err := readString(input, leadingContentPreProcessing)
+	if err != nil {
+		return "", err
+	}
+
+	var currentIndex uint
+	decoder.Init(reader)
+	for {
+		var dataBucket yaml.Node
+		errorReading := decoder.Decode(&dataBucket)
+
+		if errors.Is(errorReading, io.EOF) {
+			s.fileIndex = s.fileIndex + 1
+			return out.String(), nil
+		} else if errorReading != nil {
+			return "", fmt.Errorf("bad input '%v': %w", input, errorReading)
+		}
+
+		candidateNode := &CandidateNode{
+			Document:  currentIndex,
+			Node:      &dataBucket,
+			FileIndex: s.fileIndex,
+		}
+		// move document comments into candidate node
+		// otherwise unwrap drops them.
+		candidateNode.TrailingContent = dataBucket.FootComment
+		dataBucket.FootComment = ""
+
+		if currentIndex == 0 {
+			candidateNode.LeadingContent = leadingContent
+		}
+		inputList := list.New()
+		inputList.PushBack(candidateNode)
+
+		result, errorParsing := s.treeNavigator.GetMatchingNodes(Context{MatchingNodes: inputList}, node)
+		if errorParsing != nil {
+			return "", errorParsing
+		}
+		err = printer.PrintResults(result.MatchingNodes)
+
+		if err != nil {
+			return "", err
+		}
+		currentIndex = currentIndex + 1
+	}
+}

pkg/yqlib/string_evaluator_test.go

@@ -0,0 +1,30 @@
+package yqlib
+
+import (
+	"testing"
+
+	"github.com/mikefarah/yq/v4/test"
+)
+
+func TestStringEvaluator_Evaluate_Nominal(t *testing.T) {
+	expected_output := `` +
+		`yq` + "\n" +
+		`---` + "\n" +
+		`jq` + "\n"
+	expression := ".[].name"
+	input := `` +
+		` - name: yq` + "\n" +
+		`   description: yq is a portable command-line YAML, JSON and XML processor` + "\n" +
+		`---` + "\n" +
+		` - name: jq` + "\n" +
+		`   description: Command-line JSON processor` + "\n"
+	encoder := NewYamlEncoder(2, true, true, true)
+	decoder := NewYamlDecoder()
+
+	result, err := NewStringEvaluator().Evaluate(expression, input, encoder, true, decoder)
+	if err != nil {
+		t.Error(err)
+	}
+
+	test.AssertResult(t, expected_output, result)
+}

pkg/yqlib/utils.go

@@ -33,6 +33,14 @@ func readStream(filename string, leadingContentPreProcessing bool) (io.Reader, s
 	return processReadStream(reader)
 }
 
+func readString(input string, leadingContentPreProcessing bool) (io.Reader, string, error) {
+	reader := bufio.NewReader(strings.NewReader(input))
+	if !leadingContentPreProcessing {
+		return reader, "", nil
+	}
+	return processReadStream(reader)
+}
+
 func writeString(writer io.Writer, txt string) error {
 	_, errorWriting := writer.Write([]byte(txt))
 	return errorWriting

release_notes.txt

@@ -1,5 +1,24 @@
+4.27.5:
+  - Fixed relative merge bug #1333
+
+4.27.4:
+  - Fixed bug in alternative (//) operator, RHS being evaluated when it didn't need to be
+  - Fixed footer comment issue #1231
+  - Github action now runs as root (as recommended by Github Actions doc)
+  - Updated dependencies
+
+4.27.3:
+  - Added new 'c' merge and assign flag that clobbers custom tags
+  - Bumped go dependency to fix CVE (#1316)
+  - Updated dependencies
+
+4.27.2:
+  - Fixed JSON decoder to maintain object key order.
+
 4.27.1:
   - Added 'json' decoder for support for multiple JSON documents in a single file (e.g. NDJSON)
+  - Added 'csv' decoding, array of objects encoding, and round-triping
+  - New StringEvaluator when using yq as a lib (thanks @leviliangtw)
   - Fixed XML decoding issue (#1284)
 
 4.26.1:

scripts/devtools.sh

@@ -1,5 +1,5 @@
 #!/bin/sh
 set -ex
 go mod download golang.org/x/tools@latest
-curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(go env GOPATH)/bin v1.45.2
-wget -O- -nv https://raw.githubusercontent.com/securego/gosec/master/install.sh | sh -s v2.11.0
+curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(go env GOPATH)/bin v1.49.0
+wget -O- -nv https://raw.githubusercontent.com/securego/gosec/master/install.sh | sh -s v2.13.1

snap/snapcraft.yaml

@@ -1,5 +1,5 @@
 name: yq
-version: '4.26.1'
+version: '4.27.5'
 summary: A lightweight and portable command-line YAML processor
 description: |
   The aim of the project is to be the jq or sed of yaml files.
@@ -16,7 +16,7 @@ apps:
 parts:
   yq:
     plugin: go
-    go-channel: 1.18/stable
+    go-channel: 1.19/stable
     source: .
     source-type: git
     go-importpath: github.com/mikefarah/yq