Bumping version

Preparing release
XML Decoder: Comment parsing tweak
2026-08-24 16:39:58 +08:00 · 2022-11-28 08:36:05 +11:00 · 2022-11-28 08:34:58 +11:00 · 2022-11-27 19:14:41 +11:00 · 2022-11-27 17:58:37 +11:00 · 2022-11-27 17:46:50 +11:00
195 changed files with 4094 additions and 1122 deletions
@@ -60,3 +60,23 @@ jobs:
            -t "${IMAGE_NAME}:4" \
            -t "${IMAGE_NAME}:latest" \
            .
+          
+          cd github-action
+          docker buildx build \
+            --label "org.opencontainers.image.authors=https://github.com/mikefarah/yq/graphs/contributors" \
+            --label "org.opencontainers.image.created=$(date --rfc-3339=seconds)" \
+            --label "org.opencontainers.image.description=yq is a portable command-line YAML processor" \
+            --label "org.opencontainers.image.documentation=https://mikefarah.gitbook.io/yq/" \
+            --label "org.opencontainers.image.licenses=MIT" \
+            --label "org.opencontainers.image.revision=$(git rev-parse HEAD)" \
+            --label "org.opencontainers.image.source=https://github.com/mikefarah/yq" \
+            --label "org.opencontainers.image.title=yq" \
+            --label "org.opencontainers.image.url=https://mikefarah.gitbook.io/yq/" \
+            --label "org.opencontainers.image.version=${IMAGE_VERSION}" \
+            --platform "${PLATFORMS}" \
+            --pull \
+            --push \
+            -t "${IMAGE_NAME}:${IMAGE_VERSION}-githubaction" \
+            -t "${IMAGE_NAME}:4-githubaction" \
+            -t "${IMAGE_NAME}:latest-githubaction" \
+            .
@@ -10,10 +10,10 @@ jobs:
    runs-on: ubuntu-latest
    steps:

-    - name: Set up Go 1.18
+    - name: Set up Go 1.19
      uses: actions/setup-go@v3
      with:
-        go-version: 1.18
+        go-version: 1.19
      id: go

    - name: Check out code into the Go module directory
@@ -12,16 +12,16 @@ jobs:
      - uses: actions/checkout@v3
      - uses: actions/setup-go@v3
        with:
-          go-version: '^1.18'
+          go-version: '^1.19'
+          check-latest: true
      - name: Compile man page markup
        id: gen-man-page-md
        run: |
          ./scripts/generate-man-page-md.sh
-          echo "::set-output name=man-page-md::man.md"

      - name: Get the version
        id: get_version
-        run: echo ::set-output name=VERSION::${GITHUB_REF##*/}
+        run: echo "VERSION=${GITHUB_REF##*/}" >> "${GITHUB_OUTPUT}"

      - name: Generate man page
        uses: docker://pandoc/core:2.14.2
@@ -35,7 +35,7 @@ jobs:
            --variable=header:"yq (https://github.com/mikefarah/yq/) version ${{ steps.get_version.outputs.VERSION }}"
            --variable=author:"Mike Farah"
            --output=yq.1
-            ${{ steps.gen-man-page-md.outputs.man-page-md }}
+            man.md

      - name: Cross compile
        run: |
@@ -0,0 +1,23 @@
+name: Release Snap
+
+on:
+  release:
+    types: [released]
+
+  # Allows you to run this workflow manually from the Actions tab
+  workflow_dispatch:
+
+jobs:
+    buildSnap:
+      environment: snap
+      runs-on: ubuntu-latest
+      steps:
+        - uses: actions/checkout@v3
+        - uses: snapcore/action-build@v1
+          id: build
+        - uses: snapcore/action-publish@v1
+          env:
+            SNAPCRAFT_STORE_CREDENTIALS: ${{ secrets.STORE_LOGIN }}
+          with:
+            snap: ${{ steps.build.outputs.snap }}
+            release: stable
@@ -1,5 +1,8 @@
 name: Test Yq Action
-on: [push]
+on: 
+  push:
+  workflow_dispatch:
+
 permissions:
  contents: read

@@ -12,11 +15,14 @@ jobs:
    steps:
      - uses: actions/checkout@v3
      - name: Get test
+        id: get_value
        uses: mikefarah/yq@master
        with:
-          cmd: yq eval '.a' examples/sample.yaml
+          cmd: yq '.test' examples/multiline-text.yaml
+      - name: Multiline test
+        run: echo "### It was [${{ steps.get_value.outputs.result }}]" >> $GITHUB_STEP_SUMMARY
      - name: Write inplace test
        id: lookupSdkVersion
        uses: mikefarah/yq@master
        with:
-          cmd: yq eval -i '.b.c = 5' examples/sample.yaml
+          cmd: yq -i '.b.c = 5' examples/sample.yaml
@@ -56,4 +56,9 @@ _build
 debian/files

 # intellij
-/.idea
+/.idea
+
+# vscode
+.vscode
+
+yq3
@@ -1,4 +1,4 @@
-FROM golang:1.18.4 as builder
+FROM golang:1.19.3 as builder

 WORKDIR /go/src/mikefarah/yq

@@ -1,4 +1,4 @@
-FROM golang:1.18.4
+FROM golang:1.19.3

 COPY scripts/devtools.sh /opt/devtools.sh

@@ -3,31 +3,10 @@
 ![Build](https://github.com/mikefarah/yq/workflows/Build/badge.svg)  ![Docker Pulls](https://img.shields.io/docker/pulls/mikefarah/yq.svg) ![Github Releases (by Release)](https://img.shields.io/github/downloads/mikefarah/yq/total.svg) ![Go Report](https://goreportcard.com/badge/github.com/mikefarah/yq) ![CodeQL](https://github.com/mikefarah/yq/workflows/CodeQL/badge.svg)


-a lightweight and portable command-line YAML, JSON and XML processor. `yq` uses [jq](https://github.com/stedolan/jq) like syntax but works with yaml files as well as json and xml. It doesn't yet support everything `jq` does - but it does support the most common operations and functions, and more is being added continuously.
+a lightweight and portable command-line YAML, JSON and XML processor. `yq` uses [jq](https://github.com/stedolan/jq) like syntax but works with yaml files as well as json, xml, properties, csv and tsv. It doesn't yet support everything `jq` does - but it does support the most common operations and functions, and more is being added continuously.

 yq is written in go - so you can download a dependency free binary for your platform and you are good to go! If you prefer there are a variety of package managers that can be used as well as Docker and Podman, all listed below.

-## Notice for v4.x versions prior to 4.18.1
-Since 4.18.1, yq's 'eval/e' command is the _default_ command and no longer needs to be specified.
-
-Older versions will still need to specify 'eval/e'.
-
-Similarly, '-' is no longer required as a filename to read from STDIN (unless reading from one or more files).
-
-TLDR:
-
-Prior to 4.18.1 
-```bash
-yq e '.cool' - < file.yaml
-```
-
-4.18+ 
-```bash
-yq '.cool' < file.yaml
-```
-
-When merging multiple files together, `eval-all/ea` is still required to tell `yq` to run the expression against all the document at once.
-
 ## Quick Usage Guide

 Read a value:
@@ -73,12 +52,17 @@ yq -P sample.json

 See the [documentation](https://mikefarah.gitbook.io/yq/) for more examples.

+Take a look at the discussions for [common questions](https://github.com/mikefarah/yq/discussions/categories/q-a), and [cool ideas](https://github.com/mikefarah/yq/discussions/categories/show-and-tell)
+
 ## Install

 ### [Download the latest binary](https://github.com/mikefarah/yq/releases/latest)

 ### wget
-Use wget to download the pre-compiled binaries:
+Use wget to download, gzipped pre-compiled binaries:
+
+
+For instance, VERSION=v4.2.0 and BINARY=yq_linux_amd64

 #### Compressed via tar.gz
 ```bash
@@ -93,7 +77,12 @@ wget https://github.com/mikefarah/yq/releases/download/${VERSION}/${BINARY} -O /
    chmod +x /usr/bin/yq
 ```

-For instance, VERSION=v4.2.0 and BINARY=yq_linux_amd64
+#### Latest version
+
+```bash
+wget https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -O /usr/bin/yq &&\
+    chmod +x /usr/bin/yq
+```

 ### MacOS / Linux via Homebrew:
 Using [Homebrew](https://brew.sh/)
@@ -207,12 +196,27 @@ RUN apk add --no-cache tzdata
 USER yq
 ```

+#### Podman with SELinux
+
+If you are using podman with SELinux, you will need to set the shared volume flag `:z` on the volume mount:
+
+```
+-v "${PWD}":/workdir:z
+```
+
 ### GitHub Action
 ```
  - name: Set foobar to cool
    uses: mikefarah/yq@master
    with:
      cmd: yq -i '.foo.bar = "cool"' 'config.yml'
+  - name: Get an entry with a variable that might contain dots or spaces
+    id: get_username
+    uses: mikefarah/yq@master
+    with:
+      cmd: yq '.all.children.["${{ matrix.ip_address }}"].username' ops/inventories/production.yml
+  - name: Reuse a variable obtained in another step
+    run: echo ${{ steps.get_username.outputs.result }}
 ```

 See https://mikefarah.gitbook.io/yq/usage/github-action for more.
@@ -225,7 +229,17 @@ go install github.com/mikefarah/yq/v4@latest
 ## Community Supported Installation methods
 As these are supported by the community :heart: - however, they may be out of date with the officially supported releases.

-# Webi
+
+### Nix
+
+```
+nix profile install nixpkgs#yq-go
+```
+
+See [here](https://search.nixos.org/packages?channel=unstable&show=yq-go&from=0&size=50&sort=relevance&type=packages&query=yq-go)
+
+
+### Webi

 ```
 webi yq
@@ -248,6 +262,13 @@ choco install yq
 ```
 Supported by @chillum (https://chocolatey.org/packages/yq)

+and
+
+### Winget
+winget install yq
+
+https://winget.run/pkg/MikeFarah/yq
+
 ### Mac:
 Using [MacPorts](https://www.macports.org/)
 ```
@@ -290,10 +311,10 @@ Supported by @rmescandon (https://launchpad.net/~rmescandon/+archive/ubuntu/yq)
 - Keeps yaml formatting and comments when updating (though there are issues with whitespace)
 - [Decode/Encode base64 data](https://mikefarah.gitbook.io/yq/operators/encode-decode)
 - [Load content from other files](https://mikefarah.gitbook.io/yq/operators/load)
- [Convert to/from json](https://mikefarah.gitbook.io/yq/v/v4.x/usage/convert)
+- [Convert to/from json/ndjson](https://mikefarah.gitbook.io/yq/v/v4.x/usage/convert)
 - [Convert to/from xml](https://mikefarah.gitbook.io/yq/v/v4.x/usage/xml)
 - [Convert to/from properties](https://mikefarah.gitbook.io/yq/v/v4.x/usage/properties)
- [Convert to csv/tsv](https://mikefarah.gitbook.io/yq/usage/csv-tsv)
+- [Convert to/from csv/tsv](https://mikefarah.gitbook.io/yq/usage/csv-tsv)
 - [General shell completion scripts (bash/zsh/fish/powershell)](https://mikefarah.gitbook.io/yq/v/v4.x/commands/shell-completion)
 - [Reduce](https://mikefarah.gitbook.io/yq/operators/reduce) to merge multiple files or sum an array or other fancy things.
 - [Github Action](https://mikefarah.gitbook.io/yq/usage/github-action) to use in your automated pipeline (thanks @devorbitus)
@@ -9,7 +9,7 @@ EOL

 testEmptyEval() {
  X=$(./yq e test.yml)
-  expected=$(cat test.yml)
+  expected="# comment"
  assertEquals 0 $?
  assertEquals "$expected" "$X"
 }
@@ -0,0 +1,43 @@
+#!/bin/bash
+
+setUp() {
+  rm test*.yml || true
+  
+}
+
+testLineCountFirstLineComment() {
+  cat >test.yml <<EOL
+#test123 
+abc: 123 
+test123: 123123 
+#comment 
+lalilu: lalilu
+EOL
+
+  X=$(./yq '.lalilu | line' --header-preprocess=false < test.yml)
+  assertEquals "5" "$X"
+}
+
+testArrayOfDocs() {
+  cat >test.yml <<EOL
+---
+# leading comment doc 1
+a: 1
+---
+# leading comment doc 2
+a: 2
+EOL
+
+read -r -d '' expected << EOM
+- # leading comment doc 1
+  a: 1
+- # leading comment doc 2
+  a: 2
+EOM
+
+  X=$(./yq ea '[.]' --header-preprocess=false < test.yml)
+  assertEquals "$expected" "$X"
+
+}
+
+source ./scripts/shunit2
@@ -63,6 +63,20 @@ EOM
  assertEquals "$expected" "$X"
 }

+testInputCSVUTF8() {
+  read -r -d '' expected << EOM
+- id: 1
+  first: john
+  last: smith
+- id: 1
+  first: jane
+  last: smith
+EOM
+
+  X=$(./yq -p=csv utf8.csv)
+  assertEquals "$expected" "$X"
+}
+
 testInputTSV() {
  cat >test.tsv <<EOL
 fruit	yumLevel
@@ -95,7 +109,7 @@ EOL
  read -r -d '' expected << EOM
 cat:
  +content: BiBi
-  +legs: "4"
+  +@legs: "4"
 EOM

  X=$(./yq e -p=xml test.yml)
@@ -113,10 +127,11 @@ testInputXmlNamespaces() {
 EOL

  read -r -d '' expected << EOM
+p_xml: version="1.0"
 map:
-  +xmlns: some-namespace
-  +xmlns:xsi: some-instance
-  +xsi:schemaLocation: some-url
+  +@xmlns: some-namespace
+  +@xmlns:xsi: some-instance
+  +@xsi:schemaLocation: some-url
 EOM

  X=$(./yq e -p=xml test.yml)
@@ -126,6 +141,26 @@ EOM
  assertEquals "$expected" "$X"
 }

+testInputXmlRoundtrip() {
+  cat >test.yml <<EOL
+<?xml version="1.0"?>
+<!DOCTYPE config SYSTEM "/etc/iwatch/iwatch.dtd" >
+<map xmlns="some-namespace" xmlns:xsi="some-instance" xsi:schemaLocation="some-url">Meow</map>
+EOL
+
+  read -r -d '' expected << EOM
+<?xml version="1.0"?>
+<!DOCTYPE config SYSTEM "/etc/iwatch/iwatch.dtd" >
+<map xmlns="some-namespace" xmlns:xsi="some-instance" xsi:schemaLocation="some-url">Meow</map>
+EOM
+
+  X=$(./yq -p=xml -o=xml test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea -p=xml -o=xml test.yml)
+  assertEquals "$expected" "$X"
+}
+

 testInputXmlStrict() {
  cat >test.yml <<EOL
@@ -139,11 +174,11 @@ testInputXmlStrict() {
 </root>
 EOL

-  X=$(./yq -p=xml --xml-strict-mode test.yml 2>&1)
+  X=$(./yq -p=xml --xml-strict-mode test.yml -o=xml 2>&1)
  assertEquals 1 $?
  assertEquals "Error: bad file 'test.yml': XML syntax error on line 7: invalid character entity &writer;" "$X"

-  X=$(./yq ea -p=xml --xml-strict-mode test.yml 2>&1)
+  X=$(./yq ea -p=xml --xml-strict-mode test.yml -o=xml 2>&1)
  assertEquals "Error: bad file 'test.yml': XML syntax error on line 7: invalid character entity &writer;" "$X"
 }

@@ -155,7 +190,7 @@ EOL
  read -r -d '' expected << EOM
 cat:
  +content: BiBi
-  +legs: "4"
+  +@legs: "4"
 EOM

  X=$(cat /dev/null | ./yq e -p=xml test.yml)
@@ -1,5 +1,8 @@
 #!/bin/bash

+
+# examples where header-preprocess is required
+
 setUp() {
  rm test*.yml || true
  cat >test.yml <<EOL
@@ -31,6 +34,24 @@ EOM
  assertEquals "$expected" "$X"
 }

+testLeadingSeperatorWithDirective() {
+  cat >test.yml <<EOL
+%YAML 1.1
+---
+this: should really work
+EOL
+
+  read -r -d '' expected << EOM
+%YAML 1.1
+---
+this: should really work
+EOM
+
+  X=$(./yq < test.yml)
+  assertEquals "$expected" "$X"
+}
+
+
 testLeadingSeperatorPipeIntoEvalSeq() {
  X=$(./yq e - < test.yml)
  expected=$(cat test.yml)
@@ -48,6 +48,55 @@ EOM
  assertEquals "$expected" "$X"
 }

+testOutputYamlRawDefault() {
+  cat >test.yml <<EOL
+a: "cat"
+EOL
+
+  X=$(./yq e  '.a' test.yml)
+  assertEquals "cat" "$X"
+
+  X=$(./yq ea '.a' test.yml)
+  assertEquals "cat" "$X"
+}
+
+testOutputYamlRawOff() {
+  cat >test.yml <<EOL
+a: "cat"
+EOL
+
+  X=$(./yq e -r=false '.a' test.yml)
+  assertEquals "\"cat\"" "$X"
+
+  X=$(./yq ea -r=false '.a' test.yml)
+  assertEquals "\"cat\"" "$X"
+}
+
+testOutputJsonRaw() {
+  cat >test.yml <<EOL
+a: cat
+EOL
+
+  X=$(./yq e -r --output-format=json '.a' test.yml)
+  assertEquals "cat" "$X"
+
+  X=$(./yq ea -r --output-format=json '.a' test.yml)
+  assertEquals "cat" "$X"
+}
+
+testOutputJsonDefault() {
+  cat >test.yml <<EOL
+a: cat
+EOL
+
+  X=$(./yq e --output-format=json '.a' test.yml)
+  assertEquals "\"cat\"" "$X"
+
+  X=$(./yq ea --output-format=json '.a' test.yml)
+  assertEquals "\"cat\"" "$X"
+}
+
+
 testOutputJsonShort() {
  cat >test.yml <<EOL
 a: {b: ["cat"]}
@@ -72,11 +121,11 @@ EOM

 testOutputProperties() {
  cat >test.yml <<EOL
-a: {b: {c: ["cat"]}}
+a: {b: {c: ["cat cat"]}}
 EOL

  read -r -d '' expected << EOM
-a.b.c.0 = cat
+a.b.c.0 = cat cat
 EOM

  X=$(./yq e --output-format=props test.yml)
@@ -86,13 +135,30 @@ EOM
  assertEquals "$expected" "$X"
 }

-testOutputPropertiesShort() {
+testOutputPropertiesDontUnwrap() {
  cat >test.yml <<EOL
-a: {b: {c: ["cat"]}}
+a: {b: {c: ["cat cat"]}}
 EOL

  read -r -d '' expected << EOM
-a.b.c.0 = cat
+a.b.c.0 = "cat cat"
+EOM
+
+  X=$(./yq e -r=false --output-format=props test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea -r=false --output-format=props test.yml)
+  assertEquals "$expected" "$X"
+}
+
+
+testOutputPropertiesShort() {
+  cat >test.yml <<EOL
+a: {b: {c: ["cat cat"]}}
+EOL
+
+  read -r -d '' expected << EOM
+a.b.c.0 = cat cat
 EOM

  X=$(./yq e -o=p test.yml)
@@ -186,7 +252,7 @@ EOM

 testOutputXmComplex() {
  cat >test.yml <<EOL
-a: {b: {c: ["cat", "dog"], +f: meow}}
+a: {b: {c: ["cat", "dog"], +@f: meow}}
 EOL

  read -r -d '' expected << EOM
@@ -12,6 +12,6 @@ outputs:
    description: "The complete result from the yq command being run"
 runs:
  using: 'docker'
-  image: 'github-action/Dockerfile'
+  image: 'docker://mikefarah/yq:4-githubaction'
  args:
    - ${{ inputs.cmd }}
@@ -1,19 +1,14 @@
 package cmd

-var leadingContentPreProcessing = true
-var unwrapScalar = true
+var unwrapScalarFlag = newUnwrapFlag()
+
+var unwrapScalar = false

 var writeInplace = false
 var outputToJSON = false
 var outputFormat = "yaml"
 var inputFormat = "yaml"

-var xmlAttributePrefix = "+"
-var xmlContentName = "+content"
-var xmlStrictMode = false
-var xmlKeepNamespace = true
-var xmlUseRawToken = true
-
 var exitStatus = false
 var forceColor = false
 var forceNoColor = false
@@ -75,7 +75,7 @@ func evaluateAll(cmd *cobra.Command, args []string) (cmdError error) {
 		return err
 	}

-	decoder, err := configureDecoder()
+	decoder, err := configureDecoder(true)
 	if err != nil {
 		return err
 	}
@@ -109,13 +109,13 @@ func evaluateAll(cmd *cobra.Command, args []string) (cmdError error) {
 	switch len(args) {
 	case 0:
 		if nullInput {
-			err = yqlib.NewStreamEvaluator().EvaluateNew(processExpression(expression), printer, "")
+			err = yqlib.NewStreamEvaluator().EvaluateNew(processExpression(expression), printer)
 		} else {
 			cmd.Println(cmd.UsageString())
 			return nil
 		}
 	default:
-		err = allAtOnceEvaluator.EvaluateFiles(processExpression(expression), args, printer, leadingContentPreProcessing, decoder)
+		err = allAtOnceEvaluator.EvaluateFiles(processExpression(expression), args, printer, decoder)
 	}

 	completedSuccessfully = err == nil
@@ -97,7 +97,7 @@ func evaluateSequence(cmd *cobra.Command, args []string) (cmdError error) {

 	printer := yqlib.NewPrinter(encoder, printerWriter)

-	decoder, err := configureDecoder()
+	decoder, err := configureDecoder(false)
 	if err != nil {
 		return err
 	}
@@ -123,13 +123,13 @@ func evaluateSequence(cmd *cobra.Command, args []string) (cmdError error) {
 	switch len(args) {
 	case 0:
 		if nullInput {
-			err = streamEvaluator.EvaluateNew(processExpression(expression), printer, "")
+			err = streamEvaluator.EvaluateNew(processExpression(expression), printer)
 		} else {
 			cmd.Println(cmd.UsageString())
 			return nil
 		}
 	default:
-		err = streamEvaluator.EvaluateFiles(processExpression(expression), args, printer, leadingContentPreProcessing, decoder)
+		err = streamEvaluator.EvaluateFiles(processExpression(expression), args, printer, decoder)
 	}
 	completedSuccessfully = err == nil

@@ -35,7 +35,7 @@ yq -P sample.json
 			return evaluateSequence(cmd, args)

 		},
-		PersistentPreRun: func(cmd *cobra.Command, args []string) {
+		PersistentPreRunE: func(cmd *cobra.Command, args []string) error {
 			cmd.SetOut(cmd.OutOrStdout())

 			var format = logging.MustStringFormatter(
@@ -47,14 +47,32 @@ yq -P sample.json
 			if verbose {
 				backend.SetLevel(logging.DEBUG, "")
 			} else {
-				backend.SetLevel(logging.ERROR, "")
+				backend.SetLevel(logging.WARNING, "")
 			}

 			logging.SetBackend(backend)
 			yqlib.InitExpressionParser()
-			yqlib.XMLPreferences.AttributePrefix = xmlAttributePrefix
-			yqlib.XMLPreferences.ContentName = xmlContentName
-			yqlib.XMLPreferences.StrictMode = xmlStrictMode
+
+			outputFormatType, err := yqlib.OutputFormatFromString(outputFormat)
+
+			if err != nil {
+				return err
+			}
+
+			if outputFormatType == yqlib.YamlOutputFormat ||
+				outputFormatType == yqlib.PropsOutputFormat {
+				unwrapScalar = true
+			}
+			if unwrapScalarFlag.IsExplicitySet() {
+				unwrapScalar = unwrapScalarFlag.IsSet()
+			}
+
+			//copy preference form global setting
+			yqlib.ConfiguredYamlPreferences.UnwrapScalar = unwrapScalar
+
+			yqlib.ConfiguredYamlPreferences.PrintDocSeparators = !noDocSeparators
+
+			return nil
 		},
 	}

@@ -69,11 +87,15 @@ yq -P sample.json
 	rootCmd.PersistentFlags().StringVarP(&outputFormat, "output-format", "o", "yaml", "[yaml|y|json|j|props|p|xml|x] output format type.")
 	rootCmd.PersistentFlags().StringVarP(&inputFormat, "input-format", "p", "yaml", "[yaml|y|props|p|xml|x] parse format for input. Note that json is a subset of yaml.")

-	rootCmd.PersistentFlags().StringVar(&xmlAttributePrefix, "xml-attribute-prefix", "+", "prefix for xml attributes")
-	rootCmd.PersistentFlags().StringVar(&xmlContentName, "xml-content-name", "+content", "name for xml content (if no attribute name is present).")
-	rootCmd.PersistentFlags().BoolVar(&xmlStrictMode, "xml-strict-mode", false, "enables strict parsing of XML. See https://pkg.go.dev/encoding/xml for more details.")
-	rootCmd.PersistentFlags().BoolVar(&xmlKeepNamespace, "xml-keep-namespace", true, "enables keeping namespace after parsing attributes")
-	rootCmd.PersistentFlags().BoolVar(&xmlUseRawToken, "xml-raw-token", true, "enables using RawToken method instead Token. Commonly disables namespace translations. See https://pkg.go.dev/encoding/xml#Decoder.RawToken for details.")
+	rootCmd.PersistentFlags().StringVar(&yqlib.ConfiguredXMLPreferences.AttributePrefix, "xml-attribute-prefix", yqlib.ConfiguredXMLPreferences.AttributePrefix, "prefix for xml attributes")
+	rootCmd.PersistentFlags().StringVar(&yqlib.ConfiguredXMLPreferences.ContentName, "xml-content-name", yqlib.ConfiguredXMLPreferences.ContentName, "name for xml content (if no attribute name is present).")
+	rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredXMLPreferences.StrictMode, "xml-strict-mode", yqlib.ConfiguredXMLPreferences.StrictMode, "enables strict parsing of XML. See https://pkg.go.dev/encoding/xml for more details.")
+	rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredXMLPreferences.KeepNamespace, "xml-keep-namespace", yqlib.ConfiguredXMLPreferences.KeepNamespace, "enables keeping namespace after parsing attributes")
+	rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredXMLPreferences.UseRawToken, "xml-raw-token", yqlib.ConfiguredXMLPreferences.UseRawToken, "enables using RawToken method instead Token. Commonly disables namespace translations. See https://pkg.go.dev/encoding/xml#Decoder.RawToken for details.")
+	rootCmd.PersistentFlags().StringVar(&yqlib.ConfiguredXMLPreferences.ProcInstPrefix, "xml-proc-inst-prefix", yqlib.ConfiguredXMLPreferences.ProcInstPrefix, "prefix for xml processing instructions (e.g. <?xml version=\"1\"?>)")
+	rootCmd.PersistentFlags().StringVar(&yqlib.ConfiguredXMLPreferences.DirectiveName, "xml-directive-name", yqlib.ConfiguredXMLPreferences.DirectiveName, "name for xml directives (e.g. <!DOCTYPE thing cat>)")
+	rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredXMLPreferences.SkipProcInst, "xml-skip-proc-inst", yqlib.ConfiguredXMLPreferences.SkipProcInst, "skip over process instructions (e.g. <?xml version=\"1\"?>)")
+	rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredXMLPreferences.SkipDirectives, "xml-skip-directives", yqlib.ConfiguredXMLPreferences.SkipDirectives, "skip over directives (e.g. <!DOCTYPE thing cat>)")

 	rootCmd.PersistentFlags().BoolVarP(&nullInput, "null-input", "n", false, "Don't read input, simply evaluate the expression given. Useful for creating docs from scratch.")
 	rootCmd.PersistentFlags().BoolVarP(&noDocSeparators, "no-doc", "N", false, "Don't print document separators (---)")
@@ -81,7 +103,9 @@ yq -P sample.json
 	rootCmd.PersistentFlags().IntVarP(&indent, "indent", "I", 2, "sets indent level for output")
 	rootCmd.Flags().BoolVarP(&version, "version", "V", false, "Print version information and quit")
 	rootCmd.PersistentFlags().BoolVarP(&writeInplace, "inplace", "i", false, "update the file inplace of first file given.")
-	rootCmd.PersistentFlags().BoolVarP(&unwrapScalar, "unwrapScalar", "r", true, "unwrap scalar, print the value with no quotes, colors or comments")
+	rootCmd.PersistentFlags().VarP(unwrapScalarFlag, "unwrapScalar", "r", "unwrap scalar, print the value with no quotes, colors or comments. Defaults to true for yaml")
+	rootCmd.PersistentFlags().Lookup("unwrapScalar").NoOptDefVal = "true"
+
 	rootCmd.PersistentFlags().BoolVarP(&prettyPrint, "prettyPrint", "P", false, "pretty print, shorthand for '... style = \"\"'")
 	rootCmd.PersistentFlags().BoolVarP(&exitStatus, "exit-status", "e", false, "set exit status if there are no matches or null or false is returned")

@@ -89,7 +113,7 @@ yq -P sample.json
 	rootCmd.PersistentFlags().BoolVarP(&forceNoColor, "no-colors", "M", false, "force print with no colors")
 	rootCmd.PersistentFlags().StringVarP(&frontMatter, "front-matter", "f", "", "(extract|process) first input as yaml front-matter. Extract will pull out the yaml content, process will run the expression against the yaml content, leaving the remaining data intact")
 	rootCmd.PersistentFlags().StringVarP(&forceExpression, "expression", "", "", "forcibly set the expression argument. Useful when yq argument detection thinks your expression is a file.")
-	rootCmd.PersistentFlags().BoolVarP(&leadingContentPreProcessing, "header-preprocess", "", true, "Slurp any header comments and separators before processing expression.")
+	rootCmd.PersistentFlags().BoolVarP(&yqlib.ConfiguredYamlPreferences.LeadingContentPreProcessing, "header-preprocess", "", true, "Slurp any header comments and separators before processing expression.")

 	rootCmd.PersistentFlags().StringVarP(&splitFileExp, "split-exp", "s", "", "print each result (or doc) into a file named (exp). [exp] argument must return a string. You can use $index in the expression as the result counter.")
 	rootCmd.PersistentFlags().StringVarP(&splitFileExpFile, "split-exp-file", "", "", "Use a file to specify the split-exp expression.")
@@ -42,7 +42,7 @@ $ yq shell-completion fish > ~/.config/fish/completions/yq.fish
 `,
 	DisableFlagsInUseLine: true,
 	ValidArgs:             []string{"bash", "zsh", "fish", "powershell"},
-	Args:                  cobra.ExactValidArgs(1),
+	Args:                  cobra.MatchAll(cobra.ExactArgs(1), cobra.OnlyValidArgs),
 	RunE: func(cmd *cobra.Command, args []string) error {
 		var err error = nil
 		switch args[0] {
@@ -0,0 +1,46 @@
+package cmd
+
+import (
+	"strconv"
+
+	"github.com/spf13/pflag"
+)
+
+type boolFlag interface {
+	pflag.Value
+	IsExplicitySet() bool
+	IsSet() bool
+}
+
+type unwrapScalarFlagStrc struct {
+	explicitySet bool
+	value        bool
+}
+
+func newUnwrapFlag() boolFlag {
+	return &unwrapScalarFlagStrc{value: true}
+}
+
+func (f *unwrapScalarFlagStrc) IsExplicitySet() bool {
+	return f.explicitySet
+}
+
+func (f *unwrapScalarFlagStrc) IsSet() bool {
+	return f.value
+}
+
+func (f *unwrapScalarFlagStrc) String() string {
+	return strconv.FormatBool(f.value)
+}
+
+func (f *unwrapScalarFlagStrc) Set(value string) error {
+
+	v, err := strconv.ParseBool(value)
+	f.value = v
+	f.explicitySet = true
+	return err
+}
+
+func (*unwrapScalarFlagStrc) Type() string {
+	return "bool"
+}
@@ -56,14 +56,14 @@ func initCommand(cmd *cobra.Command, args []string) (string, []string, error) {
 	return expression, args, nil
 }

-func configureDecoder() (yqlib.Decoder, error) {
+func configureDecoder(evaluateTogether bool) (yqlib.Decoder, error) {
 	yqlibInputFormat, err := yqlib.InputFormatFromString(inputFormat)
 	if err != nil {
 		return nil, err
 	}
 	switch yqlibInputFormat {
 	case yqlib.XMLInputFormat:
-		return yqlib.NewXMLDecoder(xmlAttributePrefix, xmlContentName, xmlStrictMode, xmlKeepNamespace, xmlUseRawToken), nil
+		return yqlib.NewXMLDecoder(yqlib.ConfiguredXMLPreferences), nil
 	case yqlib.PropertiesInputFormat:
 		return yqlib.NewPropertiesDecoder(), nil
 	case yqlib.JsonInputFormat:
@@ -73,8 +73,9 @@ func configureDecoder() (yqlib.Decoder, error) {
 	case yqlib.TSVObjectInputFormat:
 		return yqlib.NewCSVObjectDecoder('\t'), nil
 	}
-
-	return yqlib.NewYamlDecoder(), nil
+	prefs := yqlib.ConfiguredYamlPreferences
+	prefs.EvaluateTogether = evaluateTogether
+	return yqlib.NewYamlDecoder(prefs), nil
 }

 func configurePrinterWriter(format yqlib.PrinterOutputFormat, out io.Writer) (yqlib.PrinterWriter, error) {
@@ -97,7 +98,7 @@ func configurePrinterWriter(format yqlib.PrinterOutputFormat, out io.Writer) (yq
 func configureEncoder(format yqlib.PrinterOutputFormat) yqlib.Encoder {
 	switch format {
 	case yqlib.JSONOutputFormat:
-		return yqlib.NewJSONEncoder(indent, colorsEnabled)
+		return yqlib.NewJSONEncoder(indent, colorsEnabled, unwrapScalar)
 	case yqlib.PropsOutputFormat:
 		return yqlib.NewPropertiesEncoder(unwrapScalar)
 	case yqlib.CSVOutputFormat:
@@ -105,9 +106,9 @@ func configureEncoder(format yqlib.PrinterOutputFormat) yqlib.Encoder {
 	case yqlib.TSVOutputFormat:
 		return yqlib.NewCsvEncoder('\t')
 	case yqlib.YamlOutputFormat:
-		return yqlib.NewYamlEncoder(indent, colorsEnabled, !noDocSeparators, unwrapScalar)
+		return yqlib.NewYamlEncoder(indent, colorsEnabled, yqlib.ConfiguredYamlPreferences)
 	case yqlib.XMLOutputFormat:
-		return yqlib.NewXMLEncoder(indent, xmlAttributePrefix, xmlContentName)
+		return yqlib.NewXMLEncoder(indent, yqlib.ConfiguredXMLPreferences)
 	}
 	panic("invalid encoder")
 }
@@ -11,7 +11,7 @@ var (
 	GitDescribe string

 	// Version is main version number that is being run at the moment.
-	Version = "4.26.1"
+	Version = "v4.30.5"

 	// VersionPrerelease is a pre-release marker for the version. If this is "" (empty string)
 	// then it means that it is a final release. Otherwise, this is a pre-release
@@ -1,11 +1,2 @@
---
- become: true
-  gather_facts: false
-  hosts: lalaland
-  name: "Apply smth"
-  roles:
-    - lala
-    - land
-  serial: 1
- become: false
-  gather_facts: true
+- [cat, dog, frog, cow]
+- [apple, banana, grape, mango]
@@ -1,12 +1 @@
-# --------------------------------------------------
-# It's a test with comment
-# --------------------------------------------------
-test:
-  - name: a
-  - name: b
-  - name: c
-groups:
-  - name: a
-  - name: b
-  - name: c
-  - name: c
+["foobar", "foobaz", "blarp"]
@@ -1,14 +1,5 @@
-<!-- before cat -->
-<cat>
-	<!-- in cat before -->
-	<x>3<!-- multi
-line comment
-for x --></x>
-	<y>
-		<!-- in y before -->
-		<d><!-- in d before -->4<!-- in d after --></d>
-		<!-- in y after -->
-	</y>
-	<!-- in_cat_after -->
-</cat>
-<!-- after cat -->
+date: "2022-11-25"
+raw:
+  id: 1
+  XML: text_outside_1<Tag1 />text_outside_2<Tag2Kling Klong</Tag2>text_outside_3
+  time: 09:19:00
@@ -1,3 +1,3 @@
 name,numberOfCats,likesApples,height
-Gary,1,true,168.8
+,1,true,168.8
 Samantha's Rabbit,2,false,-188.8
@@ -0,0 +1,4 @@
+---
+# comment
+# about things
+a: cat
@@ -2,7 +2,8 @@ FROM mikefarah/yq:4

 COPY entrypoint.sh /entrypoint.sh

-# this seems to be the default user in github actions
-USER 1001
+# github action recommendation is to run as root.
+# https://docs.github.com/en/actions/creating-actions/dockerfile-support-for-github-actions#user
+USER root

 ENTRYPOINT ["/entrypoint.sh"]
@@ -2,8 +2,11 @@
 set -e
 echo "::debug::\$cmd: $1"
 RESULT=$(eval "$1")
-RESULT="${RESULT//'%'/'%25'}"
-RESULT="${RESULT//$'\n'/'%0A'}"
-RESULT="${RESULT//$'\r'/'%0D'}"
 echo "::debug::\$RESULT: $RESULT"
-echo ::set-output name=result::"$RESULT"
+# updating from 
+# https://github.com/orgs/community/discussions/26288#discussioncomment-3876281
+# https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#setting-an-output-parameter
+delimiter=$(cat /proc/sys/kernel/random/uuid)
+echo "result<<${delimiter}" >> "${GITHUB_OUTPUT}"
+echo "${RESULT}" >> "${GITHUB_OUTPUT}"
+echo "${delimiter}" >> "${GITHUB_OUTPUT}"
@@ -2,29 +2,30 @@ module github.com/mikefarah/yq/v4

 require (
 	github.com/a8m/envsubst v1.3.0
-	github.com/alecthomas/participle/v2 v2.0.0-beta.4
-	github.com/alecthomas/repr v0.1.0
-	github.com/elliotchance/orderedmap v1.4.0
+	github.com/alecthomas/participle/v2 v2.0.0-beta.5
+	github.com/alecthomas/repr v0.1.1
+	github.com/dimchansky/utfbom v1.1.1
+	github.com/elliotchance/orderedmap v1.5.0
 	github.com/fatih/color v1.13.0
-	github.com/goccy/go-json v0.9.10
-	github.com/goccy/go-yaml v1.9.5
+	github.com/goccy/go-json v0.9.11
+	github.com/goccy/go-yaml v1.9.6
 	github.com/jinzhu/copier v0.3.5
 	github.com/magiconair/properties v1.8.6
 	github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e
-	github.com/spf13/cobra v1.5.0
+	github.com/spf13/cobra v1.6.1
+	github.com/spf13/pflag v1.0.5
 	golang.org/x/net v0.0.0-20220708220712-1185a9018129
 	gopkg.in/op/go-logging.v1 v1.0.0-20160211212156-b2cb9fa56473
 	gopkg.in/yaml.v3 v3.0.1
 )

 require (
-	github.com/inconshreveable/mousetrap v1.0.0 // indirect
+	github.com/inconshreveable/mousetrap v1.0.1 // indirect
 	github.com/mattn/go-colorable v0.1.12 // indirect
 	github.com/mattn/go-isatty v0.0.14 // indirect
-	github.com/spf13/pflag v1.0.5 // indirect
 	golang.org/x/sys v0.0.0-20220712014510-0a85c31ab51e // indirect
 	golang.org/x/text v0.3.7 // indirect
 	golang.org/x/xerrors v0.0.0-20220609144429-65e65417b02f // indirect
 )

-go 1.18
+go 1.19
@@ -1,15 +1,17 @@
 github.com/a8m/envsubst v1.3.0 h1:GmXKmVssap0YtlU3E230W98RWtWCyIZzjtf1apWWyAg=
 github.com/a8m/envsubst v1.3.0/go.mod h1:MVUTQNGQ3tsjOOtKCNd+fl8RzhsXcDvvAEzkhGtlsbY=
 github.com/alecthomas/assert/v2 v2.0.3 h1:WKqJODfOiQG0nEJKFKzDIG3E29CN2/4zR9XGJzKIkbg=
-github.com/alecthomas/participle/v2 v2.0.0-beta.4 h1:ublfGBm+x+p2j7KotHhrUMbKtejT7M0Gv1Mt1u3absw=
-github.com/alecthomas/participle/v2 v2.0.0-beta.4/go.mod h1:RC764t6n4L8D8ITAJv0qdokritYSNR3wV5cVwmIEaMM=
-github.com/alecthomas/repr v0.1.0 h1:ENn2e1+J3k09gyj2shc0dHr/yjaWSHRlrJ4DPMevDqE=
-github.com/alecthomas/repr v0.1.0/go.mod h1:2kn6fqh/zIyPLmm3ugklbEi5hg5wS435eygvNfaDQL8=
+github.com/alecthomas/participle/v2 v2.0.0-beta.5 h1:y6dsSYVb1G5eK6mgmy+BgI3Mw35a3WghArZ/Hbebrjo=
+github.com/alecthomas/participle/v2 v2.0.0-beta.5/go.mod h1:RC764t6n4L8D8ITAJv0qdokritYSNR3wV5cVwmIEaMM=
+github.com/alecthomas/repr v0.1.1 h1:87P60cSmareLAxMc4Hro0r2RBY4ROm0dYwkJNpS4pPs=
+github.com/alecthomas/repr v0.1.1/go.mod h1:Fr0507jx4eOXV7AlPV6AVZLYrLIuIeSOWtW57eE/O/4=
 github.com/cpuguy83/go-md2man/v2 v2.0.2/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
 github.com/davecgh/go-spew v1.1.0 h1:ZDRjVQ15GmhC3fiQ8ni8+OwkZQO4DARzQgrnXU1Liz8=
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
-github.com/elliotchance/orderedmap v1.4.0 h1:wZtfeEONCbx6in1CZyE6bELEt/vFayMvsxqI5SgsR+A=
-github.com/elliotchance/orderedmap v1.4.0/go.mod h1:wsDwEaX5jEoyhbs7x93zk2H/qv0zwuhg4inXhDkYqys=
+github.com/dimchansky/utfbom v1.1.1 h1:vV6w1AhK4VMnhBno/TPVCoK9U/LP0PkLCS9tbxHdi/U=
+github.com/dimchansky/utfbom v1.1.1/go.mod h1:SxdoEBH5qIqFocHMyGOXVAybYJdr71b1Q/j0mACtrfE=
+github.com/elliotchance/orderedmap v1.5.0 h1:1IsExUsjv5XNBD3ZdC7jkAAqLWOOKdbPTmkHx63OsBg=
+github.com/elliotchance/orderedmap v1.5.0/go.mod h1:wsDwEaX5jEoyhbs7x93zk2H/qv0zwuhg4inXhDkYqys=
 github.com/fatih/color v1.10.0/go.mod h1:ELkj/draVOlAH/xkhN6mQ50Qd0MPOk5AAr3maGEBuJM=
 github.com/fatih/color v1.13.0 h1:8LOYc1KYPPmyKMuN8QV2DNRWNbLo6LZ0iLs8+mlH53w=
 github.com/fatih/color v1.13.0/go.mod h1:kLAiJbzzSOZDVNGyDpeOxJ47H46qBXwg5ILebYFFOfk=
@@ -17,13 +19,13 @@ github.com/go-playground/assert/v2 v2.0.1/go.mod h1:VDjEfimB/XKnb+ZQfWdccd7VUvSc
 github.com/go-playground/locales v0.13.0/go.mod h1:taPMhCMXrRLJO55olJkUXHZBHCxTMfnGwq/HNwmWNS8=
 github.com/go-playground/universal-translator v0.17.0/go.mod h1:UkSxE5sNxxRwHyU+Scu5vgOQjsIJAF8j9muTVoKLVtA=
 github.com/go-playground/validator/v10 v10.4.1/go.mod h1:nlOn6nFhuKACm19sB/8EGNn9GlaMV7XkbRSipzJ0Ii4=
-github.com/goccy/go-json v0.9.10 h1:hCeNmprSNLB8B8vQKWl6DpuH0t60oEs+TAk9a7CScKc=
-github.com/goccy/go-json v0.9.10/go.mod h1:6MelG93GURQebXPDq3khkgXZkazVtN9CRI+MGFi0w8I=
-github.com/goccy/go-yaml v1.9.5 h1:Eh/+3uk9kLxG4koCX6lRMAPS1OaMSAi+FJcya0INdB0=
-github.com/goccy/go-yaml v1.9.5/go.mod h1:U/jl18uSupI5rdI2jmuCswEA2htH9eXfferR3KfscvA=
+github.com/goccy/go-json v0.9.11 h1:/pAaQDLHEoCq/5FFmSKBswWmK6H0e8g4159Kc/X/nqk=
+github.com/goccy/go-json v0.9.11/go.mod h1:6MelG93GURQebXPDq3khkgXZkazVtN9CRI+MGFi0w8I=
+github.com/goccy/go-yaml v1.9.6 h1:KhAu1zf9JXnm3vbG49aDE0E5uEBUsM4uwD31/58ZWyI=
+github.com/goccy/go-yaml v1.9.6/go.mod h1:JubOolP3gh0HpiBc4BLRD4YmjEjHAmIIB2aaXKkTfoE=
 github.com/hexops/gotextdiff v1.0.3 h1:gitA9+qJrrTCsiCl7+kh75nPqQt1cx4ZkudSTLoUqJM=
-github.com/inconshreveable/mousetrap v1.0.0 h1:Z8tu5sraLXCXIcARxBp/8cbvlwVa7Z1NHg9XEKhtSvM=
-github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
+github.com/inconshreveable/mousetrap v1.0.1 h1:U3uMjPSQEBMNp1lFxmllqCPM6P5u/Xq7Pgzkat/bFNc=
+github.com/inconshreveable/mousetrap v1.0.1/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
 github.com/jinzhu/copier v0.3.5 h1:GlvfUwHk62RokgqVNvYsku0TATCF7bAHVwEXoBh3iJg=
 github.com/jinzhu/copier v0.3.5/go.mod h1:DfbEm0FYsaqBcKcFuvmOZb218JkPGtvSHsKg8S8hyyg=
 github.com/leodido/go-urn v1.2.0/go.mod h1:+8+nEpDfqqsY+g338gtMEUOtuK+4dEMhiQEgxpxOKII=
@@ -41,8 +43,8 @@ github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e/go.mod h1:pJLUxLENpZxwdsK
 github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
 github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
-github.com/spf13/cobra v1.5.0 h1:X+jTBEBqF0bHN+9cSMgmfuvv2VHJ9ezmFNf9Y/XstYU=
-github.com/spf13/cobra v1.5.0/go.mod h1:dWXEIy2H428czQCjInthrTRUg7yKbok+2Qi/yBIJoUM=
+github.com/spf13/cobra v1.6.1 h1:o94oiPyS4KD1mPy2fmcYYHHfCxLqYjJOhGsCHFZtEzA=
+github.com/spf13/cobra v1.6.1/go.mod h1:IOw/AERYS7UzyrGinqmz6HLUo219MORXGxhbaJUqzrY=
 github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
 github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
 github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
@@ -60,6 +62,7 @@ golang.org/x/sys v0.0.0-20200116001909-b77594299b42/go.mod h1:h1NjWce9XRLGQEsW7w
 golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20210927094055-39ccf1dd6fa6/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20220406163625-3f8b81556e12/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20220712014510-0a85c31ab51e h1:NHvCuwuS43lGnYhten69ZWqi2QOj/CiDNcKbVqwVoew=
 golang.org/x/sys v0.0.0-20220712014510-0a85c31ab51e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
@@ -75,7 +78,6 @@ gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8
 gopkg.in/op/go-logging.v1 v1.0.0-20160211212156-b2cb9fa56473 h1:6D+BvnJ/j6e222UW8s2qTSe3wGBtvo0MbVQG/c5k8RE=
 gopkg.in/op/go-logging.v1 v1.0.0-20160211212156-b2cb9fa56473/go.mod h1:N1eN2tsCx0Ydtgjl4cqmbRCsY4/+z4cYDeqwZTk6zog=
 gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
-gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
 gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
 gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
 gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
@@ -37,7 +37,7 @@ The `=` operator then pipes the 'root' context through the `rhs` expression of `
 dog
 ```

-Both sides have now been evaluated, so now the operator copies across the value from the RHS (`.b`) to the the LHS (`.a`), and it returns the now updated context:
+Both sides have now been evaluated, so now the operator copies across the value from the RHS (`.b`) to the LHS (`.a`), and it returns the now updated context:

 ```yaml
 a: dog
@@ -8,7 +8,7 @@ import (

 // A yaml expression evaluator that runs the expression once against all files/nodes in memory.
 type Evaluator interface {
-	EvaluateFiles(expression string, filenames []string, printer Printer, leadingContentPreProcessing bool, decoder Decoder) error
+	EvaluateFiles(expression string, filenames []string, printer Printer, decoder Decoder) error

 	// EvaluateNodes takes an expression and one or more yaml nodes, returning a list of matching candidate nodes
 	EvaluateNodes(expression string, nodes ...*yaml.Node) (*list.List, error)
@@ -46,21 +46,16 @@ func (e *allAtOnceEvaluator) EvaluateCandidateNodes(expression string, inputCand
 	return context.MatchingNodes, nil
 }

-func (e *allAtOnceEvaluator) EvaluateFiles(expression string, filenames []string, printer Printer, leadingContentPreProcessing bool, decoder Decoder) error {
+func (e *allAtOnceEvaluator) EvaluateFiles(expression string, filenames []string, printer Printer, decoder Decoder) error {
 	fileIndex := 0
-	firstFileLeadingContent := ""

 	var allDocuments = list.New()
 	for _, filename := range filenames {
-		reader, leadingContent, err := readStream(filename, fileIndex == 0 && leadingContentPreProcessing)
+		reader, err := readStream(filename)
 		if err != nil {
 			return err
 		}

-		if fileIndex == 0 {
-			firstFileLeadingContent = leadingContent
-		}
-
 		fileDocuments, err := readDocuments(reader, filename, fileIndex, decoder)
 		if err != nil {
 			return err
@@ -75,11 +70,9 @@ func (e *allAtOnceEvaluator) EvaluateFiles(expression string, filenames []string
 			Filename:       "",
 			Node:           &yaml.Node{Kind: yaml.DocumentNode, Content: []*yaml.Node{{Tag: "!!null", Kind: yaml.ScalarNode}}},
 			FileIndex:      0,
-			LeadingContent: firstFileLeadingContent,
+			LeadingContent: "",
 		}
 		allDocuments.PushBack(candidateNode)
-	} else {
-		allDocuments.Front().Value.(*CandidateNode).LeadingContent = firstFileLeadingContent
 	}

 	matches, err := e.EvaluateCandidateNodes(expression, allDocuments)
@@ -157,7 +157,7 @@ func (n *CandidateNode) UpdateAttributesFrom(other *CandidateNode, prefs assignP
 	n.Node.Kind = other.Node.Kind

 	// don't clobber custom tags...
-	if strings.HasPrefix(n.Node.Tag, "!!") || n.Node.Tag == "" {
+	if prefs.ClobberCustomTags || strings.HasPrefix(n.Node.Tag, "!!") || n.Node.Tag == "" {
 		n.Node.Tag = other.Node.Tag
 	}

@@ -16,7 +16,7 @@ type compareTypePref struct {
 func compareOperator(d *dataTreeNavigator, context Context, expressionNode *ExpressionNode) (Context, error) {
 	log.Debugf("-- compareOperator")
 	prefs := expressionNode.Operation.Preferences.(compareTypePref)
-	return crossFunction(d, context.ReadOnlyClone(), expressionNode, compare(prefs), true)
+	return crossFunction(d, context, expressionNode, compare(prefs), true)
 }

 func compare(prefs compareTypePref) func(d *dataTreeNavigator, context Context, lhs *CandidateNode, rhs *CandidateNode) (*CandidateNode, error) {
@@ -54,12 +54,12 @@ func compare(prefs compareTypePref) func(d *dataTreeNavigator, context Context,
 }

 func compareDateTime(layout string, prefs compareTypePref, lhs *yaml.Node, rhs *yaml.Node) (bool, error) {
-	lhsTime, err := time.Parse(layout, lhs.Value)
+	lhsTime, err := parseDateTime(layout, lhs.Value)
 	if err != nil {
 		return false, err
 	}

-	rhsTime, err := time.Parse(layout, rhs.Value)
+	rhsTime, err := parseDateTime(layout, rhs.Value)
 	if err != nil {
 		return false, err
 	}
@@ -81,7 +81,7 @@ func compareScalars(context Context, prefs compareTypePref, lhs *yaml.Node, rhs
 	isDateTime := lhs.Tag == "!!timestamp"
 	// if the lhs is a string, it might be a timestamp in a custom format.
 	if lhsTag == "!!str" && context.GetDateTimeLayout() != time.RFC3339 {
-		_, err := time.Parse(context.GetDateTimeLayout(), lhs.Value)
+		_, err := parseDateTime(context.GetDateTimeLayout(), lhs.Value)
 		isDateTime = err == nil
 	}
 	if isDateTime {
@@ -127,6 +127,10 @@ func compareScalars(context Context, prefs compareTypePref, lhs *yaml.Node, rhs
 			return lhs.Value > rhs.Value, nil
 		}
 		return lhs.Value < rhs.Value, nil
+	} else if lhsTag == "!!null" && rhsTag == "!!null" && prefs.OrEqual {
+		return true, nil
+	} else if lhsTag == "!!null" || rhsTag == "!!null" {
+		return false, nil
 	}

 	return false, fmt.Errorf("%v not yet supported for comparison", lhs.Tag)
@@ -14,6 +14,13 @@ var compareOperatorScenarios = []expressionScenario{
 			"D0, P[a], (!!bool)::true\n",
 		},
 	},
+	{
+		skipDoc:    true,
+		expression: "(.k | length) >= 0",
+		expected: []string{
+			"D0, P[k], (!!bool)::true\n",
+		},
+	},
 	{
 		skipDoc:    true,
 		document:   "a: 5\nb: 4",
@@ -285,28 +292,28 @@ var compareOperatorScenarios = []expressionScenario{
 		description: "Both sides are null: > is false",
 		expression:  ".a > .b",
 		expected: []string{
-			"D0, P[], (!!bool)::false\n",
+			"D0, P[a], (!!bool)::false\n",
 		},
 	},
 	{
 		skipDoc:    true,
 		expression: ".a < .b",
 		expected: []string{
-			"D0, P[], (!!bool)::false\n",
+			"D0, P[a], (!!bool)::false\n",
 		},
 	},
 	{
 		description: "Both sides are null: >= is true",
 		expression:  ".a >= .b",
 		expected: []string{
-			"D0, P[], (!!bool)::true\n",
+			"D0, P[a], (!!bool)::true\n",
 		},
 	},
 	{
 		skipDoc:    true,
 		expression: ".a <= .b",
 		expected: []string{
-			"D0, P[], (!!bool)::true\n",
+			"D0, P[a], (!!bool)::true\n",
 		},
 	},

@@ -350,7 +357,7 @@ var compareOperatorScenarios = []expressionScenario{
 		document:   `a: 5`,
 		expression: ".b <= .a",
 		expected: []string{
-			"D0, P[a], (!!bool)::false\n",
+			"D0, P[b], (!!bool)::false\n",
 		},
 	},
 	{
@@ -358,7 +365,7 @@ var compareOperatorScenarios = []expressionScenario{
 		document:   `a: 5`,
 		expression: ".b < .a",
 		expected: []string{
-			"D0, P[a], (!!bool)::false\n",
+			"D0, P[b], (!!bool)::false\n",
 		},
 	},
 }
@@ -12,7 +12,9 @@ const csvSimple = `name,numberOfCats,likesApples,height
 Gary,1,true,168.8
 Samantha's Rabbit,2,false,-188.8
 `
-
+const csvMissing = `name,numberOfCats,likesApples,height
+,null,,168.8
+`
 const expectedUpdatedSimpleCsv = `name,numberOfCats,likesApples,height
 Gary,3,true,168.8
 Samantha's Rabbit,2,false,-188.8
@@ -110,6 +112,13 @@ var csvScenarios = []formatScenario{
 		expected:       csvSimpleMissingData,
 		scenarioType:   "encode-csv",
 	},
+	{
+		description:  "decode csv missing",
+		skipDoc:      true,
+		input:        csvMissing,
+		expected:     csvMissing,
+		scenarioType: "roundtrip-csv",
+	},
 	{
 		description:    "Parse CSV into an array of objects",
 		subdescription: "First row is assumed to be the header row.",
@@ -136,21 +145,21 @@ var csvScenarios = []formatScenario{
 func testCSVScenario(t *testing.T, s formatScenario) {
 	switch s.scenarioType {
 	case "encode-csv":
-		test.AssertResultWithContext(t, s.expected, processFormatScenario(s, NewYamlDecoder(), NewCsvEncoder(',')), s.description)
+		test.AssertResultWithContext(t, s.expected, mustProcessFormatScenario(s, NewYamlDecoder(ConfiguredYamlPreferences), NewCsvEncoder(',')), s.description)
 	case "encode-tsv":
-		test.AssertResultWithContext(t, s.expected, processFormatScenario(s, NewYamlDecoder(), NewCsvEncoder('\t')), s.description)
+		test.AssertResultWithContext(t, s.expected, mustProcessFormatScenario(s, NewYamlDecoder(ConfiguredYamlPreferences), NewCsvEncoder('\t')), s.description)
 	case "decode-csv-object":
-		test.AssertResultWithContext(t, s.expected, processFormatScenario(s, NewCSVObjectDecoder(','), NewYamlEncoder(2, false, true, true)), s.description)
+		test.AssertResultWithContext(t, s.expected, mustProcessFormatScenario(s, NewCSVObjectDecoder(','), NewYamlEncoder(2, false, ConfiguredYamlPreferences)), s.description)
 	case "decode-tsv-object":
-		test.AssertResultWithContext(t, s.expected, processFormatScenario(s, NewCSVObjectDecoder('\t'), NewYamlEncoder(2, false, true, true)), s.description)
+		test.AssertResultWithContext(t, s.expected, mustProcessFormatScenario(s, NewCSVObjectDecoder('\t'), NewYamlEncoder(2, false, ConfiguredYamlPreferences)), s.description)
 	case "roundtrip-csv":
-		test.AssertResultWithContext(t, s.expected, processFormatScenario(s, NewCSVObjectDecoder(','), NewCsvEncoder(',')), s.description)
+		test.AssertResultWithContext(t, s.expected, mustProcessFormatScenario(s, NewCSVObjectDecoder(','), NewCsvEncoder(',')), s.description)
 	default:
 		panic(fmt.Sprintf("unhandled scenario type %q", s.scenarioType))
 	}
 }

-func documentCSVDecodeObjectScenario(t *testing.T, w *bufio.Writer, s formatScenario, formatType string) {
+func documentCSVDecodeObjectScenario(w *bufio.Writer, s formatScenario, formatType string) {
 	writeOrPanic(w, fmt.Sprintf("## %v\n", s.description))

 	if s.subdescription != "" {
@@ -171,7 +180,7 @@ func documentCSVDecodeObjectScenario(t *testing.T, w *bufio.Writer, s formatScen
 	}

 	writeOrPanic(w, fmt.Sprintf("```yaml\n%v```\n\n",
-		processFormatScenario(s, NewCSVObjectDecoder(separator), NewYamlEncoder(s.indent, false, true, true))),
+		mustProcessFormatScenario(s, NewCSVObjectDecoder(separator), NewYamlEncoder(s.indent, false, ConfiguredYamlPreferences))),
 	)
 }

@@ -203,7 +212,7 @@ func documentCSVEncodeScenario(w *bufio.Writer, s formatScenario, formatType str
 	}

 	writeOrPanic(w, fmt.Sprintf("```%v\n%v```\n\n", formatType,
-		processFormatScenario(s, NewYamlDecoder(), NewCsvEncoder(separator))),
+		mustProcessFormatScenario(s, NewYamlDecoder(ConfiguredYamlPreferences), NewCsvEncoder(separator))),
 	)
 }

@@ -235,7 +244,7 @@ func documentCSVRoundTripScenario(w *bufio.Writer, s formatScenario, formatType
 	}

 	writeOrPanic(w, fmt.Sprintf("```%v\n%v```\n\n", formatType,
-		processFormatScenario(s, NewCSVObjectDecoder(separator), NewCsvEncoder(separator))),
+		mustProcessFormatScenario(s, NewCSVObjectDecoder(separator), NewCsvEncoder(separator))),
 	)
 }

@@ -250,9 +259,9 @@ func documentCSVScenario(t *testing.T, w *bufio.Writer, i interface{}) {
 	case "encode-tsv":
 		documentCSVEncodeScenario(w, s, "tsv")
 	case "decode-csv-object":
-		documentCSVDecodeObjectScenario(t, w, s, "csv")
+		documentCSVDecodeObjectScenario(w, s, "csv")
 	case "decode-tsv-object":
-		documentCSVDecodeObjectScenario(t, w, s, "tsv")
+		documentCSVDecodeObjectScenario(w, s, "tsv")
 	case "roundtrip-csv":
 		documentCSVRoundTripScenario(w, s, "csv")

@@ -3,8 +3,6 @@ package yqlib
 import (
 	"fmt"
 	"io"
-
-	yaml "gopkg.in/yaml.v3"
 )

 type InputFormat uint
@@ -20,8 +18,8 @@ const (
 )

 type Decoder interface {
-	Init(reader io.Reader)
-	Decode(node *yaml.Node) error
+	Init(reader io.Reader) error
+	Decode() (*CandidateNode, error)
 }

 func InputFormatFromString(format string) (InputFormat, error) {
@@ -19,21 +19,22 @@ func NewBase64Decoder() Decoder {
 	return &base64Decoder{finished: false, encoding: *base64.StdEncoding}
 }

-func (dec *base64Decoder) Init(reader io.Reader) {
+func (dec *base64Decoder) Init(reader io.Reader) error {
 	dec.reader = reader
 	dec.readAnything = false
 	dec.finished = false
+	return nil
 }

-func (dec *base64Decoder) Decode(rootYamlNode *yaml.Node) error {
+func (dec *base64Decoder) Decode() (*CandidateNode, error) {
 	if dec.finished {
-		return io.EOF
+		return nil, io.EOF
 	}
 	base64Reader := base64.NewDecoder(&dec.encoding, dec.reader)
 	buf := new(bytes.Buffer)

 	if _, err := buf.ReadFrom(base64Reader); err != nil {
-		return err
+		return nil, err
 	}
 	if buf.Len() == 0 {
 		dec.finished = true
@@ -42,12 +43,15 @@ func (dec *base64Decoder) Decode(rootYamlNode *yaml.Node) error {
 		// otherwise if we've already read some bytes, and now we get
 		// an empty string, then we are done.
 		if dec.readAnything {
-			return io.EOF
+			return nil, io.EOF
 		}
 	}
 	dec.readAnything = true
-	rootYamlNode.Kind = yaml.ScalarNode
-	rootYamlNode.Tag = "!!str"
-	rootYamlNode.Value = buf.String()
-	return nil
+	return &CandidateNode{
+		Node: &yaml.Node{
+			Kind:  yaml.ScalarNode,
+			Tag:   "!!str",
+			Value: buf.String(),
+		},
+	}, nil
 }
@@ -5,6 +5,7 @@ import (
 	"errors"
 	"io"

+	"github.com/dimchansky/utfbom"
 	yaml "gopkg.in/yaml.v3"
 )

@@ -18,10 +19,13 @@ func NewCSVObjectDecoder(separator rune) Decoder {
 	return &csvObjectDecoder{separator: separator}
 }

-func (dec *csvObjectDecoder) Init(reader io.Reader) {
-	dec.reader = *csv.NewReader(reader)
+func (dec *csvObjectDecoder) Init(reader io.Reader) error {
+	cleanReader, enc := utfbom.Skip(reader)
+	log.Debugf("Detected encoding: %s\n", enc)
+	dec.reader = *csv.NewReader(cleanReader)
 	dec.reader.Comma = dec.separator
 	dec.finished = false
+	return nil
 }

 func (dec *csvObjectDecoder) convertToYamlNode(content string) *yaml.Node {
@@ -44,14 +48,14 @@ func (dec *csvObjectDecoder) createObject(headerRow []string, contentRow []strin
 	return objectNode
 }

-func (dec *csvObjectDecoder) Decode(rootYamlNode *yaml.Node) error {
+func (dec *csvObjectDecoder) Decode() (*CandidateNode, error) {
 	if dec.finished {
-		return io.EOF
+		return nil, io.EOF
 	}
 	headerRow, err := dec.reader.Read()
 	log.Debugf(": headerRow%v", headerRow)
 	if err != nil {
-		return err
+		return nil, err
 	}

 	rootArray := &yaml.Node{Kind: yaml.SequenceNode, Tag: "!!seq"}
@@ -65,13 +69,13 @@ func (dec *csvObjectDecoder) Decode(rootYamlNode *yaml.Node) error {
 		log.Debugf("Read next contentRow: %v, %v", contentRow, err)
 	}
 	if !errors.Is(err, io.EOF) {
-		return err
+		return nil, err
 	}

-	log.Debugf("finished, contentRow%v", contentRow)
-	log.Debugf("err: %v", err)
-
-	rootYamlNode.Kind = yaml.DocumentNode
-	rootYamlNode.Content = []*yaml.Node{rootArray}
-	return nil
+	return &CandidateNode{
+		Node: &yaml.Node{
+			Kind:    yaml.DocumentNode,
+			Content: []*yaml.Node{rootArray},
+		},
+	}, nil
 }
@@ -16,63 +16,65 @@ func NewJSONDecoder() Decoder {
 	return &jsonDecoder{}
 }

-func (dec *jsonDecoder) Init(reader io.Reader) {
+func (dec *jsonDecoder) Init(reader io.Reader) error {
 	dec.decoder = *json.NewDecoder(reader)
-}
-
-func (dec *jsonDecoder) Decode(rootYamlNode *yaml.Node) error {
-
-	var dataBucket interface{}
-	log.Debug("going to decode")
-	err := dec.decoder.Decode(&dataBucket)
-	if err != nil {
-		return err
-	}
-	node, err := dec.convertToYamlNode(dataBucket)
-
-	if err != nil {
-		return err
-	}
-	rootYamlNode.Kind = yaml.DocumentNode
-	rootYamlNode.Content = []*yaml.Node{node}
 	return nil
 }

-func (dec *jsonDecoder) convertToYamlNode(data interface{}) (*yaml.Node, error) {
-	switch data := data.(type) {
-	case nil:
-		return createScalarNode(nil, "null"), nil
-	case float64, float32:
-		// json decoder returns ints as float.
-		return parseSnippet(fmt.Sprintf("%v", data))
-	case int, int64, int32, string, bool:
-		return createScalarNode(data, fmt.Sprintf("%v", data)), nil
-	case map[string]interface{}:
-		return dec.parseMap(data)
-	case []interface{}:
-		return dec.parseArray(data)
-	default:
-		return nil, fmt.Errorf("unrecognised type :(")
+func (dec *jsonDecoder) Decode() (*CandidateNode, error) {
+
+	var dataBucket orderedMap
+	log.Debug("going to decode")
+	err := dec.decoder.Decode(&dataBucket)
+	if err != nil {
+		return nil, err
 	}
+	node, err := dec.convertToYamlNode(&dataBucket)
+
+	if err != nil {
+		return nil, err
+	}
+
+	return &CandidateNode{
+		Node: &yaml.Node{
+			Kind:    yaml.DocumentNode,
+			Content: []*yaml.Node{node},
+		},
+	}, nil
 }

-func (dec *jsonDecoder) parseMap(dataMap map[string]interface{}) (*yaml.Node, error) {
+func (dec *jsonDecoder) convertToYamlNode(data *orderedMap) (*yaml.Node, error) {
+	if data.kv == nil {
+		switch rawData := data.altVal.(type) {
+		case nil:
+			return createScalarNode(nil, "null"), nil
+		case float64, float32:
+			// json decoder returns ints as float.
+			return parseSnippet(fmt.Sprintf("%v", rawData))
+		case int, int64, int32, string, bool:
+			return createScalarNode(rawData, fmt.Sprintf("%v", rawData)), nil
+		case []*orderedMap:
+			return dec.parseArray(rawData)
+		default:
+			return nil, fmt.Errorf("unrecognised type :( %v", rawData)
+		}
+	}

-	var yamlMap = &yaml.Node{Kind: yaml.MappingNode}
-
-	for key, value := range dataMap {
-		yamlValue, err := dec.convertToYamlNode(value)
+	var yamlMap = &yaml.Node{Kind: yaml.MappingNode, Tag: "!!map"}
+	for _, keyValuePair := range data.kv {
+		yamlValue, err := dec.convertToYamlNode(&keyValuePair.V)
 		if err != nil {
 			return nil, err
 		}
-		yamlMap.Content = append(yamlMap.Content, createScalarNode(key, fmt.Sprintf("%v", key)), yamlValue)
+		yamlMap.Content = append(yamlMap.Content, createScalarNode(keyValuePair.K, keyValuePair.K), yamlValue)
 	}
 	return yamlMap, nil
+
 }

-func (dec *jsonDecoder) parseArray(dataArray []interface{}) (*yaml.Node, error) {
+func (dec *jsonDecoder) parseArray(dataArray []*orderedMap) (*yaml.Node, error) {

-	var yamlMap = &yaml.Node{Kind: yaml.SequenceNode}
+	var yamlMap = &yaml.Node{Kind: yaml.SequenceNode, Tag: "!!seq"}

 	for _, value := range dataArray {
 		yamlValue, err := dec.convertToYamlNode(value)
@@ -2,6 +2,7 @@ package yqlib

 import (
 	"bytes"
+	"fmt"
 	"io"
 	"strconv"
 	"strings"
@@ -20,9 +21,10 @@ func NewPropertiesDecoder() Decoder {
 	return &propertiesDecoder{d: NewDataTreeNavigator(), finished: false}
 }

-func (dec *propertiesDecoder) Init(reader io.Reader) {
+func (dec *propertiesDecoder) Init(reader io.Reader) error {
 	dec.reader = reader
 	dec.finished = false
+	return nil
 }

 func parsePropKey(key string) []interface{} {
@@ -46,15 +48,49 @@ func (dec *propertiesDecoder) processComment(c string) string {
 	return "# " + c
 }

-func (dec *propertiesDecoder) applyProperty(properties *properties.Properties, context Context, key string) error {
+func (dec *propertiesDecoder) applyPropertyComments(context Context, path []interface{}, comments []string) error {
+	assignmentOp := &Operation{OperationType: assignOpType, Preferences: assignPreferences{}}
+
+	rhsCandidateNode := &CandidateNode{
+		Path: path,
+		Node: &yaml.Node{
+			Tag:         "!!str",
+			Value:       fmt.Sprintf("%v", path[len(path)-1]),
+			HeadComment: dec.processComment(strings.Join(comments, "\n")),
+			Kind:        yaml.ScalarNode,
+		},
+	}
+
+	rhsCandidateNode.Node.Tag = guessTagFromCustomType(rhsCandidateNode.Node)
+
+	rhsOp := &Operation{OperationType: valueOpType, CandidateNode: rhsCandidateNode}
+
+	assignmentOpNode := &ExpressionNode{
+		Operation: assignmentOp,
+		LHS:       createTraversalTree(path, traversePreferences{}, true),
+		RHS:       &ExpressionNode{Operation: rhsOp},
+	}
+
+	_, err := dec.d.GetMatchingNodes(context, assignmentOpNode)
+	return err
+}
+
+func (dec *propertiesDecoder) applyProperty(context Context, properties *properties.Properties, key string) error {
 	value, _ := properties.Get(key)
 	path := parsePropKey(key)

+	propertyComments := properties.GetComments(key)
+	if len(propertyComments) > 0 {
+		err := dec.applyPropertyComments(context, path, propertyComments)
+		if err != nil {
+			return nil
+		}
+	}
+
 	rhsNode := &yaml.Node{
-		Value:       value,
-		Tag:         "!!str",
-		Kind:        yaml.ScalarNode,
-		LineComment: dec.processComment(properties.GetComment(key)),
+		Value: value,
+		Tag:   "!!str",
+		Kind:  yaml.ScalarNode,
 	}

 	rhsNode.Tag = guessTagFromCustomType(rhsNode)
@@ -78,22 +114,22 @@ func (dec *propertiesDecoder) applyProperty(properties *properties.Properties, c
 	return err
 }

-func (dec *propertiesDecoder) Decode(rootYamlNode *yaml.Node) error {
+func (dec *propertiesDecoder) Decode() (*CandidateNode, error) {
 	if dec.finished {
-		return io.EOF
+		return nil, io.EOF
 	}
 	buf := new(bytes.Buffer)

 	if _, err := buf.ReadFrom(dec.reader); err != nil {
-		return err
+		return nil, err
 	}
 	if buf.Len() == 0 {
 		dec.finished = true
-		return io.EOF
+		return nil, io.EOF
 	}
 	properties, err := properties.LoadString(buf.String())
 	if err != nil {
-		return err
+		return nil, err
 	}
 	properties.DisableExpansion = true

@@ -108,15 +144,18 @@ func (dec *propertiesDecoder) Decode(rootYamlNode *yaml.Node) error {
 	context = context.SingleChildContext(rootMap)

 	for _, key := range properties.Keys() {
-		if err := dec.applyProperty(properties, context, key); err != nil {
-			return err
+		if err := dec.applyProperty(context, properties, key); err != nil {
+			return nil, err
 		}

 	}
-
-	rootYamlNode.Kind = yaml.DocumentNode
-	rootYamlNode.Content = []*yaml.Node{rootMap.Node}
 	dec.finished = true
-	return nil
+
+	return &CandidateNode{
+		Node: &yaml.Node{
+			Kind:    yaml.DocumentNode,
+			Content: []*yaml.Node{rootMap.Node},
+		},
+	}, nil

 }
@@ -15,20 +15,20 @@ type formatScenario struct {
 	subdescription string
 	skipDoc        bool
 	scenarioType   string
+	expectedError  string
 }

-func processFormatScenario(s formatScenario, decoder Decoder, encoder Encoder) string {
-
+func processFormatScenario(s formatScenario, decoder Decoder, encoder Encoder) (string, error) {
 	var output bytes.Buffer
 	writer := bufio.NewWriter(&output)

 	if decoder == nil {
-		decoder = NewYamlDecoder()
+		decoder = NewYamlDecoder(ConfiguredYamlPreferences)
 	}

 	inputs, err := readDocuments(strings.NewReader(s.input), "sample.yml", 0, decoder)
 	if err != nil {
-		panic(err)
+		return "", err
 	}

 	expression := s.expression
@@ -39,22 +39,31 @@ func processFormatScenario(s formatScenario, decoder Decoder, encoder Encoder) s
 	exp, err := getExpressionParser().ParseExpression(expression)

 	if err != nil {
-		panic(err)
+		return "", err
 	}

 	context, err := NewDataTreeNavigator().GetMatchingNodes(Context{MatchingNodes: inputs}, exp)

 	if err != nil {
-		panic(err)
+		return "", err
 	}

 	printer := NewPrinter(encoder, NewSinglePrinterWriter(writer))
 	err = printer.PrintResults(context.MatchingNodes)
 	if err != nil {
-		panic(err)
+		return "", err
 	}
 	writer.Flush()

-	return output.String()
+	return output.String(), nil
+}
+
+func mustProcessFormatScenario(s formatScenario, decoder Decoder, encoder Encoder) string {
+
+	result, err := processFormatScenario(s, decoder, encoder)
+	if err != nil {
+		panic(err)
+	}
+	return result

 }
@@ -3,6 +3,7 @@ package yqlib
 import (
 	"encoding/xml"
 	"errors"
+	"fmt"
 	"io"
 	"strings"
 	"unicode"
@@ -12,34 +13,24 @@ import (
 )

 type xmlDecoder struct {
-	reader          io.Reader
-	readAnything    bool
-	attributePrefix string
-	contentName     string
-	strictMode      bool
-	keepNamespace   bool
-	useRawToken     bool
-	finished        bool
+	reader       io.Reader
+	readAnything bool
+	finished     bool
+	prefs        XmlPreferences
 }

-func NewXMLDecoder(attributePrefix string, contentName string, strictMode bool, keepNamespace bool, useRawToken bool) Decoder {
-	if contentName == "" {
-		contentName = "content"
-	}
+func NewXMLDecoder(prefs XmlPreferences) Decoder {
 	return &xmlDecoder{
-		attributePrefix: attributePrefix,
-		contentName:     contentName,
-		finished:        false,
-		strictMode:      strictMode,
-		keepNamespace:   keepNamespace,
-		useRawToken:     useRawToken,
+		finished: false,
+		prefs:    prefs,
 	}
 }

-func (dec *xmlDecoder) Init(reader io.Reader) {
+func (dec *xmlDecoder) Init(reader io.Reader) error {
 	dec.reader = reader
 	dec.readAnything = false
 	dec.finished = false
+	return nil
 }

 func (dec *xmlDecoder) createSequence(nodes []*xmlNode) (*yaml.Node, error) {
@@ -63,15 +54,17 @@ func (dec *xmlDecoder) processComment(c string) string {
 }

 func (dec *xmlDecoder) createMap(n *xmlNode) (*yaml.Node, error) {
-	log.Debug("createMap: headC: %v, footC: %v", n.HeadComment, n.FootComment)
+	log.Debug("createMap: headC: %v, lineC: %v, footC: %v", n.HeadComment, n.LineComment, n.FootComment)
 	yamlNode := &yaml.Node{Kind: yaml.MappingNode, Tag: "!!map"}

 	if len(n.Data) > 0 {
-		label := dec.contentName
+		log.Debug("creating content node for map")
+		label := dec.prefs.ContentName
 		labelNode := createScalarNode(label, label)
 		labelNode.HeadComment = dec.processComment(n.HeadComment)
+		labelNode.LineComment = dec.processComment(n.LineComment)
 		labelNode.FootComment = dec.processComment(n.FootComment)
-		yamlNode.Content = append(yamlNode.Content, labelNode, createScalarNode(n.Data, n.Data))
+		yamlNode.Content = append(yamlNode.Content, labelNode, dec.createValueNodeFromData(n.Data))
 	}

 	for i, keyValuePair := range n.Children {
@@ -86,9 +79,7 @@ func (dec *xmlDecoder) createMap(n *xmlNode) (*yaml.Node, error) {

 		}

-		// if i == len(n.Children)-1 {
 		labelNode.FootComment = dec.processComment(keyValuePair.FootComment)
-		// }

 		log.Debug("len of children in %v is %v", label, len(children))
 		if len(children) > 1 {
@@ -101,8 +92,16 @@ func (dec *xmlDecoder) createMap(n *xmlNode) (*yaml.Node, error) {
 			// if the value is a scalar, the head comment of the scalar needs to go on the key?
 			// add tests for <z/> as well as multiple <ds> of inputXmlWithComments > yaml
 			if len(children[0].Children) == 0 && children[0].HeadComment != "" {
-				labelNode.HeadComment = labelNode.HeadComment + "\n" + strings.TrimSpace(children[0].HeadComment)
-				children[0].HeadComment = ""
+				if len(children[0].Data) > 0 {
+
+					log.Debug("scalar comment hack, currentlabel [%v]", labelNode.HeadComment)
+					labelNode.HeadComment = joinComments([]string{labelNode.HeadComment, strings.TrimSpace(children[0].HeadComment)}, "\n")
+					children[0].HeadComment = ""
+				} else {
+					// child is null, put the headComment as a linecomment for reasons
+					children[0].LineComment = children[0].HeadComment
+					children[0].HeadComment = ""
+				}
 			}
 			valueNode, err = dec.convertToYamlNode(children[0])
 			if err != nil {
@@ -115,48 +114,75 @@ func (dec *xmlDecoder) createMap(n *xmlNode) (*yaml.Node, error) {
 	return yamlNode, nil
 }

+func (dec *xmlDecoder) createValueNodeFromData(values []string) *yaml.Node {
+	switch len(values) {
+	case 0:
+		return createScalarNode(nil, "")
+	case 1:
+		return createScalarNode(values[0], values[0])
+	default:
+		content := make([]*yaml.Node, 0)
+		for _, value := range values {
+			content = append(content, createScalarNode(value, value))
+		}
+		return &yaml.Node{
+			Kind:    yaml.SequenceNode,
+			Tag:     "!!seq",
+			Content: content,
+		}
+	}
+}
+
 func (dec *xmlDecoder) convertToYamlNode(n *xmlNode) (*yaml.Node, error) {
 	if len(n.Children) > 0 {
 		return dec.createMap(n)
 	}
-	scalar := createScalarNode(n.Data, n.Data)
-	if n.Data == "" {
-		scalar = createScalarNode(nil, "")
-	}
-	log.Debug("scalar headC: %v, footC: %v", n.HeadComment, n.FootComment)
+
+	scalar := dec.createValueNodeFromData(n.Data)
+
+	log.Debug("scalar (%v), headC: %v, lineC: %v, footC: %v", scalar.Tag, n.HeadComment, n.LineComment, n.FootComment)
 	scalar.HeadComment = dec.processComment(n.HeadComment)
 	scalar.LineComment = dec.processComment(n.LineComment)
+	if scalar.Tag == "!!seq" {
+		scalar.Content[0].HeadComment = scalar.LineComment
+		scalar.LineComment = ""
+	}
+
 	scalar.FootComment = dec.processComment(n.FootComment)

 	return scalar, nil
 }

-func (dec *xmlDecoder) Decode(rootYamlNode *yaml.Node) error {
+func (dec *xmlDecoder) Decode() (*CandidateNode, error) {
 	if dec.finished {
-		return io.EOF
+		return nil, io.EOF
 	}
 	root := &xmlNode{}
 	// cant use xj - it doesn't keep map order.
 	err := dec.decodeXML(root)

 	if err != nil {
-		return err
+		return nil, err
 	}
 	firstNode, err := dec.convertToYamlNode(root)

 	if err != nil {
-		return err
+		return nil, err
 	} else if firstNode.Tag == "!!null" {
 		dec.finished = true
 		if dec.readAnything {
-			return io.EOF
+			return nil, io.EOF
 		}
 	}
 	dec.readAnything = true
-	rootYamlNode.Kind = yaml.DocumentNode
-	rootYamlNode.Content = []*yaml.Node{firstNode}
 	dec.finished = true
-	return nil
+
+	return &CandidateNode{
+		Node: &yaml.Node{
+			Kind:    yaml.DocumentNode,
+			Content: []*yaml.Node{firstNode},
+		},
+	}, nil
 }

 type xmlNode struct {
@@ -164,7 +190,7 @@ type xmlNode struct {
 	HeadComment string
 	FootComment string
 	LineComment string
-	Data        string
+	Data        []string
 }

 type xmlChildrenKv struct {
@@ -205,10 +231,12 @@ type element struct {
 // of the map keys.
 func (dec *xmlDecoder) decodeXML(root *xmlNode) error {
 	xmlDec := xml.NewDecoder(dec.reader)
-	xmlDec.Strict = dec.strictMode
+	xmlDec.Strict = dec.prefs.StrictMode
 	// That will convert the charset if the provided XML is non-UTF-8
 	xmlDec.CharsetReader = charset.NewReaderLabel

+	started := false
+
 	// Create first element from the root node
 	elem := &element{
 		parent: nil,
@@ -216,7 +244,7 @@ func (dec *xmlDecoder) decodeXML(root *xmlNode) error {
 	}

 	getToken := func() (xml.Token, error) {
-		if dec.useRawToken {
+		if dec.prefs.UseRawToken {
 			return xmlDec.RawToken()
 		}
 		return xmlDec.Token()
@@ -244,17 +272,23 @@ func (dec *xmlDecoder) decodeXML(root *xmlNode) error {

 			// Extract attributes as children
 			for _, a := range se.Attr {
-				if dec.keepNamespace {
+				if dec.prefs.KeepNamespace {
 					if a.Name.Space != "" {
 						a.Name.Local = a.Name.Space + ":" + a.Name.Local
 					}
 				}
-				elem.n.AddChild(dec.attributePrefix+a.Name.Local, &xmlNode{Data: a.Value})
+				elem.n.AddChild(dec.prefs.AttributePrefix+a.Name.Local, &xmlNode{Data: []string{a.Value}})
 			}
 		case xml.CharData:
+
 			// Extract XML data (if any)
-			elem.n.Data = trimNonGraphic(string(se))
-			if elem.n.Data != "" {
+			newBit := trimNonGraphic(string(se))
+			if !started && len(newBit) > 0 {
+				return fmt.Errorf("invalid XML: Encountered chardata [%v] outside of XML node", newBit)
+			}
+
+			if len(newBit) > 0 {
+				elem.n.Data = append(elem.n.Data, newBit)
 				elem.state = "chardata"
 				log.Debug("chardata [%v] for %v", elem.n.Data, elem.label)
 			}
@@ -276,13 +310,22 @@ func (dec *xmlDecoder) decodeXML(root *xmlNode) error {

 			} else if elem.state == "chardata" {
 				log.Debug("got a line comment for (%v) %v: [%v]", elem.state, elem.label, commentStr)
-				elem.n.LineComment = joinFilter([]string{elem.n.LineComment, commentStr})
+				elem.n.LineComment = joinComments([]string{elem.n.LineComment, commentStr}, " ")
 			} else {
 				log.Debug("got a head comment for (%v) %v: [%v]", elem.state, elem.label, commentStr)
-				elem.n.HeadComment = joinFilter([]string{elem.n.HeadComment, commentStr})
+				elem.n.HeadComment = joinComments([]string{elem.n.HeadComment, commentStr}, " ")
 			}

+		case xml.ProcInst:
+			if !dec.prefs.SkipProcInst {
+				elem.n.AddChild(dec.prefs.ProcInstPrefix+se.Target, &xmlNode{Data: []string{string(se.Inst)}})
+			}
+		case xml.Directive:
+			if !dec.prefs.SkipDirectives {
+				elem.n.AddChild(dec.prefs.DirectiveName, &xmlNode{Data: []string{string(se)}})
+			}
 		}
+		started = true
 	}

 	return nil
@@ -295,21 +338,21 @@ func applyFootComment(elem *element, commentStr string) {
 		lastChildIndex := len(elem.n.Children) - 1
 		childKv := elem.n.Children[lastChildIndex]
 		log.Debug("got a foot comment for %v: [%v]", childKv.K, commentStr)
-		childKv.FootComment = joinFilter([]string{elem.n.FootComment, commentStr})
+		childKv.FootComment = joinComments([]string{elem.n.FootComment, commentStr}, " ")
 	} else {
 		log.Debug("got a foot comment for %v: [%v]", elem.label, commentStr)
-		elem.n.FootComment = joinFilter([]string{elem.n.FootComment, commentStr})
+		elem.n.FootComment = joinComments([]string{elem.n.FootComment, commentStr}, " ")
 	}
 }

-func joinFilter(rawStrings []string) string {
+func joinComments(rawStrings []string, joinStr string) string {
 	stringsToJoin := make([]string, 0)
 	for _, str := range rawStrings {
 		if str != "" {
 			stringsToJoin = append(stringsToJoin, str)
 		}
 	}
-	return strings.Join(stringsToJoin, " ")
+	return strings.Join(stringsToJoin, joinStr)
 }

 // trimNonGraphic returns a slice of the string s, with all leading and trailing
@@ -1,23 +1,135 @@
 package yqlib

 import (
+	"bufio"
+	"bytes"
+	"errors"
 	"io"
+	"regexp"
+	"strings"

 	yaml "gopkg.in/yaml.v3"
 )

 type yamlDecoder struct {
 	decoder yaml.Decoder
+
+	prefs YamlPreferences
+
+	// work around of various parsing issues by yaml.v3 with document headers
+	leadingContent string
+	bufferRead     bytes.Buffer
+
+	readAnything bool
+	firstFile    bool
 }

-func NewYamlDecoder() Decoder {
-	return &yamlDecoder{}
+func NewYamlDecoder(prefs YamlPreferences) Decoder {
+	return &yamlDecoder{prefs: prefs, firstFile: true}
 }

-func (dec *yamlDecoder) Init(reader io.Reader) {
-	dec.decoder = *yaml.NewDecoder(reader)
+func (dec *yamlDecoder) processReadStream(reader *bufio.Reader) (io.Reader, string, error) {
+	var commentLineRegEx = regexp.MustCompile(`^\s*#`)
+	var yamlDirectiveLineRegEx = regexp.MustCompile(`^\s*%YA`)
+	var sb strings.Builder
+	for {
+		peekBytes, err := reader.Peek(3)
+		if errors.Is(err, io.EOF) {
+			// EOF are handled else where..
+			return reader, sb.String(), nil
+		} else if err != nil {
+			return reader, sb.String(), err
+		} else if string(peekBytes) == "---" {
+			_, err := reader.ReadString('\n')
+			sb.WriteString("$yqDocSeperator$\n")
+			if errors.Is(err, io.EOF) {
+				return reader, sb.String(), nil
+			} else if err != nil {
+				return reader, sb.String(), err
+			}
+		} else if commentLineRegEx.MatchString(string(peekBytes)) || yamlDirectiveLineRegEx.MatchString(string(peekBytes)) {
+			line, err := reader.ReadString('\n')
+			sb.WriteString(line)
+			if errors.Is(err, io.EOF) {
+				return reader, sb.String(), nil
+			} else if err != nil {
+				return reader, sb.String(), err
+			}
+		} else {
+			return reader, sb.String(), nil
+		}
+	}
 }

-func (dec *yamlDecoder) Decode(rootYamlNode *yaml.Node) error {
-	return dec.decoder.Decode(rootYamlNode)
+func (dec *yamlDecoder) Init(reader io.Reader) error {
+	readerToUse := reader
+	leadingContent := ""
+	dec.bufferRead = bytes.Buffer{}
+	var err error
+	// if we 'evaluating together' - we only process the leading content
+	// of the first file - this ensures comments from subsequent files are
+	// merged together correctly.
+	if dec.prefs.LeadingContentPreProcessing && (!dec.prefs.EvaluateTogether || dec.firstFile) {
+		readerToUse, leadingContent, err = dec.processReadStream(bufio.NewReader(reader))
+		if err != nil {
+			return err
+		}
+	} else if !dec.prefs.LeadingContentPreProcessing {
+		// if we're not process the leading content
+		// keep a copy of what we've read. This is incase its a
+		// doc with only comments - the decoder will return nothing
+		// then we can read the comments from bufferRead
+		readerToUse = io.TeeReader(reader, &dec.bufferRead)
+	}
+	dec.leadingContent = leadingContent
+	dec.readAnything = false
+	dec.decoder = *yaml.NewDecoder(readerToUse)
+	dec.firstFile = false
+	return nil
+}
+
+func (dec *yamlDecoder) Decode() (*CandidateNode, error) {
+	var dataBucket yaml.Node
+	err := dec.decoder.Decode(&dataBucket)
+	if errors.Is(err, io.EOF) && dec.leadingContent != "" && !dec.readAnything {
+		// force returning an empty node with a comment.
+		dec.readAnything = true
+		return dec.blankNodeWithComment(), nil
+	} else if errors.Is(err, io.EOF) && !dec.prefs.LeadingContentPreProcessing && !dec.readAnything {
+		// didn't find any yaml,
+		// check the tee buffer, maybe there were comments
+		dec.readAnything = true
+		dec.leadingContent = dec.bufferRead.String()
+		if dec.leadingContent != "" {
+			return dec.blankNodeWithComment(), nil
+		}
+		return nil, err
+	} else if err != nil {
+		return nil, err
+	}
+
+	candidateNode := &CandidateNode{
+		Node: &dataBucket,
+	}
+
+	if dec.leadingContent != "" {
+		candidateNode.LeadingContent = dec.leadingContent
+		dec.leadingContent = ""
+	}
+	dec.readAnything = true
+	// move document comments into candidate node
+	// otherwise unwrap drops them.
+	candidateNode.TrailingContent = dataBucket.FootComment
+	dataBucket.FootComment = ""
+	return candidateNode, nil
+}
+
+func (dec *yamlDecoder) blankNodeWithComment() *CandidateNode {
+	return &CandidateNode{
+		Document:       0,
+		Filename:       "",
+		Node:           &yaml.Node{Kind: yaml.DocumentNode, Content: []*yaml.Node{{Tag: "!!null", Kind: yaml.ScalarNode}}},
+		FileIndex:      0,
+		LeadingContent: dec.leadingContent,
+	}
 }
@@ -1,6 +0,0 @@
-
-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;
-
-`yq e <exp> <file>`
-{% endhint %}
@@ -9,12 +9,6 @@ Add behaves differently according to the type of the LHS:
 Use `+=` as a relative append assign for things like increment. Note that `.a += .x` is equivalent to running `.a = .a + .x`.


-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;
-
-`yq e <exp> <file>`
-{% endhint %}
-
 ## Concatenate arrays
 Given a sample.yml file of:
 ```yaml
@@ -2,12 +2,6 @@

 This operator is used to provide alternative (or default) values when a particular expression is either null or false.

-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;
-
-`yq e <exp> <file>`
-{% endhint %}
-
 ## LHS is defined
 Given a sample.yml file of:
 ```yaml
@@ -79,3 +73,36 @@ will output
 cat
 ```

+## Update or create - entity exists
+This initialises `a` if it's not present
+
+Given a sample.yml file of:
+```yaml
+a: 1
+```
+then
+```bash
+yq '(.a // (.a = 0)) += 1' sample.yml
+```
+will output
+```yaml
+a: 2
+```
+
+## Update or create - entity does not exist
+This initialises `a` if it's not present
+
+Given a sample.yml file of:
+```yaml
+b: camel
+```
+then
+```bash
+yq '(.a // (.a = 0)) += 1' sample.yml
+```
+will output
+```yaml
+b: camel
+a: 1
+```
+
@@ -5,12 +5,6 @@ Use the `alias` and `anchor` operators to read and write yaml aliases and anchor
 `yq` supports merge aliases (like `<<: *blah`) however this is no longer in the standard yaml spec (1.2) and so `yq` will automatically add the `!!merge` tag to these nodes as it is effectively a custom tag.


-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;
-
-`yq e <exp> <file>`
-{% endhint %}
-
 ## Merge one map
 see https://yaml.org/type/merge.html

@@ -0,0 +1,28 @@
+# Array to Map
+
+Use this operator to convert an array to..a map. The indices are used as map keys, null values in the array are skipped over.
+
+Behind the scenes, this is implemented using reduce:
+
+```
+(.[] | select(. != null) ) as $i ireduce({}; .[$i | key] = $i)
+```
+
+## Simple example
+Given a sample.yml file of:
+```yaml
+cool:
+  - null
+  - null
+  - hello
+```
+then
+```bash
+yq '.cool |= array_to_map' sample.yml
+```
+will output
+```yaml
+cool:
+  2: hello
+```
+
@@ -7,11 +7,10 @@ Which will assign the LHS node values to the RHS node values. The RHS expression

 ### relative form: `|=`
 This will do a similar thing to the plain form, however, the RHS expression is run against _the LHS nodes_. This is useful for updating values based on old values, e.g. increment.
-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;

-`yq e <exp> <file>`
-{% endhint %}
+
+### Flags
+- `c` clobber custom tags

 ## Create yaml file
 Running
@@ -235,3 +234,37 @@ a:
    - bogs
 ```

+## Custom types are maintained by default
+Given a sample.yml file of:
+```yaml
+a: !cat meow
+b: !dog woof
+```
+then
+```bash
+yq '.a = .b' sample.yml
+```
+will output
+```yaml
+a: !cat woof
+b: !dog woof
+```
+
+## Custom types: clovver
+Use the `c` option to clobber custom tags
+
+Given a sample.yml file of:
+```yaml
+a: !cat meow
+b: !dog woof
+```
+then
+```bash
+yq '.a =c .b' sample.yml
+```
+will output
+```yaml
+a: !dog woof
+b: !dog woof
+```
+
@@ -16,12 +16,6 @@ These are most commonly used with the `select` operator to filter particular nod
 - comparison (`>=`, `<` etc) operators [here](https://mikefarah.gitbook.io/yq/operators/compare)
 - select operator [here](https://mikefarah.gitbook.io/yq/operators/select)

-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;
-
-`yq e <exp> <file>`
-{% endhint %}
-
 ## `or` example
 Running
 ```bash
@@ -3,12 +3,6 @@
 This creates an array using the expression between the square brackets.


-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;
-
-`yq e <exp> <file>`
-{% endhint %}
-
 ## Collect empty
 Running
 ```bash
@@ -2,12 +2,6 @@

 Returns the column of the matching node. Starts from 1, 0 indicates there was no column data.

-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;
-
-`yq e <exp> <file>`
-{% endhint %}
-
 ## Returns column of _value_ node
 Given a sample.yml file of:
 ```yaml
@@ -1,6 +1,6 @@
 # Comment Operators

-Use these comment operators to set or retrieve comments.
+Use these comment operators to set or retrieve comments. Note that line comments on maps/arrays are actually set on the _key_ node as opposed to the _value_ (map/array). See below for examples.

 Like the `=` and `|=` assign operators, the same syntax applies when updating comments:

@@ -10,13 +10,9 @@ This will assign the LHS nodes comments to the expression on the RHS. The RHS is
 ### relative form: `|=` 
 Similar to the plain form, however the RHS evaluates against each matching LHS node! This is useful if you want to set the comments as a relative expression of the node, for instance its value or path.

-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;
-
-`yq e <exp> <file>`
-{% endhint %}
-
 ## Set line comment
+Set the comment on the key node for more reliability (see below).
+
 Given a sample.yml file of:
 ```yaml
 a: cat
@@ -30,6 +26,24 @@ will output
 a: cat # single
 ```

+## Set line comment of a maps/arrays
+For maps and arrays, you need to set the line comment on the _key_ node. This will also work for scalars.
+
+Given a sample.yml file of:
+```yaml
+a:
+  b: things
+```
+then
+```bash
+yq '(.a | key) line_comment="single"' sample.yml
+```
+will output
+```yaml
+a: # single
+  b: things
+```
+
 ## Use update assign to perform relative updates
 Given a sample.yml file of:
 ```yaml
@@ -46,6 +60,121 @@ a: cat # cat
 b: dog # dog
 ```

+## Where is the comment - map key example
+The underlying yaml parser can assign comments in a document to surprising nodes. Use an expression like this to find where you comment is. 'p' indicates the path, 'isKey' is if the node is a map key (as opposed to a map value).
+From this, you can see the 'hello-world-comment' is actually on the 'hello' key
+
+Given a sample.yml file of:
+```yaml
+hello: # hello-world-comment
+  message: world
+```
+then
+```bash
+yq '[... | {"p": path | join("."), "isKey": is_key, "hc": headComment, "lc": lineComment, "fc": footComment}]' sample.yml
+```
+will output
+```yaml
+- p: ""
+  isKey: false
+  hc: ""
+  lc: ""
+  fc: ""
+- p: hello
+  isKey: true
+  hc: ""
+  lc: hello-world-comment
+  fc: ""
+- p: hello
+  isKey: false
+  hc: ""
+  lc: ""
+  fc: ""
+- p: hello.message
+  isKey: true
+  hc: ""
+  lc: ""
+  fc: ""
+- p: hello.message
+  isKey: false
+  hc: ""
+  lc: ""
+  fc: ""
+```
+
+## Retrieve comment - map key example
+From the previous example, we know that the comment is on the 'hello' _key_ as a lineComment
+
+Given a sample.yml file of:
+```yaml
+hello: # hello-world-comment
+  message: world
+```
+then
+```bash
+yq '.hello | key | line_comment' sample.yml
+```
+will output
+```yaml
+hello-world-comment
+```
+
+## Where is the comment - array example
+The underlying yaml parser can assign comments in a document to surprising nodes. Use an expression like this to find where you comment is. 'p' indicates the path, 'isKey' is if the node is a map key (as opposed to a map value).
+From this, you can see the 'under-name-comment' is actually on the first child
+
+Given a sample.yml file of:
+```yaml
+name:
+  # under-name-comment
+  - first-array-child
+```
+then
+```bash
+yq '[... | {"p": path | join("."), "isKey": is_key, "hc": headComment, "lc": lineComment, "fc": footComment}]' sample.yml
+```
+will output
+```yaml
+- p: ""
+  isKey: false
+  hc: ""
+  lc: ""
+  fc: ""
+- p: name
+  isKey: true
+  hc: ""
+  lc: ""
+  fc: ""
+- p: name
+  isKey: false
+  hc: ""
+  lc: ""
+  fc: ""
+- p: name.0
+  isKey: false
+  hc: under-name-comment
+  lc: ""
+  fc: ""
+```
+
+## Retrieve comment - array example
+From the previous example, we know that the comment is on the first child as a headComment
+
+Given a sample.yml file of:
+```yaml
+name:
+  # under-name-comment
+  - first-array-child
+```
+then
+```bash
+yq '.name[0] | headComment' sample.yml
+```
+will output
+```yaml
+under-name-comment
+```
+
 ## Set head comment
 Given a sample.yml file of:
 ```yaml
@@ -62,6 +191,25 @@ will output
 a: cat
 ```

+## Set head comment of a map entry
+Given a sample.yml file of:
+```yaml
+f: foo
+a:
+  b: cat
+```
+then
+```bash
+yq '(.a | key) head_comment="single"' sample.yml
+```
+will output
+```yaml
+f: foo
+# single
+a:
+  b: cat
+```
+
 ## Set foot comment, using an expression
 Given a sample.yml file of:
 ```yaml
@@ -98,6 +246,7 @@ Note the use of `...` to ensure key nodes are included.

 Given a sample.yml file of:
 ```yaml
+# hi
 a: cat # comment
 # great
 b: # key comment
@@ -115,6 +264,7 @@ b:
 ## Get line comment
 Given a sample.yml file of:
 ```yaml
+# welcome!
 a: cat # meow
 # have a great day
 ```
@@ -14,12 +14,6 @@ The following types are currently supported:
 - boolean operators (`and`, `or`, `any` etc) [here](https://mikefarah.gitbook.io/yq/operators/boolean-operators)
 - select operator [here](https://mikefarah.gitbook.io/yq/operators/select)

-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;
-
-`yq e <exp> <file>`
-{% endhint %}
-
 ## Compare numbers (>)
 Given a sample.yml file of:
 ```yaml
@@ -1,11 +1,13 @@
 # Contains

-This returns `true` if the context contains the passed in parameter, and false otherwise.
+This returns `true` if the context contains the passed in parameter, and false otherwise. For arrays, this will return true if the passed in array is contained within the array. For strings, it will return true if the string is a substring.

 {% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;

-`yq e <exp> <file>`
+_Note_ that, just like jq, when checking if an array of strings `contains` another, this will use `contains` and _not_ equals to check each string. This means an array like `contains(["cat"])` will return true for an array `["cats"]`.
+
+See the "Array has a subset array" example below on how to check for a subset.
+
 {% endhint %}

 ## Array contains array
@@ -26,6 +28,24 @@ will output
 true
 ```

+## Array has a subset array
+Subtract the superset array from the subset, if there's anything left, it's not a subset
+
+Given a sample.yml file of:
+```yaml
+- foobar
+- foobaz
+- blarp
+```
+then
+```bash
+yq '["baz", "bar"] - . | length == 0' sample.yml
+```
+will output
+```yaml
+false
+```
+
 ## Object included in array
 Given a sample.yml file of:
 ```yaml
@@ -2,12 +2,6 @@

 This is used to construct objects (or maps). This can be used against existing yaml, or to create fresh yaml documents.

-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;
-
-`yq e <exp> <file>`
-{% endhint %}
-
 ## Collect empty object
 Running
 ```bash
@@ -13,11 +13,11 @@ To specify a custom parsing format, use the `with_dtf` operator. The first param
 yq 'with_dtf("myformat"; .a + "3h" | tz("Australia/Melbourne"))'
 ```

-See https://pkg.go.dev/time#pkg-constants for examples of formatting options.
+See the [library docs](https://pkg.go.dev/time#pkg-constants) for examples of formatting options.


 ## Timezones
-This uses golangs built in LoadLocation function to parse timezones strings. See https://pkg.go.dev/time#LoadLocation for more details.
+This uses golangs built in LoadLocation function to parse timezones strings. See the [library docs](https://pkg.go.dev/time#LoadLocation) for more details.


 ## Durations
@@ -25,12 +25,6 @@ Durations are parsed using golangs built in [ParseDuration](https://pkg.go.dev/t

 You can durations to time using the `+` operator.

-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;
-
-`yq e <exp> <file>`
-{% endhint %}
-
 ## Format: from standard RFC3339 format
 Providing a single parameter assumes a standard RFC3339 datetime format. If the target format is not a valid yaml datetime format, the result will be a string tagged node.

@@ -66,7 +60,7 @@ a: 2001-12-15
 ## Format: get the day of the week
 Given a sample.yml file of:
 ```yaml
-a: 2001-12-15T02:59:43.1Z
+a: 2001-12-15
 ```
 then
 ```bash
@@ -156,7 +150,7 @@ a: 2021-01-01T03:10:00Z
 ```

 ## Date subtraction
-You can subtract durations from dates. Assumes RFC3339 date time format, see [date-time operators](https://mikefarah.gitbook.io/yq/operators/date-time-operators) for more information.
+You can subtract durations from dates. Assumes RFC3339 date time format, see [date-time operators](https://mikefarah.gitbook.io/yq/operators/datetime#date-time-formattings) for more information.

 Given a sample.yml file of:
 ```yaml
@@ -2,12 +2,6 @@

 Deletes matching entries in maps or arrays.

-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;
-
-`yq e <exp> <file>`
-{% endhint %}
-
 ## Delete entry in map
 Given a sample.yml file of:
 ```yaml
@@ -2,12 +2,6 @@

 Use the `documentIndex` operator (or the `di` shorthand) to select nodes of a particular document.

-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;
-
-`yq e <exp> <file>`
-{% endhint %}
-
 ## Retrieve a document index
 Given a sample.yml file of:
 ```yaml
@@ -9,12 +9,12 @@ These operators are useful to process yaml documents that have stringified embed

 | Format | Decode (from string) | Encode (to string) |
 | --- | -- | --|
-| Yaml | from_yaml | to_yaml(i)/@yaml |
-| JSON | from_json | to_json(i)/@json |
+| Yaml | from_yaml/@yamld | to_yaml(i)/@yaml |
+| JSON | from_json/@jsond | to_json(i)/@json |
 | Properties | from_props/@propsd  | to_props/@props |
 | CSV | from_csv/@csvd | to_csv/@csv |
 | TSV | from_tsv/@tsvd | to_tsv/@tsv |
-| XML | from_xml | to_xml(i)/@xml |
+| XML | from_xml/@xmld | to_xml(i)/@xml |
 | Base64 | @base64d | @base64 |


@@ -25,12 +25,6 @@ XML uses the `--xml-attribute-prefix` and `xml-content-name` flags to identify a

 Base64 assumes [rfc4648](https://rfc-editor.org/rfc/rfc4648.html) encoding. Encoding and decoding both assume that the content is a string.

-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;
-
-`yq e <exp> <file>`
-{% endhint %}
-
 ## Encode value as json string
 Given a sample.yml file of:
 ```yaml
@@ -343,7 +337,7 @@ Given a sample.yml file of:
 a:
  cool:
    foo: bar
-    +id: hi
+    +@id: hi
 ```
 then
 ```bash
@@ -363,7 +357,7 @@ Given a sample.yml file of:
 a:
  cool:
    foo: bar
-    +id: hi
+    +@id: hi
 ```
 then
 ```bash
@@ -381,7 +375,7 @@ Given a sample.yml file of:
 a:
  cool:
    foo: bar
-    +id: hi
+    +@id: hi
 ```
 then
 ```bash
@@ -2,12 +2,6 @@

 Similar to the same named functions in `jq` these functions convert to/from an object and an array of key-value pairs. This is most useful for performing operations on keys of maps.

-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;
-
-`yq e <exp> <file>`
-{% endhint %}
-
 ## to_entries Map
 Given a sample.yml file of:
 ```yaml
@@ -107,6 +101,26 @@ KEY_a: 1
 KEY_b: 2
 ```

+## Custom sort map keys
+Use to_entries to convert to an array of key/value pairs, sort the array using sort/sort_by/etc, and convert it back.
+
+Given a sample.yml file of:
+```yaml
+a: 1
+c: 3
+b: 2
+```
+then
+```bash
+yq 'to_entries | sort_by(.key) | reverse | from_entries' sample.yml
+```
+will output
+```yaml
+c: 3
+b: 2
+a: 1
+```
+
 ## Use with_entries to filter the map
 Given a sample.yml file of:
 ```yaml
@@ -30,12 +30,6 @@ yq '(.. | select(tag == "!!str")) |= envsubst' file.yaml
 ```


-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;
-
-`yq e <exp> <file>`
-{% endhint %}
-
 ## Read string environment variable
 Running
 ```bash
@@ -21,12 +21,6 @@ The not equals `!=` operator returns `false` if the LHS is equal to the RHS.
 - select operator [here](https://mikefarah.gitbook.io/yq/operators/select)


-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;
-
-`yq e <exp> <file>`
-{% endhint %}
-
 ## Match string
 Given a sample.yml file of:
 ```yaml
@@ -2,12 +2,6 @@

 Use this operation to short-circuit expressions. Useful for validation.

-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;
-
-`yq e <exp> <file>`
-{% endhint %}
-
 ## Validate a particular value
 Given a sample.yml file of:
 ```yaml
@@ -6,12 +6,6 @@ Use `eval` to dynamically process an expression - for instance from an environme

 Tip: This can be useful way parameterise complex scripts.

-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;
-
-`yq e <exp> <file>`
-{% endhint %}
-
 ## Dynamically evaluate a path
 Given a sample.yml file of:
 ```yaml
@@ -10,12 +10,6 @@ Note the use of eval-all to ensure all documents are loaded into memory.
 yq eval-all 'select(fi == 0) * select(filename == "file2.yaml")' file1.yaml file2.yaml
 ```

-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;
-
-`yq e <exp> <file>`
-{% endhint %}
-
 ## Get filename
 Given a sample.yml file of:
 ```yaml
@@ -1,12 +1,6 @@
 # Flatten
 This recursively flattens arrays.

-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;
-
-`yq e <exp> <file>`
-{% endhint %}
-
 ## Flatten
 Recursively flattens all arrays

@@ -2,12 +2,6 @@

 This is used to group items in an array by an expression.

-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;
-
-`yq e <exp> <file>`
-{% endhint %}
-
 ## Group by field
 Given a sample.yml file of:
 ```yaml
@@ -2,12 +2,6 @@

 This is operation that returns true if the key exists in a map (or index in an array), false otherwise.

-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;
-
-`yq e <exp> <file>`
-{% endhint %}
-
 ## Has map key
 Given a sample.yml file of:
 ```yaml
@@ -0,0 +1,9 @@
+# Array to Map
+
+Use this operator to convert an array to..a map. The indices are used as map keys, null values in the array are skipped over.
+
+Behind the scenes, this is implemented using reduce:
+
+```
+(.[] | select(. != null) ) as $i ireduce({}; .[$i | key] = $i)
+```
@@ -6,4 +6,8 @@ This operator is used to update node values. It can be used in either the:
 Which will assign the LHS node values to the RHS node values. The RHS expression is run against the matching nodes in the pipeline.

 ### relative form: `|=`
-This will do a similar thing to the plain form, however, the RHS expression is run against _the LHS nodes_. This is useful for updating values based on old values, e.g. increment.
+This will do a similar thing to the plain form, however, the RHS expression is run against _the LHS nodes_. This is useful for updating values based on old values, e.g. increment.
+
+
+### Flags
+- `c` clobber custom tags
@@ -1,6 +1,6 @@
 # Comment Operators

-Use these comment operators to set or retrieve comments.
+Use these comment operators to set or retrieve comments. Note that line comments on maps/arrays are actually set on the _key_ node as opposed to the _value_ (map/array). See below for examples.

 Like the `=` and `|=` assign operators, the same syntax applies when updating comments:

@@ -1,3 +1,11 @@
 # Contains

-This returns `true` if the context contains the passed in parameter, and false otherwise.
+This returns `true` if the context contains the passed in parameter, and false otherwise. For arrays, this will return true if the passed in array is contained within the array. For strings, it will return true if the string is a substring.
+
+{% hint style="warning" %}
+
+_Note_ that, just like jq, when checking if an array of strings `contains` another, this will use `contains` and _not_ equals to check each string. This means an array like `contains(["cat"])` will return true for an array `["cats"]`.
+
+See the "Array has a subset array" example below on how to check for a subset.
+
+{% endhint %}
@@ -13,11 +13,11 @@ To specify a custom parsing format, use the `with_dtf` operator. The first param
 yq 'with_dtf("myformat"; .a + "3h" | tz("Australia/Melbourne"))'
 ```

-See https://pkg.go.dev/time#pkg-constants for examples of formatting options.
+See the [library docs](https://pkg.go.dev/time#pkg-constants) for examples of formatting options.


 ## Timezones
-This uses golangs built in LoadLocation function to parse timezones strings. See https://pkg.go.dev/time#LoadLocation for more details.
+This uses golangs built in LoadLocation function to parse timezones strings. See the [library docs](https://pkg.go.dev/time#LoadLocation) for more details.


 ## Durations
@@ -9,12 +9,12 @@ These operators are useful to process yaml documents that have stringified embed

 | Format | Decode (from string) | Encode (to string) |
 | --- | -- | --|
-| Yaml | from_yaml | to_yaml(i)/@yaml |
-| JSON | from_json | to_json(i)/@json |
+| Yaml | from_yaml/@yamld | to_yaml(i)/@yaml |
+| JSON | from_json/@jsond | to_json(i)/@json |
 | Properties | from_props/@propsd  | to_props/@props |
 | CSV | from_csv/@csvd | to_csv/@csv |
 | TSV | from_tsv/@tsvd | to_tsv/@tsv |
-| XML | from_xml | to_xml(i)/@xml |
+| XML | from_xml/@xmld | to_xml(i)/@xml |
 | Base64 | @base64d | @base64 |


@@ -14,6 +14,7 @@ You can control how objects are merged by using one or more of the following fla
 - `d` deeply merge arrays
 - `?` only merge _existing_ fields
 - `n` only merge _new_ fields
+- `c` clobber custom tags


 ### Merge two files together
@@ -1,5 +1,8 @@
 # Path

-The path operator can be used to get the traversal paths of matching nodes in an expression. The path is returned as an array, which if traversed in order will lead to the matching node.
+The `path` operator can be used to get the traversal paths of matching nodes in an expression. The path is returned as an array, which if traversed in order will lead to the matching node.

 You can get the key/index of matching nodes by using the `path` operator to return the path array then piping that through `.[-1]` to get the last element of that array, the key.
+
+Use `setpath` to set a value to the path array returned by `path`, and similarly `delpaths` for an array of path arrays.
+
@@ -0,0 +1,5 @@
+# Slice/Splice Array
+
+The slice array operator takes an array as input and returns a subarray. Like the `jq` equivalent, `.[10:15]` will return an array of length 5, starting from index 10 inclusive, up to index 15 exclusive. Negative numbers count backwards from the end of the array.
+
+You may leave out the first or second number, which will will refer to the start or end of the array respectively.
@@ -11,3 +11,6 @@ diff file1.yml file2.yml
 ```

 Note that `yq` does not yet consider anchors when sorting by keys - this may result in invalid yaml documents if your are using merge anchors.
+
+For more advanced sorting, using `to_entries` to convert the map to an array, then sort/process the array as you like (e.g. using `sort_by`) and convert back to a map using `from_entries`.
+See [here](https://mikefarah.gitbook.io/yq/operators/entries#custom-sort-map-keys) for an example. 
@@ -2,12 +2,6 @@

 Use the `keys` operator to return map keys or array indices. 

-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;
-
-`yq e <exp> <file>`
-{% endhint %}
-
 ## Map keys
 Given a sample.yml file of:
 ```yaml
@@ -118,3 +112,43 @@ will output
 comment on key
 ```

+## Check node is a key
+Given a sample.yml file of:
+```yaml
+a:
+  b:
+    - cat
+  c: frog
+```
+then
+```bash
+yq '[... | { "p": path | join("."), "isKey": is_key, "tag": tag }]' sample.yml
+```
+will output
+```yaml
+- p: ""
+  isKey: false
+  tag: '!!map'
+- p: a
+  isKey: true
+  tag: '!!str'
+- p: a
+  isKey: false
+  tag: '!!map'
+- p: a.b
+  isKey: true
+  tag: '!!str'
+- p: a.b
+  isKey: false
+  tag: '!!seq'
+- p: a.b.0
+  isKey: false
+  tag: '!!str'
+- p: a.c
+  isKey: true
+  tag: '!!str'
+- p: a.c
+  isKey: false
+  tag: '!!str'
+```
+
@@ -2,12 +2,6 @@

 Returns the lengths of the nodes. Length is defined according to the type of the node.

-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;
-
-`yq e <exp> <file>`
-{% endhint %}
-
 ## String length
 returns length of string

@@ -2,12 +2,6 @@

 Returns the line of the matching node. Starts from 1, 0 indicates there was no line data.

-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;
-
-`yq e <exp> <file>`
-{% endhint %}
-
 ## Returns line of _value_ node
 Given a sample.yml file of:
 ```yaml
@@ -45,12 +45,6 @@ this.is = a properties file
 bXkgc2VjcmV0IGNoaWxsaSByZWNpcGUgaXMuLi4u
 ```

-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;
-
-`yq e <exp> <file>`
-{% endhint %}
-
 ## Simple example
 Given a sample.yml file of:
 ```yaml
@@ -2,12 +2,6 @@

 Maps values of an array. Use `map_values` to map values of an object.

-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;
-
-`yq e <exp> <file>`
-{% endhint %}
-
 ## Map array
 Given a sample.yml file of:
 ```yaml
@@ -14,6 +14,7 @@ You can control how objects are merged by using one or more of the following fla
 - `d` deeply merge arrays
 - `?` only merge _existing_ fields
 - `n` only merge _new_ fields
+- `c` clobber custom tags


 ### Merge two files together
@@ -35,12 +36,6 @@ By default - `yq` merge is naive. It merges maps when they match the key name, a
 For more complex array merging (e.g. merging items that match on a certain key) please see the example [here](https://mikefarah.gitbook.io/yq/operators/multiply-merge#merge-arrays-of-objects-together-matching-on-a-key)


-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;
-
-`yq e <exp> <file>`
-{% endhint %}
-
 ## Multiply integers
 Given a sample.yml file of:
 ```yaml
@@ -481,3 +476,26 @@ b: !goat
  dog: woof
 ```

+## Custom types: clobber tags
+Use the `c` option to clobber custom tags. Note that the second tag is now used
+
+Given a sample.yml file of:
+```yaml
+a: !horse
+  cat: meow
+b: !goat
+  dog: woof
+```
+then
+```bash
+yq '.a *=c .b' sample.yml
+```
+will output
+```yaml
+a: !goat
+  cat: meow
+  dog: woof
+b: !goat
+  dog: woof
+```
+
@@ -2,12 +2,6 @@

 Parent simply returns the parent nodes of the matching nodes.

-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;
-
-`yq e <exp> <file>`
-{% endhint %}
-
 ## Simple example
 Given a sample.yml file of:
 ```yaml
@@ -1,14 +1,11 @@
 # Path

-The path operator can be used to get the traversal paths of matching nodes in an expression. The path is returned as an array, which if traversed in order will lead to the matching node.
+The `path` operator can be used to get the traversal paths of matching nodes in an expression. The path is returned as an array, which if traversed in order will lead to the matching node.

 You can get the key/index of matching nodes by using the `path` operator to return the path array then piping that through `.[-1]` to get the last element of that array, the key.

-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;
+Use `setpath` to set a value to the path array returned by `path`, and similarly `delpaths` for an array of path arrays.

-`yq e <exp> <file>`
-{% endhint %}

 ## Map path
 Given a sample.yml file of:
@@ -98,3 +95,140 @@ will output
  value: frog
 ```

+## Set path
+Given a sample.yml file of:
+```yaml
+a:
+  b: cat
+```
+then
+```bash
+yq 'setpath(["a", "b"]; "things")' sample.yml
+```
+will output
+```yaml
+a:
+  b: things
+```
+
+## Set on empty document
+Running
+```bash
+yq --null-input 'setpath(["a", "b"]; "things")'
+```
+will output
+```yaml
+a:
+  b: things
+```
+
+## Set path to prune deep paths
+Like pick but recursive. This uses `ireduce` to deeply set the selected paths into an empty object,
+
+Given a sample.yml file of:
+```yaml
+parentA: bob
+parentB:
+  child1: i am child1
+  child2: i am child2
+parentC:
+  child1: me child1
+  child2: me child2
+```
+then
+```bash
+yq '(.parentB.child2, .parentC.child1) as $i
+  ireduce({}; setpath($i | path; $i))' sample.yml
+```
+will output
+```yaml
+parentB:
+  child2: i am child2
+parentC:
+  child1: me child1
+```
+
+## Set array path
+Given a sample.yml file of:
+```yaml
+a:
+  - cat
+  - frog
+```
+then
+```bash
+yq 'setpath(["a", 0]; "things")' sample.yml
+```
+will output
+```yaml
+a:
+  - things
+  - frog
+```
+
+## Set array path empty
+Running
+```bash
+yq --null-input 'setpath(["a", 0]; "things")'
+```
+will output
+```yaml
+a:
+  - things
+```
+
+## Delete path
+Notice delpaths takes an _array_ of paths.
+
+Given a sample.yml file of:
+```yaml
+a:
+  b: cat
+  c: dog
+  d: frog
+```
+then
+```bash
+yq 'delpaths([["a", "c"], ["a", "d"]])' sample.yml
+```
+will output
+```yaml
+a:
+  b: cat
+```
+
+## Delete array path
+Given a sample.yml file of:
+```yaml
+a:
+  - cat
+  - frog
+```
+then
+```bash
+yq 'delpaths([["a", 0]])' sample.yml
+```
+will output
+```yaml
+a:
+  - frog
+```
+
+## Delete - wrong parameter
+delpaths does not work with a single path array
+
+Given a sample.yml file of:
+```yaml
+a:
+  - cat
+  - frog
+```
+then
+```bash
+yq 'delpaths(["a", 0])' sample.yml
+```
+will output
+```bash
+Error: DELPATHS: expected entry [0] to be a sequence, but its a !!str. Note that delpaths takes an array of path arrays, e.g. [["a", "b"]]
+```
+
@@ -4,12 +4,6 @@ Filter a map by the specified list of keys. Map is returned with the key in the

 Similarly, filter an array by the specified list of indices.

-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;
-
-`yq e <exp> <file>`
-{% endhint %}
-
 ## Pick keys from map
 Note that the order of the keys matches the pick order and non existent keys are skipped.

@@ -2,12 +2,6 @@

 Pipe the results of an expression into another. Like the bash operator.

-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;
-
-`yq e <exp> <file>`
-{% endhint %}
-
 ## Simple Pipe
 Given a sample.yml file of:
 ```yaml
@@ -19,12 +19,6 @@ For instance to set the `style` of all nodes in a yaml doc, including the map ke
 ```bash
 yq '... style= "flow"' file.yaml
 ```
-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;
-
-`yq e <exp> <file>`
-{% endhint %}
-
 ## Recurse map (values only)
 Given a sample.yml file of:
 ```yaml
@@ -21,12 +21,6 @@ Reduce syntax in `yq` is a little different from `jq` - as `yq` (currently) isn'

 To that end, the reduce operator is called `ireduce` for backwards compatibility if a `jq` like prefix version of `reduce` is ever added.

-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;
-
-`yq e <exp> <file>`
-{% endhint %}
-
 ## Sum numbers
 Given a sample.yml file of:
 ```yaml
@@ -2,12 +2,6 @@

 Reverses the order of the items in an array 

-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;
-
-`yq e <exp> <file>`
-{% endhint %}
-
 ## Reverse
 Given a sample.yml file of:
 ```yaml
@@ -8,12 +8,6 @@ Select is used to filter arrays and maps by a boolean expression.
 - comparison (`>=`, `<` etc) operators [here](https://mikefarah.gitbook.io/yq/operators/compare)
 - boolean operators (`and`, `or`, `any` etc) [here](https://mikefarah.gitbook.io/yq/operators/boolean-operators)

-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;
-
-`yq e <exp> <file>`
-{% endhint %}
-
 ## Select elements from array using wildcard prefix
 Given a sample.yml file of:
 ```yaml
@@ -0,0 +1,105 @@
+# Slice/Splice Array
+
+The slice array operator takes an array as input and returns a subarray. Like the `jq` equivalent, `.[10:15]` will return an array of length 5, starting from index 10 inclusive, up to index 15 exclusive. Negative numbers count backwards from the end of the array.
+
+You may leave out the first or second number, which will will refer to the start or end of the array respectively.
+
+## Slicing arrays
+Given a sample.yml file of:
+```yaml
+- cat
+- dog
+- frog
+- cow
+```
+then
+```bash
+yq '.[1:3]' sample.yml
+```
+will output
+```yaml
+- dog
+- frog
+```
+
+## Slicing arrays - without the first number
+Starts from the start of the array
+
+Given a sample.yml file of:
+```yaml
+- cat
+- dog
+- frog
+- cow
+```
+then
+```bash
+yq '.[:2]' sample.yml
+```
+will output
+```yaml
+- cat
+- dog
+```
+
+## Slicing arrays - without the second number
+Finishes at the end of the array
+
+Given a sample.yml file of:
+```yaml
+- cat
+- dog
+- frog
+- cow
+```
+then
+```bash
+yq '.[2:]' sample.yml
+```
+will output
+```yaml
+- frog
+- cow
+```
+
+## Slicing arrays - use negative numbers to count backwards from the end
+Given a sample.yml file of:
+```yaml
+- cat
+- dog
+- frog
+- cow
+```
+then
+```bash
+yq '.[1:-1]' sample.yml
+```
+will output
+```yaml
+- dog
+- frog
+```
+
+## Inserting into the middle of an array
+using an expression to find the index
+
+Given a sample.yml file of:
+```yaml
+- cat
+- dog
+- frog
+- cow
+```
+then
+```bash
+yq '(.[] | select(. == "dog") | key + 1) as $pos | .[0:($pos)] + ["rabbit"] + .[$pos:]' sample.yml
+```
+will output
+```yaml
+- cat
+- dog
+- rabbit
+- frog
+- cow
+```
+
@@ -12,11 +12,8 @@ diff file1.yml file2.yml

 Note that `yq` does not yet consider anchors when sorting by keys - this may result in invalid yaml documents if your are using merge anchors.

-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;
-
-`yq e <exp> <file>`
-{% endhint %}
+For more advanced sorting, using `to_entries` to convert the map to an array, then sort/process the array as you like (e.g. using `sort_by`) and convert back to a map using `from_entries`.
+See [here](https://mikefarah.gitbook.io/yq/operators/entries#custom-sort-map-keys) for an example. 

 ## Sort keys of map
 Given a sample.yml file of:
@@ -7,12 +7,6 @@ To sort by descending order, pipe the results through the `reverse` operator aft
 Note that at this stage, `yq` only sorts scalar fields.


-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;
-
-`yq e <exp> <file>`
-{% endhint %}
-
 ## Sort by string field
 Given a sample.yml file of:
 ```yaml
@@ -141,6 +135,24 @@ will output
 - a: 100
 ```

+## Sort by custom date field
+Given a sample.yml file of:
+```yaml
+- a: 12-Jun-2011
+- a: 23-Dec-2010
+- a: 10-Aug-2011
+```
+then
+```bash
+yq 'with_dtf("02-Jan-2006"; sort_by(.a))' sample.yml
+```
+will output
+```yaml
+- a: 23-Dec-2010
+- a: 12-Jun-2011
+- a: 10-Aug-2011
+```
+
 ## Sort, nulls come first
 Given a sample.yml file of:
 ```yaml
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Mike Farah	42e7c3cdf0	Bumping version	2022-11-28 08:36:05 +11:00
Mike Farah	f9a8ba91e4	Preparing release	2022-11-28 08:34:58 +11:00
Mike Farah	1617e5de19	XML Decoder: Comment parsing tweak	2022-11-27 19:14:41 +11:00
Mike Farah	77998d1bb3	XML Decoder: Fixed processing comments in empty XML #1446	2022-11-27 17:58:37 +11:00
Mike Farah	68f47c02c8	XML Decoder: Checking for invalid content outside of a root node #1448	2022-11-27 17:46:50 +11:00
Mike Farah	02be2b2918	Fixed issue where content surrounding tags are lost #1447	2022-11-27 17:29:27 +11:00
Mike Farah	43233ce62e	Fixed xml decode bug when there is content after a comment	2022-11-25 13:21:19 +11:00
Mike Farah	51043770bf	Fixed log level, deleted commented code	2022-11-25 12:09:45 +11:00
Mike Farah	4478bd14c9	Fixed loading yaml with header issue #1445	2022-11-25 12:05:56 +11:00
Mike Farah	91dc315fdb	updating release notes	2022-11-15 21:20:41 +11:00
Mike Farah	a6d1a52e33	Fixing versioning code for release	2022-11-15 20:08:58 +11:00
Mike Farah	fd35530f35	Set tags on maps,arrays when decoding json #1422	2022-11-15 11:42:31 +11:00
Mike Farah	cb609a1886	Fixed bad line entry for #1422	2022-11-15 11:35:31 +11:00
Mike Farah	b202ccc5dc	Updating snapcraft release notes	2022-11-15 11:05:56 +11:00
Mike Farah	ee226b47fc	Preparing release	2022-11-15 10:05:08 +11:00
Mike Farah	9edff1f22c	Use new github action syntax	2022-11-14 17:40:12 +11:00
Mike Farah	67864ffdab	Use new github action syntax	2022-11-14 17:35:52 +11:00
Mike Farah	b55381f34e	fixing snap release	2022-11-14 17:28:44 +11:00
Mike Farah	b6ad314dbb	fixing snap release	2022-11-14 17:28:15 +11:00
Mike Farah	762f46ed54	include yaml directives #1424	2022-11-14 17:22:13 +11:00
Mike Farah	63db5de4e2	(attempt) to automate snap release	2022-11-14 17:00:50 +11:00
Mike Farah	9606957a0e	Fixed auto versioning in xcompile.sh	2022-11-14 16:43:06 +11:00
Mike Farah	e02bb71948	Fixed parsing of newline character in string expression #1430	2022-11-14 16:40:59 +11:00
Takumi SueandGitHub	1fd96e168e	Fix mis-versioning due to hardcoding cmd.Version (#1431 )	2022-11-14 16:38:43 +11:00
Mike Farah	ffc20f7e2c	Length compares to null instead of 0 issue #1427	2022-11-14 16:14:08 +11:00
Mike Farah	3e795d020d	Updating docs	2022-11-13 11:16:01 +11:00
Mike Farah	88ce6ffcbe	Hotfix: actually update the default xml attribute prefix #1428	2022-11-13 11:13:34 +11:00
Mike Farah	83ef7ee5cf	Releasing yq	2022-11-11 14:55:03 +11:00
Mike Farah	817287ec90	Check merge alias is a map #1425	2022-11-11 14:53:45 +11:00
Mike Farah	c5994a8b28	Simplified json error check	2022-11-11 09:50:22 +11:00
Mike Farah	3435fee1f9	Fixed handling of json decoding error #1423	2022-11-10 22:33:43 +11:00
Mike Farah	ececd00fbd	Updated default xml naming prefix	2022-11-10 22:22:55 +11:00
Mike Farah	46dbd0c859	Updating release notes	2022-11-10 22:09:30 +11:00
Mike Farah	cad809515c	Refactoring	2022-11-10 22:07:53 +11:00
Mike Farah	1d35134310	Unwrap scalar now works for JSON encoding when explicitly set #1409	2022-11-10 19:21:18 +11:00
Mike Farah	fae1dbf0be	Preparing release notes	2022-11-10 18:06:43 +11:00
Mike Farah	04847502bf	Can use expressions in slice #1419	2022-11-10 18:03:18 +11:00
Mike Farah	af7e36bd47	Improved set path example	2022-11-10 10:28:29 +11:00
Mike Farah	fdad478684	Improved set path example	2022-11-10 10:22:08 +11:00
Mike Farah	22f376bbfd	Added set path example	2022-11-10 10:17:08 +11:00
Mike Farah	688fe55bb9	Added test for err fix	2022-11-09 21:46:00 +11:00
JingyuandGitHub	91b3fb2af3	fix lib.go handled error (#1418 )	2022-11-09 21:36:53 +11:00
Mike Farah	bd5e5dc965	Array to map operator for #1415	2022-11-08 13:40:00 +11:00
Mike Farah	c915113120	Fixed doc regarding getting the latest version	2022-11-08 08:50:48 +11:00
Mike Farah	cf02b90624	Fixed sorting by date #1412	2022-11-04 12:21:12 +11:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	7f5dda93c6	Bump golang from 1.19.2 to 1.19.3 (#1411 ) Bumps golang from 1.19.2 to 1.19.3. --- updated-dependencies: - dependency-name: golang dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-11-03 14:11:41 +11:00
Mike Farah	a0be871a9d	Only maps can be encoded to XML #1408	2022-11-02 21:41:39 +11:00
Mike Farah	5af062a86f	Added subset example	2022-11-01 09:29:05 +11:00
Mike Farah	ed551bf339	Added extra parseSnippet test	2022-11-01 09:15:59 +11:00
Mike Farah	5e490527de	Bumping version	2022-10-31 09:05:56 +11:00
Mike Farah	c887042a1b	Fixing null csv bug #1404	2022-10-30 22:02:08 +11:00
Mike Farah	0cc5e75432	Bumping version	2022-10-29 18:24:19 +11:00
Mike Farah	6d6cd43255	docs	2022-10-29 18:22:30 +11:00
Mike FarahandGitHub	d99614f55a	Slice array (#1403 )	2022-10-29 18:15:21 +11:00
Mike FarahandGitHub	880397d549	Refactored decoder responsibilities (#1402 ) - improved comment handling - yaml decoder now responsible for leading content work around	2022-10-28 14:16:46 +11:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	46c32f4c79	Bump github.com/spf13/cobra from 1.6.0 to 1.6.1 (#1399 ) Bumps [github.com/spf13/cobra](https://github.com/spf13/cobra) from 1.6.0 to 1.6.1. - [Release notes](https://github.com/spf13/cobra/releases) - [Commits](https://github.com/spf13/cobra/compare/v1.6.0...v1.6.1) --- updated-dependencies: - dependency-name: github.com/spf13/cobra dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-10-27 15:11:06 +11:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	b25114c1fc	Bump github.com/goccy/go-yaml from 1.9.5 to 1.9.6 (#1400 ) Bumps [github.com/goccy/go-yaml](https://github.com/goccy/go-yaml) from 1.9.5 to 1.9.6. - [Release notes](https://github.com/goccy/go-yaml/releases) - [Changelog](https://github.com/goccy/go-yaml/blob/master/CHANGELOG.md) - [Commits](https://github.com/goccy/go-yaml/compare/v1.9.5...v1.9.6) --- updated-dependencies: - dependency-name: github.com/goccy/go-yaml dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-10-27 15:10:56 +11:00
Mike Farah	44552c151b	Improving readme	2022-10-26 13:12:20 +11:00
Sho CeandGitHub	108cfeb4bb	README: how to easily download the latest release (#1397 )	2022-10-25 20:43:24 +11:00
Mike Farah	24bbffd71e	xml prefs should be part of API	2022-10-25 14:27:16 +11:00
Mike Farah	c62e18f9b2	Fixed load operator bug	2022-10-25 13:30:38 +11:00
Mike Farah	c1640fb10d	Removing old version notice	2022-10-25 12:47:13 +11:00
Mike Farah	0c33690596	Updating release notes	2022-10-25 09:36:00 +11:00
Mike Farah	6bf3defe85	Fixing updates in square brackets #1342	2022-10-24 17:55:19 +11:00
Mike FarahandGitHub	6d6b693fb3	Added XML processing instructions and directive support (#1396 )	2022-10-24 10:09:42 +11:00
Mike Farah	14f827b9b7	Added nix installation instructions	2022-10-24 09:42:22 +11:00
Mike Farah	eac988abdc	Updated release to always use latest go	2022-10-23 14:50:00 +11:00
Mike Farah	ea4a50f6e9	Fixing matt phelps username reference	2022-10-20 09:59:36 +11:00
Mike Farah	6c94869329	test yq action can now be run manually	2022-10-19 12:04:52 +11:00
Mike Farah	5ef9101913	Updated release notes	2022-10-19 11:25:56 +11:00
Mike Farah	1f0881fb5f	Bumping version	2022-10-19 11:14:32 +11:00
Mike Farah	600e132fb6	Actions use built docker-images	2022-10-19 11:10:50 +11:00
Mike Farah	f571b163e5	Merge branch 'action-use-published-docker'	2022-10-19 11:09:06 +11:00
Mike Farah	0b235b367d	action uses built docker image	2022-10-19 11:06:41 +11:00
Mike Farah	a264833c06	test action using built docker image	2022-10-19 11:05:09 +11:00
Mike Farah	22c44d3ecb	action uses built docker image	2022-10-19 11:04:53 +11:00
Mike Farah	8ed817916f	muiltiline github action test	2022-10-19 10:59:09 +11:00
Mike Farah	cb4c41eebf	different name to docker github action	2022-10-19 10:49:04 +11:00
Mike Farah	ecc490a0fa	Attempt to prebuild github action	2022-10-19 10:48:16 +11:00
mattphelps-8451andGitHub	8a45e02cbb	Fix broken multiline strings from `dee0c94` (#1390 ) 1. Escaping behavior changed as a result of set-output deprecation. - Otherwise you'll encounter breaking changes for example "appVersion: "1.0.0"%0Aapplica..." where %0A is escaped newline \n but shouldn't be escaped. 2. Delimeter is required for multiline strings. - See here: https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#multiline-strings - Delimeter must be random. UUID is a solid choice. I verified that /proc/sys/kernel/random/uuid does indeed exist on the yq image.	2022-10-19 10:16:07 +11:00
mattphelps-8451andGitHub	28a6c9c532	Fix set-output deprecate; keys are case sensitive (#1383 )	2022-10-18 13:41:58 +11:00
Mike Farah	dee0c94346	Updating action to new entrypoint format	2022-10-17 15:33:12 +11:00
Mike Farah	780ae29c2b	Improved github action test	2022-10-17 15:28:38 +11:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>Mike Farah	9ca20451b7	Bump github.com/spf13/cobra from 1.5.0 to 1.6.0 (#1380 ) * Bump github.com/spf13/cobra from 1.5.0 to 1.6.0 Bumps [github.com/spf13/cobra](https://github.com/spf13/cobra) from 1.5.0 to 1.6.0. - [Release notes](https://github.com/spf13/cobra/releases) - [Commits](https://github.com/spf13/cobra/compare/v1.5.0...v1.6.0) --- updated-dependencies: - dependency-name: github.com/spf13/cobra dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Fixed deprecated usage of ExactArgs Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Mike Farah <mikefarah@gmail.com>	2022-10-17 15:15:19 +11:00
Mike Farah	6db7acbf69	Added custom sort map example	2022-10-17 15:03:47 +11:00
Mike Farah	d16ee386d2	Fixed broken link	2022-10-14 14:24:57 +11:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	c3f4bb8156	Bump github.com/alecthomas/repr from 0.1.0 to 0.1.1 (#1379 ) Bumps [github.com/alecthomas/repr](https://github.com/alecthomas/repr) from 0.1.0 to 0.1.1. - [Release notes](https://github.com/alecthomas/repr/releases) - [Commits](https://github.com/alecthomas/repr/compare/v0.1.0...v0.1.1) --- updated-dependencies: - dependency-name: github.com/alecthomas/repr dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-10-14 09:33:34 +11:00
Mike Farah	c640888133	Fixed bug - can now delete documents #1377	2022-10-11 11:48:18 +11:00
Mike Farah	33ec66cfdd	Fixed handling of UTF8 encoded CSVs #1373	2022-10-08 13:12:50 +11:00
Mike Farah	1b8d399de4	Detect and fail on missing closing brackets #1366	2022-10-08 12:44:54 +11:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	48c36efe07	Bump golang from 1.19.1 to 1.19.2 (#1375 ) Bumps golang from 1.19.1 to 1.19.2. --- updated-dependencies: - dependency-name: golang dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-10-06 15:07:30 +11:00
Mike Farah	f27a3c5c92	Added winget instructions	2022-10-05 20:51:48 +11:00
Mike Farah	ffe635f943	Bumping version	2022-10-05 20:15:02 +11:00
Mike Farah	b5b81abb90	Added DELPATHS operator	2022-10-05 20:09:53 +11:00
Mike Farah	da3f3b93b4	Added SETPATH operator	2022-10-05 14:12:08 +11:00
Mike Farah	7f4c8e1c78	Added comment operator examples	2022-09-30 11:30:31 +10:00
Mike Farah	60f30f8a48	Added comment operator examples	2022-09-30 11:26:36 +10:00
Mike Farah	2362451fda	Added comment operator examples	2022-09-30 11:22:58 +10:00
Mike Farah	55a7fdfd8a	Added is_key operator	2022-09-30 10:27:35 +10:00
Mike Farah	3dd5f0c80c	Added another comment test	2022-09-30 10:15:41 +10:00
Mike Farah	be304a1102	Added another comment test	2022-09-30 09:46:07 +10:00
Mike Farah	a4b60039ca	Added examples of header-preprocess needing to be turned off	2022-09-30 09:36:38 +10:00
Mike Farah	b20477210f	Added missing validation for adding sequences to maps #1341	2022-09-16 10:04:48 +10:00
Mike Farah	9b6961875e	Bumping version	2022-09-09 12:29:41 +10:00
Mike Farah	51b64e6cf3	Fixing relative merge bug #1333	2022-09-09 12:28:38 +10:00
Mike Farah	851d93e5a8	Bumping version	2022-09-09 11:23:14 +10:00
Mike Farah	9291865eed	Preparing release notes	2022-09-09 11:21:53 +10:00
Mike Farah	fb330ebcb0	Updated github action to run as root	2022-09-09 11:14:25 +10:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	ae7a48df2d	Bump github.com/elliotchance/orderedmap from 1.4.0 to 1.5.0 (#1331 ) Bumps [github.com/elliotchance/orderedmap](https://github.com/elliotchance/orderedmap) from 1.4.0 to 1.5.0. - [Release notes](https://github.com/elliotchance/orderedmap/releases) - [Commits](https://github.com/elliotchance/orderedmap/compare/v1.4.0...v1.5.0) --- updated-dependencies: - dependency-name: github.com/elliotchance/orderedmap dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-09-09 11:05:46 +10:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2e3c50c535	Bump golang from 1.19.0 to 1.19.1 (#1330 ) Bumps golang from 1.19.0 to 1.19.1. --- updated-dependencies: - dependency-name: golang dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-09-09 11:05:36 +10:00
Mike Farah	a64eea3b1a	Fixed bug in alternative op, dont evaluate RHS if LHS is truthy	2022-09-09 11:05:12 +10:00
AbirdcflyandGitHub	a12c1a6610	chore: remove duplicate word in doc (#1322 ) Signed-off-by: Abirdcfly <fp544037857@gmail.com> Signed-off-by: Abirdcfly <fp544037857@gmail.com>	2022-08-31 18:43:03 +10:00
Mike Farah	bf6b4d3941	Fixed footer comment issue #1321	2022-08-30 11:24:26 +10:00
Mike Farah	3d11b0545d	Bumping version	2022-08-29 15:41:07 +10:00
Mike Farah	b4ca184108	Added new "c" flag to clobber custom tags when needed	2022-08-29 15:38:21 +10:00
Mike Farah	b1a40a9fb7	Bumped go compiler; bumped quality tools	2022-08-29 14:13:15 +10:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	187d9523c6	Bump github.com/goccy/go-json from 0.9.10 to 0.9.11 (#1314 ) Bumps [github.com/goccy/go-json](https://github.com/goccy/go-json) from 0.9.10 to 0.9.11. - [Release notes](https://github.com/goccy/go-json/releases) - [Changelog](https://github.com/goccy/go-json/blob/master/CHANGELOG.md) - [Commits](https://github.com/goccy/go-json/compare/v0.9.10...v0.9.11) --- updated-dependencies: - dependency-name: github.com/goccy/go-json dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-08-25 13:41:11 +10:00
Matti SchneiderandGitHub	a44938101d	Document dynamic get and output in GitHub Action (#1303 )	2022-08-14 10:40:41 +10:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	a4634cdc86	Bump github.com/alecthomas/participle/v2 (#1302 ) Bumps [github.com/alecthomas/participle/v2](https://github.com/alecthomas/participle) from 2.0.0-beta.4 to 2.0.0-beta.5. - [Release notes](https://github.com/alecthomas/participle/releases) - [Changelog](https://github.com/alecthomas/participle/blob/master/CHANGES.md) - [Commits](https://github.com/alecthomas/participle/compare/v2.0.0-beta.4...v2.0.0-beta.5) --- updated-dependencies: - dependency-name: github.com/alecthomas/participle/v2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-08-11 13:47:22 +10:00
Mike Farah	a696dceea4	Added string trim operator	2022-08-08 13:35:57 +10:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	989dabac5a	Bump golang from 1.18.4 to 1.19.0 (#1292 ) Bumps golang from 1.18.4 to 1.19.0. --- updated-dependencies: - dependency-name: golang dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-08-04 14:23:20 +10:00
Mike Farah	8eac186a35	Add link to discussions on readme	2022-08-01 16:55:55 +10:00
Mike Farah	1c7dc0e88a	Bug fix	2022-08-01 16:27:42 +10:00
Mike Farah	2c9b5be408	Fixed json decode to maintain key order	2022-08-01 16:26:43 +10:00
Mike Farah	a91a8ccc66	Improving docs	2022-08-01 14:12:35 +10:00
Mike Farah	6dfef716d7	Bumping version	2022-08-01 14:00:13 +10:00
Mike Farah	06a2b9d426	updating release notes	2022-08-01 13:47:24 +10:00
Mike Farah	6cd0892d59	Added SELinux note #1289	2022-08-01 13:47:16 +10:00
Mike Farah	51b3985aa0	Preparing release notes	2022-08-01 13:40:41 +10:00
Mike FarahandGitHub	c8815f5ab9	Csv decoder (#1290 ) * WIP: adding CSV decoder * Adding CSV decoder * Added CSV roundtrip * Fixing from review	2022-08-01 10:28:34 +10:00
Jih-Wei, LiangandGitHub	3c222d8707	Added StringEvaluator for evaluating string input #1266 (#1278 )	2022-08-01 08:50:56 +10:00