Fixing merge anchor key order

also add a unit test for func FormatStringFromFilename to cover such case

.github/ISSUE_TEMPLATE/bug_report_v3.md

@@ -1,51 +0,0 @@
----
-name: Bug report - V3
-about: Create a report to help us improve
-title: ''
-labels: bug, v3
-assignees: ''
-
----
-
-**Describe the bug**
-A clear and concise description of what the bug is.
-
-Note that any how to questions should be posted in the discussion board and not raised as an issue.
-
-Version of yq: 3.X.X
-Operating system: mac/linux/windows/....
-Installed via: docker/binary release/homebrew/snap/...
-
-**Input Yaml**
-Concise yaml document(s) (as simple as possible to show the bug, please keep it to 10 lines or less)
-data1.yml:
-```yaml
-this: should really work
-```
-
-data2.yml:
-```yaml
-but: it strangely didn't
-```
-
-**Command**
-The command you ran:
-```
-yq merge data1.yml data2.yml
-```
-
-**Actual behavior**
-
-```yaml
-cat: meow
-```
-
-**Expected behavior**
-
-```yaml
-this: should really work
-but: it strangely didn't
-```
-
-**Additional context**
-Add any other context about the problem here.

.github/workflows/codeql.yml

@@ -38,11 +38,11 @@ jobs:
 
     steps:
     - name: Checkout repository
-      uses: actions/checkout@v3
+      uses: actions/checkout@v4
 
     # Initializes the CodeQL tools for scanning.
     - name: Initialize CodeQL
-      uses: github/codeql-action/init@v2
+      uses: github/codeql-action/init@v3
       with:
         languages: ${{ matrix.language }}
         # If you wish to specify custom queries, you can do so here or in a config file.
@@ -53,7 +53,7 @@ jobs:
     # Autobuild attempts to build any compiled languages  (C/C++, C#, or Java).
     # If this step fails, then you should remove it and run the build manually (see below)
     - name: Autobuild
-      uses: github/codeql-action/autobuild@v2
+      uses: github/codeql-action/autobuild@v3
 
     # ℹ️ Command-line programs to run using the OS shell.
     # 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun
@@ -67,4 +67,4 @@ jobs:
     #   make release
 
     - name: Perform CodeQL Analysis
-      uses: github/codeql-action/analyze@v2
+      uses: github/codeql-action/analyze@v3

.github/workflows/docker-release.yml

@@ -14,22 +14,35 @@ jobs:
       IMAGE_NAME: mikefarah/yq
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
 
       - name: Set up QEMU
-        uses: docker/setup-qemu-action@v2
+        uses: docker/setup-qemu-action@v3
         with:
           platforms: all
 
       - name: Set up Docker Buildx
         id: buildx
-        uses: docker/setup-buildx-action@v2
+        uses: docker/setup-buildx-action@v3
         with:
           version: latest
 
       - name: Available platforms
         run: echo ${{ steps.buildx.outputs.platforms }} && docker version
 
+      - name: Login to Docker Hub
+        uses: docker/login-action@v3
+        with:
+          username: ${{ secrets.DOCKER_USERNAME }}
+          password: ${{ secrets.DOCKER_PASSWORD }}
+
+      - name: Login to GitHub Container Registry
+        uses: docker/login-action@v3
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+
       - name: Build and push image
         run: |
           echo "GithubRef: ${GITHUB_REF}"
@@ -38,14 +51,13 @@ jobs:
           IMAGE_VERSION=${VERSION:1}
           echo "IMAGE_VERSION: ${IMAGE_VERSION}"
 
-          PLATFORMS="linux/amd64,linux/ppc64le,linux/arm64"
+          PLATFORMS="linux/amd64,linux/ppc64le,linux/arm64,linux/arm/v7,linux/s390x"
 
           echo "Building and pushing version ${IMAGE_VERSION} of image ${IMAGE_NAME}"
-          echo '${{ secrets.DOCKER_PASSWORD }}' | docker login -u '${{ secrets.DOCKER_USERNAME }}' --password-stdin
           docker buildx build \
             --label "org.opencontainers.image.authors=https://github.com/mikefarah/yq/graphs/contributors" \
             --label "org.opencontainers.image.created=$(date --rfc-3339=seconds)" \
-            --label "org.opencontainers.image.description=yq is a portable command-line YAML processor" \
+            --label "org.opencontainers.image.description=yq is a portable command-line data file processor" \
             --label "org.opencontainers.image.documentation=https://mikefarah.gitbook.io/yq/" \
             --label "org.opencontainers.image.licenses=MIT" \
             --label "org.opencontainers.image.revision=$(git rev-parse HEAD)" \
@@ -59,4 +71,30 @@ jobs:
             -t "${IMAGE_NAME}:${IMAGE_VERSION}" \
             -t "${IMAGE_NAME}:4" \
             -t "${IMAGE_NAME}:latest" \
+            -t "ghcr.io/${IMAGE_NAME}:${IMAGE_VERSION}" \
+            -t "ghcr.io/${IMAGE_NAME}:4" \
+            -t "ghcr.io/${IMAGE_NAME}:latest" \
+            .
+
+          cd github-action
+          docker buildx build \
+            --label "org.opencontainers.image.authors=https://github.com/mikefarah/yq/graphs/contributors" \
+            --label "org.opencontainers.image.created=$(date --rfc-3339=seconds)" \
+            --label "org.opencontainers.image.description=yq is a portable command-line data file processor" \
+            --label "org.opencontainers.image.documentation=https://mikefarah.gitbook.io/yq/" \
+            --label "org.opencontainers.image.licenses=MIT" \
+            --label "org.opencontainers.image.revision=$(git rev-parse HEAD)" \
+            --label "org.opencontainers.image.source=https://github.com/mikefarah/yq" \
+            --label "org.opencontainers.image.title=yq" \
+            --label "org.opencontainers.image.url=https://mikefarah.gitbook.io/yq/" \
+            --label "org.opencontainers.image.version=${IMAGE_VERSION}" \
+            --platform "${PLATFORMS}" \
+            --pull \
+            --push \
+            -t "${IMAGE_NAME}:${IMAGE_VERSION}-githubaction" \
+            -t "${IMAGE_NAME}:4-githubaction" \
+            -t "${IMAGE_NAME}:latest-githubaction" \
+            -t "ghcr.io/${IMAGE_NAME}:${IMAGE_VERSION}-githubaction" \
+            -t "ghrc.io/${IMAGE_NAME}:4-githubaction" \
+            -t "ghcr.io/${IMAGE_NAME}:latest-githubaction" \
             .

.github/workflows/go.yml

@@ -10,14 +10,14 @@ jobs:
     runs-on: ubuntu-latest
     steps:
 
-    - name: Set up Go 1.19
-      uses: actions/setup-go@v3
+    - name: Set up Go
+      uses: actions/setup-go@v5
       with:
-        go-version: 1.19
+        go-version: '^1.20'
       id: go
 
     - name: Check out code into the Go module directory
-      uses: actions/checkout@v3
+      uses: actions/checkout@v4
 
     - name: Get dependencies
       run: |

.github/workflows/release.yml

@@ -2,26 +2,26 @@ name: Release YQ
 on:
   push:
     tags:
-      - 'v*'
+      - 'v4.*'
       - 'draft-*'
 
 jobs:
   publishGitRelease:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
-      - uses: actions/setup-go@v3
+      - uses: actions/checkout@v4
+      - uses: actions/setup-go@v5
         with:
-          go-version: '^1.19'
+          go-version: '^1.20'
+          check-latest: true
       - name: Compile man page markup
         id: gen-man-page-md
         run: |
           ./scripts/generate-man-page-md.sh
-          echo "::set-output name=man-page-md::man.md"
 
       - name: Get the version
         id: get_version
-        run: echo ::set-output name=VERSION::${GITHUB_REF##*/}
+        run: echo "VERSION=${GITHUB_REF##*/}" >> "${GITHUB_OUTPUT}"
 
       - name: Generate man page
         uses: docker://pandoc/core:2.14.2
@@ -35,14 +35,12 @@ jobs:
             --variable=header:"yq (https://github.com/mikefarah/yq/) version ${{ steps.get_version.outputs.VERSION }}"
             --variable=author:"Mike Farah"
             --output=yq.1
-            ${{ steps.gen-man-page-md.outputs.man-page-md }}
+            man.md
 
       - name: Cross compile
         run: |
           sudo apt-get install rhash -y
-          go install github.com/mitchellh/gox@v1.0.1
-          mkdir -p build
-          cp yq.1 build/yq.1
+          go install github.com/goreleaser/goreleaser/v2@latest
           ./scripts/xcompile.sh
 
       - name: Release

.github/workflows/snap-release.yml

@@ -0,0 +1,23 @@
+name: Release Snap
+
+on:
+  release:
+    types: [released]
+
+  # Allows you to run this workflow manually from the Actions tab
+  workflow_dispatch:
+
+jobs:
+    buildSnap:
+      environment: snap
+      runs-on: ubuntu-latest
+      steps:
+        - uses: actions/checkout@v4
+        - uses: snapcore/action-build@v1
+          id: build
+        - uses: snapcore/action-publish@v1
+          env:
+            SNAPCRAFT_STORE_CREDENTIALS: ${{ secrets.STORE_LOGIN }}
+          with:
+            snap: ${{ steps.build.outputs.snap }}
+            release: stable

.github/workflows/test-yq.yml

@@ -1,5 +1,8 @@
 name: Test Yq Action
-on: [push]
+on: 
+  push:
+  workflow_dispatch:
+
 permissions:
   contents: read
 
@@ -10,13 +13,16 @@ jobs:
     runs-on: ubuntu-latest
 
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
       - name: Get test
+        id: get_value
         uses: mikefarah/yq@master
         with:
-          cmd: yq eval '.a' examples/sample.yaml
+          cmd: yq '.test' examples/multiline-text.yaml
+      - name: Multiline test
+        run: echo "### It was [${{ steps.get_value.outputs.result }}]" >> $GITHUB_STEP_SUMMARY
       - name: Write inplace test
         id: lookupSdkVersion
         uses: mikefarah/yq@master
         with:
-          cmd: yq eval -i '.b.c = 5' examples/sample.yaml
+          cmd: yq -i '.b.c = 5' examples/sample.yaml

.gitignore

@@ -22,6 +22,7 @@ _cgo_gotypes.go
 _cgo_export.*
 
 _testmain.go
+cover.out
 coverage.out
 coverage.html
 *.exe
@@ -41,7 +42,11 @@ yq*.snap
 
 test.yml
 test*.yml
+test*.xml
+test*.toml
 test*.yaml
+test_dir1/
+test_dir2/
 0.yml
 1.yml
 2.yml

.golangci.bck.yml

@@ -0,0 +1,38 @@
+run:
+  timeout: 5m
+linters:
+  enable:
+    - asciicheck
+    - depguard
+    - errorlint
+    - gci
+    - gochecknoinits
+    - gofmt
+    - goimports
+    - gosec
+    - gosimple
+    - staticcheck
+    - unused
+    - misspell
+    - nakedret
+    - nolintlint
+    - predeclared
+    - revive
+    - unconvert
+    - unparam
+linters-settings:
+  depguard:
+    rules:
+      prevent_unmaintained_packages:
+        list-mode: lax
+        files:
+          - $all
+          - "!$test"
+        deny:
+          - pkg: io/ioutil
+            desc: "replaced by io and os packages since Go 1.16: https://tip.golang.org/doc/go1.16#ioutil"
+issues:
+  exclude-rules:
+  - linters:
+    - revive
+    text: "var-naming"

.golangci.yml

@@ -1,16 +1,11 @@
-run:
-  timeout: 5m
+version: "2"
 linters:
   enable:
     - asciicheck
     - depguard
     - errorlint
-    - gci
     - gochecknoinits
-    - gofmt
-    - goimports
     - gosec
-    - megacheck
     - misspell
     - nakedret
     - nolintlint
@@ -18,23 +13,40 @@ linters:
     - revive
     - unconvert
     - unparam
-linters-settings:
-  depguard:
-    list-type: blacklist
-    include-go-root: true
-    packages:
-      - io/ioutil
-    packages-with-error-message:
-      - io/ioutil: "The 'io/ioutil' package is deprecated. Use corresponding 'os' or 'io' functions instead."
-issues:
-  exclude-rules:
-  - linters:
-    - gosec
-    text:  "Implicit memory aliasing in for loop."
-    path: _test\.go
-  - linters:
-    - revive
-    text: "unexported-return"
-  - linters:
-    - revive
-    text: "var-naming"
+  settings:
+    depguard:
+      rules:
+        prevent_unmaintained_packages:
+          list-mode: lax
+          files:
+            - $all
+            - '!$test'
+          deny:
+            - pkg: io/ioutil
+              desc: 'replaced by io and os packages since Go 1.16: https://tip.golang.org/doc/go1.16#ioutil'
+  exclusions:
+    generated: lax
+    presets:
+      - comments
+      - common-false-positives
+      - legacy
+      - std-error-handling
+    rules:
+      - linters:
+          - revive
+        text: var-naming
+    paths:
+      - third_party$
+      - builtin$
+      - examples$
+formatters:
+  enable:
+    - gci
+    - gofmt
+    - goimports
+  exclusions:
+    generated: lax
+    paths:
+      - third_party$
+      - builtin$
+      - examples$

.goreleaser.yaml

@@ -0,0 +1,48 @@
+version: 2
+
+dist: build
+
+builds:
+  - id: yq
+
+    binary: yq_{{ .Os }}_{{ .Arch }}
+
+    ldflags:
+      - -s -w
+
+    env:
+      - CGO_ENABLED=0
+
+    targets:
+      - darwin_amd64
+      - darwin_arm64
+      - freebsd_386
+      - freebsd_amd64
+      - freebsd_arm
+      - linux_386
+      - linux_amd64
+      - linux_arm
+      - linux_arm64
+      - linux_mips
+      - linux_mips64
+      - linux_mips64le
+      - linux_mipsle
+      - linux_ppc64
+      - linux_ppc64le
+      - linux_riscv64
+      - linux_s390x
+      - netbsd_386
+      - netbsd_amd64
+      - netbsd_arm
+      - openbsd_386
+      - openbsd_amd64
+      - windows_386
+      - windows_amd64
+      - windows_arm
+      - windows_arm64
+
+    no_unique_dist_dir: true
+
+release:
+  disable: true
+  skip_upload: true

CONTRIBUTING.md

@@ -1,6 +1,6 @@
-# Development
+.# Development
 
-1. Install (golang)[https://golang.org/]
+1. Install (Golang)[https://golang.org/]
 1. Run  `scripts/devtools.sh` to install the required devtools
 2. Run `make [local] vendor` to install the vendor dependencies
 2. Run `make [local] test` to ensure you can run the existing tests
@@ -12,7 +12,7 @@
 
 # Documentation
 
-The documentation is a bit of a mixed bag (sorry in advanced, I do plan on simplifying it...) - with some parts automatically generated and stiched together and some statically defined.
+The documentation is a bit of a mixed bag (sorry in advance, I do plan on simplifying it...) - with some parts automatically generated and stiched together and some statically defined.
 
 Documentation is written in markdown, and is published in the 'gitbook' branch.
 
@@ -25,6 +25,8 @@ The pipeline will run the tests and automatically concatenate the files together
 
 The first step is to find if what you want is automatically generated or not - start by looking in the master branch. 
 
+Note that PRs with small changes (e.g. minor typos) may not be merged (see https://joel.net/how-one-guy-ruined-hacktoberfest2020-drama).
+
 ### Updating dynamic documentation from master
 - Search for the documentation you want to update. If you find matches in a `*_test.go` file - update that, as that will automatically update the matching `*.md` file 
 - Assuming you are updating a `*_test.go` file, once updated, run the test to regenerated the docs. E.g. for the 'Add' test generated docs, from the pkg/yqlib folder run:

Dockerfile

@@ -1,16 +1,16 @@
-FROM golang:1.19.1 as builder
+FROM golang:1.24.5 AS builder
 
 WORKDIR /go/src/mikefarah/yq
 
 COPY . .
 
-RUN CGO_ENABLED=0 go build .
+RUN CGO_ENABLED=0 go build -ldflags "-s -w" .
 # RUN ./scripts/test.sh -- this too often times out in the github pipeline.
 RUN ./scripts/acceptance.sh
 
 # Choose alpine as a base image to make this useful for CI, as many
 # CI tools expect an interactive shell inside the container
-FROM alpine:3 as production
+FROM alpine:3 AS production
 LABEL maintainer="Mike Farah <mikefarah@users.noreply.github.com>"
 
 COPY --from=builder /go/src/mikefarah/yq/yq /usr/bin/yq

Dockerfile.dev

@@ -1,9 +1,14 @@
-FROM golang:1.19.1
+FROM golang:1.24.5
+
+RUN apt-get update && \
+    apt-get install -y npm && \
+    npm install -g npx cspell@latest
 
 COPY scripts/devtools.sh /opt/devtools.sh
 
-RUN set -e -x \
-    && /opt/devtools.sh
+RUN set -e -x && \
+    /opt/devtools.sh
+
 ENV PATH=/go/bin:$PATH
 
 ENV CGO_ENABLED 0

Makefile

@@ -2,7 +2,7 @@ MAKEFLAGS += --warn-undefined-variables
 SHELL := /bin/bash
 .SHELLFLAGS := -o pipefail -euc
 .DEFAULT_GOAL := install
-ENGINE := $(shell { command -v podman || command -v docker; } 2>/dev/null)
+ENGINE := $(shell { (podman version > /dev/null 2>&1 && command -v podman) || command -v docker; } 2>/dev/null)
 
 include Makefile.variables
 
@@ -84,8 +84,12 @@ format: vendor
 	${ENGINERUN} bash ./scripts/format.sh
 
 
+.PHONY: spelling
+spelling: format
+	${ENGINERUN} bash ./scripts/spelling.sh
+
 .PHONY: secure
-secure: format
+secure: spelling
 	${ENGINERUN} bash ./scripts/secure.sh
 
 .PHONY: check

Makefile.variables

@@ -7,6 +7,7 @@ export GIT_DESCRIBE = $(shell git describe --tags --always)
 LDFLAGS :=
 LDFLAGS += -X main.GitCommit=${GIT_COMMIT}${GIT_DIRTY}
 LDFLAGS += -X main.GitDescribe=${GIT_DESCRIBE}
+LDFLAGS += -w -s
 
 GITHUB_TOKEN ?=
 
@@ -25,6 +26,7 @@ ifeq ($(CYG_CHECK),1)
 else
 	# all non-windows environments
 	ROOT := $(shell pwd)
+	SELINUX := $(shell which getenforce 2>&1 >/dev/null && echo :z)
 endif
 
 DEV_IMAGE := ${PROJECT}_dev
@@ -32,7 +34,7 @@ DEV_IMAGE := ${PROJECT}_dev
 ENGINERUN := ${ENGINE} run --rm \
 	-e LDFLAGS="${LDFLAGS}" \
 	-e GITHUB_TOKEN="${GITHUB_TOKEN}" \
-	-v ${ROOT}/vendor:/go/src \
-	-v ${ROOT}:/${PROJECT}/src/${IMPORT_PATH} \
+	-v ${ROOT}/vendor:/go/src${SELINUX} \
+	-v ${ROOT}:/${PROJECT}/src/${IMPORT_PATH}${SELINUX} \
 	-w /${PROJECT}/src/${IMPORT_PATH} \
 	${DEV_IMAGE}

README.md

@@ -3,31 +3,10 @@
 ![Build](https://github.com/mikefarah/yq/workflows/Build/badge.svg)  ![Docker Pulls](https://img.shields.io/docker/pulls/mikefarah/yq.svg) ![Github Releases (by Release)](https://img.shields.io/github/downloads/mikefarah/yq/total.svg) ![Go Report](https://goreportcard.com/badge/github.com/mikefarah/yq) ![CodeQL](https://github.com/mikefarah/yq/workflows/CodeQL/badge.svg)
 
 
-a lightweight and portable command-line YAML, JSON and XML processor. `yq` uses [jq](https://github.com/stedolan/jq) like syntax but works with yaml files as well as json, xml, properties, csv and tsv. It doesn't yet support everything `jq` does - but it does support the most common operations and functions, and more is being added continuously.
+a lightweight and portable command-line YAML, JSON, INI and XML processor. `yq` uses [jq](https://github.com/stedolan/jq) like syntax but works with yaml files as well as json, xml, ini, properties, csv and tsv. It doesn't yet support everything `jq` does - but it does support the most common operations and functions, and more is being added continuously.
 
 yq is written in go - so you can download a dependency free binary for your platform and you are good to go! If you prefer there are a variety of package managers that can be used as well as Docker and Podman, all listed below.
 
-## Notice for v4.x versions prior to 4.18.1
-Since 4.18.1, yq's 'eval/e' command is the _default_ command and no longer needs to be specified.
-
-Older versions will still need to specify 'eval/e'.
-
-Similarly, '-' is no longer required as a filename to read from STDIN (unless reading from one or more files).
-
-TLDR:
-
-Prior to 4.18.1 
-```bash
-yq e '.cool' - < file.yaml
-```
-
-4.18+ 
-```bash
-yq '.cool' < file.yaml
-```
-
-When merging multiple files together, `eval-all/ea` is still required to tell `yq` to run the expression against all the document at once.
-
 ## Quick Usage Guide
 
 Read a value:
@@ -40,7 +19,7 @@ Pipe from STDIN:
 yq '.a.b[0].c' < file.yaml
 ```
 
-Update a yaml file, inplace
+Update a yaml file, in place
 ```bash
 yq -i '.a.b[0].c = "cool"' file.yaml
 ```
@@ -52,6 +31,10 @@ NAME=mike yq -i '.a.b[0].c = strenv(NAME)' file.yaml
 
 Merge multiple files
 ```bash
+# merge two files
+yq -n 'load("file1.yaml") * load("file2.yaml")'
+
+# merge using globs:
 # note the use of `ea` to evaluate all the files at once
 # instead of in sequence
 yq ea '. as $item ireduce ({}; . * $item )' path/to/*.yml
@@ -66,12 +49,17 @@ yq -i '
 ' file.yaml
 ```
 
-Convert JSON to YAML
+Find and update an item in an array:
 ```bash
-yq -P sample.json
+yq '(.[] | select(.name == "foo") | .address) = "12 cat st"'
 ```
 
-See the [documentation](https://mikefarah.gitbook.io/yq/) for more examples.
+Convert JSON to YAML
+```bash
+yq -Poy sample.json
+```
+
+See [recipes](https://mikefarah.gitbook.io/yq/recipes) for more examples and the [documentation](https://mikefarah.gitbook.io/yq/) for more information.
 
 Take a look at the discussions for [common questions](https://github.com/mikefarah/yq/discussions/categories/q-a), and [cool ideas](https://github.com/mikefarah/yq/discussions/categories/show-and-tell)
 
@@ -80,22 +68,30 @@ Take a look at the discussions for [common questions](https://github.com/mikefar
 ### [Download the latest binary](https://github.com/mikefarah/yq/releases/latest)
 
 ### wget
-Use wget to download the pre-compiled binaries:
+Use wget to download, gzipped pre-compiled binaries:
+
+
+For instance, VERSION=v4.2.0 and BINARY=yq_linux_amd64
 
 #### Compressed via tar.gz
 ```bash
 wget https://github.com/mikefarah/yq/releases/download/${VERSION}/${BINARY}.tar.gz -O - |\
-  tar xz && mv ${BINARY} /usr/bin/yq
+  tar xz && mv ${BINARY} /usr/local/bin/yq
 ```
 
 #### Plain binary
 
 ```bash
-wget https://github.com/mikefarah/yq/releases/download/${VERSION}/${BINARY} -O /usr/bin/yq &&\
-    chmod +x /usr/bin/yq
+wget https://github.com/mikefarah/yq/releases/download/${VERSION}/${BINARY} -O /usr/local/bin/yq &&\
+    chmod +x /usr/local/bin/yq
 ```
 
-For instance, VERSION=v4.2.0 and BINARY=yq_linux_amd64
+#### Latest version
+
+```bash
+wget https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -O /usr/local/bin/yq &&\
+    chmod +x /usr/local/bin/yq
+```
 
 ### MacOS / Linux via Homebrew:
 Using [Homebrew](https://brew.sh/)
@@ -242,7 +238,27 @@ go install github.com/mikefarah/yq/v4@latest
 ## Community Supported Installation methods
 As these are supported by the community :heart: - however, they may be out of date with the officially supported releases.
 
-# Webi
+_Please note that the Debian package (previously supported by @rmescandon) is no longer maintained. Please use an alternative installation method._
+
+
+### X-CMD
+Checkout `yq` on x-cmd: https://x-cmd.com/mod/yq
+
+- Instant Results: See the output of your yq filter in real-time.
+- Error Handling: Encounter a syntax error? It will display the error message and the results of the closest valid filter
+
+Thanks @edwinjhlee!
+
+### Nix
+
+```
+nix profile install nixpkgs#yq-go
+```
+
+See [here](https://search.nixos.org/packages?channel=unstable&show=yq-go&from=0&size=50&sort=relevance&type=packages&query=yq-go)
+
+
+### Webi
 
 ```
 webi yq
@@ -258,6 +274,9 @@ pacman -S go-yq
 ```
 
 ### Windows:
+
+Using [Chocolatey](https://chocolatey.org)
+
 [![Chocolatey](https://img.shields.io/chocolatey/v/yq.svg)](https://chocolatey.org/packages/yq)
 [![Chocolatey](https://img.shields.io/chocolatey/dt/yq.svg)](https://chocolatey.org/packages/yq)
 ```
@@ -265,6 +284,16 @@ choco install yq
 ```
 Supported by @chillum (https://chocolatey.org/packages/yq)
 
+Using [scoop](https://scoop.sh/)
+```
+scoop install main/yq
+```
+
+Using [winget](https://learn.microsoft.com/en-us/windows/package-manager/)
+```
+winget install --id MikeFarah.yq
+```
+
 ### Mac:
 Using [MacPorts](https://www.macports.org/)
 ```
@@ -274,27 +303,31 @@ sudo port install yq
 Supported by @herbygillot (https://ports.macports.org/maintainer/github/herbygillot)
 
 ### Alpine Linux
-- Enable edge/community repo by adding ```$MIRROR/alpine/edge/community``` to ```/etc/apk/repositories```
-- Update database index with ```apk update```
-- Install yq with ```apk add yq```
 
-Supported by Tuan Hoang
-https://pkgs.alpinelinux.org/package/edge/community/x86/yq
+Alpine Linux v3.20+ (and Edge):
+```
+apk add yq-go
+```
 
+Alpine Linux up to v3.19:
+```
+apk add yq
+```
 
-### On Ubuntu 16.04 or higher from Debian package:
-```sh
-sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys CC86BB64
-sudo add-apt-repository ppa:rmescandon/yq
-sudo apt update
-sudo apt install yq -y
+Supported by Tuan Hoang (https://pkgs.alpinelinux.org/packages?name=yq-go)
+
+### Flox:
+
+Flox can be used to install yq on Linux, MacOS, and Windows through WSL.
+
+```
+flox install yq
 ```
-Supported by @rmescandon (https://launchpad.net/~rmescandon/+archive/ubuntu/yq)
 
 ## Features
 - [Detailed documentation with many examples](https://mikefarah.gitbook.io/yq/)
 - Written in portable go, so you can download a lovely dependency free binary
-- Uses similar syntax as `jq` but works with YAML, [JSON](https://mikefarah.gitbook.io/yq/usage/convert) and [XML](https://mikefarah.gitbook.io/yq/usage/xml) files
+- Uses similar syntax as `jq` but works with YAML, INI, [JSON](https://mikefarah.gitbook.io/yq/usage/convert) and [XML](https://mikefarah.gitbook.io/yq/usage/xml) files
 - Fully supports multi document yaml files
 - Supports yaml [front matter](https://mikefarah.gitbook.io/yq/usage/front-matter) blocks (e.g. jekyll/assemble)
 - Colorized yaml output
@@ -302,7 +335,7 @@ Supported by @rmescandon (https://launchpad.net/~rmescandon/+archive/ubuntu/yq)
 - [Deeply data structures](https://mikefarah.gitbook.io/yq/operators/traverse-read)
 - [Sort keys](https://mikefarah.gitbook.io/yq/operators/sort-keys)
 - Manipulate yaml [comments](https://mikefarah.gitbook.io/yq/operators/comment-operators), [styling](https://mikefarah.gitbook.io/yq/operators/style), [tags](https://mikefarah.gitbook.io/yq/operators/tag) and [anchors and aliases](https://mikefarah.gitbook.io/yq/operators/anchor-and-alias-operators).
-- [Update inplace](https://mikefarah.gitbook.io/yq/v/v4.x/commands/evaluate#flags)
+- [Update in place](https://mikefarah.gitbook.io/yq/v/v4.x/commands/evaluate#flags)
 - [Complex expressions to select and update](https://mikefarah.gitbook.io/yq/operators/select#select-and-update-matching-values-in-map)
 - Keeps yaml formatting and comments when updating (though there are issues with whitespace)
 - [Decode/Encode base64 data](https://mikefarah.gitbook.io/yq/operators/encode-decode)
@@ -329,41 +362,62 @@ Examples:
 # yq defaults to 'eval' command if no command is specified. See "yq eval --help" for more examples.
 yq '.stuff' < myfile.yml # outputs the data at the "stuff" node from "myfile.yml"
 
-yq -i '.stuff = "foo"' myfile.yml # update myfile.yml inplace
+yq -i '.stuff = "foo"' myfile.yml # update myfile.yml in place
 
 
 Available Commands:
-  completion       Generate the autocompletion script for the specified shell
-  eval             (default) Apply the expression to each document in each yaml file in sequence
-  eval-all         Loads _all_ yaml documents of _all_ yaml files and runs expression once
-  help             Help about any command
-  shell-completion Generate completion script
+  completion  Generate the autocompletion script for the specified shell
+  eval        (default) Apply the expression to each document in each yaml file in sequence
+  eval-all    Loads _all_ yaml documents of _all_ yaml files and runs expression once
+  help        Help about any command
 
 Flags:
   -C, --colors                        force print with colors
+      --csv-auto-parse                parse CSV YAML/JSON values (default true)
+      --csv-separator char            CSV Separator character (default ,)
   -e, --exit-status                   set exit status if there are no matches or null or false is returned
+      --expression string             forcibly set the expression argument. Useful when yq argument detection thinks your expression is a file.
+      --from-file string              Load expression from specified file.
   -f, --front-matter string           (extract|process) first input as yaml front-matter. Extract will pull out the yaml content, process will run the expression against the yaml content, leaving the remaining data intact
       --header-preprocess             Slurp any header comments and separators before processing expression. (default true)
   -h, --help                          help for yq
   -I, --indent int                    sets indent level for output (default 2)
-  -i, --inplace                       update the file inplace of first file given.
-  -p, --input-format string           [yaml|y|xml|x] parse format for input. Note that json is a subset of yaml. (default "yaml")
+  -i, --inplace                       update the file in place of first file given.
+  -p, --input-format string           [auto|a|yaml|y|json|j|props|p|csv|c|tsv|t|xml|x|base64|uri|toml|lua|l|ini|i] parse format for input. (default "auto")
+      --lua-globals                   output keys as top-level global variables
+      --lua-prefix string             prefix (default "return ")
+      --lua-suffix string             suffix (default ";\n")
+      --lua-unquoted                  output unquoted string keys (e.g. {foo="bar"})
   -M, --no-colors                     force print with no colors
   -N, --no-doc                        Don't print document separators (---)
+  -0, --nul-output                    Use NUL char to separate values. If unwrap scalar is also set, fail if unwrapped scalar contains NUL char.
   -n, --null-input                    Don't read input, simply evaluate the expression given. Useful for creating docs from scratch.
-  -o, --output-format string          [yaml|y|json|j|props|p|xml|x] output format type. (default "yaml")
+  -o, --output-format string          [auto|a|yaml|y|json|j|props|p|csv|c|tsv|t|xml|x|base64|uri|toml|shell|s|lua|l|ini|i] output format type. (default "auto")
   -P, --prettyPrint                   pretty print, shorthand for '... style = ""'
-  -s, --split-exp string              print each result (or doc) into a file named (exp). [exp] argument must return a string. You can use $index in the expression as the result counter.
-      --unwrapScalar                  unwrap scalar, print the value with no quotes, colors or comments (default true)
+      --properties-array-brackets     use [x] in array paths (e.g. for SpringBoot)
+      --properties-separator string   separator to use between keys and values (default " = ")
+  -s, --split-exp string              print each result (or doc) into a file named (exp). [exp] argument must return a string. You can use $index in the expression as the result counter. The necessary directories will be created.
+      --split-exp-file string         Use a file to specify the split-exp expression.
+      --string-interpolation          Toggles strings interpolation of \(exp) (default true)
+      --tsv-auto-parse                parse TSV YAML/JSON values (default true)
+  -r, --unwrapScalar                  unwrap scalar, print the value with no quotes, colors or comments. Defaults to true for yaml (default true)
   -v, --verbose                       verbose mode
   -V, --version                       Print version information and quit
-      --xml-attribute-prefix string   prefix for xml attributes (default "+")
+      --xml-attribute-prefix string   prefix for xml attributes (default "+@")
       --xml-content-name string       name for xml content (if no attribute name is present). (default "+content")
+      --xml-directive-name string     name for xml directives (e.g. <!DOCTYPE thing cat>) (default "+directive")
+      --xml-keep-namespace            enables keeping namespace after parsing attributes (default true)
+      --xml-proc-inst-prefix string   prefix for xml processing instructions (e.g. <?xml version="1"?>) (default "+p_")
+      --xml-raw-token                 enables using RawToken method instead Token. Commonly disables namespace translations. See https://pkg.go.dev/encoding/xml#Decoder.RawToken for details. (default true)
+      --xml-skip-directives           skip over directives (e.g. <!DOCTYPE thing cat>)
+      --xml-skip-proc-inst            skip over process instructions (e.g. <?xml version="1"?>)
+      --xml-strict-mode               enables strict parsing of XML. See https://pkg.go.dev/encoding/xml for more details.
 
 Use "yq [command] --help" for more information about a command.
 ```
 ## Known Issues / Missing Features
 - `yq` attempts to preserve comment positions and whitespace as much as possible, but it does not handle all scenarios (see https://github.com/go-yaml/yaml/tree/v3 for details)
 - Powershell has its own...[opinions on quoting yq](https://mikefarah.gitbook.io/yq/usage/tips-and-tricks#quotes-in-windows-powershell)
+- "yes", "no" were dropped as boolean values in the yaml 1.2 standard - which is the standard yq assumes.
 
 See [tips and tricks](https://mikefarah.gitbook.io/yq/usage/tips-and-tricks) for more common problems and solutions.

acceptance_tests/bad_args.sh

@@ -3,25 +3,25 @@
 testWriteInPlacePipeIn() {
   result=$(./yq e -i -n '.a' 2>&1)
   assertEquals 1 $?
-  assertEquals "Error: write inplace flag only applicable when giving an expression and at least one file" "$result"
+  assertEquals "Error: write in place flag only applicable when giving an expression and at least one file" "$result"
 }
 
 testWriteInPlacePipeInEvalall() {
   result=$(./yq ea -i -n '.a' 2>&1)
   assertEquals 1 $?
-  assertEquals "Error: write inplace flag only applicable when giving an expression and at least one file" "$result"
+  assertEquals "Error: write in place flag only applicable when giving an expression and at least one file" "$result"
 }
 
 testWriteInPlaceWithSplit() {
   result=$(./yq e -s "cat" -i '.a = "thing"' test.yml 2>&1)
   assertEquals 1 $?
-  assertEquals "Error: write inplace cannot be used with split file" "$result"
+  assertEquals "Error: write in place cannot be used with split file" "$result"
 }
 
 testWriteInPlaceWithSplitEvalAll() {
   result=$(./yq ea -s "cat" -i '.a = "thing"' test.yml 2>&1)
   assertEquals 1 $?
-  assertEquals "Error: write inplace cannot be used with split file" "$result"
+  assertEquals "Error: write in place cannot be used with split file" "$result"
 }
 
 testNullWithFiles() {

acceptance_tests/basic.sh

@@ -96,6 +96,17 @@ testBasicExpressionFromFile() {
   assertEquals '{"xyz":"meow","cool":"frog"}' "$X"
 }
 
+testBasicExpressionFromFileDos() {
+  ./yq -n ".xyz = 123" > test.yml
+  echo '.xyz = "meow" | .cool = "frog"' | sed 's/$'"/`echo \\\r`/" > instructions.txt
+  
+  X=$(./yq --from-file instructions.txt test.yml -o=j -I=0)
+  assertEquals '{"xyz":"meow","cool":"frog"}' "$X"
+
+  X=$(./yq ea --from-file instructions.txt test.yml -o=j -I=0)
+  assertEquals '{"xyz":"meow","cool":"frog"}' "$X"
+}
+
 testBasicGitHubAction() {
   ./yq -n ".a = 123" > test.yml
   X=$(cat /dev/null | ./yq test.yml)
@@ -143,7 +154,7 @@ testBasicCatWithFilesNoDash() {
 }
 
 # when the nullinput flag is used
-# dont automatically read STDIN (this breaks github actions)
+# don't automatically read STDIN (this breaks github actions)
 testBasicCreateFileGithubAction() {
   cat /dev/null | ./yq -n ".a = 123" > test.yml
 }
@@ -302,7 +313,7 @@ testBasicExitStatusNoEval() {
   assertEquals 1 "$?"
 }
 
-testBasicExtractFieldWithSeperator() {
+testBasicExtractFieldWithSeparator() {
     cat >test.yml <<EOL
 ---
 name: chart-name
@@ -312,7 +323,7 @@ EOL
   assertEquals "chart-name" "$X"
 }
 
-testBasicExtractMultipleFieldWithSeperator() {
+testBasicExtractMultipleFieldWithSeparator() {
     cat >test.yml <<EOL
 ---
 name: chart-name
@@ -351,6 +362,12 @@ EOM
   assertEquals "$expected" "$X"
 }
 
-
+testBasicClosedStdIn() {
+  cat >test.yml <<EOL
+a: 1
+EOL
+  X=$(./yq e '.a' test.yml <&-)
+  assertEquals "1" "$X"
+}
 
 source ./scripts/shunit2

acceptance_tests/empty.sh

@@ -9,7 +9,7 @@ EOL
 
 testEmptyEval() {
   X=$(./yq e test.yml)
-  expected=$(cat test.yml)
+  expected="# comment"
   assertEquals 0 $?
   assertEquals "$expected" "$X"
 }

acceptance_tests/flags.sh

@@ -0,0 +1,21 @@
+#!/bin/bash
+
+setUp() {
+  rm test*.yml || true
+  cat >test.yml <<EOL
+# comment
+EOL
+}
+
+testStringInterpolation() {
+    X=$(./yq -n '"Mike \(3 + 4)"')
+    assertEquals "Mike 7" "$X"
+}
+
+testNoStringInterpolation() {
+    X=$(./yq --string-interpolation=f -n '"Mike \(3 + 4)"')
+    assertEquals "Mike \(3 + 4)" "$X"
+}
+
+
+source ./scripts/shunit2

acceptance_tests/header-processing-off.sh

@@ -0,0 +1,43 @@
+#!/bin/bash
+
+setUp() {
+  rm test*.yml || true
+  
+}
+
+testLineCountFirstLineComment() {
+  cat >test.yml <<EOL
+#test123 
+abc: 123 
+test123: 123123 
+#comment 
+lalilu: lalilu
+EOL
+
+  X=$(./yq '.lalilu | line' --header-preprocess=false < test.yml)
+  assertEquals "5" "$X"
+}
+
+testArrayOfDocs() {
+  cat >test.yml <<EOL
+---
+# leading comment doc 1
+a: 1
+---
+# leading comment doc 2
+a: 2
+EOL
+
+read -r -d '' expected << EOM
+- # leading comment doc 1
+  a: 1
+- # leading comment doc 2
+  a: 2
+EOM
+
+  X=$(./yq ea '[.]' --header-preprocess=false < test.yml)
+  assertEquals "$expected" "$X"
+
+}
+
+source ./scripts/shunit2

acceptance_tests/inputs-format-auto.sh

@@ -0,0 +1,249 @@
+#!/bin/bash
+
+setUp() {
+  rm test*.yml 2>/dev/null || true
+  rm test*.toml 2>/dev/null || true
+  rm test*.tfstate 2>/dev/null || true
+  rm test*.json 2>/dev/null || true
+  rm test*.properties 2>/dev/null || true
+  rm test*.csv 2>/dev/null || true
+  rm test*.tsv 2>/dev/null || true
+  rm test*.xml 2>/dev/null || true
+}
+
+testInputJson() {
+  cat >test.json <<EOL
+{ "mike" : { "things": "cool" } }
+EOL
+
+  read -r -d '' expected << EOM
+{
+  "mike": {
+    "things": "cool"
+  }
+}
+EOM
+
+  X=$(./yq test.json)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea test.json)
+  assertEquals "$expected" "$X"
+}
+
+testInputToml() {
+  cat >test.toml <<EOL
+[owner]
+name = "Tom Preston-Werner"
+dob = 1979-05-27T07:32:00-08:00
+EOL
+
+  read -r -d '' expected << EOM
+owner:
+  name: Tom Preston-Werner
+  dob: 1979-05-27T07:32:00-08:00
+EOM
+
+  X=$(./yq -oy test.toml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea -oy test.toml)
+  assertEquals "$expected" "$X"
+}
+
+testInputTfstate() {
+  cat >test.tfstate <<EOL
+{ "mike" : { "things": "cool" } }
+EOL
+
+  read -r -d '' expected << EOM
+{"mike": {"things": "cool"}}
+EOM
+
+  X=$(./yq test.tfstate)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea test.tfstate)
+  assertEquals "$expected" "$X"
+}
+
+testInputJsonOutputYaml() {
+  cat >test.json <<EOL
+{ "mike" : { "things": "cool" } }
+EOL
+
+  read -r -d '' expected << EOM
+mike:
+  things: cool
+EOM
+
+  X=$(./yq test.json -oy)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea test.json -oy)
+  assertEquals "$expected" "$X"
+}
+
+testInputProperties() {
+  cat >test.properties <<EOL
+mike.things = hello
+EOL
+
+  read -r -d '' expected << EOM
+mike.things = hello
+EOM
+
+  X=$(./yq e test.properties)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq test.properties)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea test.properties)
+  assertEquals "$expected" "$X"
+}
+
+testInputPropertiesGitHubAction() {
+  cat >test.properties <<EOL
+mike.things = hello
+EOL
+
+  read -r -d '' expected << EOM
+mike.things = hello
+EOM
+
+  X=$(cat /dev/null | ./yq e test.properties)
+  assertEquals "$expected" "$X"
+
+  X=$(cat /dev/null | ./yq ea test.properties)
+  assertEquals "$expected" "$X"
+}
+
+testInputCSV() {
+  cat >test.csv <<EOL
+fruit,yumLevel
+apple,5
+banana,4
+EOL
+
+  read -r -d '' expected << EOM
+fruit,yumLevel
+apple,5
+banana,4
+EOM
+
+  X=$(./yq e test.csv)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea test.csv)
+  assertEquals "$expected" "$X"
+}
+
+testInputCSVUTF8() {
+  read -r -d '' expected << EOM
+id,first,last
+1,john,smith
+1,jane,smith
+EOM
+
+  X=$(./yq utf8.csv)
+  assertEquals "$expected" "$X"
+}
+
+testInputTSV() {
+  cat >test.tsv <<EOL
+fruit	yumLevel
+apple	5
+banana	4
+EOL
+
+  read -r -d '' expected << EOM
+fruit	yumLevel
+apple	5
+banana	4
+EOM
+
+  X=$(./yq e test.tsv)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea test.tsv)
+  assertEquals "$expected" "$X"
+}
+
+
+
+
+testInputXml() {
+  cat >test.xml <<EOL
+<cat legs="4">BiBi</cat>
+EOL
+
+  read -r -d '' expected << EOM
+<cat legs="4">BiBi</cat>
+EOM
+
+  X=$(./yq e test.xml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea test.xml)
+  assertEquals "$expected" "$X"
+}
+
+testInputXmlNamespaces() {
+  cat >test.xml <<EOL
+<?xml version="1.0"?>
+<map xmlns="some-namespace" xmlns:xsi="some-instance" xsi:schemaLocation="some-url">
+</map>
+EOL
+
+  read -r -d '' expected << EOM
+<?xml version="1.0"?>
+<map xmlns="some-namespace" xmlns:xsi="some-instance" xsi:schemaLocation="some-url"></map>
+EOM
+
+  X=$(./yq e test.xml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea test.xml)
+  assertEquals "$expected" "$X"
+}
+
+
+
+testInputXmlStrict() {
+  cat >test.xml <<EOL
+<?xml version="1.0"?>
+<!DOCTYPE root [
+<!ENTITY writer "Catherine.">
+<!ENTITY copyright "(r) Great">
+]>
+<root>
+    <item>&writer;&copyright;</item>
+</root>
+EOL
+
+  X=$(./yq --xml-strict-mode test.xml  2>&1)
+  assertEquals 1 $?
+  assertEquals "Error: bad file 'test.xml': XML syntax error on line 7: invalid character entity &writer;" "$X"
+
+  X=$(./yq ea --xml-strict-mode test.xml  2>&1)
+  assertEquals "Error: bad file 'test.xml': XML syntax error on line 7: invalid character entity &writer;" "$X"
+}
+
+testInputXmlGithubAction() {
+  cat >test.xml <<EOL
+<cat legs="4">BiBi</cat>
+EOL
+
+  read -r -d '' expected << EOM
+<cat legs="4">BiBi</cat>
+EOM
+
+  X=$(cat /dev/null | ./yq e test.xml)
+  assertEquals "$expected" "$X"
+
+  X=$(cat /dev/null | ./yq ea test.xml)
+  assertEquals "$expected" "$X"
+}
+
+source ./scripts/shunit2

acceptance_tests/inputs-format.sh

@@ -63,6 +63,75 @@ EOM
   assertEquals "$expected" "$X"
 }
 
+testInputCSVCustomSeparator() {
+  cat >test.csv <<EOL
+fruit;yumLevel
+apple;5
+banana;4
+EOL
+
+  read -r -d '' expected << EOM
+- fruit: apple
+  yumLevel: 5
+- fruit: banana
+  yumLevel: 4
+EOM
+
+  X=$(./yq -p=csv --csv-separator ";" test.csv)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea -p=csv --csv-separator ";" test.csv)
+  assertEquals "$expected" "$X"
+}
+
+testInputCSVNoAuto() {
+  cat >test.csv <<EOL
+thing1
+name: cat
+EOL
+
+  read -r -d '' expected << EOM
+- thing1: 'name: cat'
+EOM
+
+  X=$(./yq --csv-auto-parse=f test.csv -oy)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea --csv-auto-parse=f test.csv -oy)
+  assertEquals "$expected" "$X"
+}
+
+testInputTSVNoAuto() {
+  cat >test.tsv <<EOL
+thing1
+name: cat
+EOL
+
+  read -r -d '' expected << EOM
+- thing1: 'name: cat'
+EOM
+
+  X=$(./yq --tsv-auto-parse=f test.tsv -oy)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea --tsv-auto-parse=f test.tsv -oy)
+  assertEquals "$expected" "$X"
+}
+
+testInputCSVUTF8() {
+  read -r -d '' expected << EOM
+- id: 1
+  first: john
+  last: smith
+- id: 1
+  first: jane
+  last: smith
+EOM
+
+  X=$(./yq -p=csv utf8.csv)
+  assertEquals "$expected" "$X"
+}
+
 testInputTSV() {
   cat >test.tsv <<EOL
 fruit	yumLevel
@@ -95,7 +164,7 @@ EOL
   read -r -d '' expected << EOM
 cat:
   +content: BiBi
-  +legs: "4"
+  +@legs: "4"
 EOM
 
   X=$(./yq e -p=xml test.yml)
@@ -106,23 +175,44 @@ EOM
 }
 
 testInputXmlNamespaces() {
-  cat >test.yml <<EOL
+  cat >test.xml <<EOL
 <?xml version="1.0"?>
 <map xmlns="some-namespace" xmlns:xsi="some-instance" xsi:schemaLocation="some-url">
 </map>
 EOL
 
   read -r -d '' expected << EOM
++p_xml: version="1.0"
 map:
-  +xmlns: some-namespace
-  +xmlns:xsi: some-instance
-  +xsi:schemaLocation: some-url
+  +@xmlns: some-namespace
+  +@xmlns:xsi: some-instance
+  +@xsi:schemaLocation: some-url
 EOM
 
-  X=$(./yq e -p=xml test.yml)
+  X=$(./yq e -p=xml test.xml)
   assertEquals "$expected" "$X"
 
-  X=$(./yq ea -p=xml test.yml)
+  X=$(./yq ea -p=xml test.xml)
+  assertEquals "$expected" "$X"
+}
+
+testInputXmlRoundtrip() {
+  cat >test.yml <<EOL
+<?xml version="1.0"?>
+<!DOCTYPE config SYSTEM "/etc/iwatch/iwatch.dtd" >
+<map xmlns="some-namespace" xmlns:xsi="some-instance" xsi:schemaLocation="some-url">Meow</map>
+EOL
+
+  read -r -d '' expected << EOM
+<?xml version="1.0"?>
+<!DOCTYPE config SYSTEM "/etc/iwatch/iwatch.dtd" >
+<map xmlns="some-namespace" xmlns:xsi="some-instance" xsi:schemaLocation="some-url">Meow</map>
+EOM
+
+  X=$(./yq -p=xml -o=xml test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea -p=xml -o=xml test.yml)
   assertEquals "$expected" "$X"
 }
 
@@ -139,11 +229,11 @@ testInputXmlStrict() {
 </root>
 EOL
 
-  X=$(./yq -p=xml --xml-strict-mode test.yml 2>&1)
+  X=$(./yq -p=xml --xml-strict-mode test.yml -o=xml 2>&1)
   assertEquals 1 $?
   assertEquals "Error: bad file 'test.yml': XML syntax error on line 7: invalid character entity &writer;" "$X"
 
-  X=$(./yq ea -p=xml --xml-strict-mode test.yml 2>&1)
+  X=$(./yq ea -p=xml --xml-strict-mode test.yml -o=xml 2>&1)
   assertEquals "Error: bad file 'test.yml': XML syntax error on line 7: invalid character entity &writer;" "$X"
 }
 
@@ -155,7 +245,7 @@ EOL
   read -r -d '' expected << EOM
 cat:
   +content: BiBi
-  +legs: "4"
+  +@legs: "4"
 EOM
 
   X=$(cat /dev/null | ./yq e -p=xml test.yml)

acceptance_tests/leading-separator.sh

@@ -1,5 +1,8 @@
 #!/bin/bash
 
+
+# examples where header-preprocess is required
+
 setUp() {
   rm test*.yml || true
   cat >test.yml <<EOL
@@ -8,7 +11,7 @@ a: test
 EOL
 }
 
-testLeadingSeperatorWithDoc() {
+testLeadingSeparatorWithDoc() {
   cat >test.yml <<EOL
 # hi peeps
 # cool
@@ -31,18 +34,94 @@ EOM
   assertEquals "$expected" "$X"
 }
 
-testLeadingSeperatorPipeIntoEvalSeq() {
+
+testLeadingSeparatorWithNewlinesNewDoc() {
+  cat >test.yml <<EOL
+# hi peeps
+# cool
+
+
+---
+a: test
+---
+b: cool
+EOL
+
+  read -r -d '' expected << EOM
+# hi peeps
+# cool
+
+
+---
+a: thing
+---
+b: cool
+EOM
+
+  X=$(./yq e '(select(di == 0) | .a) = "thing"' - < test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testLeadingSeparatorWithNewlinesMoreComments() {
+  cat >test.yml <<EOL
+# hi peeps
+# cool
+
+---
+# great
+
+a: test
+---
+b: cool
+EOL
+
+  read -r -d '' expected << EOM
+# hi peeps
+# cool
+
+---
+# great
+
+a: thing
+---
+b: cool
+EOM
+
+  X=$(./yq e '(select(di == 0) | .a) = "thing"' - < test.yml)
+  assertEquals "$expected" "$X"
+}
+
+
+testLeadingSeparatorWithDirective() {
+  cat >test.yml <<EOL
+%YAML 1.1
+---
+this: should really work
+EOL
+
+  read -r -d '' expected << EOM
+%YAML 1.1
+---
+this: should really work
+EOM
+
+  X=$(./yq < test.yml)
+  assertEquals "$expected" "$X"
+}
+
+
+testLeadingSeparatorPipeIntoEvalSeq() {
   X=$(./yq e - < test.yml)
   expected=$(cat test.yml)
   assertEquals "$expected" "$X"
 }
 
-testLeadingSeperatorExtractField() {
+testLeadingSeparatorExtractField() {
   X=$(./yq e '.a' - < test.yml)
   assertEquals "test" "$X"
 }
 
-testLeadingSeperatorExtractFieldWithCommentsAfterSep() {
+testLeadingSeparatorExtractFieldWithCommentsAfterSep() {
   cat >test.yml <<EOL
 ---
 # hi peeps
@@ -53,7 +132,7 @@ EOL
   assertEquals "test" "$X"
 }
 
-testLeadingSeperatorExtractFieldWithCommentsBeforeSep() {
+testLeadingSeparatorExtractFieldWithCommentsBeforeSep() {
   cat >test.yml <<EOL
 # hi peeps
 # cool
@@ -65,7 +144,7 @@ EOL
 }
 
 
-testLeadingSeperatorExtractFieldMultiDoc() {
+testLeadingSeparatorExtractFieldMultiDoc() {
   cat >test.yml <<EOL
 ---
 a: test
@@ -82,13 +161,13 @@ EOM
   assertEquals "$expected" "$X"
 }
 
-testLeadingSeperatorExtractFieldMultiDocWithComments() {
+testLeadingSeparatorExtractFieldMultiDocWithComments() {
   cat >test.yml <<EOL
 # here
 ---
 # there
 a: test
-# whereever
+# wherever
 ---
 # you are
 a: test2
@@ -105,26 +184,26 @@ EOM
 }
 
 
-testLeadingSeperatorEvalSeq() {
+testLeadingSeparatorEvalSeq() {
   X=$(./yq e test.yml)
   expected=$(cat test.yml)
   assertEquals "$expected" "$X"
 }
 
-testLeadingSeperatorPipeIntoEvalAll() {
+testLeadingSeparatorPipeIntoEvalAll() {
   X=$(./yq ea - < test.yml)
   expected=$(cat test.yml)
   assertEquals "$expected" "$X"
 }
 
 
-testLeadingSeperatorEvalAll() {
+testLeadingSeparatorEvalAll() {
   X=$(./yq ea test.yml)
   expected=$(cat test.yml)
   assertEquals "$expected" "$X"
 }
 
-testLeadingSeperatorMultiDocEvalSimple() {
+testLeadingSeparatorMultiDocEvalSimple() {
   read -r -d '' expected << EOM
 ---
 a: test
@@ -138,7 +217,7 @@ EOM
   assertEquals "$expected" "$X"
 }
 
-testLeadingSeperatorMultiDocInOneFile() {
+testLeadingSeparatorMultiDocInOneFile() {
   cat >test.yml <<EOL
 ---
 # hi peeps
@@ -152,7 +231,7 @@ EOL
   assertEquals "$expected" "$X"
 }
 
-testLeadingSeperatorMultiDocInOneFileEvalAll() {
+testLeadingSeparatorMultiDocInOneFileEvalAll() {
   cat >test.yml <<EOL
 ---
 # hi peeps
@@ -166,7 +245,7 @@ EOL
   assertEquals "$expected" "$X"
 }
 
-testLeadingSeperatorMultiDocEvalComments() {
+testLeadingSeparatorMultiDocEvalComments() {
   cat >test.yml <<EOL
 # hi peeps
 # cool
@@ -194,7 +273,7 @@ EOM
   assertEquals "$expected" "$X"
 }
 
-testLeadingSeperatorMultiDocEvalCommentsTrailingSep() {
+testLeadingSeparatorMultiDocEvalCommentsTrailingSep() {
   cat >test.yml <<EOL
 # hi peeps
 # cool
@@ -226,7 +305,7 @@ EOM
   assertEquals "$expected" "$X"
 }
 
-testLeadingSeperatorMultiMultiDocEvalCommentsTrailingSep() {
+testLeadingSeparatorMultiMultiDocEvalCommentsTrailingSep() {
   cat >test.yml <<EOL
 # hi peeps
 # cool
@@ -266,7 +345,7 @@ EOM
   assertEquals "$expected" "$X"
 }
 
-testLeadingSeperatorMultiDocEvalCommentsLeadingSep() {
+testLeadingSeparatorMultiDocEvalCommentsLeadingSep() {
   cat >test.yml <<EOL
 ---
 # hi peeps
@@ -326,7 +405,7 @@ EOM
   assertEquals "$expected" "$X"
 }
 
-testLeadingSeperatorMultiDocEvalCommentsStripComments() {
+testLeadingSeparatorMultiDocEvalCommentsStripComments() {
   cat >test.yml <<EOL
 ---
 # hi peeps
@@ -349,7 +428,7 @@ EOM
   assertEquals "$expected" "$X"
 }
 
-testLeadingSeperatorMultiDocEvalCommentsLeadingSepNoDocFlag() {
+testLeadingSeparatorMultiDocEvalCommentsLeadingSepNoDocFlag() {
   cat >test.yml <<EOL
 ---
 # hi peeps
@@ -375,7 +454,7 @@ EOM
   assertEquals "$expected" "$X"
 }
 
-testLeadingSeperatorMultiDocEvalJsonFlag() {
+testLeadingSeparatorMultiDocEvalJsonFlag() {
   cat >test.yml <<EOL
 ---
 # hi peeps
@@ -404,7 +483,7 @@ EOM
   assertEquals "$expected" "$X"
 }
 
-testLeadingSeperatorMultiDocEvalAllJsonFlag() {
+testLeadingSeparatorMultiDocEvalAllJsonFlag() {
   cat >test.yml <<EOL
 ---
 # hi peeps
@@ -433,7 +512,7 @@ EOM
   assertEquals "$expected" "$X"
 }
 
-testLeadingSeperatorMultiDocEvalAll() {
+testLeadingSeparatorMultiDocEvalAll() {
   read -r -d '' expected << EOM
 ---
 a: test

acceptance_tests/load-file.sh

@@ -3,25 +3,25 @@
 testLoadFileNotExist() {
   result=$(./yq e -n 'load("cat.yml")' 2>&1)
   assertEquals 1 $?
-  assertEquals "Error: Failed to load cat.yml: open cat.yml: no such file or directory" "$result"
+  assertEquals "Error: failed to load cat.yml: open cat.yml: no such file or directory" "$result"
 }
 
 testLoadFileExpNotExist() {
   result=$(./yq e -n 'load(.a)' 2>&1)
   assertEquals 1 $?
-  assertEquals "Error: Filename expression returned nil" "$result"
+  assertEquals "Error: filename expression returned nil" "$result"
 }
 
 testStrLoadFileNotExist() {
   result=$(./yq e -n 'strload("cat.yml")' 2>&1)
   assertEquals 1 $?
-  assertEquals "Error: Failed to load cat.yml: open cat.yml: no such file or directory" "$result"
+  assertEquals "Error: failed to load cat.yml: open cat.yml: no such file or directory" "$result"
 }
 
 testStrLoadFileExpNotExist() {
   result=$(./yq e -n 'strload(.a)' 2>&1)
   assertEquals 1 $?
-  assertEquals "Error: Filename expression returned nil" "$result"
+  assertEquals "Error: filename expression returned nil" "$result"
 }
 
 source ./scripts/shunit2

acceptance_tests/nul-separator.sh

@@ -0,0 +1,286 @@
+#!/bin/bash
+
+setUp() {
+    rm test*.yml || true
+}
+
+## Convenient bash shortcut to read records of NUL separated values
+## from stdin the safe way. See example usage in the next tests.
+read-0() {
+    local eof="" IFS=''
+    while [ "$1" ]; do
+        ## - The `-r` avoids bad surprise with '\n' and other interpreted
+        ##   sequences that can be read.
+        ## - The `-d ''` is the (strange?) way to refer to NUL delimiter.
+        ## - The `--` is how to avoid unpleasant surprises if your
+        ##   "$1" starts with "-" (minus) sign. This protection also
+        ##   will produce a readable error if you want to try to start
+        ##   your variable names with a "-".
+        read -r -d '' -- "$1" || eof=1
+        shift
+    done
+    [ -z "$eof" ] ## fail on EOF
+}
+
+## Convenient bash shortcut to be used with the next function `p-err`
+## to read NUL separated values the safe way AND catch any errors from
+## the process creating the stream of NUL separated data.  See example
+## usage in the tests.
+read-0-err() {
+    local ret="$1" eof="" idx=0 last=
+    read -r -- "${ret?}" <<<"0"
+    shift
+    while [ "$1" ]; do
+        last=$idx
+        read -r -d '' -- "$1" || {
+            ## Put this last value in ${!ret}
+            eof="$1"
+            read -r -- "$ret" <<<"${!eof}"
+            break
+        }
+        ((idx++))
+        shift
+    done
+    [ -z "$eof" ] || {
+        if [ "$last" != 0 ]; then
+            ## Uhoh, we have no idea if the errorlevel of the internal
+            ## command was properly delimited with a NUL char, and
+            ## anyway something went really wrong at least about the
+            ## number of fields separated by NUL char and the one
+            ## expected.
+            echo "Error: read-0-err couldn't fill all value $ret = '${!ret}', '$eof', '${!eof}'" >&2
+            read -r -- "$ret" <<<"not-enough-values"
+        else
+            if ! [[ "${!ret}" =~ ^[0-9]+$ && "${!ret}" -ge 0 && "${!ret}" -le 127 ]]; then
+                ## This could happen if you don't use `p-err` wrapper,
+                ## or used stdout in unexpected ways in your inner
+                ## command.
+                echo "Error: last value is not a number, did you finish with an errorlevel ?" >&2
+                read -r -- "$ret" <<<"last-value-not-a-number"
+            fi
+        fi
+        false
+    }
+}
+
+## Simply runs command given as argument and adds errorlevel in the
+## standard output. Is expected to be used in tandem with
+## `read-0-err`.
+p-err() {
+    local exp="$1"
+    "$@"
+    printf "%s" "$?"
+}
+
+wyq-r() {
+    local exp="$1"
+    ./yq e -0 -r=false "$1"
+    printf "%s" "$?"
+}
+
+testBasicUsageRaw() {
+  cat >test.yml <<EOL
+a: foo
+b: bar
+EOL
+
+  printf "foo\0bar\0" > expected.out
+
+  ## We need to compare binary content here. We have to filter the compared
+  ## content through a representation that gets rid of NUL chars but accurately
+  ## transcribe the content.
+  ## Also as it would be nice to have a pretty output in case the test fails,
+  ## we use here 'hd': a widely available shortcut to 'hexdump' that will
+  ## pretty-print any binary to it's hexadecimal representation.
+  ##
+  ## Note that the standard `assertEquals` compare its arguments
+  ## value, but they can't hold NUL characters (this comes from the
+  ## limitation of the C API of `exec*(..)` functions that requires
+  ## `const char *arv[]`). And these are NUL terminated strings.  As a
+  ## consequence, the NUL characters gets removed in bash arguments.
+  assertEquals "$(hd expected.out)" \
+               "$(./yq e -0 '.a, .b' test.yml | hd)"
+
+  rm expected.out
+}
+
+testBasicUsage() {
+  local a b
+  cat >test.yml <<EOL
+a: foo
+b: bar
+EOL
+
+  ## We provide 2 values, and ask to fill 2 variables.
+  read-0 a b < <(./yq e -0 '.a, .b' test.yml)
+  assertEquals "$?" "0"      ## Everything is fine
+  assertEquals "foo" "$a"    ## Values are correctly parsed
+  assertEquals "bar" "$b"
+
+  a=YYY ; b=XXX
+  ## Not enough values provided to fill `a` and `b`.
+  read-0 a b < <(./yq e -0 '.a' test.yml)
+  assertEquals "$?" "1"      ## An error was emitted
+  assertEquals "foo" "$a"    ## First value was correctly parsed
+  assertEquals "" "$b"       ## Second was still reset
+
+  ## Error from inner command are not catchable !. Use
+  ## `read-0-err`/`p-err` for that.
+  read-0 a < <(printf "\0"; ./yq e -0 'xxx' test.yml; )
+  assertEquals "$?" "0"
+
+}
+
+testBasicUsageJson() {
+  cat >test.yml <<EOL
+a:
+  x: foo
+b: bar
+EOL
+
+  read-0 a b < <(./yq e -0 -o=json '.a, .b' test.yml)
+
+  assertEquals '{
+  "x": "foo"
+}' "$a"
+  assertEquals '"bar"' "$b"
+
+}
+
+testFailWithValueContainingNUL() {
+  local a b c
+  ## Note that value of field 'a' actually contains a NUL char !
+  cat >test.yml <<EOL
+a: "foo\u0000bar"
+b: 1
+c: |
+  wiz
+  boom
+EOL
+
+  ## We are looking for trouble with asking to separated fields with NUL
+  ## char and requested value `.a` actually contains itself a NUL char !
+  read-0 a b c < <(./yq e -0 '.a, .b, .c' test.yml)
+  assertNotEquals "0" "$?"   ## read-0 failed to fill all values
+
+  ## But here, we can request for one value, even if `./yq` fails
+  read-0 b < <(./yq e -0 '.b, .a' test.yml)
+  assertEquals "0" "$?"   ## read-0 succeeds at feeding the first value
+  ## Note: to catch the failure of `yq`, see in the next tests the usage
+  ## of `read-0-err`.
+
+  ## using -r=false solves any NUL containing value issues, but keeps
+  ## all in YAML representation:
+  read-0 a b c < <(./yq e -0 -r=false '.a, .b, .c' test.yml)
+  assertEquals "0" "$?"    ## All goes well despite asking for `a` value
+
+  assertEquals '"foo\0bar"' "$a"   ## This is a YAML string representation
+  assertEquals '1' "$b"
+  assertEquals '|
+  wiz
+  boom' "$c"
+}
+
+testStandardLoop() {
+    local E a b res
+
+    ## Here everything is normal: 4 values, that will be paired
+    ## in key/values.
+    cat >test.yml <<EOL
+- yay
+- wiz
+- hop
+- pow
+EOL
+
+    res=""
+    while read-0-err E a b; do
+        res+="$a: $b;"
+    done < <(p-err ./yq -0 '.[]' test.yml)
+
+    assertEquals "0" "$E"                     ## errorlevel of internal command
+    assertEquals "yay: wiz;hop: pow;" "$res"  ## expected result
+}
+
+testStandardLoopWithoutEnoughValues() {
+    local E a b res
+
+    ## Here 5 values, there will be a missing value when reading
+    ## pairs of value.
+    cat >test.yml <<EOL
+- yay
+- wiz
+- hop
+- pow
+- kwak
+EOL
+
+    res=""
+    ## The loop will succeed 2 times then fail
+    while read-0-err E a b; do
+        res+="$a: $b;"
+    done < <(p-err ./yq -0 '.[]' test.yml)
+
+    assertEquals "not-enough-values" "$E"     ## Not enough value error
+    assertEquals "yay: wiz;hop: pow;" "$res"  ## the 2 full key/value pairs
+
+}
+
+testStandardLoopWithInternalCmdError() {
+    local E a b res
+
+    ## Note the third value contains a NUL char !
+    cat >test.yml <<EOL
+- yay
+- wiz
+- "foo\0bar"
+- hop
+- pow
+EOL
+
+    res=""
+    ## It should be only upon the second pass in the loop that
+    ## read-0-err will catch the fact that there is an error !
+    while read-0-err E a b; do
+        res+="$a: $b;"
+    done < <(p-err ./yq -0 '.[]' test.yml)
+    assertEquals "1" "$E"            ## Internal command errorlevel (from `./yq`)
+    assertEquals "yay: wiz;" "$res"  ## first 2 values were ok at least
+
+}
+
+testStandardLoopNotEnoughErrorEatsCmdError() {
+    local E a b res
+
+    ## Because of possible edge cases where the internal errorlevel
+    ## reported by `p-err` in the standard output might be mangled
+    ## with the unfinished record, `read-0-err E ...` will NOT report
+    ## the internal command error in the variable E and instead will
+    ## store the value 'not-enough-values'. In real world, anyway, you
+    ## will want to react the same if the internal command failed
+    ## and/or you didn't get as much values as expected while
+    ## reading. Keep in mind also that standard error is not
+    ## swallowed, so you can read reports from the inner command AND
+    ## from `read-0-err`.
+
+    ## Here, note that the fourth value contains a NUL char !
+    cat >test.yml <<EOL
+- yay
+- wiz
+- hop
+- "foo\0bar"
+- pow
+EOL
+
+    res=""
+    ## It should be only upon the second loop that read-0-err will catch
+    ## the fact that there are not enough data to fill the requested variables
+    while read-0-err E a b; do
+        res+="$a: $b;"
+    done < <(p-err ./yq -0 '.[]' test.yml)
+    assertEquals "not-enough-values" "$E"          ## Not enough values error eats internal error !
+    assertEquals "yay: wiz;" "$res"  ## first 2 values were ok at least
+}
+
+
+source ./scripts/shunit2

acceptance_tests/output-format.sh

@@ -48,6 +48,64 @@ EOM
   assertEquals "$expected" "$X"
 }
 
+testOutputYamlRawDefault() {
+  cat >test.yml <<EOL
+a: "cat"
+EOL
+
+  X=$(./yq e  '.a' test.yml)
+  assertEquals "cat" "$X"
+
+  X=$(./yq ea '.a' test.yml)
+  assertEquals "cat" "$X"
+}
+
+testOutputYamlRawOff() {
+  cat >test.yml <<EOL
+a: "cat"
+EOL
+
+  X=$(./yq e -r=false '.a' test.yml)
+  assertEquals "\"cat\"" "$X"
+
+  X=$(./yq ea -r=false '.a' test.yml)
+  assertEquals "\"cat\"" "$X"
+}
+
+testOutputYamlRawOnRoot() {
+  cat >test.yml <<EOL
+'a'
+EOL
+
+  X=$(./yq e -r '.' test.yml)
+  assertEquals "a" "$X"
+}
+
+testOutputJsonRaw() {
+  cat >test.yml <<EOL
+a: cat
+EOL
+
+  X=$(./yq e -r --output-format=json '.a' test.yml)
+  assertEquals "cat" "$X"
+
+  X=$(./yq ea -r --output-format=json '.a' test.yml)
+  assertEquals "cat" "$X"
+}
+
+testOutputJsonDefault() {
+  cat >test.yml <<EOL
+a: cat
+EOL
+
+  X=$(./yq e --output-format=json '.a' test.yml)
+  assertEquals "\"cat\"" "$X"
+
+  X=$(./yq ea --output-format=json '.a' test.yml)
+  assertEquals "\"cat\"" "$X"
+}
+
+
 testOutputJsonShort() {
   cat >test.yml <<EOL
 a: {b: ["cat"]}
@@ -72,11 +130,11 @@ EOM
 
 testOutputProperties() {
   cat >test.yml <<EOL
-a: {b: {c: ["cat"]}}
+a: {b: {c: ["cat cat"]}}
 EOL
 
   read -r -d '' expected << EOM
-a.b.c.0 = cat
+a.b.c.0 = cat cat
 EOM
 
   X=$(./yq e --output-format=props test.yml)
@@ -86,13 +144,30 @@ EOM
   assertEquals "$expected" "$X"
 }
 
-testOutputPropertiesShort() {
+testOutputPropertiesDontUnwrap() {
   cat >test.yml <<EOL
-a: {b: {c: ["cat"]}}
+a: {b: {c: ["cat cat"]}}
 EOL
 
   read -r -d '' expected << EOM
-a.b.c.0 = cat
+a.b.c.0 = "cat cat"
+EOM
+
+  X=$(./yq e -r=false --output-format=props test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea -r=false --output-format=props test.yml)
+  assertEquals "$expected" "$X"
+}
+
+
+testOutputPropertiesShort() {
+  cat >test.yml <<EOL
+a: {b: {c: ["cat cat"]}}
+EOL
+
+  read -r -d '' expected << EOM
+a.b.c.0 = cat cat
 EOM
 
   X=$(./yq e -o=p test.yml)
@@ -123,6 +198,27 @@ EOM
   assertEquals "$expected" "$X"
 }
 
+testOutputCSVCustomSeparator() {
+  cat >test.yml <<EOL
+- fruit: apple
+  yumLevel: 5
+- fruit: banana
+  yumLevel: 4
+EOL
+
+  read -r -d '' expected << EOM
+fruit;yumLevel
+apple;5
+banana;4
+EOM
+
+  X=$(./yq -oc --csv-separator ";" test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea -o=csv --csv-separator ";" test.yml)
+  assertEquals "$expected" "$X"
+}
+
 testOutputTSV() {
   cat >test.yml <<EOL
 - fruit: apple
@@ -186,7 +282,7 @@ EOM
 
 testOutputXmComplex() {
   cat >test.yml <<EOL
-a: {b: {c: ["cat", "dog"], +f: meow}}
+a: {b: {c: ["cat", "dog"], +@f: meow}}
 EOL
 
   read -r -d '' expected << EOM
@@ -205,4 +301,43 @@ EOM
   assertEquals "$expected" "$X"
 }
 
-source ./scripts/shunit2
+testLuaOutputPretty() {
+  cat >test.yml <<EOL
+animals:
+  cat: meow
+EOL
+
+  read -r -d '' expected << EOM
+return {
+	["animals"] = {
+		["cat"] = "meow";
+	};
+};
+EOM
+
+  X=$(./yq e --output-format=lua test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq e --output-format=lua --prettyPrint test.yml)
+  assertEquals "$expected" "$X"
+
+}
+
+testLuaOutputSubset() {
+  cat >test.yml <<EOL
+animals:
+  cat: meow
+EOL
+
+  read -r -d '' expected << EOM
+return {
+	["cat"] = "meow";
+};
+EOM
+
+  X=$(./yq e --output-format=lua '.animals' test.yml)
+  assertEquals "$expected" "$X"
+
+}
+
+source ./scripts/shunit2

acceptance_tests/shebang.sh

@@ -0,0 +1,26 @@
+#!/bin/bash
+
+setUp() {
+  rm test*.yq || true
+  cat >test.yq <<EOL
+#!./yq
+.a.b
+EOL
+chmod +x test.yq
+
+rm test*.yml || true
+  cat >test.yml <<EOL
+a: {b: apple}
+EOL
+}
+
+testCanExecYqFile() {
+  read -r -d '' expected << EOM
+apple
+EOM
+   X=$(./test.yq test.yml)
+  assertEquals "$expected" "$X"
+}
+
+source ./scripts/shunit2
+

acceptance_tests/split-printer.sh

@@ -2,6 +2,7 @@
 
 setUp() {
   rm test*.yml || true
+  rm -rf test_dir* || true
 }
 
 testBasicSplitWithName() {
@@ -204,4 +205,23 @@ EOM
   assertEquals "$expectedDoc3" "$doc3"
 }
 
-source ./scripts/shunit2
+testSplitWithDirectories() {
+  cat >test.yml <<EOL
+f: test_dir1/test_file1
+---
+f: test_dir2/dir22/test_file2
+---
+f: test_file3
+EOL
+
+  ./yq e --no-doc -s ".f" test.yml
+
+  doc1=$(cat test_dir1/test_file1.yml)
+  assertEquals "f: test_dir1/test_file1" "$doc1"
+  doc2=$(cat test_dir2/dir22/test_file2.yml)
+  assertEquals "f: test_dir2/dir22/test_file2" "$doc2"
+  doc3=$(cat test_file3.yml)
+  assertEquals "f: test_file3" "$doc3"
+}
+
+source ./scripts/shunit2

action.yml

@@ -12,6 +12,6 @@ outputs:
     description: "The complete result from the yq command being run"
 runs:
   using: 'docker'
-  image: 'github-action/Dockerfile'
+  image: 'docker://mikefarah/yq:4-githubaction'
   args:
     - ${{ inputs.cmd }}

cmd/completion.go

@@ -7,19 +7,20 @@ import (
 )
 
 var completionCmd = &cobra.Command{
-	Use:   "shell-completion [bash|zsh|fish|powershell]",
-	Short: "Generate completion script",
+	Use:     "completion [bash|zsh|fish|powershell]",
+	Aliases: []string{"shell-completion"},
+	Short:   "Generate the autocompletion script for the specified shell",
 	Long: `To load completions:
 
 Bash:
 
-$ source <(yq shell-completion bash)
+$ source <(yq completion bash)
 
 # To load completions for each session, execute once:
 Linux:
-  $ yq shell-completion bash > /etc/bash_completion.d/yq
+  $ yq completion bash > /etc/bash_completion.d/yq
 MacOS:
-  $ yq shell-completion bash > /usr/local/etc/bash_completion.d/yq
+  $ yq completion bash > /usr/local/etc/bash_completion.d/yq
 
 Zsh:
 
@@ -29,25 +30,25 @@ Zsh:
 $ echo "autoload -U compinit; compinit" >> ~/.zshrc
 
 # To load completions for each session, execute once:
-$ yq shell-completion zsh > "${fpath[1]}/_yq"
+$ yq completion zsh > "${fpath[1]}/_yq"
 
 # You will need to start a new shell for this setup to take effect.
 
 Fish:
 
-$ yq shell-completion fish | source
+$ yq completion fish | source
 
 # To load completions for each session, execute once:
-$ yq shell-completion fish > ~/.config/fish/completions/yq.fish
+$ yq completion fish > ~/.config/fish/completions/yq.fish
 `,
 	DisableFlagsInUseLine: true,
 	ValidArgs:             []string{"bash", "zsh", "fish", "powershell"},
-	Args:                  cobra.ExactValidArgs(1),
+	Args:                  cobra.MatchAll(cobra.ExactArgs(1), cobra.OnlyValidArgs),
 	RunE: func(cmd *cobra.Command, args []string) error {
 		var err error = nil
 		switch args[0] {
 		case "bash":
-			err = cmd.Root().GenBashCompletion(os.Stdout)
+			err = cmd.Root().GenBashCompletionV2(os.Stdout, true)
 		case "zsh":
 			err = cmd.Root().GenZshCompletion(os.Stdout)
 		case "fish":

cmd/constant.go

@@ -1,30 +1,31 @@
 package cmd
 
-var leadingContentPreProcessing = true
-var unwrapScalar = true
+var unwrapScalarFlag = newUnwrapFlag()
+
+var printNodeInfo = false
+
+var unwrapScalar = false
 
 var writeInplace = false
 var outputToJSON = false
-var outputFormat = "yaml"
-var inputFormat = "yaml"
 
-var xmlAttributePrefix = "+"
-var xmlContentName = "+content"
-var xmlStrictMode = false
-var xmlKeepNamespace = true
-var xmlUseRawToken = true
+var outputFormat = ""
+
+var inputFormat = ""
 
 var exitStatus = false
-var forceColor = false
-var forceNoColor = false
-var colorsEnabled = false
 var indent = 2
 var noDocSeparators = false
 var nullInput = false
+var nulSepOutput = false
 var verbose = false
 var version = false
 var prettyPrint = false
 
+var forceColor = false
+var forceNoColor = false
+var colorsEnabled = false
+
 // can be either "" (off), "extract" or "process"
 var frontMatter = ""
 

cmd/evaluate_all_command.go

@@ -12,8 +12,14 @@ func createEvaluateAllCommand() *cobra.Command {
 		Use:     "eval-all [expression] [yaml_file1]...",
 		Aliases: []string{"ea"},
 		Short:   "Loads _all_ yaml documents of _all_ yaml files and runs expression once",
+		ValidArgsFunction: func(_ *cobra.Command, args []string, _ string) ([]string, cobra.ShellCompDirective) {
+			if len(args) == 0 {
+				return nil, cobra.ShellCompDirectiveNoFileComp
+			}
+			return nil, cobra.ShellCompDirectiveDefault
+		},
 		Example: `
-# Merge f2.yml into f1.yml (inplace)
+# Merge f2.yml into f1.yml (in place)
 yq eval-all --inplace 'select(fileIndex == 0) * select(fileIndex == 1)' f1.yml f2.yml
 ## the same command and expression using shortened names:
 yq ea -i 'select(fi == 0) * select(fi == 1)' f1.yml f2.yml
@@ -26,7 +32,7 @@ yq ea '. as $item ireduce ({}; . * $item )' file1.yml file2.yml ...
 ## use '-' as a filename to pipe from STDIN
 cat file2.yml | yq ea '.a.b' file1.yml - file3.yml
 `,
-		Long: `yq is a portable command-line YAML processor (https://github.com/mikefarah/yq/) 
+		Long: `yq is a portable command-line data file processor (https://github.com/mikefarah/yq/) 
 See https://mikefarah.gitbook.io/yq/ for detailed documentation and examples.
 
 ## Evaluate All ##
@@ -70,12 +76,12 @@ func evaluateAll(cmd *cobra.Command, args []string) (cmdError error) {
 		}()
 	}
 
-	format, err := yqlib.OutputFormatFromString(outputFormat)
+	format, err := yqlib.FormatFromString(outputFormat)
 	if err != nil {
 		return err
 	}
 
-	decoder, err := configureDecoder()
+	decoder, err := configureDecoder(true)
 	if err != nil {
 		return err
 	}
@@ -84,9 +90,15 @@ func evaluateAll(cmd *cobra.Command, args []string) (cmdError error) {
 	if err != nil {
 		return err
 	}
-	encoder := configureEncoder(format)
+	encoder, err := configureEncoder()
+	if err != nil {
+		return err
+	}
 
 	printer := yqlib.NewPrinter(encoder, printerWriter)
+	if nulSepOutput {
+		printer.SetNulSepOutput(true)
+	}
 
 	if frontMatter != "" {
 		frontMatterHandler := yqlib.NewFrontMatterHandler(args[0])
@@ -109,13 +121,13 @@ func evaluateAll(cmd *cobra.Command, args []string) (cmdError error) {
 	switch len(args) {
 	case 0:
 		if nullInput {
-			err = yqlib.NewStreamEvaluator().EvaluateNew(processExpression(expression), printer, "")
+			err = yqlib.NewStreamEvaluator().EvaluateNew(processExpression(expression), printer)
 		} else {
 			cmd.Println(cmd.UsageString())
 			return nil
 		}
 	default:
-		err = allAtOnceEvaluator.EvaluateFiles(processExpression(expression), args, printer, leadingContentPreProcessing, decoder)
+		err = allAtOnceEvaluator.EvaluateFiles(processExpression(expression), args, printer, decoder)
 	}
 
 	completedSuccessfully = err == nil

cmd/evaluate_sequence_command.go

@@ -13,6 +13,12 @@ func createEvaluateSequenceCommand() *cobra.Command {
 		Use:     "eval [expression] [yaml_file1]...",
 		Aliases: []string{"e"},
 		Short:   "(default) Apply the expression to each document in each yaml file in sequence",
+		ValidArgsFunction: func(_ *cobra.Command, args []string, _ string) ([]string, cobra.ShellCompDirective) {
+			if len(args) == 0 {
+				return nil, cobra.ShellCompDirectiveNoFileComp
+			}
+			return nil, cobra.ShellCompDirectiveDefault
+		},
 		Example: `
 # Reads field under the given path for each file
 yq e '.a.b' f1.yml f2.yml 
@@ -28,10 +34,10 @@ cat file2.yml | yq e '.a.b' file1.yml - file3.yml
 ## Note that editing an empty file does not work.
 yq e -n '.a.b.c = "cat"' 
 
-# Update a file inplace
+# Update a file in place
 yq e '.a.b = "cool"' -i file.yaml 
 `,
-		Long: `yq is a portable command-line YAML processor (https://github.com/mikefarah/yq/) 
+		Long: `yq is a portable command-line data file processor (https://github.com/mikefarah/yq/) 
 See https://mikefarah.gitbook.io/yq/ for detailed documentation and examples.
 
 ## Evaluate Sequence ##
@@ -84,7 +90,7 @@ func evaluateSequence(cmd *cobra.Command, args []string) (cmdError error) {
 		}()
 	}
 
-	format, err := yqlib.OutputFormatFromString(outputFormat)
+	format, err := yqlib.FormatFromString(outputFormat)
 	if err != nil {
 		return err
 	}
@@ -93,11 +99,22 @@ func evaluateSequence(cmd *cobra.Command, args []string) (cmdError error) {
 	if err != nil {
 		return err
 	}
-	encoder := configureEncoder(format)
+	encoder, err := configureEncoder()
+	if err != nil {
+		return err
+	}
 
 	printer := yqlib.NewPrinter(encoder, printerWriter)
 
-	decoder, err := configureDecoder()
+	if printNodeInfo {
+		printer = yqlib.NewNodeInfoPrinter(printerWriter)
+	}
+
+	if nulSepOutput {
+		printer.SetNulSepOutput(true)
+	}
+
+	decoder, err := configureDecoder(false)
 	if err != nil {
 		return err
 	}
@@ -123,13 +140,13 @@ func evaluateSequence(cmd *cobra.Command, args []string) (cmdError error) {
 	switch len(args) {
 	case 0:
 		if nullInput {
-			err = streamEvaluator.EvaluateNew(processExpression(expression), printer, "")
+			err = streamEvaluator.EvaluateNew(processExpression(expression), printer)
 		} else {
 			cmd.Println(cmd.UsageString())
 			return nil
 		}
 	default:
-		err = streamEvaluator.EvaluateFiles(processExpression(expression), args, printer, leadingContentPreProcessing, decoder)
+		err = streamEvaluator.EvaluateFiles(processExpression(expression), args, printer, decoder)
 	}
 	completedSuccessfully = err == nil
 

cmd/root.go

@@ -1,21 +1,52 @@
 package cmd
 
 import (
+	"fmt"
 	"os"
+	"strings"
 
 	"github.com/mikefarah/yq/v4/pkg/yqlib"
 	"github.com/spf13/cobra"
 	logging "gopkg.in/op/go-logging.v1"
 )
 
+type runeValue rune
+
+func newRuneVar(p *rune) *runeValue {
+	return (*runeValue)(p)
+}
+
+func (r *runeValue) String() string {
+	return string(*r)
+}
+
+func (r *runeValue) Set(rawVal string) error {
+	val := strings.ReplaceAll(rawVal, "\\n", "\n")
+	val = strings.ReplaceAll(val, "\\t", "\t")
+	val = strings.ReplaceAll(val, "\\r", "\r")
+	val = strings.ReplaceAll(val, "\\f", "\f")
+	val = strings.ReplaceAll(val, "\\v", "\v")
+	if len(val) != 1 {
+		return fmt.Errorf("[%v] is not a valid character. Must be length 1 was %v", val, len(val))
+	}
+	*r = runeValue(rune(val[0]))
+	return nil
+}
+
+func (r *runeValue) Type() string {
+	return "char"
+}
+
 func New() *cobra.Command {
 	var rootCmd = &cobra.Command{
 		Use:   "yq",
-		Short: "yq is a lightweight and portable command-line YAML processor.",
-		Long: `yq is a portable command-line YAML processor (https://github.com/mikefarah/yq/) 
+		Short: "yq is a lightweight and portable command-line data file processor.",
+		Long: `yq is a portable command-line data file processor (https://github.com/mikefarah/yq/) 
 See https://mikefarah.gitbook.io/yq/ for detailed documentation and examples.`,
 		Example: `
-# yq defaults to 'eval' command if no command is specified. See "yq eval --help" for more examples.
+# yq tries to auto-detect the file format based off the extension, and defaults to YAML if it's unknown (or piping through STDIN)
+# Use the '-p/--input-format' flag to specify a format type.
+cat file.xml | yq -p xml
 
 # read the "stuff" node from "myfile.yml"
 yq '.stuff' < myfile.yml
@@ -24,7 +55,7 @@ yq '.stuff' < myfile.yml
 yq -i '.stuff = "foo"' myfile.yml
 
 # print contents of sample.json as idiomatic YAML
-yq -P sample.json
+yq -P -oy sample.json
 `,
 
 		RunE: func(cmd *cobra.Command, args []string) error {
@@ -35,30 +66,40 @@ yq -P sample.json
 			return evaluateSequence(cmd, args)
 
 		},
-		PersistentPreRun: func(cmd *cobra.Command, args []string) {
+		PersistentPreRunE: func(cmd *cobra.Command, _ []string) error {
 			cmd.SetOut(cmd.OutOrStdout())
+			level := logging.WARNING
+			stringFormat := `[%{level}] %{color}%{time:15:04:05}%{color:reset} %{message}`
 
-			var format = logging.MustStringFormatter(
-				`%{color}%{time:15:04:05} %{shortfunc} [%{level:.4s}]%{color:reset} %{message}`,
-			)
+			// when NO_COLOR environment variable presents and not an empty string the coloured output should be disabled;
+			// refer to no-color.org
+			forceNoColor = forceNoColor || os.Getenv("NO_COLOR") != ""
+
+			if verbose && forceNoColor {
+				level = logging.DEBUG
+				stringFormat = `[%{level:5.5s}] %{time:15:04:05} %{shortfile:-33s} %{shortfunc:-25s} %{message}`
+			} else if verbose {
+				level = logging.DEBUG
+				stringFormat = `[%{level:5.5s}] %{color}%{time:15:04:05}%{color:bold} %{shortfile:-33s} %{shortfunc:-25s}%{color:reset} %{message}`
+			} else if forceNoColor {
+				stringFormat = `[%{level}] %{time:15:04:05} %{message}`
+			}
+
+			var format = logging.MustStringFormatter(stringFormat)
 			var backend = logging.AddModuleLevel(
 				logging.NewBackendFormatter(logging.NewLogBackend(os.Stderr, "", 0), format))
 
-			if verbose {
-				backend.SetLevel(logging.DEBUG, "")
-			} else {
-				backend.SetLevel(logging.ERROR, "")
-			}
+			backend.SetLevel(level, "")
 
 			logging.SetBackend(backend)
 			yqlib.InitExpressionParser()
-			yqlib.XMLPreferences.AttributePrefix = xmlAttributePrefix
-			yqlib.XMLPreferences.ContentName = xmlContentName
-			yqlib.XMLPreferences.StrictMode = xmlStrictMode
+
+			return nil
 		},
 	}
 
 	rootCmd.PersistentFlags().BoolVarP(&verbose, "verbose", "v", false, "verbose mode")
+	rootCmd.PersistentFlags().BoolVarP(&printNodeInfo, "debug-node-info", "", false, "debug node info")
 
 	rootCmd.PersistentFlags().BoolVarP(&outputToJSON, "tojson", "j", false, "(deprecated) output as json. Set indent to 0 to print json in one line.")
 	err := rootCmd.PersistentFlags().MarkDeprecated("tojson", "please use -o=json instead")
@@ -66,35 +107,111 @@ yq -P sample.json
 		panic(err)
 	}
 
-	rootCmd.PersistentFlags().StringVarP(&outputFormat, "output-format", "o", "yaml", "[yaml|y|json|j|props|p|xml|x] output format type.")
-	rootCmd.PersistentFlags().StringVarP(&inputFormat, "input-format", "p", "yaml", "[yaml|y|props|p|xml|x] parse format for input. Note that json is a subset of yaml.")
+	rootCmd.PersistentFlags().StringVarP(&outputFormat, "output-format", "o", "auto", fmt.Sprintf("[auto|a|%v] output format type.", yqlib.GetAvailableOutputFormatString()))
+	var outputCompletions = []string{"auto"}
+	for _, formats := range yqlib.GetAvailableOutputFormats() {
+		outputCompletions = append(outputCompletions, formats.FormalName)
+	}
 
-	rootCmd.PersistentFlags().StringVar(&xmlAttributePrefix, "xml-attribute-prefix", "+", "prefix for xml attributes")
-	rootCmd.PersistentFlags().StringVar(&xmlContentName, "xml-content-name", "+content", "name for xml content (if no attribute name is present).")
-	rootCmd.PersistentFlags().BoolVar(&xmlStrictMode, "xml-strict-mode", false, "enables strict parsing of XML. See https://pkg.go.dev/encoding/xml for more details.")
-	rootCmd.PersistentFlags().BoolVar(&xmlKeepNamespace, "xml-keep-namespace", true, "enables keeping namespace after parsing attributes")
-	rootCmd.PersistentFlags().BoolVar(&xmlUseRawToken, "xml-raw-token", true, "enables using RawToken method instead Token. Commonly disables namespace translations. See https://pkg.go.dev/encoding/xml#Decoder.RawToken for details.")
+	if err = rootCmd.RegisterFlagCompletionFunc("output-format", cobra.FixedCompletions(outputCompletions, cobra.ShellCompDirectiveNoFileComp)); err != nil {
+		panic(err)
+	}
+	rootCmd.PersistentFlags().StringVarP(&inputFormat, "input-format", "p", "auto", fmt.Sprintf("[auto|a|%v] parse format for input.", yqlib.GetAvailableInputFormatString()))
+
+	var inputCompletions = []string{"auto"}
+	for _, formats := range yqlib.GetAvailableInputFormats() {
+		inputCompletions = append(inputCompletions, formats.FormalName)
+	}
+
+	if err = rootCmd.RegisterFlagCompletionFunc("input-format", cobra.FixedCompletions(inputCompletions, cobra.ShellCompDirectiveNoFileComp)); err != nil {
+		panic(err)
+	}
+
+	rootCmd.PersistentFlags().StringVar(&yqlib.ConfiguredXMLPreferences.AttributePrefix, "xml-attribute-prefix", yqlib.ConfiguredXMLPreferences.AttributePrefix, "prefix for xml attributes")
+	if err = rootCmd.RegisterFlagCompletionFunc("xml-attribute-prefix", cobra.NoFileCompletions); err != nil {
+		panic(err)
+	}
+	rootCmd.PersistentFlags().StringVar(&yqlib.ConfiguredXMLPreferences.ContentName, "xml-content-name", yqlib.ConfiguredXMLPreferences.ContentName, "name for xml content (if no attribute name is present).")
+	if err = rootCmd.RegisterFlagCompletionFunc("xml-content-name", cobra.NoFileCompletions); err != nil {
+		panic(err)
+	}
+	rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredXMLPreferences.StrictMode, "xml-strict-mode", yqlib.ConfiguredXMLPreferences.StrictMode, "enables strict parsing of XML. See https://pkg.go.dev/encoding/xml for more details.")
+	rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredXMLPreferences.KeepNamespace, "xml-keep-namespace", yqlib.ConfiguredXMLPreferences.KeepNamespace, "enables keeping namespace after parsing attributes")
+	rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredXMLPreferences.UseRawToken, "xml-raw-token", yqlib.ConfiguredXMLPreferences.UseRawToken, "enables using RawToken method instead Token. Commonly disables namespace translations. See https://pkg.go.dev/encoding/xml#Decoder.RawToken for details.")
+	rootCmd.PersistentFlags().StringVar(&yqlib.ConfiguredXMLPreferences.ProcInstPrefix, "xml-proc-inst-prefix", yqlib.ConfiguredXMLPreferences.ProcInstPrefix, "prefix for xml processing instructions (e.g. <?xml version=\"1\"?>)")
+	if err = rootCmd.RegisterFlagCompletionFunc("xml-proc-inst-prefix", cobra.NoFileCompletions); err != nil {
+		panic(err)
+	}
+	rootCmd.PersistentFlags().StringVar(&yqlib.ConfiguredXMLPreferences.DirectiveName, "xml-directive-name", yqlib.ConfiguredXMLPreferences.DirectiveName, "name for xml directives (e.g. <!DOCTYPE thing cat>)")
+	if err = rootCmd.RegisterFlagCompletionFunc("xml-directive-name", cobra.NoFileCompletions); err != nil {
+		panic(err)
+	}
+	rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredXMLPreferences.SkipProcInst, "xml-skip-proc-inst", yqlib.ConfiguredXMLPreferences.SkipProcInst, "skip over process instructions (e.g. <?xml version=\"1\"?>)")
+	rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredXMLPreferences.SkipDirectives, "xml-skip-directives", yqlib.ConfiguredXMLPreferences.SkipDirectives, "skip over directives (e.g. <!DOCTYPE thing cat>)")
+
+	rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredCsvPreferences.AutoParse, "csv-auto-parse", yqlib.ConfiguredCsvPreferences.AutoParse, "parse CSV YAML/JSON values")
+	rootCmd.PersistentFlags().Var(newRuneVar(&yqlib.ConfiguredCsvPreferences.Separator), "csv-separator", "CSV Separator character")
+
+	rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredTsvPreferences.AutoParse, "tsv-auto-parse", yqlib.ConfiguredTsvPreferences.AutoParse, "parse TSV YAML/JSON values")
+
+	rootCmd.PersistentFlags().StringVar(&yqlib.ConfiguredLuaPreferences.DocPrefix, "lua-prefix", yqlib.ConfiguredLuaPreferences.DocPrefix, "prefix")
+	if err = rootCmd.RegisterFlagCompletionFunc("lua-prefix", cobra.NoFileCompletions); err != nil {
+		panic(err)
+	}
+	rootCmd.PersistentFlags().StringVar(&yqlib.ConfiguredLuaPreferences.DocSuffix, "lua-suffix", yqlib.ConfiguredLuaPreferences.DocSuffix, "suffix")
+	if err = rootCmd.RegisterFlagCompletionFunc("lua-suffix", cobra.NoFileCompletions); err != nil {
+		panic(err)
+	}
+	rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredLuaPreferences.UnquotedKeys, "lua-unquoted", yqlib.ConfiguredLuaPreferences.UnquotedKeys, "output unquoted string keys (e.g. {foo=\"bar\"})")
+	rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredLuaPreferences.Globals, "lua-globals", yqlib.ConfiguredLuaPreferences.Globals, "output keys as top-level global variables")
+
+	rootCmd.PersistentFlags().StringVar(&yqlib.ConfiguredPropertiesPreferences.KeyValueSeparator, "properties-separator", yqlib.ConfiguredPropertiesPreferences.KeyValueSeparator, "separator to use between keys and values")
+	rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredPropertiesPreferences.UseArrayBrackets, "properties-array-brackets", yqlib.ConfiguredPropertiesPreferences.UseArrayBrackets, "use [x] in array paths (e.g. for SpringBoot)")
+
+	rootCmd.PersistentFlags().BoolVar(&yqlib.StringInterpolationEnabled, "string-interpolation", yqlib.StringInterpolationEnabled, "Toggles strings interpolation of \\(exp)")
 
 	rootCmd.PersistentFlags().BoolVarP(&nullInput, "null-input", "n", false, "Don't read input, simply evaluate the expression given. Useful for creating docs from scratch.")
 	rootCmd.PersistentFlags().BoolVarP(&noDocSeparators, "no-doc", "N", false, "Don't print document separators (---)")
 
 	rootCmd.PersistentFlags().IntVarP(&indent, "indent", "I", 2, "sets indent level for output")
+	if err = rootCmd.RegisterFlagCompletionFunc("indent", cobra.NoFileCompletions); err != nil {
+		panic(err)
+	}
 	rootCmd.Flags().BoolVarP(&version, "version", "V", false, "Print version information and quit")
-	rootCmd.PersistentFlags().BoolVarP(&writeInplace, "inplace", "i", false, "update the file inplace of first file given.")
-	rootCmd.PersistentFlags().BoolVarP(&unwrapScalar, "unwrapScalar", "r", true, "unwrap scalar, print the value with no quotes, colors or comments")
+	rootCmd.PersistentFlags().BoolVarP(&writeInplace, "inplace", "i", false, "update the file in place of first file given.")
+	rootCmd.PersistentFlags().VarP(unwrapScalarFlag, "unwrapScalar", "r", "unwrap scalar, print the value with no quotes, colors or comments. Defaults to true for yaml")
+	rootCmd.PersistentFlags().Lookup("unwrapScalar").NoOptDefVal = "true"
+	rootCmd.PersistentFlags().BoolVarP(&nulSepOutput, "nul-output", "0", false, "Use NUL char to separate values. If unwrap scalar is also set, fail if unwrapped scalar contains NUL char.")
+
 	rootCmd.PersistentFlags().BoolVarP(&prettyPrint, "prettyPrint", "P", false, "pretty print, shorthand for '... style = \"\"'")
 	rootCmd.PersistentFlags().BoolVarP(&exitStatus, "exit-status", "e", false, "set exit status if there are no matches or null or false is returned")
 
 	rootCmd.PersistentFlags().BoolVarP(&forceColor, "colors", "C", false, "force print with colors")
-	rootCmd.PersistentFlags().BoolVarP(&forceNoColor, "no-colors", "M", false, "force print with no colors")
+	rootCmd.PersistentFlags().BoolVarP(&forceNoColor, "no-colors", "M", forceNoColor, "force print with no colors")
 	rootCmd.PersistentFlags().StringVarP(&frontMatter, "front-matter", "f", "", "(extract|process) first input as yaml front-matter. Extract will pull out the yaml content, process will run the expression against the yaml content, leaving the remaining data intact")
+	if err = rootCmd.RegisterFlagCompletionFunc("front-matter", cobra.FixedCompletions([]string{"extract", "process"}, cobra.ShellCompDirectiveNoFileComp)); err != nil {
+		panic(err)
+	}
 	rootCmd.PersistentFlags().StringVarP(&forceExpression, "expression", "", "", "forcibly set the expression argument. Useful when yq argument detection thinks your expression is a file.")
-	rootCmd.PersistentFlags().BoolVarP(&leadingContentPreProcessing, "header-preprocess", "", true, "Slurp any header comments and separators before processing expression.")
+	if err = rootCmd.RegisterFlagCompletionFunc("expression", cobra.NoFileCompletions); err != nil {
+		panic(err)
+	}
+	rootCmd.PersistentFlags().BoolVarP(&yqlib.ConfiguredYamlPreferences.LeadingContentPreProcessing, "header-preprocess", "", true, "Slurp any header comments and separators before processing expression.")
+	rootCmd.PersistentFlags().BoolVarP(&yqlib.ConfiguredYamlPreferences.FixMergeAnchorToSpec, "yaml-fix-merge-anchor-to-spec", "", false, "Fix merge anchor to match YAML spec. Will default to true in late 2025")
 
-	rootCmd.PersistentFlags().StringVarP(&splitFileExp, "split-exp", "s", "", "print each result (or doc) into a file named (exp). [exp] argument must return a string. You can use $index in the expression as the result counter.")
+	rootCmd.PersistentFlags().StringVarP(&splitFileExp, "split-exp", "s", "", "print each result (or doc) into a file named (exp). [exp] argument must return a string. You can use $index in the expression as the result counter. The necessary directories will be created.")
+	if err = rootCmd.RegisterFlagCompletionFunc("split-exp", cobra.NoFileCompletions); err != nil {
+		panic(err)
+	}
 	rootCmd.PersistentFlags().StringVarP(&splitFileExpFile, "split-exp-file", "", "", "Use a file to specify the split-exp expression.")
+	if err = rootCmd.MarkPersistentFlagFilename("split-exp-file"); err != nil {
+		panic(err)
+	}
 
 	rootCmd.PersistentFlags().StringVarP(&expressionFile, "from-file", "", "", "Load expression from specified file.")
+	if err = rootCmd.MarkPersistentFlagFilename("from-file"); err != nil {
+		panic(err)
+	}
 
 	rootCmd.AddCommand(
 		createEvaluateSequenceCommand(),

cmd/unwrap_flag.go

@@ -0,0 +1,46 @@
+package cmd
+
+import (
+	"strconv"
+
+	"github.com/spf13/pflag"
+)
+
+type boolFlag interface {
+	pflag.Value
+	IsExplicitlySet() bool
+	IsSet() bool
+}
+
+type unwrapScalarFlagStrc struct {
+	explicitlySet bool
+	value         bool
+}
+
+func newUnwrapFlag() boolFlag {
+	return &unwrapScalarFlagStrc{value: true}
+}
+
+func (f *unwrapScalarFlagStrc) IsExplicitlySet() bool {
+	return f.explicitlySet
+}
+
+func (f *unwrapScalarFlagStrc) IsSet() bool {
+	return f.value
+}
+
+func (f *unwrapScalarFlagStrc) String() string {
+	return strconv.FormatBool(f.value)
+}
+
+func (f *unwrapScalarFlagStrc) Set(value string) error {
+
+	v, err := strconv.ParseBool(value)
+	f.value = v
+	f.explicitlySet = true
+	return err
+}
+
+func (*unwrapScalarFlagStrc) Type() string {
+	return "bool"
+}

cmd/utils.go

@@ -4,80 +4,176 @@ import (
 	"fmt"
 	"io"
 	"os"
+	"strings"
 
 	"github.com/mikefarah/yq/v4/pkg/yqlib"
 	"github.com/spf13/cobra"
 	"gopkg.in/op/go-logging.v1"
 )
 
+func isAutomaticOutputFormat() bool {
+	return outputFormat == "" || outputFormat == "auto" || outputFormat == "a"
+}
+
 func initCommand(cmd *cobra.Command, args []string) (string, []string, error) {
 	cmd.SilenceUsage = true
 
-	fileInfo, _ := os.Stdout.Stat()
-
-	if forceColor || (!forceNoColor && (fileInfo.Mode()&os.ModeCharDevice) != 0) {
-		colorsEnabled = true
-	}
+	setupColors()
 
 	expression, args, err := processArgs(args)
 	if err != nil {
 		return "", nil, err
 	}
 
-	if splitFileExpFile != "" {
-		splitExpressionBytes, err := os.ReadFile(splitFileExpFile)
-		if err != nil {
-			return "", nil, err
-		}
-		splitFileExp = string(splitExpressionBytes)
+	if err := loadSplitFileExpression(); err != nil {
+		return "", nil, err
 	}
 
-	// backwards compatibility
-	if outputToJSON {
-		outputFormat = "json"
+	handleBackwardsCompatibility()
+
+	if err := validateCommandFlags(args); err != nil {
+		return "", nil, err
 	}
 
-	if writeInplace && (len(args) == 0 || args[0] == "-") {
-		return "", nil, fmt.Errorf("write inplace flag only applicable when giving an expression and at least one file")
+	if err := configureFormats(args); err != nil {
+		return "", nil, err
 	}
 
-	if frontMatter != "" && len(args) == 0 {
-		return "", nil, fmt.Errorf("front matter flag only applicable when giving an expression and at least one file")
-	}
-
-	if writeInplace && splitFileExp != "" {
-		return "", nil, fmt.Errorf("write inplace cannot be used with split file")
-	}
-
-	if nullInput && len(args) > 0 {
-		return "", nil, fmt.Errorf("cannot pass files in when using null-input flag")
-	}
+	configureUnwrapScalar()
 
 	return expression, args, nil
 }
 
-func configureDecoder() (yqlib.Decoder, error) {
-	yqlibInputFormat, err := yqlib.InputFormatFromString(inputFormat)
+func setupColors() {
+	fileInfo, _ := os.Stdout.Stat()
+
+	if forceColor || (!forceNoColor && (fileInfo.Mode()&os.ModeCharDevice) != 0) {
+		colorsEnabled = true
+	}
+}
+
+func loadSplitFileExpression() error {
+	if splitFileExpFile != "" {
+		splitExpressionBytes, err := os.ReadFile(splitFileExpFile)
+		if err != nil {
+			return err
+		}
+		splitFileExp = string(splitExpressionBytes)
+	}
+	return nil
+}
+
+func handleBackwardsCompatibility() {
+	// backwards compatibility
+	if outputToJSON {
+		outputFormat = "json"
+	}
+}
+
+func validateCommandFlags(args []string) error {
+	if writeInplace && (len(args) == 0 || args[0] == "-") {
+		return fmt.Errorf("write in place flag only applicable when giving an expression and at least one file")
+	}
+
+	if frontMatter != "" && len(args) == 0 {
+		return fmt.Errorf("front matter flag only applicable when giving an expression and at least one file")
+	}
+
+	if writeInplace && splitFileExp != "" {
+		return fmt.Errorf("write in place cannot be used with split file")
+	}
+
+	if nullInput && len(args) > 0 {
+		return fmt.Errorf("cannot pass files in when using null-input flag")
+	}
+
+	return nil
+}
+
+func configureFormats(args []string) error {
+	inputFilename := ""
+	if len(args) > 0 {
+		inputFilename = args[0]
+	}
+
+	if err := configureInputFormat(inputFilename); err != nil {
+		return err
+	}
+
+	if err := configureOutputFormat(); err != nil {
+		return err
+	}
+
+	yqlib.GetLogger().Debug("Using input format %v", inputFormat)
+	yqlib.GetLogger().Debug("Using output format %v", outputFormat)
+
+	return nil
+}
+
+func configureInputFormat(inputFilename string) error {
+	if inputFormat == "" || inputFormat == "auto" || inputFormat == "a" {
+		inputFormat = yqlib.FormatStringFromFilename(inputFilename)
+
+		_, err := yqlib.FormatFromString(inputFormat)
+		if err != nil {
+			// unknown file type, default to yaml
+			yqlib.GetLogger().Debug("Unknown file format extension '%v', defaulting to yaml", inputFormat)
+			inputFormat = "yaml"
+			if isAutomaticOutputFormat() {
+				outputFormat = "yaml"
+			}
+		} else if isAutomaticOutputFormat() {
+			outputFormat = inputFormat
+		}
+	} else if isAutomaticOutputFormat() {
+		// backwards compatibility -
+		// before this was introduced, `yq -pcsv things.csv`
+		// would produce *yaml* output.
+		//
+		outputFormat = yqlib.FormatStringFromFilename(inputFilename)
+		if inputFilename != "-" {
+			yqlib.GetLogger().Warning("yq default output is now 'auto' (based on the filename extension). Normally yq would output '%v', but for backwards compatibility 'yaml' has been set. Please use -oy to specify yaml, or drop the -p flag.", outputFormat)
+		}
+		outputFormat = "yaml"
+	}
+	return nil
+}
+
+func configureOutputFormat() error {
+	outputFormatType, err := yqlib.FormatFromString(outputFormat)
+	if err != nil {
+		return err
+	}
+
+	if outputFormatType == yqlib.YamlFormat ||
+		outputFormatType == yqlib.PropertiesFormat {
+		unwrapScalar = true
+	}
+
+	return nil
+}
+
+func configureUnwrapScalar() {
+	if unwrapScalarFlag.IsExplicitlySet() {
+		unwrapScalar = unwrapScalarFlag.IsSet()
+	}
+}
+
+func configureDecoder(evaluateTogether bool) (yqlib.Decoder, error) {
+	format, err := yqlib.FormatFromString(inputFormat)
 	if err != nil {
 		return nil, err
 	}
-	switch yqlibInputFormat {
-	case yqlib.XMLInputFormat:
-		return yqlib.NewXMLDecoder(xmlAttributePrefix, xmlContentName, xmlStrictMode, xmlKeepNamespace, xmlUseRawToken), nil
-	case yqlib.PropertiesInputFormat:
-		return yqlib.NewPropertiesDecoder(), nil
-	case yqlib.JsonInputFormat:
-		return yqlib.NewJSONDecoder(), nil
-	case yqlib.CSVObjectInputFormat:
-		return yqlib.NewCSVObjectDecoder(','), nil
-	case yqlib.TSVObjectInputFormat:
-		return yqlib.NewCSVObjectDecoder('\t'), nil
-	}
+	yqlib.ConfiguredYamlPreferences.EvaluateTogether = evaluateTogether
 
-	return yqlib.NewYamlDecoder(), nil
+	yqlibDecoder := format.DecoderFactory()
+	if yqlibDecoder == nil {
+		return nil, fmt.Errorf("no support for %s input format", inputFormat)
+	}
+	return yqlibDecoder, nil
 }
 
-func configurePrinterWriter(format yqlib.PrinterOutputFormat, out io.Writer) (yqlib.PrinterWriter, error) {
+func configurePrinterWriter(format *yqlib.Format, out io.Writer) (yqlib.PrinterWriter, error) {
 
 	var printerWriter yqlib.PrinterWriter
 
@@ -94,22 +190,30 @@ func configurePrinterWriter(format yqlib.PrinterOutputFormat, out io.Writer) (yq
 	return printerWriter, nil
 }
 
-func configureEncoder(format yqlib.PrinterOutputFormat) yqlib.Encoder {
-	switch format {
-	case yqlib.JSONOutputFormat:
-		return yqlib.NewJSONEncoder(indent, colorsEnabled)
-	case yqlib.PropsOutputFormat:
-		return yqlib.NewPropertiesEncoder(unwrapScalar)
-	case yqlib.CSVOutputFormat:
-		return yqlib.NewCsvEncoder(',')
-	case yqlib.TSVOutputFormat:
-		return yqlib.NewCsvEncoder('\t')
-	case yqlib.YamlOutputFormat:
-		return yqlib.NewYamlEncoder(indent, colorsEnabled, !noDocSeparators, unwrapScalar)
-	case yqlib.XMLOutputFormat:
-		return yqlib.NewXMLEncoder(indent, xmlAttributePrefix, xmlContentName)
+func configureEncoder() (yqlib.Encoder, error) {
+	yqlibOutputFormat, err := yqlib.FormatFromString(outputFormat)
+	if err != nil {
+		return nil, err
 	}
-	panic("invalid encoder")
+	yqlib.ConfiguredXMLPreferences.Indent = indent
+	yqlib.ConfiguredYamlPreferences.Indent = indent
+	yqlib.ConfiguredJSONPreferences.Indent = indent
+
+	yqlib.ConfiguredYamlPreferences.UnwrapScalar = unwrapScalar
+	yqlib.ConfiguredPropertiesPreferences.UnwrapScalar = unwrapScalar
+	yqlib.ConfiguredJSONPreferences.UnwrapScalar = unwrapScalar
+
+	yqlib.ConfiguredYamlPreferences.ColorsEnabled = colorsEnabled
+	yqlib.ConfiguredJSONPreferences.ColorsEnabled = colorsEnabled
+
+	yqlib.ConfiguredYamlPreferences.PrintDocSeparators = !noDocSeparators
+
+	encoder := yqlibOutputFormat.EncoderFactory()
+
+	if encoder == nil {
+		return nil, fmt.Errorf("no support for %s output format", outputFormat)
+	}
+	return encoder, err
 }
 
 // this is a hack to enable backwards compatibility with githubactions (which pipe /dev/null into everything)
@@ -133,8 +237,11 @@ func maybeFile(str string) bool {
 }
 
 func processStdInArgs(args []string) []string {
-	stat, _ := os.Stdin.Stat()
-	pipingStdin := (stat.Mode() & os.ModeCharDevice) == 0
+	stat, err := os.Stdin.Stat()
+	if err != nil {
+		yqlib.GetLogger().Debugf("error getting stdin: %v", err)
+	}
+	pipingStdin := stat != nil && (stat.Mode()&os.ModeCharDevice) == 0
 
 	// if we've been given a file, don't automatically
 	// read from stdin.
@@ -158,15 +265,25 @@ func processStdInArgs(args []string) []string {
 
 func processArgs(originalArgs []string) (string, []string, error) {
 	expression := forceExpression
+	args := processStdInArgs(originalArgs)
+	maybeFirstArgIsAFile := len(args) > 0 && maybeFile(args[0])
+
+	if expressionFile == "" && maybeFirstArgIsAFile && strings.HasSuffix(args[0], ".yq") {
+		// lets check if an expression file was given
+		yqlib.GetLogger().Debug("Assuming arg %v is an expression file", args[0])
+		expressionFile = args[0]
+		args = args[1:]
+	}
+
 	if expressionFile != "" {
 		expressionBytes, err := os.ReadFile(expressionFile)
 		if err != nil {
 			return "", nil, err
 		}
-		expression = string(expressionBytes)
+		//replace \r\n (windows) with good ol' unix file endings.
+		expression = strings.ReplaceAll(string(expressionBytes), "\r\n", "\n")
 	}
 
-	args := processStdInArgs(originalArgs)
 	yqlib.GetLogger().Debugf("processed args: %v", args)
 	if expression == "" && len(args) > 0 && args[0] != "-" && !maybeFile(args[0]) {
 		yqlib.GetLogger().Debug("assuming expression is '%v'", args[0])

cmd/version.go

@@ -11,7 +11,7 @@ var (
 	GitDescribe string
 
 	// Version is main version number that is being run at the moment.
-	Version = "4.27.5"
+	Version = "v4.46.1"
 
 	// VersionPrerelease is a pre-release marker for the version. If this is "" (empty string)
 	// then it means that it is a final release. Otherwise, this is a pre-release
@@ -45,5 +45,5 @@ func getHumanVersion() string {
 	}
 
 	// Strip off any single quotes added by the git information.
-	return strings.Replace(version, "'", "", -1)
+	return strings.ReplaceAll(version, "'", "")
 }

cspell.config.yaml

@@ -0,0 +1,14 @@
+---
+$schema: https://raw.githubusercontent.com/streetsidesoftware/cspell/main/cspell.schema.json
+version: '0.2'
+language: en-GB
+dictionaryDefinitions:
+  - name: project-words
+    path: './project-words.txt'
+    addWords: true
+dictionaries:
+  - project-words
+ignorePaths:
+  - 'vendor'
+  - 'bin'
+  - '/project-words.txt'

debian/changelog

@@ -1,210 +0,0 @@
-yq (4.16.2) focal; urgency=medium
-
-  * Fixed with semicolon space issue
-  * Updating with documentation
-  * Added STDIN example to the top
-  * minor readme cleanup
-  * Help text tweak
-  * Fixed docker timeout - simplify docker builds
-  * New release with docker build fixes
-  * Updating to go 1.17 to fix CVE #944
-  * Fix a typo in root.go
-  * Skip the tests if the nocheck Debian build option is specified
-  * Fixed select bug (#958)
-  * Sped up explode operator
-  * Slight performance improvement to context.ChildContext
-  * Speed up multiply
-  * Update README with recently added / changed options
-  * Make deepMatch report in linear time
-  * Removed leadingContentPreProcessing flag - header preprocessing is stable
-  * Revert "Removed leadingContentPreProcessing flag - header preprocessing is stable"
-  * Keep flag, it is needed in corner cases
-  * Updated Readme
-  * Man page
-  * Fixed expression parsing bug #970
-  * Bumping go-lang, docker versions
-  * Added test release flow
-  * Updated github action release to generate man page
-  * Bumping version
-  * Removing no longer needed github action
-  * Added decoder op
-  * Fixed newline handling when decoding/encoding
-  * Fixed newline handling in encoder/decoder
-  * Can specify indent in encode ops
-  * Added group_by operator
-  * Added flatten operator
-  * Fixed flatten error message
-  * Improving docs
-  * Split printer
-  * Refactored command logic
-  * Fix JSON encoding removing null #985
-  * Fixed acceptance tests
-  * gitbook
-  * Update document generation script
-  * Updating README
-  * Updating release instructions
-  * github action no longer uses data1.yml
-  * Create dependabot.yml
-  * Bump actions/create-release from 1.0.0 to 1.1.4
-  * Bump actions/setup-go from 1 to 2.1.4
-  * Bump github.com/goccy/go-yaml from 1.8.9 to 1.9.4
-  * Bump github.com/jinzhu/copier from 0.2.8 to 0.3.2
-  * Bump github.com/fatih/color from 1.10.0 to 1.13.0
-  * Bump github.com/spf13/cobra from 1.1.3 to 1.2.1
-  * Update dependabot.yml
-  * Update go.yml
-  * add build check to PRs
-  * Include secure as part of build process
-  * Fixing bad label in github action
-  * fixed printer test
-  * remove leading content indicator
-  * Fixed header preprocessing!
-  * lint : define golangci configuration file
-  * Update check.sh
-  * Load file acceptance test
-  * Minor improvement on handling front matter
-  * Improved load doc
-  * feature: detect MANPATh and install there
-  * Update install-man-page.sh
-  * simplify prod stage, move version label to action
-  * add labels, quote some values
-  * enable errorlint linter
-  * Added errorlint to devtools
-  * Added key operator
-  * Added more tests
-  * Fixing comments
-  * Attempt to fix golint problem
-  * Include version query for tools
-  * Clean up errored file?
-  * enable misspell linter
-  * updated readme
-  * update Golangci version to v1.43.0
-  * gci linter
-  * Better merge array by key example
-  * Added credit for merge by array example
-  * Better formatting of merge arrays example
-  * Better merge example
-  * Add accessor for the yq logger instance (#1013)
-  * Fixed collect op when working with multiple nodes
-  * Added map, map_values
-  * Add support for Podman as well as Docker (#1026)
-  * Bump github.com/jinzhu/copier from 0.3.2 to 0.3.4 (#1027)
-  * Added csv, tsv output formats
-  * Added encoder tests
-  * Cleanup test
-  * Fixed docker permission issue #1014
-  * Recording release notes for next release
-  * Assignment op no longer clobbers anchor (#1029)
-  * Added sort_by operator
-  * Improved error message
-  * Improved tips and tricks
-  * Report while filename failed to parse #1030
-  * Added script for extracting checksums
-  * Improved extract-checksum.sh
-  * Bump github.com/spf13/cobra from 1.2.1 to 1.3.0 (#1039)
-  * enable more linters (#1043)
-  * Bump golang compiler #1037
-
- -- Roberto Mier Escandon <rmescandon@gmail.com>  Tue, 21 Dec 2021 09:41:44 +0000
-
-yq (4.13.0) focal; urgency=medium
-
-  * New `with` operator for making multiple changes to a given path
-  * New `contains` operator, works like the `jq` equivalent
-  * Subtract operator now supports subtracting elements from arrays!
-  * Fixed Swapping values using variables #934
-  * Github Action now properly supports multiline output #936, thanks @pjxiao
-  * Fixed missing closing bracket validation #932
-  * Fix processing of hex numbers #929
-  * Fixed alternative and union operator issues #930
-  * Can now convert yaml to properties properties format (`-o=props`), See [docs](https://mikefarah.gitbook.io/yq/v/v4.x/usage/properties) for more info.
-  * Fixed document header/footer comment handling when merging (https://github.com/mikefarah/yq/issues/919)
-  * pretty print yaml 1.1 compatibility  (https://github.com/mikefarah/yq/issues/914)
-
- -- Roberto Mier Escandon <rmescandon@gmail.com>  Thu, 16 Sep 2021 20:58:30 +0200
-
-yq (4.9.6) focal; urgency=medium
-
-  * Added darwin/arm64 build, thanks @alecthomas
-  * Incremented docker alpine base version, thanks @da6d6i7-bronga
-  * Bug fix: multine expression
-  * Bug fix: special character
-
- -- Roberto Mier Escandon <rmescandon@gmail.com>  Tue, 29 Jun 2021 21:32:14 +0200
-
-yq (3.3.2) focal; urgency=medium
-
-  * Bug fix: existStatus bug (#459)
-  * Automatically makes a os temp directory if it does not exist (#461)
-
- -- Roberto Mier Escandon <rmescandon@gmail.com>  Fri, 07 Aug 2020 18:53:01 +0200
-
-yq (3.3-0) focal; urgency=medium
-
-  * You can control string styles (quotes) using the new --style flag
-  * String values now always have quotes when outputting to json
-  * Negative array indices now traverse the array backwards
-  * Added a --stripComments flag to print yaml without any comments
-  * Bumped go to version 1.14
-
- -- Roberto Mier Escandon <rmescandon@gmail.com>  Thu, 30 Apr 2020 20:45:44 +0200
-
-yq (3.1-2) eoan; urgency=medium
-
-  * Bug fix: yq 3 was removing empty inline-style objects and arrays (#355)
-  * Bug fix: Merge option returned different output when switching order of 
-    merging files(#347)
-  * Bug fix: Add new object to existing array object was failing in 3.1.1 (#361)
-  * Bug fix: yq 3 empty keys did not allow merging of values (#356)
-  * Bug fix: keys quoted during merge (#363)
-  * Bug fix: Correct length with wc -l (#362)
-  * Bug fix: Write to empty document removed path (#359)
-
- -- Roberto Mier Escandon <rmescandon@gmail.com>  Mon, 24 Feb 2020 20:31:58 +0100
-
-yq (3.1-1) eoan; urgency=medium
-
-  * Keeps yaml comments and formatting, can specify yaml tags when updating.
-  * Handles anchors
-  * Can print out matching paths and values when splatting
-  * JSON output works for all commands
-  * Yaml files with multiple documents are printed out as one JSON 
-    document per line.
-  * Deep splat (**) to match arbitrary paths
-  * Update scripts file format has changed to be more powerful
-  * Reading and splatting, matching results are printed once per line
-  * Bugfixing
-
- -- Roberto Mier Escandon <rmescandon@gmail.com>  Tue, 11 Feb 2020 22:18:24 +0100
-
-yq (2.2-1) bionic; urgency=medium
-
-  * Added Windows support for the "--inplace" command flag
-  * Prefix now supports arrays
-  * Add prefix command
-  * Bump Alpine version to 3.8
-  * Improved docker build process
-  * Lint fixes
-  * Build support for all linux architectures supported by gox
-
- -- Roberto Mier Escandon <rmescandon@gmail.com>  Sat, 19 Jan 2019 15:50:47 +0100
-
-yq (2.1-0) bionic; urgency=medium
-
-  * Ability to read multiple documents in a single file
-  * Ability to append list items instead of overwriting
-
- -- Roberto Mier Escandón <rmescandon@gmail.com>  Tue, 10 Jul 2018 14:02:42 +0200
-
-yq (2.0-0) bionic; urgency=medium
-
-  * Release 2.0.0
-
- -- Roberto Mier Escandón <rmescandon@gmail.com>  Wed, 20 Jun 2018 10:29:53 +0200
-
-yq (1.15-0) bionic; urgency=medium
-
-  * Release 1.15
-
- -- Roberto Mier Escandón <rmescandon@gmail.com>  Wed, 06 Jun 2018 11:32:03 +0200
-

debian/compat

@@ -1 +0,0 @@
-10

debian/control

@@ -1,22 +0,0 @@
-Source: yq
-Section: devel
-Priority: optional
-Maintainer: Roberto Mier Escandón <rmescandon@gmail.com>
-Build-Depends: debhelper (>=10),
-               golang-1.17-go,
-               pandoc,
-               rsync
-Standards-Version: 4.1.4
-Homepage: https://github.com/mikefarah/yq.git
-Vcs-Browser: https://github.com/mikefarah/yq.git
-Vcs-Git: https://github.com/mikefarah/yq.git
-XS-Go-Import-Path: github.com/mikefarah/yq
-XSBC-Original-Maintainer: Roberto Mier Escandón <rmescandon@gmail.com>
-
-Package: yq
-Architecture: any
-Depends: ${shlibs:Depends}, ${misc:Depends}
-Description: lightweight and portable command-line YAML processor
- .
- The aim of the project is to be the 
- [jq](https://github.com/stedolan/jq) or sed of yaml files.

debian/copyright

@@ -1,24 +0,0 @@
-Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
-Upstream-Name: yq
-Source: https://github.com/mikefarah/yq.git
-
-Files: *
-Copyright: 2017 Mike Farah
-License: Expat
-    Permission is hereby granted, free of charge, to any person obtaining a copy
-    of this software and associated documentation files (the "Software"), to deal
-    in the Software without restriction, including without limitation the rights
-    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-    copies of the Software, and to permit persons to whom the Software is
-    furnished to do so, subject to the following conditions:
-    .
-    The above copyright notice and this permission notice shall be included in all
-    copies or substantial portions of the Software.
-    .
-    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-    SOFTWARE.

debian/gbp.conf

@@ -1,2 +0,0 @@
-[DEFAULT]
-pristine-tar = True

debian/rules

@@ -1,74 +0,0 @@
-#!/usr/bin/make -f
-#
-# Copyright (C) 2018-2021 Roberto Mier Escandón <rmescandon@gmail.com>
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 3 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-PROJECT := yq
-OWNER := mikefarah
-REPO := github.com
-
-export DH_OPTIONS
-export DH_GOPKG := ${REPO}/${OWNER}/${PROJECT}
-export GOROOT := /usr/local/go
-export GOPATH := ${CURDIR}/_build
-export GOBIN := ${GOPATH}/bin
-export PATH := ${GOROOT}/bin:${GOBIN}:${PATH}
-export GOCACHE := /tmp/gocache
-export GOFLAGS := -mod=vendor
-export GO111MODULE := on
-
-SRCDIR := ${GOPATH}/src/${DH_GOPKG}
-DESTDIR := ${CURDIR}/debian/${PROJECT}
-BINDIR := /usr/bin
-MANDIR := /usr/share/man/man1/
-ASSETSDIR := /usr/share/${PROJECT}
-
-%:
-	dh $@
-
-override_dh_auto_build:
-	mkdir -p ${SRCDIR}
-	mkdir -p ${GOBIN}
-	# copy project to local srcdir to build from there
-	rsync -avz --progress --exclude=_build --exclude=debian --exclude=tmp. --exclude=go.mod --exclude=docs . $(SRCDIR)
-	# build go code
-	( \
-		cd ${SRCDIR} && \
-		go install -buildmode=pie ./... \
-	)
-
-	# build man page
-	( \
-		cd ${SRCDIR} && \
-		./scripts/generate-man-page-md.sh && \
-		./scripts/generate-man-page.sh \
-	)
-	
-override_dh_auto_test:
-ifeq (,$(filter nocheck,$(DEB_BUILD_OPTIONS)))
-	(cd ${SRCDIR} && go test -v ./...)
-endif
-
-override_dh_auto_install:
-	cp ${GOBIN}/yq ${DESTDIR}/${BINDIR}
-	cp -f ${SRCDIR}/LICENSE ${DESTDIR}/${ASSETSDIR}
-	chmod a+x ${DESTDIR}/${BINDIR}/yq
-
-	# man
-	mkdir -p "${DESTDIR}"/"${MANDIR}"
-	cp "${SRCDIR}"/yq.1 "${DESTDIR}"/"${MANDIR}" \
-
-override_dh_auto_clean:
-	dh_clean
-	rm -rf ${CURDIR}/_build

debian/source/format

@@ -1 +0,0 @@
-3.0 (native)

debian/yq.dirs

@@ -1,3 +0,0 @@
-usr/bin
-usr/share/yq
-usr/share/man/man1

examples/array.yaml

@@ -1,11 +1,2 @@
----
-- become: true
-  gather_facts: false
-  hosts: lalaland
-  name: "Apply smth"
-  roles:
-    - lala
-    - land
-  serial: 1
-- become: false
-  gather_facts: true
+- [cat, dog, frog, cow]
+- [apple, banana, grape, mango]

examples/data.lua

@@ -0,0 +1,9 @@
+return {
+	["country"] = "Australia"; -- this place
+	["cities"] = {
+		"Sydney",
+		"Melbourne",
+		"Brisbane",
+		"Perth",
+	};
+};

examples/data1.yaml

@@ -1,12 +1,8 @@
-# --------------------------------------------------
-# It's a test with comment
-# --------------------------------------------------
-test:
-  - name: a
-  - name: b
-  - name: c
-groups:
-  - name: a
-  - name: b
-  - name: c
-  - name: c
+# 001
+---
+abc:  # 001
+- 1   # one
+- 2   # two
+
+---
+def   # 002

examples/environment.yq

@@ -0,0 +1,5 @@
+#! yq
+.[] |(
+    ( select(kind == "scalar") | key + "='" + . + "'"),
+    ( select(kind == "seq") | key + "=(" + (map("'" + . + "'") | join(",")) + ")")
+)

examples/front-matter.yaml

@@ -1,6 +1,6 @@
 ---
 a: apple
-b: bannana
+b: banana
 ---
 hello there
 apples: great

examples/mike.xml

@@ -1,14 +1 @@
-<!-- before cat -->
-<cat>
-	<!-- in cat before -->
-	<x>3<!-- multi
-line comment
-for x --></x>
-	<y>
-		<!-- in y before -->
-		<d><!-- in d before -->4<!-- in d after --></d>
-		<!-- in y after -->
-	</y>
-	<!-- in_cat_after -->
-</cat>
-<!-- after cat -->
+<zoo><thing><frog>boing</frog></thing></zoo>

examples/sample.ini

@@ -0,0 +1,14 @@
+
+; This is a INI document
+
+[owner]
+name = "Tom Preston-Werner"
+dob = 1979-05-27T07:32:00-08:00
+
+[database]
+db_host = "localhost"
+db_port = 5432
+db_user = "postgres"
+db_password = ""
+db_name = "postgres"
+

examples/sample.toml

@@ -0,0 +1,26 @@
+
+
+# This is a TOML document
+
+title = "TOML Example"
+
+[owner]
+name = "Tom Preston-Werner"
+dob = 1979-05-27T07:32:00-08:00
+
+[database]
+enabled = true
+ports = [ 8000, 8001, 8002 ]
+data = [ ["delta", "phi"], [3.14] ]
+temp_targets = { cpu = 79.5, case = 72.0 }
+
+[servers]
+
+[servers.alpha]
+ip = "10.0.0.1"
+role = "frontend"
+
+[servers.beta]
+ip = "10.0.0.2"
+role = "backend"
+

examples/sample.yaml

@@ -1,11 +1,2 @@
-# Some doc
-
-a: true
-b:
-  c: 2
-  d: [3, 4, 5]
-  e:
-    - name: fred
-      value: 3
-    - name: sam
-      value: 4
+ # things
+a: apple

examples/sample_no_sections.ini

@@ -0,0 +1,8 @@
+
+; This is a INI document
+name = "Tom Preston-Werner"
+dob = 1979-05-27T07:32:00-08:00
+enabled = true
+ip = "10.0.0.1"
+role = "frontend"
+treads = 4

examples/sample_objects.csv

@@ -1,3 +1,3 @@
 name,numberOfCats,likesApples,height
-Gary,1,true,168.8
+,1,true,168.8
 Samantha's Rabbit,2,false,-188.8

examples/small.yaml

@@ -0,0 +1,4 @@
+---
+# comment
+# about things
+a: cat

github-action/entrypoint.sh

@@ -2,8 +2,11 @@
 set -e
 echo "::debug::\$cmd: $1"
 RESULT=$(eval "$1")
-RESULT="${RESULT//'%'/'%25'}"
-RESULT="${RESULT//$'\n'/'%0A'}"
-RESULT="${RESULT//$'\r'/'%0D'}"
 echo "::debug::\$RESULT: $RESULT"
-echo ::set-output name=result::"$RESULT"
+# updating from 
+# https://github.com/orgs/community/discussions/26288#discussioncomment-3876281
+# https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#setting-an-output-parameter
+delimiter=$(cat /proc/sys/kernel/random/uuid)
+echo "result<<${delimiter}" >> "${GITHUB_OUTPUT}"
+echo "${RESULT}" >> "${GITHUB_OUTPUT}"
+echo "${delimiter}" >> "${GITHUB_OUTPUT}"

go.mod

@@ -1,30 +1,35 @@
 module github.com/mikefarah/yq/v4
 
 require (
-	github.com/a8m/envsubst v1.3.0
-	github.com/alecthomas/participle/v2 v2.0.0-beta.5
-	github.com/alecthomas/repr v0.1.0
-	github.com/elliotchance/orderedmap v1.5.0
-	github.com/fatih/color v1.13.0
-	github.com/goccy/go-json v0.9.11
-	github.com/goccy/go-yaml v1.9.5
-	github.com/jinzhu/copier v0.3.5
-	github.com/magiconair/properties v1.8.6
+	github.com/a8m/envsubst v1.4.3
+	github.com/alecthomas/participle/v2 v2.1.4
+	github.com/alecthomas/repr v0.4.0
+	github.com/dimchansky/utfbom v1.1.1
+	github.com/elliotchance/orderedmap v1.8.0
+	github.com/fatih/color v1.18.0
+	github.com/go-ini/ini v1.67.0
+	github.com/goccy/go-json v0.10.5
+	github.com/goccy/go-yaml v1.18.0
+	github.com/jinzhu/copier v0.4.0
+	github.com/magiconair/properties v1.8.10
+	github.com/pelletier/go-toml/v2 v2.2.4
 	github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e
-	github.com/spf13/cobra v1.5.0
-	golang.org/x/net v0.0.0-20220708220712-1185a9018129
+	github.com/spf13/cobra v1.9.1
+	github.com/spf13/pflag v1.0.6
+	github.com/yuin/gopher-lua v1.1.1
+	go.yaml.in/yaml/v3 v3.0.4
+	golang.org/x/net v0.41.0
+	golang.org/x/text v0.27.0
 	gopkg.in/op/go-logging.v1 v1.0.0-20160211212156-b2cb9fa56473
-	gopkg.in/yaml.v3 v3.0.1
 )
 
 require (
-	github.com/inconshreveable/mousetrap v1.0.0 // indirect
-	github.com/mattn/go-colorable v0.1.12 // indirect
-	github.com/mattn/go-isatty v0.0.14 // indirect
-	github.com/spf13/pflag v1.0.5 // indirect
-	golang.org/x/sys v0.0.0-20220712014510-0a85c31ab51e // indirect
-	golang.org/x/text v0.3.7 // indirect
-	golang.org/x/xerrors v0.0.0-20220609144429-65e65417b02f // indirect
+	github.com/inconshreveable/mousetrap v1.1.0 // indirect
+	github.com/mattn/go-colorable v0.1.14 // indirect
+	github.com/mattn/go-isatty v0.0.20 // indirect
+	golang.org/x/sys v0.33.0 // indirect
 )
 
-go 1.19
+go 1.24
+
+toolchain go1.24.1

go.sum

@@ -1,81 +1,67 @@
-github.com/a8m/envsubst v1.3.0 h1:GmXKmVssap0YtlU3E230W98RWtWCyIZzjtf1apWWyAg=
-github.com/a8m/envsubst v1.3.0/go.mod h1:MVUTQNGQ3tsjOOtKCNd+fl8RzhsXcDvvAEzkhGtlsbY=
-github.com/alecthomas/assert/v2 v2.0.3 h1:WKqJODfOiQG0nEJKFKzDIG3E29CN2/4zR9XGJzKIkbg=
-github.com/alecthomas/participle/v2 v2.0.0-beta.5 h1:y6dsSYVb1G5eK6mgmy+BgI3Mw35a3WghArZ/Hbebrjo=
-github.com/alecthomas/participle/v2 v2.0.0-beta.5/go.mod h1:RC764t6n4L8D8ITAJv0qdokritYSNR3wV5cVwmIEaMM=
-github.com/alecthomas/repr v0.1.0 h1:ENn2e1+J3k09gyj2shc0dHr/yjaWSHRlrJ4DPMevDqE=
-github.com/alecthomas/repr v0.1.0/go.mod h1:2kn6fqh/zIyPLmm3ugklbEi5hg5wS435eygvNfaDQL8=
-github.com/cpuguy83/go-md2man/v2 v2.0.2/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
+github.com/a8m/envsubst v1.4.3 h1:kDF7paGK8QACWYaQo6KtyYBozY2jhQrTuNNuUxQkhJY=
+github.com/a8m/envsubst v1.4.3/go.mod h1:4jjHWQlZoaXPoLQUb7H2qT4iLkZDdmEQiOUogdUmqVU=
+github.com/alecthomas/assert/v2 v2.11.0 h1:2Q9r3ki8+JYXvGsDyBXwH3LcJ+WK5D0gc5E8vS6K3D0=
+github.com/alecthomas/assert/v2 v2.11.0/go.mod h1:Bze95FyfUr7x34QZrjL+XP+0qgp/zg8yS+TtBj1WA3k=
+github.com/alecthomas/participle/v2 v2.1.4 h1:W/H79S8Sat/krZ3el6sQMvMaahJ+XcM9WSI2naI7w2U=
+github.com/alecthomas/participle/v2 v2.1.4/go.mod h1:8tqVbpTX20Ru4NfYQgZf4mP18eXPTBViyMWiArNEgGI=
+github.com/alecthomas/repr v0.4.0 h1:GhI2A8MACjfegCPVq9f1FLvIBS+DrQ2KQBFZP1iFzXc=
+github.com/alecthomas/repr v0.4.0/go.mod h1:Fr0507jx4eOXV7AlPV6AVZLYrLIuIeSOWtW57eE/O/4=
+github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
 github.com/davecgh/go-spew v1.1.0 h1:ZDRjVQ15GmhC3fiQ8ni8+OwkZQO4DARzQgrnXU1Liz8=
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
-github.com/elliotchance/orderedmap v1.5.0 h1:1IsExUsjv5XNBD3ZdC7jkAAqLWOOKdbPTmkHx63OsBg=
-github.com/elliotchance/orderedmap v1.5.0/go.mod h1:wsDwEaX5jEoyhbs7x93zk2H/qv0zwuhg4inXhDkYqys=
-github.com/fatih/color v1.10.0/go.mod h1:ELkj/draVOlAH/xkhN6mQ50Qd0MPOk5AAr3maGEBuJM=
-github.com/fatih/color v1.13.0 h1:8LOYc1KYPPmyKMuN8QV2DNRWNbLo6LZ0iLs8+mlH53w=
-github.com/fatih/color v1.13.0/go.mod h1:kLAiJbzzSOZDVNGyDpeOxJ47H46qBXwg5ILebYFFOfk=
-github.com/go-playground/assert/v2 v2.0.1/go.mod h1:VDjEfimB/XKnb+ZQfWdccd7VUvScMdVu0Titje2rxJ4=
-github.com/go-playground/locales v0.13.0/go.mod h1:taPMhCMXrRLJO55olJkUXHZBHCxTMfnGwq/HNwmWNS8=
-github.com/go-playground/universal-translator v0.17.0/go.mod h1:UkSxE5sNxxRwHyU+Scu5vgOQjsIJAF8j9muTVoKLVtA=
-github.com/go-playground/validator/v10 v10.4.1/go.mod h1:nlOn6nFhuKACm19sB/8EGNn9GlaMV7XkbRSipzJ0Ii4=
-github.com/goccy/go-json v0.9.11 h1:/pAaQDLHEoCq/5FFmSKBswWmK6H0e8g4159Kc/X/nqk=
-github.com/goccy/go-json v0.9.11/go.mod h1:6MelG93GURQebXPDq3khkgXZkazVtN9CRI+MGFi0w8I=
-github.com/goccy/go-yaml v1.9.5 h1:Eh/+3uk9kLxG4koCX6lRMAPS1OaMSAi+FJcya0INdB0=
-github.com/goccy/go-yaml v1.9.5/go.mod h1:U/jl18uSupI5rdI2jmuCswEA2htH9eXfferR3KfscvA=
+github.com/dimchansky/utfbom v1.1.1 h1:vV6w1AhK4VMnhBno/TPVCoK9U/LP0PkLCS9tbxHdi/U=
+github.com/dimchansky/utfbom v1.1.1/go.mod h1:SxdoEBH5qIqFocHMyGOXVAybYJdr71b1Q/j0mACtrfE=
+github.com/elliotchance/orderedmap v1.8.0 h1:TrOREecvh3JbS+NCgwposXG5ZTFHtEsQiCGOhPElnMw=
+github.com/elliotchance/orderedmap v1.8.0/go.mod h1:wsDwEaX5jEoyhbs7x93zk2H/qv0zwuhg4inXhDkYqys=
+github.com/fatih/color v1.18.0 h1:S8gINlzdQ840/4pfAwic/ZE0djQEH3wM94VfqLTZcOM=
+github.com/fatih/color v1.18.0/go.mod h1:4FelSpRwEGDpQ12mAdzqdOukCy4u8WUtOY6lkT/6HfU=
+github.com/go-ini/ini v1.67.0 h1:z6ZrTEZqSWOTyH2FlglNbNgARyHG8oLW9gMELqKr06A=
+github.com/go-ini/ini v1.67.0/go.mod h1:ByCAeIL28uOIIG0E3PJtZPDL8WnHpFKFOtgjp+3Ies8=
+github.com/goccy/go-json v0.10.5 h1:Fq85nIqj+gXn/S5ahsiTlK3TmC85qgirsdTP/+DeaC4=
+github.com/goccy/go-json v0.10.5/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
+github.com/goccy/go-yaml v1.18.0 h1:8W7wMFS12Pcas7KU+VVkaiCng+kG8QiFeFwzFb+rwuw=
+github.com/goccy/go-yaml v1.18.0/go.mod h1:XBurs7gK8ATbW4ZPGKgcbrY1Br56PdM69F7LkFRi1kA=
 github.com/hexops/gotextdiff v1.0.3 h1:gitA9+qJrrTCsiCl7+kh75nPqQt1cx4ZkudSTLoUqJM=
-github.com/inconshreveable/mousetrap v1.0.0 h1:Z8tu5sraLXCXIcARxBp/8cbvlwVa7Z1NHg9XEKhtSvM=
-github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
-github.com/jinzhu/copier v0.3.5 h1:GlvfUwHk62RokgqVNvYsku0TATCF7bAHVwEXoBh3iJg=
-github.com/jinzhu/copier v0.3.5/go.mod h1:DfbEm0FYsaqBcKcFuvmOZb218JkPGtvSHsKg8S8hyyg=
-github.com/leodido/go-urn v1.2.0/go.mod h1:+8+nEpDfqqsY+g338gtMEUOtuK+4dEMhiQEgxpxOKII=
-github.com/magiconair/properties v1.8.6 h1:5ibWZ6iY0NctNGWo87LalDlEZ6R41TqbbDamhfG/Qzo=
-github.com/magiconair/properties v1.8.6/go.mod h1:y3VJvCyxH9uVvJTWEGAELF3aiYNyPKd5NZ3oSwXrF60=
-github.com/mattn/go-colorable v0.1.8/go.mod h1:u6P/XSegPjTcexA+o6vUJrdnUu04hMope9wVRipJSqc=
-github.com/mattn/go-colorable v0.1.9/go.mod h1:u6P/XSegPjTcexA+o6vUJrdnUu04hMope9wVRipJSqc=
-github.com/mattn/go-colorable v0.1.12 h1:jF+Du6AlPIjs2BiUiQlKOX0rt3SujHxPnksPKZbaA40=
-github.com/mattn/go-colorable v0.1.12/go.mod h1:u5H1YNBxpqRaxsYJYSkiCWKzEfiAb1Gb520KVy5xxl4=
-github.com/mattn/go-isatty v0.0.12/go.mod h1:cbi8OIDigv2wuxKPP5vlRcQ1OAZbq2CE4Kysco4FUpU=
-github.com/mattn/go-isatty v0.0.14 h1:yVuAays6BHfxijgZPzw+3Zlu5yQgKGP2/hcQbHb7S9Y=
-github.com/mattn/go-isatty v0.0.14/go.mod h1:7GGIvUiUoEMVVmxf/4nioHXj79iQHKdU27kJ6hsGG94=
+github.com/hexops/gotextdiff v1.0.3/go.mod h1:pSWU5MAI3yDq+fZBTazCSJysOMbxWL1BSow5/V2vxeg=
+github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
+github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
+github.com/jinzhu/copier v0.4.0 h1:w3ciUoD19shMCRargcpm0cm91ytaBhDvuRpz1ODO/U8=
+github.com/jinzhu/copier v0.4.0/go.mod h1:DfbEm0FYsaqBcKcFuvmOZb218JkPGtvSHsKg8S8hyyg=
+github.com/magiconair/properties v1.8.10 h1:s31yESBquKXCV9a/ScB3ESkOjUYYv+X0rg8SYxI99mE=
+github.com/magiconair/properties v1.8.10/go.mod h1:Dhd985XPs7jluiymwWYZ0G4Z61jb3vdS329zhj2hYo0=
+github.com/mattn/go-colorable v0.1.14 h1:9A9LHSqF/7dyVVX6g0U9cwm9pG3kP9gSzcuIPHPsaIE=
+github.com/mattn/go-colorable v0.1.14/go.mod h1:6LmQG8QLFO4G5z1gPvYEzlUgJ2wF+stgPZH1UqBm1s8=
+github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
+github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
+github.com/pelletier/go-toml/v2 v2.2.4 h1:mye9XuhQ6gvn5h28+VilKrrPoQVanw5PMw/TB0t5Ec4=
+github.com/pelletier/go-toml/v2 v2.2.4/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8t9Stt+h56mCY=
 github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e h1:aoZm08cpOy4WuID//EZDgcC4zIxODThtZNPirFr42+A=
 github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e/go.mod h1:pJLUxLENpZxwdsKMEsNbx1VGcRFpLqf3715MtcvvzbA=
 github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
 github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
-github.com/spf13/cobra v1.5.0 h1:X+jTBEBqF0bHN+9cSMgmfuvv2VHJ9ezmFNf9Y/XstYU=
-github.com/spf13/cobra v1.5.0/go.mod h1:dWXEIy2H428czQCjInthrTRUg7yKbok+2Qi/yBIJoUM=
-github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
-github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
+github.com/spf13/cobra v1.9.1 h1:CXSaggrXdbHK9CF+8ywj8Amf7PBRmPCOJugH954Nnlo=
+github.com/spf13/cobra v1.9.1/go.mod h1:nDyEzZ8ogv936Cinf6g1RU9MRY64Ir93oCnqb9wxYW0=
+github.com/spf13/pflag v1.0.6 h1:jFzHGLGAlb3ruxLB8MhbI6A8+AQX/2eW4qeyNZXNp2o=
+github.com/spf13/pflag v1.0.6/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
 github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
-github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
 github.com/stretchr/testify v1.7.0 h1:nwc3DEeHmmLAfoZucVR881uASk0Mfjw8xYJ99tb5CcY=
 github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
-golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
-golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
-golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
-golang.org/x/net v0.0.0-20220708220712-1185a9018129 h1:vucSRfWwTsoXro7P+3Cjlr6flUMtzCwzlvkxEQtHHB0=
-golang.org/x/net v0.0.0-20220708220712-1185a9018129/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
-golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
-golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20200116001909-b77594299b42/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.0.0-20210927094055-39ccf1dd6fa6/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.0.0-20220712014510-0a85c31ab51e h1:NHvCuwuS43lGnYhten69ZWqi2QOj/CiDNcKbVqwVoew=
-golang.org/x/sys v0.0.0-20220712014510-0a85c31ab51e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
-golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
-golang.org/x/text v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
-golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
-golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
-golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
-golang.org/x/xerrors v0.0.0-20220609144429-65e65417b02f h1:uF6paiQQebLeSXkrTqHqz0MXhXXS1KgF41eUdBNvxK0=
-golang.org/x/xerrors v0.0.0-20220609144429-65e65417b02f/go.mod h1:K8+ghG5WaK9qNqU5K3HdILfMLy1f3aNYFI/wnl100a8=
+github.com/yuin/gopher-lua v1.1.1 h1:kYKnWBjvbNP4XLT3+bPEwAXJx262OhaHDWDVOPjL46M=
+github.com/yuin/gopher-lua v1.1.1/go.mod h1:GBR0iDaNXjAgGg9zfCvksxSRnQx76gclCIb7kdAd1Pw=
+go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc=
+go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
+golang.org/x/net v0.41.0 h1:vBTly1HeNPEn3wtREYfy4GZ/NECgw2Cnl+nK6Nz3uvw=
+golang.org/x/net v0.41.0/go.mod h1:B/K4NNqkfmg07DQYrbwvSluqCJOOXwUjeb/5lOisjbA=
+golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.33.0 h1:q3i8TbbEz+JRD9ywIRlyRAQbM0qF7hu24q3teo2hbuw=
+golang.org/x/sys v0.33.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
+golang.org/x/text v0.27.0 h1:4fGWRpyh641NLlecmyl4LOe6yDdfaYNrGb2zdfo4JV4=
+golang.org/x/text v0.27.0/go.mod h1:1D28KMCvyooCX9hBiosv5Tz/+YLxj0j7XhWjpSUF7CU=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/op/go-logging.v1 v1.0.0-20160211212156-b2cb9fa56473 h1:6D+BvnJ/j6e222UW8s2qTSe3wGBtvo0MbVQG/c5k8RE=
 gopkg.in/op/go-logging.v1 v1.0.0-20160211212156-b2cb9fa56473/go.mod h1:N1eN2tsCx0Ydtgjl4cqmbRCsY4/+z4cYDeqwZTk6zog=
-gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
-gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
 gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
 gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
 gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

how-it-works.md

@@ -29,7 +29,7 @@ It pipes the current, lets call it 'root' context through the `lhs` expression o
 cat
 ```
 
-Sidenote: this node holds not only its value 'cat', but comments and metadata too, including path and parent information.
+Side note: this node holds not only its value 'cat', but comments and metadata too, including path and parent information.
 
 The `=` operator then pipes the 'root' context through the `rhs` expression of `.b` to return the node
 

pkg/yqlib/all_at_once_evaluator.go

@@ -2,16 +2,14 @@ package yqlib
 
 import (
 	"container/list"
-
-	yaml "gopkg.in/yaml.v3"
 )
 
 // A yaml expression evaluator that runs the expression once against all files/nodes in memory.
 type Evaluator interface {
-	EvaluateFiles(expression string, filenames []string, printer Printer, leadingContentPreProcessing bool, decoder Decoder) error
+	EvaluateFiles(expression string, filenames []string, printer Printer, decoder Decoder) error
 
 	// EvaluateNodes takes an expression and one or more yaml nodes, returning a list of matching candidate nodes
-	EvaluateNodes(expression string, nodes ...*yaml.Node) (*list.List, error)
+	EvaluateNodes(expression string, nodes ...*CandidateNode) (*list.List, error)
 
 	// EvaluateCandidateNodes takes an expression and list of candidate nodes, returning a list of matching candidate nodes
 	EvaluateCandidateNodes(expression string, inputCandidateNodes *list.List) (*list.List, error)
@@ -26,10 +24,10 @@ func NewAllAtOnceEvaluator() Evaluator {
 	return &allAtOnceEvaluator{treeNavigator: NewDataTreeNavigator()}
 }
 
-func (e *allAtOnceEvaluator) EvaluateNodes(expression string, nodes ...*yaml.Node) (*list.List, error) {
+func (e *allAtOnceEvaluator) EvaluateNodes(expression string, nodes ...*CandidateNode) (*list.List, error) {
 	inputCandidates := list.New()
 	for _, node := range nodes {
-		inputCandidates.PushBack(&CandidateNode{Node: node})
+		inputCandidates.PushBack(node)
 	}
 	return e.EvaluateCandidateNodes(expression, inputCandidates)
 }
@@ -46,21 +44,16 @@ func (e *allAtOnceEvaluator) EvaluateCandidateNodes(expression string, inputCand
 	return context.MatchingNodes, nil
 }
 
-func (e *allAtOnceEvaluator) EvaluateFiles(expression string, filenames []string, printer Printer, leadingContentPreProcessing bool, decoder Decoder) error {
+func (e *allAtOnceEvaluator) EvaluateFiles(expression string, filenames []string, printer Printer, decoder Decoder) error {
 	fileIndex := 0
-	firstFileLeadingContent := ""
 
 	var allDocuments = list.New()
 	for _, filename := range filenames {
-		reader, leadingContent, err := readStream(filename, fileIndex == 0 && leadingContentPreProcessing)
+		reader, err := readStream(filename)
 		if err != nil {
 			return err
 		}
 
-		if fileIndex == 0 {
-			firstFileLeadingContent = leadingContent
-		}
-
 		fileDocuments, err := readDocuments(reader, filename, fileIndex, decoder)
 		if err != nil {
 			return err
@@ -70,16 +63,8 @@ func (e *allAtOnceEvaluator) EvaluateFiles(expression string, filenames []string
 	}
 
 	if allDocuments.Len() == 0 {
-		candidateNode := &CandidateNode{
-			Document:       0,
-			Filename:       "",
-			Node:           &yaml.Node{Kind: yaml.DocumentNode, Content: []*yaml.Node{{Tag: "!!null", Kind: yaml.ScalarNode}}},
-			FileIndex:      0,
-			LeadingContent: firstFileLeadingContent,
-		}
+		candidateNode := createScalarNode(nil, "")
 		allDocuments.PushBack(candidateNode)
-	} else {
-		allDocuments.Front().Value.(*CandidateNode).LeadingContent = firstFileLeadingContent
 	}
 
 	matches, err := e.EvaluateCandidateNodes(expression, allDocuments)

pkg/yqlib/all_at_once_evaluator_test.go

@@ -1,6 +1,8 @@
 package yqlib
 
 import (
+	"bufio"
+	"strings"
 	"testing"
 
 	"github.com/mikefarah/yq/v4/test"
@@ -18,7 +20,7 @@ var evaluateNodesScenario = []expressionScenario{
 		document:   `a: hello`,
 		expression: `.`,
 		expected: []string{
-			"D0, P[], (doc)::a: hello\n",
+			"D0, P[], (!!map)::a: hello\n",
 		},
 	},
 	{
@@ -32,9 +34,23 @@ var evaluateNodesScenario = []expressionScenario{
 
 func TestAllAtOnceEvaluateNodes(t *testing.T) {
 	var evaluator = NewAllAtOnceEvaluator()
+	// logging.SetLevel(logging.DEBUG, "")
 	for _, tt := range evaluateNodesScenario {
-		node := test.ParseData(tt.document)
-		list, _ := evaluator.EvaluateNodes(tt.expression, &node)
+		decoder := NewYamlDecoder(ConfiguredYamlPreferences)
+		reader := bufio.NewReader(strings.NewReader(tt.document))
+		err := decoder.Init(reader)
+		if err != nil {
+			t.Error(err)
+			return
+		}
+		candidateNode, errorReading := decoder.Decode()
+
+		if errorReading != nil {
+			t.Error(errorReading)
+			return
+		}
+
+		list, _ := evaluator.EvaluateNodes(tt.expression, candidateNode)
 		test.AssertResultComplex(t, tt.expected, resultsToString(t, list))
 	}
 }

pkg/yqlib/candidate_node.go

@@ -3,51 +3,215 @@ package yqlib
 import (
 	"container/list"
 	"fmt"
+	"strconv"
 	"strings"
-
-	"github.com/jinzhu/copier"
-	yaml "gopkg.in/yaml.v3"
 )
 
+type Kind uint32
+
+const (
+	SequenceNode Kind = 1 << iota
+	MappingNode
+	ScalarNode
+	AliasNode
+)
+
+type Style uint32
+
+const (
+	TaggedStyle Style = 1 << iota
+	DoubleQuotedStyle
+	SingleQuotedStyle
+	LiteralStyle
+	FoldedStyle
+	FlowStyle
+)
+
+func createStringScalarNode(stringValue string) *CandidateNode {
+	var node = &CandidateNode{Kind: ScalarNode}
+	node.Value = stringValue
+	node.Tag = "!!str"
+	return node
+}
+
+func createScalarNode(value interface{}, stringValue string) *CandidateNode {
+	var node = &CandidateNode{Kind: ScalarNode}
+	node.Value = stringValue
+
+	switch value.(type) {
+	case float32, float64:
+		node.Tag = "!!float"
+	case int, int64, int32:
+		node.Tag = "!!int"
+	case bool:
+		node.Tag = "!!bool"
+	case string:
+		node.Tag = "!!str"
+	case nil:
+		node.Tag = "!!null"
+	}
+	return node
+}
+
+type NodeInfo struct {
+	Kind        string      `yaml:"kind"`
+	Style       string      `yaml:"style,omitempty"`
+	Anchor      string      `yaml:"anchor,omitempty"`
+	Tag         string      `yaml:"tag,omitempty"`
+	HeadComment string      `yaml:"headComment,omitempty"`
+	LineComment string      `yaml:"lineComment,omitempty"`
+	FootComment string      `yaml:"footComment,omitempty"`
+	Value       string      `yaml:"value,omitempty"`
+	Line        int         `yaml:"line,omitempty"`
+	Column      int         `yaml:"column,omitempty"`
+	Content     []*NodeInfo `yaml:"content,omitempty"`
+}
+
 type CandidateNode struct {
-	Node   *yaml.Node     // the actual node
+	Kind  Kind
+	Style Style
+
+	Tag     string
+	Value   string
+	Anchor  string
+	Alias   *CandidateNode
+	Content []*CandidateNode
+
+	HeadComment string
+	LineComment string
+	FootComment string
+
 	Parent *CandidateNode // parent node
-	Key    *yaml.Node     // node key, if this is a value from a map (or index in an array)
+	Key    *CandidateNode // node key, if this is a value from a map (or index in an array)
 
-	LeadingContent  string
-	TrailingContent string
+	LeadingContent string
 
-	Path      []interface{} /// the path we took to get to this node
-	Document  uint          // the document index of this node
-	Filename  string
-	FileIndex int
+	document uint // the document index of this node
+	filename string
+
+	Line   int
+	Column int
+
+	fileIndex int
 	// when performing op against all nodes given, this will treat all the nodes as one
 	// (e.g. top level cross document merge). This property does not propagate to child nodes.
 	EvaluateTogether bool
 	IsMapKey         bool
 }
 
+func (n *CandidateNode) CreateChild() *CandidateNode {
+	return &CandidateNode{
+		Parent: n,
+	}
+}
+
+func (n *CandidateNode) SetDocument(idx uint) {
+	n.document = idx
+}
+
+func (n *CandidateNode) GetDocument() uint {
+	// defer to parent
+	if n.Parent != nil {
+		return n.Parent.GetDocument()
+	}
+	return n.document
+}
+
+func (n *CandidateNode) SetFilename(name string) {
+	n.filename = name
+}
+
+func (n *CandidateNode) GetFilename() string {
+	if n.Parent != nil {
+		return n.Parent.GetFilename()
+	}
+	return n.filename
+}
+
+func (n *CandidateNode) SetFileIndex(idx int) {
+	n.fileIndex = idx
+}
+
+func (n *CandidateNode) GetFileIndex() int {
+	if n.Parent != nil {
+		return n.Parent.GetFileIndex()
+	}
+	return n.fileIndex
+}
+
 func (n *CandidateNode) GetKey() string {
 	keyPrefix := ""
 	if n.IsMapKey {
-		keyPrefix = "key-"
+		keyPrefix = fmt.Sprintf("key-%v-", n.Value)
 	}
-	return fmt.Sprintf("%v%v - %v", keyPrefix, n.Document, n.Path)
+	key := ""
+	if n.Key != nil {
+		key = n.Key.Value
+	}
+	return fmt.Sprintf("%v%v - %v", keyPrefix, n.GetDocument(), key)
 }
 
-func (n *CandidateNode) GetNiceTag() string {
-	return unwrapDoc(n.Node).Tag
+func (n *CandidateNode) getParsedKey() interface{} {
+	if n.IsMapKey {
+		return n.Value
+	}
+	if n.Key == nil {
+		return nil
+	}
+	if n.Key.Tag == "!!str" {
+		return n.Key.Value
+	}
+	index, err := parseInt(n.Key.Value)
+	if err != nil {
+		return n.Key.Value
+	}
+	return index
+
+}
+
+func (n *CandidateNode) FilterMapContentByKey(keyPredicate func(*CandidateNode) bool) []*CandidateNode {
+	var result []*CandidateNode
+	for index := 0; index < len(n.Content); index = index + 2 {
+		keyNode := n.Content[index]
+		valueNode := n.Content[index+1]
+		if keyPredicate(keyNode) {
+			result = append(result, keyNode, valueNode)
+		}
+	}
+	return result
+}
+
+func (n *CandidateNode) GetPath() []interface{} {
+	key := n.getParsedKey()
+	if n.Parent != nil && key != nil {
+		return append(n.Parent.GetPath(), key)
+	}
+
+	if key != nil {
+		return []interface{}{key}
+	}
+	return make([]interface{}, 0)
 }
 
 func (n *CandidateNode) GetNicePath() string {
-	if n.Path != nil && len(n.Path) >= 0 {
-		pathStr := make([]string, len(n.Path))
-		for i, v := range n.Path {
-			pathStr[i] = fmt.Sprintf("%v", v)
+	var sb strings.Builder
+	path := n.GetPath()
+	for i, element := range path {
+		elementStr := fmt.Sprintf("%v", element)
+		switch element.(type) {
+		case int:
+			sb.WriteString("[" + elementStr + "]")
+		default:
+			if i == 0 {
+				sb.WriteString(elementStr)
+			} else if strings.ContainsRune(elementStr, '.') {
+				sb.WriteString("[" + elementStr + "]")
+			} else {
+				sb.WriteString("." + elementStr)
+			}
 		}
-		return strings.Join(pathStr, ".")
 	}
-	return ""
+	return sb.String()
 }
 
 func (n *CandidateNode) AsList() *list.List {
@@ -56,134 +220,321 @@ func (n *CandidateNode) AsList() *list.List {
 	return elMap
 }
 
-func (n *CandidateNode) CreateChildInMap(key *yaml.Node, node *yaml.Node) *CandidateNode {
-	var value interface{}
-	if key != nil {
-		value = key.Value
+func (n *CandidateNode) SetParent(parent *CandidateNode) {
+	n.Parent = parent
+}
+
+type ValueVisitor func(*CandidateNode) error
+
+func (n *CandidateNode) VisitValues(visitor ValueVisitor) error {
+	switch n.Kind {
+	case MappingNode:
+		for i := 1; i < len(n.Content); i = i + 2 {
+			if err := visitor(n.Content[i]); err != nil {
+				return err
+			}
+		}
+	case SequenceNode:
+		for i := 0; i < len(n.Content); i = i + 1 {
+			if err := visitor(n.Content[i]); err != nil {
+				return err
+			}
+		}
 	}
-	return &CandidateNode{
-		Node:      node,
-		Path:      n.createChildPath(value),
-		Parent:    n,
-		Key:       key,
-		Document:  n.Document,
-		Filename:  n.Filename,
-		FileIndex: n.FileIndex,
+	return nil
+}
+
+func (n *CandidateNode) CanVisitValues() bool {
+	return n.Kind == MappingNode || n.Kind == SequenceNode
+}
+
+func (n *CandidateNode) AddKeyValueChild(rawKey *CandidateNode, rawValue *CandidateNode) (*CandidateNode, *CandidateNode) {
+	key := rawKey.Copy()
+	key.SetParent(n)
+	key.IsMapKey = true
+
+	value := rawValue.Copy()
+	value.SetParent(n)
+	value.IsMapKey = false // force this, incase we are creating a value from a key
+	value.Key = key
+
+	n.Content = append(n.Content, key, value)
+	return key, value
+}
+
+func (n *CandidateNode) AddChild(rawChild *CandidateNode) {
+	value := rawChild.Copy()
+	value.SetParent(n)
+	value.IsMapKey = false
+
+	index := len(n.Content)
+	keyNode := createScalarNode(index, fmt.Sprintf("%v", index))
+	keyNode.SetParent(n)
+	value.Key = keyNode
+
+	n.Content = append(n.Content, value)
+}
+
+func (n *CandidateNode) AddChildren(children []*CandidateNode) {
+	if n.Kind == MappingNode {
+		for i := 0; i < len(children); i += 2 {
+			key := children[i]
+			value := children[i+1]
+			n.AddKeyValueChild(key, value)
+		}
+
+	} else {
+		for _, rawChild := range children {
+			n.AddChild(rawChild)
+		}
 	}
 }
 
-func (n *CandidateNode) CreateChildInArray(index int, node *yaml.Node) *CandidateNode {
-	return &CandidateNode{
-		Node:      node,
-		Path:      n.createChildPath(index),
-		Parent:    n,
-		Key:       &yaml.Node{Kind: yaml.ScalarNode, Value: fmt.Sprintf("%v", index), Tag: "!!int"},
-		Document:  n.Document,
-		Filename:  n.Filename,
-		FileIndex: n.FileIndex,
+func (n *CandidateNode) GetValueRep() (interface{}, error) {
+	log.Debugf("GetValueRep for %v value: %v", n.GetNicePath(), n.Value)
+	realTag := n.guessTagFromCustomType()
+
+	switch realTag {
+	case "!!int":
+		_, val, err := parseInt64(n.Value)
+		return val, err
+	case "!!float":
+		// need to test this
+		return strconv.ParseFloat(n.Value, 64)
+	case "!!bool":
+		return isTruthyNode(n), nil
+	case "!!null":
+		return nil, nil
 	}
+
+	return n.Value, nil
 }
 
-func (n *CandidateNode) CreateReplacement(node *yaml.Node) *CandidateNode {
-	return &CandidateNode{
-		Node:      node,
-		Path:      n.createChildPath(nil),
-		Parent:    n.Parent,
-		Key:       n.Key,
-		IsMapKey:  n.IsMapKey,
-		Document:  n.Document,
-		Filename:  n.Filename,
-		FileIndex: n.FileIndex,
+func (n *CandidateNode) guessTagFromCustomType() string {
+	if strings.HasPrefix(n.Tag, "!!") {
+		return n.Tag
+	} else if n.Value == "" {
+		log.Debug("guessTagFromCustomType: node has no value to guess the type with")
+		return n.Tag
 	}
+	dataBucket, errorReading := parseSnippet(n.Value)
+
+	if errorReading != nil {
+		log.Debug("guessTagFromCustomType: could not guess underlying tag type %v", errorReading)
+		return n.Tag
+	}
+	guessedTag := dataBucket.Tag
+	log.Info("im guessing the tag %v is a %v", n.Tag, guessedTag)
+	return guessedTag
 }
 
-func (n *CandidateNode) CreateReplacementWithDocWrappers(node *yaml.Node) *CandidateNode {
-	replacement := n.CreateReplacement(node)
+func (n *CandidateNode) CreateReplacement(kind Kind, tag string, value string) *CandidateNode {
+	node := &CandidateNode{
+		Kind:  kind,
+		Tag:   tag,
+		Value: value,
+	}
+	return n.CopyAsReplacement(node)
+}
+
+func (n *CandidateNode) CopyAsReplacement(replacement *CandidateNode) *CandidateNode {
+	newCopy := replacement.Copy()
+	newCopy.Parent = n.Parent
+
+	if n.IsMapKey {
+		newCopy.Key = n
+	} else {
+		newCopy.Key = n.Key
+	}
+
+	return newCopy
+}
+
+func (n *CandidateNode) CreateReplacementWithComments(kind Kind, tag string, style Style) *CandidateNode {
+	replacement := n.CreateReplacement(kind, tag, "")
 	replacement.LeadingContent = n.LeadingContent
-	replacement.TrailingContent = n.TrailingContent
+	replacement.HeadComment = n.HeadComment
+	replacement.LineComment = n.LineComment
+	replacement.FootComment = n.FootComment
+	replacement.Style = style
 	return replacement
 }
 
-func (n *CandidateNode) createChildPath(path interface{}) []interface{} {
-	if path == nil {
-		newPath := make([]interface{}, len(n.Path))
-		copy(newPath, n.Path)
-		return newPath
-	}
-
-	//don't use append as they may actually modify the path of the orignal node!
-	newPath := make([]interface{}, len(n.Path)+1)
-	copy(newPath, n.Path)
-	newPath[len(n.Path)] = path
-	return newPath
+func (n *CandidateNode) Copy() *CandidateNode {
+	return n.doCopy(true)
 }
 
-func (n *CandidateNode) Copy() (*CandidateNode, error) {
-	clone := &CandidateNode{}
-	err := copier.Copy(clone, n)
-	if err != nil {
-		return nil, err
+func (n *CandidateNode) CopyWithoutContent() *CandidateNode {
+	return n.doCopy(false)
+}
+
+func (n *CandidateNode) doCopy(cloneContent bool) *CandidateNode {
+	var content []*CandidateNode
+
+	var copyKey *CandidateNode
+	if n.Key != nil {
+		copyKey = n.Key.Copy()
 	}
-	clone.Node = deepClone(n.Node)
-	return clone, nil
+
+	clone := &CandidateNode{
+		Kind:  n.Kind,
+		Style: n.Style,
+
+		Tag:    n.Tag,
+		Value:  n.Value,
+		Anchor: n.Anchor,
+
+		// ok not to clone this,
+		// as its a reference to somewhere else.
+		Alias:   n.Alias,
+		Content: content,
+
+		HeadComment: n.HeadComment,
+		LineComment: n.LineComment,
+		FootComment: n.FootComment,
+
+		Parent: n.Parent,
+		Key:    copyKey,
+
+		LeadingContent: n.LeadingContent,
+
+		document:  n.document,
+		filename:  n.filename,
+		fileIndex: n.fileIndex,
+
+		Line:   n.Line,
+		Column: n.Column,
+
+		EvaluateTogether: n.EvaluateTogether,
+		IsMapKey:         n.IsMapKey,
+	}
+
+	if cloneContent {
+		clone.AddChildren(n.Content)
+	}
+
+	return clone
 }
 
 // updates this candidate from the given candidate node
 func (n *CandidateNode) UpdateFrom(other *CandidateNode, prefs assignPreferences) {
-
+	if n == other {
+		log.Debugf("UpdateFrom, no need to update from myself.")
+		return
+	}
 	// if this is an empty map or empty array, use the style of other node.
-	if (n.Node.Kind != yaml.ScalarNode && len(n.Node.Content) == 0) ||
+	if (n.Kind != ScalarNode && len(n.Content) == 0) ||
 		// if the tag has changed (e.g. from str to bool)
-		(guessTagFromCustomType(n.Node) != guessTagFromCustomType(other.Node)) {
-		n.Node.Style = other.Node.Style
+		(n.guessTagFromCustomType() != other.guessTagFromCustomType()) {
+		n.Style = other.Style
 	}
 
-	n.Node.Content = deepCloneContent(other.Node.Content)
-	n.Node.Kind = other.Node.Kind
-	n.Node.Value = other.Node.Value
+	n.Content = make([]*CandidateNode, 0)
+	n.Kind = other.Kind
+	n.AddChildren(other.Content)
+
+	n.Value = other.Value
 
 	n.UpdateAttributesFrom(other, prefs)
 
 }
 
 func (n *CandidateNode) UpdateAttributesFrom(other *CandidateNode, prefs assignPreferences) {
-	log.Debug("UpdateAttributesFrom: n: %v other: %v", n.GetKey(), other.GetKey())
-	if n.Node.Kind != other.Node.Kind {
+	log.Debug("UpdateAttributesFrom: n: %v other: %v", NodeToString(n), NodeToString(other))
+	if n.Kind != other.Kind {
 		// clear out the contents when switching to a different type
 		// e.g. map to array
-		n.Node.Content = make([]*yaml.Node, 0)
-		n.Node.Value = ""
+		n.Content = make([]*CandidateNode, 0)
+		n.Value = ""
 	}
-	n.Node.Kind = other.Node.Kind
+	n.Kind = other.Kind
 
 	// don't clobber custom tags...
-	if prefs.ClobberCustomTags || strings.HasPrefix(n.Node.Tag, "!!") || n.Node.Tag == "" {
-		n.Node.Tag = other.Node.Tag
+	if prefs.ClobberCustomTags || strings.HasPrefix(n.Tag, "!!") || n.Tag == "" {
+		n.Tag = other.Tag
 	}
 
-	n.Node.Alias = other.Node.Alias
+	n.Alias = other.Alias
 
 	if !prefs.DontOverWriteAnchor {
-		n.Node.Anchor = other.Node.Anchor
+		n.Anchor = other.Anchor
 	}
 
 	// merge will pickup the style of the new thing
 	// when autocreating nodes
 
-	if n.Node.Style == 0 {
-		n.Node.Style = other.Node.Style
+	if n.Style == 0 {
+		n.Style = other.Style
 	}
 
-	if other.Node.FootComment != "" {
-		n.Node.FootComment = other.Node.FootComment
+	if other.FootComment != "" {
+		n.FootComment = other.FootComment
 	}
-	if other.TrailingContent != "" {
-		n.TrailingContent = other.TrailingContent
+	if other.HeadComment != "" {
+		n.HeadComment = other.HeadComment
 	}
-	if other.Node.HeadComment != "" {
-		n.Node.HeadComment = other.Node.HeadComment
-	}
-	if other.Node.LineComment != "" {
-		n.Node.LineComment = other.Node.LineComment
+	if other.LineComment != "" {
+		n.LineComment = other.LineComment
 	}
 }
+
+func (n *CandidateNode) ConvertToNodeInfo() *NodeInfo {
+	info := &NodeInfo{
+		Kind:        kindToString(n.Kind),
+		Style:       styleToString(n.Style),
+		Anchor:      n.Anchor,
+		Tag:         n.Tag,
+		HeadComment: n.HeadComment,
+		LineComment: n.LineComment,
+		FootComment: n.FootComment,
+		Value:       n.Value,
+		Line:        n.Line,
+		Column:      n.Column,
+	}
+	if len(n.Content) > 0 {
+		info.Content = make([]*NodeInfo, len(n.Content))
+		for i, child := range n.Content {
+			info.Content[i] = child.ConvertToNodeInfo()
+		}
+	}
+	return info
+}
+
+// Helper functions to convert Kind and Style to string for NodeInfo
+func kindToString(k Kind) string {
+	switch k {
+	case SequenceNode:
+		return "SequenceNode"
+	case MappingNode:
+		return "MappingNode"
+	case ScalarNode:
+		return "ScalarNode"
+	case AliasNode:
+		return "AliasNode"
+	default:
+		return "Unknown"
+	}
+}
+
+func styleToString(s Style) string {
+	var styles []string
+	if s&TaggedStyle != 0 {
+		styles = append(styles, "TaggedStyle")
+	}
+	if s&DoubleQuotedStyle != 0 {
+		styles = append(styles, "DoubleQuotedStyle")
+	}
+	if s&SingleQuotedStyle != 0 {
+		styles = append(styles, "SingleQuotedStyle")
+	}
+	if s&LiteralStyle != 0 {
+		styles = append(styles, "LiteralStyle")
+	}
+	if s&FoldedStyle != 0 {
+		styles = append(styles, "FoldedStyle")
+	}
+	if s&FlowStyle != 0 {
+		styles = append(styles, "FlowStyle")
+	}
+	return strings.Join(styles, ",")
+}

pkg/yqlib/candidate_node_goccy_yaml.go

@@ -0,0 +1,210 @@
+package yqlib
+
+import (
+	"fmt"
+	"strings"
+
+	yaml "github.com/goccy/go-yaml"
+	"github.com/goccy/go-yaml/ast"
+	goccyToken "github.com/goccy/go-yaml/token"
+)
+
+func (o *CandidateNode) goccyDecodeIntoChild(childNode ast.Node, cm yaml.CommentMap, anchorMap map[string]*CandidateNode) (*CandidateNode, error) {
+	newChild := o.CreateChild()
+
+	err := newChild.UnmarshalGoccyYAML(childNode, cm, anchorMap)
+	return newChild, err
+}
+
+func (o *CandidateNode) UnmarshalGoccyYAML(node ast.Node, cm yaml.CommentMap, anchorMap map[string]*CandidateNode) error {
+	log.Debugf("UnmarshalYAML %v", node)
+	log.Debugf("UnmarshalYAML %v", node.Type().String())
+	log.Debugf("UnmarshalYAML Node Value: %v", node.String())
+	log.Debugf("UnmarshalYAML Node GetComment: %v", node.GetComment())
+
+	if node.GetComment() != nil {
+		commentMapComments := cm[node.GetPath()]
+		for _, comment := range node.GetComment().Comments {
+			// need to use the comment map to find the position :/
+			log.Debugf("%v has a comment of [%v]", node.GetPath(), comment.Token.Value)
+			for _, commentMapComment := range commentMapComments {
+				commentMapValue := strings.Join(commentMapComment.Texts, "\n")
+				if commentMapValue == comment.Token.Value {
+					log.Debug("found a matching entry in comment map")
+					// we found the comment in the comment map,
+					// now we can process the position
+					switch commentMapComment.Position {
+					case yaml.CommentHeadPosition:
+						o.HeadComment = comment.String()
+						log.Debug("its a head comment %v", comment.String())
+					case yaml.CommentLinePosition:
+						o.LineComment = comment.String()
+						log.Debug("its a line comment %v", comment.String())
+					case yaml.CommentFootPosition:
+						o.FootComment = comment.String()
+						log.Debug("its a foot comment %v", comment.String())
+					}
+				}
+			}
+
+		}
+	}
+
+	o.Value = node.String()
+	o.Line = node.GetToken().Position.Line
+	o.Column = node.GetToken().Position.Column
+
+	switch node.Type() {
+	case ast.IntegerType:
+		o.Kind = ScalarNode
+		o.Tag = "!!int"
+	case ast.FloatType:
+		o.Kind = ScalarNode
+		o.Tag = "!!float"
+	case ast.BoolType:
+		o.Kind = ScalarNode
+		o.Tag = "!!bool"
+	case ast.NullType:
+		log.Debugf("its a null type with value %v", node.GetToken().Value)
+		o.Kind = ScalarNode
+		o.Tag = "!!null"
+		o.Value = node.GetToken().Value
+		if node.GetToken().Type == goccyToken.ImplicitNullType {
+			o.Value = ""
+		}
+	case ast.StringType:
+		o.Kind = ScalarNode
+		o.Tag = "!!str"
+		switch node.GetToken().Type {
+		case goccyToken.SingleQuoteType:
+			o.Style = SingleQuotedStyle
+		case goccyToken.DoubleQuoteType:
+			o.Style = DoubleQuotedStyle
+		}
+		o.Value = node.(*ast.StringNode).Value
+		log.Debugf("string value %v", node.(*ast.StringNode).Value)
+	case ast.LiteralType:
+		o.Kind = ScalarNode
+		o.Tag = "!!str"
+		o.Style = LiteralStyle
+		astLiteral := node.(*ast.LiteralNode)
+		log.Debugf("astLiteral.Start.Type %v", astLiteral.Start.Type)
+		if astLiteral.Start.Type == goccyToken.FoldedType {
+			log.Debugf("folded Type %v", astLiteral.Start.Type)
+			o.Style = FoldedStyle
+		}
+		log.Debug("start value: %v ", node.(*ast.LiteralNode).Start.Value)
+		log.Debug("start value: %v ", node.(*ast.LiteralNode).Start.Type)
+		// TODO: here I could put the original value with line breaks
+		// to solve the multiline > problem
+		o.Value = astLiteral.Value.Value
+	case ast.TagType:
+		if err := o.UnmarshalGoccyYAML(node.(*ast.TagNode).Value, cm, anchorMap); err != nil {
+			return err
+		}
+		o.Tag = node.(*ast.TagNode).Start.Value
+	case ast.MappingType:
+		log.Debugf("UnmarshalYAML -  a mapping node")
+		o.Kind = MappingNode
+		o.Tag = "!!map"
+
+		mappingNode := node.(*ast.MappingNode)
+		if mappingNode.IsFlowStyle {
+			o.Style = FlowStyle
+		}
+		for _, mappingValueNode := range mappingNode.Values {
+			err := o.goccyProcessMappingValueNode(mappingValueNode, cm, anchorMap)
+			if err != nil {
+				return err
+			}
+		}
+		if mappingNode.FootComment != nil {
+			log.Debugf("mapping node has a foot comment of: %v", mappingNode.FootComment)
+			o.FootComment = mappingNode.FootComment.String()
+		}
+	case ast.MappingValueType:
+		log.Debugf("UnmarshalYAML -  a mapping node")
+		o.Kind = MappingNode
+		o.Tag = "!!map"
+		mappingValueNode := node.(*ast.MappingValueNode)
+		err := o.goccyProcessMappingValueNode(mappingValueNode, cm, anchorMap)
+		if err != nil {
+			return err
+		}
+	case ast.SequenceType:
+		log.Debugf("UnmarshalYAML -  a sequence node")
+		o.Kind = SequenceNode
+		o.Tag = "!!seq"
+		sequenceNode := node.(*ast.SequenceNode)
+		if sequenceNode.IsFlowStyle {
+			o.Style = FlowStyle
+		}
+		astSeq := sequenceNode.Values
+		o.Content = make([]*CandidateNode, len(astSeq))
+		for i := 0; i < len(astSeq); i++ {
+			keyNode := o.CreateChild()
+			keyNode.IsMapKey = true
+			keyNode.Tag = "!!int"
+			keyNode.Kind = ScalarNode
+			keyNode.Value = fmt.Sprintf("%v", i)
+
+			valueNode, err := o.goccyDecodeIntoChild(astSeq[i], cm, anchorMap)
+			if err != nil {
+				return err
+			}
+
+			valueNode.Key = keyNode
+			o.Content[i] = valueNode
+		}
+	case ast.AnchorType:
+		log.Debugf("UnmarshalYAML -  an anchor node")
+		anchorNode := node.(*ast.AnchorNode)
+		err := o.UnmarshalGoccyYAML(anchorNode.Value, cm, anchorMap)
+		if err != nil {
+			return err
+		}
+		o.Anchor = anchorNode.Name.String()
+		anchorMap[o.Anchor] = o
+
+	case ast.AliasType:
+		log.Debugf("UnmarshalYAML -  an alias node")
+		aliasNode := node.(*ast.AliasNode)
+		o.Kind = AliasNode
+		o.Value = aliasNode.Value.String()
+		o.Alias = anchorMap[o.Value]
+
+	case ast.MergeKeyType:
+		log.Debugf("UnmarshalYAML -  a merge key")
+		o.Kind = ScalarNode
+		o.Tag = "!!merge" // note - I should be able to get rid of this.
+		o.Value = "<<"
+
+	default:
+		log.Debugf("UnmarshalYAML -  no idea of the type!!\n%v: %v", node.Type(), node.String())
+	}
+	log.Debugf("KIND: %v", o.Kind)
+	return nil
+}
+
+func (o *CandidateNode) goccyProcessMappingValueNode(mappingEntry *ast.MappingValueNode, cm yaml.CommentMap, anchorMap map[string]*CandidateNode) error {
+	log.Debug("UnmarshalYAML MAP KEY entry %v", mappingEntry.Key)
+
+	// AddKeyValueFirst because it clones the nodes, and we want to have the real refs when Unmarshalling
+	// particularly for the anchorMap
+	keyNode, valueNode := o.AddKeyValueChild(&CandidateNode{}, &CandidateNode{})
+
+	if err := keyNode.UnmarshalGoccyYAML(mappingEntry.Key, cm, anchorMap); err != nil {
+		return err
+	}
+
+	log.Debug("UnmarshalYAML MAP VALUE entry %v", mappingEntry.Value)
+	if err := valueNode.UnmarshalGoccyYAML(mappingEntry.Value, cm, anchorMap); err != nil {
+		return err
+	}
+
+	if mappingEntry.FootComment != nil {
+		valueNode.FootComment = mappingEntry.FootComment.String()
+	}
+
+	return nil
+}

pkg/yqlib/candidate_node_test.go

@@ -0,0 +1,206 @@
+package yqlib
+
+import (
+	"fmt"
+	"testing"
+
+	"github.com/mikefarah/yq/v4/test"
+)
+
+type valueRepScenario struct {
+	input    string
+	tag      string
+	expected interface{}
+}
+
+var valueRepScenarios = []valueRepScenario{
+	{
+		input:    `"cat"`,
+		expected: `"cat"`,
+	},
+	{
+		input:    `3`,
+		expected: int64(3),
+	},
+	{
+		input:    `3.1`,
+		expected: float64(3.1),
+	},
+	{
+		input:    `true`,
+		expected: true,
+	},
+	{
+		input:    `y`,
+		tag:      "!!bool",
+		expected: true,
+	},
+	{
+		tag:      "!!null",
+		expected: nil,
+	},
+}
+
+func TestCandidateNodeGetValueRepScenarios(t *testing.T) {
+	for _, tt := range valueRepScenarios {
+		node := CandidateNode{Value: tt.input, Tag: tt.tag}
+		actual, err := node.GetValueRep()
+		if err != nil {
+			t.Error(err)
+			return
+		}
+		test.AssertResult(t, tt.expected, actual)
+	}
+}
+
+func TestCandidateNodeChildWhenParentUpdated(t *testing.T) {
+	parent := CandidateNode{}
+	child := parent.CreateChild()
+	parent.SetDocument(1)
+	parent.SetFileIndex(2)
+	parent.SetFilename("meow")
+	test.AssertResultWithContext(t, "meow", child.GetFilename(), "filename")
+	test.AssertResultWithContext(t, 2, child.GetFileIndex(), "file index")
+	test.AssertResultWithContext(t, uint(1), child.GetDocument(), "document index")
+}
+
+type createScalarNodeScenario struct {
+	value       interface{}
+	stringValue string
+	expectedTag string
+}
+
+var createScalarScenarios = []createScalarNodeScenario{
+	{
+		value:       "mike",
+		stringValue: "mike",
+		expectedTag: "!!str",
+	},
+	{
+		value:       3,
+		stringValue: "3",
+		expectedTag: "!!int",
+	},
+	{
+		value:       3.1,
+		stringValue: "3.1",
+		expectedTag: "!!float",
+	},
+	{
+		value:       true,
+		stringValue: "true",
+		expectedTag: "!!bool",
+	},
+	{
+		value:       nil,
+		stringValue: "~",
+		expectedTag: "!!null",
+	},
+}
+
+func TestCreateScalarNodeScenarios(t *testing.T) {
+	for _, tt := range createScalarScenarios {
+		actual := createScalarNode(tt.value, tt.stringValue)
+		test.AssertResultWithContext(t, tt.stringValue, actual.Value, fmt.Sprintf("Value for: Value: [%v], String: %v", tt.value, tt.stringValue))
+		test.AssertResultWithContext(t, tt.expectedTag, actual.Tag, fmt.Sprintf("Value for: Value: [%v], String: %v", tt.value, tt.stringValue))
+	}
+}
+
+func TestGetKeyForMapValue(t *testing.T) {
+	key := createStringScalarNode("yourKey")
+	n := CandidateNode{Key: key, Value: "meow", document: 3}
+	test.AssertResult(t, "3 - yourKey", n.GetKey())
+}
+
+func TestGetKeyForMapKey(t *testing.T) {
+	key := createStringScalarNode("yourKey")
+	key.IsMapKey = true
+	key.document = 3
+	test.AssertResult(t, "key-yourKey-3 - ", key.GetKey())
+}
+
+func TestGetKeyForValue(t *testing.T) {
+	n := CandidateNode{Value: "meow", document: 3}
+	test.AssertResult(t, "3 - ", n.GetKey())
+}
+
+func TestGetParsedKeyForMapKey(t *testing.T) {
+	key := createStringScalarNode("yourKey")
+	key.IsMapKey = true
+	key.document = 3
+	test.AssertResult(t, "yourKey", key.getParsedKey())
+}
+
+func TestGetParsedKeyForLooseValue(t *testing.T) {
+	n := CandidateNode{Value: "meow", document: 3}
+	test.AssertResult(t, nil, n.getParsedKey())
+}
+
+func TestGetParsedKeyForMapValue(t *testing.T) {
+	key := createStringScalarNode("yourKey")
+	n := CandidateNode{Key: key, Value: "meow", document: 3}
+	test.AssertResult(t, "yourKey", n.getParsedKey())
+}
+
+func TestGetParsedKeyForArrayValue(t *testing.T) {
+	key := createScalarNode(4, "4")
+	n := CandidateNode{Key: key, Value: "meow", document: 3}
+	test.AssertResult(t, 4, n.getParsedKey())
+}
+
+func TestCandidateNodeAddKeyValueChild(t *testing.T) {
+	key := CandidateNode{Value: "cool", IsMapKey: true}
+	node := CandidateNode{}
+
+	// if we use a key in a new node as a value, it should no longer be marked as a key
+
+	_, keyIsValueNow := node.AddKeyValueChild(&CandidateNode{Value: "newKey"}, &key)
+
+	test.AssertResult(t, keyIsValueNow.IsMapKey, false)
+	test.AssertResult(t, key.IsMapKey, true)
+
+}
+
+func TestConvertToNodeInfo(t *testing.T) {
+	child := &CandidateNode{
+		Kind:   ScalarNode,
+		Style:  DoubleQuotedStyle,
+		Tag:    "!!str",
+		Value:  "childValue",
+		Line:   2,
+		Column: 3,
+	}
+	parent := &CandidateNode{
+		Kind:        MappingNode,
+		Style:       TaggedStyle,
+		Tag:         "!!map",
+		Value:       "",
+		Line:        1,
+		Column:      1,
+		Content:     []*CandidateNode{child},
+		HeadComment: "head",
+		LineComment: "line",
+		FootComment: "foot",
+		Anchor:      "anchor",
+	}
+	info := parent.ConvertToNodeInfo()
+	test.AssertResult(t, "MappingNode", info.Kind)
+	test.AssertResult(t, "TaggedStyle", info.Style)
+	test.AssertResult(t, "!!map", info.Tag)
+	test.AssertResult(t, "head", info.HeadComment)
+	test.AssertResult(t, "line", info.LineComment)
+	test.AssertResult(t, "foot", info.FootComment)
+	test.AssertResult(t, "anchor", info.Anchor)
+	test.AssertResult(t, 1, info.Line)
+	test.AssertResult(t, 1, info.Column)
+	if len(info.Content) != 1 {
+		t.Fatalf("Expected 1 child, got %d", len(info.Content))
+	}
+	childInfo := info.Content[0]
+	test.AssertResult(t, "ScalarNode", childInfo.Kind)
+	test.AssertResult(t, "DoubleQuotedStyle", childInfo.Style)
+	test.AssertResult(t, "!!str", childInfo.Tag)
+	test.AssertResult(t, "childValue", childInfo.Value)
+	test.AssertResult(t, 2, childInfo.Line)
+	test.AssertResult(t, 3, childInfo.Column)
+}

pkg/yqlib/candidate_node_yaml.go

@@ -0,0 +1,215 @@
+package yqlib
+
+import (
+	"fmt"
+
+	yaml "go.yaml.in/yaml/v3"
+)
+
+func MapYamlStyle(original yaml.Style) Style {
+	switch original {
+	case yaml.TaggedStyle:
+		return TaggedStyle
+	case yaml.DoubleQuotedStyle:
+		return DoubleQuotedStyle
+	case yaml.SingleQuotedStyle:
+		return SingleQuotedStyle
+	case yaml.LiteralStyle:
+		return LiteralStyle
+	case yaml.FoldedStyle:
+		return FoldedStyle
+	case yaml.FlowStyle:
+		return FlowStyle
+	case 0:
+		return 0
+	}
+	return Style(original)
+}
+
+func MapToYamlStyle(original Style) yaml.Style {
+	switch original {
+	case TaggedStyle:
+		return yaml.TaggedStyle
+	case DoubleQuotedStyle:
+		return yaml.DoubleQuotedStyle
+	case SingleQuotedStyle:
+		return yaml.SingleQuotedStyle
+	case LiteralStyle:
+		return yaml.LiteralStyle
+	case FoldedStyle:
+		return yaml.FoldedStyle
+	case FlowStyle:
+		return yaml.FlowStyle
+	case 0:
+		return 0
+	}
+	return yaml.Style(original)
+}
+
+func (o *CandidateNode) copyFromYamlNode(node *yaml.Node, anchorMap map[string]*CandidateNode) {
+	o.Style = MapYamlStyle(node.Style)
+
+	o.Tag = node.Tag
+	o.Value = node.Value
+	o.Anchor = node.Anchor
+
+	if o.Anchor != "" {
+		anchorMap[o.Anchor] = o
+		log.Debug("set anchor %v to %v", o.Anchor, NodeToString(o))
+	}
+
+	// its a single alias
+	if node.Alias != nil && node.Alias.Anchor != "" {
+		o.Alias = anchorMap[node.Alias.Anchor]
+		log.Debug("set alias to %v", NodeToString(anchorMap[node.Alias.Anchor]))
+	}
+	o.HeadComment = node.HeadComment
+	o.LineComment = node.LineComment
+	o.FootComment = node.FootComment
+
+	o.Line = node.Line
+	o.Column = node.Column
+}
+
+func (o *CandidateNode) copyToYamlNode(node *yaml.Node) {
+	node.Style = MapToYamlStyle(o.Style)
+
+	node.Tag = o.Tag
+	node.Value = o.Value
+	node.Anchor = o.Anchor
+
+	node.HeadComment = o.HeadComment
+
+	node.LineComment = o.LineComment
+	node.FootComment = o.FootComment
+
+	node.Line = o.Line
+	node.Column = o.Column
+}
+
+func (o *CandidateNode) decodeIntoChild(childNode *yaml.Node, anchorMap map[string]*CandidateNode) (*CandidateNode, error) {
+	newChild := o.CreateChild()
+
+	// null yaml.Nodes to not end up calling UnmarshalYAML
+	// so we call it explicitly
+	if childNode.Tag == "!!null" {
+		newChild.Kind = ScalarNode
+		newChild.copyFromYamlNode(childNode, anchorMap)
+		return newChild, nil
+	}
+
+	err := newChild.UnmarshalYAML(childNode, anchorMap)
+	return newChild, err
+}
+
+func (o *CandidateNode) UnmarshalYAML(node *yaml.Node, anchorMap map[string]*CandidateNode) error {
+	log.Debugf("UnmarshalYAML %v", node.Tag)
+	switch node.Kind {
+	case yaml.AliasNode:
+		log.Debug("UnmarshalYAML - alias from yaml: %v", o.Tag)
+		o.Kind = AliasNode
+		o.copyFromYamlNode(node, anchorMap)
+		return nil
+	case yaml.ScalarNode:
+		log.Debugf("UnmarshalYAML -  a scalar")
+		o.Kind = ScalarNode
+		o.copyFromYamlNode(node, anchorMap)
+		return nil
+	case yaml.MappingNode:
+		log.Debugf("UnmarshalYAML -  a mapping node")
+		o.Kind = MappingNode
+		o.copyFromYamlNode(node, anchorMap)
+		o.Content = make([]*CandidateNode, len(node.Content))
+		for i := 0; i < len(node.Content); i += 2 {
+
+			keyNode, err := o.decodeIntoChild(node.Content[i], anchorMap)
+			if err != nil {
+				return err
+			}
+
+			keyNode.IsMapKey = true
+
+			valueNode, err := o.decodeIntoChild(node.Content[i+1], anchorMap)
+			if err != nil {
+				return err
+			}
+
+			valueNode.Key = keyNode
+
+			o.Content[i] = keyNode
+			o.Content[i+1] = valueNode
+		}
+		log.Debugf("UnmarshalYAML -  finished mapping node")
+		return nil
+	case yaml.SequenceNode:
+		log.Debugf("UnmarshalYAML -  a sequence: %v", len(node.Content))
+		o.Kind = SequenceNode
+
+		o.copyFromYamlNode(node, anchorMap)
+		log.Debugf("node Style: %v", node.Style)
+		log.Debugf("o Style: %v", o.Style)
+		o.Content = make([]*CandidateNode, len(node.Content))
+		for i := 0; i < len(node.Content); i++ {
+			keyNode := o.CreateChild()
+			keyNode.IsMapKey = true
+			keyNode.Tag = "!!int"
+			keyNode.Kind = ScalarNode
+			keyNode.Value = fmt.Sprintf("%v", i)
+
+			valueNode, err := o.decodeIntoChild(node.Content[i], anchorMap)
+			if err != nil {
+				return err
+			}
+
+			valueNode.Key = keyNode
+			o.Content[i] = valueNode
+		}
+		return nil
+	case 0:
+		// not sure when this happens
+		o.copyFromYamlNode(node, anchorMap)
+		log.Debugf("UnmarshalYAML -  err.. %v", NodeToString(o))
+		return nil
+	default:
+		return fmt.Errorf("orderedMap: invalid yaml node")
+	}
+}
+
+func (o *CandidateNode) MarshalYAML() (*yaml.Node, error) {
+	log.Debug("MarshalYAML to yaml: %v", o.Tag)
+	switch o.Kind {
+	case AliasNode:
+		log.Debug("MarshalYAML - alias to yaml: %v", o.Tag)
+		target := &yaml.Node{Kind: yaml.AliasNode}
+		o.copyToYamlNode(target)
+		return target, nil
+	case ScalarNode:
+		log.Debug("MarshalYAML - scalar: %v", o.Value)
+		target := &yaml.Node{Kind: yaml.ScalarNode}
+		o.copyToYamlNode(target)
+		return target, nil
+	case MappingNode, SequenceNode:
+		targetKind := yaml.MappingNode
+		if o.Kind == SequenceNode {
+			targetKind = yaml.SequenceNode
+		}
+		target := &yaml.Node{Kind: targetKind}
+		o.copyToYamlNode(target)
+		log.Debugf("original style: %v", o.Style)
+		log.Debugf("original: %v, tag: %v, style: %v, kind: %v", NodeToString(o), target.Tag, target.Style, target.Kind == yaml.SequenceNode)
+		target.Content = make([]*yaml.Node, len(o.Content))
+		for i := 0; i < len(o.Content); i++ {
+
+			child, err := o.Content[i].MarshalYAML()
+
+			if err != nil {
+				return nil, err
+			}
+			target.Content[i] = child
+		}
+		return target, nil
+	}
+	target := &yaml.Node{}
+	o.copyToYamlNode(target)
+	return target, nil
+}

pkg/yqlib/candidiate_node_json.go

@@ -0,0 +1,179 @@
+//go:build !yq_nojson
+
+package yqlib
+
+import (
+	"bytes"
+	"errors"
+	"fmt"
+	"io"
+
+	"github.com/goccy/go-json"
+)
+
+func (o *CandidateNode) setScalarFromJson(value interface{}) error {
+	o.Kind = ScalarNode
+	switch rawData := value.(type) {
+	case nil:
+		o.Tag = "!!null"
+		o.Value = "null"
+	case float64, float32:
+		o.Value = fmt.Sprintf("%v", value)
+		o.Tag = "!!float"
+		// json decoder returns ints as float.
+		if value == float64(int64(rawData.(float64))) {
+			// aha it's an int disguised as a float
+			o.Tag = "!!int"
+			o.Value = fmt.Sprintf("%v", int64(value.(float64)))
+		}
+	case int, int64, int32:
+		o.Value = fmt.Sprintf("%v", value)
+		o.Tag = "!!int"
+	case bool:
+		o.Value = fmt.Sprintf("%v", value)
+		o.Tag = "!!bool"
+	case string:
+		o.Value = rawData
+		o.Tag = "!!str"
+	default:
+		return fmt.Errorf("unrecognised type :( %v", rawData)
+	}
+	return nil
+}
+
+func (o *CandidateNode) UnmarshalJSON(data []byte) error {
+	log.Debug("UnmarshalJSON")
+	switch data[0] {
+	case '{':
+		log.Debug("UnmarshalJSON -  its a map!")
+		// its a map
+		o.Kind = MappingNode
+		o.Tag = "!!map"
+
+		dec := json.NewDecoder(bytes.NewReader(data))
+		_, err := dec.Token() // open object
+		if err != nil {
+			return err
+		}
+
+		// cycle through k/v
+		var tok json.Token
+		for tok, err = dec.Token(); err == nil; tok, err = dec.Token() {
+			// we can expect two types: string or Delim. Delim automatically means
+			// that it is the closing bracket of the object, whereas string means
+			// that there is another key.
+			if _, ok := tok.(json.Delim); ok {
+				break
+			}
+
+			childKey := o.CreateChild()
+			childKey.IsMapKey = true
+			childKey.Value = tok.(string)
+			childKey.Kind = ScalarNode
+			childKey.Tag = "!!str"
+
+			childValue := o.CreateChild()
+			childValue.Key = childKey
+
+			if err := dec.Decode(childValue); err != nil {
+				return err
+			}
+			o.Content = append(o.Content, childKey, childValue)
+		}
+		// unexpected error
+		if err != nil && !errors.Is(err, io.EOF) {
+			return err
+		}
+		return nil
+	case '[':
+		o.Kind = SequenceNode
+		o.Tag = "!!seq"
+		log.Debug("UnmarshalJSON -  its an array!")
+		var children []*CandidateNode
+		if err := json.Unmarshal(data, &children); err != nil {
+			return err
+		}
+		// now we put the children into the content, and set a key value for them
+		for i, child := range children {
+
+			if child == nil {
+				// need to represent it as a null scalar
+				child = createScalarNode(nil, "null")
+			}
+			childKey := o.CreateChild()
+			childKey.Kind = ScalarNode
+			childKey.Tag = "!!int"
+			childKey.Value = fmt.Sprintf("%v", i)
+			childKey.IsMapKey = true
+
+			child.Parent = o
+			child.Key = childKey
+			o.Content = append(o.Content, child)
+		}
+		return nil
+	}
+	log.Debug("UnmarshalJSON -  its a scalar!")
+	// otherwise, must be a scalar
+	var scalar interface{}
+	err := json.Unmarshal(data, &scalar)
+
+	if err != nil {
+		return err
+	}
+	log.Debug("UnmarshalJSON -  scalar is %v", scalar)
+
+	return o.setScalarFromJson(scalar)
+
+}
+
+func (o *CandidateNode) MarshalJSON() ([]byte, error) {
+	log.Debugf("MarshalJSON %v", NodeToString(o))
+	buf := new(bytes.Buffer)
+	enc := json.NewEncoder(buf)
+	enc.SetIndent("", " ")
+	enc.SetEscapeHTML(false) // do not escape html chars e.g. &, <, >
+
+	switch o.Kind {
+	case AliasNode:
+		log.Debugf("MarshalJSON AliasNode")
+		err := enc.Encode(o.Alias)
+		return buf.Bytes(), err
+	case ScalarNode:
+		log.Debugf("MarshalJSON ScalarNode")
+		value, err := o.GetValueRep()
+		if err != nil {
+			return buf.Bytes(), err
+		}
+		err = enc.Encode(value)
+		return buf.Bytes(), err
+	case MappingNode:
+		log.Debugf("MarshalJSON MappingNode")
+		buf.WriteByte('{')
+		for i := 0; i < len(o.Content); i += 2 {
+			if err := enc.Encode(o.Content[i].Value); err != nil {
+				return nil, err
+			}
+			buf.WriteByte(':')
+			if err := enc.Encode(o.Content[i+1]); err != nil {
+				return nil, err
+			}
+			if i != len(o.Content)-2 {
+				buf.WriteByte(',')
+			}
+		}
+		buf.WriteByte('}')
+		return buf.Bytes(), nil
+	case SequenceNode:
+		log.Debugf("MarshalJSON SequenceNode, %v, len: %v", o.Content, len(o.Content))
+		var err error
+		if len(o.Content) == 0 {
+			buf.WriteString("[]")
+		} else {
+			err = enc.Encode(o.Content)
+		}
+		return buf.Bytes(), err
+	default:
+		err := enc.Encode(nil)
+		return buf.Bytes(), err
+	}
+}

pkg/yqlib/chown_linux.go

@@ -0,0 +1,25 @@
+//go:build linux
+
+package yqlib
+
+import (
+	"io/fs"
+	"os"
+	"syscall"
+)
+
+func changeOwner(info fs.FileInfo, file *os.File) error {
+	if stat, ok := info.Sys().(*syscall.Stat_t); ok {
+		uid := int(stat.Uid)
+		gid := int(stat.Gid)
+
+		err := os.Chown(file.Name(), uid, gid)
+		if err != nil {
+			// this happens with snap confinement
+			// not really a big issue as users can chown
+			// the file themselves if required.
+			log.Info("Skipping chown: %v", err)
+		}
+	}
+	return nil
+}

pkg/yqlib/chown_not_linux_os.go

@@ -0,0 +1,12 @@
+//go:build !linux
+
+package yqlib
+
+import (
+	"io/fs"
+	"os"
+)
+
+func changeOwner(_ fs.FileInfo, _ *os.File) error {
+	return nil
+}

pkg/yqlib/color_print.go

@@ -56,6 +56,12 @@ func colorizeAndPrint(yamlBytes []byte, writer io.Writer) error {
 			Suffix: format(color.Reset),
 		}
 	}
+	p.Comment = func() *printer.Property {
+		return &printer.Property{
+			Prefix: format(color.FgHiBlack),
+			Suffix: format(color.Reset),
+		}
+	}
 	_, err := writer.Write([]byte(p.PrintTokens(tokens) + "\n"))
 	return err
 }

pkg/yqlib/context.go

@@ -5,7 +5,6 @@ import (
 	"fmt"
 	"time"
 
-	"github.com/jinzhu/copier"
 	logging "gopkg.in/op/go-logging.v1"
 )
 
@@ -58,13 +57,19 @@ func (n *Context) SetVariable(name string, value *list.List) {
 func (n *Context) ChildContext(results *list.List) Context {
 	clone := Context{DontAutoCreate: n.DontAutoCreate, datetimeLayout: n.datetimeLayout}
 	clone.Variables = make(map[string]*list.List)
-	if len(n.Variables) > 0 {
-		err := copier.Copy(&clone.Variables, n.Variables)
-		if err != nil {
-			log.Error("Error cloning context :(")
-			panic(err)
+	for variableKey, originalValueList := range n.Variables {
+
+		variableCopyList := list.New()
+		for el := originalValueList.Front(); el != nil; el = el.Next() {
+			// note that we dont make a copy of the candidate node
+			// this is so the 'ref' operator can work correctly.
+			clonedNode := el.Value.(*CandidateNode)
+			variableCopyList.PushBack(clonedNode)
 		}
+
+		clone.Variables[variableKey] = variableCopyList
 	}
+
 	clone.MatchingNodes = results
 	return clone
 }
@@ -78,35 +83,18 @@ func (n *Context) ToString() string {
 }
 
 func (n *Context) DeepClone() Context {
-	clone := Context{}
-	err := copier.Copy(&clone, n)
-	// copier doesn't do lists properly for some reason
-	clone.MatchingNodes = list.New()
+
+	clonedContent := list.New()
 	for el := n.MatchingNodes.Front(); el != nil; el = el.Next() {
-		clonedNode, err := el.Value.(*CandidateNode).Copy()
-		if err != nil {
-			log.Error("Error cloning context :(")
-			panic(err)
-		}
-		clone.MatchingNodes.PushBack(clonedNode)
+		clonedNode := el.Value.(*CandidateNode).Copy()
+		clonedContent.PushBack(clonedNode)
 	}
 
-	if err != nil {
-		log.Error("Error cloning context :(")
-		panic(err)
-	}
-	return clone
+	return n.ChildContext(clonedContent)
 }
 
 func (n *Context) Clone() Context {
-	clone := Context{}
-	err := copier.Copy(&clone, n)
-
-	if err != nil {
-		log.Error("Error cloning context :(")
-		panic(err)
-	}
-	return clone
+	return n.ChildContext(n.MatchingNodes)
 }
 
 func (n *Context) ReadOnlyClone() Context {

pkg/yqlib/context_test.go

@@ -0,0 +1,261 @@
+package yqlib
+
+import (
+	"container/list"
+	"strings"
+	"testing"
+
+	"github.com/mikefarah/yq/v4/test"
+	logging "gopkg.in/op/go-logging.v1"
+)
+
+func TestChildContext(t *testing.T) {
+
+	expectedOriginal := make(map[string]*list.List)
+	expectedOriginal["dog"] = list.New()
+	expectedOriginal["dog"].PushBack(&CandidateNode{Value: "woof"})
+
+	originalVariables := make(map[string]*list.List)
+	originalVariables["dog"] = list.New()
+	originalVariables["dog"].PushBack(&CandidateNode{Value: "woof"})
+
+	original := Context{
+		DontAutoCreate: true,
+		datetimeLayout: "cat",
+		Variables:      originalVariables,
+	}
+
+	newResults := list.New()
+	newResults.PushBack(&CandidateNode{Value: "bar"})
+
+	clone := original.ChildContext(newResults)
+	test.AssertResultComplex(t, originalVariables, clone.Variables)
+
+	clone.Variables["dog"].PushBack("bark")
+	// ensure this is a separate copy
+	test.AssertResultComplex(t, 1, originalVariables["dog"].Len())
+
+}
+
+func TestChildContextNoVariables(t *testing.T) {
+
+	original := Context{
+		DontAutoCreate: true,
+		datetimeLayout: "cat",
+	}
+
+	newResults := list.New()
+	newResults.PushBack(&CandidateNode{Value: "bar"})
+
+	clone := original.ChildContext(newResults)
+	test.AssertResultComplex(t, make(map[string]*list.List), clone.Variables)
+
+}
+
+func TestSingleReadonlyChildContext(t *testing.T) {
+	original := Context{
+		DontAutoCreate: false,
+		datetimeLayout: "2006-01-02",
+	}
+
+	candidate := &CandidateNode{Value: "test"}
+	clone := original.SingleReadonlyChildContext(candidate)
+
+	// Should have DontAutoCreate set to true
+	test.AssertResultComplex(t, true, clone.DontAutoCreate)
+
+	// Should have the candidate node in MatchingNodes
+	test.AssertResultComplex(t, 1, clone.MatchingNodes.Len())
+	test.AssertResultComplex(t, candidate, clone.MatchingNodes.Front().Value)
+}
+
+func TestSingleChildContext(t *testing.T) {
+	original := Context{
+		DontAutoCreate: true,
+		datetimeLayout: "2006-01-02",
+	}
+
+	candidate := &CandidateNode{Value: "test"}
+	clone := original.SingleChildContext(candidate)
+
+	// Should preserve DontAutoCreate
+	test.AssertResultComplex(t, true, clone.DontAutoCreate)
+
+	// Should have the candidate node in MatchingNodes
+	test.AssertResultComplex(t, 1, clone.MatchingNodes.Len())
+	test.AssertResultComplex(t, candidate, clone.MatchingNodes.Front().Value)
+}
+
+func TestSetDateTimeLayout(t *testing.T) {
+	context := Context{}
+
+	// Test setting datetime layout
+	context.SetDateTimeLayout("2006-01-02T15:04:05Z07:00")
+	test.AssertResultComplex(t, "2006-01-02T15:04:05Z07:00", context.datetimeLayout)
+}
+
+func TestGetDateTimeLayout(t *testing.T) {
+	// Test with custom layout
+	context := Context{datetimeLayout: "2006-01-02"}
+	result := context.GetDateTimeLayout()
+	test.AssertResultComplex(t, "2006-01-02", result)
+
+	// Test with empty layout (should return default)
+	context = Context{}
+	result = context.GetDateTimeLayout()
+	test.AssertResultComplex(t, "2006-01-02T15:04:05Z07:00", result)
+}
+
+func TestGetVariable(t *testing.T) {
+	// Test with nil Variables
+	context := Context{}
+	result := context.GetVariable("nonexistent")
+	test.AssertResultComplex(t, (*list.List)(nil), result)
+
+	// Test with existing variable
+	variables := make(map[string]*list.List)
+	variables["test"] = list.New()
+	variables["test"].PushBack(&CandidateNode{Value: "value"})
+
+	context = Context{Variables: variables}
+	result = context.GetVariable("test")
+	test.AssertResultComplex(t, variables["test"], result)
+
+	// Test with non-existent variable
+	result = context.GetVariable("nonexistent")
+	test.AssertResultComplex(t, (*list.List)(nil), result)
+}
+
+func TestSetVariable(t *testing.T) {
+	// Test setting variable when Variables is nil
+	context := Context{}
+	value := list.New()
+	value.PushBack(&CandidateNode{Value: "test"})
+
+	context.SetVariable("key", value)
+	test.AssertResultComplex(t, value, context.Variables["key"])
+
+	// Test setting variable when Variables already exists
+	context.SetVariable("key2", value)
+	test.AssertResultComplex(t, value, context.Variables["key2"])
+}
+
+func TestToString(t *testing.T) {
+	context := Context{
+		DontAutoCreate: true,
+		MatchingNodes:  list.New(),
+	}
+
+	// Add a node to test the full string representation
+	node := &CandidateNode{Value: "test"}
+	context.MatchingNodes.PushBack(node)
+
+	// Test with debug logging disabled (default)
+	result := context.ToString()
+	test.AssertResultComplex(t, "", result)
+
+	// Test with debug logging enabled
+	logging.SetLevel(logging.DEBUG, "")
+	defer logging.SetLevel(logging.INFO, "") // Reset to default
+
+	result2 := context.ToString()
+	test.AssertResultComplex(t, true, len(result2) > 0)
+	test.AssertResultComplex(t, true, strings.Contains(result2, "Context"))
+	test.AssertResultComplex(t, true, strings.Contains(result2, "DontAutoCreate: true"))
+}
+
+func TestDeepClone(t *testing.T) {
+	// Create original context with variables and matching nodes
+	originalVariables := make(map[string]*list.List)
+	originalVariables["test"] = list.New()
+	originalVariables["test"].PushBack(&CandidateNode{Value: "original"})
+
+	original := Context{
+		DontAutoCreate: true,
+		datetimeLayout: "2006-01-02",
+		Variables:      originalVariables,
+		MatchingNodes:  list.New(),
+	}
+
+	// Add a node to MatchingNodes
+	node := &CandidateNode{Value: "test"}
+	original.MatchingNodes.PushBack(node)
+
+	clone := original.DeepClone()
+
+	// Should preserve DontAutoCreate and datetimeLayout
+	test.AssertResultComplex(t, original.DontAutoCreate, clone.DontAutoCreate)
+	test.AssertResultComplex(t, original.datetimeLayout, clone.datetimeLayout)
+
+	// Should have copied variables
+	test.AssertResultComplex(t, 1, len(clone.Variables))
+	test.AssertResultComplex(t, "original", clone.Variables["test"].Front().Value.(*CandidateNode).Value)
+
+	// Should have deep copied MatchingNodes
+	test.AssertResultComplex(t, 1, clone.MatchingNodes.Len())
+
+	// Verify it's a deep copy by modifying the original
+	original.MatchingNodes.Front().Value.(*CandidateNode).Value = "modified"
+	test.AssertResultComplex(t, "test", clone.MatchingNodes.Front().Value.(*CandidateNode).Value)
+}
+
+func TestClone(t *testing.T) {
+	// Create original context
+	original := Context{
+		DontAutoCreate: true,
+		datetimeLayout: "2006-01-02",
+		MatchingNodes:  list.New(),
+	}
+
+	node := &CandidateNode{Value: "test"}
+	original.MatchingNodes.PushBack(node)
+
+	clone := original.Clone()
+
+	// Should preserve DontAutoCreate and datetimeLayout
+	test.AssertResultComplex(t, original.DontAutoCreate, clone.DontAutoCreate)
+	test.AssertResultComplex(t, original.datetimeLayout, clone.datetimeLayout)
+
+	// Should have the same MatchingNodes reference
+	test.AssertResultComplex(t, original.MatchingNodes, clone.MatchingNodes)
+}
+
+func TestReadOnlyClone(t *testing.T) {
+	original := Context{
+		DontAutoCreate: false,
+		datetimeLayout: "2006-01-02",
+		MatchingNodes:  list.New(),
+	}
+
+	node := &CandidateNode{Value: "test"}
+	original.MatchingNodes.PushBack(node)
+
+	clone := original.ReadOnlyClone()
+
+	// Should set DontAutoCreate to true
+	test.AssertResultComplex(t, true, clone.DontAutoCreate)
+
+	// Should preserve other fields
+	test.AssertResultComplex(t, original.datetimeLayout, clone.datetimeLayout)
+	test.AssertResultComplex(t, original.MatchingNodes, clone.MatchingNodes)
+}
+
+func TestWritableClone(t *testing.T) {
+	original := Context{
+		DontAutoCreate: true,
+		datetimeLayout: "2006-01-02",
+		MatchingNodes:  list.New(),
+	}
+
+	node := &CandidateNode{Value: "test"}
+	original.MatchingNodes.PushBack(node)
+
+	clone := original.WritableClone()
+
+	// Should set DontAutoCreate to false
+	test.AssertResultComplex(t, false, clone.DontAutoCreate)
+
+	// Should preserve other fields
+	test.AssertResultComplex(t, original.datetimeLayout, clone.datetimeLayout)
+	test.AssertResultComplex(t, original.MatchingNodes, clone.MatchingNodes)
+}

pkg/yqlib/csv.go

@@ -0,0 +1,23 @@
+package yqlib
+
+type CsvPreferences struct {
+	Separator rune
+	AutoParse bool
+}
+
+func NewDefaultCsvPreferences() CsvPreferences {
+	return CsvPreferences{
+		Separator: ',',
+		AutoParse: true,
+	}
+}
+
+func NewDefaultTsvPreferences() CsvPreferences {
+	return CsvPreferences{
+		Separator: '\t',
+		AutoParse: true,
+	}
+}
+
+var ConfiguredCsvPreferences = NewDefaultCsvPreferences()
+var ConfiguredTsvPreferences = NewDefaultTsvPreferences()

pkg/yqlib/csv_test.go

@@ -13,6 +13,13 @@ Gary,1,true,168.8
 Samantha's Rabbit,2,false,-188.8
 `
 
+const csvSimpleWithObject = `name,numberOfCats,likesApples,height,facts
+Gary,1,true,168.8,cool: true
+Samantha's Rabbit,2,false,-188.8,tall: indeed
+`
+const csvMissing = `name,numberOfCats,likesApples,height
+,null,,168.8
+`
 const expectedUpdatedSimpleCsv = `name,numberOfCats,likesApples,height
 Gary,3,true,168.8
 Samantha's Rabbit,2,false,-188.8
@@ -37,6 +44,31 @@ const expectedYamlFromCSV = `- name: Gary
   likesApples: false
   height: -188.8
 `
+const expectedYamlFromCSVWithObject = `- name: Gary
+  numberOfCats: 1
+  likesApples: true
+  height: 168.8
+  facts:
+    cool: true
+- name: Samantha's Rabbit
+  numberOfCats: 2
+  likesApples: false
+  height: -188.8
+  facts:
+    tall: indeed
+`
+
+const expectedYamlFromCSVNoParsing = `- name: Gary
+  numberOfCats: 1
+  likesApples: true
+  height: 168.8
+  facts: 'cool: true'
+- name: Samantha's Rabbit
+  numberOfCats: 2
+  likesApples: false
+  height: -188.8
+  facts: 'tall: indeed'
+`
 
 const expectedYamlFromCSVMissingData = `- name: Gary
   numberOfCats: 1
@@ -110,12 +142,71 @@ var csvScenarios = []formatScenario{
 		expected:       csvSimpleMissingData,
 		scenarioType:   "encode-csv",
 	},
+	{
+		description:  "decode csv missing",
+		skipDoc:      true,
+		input:        csvMissing,
+		expected:     csvMissing,
+		scenarioType: "roundtrip-csv",
+	},
+	{
+		description:  "decode csv key",
+		skipDoc:      true,
+		input:        csvSimple,
+		expression:   ".[0].name | key",
+		expected:     "name\n",
+		scenarioType: "decode-csv",
+	},
+	{
+		description:  "decode csv parent",
+		skipDoc:      true,
+		input:        csvSimple,
+		expression:   ".[0].name | parent | .height",
+		expected:     "168.8\n",
+		scenarioType: "decode-csv",
+	},
 	{
 		description:    "Parse CSV into an array of objects",
-		subdescription: "First row is assumed to be the header row.",
-		input:          csvSimple,
-		expected:       expectedYamlFromCSV,
-		scenarioType:   "decode-csv-object",
+		subdescription: "First row is assumed to be the header row. By default, entries with YAML/JSON formatting will be parsed!",
+		input:          csvSimpleWithObject,
+		expected:       expectedYamlFromCSVWithObject,
+		scenarioType:   "decode-csv",
+	},
+	{
+		description:  "Decode CSV line breaks",
+		skipDoc:      true,
+		input:        "heading1\n\"some data\nwith a line break\"\n",
+		expected:     "- heading1: |-\n    some data\n    with a line break\n",
+		scenarioType: "decode-csv",
+	},
+	{
+		description:    "Parse CSV into an array of objects, no auto-parsing",
+		subdescription: "First row is assumed to be the header row. Entries with YAML/JSON will be left as strings.",
+		input:          csvSimpleWithObject,
+		expected:       expectedYamlFromCSVNoParsing,
+		scenarioType:   "decode-csv-no-auto",
+	},
+	{
+		description:  "values starting with #, no auto parse",
+		skipDoc:      true,
+		input:        "value\n#ffff",
+		expected:     "- value: '#ffff'\n",
+		scenarioType: "decode-csv-no-auto",
+	},
+	{
+		description:  "values starting with #",
+		skipDoc:      true,
+		input:        "value\n#ffff",
+		expected:     "- value: #ffff\n",
+		scenarioType: "decode-csv",
+	},
+	{
+		description:  "Scalar roundtrip",
+		skipDoc:      true,
+		input:        "mike\ncat",
+		expression:   ".[0].mike",
+		expected:     "cat\n",
+		scenarioType: "roundtrip-csv",
 	},
 	{
 		description:    "Parse TSV into an array of objects",
@@ -136,15 +227,17 @@ var csvScenarios = []formatScenario{
 func testCSVScenario(t *testing.T, s formatScenario) {
 	switch s.scenarioType {
 	case "encode-csv":
-		test.AssertResultWithContext(t, s.expected, processFormatScenario(s, NewYamlDecoder(), NewCsvEncoder(',')), s.description)
+		test.AssertResultWithContext(t, s.expected, mustProcessFormatScenario(s, NewYamlDecoder(ConfiguredYamlPreferences), NewCsvEncoder(ConfiguredCsvPreferences)), s.description)
 	case "encode-tsv":
-		test.AssertResultWithContext(t, s.expected, processFormatScenario(s, NewYamlDecoder(), NewCsvEncoder('\t')), s.description)
-	case "decode-csv-object":
-		test.AssertResultWithContext(t, s.expected, processFormatScenario(s, NewCSVObjectDecoder(','), NewYamlEncoder(2, false, true, true)), s.description)
+		test.AssertResultWithContext(t, s.expected, mustProcessFormatScenario(s, NewYamlDecoder(ConfiguredYamlPreferences), NewCsvEncoder(ConfiguredTsvPreferences)), s.description)
+	case "decode-csv":
+		test.AssertResultWithContext(t, s.expected, mustProcessFormatScenario(s, NewCSVObjectDecoder(ConfiguredCsvPreferences), NewYamlEncoder(ConfiguredYamlPreferences)), s.description)
+	case "decode-csv-no-auto":
+		test.AssertResultWithContext(t, s.expected, mustProcessFormatScenario(s, NewCSVObjectDecoder(CsvPreferences{Separator: ',', AutoParse: false}), NewYamlEncoder(ConfiguredYamlPreferences)), s.description)
 	case "decode-tsv-object":
-		test.AssertResultWithContext(t, s.expected, processFormatScenario(s, NewCSVObjectDecoder('\t'), NewYamlEncoder(2, false, true, true)), s.description)
+		test.AssertResultWithContext(t, s.expected, mustProcessFormatScenario(s, NewCSVObjectDecoder(ConfiguredTsvPreferences), NewYamlEncoder(ConfiguredYamlPreferences)), s.description)
 	case "roundtrip-csv":
-		test.AssertResultWithContext(t, s.expected, processFormatScenario(s, NewCSVObjectDecoder(','), NewCsvEncoder(',')), s.description)
+		test.AssertResultWithContext(t, s.expected, mustProcessFormatScenario(s, NewCSVObjectDecoder(ConfiguredCsvPreferences), NewCsvEncoder(ConfiguredCsvPreferences)), s.description)
 	default:
 		panic(fmt.Sprintf("unhandled scenario type %q", s.scenarioType))
 	}
@@ -171,7 +264,32 @@ func documentCSVDecodeObjectScenario(w *bufio.Writer, s formatScenario, formatTy
 	}
 
 	writeOrPanic(w, fmt.Sprintf("```yaml\n%v```\n\n",
-		processFormatScenario(s, NewCSVObjectDecoder(separator), NewYamlEncoder(s.indent, false, true, true))),
+		mustProcessFormatScenario(s, NewCSVObjectDecoder(CsvPreferences{Separator: separator, AutoParse: true}), NewYamlEncoder(ConfiguredYamlPreferences))),
+	)
+}
+
+func documentCSVDecodeObjectNoAutoScenario(w *bufio.Writer, s formatScenario, formatType string) {
+	writeOrPanic(w, fmt.Sprintf("## %v\n", s.description))
+
+	if s.subdescription != "" {
+		writeOrPanic(w, s.subdescription)
+		writeOrPanic(w, "\n\n")
+	}
+
+	writeOrPanic(w, fmt.Sprintf("Given a sample.%v file of:\n", formatType))
+	writeOrPanic(w, fmt.Sprintf("```%v\n%v\n```\n", formatType, s.input))
+
+	writeOrPanic(w, "then\n")
+	writeOrPanic(w, fmt.Sprintf("```bash\nyq -p=%v --csv-auto-parse=f sample.%v\n```\n", formatType, formatType))
+	writeOrPanic(w, "will output\n")
+
+	separator := ','
+	if formatType == "tsv" {
+		separator = '\t'
+	}
+
+	writeOrPanic(w, fmt.Sprintf("```yaml\n%v```\n\n",
+		mustProcessFormatScenario(s, NewCSVObjectDecoder(CsvPreferences{Separator: separator, AutoParse: false}), NewYamlEncoder(ConfiguredYamlPreferences))),
 	)
 }
 
@@ -201,9 +319,10 @@ func documentCSVEncodeScenario(w *bufio.Writer, s formatScenario, formatType str
 	if formatType == "tsv" {
 		separator = '\t'
 	}
-
+	csvPrefs := NewDefaultCsvPreferences()
+	csvPrefs.Separator = separator
 	writeOrPanic(w, fmt.Sprintf("```%v\n%v```\n\n", formatType,
-		processFormatScenario(s, NewYamlDecoder(), NewCsvEncoder(separator))),
+		mustProcessFormatScenario(s, NewYamlDecoder(ConfiguredYamlPreferences), NewCsvEncoder(csvPrefs))),
 	)
 }
 
@@ -234,12 +353,15 @@ func documentCSVRoundTripScenario(w *bufio.Writer, s formatScenario, formatType
 		separator = '\t'
 	}
 
+	csvPrefs := NewDefaultCsvPreferences()
+	csvPrefs.Separator = separator
+
 	writeOrPanic(w, fmt.Sprintf("```%v\n%v```\n\n", formatType,
-		processFormatScenario(s, NewCSVObjectDecoder(separator), NewCsvEncoder(separator))),
+		mustProcessFormatScenario(s, NewCSVObjectDecoder(CsvPreferences{Separator: separator, AutoParse: true}), NewCsvEncoder(csvPrefs))),
 	)
 }
 
-func documentCSVScenario(t *testing.T, w *bufio.Writer, i interface{}) {
+func documentCSVScenario(_ *testing.T, w *bufio.Writer, i interface{}) {
 	s := i.(formatScenario)
 	if s.skipDoc {
 		return
@@ -249,8 +371,10 @@ func documentCSVScenario(t *testing.T, w *bufio.Writer, i interface{}) {
 		documentCSVEncodeScenario(w, s, "csv")
 	case "encode-tsv":
 		documentCSVEncodeScenario(w, s, "tsv")
-	case "decode-csv-object":
+	case "decode-csv":
 		documentCSVDecodeObjectScenario(w, s, "csv")
+	case "decode-csv-no-auto":
+		documentCSVDecodeObjectNoAutoScenario(w, s, "csv")
 	case "decode-tsv-object":
 		documentCSVDecodeObjectScenario(w, s, "tsv")
 	case "roundtrip-csv":

pkg/yqlib/data_tree_navigator.go

@@ -7,10 +7,12 @@ import (
 )
 
 type DataTreeNavigator interface {
-	// given the context and a expressionNode,
+	// given the context and an expressionNode,
 	// this will process the against the given expressionNode and return
 	// a new context of matching candidates
 	GetMatchingNodes(context Context, expressionNode *ExpressionNode) (Context, error)
+
+	DeeplyAssign(context Context, path []interface{}, rhsNode *CandidateNode) error
 }
 
 type dataTreeNavigator struct {
@@ -20,6 +22,33 @@ func NewDataTreeNavigator() DataTreeNavigator {
 	return &dataTreeNavigator{}
 }
 
+func (d *dataTreeNavigator) DeeplyAssign(context Context, path []interface{}, rhsCandidateNode *CandidateNode) error {
+
+	assignmentOp := &Operation{OperationType: assignOpType, Preferences: assignPreferences{}}
+
+	if rhsCandidateNode.Kind == MappingNode {
+		log.Debug("DeeplyAssign: deeply merging object")
+		// if the rhs is a map, we need to deeply merge it in.
+		// otherwise we'll clobber any existing fields
+		assignmentOp = &Operation{OperationType: multiplyAssignOpType, Preferences: multiplyPreferences{
+			AppendArrays:  true,
+			TraversePrefs: traversePreferences{DontFollowAlias: true},
+			AssignPrefs:   assignPreferences{},
+		}}
+	}
+
+	rhsOp := &Operation{OperationType: valueOpType, CandidateNode: rhsCandidateNode}
+
+	assignmentOpNode := &ExpressionNode{
+		Operation: assignmentOp,
+		LHS:       createTraversalTree(path, traversePreferences{}, false),
+		RHS:       &ExpressionNode{Operation: rhsOp},
+	}
+
+	_, err := d.GetMatchingNodes(context, assignmentOpNode)
+	return err
+}
+
 func (d *dataTreeNavigator) GetMatchingNodes(context Context, expressionNode *ExpressionNode) (Context, error) {
 	if expressionNode == nil {
 		log.Debugf("getMatchingNodes - nothing to do")
@@ -31,11 +60,10 @@ func (d *dataTreeNavigator) GetMatchingNodes(context Context, expressionNode *Ex
 			log.Debug(NodeToString(el.Value.(*CandidateNode)))
 		}
 	}
-	log.Debug(">>")
 	handler := expressionNode.Operation.OperationType.Handler
 	if handler != nil {
 		return handler(d, context, expressionNode)
 	}
-	return Context{}, fmt.Errorf("Unknown operator %v", expressionNode.Operation.OperationType)
+	return Context{}, fmt.Errorf("unknown operator %v", expressionNode.Operation.OperationType.Type)
 
 }

pkg/yqlib/data_tree_navigator_test.go

@@ -0,0 +1,437 @@
+package yqlib
+
+import (
+	"container/list"
+	"testing"
+
+	"github.com/mikefarah/yq/v4/test"
+)
+
+func TestGetMatchingNodes_NilExpressionNode(t *testing.T) {
+	navigator := NewDataTreeNavigator()
+	context := Context{
+		MatchingNodes: list.New(),
+	}
+
+	result, err := navigator.GetMatchingNodes(context, nil)
+
+	test.AssertResult(t, nil, err)
+	test.AssertResultComplex(t, context, result)
+}
+
+func TestGetMatchingNodes_UnknownOperator(t *testing.T) {
+	navigator := NewDataTreeNavigator()
+	context := Context{
+		MatchingNodes: list.New(),
+	}
+
+	// Create an expression node with an unknown operation type
+	unknownOpType := &operationType{Type: "UNKNOWN", Handler: nil}
+	expressionNode := &ExpressionNode{
+		Operation: &Operation{OperationType: unknownOpType},
+	}
+
+	result, err := navigator.GetMatchingNodes(context, expressionNode)
+
+	test.AssertResult(t, "unknown operator UNKNOWN", err.Error())
+	test.AssertResultComplex(t, Context{}, result)
+}
+
+func TestGetMatchingNodes_ValidOperator(t *testing.T) {
+	navigator := NewDataTreeNavigator()
+
+	// Create a simple context with a scalar node
+	scalarNode := &CandidateNode{
+		Kind:  ScalarNode,
+		Tag:   "!!str",
+		Value: "test",
+	}
+	context := Context{
+		MatchingNodes: list.New(),
+	}
+	context.MatchingNodes.PushBack(scalarNode)
+
+	// Create an expression node with a valid operation (self reference)
+	expressionNode := &ExpressionNode{
+		Operation: &Operation{OperationType: selfReferenceOpType},
+	}
+
+	result, err := navigator.GetMatchingNodes(context, expressionNode)
+
+	test.AssertResult(t, nil, err)
+	test.AssertResult(t, 1, result.MatchingNodes.Len())
+
+	// Verify the result contains the same node
+	resultNode := result.MatchingNodes.Front().Value.(*CandidateNode)
+	test.AssertResult(t, scalarNode, resultNode)
+}
+
+func TestDeeplyAssign_ScalarNode(t *testing.T) {
+	navigator := NewDataTreeNavigator()
+
+	// Create a context with a root mapping node
+	rootNode := &CandidateNode{
+		Kind: MappingNode,
+		Tag:  "!!map",
+		Content: []*CandidateNode{
+			{Kind: ScalarNode, Tag: "!!str", Value: "existing", IsMapKey: true},
+			{Kind: ScalarNode, Tag: "!!str", Value: "old_value"},
+		},
+	}
+	context := Context{
+		MatchingNodes: list.New(),
+	}
+	context.MatchingNodes.PushBack(rootNode)
+
+	// Create a scalar node to assign
+	scalarNode := &CandidateNode{
+		Kind:  ScalarNode,
+		Tag:   "!!str",
+		Value: "new_value",
+	}
+
+	// Assign to path ["new_key"]
+	path := []interface{}{"new_key"}
+	err := navigator.DeeplyAssign(context, path, scalarNode)
+
+	test.AssertResult(t, nil, err)
+
+	// Verify the assignment was made
+	// The root node should now have the new key-value pair
+	test.AssertResult(t, 4, len(rootNode.Content)) // 2 original + 2 new
+
+	// Find the new key-value pair
+	found := false
+	for i := 0; i < len(rootNode.Content)-1; i += 2 {
+		key := rootNode.Content[i]
+		value := rootNode.Content[i+1]
+		if key.Value == "new_key" && value.Value == "new_value" {
+			found = true
+			break
+		}
+	}
+	test.AssertResult(t, true, found)
+}
+
+func TestDeeplyAssign_MappingNode(t *testing.T) {
+	navigator := NewDataTreeNavigator()
+
+	// Create a context with a root mapping node
+	rootNode := &CandidateNode{
+		Kind: MappingNode,
+		Tag:  "!!map",
+		Content: []*CandidateNode{
+			{Kind: ScalarNode, Tag: "!!str", Value: "existing", IsMapKey: true},
+			{Kind: ScalarNode, Tag: "!!str", Value: "old_value"},
+		},
+	}
+	context := Context{
+		MatchingNodes: list.New(),
+	}
+	context.MatchingNodes.PushBack(rootNode)
+
+	// Create a mapping node to assign (this should trigger deep merge)
+	mappingNode := &CandidateNode{
+		Kind: MappingNode,
+		Tag:  "!!map",
+		Content: []*CandidateNode{
+			{Kind: ScalarNode, Tag: "!!str", Value: "nested_key", IsMapKey: true},
+			{Kind: ScalarNode, Tag: "!!str", Value: "nested_value"},
+		},
+	}
+
+	// Assign to path ["new_map"]
+	path := []interface{}{"new_map"}
+	err := navigator.DeeplyAssign(context, path, mappingNode)
+
+	test.AssertResult(t, nil, err)
+
+	// Verify the assignment was made
+	// The root node should now have the new mapping
+	test.AssertResult(t, 4, len(rootNode.Content)) // 2 original + 2 new
+
+	// Find the new mapping
+	found := false
+	for i := 0; i < len(rootNode.Content); i += 2 {
+		if i+1 < len(rootNode.Content) {
+			key := rootNode.Content[i]
+			value := rootNode.Content[i+1]
+			if key.Value == "new_map" && value.Kind == MappingNode {
+				found = true
+				// Verify the nested content
+				test.AssertResult(t, 2, len(value.Content))
+				test.AssertResult(t, "nested_key", value.Content[0].Value)
+				test.AssertResult(t, "nested_value", value.Content[1].Value)
+				break
+			}
+		}
+	}
+	test.AssertResult(t, true, found)
+}
+
+func TestDeeplyAssign_DeepPath(t *testing.T) {
+	navigator := NewDataTreeNavigator()
+
+	// Create a context with a root mapping node
+	rootNode := &CandidateNode{
+		Kind: MappingNode,
+		Tag:  "!!map",
+		Content: []*CandidateNode{
+			{Kind: ScalarNode, Tag: "!!str", Value: "level1", IsMapKey: true},
+			{Kind: MappingNode, Tag: "!!map", Content: []*CandidateNode{}},
+		},
+	}
+	context := Context{
+		MatchingNodes: list.New(),
+	}
+	context.MatchingNodes.PushBack(rootNode)
+
+	// Create a scalar node to assign
+	scalarNode := &CandidateNode{
+		Kind:  ScalarNode,
+		Tag:   "!!str",
+		Value: "deep_value",
+	}
+
+	// Assign to deep path ["level1", "level2", "level3"]
+	path := []interface{}{"level1", "level2", "level3"}
+	err := navigator.DeeplyAssign(context, path, scalarNode)
+
+	test.AssertResult(t, nil, err)
+
+	// Verify the deep assignment was made
+	level1Node := rootNode.Content[1]                // The mapping node
+	test.AssertResult(t, 2, len(level1Node.Content)) // Should have level2 key-value
+
+	level2Key := level1Node.Content[0]
+	level2Value := level1Node.Content[1]
+	test.AssertResult(t, "level2", level2Key.Value)
+	test.AssertResult(t, MappingNode, level2Value.Kind)
+
+	level3Key := level2Value.Content[0]
+	level3Value := level2Value.Content[1]
+	test.AssertResult(t, "level3", level3Key.Value)
+	test.AssertResult(t, "deep_value", level3Value.Value)
+}
+
+func TestDeeplyAssign_ArrayPath(t *testing.T) {
+	navigator := NewDataTreeNavigator()
+
+	// Create a context with a root mapping node containing an array
+	rootNode := &CandidateNode{
+		Kind: MappingNode,
+		Tag:  "!!map",
+		Content: []*CandidateNode{
+			{Kind: ScalarNode, Tag: "!!str", Value: "array", IsMapKey: true},
+			{Kind: SequenceNode, Tag: "!!seq", Content: []*CandidateNode{}},
+		},
+	}
+	context := Context{
+		MatchingNodes: list.New(),
+	}
+	context.MatchingNodes.PushBack(rootNode)
+
+	// Create a scalar node to assign
+	scalarNode := &CandidateNode{
+		Kind:  ScalarNode,
+		Tag:   "!!str",
+		Value: "array_value",
+	}
+
+	// Assign to array path ["array", 0]
+	path := []interface{}{"array", 0}
+	err := navigator.DeeplyAssign(context, path, scalarNode)
+
+	test.AssertResult(t, nil, err)
+
+	// Verify the array assignment was made
+	arrayNode := rootNode.Content[1]                // The sequence node
+	test.AssertResult(t, 1, len(arrayNode.Content)) // Should have one element
+
+	arrayElement := arrayNode.Content[0]
+	test.AssertResult(t, "array_value", arrayElement.Value)
+}
+
+func TestDeeplyAssign_OverwriteExisting(t *testing.T) {
+	navigator := NewDataTreeNavigator()
+
+	// Create a context with a root mapping node
+	rootNode := &CandidateNode{
+		Kind: MappingNode,
+		Tag:  "!!map",
+		Content: []*CandidateNode{
+			{Kind: ScalarNode, Tag: "!!str", Value: "key", IsMapKey: true},
+			{Kind: ScalarNode, Tag: "!!str", Value: "old_value"},
+		},
+	}
+	context := Context{
+		MatchingNodes: list.New(),
+	}
+	context.MatchingNodes.PushBack(rootNode)
+
+	// Create a scalar node to assign
+	scalarNode := &CandidateNode{
+		Kind:  ScalarNode,
+		Tag:   "!!str",
+		Value: "new_value",
+	}
+
+	// Assign to existing path ["key"]
+	path := []interface{}{"key"}
+	err := navigator.DeeplyAssign(context, path, scalarNode)
+
+	test.AssertResult(t, nil, err)
+
+	// Verify the value was overwritten
+	test.AssertResult(t, 2, len(rootNode.Content)) // Should still have 2 elements
+
+	key := rootNode.Content[0]
+	value := rootNode.Content[1]
+	test.AssertResult(t, "key", key.Value)
+	test.AssertResult(t, "new_value", value.Value) // Should be overwritten
+}
+
+func TestDeeplyAssign_ErrorHandling(t *testing.T) {
+	navigator := NewDataTreeNavigator()
+
+	// Create a context with a scalar node (not a mapping)
+	scalarNode := &CandidateNode{
+		Kind:  ScalarNode,
+		Tag:   "!!str",
+		Value: "not_a_map",
+	}
+	context := Context{
+		MatchingNodes: list.New(),
+	}
+	context.MatchingNodes.PushBack(scalarNode)
+
+	// Create a scalar node to assign
+	assignNode := &CandidateNode{
+		Kind:  ScalarNode,
+		Tag:   "!!str",
+		Value: "value",
+	}
+
+	// Try to assign to a path on a scalar (should fail)
+	path := []interface{}{"key"}
+	err := navigator.DeeplyAssign(context, path, assignNode)
+
+	// Print the actual error for debugging
+	if err != nil {
+		t.Logf("Actual error: %v", err)
+	}
+
+	// This should fail because we can't assign to a scalar
+	test.AssertResult(t, nil, err)
+}
+
+func TestGetMatchingNodes_WithVariables(t *testing.T) {
+	navigator := NewDataTreeNavigator()
+
+	// Create a context with variables
+	variables := make(map[string]*list.List)
+	varList := list.New()
+	varList.PushBack(&CandidateNode{Kind: ScalarNode, Tag: "!!str", Value: "var_value"})
+	variables["test_var"] = varList
+
+	context := Context{
+		MatchingNodes: list.New(),
+		Variables:     variables,
+	}
+
+	// Create an expression node that gets a variable
+	expressionNode := &ExpressionNode{
+		Operation: &Operation{OperationType: getVariableOpType, StringValue: "test_var"},
+	}
+
+	result, err := navigator.GetMatchingNodes(context, expressionNode)
+
+	test.AssertResult(t, nil, err)
+	test.AssertResult(t, 1, result.MatchingNodes.Len())
+
+	// Verify the variable was retrieved
+	resultNode := result.MatchingNodes.Front().Value.(*CandidateNode)
+	test.AssertResult(t, "var_value", resultNode.Value)
+}
+
+func TestGetMatchingNodes_EmptyContext(t *testing.T) {
+	navigator := NewDataTreeNavigator()
+
+	// Create an empty context
+	context := Context{
+		MatchingNodes: list.New(),
+	}
+
+	// Create an expression node with self reference
+	expressionNode := &ExpressionNode{
+		Operation: &Operation{OperationType: selfReferenceOpType},
+	}
+
+	result, err := navigator.GetMatchingNodes(context, expressionNode)
+
+	test.AssertResult(t, nil, err)
+	test.AssertResult(t, 0, result.MatchingNodes.Len())
+}
+
+func TestDeeplyAssign_ComplexMappingMerge(t *testing.T) {
+	navigator := NewDataTreeNavigator()
+
+	// Create a context with a root mapping node containing nested data
+	rootNode := &CandidateNode{
+		Kind: MappingNode,
+		Tag:  "!!map",
+		Content: []*CandidateNode{
+			{Kind: ScalarNode, Tag: "!!str", Value: "config", IsMapKey: true},
+			{Kind: MappingNode, Tag: "!!map", Content: []*CandidateNode{
+				{Kind: ScalarNode, Tag: "!!str", Value: "existing_key", IsMapKey: true},
+				{Kind: ScalarNode, Tag: "!!str", Value: "existing_value"},
+			}},
+		},
+	}
+	context := Context{
+		MatchingNodes: list.New(),
+	}
+	context.MatchingNodes.PushBack(rootNode)
+
+	// Create a mapping node to merge
+	mappingNode := &CandidateNode{
+		Kind: MappingNode,
+		Tag:  "!!map",
+		Content: []*CandidateNode{
+			{Kind: ScalarNode, Tag: "!!str", Value: "new_key", IsMapKey: true},
+			{Kind: ScalarNode, Tag: "!!str", Value: "new_value"},
+			{Kind: ScalarNode, Tag: "!!str", Value: "existing_key", IsMapKey: true},
+			{Kind: ScalarNode, Tag: "!!str", Value: "updated_value"},
+		},
+	}
+
+	// Assign to path ["config"] (should merge with existing mapping)
+	path := []interface{}{"config"}
+	err := navigator.DeeplyAssign(context, path, mappingNode)
+
+	test.AssertResult(t, nil, err)
+
+	// Verify the merge was successful
+	configNode := rootNode.Content[1]                // The config mapping node
+	test.AssertResult(t, 4, len(configNode.Content)) // Should have 2 key-value pairs
+
+	// Check that both existing and new keys are present
+	foundExisting := false
+	foundNew := false
+	for i := 0; i < len(configNode.Content); i += 2 {
+		if i+1 < len(configNode.Content) {
+			key := configNode.Content[i]
+			value := configNode.Content[i+1]
+			switch key.Value {
+			case "existing_key":
+				foundExisting = true
+				test.AssertResult(t, "updated_value", value.Value) // Should be updated
+			case "new_key":
+				foundNew = true
+				test.AssertResult(t, "new_value", value.Value)
+			}
+		}
+	}
+	test.AssertResult(t, true, foundExisting)
+	test.AssertResult(t, true, foundNew)
+}

pkg/yqlib/decoder.go

@@ -1,44 +1,10 @@
 package yqlib
 
 import (
-	"fmt"
 	"io"
-
-	yaml "gopkg.in/yaml.v3"
-)
-
-type InputFormat uint
-
-const (
-	YamlInputFormat = 1 << iota
-	XMLInputFormat
-	PropertiesInputFormat
-	Base64InputFormat
-	JsonInputFormat
-	CSVObjectInputFormat
-	TSVObjectInputFormat
 )
 
 type Decoder interface {
-	Init(reader io.Reader)
-	Decode(node *yaml.Node) error
-}
-
-func InputFormatFromString(format string) (InputFormat, error) {
-	switch format {
-	case "yaml", "y":
-		return YamlInputFormat, nil
-	case "xml", "x":
-		return XMLInputFormat, nil
-	case "props", "p":
-		return PropertiesInputFormat, nil
-	case "json", "ndjson", "j":
-		return JsonInputFormat, nil
-	case "csv", "c":
-		return CSVObjectInputFormat, nil
-	case "tsv", "t":
-		return TSVObjectInputFormat, nil
-	default:
-		return 0, fmt.Errorf("unknown format '%v' please use [yaml|xml|props]", format)
-	}
+	Init(reader io.Reader) error
+	Decode() (*CandidateNode, error)
 }

pkg/yqlib/decoder_base64.go

@@ -1,13 +1,36 @@
+//go:build !yq_nobase64
+
 package yqlib
 
 import (
 	"bytes"
 	"encoding/base64"
 	"io"
-
-	yaml "gopkg.in/yaml.v3"
+	"strings"
 )
 
+type base64Padder struct {
+	count int
+	io.Reader
+}
+
+func (c *base64Padder) pad(buf []byte) (int, error) {
+	pad := strings.Repeat("=", (4 - c.count%4))
+	n, err := strings.NewReader(pad).Read(buf)
+	c.count += n
+	return n, err
+}
+
+func (c *base64Padder) Read(buf []byte) (int, error) {
+	n, err := c.Reader.Read(buf)
+	c.count += n
+
+	if err == io.EOF && c.count%4 != 0 {
+		return c.pad(buf)
+	}
+	return n, err
+}
+
 type base64Decoder struct {
 	reader       io.Reader
 	finished     bool
@@ -19,21 +42,22 @@ func NewBase64Decoder() Decoder {
 	return &base64Decoder{finished: false, encoding: *base64.StdEncoding}
 }
 
-func (dec *base64Decoder) Init(reader io.Reader) {
-	dec.reader = reader
+func (dec *base64Decoder) Init(reader io.Reader) error {
+	dec.reader = &base64Padder{Reader: reader}
 	dec.readAnything = false
 	dec.finished = false
+	return nil
 }
 
-func (dec *base64Decoder) Decode(rootYamlNode *yaml.Node) error {
+func (dec *base64Decoder) Decode() (*CandidateNode, error) {
 	if dec.finished {
-		return io.EOF
+		return nil, io.EOF
 	}
 	base64Reader := base64.NewDecoder(&dec.encoding, dec.reader)
 	buf := new(bytes.Buffer)
 
 	if _, err := buf.ReadFrom(base64Reader); err != nil {
-		return err
+		return nil, err
 	}
 	if buf.Len() == 0 {
 		dec.finished = true
@@ -42,12 +66,9 @@ func (dec *base64Decoder) Decode(rootYamlNode *yaml.Node) error {
 		// otherwise if we've already read some bytes, and now we get
 		// an empty string, then we are done.
 		if dec.readAnything {
-			return io.EOF
+			return nil, io.EOF
 		}
 	}
 	dec.readAnything = true
-	rootYamlNode.Kind = yaml.ScalarNode
-	rootYamlNode.Tag = "!!str"
-	rootYamlNode.Value = buf.String()
-	return nil
+	return createStringScalarNode(buf.String()), nil
 }

pkg/yqlib/decoder_csv_object.go

@@ -1,3 +1,5 @@
+//go:build !yq_nocsv
+
 package yqlib
 
 import (
@@ -5,73 +7,70 @@ import (
 	"errors"
 	"io"
 
-	yaml "gopkg.in/yaml.v3"
+	"github.com/dimchansky/utfbom"
 )
 
 type csvObjectDecoder struct {
-	separator rune
-	reader    csv.Reader
-	finished  bool
+	prefs    CsvPreferences
+	reader   csv.Reader
+	finished bool
 }
 
-func NewCSVObjectDecoder(separator rune) Decoder {
-	return &csvObjectDecoder{separator: separator}
+func NewCSVObjectDecoder(prefs CsvPreferences) Decoder {
+	return &csvObjectDecoder{prefs: prefs}
 }
 
-func (dec *csvObjectDecoder) Init(reader io.Reader) {
-	dec.reader = *csv.NewReader(reader)
-	dec.reader.Comma = dec.separator
+func (dec *csvObjectDecoder) Init(reader io.Reader) error {
+	cleanReader, enc := utfbom.Skip(reader)
+	log.Debugf("Detected encoding: %s\n", enc)
+	dec.reader = *csv.NewReader(cleanReader)
+	dec.reader.Comma = dec.prefs.Separator
 	dec.finished = false
+	return nil
 }
 
-func (dec *csvObjectDecoder) convertToYamlNode(content string) *yaml.Node {
+func (dec *csvObjectDecoder) convertToNode(content string) *CandidateNode {
 	node, err := parseSnippet(content)
-	if err != nil {
+	// if we're not auto-parsing, then we wont put in parsed objects or arrays
+	// but we still parse scalars
+	if err != nil || (!dec.prefs.AutoParse && (node.Kind != ScalarNode || node.Value != content)) {
 		return createScalarNode(content, content)
 	}
 	return node
 }
 
-func (dec *csvObjectDecoder) createObject(headerRow []string, contentRow []string) *yaml.Node {
-	objectNode := &yaml.Node{Kind: yaml.MappingNode, Tag: "!!map"}
+func (dec *csvObjectDecoder) createObject(headerRow []string, contentRow []string) *CandidateNode {
+	objectNode := &CandidateNode{Kind: MappingNode, Tag: "!!map"}
 
 	for i, header := range headerRow {
-		objectNode.Content = append(
-			objectNode.Content,
-			createScalarNode(header, header),
-			dec.convertToYamlNode(contentRow[i]))
+		objectNode.AddKeyValueChild(createScalarNode(header, header), dec.convertToNode(contentRow[i]))
 	}
 	return objectNode
 }
 
-func (dec *csvObjectDecoder) Decode(rootYamlNode *yaml.Node) error {
+func (dec *csvObjectDecoder) Decode() (*CandidateNode, error) {
 	if dec.finished {
-		return io.EOF
+		return nil, io.EOF
 	}
 	headerRow, err := dec.reader.Read()
 	log.Debugf(": headerRow%v", headerRow)
 	if err != nil {
-		return err
+		return nil, err
 	}
 
-	rootArray := &yaml.Node{Kind: yaml.SequenceNode, Tag: "!!seq"}
+	rootArray := &CandidateNode{Kind: SequenceNode, Tag: "!!seq"}
 
 	contentRow, err := dec.reader.Read()
 
 	for err == nil && len(contentRow) > 0 {
 		log.Debugf("Adding contentRow: %v", contentRow)
-		rootArray.Content = append(rootArray.Content, dec.createObject(headerRow, contentRow))
+		rootArray.AddChild(dec.createObject(headerRow, contentRow))
 		contentRow, err = dec.reader.Read()
 		log.Debugf("Read next contentRow: %v, %v", contentRow, err)
 	}
 	if !errors.Is(err, io.EOF) {
-		return err
+		return nil, err
 	}
 
-	log.Debugf("finished, contentRow%v", contentRow)
-	log.Debugf("err: %v", err)
-
-	rootYamlNode.Kind = yaml.DocumentNode
-	rootYamlNode.Content = []*yaml.Node{rootArray}
-	return nil
+	return rootArray, nil
 }

pkg/yqlib/decoder_goccy_yaml.go

@@ -0,0 +1,49 @@
+//go:build !yq_noyaml
+
+//
+// NOTE this is still a WIP - not yet ready.
+//
+
+package yqlib
+
+import (
+	"io"
+
+	yaml "github.com/goccy/go-yaml"
+	"github.com/goccy/go-yaml/ast"
+)
+
+type goccyYamlDecoder struct {
+	decoder yaml.Decoder
+	cm      yaml.CommentMap
+	// anchor map persists over multiple documents for convenience.
+	anchorMap map[string]*CandidateNode
+}
+
+func NewGoccyYAMLDecoder() Decoder {
+	return &goccyYamlDecoder{}
+}
+
+func (dec *goccyYamlDecoder) Init(reader io.Reader) error {
+	dec.cm = yaml.CommentMap{}
+	dec.decoder = *yaml.NewDecoder(reader, yaml.CommentToMap(dec.cm), yaml.UseOrderedMap())
+	dec.anchorMap = make(map[string]*CandidateNode)
+	return nil
+}
+
+func (dec *goccyYamlDecoder) Decode() (*CandidateNode, error) {
+
+	var ast ast.Node
+
+	err := dec.decoder.Decode(&ast)
+	if err != nil {
+		return nil, err
+	}
+
+	candidateNode := &CandidateNode{}
+	if err := candidateNode.UnmarshalGoccyYAML(ast, dec.cm, dec.anchorMap); err != nil {
+		return nil, err
+	}
+
+	return candidateNode, nil
+}

pkg/yqlib/decoder_ini.go

@@ -0,0 +1,106 @@
+//go:build !yq_noini
+
+package yqlib
+
+import (
+	"fmt"
+	"io"
+
+	"github.com/go-ini/ini"
+)
+
+type iniDecoder struct {
+	reader   io.Reader
+	finished bool // Flag to signal completion of processing
+}
+
+func NewINIDecoder() Decoder {
+	return &iniDecoder{
+		finished: false, // Initialize the flag as false
+	}
+}
+
+func (dec *iniDecoder) Init(reader io.Reader) error {
+	// Store the reader for use in Decode
+	dec.reader = reader
+	return nil
+}
+
+func (dec *iniDecoder) Decode() (*CandidateNode, error) {
+	// If processing is already finished, return io.EOF
+	if dec.finished {
+		return nil, io.EOF
+	}
+
+	// Read all content from the stored reader
+	content, err := io.ReadAll(dec.reader)
+	if err != nil {
+		return nil, fmt.Errorf("failed to read INI content: %w", err)
+	}
+
+	// Parse the INI content
+	cfg, err := ini.Load(content)
+	if err != nil {
+		return nil, fmt.Errorf("failed to parse INI content: %w", err)
+	}
+
+	// Create a root CandidateNode as a MappingNode (since INI is key-value based)
+	root := &CandidateNode{
+		Kind:  MappingNode,
+		Tag:   "!!map",
+		Value: "",
+	}
+
+	// Process each section in the INI file
+	for _, section := range cfg.Sections() {
+		sectionName := section.Name()
+
+		if sectionName == ini.DefaultSection {
+			// For the default section, add key-value pairs directly to the root node
+			for _, key := range section.Keys() {
+				keyName := key.Name()
+				keyValue := key.String()
+
+				// Create a key node (scalar for the key name)
+				keyNode := createStringScalarNode(keyName)
+				// Create a value node (scalar for the value)
+				valueNode := createStringScalarNode(keyValue)
+
+				// Add key-value pair to the root node
+				root.AddKeyValueChild(keyNode, valueNode)
+			}
+		} else {
+			// For named sections, create a nested map
+			sectionNode := &CandidateNode{
+				Kind:  MappingNode,
+				Tag:   "!!map",
+				Value: "",
+			}
+
+			// Add key-value pairs to the section node
+			for _, key := range section.Keys() {
+				keyName := key.Name()
+				keyValue := key.String()
+
+				// Create a key node (scalar for the key name)
+				keyNode := createStringScalarNode(keyName)
+				// Create a value node (scalar for the value)
+				valueNode := createStringScalarNode(keyValue)
+
+				// Add key-value pair to the section node
+				sectionNode.AddKeyValueChild(keyNode, valueNode)
+			}
+
+			// Create a key node for the section name
+			sectionKeyNode := createStringScalarNode(sectionName)
+			// Add the section as a nested map to the root node
+			root.AddKeyValueChild(sectionKeyNode, sectionNode)
+		}
+	}
+
+	// Set the finished flag to true to prevent further Decode calls
+	dec.finished = true
+
+	// Return the root node
+	return root, nil
+}

pkg/yqlib/decoder_json.go

@@ -1,11 +1,11 @@
+//go:build !yq_nojson
+
 package yqlib
 
 import (
-	"fmt"
 	"io"
 
 	"github.com/goccy/go-json"
-	yaml "gopkg.in/yaml.v3"
 )
 
 type jsonDecoder struct {
@@ -16,67 +16,18 @@ func NewJSONDecoder() Decoder {
 	return &jsonDecoder{}
 }
 
-func (dec *jsonDecoder) Init(reader io.Reader) {
+func (dec *jsonDecoder) Init(reader io.Reader) error {
 	dec.decoder = *json.NewDecoder(reader)
-}
-
-func (dec *jsonDecoder) Decode(rootYamlNode *yaml.Node) error {
-
-	var dataBucket orderedMap
-	log.Debug("going to decode")
-	err := dec.decoder.Decode(&dataBucket)
-	if err != nil {
-		return err
-	}
-	node, err := dec.convertToYamlNode(&dataBucket)
-
-	if err != nil {
-		return err
-	}
-	rootYamlNode.Kind = yaml.DocumentNode
-	rootYamlNode.Content = []*yaml.Node{node}
 	return nil
 }
 
-func (dec *jsonDecoder) convertToYamlNode(data *orderedMap) (*yaml.Node, error) {
-	if data.kv == nil {
-		switch rawData := data.altVal.(type) {
-		case nil:
-			return createScalarNode(nil, "null"), nil
-		case float64, float32:
-			// json decoder returns ints as float.
-			return parseSnippet(fmt.Sprintf("%v", rawData))
-		case int, int64, int32, string, bool:
-			return createScalarNode(rawData, fmt.Sprintf("%v", rawData)), nil
-		case []*orderedMap:
-			return dec.parseArray(rawData)
-		default:
-			return nil, fmt.Errorf("unrecognised type :( %v", rawData)
-		}
+func (dec *jsonDecoder) Decode() (*CandidateNode, error) {
+
+	var dataBucket CandidateNode
+	err := dec.decoder.Decode(&dataBucket)
+	if err != nil {
+		return nil, err
 	}
 
-	var yamlMap = &yaml.Node{Kind: yaml.MappingNode}
-	for _, keyValuePair := range data.kv {
-		yamlValue, err := dec.convertToYamlNode(&keyValuePair.V)
-		if err != nil {
-			return nil, err
-		}
-		yamlMap.Content = append(yamlMap.Content, createScalarNode(keyValuePair.K, keyValuePair.K), yamlValue)
-	}
-	return yamlMap, nil
-
-}
-
-func (dec *jsonDecoder) parseArray(dataArray []*orderedMap) (*yaml.Node, error) {
-
-	var yamlMap = &yaml.Node{Kind: yaml.SequenceNode}
-
-	for _, value := range dataArray {
-		yamlValue, err := dec.convertToYamlNode(value)
-		if err != nil {
-			return nil, err
-		}
-		yamlMap.Content = append(yamlMap.Content, yamlValue)
-	}
-	return yamlMap, nil
+	return &dataBucket, nil
 }

pkg/yqlib/decoder_lua.go

@@ -0,0 +1,165 @@
+//go:build !yq_nolua
+
+package yqlib
+
+import (
+	"fmt"
+	"io"
+	"math"
+
+	lua "github.com/yuin/gopher-lua"
+)
+
+type luaDecoder struct {
+	reader   io.Reader
+	finished bool
+	prefs    LuaPreferences
+}
+
+func NewLuaDecoder(prefs LuaPreferences) Decoder {
+	return &luaDecoder{
+		prefs: prefs,
+	}
+}
+
+func (dec *luaDecoder) Init(reader io.Reader) error {
+	dec.reader = reader
+	return nil
+}
+
+func (dec *luaDecoder) convertToYamlNode(ls *lua.LState, lv lua.LValue) *CandidateNode {
+	switch lv.Type() {
+	case lua.LTNil:
+		return &CandidateNode{
+			Kind:  ScalarNode,
+			Tag:   "!!null",
+			Value: "",
+		}
+	case lua.LTBool:
+		return &CandidateNode{
+			Kind:  ScalarNode,
+			Tag:   "!!bool",
+			Value: lv.String(),
+		}
+	case lua.LTNumber:
+		n := float64(lua.LVAsNumber(lv))
+		// various special case floats
+		if math.IsNaN(n) {
+			return &CandidateNode{
+				Kind:  ScalarNode,
+				Tag:   "!!float",
+				Value: ".nan",
+			}
+		}
+		if math.IsInf(n, 1) {
+			return &CandidateNode{
+				Kind:  ScalarNode,
+				Tag:   "!!float",
+				Value: ".inf",
+			}
+		}
+		if math.IsInf(n, -1) {
+			return &CandidateNode{
+				Kind:  ScalarNode,
+				Tag:   "!!float",
+				Value: "-.inf",
+			}
+		}
+
+		// does it look like an integer?
+		if n == float64(int(n)) {
+			return &CandidateNode{
+				Kind:  ScalarNode,
+				Tag:   "!!int",
+				Value: lv.String(),
+			}
+		}
+
+		return &CandidateNode{
+			Kind:  ScalarNode,
+			Tag:   "!!float",
+			Value: lv.String(),
+		}
+	case lua.LTString:
+		return &CandidateNode{
+			Kind:  ScalarNode,
+			Tag:   "!!str",
+			Value: lv.String(),
+		}
+	case lua.LTFunction:
+		return &CandidateNode{
+			Kind:  ScalarNode,
+			Tag:   "tag:lua.org,2006,function",
+			Value: lv.String(),
+		}
+	case lua.LTTable:
+		// Simultaneously create a sequence and a map, pick which one to return
+		// based on whether all keys were consecutive integers
+		i := 1
+		yaml_sequence := &CandidateNode{
+			Kind: SequenceNode,
+			Tag:  "!!seq",
+		}
+		yaml_map := &CandidateNode{
+			Kind: MappingNode,
+			Tag:  "!!map",
+		}
+		t := lv.(*lua.LTable)
+		k, v := ls.Next(t, lua.LNil)
+		for k != lua.LNil {
+			if ki, ok := k.(lua.LNumber); i != 0 && ok && math.Mod(float64(ki), 1) == 0 && int(ki) == i {
+				i++
+			} else {
+				i = 0
+			}
+			newKey := dec.convertToYamlNode(ls, k)
+
+			yv := dec.convertToYamlNode(ls, v)
+			yaml_map.AddKeyValueChild(newKey, yv)
+
+			if i != 0 {
+				yaml_sequence.AddChild(yv)
+			}
+			k, v = ls.Next(t, k)
+		}
+		if i != 0 {
+			return yaml_sequence
+		}
+		return yaml_map
+	default:
+		return &CandidateNode{
+			Kind:        ScalarNode,
+			LineComment: fmt.Sprintf("Unhandled Lua type: %s", lv.Type().String()),
+			Tag:         "!!null",
+			Value:       lv.String(),
+		}
+	}
+}
+
+func (dec *luaDecoder) decideTopLevelNode(ls *lua.LState) *CandidateNode {
+	if ls.GetTop() == 0 {
+		// no items were explicitly returned, encode the globals table instead
+		return dec.convertToYamlNode(ls, ls.Get(lua.GlobalsIndex))
+	}
+	return dec.convertToYamlNode(ls, ls.Get(1))
+}
+
+func (dec *luaDecoder) Decode() (*CandidateNode, error) {
+	if dec.finished {
+		return nil, io.EOF
+	}
+	ls := lua.NewState(lua.Options{SkipOpenLibs: true})
+	defer ls.Close()
+	fn, err := ls.Load(dec.reader, "@input")
+	if err != nil {
+		return nil, err
+	}
+	ls.Push(fn)
+	err = ls.PCall(0, lua.MultRet, nil)
+	if err != nil {
+		return nil, err
+	}
+	firstNode := dec.decideTopLevelNode(ls)
+	dec.finished = true
+	return firstNode, nil
+}

pkg/yqlib/decoder_properties.go

@@ -1,13 +1,15 @@
+//go:build !yq_noprops
+
 package yqlib
 
 import (
 	"bytes"
+	"fmt"
 	"io"
 	"strconv"
 	"strings"
 
 	"github.com/magiconair/properties"
-	"gopkg.in/yaml.v3"
 )
 
 type propertiesDecoder struct {
@@ -20,9 +22,10 @@ func NewPropertiesDecoder() Decoder {
 	return &propertiesDecoder{d: NewDataTreeNavigator(), finished: false}
 }
 
-func (dec *propertiesDecoder) Init(reader io.Reader) {
+func (dec *propertiesDecoder) Init(reader io.Reader) error {
 	dec.reader = reader
 	dec.finished = false
+	return nil
 }
 
 func parsePropKey(key string) []interface{} {
@@ -46,31 +49,23 @@ func (dec *propertiesDecoder) processComment(c string) string {
 	return "# " + c
 }
 
-func (dec *propertiesDecoder) applyProperty(properties *properties.Properties, context Context, key string) error {
-	value, _ := properties.Get(key)
-	path := parsePropKey(key)
-
-	rhsNode := &yaml.Node{
-		Value:       value,
-		Tag:         "!!str",
-		Kind:        yaml.ScalarNode,
-		LineComment: dec.processComment(properties.GetComment(key)),
-	}
-
-	rhsNode.Tag = guessTagFromCustomType(rhsNode)
-
-	rhsCandidateNode := &CandidateNode{
-		Path: path,
-		Node: rhsNode,
-	}
-
+func (dec *propertiesDecoder) applyPropertyComments(context Context, path []interface{}, comments []string) error {
 	assignmentOp := &Operation{OperationType: assignOpType, Preferences: assignPreferences{}}
 
-	rhsOp := &Operation{OperationType: valueOpType, CandidateNode: rhsCandidateNode}
+	rhsCandidateNode := &CandidateNode{
+		Tag:         "!!str",
+		Value:       fmt.Sprintf("%v", path[len(path)-1]),
+		HeadComment: dec.processComment(strings.Join(comments, "\n")),
+		Kind:        ScalarNode,
+	}
+
+	rhsCandidateNode.Tag = rhsCandidateNode.guessTagFromCustomType()
+
+	rhsOp := &Operation{OperationType: referenceOpType, CandidateNode: rhsCandidateNode}
 
 	assignmentOpNode := &ExpressionNode{
 		Operation: assignmentOp,
-		LHS:       createTraversalTree(path, traversePreferences{}, false),
+		LHS:       createTraversalTree(path, traversePreferences{}, true),
 		RHS:       &ExpressionNode{Operation: rhsOp},
 	}
 
@@ -78,45 +73,59 @@ func (dec *propertiesDecoder) applyProperty(properties *properties.Properties, c
 	return err
 }
 
-func (dec *propertiesDecoder) Decode(rootYamlNode *yaml.Node) error {
+func (dec *propertiesDecoder) applyProperty(context Context, properties *properties.Properties, key string) error {
+	value, _ := properties.Get(key)
+	path := parsePropKey(key)
+
+	propertyComments := properties.GetComments(key)
+	if len(propertyComments) > 0 {
+		err := dec.applyPropertyComments(context, path, propertyComments)
+		if err != nil {
+			return nil
+		}
+	}
+
+	rhsNode := createStringScalarNode(value)
+	rhsNode.Tag = rhsNode.guessTagFromCustomType()
+
+	return dec.d.DeeplyAssign(context, path, rhsNode)
+}
+
+func (dec *propertiesDecoder) Decode() (*CandidateNode, error) {
 	if dec.finished {
-		return io.EOF
+		return nil, io.EOF
 	}
 	buf := new(bytes.Buffer)
 
 	if _, err := buf.ReadFrom(dec.reader); err != nil {
-		return err
+		return nil, err
 	}
 	if buf.Len() == 0 {
 		dec.finished = true
-		return io.EOF
+		return nil, io.EOF
 	}
 	properties, err := properties.LoadString(buf.String())
 	if err != nil {
-		return err
+		return nil, err
 	}
 	properties.DisableExpansion = true
 
 	rootMap := &CandidateNode{
-		Node: &yaml.Node{
-			Kind: yaml.MappingNode,
-			Tag:  "!!map",
-		},
+		Kind: MappingNode,
+		Tag:  "!!map",
 	}
 
 	context := Context{}
 	context = context.SingleChildContext(rootMap)
 
 	for _, key := range properties.Keys() {
-		if err := dec.applyProperty(properties, context, key); err != nil {
-			return err
+		if err := dec.applyProperty(context, properties, key); err != nil {
+			return nil, err
 		}
 
 	}
-
-	rootYamlNode.Kind = yaml.DocumentNode
-	rootYamlNode.Content = []*yaml.Node{rootMap.Node}
 	dec.finished = true
-	return nil
+
+	return rootMap, nil
 
 }

pkg/yqlib/decoder_test.go

@@ -3,6 +3,7 @@ package yqlib
 import (
 	"bufio"
 	"bytes"
+	"fmt"
 	"strings"
 )
 
@@ -15,22 +16,25 @@ type formatScenario struct {
 	subdescription string
 	skipDoc        bool
 	scenarioType   string
+	expectedError  string
 }
 
-func processFormatScenario(s formatScenario, decoder Decoder, encoder Encoder) string {
-
+func processFormatScenario(s formatScenario, decoder Decoder, encoder Encoder) (string, error) {
 	var output bytes.Buffer
 	writer := bufio.NewWriter(&output)
 
 	if decoder == nil {
-		decoder = NewYamlDecoder()
+		decoder = NewYamlDecoder(ConfiguredYamlPreferences)
 	}
 
+	log.Debugf("reading docs")
 	inputs, err := readDocuments(strings.NewReader(s.input), "sample.yml", 0, decoder)
 	if err != nil {
-		panic(err)
+		return "", err
 	}
 
+	log.Debugf("done reading the documents")
+
 	expression := s.expression
 	if expression == "" {
 		expression = "."
@@ -39,22 +43,34 @@ func processFormatScenario(s formatScenario, decoder Decoder, encoder Encoder) s
 	exp, err := getExpressionParser().ParseExpression(expression)
 
 	if err != nil {
-		panic(err)
+		return "", err
 	}
 
 	context, err := NewDataTreeNavigator().GetMatchingNodes(Context{MatchingNodes: inputs}, exp)
 
+	log.Debugf("Going to print: %v", NodesToString(context.MatchingNodes))
+
 	if err != nil {
-		panic(err)
+		return "", err
 	}
 
 	printer := NewPrinter(encoder, NewSinglePrinterWriter(writer))
 	err = printer.PrintResults(context.MatchingNodes)
 	if err != nil {
-		panic(err)
+		return "", err
 	}
 	writer.Flush()
 
-	return output.String()
+	return output.String(), nil
+}
+
+func mustProcessFormatScenario(s formatScenario, decoder Decoder, encoder Encoder) string {
+
+	result, err := processFormatScenario(s, decoder, encoder)
+	if err != nil {
+		log.Error("Bad scenario %v: %w", s.description, err)
+		return fmt.Sprintf("Bad scenario %v: %v", s.description, err.Error())
+	}
+	return result
 
 }

pkg/yqlib/decoder_toml.go

@@ -0,0 +1,358 @@
+//go:build !yq_notoml
+
+package yqlib
+
+import (
+	"bytes"
+	"errors"
+	"fmt"
+	"io"
+	"strconv"
+	"time"
+
+	toml "github.com/pelletier/go-toml/v2/unstable"
+)
+
+type tomlDecoder struct {
+	parser   toml.Parser
+	finished bool
+	d        DataTreeNavigator
+	rootMap  *CandidateNode
+}
+
+func NewTomlDecoder() Decoder {
+	return &tomlDecoder{
+		finished: false,
+		d:        NewDataTreeNavigator(),
+	}
+}
+
+func (dec *tomlDecoder) Init(reader io.Reader) error {
+	dec.parser = toml.Parser{}
+	buf := new(bytes.Buffer)
+	_, err := buf.ReadFrom(reader)
+	if err != nil {
+		return err
+	}
+	dec.parser.Reset(buf.Bytes())
+	dec.rootMap = &CandidateNode{
+		Kind: MappingNode,
+		Tag:  "!!map",
+	}
+	return nil
+}
+
+func (dec *tomlDecoder) getFullPath(tomlNode *toml.Node) []interface{} {
+	path := make([]interface{}, 0)
+	for {
+		path = append(path, string(tomlNode.Data))
+		tomlNode = tomlNode.Next()
+		if tomlNode == nil {
+			return path
+		}
+	}
+}
+
+func (dec *tomlDecoder) processKeyValueIntoMap(rootMap *CandidateNode, tomlNode *toml.Node) error {
+	value := tomlNode.Value()
+	path := dec.getFullPath(value.Next())
+	log.Debug("processKeyValueIntoMap: %v", path)
+
+	valueNode, err := dec.decodeNode(value)
+	if err != nil {
+		return err
+	}
+
+	context := Context{}
+	context = context.SingleChildContext(rootMap)
+
+	return dec.d.DeeplyAssign(context, path, valueNode)
+}
+
+func (dec *tomlDecoder) decodeKeyValuesIntoMap(rootMap *CandidateNode, tomlNode *toml.Node) (bool, error) {
+	log.Debug("decodeKeyValuesIntoMap -- processing first (current) entry")
+	if err := dec.processKeyValueIntoMap(rootMap, tomlNode); err != nil {
+		return false, err
+	}
+
+	for dec.parser.NextExpression() {
+		nextItem := dec.parser.Expression()
+		log.Debug("decodeKeyValuesIntoMap -- next exp, its a %v", nextItem.Kind)
+
+		if nextItem.Kind == toml.KeyValue {
+			if err := dec.processKeyValueIntoMap(rootMap, nextItem); err != nil {
+				return false, err
+			}
+		} else {
+			// run out of key values
+			log.Debug("done in decodeKeyValuesIntoMap, gota a %v", nextItem.Kind)
+			return true, nil
+		}
+	}
+	log.Debug("no more things to read in")
+	return false, nil
+}
+
+func (dec *tomlDecoder) createInlineTableMap(tomlNode *toml.Node) (*CandidateNode, error) {
+	content := make([]*CandidateNode, 0)
+	log.Debug("createInlineTableMap")
+
+	iterator := tomlNode.Children()
+	for iterator.Next() {
+		child := iterator.Node()
+		if child.Kind != toml.KeyValue {
+			return nil, fmt.Errorf("only keyvalue pairs are supported in inlinetables, got %v instead", child.Kind)
+		}
+
+		keyValues := &CandidateNode{
+			Kind: MappingNode,
+			Tag:  "!!map",
+		}
+
+		if err := dec.processKeyValueIntoMap(keyValues, child); err != nil {
+			return nil, err
+		}
+
+		content = append(content, keyValues.Content...)
+	}
+
+	return &CandidateNode{
+		Kind:    MappingNode,
+		Tag:     "!!map",
+		Content: content,
+	}, nil
+}
+
+func (dec *tomlDecoder) createArray(tomlNode *toml.Node) (*CandidateNode, error) {
+	content := make([]*CandidateNode, 0)
+	iterator := tomlNode.Children()
+	for iterator.Next() {
+		child := iterator.Node()
+		yamlNode, err := dec.decodeNode(child)
+		if err != nil {
+			return nil, err
+		}
+		content = append(content, yamlNode)
+	}
+
+	return &CandidateNode{
+		Kind:    SequenceNode,
+		Tag:     "!!seq",
+		Content: content,
+	}, nil
+
+}
+
+func (dec *tomlDecoder) createStringScalar(tomlNode *toml.Node) (*CandidateNode, error) {
+	content := string(tomlNode.Data)
+	return createScalarNode(content, content), nil
+}
+
+func (dec *tomlDecoder) createBoolScalar(tomlNode *toml.Node) (*CandidateNode, error) {
+	content := string(tomlNode.Data)
+	return createScalarNode(content == "true", content), nil
+}
+
+func (dec *tomlDecoder) createIntegerScalar(tomlNode *toml.Node) (*CandidateNode, error) {
+	content := string(tomlNode.Data)
+	_, num, err := parseInt64(content)
+	return createScalarNode(num, content), err
+}
+
+func (dec *tomlDecoder) createDateTimeScalar(tomlNode *toml.Node) (*CandidateNode, error) {
+	content := string(tomlNode.Data)
+	val, err := parseDateTime(time.RFC3339, content)
+	return createScalarNode(val, content), err
+}
+
+func (dec *tomlDecoder) createFloatScalar(tomlNode *toml.Node) (*CandidateNode, error) {
+	content := string(tomlNode.Data)
+	num, err := strconv.ParseFloat(content, 64)
+	return createScalarNode(num, content), err
+}
+
+func (dec *tomlDecoder) decodeNode(tomlNode *toml.Node) (*CandidateNode, error) {
+	switch tomlNode.Kind {
+	case toml.Key, toml.String:
+		return dec.createStringScalar(tomlNode)
+	case toml.Bool:
+		return dec.createBoolScalar(tomlNode)
+	case toml.Integer:
+		return dec.createIntegerScalar(tomlNode)
+	case toml.DateTime:
+		return dec.createDateTimeScalar(tomlNode)
+	case toml.Float:
+		return dec.createFloatScalar(tomlNode)
+	case toml.Array:
+		return dec.createArray(tomlNode)
+	case toml.InlineTable:
+		return dec.createInlineTableMap(tomlNode)
+	default:
+		return nil, fmt.Errorf("unsupported type %v", tomlNode.Kind)
+	}
+
+}
+
+func (dec *tomlDecoder) Decode() (*CandidateNode, error) {
+	if dec.finished {
+		return nil, io.EOF
+	}
+	//
+	// toml library likes to panic
+	var deferredError error
+	defer func() { //catch or finally
+		if r := recover(); r != nil {
+			var ok bool
+			deferredError, ok = r.(error)
+			if !ok {
+				deferredError = fmt.Errorf("pkg: %v", r)
+			}
+		}
+	}()
+
+	log.Debug("ok here we go")
+	var runAgainstCurrentExp = false
+	var err error
+	for runAgainstCurrentExp || dec.parser.NextExpression() {
+
+		if runAgainstCurrentExp {
+			log.Debug("running against current exp")
+		}
+
+		currentNode := dec.parser.Expression()
+
+		log.Debug("currentNode: %v ", currentNode.Kind)
+		runAgainstCurrentExp, err = dec.processTopLevelNode(currentNode)
+		if err != nil {
+			return dec.rootMap, err
+		}
+
+	}
+
+	err = dec.parser.Error()
+	if err != nil {
+		return nil, err
+	}
+
+	// must have finished
+	dec.finished = true
+
+	if len(dec.rootMap.Content) == 0 {
+		return nil, io.EOF
+	}
+
+	return dec.rootMap, deferredError
+
+}
+
+func (dec *tomlDecoder) processTopLevelNode(currentNode *toml.Node) (bool, error) {
+	var runAgainstCurrentExp bool
+	var err error
+	log.Debug("processTopLevelNode: Going to process %v state is current %v", currentNode.Kind, NodeToString(dec.rootMap))
+	switch currentNode.Kind {
+	case toml.Table:
+		runAgainstCurrentExp, err = dec.processTable(currentNode)
+	case toml.ArrayTable:
+		runAgainstCurrentExp, err = dec.processArrayTable(currentNode)
+	default:
+		runAgainstCurrentExp, err = dec.decodeKeyValuesIntoMap(dec.rootMap, currentNode)
+	}
+
+	log.Debug("processTopLevelNode: DONE Processing state is now %v", NodeToString(dec.rootMap))
+	return runAgainstCurrentExp, err
+}
+
+func (dec *tomlDecoder) processTable(currentNode *toml.Node) (bool, error) {
+	log.Debug("Enter processTable")
+	fullPath := dec.getFullPath(currentNode.Child())
+	log.Debug("fullpath: %v", fullPath)
+
+	tableNodeValue := &CandidateNode{
+		Kind:    MappingNode,
+		Tag:     "!!map",
+		Content: make([]*CandidateNode, 0),
+	}
+
+	var tableValue *toml.Node
+	runAgainstCurrentExp := false
+	var err error
+	hasValue := dec.parser.NextExpression()
+	// check to see if there is any table data
+	if hasValue {
+		tableValue = dec.parser.Expression()
+		// next expression is not table data, so we are done
+		if tableValue.Kind != toml.KeyValue {
+			log.Debug("got an empty table, returning")
+			return true, nil
+		}
+
+		runAgainstCurrentExp, err = dec.decodeKeyValuesIntoMap(tableNodeValue, tableValue)
+		if err != nil && !errors.Is(err, io.EOF) {
+			return false, err
+		}
+	}
+
+	c := Context{}
+	c = c.SingleChildContext(dec.rootMap)
+	err = dec.d.DeeplyAssign(c, fullPath, tableNodeValue)
+	if err != nil {
+		return false, err
+	}
+	return runAgainstCurrentExp, nil
+}
+
+func (dec *tomlDecoder) arrayAppend(context Context, path []interface{}, rhsNode *CandidateNode) error {
+	log.Debug("arrayAppend to path: %v,%v", path, NodeToString(rhsNode))
+	rhsCandidateNode := &CandidateNode{
+		Kind:    SequenceNode,
+		Tag:     "!!seq",
+		Content: []*CandidateNode{rhsNode},
+	}
+
+	assignmentOp := &Operation{OperationType: addAssignOpType}
+
+	rhsOp := &Operation{OperationType: valueOpType, CandidateNode: rhsCandidateNode}
+
+	assignmentOpNode := &ExpressionNode{
+		Operation: assignmentOp,
+		LHS:       createTraversalTree(path, traversePreferences{}, false),
+		RHS:       &ExpressionNode{Operation: rhsOp},
+	}
+
+	_, err := dec.d.GetMatchingNodes(context, assignmentOpNode)
+	return err
+}
+
+func (dec *tomlDecoder) processArrayTable(currentNode *toml.Node) (bool, error) {
+	log.Debug("Entering processArrayTable")
+	fullPath := dec.getFullPath(currentNode.Child())
+	log.Debug("Fullpath: %v", fullPath)
+
+	// need to use the array append exp to add another entry to
+	// this array: fullpath += [ thing ]
+
+	hasValue := dec.parser.NextExpression()
+	if !hasValue {
+		return false, fmt.Errorf("error retrieving table %v value: %w", fullPath, dec.parser.Error())
+	}
+
+	tableNodeValue := &CandidateNode{
+		Kind: MappingNode,
+		Tag:  "!!map",
+	}
+
+	tableValue := dec.parser.Expression()
+	runAgainstCurrentExp, err := dec.decodeKeyValuesIntoMap(tableNodeValue, tableValue)
+	log.Debugf("table node err: %w", err)
+	if err != nil && !errors.Is(err, io.EOF) {
+		return false, err
+	}
+	c := Context{}
+
+	c = c.SingleChildContext(dec.rootMap)
+
+	// += function
+	err = dec.arrayAppend(c, fullPath, tableNodeValue)
+
+	return runAgainstCurrentExp, err
+}

pkg/yqlib/decoder_uri.go

@@ -0,0 +1,54 @@
+//go:build !yq_nouri
+
+package yqlib
+
+import (
+	"bytes"
+	"io"
+	"net/url"
+)
+
+type uriDecoder struct {
+	reader       io.Reader
+	finished     bool
+	readAnything bool
+}
+
+func NewUriDecoder() Decoder {
+	return &uriDecoder{finished: false}
+}
+
+func (dec *uriDecoder) Init(reader io.Reader) error {
+	dec.reader = reader
+	dec.readAnything = false
+	dec.finished = false
+	return nil
+}
+
+func (dec *uriDecoder) Decode() (*CandidateNode, error) {
+	if dec.finished {
+		return nil, io.EOF
+	}
+
+	buf := new(bytes.Buffer)
+
+	if _, err := buf.ReadFrom(dec.reader); err != nil {
+		return nil, err
+	}
+	if buf.Len() == 0 {
+		dec.finished = true
+
+		// if we've read _only_ an empty string, lets return that
+		// otherwise if we've already read some bytes, and now we get
+		// an empty string, then we are done.
+		if dec.readAnything {
+			return nil, io.EOF
+		}
+	}
+	newValue, err := url.QueryUnescape(buf.String())
+	if err != nil {
+		return nil, err
+	}
+	dec.readAnything = true
+	return createStringScalarNode(newValue), nil
+}

pkg/yqlib/decoder_xml.go

@@ -1,94 +1,96 @@
+//go:build !yq_noxml
+
 package yqlib
 
 import (
 	"encoding/xml"
 	"errors"
+	"fmt"
 	"io"
+	"regexp"
 	"strings"
 	"unicode"
 
 	"golang.org/x/net/html/charset"
-	yaml "gopkg.in/yaml.v3"
 )
 
 type xmlDecoder struct {
-	reader          io.Reader
-	readAnything    bool
-	attributePrefix string
-	contentName     string
-	strictMode      bool
-	keepNamespace   bool
-	useRawToken     bool
-	finished        bool
+	reader       io.Reader
+	readAnything bool
+	finished     bool
+	prefs        XmlPreferences
 }
 
-func NewXMLDecoder(attributePrefix string, contentName string, strictMode bool, keepNamespace bool, useRawToken bool) Decoder {
-	if contentName == "" {
-		contentName = "content"
-	}
+func NewXMLDecoder(prefs XmlPreferences) Decoder {
 	return &xmlDecoder{
-		attributePrefix: attributePrefix,
-		contentName:     contentName,
-		finished:        false,
-		strictMode:      strictMode,
-		keepNamespace:   keepNamespace,
-		useRawToken:     useRawToken,
+		finished: false,
+		prefs:    prefs,
 	}
 }
 
-func (dec *xmlDecoder) Init(reader io.Reader) {
+func (dec *xmlDecoder) Init(reader io.Reader) error {
 	dec.reader = reader
 	dec.readAnything = false
 	dec.finished = false
+	return nil
 }
 
-func (dec *xmlDecoder) createSequence(nodes []*xmlNode) (*yaml.Node, error) {
-	yamlNode := &yaml.Node{Kind: yaml.SequenceNode, Tag: "!!seq"}
+func (dec *xmlDecoder) createSequence(nodes []*xmlNode) (*CandidateNode, error) {
+	yamlNode := &CandidateNode{Kind: SequenceNode, Tag: "!!seq"}
 	for _, child := range nodes {
 		yamlChild, err := dec.convertToYamlNode(child)
 		if err != nil {
 			return nil, err
 		}
-		yamlNode.Content = append(yamlNode.Content, yamlChild)
+		yamlNode.AddChild(yamlChild)
 	}
 
 	return yamlNode, nil
 }
 
+var decoderCommentPrefix = regexp.MustCompile(`(^|\n)([[:alpha:]])`)
+
 func (dec *xmlDecoder) processComment(c string) string {
 	if c == "" {
 		return ""
 	}
-	return "#" + strings.TrimRight(c, " ")
+	//need to replace "cat " with "# cat"
+	// "\ncat\n" with "\n cat\n"
+	// ensure non-empty comments starting with newline have a space in front
+
+	replacement := decoderCommentPrefix.ReplaceAllString(c, "$1 $2")
+	replacement = "#" + strings.ReplaceAll(strings.TrimRight(replacement, " "), "\n", "\n#")
+	return replacement
 }
 
-func (dec *xmlDecoder) createMap(n *xmlNode) (*yaml.Node, error) {
-	log.Debug("createMap: headC: %v, footC: %v", n.HeadComment, n.FootComment)
-	yamlNode := &yaml.Node{Kind: yaml.MappingNode, Tag: "!!map"}
+func (dec *xmlDecoder) createMap(n *xmlNode) (*CandidateNode, error) {
+	log.Debug("createMap: headC: %v, lineC: %v, footC: %v", n.HeadComment, n.LineComment, n.FootComment)
+	yamlNode := &CandidateNode{Kind: MappingNode, Tag: "!!map"}
 
 	if len(n.Data) > 0 {
-		label := dec.contentName
+		log.Debugf("creating content node for map: %v", dec.prefs.ContentName)
+		label := dec.prefs.ContentName
 		labelNode := createScalarNode(label, label)
 		labelNode.HeadComment = dec.processComment(n.HeadComment)
+		labelNode.LineComment = dec.processComment(n.LineComment)
 		labelNode.FootComment = dec.processComment(n.FootComment)
-		yamlNode.Content = append(yamlNode.Content, labelNode, createScalarNode(n.Data, n.Data))
+		yamlNode.AddKeyValueChild(labelNode, dec.createValueNodeFromData(n.Data))
 	}
 
 	for i, keyValuePair := range n.Children {
 		label := keyValuePair.K
 		children := keyValuePair.V
 		labelNode := createScalarNode(label, label)
-		var valueNode *yaml.Node
+		var valueNode *CandidateNode
 		var err error
 
 		if i == 0 {
+			log.Debugf("head comment here")
 			labelNode.HeadComment = dec.processComment(n.HeadComment)
 
 		}
-
-		// if i == len(n.Children)-1 {
+		log.Debugf("label=%v, i=%v, keyValuePair.FootComment: %v", label, i, keyValuePair.FootComment)
 		labelNode.FootComment = dec.processComment(keyValuePair.FootComment)
-		// }
 
 		log.Debug("len of children in %v is %v", label, len(children))
 		if len(children) > 1 {
@@ -101,62 +103,92 @@ func (dec *xmlDecoder) createMap(n *xmlNode) (*yaml.Node, error) {
 			// if the value is a scalar, the head comment of the scalar needs to go on the key?
 			// add tests for <z/> as well as multiple <ds> of inputXmlWithComments > yaml
 			if len(children[0].Children) == 0 && children[0].HeadComment != "" {
-				labelNode.HeadComment = labelNode.HeadComment + "\n" + strings.TrimSpace(children[0].HeadComment)
-				children[0].HeadComment = ""
+				if len(children[0].Data) > 0 {
+
+					log.Debug("scalar comment hack, currentlabel [%v]", labelNode.HeadComment)
+					labelNode.HeadComment = joinComments([]string{labelNode.HeadComment, strings.TrimSpace(children[0].HeadComment)}, "\n")
+					children[0].HeadComment = ""
+				} else {
+					// child is null, put the headComment as a linecomment for reasons
+					children[0].LineComment = children[0].HeadComment
+					children[0].HeadComment = ""
+				}
 			}
 			valueNode, err = dec.convertToYamlNode(children[0])
 			if err != nil {
 				return nil, err
 			}
 		}
-		yamlNode.Content = append(yamlNode.Content, labelNode, valueNode)
+		yamlNode.AddKeyValueChild(labelNode, valueNode)
 	}
 
 	return yamlNode, nil
 }
 
-func (dec *xmlDecoder) convertToYamlNode(n *xmlNode) (*yaml.Node, error) {
+func (dec *xmlDecoder) createValueNodeFromData(values []string) *CandidateNode {
+	switch len(values) {
+	case 0:
+		return createScalarNode(nil, "")
+	case 1:
+		return createScalarNode(values[0], values[0])
+	default:
+		content := make([]*CandidateNode, 0)
+		for _, value := range values {
+			content = append(content, createScalarNode(value, value))
+		}
+		return &CandidateNode{
+			Kind:    SequenceNode,
+			Tag:     "!!seq",
+			Content: content,
+		}
+	}
+}
+
+func (dec *xmlDecoder) convertToYamlNode(n *xmlNode) (*CandidateNode, error) {
 	if len(n.Children) > 0 {
 		return dec.createMap(n)
 	}
-	scalar := createScalarNode(n.Data, n.Data)
-	if n.Data == "" {
-		scalar = createScalarNode(nil, "")
-	}
-	log.Debug("scalar headC: %v, footC: %v", n.HeadComment, n.FootComment)
+
+	scalar := dec.createValueNodeFromData(n.Data)
+
+	log.Debug("scalar (%v), headC: %v, lineC: %v, footC: %v", scalar.Tag, n.HeadComment, n.LineComment, n.FootComment)
 	scalar.HeadComment = dec.processComment(n.HeadComment)
 	scalar.LineComment = dec.processComment(n.LineComment)
+	if scalar.Tag == "!!seq" {
+		scalar.Content[0].HeadComment = scalar.LineComment
+		scalar.LineComment = ""
+	}
+
 	scalar.FootComment = dec.processComment(n.FootComment)
 
 	return scalar, nil
 }
 
-func (dec *xmlDecoder) Decode(rootYamlNode *yaml.Node) error {
+func (dec *xmlDecoder) Decode() (*CandidateNode, error) {
 	if dec.finished {
-		return io.EOF
+		return nil, io.EOF
 	}
 	root := &xmlNode{}
 	// cant use xj - it doesn't keep map order.
 	err := dec.decodeXML(root)
 
 	if err != nil {
-		return err
+		return nil, err
 	}
 	firstNode, err := dec.convertToYamlNode(root)
 
 	if err != nil {
-		return err
+		return nil, err
 	} else if firstNode.Tag == "!!null" {
 		dec.finished = true
 		if dec.readAnything {
-			return io.EOF
+			return nil, io.EOF
 		}
 	}
 	dec.readAnything = true
-	rootYamlNode.Kind = yaml.DocumentNode
-	rootYamlNode.Content = []*yaml.Node{firstNode}
 	dec.finished = true
-	return nil
+
+	return firstNode, nil
 }
 
 type xmlNode struct {
@@ -164,7 +196,7 @@ type xmlNode struct {
 	HeadComment string
 	FootComment string
 	LineComment string
-	Data        string
+	Data        []string
 }
 
 type xmlChildrenKv struct {
@@ -205,10 +237,12 @@ type element struct {
 // of the map keys.
 func (dec *xmlDecoder) decodeXML(root *xmlNode) error {
 	xmlDec := xml.NewDecoder(dec.reader)
-	xmlDec.Strict = dec.strictMode
+	xmlDec.Strict = dec.prefs.StrictMode
 	// That will convert the charset if the provided XML is non-UTF-8
 	xmlDec.CharsetReader = charset.NewReaderLabel
 
+	started := false
+
 	// Create first element from the root node
 	elem := &element{
 		parent: nil,
@@ -216,7 +250,7 @@ func (dec *xmlDecoder) decodeXML(root *xmlNode) error {
 	}
 
 	getToken := func() (xml.Token, error) {
-		if dec.useRawToken {
+		if dec.prefs.UseRawToken {
 			return xmlDec.RawToken()
 		}
 		return xmlDec.Token()
@@ -244,21 +278,31 @@ func (dec *xmlDecoder) decodeXML(root *xmlNode) error {
 
 			// Extract attributes as children
 			for _, a := range se.Attr {
-				if dec.keepNamespace {
+				if dec.prefs.KeepNamespace {
 					if a.Name.Space != "" {
 						a.Name.Local = a.Name.Space + ":" + a.Name.Local
 					}
 				}
-				elem.n.AddChild(dec.attributePrefix+a.Name.Local, &xmlNode{Data: a.Value})
+				elem.n.AddChild(dec.prefs.AttributePrefix+a.Name.Local, &xmlNode{Data: []string{a.Value}})
 			}
 		case xml.CharData:
+
 			// Extract XML data (if any)
-			elem.n.Data = trimNonGraphic(string(se))
-			if elem.n.Data != "" {
+			newBit := trimNonGraphic(string(se))
+			if !started && len(newBit) > 0 {
+				return fmt.Errorf("invalid XML: Encountered chardata [%v] outside of XML node", newBit)
+			}
+
+			if len(newBit) > 0 {
+				elem.n.Data = append(elem.n.Data, newBit)
 				elem.state = "chardata"
 				log.Debug("chardata [%v] for %v", elem.n.Data, elem.label)
 			}
 		case xml.EndElement:
+			if elem == nil {
+				log.Debug("no element, probably bad xml")
+				continue
+			}
 			log.Debug("end element %v", elem.label)
 			elem.state = "finished"
 			// And add it to its parent list
@@ -271,18 +315,28 @@ func (dec *xmlDecoder) decodeXML(root *xmlNode) error {
 		case xml.Comment:
 
 			commentStr := string(xml.CharData(se))
-			if elem.state == "started" {
+			switch elem.state {
+			case "started":
 				applyFootComment(elem, commentStr)
 
-			} else if elem.state == "chardata" {
+			case "chardata":
 				log.Debug("got a line comment for (%v) %v: [%v]", elem.state, elem.label, commentStr)
-				elem.n.LineComment = joinFilter([]string{elem.n.LineComment, commentStr})
-			} else {
+				elem.n.LineComment = joinComments([]string{elem.n.LineComment, commentStr}, " ")
+			default:
 				log.Debug("got a head comment for (%v) %v: [%v]", elem.state, elem.label, commentStr)
-				elem.n.HeadComment = joinFilter([]string{elem.n.HeadComment, commentStr})
+				elem.n.HeadComment = joinComments([]string{elem.n.HeadComment, commentStr}, " ")
 			}
 
+		case xml.ProcInst:
+			if !dec.prefs.SkipProcInst {
+				elem.n.AddChild(dec.prefs.ProcInstPrefix+se.Target, &xmlNode{Data: []string{string(se.Inst)}})
+			}
+		case xml.Directive:
+			if !dec.prefs.SkipDirectives {
+				elem.n.AddChild(dec.prefs.DirectiveName, &xmlNode{Data: []string{string(se)}})
+			}
 		}
+		started = true
 	}
 
 	return nil
@@ -294,22 +348,28 @@ func applyFootComment(elem *element, commentStr string) {
 	if len(elem.n.Children) > 0 {
 		lastChildIndex := len(elem.n.Children) - 1
 		childKv := elem.n.Children[lastChildIndex]
-		log.Debug("got a foot comment for %v: [%v]", childKv.K, commentStr)
-		childKv.FootComment = joinFilter([]string{elem.n.FootComment, commentStr})
+		log.Debug("got a foot comment, putting on last child for %v: [%v]", childKv.K, commentStr)
+		// if it's an array of scalars, put the foot comment on the scalar itself
+		if len(childKv.V) > 0 && len(childKv.V[0].Children) == 0 {
+			nodeToUpdate := childKv.V[len(childKv.V)-1]
+			nodeToUpdate.FootComment = joinComments([]string{nodeToUpdate.FootComment, commentStr}, " ")
+		} else {
+			childKv.FootComment = joinComments([]string{elem.n.FootComment, commentStr}, " ")
+		}
 	} else {
 		log.Debug("got a foot comment for %v: [%v]", elem.label, commentStr)
-		elem.n.FootComment = joinFilter([]string{elem.n.FootComment, commentStr})
+		elem.n.FootComment = joinComments([]string{elem.n.FootComment, commentStr}, " ")
 	}
 }
 
-func joinFilter(rawStrings []string) string {
+func joinComments(rawStrings []string, joinStr string) string {
 	stringsToJoin := make([]string, 0)
 	for _, str := range rawStrings {
 		if str != "" {
 			stringsToJoin = append(stringsToJoin, str)
 		}
 	}
-	return strings.Join(stringsToJoin, " ")
+	return strings.Join(stringsToJoin, joinStr)
 }
 
 // trimNonGraphic returns a slice of the string s, with all leading and trailing

pkg/yqlib/decoder_yaml.go

@@ -1,23 +1,157 @@
 package yqlib
 
 import (
+	"bufio"
+	"bytes"
+	"errors"
 	"io"
+	"regexp"
+	"strings"
 
-	yaml "gopkg.in/yaml.v3"
+	yaml "go.yaml.in/yaml/v3"
 )
 
 type yamlDecoder struct {
 	decoder yaml.Decoder
+
+	prefs YamlPreferences
+
+	// work around of various parsing issues by yaml.v3 with document headers
+	leadingContent string
+	bufferRead     bytes.Buffer
+
+	// anchor map persists over multiple documents for convenience.
+	anchorMap map[string]*CandidateNode
+
+	readAnything  bool
+	firstFile     bool
+	documentIndex uint
 }
 
-func NewYamlDecoder() Decoder {
-	return &yamlDecoder{}
+func NewYamlDecoder(prefs YamlPreferences) Decoder {
+	return &yamlDecoder{prefs: prefs, firstFile: true}
 }
 
-func (dec *yamlDecoder) Init(reader io.Reader) {
-	dec.decoder = *yaml.NewDecoder(reader)
+func (dec *yamlDecoder) processReadStream(reader *bufio.Reader) (io.Reader, string, error) {
+	var commentLineRegEx = regexp.MustCompile(`^\s*#`)
+	var yamlDirectiveLineRegEx = regexp.MustCompile(`^\s*%YA`)
+	var sb strings.Builder
+	for {
+		peekBytes, err := reader.Peek(4)
+		if errors.Is(err, io.EOF) {
+			// EOF are handled else where..
+			return reader, sb.String(), nil
+		} else if err != nil {
+			return reader, sb.String(), err
+		} else if string(peekBytes[0]) == "\n" {
+			_, err := reader.ReadString('\n')
+			sb.WriteString("\n")
+			if errors.Is(err, io.EOF) {
+				return reader, sb.String(), nil
+			} else if err != nil {
+				return reader, sb.String(), err
+			}
+		} else if string(peekBytes) == "--- " {
+			_, err := reader.ReadString(' ')
+			sb.WriteString("$yqDocSeparator$\n")
+			if errors.Is(err, io.EOF) {
+				return reader, sb.String(), nil
+			} else if err != nil {
+				return reader, sb.String(), err
+			}
+		} else if string(peekBytes) == "---\n" {
+			_, err := reader.ReadString('\n')
+			sb.WriteString("$yqDocSeparator$\n")
+			if errors.Is(err, io.EOF) {
+				return reader, sb.String(), nil
+			} else if err != nil {
+				return reader, sb.String(), err
+			}
+		} else if commentLineRegEx.MatchString(string(peekBytes)) || yamlDirectiveLineRegEx.MatchString(string(peekBytes)) {
+			line, err := reader.ReadString('\n')
+			sb.WriteString(line)
+			if errors.Is(err, io.EOF) {
+				return reader, sb.String(), nil
+			} else if err != nil {
+				return reader, sb.String(), err
+			}
+		} else {
+			return reader, sb.String(), nil
+		}
+	}
 }
 
-func (dec *yamlDecoder) Decode(rootYamlNode *yaml.Node) error {
-	return dec.decoder.Decode(rootYamlNode)
+func (dec *yamlDecoder) Init(reader io.Reader) error {
+	readerToUse := reader
+	leadingContent := ""
+	dec.bufferRead = bytes.Buffer{}
+	var err error
+	// if we 'evaluating together' - we only process the leading content
+	// of the first file - this ensures comments from subsequent files are
+	// merged together correctly.
+	if dec.prefs.LeadingContentPreProcessing && (!dec.prefs.EvaluateTogether || dec.firstFile) {
+		readerToUse, leadingContent, err = dec.processReadStream(bufio.NewReader(reader))
+		if err != nil {
+			return err
+		}
+	} else if !dec.prefs.LeadingContentPreProcessing {
+		// if we're not process the leading content
+		// keep a copy of what we've read. This is incase its a
+		// doc with only comments - the decoder will return nothing
+		// then we can read the comments from bufferRead
+		readerToUse = io.TeeReader(reader, &dec.bufferRead)
+	}
+	dec.leadingContent = leadingContent
+	dec.readAnything = false
+	dec.decoder = *yaml.NewDecoder(readerToUse)
+	dec.firstFile = false
+	dec.documentIndex = 0
+	dec.anchorMap = make(map[string]*CandidateNode)
+	return nil
+}
+
+func (dec *yamlDecoder) Decode() (*CandidateNode, error) {
+	var yamlNode yaml.Node
+	err := dec.decoder.Decode(&yamlNode)
+
+	if errors.Is(err, io.EOF) && dec.leadingContent != "" && !dec.readAnything {
+		// force returning an empty node with a comment.
+		dec.readAnything = true
+		return dec.blankNodeWithComment(), nil
+	} else if errors.Is(err, io.EOF) && !dec.prefs.LeadingContentPreProcessing && !dec.readAnything {
+		// didn't find any yaml,
+		// check the tee buffer, maybe there were comments
+		dec.readAnything = true
+		dec.leadingContent = dec.bufferRead.String()
+		if dec.leadingContent != "" {
+			return dec.blankNodeWithComment(), nil
+		}
+		return nil, err
+	} else if err != nil {
+		return nil, err
+	}
+
+	candidateNode := CandidateNode{document: dec.documentIndex}
+	// don't bother with the DocumentNode
+	err = candidateNode.UnmarshalYAML(yamlNode.Content[0], dec.anchorMap)
+	if err != nil {
+		return nil, err
+	}
+
+	candidateNode.HeadComment = yamlNode.HeadComment + candidateNode.HeadComment
+	candidateNode.FootComment = yamlNode.FootComment + candidateNode.FootComment
+
+	if dec.leadingContent != "" {
+		candidateNode.LeadingContent = dec.leadingContent
+		dec.leadingContent = ""
+	}
+	dec.readAnything = true
+	dec.documentIndex++
+	return &candidateNode, nil
+}
+
+func (dec *yamlDecoder) blankNodeWithComment() *CandidateNode {
+	node := createScalarNode(nil, "")
+	node.LeadingContent = dec.leadingContent
+	return node
 }

pkg/yqlib/doc/notification-snippet.md

@@ -1,6 +0,0 @@
-
-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified. 
-
-`yq e <exp> <file>`
-{% endhint %}

pkg/yqlib/doc/operators/add.md

@@ -9,12 +9,6 @@ Add behaves differently according to the type of the LHS:
 Use `+=` as a relative append assign for things like increment. Note that `.a += .x` is equivalent to running `.a = .a + .x`.
 
 
-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified. 
-
-`yq e <exp> <file>`
-{% endhint %}
-
 ## Concatenate arrays
 Given a sample.yml file of:
 ```yaml
@@ -92,6 +86,23 @@ will output
 a: ['dog', 'cat']
 ```
 
+## Prepend to existing array
+Given a sample.yml file of:
+```yaml
+a:
+  - dog
+```
+then
+```bash
+yq '.a = ["cat"] + .a' sample.yml
+```
+will output
+```yaml
+a:
+  - cat
+  - dog
+```
+
 ## Add new object to array
 Given a sample.yml file of:
 ```yaml

pkg/yqlib/doc/operators/alternative-default-value.md

@@ -2,12 +2,6 @@
 
 This operator is used to provide alternative (or default) values when a particular expression is either null or false.
 
-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified. 
-
-`yq e <exp> <file>`
-{% endhint %}
-
 ## LHS is defined
 Given a sample.yml file of:
 ```yaml

pkg/yqlib/doc/operators/anchor-and-alias-operators.md

@@ -5,12 +5,6 @@ Use the `alias` and `anchor` operators to read and write yaml aliases and anchor
 `yq` supports merge aliases (like `<<: *blah`) however this is no longer in the standard yaml spec (1.2) and so `yq` will automatically add the `!!merge` tag to these nodes as it is effectively a custom tag.
 
 
-{% hint style="warning" %}
-Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#x20;
-
-`yq e <exp> <file>`
-{% endhint %}
-
 ## Merge one map
 see https://yaml.org/type/merge.html
 
@@ -40,68 +34,6 @@ y: 2
 r: 10
 ```
 
-## Merge multiple maps
-see https://yaml.org/type/merge.html
-
-Given a sample.yml file of:
-```yaml
-- &CENTER
-  x: 1
-  y: 2
-- &LEFT
-  x: 0
-  y: 2
-- &BIG
-  r: 10
-- &SMALL
-  r: 1
-- !!merge <<:
-    - *CENTER
-    - *BIG
-```
-then
-```bash
-yq '.[4] | explode(.)' sample.yml
-```
-will output
-```yaml
-r: 10
-x: 1
-y: 2
-```
-
-## Override
-see https://yaml.org/type/merge.html
-
-Given a sample.yml file of:
-```yaml
-- &CENTER
-  x: 1
-  y: 2
-- &LEFT
-  x: 0
-  y: 2
-- &BIG
-  r: 10
-- &SMALL
-  r: 1
-- !!merge <<:
-    - *BIG
-    - *LEFT
-    - *SMALL
-  x: 1
-```
-then
-```bash
-yq '.[4] | explode(.)' sample.yml
-```
-will output
-```yaml
-r: 10
-x: 1
-y: 2
-```
-
 ## Get anchor
 Given a sample.yml file of:
 ```yaml
@@ -260,7 +192,101 @@ f:
   cat: b
 ```
 
-## Explode with merge anchors
+## Dereference and update a field
+Use explode with multiply to dereference an object
+
+Given a sample.yml file of:
+```yaml
+item_value: &item_value
+  value: true
+thingOne:
+  name: item_1
+  !!merge <<: *item_value
+thingTwo:
+  name: item_2
+  !!merge <<: *item_value
+```
+then
+```bash
+yq '.thingOne |= explode(.) * {"value": false}' sample.yml
+```
+will output
+```yaml
+item_value: &item_value
+  value: true
+thingOne:
+  name: item_1
+  value: false
+thingTwo:
+  name: item_2
+  !!merge <<: *item_value
+```
+
+## Merge multiple maps
+see https://yaml.org/type/merge.html
+
+Given a sample.yml file of:
+```yaml
+- &CENTER
+  x: 1
+  y: 2
+- &LEFT
+  x: 0
+  y: 2
+- &BIG
+  r: 10
+- &SMALL
+  r: 1
+- !!merge <<:
+    - *CENTER
+    - *BIG
+```
+then
+```bash
+yq '.[4] | explode(.)' sample.yml
+```
+will output
+```yaml
+r: 10
+x: 1
+y: 2
+```
+
+## Override
+see https://yaml.org/type/merge.html
+
+Given a sample.yml file of:
+```yaml
+- &CENTER
+  x: 1
+  y: 2
+- &LEFT
+  x: 0
+  y: 2
+- &BIG
+  r: 10
+- &SMALL
+  r: 1
+- !!merge <<:
+    - *BIG
+    - *LEFT
+    - *SMALL
+  x: 1
+```
+then
+```bash
+yq '.[4] | explode(.)' sample.yml
+```
+will output
+```yaml
+r: 10
+x: 1
+y: 2
+```
+
+## LEGACY: Explode with merge anchors
+Caution: this is for when --yaml-fix-merge-anchor-to-spec=false; it's not to YAML spec because the merge anchors incorrectly override the object values. Flag will default to true in late 2025
+
 Given a sample.yml file of:
 ```yaml
 foo: &foo
@@ -307,33 +333,52 @@ foobar:
   thing: foobar_thing
 ```
 
-## Dereference and update a field
-`Use explode with multiply to dereference an object
+## FIXED: Explode with merge anchors
+Set `--yaml-fix-merge-anchor-to-spec=true` to get this correct merge behaviour. Flag will default to true in late 2025 
 
 Given a sample.yml file of:
 ```yaml
-item_value: &item_value
-  value: true
-thingOne:
-  name: item_1
-  !!merge <<: *item_value
-thingTwo:
-  name: item_2
-  !!merge <<: *item_value
+foo: &foo
+  a: foo_a
+  thing: foo_thing
+  c: foo_c
+bar: &bar
+  b: bar_b
+  thing: bar_thing
+  c: bar_c
+foobarList:
+  b: foobarList_b
+  !!merge <<:
+    - *foo
+    - *bar
+  c: foobarList_c
+foobar:
+  c: foobar_c
+  !!merge <<: *foo
+  thing: foobar_thing
 ```
 then
 ```bash
-yq '.thingOne |= explode(.) * {"value": false}' sample.yml
+yq 'explode(.)' sample.yml
 ```
 will output
 ```yaml
-item_value: &item_value
-  value: true
-thingOne:
-  name: item_1
-  value: false
-thingTwo:
-  name: item_2
-  !!merge <<: *item_value
+foo:
+  a: foo_a
+  thing: foo_thing
+  c: foo_c
+bar:
+  b: bar_b
+  thing: bar_thing
+  c: bar_c
+foobarList:
+  b: foobarList_b
+  a: foo_a
+  thing: foo_thing
+  c: foobarList_c
+foobar:
+  c: foobar_c
+  a: foo_a
+  thing: foobar_thing
 ```