Bumping version

Bumps [golang.org/x/net](https://github.com/golang/net) from 0.47.0 to 0.48.0.
- [Commits](https://github.com/golang/net/compare/v0.47.0...v0.48.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-version: 0.48.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

.github/ISSUE_TEMPLATE/bug_report_v3.md

@@ -1,51 +0,0 @@
----
-name: Bug report - V3
-about: Create a report to help us improve
-title: ''
-labels: bug, v3
-assignees: ''
-
----
-
-**Describe the bug**
-A clear and concise description of what the bug is.
-
-Note that any how to questions should be posted in the discussion board and not raised as an issue.
-
-Version of yq: 3.X.X
-Operating system: mac/linux/windows/....
-Installed via: docker/binary release/homebrew/snap/...
-
-**Input Yaml**
-Concise yaml document(s) (as simple as possible to show the bug, please keep it to 10 lines or less)
-data1.yml:
-```yaml
-this: should really work
-```
-
-data2.yml:
-```yaml
-but: it strangely didn't
-```
-
-**Command**
-The command you ran:
-```
-yq merge data1.yml data2.yml
-```
-
-**Actual behavior**
-
-```yaml
-cat: meow
-```
-
-**Expected behavior**
-
-```yaml
-this: should really work
-but: it strangely didn't
-```
-
-**Additional context**
-Add any other context about the problem here.

.github/workflows/codeql.yml

@@ -38,11 +38,11 @@ jobs:
 
     steps:
     - name: Checkout repository
-      uses: actions/checkout@v4
+      uses: actions/checkout@v6
 
     # Initializes the CodeQL tools for scanning.
     - name: Initialize CodeQL
-      uses: github/codeql-action/init@v3
+      uses: github/codeql-action/init@v4
       with:
         languages: ${{ matrix.language }}
         # If you wish to specify custom queries, you can do so here or in a config file.
@@ -53,7 +53,7 @@ jobs:
     # Autobuild attempts to build any compiled languages  (C/C++, C#, or Java).
     # If this step fails, then you should remove it and run the build manually (see below)
     - name: Autobuild
-      uses: github/codeql-action/autobuild@v3
+      uses: github/codeql-action/autobuild@v4
 
     # ℹ️ Command-line programs to run using the OS shell.
     # 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun
@@ -67,4 +67,4 @@ jobs:
     #   make release
 
     - name: Perform CodeQL Analysis
-      uses: github/codeql-action/analyze@v3
+      uses: github/codeql-action/analyze@v4

.github/workflows/docker-release.yml

@@ -14,7 +14,7 @@ jobs:
       IMAGE_NAME: mikefarah/yq
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@v6
 
       - name: Set up QEMU
         uses: docker/setup-qemu-action@v3
@@ -30,6 +30,19 @@ jobs:
       - name: Available platforms
         run: echo ${{ steps.buildx.outputs.platforms }} && docker version
 
+      - name: Login to Docker Hub
+        uses: docker/login-action@v3
+        with:
+          username: ${{ secrets.DOCKER_USERNAME }}
+          password: ${{ secrets.DOCKER_PASSWORD }}
+
+      - name: Login to GitHub Container Registry
+        uses: docker/login-action@v3
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+
       - name: Build and push image
         run: |
           echo "GithubRef: ${GITHUB_REF}"
@@ -38,10 +51,9 @@ jobs:
           IMAGE_VERSION=${VERSION:1}
           echo "IMAGE_VERSION: ${IMAGE_VERSION}"
 
-          PLATFORMS="linux/amd64,linux/ppc64le,linux/arm64"
+          PLATFORMS="linux/amd64,linux/ppc64le,linux/arm64,linux/arm/v7,linux/s390x"
 
           echo "Building and pushing version ${IMAGE_VERSION} of image ${IMAGE_NAME}"
-          echo '${{ secrets.DOCKER_PASSWORD }}' | docker login -u '${{ secrets.DOCKER_USERNAME }}' --password-stdin
           docker buildx build \
             --label "org.opencontainers.image.authors=https://github.com/mikefarah/yq/graphs/contributors" \
             --label "org.opencontainers.image.created=$(date --rfc-3339=seconds)" \
@@ -59,8 +71,11 @@ jobs:
             -t "${IMAGE_NAME}:${IMAGE_VERSION}" \
             -t "${IMAGE_NAME}:4" \
             -t "${IMAGE_NAME}:latest" \
+            -t "ghcr.io/${IMAGE_NAME}:${IMAGE_VERSION}" \
+            -t "ghcr.io/${IMAGE_NAME}:4" \
+            -t "ghcr.io/${IMAGE_NAME}:latest" \
             .
-          
+
           cd github-action
           docker buildx build \
             --label "org.opencontainers.image.authors=https://github.com/mikefarah/yq/graphs/contributors" \
@@ -79,4 +94,7 @@ jobs:
             -t "${IMAGE_NAME}:${IMAGE_VERSION}-githubaction" \
             -t "${IMAGE_NAME}:4-githubaction" \
             -t "${IMAGE_NAME}:latest-githubaction" \
+            -t "ghcr.io/${IMAGE_NAME}:${IMAGE_VERSION}-githubaction" \
+            -t "ghcr.io/${IMAGE_NAME}:4-githubaction" \
+            -t "ghcr.io/${IMAGE_NAME}:latest-githubaction" \
             .

.github/workflows/go.yml

@@ -17,7 +17,7 @@ jobs:
       id: go
 
     - name: Check out code into the Go module directory
-      uses: actions/checkout@v4
+      uses: actions/checkout@v6
 
     - name: Get dependencies
       run: |

.github/workflows/release.yml

@@ -9,7 +9,7 @@ jobs:
   publishGitRelease:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@v6
       - uses: actions/setup-go@v5
         with:
           go-version: '^1.20'
@@ -40,9 +40,7 @@ jobs:
       - name: Cross compile
         run: |
           sudo apt-get install rhash -y
-          go install github.com/mitchellh/gox@v1.0.1
-          mkdir -p build
-          cp yq.1 build/yq.1
+          go install github.com/goreleaser/goreleaser/v2@latest
           ./scripts/xcompile.sh
 
       - name: Release

.github/workflows/snap-release.yml

@@ -12,12 +12,16 @@ jobs:
       environment: snap
       runs-on: ubuntu-latest
       steps:
-        - uses: actions/checkout@v4
+        - uses: actions/checkout@v6
         - uses: snapcore/action-build@v1
           id: build
+          env:
+            SNAPCRAFT_STORE_CREDENTIALS: ${{ secrets.STORE_LOGIN }}
+          with:
+            snapcraft-args: "remote-build --launchpad-accept-public-upload"
         - uses: snapcore/action-publish@v1
           env:
             SNAPCRAFT_STORE_CREDENTIALS: ${{ secrets.STORE_LOGIN }}
           with:
             snap: ${{ steps.build.outputs.snap }}
-            release: stable
+            release: stable

.github/workflows/test-yq.yml

@@ -13,7 +13,7 @@ jobs:
     runs-on: ubuntu-latest
 
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@v6
       - name: Get test
         id: get_value
         uses: mikefarah/yq@master

.gitignore

@@ -22,8 +22,10 @@ _cgo_gotypes.go
 _cgo_export.*
 
 _testmain.go
+cover.out
 coverage.out
 coverage.html
+coverage_sorted.txt
 *.exe
 *.test
 *.prof
@@ -44,6 +46,8 @@ test*.yml
 test*.xml
 test*.toml
 test*.yaml
+test_dir1/
+test_dir2/
 0.yml
 1.yml
 2.yml

.golangci.bck.yml

@@ -0,0 +1,38 @@
+run:
+  timeout: 5m
+linters:
+  enable:
+    - asciicheck
+    - depguard
+    - errorlint
+    - gci
+    - gochecknoinits
+    - gofmt
+    - goimports
+    - gosec
+    - gosimple
+    - staticcheck
+    - unused
+    - misspell
+    - nakedret
+    - nolintlint
+    - predeclared
+    - revive
+    - unconvert
+    - unparam
+linters-settings:
+  depguard:
+    rules:
+      prevent_unmaintained_packages:
+        list-mode: lax
+        files:
+          - $all
+          - "!$test"
+        deny:
+          - pkg: io/ioutil
+            desc: "replaced by io and os packages since Go 1.16: https://tip.golang.org/doc/go1.16#ioutil"
+issues:
+  exclude-rules:
+  - linters:
+    - revive
+    text: "var-naming"

.golangci.yml

@@ -1,16 +1,11 @@
-run:
-  timeout: 5m
+version: "2"
 linters:
   enable:
     - asciicheck
     - depguard
     - errorlint
-    - gci
     - gochecknoinits
-    - gofmt
-    - goimports
     - gosec
-    - megacheck
     - misspell
     - nakedret
     - nolintlint
@@ -18,26 +13,40 @@ linters:
     - revive
     - unconvert
     - unparam
-linters-settings:
-  depguard:
+  settings:
+    depguard:
+      rules:
+        prevent_unmaintained_packages:
+          list-mode: lax
+          files:
+            - $all
+            - '!$test'
+          deny:
+            - pkg: io/ioutil
+              desc: 'replaced by io and os packages since Go 1.16: https://tip.golang.org/doc/go1.16#ioutil'
+  exclusions:
+    generated: lax
+    presets:
+      - comments
+      - common-false-positives
+      - legacy
+      - std-error-handling
     rules:
-      prevent_unmaintained_packages:
-        list-mode: lax
-        files:
-          - $all
-          - "!$test"
-        deny:
-          - pkg: io/ioutil
-            desc: "replaced by io and os packages since Go 1.16: https://tip.golang.org/doc/go1.16#ioutil"
-issues:
-  exclude-rules:
-  - linters:
-    - gosec
-    text:  "Implicit memory aliasing in for loop."
-    path: _test\.go
-  - linters:
-    - revive
-    text: "unexported-return"
-  - linters:
-    - revive
-    text: "var-naming"
+      - linters:
+          - revive
+        text: var-naming
+    paths:
+      - third_party$
+      - builtin$
+      - examples$
+formatters:
+  enable:
+    - gci
+    - gofmt
+    - goimports
+  exclusions:
+    generated: lax
+    paths:
+      - third_party$
+      - builtin$
+      - examples$

.goreleaser.yaml

@@ -0,0 +1,49 @@
+version: 2
+
+dist: build
+
+builds:
+  - id: yq
+
+    binary: yq_{{ .Os }}_{{ .Arch }}
+
+    ldflags:
+      - -s -w
+
+    env:
+      - CGO_ENABLED=0
+
+    targets:
+      - darwin_amd64
+      - darwin_arm64
+      - freebsd_386
+      - freebsd_amd64
+      - freebsd_arm
+      - linux_386
+      - linux_amd64
+      - linux_arm
+      - linux_arm64
+      - linux_loong64
+      - linux_mips
+      - linux_mips64
+      - linux_mips64le
+      - linux_mipsle
+      - linux_ppc64
+      - linux_ppc64le
+      - linux_riscv64
+      - linux_s390x
+      - netbsd_386
+      - netbsd_amd64
+      - netbsd_arm
+      - openbsd_386
+      - openbsd_amd64
+      - windows_386
+      - windows_amd64
+      - windows_arm
+      - windows_arm64
+
+    no_unique_dist_dir: true
+
+release:
+  disable: true
+  skip_upload: true

CONTRIBUTING.md

@@ -1,40 +1,214 @@
+# Before you begin 
+Not all new PRs will be merged in 
+
+It's recommended to check with the owner first (e.g. raise an issue) to discuss a new feature before developing, to ensure your hard efforts don't go to waste.
+
+PRs to fix bugs and issues are almost always welcome :pray: please ensure you write tests as well.
+
+The following types of PRs will _not_ be accepted:
+- **Significant refactors** take a lot of time to understand and can have all sorts of unintended side effects. If you think there's a better way to do things (that requires significant changes) raise an issue for discussion first :)
+- **Release pipeline PRs** are a security risk - it's too easy for a serious vulnerability to sneak in (either intended or not). If there is a new cool way of releasing things, raise an issue for discussion first - it will need to be gone over with a fine tooth comb.
+- **Version bumps** are handled by dependabot, the bot will auto-raise PRs and they will be regularly merged in. 
+- **New release platforms** At this stage, yq is not going to maintain any other release platforms other than GitHub and Docker - that said, I'm more than happy to put in other community maintained methods in the README for visibility :heart:
+
+
 # Development
 
-1. Install (Golang)[https://golang.org/]
-1. Run  `scripts/devtools.sh` to install the required devtools
-2. Run `make [local] vendor` to install the vendor dependencies
-2. Run `make [local] test` to ensure you can run the existing tests
-3. Write unit tests - (see existing examples). Changes will not be accepted without corresponding unit tests.
-4. Make the code changes.
-5. `make [local] test` to lint code and run tests
-6. Profit! ok no profit, but raise a PR and get kudos :)
+## Initial Setup
 
+1. Install [Golang](https://golang.org/) (version 1.24.0 or later)
+2. Run `scripts/devtools.sh` to install required development tools:
+   - golangci-lint for code linting
+   - gosec for security analysis
+3. Run `make [local] vendor` to install vendor dependencies
+4. Run `make [local] test` to ensure you can run the existing tests
+
+## Development Workflow
+
+1. **Write unit tests first** - Changes will not be accepted without corresponding unit tests (see Testing section below)
+2. **Make your code changes**
+3. **Run tests and linting**: `make [local] test` (this runs formatting, linting, security checks, and tests)
+4. **Create your PR** and get kudos! :)
+
+## Make Commands
+
+- Use `make [local] <command>` for local development (runs in Docker container)
+- Use `make <command>` for CI/CD environments
+- Common commands:
+  - `make [local] vendor` - Install dependencies
+  - `make [local] test` - Run all checks and tests
+  - `make [local] build` - Build the yq binary
+  - `make [local] format` - Format code
+  - `make [local] check` - Run linting and security checks
+
+# Code Quality
+
+## Linting and Formatting
+
+The project uses strict linting rules defined in `.golangci.yml`. All code must pass:
+
+- **Code formatting**: gofmt, goimports, gci
+- **Linting**: revive, errorlint, gosec, misspell, and others
+- **Security checks**: gosec security analysis
+- **Spelling checks**: misspell detection
+
+Run `make [local] check` to verify your code meets all quality standards.
+
+## Code Style Guidelines
+
+- Follow standard Go conventions
+- Use meaningful variable names
+- Add comments for public functions and complex logic
+- Keep functions focused and reasonably sized
+- Use the project's existing patterns and conventions
+
+# Testing
+
+## Test Structure
+
+Tests in yq use the `expressionScenario` pattern. Each test scenario includes:
+- `expression`: The yq expression to test
+- `document`: Input YAML/JSON (optional)
+- `expected`: Expected output
+- `skipDoc`: Whether to skip documentation generation
+
+## Writing Tests
+
+1. **Find the appropriate test file** (e.g., `operator_add_test.go` for addition operations)
+2. **Add your test scenario** to the `*OperatorScenarios` slice
+3. **Run the specific test**: `go test -run TestAddOperatorScenarios` (replace with appropriate test name)
+4. **Verify documentation generation** (see Documentation section)
+
+## Test Examples
+
+```go
+var addOperatorScenarios = []expressionScenario{
+    {
+        skipDoc:    true,
+        expression: `"foo" + "bar"`,
+        expected: []string{
+            "D0, P[], (!!str)::foobar\n",
+        },
+    },
+    {
+        document:   "apples: 3",
+        expression: `.apples + 3`,
+        expected: []string{
+            "D0, P[apples], (!!int)::6\n",
+        },
+    },
+}
+```
+
+## Running Tests
+
+- **All tests**: `make [local] test`
+- **Specific test**: `go test -run TestName`
+- **With coverage**: `make [local] cover`
 
 # Documentation
 
-The documentation is a bit of a mixed bag (sorry in advance, I do plan on simplifying it...) - with some parts automatically generated and stiched together and some statically defined.
+## Documentation Generation
 
-Documentation is written in markdown, and is published in the 'gitbook' branch.
+The project uses a documentation system that combines static headers with dynamically generated content from tests.
 
-The various operator documentation (e.g. 'strings') are generated from the 'master' branch, and have a statically defined header (e.g. `pkg/yqlib/doc/operators/headers/add.md`) and the bulk of the docs are generated from the unit tests e.g. `pkg/yqlib/operator_add_test.go`.
+### How It Works
 
-The pipeline will run the tests and automatically concatenate the files together, and put them under 
-`pkg/qylib/doc/add.md`. These files are checked in the master branch (and are copied to the gitbook branch as part of the release process).
+1. **Static headers** are defined in `pkg/yqlib/doc/operators/headers/*.md`
+2. **Dynamic content** is generated from test scenarios in `*_test.go` files
+3. **Generated docs** are created in `pkg/yqlib/doc/*.md` by concatenating headers with test-generated content
+4. **Documentation is synced** to the gitbook branch for the website
 
-## How to contribute
+### Updating Operator Documentation
 
-The first step is to find if what you want is automatically generated or not - start by looking in the master branch. 
+#### For Test-Generated Documentation
 
-### Updating dynamic documentation from master
-- Search for the documentation you want to update. If you find matches in a `*_test.go` file - update that, as that will automatically update the matching `*.md` file 
-- Assuming you are updating a `*_test.go` file, once updated, run the test to regenerated the docs. E.g. for the 'Add' test generated docs, from the pkg/yqlib folder run:
-`go test -run TestAddOperatorScenarios` which will run that test defined in the `operator_add_test.go` file.
-- Ensure the tests still pass, and check the generated documentation have your update.
-- Note: If the documentation is only in a `headers/*.md` file, then just update that directly
-- Raise a PR to merge the changes into master!
+Most operator documentation is generated from tests. To update:
 
-### Updating static documentation from the gitbook branch
-If you haven't found what you want to update in the master branch, then check the gitbook branch directly as there are a few pages in there that are not in master.
+1. **Find the test file** (e.g., `operator_add_test.go`)
+2. **Update test scenarios** - each `expressionScenario` with `skipDoc: false` becomes documentation
+3. **Run the test** to regenerate docs:
+   ```bash
+   cd pkg/yqlib
+   go test -run TestAddOperatorScenarios
+   ```
+4. **Verify the generated documentation** in `pkg/yqlib/doc/add.md`
+5. **Create a PR** with your changes
 
-- Update the `*.md` files
-- Raise a PR to merge the changes into gitbook.
+#### For Header-Only Documentation
+
+If documentation exists only in `headers/*.md` files:
+1. **Update the header file directly** (e.g., `pkg/yqlib/doc/operators/headers/add.md`)
+2. **Create a PR** with your changes
+
+### Updating Static Documentation
+
+For documentation not in the master branch:
+
+1. **Check the gitbook branch** for additional pages
+2. **Update the `*.md` files** directly
+3. **Create a PR** to the gitbook branch
+
+### Documentation Best Practices
+
+- **Write clear, concise examples** in test scenarios
+- **Use meaningful variable names** in examples
+- **Include edge cases** and error conditions
+- **Test your documentation changes** by running the specific test
+- **Verify generated output** matches expectations
+
+Note: PRs with small changes (e.g. minor typos) may not be merged (see https://joel.net/how-one-guy-ruined-hacktoberfest2020-drama).
+
+# Troubleshooting
+
+## Common Setup Issues
+
+### Docker/Podman Issues
+- **Problem**: `make` commands fail with Docker errors
+- **Solution**: Ensure Docker or Podman is running and accessible
+- **Alternative**: Use `make local <command>` to run in containers
+
+### Go Version Issues
+- **Problem**: Build fails with Go version errors
+- **Solution**: Ensure you have Go 1.24.0 or later installed
+- **Check**: Run `go version` to verify
+
+### Vendor Dependencies
+- **Problem**: `make vendor` fails or dependencies are outdated
+- **Solution**: 
+  ```bash
+  go mod tidy
+  make [local] vendor
+  ```
+
+### Linting Failures
+- **Problem**: `make check` fails with linting errors
+- **Solution**: 
+  ```bash
+  make [local] format  # Auto-fix formatting
+  # Manually fix remaining linting issues
+  make [local] check   # Verify fixes
+  ```
+
+### Test Failures
+- **Problem**: Tests fail locally but pass in CI
+- **Solution**: 
+  ```bash
+  make [local] test    # Run in Docker container
+  ```
+
+### Documentation Generation Issues
+- **Problem**: Generated docs don't update after test changes
+- **Solution**: 
+  ```bash
+  cd pkg/yqlib
+  go test -run TestSpecificOperatorScenarios
+  # Check if generated file updated in pkg/yqlib/doc/
+  ```
+
+## Getting Help
+
+- **Check existing issues**: Search GitHub issues for similar problems
+- **Create an issue**: If you can't find a solution, create a detailed issue
+- **Ask questions**: Use GitHub Discussions for general questions
+- **Join the community**: Check the project's community channels

Dockerfile

@@ -1,4 +1,4 @@
-FROM golang:1.22.1 as builder
+FROM golang:1.25.5 AS builder
 
 WORKDIR /go/src/mikefarah/yq
 
@@ -10,7 +10,7 @@ RUN ./scripts/acceptance.sh
 
 # Choose alpine as a base image to make this useful for CI, as many
 # CI tools expect an interactive shell inside the container
-FROM alpine:3 as production
+FROM alpine:3 AS production
 LABEL maintainer="Mike Farah <mikefarah@users.noreply.github.com>"
 
 COPY --from=builder /go/src/mikefarah/yq/yq /usr/bin/yq

Dockerfile.dev

@@ -1,12 +1,15 @@
-FROM golang:1.22.1
+FROM golang:1.25.5
+
+RUN apt-get update && \
+    apt-get install -y npm && \
+    npm install -g npx cspell@latest
 
 COPY scripts/devtools.sh /opt/devtools.sh
 
-RUN set -e -x \
-    && /opt/devtools.sh
-ENV PATH=/go/bin:$PATH
+RUN set -e -x && \
+    /opt/devtools.sh
 
-RUN apt-get update && apt-get install -y npm && npm install -g npx cspell@latest
+ENV PATH=/go/bin:$PATH
 
 ENV CGO_ENABLED 0
 ENV GOPATH /go:/yq

Makefile

@@ -2,7 +2,7 @@ MAKEFLAGS += --warn-undefined-variables
 SHELL := /bin/bash
 .SHELLFLAGS := -o pipefail -euc
 .DEFAULT_GOAL := install
-ENGINE := $(shell { command -v podman || command -v docker; } 2>/dev/null)
+ENGINE := $(shell { (podman version > /dev/null 2>&1 && command -v podman) || command -v docker; } 2>/dev/null)
 
 include Makefile.variables
 

Makefile.variables

@@ -26,6 +26,7 @@ ifeq ($(CYG_CHECK),1)
 else
 	# all non-windows environments
 	ROOT := $(shell pwd)
+	SELINUX := $(shell which getenforce 2>&1 >/dev/null && echo :z)
 endif
 
 DEV_IMAGE := ${PROJECT}_dev
@@ -33,7 +34,7 @@ DEV_IMAGE := ${PROJECT}_dev
 ENGINERUN := ${ENGINE} run --rm \
 	-e LDFLAGS="${LDFLAGS}" \
 	-e GITHUB_TOKEN="${GITHUB_TOKEN}" \
-	-v ${ROOT}/vendor:/go/src \
-	-v ${ROOT}:/${PROJECT}/src/${IMPORT_PATH} \
+	-v ${ROOT}/vendor:/go/src${SELINUX} \
+	-v ${ROOT}:/${PROJECT}/src/${IMPORT_PATH}${SELINUX} \
 	-w /${PROJECT}/src/${IMPORT_PATH} \
 	${DEV_IMAGE}

README.md

@@ -3,44 +3,46 @@
 ![Build](https://github.com/mikefarah/yq/workflows/Build/badge.svg)  ![Docker Pulls](https://img.shields.io/docker/pulls/mikefarah/yq.svg) ![Github Releases (by Release)](https://img.shields.io/github/downloads/mikefarah/yq/total.svg) ![Go Report](https://goreportcard.com/badge/github.com/mikefarah/yq) ![CodeQL](https://github.com/mikefarah/yq/workflows/CodeQL/badge.svg)
 
 
-a lightweight and portable command-line YAML, JSON and XML processor. `yq` uses [jq](https://github.com/stedolan/jq) like syntax but works with yaml files as well as json, xml, properties, csv and tsv. It doesn't yet support everything `jq` does - but it does support the most common operations and functions, and more is being added continuously.
+A lightweight and portable command-line YAML, JSON, INI and XML processor. `yq` uses [jq](https://github.com/stedolan/jq) (a popular JSON processor) like syntax but works with yaml files as well as json, xml, ini, properties, csv and tsv. It doesn't yet support everything `jq` does - but it does support the most common operations and functions, and more is being added continuously.
 
-yq is written in go - so you can download a dependency free binary for your platform and you are good to go! If you prefer there are a variety of package managers that can be used as well as Docker and Podman, all listed below.
+yq is written in Go - so you can download a dependency free binary for your platform and you are good to go! If you prefer there are a variety of package managers that can be used as well as Docker and Podman, all listed below.
 
 ## Quick Usage Guide
 
-Read a value:
+### Basic Operations
+
+**Read a value:**
 ```bash
 yq '.a.b[0].c' file.yaml
 ```
 
-Pipe from STDIN:
+**Pipe from STDIN:**
 ```bash
 yq '.a.b[0].c' < file.yaml
 ```
 
-Update a yaml file, in place
+**Update a yaml file in place:**
 ```bash
 yq -i '.a.b[0].c = "cool"' file.yaml
 ```
 
-Update using environment variables
+**Update using environment variables:**
 ```bash
 NAME=mike yq -i '.a.b[0].c = strenv(NAME)' file.yaml
 ```
 
-Merge multiple files
+### Advanced Operations
+
+**Merge multiple files:**
 ```bash
 # merge two files
 yq -n 'load("file1.yaml") * load("file2.yaml")'
 
-# merge using globs:
-# note the use of `ea` to evaluate all the files at once
-# instead of in sequence
+# merge using globs (note: `ea` evaluates all files at once instead of in sequence)
 yq ea '. as $item ireduce ({}; . * $item )' path/to/*.yml
 ```
 
-Multiple updates to a yaml file
+**Multiple updates to a yaml file:**
 ```bash
 yq -i '
   .a.b[0].c = "cool" |
@@ -49,14 +51,22 @@ yq -i '
 ' file.yaml
 ```
 
-Find and update an item in an array:
+**Find and update an item in an array:**
 ```bash
-yq '(.[] | select(.name == "foo") | .address) = "12 cat st"'
+# Note: requires input file - add your file at the end
+yq -i '(.[] | select(.name == "foo") | .address) = "12 cat st"' data.yaml
 ```
 
-Convert JSON to YAML
+**Convert between formats:**
 ```bash
+# Convert JSON to YAML (pretty print)
 yq -Poy sample.json
+
+# Convert YAML to JSON
+yq -o json file.yaml
+
+# Convert XML to YAML
+yq -o yaml file.xml
 ```
 
 See [recipes](https://mikefarah.gitbook.io/yq/recipes) for more examples and the [documentation](https://mikefarah.gitbook.io/yq/) for more information.
@@ -68,30 +78,30 @@ Take a look at the discussions for [common questions](https://github.com/mikefar
 ### [Download the latest binary](https://github.com/mikefarah/yq/releases/latest)
 
 ### wget
-Use wget to download, gzipped pre-compiled binaries:
+Use wget to download pre-compiled binaries. Choose your platform and architecture:
 
-
-For instance, VERSION=v4.2.0 and BINARY=yq_linux_amd64
-
-#### Compressed via tar.gz
+**For Linux (example):**
 ```bash
-wget https://github.com/mikefarah/yq/releases/download/${VERSION}/${BINARY}.tar.gz -O - |\
-  tar xz && mv ${BINARY} /usr/bin/yq
+# Set your platform variables (adjust as needed)
+VERSION=v4.2.0
+PLATFORM=linux_amd64
+
+# Download compressed binary
+wget https://github.com/mikefarah/yq/releases/download/${VERSION}/yq_${PLATFORM}.tar.gz -O - |\
+  tar xz && sudo mv yq_${PLATFORM} /usr/local/bin/yq
+
+# Or download plain binary
+wget https://github.com/mikefarah/yq/releases/download/${VERSION}/yq_${PLATFORM} -O /usr/local/bin/yq &&\
+    chmod +x /usr/local/bin/yq
 ```
 
-#### Plain binary
-
+**Latest version (Linux AMD64):**
 ```bash
-wget https://github.com/mikefarah/yq/releases/download/${VERSION}/${BINARY} -O /usr/bin/yq &&\
-    chmod +x /usr/bin/yq
+wget https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -O /usr/local/bin/yq &&\
+    chmod +x /usr/local/bin/yq
 ```
 
-#### Latest version
-
-```bash
-wget https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -O /usr/bin/yq &&\
-    chmod +x /usr/bin/yq
-```
+**Available platforms:** `linux_amd64`, `linux_arm64`, `linux_arm`, `linux_386`, `darwin_amd64`, `darwin_arm64`, `windows_amd64`, `windows_386`, etc.
 
 ### MacOS / Linux via Homebrew:
 Using [Homebrew](https://brew.sh/)
@@ -123,28 +133,31 @@ rm /etc/myfile.tmp
 ```
 
 ### Run with Docker or Podman
-#### Oneshot use:
 
+#### One-time use:
 ```bash
-docker run --rm -v "${PWD}":/workdir mikefarah/yq [command] [flags] [expression ]FILE...
+# Docker - process files in current directory
+docker run --rm -v "${PWD}":/workdir mikefarah/yq '.a.b[0].c' file.yaml
+
+# Podman - same usage as Docker
+podman run --rm -v "${PWD}":/workdir mikefarah/yq '.a.b[0].c' file.yaml
 ```
 
-Note that you can run `yq` in docker without network access and other privileges if you desire,
-namely `--security-opt=no-new-privileges --cap-drop all --network none`.
-
+**Security note:** You can run `yq` in Docker with restricted privileges:
 ```bash
-podman run --rm -v "${PWD}":/workdir mikefarah/yq [command] [flags] [expression ]FILE...
+docker run --rm --security-opt=no-new-privileges --cap-drop all --network none \
+  -v "${PWD}":/workdir mikefarah/yq '.a.b[0].c' file.yaml
 ```
 
-#### Pipe in via STDIN:
+#### Pipe data via STDIN:
 
-You'll need to pass the `-i\--interactive` flag to docker:
+You'll need to pass the `-i --interactive` flag to Docker/Podman:
 
 ```bash
+# Process piped data
 docker run -i --rm mikefarah/yq '.this.thing' < myfile.yml
-```
 
-```bash
+# Same with Podman
 podman run -i --rm mikefarah/yq '.this.thing' < myfile.yml
 ```
 
@@ -241,6 +254,14 @@ As these are supported by the community :heart: - however, they may be out of da
 _Please note that the Debian package (previously supported by @rmescandon) is no longer maintained. Please use an alternative installation method._
 
 
+### X-CMD
+Checkout `yq` on x-cmd: https://x-cmd.com/mod/yq
+
+- Instant Results: See the output of your yq filter in real-time.
+- Error Handling: Encounter a syntax error? It will display the error message and the results of the closest valid filter
+
+Thanks @edwinjhlee!
+
 ### Nix
 
 ```
@@ -286,7 +307,7 @@ Using [winget](https://learn.microsoft.com/en-us/windows/package-manager/)
 winget install --id MikeFarah.yq
 ```
 
-### Mac:
+### MacPorts:
 Using [MacPorts](https://www.macports.org/)
 ```
 sudo port selfupdate
@@ -295,23 +316,44 @@ sudo port install yq
 Supported by @herbygillot (https://ports.macports.org/maintainer/github/herbygillot)
 
 ### Alpine Linux
-- Enable edge/community repo by adding ```$MIRROR/alpine/edge/community``` to ```/etc/apk/repositories```
-- Update database index with ```apk update```
-- Install yq with ```apk add yq```
 
-Supported by Tuan Hoang
-https://pkgs.alpinelinux.org/package/edge/community/x86/yq
+Alpine Linux v3.20+ (and Edge):
+```
+apk add yq-go
+```
 
+Alpine Linux up to v3.19:
+```
+apk add yq
+```
+
+Supported by Tuan Hoang (https://pkgs.alpinelinux.org/packages?name=yq-go)
+
+### Flox:
+
+Flox can be used to install yq on Linux, MacOS, and Windows through WSL.
+
+```
+flox install yq
+```
+
+
+### MacOS / Linux via gah:
+Using [gah](https://github.com/marverix/gah)
+
+```
+gah install yq
+```
 
 ## Features
 - [Detailed documentation with many examples](https://mikefarah.gitbook.io/yq/)
 - Written in portable go, so you can download a lovely dependency free binary
-- Uses similar syntax as `jq` but works with YAML, [JSON](https://mikefarah.gitbook.io/yq/usage/convert) and [XML](https://mikefarah.gitbook.io/yq/usage/xml) files
+- Uses similar syntax as `jq` but works with YAML, INI, [JSON](https://mikefarah.gitbook.io/yq/usage/convert) and [XML](https://mikefarah.gitbook.io/yq/usage/xml) files
 - Fully supports multi document yaml files
 - Supports yaml [front matter](https://mikefarah.gitbook.io/yq/usage/front-matter) blocks (e.g. jekyll/assemble)
 - Colorized yaml output
 - [Date/Time manipulation and formatting with TZ](https://mikefarah.gitbook.io/yq/operators/datetime)
-- [Deeply data structures](https://mikefarah.gitbook.io/yq/operators/traverse-read)
+- [Deep data structures](https://mikefarah.gitbook.io/yq/operators/traverse-read)
 - [Sort keys](https://mikefarah.gitbook.io/yq/operators/sort-keys)
 - Manipulate yaml [comments](https://mikefarah.gitbook.io/yq/operators/comment-operators), [styling](https://mikefarah.gitbook.io/yq/operators/style), [tags](https://mikefarah.gitbook.io/yq/operators/tag) and [anchors and aliases](https://mikefarah.gitbook.io/yq/operators/anchor-and-alias-operators).
 - [Update in place](https://mikefarah.gitbook.io/yq/v/v4.x/commands/evaluate#flags)
@@ -338,42 +380,96 @@ Usage:
 
 Examples:
 
-# yq defaults to 'eval' command if no command is specified. See "yq eval --help" for more examples.
-yq '.stuff' < myfile.yml # outputs the data at the "stuff" node from "myfile.yml"
+# yq tries to auto-detect the file format based off the extension, and defaults to YAML if it's unknown (or piping through STDIN)
+# Use the '-p/--input-format' flag to specify a format type.
+cat file.xml | yq -p xml
 
-yq -i '.stuff = "foo"' myfile.yml # update myfile.yml in place
+# read the "stuff" node from "myfile.yml"
+yq '.stuff' < myfile.yml
+
+# update myfile.yml in place
+yq -i '.stuff = "foo"' myfile.yml
+
+# print contents of sample.json as idiomatic YAML
+yq -P -oy sample.json
 
 
 Available Commands:
-  completion       Generate the autocompletion script for the specified shell
-  eval             (default) Apply the expression to each document in each yaml file in sequence
-  eval-all         Loads _all_ yaml documents of _all_ yaml files and runs expression once
-  help             Help about any command
-  shell-completion Generate completion script
+  completion  Generate the autocompletion script for the specified shell
+  eval        (default) Apply the expression to each document in each yaml file in sequence
+  eval-all    Loads _all_ yaml documents of _all_ yaml files and runs expression once
+  help        Help about any command
 
 Flags:
-  -C, --colors                        force print with colors
-  -e, --exit-status                   set exit status if there are no matches or null or false is returned
-  -f, --front-matter string           (extract|process) first input as yaml front-matter. Extract will pull out the yaml content, process will run the expression against the yaml content, leaving the remaining data intact
-      --header-preprocess             Slurp any header comments and separators before processing expression. (default true)
-  -h, --help                          help for yq
-  -I, --indent int                    sets indent level for output (default 2)
-  -i, --inplace                       update the file in place of first file given.
-  -p, --input-format string           [yaml|y|xml|x] parse format for input. Note that json is a subset of yaml. (default "yaml")
-  -M, --no-colors                     force print with no colors
-  -N, --no-doc                        Don't print document separators (---)
-  -n, --null-input                    Don't read input, simply evaluate the expression given. Useful for creating docs from scratch.
-  -o, --output-format string          [yaml|y|json|j|props|p|xml|x] output format type. (default "yaml")
-  -P, --prettyPrint                   pretty print, shorthand for '... style = ""'
-  -s, --split-exp string              print each result (or doc) into a file named (exp). [exp] argument must return a string. You can use $index in the expression as the result counter.
-      --unwrapScalar                  unwrap scalar, print the value with no quotes, colors or comments (default true)
-  -v, --verbose                       verbose mode
-  -V, --version                       Print version information and quit
-      --xml-attribute-prefix string   prefix for xml attributes (default "+")
-      --xml-content-name string       name for xml content (if no attribute name is present). (default "+content")
+  -C, --colors                          force print with colors
+      --csv-auto-parse                  parse CSV YAML/JSON values (default true)
+      --csv-separator char              CSV Separator character (default ,)
+      --debug-node-info                 debug node info
+  -e, --exit-status                     set exit status if there are no matches or null or false is returned
+      --expression string               forcibly set the expression argument. Useful when yq argument detection thinks your expression is a file.
+      --from-file string                Load expression from specified file.
+  -f, --front-matter string             (extract|process) first input as yaml front-matter. Extract will pull out the yaml content, process will run the expression against the yaml content, leaving the remaining data intact
+      --header-preprocess               Slurp any header comments and separators before processing expression. (default true)
+  -h, --help                            help for yq
+  -I, --indent int                      sets indent level for output (default 2)
+  -i, --inplace                         update the file in place of first file given.
+  -p, --input-format string             [auto|a|yaml|y|json|j|props|p|csv|c|tsv|t|xml|x|base64|uri|toml|lua|l|ini|i] parse format for input. (default "auto")
+      --lua-globals                     output keys as top-level global variables
+      --lua-prefix string               prefix (default "return ")
+      --lua-suffix string               suffix (default ";\n")
+      --lua-unquoted                    output unquoted string keys (e.g. {foo="bar"})
+  -M, --no-colors                       force print with no colors
+  -N, --no-doc                          Don't print document separators (---)
+  -0, --nul-output                      Use NUL char to separate values. If unwrap scalar is also set, fail if unwrapped scalar contains NUL char.
+  -n, --null-input                      Don't read input, simply evaluate the expression given. Useful for creating docs from scratch.
+  -o, --output-format string            [auto|a|yaml|y|json|j|props|p|csv|c|tsv|t|xml|x|base64|uri|toml|shell|s|lua|l|ini|i] output format type. (default "auto")
+  -P, --prettyPrint                     pretty print, shorthand for '... style = ""'
+      --properties-array-brackets       use [x] in array paths (e.g. for SpringBoot)
+      --properties-separator string     separator to use between keys and values (default " = ")
+      --security-disable-env-ops        Disable env related operations.
+      --security-disable-file-ops       Disable file related operations (e.g. load)
+      --shell-key-separator string      separator for shell variable key paths (default "_")
+  -s, --split-exp string                print each result (or doc) into a file named (exp). [exp] argument must return a string. You can use $index in the expression as the result counter. The necessary directories will be created.
+      --split-exp-file string           Use a file to specify the split-exp expression.
+      --string-interpolation            Toggles strings interpolation of \(exp) (default true)
+      --tsv-auto-parse                  parse TSV YAML/JSON values (default true)
+  -r, --unwrapScalar                    unwrap scalar, print the value with no quotes, colors or comments. Defaults to true for yaml (default true)
+  -v, --verbose                         verbose mode
+  -V, --version                         Print version information and quit
+      --xml-attribute-prefix string     prefix for xml attributes (default "+@")
+      --xml-content-name string         name for xml content (if no attribute name is present). (default "+content")
+      --xml-directive-name string       name for xml directives (e.g. <!DOCTYPE thing cat>) (default "+directive")
+      --xml-keep-namespace              enables keeping namespace after parsing attributes (default true)
+      --xml-proc-inst-prefix string     prefix for xml processing instructions (e.g. <?xml version="1"?>) (default "+p_")
+      --xml-raw-token                   enables using RawToken method instead Token. Commonly disables namespace translations. See https://pkg.go.dev/encoding/xml#Decoder.RawToken for details. (default true)
+      --xml-skip-directives             skip over directives (e.g. <!DOCTYPE thing cat>)
+      --xml-skip-proc-inst              skip over process instructions (e.g. <?xml version="1"?>)
+      --xml-strict-mode                 enables strict parsing of XML. See https://pkg.go.dev/encoding/xml for more details.
+      --yaml-fix-merge-anchor-to-spec   Fix merge anchor to match YAML spec. Will default to true in late 2025
 
 Use "yq [command] --help" for more information about a command.
 ```
+
+## Troubleshooting
+
+### Common Issues
+
+**PowerShell quoting issues:**
+```powershell
+# Use single quotes for expressions
+yq '.a.b[0].c' file.yaml
+
+# Or escape double quotes
+yq ".a.b[0].c = \"value\"" file.yaml
+```
+
+### Getting Help
+
+- **Check existing issues**: [GitHub Issues](https://github.com/mikefarah/yq/issues)
+- **Ask questions**: [GitHub Discussions](https://github.com/mikefarah/yq/discussions)
+- **Documentation**: [Complete documentation](https://mikefarah.gitbook.io/yq/)
+- **Examples**: [Recipes and examples](https://mikefarah.gitbook.io/yq/recipes)
+
 ## Known Issues / Missing Features
 - `yq` attempts to preserve comment positions and whitespace as much as possible, but it does not handle all scenarios (see https://github.com/go-yaml/yaml/tree/v3 for details)
 - Powershell has its own...[opinions on quoting yq](https://mikefarah.gitbook.io/yq/usage/tips-and-tricks#quotes-in-windows-powershell)

acceptance_tests/load-file.sh

@@ -3,7 +3,7 @@
 testLoadFileNotExist() {
   result=$(./yq e -n 'load("cat.yml")' 2>&1)
   assertEquals 1 $?
-  assertEquals "Error: Failed to load cat.yml: open cat.yml: no such file or directory" "$result"
+  assertEquals "Error: failed to load cat.yml: open cat.yml: no such file or directory" "$result"
 }
 
 testLoadFileExpNotExist() {
@@ -15,7 +15,7 @@ testLoadFileExpNotExist() {
 testStrLoadFileNotExist() {
   result=$(./yq e -n 'strload("cat.yml")' 2>&1)
   assertEquals 1 $?
-  assertEquals "Error: Failed to load cat.yml: open cat.yml: no such file or directory" "$result"
+  assertEquals "Error: failed to load cat.yml: open cat.yml: no such file or directory" "$result"
 }
 
 testStrLoadFileExpNotExist() {

acceptance_tests/split-printer.sh

@@ -2,6 +2,7 @@
 
 setUp() {
   rm test*.yml || true
+  rm -rf test_dir* || true
 }
 
 testBasicSplitWithName() {
@@ -204,4 +205,23 @@ EOM
   assertEquals "$expectedDoc3" "$doc3"
 }
 
-source ./scripts/shunit2
+testSplitWithDirectories() {
+  cat >test.yml <<EOL
+f: test_dir1/test_file1
+---
+f: test_dir2/dir22/test_file2
+---
+f: test_file3
+EOL
+
+  ./yq e --no-doc -s ".f" test.yml
+
+  doc1=$(cat test_dir1/test_file1.yml)
+  assertEquals "f: test_dir1/test_file1" "$doc1"
+  doc2=$(cat test_dir2/dir22/test_file2.yml)
+  assertEquals "f: test_dir2/dir22/test_file2" "$doc2"
+  doc3=$(cat test_file3.yml)
+  assertEquals "f: test_file3" "$doc3"
+}
+
+source ./scripts/shunit2

agents.md

@@ -0,0 +1,414 @@
+# Adding a New Encoder/Decoder
+
+This guide explains how to add support for a new format (encoder/decoder) to yq without modifying `candidate_node.go`.
+
+## Overview
+
+The encoder/decoder architecture in yq is based on two main interfaces:
+
+- **Encoder**: Converts a `CandidateNode` to output in a specific format
+- **Decoder**: Reads input in a specific format and creates a `CandidateNode`
+
+Each format is registered in `pkg/yqlib/format.go` and made available through factory functions.
+
+## Architecture
+
+### Key Files
+
+- `pkg/yqlib/encoder.go` - Defines the `Encoder` interface
+- `pkg/yqlib/decoder.go` - Defines the `Decoder` interface
+- `pkg/yqlib/format.go` - Format registry and factory functions
+- `pkg/yqlib/operator_encoder_decoder.go` - Encode/decode operators
+- `pkg/yqlib/encoder_*.go` - Encoder implementations
+- `pkg/yqlib/decoder_*.go` - Decoder implementations
+
+### Interfaces
+
+**Encoder Interface:**
+```go
+type Encoder interface {
+    Encode(writer io.Writer, node *CandidateNode) error
+    PrintDocumentSeparator(writer io.Writer) error
+    PrintLeadingContent(writer io.Writer, content string) error
+    CanHandleAliases() bool
+}
+```
+
+**Decoder Interface:**
+```go
+type Decoder interface {
+    Init(reader io.Reader) error
+    Decode() (*CandidateNode, error)
+}
+```
+
+## Step-by-Step: Adding a New Encoder/Decoder
+
+### Step 1: Create the Encoder File
+
+Create `pkg/yqlib/encoder_<format>.go` implementing the `Encoder` interface:
+- `Encode()` - Convert a `CandidateNode` to your format and write to the output writer
+- `PrintDocumentSeparator()` - Handle document separators if your format requires them
+- `PrintLeadingContent()` - Handle leading content/comments if supported
+- `CanHandleAliases()` - Return whether your format supports YAML aliases
+
+See `encoder_json.go` or `encoder_base64.go` for examples.
+
+### Step 2: Create the Decoder File
+
+Create `pkg/yqlib/decoder_<format>.go` implementing the `Decoder` interface:
+- `Init()` - Initialize the decoder with the input reader and set up any needed state
+- `Decode()` - Decode one document from the input and return a `CandidateNode`, or `io.EOF` when finished
+
+See `decoder_json.go` or `decoder_base64.go` for examples.
+
+### Step 3: Create Tests (Mandatory)
+
+Create a test file `pkg/yqlib/<format>_test.go` using the `formatScenario` pattern:
+- Define test scenarios as `formatScenario` structs with fields: `description`, `input`, `expected`, `scenarioType`
+- `scenarioType` can be `"decode"` (test decoding to YAML) or `"roundtrip"` (encode/decode preservation)
+- Create a helper function `test<Format>Scenario()` that switches on `scenarioType`
+- Create main test function `Test<Format>FormatScenarios()` that iterates over scenarios
+
+Test coverage must include:
+- Basic data types (scalars, arrays, objects/maps)
+- Nested structures
+- Edge cases (empty inputs, special characters, escape sequences)
+- Format-specific features or syntax
+- Round-trip tests: decode → encode → decode should preserve data
+
+See `hcl_test.go` for a complete example.
+
+### Step 4: Register the Format in format.go
+
+Edit `pkg/yqlib/format.go`:
+
+1. Add a new format variable:
+   - `"<format>"` is the formal name (e.g., "json", "yaml")
+   - `[]string{...}` contains short aliases (can be empty)
+   - The first function creates an encoder (can be nil for encode-only formats)
+   - The second function creates a decoder (can be nil for decode-only formats)
+
+2. Add the format to the `Formats` slice in the same file
+
+See existing formats in `format.go` for the exact structure.
+
+### Step 5: Handle Encoder Configuration (if needed)
+
+If your format has preferences/configuration options:
+
+1. Create a preferences struct with your configuration fields
+2. Update the encoder to accept preferences in its factory function
+3. Update `format.go` to pass the configured preferences
+4. Update `operator_encoder_decoder.go` if special indent handling is needed (see existing formats like JSON and YAML for the pattern)
+
+This pattern is optional and only needed if your format has user-configurable options.
+
+## Build Tags
+
+Use build tags to allow optional compilation of formats:
+- Add `//go:build !yq_no<format>` at the top of your encoder and decoder files
+- Create a no-build version in `pkg/yqlib/no_<format>.go` that returns nil for encoder/decoder factories
+
+This allows users to compile yq without certain formats using: `go build -tags yq_no<format>`
+
+## Working with CandidateNode
+
+The `CandidateNode` struct represents a YAML node with:
+- `Kind`: The node type (ScalarNode, SequenceNode, MappingNode)
+- `Tag`: The YAML tag (e.g., "!!str", "!!int", "!!map")
+- `Value`: The scalar value (for ScalarNode only)
+- `Content`: Child nodes (for SequenceNode and MappingNode)
+
+Key methods:
+- `node.guessTagFromCustomType()` - Infer the tag from Go type
+- `node.AsList()` - Convert to a list for processing
+- `node.CreateReplacement()` - Create a new replacement node
+- `NewCandidate()` - Create a new CandidateNode
+
+## Key Points
+
+✅ **DO:**
+- Implement only the `Encoder` and `Decoder` interfaces
+- Register your format in `format.go` only
+- Keep format-specific logic in your encoder/decoder files
+- Use the candidate_node style attribute to store style information for round-trip. Ask if this needs to be updated with new styles.
+- Use build tags for optional compilation
+- Add comprehensive tests
+- Run the specific encoder/decoder test (e.g. <format>_test.go) whenever you make ay changes to the encoder_<format> or decoder_<format>
+- Handle errors gracefully
+- Add the no build directive, like the xml encoder and decoder, that enables a minimal yq builds. e.g.  `//go:build !yq_<format>`. Be sure to also update the build_small-yq.sh and build-tinygo-yq.sh to not include the new format.
+
+❌ **DON'T:**
+- Modify `candidate_node.go` to add format-specific logic
+- Add format-specific fields to `CandidateNode`
+- Create special cases in core navigation or evaluation logic
+- Bypass the encoder/decoder interfaces
+- Use candidate_node tag attribute for anything other than indicate the data type
+
+## Examples
+
+Refer to existing format implementations for patterns:
+
+- **Simple encoder/decoder**: `encoder_json.go`, `decoder_json.go`
+- **Complex with preferences**: `encoder_yaml.go`, `decoder_yaml.go`
+- **Encoder-only**: `encoder_sh.go` (ShFormat has nil decoder)
+- **String-only operations**: `encoder_base64.go`, `decoder_base64.go`
+
+## Testing Your Implementation (Mandatory)
+
+Tests must be implemented in `<format>_test.go` following the `formatScenario` pattern:
+
+1. **Create test scenarios** using the `formatScenario` struct with fields:
+   - `description`: Brief description of what's being tested
+   - `input`: Sample input in your format
+   - `expected`: Expected output (typically in YAML for decode tests)
+   - `scenarioType`: Either `"decode"` or `"roundtrip"`
+
+2. **Test coverage must include:**
+   - Basic data types (scalars, arrays, objects/maps)
+   - Nested structures
+   - Edge cases (empty inputs, special characters, escape sequences)
+   - Format-specific features or syntax
+   - Round-trip tests: decode → encode → decode should preserve data
+
+3. **Test function pattern:**
+   - `test<Format>Scenario()`: Helper function that switches on `scenarioType`
+   - `Test<Format>FormatScenarios()`: Main test function that iterates over scenarios
+
+4. **Example from existing formats:**
+   - See `hcl_test.go` for a complete example
+   - See `yaml_test.go` for YAML-specific patterns
+   - See `json_test.go` for more complex scenarios
+
+## Common Patterns
+
+### Scalar-Only Formats
+Some formats only work with scalars (like base64, uri):
+```go
+if node.guessTagFromCustomType() != "!!str" {
+    return fmt.Errorf("cannot encode %v as <format>, can only operate on strings", node.Tag)
+}
+```
+
+### Format with Indentation
+Use preferences to control output formatting:
+```go
+type <format>Preferences struct {
+    Indent int
+}
+
+func (prefs *<format>Preferences) Copy() <format>Preferences {
+    return *prefs
+}
+```
+
+### Multiple Documents
+Decoders should support reading multiple documents:
+```go
+func (dec *<format>Decoder) Decode() (*CandidateNode, error) {
+    if dec.finished {
+        return nil, io.EOF
+    }
+    // ... decode next document ...
+    if noMoreDocuments {
+        dec.finished = true
+    }
+    return candidate, nil
+}
+```
+
+---
+
+# Adding a New Operator
+
+This guide explains how to add a new operator to yq. Operators are the core of yq's expression language and process `CandidateNode` objects without requiring modifications to `candidate_node.go` itself.
+
+## Overview
+
+Operators transform data by implementing a handler function that processes a `Context` containing `CandidateNode` objects. Each operator is:
+
+1. Defined as an `operationType` in `operation.go`
+2. Registered in the lexer in `lexer_participle.go`
+3. Implemented in its own `operator_<type>.go` file
+4. Tested in `operator_<type>_test.go`
+5. Documented in `pkg/yqlib/doc/operators/headers/<type>.md`
+
+## Architecture
+
+### Key Files
+
+- `pkg/yqlib/operation.go` - Defines `operationType` and operator registry
+- `pkg/yqlib/lexer_participle.go` - Registers operators with their syntax patterns
+- `pkg/yqlib/operator_<type>.go` - Operator implementation
+- `pkg/yqlib/operator_<type>_test.go` - Operator tests using `expressionScenario`
+- `pkg/yqlib/doc/operators/headers/<type>.md` - Documentation header
+
+### Core Types
+
+**operationType:**
+```go
+type operationType struct {
+    Type                 string          // Unique operator name (e.g., "REVERSE")
+    NumArgs              uint            // Number of arguments (0 for no args)
+    Precedence           uint            // Operator precedence (higher = higher precedence)
+    Handler              operatorHandler // The function that executes the operator
+    CheckForPostTraverse bool            // Whether to apply post-traversal logic
+    ToString             func(*Operation) string // Custom string representation
+}
+```
+
+**operatorHandler signature:**
+```go
+type operatorHandler func(*dataTreeNavigator, Context, *ExpressionNode) (Context, error)
+```
+
+**expressionScenario for tests:**
+```go
+type expressionScenario struct {
+    description      string
+    subdescription   string
+    document         string
+    expression       string
+    expected         []string
+    skipDoc          bool
+    expectedError    string
+}
+```
+
+## Step-by-Step: Adding a New Operator
+
+### Step 1: Create the Operator Implementation File
+
+Create `pkg/yqlib/operator_<type>.go` implementing the operator handler function:
+- Implement the `operatorHandler` function signature
+- Process nodes from `context.MatchingNodes`
+- Return a new `Context` with results using `context.ChildContext()`
+- Use `candidate.CreateReplacement()` or `candidate.CreateReplacementWithComments()` to create new nodes
+- Handle errors gracefully with meaningful error messages
+
+See `operator_reverse.go` or `operator_keys.go` for examples.
+
+### Step 2: Register the Operator in operation.go
+
+Add the operator type definition to `pkg/yqlib/operation.go`:
+
+```go
+var <type>OpType = &operationType{
+    Type:       "<TYPE>",          // All caps, matches pattern in lexer
+    NumArgs:    0,                 // 0 for no args, 1+ for args
+    Precedence: 50,                // Typical range: 40-55
+    Handler:    <type>Operator,    // Reference to handler function
+}
+```
+
+**Precedence guidelines:**
+- 10-20: Logical operators (OR, AND, UNION)
+- 30: Pipe operator
+- 40: Assignment and comparison operators
+- 42: Arithmetic operators (ADD, SUBTRACT, MULTIPLY, DIVIDE)
+- 50-52: Most other operators
+- 55: High precedence (e.g., GET_VARIABLE)
+
+**Optional fields:**
+- `CheckForPostTraverse: true` - If your operator can have another directly after it without the pipe character. Most of the time this is false.
+- `ToString: customToString` - Custom string representation (rarely needed)
+
+### Step 3: Register the Operator in lexer_participle.go
+
+Edit `pkg/yqlib/lexer_participle.go` to add the operator to the lexer rules:
+- Use `simpleOp()` for simple keyword patterns
+- Use object syntax for regex patterns or complex syntax
+- Support optional characters with `_?` and aliases with `|`
+
+See existing operators in `lexer_participle.go` for pattern examples.
+
+### Step 4: Create Tests (Mandatory)
+
+Create `pkg/yqlib/operator_<type>_test.go` using the `expressionScenario` pattern:
+- Define test scenarios with `description`, `document`, `expression`, and `expected` fields
+- `expected` is a slice of strings showing output format: `"D<doc>, P[<path>], (<tag>)::<value>\n"`
+- Set `skipDoc: true` for edge cases you don't want in generated documentation
+- Include `subdescription` for longer test names
+- Set `expectedError` if testing error cases
+- Create main test function that iterates over scenarios
+
+Test coverage must include:
+- Basic data types and nested structures
+- Edge cases (empty inputs, special characters, type errors)
+- Multiple outputs if applicable
+- Format-specific features
+
+See `operator_reverse_test.go` for a simple example and `operator_keys_test.go` for complex cases.
+
+### Step 5: Create Documentation Header
+
+Create `pkg/yqlib/doc/operators/headers/<type>.md`:
+- Use the exact operator name as the title
+- Include a concise 1-2 sentence summary
+- Add additional context or examples if the operator is complex
+
+See existing headers in `doc/operators/headers/` for examples.
+
+## Working with Context and CandidateNode
+
+### Context Management
+- `context.ChildContext(results)` - Create child context with results
+- `context.GetVariable("varName")` - Get variables stored in context
+- `context.SetVariable("varName", value)` - Set variables in context
+
+### CandidateNode Operations
+- `candidate.CreateReplacement(ScalarNode, "!!str", stringValue)` - Create a replacement node
+- `candidate.CreateReplacementWithComments(SequenceNode, "!!seq", candidate.Style)` - With style preserved
+- `candidate.Kind` - The node type (ScalarNode, SequenceNode, MappingNode)
+- `candidate.Tag` - The YAML tag (!!str, !!int, etc.)
+- `candidate.Value` - The scalar value (for ScalarNode only)
+- `candidate.Content` - Child nodes (for SequenceNode and MappingNode)
+- `candidate.guessTagFromCustomType()` - Infer the tag from Go type
+- `candidate.AsList()` - Convert to a list representation
+
+## Key Points
+
+✅ **DO:**
+- Implement the operator handler with the correct signature
+- Register in `operation.go` with appropriate precedence
+- Add the lexer pattern in `lexer_participle.go`
+- Write comprehensive tests covering normal and edge cases
+- Create a documentation header in `doc/operators/headers/`
+- Use `Context.ChildContext()` for proper context threading
+- Handle all node types gracefully
+- Return meaningful error messages
+
+❌ **DON'T:**
+- Modify `candidate_node.go` (operators shouldn't need this)
+- Modify core navigation or evaluation logic
+- Bypass the handler function pattern
+- Add format-specific or operator-specific fields to `CandidateNode`
+- Skip tests or documentation
+
+## Examples
+
+Refer to existing operator implementations for patterns:
+
+- **No-argument operator**: `operator_reverse.go` - Processes arrays/sequences
+- **Single-argument operator**: `operator_map.go` - Takes an expression argument
+- **Complex multi-output**: `operator_keys.go` - Produces multiple results
+- **With preferences**: `operator_to_number.go` - Configuration options
+- **Error handling**: `operator_error.go` - Control flow with errors
+- **String operations**: `operator_strings.go` - Multiple related operators
+
+## Testing Patterns
+
+Refer to existing test files for specific patterns:
+- Basic expression tests in `operator_reverse_test.go`
+- Multi-output tests in `operator_keys_test.go`
+- Error handling tests in `operator_error_test.go`
+- Tests with `skipDoc` flag to exclude from generated documentation
+
+## Common Patterns
+
+Refer to existing operator implementations for these patterns:
+- Simple transformation: see `operator_reverse.go`
+- Type checking: see `operator_error.go`
+- Working with arguments: see `operator_map.go`
+- Post-traversal operators: see `operator_with.go`

cmd/completion.go

@@ -7,19 +7,20 @@ import (
 )
 
 var completionCmd = &cobra.Command{
-	Use:   "shell-completion [bash|zsh|fish|powershell]",
-	Short: "Generate completion script",
+	Use:     "completion [bash|zsh|fish|powershell]",
+	Aliases: []string{"shell-completion"},
+	Short:   "Generate the autocompletion script for the specified shell",
 	Long: `To load completions:
 
 Bash:
 
-$ source <(yq shell-completion bash)
+$ source <(yq completion bash)
 
 # To load completions for each session, execute once:
 Linux:
-  $ yq shell-completion bash > /etc/bash_completion.d/yq
+  $ yq completion bash > /etc/bash_completion.d/yq
 MacOS:
-  $ yq shell-completion bash > /usr/local/etc/bash_completion.d/yq
+  $ yq completion bash > /usr/local/etc/bash_completion.d/yq
 
 Zsh:
 
@@ -29,16 +30,16 @@ Zsh:
 $ echo "autoload -U compinit; compinit" >> ~/.zshrc
 
 # To load completions for each session, execute once:
-$ yq shell-completion zsh > "${fpath[1]}/_yq"
+$ yq completion zsh > "${fpath[1]}/_yq"
 
 # You will need to start a new shell for this setup to take effect.
 
 Fish:
 
-$ yq shell-completion fish | source
+$ yq completion fish | source
 
 # To load completions for each session, execute once:
-$ yq shell-completion fish > ~/.config/fish/completions/yq.fish
+$ yq completion fish > ~/.config/fish/completions/yq.fish
 `,
 	DisableFlagsInUseLine: true,
 	ValidArgs:             []string{"bash", "zsh", "fish", "powershell"},

cmd/constant.go

@@ -2,6 +2,8 @@ package cmd
 
 var unwrapScalarFlag = newUnwrapFlag()
 
+var printNodeInfo = false
+
 var unwrapScalar = false
 
 var writeInplace = false
@@ -12,9 +14,6 @@ var outputFormat = ""
 var inputFormat = ""
 
 var exitStatus = false
-var forceColor = false
-var forceNoColor = false
-var colorsEnabled = false
 var indent = 2
 var noDocSeparators = false
 var nullInput = false
@@ -23,6 +22,10 @@ var verbose = false
 var version = false
 var prettyPrint = false
 
+var forceColor = false
+var forceNoColor = false
+var colorsEnabled = false
+
 // can be either "" (off), "extract" or "process"
 var frontMatter = ""
 

cmd/evaluate_all_command.go

@@ -12,7 +12,7 @@ func createEvaluateAllCommand() *cobra.Command {
 		Use:     "eval-all [expression] [yaml_file1]...",
 		Aliases: []string{"ea"},
 		Short:   "Loads _all_ yaml documents of _all_ yaml files and runs expression once",
-		ValidArgsFunction: func(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) {
+		ValidArgsFunction: func(_ *cobra.Command, args []string, _ string) ([]string, cobra.ShellCompDirective) {
 			if len(args) == 0 {
 				return nil, cobra.ShellCompDirectiveNoFileComp
 			}

cmd/evaluate_all_command_test.go

@@ -0,0 +1,328 @@
+package cmd
+
+import (
+	"bytes"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+)
+
+func TestCreateEvaluateAllCommand(t *testing.T) {
+	cmd := createEvaluateAllCommand()
+
+	if cmd == nil {
+		t.Fatal("createEvaluateAllCommand returned nil")
+	}
+
+	// Test basic command properties
+	if cmd.Use != "eval-all [expression] [yaml_file1]..." {
+		t.Errorf("Expected Use to be 'eval-all [expression] [yaml_file1]...', got %q", cmd.Use)
+	}
+
+	if cmd.Short == "" {
+		t.Error("Expected Short description to be non-empty")
+	}
+
+	if cmd.Long == "" {
+		t.Error("Expected Long description to be non-empty")
+	}
+
+	// Test aliases
+	expectedAliases := []string{"ea"}
+	if len(cmd.Aliases) != len(expectedAliases) {
+		t.Errorf("Expected %d aliases, got %d", len(expectedAliases), len(cmd.Aliases))
+	}
+
+	for i, expected := range expectedAliases {
+		if i >= len(cmd.Aliases) || cmd.Aliases[i] != expected {
+			t.Errorf("Expected alias %d to be %q, got %q", i, expected, cmd.Aliases[i])
+		}
+	}
+}
+
+func TestEvaluateAll_NoArgs(t *testing.T) {
+	// Create a temporary command
+	cmd := createEvaluateAllCommand()
+
+	// Set up command to capture output
+	var output bytes.Buffer
+	cmd.SetOut(&output)
+
+	// Test with no arguments and no null input
+	nullInput = false
+	defer func() { nullInput = false }()
+
+	err := evaluateAll(cmd, []string{})
+
+	// Should not error, but should print usage
+	if err != nil {
+		t.Errorf("evaluateAll with no args should not error, got: %v", err)
+	}
+
+	// Should have printed usage information
+	if output.Len() == 0 {
+		t.Error("Expected usage information to be printed")
+	}
+}
+
+func TestEvaluateAll_NullInput(t *testing.T) {
+	// Create a temporary command
+	cmd := createEvaluateAllCommand()
+
+	// Set up command to capture output
+	var output bytes.Buffer
+	cmd.SetOut(&output)
+
+	// Test with null input
+	nullInput = true
+	defer func() { nullInput = false }()
+
+	err := evaluateAll(cmd, []string{})
+
+	// Should not error when using null input
+	if err != nil {
+		t.Errorf("evaluateAll with null input should not error, got: %v", err)
+	}
+}
+
+func TestEvaluateAll_WithSingleFile(t *testing.T) {
+	// Create a temporary YAML file
+	tempDir := t.TempDir()
+	yamlFile := filepath.Join(tempDir, "test.yaml")
+	yamlContent := []byte("name: test\nage: 25\n")
+	err := os.WriteFile(yamlFile, yamlContent, 0600)
+	if err != nil {
+		t.Fatalf("Failed to create test YAML file: %v", err)
+	}
+
+	// Create a temporary command
+	cmd := createEvaluateAllCommand()
+
+	// Set up command to capture output
+	var output bytes.Buffer
+	cmd.SetOut(&output)
+
+	// Test with a single file
+	err = evaluateAll(cmd, []string{yamlFile})
+
+	// Should not error
+	if err != nil {
+		t.Errorf("evaluateAll with single file should not error, got: %v", err)
+	}
+
+	// Should have some output
+	if output.Len() == 0 {
+		t.Error("Expected output from evaluateAll with single file")
+	}
+}
+
+func TestEvaluateAll_WithMultipleFiles(t *testing.T) {
+	// Create temporary YAML files
+	tempDir := t.TempDir()
+
+	yamlFile1 := filepath.Join(tempDir, "test1.yaml")
+	yamlContent1 := []byte("name: test1\nage: 25\n")
+	err := os.WriteFile(yamlFile1, yamlContent1, 0600)
+	if err != nil {
+		t.Fatalf("Failed to create test YAML file 1: %v", err)
+	}
+
+	yamlFile2 := filepath.Join(tempDir, "test2.yaml")
+	yamlContent2 := []byte("name: test2\nage: 30\n")
+	err = os.WriteFile(yamlFile2, yamlContent2, 0600)
+	if err != nil {
+		t.Fatalf("Failed to create test YAML file 2: %v", err)
+	}
+
+	// Create a temporary command
+	cmd := createEvaluateAllCommand()
+
+	// Set up command to capture output
+	var output bytes.Buffer
+	cmd.SetOut(&output)
+
+	// Test with multiple files
+	err = evaluateAll(cmd, []string{yamlFile1, yamlFile2})
+
+	// Should not error
+	if err != nil {
+		t.Errorf("evaluateAll with multiple files should not error, got: %v", err)
+	}
+
+	// Should have output
+	if output.Len() == 0 {
+		t.Error("Expected output from evaluateAll with multiple files")
+	}
+}
+
+func TestEvaluateAll_WithExpression(t *testing.T) {
+	// Create a temporary YAML file
+	tempDir := t.TempDir()
+	yamlFile := filepath.Join(tempDir, "test.yaml")
+	yamlContent := []byte("name: test\nage: 25\n")
+	err := os.WriteFile(yamlFile, yamlContent, 0600)
+	if err != nil {
+		t.Fatalf("Failed to create test YAML file: %v", err)
+	}
+
+	// Create a temporary command
+	cmd := createEvaluateAllCommand()
+
+	// Set up command to capture output
+	var output bytes.Buffer
+	cmd.SetOut(&output)
+
+	// Test with expression
+	err = evaluateAll(cmd, []string{".name", yamlFile})
+
+	// Should not error
+	if err != nil {
+		t.Errorf("evaluateAll with expression should not error, got: %v", err)
+	}
+
+	// Should have output
+	if output.Len() == 0 {
+		t.Error("Expected output from evaluateAll with expression")
+	}
+}
+
+func TestEvaluateAll_WriteInPlace(t *testing.T) {
+	// Create a temporary YAML file
+	tempDir := t.TempDir()
+	yamlFile := filepath.Join(tempDir, "test.yaml")
+	yamlContent := []byte("name: test\nage: 25\n")
+	err := os.WriteFile(yamlFile, yamlContent, 0600)
+	if err != nil {
+		t.Fatalf("Failed to create test YAML file: %v", err)
+	}
+
+	// Create a temporary command
+	cmd := createEvaluateAllCommand()
+
+	// Set up command to capture output
+	var output bytes.Buffer
+	cmd.SetOut(&output)
+
+	// Enable write in place
+	originalWriteInplace := writeInplace
+	writeInplace = true
+	defer func() { writeInplace = originalWriteInplace }()
+
+	// Test with write in place
+	err = evaluateAll(cmd, []string{".name = \"updated\"", yamlFile})
+
+	// Should not error
+	if err != nil {
+		t.Errorf("evaluateAll with write in place should not error, got: %v", err)
+	}
+
+	// Verify the file was updated
+	updatedContent, err := os.ReadFile(yamlFile)
+	if err != nil {
+		t.Fatalf("Failed to read updated file: %v", err)
+	}
+
+	// Should contain the updated content
+	if !strings.Contains(string(updatedContent), "updated") {
+		t.Errorf("Expected file to contain 'updated', got: %s", string(updatedContent))
+	}
+}
+
+func TestEvaluateAll_ExitStatus(t *testing.T) {
+	// Create a temporary YAML file
+	tempDir := t.TempDir()
+	yamlFile := filepath.Join(tempDir, "test.yaml")
+	yamlContent := []byte("name: test\nage: 25\n")
+	err := os.WriteFile(yamlFile, yamlContent, 0600)
+	if err != nil {
+		t.Fatalf("Failed to create test YAML file: %v", err)
+	}
+
+	// Create a temporary command
+	cmd := createEvaluateAllCommand()
+
+	// Set up command to capture output
+	var output bytes.Buffer
+	cmd.SetOut(&output)
+
+	// Enable exit status
+	originalExitStatus := exitStatus
+	exitStatus = true
+	defer func() { exitStatus = originalExitStatus }()
+
+	// Test with expression that should find no matches
+	err = evaluateAll(cmd, []string{".nonexistent", yamlFile})
+
+	// Should error when no matches found and exit status is enabled
+	if err == nil {
+		t.Error("Expected error when no matches found and exit status is enabled")
+	}
+}
+
+func TestEvaluateAll_WithMultipleDocuments(t *testing.T) {
+	// Create a temporary YAML file with multiple documents
+	tempDir := t.TempDir()
+	yamlFile := filepath.Join(tempDir, "test.yaml")
+	yamlContent := []byte("---\nname: doc1\nage: 25\n---\nname: doc2\nage: 30\n")
+	err := os.WriteFile(yamlFile, yamlContent, 0600)
+	if err != nil {
+		t.Fatalf("Failed to create test YAML file: %v", err)
+	}
+
+	// Create a temporary command
+	cmd := createEvaluateAllCommand()
+
+	// Set up command to capture output
+	var output bytes.Buffer
+	cmd.SetOut(&output)
+
+	// Test with multiple documents
+	err = evaluateAll(cmd, []string{".", yamlFile})
+
+	// Should not error
+	if err != nil {
+		t.Errorf("evaluateAll with multiple documents should not error, got: %v", err)
+	}
+
+	// Should have output
+	if output.Len() == 0 {
+		t.Error("Expected output from evaluateAll with multiple documents")
+	}
+}
+
+func TestEvaluateAll_NulSepOutput(t *testing.T) {
+	// Create a temporary YAML file
+	tempDir := t.TempDir()
+	yamlFile := filepath.Join(tempDir, "test.yaml")
+	yamlContent := []byte("name: test\nage: 25\n")
+	err := os.WriteFile(yamlFile, yamlContent, 0600)
+	if err != nil {
+		t.Fatalf("Failed to create test YAML file: %v", err)
+	}
+
+	// Create a temporary command
+	cmd := createEvaluateAllCommand()
+
+	// Set up command to capture output
+	var output bytes.Buffer
+	cmd.SetOut(&output)
+
+	// Enable nul separator output
+	originalNulSepOutput := nulSepOutput
+	nulSepOutput = true
+	defer func() { nulSepOutput = originalNulSepOutput }()
+
+	// Test with nul separator output
+	err = evaluateAll(cmd, []string{".name", yamlFile})
+
+	// Should not error
+	if err != nil {
+		t.Errorf("evaluateAll with nul separator output should not error, got: %v", err)
+	}
+
+	// Should have output
+	if output.Len() == 0 {
+		t.Error("Expected output from evaluateAll with nul separator output")
+	}
+}

cmd/evaluate_sequence_command.go

@@ -13,7 +13,7 @@ func createEvaluateSequenceCommand() *cobra.Command {
 		Use:     "eval [expression] [yaml_file1]...",
 		Aliases: []string{"e"},
 		Short:   "(default) Apply the expression to each document in each yaml file in sequence",
-		ValidArgsFunction: func(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) {
+		ValidArgsFunction: func(_ *cobra.Command, args []string, _ string) ([]string, cobra.ShellCompDirective) {
 			if len(args) == 0 {
 				return nil, cobra.ShellCompDirectiveNoFileComp
 			}
@@ -105,6 +105,11 @@ func evaluateSequence(cmd *cobra.Command, args []string) (cmdError error) {
 	}
 
 	printer := yqlib.NewPrinter(encoder, printerWriter)
+
+	if printNodeInfo {
+		printer = yqlib.NewNodeInfoPrinter(printerWriter)
+	}
+
 	if nulSepOutput {
 		printer.SetNulSepOutput(true)
 	}

cmd/evaluate_sequence_command_test.go

@@ -0,0 +1,276 @@
+package cmd
+
+import (
+	"bytes"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+)
+
+func TestCreateEvaluateSequenceCommand(t *testing.T) {
+	cmd := createEvaluateSequenceCommand()
+
+	if cmd == nil {
+		t.Fatal("createEvaluateSequenceCommand returned nil")
+	}
+
+	// Test basic command properties
+	if cmd.Use != "eval [expression] [yaml_file1]..." {
+		t.Errorf("Expected Use to be 'eval [expression] [yaml_file1]...', got %q", cmd.Use)
+	}
+
+	if cmd.Short == "" {
+		t.Error("Expected Short description to be non-empty")
+	}
+
+	if cmd.Long == "" {
+		t.Error("Expected Long description to be non-empty")
+	}
+
+	// Test aliases
+	expectedAliases := []string{"e"}
+	if len(cmd.Aliases) != len(expectedAliases) {
+		t.Errorf("Expected %d aliases, got %d", len(expectedAliases), len(cmd.Aliases))
+	}
+
+	for i, expected := range expectedAliases {
+		if i >= len(cmd.Aliases) || cmd.Aliases[i] != expected {
+			t.Errorf("Expected alias %d to be %q, got %q", i, expected, cmd.Aliases[i])
+		}
+	}
+}
+
+func TestProcessExpression(t *testing.T) {
+	// Reset global variables
+	originalPrettyPrint := prettyPrint
+	defer func() { prettyPrint = originalPrettyPrint }()
+
+	tests := []struct {
+		name        string
+		prettyPrint bool
+		expression  string
+		expected    string
+	}{
+		{
+			name:        "empty expression without pretty print",
+			prettyPrint: false,
+			expression:  "",
+			expected:    "",
+		},
+		{
+			name:        "empty expression with pretty print",
+			prettyPrint: true,
+			expression:  "",
+			expected:    `(... | (select(tag != "!!str"), select(tag == "!!str") | select(test("(?i)^(y|yes|n|no|on|off)$") | not))  ) style=""`,
+		},
+		{
+			name:        "simple expression without pretty print",
+			prettyPrint: false,
+			expression:  ".a.b",
+			expected:    ".a.b",
+		},
+		{
+			name:        "simple expression with pretty print",
+			prettyPrint: true,
+			expression:  ".a.b",
+			expected:    `.a.b | (... | (select(tag != "!!str"), select(tag == "!!str") | select(test("(?i)^(y|yes|n|no|on|off)$") | not))  ) style=""`,
+		},
+		{
+			name:        "complex expression with pretty print",
+			prettyPrint: true,
+			expression:  ".items[] | select(.active == true)",
+			expected:    `.items[] | select(.active == true) | (... | (select(tag != "!!str"), select(tag == "!!str") | select(test("(?i)^(y|yes|n|no|on|off)$") | not))  ) style=""`,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			prettyPrint = tt.prettyPrint
+			result := processExpression(tt.expression)
+			if result != tt.expected {
+				t.Errorf("processExpression(%q) = %q, want %q", tt.expression, result, tt.expected)
+			}
+		})
+	}
+}
+
+func TestEvaluateSequence_NoArgs(t *testing.T) {
+	// Create a temporary command
+	cmd := createEvaluateSequenceCommand()
+
+	// Set up command to capture output
+	var output bytes.Buffer
+	cmd.SetOut(&output)
+
+	// Test with no arguments and no null input
+	nullInput = false
+	defer func() { nullInput = false }()
+
+	err := evaluateSequence(cmd, []string{})
+
+	// Should not error, but should print usage
+	if err != nil {
+		t.Errorf("evaluateSequence with no args should not error, got: %v", err)
+	}
+
+	// Should have printed usage information
+	if output.Len() == 0 {
+		t.Error("Expected usage information to be printed")
+	}
+}
+
+func TestEvaluateSequence_NullInput(t *testing.T) {
+	// Create a temporary command
+	cmd := createEvaluateSequenceCommand()
+
+	// Set up command to capture output
+	var output bytes.Buffer
+	cmd.SetOut(&output)
+
+	// Test with null input
+	nullInput = true
+	defer func() { nullInput = false }()
+
+	err := evaluateSequence(cmd, []string{})
+
+	// Should not error when using null input
+	if err != nil {
+		t.Errorf("evaluateSequence with null input should not error, got: %v", err)
+	}
+}
+
+func TestEvaluateSequence_WithFile(t *testing.T) {
+	// Create a temporary YAML file
+	tempDir := t.TempDir()
+	yamlFile := filepath.Join(tempDir, "test.yaml")
+	yamlContent := []byte("name: test\nage: 25\n")
+	err := os.WriteFile(yamlFile, yamlContent, 0600)
+	if err != nil {
+		t.Fatalf("Failed to create test YAML file: %v", err)
+	}
+
+	// Create a temporary command
+	cmd := createEvaluateSequenceCommand()
+
+	// Set up command to capture output
+	var output bytes.Buffer
+	cmd.SetOut(&output)
+
+	// Test with a file
+	err = evaluateSequence(cmd, []string{yamlFile})
+
+	// Should not error
+	if err != nil {
+		t.Errorf("evaluateSequence with file should not error, got: %v", err)
+	}
+
+	// Should have some output
+	if output.Len() == 0 {
+		t.Error("Expected output from evaluateSequence with file")
+	}
+}
+
+func TestEvaluateSequence_WithExpressionAndFile(t *testing.T) {
+	// Create a temporary YAML file
+	tempDir := t.TempDir()
+	yamlFile := filepath.Join(tempDir, "test.yaml")
+	yamlContent := []byte("name: test\nage: 25\n")
+	err := os.WriteFile(yamlFile, yamlContent, 0600)
+	if err != nil {
+		t.Fatalf("Failed to create test YAML file: %v", err)
+	}
+
+	// Create a temporary command
+	cmd := createEvaluateSequenceCommand()
+
+	// Set up command to capture output
+	var output bytes.Buffer
+	cmd.SetOut(&output)
+
+	// Test with expression and file
+	err = evaluateSequence(cmd, []string{".name", yamlFile})
+
+	// Should not error
+	if err != nil {
+		t.Errorf("evaluateSequence with expression and file should not error, got: %v", err)
+	}
+
+	// Should have output
+	if output.Len() == 0 {
+		t.Error("Expected output from evaluateSequence with expression and file")
+	}
+}
+
+func TestEvaluateSequence_WriteInPlace(t *testing.T) {
+	// Create a temporary YAML file
+	tempDir := t.TempDir()
+	yamlFile := filepath.Join(tempDir, "test.yaml")
+	yamlContent := []byte("name: test\nage: 25\n")
+	err := os.WriteFile(yamlFile, yamlContent, 0600)
+	if err != nil {
+		t.Fatalf("Failed to create test YAML file: %v", err)
+	}
+
+	// Create a temporary command
+	cmd := createEvaluateSequenceCommand()
+
+	// Set up command to capture output
+	var output bytes.Buffer
+	cmd.SetOut(&output)
+
+	// Enable write in place
+	originalWriteInplace := writeInplace
+	writeInplace = true
+	defer func() { writeInplace = originalWriteInplace }()
+
+	// Test with write in place
+	err = evaluateSequence(cmd, []string{".name = \"updated\"", yamlFile})
+
+	// Should not error
+	if err != nil {
+		t.Errorf("evaluateSequence with write in place should not error, got: %v", err)
+	}
+
+	// Verify the file was updated
+	updatedContent, err := os.ReadFile(yamlFile)
+	if err != nil {
+		t.Fatalf("Failed to read updated file: %v", err)
+	}
+
+	// Should contain the updated content
+	if !strings.Contains(string(updatedContent), "updated") {
+		t.Errorf("Expected file to contain 'updated', got: %s", string(updatedContent))
+	}
+}
+
+func TestEvaluateSequence_ExitStatus(t *testing.T) {
+	// Create a temporary YAML file
+	tempDir := t.TempDir()
+	yamlFile := filepath.Join(tempDir, "test.yaml")
+	yamlContent := []byte("name: test\nage: 25\n")
+	err := os.WriteFile(yamlFile, yamlContent, 0600)
+	if err != nil {
+		t.Fatalf("Failed to create test YAML file: %v", err)
+	}
+
+	// Create a temporary command
+	cmd := createEvaluateSequenceCommand()
+
+	// Set up command to capture output
+	var output bytes.Buffer
+	cmd.SetOut(&output)
+
+	// Enable exit status
+	originalExitStatus := exitStatus
+	exitStatus = true
+	defer func() { exitStatus = originalExitStatus }()
+
+	// Test with expression that should find no matches
+	err = evaluateSequence(cmd, []string{".nonexistent", yamlFile})
+
+	// Should error when no matches found and exit status is enabled
+	if err == nil {
+		t.Error("Expected error when no matches found and exit status is enabled")
+	}
+}

cmd/root.go

@@ -66,14 +66,23 @@ yq -P -oy sample.json
 			return evaluateSequence(cmd, args)
 
 		},
-		PersistentPreRunE: func(cmd *cobra.Command, args []string) error {
+		PersistentPreRunE: func(cmd *cobra.Command, _ []string) error {
 			cmd.SetOut(cmd.OutOrStdout())
 			level := logging.WARNING
 			stringFormat := `[%{level}] %{color}%{time:15:04:05}%{color:reset} %{message}`
 
-			if verbose {
+			// when NO_COLOR environment variable presents and not an empty string the coloured output should be disabled;
+			// refer to no-color.org
+			forceNoColor = forceNoColor || os.Getenv("NO_COLOR") != ""
+
+			if verbose && forceNoColor {
+				level = logging.DEBUG
+				stringFormat = `[%{level:5.5s}] %{time:15:04:05} %{shortfile:-33s} %{shortfunc:-25s} %{message}`
+			} else if verbose {
 				level = logging.DEBUG
 				stringFormat = `[%{level:5.5s}] %{color}%{time:15:04:05}%{color:bold} %{shortfile:-33s} %{shortfunc:-25s}%{color:reset} %{message}`
+			} else if forceNoColor {
+				stringFormat = `[%{level}] %{time:15:04:05} %{message}`
 			}
 
 			var format = logging.MustStringFormatter(stringFormat)
@@ -90,6 +99,7 @@ yq -P -oy sample.json
 	}
 
 	rootCmd.PersistentFlags().BoolVarP(&verbose, "verbose", "v", false, "verbose mode")
+	rootCmd.PersistentFlags().BoolVarP(&printNodeInfo, "debug-node-info", "", false, "debug node info")
 
 	rootCmd.PersistentFlags().BoolVarP(&outputToJSON, "tojson", "j", false, "(deprecated) output as json. Set indent to 0 to print json in one line.")
 	err := rootCmd.PersistentFlags().MarkDeprecated("tojson", "please use -o=json instead")
@@ -158,6 +168,11 @@ yq -P -oy sample.json
 	rootCmd.PersistentFlags().StringVar(&yqlib.ConfiguredPropertiesPreferences.KeyValueSeparator, "properties-separator", yqlib.ConfiguredPropertiesPreferences.KeyValueSeparator, "separator to use between keys and values")
 	rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredPropertiesPreferences.UseArrayBrackets, "properties-array-brackets", yqlib.ConfiguredPropertiesPreferences.UseArrayBrackets, "use [x] in array paths (e.g. for SpringBoot)")
 
+	rootCmd.PersistentFlags().StringVar(&yqlib.ConfiguredShellVariablesPreferences.KeySeparator, "shell-key-separator", yqlib.ConfiguredShellVariablesPreferences.KeySeparator, "separator for shell variable key paths")
+	if err = rootCmd.RegisterFlagCompletionFunc("shell-key-separator", cobra.NoFileCompletions); err != nil {
+		panic(err)
+	}
+
 	rootCmd.PersistentFlags().BoolVar(&yqlib.StringInterpolationEnabled, "string-interpolation", yqlib.StringInterpolationEnabled, "Toggles strings interpolation of \\(exp)")
 
 	rootCmd.PersistentFlags().BoolVarP(&nullInput, "null-input", "n", false, "Don't read input, simply evaluate the expression given. Useful for creating docs from scratch.")
@@ -177,7 +192,7 @@ yq -P -oy sample.json
 	rootCmd.PersistentFlags().BoolVarP(&exitStatus, "exit-status", "e", false, "set exit status if there are no matches or null or false is returned")
 
 	rootCmd.PersistentFlags().BoolVarP(&forceColor, "colors", "C", false, "force print with colors")
-	rootCmd.PersistentFlags().BoolVarP(&forceNoColor, "no-colors", "M", false, "force print with no colors")
+	rootCmd.PersistentFlags().BoolVarP(&forceNoColor, "no-colors", "M", forceNoColor, "force print with no colors")
 	rootCmd.PersistentFlags().StringVarP(&frontMatter, "front-matter", "f", "", "(extract|process) first input as yaml front-matter. Extract will pull out the yaml content, process will run the expression against the yaml content, leaving the remaining data intact")
 	if err = rootCmd.RegisterFlagCompletionFunc("front-matter", cobra.FixedCompletions([]string{"extract", "process"}, cobra.ShellCompDirectiveNoFileComp)); err != nil {
 		panic(err)
@@ -187,8 +202,9 @@ yq -P -oy sample.json
 		panic(err)
 	}
 	rootCmd.PersistentFlags().BoolVarP(&yqlib.ConfiguredYamlPreferences.LeadingContentPreProcessing, "header-preprocess", "", true, "Slurp any header comments and separators before processing expression.")
+	rootCmd.PersistentFlags().BoolVarP(&yqlib.ConfiguredYamlPreferences.FixMergeAnchorToSpec, "yaml-fix-merge-anchor-to-spec", "", false, "Fix merge anchor to match YAML spec. Will default to true in late 2025")
 
-	rootCmd.PersistentFlags().StringVarP(&splitFileExp, "split-exp", "s", "", "print each result (or doc) into a file named (exp). [exp] argument must return a string. You can use $index in the expression as the result counter.")
+	rootCmd.PersistentFlags().StringVarP(&splitFileExp, "split-exp", "s", "", "print each result (or doc) into a file named (exp). [exp] argument must return a string. You can use $index in the expression as the result counter. The necessary directories will be created.")
 	if err = rootCmd.RegisterFlagCompletionFunc("split-exp", cobra.NoFileCompletions); err != nil {
 		panic(err)
 	}
@@ -202,6 +218,9 @@ yq -P -oy sample.json
 		panic(err)
 	}
 
+	rootCmd.PersistentFlags().BoolVarP(&yqlib.ConfiguredSecurityPreferences.DisableEnvOps, "security-disable-env-ops", "", false, "Disable env related operations.")
+	rootCmd.PersistentFlags().BoolVarP(&yqlib.ConfiguredSecurityPreferences.DisableFileOps, "security-disable-file-ops", "", false, "Disable file related operations (e.g. load)")
+
 	rootCmd.AddCommand(
 		createEvaluateSequenceCommand(),
 		createEvaluateAllCommand(),

cmd/root_test.go

@@ -0,0 +1,264 @@
+package cmd
+
+import (
+	"strings"
+	"testing"
+)
+
+func TestNewRuneVar(t *testing.T) {
+	var r rune
+	runeVar := newRuneVar(&r)
+
+	if runeVar == nil {
+		t.Fatal("newRuneVar returned nil")
+	}
+}
+
+func TestRuneValue_String(t *testing.T) {
+	tests := []struct {
+		name     string
+		runeVal  rune
+		expected string
+	}{
+		{
+			name:     "simple character",
+			runeVal:  'a',
+			expected: "a",
+		},
+		{
+			name:     "special character",
+			runeVal:  '\n',
+			expected: "\n",
+		},
+		{
+			name:     "unicode character",
+			runeVal:  'ñ',
+			expected: "ñ",
+		},
+		{
+			name:     "zero rune",
+			runeVal:  0,
+			expected: string(rune(0)),
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			runeVal := runeValue(tt.runeVal)
+			result := runeVal.String()
+			if result != tt.expected {
+				t.Errorf("runeValue.String() = %q, want %q", result, tt.expected)
+			}
+		})
+	}
+}
+
+func TestRuneValue_Set(t *testing.T) {
+	tests := []struct {
+		name        string
+		input       string
+		expected    rune
+		expectError bool
+	}{
+		{
+			name:        "simple character",
+			input:       "a",
+			expected:    'a',
+			expectError: false,
+		},
+		{
+			name:        "newline escape",
+			input:       "\\n",
+			expected:    '\n',
+			expectError: false,
+		},
+		{
+			name:        "tab escape",
+			input:       "\\t",
+			expected:    '\t',
+			expectError: false,
+		},
+		{
+			name:        "carriage return escape",
+			input:       "\\r",
+			expected:    '\r',
+			expectError: false,
+		},
+		{
+			name:        "form feed escape",
+			input:       "\\f",
+			expected:    '\f',
+			expectError: false,
+		},
+		{
+			name:        "vertical tab escape",
+			input:       "\\v",
+			expected:    '\v',
+			expectError: false,
+		},
+		{
+			name:        "empty string",
+			input:       "",
+			expected:    0,
+			expectError: true,
+		},
+		{
+			name:        "multiple characters",
+			input:       "ab",
+			expected:    0,
+			expectError: true,
+		},
+		{
+			name:        "special character",
+			input:       "ñ",
+			expected:    'ñ',
+			expectError: true, // This will fail because the Set function checks len(val) != 1
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			var r rune
+			runeVal := newRuneVar(&r)
+
+			err := runeVal.Set(tt.input)
+
+			if tt.expectError {
+				if err == nil {
+					t.Errorf("Expected error for input %q, but got none", tt.input)
+				}
+			} else {
+				if err != nil {
+					t.Errorf("Unexpected error for input %q: %v", tt.input, err)
+				}
+				if r != tt.expected {
+					t.Errorf("Expected rune %q (%d), got %q (%d)",
+						string(tt.expected), tt.expected, string(r), r)
+				}
+			}
+		})
+	}
+}
+
+func TestRuneValue_Set_ErrorMessages(t *testing.T) {
+	tests := []struct {
+		name          string
+		input         string
+		expectedError string
+	}{
+		{
+			name:          "empty string error",
+			input:         "",
+			expectedError: "[] is not a valid character. Must be length 1 was 0",
+		},
+		{
+			name:          "multiple characters error",
+			input:         "abc",
+			expectedError: "[abc] is not a valid character. Must be length 1 was 3",
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			var r rune
+			runeVal := newRuneVar(&r)
+
+			err := runeVal.Set(tt.input)
+
+			if err == nil {
+				t.Errorf("Expected error for input %q, but got none", tt.input)
+				return
+			}
+
+			if !strings.Contains(err.Error(), tt.expectedError) {
+				t.Errorf("Expected error message to contain %q, got %q",
+					tt.expectedError, err.Error())
+			}
+		})
+	}
+}
+
+func TestRuneValue_Type(t *testing.T) {
+	var r rune
+	runeVal := newRuneVar(&r)
+
+	result := runeVal.Type()
+	expected := "char"
+
+	if result != expected {
+		t.Errorf("runeValue.Type() = %q, want %q", result, expected)
+	}
+}
+
+func TestNew(t *testing.T) {
+	rootCmd := New()
+
+	if rootCmd == nil {
+		t.Fatal("New() returned nil")
+	}
+
+	// Test basic command properties
+	if rootCmd.Use != "yq" {
+		t.Errorf("Expected Use to be 'yq', got %q", rootCmd.Use)
+	}
+
+	if rootCmd.Short == "" {
+		t.Error("Expected Short description to be non-empty")
+	}
+
+	if rootCmd.Long == "" {
+		t.Error("Expected Long description to be non-empty")
+	}
+
+	// Test that the command has the expected subcommands
+	expectedCommands := []string{"eval", "eval-all", "completion"}
+	actualCommands := make([]string, 0, len(rootCmd.Commands()))
+
+	for _, cmd := range rootCmd.Commands() {
+		actualCommands = append(actualCommands, cmd.Name())
+	}
+
+	for _, expected := range expectedCommands {
+		found := false
+		for _, actual := range actualCommands {
+			if actual == expected {
+				found = true
+				break
+			}
+		}
+		if !found {
+			t.Errorf("Expected command %q not found in actual commands: %v",
+				expected, actualCommands)
+		}
+	}
+}
+
+func TestNew_FlagCompletions(t *testing.T) {
+	rootCmd := New()
+
+	// Test that flag completion functions are registered
+	// This is a basic smoke test - we can't easily test the actual completion logic
+	// without more complex setup
+	flags := []string{
+		"output-format",
+		"input-format",
+		"xml-attribute-prefix",
+		"xml-content-name",
+		"xml-proc-inst-prefix",
+		"xml-directive-name",
+		"lua-prefix",
+		"lua-suffix",
+		"properties-separator",
+		"indent",
+		"front-matter",
+		"expression",
+		"split-exp",
+	}
+
+	for _, flagName := range flags {
+		flag := rootCmd.PersistentFlags().Lookup(flagName)
+		if flag == nil {
+			t.Errorf("Expected flag %q to exist", flagName)
+		}
+	}
+}

cmd/utils.go

@@ -18,52 +18,100 @@ func isAutomaticOutputFormat() bool {
 func initCommand(cmd *cobra.Command, args []string) (string, []string, error) {
 	cmd.SilenceUsage = true
 
-	fileInfo, _ := os.Stdout.Stat()
-
-	if forceColor || (!forceNoColor && (fileInfo.Mode()&os.ModeCharDevice) != 0) {
-		colorsEnabled = true
-	}
+	setupColors()
 
 	expression, args, err := processArgs(args)
 	if err != nil {
 		return "", nil, err
 	}
 
+	if err := loadSplitFileExpression(); err != nil {
+		return "", nil, err
+	}
+
+	handleBackwardsCompatibility()
+
+	if err := validateCommandFlags(args); err != nil {
+		return "", nil, err
+	}
+
+	if err := configureFormats(args); err != nil {
+		return "", nil, err
+	}
+
+	configureUnwrapScalar()
+
+	return expression, args, nil
+}
+
+func setupColors() {
+	fileInfo, _ := os.Stdout.Stat()
+
+	if forceColor || (!forceNoColor && (fileInfo.Mode()&os.ModeCharDevice) != 0) {
+		colorsEnabled = true
+	}
+}
+
+func loadSplitFileExpression() error {
 	if splitFileExpFile != "" {
 		splitExpressionBytes, err := os.ReadFile(splitFileExpFile)
 		if err != nil {
-			return "", nil, err
+			return err
 		}
 		splitFileExp = string(splitExpressionBytes)
 	}
+	return nil
+}
 
+func handleBackwardsCompatibility() {
 	// backwards compatibility
 	if outputToJSON {
 		outputFormat = "json"
 	}
+}
 
+func validateCommandFlags(args []string) error {
 	if writeInplace && (len(args) == 0 || args[0] == "-") {
-		return "", nil, fmt.Errorf("write in place flag only applicable when giving an expression and at least one file")
+		return fmt.Errorf("write in place flag only applicable when giving an expression and at least one file")
 	}
 
 	if frontMatter != "" && len(args) == 0 {
-		return "", nil, fmt.Errorf("front matter flag only applicable when giving an expression and at least one file")
+		return fmt.Errorf("front matter flag only applicable when giving an expression and at least one file")
 	}
 
 	if writeInplace && splitFileExp != "" {
-		return "", nil, fmt.Errorf("write in place cannot be used with split file")
+		return fmt.Errorf("write in place cannot be used with split file")
 	}
 
 	if nullInput && len(args) > 0 {
-		return "", nil, fmt.Errorf("cannot pass files in when using null-input flag")
+		return fmt.Errorf("cannot pass files in when using null-input flag")
 	}
 
+	return nil
+}
+
+func configureFormats(args []string) error {
 	inputFilename := ""
 	if len(args) > 0 {
 		inputFilename = args[0]
 	}
-	if inputFormat == "" || inputFormat == "auto" || inputFormat == "a" {
 
+	if err := configureInputFormat(inputFilename); err != nil {
+		return err
+	}
+
+	if err := configureOutputFormat(); err != nil {
+		return err
+	}
+
+	yqlib.GetLogger().Debug("Using input format %v", inputFormat)
+	yqlib.GetLogger().Debug("Using output format %v", outputFormat)
+
+	return nil
+}
+
+func configureInputFormat(inputFilename string) error {
+	if inputFormat == "" || inputFormat == "auto" || inputFormat == "a" {
 		inputFormat = yqlib.FormatStringFromFilename(inputFilename)
 
 		_, err := yqlib.FormatFromString(inputFormat)
@@ -75,10 +123,6 @@ func initCommand(cmd *cobra.Command, args []string) (string, []string, error) {
 				outputFormat = "yaml"
 			}
 		} else if isAutomaticOutputFormat() {
-			// automatic input worked, we can do it for output too unless specified
-			if inputFormat == "json" {
-				yqlib.GetLogger().Warning("JSON file output is now JSON by default (instead of yaml). Use '-oy' or '--output-format=yaml' for yaml output")
-			}
 			outputFormat = inputFormat
 		}
 	} else if isAutomaticOutputFormat() {
@@ -92,24 +136,27 @@ func initCommand(cmd *cobra.Command, args []string) (string, []string, error) {
 		}
 		outputFormat = "yaml"
 	}
+	return nil
+}
 
+func configureOutputFormat() error {
 	outputFormatType, err := yqlib.FormatFromString(outputFormat)
-
 	if err != nil {
-		return "", nil, err
+		return err
 	}
-	yqlib.GetLogger().Debug("Using input format %v", inputFormat)
-	yqlib.GetLogger().Debug("Using output format %v", outputFormat)
 
 	if outputFormatType == yqlib.YamlFormat ||
 		outputFormatType == yqlib.PropertiesFormat {
 		unwrapScalar = true
 	}
+
+	return nil
+}
+
+func configureUnwrapScalar() {
 	if unwrapScalarFlag.IsExplicitlySet() {
 		unwrapScalar = unwrapScalarFlag.IsSet()
 	}
-
-	return expression, args, nil
 }
 
 func configureDecoder(evaluateTogether bool) (yqlib.Decoder, error) {
@@ -158,6 +205,7 @@ func configureEncoder() (yqlib.Encoder, error) {
 
 	yqlib.ConfiguredYamlPreferences.ColorsEnabled = colorsEnabled
 	yqlib.ConfiguredJSONPreferences.ColorsEnabled = colorsEnabled
+	yqlib.ConfiguredHclPreferences.ColorsEnabled = colorsEnabled
 
 	yqlib.ConfiguredYamlPreferences.PrintDocSeparators = !noDocSeparators
 

cmd/version.go

@@ -11,7 +11,7 @@ var (
 	GitDescribe string
 
 	// Version is main version number that is being run at the moment.
-	Version = "v4.42.1"
+	Version = "v4.50.1"
 
 	// VersionPrerelease is a pre-release marker for the version. If this is "" (empty string)
 	// then it means that it is a final release. Otherwise, this is a pre-release
@@ -45,5 +45,5 @@ func getHumanVersion() string {
 	}
 
 	// Strip off any single quotes added by the git information.
-	return strings.Replace(version, "'", "", -1)
+	return strings.ReplaceAll(version, "'", "")
 }

debian/changelog

@@ -1,210 +0,0 @@
-yq (4.16.2) focal; urgency=medium
-
-  * Fixed with semicolon space issue
-  * Updating with documentation
-  * Added STDIN example to the top
-  * minor readme cleanup
-  * Help text tweak
-  * Fixed docker timeout - simplify docker builds
-  * New release with docker build fixes
-  * Updating to go 1.17 to fix CVE #944
-  * Fix a typo in root.go
-  * Skip the tests if the nocheck Debian build option is specified
-  * Fixed select bug (#958)
-  * Sped up explode operator
-  * Slight performance improvement to context.ChildContext
-  * Speed up multiply
-  * Update README with recently added / changed options
-  * Make deepMatch report in linear time
-  * Removed leadingContentPreProcessing flag - header preprocessing is stable
-  * Revert "Removed leadingContentPreProcessing flag - header preprocessing is stable"
-  * Keep flag, it is needed in corner cases
-  * Updated Readme
-  * Man page
-  * Fixed expression parsing bug #970
-  * Bumping go-lang, docker versions
-  * Added test release flow
-  * Updated github action release to generate man page
-  * Bumping version
-  * Removing no longer needed github action
-  * Added decoder op
-  * Fixed newline handling when decoding/encoding
-  * Fixed newline handling in encoder/decoder
-  * Can specify indent in encode ops
-  * Added group_by operator
-  * Added flatten operator
-  * Fixed flatten error message
-  * Improving docs
-  * Split printer
-  * Refactored command logic
-  * Fix JSON encoding removing null #985
-  * Fixed acceptance tests
-  * gitbook
-  * Update document generation script
-  * Updating README
-  * Updating release instructions
-  * github action no longer uses data1.yml
-  * Create dependabot.yml
-  * Bump actions/create-release from 1.0.0 to 1.1.4
-  * Bump actions/setup-go from 1 to 2.1.4
-  * Bump github.com/goccy/go-yaml from 1.8.9 to 1.9.4
-  * Bump github.com/jinzhu/copier from 0.2.8 to 0.3.2
-  * Bump github.com/fatih/color from 1.10.0 to 1.13.0
-  * Bump github.com/spf13/cobra from 1.1.3 to 1.2.1
-  * Update dependabot.yml
-  * Update go.yml
-  * add build check to PRs
-  * Include secure as part of build process
-  * Fixing bad label in github action
-  * fixed printer test
-  * remove leading content indicator
-  * Fixed header preprocessing!
-  * lint : define golangci configuration file
-  * Update check.sh
-  * Load file acceptance test
-  * Minor improvement on handling front matter
-  * Improved load doc
-  * feature: detect MANPATh and install there
-  * Update install-man-page.sh
-  * simplify prod stage, move version label to action
-  * add labels, quote some values
-  * enable errorlint linter
-  * Added errorlint to devtools
-  * Added key operator
-  * Added more tests
-  * Fixing comments
-  * Attempt to fix golint problem
-  * Include version query for tools
-  * Clean up errored file?
-  * enable misspell linter
-  * updated readme
-  * update Golangci version to v1.43.0
-  * gci linter
-  * Better merge array by key example
-  * Added credit for merge by array example
-  * Better formatting of merge arrays example
-  * Better merge example
-  * Add accessor for the yq logger instance (#1013)
-  * Fixed collect op when working with multiple nodes
-  * Added map, map_values
-  * Add support for Podman as well as Docker (#1026)
-  * Bump github.com/jinzhu/copier from 0.3.2 to 0.3.4 (#1027)
-  * Added csv, tsv output formats
-  * Added encoder tests
-  * Cleanup test
-  * Fixed docker permission issue #1014
-  * Recording release notes for next release
-  * Assignment op no longer clobbers anchor (#1029)
-  * Added sort_by operator
-  * Improved error message
-  * Improved tips and tricks
-  * Report while filename failed to parse #1030
-  * Added script for extracting checksums
-  * Improved extract-checksum.sh
-  * Bump github.com/spf13/cobra from 1.2.1 to 1.3.0 (#1039)
-  * enable more linters (#1043)
-  * Bump Golang compiler #1037
-
- -- Roberto Mier Escandon <rmescandon@gmail.com>  Tue, 21 Dec 2021 09:41:44 +0000
-
-yq (4.13.0) focal; urgency=medium
-
-  * New `with` operator for making multiple changes to a given path
-  * New `contains` operator, works like the `jq` equivalent
-  * Subtract operator now supports subtracting elements from arrays!
-  * Fixed Swapping values using variables #934
-  * Github Action now properly supports multiline output #936, thanks @pjxiao
-  * Fixed missing closing bracket validation #932
-  * Fix processing of hex numbers #929
-  * Fixed alternative and union operator issues #930
-  * Can now convert yaml to properties properties format (`-o=props`), See [docs](https://mikefarah.gitbook.io/yq/v/v4.x/usage/properties) for more info.
-  * Fixed document header/footer comment handling when merging (https://github.com/mikefarah/yq/issues/919)
-  * pretty print yaml 1.1 compatibility  (https://github.com/mikefarah/yq/issues/914)
-
- -- Roberto Mier Escandon <rmescandon@gmail.com>  Thu, 16 Sep 2021 20:58:30 +0200
-
-yq (4.9.6) focal; urgency=medium
-
-  * Added darwin/arm64 build, thanks @alecthomas
-  * Incremented docker alpine base version, thanks @da6d6i7-bronga
-  * Bug fix: multiline expression
-  * Bug fix: special character
-
- -- Roberto Mier Escandon <rmescandon@gmail.com>  Tue, 29 Jun 2021 21:32:14 +0200
-
-yq (3.3.2) focal; urgency=medium
-
-  * Bug fix: existStatus bug (#459)
-  * Automatically makes an os temp directory if it does not exist (#461)
-
- -- Roberto Mier Escandon <rmescandon@gmail.com>  Fri, 07 Aug 2020 18:53:01 +0200
-
-yq (3.3-0) focal; urgency=medium
-
-  * You can control string styles (quotes) using the new --style flag
-  * String values now always have quotes when outputting to json
-  * Negative array indices now traverse the array backwards
-  * Added a --stripComments flag to print yaml without any comments
-  * Bumped go to version 1.14
-
- -- Roberto Mier Escandon <rmescandon@gmail.com>  Thu, 30 Apr 2020 20:45:44 +0200
-
-yq (3.1-2) eoan; urgency=medium
-
-  * Bug fix: yq 3 was removing empty inline-style objects and arrays (#355)
-  * Bug fix: Merge option returned different output when switching order of 
-    merging files(#347)
-  * Bug fix: Add new object to existing array object was failing in 3.1.1 (#361)
-  * Bug fix: yq 3 empty keys did not allow merging of values (#356)
-  * Bug fix: keys quoted during merge (#363)
-  * Bug fix: Correct length with wc -l (#362)
-  * Bug fix: Write to empty document removed path (#359)
-
- -- Roberto Mier Escandon <rmescandon@gmail.com>  Mon, 24 Feb 2020 20:31:58 +0100
-
-yq (3.1-1) eoan; urgency=medium
-
-  * Keeps yaml comments and formatting, can specify yaml tags when updating.
-  * Handles anchors
-  * Can print out matching paths and values when splatting
-  * JSON output works for all commands
-  * Yaml files with multiple documents are printed out as one JSON 
-    document per line.
-  * Deep splat (**) to match arbitrary paths
-  * Update scripts file format has changed to be more powerful
-  * Reading and splatting, matching results are printed once per line
-  * Bugfixing
-
- -- Roberto Mier Escandon <rmescandon@gmail.com>  Tue, 11 Feb 2020 22:18:24 +0100
-
-yq (2.2-1) bionic; urgency=medium
-
-  * Added Windows support for the "--inplace" command flag
-  * Prefix now supports arrays
-  * Add prefix command
-  * Bump Alpine version to 3.8
-  * Improved docker build process
-  * Lint fixes
-  * Build support for all linux architectures supported by gox
-
- -- Roberto Mier Escandon <rmescandon@gmail.com>  Sat, 19 Jan 2019 15:50:47 +0100
-
-yq (2.1-0) bionic; urgency=medium
-
-  * Ability to read multiple documents in a single file
-  * Ability to append list items instead of overwriting
-
- -- Roberto Mier Escandón <rmescandon@gmail.com>  Tue, 10 Jul 2018 14:02:42 +0200
-
-yq (2.0-0) bionic; urgency=medium
-
-  * Release 2.0.0
-
- -- Roberto Mier Escandón <rmescandon@gmail.com>  Wed, 20 Jun 2018 10:29:53 +0200
-
-yq (1.15-0) bionic; urgency=medium
-
-  * Release 1.15
-
- -- Roberto Mier Escandón <rmescandon@gmail.com>  Wed, 06 Jun 2018 11:32:03 +0200
-

debian/compat

@@ -1 +0,0 @@
-10

debian/control

@@ -1,22 +0,0 @@
-Source: yq
-Section: devel
-Priority: optional
-Maintainer: Roberto Mier Escandón <rmescandon@gmail.com>
-Build-Depends: debhelper (>=10),
-               golang-1.17-go,
-               pandoc,
-               rsync
-Standards-Version: 4.1.4
-Homepage: https://github.com/mikefarah/yq.git
-Vcs-Browser: https://github.com/mikefarah/yq.git
-Vcs-Git: https://github.com/mikefarah/yq.git
-XS-Go-Import-Path: github.com/mikefarah/yq
-XSBC-Original-Maintainer: Roberto Mier Escandón <rmescandon@gmail.com>
-
-Package: yq
-Architecture: any
-Depends: ${shlibs:Depends}, ${misc:Depends}
-Description: lightweight and portable command-line data file processor
- .
- The aim of the project is to be the 
- [jq](https://github.com/stedolan/jq) or sed of yaml files.

debian/copyright

@@ -1,24 +0,0 @@
-Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
-Upstream-Name: yq
-Source: https://github.com/mikefarah/yq.git
-
-Files: *
-Copyright: 2017 Mike Farah
-License: Expat
-    Permission is hereby granted, free of charge, to any person obtaining a copy
-    of this software and associated documentation files (the "Software"), to deal
-    in the Software without restriction, including without limitation the rights
-    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-    copies of the Software, and to permit persons to whom the Software is
-    furnished to do so, subject to the following conditions:
-    .
-    The above copyright notice and this permission notice shall be included in all
-    copies or substantial portions of the Software.
-    .
-    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-    SOFTWARE.

debian/gbp.conf

@@ -1,2 +0,0 @@
-[DEFAULT]
-pristine-tar = True

debian/rules

@@ -1,74 +0,0 @@
-#!/usr/bin/make -f
-#
-# Copyright (C) 2018-2021 Roberto Mier Escandón <rmescandon@gmail.com>
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 3 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-PROJECT := yq
-OWNER := mikefarah
-REPO := github.com
-
-export DH_OPTIONS
-export DH_GOPKG := ${REPO}/${OWNER}/${PROJECT}
-export GOROOT := /usr/local/go
-export GOPATH := ${CURDIR}/_build
-export GOBIN := ${GOPATH}/bin
-export PATH := ${GOROOT}/bin:${GOBIN}:${PATH}
-export GOCACHE := /tmp/gocache
-export GOFLAGS := -mod=vendor
-export GO111MODULE := on
-
-SRCDIR := ${GOPATH}/src/${DH_GOPKG}
-DESTDIR := ${CURDIR}/debian/${PROJECT}
-BINDIR := /usr/bin
-MANDIR := /usr/share/man/man1/
-ASSETSDIR := /usr/share/${PROJECT}
-
-%:
-	dh $@
-
-override_dh_auto_build:
-	mkdir -p ${SRCDIR}
-	mkdir -p ${GOBIN}
-	# copy project to local srcdir to build from there
-	rsync -avz --progress --exclude=_build --exclude=debian --exclude=tmp. --exclude=go.mod --exclude=docs . $(SRCDIR)
-	# build go code
-	( \
-		cd ${SRCDIR} && \
-		go install -buildmode=pie ./... \
-	)
-
-	# build man page
-	( \
-		cd ${SRCDIR} && \
-		./scripts/generate-man-page-md.sh && \
-		./scripts/generate-man-page.sh \
-	)
-	
-override_dh_auto_test:
-ifeq (,$(filter nocheck,$(DEB_BUILD_OPTIONS)))
-	(cd ${SRCDIR} && go test -v ./...)
-endif
-
-override_dh_auto_install:
-	cp ${GOBIN}/yq ${DESTDIR}/${BINDIR}
-	cp -f ${SRCDIR}/LICENSE ${DESTDIR}/${ASSETSDIR}
-	chmod a+x ${DESTDIR}/${BINDIR}/yq
-
-	# man
-	mkdir -p "${DESTDIR}"/"${MANDIR}"
-	cp "${SRCDIR}"/yq.1 "${DESTDIR}"/"${MANDIR}" \
-
-override_dh_auto_clean:
-	dh_clean
-	rm -rf ${CURDIR}/_build

debian/source/format

@@ -1 +0,0 @@
-3.0 (native)

debian/yq.dirs

@@ -1,3 +0,0 @@
-usr/bin
-usr/share/yq
-usr/share/man/man1

examples/data1.yaml

@@ -1,11 +1 @@
-# block comments come through
-person: # neither do comments on maps
-    name: Mike Wazowski # comments on values appear
-    pets: 
-    - cat # comments on array values appear
-    - dog # comments on array values appear
-    - things:
-        - frog
-    food: [pizza] # comments on arrays do not
-emptyArray: []
-emptyMap: []
+a: apple

examples/front-matter.yaml

@@ -1,6 +1,6 @@
 ---
 a: apple
-b: bannana
+b: banana
 ---
 hello there
 apples: great

examples/sample.hcl

@@ -0,0 +1,8 @@
+# Arithmetic with literals and application-provided variables
+sum = 1 + addend
+
+# String interpolation and templates
+message = "Hello, ${name}!"
+
+# Application-provided functions
+shouty_message = upper(message)

examples/sample.ini

@@ -0,0 +1,14 @@
+
+; This is a INI document
+
+[owner]
+name = "Tom Preston-Werner"
+dob = 1979-05-27T07:32:00-08:00
+
+[database]
+db_host = "localhost"
+db_port = 5432
+db_user = "postgres"
+db_password = ""
+db_name = "postgres"
+

examples/sample.toml

@@ -1,26 +1,6 @@
+[[fruits]]
 
+[animals]
 
-# This is a TOML document
-
-title = "TOML Example"
-
-[owner]
-name = "Tom Preston-Werner"
-dob = 1979-05-27T07:32:00-08:00
-
-[database]
-enabled = true
-ports = [ 8000, 8001, 8002 ]
-data = [ ["delta", "phi"], [3.14] ]
-temp_targets = { cpu = 79.5, case = 72.0 }
-
-[servers]
-
-[servers.alpha]
-ip = "10.0.0.1"
-role = "frontend"
-
-[servers.beta]
-ip = "10.0.0.2"
-role = "backend"
-
+[[fruits.varieties]]  # nested array of tables
+name = "red delicious"

examples/sample2.hcl

@@ -0,0 +1,8 @@
+# Arithmetic with literals and application-provided variables
+sum = 1 + addend
+
+# String interpolation and templates
+message = "Hello, ${name}!"
+
+# Application-provided functions
+shouty_message = upper(message)

examples/sample_no_sections.ini

@@ -0,0 +1,8 @@
+
+; This is a INI document
+name = "Tom Preston-Werner"
+dob = 1979-05-27T07:32:00-08:00
+enabled = true
+ip = "10.0.0.1"
+role = "frontend"
+treads = 4

go.mod

@@ -1,33 +1,44 @@
 module github.com/mikefarah/yq/v4
 
 require (
-	github.com/a8m/envsubst v1.4.2
-	github.com/alecthomas/participle/v2 v2.1.1
-	github.com/alecthomas/repr v0.4.0
+	github.com/a8m/envsubst v1.4.3
+	github.com/alecthomas/participle/v2 v2.1.4
+	github.com/alecthomas/repr v0.5.2
 	github.com/dimchansky/utfbom v1.1.1
-	github.com/elliotchance/orderedmap v1.5.1
-	github.com/fatih/color v1.16.0
-	github.com/goccy/go-json v0.10.2
-	github.com/goccy/go-yaml v1.11.3
+	github.com/elliotchance/orderedmap v1.8.0
+	github.com/fatih/color v1.18.0
+	github.com/go-ini/ini v1.67.0
+	github.com/goccy/go-json v0.10.5
+	github.com/goccy/go-yaml v1.19.0
+	github.com/hashicorp/hcl/v2 v2.24.0
 	github.com/jinzhu/copier v0.4.0
-	github.com/magiconair/properties v1.8.7
-	github.com/pelletier/go-toml/v2 v2.1.1
+	github.com/magiconair/properties v1.8.10
+	github.com/pelletier/go-toml/v2 v2.2.4
 	github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e
-	github.com/spf13/cobra v1.8.0
-	github.com/spf13/pflag v1.0.5
+	github.com/spf13/cobra v1.10.2
+	github.com/spf13/pflag v1.0.10
 	github.com/yuin/gopher-lua v1.1.1
-	golang.org/x/net v0.22.0
-	golang.org/x/text v0.14.0
+	github.com/zclconf/go-cty v1.17.0
+	go.yaml.in/yaml/v4 v4.0.0-rc.3
+	golang.org/x/net v0.48.0
+	golang.org/x/text v0.32.0
 	gopkg.in/op/go-logging.v1 v1.0.0-20160211212156-b2cb9fa56473
-	gopkg.in/yaml.v3 v3.0.1
 )
 
 require (
+	github.com/agext/levenshtein v1.2.1 // indirect
+	github.com/apparentlymart/go-textseg/v15 v15.0.0 // indirect
+	github.com/google/go-cmp v0.6.0 // indirect
 	github.com/inconshreveable/mousetrap v1.1.0 // indirect
-	github.com/mattn/go-colorable v0.1.13 // indirect
+	github.com/mattn/go-colorable v0.1.14 // indirect
 	github.com/mattn/go-isatty v0.0.20 // indirect
-	golang.org/x/sys v0.18.0 // indirect
-	golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 // indirect
+	github.com/mitchellh/go-wordwrap v1.0.1 // indirect
+	golang.org/x/mod v0.30.0 // indirect
+	golang.org/x/sync v0.19.0 // indirect
+	golang.org/x/sys v0.39.0 // indirect
+	golang.org/x/tools v0.39.0 // indirect
 )
 
-go 1.21
+go 1.24.0
+
+toolchain go1.24.1

go.sum

@@ -1,85 +1,90 @@
-github.com/a8m/envsubst v1.4.2 h1:4yWIHXOLEJHQEFd4UjrWDrYeYlV7ncFWJOCBRLOZHQg=
-github.com/a8m/envsubst v1.4.2/go.mod h1:MVUTQNGQ3tsjOOtKCNd+fl8RzhsXcDvvAEzkhGtlsbY=
-github.com/alecthomas/assert/v2 v2.3.0 h1:mAsH2wmvjsuvyBvAmCtm7zFsBlb8mIHx5ySLVdDZXL0=
-github.com/alecthomas/assert/v2 v2.3.0/go.mod h1:pXcQ2Asjp247dahGEmsZ6ru0UVwnkhktn7S0bBDLxvQ=
-github.com/alecthomas/participle/v2 v2.1.1 h1:hrjKESvSqGHzRb4yW1ciisFJ4p3MGYih6icjJvbsmV8=
-github.com/alecthomas/participle/v2 v2.1.1/go.mod h1:Y1+hAs8DHPmc3YUFzqllV+eSQ9ljPTk0ZkPMtEdAx2c=
-github.com/alecthomas/repr v0.4.0 h1:GhI2A8MACjfegCPVq9f1FLvIBS+DrQ2KQBFZP1iFzXc=
-github.com/alecthomas/repr v0.4.0/go.mod h1:Fr0507jx4eOXV7AlPV6AVZLYrLIuIeSOWtW57eE/O/4=
-github.com/cpuguy83/go-md2man/v2 v2.0.3/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
+github.com/a8m/envsubst v1.4.3 h1:kDF7paGK8QACWYaQo6KtyYBozY2jhQrTuNNuUxQkhJY=
+github.com/a8m/envsubst v1.4.3/go.mod h1:4jjHWQlZoaXPoLQUb7H2qT4iLkZDdmEQiOUogdUmqVU=
+github.com/agext/levenshtein v1.2.1 h1:QmvMAjj2aEICytGiWzmxoE0x2KZvE0fvmqMOfy2tjT8=
+github.com/agext/levenshtein v1.2.1/go.mod h1:JEDfjyjHDjOF/1e4FlBE/PkbqA9OfWu2ki2W0IB5558=
+github.com/alecthomas/assert/v2 v2.11.0 h1:2Q9r3ki8+JYXvGsDyBXwH3LcJ+WK5D0gc5E8vS6K3D0=
+github.com/alecthomas/assert/v2 v2.11.0/go.mod h1:Bze95FyfUr7x34QZrjL+XP+0qgp/zg8yS+TtBj1WA3k=
+github.com/alecthomas/participle/v2 v2.1.4 h1:W/H79S8Sat/krZ3el6sQMvMaahJ+XcM9WSI2naI7w2U=
+github.com/alecthomas/participle/v2 v2.1.4/go.mod h1:8tqVbpTX20Ru4NfYQgZf4mP18eXPTBViyMWiArNEgGI=
+github.com/alecthomas/repr v0.5.2 h1:SU73FTI9D1P5UNtvseffFSGmdNci/O6RsqzeXJtP0Qs=
+github.com/alecthomas/repr v0.5.2/go.mod h1:Fr0507jx4eOXV7AlPV6AVZLYrLIuIeSOWtW57eE/O/4=
+github.com/apparentlymart/go-textseg/v15 v15.0.0 h1:uYvfpb3DyLSCGWnctWKGj857c6ew1u1fNQOlOtuGxQY=
+github.com/apparentlymart/go-textseg/v15 v15.0.0/go.mod h1:K8XmNZdhEBkdlyDdvbmmsvpAG721bKi0joRfFdHIWJ4=
+github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/dimchansky/utfbom v1.1.1 h1:vV6w1AhK4VMnhBno/TPVCoK9U/LP0PkLCS9tbxHdi/U=
 github.com/dimchansky/utfbom v1.1.1/go.mod h1:SxdoEBH5qIqFocHMyGOXVAybYJdr71b1Q/j0mACtrfE=
-github.com/elliotchance/orderedmap v1.5.1 h1:G1X4PYlljzimbdQ3RXmtIZiQ9d6aRQ3sH1nzjq5mECE=
-github.com/elliotchance/orderedmap v1.5.1/go.mod h1:wsDwEaX5jEoyhbs7x93zk2H/qv0zwuhg4inXhDkYqys=
-github.com/fatih/color v1.16.0 h1:zmkK9Ngbjj+K0yRhTVONQh1p/HknKYSlNT+vZCzyokM=
-github.com/fatih/color v1.16.0/go.mod h1:fL2Sau1YI5c0pdGEVCbKQbLXB6edEj1ZgiY4NijnWvE=
-github.com/go-playground/locales v0.13.0 h1:HyWk6mgj5qFqCT5fjGBuRArbVDfE4hi8+e8ceBS/t7Q=
-github.com/go-playground/locales v0.13.0/go.mod h1:taPMhCMXrRLJO55olJkUXHZBHCxTMfnGwq/HNwmWNS8=
-github.com/go-playground/universal-translator v0.17.0 h1:icxd5fm+REJzpZx7ZfpaD876Lmtgy7VtROAbHHXk8no=
-github.com/go-playground/universal-translator v0.17.0/go.mod h1:UkSxE5sNxxRwHyU+Scu5vgOQjsIJAF8j9muTVoKLVtA=
-github.com/go-playground/validator/v10 v10.4.1 h1:pH2c5ADXtd66mxoE0Zm9SUhxE20r7aM3F26W0hOn+GE=
-github.com/go-playground/validator/v10 v10.4.1/go.mod h1:nlOn6nFhuKACm19sB/8EGNn9GlaMV7XkbRSipzJ0Ii4=
-github.com/goccy/go-json v0.10.2 h1:CrxCmQqYDkv1z7lO7Wbh2HN93uovUHgrECaO5ZrCXAU=
-github.com/goccy/go-json v0.10.2/go.mod h1:6MelG93GURQebXPDq3khkgXZkazVtN9CRI+MGFi0w8I=
-github.com/goccy/go-yaml v1.11.3 h1:B3W9IdWbvrUu2OYQGwvU1nZtvMQJPBKgBUuweJjLj6I=
-github.com/goccy/go-yaml v1.11.3/go.mod h1:wKnAMd44+9JAAnGQpWVEgBzGt3YuTaQ4uXoHvE4m7WU=
-github.com/google/go-cmp v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38=
-github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
+github.com/elliotchance/orderedmap v1.8.0 h1:TrOREecvh3JbS+NCgwposXG5ZTFHtEsQiCGOhPElnMw=
+github.com/elliotchance/orderedmap v1.8.0/go.mod h1:wsDwEaX5jEoyhbs7x93zk2H/qv0zwuhg4inXhDkYqys=
+github.com/fatih/color v1.18.0 h1:S8gINlzdQ840/4pfAwic/ZE0djQEH3wM94VfqLTZcOM=
+github.com/fatih/color v1.18.0/go.mod h1:4FelSpRwEGDpQ12mAdzqdOukCy4u8WUtOY6lkT/6HfU=
+github.com/go-ini/ini v1.67.0 h1:z6ZrTEZqSWOTyH2FlglNbNgARyHG8oLW9gMELqKr06A=
+github.com/go-ini/ini v1.67.0/go.mod h1:ByCAeIL28uOIIG0E3PJtZPDL8WnHpFKFOtgjp+3Ies8=
+github.com/go-test/deep v1.0.3 h1:ZrJSEWsXzPOxaZnFteGEfooLba+ju3FYIbOrS+rQd68=
+github.com/go-test/deep v1.0.3/go.mod h1:wGDj63lr65AM2AQyKZd/NYHGb0R+1RLqB8NKt3aSFNA=
+github.com/goccy/go-json v0.10.5 h1:Fq85nIqj+gXn/S5ahsiTlK3TmC85qgirsdTP/+DeaC4=
+github.com/goccy/go-json v0.10.5/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
+github.com/goccy/go-yaml v1.19.0 h1:EmkZ9RIsX+Uq4DYFowegAuJo8+xdX3T/2dwNPXbxEYE=
+github.com/goccy/go-yaml v1.19.0/go.mod h1:XBurs7gK8ATbW4ZPGKgcbrY1Br56PdM69F7LkFRi1kA=
+github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
+github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
+github.com/hashicorp/hcl/v2 v2.24.0 h1:2QJdZ454DSsYGoaE6QheQZjtKZSUs9Nh2izTWiwQxvE=
+github.com/hashicorp/hcl/v2 v2.24.0/go.mod h1:oGoO1FIQYfn/AgyOhlg9qLC6/nOJPX3qGbkZpYAcqfM=
 github.com/hexops/gotextdiff v1.0.3 h1:gitA9+qJrrTCsiCl7+kh75nPqQt1cx4ZkudSTLoUqJM=
 github.com/hexops/gotextdiff v1.0.3/go.mod h1:pSWU5MAI3yDq+fZBTazCSJysOMbxWL1BSow5/V2vxeg=
 github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
 github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
 github.com/jinzhu/copier v0.4.0 h1:w3ciUoD19shMCRargcpm0cm91ytaBhDvuRpz1ODO/U8=
 github.com/jinzhu/copier v0.4.0/go.mod h1:DfbEm0FYsaqBcKcFuvmOZb218JkPGtvSHsKg8S8hyyg=
-github.com/leodido/go-urn v1.2.0 h1:hpXL4XnriNwQ/ABnpepYM/1vCLWNDfUNts8dX3xTG6Y=
-github.com/leodido/go-urn v1.2.0/go.mod h1:+8+nEpDfqqsY+g338gtMEUOtuK+4dEMhiQEgxpxOKII=
-github.com/magiconair/properties v1.8.7 h1:IeQXZAiQcpL9mgcAe1Nu6cX9LLw6ExEHKjN0VQdvPDY=
-github.com/magiconair/properties v1.8.7/go.mod h1:Dhd985XPs7jluiymwWYZ0G4Z61jb3vdS329zhj2hYo0=
-github.com/mattn/go-colorable v0.1.13 h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA=
-github.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovkB8vQcUbaXHg=
-github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM=
+github.com/magiconair/properties v1.8.10 h1:s31yESBquKXCV9a/ScB3ESkOjUYYv+X0rg8SYxI99mE=
+github.com/magiconair/properties v1.8.10/go.mod h1:Dhd985XPs7jluiymwWYZ0G4Z61jb3vdS329zhj2hYo0=
+github.com/mattn/go-colorable v0.1.14 h1:9A9LHSqF/7dyVVX6g0U9cwm9pG3kP9gSzcuIPHPsaIE=
+github.com/mattn/go-colorable v0.1.14/go.mod h1:6LmQG8QLFO4G5z1gPvYEzlUgJ2wF+stgPZH1UqBm1s8=
 github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
 github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
-github.com/pelletier/go-toml/v2 v2.1.1 h1:LWAJwfNvjQZCFIDKWYQaM62NcYeYViCmWIwmOStowAI=
-github.com/pelletier/go-toml/v2 v2.1.1/go.mod h1:tJU2Z3ZkXwnxa4DPO899bsyIoywizdUvyaeZurnPPDc=
+github.com/mitchellh/go-wordwrap v1.0.1 h1:TLuKupo69TCn6TQSyGxwI1EblZZEsQ0vMlAFQflz0v0=
+github.com/mitchellh/go-wordwrap v1.0.1/go.mod h1:R62XHJLzvMFRBbcrT7m7WgmE1eOyTSsCt+hzestvNj0=
+github.com/pelletier/go-toml/v2 v2.2.4 h1:mye9XuhQ6gvn5h28+VilKrrPoQVanw5PMw/TB0t5Ec4=
+github.com/pelletier/go-toml/v2 v2.2.4/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8t9Stt+h56mCY=
 github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e h1:aoZm08cpOy4WuID//EZDgcC4zIxODThtZNPirFr42+A=
 github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e/go.mod h1:pJLUxLENpZxwdsKMEsNbx1VGcRFpLqf3715MtcvvzbA=
 github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
 github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
-github.com/spf13/cobra v1.8.0 h1:7aJaZx1B85qltLMc546zn58BxxfZdR/W22ej9CFoEf0=
-github.com/spf13/cobra v1.8.0/go.mod h1:WXLWApfZ71AjXPya3WOlMsY9yMs7YeiHhFVlvLyhcho=
-github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
-github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
+github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU=
+github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiTUUS4=
+github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
+github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk=
+github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
 github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
-github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
-github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
+github.com/stretchr/testify v1.7.0 h1:nwc3DEeHmmLAfoZucVR881uASk0Mfjw8xYJ99tb5CcY=
 github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
-github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
-github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
-github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk=
-github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
 github.com/yuin/gopher-lua v1.1.1 h1:kYKnWBjvbNP4XLT3+bPEwAXJx262OhaHDWDVOPjL46M=
 github.com/yuin/gopher-lua v1.1.1/go.mod h1:GBR0iDaNXjAgGg9zfCvksxSRnQx76gclCIb7kdAd1Pw=
-golang.org/x/crypto v0.21.0 h1:X31++rzVUdKhX5sWmSOFZxx8UW/ldWx55cbf08iNAMA=
-golang.org/x/crypto v0.21.0/go.mod h1:0BP7YvVV9gBbVKyeTG0Gyn+gZm94bibOW5BjDEYAOMs=
-golang.org/x/net v0.22.0 h1:9sGLhx7iRIHEiX0oAJ3MRZMUCElJgy7Br1nO+AMN3Tc=
-golang.org/x/net v0.22.0/go.mod h1:JKghWKKOSdJwpW2GEx0Ja7fmaKnMsbu+MWVZTokSYmg=
-golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+github.com/zclconf/go-cty v1.17.0 h1:seZvECve6XX4tmnvRzWtJNHdscMtYEx5R7bnnVyd/d0=
+github.com/zclconf/go-cty v1.17.0/go.mod h1:wqFzcImaLTI6A5HfsRwB0nj5n0MRZFwmey8YoFPPs3U=
+github.com/zclconf/go-cty-debug v0.0.0-20240509010212-0d6042c53940 h1:4r45xpDWB6ZMSMNJFMOjqrGHynW3DIBuR2H9j0ug+Mo=
+github.com/zclconf/go-cty-debug v0.0.0-20240509010212-0d6042c53940/go.mod h1:CmBdvvj3nqzfzJ6nTCIwDTPZ56aVGvDrmztiO5g3qrM=
+go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
+go.yaml.in/yaml/v4 v4.0.0-rc.3 h1:3h1fjsh1CTAPjW7q/EMe+C8shx5d8ctzZTrLcs/j8Go=
+go.yaml.in/yaml/v4 v4.0.0-rc.3/go.mod h1:aZqd9kCMsGL7AuUv/m/PvWLdg5sjJsZ4oHDEnfPPfY0=
+golang.org/x/mod v0.30.0 h1:fDEXFVZ/fmCKProc/yAXXUijritrDzahmwwefnjoPFk=
+golang.org/x/mod v0.30.0/go.mod h1:lAsf5O2EvJeSFMiBxXDki7sCgAxEUcZHXoXMKT4GJKc=
+golang.org/x/net v0.48.0 h1:zyQRTTrjc33Lhh0fBgT/H3oZq9WuvRR5gPC70xpDiQU=
+golang.org/x/net v0.48.0/go.mod h1:+ndRgGjkh8FGtu1w1FGbEC31if4VrNVMuKTgcAAnQRY=
+golang.org/x/sync v0.19.0 h1:vV+1eWNmZ5geRlYjzm2adRgW2/mcpevXNg50YZtPCE4=
+golang.org/x/sync v0.19.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
 golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.18.0 h1:DBdB3niSjOA/O0blCZBqDefyWNYveAYMNF1Wum0DYQ4=
-golang.org/x/sys v0.18.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
-golang.org/x/text v0.14.0 h1:ScX5w1eTa3QqT8oi6+ziP7dTV1S2+ALU0bI+0zXKWiQ=
-golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
-golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 h1:H2TDz8ibqkAF6YGhCdN3jS9O0/s90v0rJh3X/OLHEUk=
-golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2/go.mod h1:K8+ghG5WaK9qNqU5K3HdILfMLy1f3aNYFI/wnl100a8=
-gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
+golang.org/x/sys v0.39.0 h1:CvCKL8MeisomCi6qNZ+wbb0DN9E5AATixKsvNtMoMFk=
+golang.org/x/sys v0.39.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
+golang.org/x/text v0.32.0 h1:ZD01bjUt1FQ9WJ0ClOL5vxgxOI/sVCNgX1YtKwcY0mU=
+golang.org/x/text v0.32.0/go.mod h1:o/rUWzghvpD5TXrTIBuJU77MTaN0ljMWE47kxGJQ7jY=
+golang.org/x/tools v0.39.0 h1:ik4ho21kwuQln40uelmciQPp9SipgNDdrafrYA4TmQQ=
+golang.org/x/tools v0.39.0/go.mod h1:JnefbkDPyD8UU2kI5fuf8ZX4/yUeh9W877ZeBONxUqQ=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/op/go-logging.v1 v1.0.0-20160211212156-b2cb9fa56473 h1:6D+BvnJ/j6e222UW8s2qTSe3wGBtvo0MbVQG/c5k8RE=
 gopkg.in/op/go-logging.v1 v1.0.0-20160211212156-b2cb9fa56473/go.mod h1:N1eN2tsCx0Ydtgjl4cqmbRCsY4/+z4cYDeqwZTk6zog=
+gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c h1:dUUwHk2QECo/6vqA44rthZ8ie2QXMNeKRTHCNY2nXvo=
 gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
-gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
-gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

how-it-works.md

@@ -1,8 +1,63 @@
-# How it works
+# Expression Syntax: A Visual Guide
+In `yq`, expressions are made up of operators and pipes. A context of nodes is passed through the expression, and each operation takes the context as input and returns a new context as output. That output is piped in as input for the next operation in the expression.
 
-In `yq` expressions are made up of operators and pipes. A context of nodes is passed through the expression and each operation takes the context as input and returns a new context as output. That output is piped in as input for the next operation in the expression. To begin with, the context is set to the first yaml document of the first yaml file (if processing in sequence using eval).
+Let's break down the process step by step using a diagram. We'll start with a single YAML document, apply an expression, and observe how the context changes at each step.
 
-Lets look at a couple of examples.
+Given a document like:
+
+```yaml
+root:
+  items:
+    - name: apple
+      type: fruit
+    - name: carrot
+      type: vegetable
+    - name: banana
+      type: fruit
+```
+
+You can use dot notation to access nested structures. For example, to access the `name` of the first item, you would use the expression `.root.items[0].name`, which would return `apple`.
+
+But lets see how we could find all the fruit under `items`
+
+## Step 1: Initial Context
+The context starts at the root of the YAML document. In this case, the entire document is the initial context. 
+
+```
+root
+└── items
+    ├── name: apple
+    │   type: fruit
+    ├── name: carrot
+    │   type: vegetable
+    └── name: banana
+        type: fruit
+```
+
+## Step 2: Splatting the Array
+Using the expression `.root.items[]`, we "splat" the items array. This means each element of the array becomes its own node in the context:
+
+```
+Node 1: { name: apple, type: fruit }
+Node 2: { name: carrot, type: vegetable }
+Node 3: { name: banana, type: fruit }
+```
+
+## Step 3: Filtering the Nodes
+Next, we apply a filter to select only the nodes where type is fruit. The expression `.root.items[] | select(.type == "fruit")` filters the nodes:
+
+```
+Filtered Node 1: { name: apple, type: fruit }
+Filtered Node 2: { name: banana, type: fruit }
+```
+
+## Step 4: Extracting a Field
+Finally, we extract the name field from the filtered nodes using `.root.items[] | select(.type == "fruit") | .name` This results in:
+
+```
+apple
+banana
+```
 
 ## Simple assignment example
 
@@ -44,7 +99,6 @@ a: dog
 b: dog
 ```
 
-
 ## Complex assignment, operator precedence rules
 
 Just like math expressions - `yq` expressions have an order of precedence. The pipe `|` operator has a low order of precedence, so operators with higher precedence will get evaluated first. 
@@ -73,7 +127,7 @@ name: sally
 fruit: mango
 ```
 
-To properly update this yaml, you will need to use brackets (think BODMAS from maths) and wrap the entire LHS:
+**Important**: To properly update this YAML, you must wrap the entire LHS in parentheses. Think of it like using brackets in math to ensure the correct order of operations.
 `(.[] | select(.name == "sally") | .fruit) = "mango"`
 
 
@@ -126,4 +180,4 @@ The assignment operator then copies across the value from the RHS to the value o
 ```yaml
 a: 2
 b: thing
-```
+```

pkg/yqlib/base64_test.go

@@ -0,0 +1,269 @@
+//go:build !yq_nobase64
+
+package yqlib
+
+import (
+	"bufio"
+	"fmt"
+	"testing"
+
+	"github.com/mikefarah/yq/v4/test"
+)
+
+const base64EncodedSimple = "YSBzcGVjaWFsIHN0cmluZw=="
+const base64DecodedSimpleExtraSpaces = "\n " + base64EncodedSimple + "  \n"
+const base64DecodedSimple = "a special string"
+
+const base64EncodedUTF8 = "V29ya3Mgd2l0aCBVVEYtMTYg8J+Yig=="
+const base64DecodedUTF8 = "Works with UTF-16 😊"
+
+const base64EncodedYaml = "YTogYXBwbGUK"
+const base64DecodedYaml = "a: apple\n"
+
+const base64EncodedEmpty = ""
+const base64DecodedEmpty = ""
+
+const base64MissingPadding = "Y2F0cw"
+const base64DecodedMissingPadding = "cats"
+
+const base64EncodedCats = "Y2F0cw=="
+const base64DecodedCats = "cats"
+
+var base64Scenarios = []formatScenario{
+	{
+		skipDoc:      true,
+		description:  "empty decode",
+		input:        base64EncodedEmpty,
+		expected:     base64DecodedEmpty + "\n",
+		scenarioType: "decode",
+	},
+	{
+		skipDoc:      true,
+		description:  "simple decode",
+		input:        base64EncodedSimple,
+		expected:     base64DecodedSimple + "\n",
+		scenarioType: "decode",
+	},
+	{
+		description:    "Decode base64: simple",
+		subdescription: "Decoded data is assumed to be a string.",
+		input:          base64EncodedSimple,
+		expected:       base64DecodedSimple + "\n",
+		scenarioType:   "decode",
+	},
+	{
+		description:    "Decode base64: UTF-8",
+		subdescription: "Base64 decoding supports UTF-8 encoded strings.",
+		input:          base64EncodedUTF8,
+		expected:       base64DecodedUTF8 + "\n",
+		scenarioType:   "decode",
+	},
+	{
+		skipDoc:      true,
+		description:  "decode missing padding",
+		input:        base64MissingPadding,
+		expected:     base64DecodedMissingPadding + "\n",
+		scenarioType: "decode",
+	},
+	{
+
+		description:    "Decode with extra spaces",
+		subdescription: "Extra leading/trailing whitespace is stripped",
+		input:          base64DecodedSimpleExtraSpaces,
+		expected:       base64DecodedSimple + "\n",
+		scenarioType:   "decode",
+	},
+	{
+		skipDoc:      true,
+		description:  "decode with padding",
+		input:        base64EncodedCats,
+		expected:     base64DecodedCats + "\n",
+		scenarioType: "decode",
+	},
+	{
+		skipDoc:      true,
+		description:  "decode yaml document",
+		input:        base64EncodedYaml,
+		expected:     base64DecodedYaml + "\n",
+		scenarioType: "decode",
+	},
+	{
+		description:  "Encode base64: string",
+		input:        "\"" + base64DecodedSimple + "\"",
+		expected:     base64EncodedSimple,
+		scenarioType: "encode",
+	},
+	{
+		description:    "Encode base64: string from document",
+		subdescription: "Extract a string field and encode it to base64.",
+		input:          "coolData: \"" + base64DecodedSimple + "\"",
+		expression:     ".coolData",
+		expected:       base64EncodedSimple,
+		scenarioType:   "encode",
+	},
+	{
+		skipDoc:      true,
+		description:  "encode empty string",
+		input:        "\"\"",
+		expected:     "",
+		scenarioType: "encode",
+	},
+	{
+		skipDoc:      true,
+		description:  "encode UTF-8 string",
+		input:        "\"" + base64DecodedUTF8 + "\"",
+		expected:     base64EncodedUTF8,
+		scenarioType: "encode",
+	},
+	{
+		skipDoc:      true,
+		description:  "encode cats",
+		input:        "\"" + base64DecodedCats + "\"",
+		expected:     base64EncodedCats,
+		scenarioType: "encode",
+	},
+	{
+		description:  "Roundtrip: simple",
+		skipDoc:      true,
+		input:        base64EncodedSimple,
+		expected:     base64EncodedSimple,
+		scenarioType: "roundtrip",
+	},
+	{
+		description:  "Roundtrip: UTF-8",
+		skipDoc:      true,
+		input:        base64EncodedUTF8,
+		expected:     base64EncodedUTF8,
+		scenarioType: "roundtrip",
+	},
+	{
+		description:  "Roundtrip: missing padding",
+		skipDoc:      true,
+		input:        base64MissingPadding,
+		expected:     base64EncodedCats,
+		scenarioType: "roundtrip",
+	},
+	{
+		description:  "Roundtrip: empty",
+		skipDoc:      true,
+		input:        base64EncodedEmpty,
+		expected:     base64EncodedEmpty,
+		scenarioType: "roundtrip",
+	},
+	{
+		description:   "Encode error: non-string",
+		skipDoc:       true,
+		input:         "123",
+		expectedError: "cannot encode !!int as base64, can only operate on strings",
+		scenarioType:  "encode-error",
+	},
+	{
+		description:   "Encode error: array",
+		skipDoc:       true,
+		input:         "[1, 2, 3]",
+		expectedError: "cannot encode !!seq as base64, can only operate on strings",
+		scenarioType:  "encode-error",
+	},
+	{
+		description:   "Encode error: map",
+		skipDoc:       true,
+		input:         "{b: c}",
+		expectedError: "cannot encode !!map as base64, can only operate on strings",
+		scenarioType:  "encode-error",
+	},
+}
+
+func testBase64Scenario(t *testing.T, s formatScenario) {
+	switch s.scenarioType {
+	case "", "decode":
+		yamlPrefs := ConfiguredYamlPreferences.Copy()
+		yamlPrefs.Indent = 4
+		test.AssertResultWithContext(t, s.expected, mustProcessFormatScenario(s, NewBase64Decoder(), NewYamlEncoder(yamlPrefs)), s.description)
+	case "encode":
+		test.AssertResultWithContext(t, s.expected, mustProcessFormatScenario(s, NewYamlDecoder(ConfiguredYamlPreferences), NewBase64Encoder()), s.description)
+	case "roundtrip":
+		test.AssertResultWithContext(t, s.expected, mustProcessFormatScenario(s, NewBase64Decoder(), NewBase64Encoder()), s.description)
+	case "encode-error":
+		result, err := processFormatScenario(s, NewYamlDecoder(ConfiguredYamlPreferences), NewBase64Encoder())
+		if err == nil {
+			t.Errorf("Expected error '%v' but it worked: %v", s.expectedError, result)
+		} else {
+			test.AssertResultComplexWithContext(t, s.expectedError, err.Error(), s.description)
+		}
+
+	default:
+		panic(fmt.Sprintf("unhandled scenario type %q", s.scenarioType))
+	}
+}
+
+func documentBase64Scenario(_ *testing.T, w *bufio.Writer, i interface{}) {
+	s := i.(formatScenario)
+
+	if s.skipDoc {
+		return
+	}
+	switch s.scenarioType {
+	case "", "decode":
+		documentBase64DecodeScenario(w, s)
+	case "encode":
+		documentBase64EncodeScenario(w, s)
+
+	default:
+		panic(fmt.Sprintf("unhandled scenario type %q", s.scenarioType))
+	}
+}
+
+func documentBase64DecodeScenario(w *bufio.Writer, s formatScenario) {
+	writeOrPanic(w, fmt.Sprintf("## %v\n", s.description))
+
+	if s.subdescription != "" {
+		writeOrPanic(w, s.subdescription)
+		writeOrPanic(w, "\n\n")
+	}
+
+	writeOrPanic(w, "Given a sample.txt file of:\n")
+	writeOrPanic(w, fmt.Sprintf("```\n%v\n```\n", s.input))
+
+	writeOrPanic(w, "then\n")
+	expression := s.expression
+	if expression == "" {
+		expression = "."
+	}
+	writeOrPanic(w, fmt.Sprintf("```bash\nyq -p=base64 -oy '%v' sample.txt\n```\n", expression))
+	writeOrPanic(w, "will output\n")
+
+	writeOrPanic(w, fmt.Sprintf("```yaml\n%v```\n\n", mustProcessFormatScenario(s, NewBase64Decoder(), NewYamlEncoder(ConfiguredYamlPreferences))))
+}
+
+func documentBase64EncodeScenario(w *bufio.Writer, s formatScenario) {
+	writeOrPanic(w, fmt.Sprintf("## %v\n", s.description))
+
+	if s.subdescription != "" {
+		writeOrPanic(w, s.subdescription)
+		writeOrPanic(w, "\n\n")
+	}
+
+	writeOrPanic(w, "Given a sample.yml file of:\n")
+	writeOrPanic(w, fmt.Sprintf("```yaml\n%v\n```\n", s.input))
+
+	writeOrPanic(w, "then\n")
+	expression := s.expression
+	if expression == "" {
+		expression = "."
+	}
+	writeOrPanic(w, fmt.Sprintf("```bash\nyq -o=base64 '%v' sample.yml\n```\n", expression))
+	writeOrPanic(w, "will output\n")
+
+	writeOrPanic(w, fmt.Sprintf("```\n%v```\n\n", mustProcessFormatScenario(s, NewYamlDecoder(ConfiguredYamlPreferences), NewBase64Encoder())))
+}
+
+func TestBase64Scenarios(t *testing.T) {
+	for _, tt := range base64Scenarios {
+		testBase64Scenario(t, tt)
+	}
+	genericScenarios := make([]interface{}, len(base64Scenarios))
+	for i, s := range base64Scenarios {
+		genericScenarios[i] = s
+	}
+	documentScenarios(t, "usage", "base64", genericScenarios, documentBase64Scenario)
+}

pkg/yqlib/candidate_node.go

@@ -53,6 +53,20 @@ func createScalarNode(value interface{}, stringValue string) *CandidateNode {
 	return node
 }
 
+type NodeInfo struct {
+	Kind        string      `yaml:"kind"`
+	Style       string      `yaml:"style,omitempty"`
+	Anchor      string      `yaml:"anchor,omitempty"`
+	Tag         string      `yaml:"tag,omitempty"`
+	HeadComment string      `yaml:"headComment,omitempty"`
+	LineComment string      `yaml:"lineComment,omitempty"`
+	FootComment string      `yaml:"footComment,omitempty"`
+	Value       string      `yaml:"value,omitempty"`
+	Line        int         `yaml:"line,omitempty"`
+	Column      int         `yaml:"column,omitempty"`
+	Content     []*NodeInfo `yaml:"content,omitempty"`
+}
+
 type CandidateNode struct {
 	Kind  Kind
 	Style Style
@@ -83,6 +97,9 @@ type CandidateNode struct {
 	// (e.g. top level cross document merge). This property does not propagate to child nodes.
 	EvaluateTogether bool
 	IsMapKey         bool
+	// For formats like HCL and TOML: indicates that child entries should be emitted as separate blocks/tables
+	// rather than consolidated into nested mappings (default behaviour)
+	EncodeSeparate bool
 }
 
 func (n *CandidateNode) CreateChild() *CandidateNode {
@@ -155,6 +172,18 @@ func (n *CandidateNode) getParsedKey() interface{} {
 
 }
 
+func (n *CandidateNode) FilterMapContentByKey(keyPredicate func(*CandidateNode) bool) []*CandidateNode {
+	var result []*CandidateNode
+	for index := 0; index < len(n.Content); index = index + 2 {
+		keyNode := n.Content[index]
+		valueNode := n.Content[index+1]
+		if keyPredicate(keyNode) {
+			result = append(result, keyNode, valueNode)
+		}
+	}
+	return result
+}
+
 func (n *CandidateNode) GetPath() []interface{} {
 	key := n.getParsedKey()
 	if n.Parent != nil && key != nil {
@@ -198,6 +227,30 @@ func (n *CandidateNode) SetParent(parent *CandidateNode) {
 	n.Parent = parent
 }
 
+type ValueVisitor func(*CandidateNode) error
+
+func (n *CandidateNode) VisitValues(visitor ValueVisitor) error {
+	switch n.Kind {
+	case MappingNode:
+		for i := 1; i < len(n.Content); i = i + 2 {
+			if err := visitor(n.Content[i]); err != nil {
+				return err
+			}
+		}
+	case SequenceNode:
+		for i := 0; i < len(n.Content); i = i + 1 {
+			if err := visitor(n.Content[i]); err != nil {
+				return err
+			}
+		}
+	}
+	return nil
+}
+
+func (n *CandidateNode) CanVisitValues() bool {
+	return n.Kind == MappingNode || n.Kind == SequenceNode
+}
+
 func (n *CandidateNode) AddKeyValueChild(rawKey *CandidateNode, rawValue *CandidateNode) (*CandidateNode, *CandidateNode) {
 	key := rawKey.Copy()
 	key.SetParent(n)
@@ -215,14 +268,13 @@ func (n *CandidateNode) AddKeyValueChild(rawKey *CandidateNode, rawValue *Candid
 func (n *CandidateNode) AddChild(rawChild *CandidateNode) {
 	value := rawChild.Copy()
 	value.SetParent(n)
-	if value.Key != nil {
-		value.Key.SetParent(n)
-	} else {
-		index := len(n.Content)
-		keyNode := createScalarNode(index, fmt.Sprintf("%v", index))
-		keyNode.SetParent(n)
-		value.Key = keyNode
-	}
+	value.IsMapKey = false
+
+	index := len(n.Content)
+	keyNode := createScalarNode(index, fmt.Sprintf("%v", index))
+	keyNode.SetParent(n)
+	value.Key = keyNode
+
 	n.Content = append(n.Content, value)
 }
 
@@ -358,6 +410,8 @@ func (n *CandidateNode) doCopy(cloneContent bool) *CandidateNode {
 
 		EvaluateTogether: n.EvaluateTogether,
 		IsMapKey:         n.IsMapKey,
+
+		EncodeSeparate: n.EncodeSeparate,
 	}
 
 	if cloneContent {
@@ -428,3 +482,64 @@ func (n *CandidateNode) UpdateAttributesFrom(other *CandidateNode, prefs assignP
 		n.LineComment = other.LineComment
 	}
 }
+
+func (n *CandidateNode) ConvertToNodeInfo() *NodeInfo {
+	info := &NodeInfo{
+		Kind:        kindToString(n.Kind),
+		Style:       styleToString(n.Style),
+		Anchor:      n.Anchor,
+		Tag:         n.Tag,
+		HeadComment: n.HeadComment,
+		LineComment: n.LineComment,
+		FootComment: n.FootComment,
+		Value:       n.Value,
+		Line:        n.Line,
+		Column:      n.Column,
+	}
+	if len(n.Content) > 0 {
+		info.Content = make([]*NodeInfo, len(n.Content))
+		for i, child := range n.Content {
+			info.Content[i] = child.ConvertToNodeInfo()
+		}
+	}
+	return info
+}
+
+// Helper functions to convert Kind and Style to string for NodeInfo
+func kindToString(k Kind) string {
+	switch k {
+	case SequenceNode:
+		return "SequenceNode"
+	case MappingNode:
+		return "MappingNode"
+	case ScalarNode:
+		return "ScalarNode"
+	case AliasNode:
+		return "AliasNode"
+	default:
+		return "Unknown"
+	}
+}
+
+func styleToString(s Style) string {
+	var styles []string
+	if s&TaggedStyle != 0 {
+		styles = append(styles, "TaggedStyle")
+	}
+	if s&DoubleQuotedStyle != 0 {
+		styles = append(styles, "DoubleQuotedStyle")
+	}
+	if s&SingleQuotedStyle != 0 {
+		styles = append(styles, "SingleQuotedStyle")
+	}
+	if s&LiteralStyle != 0 {
+		styles = append(styles, "LiteralStyle")
+	}
+	if s&FoldedStyle != 0 {
+		styles = append(styles, "FoldedStyle")
+	}
+	if s&FlowStyle != 0 {
+		styles = append(styles, "FlowStyle")
+	}
+	return strings.Join(styles, ",")
+}

pkg/yqlib/candidate_node_goccy_yaml.go

@@ -9,14 +9,14 @@ import (
 	goccyToken "github.com/goccy/go-yaml/token"
 )
 
-func (o *CandidateNode) goccyDecodeIntoChild(childNode ast.Node, cm yaml.CommentMap) (*CandidateNode, error) {
+func (o *CandidateNode) goccyDecodeIntoChild(childNode ast.Node, cm yaml.CommentMap, anchorMap map[string]*CandidateNode) (*CandidateNode, error) {
 	newChild := o.CreateChild()
 
-	err := newChild.UnmarshalGoccyYAML(childNode, cm)
+	err := newChild.UnmarshalGoccyYAML(childNode, cm, anchorMap)
 	return newChild, err
 }
 
-func (o *CandidateNode) UnmarshalGoccyYAML(node ast.Node, cm yaml.CommentMap) error {
+func (o *CandidateNode) UnmarshalGoccyYAML(node ast.Node, cm yaml.CommentMap, anchorMap map[string]*CandidateNode) error {
 	log.Debugf("UnmarshalYAML %v", node)
 	log.Debugf("UnmarshalYAML %v", node.Type().String())
 	log.Debugf("UnmarshalYAML Node Value: %v", node.String())
@@ -51,6 +51,9 @@ func (o *CandidateNode) UnmarshalGoccyYAML(node ast.Node, cm yaml.CommentMap) er
 	}
 
 	o.Value = node.String()
+	o.Line = node.GetToken().Position.Line
+	o.Column = node.GetToken().Position.Column
+
 	switch node.Type() {
 	case ast.IntegerType:
 		o.Kind = ScalarNode
@@ -62,9 +65,13 @@ func (o *CandidateNode) UnmarshalGoccyYAML(node ast.Node, cm yaml.CommentMap) er
 		o.Kind = ScalarNode
 		o.Tag = "!!bool"
 	case ast.NullType:
+		log.Debugf("its a null type with value %v", node.GetToken().Value)
 		o.Kind = ScalarNode
 		o.Tag = "!!null"
 		o.Value = node.GetToken().Value
+		if node.GetToken().Type == goccyToken.ImplicitNullType {
+			o.Value = ""
+		}
 	case ast.StringType:
 		o.Kind = ScalarNode
 		o.Tag = "!!str"
@@ -92,7 +99,7 @@ func (o *CandidateNode) UnmarshalGoccyYAML(node ast.Node, cm yaml.CommentMap) er
 		// to solve the multiline > problem
 		o.Value = astLiteral.Value.Value
 	case ast.TagType:
-		if err := o.UnmarshalGoccyYAML(node.(*ast.TagNode).Value, cm); err != nil {
+		if err := o.UnmarshalGoccyYAML(node.(*ast.TagNode).Value, cm, anchorMap); err != nil {
 			return err
 		}
 		o.Tag = node.(*ast.TagNode).Start.Value
@@ -106,9 +113,9 @@ func (o *CandidateNode) UnmarshalGoccyYAML(node ast.Node, cm yaml.CommentMap) er
 			o.Style = FlowStyle
 		}
 		for _, mappingValueNode := range mappingNode.Values {
-			err := o.goccyProcessMappingValueNode(mappingValueNode, cm)
+			err := o.goccyProcessMappingValueNode(mappingValueNode, cm, anchorMap)
 			if err != nil {
-				return ast.ErrInvalidAnchorName
+				return err
 			}
 		}
 		if mappingNode.FootComment != nil {
@@ -120,9 +127,9 @@ func (o *CandidateNode) UnmarshalGoccyYAML(node ast.Node, cm yaml.CommentMap) er
 		o.Kind = MappingNode
 		o.Tag = "!!map"
 		mappingValueNode := node.(*ast.MappingValueNode)
-		err := o.goccyProcessMappingValueNode(mappingValueNode, cm)
+		err := o.goccyProcessMappingValueNode(mappingValueNode, cm, anchorMap)
 		if err != nil {
-			return ast.ErrInvalidAnchorName
+			return err
 		}
 	case ast.SequenceType:
 		log.Debugf("UnmarshalYAML -  a sequence node")
@@ -141,7 +148,7 @@ func (o *CandidateNode) UnmarshalGoccyYAML(node ast.Node, cm yaml.CommentMap) er
 			keyNode.Kind = ScalarNode
 			keyNode.Value = fmt.Sprintf("%v", i)
 
-			valueNode, err := o.goccyDecodeIntoChild(astSeq[i], cm)
+			valueNode, err := o.goccyDecodeIntoChild(astSeq[i], cm, anchorMap)
 			if err != nil {
 				return err
 			}
@@ -149,32 +156,55 @@ func (o *CandidateNode) UnmarshalGoccyYAML(node ast.Node, cm yaml.CommentMap) er
 			valueNode.Key = keyNode
 			o.Content[i] = valueNode
 		}
+	case ast.AnchorType:
+		log.Debugf("UnmarshalYAML -  an anchor node")
+		anchorNode := node.(*ast.AnchorNode)
+		err := o.UnmarshalGoccyYAML(anchorNode.Value, cm, anchorMap)
+		if err != nil {
+			return err
+		}
+		o.Anchor = anchorNode.Name.String()
+		anchorMap[o.Anchor] = o
+
+	case ast.AliasType:
+		log.Debugf("UnmarshalYAML -  an alias node")
+		aliasNode := node.(*ast.AliasNode)
+		o.Kind = AliasNode
+		o.Value = aliasNode.Value.String()
+		o.Alias = anchorMap[o.Value]
+
+	case ast.MergeKeyType:
+		log.Debugf("UnmarshalYAML -  a merge key")
+		o.Kind = ScalarNode
+		o.Tag = "!!merge" // note - I should be able to get rid of this.
+		o.Value = "<<"
 
 	default:
-		log.Debugf("UnmarshalYAML -  node idea of the type!!")
+		log.Debugf("UnmarshalYAML -  no idea of the type!!\n%v: %v", node.Type(), node.String())
 	}
 	log.Debugf("KIND: %v", o.Kind)
 	return nil
 }
 
-func (o *CandidateNode) goccyProcessMappingValueNode(mappingEntry *ast.MappingValueNode, cm yaml.CommentMap) error {
+func (o *CandidateNode) goccyProcessMappingValueNode(mappingEntry *ast.MappingValueNode, cm yaml.CommentMap, anchorMap map[string]*CandidateNode) error {
 	log.Debug("UnmarshalYAML MAP KEY entry %v", mappingEntry.Key)
-	keyNode, err := o.goccyDecodeIntoChild(mappingEntry.Key, cm)
-	if err != nil {
+
+	// AddKeyValueFirst because it clones the nodes, and we want to have the real refs when Unmarshalling
+	// particularly for the anchorMap
+	keyNode, valueNode := o.AddKeyValueChild(&CandidateNode{}, &CandidateNode{})
+
+	if err := keyNode.UnmarshalGoccyYAML(mappingEntry.Key, cm, anchorMap); err != nil {
 		return err
 	}
-	keyNode.IsMapKey = true
 
 	log.Debug("UnmarshalYAML MAP VALUE entry %v", mappingEntry.Value)
-	valueNode, err := o.goccyDecodeIntoChild(mappingEntry.Value, cm)
-	if err != nil {
+	if err := valueNode.UnmarshalGoccyYAML(mappingEntry.Value, cm, anchorMap); err != nil {
 		return err
 	}
 
 	if mappingEntry.FootComment != nil {
 		valueNode.FootComment = mappingEntry.FootComment.String()
 	}
-	o.AddKeyValueChild(keyNode, valueNode)
 
 	return nil
 }

pkg/yqlib/candidate_node_test.go

@@ -152,11 +152,243 @@ func TestCandidateNodeAddKeyValueChild(t *testing.T) {
 	key := CandidateNode{Value: "cool", IsMapKey: true}
 	node := CandidateNode{}
 
-	// if we use a key in a new node as a value, it should no longer be marked as a key
-
 	_, keyIsValueNow := node.AddKeyValueChild(&CandidateNode{Value: "newKey"}, &key)
 
 	test.AssertResult(t, keyIsValueNow.IsMapKey, false)
 	test.AssertResult(t, key.IsMapKey, true)
 
 }
+
+func TestConvertToNodeInfo(t *testing.T) {
+	child := &CandidateNode{
+		Kind:   ScalarNode,
+		Style:  DoubleQuotedStyle,
+		Tag:    "!!str",
+		Value:  "childValue",
+		Line:   2,
+		Column: 3,
+	}
+	parent := &CandidateNode{
+		Kind:        MappingNode,
+		Style:       TaggedStyle,
+		Tag:         "!!map",
+		Value:       "",
+		Line:        1,
+		Column:      1,
+		Content:     []*CandidateNode{child},
+		HeadComment: "head",
+		LineComment: "line",
+		FootComment: "foot",
+		Anchor:      "anchor",
+	}
+	info := parent.ConvertToNodeInfo()
+	test.AssertResult(t, "MappingNode", info.Kind)
+	test.AssertResult(t, "TaggedStyle", info.Style)
+	test.AssertResult(t, "!!map", info.Tag)
+	test.AssertResult(t, "head", info.HeadComment)
+	test.AssertResult(t, "line", info.LineComment)
+	test.AssertResult(t, "foot", info.FootComment)
+	test.AssertResult(t, "anchor", info.Anchor)
+	test.AssertResult(t, 1, info.Line)
+	test.AssertResult(t, 1, info.Column)
+	if len(info.Content) != 1 {
+		t.Fatalf("Expected 1 child, got %d", len(info.Content))
+	}
+	childInfo := info.Content[0]
+	test.AssertResult(t, "ScalarNode", childInfo.Kind)
+	test.AssertResult(t, "DoubleQuotedStyle", childInfo.Style)
+	test.AssertResult(t, "!!str", childInfo.Tag)
+	test.AssertResult(t, "childValue", childInfo.Value)
+	test.AssertResult(t, 2, childInfo.Line)
+	test.AssertResult(t, 3, childInfo.Column)
+}
+
+func TestCandidateNodeGetPath(t *testing.T) {
+	// Test root node with no parent
+	root := CandidateNode{Value: "root"}
+	path := root.GetPath()
+	test.AssertResult(t, 0, len(path))
+
+	// Test node with key
+	key := createStringScalarNode("myKey")
+	node := CandidateNode{Key: key, Value: "myValue"}
+	path = node.GetPath()
+	test.AssertResult(t, 1, len(path))
+	test.AssertResult(t, "myKey", path[0])
+
+	// Test nested path
+	parent := CandidateNode{}
+	parentKey := createStringScalarNode("parent")
+	parent.Key = parentKey
+	node.Parent = &parent
+	path = node.GetPath()
+	test.AssertResult(t, 2, len(path))
+	test.AssertResult(t, "parent", path[0])
+	test.AssertResult(t, "myKey", path[1])
+}
+
+func TestCandidateNodeGetNicePath(t *testing.T) {
+	// Test simple key
+	key := createStringScalarNode("simple")
+	node := CandidateNode{Key: key}
+	nicePath := node.GetNicePath()
+	test.AssertResult(t, "simple", nicePath)
+
+	// Test array index
+	arrayKey := createScalarNode(0, "0")
+	arrayNode := CandidateNode{Key: arrayKey}
+	nicePath = arrayNode.GetNicePath()
+	test.AssertResult(t, "[0]", nicePath)
+
+	dotKey := createStringScalarNode("key.with.dots")
+	dotNode := CandidateNode{Key: dotKey}
+	nicePath = dotNode.GetNicePath()
+	test.AssertResult(t, "key.with.dots", nicePath)
+
+	// Test nested path
+	parentKey := createStringScalarNode("parent")
+	parent := CandidateNode{Key: parentKey}
+	childKey := createStringScalarNode("child")
+	child := CandidateNode{Key: childKey, Parent: &parent}
+	nicePath = child.GetNicePath()
+	test.AssertResult(t, "parent.child", nicePath)
+}
+
+func TestCandidateNodeFilterMapContentByKey(t *testing.T) {
+	// Create a map with multiple key-value pairs
+	key1 := createStringScalarNode("key1")
+	value1 := createStringScalarNode("value1")
+	key2 := createStringScalarNode("key2")
+	value2 := createStringScalarNode("value2")
+	key3 := createStringScalarNode("key3")
+	value3 := createStringScalarNode("value3")
+
+	mapNode := &CandidateNode{
+		Kind:    MappingNode,
+		Content: []*CandidateNode{key1, value1, key2, value2, key3, value3},
+	}
+
+	// Filter by key predicate that matches key1 and key3
+	filtered := mapNode.FilterMapContentByKey(func(key *CandidateNode) bool {
+		return key.Value == "key1" || key.Value == "key3"
+	})
+
+	// Should return key1, value1, key3, value3
+	test.AssertResult(t, 4, len(filtered))
+	test.AssertResult(t, "key1", filtered[0].Value)
+	test.AssertResult(t, "value1", filtered[1].Value)
+	test.AssertResult(t, "key3", filtered[2].Value)
+	test.AssertResult(t, "value3", filtered[3].Value)
+}
+
+func TestCandidateNodeVisitValues(t *testing.T) {
+	// Test mapping node
+	key1 := createStringScalarNode("key1")
+	value1 := createStringScalarNode("value1")
+	key2 := createStringScalarNode("key2")
+	value2 := createStringScalarNode("value2")
+
+	mapNode := &CandidateNode{
+		Kind:    MappingNode,
+		Content: []*CandidateNode{key1, value1, key2, value2},
+	}
+
+	var visited []string
+	err := mapNode.VisitValues(func(node *CandidateNode) error {
+		visited = append(visited, node.Value)
+		return nil
+	})
+
+	test.AssertResult(t, nil, err)
+	test.AssertResult(t, 2, len(visited))
+	test.AssertResult(t, "value1", visited[0])
+	test.AssertResult(t, "value2", visited[1])
+
+	// Test sequence node
+	item1 := createStringScalarNode("item1")
+	item2 := createStringScalarNode("item2")
+
+	seqNode := &CandidateNode{
+		Kind:    SequenceNode,
+		Content: []*CandidateNode{item1, item2},
+	}
+
+	visited = []string{}
+	err = seqNode.VisitValues(func(node *CandidateNode) error {
+		visited = append(visited, node.Value)
+		return nil
+	})
+
+	test.AssertResult(t, nil, err)
+	test.AssertResult(t, 2, len(visited))
+	test.AssertResult(t, "item1", visited[0])
+	test.AssertResult(t, "item2", visited[1])
+
+	// Test scalar node (should not visit anything)
+	scalarNode := &CandidateNode{
+		Kind:  ScalarNode,
+		Value: "scalar",
+	}
+
+	visited = []string{}
+	err = scalarNode.VisitValues(func(node *CandidateNode) error {
+		visited = append(visited, node.Value)
+		return nil
+	})
+
+	test.AssertResult(t, nil, err)
+	test.AssertResult(t, 0, len(visited))
+}
+
+func TestCandidateNodeCanVisitValues(t *testing.T) {
+	mapNode := &CandidateNode{Kind: MappingNode}
+	seqNode := &CandidateNode{Kind: SequenceNode}
+	scalarNode := &CandidateNode{Kind: ScalarNode}
+
+	test.AssertResult(t, true, mapNode.CanVisitValues())
+	test.AssertResult(t, true, seqNode.CanVisitValues())
+	test.AssertResult(t, false, scalarNode.CanVisitValues())
+}
+
+func TestCandidateNodeAddChild(t *testing.T) {
+	parent := &CandidateNode{Kind: SequenceNode}
+	child := createStringScalarNode("child")
+
+	parent.AddChild(child)
+
+	test.AssertResult(t, 1, len(parent.Content))
+	test.AssertResult(t, false, parent.Content[0].IsMapKey)
+	test.AssertResult(t, "0", parent.Content[0].Key.Value)
+	// Check that parent is set correctly
+	if parent.Content[0].Parent != parent {
+		t.Errorf("Expected parent to be set correctly")
+	}
+}
+
+func TestCandidateNodeAddChildren(t *testing.T) {
+	// Test sequence node
+	parent := &CandidateNode{Kind: SequenceNode}
+	child1 := createStringScalarNode("child1")
+	child2 := createStringScalarNode("child2")
+
+	parent.AddChildren([]*CandidateNode{child1, child2})
+
+	test.AssertResult(t, 2, len(parent.Content))
+	test.AssertResult(t, "child1", parent.Content[0].Value)
+	test.AssertResult(t, "child2", parent.Content[1].Value)
+
+	// Test mapping node
+	mapParent := &CandidateNode{Kind: MappingNode}
+	key1 := createStringScalarNode("key1")
+	value1 := createStringScalarNode("value1")
+	key2 := createStringScalarNode("key2")
+	value2 := createStringScalarNode("value2")
+
+	mapParent.AddChildren([]*CandidateNode{key1, value1, key2, value2})
+
+	test.AssertResult(t, 4, len(mapParent.Content))
+	test.AssertResult(t, true, mapParent.Content[0].IsMapKey)  // key1
+	test.AssertResult(t, false, mapParent.Content[1].IsMapKey) // value1
+	test.AssertResult(t, true, mapParent.Content[2].IsMapKey)  // key2
+	test.AssertResult(t, false, mapParent.Content[3].IsMapKey) // value2
+}

pkg/yqlib/candidate_node_yaml.go

@@ -3,7 +3,7 @@ package yqlib
 import (
 	"fmt"
 
-	yaml "gopkg.in/yaml.v3"
+	yaml "go.yaml.in/yaml/v4"
 )
 
 func MapYamlStyle(original yaml.Style) Style {
@@ -78,9 +78,6 @@ func (o *CandidateNode) copyToYamlNode(node *yaml.Node) {
 	node.Value = o.Value
 	node.Anchor = o.Anchor
 
-	// node.Alias = TODO - find Alias in our own structure
-	// might need to be a post process thing
-
 	node.HeadComment = o.HeadComment
 
 	node.LineComment = o.LineComment

pkg/yqlib/candidiate_node_json.go

@@ -1,3 +1,5 @@
+//go:build !yq_nojson
+
 package yqlib
 
 import (

pkg/yqlib/chown_linux_test.go

@@ -0,0 +1,139 @@
+//go:build linux
+
+package yqlib
+
+import (
+	"os"
+	"path/filepath"
+	"testing"
+	"time"
+)
+
+func TestChangeOwner(t *testing.T) {
+	// Create a temporary file for testing
+	tempDir := t.TempDir()
+	testFile := filepath.Join(tempDir, "testfile.txt")
+
+	// Create a test file
+	err := os.WriteFile(testFile, []byte("test content"), 0600)
+	if err != nil {
+		t.Fatalf("Failed to create test file: %v", err)
+	}
+
+	// Get file info
+	info, err := os.Stat(testFile)
+	if err != nil {
+		t.Fatalf("Failed to stat test file: %v", err)
+	}
+
+	// Create another temporary file to change ownership of
+	tempFile, err := os.CreateTemp(tempDir, "chown_test_*.txt")
+	if err != nil {
+		t.Fatalf("Failed to create temp file: %v", err)
+	}
+	defer os.Remove(tempFile.Name())
+	tempFile.Close()
+
+	// Test changeOwner function
+	err = changeOwner(info, tempFile)
+	if err != nil {
+		t.Errorf("changeOwner failed: %v", err)
+	}
+
+	// Verify that the function doesn't panic with valid input
+	tempFile2, err := os.CreateTemp(tempDir, "chown_test2_*.txt")
+	if err != nil {
+		t.Fatalf("Failed to create second temp file: %v", err)
+	}
+	defer os.Remove(tempFile2.Name())
+	tempFile2.Close()
+
+	// Test with the second file
+	err = changeOwner(info, tempFile2)
+	if err != nil {
+		t.Errorf("changeOwner failed on second file: %v", err)
+	}
+}
+
+func TestChangeOwnerWithInvalidFileInfo(t *testing.T) {
+	// Create a mock file info that doesn't have syscall.Stat_t
+	mockInfo := &mockFileInfo{
+		name: "mock",
+		size: 0,
+		mode: 0600,
+	}
+
+	// Create a temporary file
+	tempFile, err := os.CreateTemp(t.TempDir(), "chown_test_*.txt")
+	if err != nil {
+		t.Fatalf("Failed to create temp file: %v", err)
+	}
+	defer os.Remove(tempFile.Name())
+	tempFile.Close()
+
+	// Test changeOwner with mock file info (should not panic)
+	err = changeOwner(mockInfo, tempFile)
+	if err != nil {
+		t.Errorf("changeOwner failed with mock file info: %v", err)
+	}
+}
+
+func TestChangeOwnerWithNonExistentFile(t *testing.T) {
+	// Create a temporary file
+	tempFile, err := os.CreateTemp(t.TempDir(), "chown_test_*.txt")
+	if err != nil {
+		t.Fatalf("Failed to create temp file: %v", err)
+	}
+	defer os.Remove(tempFile.Name())
+	tempFile.Close()
+
+	// Get file info
+	info, err := os.Stat(tempFile.Name())
+	if err != nil {
+		t.Fatalf("Failed to stat temp file: %v", err)
+	}
+
+	// Remove the file
+	os.Remove(tempFile.Name())
+
+	err = changeOwner(info, tempFile)
+	// The function should not panic even if the file doesn't exist
+	if err != nil {
+		t.Logf("Expected error when changing owner of non-existent file: %v", err)
+	}
+}
+
+// mockFileInfo implements fs.FileInfo but doesn't have syscall.Stat_t
+type mockFileInfo struct {
+	name string
+	size int64
+	mode os.FileMode
+}
+
+func (m *mockFileInfo) Name() string       { return m.name }
+func (m *mockFileInfo) Size() int64        { return m.size }
+func (m *mockFileInfo) Mode() os.FileMode  { return m.mode }
+func (m *mockFileInfo) ModTime() time.Time { return time.Time{} }
+func (m *mockFileInfo) IsDir() bool        { return false }
+func (m *mockFileInfo) Sys() interface{}   { return nil } // This will cause the type assertion to fail
+
+func TestChangeOwnerWithSyscallStatT(t *testing.T) {
+	// Create a temporary file
+	tempFile, err := os.CreateTemp(t.TempDir(), "chown_test_*.txt")
+	if err != nil {
+		t.Fatalf("Failed to create temp file: %v", err)
+	}
+	defer os.Remove(tempFile.Name())
+	tempFile.Close()
+
+	// Get file info
+	info, err := os.Stat(tempFile.Name())
+	if err != nil {
+		t.Fatalf("Failed to stat temp file: %v", err)
+	}
+
+	err = changeOwner(info, tempFile)
+	if err != nil {
+		t.Logf("changeOwner returned error (this might be expected in some environments): %v", err)
+	}
+}

pkg/yqlib/chown_not_linux_os.go

@@ -7,6 +7,6 @@ import (
 	"os"
 )
 
-func changeOwner(info fs.FileInfo, file *os.File) error {
+func changeOwner(_ fs.FileInfo, _ *os.File) error {
 	return nil
 }

pkg/yqlib/color_print.go

@@ -56,6 +56,12 @@ func colorizeAndPrint(yamlBytes []byte, writer io.Writer) error {
 			Suffix: format(color.Reset),
 		}
 	}
+	p.Comment = func() *printer.Property {
+		return &printer.Property{
+			Prefix: format(color.FgHiBlack),
+			Suffix: format(color.Reset),
+		}
+	}
 	_, err := writer.Write([]byte(p.PrintTokens(tokens) + "\n"))
 	return err
 }

pkg/yqlib/color_print_test.go

@@ -0,0 +1,153 @@
+package yqlib
+
+import (
+	"bytes"
+	"strings"
+	"testing"
+
+	"github.com/fatih/color"
+)
+
+func TestFormat(t *testing.T) {
+	tests := []struct {
+		name     string
+		attr     color.Attribute
+		expected string
+	}{
+		{
+			name:     "reset color",
+			attr:     color.Reset,
+			expected: "\x1b[0m",
+		},
+		{
+			name:     "red color",
+			attr:     color.FgRed,
+			expected: "\x1b[31m",
+		},
+		{
+			name:     "green color",
+			attr:     color.FgGreen,
+			expected: "\x1b[32m",
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			result := format(tt.attr)
+			if result != tt.expected {
+				t.Errorf("format(%d) = %q, want %q", tt.attr, result, tt.expected)
+			}
+		})
+	}
+}
+
+func TestColorizeAndPrint(t *testing.T) {
+	tests := []struct {
+		name      string
+		yamlBytes []byte
+		expectErr bool
+	}{
+		{
+			name:      "simple yaml",
+			yamlBytes: []byte("name: test\nage: 25\n"),
+			expectErr: false,
+		},
+		{
+			name:      "yaml with strings",
+			yamlBytes: []byte("name: \"hello world\"\nactive: true\ncount: 42\n"),
+			expectErr: false,
+		},
+		{
+			name:      "yaml with anchors and aliases",
+			yamlBytes: []byte("default: &default\n  name: test\nuser: *default\n"),
+			expectErr: false,
+		},
+		{
+			name:      "yaml with comments",
+			yamlBytes: []byte("# This is a comment\nname: test\n"),
+			expectErr: false,
+		},
+		{
+			name:      "empty yaml",
+			yamlBytes: []byte(""),
+			expectErr: false,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			var buf bytes.Buffer
+			err := colorizeAndPrint(tt.yamlBytes, &buf)
+
+			if tt.expectErr && err == nil {
+				t.Error("Expected error but got none")
+			}
+			if !tt.expectErr && err != nil {
+				t.Errorf("Unexpected error: %v", err)
+			}
+
+			// Check that output contains escape sequences (color codes)
+			if !tt.expectErr && len(tt.yamlBytes) > 0 {
+				output := buf.String()
+				if !strings.Contains(output, "\x1b[") {
+					t.Error("Expected output to contain color escape sequences")
+				}
+			}
+		})
+	}
+}
+
+func TestColorizeAndPrintWithDifferentYamlTypes(t *testing.T) {
+	testCases := []struct {
+		name      string
+		yaml      string
+		expectErr bool
+	}{
+		{
+			name: "boolean values",
+			yaml: "active: true\ninactive: false\n",
+		},
+		{
+			name: "numeric values",
+			yaml: "integer: 42\nfloat: 3.14\nnegative: -10\n",
+		},
+		{
+			name: "map keys",
+			yaml: "user:\n  name: john\n  age: 30\n",
+		},
+		{
+			name: "string values",
+			yaml: "message: \"hello world\"\ndescription: 'single quotes'\n",
+		},
+		{
+			name: "mixed types",
+			yaml: "config:\n  debug: true\n  port: 8080\n  host: \"localhost\"\n",
+		},
+	}
+
+	for _, tc := range testCases {
+		t.Run(tc.name, func(t *testing.T) {
+			var buf bytes.Buffer
+			err := colorizeAndPrint([]byte(tc.yaml), &buf)
+
+			if tc.expectErr && err == nil {
+				t.Error("Expected error but got none")
+			}
+			if !tc.expectErr && err != nil {
+				t.Errorf("Unexpected error: %v", err)
+			}
+
+			// Verify output contains color codes
+			if !tc.expectErr {
+				output := buf.String()
+				if !strings.Contains(output, "\x1b[") {
+					t.Error("Expected output to contain color escape sequences")
+				}
+				// Should end with newline
+				if !strings.HasSuffix(output, "\n") {
+					t.Error("Expected output to end with newline")
+				}
+			}
+		})
+	}
+}

pkg/yqlib/context_test.go

@@ -2,9 +2,11 @@ package yqlib
 
 import (
 	"container/list"
+	"strings"
 	"testing"
 
 	"github.com/mikefarah/yq/v4/test"
+	logging "gopkg.in/op/go-logging.v1"
 )
 
 func TestChildContext(t *testing.T) {
@@ -49,3 +51,211 @@ func TestChildContextNoVariables(t *testing.T) {
 	test.AssertResultComplex(t, make(map[string]*list.List), clone.Variables)
 
 }
+
+func TestSingleReadonlyChildContext(t *testing.T) {
+	original := Context{
+		DontAutoCreate: false,
+		datetimeLayout: "2006-01-02",
+	}
+
+	candidate := &CandidateNode{Value: "test"}
+	clone := original.SingleReadonlyChildContext(candidate)
+
+	// Should have DontAutoCreate set to true
+	test.AssertResultComplex(t, true, clone.DontAutoCreate)
+
+	// Should have the candidate node in MatchingNodes
+	test.AssertResultComplex(t, 1, clone.MatchingNodes.Len())
+	test.AssertResultComplex(t, candidate, clone.MatchingNodes.Front().Value)
+}
+
+func TestSingleChildContext(t *testing.T) {
+	original := Context{
+		DontAutoCreate: true,
+		datetimeLayout: "2006-01-02",
+	}
+
+	candidate := &CandidateNode{Value: "test"}
+	clone := original.SingleChildContext(candidate)
+
+	// Should preserve DontAutoCreate
+	test.AssertResultComplex(t, true, clone.DontAutoCreate)
+
+	// Should have the candidate node in MatchingNodes
+	test.AssertResultComplex(t, 1, clone.MatchingNodes.Len())
+	test.AssertResultComplex(t, candidate, clone.MatchingNodes.Front().Value)
+}
+
+func TestSetDateTimeLayout(t *testing.T) {
+	context := Context{}
+
+	// Test setting datetime layout
+	context.SetDateTimeLayout("2006-01-02T15:04:05Z07:00")
+	test.AssertResultComplex(t, "2006-01-02T15:04:05Z07:00", context.datetimeLayout)
+}
+
+func TestGetDateTimeLayout(t *testing.T) {
+	// Test with custom layout
+	context := Context{datetimeLayout: "2006-01-02"}
+	result := context.GetDateTimeLayout()
+	test.AssertResultComplex(t, "2006-01-02", result)
+
+	// Test with empty layout (should return default)
+	context = Context{}
+	result = context.GetDateTimeLayout()
+	test.AssertResultComplex(t, "2006-01-02T15:04:05Z07:00", result)
+}
+
+func TestGetVariable(t *testing.T) {
+	// Test with nil Variables
+	context := Context{}
+	result := context.GetVariable("nonexistent")
+	test.AssertResultComplex(t, (*list.List)(nil), result)
+
+	// Test with existing variable
+	variables := make(map[string]*list.List)
+	variables["test"] = list.New()
+	variables["test"].PushBack(&CandidateNode{Value: "value"})
+
+	context = Context{Variables: variables}
+	result = context.GetVariable("test")
+	test.AssertResultComplex(t, variables["test"], result)
+
+	// Test with non-existent variable
+	result = context.GetVariable("nonexistent")
+	test.AssertResultComplex(t, (*list.List)(nil), result)
+}
+
+func TestSetVariable(t *testing.T) {
+	// Test setting variable when Variables is nil
+	context := Context{}
+	value := list.New()
+	value.PushBack(&CandidateNode{Value: "test"})
+
+	context.SetVariable("key", value)
+	test.AssertResultComplex(t, value, context.Variables["key"])
+
+	// Test setting variable when Variables already exists
+	context.SetVariable("key2", value)
+	test.AssertResultComplex(t, value, context.Variables["key2"])
+}
+
+func TestToString(t *testing.T) {
+	context := Context{
+		DontAutoCreate: true,
+		MatchingNodes:  list.New(),
+	}
+
+	// Add a node to test the full string representation
+	node := &CandidateNode{Value: "test"}
+	context.MatchingNodes.PushBack(node)
+
+	// Test with debug logging disabled (default)
+	result := context.ToString()
+	test.AssertResultComplex(t, "", result)
+
+	// Test with debug logging enabled
+	logging.SetLevel(logging.DEBUG, "")
+	defer logging.SetLevel(logging.INFO, "") // Reset to default
+
+	result2 := context.ToString()
+	test.AssertResultComplex(t, true, len(result2) > 0)
+	test.AssertResultComplex(t, true, strings.Contains(result2, "Context"))
+	test.AssertResultComplex(t, true, strings.Contains(result2, "DontAutoCreate: true"))
+}
+
+func TestDeepClone(t *testing.T) {
+	// Create original context with variables and matching nodes
+	originalVariables := make(map[string]*list.List)
+	originalVariables["test"] = list.New()
+	originalVariables["test"].PushBack(&CandidateNode{Value: "original"})
+
+	original := Context{
+		DontAutoCreate: true,
+		datetimeLayout: "2006-01-02",
+		Variables:      originalVariables,
+		MatchingNodes:  list.New(),
+	}
+
+	// Add a node to MatchingNodes
+	node := &CandidateNode{Value: "test"}
+	original.MatchingNodes.PushBack(node)
+
+	clone := original.DeepClone()
+
+	// Should preserve DontAutoCreate and datetimeLayout
+	test.AssertResultComplex(t, original.DontAutoCreate, clone.DontAutoCreate)
+	test.AssertResultComplex(t, original.datetimeLayout, clone.datetimeLayout)
+
+	// Should have copied variables
+	test.AssertResultComplex(t, 1, len(clone.Variables))
+	test.AssertResultComplex(t, "original", clone.Variables["test"].Front().Value.(*CandidateNode).Value)
+
+	// Should have deep copied MatchingNodes
+	test.AssertResultComplex(t, 1, clone.MatchingNodes.Len())
+
+	// Verify it's a deep copy by modifying the original
+	original.MatchingNodes.Front().Value.(*CandidateNode).Value = "modified"
+	test.AssertResultComplex(t, "test", clone.MatchingNodes.Front().Value.(*CandidateNode).Value)
+}
+
+func TestClone(t *testing.T) {
+	// Create original context
+	original := Context{
+		DontAutoCreate: true,
+		datetimeLayout: "2006-01-02",
+		MatchingNodes:  list.New(),
+	}
+
+	node := &CandidateNode{Value: "test"}
+	original.MatchingNodes.PushBack(node)
+
+	clone := original.Clone()
+
+	// Should preserve DontAutoCreate and datetimeLayout
+	test.AssertResultComplex(t, original.DontAutoCreate, clone.DontAutoCreate)
+	test.AssertResultComplex(t, original.datetimeLayout, clone.datetimeLayout)
+
+	// Should have the same MatchingNodes reference
+	test.AssertResultComplex(t, original.MatchingNodes, clone.MatchingNodes)
+}
+
+func TestReadOnlyClone(t *testing.T) {
+	original := Context{
+		DontAutoCreate: false,
+		datetimeLayout: "2006-01-02",
+		MatchingNodes:  list.New(),
+	}
+
+	node := &CandidateNode{Value: "test"}
+	original.MatchingNodes.PushBack(node)
+
+	clone := original.ReadOnlyClone()
+
+	// Should set DontAutoCreate to true
+	test.AssertResultComplex(t, true, clone.DontAutoCreate)
+
+	// Should preserve other fields
+	test.AssertResultComplex(t, original.datetimeLayout, clone.datetimeLayout)
+	test.AssertResultComplex(t, original.MatchingNodes, clone.MatchingNodes)
+}
+
+func TestWritableClone(t *testing.T) {
+	original := Context{
+		DontAutoCreate: true,
+		datetimeLayout: "2006-01-02",
+		MatchingNodes:  list.New(),
+	}
+
+	node := &CandidateNode{Value: "test"}
+	original.MatchingNodes.PushBack(node)
+
+	clone := original.WritableClone()
+
+	// Should set DontAutoCreate to false
+	test.AssertResultComplex(t, false, clone.DontAutoCreate)
+
+	// Should preserve other fields
+	test.AssertResultComplex(t, original.datetimeLayout, clone.datetimeLayout)
+	test.AssertResultComplex(t, original.MatchingNodes, clone.MatchingNodes)
+}

pkg/yqlib/csv_test.go

@@ -186,6 +186,20 @@ var csvScenarios = []formatScenario{
 		expected:       expectedYamlFromCSVNoParsing,
 		scenarioType:   "decode-csv-no-auto",
 	},
+	{
+		description:  "values starting with #, no auto parse",
+		skipDoc:      true,
+		input:        "value\n#ffff",
+		expected:     "- value: '#ffff'\n",
+		scenarioType: "decode-csv-no-auto",
+	},
+	{
+		description:  "values starting with #",
+		skipDoc:      true,
+		input:        "value\n#ffff",
+		expected:     "- value: #ffff\n",
+		scenarioType: "decode-csv",
+	},
 	{
 		description:  "Scalar roundtrip",
 		skipDoc:      true,

pkg/yqlib/data_tree_navigator.go

@@ -64,6 +64,6 @@ func (d *dataTreeNavigator) GetMatchingNodes(context Context, expressionNode *Ex
 	if handler != nil {
 		return handler(d, context, expressionNode)
 	}
-	return Context{}, fmt.Errorf("Unknown operator %v", expressionNode.Operation.OperationType)
+	return Context{}, fmt.Errorf("unknown operator %v", expressionNode.Operation.OperationType.Type)
 
 }

pkg/yqlib/data_tree_navigator_test.go

@@ -0,0 +1,435 @@
+package yqlib
+
+import (
+	"container/list"
+	"testing"
+
+	"github.com/mikefarah/yq/v4/test"
+)
+
+func TestGetMatchingNodes_NilExpressionNode(t *testing.T) {
+	navigator := NewDataTreeNavigator()
+	context := Context{
+		MatchingNodes: list.New(),
+	}
+
+	result, err := navigator.GetMatchingNodes(context, nil)
+
+	test.AssertResult(t, nil, err)
+	test.AssertResultComplex(t, context, result)
+}
+
+func TestGetMatchingNodes_UnknownOperator(t *testing.T) {
+	navigator := NewDataTreeNavigator()
+	context := Context{
+		MatchingNodes: list.New(),
+	}
+
+	// Create an expression node with an unknown operation type
+	unknownOpType := &operationType{Type: "UNKNOWN", Handler: nil}
+	expressionNode := &ExpressionNode{
+		Operation: &Operation{OperationType: unknownOpType},
+	}
+
+	result, err := navigator.GetMatchingNodes(context, expressionNode)
+
+	test.AssertResult(t, "unknown operator UNKNOWN", err.Error())
+	test.AssertResultComplex(t, Context{}, result)
+}
+
+func TestGetMatchingNodes_ValidOperator(t *testing.T) {
+	navigator := NewDataTreeNavigator()
+
+	// Create a simple context with a scalar node
+	scalarNode := &CandidateNode{
+		Kind:  ScalarNode,
+		Tag:   "!!str",
+		Value: "test",
+	}
+	context := Context{
+		MatchingNodes: list.New(),
+	}
+	context.MatchingNodes.PushBack(scalarNode)
+
+	// Create an expression node with a valid operation (self reference)
+	expressionNode := &ExpressionNode{
+		Operation: &Operation{OperationType: selfReferenceOpType},
+	}
+
+	result, err := navigator.GetMatchingNodes(context, expressionNode)
+
+	test.AssertResult(t, nil, err)
+	test.AssertResult(t, 1, result.MatchingNodes.Len())
+
+	// Verify the result contains the same node
+	resultNode := result.MatchingNodes.Front().Value.(*CandidateNode)
+	test.AssertResult(t, scalarNode, resultNode)
+}
+
+func TestDeeplyAssign_ScalarNode(t *testing.T) {
+	navigator := NewDataTreeNavigator()
+
+	// Create a context with a root mapping node
+	rootNode := &CandidateNode{
+		Kind: MappingNode,
+		Tag:  "!!map",
+		Content: []*CandidateNode{
+			{Kind: ScalarNode, Tag: "!!str", Value: "existing", IsMapKey: true},
+			{Kind: ScalarNode, Tag: "!!str", Value: "old_value"},
+		},
+	}
+	context := Context{
+		MatchingNodes: list.New(),
+	}
+	context.MatchingNodes.PushBack(rootNode)
+
+	// Create a scalar node to assign
+	scalarNode := &CandidateNode{
+		Kind:  ScalarNode,
+		Tag:   "!!str",
+		Value: "new_value",
+	}
+
+	// Assign to path ["new_key"]
+	path := []interface{}{"new_key"}
+	err := navigator.DeeplyAssign(context, path, scalarNode)
+
+	test.AssertResult(t, nil, err)
+
+	// Verify the assignment was made
+	// The root node should now have the new key-value pair
+	test.AssertResult(t, 4, len(rootNode.Content)) // 2 original + 2 new
+
+	// Find the new key-value pair
+	found := false
+	for i := 0; i < len(rootNode.Content)-1; i += 2 {
+		key := rootNode.Content[i]
+		value := rootNode.Content[i+1]
+		if key.Value == "new_key" && value.Value == "new_value" {
+			found = true
+			break
+		}
+	}
+	test.AssertResult(t, true, found)
+}
+
+func TestDeeplyAssign_MappingNode(t *testing.T) {
+	navigator := NewDataTreeNavigator()
+
+	// Create a context with a root mapping node
+	rootNode := &CandidateNode{
+		Kind: MappingNode,
+		Tag:  "!!map",
+		Content: []*CandidateNode{
+			{Kind: ScalarNode, Tag: "!!str", Value: "existing", IsMapKey: true},
+			{Kind: ScalarNode, Tag: "!!str", Value: "old_value"},
+		},
+	}
+	context := Context{
+		MatchingNodes: list.New(),
+	}
+	context.MatchingNodes.PushBack(rootNode)
+
+	// Create a mapping node to assign (this should trigger deep merge)
+	mappingNode := &CandidateNode{
+		Kind: MappingNode,
+		Tag:  "!!map",
+		Content: []*CandidateNode{
+			{Kind: ScalarNode, Tag: "!!str", Value: "nested_key", IsMapKey: true},
+			{Kind: ScalarNode, Tag: "!!str", Value: "nested_value"},
+		},
+	}
+
+	// Assign to path ["new_map"]
+	path := []interface{}{"new_map"}
+	err := navigator.DeeplyAssign(context, path, mappingNode)
+
+	test.AssertResult(t, nil, err)
+
+	// Verify the assignment was made
+	// The root node should now have the new mapping
+	test.AssertResult(t, 4, len(rootNode.Content)) // 2 original + 2 new
+
+	// Find the new mapping
+	found := false
+	for i := 0; i < len(rootNode.Content); i += 2 {
+		if i+1 < len(rootNode.Content) {
+			key := rootNode.Content[i]
+			value := rootNode.Content[i+1]
+			if key.Value == "new_map" && value.Kind == MappingNode {
+				found = true
+				// Verify the nested content
+				test.AssertResult(t, 2, len(value.Content))
+				test.AssertResult(t, "nested_key", value.Content[0].Value)
+				test.AssertResult(t, "nested_value", value.Content[1].Value)
+				break
+			}
+		}
+	}
+	test.AssertResult(t, true, found)
+}
+
+func TestDeeplyAssign_DeepPath(t *testing.T) {
+	navigator := NewDataTreeNavigator()
+
+	// Create a context with a root mapping node
+	rootNode := &CandidateNode{
+		Kind: MappingNode,
+		Tag:  "!!map",
+		Content: []*CandidateNode{
+			{Kind: ScalarNode, Tag: "!!str", Value: "level1", IsMapKey: true},
+			{Kind: MappingNode, Tag: "!!map", Content: []*CandidateNode{}},
+		},
+	}
+	context := Context{
+		MatchingNodes: list.New(),
+	}
+	context.MatchingNodes.PushBack(rootNode)
+
+	// Create a scalar node to assign
+	scalarNode := &CandidateNode{
+		Kind:  ScalarNode,
+		Tag:   "!!str",
+		Value: "deep_value",
+	}
+
+	// Assign to deep path ["level1", "level2", "level3"]
+	path := []interface{}{"level1", "level2", "level3"}
+	err := navigator.DeeplyAssign(context, path, scalarNode)
+
+	test.AssertResult(t, nil, err)
+
+	// Verify the deep assignment was made
+	level1Node := rootNode.Content[1]                // The mapping node
+	test.AssertResult(t, 2, len(level1Node.Content)) // Should have level2 key-value
+
+	level2Key := level1Node.Content[0]
+	level2Value := level1Node.Content[1]
+	test.AssertResult(t, "level2", level2Key.Value)
+	test.AssertResult(t, MappingNode, level2Value.Kind)
+
+	level3Key := level2Value.Content[0]
+	level3Value := level2Value.Content[1]
+	test.AssertResult(t, "level3", level3Key.Value)
+	test.AssertResult(t, "deep_value", level3Value.Value)
+}
+
+func TestDeeplyAssign_ArrayPath(t *testing.T) {
+	navigator := NewDataTreeNavigator()
+
+	// Create a context with a root mapping node containing an array
+	rootNode := &CandidateNode{
+		Kind: MappingNode,
+		Tag:  "!!map",
+		Content: []*CandidateNode{
+			{Kind: ScalarNode, Tag: "!!str", Value: "array", IsMapKey: true},
+			{Kind: SequenceNode, Tag: "!!seq", Content: []*CandidateNode{}},
+		},
+	}
+	context := Context{
+		MatchingNodes: list.New(),
+	}
+	context.MatchingNodes.PushBack(rootNode)
+
+	// Create a scalar node to assign
+	scalarNode := &CandidateNode{
+		Kind:  ScalarNode,
+		Tag:   "!!str",
+		Value: "array_value",
+	}
+
+	// Assign to array path ["array", 0]
+	path := []interface{}{"array", 0}
+	err := navigator.DeeplyAssign(context, path, scalarNode)
+
+	test.AssertResult(t, nil, err)
+
+	// Verify the array assignment was made
+	arrayNode := rootNode.Content[1]                // The sequence node
+	test.AssertResult(t, 1, len(arrayNode.Content)) // Should have one element
+
+	arrayElement := arrayNode.Content[0]
+	test.AssertResult(t, "array_value", arrayElement.Value)
+}
+
+func TestDeeplyAssign_OverwriteExisting(t *testing.T) {
+	navigator := NewDataTreeNavigator()
+
+	// Create a context with a root mapping node
+	rootNode := &CandidateNode{
+		Kind: MappingNode,
+		Tag:  "!!map",
+		Content: []*CandidateNode{
+			{Kind: ScalarNode, Tag: "!!str", Value: "key", IsMapKey: true},
+			{Kind: ScalarNode, Tag: "!!str", Value: "old_value"},
+		},
+	}
+	context := Context{
+		MatchingNodes: list.New(),
+	}
+	context.MatchingNodes.PushBack(rootNode)
+
+	// Create a scalar node to assign
+	scalarNode := &CandidateNode{
+		Kind:  ScalarNode,
+		Tag:   "!!str",
+		Value: "new_value",
+	}
+
+	// Assign to existing path ["key"]
+	path := []interface{}{"key"}
+	err := navigator.DeeplyAssign(context, path, scalarNode)
+
+	test.AssertResult(t, nil, err)
+
+	// Verify the value was overwritten
+	test.AssertResult(t, 2, len(rootNode.Content)) // Should still have 2 elements
+
+	key := rootNode.Content[0]
+	value := rootNode.Content[1]
+	test.AssertResult(t, "key", key.Value)
+	test.AssertResult(t, "new_value", value.Value) // Should be overwritten
+}
+
+func TestDeeplyAssign_ErrorHandling(t *testing.T) {
+	navigator := NewDataTreeNavigator()
+
+	// Create a context with a scalar node (not a mapping)
+	scalarNode := &CandidateNode{
+		Kind:  ScalarNode,
+		Tag:   "!!str",
+		Value: "not_a_map",
+	}
+	context := Context{
+		MatchingNodes: list.New(),
+	}
+	context.MatchingNodes.PushBack(scalarNode)
+
+	// Create a scalar node to assign
+	assignNode := &CandidateNode{
+		Kind:  ScalarNode,
+		Tag:   "!!str",
+		Value: "value",
+	}
+
+	path := []interface{}{"key"}
+	err := navigator.DeeplyAssign(context, path, assignNode)
+
+	// Print the actual error for debugging
+	if err != nil {
+		t.Logf("Actual error: %v", err)
+	}
+
+	test.AssertResult(t, nil, err)
+}
+
+func TestGetMatchingNodes_WithVariables(t *testing.T) {
+	navigator := NewDataTreeNavigator()
+
+	// Create a context with variables
+	variables := make(map[string]*list.List)
+	varList := list.New()
+	varList.PushBack(&CandidateNode{Kind: ScalarNode, Tag: "!!str", Value: "var_value"})
+	variables["test_var"] = varList
+
+	context := Context{
+		MatchingNodes: list.New(),
+		Variables:     variables,
+	}
+
+	// Create an expression node that gets a variable
+	expressionNode := &ExpressionNode{
+		Operation: &Operation{OperationType: getVariableOpType, StringValue: "test_var"},
+	}
+
+	result, err := navigator.GetMatchingNodes(context, expressionNode)
+
+	test.AssertResult(t, nil, err)
+	test.AssertResult(t, 1, result.MatchingNodes.Len())
+
+	// Verify the variable was retrieved
+	resultNode := result.MatchingNodes.Front().Value.(*CandidateNode)
+	test.AssertResult(t, "var_value", resultNode.Value)
+}
+
+func TestGetMatchingNodes_EmptyContext(t *testing.T) {
+	navigator := NewDataTreeNavigator()
+
+	// Create an empty context
+	context := Context{
+		MatchingNodes: list.New(),
+	}
+
+	// Create an expression node with self reference
+	expressionNode := &ExpressionNode{
+		Operation: &Operation{OperationType: selfReferenceOpType},
+	}
+
+	result, err := navigator.GetMatchingNodes(context, expressionNode)
+
+	test.AssertResult(t, nil, err)
+	test.AssertResult(t, 0, result.MatchingNodes.Len())
+}
+
+func TestDeeplyAssign_ComplexMappingMerge(t *testing.T) {
+	navigator := NewDataTreeNavigator()
+
+	// Create a context with a root mapping node containing nested data
+	rootNode := &CandidateNode{
+		Kind: MappingNode,
+		Tag:  "!!map",
+		Content: []*CandidateNode{
+			{Kind: ScalarNode, Tag: "!!str", Value: "config", IsMapKey: true},
+			{Kind: MappingNode, Tag: "!!map", Content: []*CandidateNode{
+				{Kind: ScalarNode, Tag: "!!str", Value: "existing_key", IsMapKey: true},
+				{Kind: ScalarNode, Tag: "!!str", Value: "existing_value"},
+			}},
+		},
+	}
+	context := Context{
+		MatchingNodes: list.New(),
+	}
+	context.MatchingNodes.PushBack(rootNode)
+
+	// Create a mapping node to merge
+	mappingNode := &CandidateNode{
+		Kind: MappingNode,
+		Tag:  "!!map",
+		Content: []*CandidateNode{
+			{Kind: ScalarNode, Tag: "!!str", Value: "new_key", IsMapKey: true},
+			{Kind: ScalarNode, Tag: "!!str", Value: "new_value"},
+			{Kind: ScalarNode, Tag: "!!str", Value: "existing_key", IsMapKey: true},
+			{Kind: ScalarNode, Tag: "!!str", Value: "updated_value"},
+		},
+	}
+
+	// Assign to path ["config"] (should merge with existing mapping)
+	path := []interface{}{"config"}
+	err := navigator.DeeplyAssign(context, path, mappingNode)
+
+	test.AssertResult(t, nil, err)
+
+	// Verify the merge was successful
+	configNode := rootNode.Content[1]                // The config mapping node
+	test.AssertResult(t, 4, len(configNode.Content)) // Should have 2 key-value pairs
+
+	// Check that both existing and new keys are present
+	foundExisting := false
+	foundNew := false
+	for i := 0; i < len(configNode.Content); i += 2 {
+		if i+1 < len(configNode.Content) {
+			key := configNode.Content[i]
+			value := configNode.Content[i+1]
+			switch key.Value {
+			case "existing_key":
+				foundExisting = true
+				test.AssertResult(t, "updated_value", value.Value) // Should be updated
+			case "new_key":
+				foundNew = true
+				test.AssertResult(t, "new_value", value.Value)
+			}
+		}
+	}
+	test.AssertResult(t, true, foundExisting)
+	test.AssertResult(t, true, foundNew)
+}

pkg/yqlib/decoder_base64.go

@@ -1,3 +1,5 @@
+//go:build !yq_nobase64
+
 package yqlib
 
 import (
@@ -7,28 +9,6 @@ import (
 	"strings"
 )
 
-type base64Padder struct {
-	count uint64
-	io.Reader
-}
-
-func (c *base64Padder) pad(buf []byte) (int, error) {
-	pad := strings.Repeat("=", int(4-c.count%4))
-	n, err := strings.NewReader(pad).Read(buf)
-	c.count += uint64(n)
-	return n, err
-}
-
-func (c *base64Padder) Read(buf []byte) (int, error) {
-	n, err := c.Reader.Read(buf)
-	c.count += uint64(n)
-
-	if err == io.EOF && c.count%4 != 0 {
-		return c.pad(buf)
-	}
-	return n, err
-}
-
 type base64Decoder struct {
 	reader       io.Reader
 	finished     bool
@@ -41,7 +21,25 @@ func NewBase64Decoder() Decoder {
 }
 
 func (dec *base64Decoder) Init(reader io.Reader) error {
-	dec.reader = &base64Padder{Reader: reader}
+	// Read all data from the reader and strip leading/trailing whitespace
+	// This is necessary because base64 decoding needs to see the complete input
+	// to handle padding correctly, and we need to strip whitespace before decoding.
+	buf := new(bytes.Buffer)
+	if _, err := buf.ReadFrom(reader); err != nil {
+		return err
+	}
+
+	// Strip leading and trailing whitespace
+	stripped := strings.TrimSpace(buf.String())
+
+	// Add padding if needed (base64 strings should be a multiple of 4 characters)
+	padLen := len(stripped) % 4
+	if padLen > 0 {
+		stripped += strings.Repeat("=", 4-padLen)
+	}
+
+	// Create a new reader from the stripped and padded data
+	dec.reader = strings.NewReader(stripped)
 	dec.readAnything = false
 	dec.finished = false
 	return nil

pkg/yqlib/decoder_csv_object.go

@@ -1,3 +1,5 @@
+//go:build !yq_nocsv
+
 package yqlib
 
 import (
@@ -31,7 +33,7 @@ func (dec *csvObjectDecoder) convertToNode(content string) *CandidateNode {
 	node, err := parseSnippet(content)
 	// if we're not auto-parsing, then we wont put in parsed objects or arrays
 	// but we still parse scalars
-	if err != nil || (!dec.prefs.AutoParse && node.Kind != ScalarNode) {
+	if err != nil || (!dec.prefs.AutoParse && (node.Kind != ScalarNode || node.Value != content)) {
 		return createScalarNode(content, content)
 	}
 	return node

pkg/yqlib/decoder_goccy_yaml.go

@@ -16,6 +16,8 @@ import (
 type goccyYamlDecoder struct {
 	decoder yaml.Decoder
 	cm      yaml.CommentMap
+	// anchor map persists over multiple documents for convenience.
+	anchorMap map[string]*CandidateNode
 }
 
 func NewGoccyYAMLDecoder() Decoder {
@@ -25,6 +27,7 @@ func NewGoccyYAMLDecoder() Decoder {
 func (dec *goccyYamlDecoder) Init(reader io.Reader) error {
 	dec.cm = yaml.CommentMap{}
 	dec.decoder = *yaml.NewDecoder(reader, yaml.CommentToMap(dec.cm), yaml.UseOrderedMap())
+	dec.anchorMap = make(map[string]*CandidateNode)
 	return nil
 }
 
@@ -38,7 +41,7 @@ func (dec *goccyYamlDecoder) Decode() (*CandidateNode, error) {
 	}
 
 	candidateNode := &CandidateNode{}
-	if err := candidateNode.UnmarshalGoccyYAML(ast, dec.cm); err != nil {
+	if err := candidateNode.UnmarshalGoccyYAML(ast, dec.cm, dec.anchorMap); err != nil {
 		return nil, err
 	}
 

pkg/yqlib/decoder_hcl.go

@@ -0,0 +1,471 @@
+//go:build !yq_nohcl
+
+package yqlib
+
+import (
+	"fmt"
+	"io"
+	"math/big"
+	"sort"
+	"strconv"
+	"strings"
+
+	"github.com/hashicorp/hcl/v2"
+	"github.com/hashicorp/hcl/v2/hclsyntax"
+	"github.com/zclconf/go-cty/cty"
+)
+
+type hclDecoder struct {
+	file          *hcl.File
+	fileBytes     []byte
+	readAnything  bool
+	documentIndex uint
+}
+
+func NewHclDecoder() Decoder {
+	return &hclDecoder{}
+}
+
+// sortedAttributes returns attributes in declaration order by source position
+func sortedAttributes(attrs hclsyntax.Attributes) []*attributeWithName {
+	var sorted []*attributeWithName
+	for name, attr := range attrs {
+		sorted = append(sorted, &attributeWithName{Name: name, Attr: attr})
+	}
+	sort.Slice(sorted, func(i, j int) bool {
+		return sorted[i].Attr.Range().Start.Byte < sorted[j].Attr.Range().Start.Byte
+	})
+	return sorted
+}
+
+type attributeWithName struct {
+	Name string
+	Attr *hclsyntax.Attribute
+}
+
+// extractLineComment extracts any inline comment after the given position
+func extractLineComment(src []byte, endPos int) string {
+	// Look for # comment after the token
+	for i := endPos; i < len(src); i++ {
+		if src[i] == '#' {
+			// Found comment, extract until end of line
+			start := i
+			for i < len(src) && src[i] != '\n' {
+				i++
+			}
+			return strings.TrimSpace(string(src[start:i]))
+		}
+		if src[i] == '\n' {
+			// Hit newline before comment
+			break
+		}
+		// Skip whitespace and other characters
+	}
+	return ""
+}
+
+// extractHeadComment extracts comments before a given start position
+func extractHeadComment(src []byte, startPos int) string {
+	var comments []string
+
+	// Start just before the token and skip trailing whitespace
+	i := startPos - 1
+	for i >= 0 && (src[i] == ' ' || src[i] == '\t' || src[i] == '\n' || src[i] == '\r') {
+		i--
+	}
+
+	for i >= 0 {
+		// Find line boundaries
+		lineEnd := i
+		for i >= 0 && src[i] != '\n' {
+			i--
+		}
+		lineStart := i + 1
+
+		line := strings.TrimRight(string(src[lineStart:lineEnd+1]), " \t\r")
+		trimmed := strings.TrimSpace(line)
+
+		if trimmed == "" {
+			break
+		}
+
+		if !strings.HasPrefix(trimmed, "#") {
+			break
+		}
+
+		comments = append([]string{trimmed}, comments...)
+
+		// Move to previous line (skip any whitespace/newlines)
+		i = lineStart - 1
+		for i >= 0 && (src[i] == ' ' || src[i] == '\t' || src[i] == '\n' || src[i] == '\r') {
+			i--
+		}
+	}
+
+	if len(comments) > 0 {
+		return strings.Join(comments, "\n")
+	}
+	return ""
+}
+
+func (dec *hclDecoder) Init(reader io.Reader) error {
+	data, err := io.ReadAll(reader)
+	if err != nil {
+		return err
+	}
+	file, diags := hclsyntax.ParseConfig(data, "input.hcl", hcl.Pos{Line: 1, Column: 1})
+	if diags != nil && diags.HasErrors() {
+		return fmt.Errorf("hcl parse error: %w", diags)
+	}
+	dec.file = file
+	dec.fileBytes = data
+	dec.readAnything = false
+	dec.documentIndex = 0
+	return nil
+}
+
+func (dec *hclDecoder) Decode() (*CandidateNode, error) {
+	if dec.readAnything {
+		return nil, io.EOF
+	}
+	dec.readAnything = true
+
+	if dec.file == nil {
+		return nil, fmt.Errorf("no hcl file parsed")
+	}
+
+	root := &CandidateNode{Kind: MappingNode}
+
+	// process attributes in declaration order
+	body := dec.file.Body.(*hclsyntax.Body)
+	firstAttr := true
+	for _, attrWithName := range sortedAttributes(body.Attributes) {
+		keyNode := createStringScalarNode(attrWithName.Name)
+		valNode := convertHclExprToNode(attrWithName.Attr.Expr, dec.fileBytes)
+
+		// Attach comments if any
+		attrRange := attrWithName.Attr.Range()
+		headComment := extractHeadComment(dec.fileBytes, attrRange.Start.Byte)
+		if firstAttr && headComment != "" {
+			// For the first attribute, apply its head comment to the root
+			root.HeadComment = headComment
+			firstAttr = false
+		} else if headComment != "" {
+			keyNode.HeadComment = headComment
+		}
+		if lineComment := extractLineComment(dec.fileBytes, attrRange.End.Byte); lineComment != "" {
+			valNode.LineComment = lineComment
+		}
+
+		root.AddKeyValueChild(keyNode, valNode)
+	}
+
+	// process blocks
+	// Count blocks by type at THIS level to detect multiple separate blocks
+	blocksByType := make(map[string]int)
+	for _, block := range body.Blocks {
+		blocksByType[block.Type]++
+	}
+
+	for _, block := range body.Blocks {
+		addBlockToMapping(root, block, dec.fileBytes, blocksByType[block.Type] > 1)
+	}
+
+	dec.documentIndex++
+	root.document = dec.documentIndex - 1
+	return root, nil
+}
+
+func hclBodyToNode(body *hclsyntax.Body, src []byte) *CandidateNode {
+	node := &CandidateNode{Kind: MappingNode}
+	for _, attrWithName := range sortedAttributes(body.Attributes) {
+		key := createStringScalarNode(attrWithName.Name)
+		val := convertHclExprToNode(attrWithName.Attr.Expr, src)
+
+		// Attach comments if any
+		attrRange := attrWithName.Attr.Range()
+		if headComment := extractHeadComment(src, attrRange.Start.Byte); headComment != "" {
+			key.HeadComment = headComment
+		}
+		if lineComment := extractLineComment(src, attrRange.End.Byte); lineComment != "" {
+			val.LineComment = lineComment
+		}
+
+		node.AddKeyValueChild(key, val)
+	}
+
+	// Process nested blocks, counting blocks by type at THIS level
+	// to detect which block types appear multiple times
+	blocksByType := make(map[string]int)
+	for _, block := range body.Blocks {
+		blocksByType[block.Type]++
+	}
+
+	for _, block := range body.Blocks {
+		addBlockToMapping(node, block, src, blocksByType[block.Type] > 1)
+	}
+	return node
+}
+
+// addBlockToMapping nests block type and labels into the parent mapping, merging children.
+// isMultipleBlocksOfType indicates if there are multiple blocks of this type at THIS level
+func addBlockToMapping(parent *CandidateNode, block *hclsyntax.Block, src []byte, isMultipleBlocksOfType bool) {
+	bodyNode := hclBodyToNode(block.Body, src)
+	current := parent
+
+	// ensure block type mapping exists
+	var typeNode *CandidateNode
+	for i := 0; i < len(current.Content); i += 2 {
+		if current.Content[i].Value == block.Type {
+			typeNode = current.Content[i+1]
+			break
+		}
+	}
+	if typeNode == nil {
+		_, typeNode = current.AddKeyValueChild(createStringScalarNode(block.Type), &CandidateNode{Kind: MappingNode})
+		// Mark the type node if there are multiple blocks of this type at this level
+		// This tells the encoder to emit them as separate blocks rather than consolidating them
+		if isMultipleBlocksOfType {
+			typeNode.EncodeSeparate = true
+		}
+	}
+	current = typeNode
+
+	// walk labels, creating/merging mappings
+	for _, label := range block.Labels {
+		var next *CandidateNode
+		for i := 0; i < len(current.Content); i += 2 {
+			if current.Content[i].Value == label {
+				next = current.Content[i+1]
+				break
+			}
+		}
+		if next == nil {
+			_, next = current.AddKeyValueChild(createStringScalarNode(label), &CandidateNode{Kind: MappingNode})
+		}
+		current = next
+	}
+
+	// merge body attributes/blocks into the final mapping
+	for i := 0; i < len(bodyNode.Content); i += 2 {
+		current.AddKeyValueChild(bodyNode.Content[i], bodyNode.Content[i+1])
+	}
+}
+
+func convertHclExprToNode(expr hclsyntax.Expression, src []byte) *CandidateNode {
+	// handle literal values directly
+	switch e := expr.(type) {
+	case *hclsyntax.LiteralValueExpr:
+		v := e.Val
+		if v.IsNull() {
+			return createScalarNode(nil, "")
+		}
+		switch {
+		case v.Type().Equals(cty.String):
+			// prefer to extract exact source (to avoid extra quoting) when available
+			// Prefer the actual cty string value
+			s := v.AsString()
+			node := createScalarNode(s, s)
+			// Don't set style for regular quoted strings - let YAML handle naturally
+			return node
+		case v.Type().Equals(cty.Bool):
+			b := v.True()
+			return createScalarNode(b, strconv.FormatBool(b))
+		case v.Type() == cty.Number:
+			// prefer integers when the numeric value is integral
+			bf := v.AsBigFloat()
+			if bf == nil {
+				// fallback to string
+				return createStringScalarNode(v.GoString())
+			}
+			// check if bf represents an exact integer
+			if intVal, acc := bf.Int(nil); acc == big.Exact {
+				s := intVal.String()
+				return createScalarNode(intVal.Int64(), s)
+			}
+			s := bf.Text('g', -1)
+			return createScalarNode(0.0, s)
+		case v.Type().IsTupleType() || v.Type().IsListType() || v.Type().IsSetType():
+			seq := &CandidateNode{Kind: SequenceNode}
+			it := v.ElementIterator()
+			for it.Next() {
+				_, val := it.Element()
+				// convert cty.Value to a node by wrapping in literal expr via string representation
+				child := convertCtyValueToNode(val)
+				seq.AddChild(child)
+			}
+			return seq
+		case v.Type().IsMapType() || v.Type().IsObjectType():
+			m := &CandidateNode{Kind: MappingNode}
+			it := v.ElementIterator()
+			for it.Next() {
+				key, val := it.Element()
+				keyStr := key.AsString()
+				keyNode := createStringScalarNode(keyStr)
+				valNode := convertCtyValueToNode(val)
+				m.AddKeyValueChild(keyNode, valNode)
+			}
+			return m
+		default:
+			// fallback to string
+			s := v.GoString()
+			return createStringScalarNode(s)
+		}
+	case *hclsyntax.TupleConsExpr:
+		// parse tuple/list into YAML sequence
+		seq := &CandidateNode{Kind: SequenceNode}
+		for _, exprVal := range e.Exprs {
+			child := convertHclExprToNode(exprVal, src)
+			seq.AddChild(child)
+		}
+		return seq
+	case *hclsyntax.ObjectConsExpr:
+		// parse object into YAML mapping
+		m := &CandidateNode{Kind: MappingNode}
+		m.Style = FlowStyle // Mark as inline object (flow style) for encoder
+		for _, item := range e.Items {
+			// evaluate key expression to get the key string
+			keyVal, keyDiags := item.KeyExpr.Value(nil)
+			if keyDiags != nil && keyDiags.HasErrors() {
+				// fallback: try to extract key from source
+				r := item.KeyExpr.Range()
+				start := r.Start.Byte
+				end := r.End.Byte
+				if start >= 0 && end >= start && end <= len(src) {
+					keyNode := createStringScalarNode(strings.TrimSpace(string(src[start:end])))
+					valNode := convertHclExprToNode(item.ValueExpr, src)
+					m.AddKeyValueChild(keyNode, valNode)
+				}
+				continue
+			}
+			keyStr := keyVal.AsString()
+			keyNode := createStringScalarNode(keyStr)
+			valNode := convertHclExprToNode(item.ValueExpr, src)
+			m.AddKeyValueChild(keyNode, valNode)
+		}
+		return m
+	case *hclsyntax.TemplateExpr:
+		// Reconstruct template string, preserving ${} syntax for interpolations
+		var parts []string
+		for _, p := range e.Parts {
+			switch lp := p.(type) {
+			case *hclsyntax.LiteralValueExpr:
+				if lp.Val.Type().Equals(cty.String) {
+					parts = append(parts, lp.Val.AsString())
+				} else {
+					parts = append(parts, lp.Val.GoString())
+				}
+			default:
+				// Non-literal expression - reconstruct with ${} wrapper
+				r := p.Range()
+				start := r.Start.Byte
+				end := r.End.Byte
+				if start >= 0 && end >= start && end <= len(src) {
+					exprText := string(src[start:end])
+					parts = append(parts, "${"+exprText+"}")
+				} else {
+					parts = append(parts, fmt.Sprintf("${%v}", p))
+				}
+			}
+		}
+		combined := strings.Join(parts, "")
+		node := createScalarNode(combined, combined)
+		// Set DoubleQuotedStyle for all templates (which includes all quoted strings in HCL)
+		// This ensures HCL roundtrips preserve quotes, and YAML properly quotes strings with ${}
+		node.Style = DoubleQuotedStyle
+		return node
+	case *hclsyntax.ScopeTraversalExpr:
+		// Simple identifier/traversal (e.g. unquoted string literal in HCL)
+		r := e.Range()
+		start := r.Start.Byte
+		end := r.End.Byte
+		if start >= 0 && end >= start && end <= len(src) {
+			text := strings.TrimSpace(string(src[start:end]))
+			return createStringScalarNode(text)
+		}
+		// Fallback to root name if source unavailable
+		if len(e.Traversal) > 0 {
+			if root, ok := e.Traversal[0].(hcl.TraverseRoot); ok {
+				return createStringScalarNode(root.Name)
+			}
+		}
+		return createStringScalarNode("")
+	case *hclsyntax.FunctionCallExpr:
+		// Preserve function calls as raw expressions for roundtrip
+		r := e.Range()
+		start := r.Start.Byte
+		end := r.End.Byte
+		if start >= 0 && end >= start && end <= len(src) {
+			text := strings.TrimSpace(string(src[start:end]))
+			node := createStringScalarNode(text)
+			node.Style = 0
+			return node
+		}
+		node := createStringScalarNode(e.Name)
+		node.Style = 0
+		return node
+	default:
+		// try to evaluate the expression (handles unary, binary ops, etc.)
+		val, diags := expr.Value(nil)
+		if diags == nil || !diags.HasErrors() {
+			// successfully evaluated, convert cty.Value to node
+			return convertCtyValueToNode(val)
+		}
+		// fallback: extract source text for the expression
+		r := expr.Range()
+		start := r.Start.Byte
+		end := r.End.Byte
+		if start >= 0 && end >= start && end <= len(src) {
+			text := string(src[start:end])
+			// Mark as unquoted expression so encoder emits without quoting
+			node := createStringScalarNode(text)
+			node.Style = 0
+			return node
+		}
+		return createStringScalarNode(fmt.Sprintf("%v", expr))
+	}
+}
+
+func convertCtyValueToNode(v cty.Value) *CandidateNode {
+	if v.IsNull() {
+		return createScalarNode(nil, "")
+	}
+	switch {
+	case v.Type().Equals(cty.String):
+		return createScalarNode("", v.AsString())
+	case v.Type().Equals(cty.Bool):
+		b := v.True()
+		return createScalarNode(b, strconv.FormatBool(b))
+	case v.Type() == cty.Number:
+		bf := v.AsBigFloat()
+		if bf == nil {
+			return createStringScalarNode(v.GoString())
+		}
+		if intVal, acc := bf.Int(nil); acc == big.Exact {
+			s := intVal.String()
+			return createScalarNode(intVal.Int64(), s)
+		}
+		s := bf.Text('g', -1)
+		return createScalarNode(0.0, s)
+	case v.Type().IsTupleType() || v.Type().IsListType() || v.Type().IsSetType():
+		seq := &CandidateNode{Kind: SequenceNode}
+		it := v.ElementIterator()
+		for it.Next() {
+			_, val := it.Element()
+			seq.AddChild(convertCtyValueToNode(val))
+		}
+		return seq
+	case v.Type().IsMapType() || v.Type().IsObjectType():
+		m := &CandidateNode{Kind: MappingNode}
+		it := v.ElementIterator()
+		for it.Next() {
+			key, val := it.Element()
+			keyNode := createStringScalarNode(key.AsString())
+			valNode := convertCtyValueToNode(val)
+			m.AddKeyValueChild(keyNode, valNode)
+		}
+		return m
+	default:
+		return createStringScalarNode(v.GoString())
+	}
+}

pkg/yqlib/decoder_ini.go

@@ -0,0 +1,106 @@
+//go:build !yq_noini
+
+package yqlib
+
+import (
+	"fmt"
+	"io"
+
+	"github.com/go-ini/ini"
+)
+
+type iniDecoder struct {
+	reader   io.Reader
+	finished bool // Flag to signal completion of processing
+}
+
+func NewINIDecoder() Decoder {
+	return &iniDecoder{
+		finished: false, // Initialize the flag as false
+	}
+}
+
+func (dec *iniDecoder) Init(reader io.Reader) error {
+	// Store the reader for use in Decode
+	dec.reader = reader
+	return nil
+}
+
+func (dec *iniDecoder) Decode() (*CandidateNode, error) {
+	// If processing is already finished, return io.EOF
+	if dec.finished {
+		return nil, io.EOF
+	}
+
+	// Read all content from the stored reader
+	content, err := io.ReadAll(dec.reader)
+	if err != nil {
+		return nil, fmt.Errorf("failed to read INI content: %w", err)
+	}
+
+	// Parse the INI content
+	cfg, err := ini.Load(content)
+	if err != nil {
+		return nil, fmt.Errorf("failed to parse INI content: %w", err)
+	}
+
+	// Create a root CandidateNode as a MappingNode (since INI is key-value based)
+	root := &CandidateNode{
+		Kind:  MappingNode,
+		Tag:   "!!map",
+		Value: "",
+	}
+
+	// Process each section in the INI file
+	for _, section := range cfg.Sections() {
+		sectionName := section.Name()
+
+		if sectionName == ini.DefaultSection {
+			// For the default section, add key-value pairs directly to the root node
+			for _, key := range section.Keys() {
+				keyName := key.Name()
+				keyValue := key.String()
+
+				// Create a key node (scalar for the key name)
+				keyNode := createStringScalarNode(keyName)
+				// Create a value node (scalar for the value)
+				valueNode := createStringScalarNode(keyValue)
+
+				// Add key-value pair to the root node
+				root.AddKeyValueChild(keyNode, valueNode)
+			}
+		} else {
+			// For named sections, create a nested map
+			sectionNode := &CandidateNode{
+				Kind:  MappingNode,
+				Tag:   "!!map",
+				Value: "",
+			}
+
+			// Add key-value pairs to the section node
+			for _, key := range section.Keys() {
+				keyName := key.Name()
+				keyValue := key.String()
+
+				// Create a key node (scalar for the key name)
+				keyNode := createStringScalarNode(keyName)
+				// Create a value node (scalar for the value)
+				valueNode := createStringScalarNode(keyValue)
+
+				// Add key-value pair to the section node
+				sectionNode.AddKeyValueChild(keyNode, valueNode)
+			}
+
+			// Create a key node for the section name
+			sectionKeyNode := createStringScalarNode(sectionName)
+			// Add the section as a nested map to the root node
+			root.AddKeyValueChild(sectionKeyNode, sectionNode)
+		}
+	}
+
+	// Set the finished flag to true to prevent further Decode calls
+	dec.finished = true
+
+	// Return the root node
+	return root, nil
+}

pkg/yqlib/decoder_properties.go

@@ -1,3 +1,5 @@
+//go:build !yq_noprops
+
 package yqlib
 
 import (

pkg/yqlib/decoder_test.go

@@ -68,7 +68,8 @@ func mustProcessFormatScenario(s formatScenario, decoder Decoder, encoder Encode
 
 	result, err := processFormatScenario(s, decoder, encoder)
 	if err != nil {
-		panic(fmt.Errorf("Bad scenario %v: %w", s.description, err))
+		log.Error("Bad scenario %v: %w", s.description, err)
+		return fmt.Sprintf("Bad scenario %v: %v", s.description, err.Error())
 	}
 	return result
 

pkg/yqlib/decoder_toml.go

@@ -249,11 +249,12 @@ func (dec *tomlDecoder) processTopLevelNode(currentNode *toml.Node) (bool, error
 	var runAgainstCurrentExp bool
 	var err error
 	log.Debug("processTopLevelNode: Going to process %v state is current %v", currentNode.Kind, NodeToString(dec.rootMap))
-	if currentNode.Kind == toml.Table {
+	switch currentNode.Kind {
+	case toml.Table:
 		runAgainstCurrentExp, err = dec.processTable(currentNode)
-	} else if currentNode.Kind == toml.ArrayTable {
+	case toml.ArrayTable:
 		runAgainstCurrentExp, err = dec.processArrayTable(currentNode)
-	} else {
+	default:
 		runAgainstCurrentExp, err = dec.decodeKeyValuesIntoMap(dec.rootMap, currentNode)
 	}
 
@@ -266,6 +267,14 @@ func (dec *tomlDecoder) processTable(currentNode *toml.Node) (bool, error) {
 	fullPath := dec.getFullPath(currentNode.Child())
 	log.Debug("fullpath: %v", fullPath)
 
+	c := Context{}
+	c = c.SingleChildContext(dec.rootMap)
+
+	fullPath, err := getPathToUse(fullPath, dec, c)
+	if err != nil {
+		return false, err
+	}
+
 	tableNodeValue := &CandidateNode{
 		Kind:    MappingNode,
 		Tag:     "!!map",
@@ -274,25 +283,22 @@ func (dec *tomlDecoder) processTable(currentNode *toml.Node) (bool, error) {
 
 	var tableValue *toml.Node
 	runAgainstCurrentExp := false
-	var err error
 	hasValue := dec.parser.NextExpression()
 	// check to see if there is any table data
 	if hasValue {
 		tableValue = dec.parser.Expression()
 		// next expression is not table data, so we are done
 		if tableValue.Kind != toml.KeyValue {
-			log.Debug("got an empty table, returning")
-			return true, nil
-		}
-
-		runAgainstCurrentExp, err = dec.decodeKeyValuesIntoMap(tableNodeValue, tableValue)
-		if err != nil && !errors.Is(io.EOF, err) {
-			return false, err
+			log.Debug("got an empty table")
+			runAgainstCurrentExp = true
+		} else {
+			runAgainstCurrentExp, err = dec.decodeKeyValuesIntoMap(tableNodeValue, tableValue)
+			if err != nil && !errors.Is(err, io.EOF) {
+				return false, err
+			}
 		}
 	}
 
-	c := Context{}
-	c = c.SingleChildContext(dec.rootMap)
 	err = dec.d.DeeplyAssign(c, fullPath, tableNodeValue)
 	if err != nil {
 		return false, err
@@ -323,35 +329,69 @@ func (dec *tomlDecoder) arrayAppend(context Context, path []interface{}, rhsNode
 }
 
 func (dec *tomlDecoder) processArrayTable(currentNode *toml.Node) (bool, error) {
-	log.Debug("Entering processArrayTable")
+	log.Debug("Enter processArrayTable")
 	fullPath := dec.getFullPath(currentNode.Child())
 	log.Debug("Fullpath: %v", fullPath)
 
+	c := Context{}
+	c = c.SingleChildContext(dec.rootMap)
+
+	fullPath, err := getPathToUse(fullPath, dec, c)
+	if err != nil {
+		return false, err
+	}
+
 	// need to use the array append exp to add another entry to
 	// this array: fullpath += [ thing ]
-
 	hasValue := dec.parser.NextExpression()
-	if !hasValue {
-		return false, fmt.Errorf("error retrieving table %v value: %w", fullPath, dec.parser.Error())
-	}
 
 	tableNodeValue := &CandidateNode{
 		Kind: MappingNode,
 		Tag:  "!!map",
 	}
-
-	tableValue := dec.parser.Expression()
-	runAgainstCurrentExp, err := dec.decodeKeyValuesIntoMap(tableNodeValue, tableValue)
-	log.Debugf("table node err: %w", err)
-	if err != nil && !errors.Is(io.EOF, err) {
-		return false, err
+	runAgainstCurrentExp := false
+	// if the next value is a ArrayTable or Table, then its not part of this declaration (not a key value pair)
+	// so lets leave that expression for the next round of parsing
+	if hasValue && (dec.parser.Expression().Kind == toml.ArrayTable || dec.parser.Expression().Kind == toml.Table) {
+		runAgainstCurrentExp = true
+	} else if hasValue {
+		// otherwise, if there is a value, it must be some key value pairs of the
+		// first object in the array!
+		tableValue := dec.parser.Expression()
+		runAgainstCurrentExp, err = dec.decodeKeyValuesIntoMap(tableNodeValue, tableValue)
+		if err != nil && !errors.Is(err, io.EOF) {
+			return false, err
+		}
 	}
-	c := Context{}
-
-	c = c.SingleChildContext(dec.rootMap)
 
 	// += function
 	err = dec.arrayAppend(c, fullPath, tableNodeValue)
 
 	return runAgainstCurrentExp, err
 }
+
+// if fullPath points to an array of maps rather than a map
+// then it should set this element into the _last_ element of that array.
+// Because TOML. So we'll inject the last index into the path.
+
+func getPathToUse(fullPath []interface{}, dec *tomlDecoder, c Context) ([]interface{}, error) {
+	pathToCheck := fullPath
+	if len(fullPath) >= 1 {
+		pathToCheck = fullPath[:len(fullPath)-1]
+	}
+	readOp := createTraversalTree(pathToCheck, traversePreferences{DontAutoCreate: true}, false)
+
+	resultContext, err := dec.d.GetMatchingNodes(c, readOp)
+	if err != nil {
+		return nil, err
+	}
+	if resultContext.MatchingNodes.Len() >= 1 {
+		match := resultContext.MatchingNodes.Front().Value.(*CandidateNode)
+		// path refers to an array, we need to add this to the last element in the array
+		if match.Kind == SequenceNode {
+			fullPath = append(pathToCheck, len(match.Content)-1, fullPath[len(fullPath)-1])
+			log.Debugf("Adding to end of %v array, using path: %v", pathToCheck, fullPath)
+		}
+	}
+	return fullPath, err
+}

pkg/yqlib/decoder_uri.go

@@ -1,3 +1,5 @@
+//go:build !yq_nouri
+
 package yqlib
 
 import (

pkg/yqlib/decoder_xml.go

@@ -270,10 +270,16 @@ func (dec *xmlDecoder) decodeXML(root *xmlNode) error {
 			log.Debug("start element %v", se.Name.Local)
 			elem.state = "started"
 			// Build new a new current element and link it to its parent
+			var label = se.Name.Local
+			if dec.prefs.KeepNamespace {
+				if se.Name.Space != "" {
+					label = se.Name.Space + ":" + se.Name.Local
+				}
+			}
 			elem = &element{
 				parent: elem,
 				n:      &xmlNode{},
-				label:  se.Name.Local,
+				label:  label,
 			}
 
 			// Extract attributes as children
@@ -315,13 +321,14 @@ func (dec *xmlDecoder) decodeXML(root *xmlNode) error {
 		case xml.Comment:
 
 			commentStr := string(xml.CharData(se))
-			if elem.state == "started" {
+			switch elem.state {
+			case "started":
 				applyFootComment(elem, commentStr)
 
-			} else if elem.state == "chardata" {
+			case "chardata":
 				log.Debug("got a line comment for (%v) %v: [%v]", elem.state, elem.label, commentStr)
 				elem.n.LineComment = joinComments([]string{elem.n.LineComment, commentStr}, " ")
-			} else {
+			default:
 				log.Debug("got a head comment for (%v) %v: [%v]", elem.state, elem.label, commentStr)
 				elem.n.HeadComment = joinComments([]string{elem.n.HeadComment, commentStr}, " ")
 			}

pkg/yqlib/decoder_yaml.go

@@ -8,7 +8,14 @@ import (
 	"regexp"
 	"strings"
 
-	yaml "gopkg.in/yaml.v3"
+	yaml "go.yaml.in/yaml/v4"
+)
+
+var (
+	commentLineRe       = regexp.MustCompile(`^\s*#`)
+	yamlDirectiveLineRe = regexp.MustCompile(`^\s*%YAML`)
+	separatorLineRe     = regexp.MustCompile(`^\s*---\s*$`)
+	separatorPrefixRe   = regexp.MustCompile(`^\s*---\s+`)
 )
 
 type yamlDecoder struct {
@@ -33,51 +40,72 @@ func NewYamlDecoder(prefs YamlPreferences) Decoder {
 }
 
 func (dec *yamlDecoder) processReadStream(reader *bufio.Reader) (io.Reader, string, error) {
-	var commentLineRegEx = regexp.MustCompile(`^\s*#`)
-	var yamlDirectiveLineRegEx = regexp.MustCompile(`^\s*%YA`)
 	var sb strings.Builder
+
 	for {
-		peekBytes, err := reader.Peek(4)
-		if errors.Is(err, io.EOF) {
-			// EOF are handled else where..
-			return reader, sb.String(), nil
-		} else if err != nil {
-			return reader, sb.String(), err
-		} else if string(peekBytes[0]) == "\n" {
-			_, err := reader.ReadString('\n')
-			sb.WriteString("\n")
-			if errors.Is(err, io.EOF) {
-				return reader, sb.String(), nil
-			} else if err != nil {
-				return reader, sb.String(), err
-			}
-		} else if string(peekBytes) == "--- " {
-			_, err := reader.ReadString(' ')
-			sb.WriteString("$yqDocSeparator$\n")
-			if errors.Is(err, io.EOF) {
-				return reader, sb.String(), nil
-			} else if err != nil {
-				return reader, sb.String(), err
-			}
-		} else if string(peekBytes) == "---\n" {
-			_, err := reader.ReadString('\n')
-			sb.WriteString("$yqDocSeparator$\n")
-			if errors.Is(err, io.EOF) {
-				return reader, sb.String(), nil
-			} else if err != nil {
-				return reader, sb.String(), err
-			}
-		} else if commentLineRegEx.MatchString(string(peekBytes)) || yamlDirectiveLineRegEx.MatchString(string(peekBytes)) {
-			line, err := reader.ReadString('\n')
-			sb.WriteString(line)
-			if errors.Is(err, io.EOF) {
-				return reader, sb.String(), nil
-			} else if err != nil {
-				return reader, sb.String(), err
-			}
-		} else {
+		line, err := reader.ReadString('\n')
+		if errors.Is(err, io.EOF) && line == "" {
+			// no more data
 			return reader, sb.String(), nil
 		}
+		if err != nil && !errors.Is(err, io.EOF) {
+			return reader, sb.String(), err
+		}
+
+		// Determine newline style and strip it for inspection
+		newline := ""
+		if strings.HasSuffix(line, "\r\n") {
+			newline = "\r\n"
+			line = strings.TrimSuffix(line, "\r\n")
+		} else if strings.HasSuffix(line, "\n") {
+			newline = "\n"
+			line = strings.TrimSuffix(line, "\n")
+		}
+
+		trimmed := strings.TrimSpace(line)
+
+		// Document separator: exact line '---' or a '--- ' prefix followed by content
+		if separatorLineRe.MatchString(trimmed) {
+			sb.WriteString("$yqDocSeparator$")
+			sb.WriteString(newline)
+			if errors.Is(err, io.EOF) {
+				return reader, sb.String(), nil
+			}
+			continue
+		}
+
+		// Handle lines that start with '--- ' followed by more content (e.g. '--- cat')
+		if separatorPrefixRe.MatchString(line) {
+			match := separatorPrefixRe.FindString(line)
+			remainder := line[len(match):]
+			// normalize separator newline: if original had none, default to LF
+			sepNewline := newline
+			if sepNewline == "" {
+				sepNewline = "\n"
+			}
+			sb.WriteString("$yqDocSeparator$")
+			sb.WriteString(sepNewline)
+			// push the remainder back onto the reader and continue processing
+			reader = bufio.NewReader(io.MultiReader(strings.NewReader(remainder), reader))
+			if errors.Is(err, io.EOF) && remainder == "" {
+				return reader, sb.String(), nil
+			}
+			continue
+		}
+
+		// Comments, YAML directives, and blank lines are leading content
+		if commentLineRe.MatchString(line) || yamlDirectiveLineRe.MatchString(line) || trimmed == "" {
+			sb.WriteString(line)
+			sb.WriteString(newline)
+			if errors.Is(err, io.EOF) {
+				return reader, sb.String(), nil
+			}
+			continue
+		}
+
+		// First non-leading line: push it back onto a reader and return
+		originalLine := line + newline
+		return io.MultiReader(strings.NewReader(originalLine), reader), sb.String(), nil
 	}
 }
 
@@ -129,6 +157,8 @@ func (dec *yamlDecoder) Decode() (*CandidateNode, error) {
 		return nil, err
 	} else if err != nil {
 		return nil, err
+	} else if len(yamlNode.Content) == 0 {
+		return nil, errors.New("yaml node has no content")
 	}
 
 	candidateNode := CandidateNode{document: dec.documentIndex}

pkg/yqlib/doc/operators/anchor-and-alias-operators.md

@@ -5,6 +5,18 @@ Use the `alias` and `anchor` operators to read and write yaml aliases and anchor
 `yq` supports merge aliases (like `<<: *blah`) however this is no longer in the standard yaml spec (1.2) and so `yq` will automatically add the `!!merge` tag to these nodes as it is effectively a custom tag.
 
 
+## NOTE --yaml-fix-merge-anchor-to-spec flag
+`yq` doesn't merge anchors `<<:` to spec, in some circumstances it incorrectly overrides existing keys when the spec documents not to do that.
+
+To minimise disruption while still fixing the issue, a flag has been added to toggle this behaviour. This will first default to false; and log warnings to users. Then it will default to true (and still allow users to specify false if needed).
+
+This flag also enables advanced merging, like inline maps, as well as fixes to ensure when exploding a particular path, neighbours are not affect ed.
+
+Long story short, you should be setting this flag to true.
+
+See examples of the flag differences below, where LEGACY is with the flag off; and FIXED is with the flag on.
+
+
 ## Merge one map
 see https://yaml.org/type/merge.html
 
@@ -34,68 +46,6 @@ y: 2
 r: 10
 ```
 
-## Merge multiple maps
-see https://yaml.org/type/merge.html
-
-Given a sample.yml file of:
-```yaml
-- &CENTER
-  x: 1
-  y: 2
-- &LEFT
-  x: 0
-  y: 2
-- &BIG
-  r: 10
-- &SMALL
-  r: 1
-- !!merge <<:
-    - *CENTER
-    - *BIG
-```
-then
-```bash
-yq '.[4] | explode(.)' sample.yml
-```
-will output
-```yaml
-r: 10
-x: 1
-y: 2
-```
-
-## Override
-see https://yaml.org/type/merge.html
-
-Given a sample.yml file of:
-```yaml
-- &CENTER
-  x: 1
-  y: 2
-- &LEFT
-  x: 0
-  y: 2
-- &BIG
-  r: 10
-- &SMALL
-  r: 1
-- !!merge <<:
-    - *BIG
-    - *LEFT
-    - *SMALL
-  x: 1
-```
-then
-```bash
-yq '.[4] | explode(.)' sample.yml
-```
-will output
-```yaml
-r: 10
-x: 1
-y: 2
-```
-
 ## Get anchor
 Given a sample.yml file of:
 ```yaml
@@ -241,7 +191,7 @@ Given a sample.yml file of:
 ```yaml
 f:
   a: &a cat
-  *a: b
+  *a : b
 ```
 then
 ```bash
@@ -254,7 +204,39 @@ f:
   cat: b
 ```
 
-## Explode with merge anchors
+## Dereference and update a field
+Use explode with multiply to dereference an object
+
+Given a sample.yml file of:
+```yaml
+item_value: &item_value
+  value: true
+thingOne:
+  name: item_1
+  !!merge <<: *item_value
+thingTwo:
+  name: item_2
+  !!merge <<: *item_value
+```
+then
+```bash
+yq '.thingOne |= (explode(.) | sort_keys(.)) * {"value": false}' sample.yml
+```
+will output
+```yaml
+item_value: &item_value
+  value: true
+thingOne:
+  name: item_1
+  value: false
+thingTwo:
+  name: item_2
+  !!merge <<: *item_value
+```
+
+## LEGACY: Explode with merge anchors
+Caution: this is for when --yaml-fix-merge-anchor-to-spec=false; it's not to YAML spec because the merge anchors incorrectly override the object values (foobarList.b is set to bar_b when it should still be foobarList_b). Flag will default to true in late 2025
+
 Given a sample.yml file of:
 ```yaml
 foo: &foo
@@ -301,33 +283,201 @@ foobar:
   thing: foobar_thing
 ```
 
-## Dereference and update a field
-Use explode with multiply to dereference an object
+## LEGACY: Merge multiple maps
+see https://yaml.org/type/merge.html. This has the correct data, but the wrong key order; set --yaml-fix-merge-anchor-to-spec=true to fix the key order.
 
 Given a sample.yml file of:
 ```yaml
-item_value: &item_value
-  value: true
-thingOne:
-  name: item_1
-  !!merge <<: *item_value
-thingTwo:
-  name: item_2
-  !!merge <<: *item_value
+- &CENTER
+  x: 1
+  y: 2
+- &LEFT
+  x: 0
+  y: 2
+- &BIG
+  r: 10
+- &SMALL
+  r: 1
+- !!merge <<:
+    - *CENTER
+    - *BIG
 ```
 then
 ```bash
-yq '.thingOne |= explode(.) * {"value": false}' sample.yml
+yq '.[4] | explode(.)' sample.yml
 ```
 will output
 ```yaml
-item_value: &item_value
-  value: true
-thingOne:
-  name: item_1
-  value: false
-thingTwo:
-  name: item_2
-  !!merge <<: *item_value
+r: 10
+x: 1
+y: 2
+```
+
+## LEGACY: Override
+see https://yaml.org/type/merge.html. This has the correct data, but the wrong key order; set --yaml-fix-merge-anchor-to-spec=true to fix the key order.
+
+Given a sample.yml file of:
+```yaml
+- &CENTER
+  x: 1
+  y: 2
+- &LEFT
+  x: 0
+  y: 2
+- &BIG
+  r: 10
+- &SMALL
+  r: 1
+- !!merge <<:
+    - *BIG
+    - *LEFT
+    - *SMALL
+  x: 1
+```
+then
+```bash
+yq '.[4] | explode(.)' sample.yml
+```
+will output
+```yaml
+r: 10
+x: 1
+y: 2
+```
+
+## FIXED: Explode with merge anchors
+Set `--yaml-fix-merge-anchor-to-spec=true` to get this correct merge behaviour (flag will default to true in late 2025).
+Observe that foobarList.b property is still foobarList_b.
+
+Given a sample.yml file of:
+```yaml
+foo: &foo
+  a: foo_a
+  thing: foo_thing
+  c: foo_c
+bar: &bar
+  b: bar_b
+  thing: bar_thing
+  c: bar_c
+foobarList:
+  b: foobarList_b
+  !!merge <<:
+    - *foo
+    - *bar
+  c: foobarList_c
+foobar:
+  c: foobar_c
+  !!merge <<: *foo
+  thing: foobar_thing
+```
+then
+```bash
+yq 'explode(.)' sample.yml
+```
+will output
+```yaml
+foo:
+  a: foo_a
+  thing: foo_thing
+  c: foo_c
+bar:
+  b: bar_b
+  thing: bar_thing
+  c: bar_c
+foobarList:
+  b: foobarList_b
+  a: foo_a
+  thing: foo_thing
+  c: foobarList_c
+foobar:
+  c: foobar_c
+  a: foo_a
+  thing: foobar_thing
+```
+
+## FIXED: Merge multiple maps
+Set `--yaml-fix-merge-anchor-to-spec=true` to get this correct merge behaviour (flag will default to true in late 2025).
+Taken from https://yaml.org/type/merge.html. Same values as legacy, but with the correct key order.
+
+Given a sample.yml file of:
+```yaml
+- &CENTER
+  x: 1
+  y: 2
+- &LEFT
+  x: 0
+  y: 2
+- &BIG
+  r: 10
+- &SMALL
+  r: 1
+- !!merge <<:
+    - *CENTER
+    - *BIG
+```
+then
+```bash
+yq '.[4] | explode(.)' sample.yml
+```
+will output
+```yaml
+x: 1
+y: 2
+r: 10
+```
+
+## FIXED: Override
+Set `--yaml-fix-merge-anchor-to-spec=true` to get this correct merge behaviour (flag will default to true in late 2025).
+Taken from https://yaml.org/type/merge.html. Same values as legacy, but with the correct key order.
+
+Given a sample.yml file of:
+```yaml
+- &CENTER
+  x: 1
+  y: 2
+- &LEFT
+  x: 0
+  y: 2
+- &BIG
+  r: 10
+- &SMALL
+  r: 1
+- !!merge <<:
+    - *BIG
+    - *LEFT
+    - *SMALL
+  x: 1
+```
+then
+```bash
+yq '.[4] | explode(.)' sample.yml
+```
+will output
+```yaml
+r: 10
+y: 2
+x: 1
+```
+
+## Exploding inline merge anchor
+Set `--yaml-fix-merge-anchor-to-spec=true` to get this correct merge behaviour (flag will default to true in late 2025).
+
+
+Given a sample.yml file of:
+```yaml
+a:
+  b: &b 42
+!!merge <<:
+  c: *b
+```
+then
+```bash
+yq 'explode(.) | sort_keys(.)' sample.yml
+```
+will output
+```yaml
+a:
+  b: 42
+c: 42
 ```
 

pkg/yqlib/doc/operators/entries.md

@@ -2,6 +2,8 @@
 
 Similar to the same named functions in `jq` these functions convert to/from an object and an array of key-value pairs. This is most useful for performing operations on keys of maps.
 
+Use `with_entries(op)` as a syntactic sugar for doing `to_entries | op | from_entries`.
+
 ## to_entries Map
 Given a sample.yml file of:
 ```yaml
@@ -101,6 +103,28 @@ KEY_a: 1
 KEY_b: 2
 ```
 
+## Use with_entries to update keys recursively
+We use (.. | select(tag="map")) to find all the maps in the doc, then |= to update each one of those maps. In the update, with_entries is used.
+
+Given a sample.yml file of:
+```yaml
+a: 1
+b:
+  b_a: nested
+  b_b: thing
+```
+then
+```bash
+yq '(.. | select(tag=="!!map")) |= with_entries(.key |= "KEY_" + .)' sample.yml
+```
+will output
+```yaml
+KEY_a: 1
+KEY_b:
+  KEY_b_a: nested
+  KEY_b_b: thing
+```
+
 ## Custom sort map keys
 Use to_entries to convert to an array of key/value pairs, sort the array using sort/sort_by/etc, and convert it back.
 

pkg/yqlib/doc/operators/env-variable-operators.md

@@ -29,6 +29,9 @@ as follows:
 yq '(.. | select(tag == "!!str")) |= envsubst' file.yaml
 ```
 
+## Disabling env operators
+If required, you can use the `--security-disable-env-ops` to disable env operations.
+
 
 ## Read string environment variable
 Running
@@ -254,3 +257,39 @@ will output
 Error: variable ${notThere} not set
 ```
 
+## env() operation fails when security is enabled
+Use `--security-disable-env-ops` to disable env operations for security.
+
+Running
+```bash
+yq --null-input 'env("MYENV")'
+```
+will output
+```bash
+Error: env operations have been disabled
+```
+
+## strenv() operation fails when security is enabled
+Use `--security-disable-env-ops` to disable env operations for security.
+
+Running
+```bash
+yq --null-input 'strenv("MYENV")'
+```
+will output
+```bash
+Error: env operations have been disabled
+```
+
+## envsubst() operation fails when security is enabled
+Use `--security-disable-env-ops` to disable env operations for security.
+
+Running
+```bash
+yq --null-input '"value: ${MYENV}" | envsubst'
+```
+will output
+```bash
+Error: env operations have been disabled
+```
+

pkg/yqlib/doc/operators/first.md

@@ -0,0 +1,319 @@
+# First
+
+Returns the first matching element in an array, or first matching value in a map.
+
+Can be given an expression to match with, otherwise will just return the first.
+
+## First matching element from array
+Given a sample.yml file of:
+```yaml
+- a: banana
+- a: cat
+- a: apple
+```
+then
+```bash
+yq 'first(.a == "cat")' sample.yml
+```
+will output
+```yaml
+a: cat
+```
+
+## First matching element from array with multiple matches
+Given a sample.yml file of:
+```yaml
+- a: banana
+- a: cat
+  b: firstCat
+- a: apple
+- a: cat
+  b: secondCat
+```
+then
+```bash
+yq 'first(.a == "cat")' sample.yml
+```
+will output
+```yaml
+a: cat
+b: firstCat
+```
+
+## First matching element from array with numeric condition
+Given a sample.yml file of:
+```yaml
+- a: 10
+- a: 100
+- a: 1
+- a: 101
+```
+then
+```bash
+yq 'first(.a > 50)' sample.yml
+```
+will output
+```yaml
+a: 100
+```
+
+## First matching element from array with boolean condition
+Given a sample.yml file of:
+```yaml
+- a: false
+- a: true
+  b: firstTrue
+- a: false
+- a: true
+  b: secondTrue
+```
+then
+```bash
+yq 'first(.a == true)' sample.yml
+```
+will output
+```yaml
+a: true
+b: firstTrue
+```
+
+## First matching element from array with null values
+Given a sample.yml file of:
+```yaml
+- a: null
+- a: cat
+- a: apple
+```
+then
+```bash
+yq 'first(.a != null)' sample.yml
+```
+will output
+```yaml
+a: cat
+```
+
+## First matching element from array with complex condition
+Given a sample.yml file of:
+```yaml
+- a: dog
+  b: 7
+- a: cat
+  b: 3
+- a: apple
+  b: 5
+```
+then
+```bash
+yq 'first(.b > 4 and .b < 6)' sample.yml
+```
+will output
+```yaml
+a: apple
+b: 5
+```
+
+## First matching element from map
+Given a sample.yml file of:
+```yaml
+x:
+  a: banana
+y:
+  a: cat
+z:
+  a: apple
+```
+then
+```bash
+yq 'first(.a == "cat")' sample.yml
+```
+will output
+```yaml
+a: cat
+```
+
+## First matching element from map with numeric condition
+Given a sample.yml file of:
+```yaml
+x:
+  a: 10
+y:
+  a: 100
+z:
+  a: 101
+```
+then
+```bash
+yq 'first(.a > 50)' sample.yml
+```
+will output
+```yaml
+a: 100
+```
+
+## First matching element from nested structure
+Given a sample.yml file of:
+```yaml
+items:
+  - a: banana
+  - a: cat
+  - a: apple
+```
+then
+```bash
+yq '.items | first(.a == "cat")' sample.yml
+```
+will output
+```yaml
+a: cat
+```
+
+## First matching element with no matches
+Given a sample.yml file of:
+```yaml
+- a: banana
+- a: cat
+- a: apple
+```
+then
+```bash
+yq 'first(.a == "dog")' sample.yml
+```
+will output
+```yaml
+```
+
+## First matching element from empty array
+Given a sample.yml file of:
+```yaml
+[]
+```
+then
+```bash
+yq 'first(.a == "cat")' sample.yml
+```
+will output
+```yaml
+```
+
+## First matching element from scalar node
+Given a sample.yml file of:
+```yaml
+hello
+```
+then
+```bash
+yq 'first(. == "hello")' sample.yml
+```
+will output
+```yaml
+```
+
+## First matching element from null node
+Given a sample.yml file of:
+```yaml
+null
+```
+then
+```bash
+yq 'first(. == "hello")' sample.yml
+```
+will output
+```yaml
+```
+
+## First matching element with string condition
+Given a sample.yml file of:
+```yaml
+- a: banana
+- a: cat
+- a: apple
+```
+then
+```bash
+yq 'first(.a | test("^c"))' sample.yml
+```
+will output
+```yaml
+a: cat
+```
+
+## First matching element with length condition
+Given a sample.yml file of:
+```yaml
+- a: hi
+- a: hello
+- a: world
+```
+then
+```bash
+yq 'first(.a | length > 4)' sample.yml
+```
+will output
+```yaml
+a: hello
+```
+
+## First matching element from array of strings
+Given a sample.yml file of:
+```yaml
+- banana
+- cat
+- apple
+```
+then
+```bash
+yq 'first(. == "cat")' sample.yml
+```
+will output
+```yaml
+cat
+```
+
+## First matching element from array of numbers
+Given a sample.yml file of:
+```yaml
+- 10
+- 100
+- 1
+```
+then
+```bash
+yq 'first(. > 50)' sample.yml
+```
+will output
+```yaml
+100
+```
+
+## First element with no filter from array
+Given a sample.yml file of:
+```yaml
+- 10
+- 100
+- 1
+```
+then
+```bash
+yq 'first' sample.yml
+```
+will output
+```yaml
+10
+```
+
+## First element with no filter from array of maps
+Given a sample.yml file of:
+```yaml
+- a: 10
+- a: 100
+```
+then
+```bash
+yq 'first' sample.yml
+```
+will output
+```yaml
+a: 10
+```
+

pkg/yqlib/doc/operators/headers/anchor-and-alias-operators.md

@@ -4,3 +4,15 @@ Use the `alias` and `anchor` operators to read and write yaml aliases and anchor
 
 `yq` supports merge aliases (like `<<: *blah`) however this is no longer in the standard yaml spec (1.2) and so `yq` will automatically add the `!!merge` tag to these nodes as it is effectively a custom tag.
 
+
+## NOTE --yaml-fix-merge-anchor-to-spec flag
+`yq` doesn't merge anchors `<<:` to spec, in some circumstances it incorrectly overrides existing keys when the spec documents not to do that.
+
+To minimise disruption while still fixing the issue, a flag has been added to toggle this behaviour. This will first default to false; and log warnings to users. Then it will default to true (and still allow users to specify false if needed).
+
+This flag also enables advanced merging, like inline maps, as well as fixes to ensure when exploding a particular path, neighbours are not affect ed.
+
+Long story short, you should be setting this flag to true.
+
+See examples of the flag differences below, where LEGACY is with the flag off; and FIXED is with the flag on.
+

pkg/yqlib/doc/operators/headers/entries.md

@@ -1,3 +1,5 @@
 # Entries
 
 Similar to the same named functions in `jq` these functions convert to/from an object and an array of key-value pairs. This is most useful for performing operations on keys of maps.
+
+Use `with_entries(op)` as a syntactic sugar for doing `to_entries | op | from_entries`.

pkg/yqlib/doc/operators/headers/env-variable-operators.md

@@ -29,3 +29,6 @@ as follows:
 yq '(.. | select(tag == "!!str")) |= envsubst' file.yaml
 ```
 
+## Disabling env operators
+If required, you can use the `--security-disable-env-ops` to disable env operations.
+

pkg/yqlib/doc/operators/headers/first.md

@@ -0,0 +1,5 @@
+# First
+
+Returns the first matching element in an array, or first matching value in a map.
+
+Can be given an expression to match with, otherwise will just return the first.

pkg/yqlib/doc/operators/headers/kind.md

@@ -0,0 +1,7 @@
+# Kind
+
+The `kind` operator identifies the type of a node as either `scalar`, `map`, or `seq`.
+
+This can be used for filtering or transforming nodes based on their type.
+
+Note that `null` values are treated as `scalar`.

pkg/yqlib/doc/operators/headers/load.md

@@ -46,3 +46,7 @@ this.is = a properties file
 ```
 bXkgc2VjcmV0IGNoaWxsaSByZWNpcGUgaXMuLi4u
 ```
+
+## Disabling file operators
+If required, you can use the `--security-disable-file-ops` to disable file operations.
+

pkg/yqlib/doc/operators/headers/max.md

@@ -0,0 +1,3 @@
+# Max
+
+Computes the maximum among an incoming sequence of scalar values.

pkg/yqlib/doc/operators/headers/min.md

@@ -0,0 +1,3 @@
+# Min
+
+Computes the minimum among an incoming sequence of scalar values.

pkg/yqlib/doc/operators/headers/omit.md

@@ -0,0 +1,3 @@
+# Omit
+
+Works like `pick`, but instead you specify the keys/indices that you _don't_ want included.

pkg/yqlib/doc/operators/headers/pivot.md

@@ -0,0 +1,3 @@
+# Pivot
+
+Emulates the `PIVOT` function supported by several popular RDBMS systems.

pkg/yqlib/doc/operators/headers/slice-array.md

@@ -2,4 +2,4 @@
 
 The slice array operator takes an array as input and returns a subarray. Like the `jq` equivalent, `.[10:15]` will return an array of length 5, starting from index 10 inclusive, up to index 15 exclusive. Negative numbers count backwards from the end of the array.
 
-You may leave out the first or second number, which will will refer to the start or end of the array respectively.
+You may leave out the first or second number, which will refer to the start or end of the array respectively.

pkg/yqlib/doc/operators/headers/sort-keys.md

@@ -12,5 +12,5 @@ diff file1.yml file2.yml
 
 Note that `yq` does not yet consider anchors when sorting by keys - this may result in invalid yaml documents if you are using merge anchors.
 
-For more advanced sorting, using `to_entries` to convert the map to an array, then sort/process the array as you like (e.g. using `sort_by`) and convert back to a map using `from_entries`.
-See [here](https://mikefarah.gitbook.io/yq/operators/entries#custom-sort-map-keys) for an example. 
+For more advanced sorting, you can use the [sort_by](https://mikefarah.gitbook.io/yq/operators/sort) function on a map, and give it a custom function like `sort_by(key | downcase)`.
+

pkg/yqlib/doc/operators/headers/string-operators.md

@@ -3,7 +3,7 @@
 ## RegEx
 This uses Golang's native regex functions under the hood - See their [docs](https://github.com/google/re2/wiki/Syntax) for the supported syntax.
 
-Case insensitive tip: prefix the regex with `(?i)` - e.g. `test("(?i)cats)"`.
+Case insensitive tip: prefix the regex with `(?i)` - e.g. `test("(?i)cats")`.
 
 ### match(regEx)
 This operator returns the substring match details of the given regEx.

pkg/yqlib/doc/operators/headers/traverse-read.md

@@ -1,3 +1,12 @@
 # Traverse (Read)
 
 This is the simplest (and perhaps most used) operator. It is used to navigate deeply into yaml structures.
+
+
+## NOTE --yaml-fix-merge-anchor-to-spec flag
+`yq` doesn't merge anchors `<<:` to spec, in some circumstances it incorrectly overrides existing keys when the spec documents not to do that.
+
+To minimise disruption while still fixing the issue, a flag has been added to toggle this behaviour. This will first default to false; and log warnings to users. Then it will default to true (and still allow users to specify false if needed)
+
+See examples of the flag differences below, where LEGACY is the flag off; and FIXED is with the flag on.
+

pkg/yqlib/doc/operators/kind.md

@@ -1,3 +1,10 @@
+# Kind
+
+The `kind` operator identifies the type of a node as either `scalar`, `map`, or `seq`.
+
+This can be used for filtering or transforming nodes based on their type.
+
+Note that `null` values are treated as `scalar`.
 
 ## Get kind
 Given a sample.yml file of:

pkg/yqlib/doc/operators/load.md

@@ -47,6 +47,10 @@ this.is = a properties file
 bXkgc2VjcmV0IGNoaWxsaSByZWNpcGUgaXMuLi4u
 ```
 
+## Disabling file operators
+If required, you can use the `--security-disable-file-ops` to disable file operations.
+
+
 ## Simple example
 Given a sample.yml file of:
 ```yaml
@@ -194,3 +198,63 @@ cool: things
 more_stuff: my secret chilli recipe is....
 ```
 
+## load() operation fails when security is enabled
+Use `--security-disable-file-ops` to disable file operations for security.
+
+Running
+```bash
+yq --null-input 'load("../../examples/thing.yml")'
+```
+will output
+```bash
+Error: file operations have been disabled
+```
+
+## load_str() operation fails when security is enabled
+Use `--security-disable-file-ops` to disable file operations for security.
+
+Running
+```bash
+yq --null-input 'load_str("../../examples/thing.yml")'
+```
+will output
+```bash
+Error: file operations have been disabled
+```
+
+## load_xml() operation fails when security is enabled
+Use `--security-disable-file-ops` to disable file operations for security.
+
+Running
+```bash
+yq --null-input 'load_xml("../../examples/small.xml")'
+```
+will output
+```bash
+Error: file operations have been disabled
+```
+
+## load_props() operation fails when security is enabled
+Use `--security-disable-file-ops` to disable file operations for security.
+
+Running
+```bash
+yq --null-input 'load_props("../../examples/small.properties")'
+```
+will output
+```bash
+Error: file operations have been disabled
+```
+
+## load_base64() operation fails when security is enabled
+Use `--security-disable-file-ops` to disable file operations for security.
+
+Running
+```bash
+yq --null-input 'load_base64("../../examples/base64.txt")'
+```
+will output
+```bash
+Error: file operations have been disabled
+```
+

pkg/yqlib/doc/operators/max.md

@@ -0,0 +1,51 @@
+# Max
+
+Computes the maximum among an incoming sequence of scalar values.
+
+## Maximum int
+Given a sample.yml file of:
+```yaml
+- 99
+- 16
+- 12
+- 6
+- 66
+```
+then
+```bash
+yq 'max' sample.yml
+```
+will output
+```yaml
+99
+```
+
+## Maximum string
+Given a sample.yml file of:
+```yaml
+- foo
+- bar
+- baz
+```
+then
+```bash
+yq 'max' sample.yml
+```
+will output
+```yaml
+foo
+```
+
+## Maximum of empty
+Given a sample.yml file of:
+```yaml
+[]
+```
+then
+```bash
+yq 'max' sample.yml
+```
+will output
+```yaml
+```
+

pkg/yqlib/doc/operators/min.md

@@ -0,0 +1,51 @@
+# Min
+
+Computes the minimum among an incoming sequence of scalar values.
+
+## Minimum int
+Given a sample.yml file of:
+```yaml
+- 99
+- 16
+- 12
+- 6
+- 66
+```
+then
+```bash
+yq 'min' sample.yml
+```
+will output
+```yaml
+6
+```
+
+## Minimum string
+Given a sample.yml file of:
+```yaml
+- foo
+- bar
+- baz
+```
+then
+```bash
+yq 'min' sample.yml
+```
+will output
+```yaml
+bar
+```
+
+## Minimum of empty
+Given a sample.yml file of:
+```yaml
+[]
+```
+then
+```bash
+yq 'min' sample.yml
+```
+will output
+```yaml
+```
+

pkg/yqlib/doc/operators/multiply-merge.md

@@ -55,6 +55,62 @@ a: 12
 b: 4
 ```
 
+## Multiply string node X int
+Given a sample.yml file of:
+```yaml
+b: banana
+```
+then
+```bash
+yq '.b * 4' sample.yml
+```
+will output
+```yaml
+bananabananabananabanana
+```
+
+## Multiply int X string node
+Given a sample.yml file of:
+```yaml
+b: banana
+```
+then
+```bash
+yq '4 * .b' sample.yml
+```
+will output
+```yaml
+bananabananabananabanana
+```
+
+## Multiply string X int node
+Given a sample.yml file of:
+```yaml
+n: 4
+```
+then
+```bash
+yq '"banana" * .n' sample.yml
+```
+will output
+```yaml
+bananabananabananabanana
+```
+
+## Multiply int node X string
+Given a sample.yml file of:
+```yaml
+n: 4
+```
+then
+```bash
+yq '.n * "banana"' sample.yml
+```
+will output
+```yaml
+bananabananabananabanana
+```
+
 ## Merge objects together, returning merged result only
 Given a sample.yml file of:
 ```yaml

pkg/yqlib/doc/operators/omit.md

@@ -0,0 +1,44 @@
+# Omit
+
+Works like `pick`, but instead you specify the keys/indices that you _don't_ want included.
+
+## Omit keys from map
+Note that non existent keys are skipped.
+
+Given a sample.yml file of:
+```yaml
+myMap:
+  cat: meow
+  dog: bark
+  thing: hamster
+  hamster: squeak
+```
+then
+```bash
+yq '.myMap |= omit(["hamster", "cat", "goat"])' sample.yml
+```
+will output
+```yaml
+myMap:
+  dog: bark
+  thing: hamster
+```
+
+## Omit indices from array
+Note that non existent indices are skipped.
+
+Given a sample.yml file of:
+```yaml
+- cat
+- leopard
+- lion
+```
+then
+```bash
+yq 'omit([2, 0, 734, -5])' sample.yml
+```
+will output
+```yaml
+- leopard
+```
+

pkg/yqlib/doc/operators/parent.md

@@ -37,6 +37,48 @@ fruit: banana
 name: sam
 ```
 
+## Get parent attribute
+Given a sample.yml file of:
+```yaml
+a:
+  fruit: apple
+  name: bob
+b:
+  fruit: banana
+  name: sam
+```
+then
+```bash
+yq '.. | select(. == "banana") | parent.name' sample.yml
+```
+will output
+```yaml
+sam
+```
+
+## Get parents
+Match all parents
+
+Given a sample.yml file of:
+```yaml
+a:
+  b:
+    c: cat
+```
+then
+```bash
+yq '.a.b.c | parents' sample.yml
+```
+will output
+```yaml
+- c: cat
+- b:
+    c: cat
+- a:
+    b:
+      c: cat
+```
+
 ## N-th parent
 You can optionally supply the number of levels to go up for the parent, the default being 1.