wip

.github/ISSUE_TEMPLATE/bug_report_v3.md

@@ -0,0 +1,51 @@
+---
+name: Bug report - V3
+about: Create a report to help us improve
+title: ''
+labels: bug, v3
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+Note that any how to questions should be posted in the discussion board and not raised as an issue.
+
+Version of yq: 3.X.X
+Operating system: mac/linux/windows/....
+Installed via: docker/binary release/homebrew/snap/...
+
+**Input Yaml**
+Concise yaml document(s) (as simple as possible to show the bug, please keep it to 10 lines or less)
+data1.yml:
+```yaml
+this: should really work
+```
+
+data2.yml:
+```yaml
+but: it strangely didn't
+```
+
+**Command**
+The command you ran:
+```
+yq merge data1.yml data2.yml
+```
+
+**Actual behavior**
+
+```yaml
+cat: meow
+```
+
+**Expected behavior**
+
+```yaml
+this: should really work
+but: it strangely didn't
+```
+
+**Additional context**
+Add any other context about the problem here.

.github/workflows/codeql.yml

@@ -38,11 +38,11 @@ jobs:
 
     steps:
     - name: Checkout repository
-      uses: actions/checkout@v5
+      uses: actions/checkout@v4
 
     # Initializes the CodeQL tools for scanning.
     - name: Initialize CodeQL
-      uses: github/codeql-action/init@v4
+      uses: github/codeql-action/init@v3
       with:
         languages: ${{ matrix.language }}
         # If you wish to specify custom queries, you can do so here or in a config file.
@@ -53,7 +53,7 @@ jobs:
     # Autobuild attempts to build any compiled languages  (C/C++, C#, or Java).
     # If this step fails, then you should remove it and run the build manually (see below)
     - name: Autobuild
-      uses: github/codeql-action/autobuild@v4
+      uses: github/codeql-action/autobuild@v3
 
     # ℹ️ Command-line programs to run using the OS shell.
     # 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun
@@ -67,4 +67,4 @@ jobs:
     #   make release
 
     - name: Perform CodeQL Analysis
-      uses: github/codeql-action/analyze@v4
+      uses: github/codeql-action/analyze@v3

.github/workflows/docker-release.yml

@@ -14,7 +14,7 @@ jobs:
       IMAGE_NAME: mikefarah/yq
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v5
+      - uses: actions/checkout@v4
 
       - name: Set up QEMU
         uses: docker/setup-qemu-action@v3
@@ -30,19 +30,6 @@ jobs:
       - name: Available platforms
         run: echo ${{ steps.buildx.outputs.platforms }} && docker version
 
-      - name: Login to Docker Hub
-        uses: docker/login-action@v3
-        with:
-          username: ${{ secrets.DOCKER_USERNAME }}
-          password: ${{ secrets.DOCKER_PASSWORD }}
-
-      - name: Login to GitHub Container Registry
-        uses: docker/login-action@v3
-        with:
-          registry: ghcr.io
-          username: ${{ github.actor }}
-          password: ${{ secrets.GITHUB_TOKEN }}
-
       - name: Build and push image
         run: |
           echo "GithubRef: ${GITHUB_REF}"
@@ -51,9 +38,10 @@ jobs:
           IMAGE_VERSION=${VERSION:1}
           echo "IMAGE_VERSION: ${IMAGE_VERSION}"
 
-          PLATFORMS="linux/amd64,linux/ppc64le,linux/arm64,linux/arm/v7,linux/s390x"
+          PLATFORMS="linux/amd64,linux/ppc64le,linux/arm64"
 
           echo "Building and pushing version ${IMAGE_VERSION} of image ${IMAGE_NAME}"
+          echo '${{ secrets.DOCKER_PASSWORD }}' | docker login -u '${{ secrets.DOCKER_USERNAME }}' --password-stdin
           docker buildx build \
             --label "org.opencontainers.image.authors=https://github.com/mikefarah/yq/graphs/contributors" \
             --label "org.opencontainers.image.created=$(date --rfc-3339=seconds)" \
@@ -71,11 +59,8 @@ jobs:
             -t "${IMAGE_NAME}:${IMAGE_VERSION}" \
             -t "${IMAGE_NAME}:4" \
             -t "${IMAGE_NAME}:latest" \
-            -t "ghcr.io/${IMAGE_NAME}:${IMAGE_VERSION}" \
-            -t "ghcr.io/${IMAGE_NAME}:4" \
-            -t "ghcr.io/${IMAGE_NAME}:latest" \
             .
-
+          
           cd github-action
           docker buildx build \
             --label "org.opencontainers.image.authors=https://github.com/mikefarah/yq/graphs/contributors" \
@@ -94,7 +79,4 @@ jobs:
             -t "${IMAGE_NAME}:${IMAGE_VERSION}-githubaction" \
             -t "${IMAGE_NAME}:4-githubaction" \
             -t "${IMAGE_NAME}:latest-githubaction" \
-            -t "ghcr.io/${IMAGE_NAME}:${IMAGE_VERSION}-githubaction" \
-            -t "ghcr.io/${IMAGE_NAME}:4-githubaction" \
-            -t "ghcr.io/${IMAGE_NAME}:latest-githubaction" \
             .

.github/workflows/go.yml

@@ -17,7 +17,7 @@ jobs:
       id: go
 
     - name: Check out code into the Go module directory
-      uses: actions/checkout@v5
+      uses: actions/checkout@v4
 
     - name: Get dependencies
       run: |

.github/workflows/release.yml

@@ -9,7 +9,7 @@ jobs:
   publishGitRelease:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v5
+      - uses: actions/checkout@v4
       - uses: actions/setup-go@v5
         with:
           go-version: '^1.20'
@@ -40,7 +40,9 @@ jobs:
       - name: Cross compile
         run: |
           sudo apt-get install rhash -y
-          go install github.com/goreleaser/goreleaser/v2@latest
+          go install github.com/mitchellh/gox@v1.0.1
+          mkdir -p build
+          cp yq.1 build/yq.1
           ./scripts/xcompile.sh
 
       - name: Release

.github/workflows/snap-release.yml

@@ -12,7 +12,7 @@ jobs:
       environment: snap
       runs-on: ubuntu-latest
       steps:
-        - uses: actions/checkout@v5
+        - uses: actions/checkout@v4
         - uses: snapcore/action-build@v1
           id: build
         - uses: snapcore/action-publish@v1

.github/workflows/test-yq.yml

@@ -13,7 +13,7 @@ jobs:
     runs-on: ubuntu-latest
 
     steps:
-      - uses: actions/checkout@v5
+      - uses: actions/checkout@v4
       - name: Get test
         id: get_value
         uses: mikefarah/yq@master

.gitignore

@@ -22,10 +22,8 @@ _cgo_gotypes.go
 _cgo_export.*
 
 _testmain.go
-cover.out
 coverage.out
 coverage.html
-coverage_sorted.txt
 *.exe
 *.test
 *.prof
@@ -46,8 +44,6 @@ test*.yml
 test*.xml
 test*.toml
 test*.yaml
-test_dir1/
-test_dir2/
 0.yml
 1.yml
 2.yml

.golangci.bck.yml

@@ -1,38 +0,0 @@
-run:
-  timeout: 5m
-linters:
-  enable:
-    - asciicheck
-    - depguard
-    - errorlint
-    - gci
-    - gochecknoinits
-    - gofmt
-    - goimports
-    - gosec
-    - gosimple
-    - staticcheck
-    - unused
-    - misspell
-    - nakedret
-    - nolintlint
-    - predeclared
-    - revive
-    - unconvert
-    - unparam
-linters-settings:
-  depguard:
-    rules:
-      prevent_unmaintained_packages:
-        list-mode: lax
-        files:
-          - $all
-          - "!$test"
-        deny:
-          - pkg: io/ioutil
-            desc: "replaced by io and os packages since Go 1.16: https://tip.golang.org/doc/go1.16#ioutil"
-issues:
-  exclude-rules:
-  - linters:
-    - revive
-    text: "var-naming"

.golangci.yml

@@ -1,11 +1,16 @@
-version: "2"
+run:
+  timeout: 5m
 linters:
   enable:
     - asciicheck
     - depguard
     - errorlint
+    - gci
     - gochecknoinits
+    - gofmt
+    - goimports
     - gosec
+    - megacheck
     - misspell
     - nakedret
     - nolintlint
@@ -13,40 +18,26 @@ linters:
     - revive
     - unconvert
     - unparam
-  settings:
-    depguard:
-      rules:
-        prevent_unmaintained_packages:
-          list-mode: lax
-          files:
-            - $all
-            - '!$test'
-          deny:
-            - pkg: io/ioutil
-              desc: 'replaced by io and os packages since Go 1.16: https://tip.golang.org/doc/go1.16#ioutil'
-  exclusions:
-    generated: lax
-    presets:
-      - comments
-      - common-false-positives
-      - legacy
-      - std-error-handling
+linters-settings:
+  depguard:
     rules:
-      - linters:
-          - revive
-        text: var-naming
-    paths:
-      - third_party$
-      - builtin$
-      - examples$
-formatters:
-  enable:
-    - gci
-    - gofmt
-    - goimports
-  exclusions:
-    generated: lax
-    paths:
-      - third_party$
-      - builtin$
-      - examples$
+      prevent_unmaintained_packages:
+        list-mode: lax
+        files:
+          - $all
+          - "!$test"
+        deny:
+          - pkg: io/ioutil
+            desc: "replaced by io and os packages since Go 1.16: https://tip.golang.org/doc/go1.16#ioutil"
+issues:
+  exclude-rules:
+  - linters:
+    - gosec
+    text:  "Implicit memory aliasing in for loop."
+    path: _test\.go
+  - linters:
+    - revive
+    text: "unexported-return"
+  - linters:
+    - revive
+    text: "var-naming"

.goreleaser.yaml

@@ -1,49 +0,0 @@
-version: 2
-
-dist: build
-
-builds:
-  - id: yq
-
-    binary: yq_{{ .Os }}_{{ .Arch }}
-
-    ldflags:
-      - -s -w
-
-    env:
-      - CGO_ENABLED=0
-
-    targets:
-      - darwin_amd64
-      - darwin_arm64
-      - freebsd_386
-      - freebsd_amd64
-      - freebsd_arm
-      - linux_386
-      - linux_amd64
-      - linux_arm
-      - linux_arm64
-      - linux_loong64
-      - linux_mips
-      - linux_mips64
-      - linux_mips64le
-      - linux_mipsle
-      - linux_ppc64
-      - linux_ppc64le
-      - linux_riscv64
-      - linux_s390x
-      - netbsd_386
-      - netbsd_amd64
-      - netbsd_arm
-      - openbsd_386
-      - openbsd_amd64
-      - windows_386
-      - windows_amd64
-      - windows_arm
-      - windows_arm64
-
-    no_unique_dist_dir: true
-
-release:
-  disable: true
-  skip_upload: true

CONTRIBUTING.md

@@ -1,214 +1,40 @@
-# Before you begin 
-Not all new PRs will be merged in 
-
-It's recommended to check with the owner first (e.g. raise an issue) to discuss a new feature before developing, to ensure your hard efforts don't go to waste.
-
-PRs to fix bugs and issues are almost always welcome :pray: please ensure you write tests as well.
-
-The following types of PRs will _not_ be accepted:
-- **Significant refactors** take a lot of time to understand and can have all sorts of unintended side effects. If you think there's a better way to do things (that requires significant changes) raise an issue for discussion first :)
-- **Release pipeline PRs** are a security risk - it's too easy for a serious vulnerability to sneak in (either intended or not). If there is a new cool way of releasing things, raise an issue for discussion first - it will need to be gone over with a fine tooth comb.
-- **Version bumps** are handled by dependabot, the bot will auto-raise PRs and they will be regularly merged in. 
-- **New release platforms** At this stage, yq is not going to maintain any other release platforms other than GitHub and Docker - that said, I'm more than happy to put in other community maintained methods in the README for visibility :heart:
-
-
 # Development
 
-## Initial Setup
+1. Install (Golang)[https://golang.org/]
+1. Run  `scripts/devtools.sh` to install the required devtools
+2. Run `make [local] vendor` to install the vendor dependencies
+2. Run `make [local] test` to ensure you can run the existing tests
+3. Write unit tests - (see existing examples). Changes will not be accepted without corresponding unit tests.
+4. Make the code changes.
+5. `make [local] test` to lint code and run tests
+6. Profit! ok no profit, but raise a PR and get kudos :)
 
-1. Install [Golang](https://golang.org/) (version 1.24.0 or later)
-2. Run `scripts/devtools.sh` to install required development tools:
-   - golangci-lint for code linting
-   - gosec for security analysis
-3. Run `make [local] vendor` to install vendor dependencies
-4. Run `make [local] test` to ensure you can run the existing tests
-
-## Development Workflow
-
-1. **Write unit tests first** - Changes will not be accepted without corresponding unit tests (see Testing section below)
-2. **Make your code changes**
-3. **Run tests and linting**: `make [local] test` (this runs formatting, linting, security checks, and tests)
-4. **Create your PR** and get kudos! :)
-
-## Make Commands
-
-- Use `make [local] <command>` for local development (runs in Docker container)
-- Use `make <command>` for CI/CD environments
-- Common commands:
-  - `make [local] vendor` - Install dependencies
-  - `make [local] test` - Run all checks and tests
-  - `make [local] build` - Build the yq binary
-  - `make [local] format` - Format code
-  - `make [local] check` - Run linting and security checks
-
-# Code Quality
-
-## Linting and Formatting
-
-The project uses strict linting rules defined in `.golangci.yml`. All code must pass:
-
-- **Code formatting**: gofmt, goimports, gci
-- **Linting**: revive, errorlint, gosec, misspell, and others
-- **Security checks**: gosec security analysis
-- **Spelling checks**: misspell detection
-
-Run `make [local] check` to verify your code meets all quality standards.
-
-## Code Style Guidelines
-
-- Follow standard Go conventions
-- Use meaningful variable names
-- Add comments for public functions and complex logic
-- Keep functions focused and reasonably sized
-- Use the project's existing patterns and conventions
-
-# Testing
-
-## Test Structure
-
-Tests in yq use the `expressionScenario` pattern. Each test scenario includes:
-- `expression`: The yq expression to test
-- `document`: Input YAML/JSON (optional)
-- `expected`: Expected output
-- `skipDoc`: Whether to skip documentation generation
-
-## Writing Tests
-
-1. **Find the appropriate test file** (e.g., `operator_add_test.go` for addition operations)
-2. **Add your test scenario** to the `*OperatorScenarios` slice
-3. **Run the specific test**: `go test -run TestAddOperatorScenarios` (replace with appropriate test name)
-4. **Verify documentation generation** (see Documentation section)
-
-## Test Examples
-
-```go
-var addOperatorScenarios = []expressionScenario{
-    {
-        skipDoc:    true,
-        expression: `"foo" + "bar"`,
-        expected: []string{
-            "D0, P[], (!!str)::foobar\n",
-        },
-    },
-    {
-        document:   "apples: 3",
-        expression: `.apples + 3`,
-        expected: []string{
-            "D0, P[apples], (!!int)::6\n",
-        },
-    },
-}
-```
-
-## Running Tests
-
-- **All tests**: `make [local] test`
-- **Specific test**: `go test -run TestName`
-- **With coverage**: `make [local] cover`
 
 # Documentation
 
-## Documentation Generation
+The documentation is a bit of a mixed bag (sorry in advance, I do plan on simplifying it...) - with some parts automatically generated and stiched together and some statically defined.
 
-The project uses a documentation system that combines static headers with dynamically generated content from tests.
+Documentation is written in markdown, and is published in the 'gitbook' branch.
 
-### How It Works
+The various operator documentation (e.g. 'strings') are generated from the 'master' branch, and have a statically defined header (e.g. `pkg/yqlib/doc/operators/headers/add.md`) and the bulk of the docs are generated from the unit tests e.g. `pkg/yqlib/operator_add_test.go`.
 
-1. **Static headers** are defined in `pkg/yqlib/doc/operators/headers/*.md`
-2. **Dynamic content** is generated from test scenarios in `*_test.go` files
-3. **Generated docs** are created in `pkg/yqlib/doc/*.md` by concatenating headers with test-generated content
-4. **Documentation is synced** to the gitbook branch for the website
+The pipeline will run the tests and automatically concatenate the files together, and put them under 
+`pkg/qylib/doc/add.md`. These files are checked in the master branch (and are copied to the gitbook branch as part of the release process).
 
-### Updating Operator Documentation
+## How to contribute
 
-#### For Test-Generated Documentation
+The first step is to find if what you want is automatically generated or not - start by looking in the master branch. 
 
-Most operator documentation is generated from tests. To update:
+### Updating dynamic documentation from master
+- Search for the documentation you want to update. If you find matches in a `*_test.go` file - update that, as that will automatically update the matching `*.md` file 
+- Assuming you are updating a `*_test.go` file, once updated, run the test to regenerated the docs. E.g. for the 'Add' test generated docs, from the pkg/yqlib folder run:
+`go test -run TestAddOperatorScenarios` which will run that test defined in the `operator_add_test.go` file.
+- Ensure the tests still pass, and check the generated documentation have your update.
+- Note: If the documentation is only in a `headers/*.md` file, then just update that directly
+- Raise a PR to merge the changes into master!
 
-1. **Find the test file** (e.g., `operator_add_test.go`)
-2. **Update test scenarios** - each `expressionScenario` with `skipDoc: false` becomes documentation
-3. **Run the test** to regenerate docs:
-   ```bash
-   cd pkg/yqlib
-   go test -run TestAddOperatorScenarios
-   ```
-4. **Verify the generated documentation** in `pkg/yqlib/doc/add.md`
-5. **Create a PR** with your changes
+### Updating static documentation from the gitbook branch
+If you haven't found what you want to update in the master branch, then check the gitbook branch directly as there are a few pages in there that are not in master.
 
-#### For Header-Only Documentation
-
-If documentation exists only in `headers/*.md` files:
-1. **Update the header file directly** (e.g., `pkg/yqlib/doc/operators/headers/add.md`)
-2. **Create a PR** with your changes
-
-### Updating Static Documentation
-
-For documentation not in the master branch:
-
-1. **Check the gitbook branch** for additional pages
-2. **Update the `*.md` files** directly
-3. **Create a PR** to the gitbook branch
-
-### Documentation Best Practices
-
-- **Write clear, concise examples** in test scenarios
-- **Use meaningful variable names** in examples
-- **Include edge cases** and error conditions
-- **Test your documentation changes** by running the specific test
-- **Verify generated output** matches expectations
-
-Note: PRs with small changes (e.g. minor typos) may not be merged (see https://joel.net/how-one-guy-ruined-hacktoberfest2020-drama).
-
-# Troubleshooting
-
-## Common Setup Issues
-
-### Docker/Podman Issues
-- **Problem**: `make` commands fail with Docker errors
-- **Solution**: Ensure Docker or Podman is running and accessible
-- **Alternative**: Use `make local <command>` to run in containers
-
-### Go Version Issues
-- **Problem**: Build fails with Go version errors
-- **Solution**: Ensure you have Go 1.24.0 or later installed
-- **Check**: Run `go version` to verify
-
-### Vendor Dependencies
-- **Problem**: `make vendor` fails or dependencies are outdated
-- **Solution**: 
-  ```bash
-  go mod tidy
-  make [local] vendor
-  ```
-
-### Linting Failures
-- **Problem**: `make check` fails with linting errors
-- **Solution**: 
-  ```bash
-  make [local] format  # Auto-fix formatting
-  # Manually fix remaining linting issues
-  make [local] check   # Verify fixes
-  ```
-
-### Test Failures
-- **Problem**: Tests fail locally but pass in CI
-- **Solution**: 
-  ```bash
-  make [local] test    # Run in Docker container
-  ```
-
-### Documentation Generation Issues
-- **Problem**: Generated docs don't update after test changes
-- **Solution**: 
-  ```bash
-  cd pkg/yqlib
-  go test -run TestSpecificOperatorScenarios
-  # Check if generated file updated in pkg/yqlib/doc/
-  ```
-
-## Getting Help
-
-- **Check existing issues**: Search GitHub issues for similar problems
-- **Create an issue**: If you can't find a solution, create a detailed issue
-- **Ask questions**: Use GitHub Discussions for general questions
-- **Join the community**: Check the project's community channels
+- Update the `*.md` files
+- Raise a PR to merge the changes into gitbook.

Dockerfile

@@ -1,4 +1,4 @@
-FROM golang:1.25.4 AS builder
+FROM golang:1.22.1 as builder
 
 WORKDIR /go/src/mikefarah/yq
 
@@ -10,7 +10,7 @@ RUN ./scripts/acceptance.sh
 
 # Choose alpine as a base image to make this useful for CI, as many
 # CI tools expect an interactive shell inside the container
-FROM alpine:3 AS production
+FROM alpine:3 as production
 LABEL maintainer="Mike Farah <mikefarah@users.noreply.github.com>"
 
 COPY --from=builder /go/src/mikefarah/yq/yq /usr/bin/yq

Dockerfile.dev

@@ -1,15 +1,12 @@
-FROM golang:1.25.4
-
-RUN apt-get update && \
-    apt-get install -y npm && \
-    npm install -g npx cspell@latest
+FROM golang:1.22.1
 
 COPY scripts/devtools.sh /opt/devtools.sh
 
-RUN set -e -x && \
-    /opt/devtools.sh
-
+RUN set -e -x \
+    && /opt/devtools.sh
 ENV PATH=/go/bin:$PATH
 
+RUN apt-get update && apt-get install -y npm && npm install -g npx cspell@latest
+
 ENV CGO_ENABLED 0
 ENV GOPATH /go:/yq

Makefile

@@ -2,7 +2,7 @@ MAKEFLAGS += --warn-undefined-variables
 SHELL := /bin/bash
 .SHELLFLAGS := -o pipefail -euc
 .DEFAULT_GOAL := install
-ENGINE := $(shell { (podman version > /dev/null 2>&1 && command -v podman) || command -v docker; } 2>/dev/null)
+ENGINE := $(shell { command -v podman || command -v docker; } 2>/dev/null)
 
 include Makefile.variables
 

Makefile.variables

@@ -26,7 +26,6 @@ ifeq ($(CYG_CHECK),1)
 else
 	# all non-windows environments
 	ROOT := $(shell pwd)
-	SELINUX := $(shell which getenforce 2>&1 >/dev/null && echo :z)
 endif
 
 DEV_IMAGE := ${PROJECT}_dev
@@ -34,7 +33,7 @@ DEV_IMAGE := ${PROJECT}_dev
 ENGINERUN := ${ENGINE} run --rm \
 	-e LDFLAGS="${LDFLAGS}" \
 	-e GITHUB_TOKEN="${GITHUB_TOKEN}" \
-	-v ${ROOT}/vendor:/go/src${SELINUX} \
-	-v ${ROOT}:/${PROJECT}/src/${IMPORT_PATH}${SELINUX} \
+	-v ${ROOT}/vendor:/go/src \
+	-v ${ROOT}:/${PROJECT}/src/${IMPORT_PATH} \
 	-w /${PROJECT}/src/${IMPORT_PATH} \
 	${DEV_IMAGE}

README.md

@@ -3,46 +3,44 @@
 ![Build](https://github.com/mikefarah/yq/workflows/Build/badge.svg)  ![Docker Pulls](https://img.shields.io/docker/pulls/mikefarah/yq.svg) ![Github Releases (by Release)](https://img.shields.io/github/downloads/mikefarah/yq/total.svg) ![Go Report](https://goreportcard.com/badge/github.com/mikefarah/yq) ![CodeQL](https://github.com/mikefarah/yq/workflows/CodeQL/badge.svg)
 
 
-A lightweight and portable command-line YAML, JSON, INI and XML processor. `yq` uses [jq](https://github.com/stedolan/jq) (a popular JSON processor) like syntax but works with yaml files as well as json, xml, ini, properties, csv and tsv. It doesn't yet support everything `jq` does - but it does support the most common operations and functions, and more is being added continuously.
+a lightweight and portable command-line YAML, JSON and XML processor. `yq` uses [jq](https://github.com/stedolan/jq) like syntax but works with yaml files as well as json, xml, properties, csv and tsv. It doesn't yet support everything `jq` does - but it does support the most common operations and functions, and more is being added continuously.
 
-yq is written in Go - so you can download a dependency free binary for your platform and you are good to go! If you prefer there are a variety of package managers that can be used as well as Docker and Podman, all listed below.
+yq is written in go - so you can download a dependency free binary for your platform and you are good to go! If you prefer there are a variety of package managers that can be used as well as Docker and Podman, all listed below.
 
 ## Quick Usage Guide
 
-### Basic Operations
-
-**Read a value:**
+Read a value:
 ```bash
 yq '.a.b[0].c' file.yaml
 ```
 
-**Pipe from STDIN:**
+Pipe from STDIN:
 ```bash
 yq '.a.b[0].c' < file.yaml
 ```
 
-**Update a yaml file in place:**
+Update a yaml file, in place
 ```bash
 yq -i '.a.b[0].c = "cool"' file.yaml
 ```
 
-**Update using environment variables:**
+Update using environment variables
 ```bash
 NAME=mike yq -i '.a.b[0].c = strenv(NAME)' file.yaml
 ```
 
-### Advanced Operations
-
-**Merge multiple files:**
+Merge multiple files
 ```bash
 # merge two files
 yq -n 'load("file1.yaml") * load("file2.yaml")'
 
-# merge using globs (note: `ea` evaluates all files at once instead of in sequence)
+# merge using globs:
+# note the use of `ea` to evaluate all the files at once
+# instead of in sequence
 yq ea '. as $item ireduce ({}; . * $item )' path/to/*.yml
 ```
 
-**Multiple updates to a yaml file:**
+Multiple updates to a yaml file
 ```bash
 yq -i '
   .a.b[0].c = "cool" |
@@ -51,22 +49,14 @@ yq -i '
 ' file.yaml
 ```
 
-**Find and update an item in an array:**
+Find and update an item in an array:
 ```bash
-# Note: requires input file - add your file at the end
-yq -i '(.[] | select(.name == "foo") | .address) = "12 cat st"' data.yaml
+yq '(.[] | select(.name == "foo") | .address) = "12 cat st"'
 ```
 
-**Convert between formats:**
+Convert JSON to YAML
 ```bash
-# Convert JSON to YAML (pretty print)
 yq -Poy sample.json
-
-# Convert YAML to JSON
-yq -o json file.yaml
-
-# Convert XML to YAML
-yq -o yaml file.xml
 ```
 
 See [recipes](https://mikefarah.gitbook.io/yq/recipes) for more examples and the [documentation](https://mikefarah.gitbook.io/yq/) for more information.
@@ -78,30 +68,30 @@ Take a look at the discussions for [common questions](https://github.com/mikefar
 ### [Download the latest binary](https://github.com/mikefarah/yq/releases/latest)
 
 ### wget
-Use wget to download pre-compiled binaries. Choose your platform and architecture:
+Use wget to download, gzipped pre-compiled binaries:
 
-**For Linux (example):**
+
+For instance, VERSION=v4.2.0 and BINARY=yq_linux_amd64
+
+#### Compressed via tar.gz
 ```bash
-# Set your platform variables (adjust as needed)
-VERSION=v4.2.0
-PLATFORM=linux_amd64
-
-# Download compressed binary
-wget https://github.com/mikefarah/yq/releases/download/${VERSION}/yq_${PLATFORM}.tar.gz -O - |\
-  tar xz && sudo mv yq_${PLATFORM} /usr/local/bin/yq
-
-# Or download plain binary
-wget https://github.com/mikefarah/yq/releases/download/${VERSION}/yq_${PLATFORM} -O /usr/local/bin/yq &&\
-    chmod +x /usr/local/bin/yq
+wget https://github.com/mikefarah/yq/releases/download/${VERSION}/${BINARY}.tar.gz -O - |\
+  tar xz && mv ${BINARY} /usr/bin/yq
 ```
 
-**Latest version (Linux AMD64):**
+#### Plain binary
+
 ```bash
-wget https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -O /usr/local/bin/yq &&\
-    chmod +x /usr/local/bin/yq
+wget https://github.com/mikefarah/yq/releases/download/${VERSION}/${BINARY} -O /usr/bin/yq &&\
+    chmod +x /usr/bin/yq
 ```
 
-**Available platforms:** `linux_amd64`, `linux_arm64`, `linux_arm`, `linux_386`, `darwin_amd64`, `darwin_arm64`, `windows_amd64`, `windows_386`, etc.
+#### Latest version
+
+```bash
+wget https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -O /usr/bin/yq &&\
+    chmod +x /usr/bin/yq
+```
 
 ### MacOS / Linux via Homebrew:
 Using [Homebrew](https://brew.sh/)
@@ -133,31 +123,28 @@ rm /etc/myfile.tmp
 ```
 
 ### Run with Docker or Podman
+#### Oneshot use:
 
-#### One-time use:
 ```bash
-# Docker - process files in current directory
-docker run --rm -v "${PWD}":/workdir mikefarah/yq '.a.b[0].c' file.yaml
-
-# Podman - same usage as Docker
-podman run --rm -v "${PWD}":/workdir mikefarah/yq '.a.b[0].c' file.yaml
+docker run --rm -v "${PWD}":/workdir mikefarah/yq [command] [flags] [expression ]FILE...
 ```
 
-**Security note:** You can run `yq` in Docker with restricted privileges:
+Note that you can run `yq` in docker without network access and other privileges if you desire,
+namely `--security-opt=no-new-privileges --cap-drop all --network none`.
+
 ```bash
-docker run --rm --security-opt=no-new-privileges --cap-drop all --network none \
-  -v "${PWD}":/workdir mikefarah/yq '.a.b[0].c' file.yaml
+podman run --rm -v "${PWD}":/workdir mikefarah/yq [command] [flags] [expression ]FILE...
 ```
 
-#### Pipe data via STDIN:
+#### Pipe in via STDIN:
 
-You'll need to pass the `-i --interactive` flag to Docker/Podman:
+You'll need to pass the `-i\--interactive` flag to docker:
 
 ```bash
-# Process piped data
 docker run -i --rm mikefarah/yq '.this.thing' < myfile.yml
+```
 
-# Same with Podman
+```bash
 podman run -i --rm mikefarah/yq '.this.thing' < myfile.yml
 ```
 
@@ -254,14 +241,6 @@ As these are supported by the community :heart: - however, they may be out of da
 _Please note that the Debian package (previously supported by @rmescandon) is no longer maintained. Please use an alternative installation method._
 
 
-### X-CMD
-Checkout `yq` on x-cmd: https://x-cmd.com/mod/yq
-
-- Instant Results: See the output of your yq filter in real-time.
-- Error Handling: Encounter a syntax error? It will display the error message and the results of the closest valid filter
-
-Thanks @edwinjhlee!
-
 ### Nix
 
 ```
@@ -307,7 +286,7 @@ Using [winget](https://learn.microsoft.com/en-us/windows/package-manager/)
 winget install --id MikeFarah.yq
 ```
 
-### MacPorts:
+### Mac:
 Using [MacPorts](https://www.macports.org/)
 ```
 sudo port selfupdate
@@ -316,44 +295,23 @@ sudo port install yq
 Supported by @herbygillot (https://ports.macports.org/maintainer/github/herbygillot)
 
 ### Alpine Linux
+- Enable edge/community repo by adding ```$MIRROR/alpine/edge/community``` to ```/etc/apk/repositories```
+- Update database index with ```apk update```
+- Install yq with ```apk add yq```
 
-Alpine Linux v3.20+ (and Edge):
-```
-apk add yq-go
-```
+Supported by Tuan Hoang
+https://pkgs.alpinelinux.org/package/edge/community/x86/yq
 
-Alpine Linux up to v3.19:
-```
-apk add yq
-```
-
-Supported by Tuan Hoang (https://pkgs.alpinelinux.org/packages?name=yq-go)
-
-### Flox:
-
-Flox can be used to install yq on Linux, MacOS, and Windows through WSL.
-
-```
-flox install yq
-```
-
-
-### MacOS / Linux via gah:
-Using [gah](https://github.com/marverix/gah)
-
-```
-gah install yq
-```
 
 ## Features
 - [Detailed documentation with many examples](https://mikefarah.gitbook.io/yq/)
 - Written in portable go, so you can download a lovely dependency free binary
-- Uses similar syntax as `jq` but works with YAML, INI, [JSON](https://mikefarah.gitbook.io/yq/usage/convert) and [XML](https://mikefarah.gitbook.io/yq/usage/xml) files
+- Uses similar syntax as `jq` but works with YAML, [JSON](https://mikefarah.gitbook.io/yq/usage/convert) and [XML](https://mikefarah.gitbook.io/yq/usage/xml) files
 - Fully supports multi document yaml files
 - Supports yaml [front matter](https://mikefarah.gitbook.io/yq/usage/front-matter) blocks (e.g. jekyll/assemble)
 - Colorized yaml output
 - [Date/Time manipulation and formatting with TZ](https://mikefarah.gitbook.io/yq/operators/datetime)
-- [Deep data structures](https://mikefarah.gitbook.io/yq/operators/traverse-read)
+- [Deeply data structures](https://mikefarah.gitbook.io/yq/operators/traverse-read)
 - [Sort keys](https://mikefarah.gitbook.io/yq/operators/sort-keys)
 - Manipulate yaml [comments](https://mikefarah.gitbook.io/yq/operators/comment-operators), [styling](https://mikefarah.gitbook.io/yq/operators/style), [tags](https://mikefarah.gitbook.io/yq/operators/tag) and [anchors and aliases](https://mikefarah.gitbook.io/yq/operators/anchor-and-alias-operators).
 - [Update in place](https://mikefarah.gitbook.io/yq/v/v4.x/commands/evaluate#flags)
@@ -387,76 +345,35 @@ yq -i '.stuff = "foo"' myfile.yml # update myfile.yml in place
 
 
 Available Commands:
-  completion  Generate the autocompletion script for the specified shell
-  eval        (default) Apply the expression to each document in each yaml file in sequence
-  eval-all    Loads _all_ yaml documents of _all_ yaml files and runs expression once
-  help        Help about any command
+  completion       Generate the autocompletion script for the specified shell
+  eval             (default) Apply the expression to each document in each yaml file in sequence
+  eval-all         Loads _all_ yaml documents of _all_ yaml files and runs expression once
+  help             Help about any command
+  shell-completion Generate completion script
 
 Flags:
   -C, --colors                        force print with colors
-      --csv-auto-parse                parse CSV YAML/JSON values (default true)
-      --csv-separator char            CSV Separator character (default ,)
   -e, --exit-status                   set exit status if there are no matches or null or false is returned
-      --expression string             forcibly set the expression argument. Useful when yq argument detection thinks your expression is a file.
-      --from-file string              Load expression from specified file.
   -f, --front-matter string           (extract|process) first input as yaml front-matter. Extract will pull out the yaml content, process will run the expression against the yaml content, leaving the remaining data intact
       --header-preprocess             Slurp any header comments and separators before processing expression. (default true)
   -h, --help                          help for yq
   -I, --indent int                    sets indent level for output (default 2)
   -i, --inplace                       update the file in place of first file given.
-  -p, --input-format string           [auto|a|yaml|y|json|j|props|p|csv|c|tsv|t|xml|x|base64|uri|toml|lua|l|ini|i] parse format for input. (default "auto")
-      --lua-globals                   output keys as top-level global variables
-      --lua-prefix string             prefix (default "return ")
-      --lua-suffix string             suffix (default ";\n")
-      --lua-unquoted                  output unquoted string keys (e.g. {foo="bar"})
+  -p, --input-format string           [yaml|y|xml|x] parse format for input. Note that json is a subset of yaml. (default "yaml")
   -M, --no-colors                     force print with no colors
   -N, --no-doc                        Don't print document separators (---)
-  -0, --nul-output                    Use NUL char to separate values. If unwrap scalar is also set, fail if unwrapped scalar contains NUL char.
   -n, --null-input                    Don't read input, simply evaluate the expression given. Useful for creating docs from scratch.
-  -o, --output-format string          [auto|a|yaml|y|json|j|props|p|csv|c|tsv|t|xml|x|base64|uri|toml|shell|s|lua|l|ini|i] output format type. (default "auto")
+  -o, --output-format string          [yaml|y|json|j|props|p|xml|x] output format type. (default "yaml")
   -P, --prettyPrint                   pretty print, shorthand for '... style = ""'
-      --properties-array-brackets     use [x] in array paths (e.g. for SpringBoot)
-      --properties-separator string   separator to use between keys and values (default " = ")
-  -s, --split-exp string              print each result (or doc) into a file named (exp). [exp] argument must return a string. You can use $index in the expression as the result counter. The necessary directories will be created.
-      --split-exp-file string         Use a file to specify the split-exp expression.
-      --string-interpolation          Toggles strings interpolation of \(exp) (default true)
-      --tsv-auto-parse                parse TSV YAML/JSON values (default true)
-  -r, --unwrapScalar                  unwrap scalar, print the value with no quotes, colors or comments. Defaults to true for yaml (default true)
+  -s, --split-exp string              print each result (or doc) into a file named (exp). [exp] argument must return a string. You can use $index in the expression as the result counter.
+      --unwrapScalar                  unwrap scalar, print the value with no quotes, colors or comments (default true)
   -v, --verbose                       verbose mode
   -V, --version                       Print version information and quit
-      --xml-attribute-prefix string   prefix for xml attributes (default "+@")
+      --xml-attribute-prefix string   prefix for xml attributes (default "+")
       --xml-content-name string       name for xml content (if no attribute name is present). (default "+content")
-      --xml-directive-name string     name for xml directives (e.g. <!DOCTYPE thing cat>) (default "+directive")
-      --xml-keep-namespace            enables keeping namespace after parsing attributes (default true)
-      --xml-proc-inst-prefix string   prefix for xml processing instructions (e.g. <?xml version="1"?>) (default "+p_")
-      --xml-raw-token                 enables using RawToken method instead Token. Commonly disables namespace translations. See https://pkg.go.dev/encoding/xml#Decoder.RawToken for details. (default true)
-      --xml-skip-directives           skip over directives (e.g. <!DOCTYPE thing cat>)
-      --xml-skip-proc-inst            skip over process instructions (e.g. <?xml version="1"?>)
-      --xml-strict-mode               enables strict parsing of XML. See https://pkg.go.dev/encoding/xml for more details.
 
 Use "yq [command] --help" for more information about a command.
 ```
-
-## Troubleshooting
-
-### Common Issues
-
-**PowerShell quoting issues:**
-```powershell
-# Use single quotes for expressions
-yq '.a.b[0].c' file.yaml
-
-# Or escape double quotes
-yq ".a.b[0].c = \"value\"" file.yaml
-```
-
-### Getting Help
-
-- **Check existing issues**: [GitHub Issues](https://github.com/mikefarah/yq/issues)
-- **Ask questions**: [GitHub Discussions](https://github.com/mikefarah/yq/discussions)
-- **Documentation**: [Complete documentation](https://mikefarah.gitbook.io/yq/)
-- **Examples**: [Recipes and examples](https://mikefarah.gitbook.io/yq/recipes)
-
 ## Known Issues / Missing Features
 - `yq` attempts to preserve comment positions and whitespace as much as possible, but it does not handle all scenarios (see https://github.com/go-yaml/yaml/tree/v3 for details)
 - Powershell has its own...[opinions on quoting yq](https://mikefarah.gitbook.io/yq/usage/tips-and-tricks#quotes-in-windows-powershell)

acceptance_tests/load-file.sh

@@ -3,7 +3,7 @@
 testLoadFileNotExist() {
   result=$(./yq e -n 'load("cat.yml")' 2>&1)
   assertEquals 1 $?
-  assertEquals "Error: failed to load cat.yml: open cat.yml: no such file or directory" "$result"
+  assertEquals "Error: Failed to load cat.yml: open cat.yml: no such file or directory" "$result"
 }
 
 testLoadFileExpNotExist() {
@@ -15,7 +15,7 @@ testLoadFileExpNotExist() {
 testStrLoadFileNotExist() {
   result=$(./yq e -n 'strload("cat.yml")' 2>&1)
   assertEquals 1 $?
-  assertEquals "Error: failed to load cat.yml: open cat.yml: no such file or directory" "$result"
+  assertEquals "Error: Failed to load cat.yml: open cat.yml: no such file or directory" "$result"
 }
 
 testStrLoadFileExpNotExist() {

acceptance_tests/split-printer.sh

@@ -2,7 +2,6 @@
 
 setUp() {
   rm test*.yml || true
-  rm -rf test_dir* || true
 }
 
 testBasicSplitWithName() {
@@ -205,23 +204,4 @@ EOM
   assertEquals "$expectedDoc3" "$doc3"
 }
 
-testSplitWithDirectories() {
-  cat >test.yml <<EOL
-f: test_dir1/test_file1
----
-f: test_dir2/dir22/test_file2
----
-f: test_file3
-EOL
-
-  ./yq e --no-doc -s ".f" test.yml
-
-  doc1=$(cat test_dir1/test_file1.yml)
-  assertEquals "f: test_dir1/test_file1" "$doc1"
-  doc2=$(cat test_dir2/dir22/test_file2.yml)
-  assertEquals "f: test_dir2/dir22/test_file2" "$doc2"
-  doc3=$(cat test_file3.yml)
-  assertEquals "f: test_file3" "$doc3"
-}
-
-source ./scripts/shunit2
+source ./scripts/shunit2

cmd/constant.go

@@ -2,8 +2,6 @@ package cmd
 
 var unwrapScalarFlag = newUnwrapFlag()
 
-var printNodeInfo = false
-
 var unwrapScalar = false
 
 var writeInplace = false
@@ -14,6 +12,9 @@ var outputFormat = ""
 var inputFormat = ""
 
 var exitStatus = false
+var forceColor = false
+var forceNoColor = false
+var colorsEnabled = false
 var indent = 2
 var noDocSeparators = false
 var nullInput = false
@@ -22,10 +23,6 @@ var verbose = false
 var version = false
 var prettyPrint = false
 
-var forceColor = false
-var forceNoColor = false
-var colorsEnabled = false
-
 // can be either "" (off), "extract" or "process"
 var frontMatter = ""
 

cmd/evaluate_all_command.go

@@ -12,7 +12,7 @@ func createEvaluateAllCommand() *cobra.Command {
 		Use:     "eval-all [expression] [yaml_file1]...",
 		Aliases: []string{"ea"},
 		Short:   "Loads _all_ yaml documents of _all_ yaml files and runs expression once",
-		ValidArgsFunction: func(_ *cobra.Command, args []string, _ string) ([]string, cobra.ShellCompDirective) {
+		ValidArgsFunction: func(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) {
 			if len(args) == 0 {
 				return nil, cobra.ShellCompDirectiveNoFileComp
 			}

cmd/evaluate_all_command_test.go

@@ -1,328 +0,0 @@
-package cmd
-
-import (
-	"bytes"
-	"os"
-	"path/filepath"
-	"strings"
-	"testing"
-)
-
-func TestCreateEvaluateAllCommand(t *testing.T) {
-	cmd := createEvaluateAllCommand()
-
-	if cmd == nil {
-		t.Fatal("createEvaluateAllCommand returned nil")
-	}
-
-	// Test basic command properties
-	if cmd.Use != "eval-all [expression] [yaml_file1]..." {
-		t.Errorf("Expected Use to be 'eval-all [expression] [yaml_file1]...', got %q", cmd.Use)
-	}
-
-	if cmd.Short == "" {
-		t.Error("Expected Short description to be non-empty")
-	}
-
-	if cmd.Long == "" {
-		t.Error("Expected Long description to be non-empty")
-	}
-
-	// Test aliases
-	expectedAliases := []string{"ea"}
-	if len(cmd.Aliases) != len(expectedAliases) {
-		t.Errorf("Expected %d aliases, got %d", len(expectedAliases), len(cmd.Aliases))
-	}
-
-	for i, expected := range expectedAliases {
-		if i >= len(cmd.Aliases) || cmd.Aliases[i] != expected {
-			t.Errorf("Expected alias %d to be %q, got %q", i, expected, cmd.Aliases[i])
-		}
-	}
-}
-
-func TestEvaluateAll_NoArgs(t *testing.T) {
-	// Create a temporary command
-	cmd := createEvaluateAllCommand()
-
-	// Set up command to capture output
-	var output bytes.Buffer
-	cmd.SetOut(&output)
-
-	// Test with no arguments and no null input
-	nullInput = false
-	defer func() { nullInput = false }()
-
-	err := evaluateAll(cmd, []string{})
-
-	// Should not error, but should print usage
-	if err != nil {
-		t.Errorf("evaluateAll with no args should not error, got: %v", err)
-	}
-
-	// Should have printed usage information
-	if output.Len() == 0 {
-		t.Error("Expected usage information to be printed")
-	}
-}
-
-func TestEvaluateAll_NullInput(t *testing.T) {
-	// Create a temporary command
-	cmd := createEvaluateAllCommand()
-
-	// Set up command to capture output
-	var output bytes.Buffer
-	cmd.SetOut(&output)
-
-	// Test with null input
-	nullInput = true
-	defer func() { nullInput = false }()
-
-	err := evaluateAll(cmd, []string{})
-
-	// Should not error when using null input
-	if err != nil {
-		t.Errorf("evaluateAll with null input should not error, got: %v", err)
-	}
-}
-
-func TestEvaluateAll_WithSingleFile(t *testing.T) {
-	// Create a temporary YAML file
-	tempDir := t.TempDir()
-	yamlFile := filepath.Join(tempDir, "test.yaml")
-	yamlContent := []byte("name: test\nage: 25\n")
-	err := os.WriteFile(yamlFile, yamlContent, 0600)
-	if err != nil {
-		t.Fatalf("Failed to create test YAML file: %v", err)
-	}
-
-	// Create a temporary command
-	cmd := createEvaluateAllCommand()
-
-	// Set up command to capture output
-	var output bytes.Buffer
-	cmd.SetOut(&output)
-
-	// Test with a single file
-	err = evaluateAll(cmd, []string{yamlFile})
-
-	// Should not error
-	if err != nil {
-		t.Errorf("evaluateAll with single file should not error, got: %v", err)
-	}
-
-	// Should have some output
-	if output.Len() == 0 {
-		t.Error("Expected output from evaluateAll with single file")
-	}
-}
-
-func TestEvaluateAll_WithMultipleFiles(t *testing.T) {
-	// Create temporary YAML files
-	tempDir := t.TempDir()
-
-	yamlFile1 := filepath.Join(tempDir, "test1.yaml")
-	yamlContent1 := []byte("name: test1\nage: 25\n")
-	err := os.WriteFile(yamlFile1, yamlContent1, 0600)
-	if err != nil {
-		t.Fatalf("Failed to create test YAML file 1: %v", err)
-	}
-
-	yamlFile2 := filepath.Join(tempDir, "test2.yaml")
-	yamlContent2 := []byte("name: test2\nage: 30\n")
-	err = os.WriteFile(yamlFile2, yamlContent2, 0600)
-	if err != nil {
-		t.Fatalf("Failed to create test YAML file 2: %v", err)
-	}
-
-	// Create a temporary command
-	cmd := createEvaluateAllCommand()
-
-	// Set up command to capture output
-	var output bytes.Buffer
-	cmd.SetOut(&output)
-
-	// Test with multiple files
-	err = evaluateAll(cmd, []string{yamlFile1, yamlFile2})
-
-	// Should not error
-	if err != nil {
-		t.Errorf("evaluateAll with multiple files should not error, got: %v", err)
-	}
-
-	// Should have output
-	if output.Len() == 0 {
-		t.Error("Expected output from evaluateAll with multiple files")
-	}
-}
-
-func TestEvaluateAll_WithExpression(t *testing.T) {
-	// Create a temporary YAML file
-	tempDir := t.TempDir()
-	yamlFile := filepath.Join(tempDir, "test.yaml")
-	yamlContent := []byte("name: test\nage: 25\n")
-	err := os.WriteFile(yamlFile, yamlContent, 0600)
-	if err != nil {
-		t.Fatalf("Failed to create test YAML file: %v", err)
-	}
-
-	// Create a temporary command
-	cmd := createEvaluateAllCommand()
-
-	// Set up command to capture output
-	var output bytes.Buffer
-	cmd.SetOut(&output)
-
-	// Test with expression
-	err = evaluateAll(cmd, []string{".name", yamlFile})
-
-	// Should not error
-	if err != nil {
-		t.Errorf("evaluateAll with expression should not error, got: %v", err)
-	}
-
-	// Should have output
-	if output.Len() == 0 {
-		t.Error("Expected output from evaluateAll with expression")
-	}
-}
-
-func TestEvaluateAll_WriteInPlace(t *testing.T) {
-	// Create a temporary YAML file
-	tempDir := t.TempDir()
-	yamlFile := filepath.Join(tempDir, "test.yaml")
-	yamlContent := []byte("name: test\nage: 25\n")
-	err := os.WriteFile(yamlFile, yamlContent, 0600)
-	if err != nil {
-		t.Fatalf("Failed to create test YAML file: %v", err)
-	}
-
-	// Create a temporary command
-	cmd := createEvaluateAllCommand()
-
-	// Set up command to capture output
-	var output bytes.Buffer
-	cmd.SetOut(&output)
-
-	// Enable write in place
-	originalWriteInplace := writeInplace
-	writeInplace = true
-	defer func() { writeInplace = originalWriteInplace }()
-
-	// Test with write in place
-	err = evaluateAll(cmd, []string{".name = \"updated\"", yamlFile})
-
-	// Should not error
-	if err != nil {
-		t.Errorf("evaluateAll with write in place should not error, got: %v", err)
-	}
-
-	// Verify the file was updated
-	updatedContent, err := os.ReadFile(yamlFile)
-	if err != nil {
-		t.Fatalf("Failed to read updated file: %v", err)
-	}
-
-	// Should contain the updated content
-	if !strings.Contains(string(updatedContent), "updated") {
-		t.Errorf("Expected file to contain 'updated', got: %s", string(updatedContent))
-	}
-}
-
-func TestEvaluateAll_ExitStatus(t *testing.T) {
-	// Create a temporary YAML file
-	tempDir := t.TempDir()
-	yamlFile := filepath.Join(tempDir, "test.yaml")
-	yamlContent := []byte("name: test\nage: 25\n")
-	err := os.WriteFile(yamlFile, yamlContent, 0600)
-	if err != nil {
-		t.Fatalf("Failed to create test YAML file: %v", err)
-	}
-
-	// Create a temporary command
-	cmd := createEvaluateAllCommand()
-
-	// Set up command to capture output
-	var output bytes.Buffer
-	cmd.SetOut(&output)
-
-	// Enable exit status
-	originalExitStatus := exitStatus
-	exitStatus = true
-	defer func() { exitStatus = originalExitStatus }()
-
-	// Test with expression that should find no matches
-	err = evaluateAll(cmd, []string{".nonexistent", yamlFile})
-
-	// Should error when no matches found and exit status is enabled
-	if err == nil {
-		t.Error("Expected error when no matches found and exit status is enabled")
-	}
-}
-
-func TestEvaluateAll_WithMultipleDocuments(t *testing.T) {
-	// Create a temporary YAML file with multiple documents
-	tempDir := t.TempDir()
-	yamlFile := filepath.Join(tempDir, "test.yaml")
-	yamlContent := []byte("---\nname: doc1\nage: 25\n---\nname: doc2\nage: 30\n")
-	err := os.WriteFile(yamlFile, yamlContent, 0600)
-	if err != nil {
-		t.Fatalf("Failed to create test YAML file: %v", err)
-	}
-
-	// Create a temporary command
-	cmd := createEvaluateAllCommand()
-
-	// Set up command to capture output
-	var output bytes.Buffer
-	cmd.SetOut(&output)
-
-	// Test with multiple documents
-	err = evaluateAll(cmd, []string{".", yamlFile})
-
-	// Should not error
-	if err != nil {
-		t.Errorf("evaluateAll with multiple documents should not error, got: %v", err)
-	}
-
-	// Should have output
-	if output.Len() == 0 {
-		t.Error("Expected output from evaluateAll with multiple documents")
-	}
-}
-
-func TestEvaluateAll_NulSepOutput(t *testing.T) {
-	// Create a temporary YAML file
-	tempDir := t.TempDir()
-	yamlFile := filepath.Join(tempDir, "test.yaml")
-	yamlContent := []byte("name: test\nage: 25\n")
-	err := os.WriteFile(yamlFile, yamlContent, 0600)
-	if err != nil {
-		t.Fatalf("Failed to create test YAML file: %v", err)
-	}
-
-	// Create a temporary command
-	cmd := createEvaluateAllCommand()
-
-	// Set up command to capture output
-	var output bytes.Buffer
-	cmd.SetOut(&output)
-
-	// Enable nul separator output
-	originalNulSepOutput := nulSepOutput
-	nulSepOutput = true
-	defer func() { nulSepOutput = originalNulSepOutput }()
-
-	// Test with nul separator output
-	err = evaluateAll(cmd, []string{".name", yamlFile})
-
-	// Should not error
-	if err != nil {
-		t.Errorf("evaluateAll with nul separator output should not error, got: %v", err)
-	}
-
-	// Should have output
-	if output.Len() == 0 {
-		t.Error("Expected output from evaluateAll with nul separator output")
-	}
-}

cmd/evaluate_sequence_command.go

@@ -13,7 +13,7 @@ func createEvaluateSequenceCommand() *cobra.Command {
 		Use:     "eval [expression] [yaml_file1]...",
 		Aliases: []string{"e"},
 		Short:   "(default) Apply the expression to each document in each yaml file in sequence",
-		ValidArgsFunction: func(_ *cobra.Command, args []string, _ string) ([]string, cobra.ShellCompDirective) {
+		ValidArgsFunction: func(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) {
 			if len(args) == 0 {
 				return nil, cobra.ShellCompDirectiveNoFileComp
 			}
@@ -105,11 +105,6 @@ func evaluateSequence(cmd *cobra.Command, args []string) (cmdError error) {
 	}
 
 	printer := yqlib.NewPrinter(encoder, printerWriter)
-
-	if printNodeInfo {
-		printer = yqlib.NewNodeInfoPrinter(printerWriter)
-	}
-
 	if nulSepOutput {
 		printer.SetNulSepOutput(true)
 	}

cmd/evaluate_sequence_command_test.go

@@ -1,276 +0,0 @@
-package cmd
-
-import (
-	"bytes"
-	"os"
-	"path/filepath"
-	"strings"
-	"testing"
-)
-
-func TestCreateEvaluateSequenceCommand(t *testing.T) {
-	cmd := createEvaluateSequenceCommand()
-
-	if cmd == nil {
-		t.Fatal("createEvaluateSequenceCommand returned nil")
-	}
-
-	// Test basic command properties
-	if cmd.Use != "eval [expression] [yaml_file1]..." {
-		t.Errorf("Expected Use to be 'eval [expression] [yaml_file1]...', got %q", cmd.Use)
-	}
-
-	if cmd.Short == "" {
-		t.Error("Expected Short description to be non-empty")
-	}
-
-	if cmd.Long == "" {
-		t.Error("Expected Long description to be non-empty")
-	}
-
-	// Test aliases
-	expectedAliases := []string{"e"}
-	if len(cmd.Aliases) != len(expectedAliases) {
-		t.Errorf("Expected %d aliases, got %d", len(expectedAliases), len(cmd.Aliases))
-	}
-
-	for i, expected := range expectedAliases {
-		if i >= len(cmd.Aliases) || cmd.Aliases[i] != expected {
-			t.Errorf("Expected alias %d to be %q, got %q", i, expected, cmd.Aliases[i])
-		}
-	}
-}
-
-func TestProcessExpression(t *testing.T) {
-	// Reset global variables
-	originalPrettyPrint := prettyPrint
-	defer func() { prettyPrint = originalPrettyPrint }()
-
-	tests := []struct {
-		name        string
-		prettyPrint bool
-		expression  string
-		expected    string
-	}{
-		{
-			name:        "empty expression without pretty print",
-			prettyPrint: false,
-			expression:  "",
-			expected:    "",
-		},
-		{
-			name:        "empty expression with pretty print",
-			prettyPrint: true,
-			expression:  "",
-			expected:    `(... | (select(tag != "!!str"), select(tag == "!!str") | select(test("(?i)^(y|yes|n|no|on|off)$") | not))  ) style=""`,
-		},
-		{
-			name:        "simple expression without pretty print",
-			prettyPrint: false,
-			expression:  ".a.b",
-			expected:    ".a.b",
-		},
-		{
-			name:        "simple expression with pretty print",
-			prettyPrint: true,
-			expression:  ".a.b",
-			expected:    `.a.b | (... | (select(tag != "!!str"), select(tag == "!!str") | select(test("(?i)^(y|yes|n|no|on|off)$") | not))  ) style=""`,
-		},
-		{
-			name:        "complex expression with pretty print",
-			prettyPrint: true,
-			expression:  ".items[] | select(.active == true)",
-			expected:    `.items[] | select(.active == true) | (... | (select(tag != "!!str"), select(tag == "!!str") | select(test("(?i)^(y|yes|n|no|on|off)$") | not))  ) style=""`,
-		},
-	}
-
-	for _, tt := range tests {
-		t.Run(tt.name, func(t *testing.T) {
-			prettyPrint = tt.prettyPrint
-			result := processExpression(tt.expression)
-			if result != tt.expected {
-				t.Errorf("processExpression(%q) = %q, want %q", tt.expression, result, tt.expected)
-			}
-		})
-	}
-}
-
-func TestEvaluateSequence_NoArgs(t *testing.T) {
-	// Create a temporary command
-	cmd := createEvaluateSequenceCommand()
-
-	// Set up command to capture output
-	var output bytes.Buffer
-	cmd.SetOut(&output)
-
-	// Test with no arguments and no null input
-	nullInput = false
-	defer func() { nullInput = false }()
-
-	err := evaluateSequence(cmd, []string{})
-
-	// Should not error, but should print usage
-	if err != nil {
-		t.Errorf("evaluateSequence with no args should not error, got: %v", err)
-	}
-
-	// Should have printed usage information
-	if output.Len() == 0 {
-		t.Error("Expected usage information to be printed")
-	}
-}
-
-func TestEvaluateSequence_NullInput(t *testing.T) {
-	// Create a temporary command
-	cmd := createEvaluateSequenceCommand()
-
-	// Set up command to capture output
-	var output bytes.Buffer
-	cmd.SetOut(&output)
-
-	// Test with null input
-	nullInput = true
-	defer func() { nullInput = false }()
-
-	err := evaluateSequence(cmd, []string{})
-
-	// Should not error when using null input
-	if err != nil {
-		t.Errorf("evaluateSequence with null input should not error, got: %v", err)
-	}
-}
-
-func TestEvaluateSequence_WithFile(t *testing.T) {
-	// Create a temporary YAML file
-	tempDir := t.TempDir()
-	yamlFile := filepath.Join(tempDir, "test.yaml")
-	yamlContent := []byte("name: test\nage: 25\n")
-	err := os.WriteFile(yamlFile, yamlContent, 0600)
-	if err != nil {
-		t.Fatalf("Failed to create test YAML file: %v", err)
-	}
-
-	// Create a temporary command
-	cmd := createEvaluateSequenceCommand()
-
-	// Set up command to capture output
-	var output bytes.Buffer
-	cmd.SetOut(&output)
-
-	// Test with a file
-	err = evaluateSequence(cmd, []string{yamlFile})
-
-	// Should not error
-	if err != nil {
-		t.Errorf("evaluateSequence with file should not error, got: %v", err)
-	}
-
-	// Should have some output
-	if output.Len() == 0 {
-		t.Error("Expected output from evaluateSequence with file")
-	}
-}
-
-func TestEvaluateSequence_WithExpressionAndFile(t *testing.T) {
-	// Create a temporary YAML file
-	tempDir := t.TempDir()
-	yamlFile := filepath.Join(tempDir, "test.yaml")
-	yamlContent := []byte("name: test\nage: 25\n")
-	err := os.WriteFile(yamlFile, yamlContent, 0600)
-	if err != nil {
-		t.Fatalf("Failed to create test YAML file: %v", err)
-	}
-
-	// Create a temporary command
-	cmd := createEvaluateSequenceCommand()
-
-	// Set up command to capture output
-	var output bytes.Buffer
-	cmd.SetOut(&output)
-
-	// Test with expression and file
-	err = evaluateSequence(cmd, []string{".name", yamlFile})
-
-	// Should not error
-	if err != nil {
-		t.Errorf("evaluateSequence with expression and file should not error, got: %v", err)
-	}
-
-	// Should have output
-	if output.Len() == 0 {
-		t.Error("Expected output from evaluateSequence with expression and file")
-	}
-}
-
-func TestEvaluateSequence_WriteInPlace(t *testing.T) {
-	// Create a temporary YAML file
-	tempDir := t.TempDir()
-	yamlFile := filepath.Join(tempDir, "test.yaml")
-	yamlContent := []byte("name: test\nage: 25\n")
-	err := os.WriteFile(yamlFile, yamlContent, 0600)
-	if err != nil {
-		t.Fatalf("Failed to create test YAML file: %v", err)
-	}
-
-	// Create a temporary command
-	cmd := createEvaluateSequenceCommand()
-
-	// Set up command to capture output
-	var output bytes.Buffer
-	cmd.SetOut(&output)
-
-	// Enable write in place
-	originalWriteInplace := writeInplace
-	writeInplace = true
-	defer func() { writeInplace = originalWriteInplace }()
-
-	// Test with write in place
-	err = evaluateSequence(cmd, []string{".name = \"updated\"", yamlFile})
-
-	// Should not error
-	if err != nil {
-		t.Errorf("evaluateSequence with write in place should not error, got: %v", err)
-	}
-
-	// Verify the file was updated
-	updatedContent, err := os.ReadFile(yamlFile)
-	if err != nil {
-		t.Fatalf("Failed to read updated file: %v", err)
-	}
-
-	// Should contain the updated content
-	if !strings.Contains(string(updatedContent), "updated") {
-		t.Errorf("Expected file to contain 'updated', got: %s", string(updatedContent))
-	}
-}
-
-func TestEvaluateSequence_ExitStatus(t *testing.T) {
-	// Create a temporary YAML file
-	tempDir := t.TempDir()
-	yamlFile := filepath.Join(tempDir, "test.yaml")
-	yamlContent := []byte("name: test\nage: 25\n")
-	err := os.WriteFile(yamlFile, yamlContent, 0600)
-	if err != nil {
-		t.Fatalf("Failed to create test YAML file: %v", err)
-	}
-
-	// Create a temporary command
-	cmd := createEvaluateSequenceCommand()
-
-	// Set up command to capture output
-	var output bytes.Buffer
-	cmd.SetOut(&output)
-
-	// Enable exit status
-	originalExitStatus := exitStatus
-	exitStatus = true
-	defer func() { exitStatus = originalExitStatus }()
-
-	// Test with expression that should find no matches
-	err = evaluateSequence(cmd, []string{".nonexistent", yamlFile})
-
-	// Should error when no matches found and exit status is enabled
-	if err == nil {
-		t.Error("Expected error when no matches found and exit status is enabled")
-	}
-}

cmd/root.go

@@ -5,6 +5,8 @@ import (
 	"os"
 	"strings"
 
+	"github.com/mikefarah/yq/v4/pkg/properties"
+	"github.com/mikefarah/yq/v4/pkg/xml"
 	"github.com/mikefarah/yq/v4/pkg/yqlib"
 	"github.com/spf13/cobra"
 	logging "gopkg.in/op/go-logging.v1"
@@ -66,23 +68,14 @@ yq -P -oy sample.json
 			return evaluateSequence(cmd, args)
 
 		},
-		PersistentPreRunE: func(cmd *cobra.Command, _ []string) error {
+		PersistentPreRunE: func(cmd *cobra.Command, args []string) error {
 			cmd.SetOut(cmd.OutOrStdout())
 			level := logging.WARNING
 			stringFormat := `[%{level}] %{color}%{time:15:04:05}%{color:reset} %{message}`
 
-			// when NO_COLOR environment variable presents and not an empty string the coloured output should be disabled;
-			// refer to no-color.org
-			forceNoColor = forceNoColor || os.Getenv("NO_COLOR") != ""
-
-			if verbose && forceNoColor {
-				level = logging.DEBUG
-				stringFormat = `[%{level:5.5s}] %{time:15:04:05} %{shortfile:-33s} %{shortfunc:-25s} %{message}`
-			} else if verbose {
+			if verbose {
 				level = logging.DEBUG
 				stringFormat = `[%{level:5.5s}] %{color}%{time:15:04:05}%{color:bold} %{shortfile:-33s} %{shortfunc:-25s}%{color:reset} %{message}`
-			} else if forceNoColor {
-				stringFormat = `[%{level}] %{time:15:04:05} %{message}`
 			}
 
 			var format = logging.MustStringFormatter(stringFormat)
@@ -99,7 +92,6 @@ yq -P -oy sample.json
 	}
 
 	rootCmd.PersistentFlags().BoolVarP(&verbose, "verbose", "v", false, "verbose mode")
-	rootCmd.PersistentFlags().BoolVarP(&printNodeInfo, "debug-node-info", "", false, "debug node info")
 
 	rootCmd.PersistentFlags().BoolVarP(&outputToJSON, "tojson", "j", false, "(deprecated) output as json. Set indent to 0 to print json in one line.")
 	err := rootCmd.PersistentFlags().MarkDeprecated("tojson", "please use -o=json instead")
@@ -127,27 +119,27 @@ yq -P -oy sample.json
 		panic(err)
 	}
 
-	rootCmd.PersistentFlags().StringVar(&yqlib.ConfiguredXMLPreferences.AttributePrefix, "xml-attribute-prefix", yqlib.ConfiguredXMLPreferences.AttributePrefix, "prefix for xml attributes")
+	rootCmd.PersistentFlags().StringVar(&xml.ConfiguredXMLPreferences.AttributePrefix, "xml-attribute-prefix", xml.ConfiguredXMLPreferences.AttributePrefix, "prefix for xml attributes")
 	if err = rootCmd.RegisterFlagCompletionFunc("xml-attribute-prefix", cobra.NoFileCompletions); err != nil {
 		panic(err)
 	}
-	rootCmd.PersistentFlags().StringVar(&yqlib.ConfiguredXMLPreferences.ContentName, "xml-content-name", yqlib.ConfiguredXMLPreferences.ContentName, "name for xml content (if no attribute name is present).")
+	rootCmd.PersistentFlags().StringVar(&xml.ConfiguredXMLPreferences.ContentName, "xml-content-name", xml.ConfiguredXMLPreferences.ContentName, "name for xml content (if no attribute name is present).")
 	if err = rootCmd.RegisterFlagCompletionFunc("xml-content-name", cobra.NoFileCompletions); err != nil {
 		panic(err)
 	}
-	rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredXMLPreferences.StrictMode, "xml-strict-mode", yqlib.ConfiguredXMLPreferences.StrictMode, "enables strict parsing of XML. See https://pkg.go.dev/encoding/xml for more details.")
-	rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredXMLPreferences.KeepNamespace, "xml-keep-namespace", yqlib.ConfiguredXMLPreferences.KeepNamespace, "enables keeping namespace after parsing attributes")
-	rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredXMLPreferences.UseRawToken, "xml-raw-token", yqlib.ConfiguredXMLPreferences.UseRawToken, "enables using RawToken method instead Token. Commonly disables namespace translations. See https://pkg.go.dev/encoding/xml#Decoder.RawToken for details.")
-	rootCmd.PersistentFlags().StringVar(&yqlib.ConfiguredXMLPreferences.ProcInstPrefix, "xml-proc-inst-prefix", yqlib.ConfiguredXMLPreferences.ProcInstPrefix, "prefix for xml processing instructions (e.g. <?xml version=\"1\"?>)")
+	rootCmd.PersistentFlags().BoolVar(&xml.ConfiguredXMLPreferences.StrictMode, "xml-strict-mode", xml.ConfiguredXMLPreferences.StrictMode, "enables strict parsing of XML. See https://pkg.go.dev/encoding/xml for more details.")
+	rootCmd.PersistentFlags().BoolVar(&xml.ConfiguredXMLPreferences.KeepNamespace, "xml-keep-namespace", xml.ConfiguredXMLPreferences.KeepNamespace, "enables keeping namespace after parsing attributes")
+	rootCmd.PersistentFlags().BoolVar(&xml.ConfiguredXMLPreferences.UseRawToken, "xml-raw-token", xml.ConfiguredXMLPreferences.UseRawToken, "enables using RawToken method instead Token. Commonly disables namespace translations. See https://pkg.go.dev/encoding/xml#Decoder.RawToken for details.")
+	rootCmd.PersistentFlags().StringVar(&xml.ConfiguredXMLPreferences.ProcInstPrefix, "xml-proc-inst-prefix", xml.ConfiguredXMLPreferences.ProcInstPrefix, "prefix for xml processing instructions (e.g. <?xml version=\"1\"?>)")
 	if err = rootCmd.RegisterFlagCompletionFunc("xml-proc-inst-prefix", cobra.NoFileCompletions); err != nil {
 		panic(err)
 	}
-	rootCmd.PersistentFlags().StringVar(&yqlib.ConfiguredXMLPreferences.DirectiveName, "xml-directive-name", yqlib.ConfiguredXMLPreferences.DirectiveName, "name for xml directives (e.g. <!DOCTYPE thing cat>)")
+	rootCmd.PersistentFlags().StringVar(&xml.ConfiguredXMLPreferences.DirectiveName, "xml-directive-name", xml.ConfiguredXMLPreferences.DirectiveName, "name for xml directives (e.g. <!DOCTYPE thing cat>)")
 	if err = rootCmd.RegisterFlagCompletionFunc("xml-directive-name", cobra.NoFileCompletions); err != nil {
 		panic(err)
 	}
-	rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredXMLPreferences.SkipProcInst, "xml-skip-proc-inst", yqlib.ConfiguredXMLPreferences.SkipProcInst, "skip over process instructions (e.g. <?xml version=\"1\"?>)")
-	rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredXMLPreferences.SkipDirectives, "xml-skip-directives", yqlib.ConfiguredXMLPreferences.SkipDirectives, "skip over directives (e.g. <!DOCTYPE thing cat>)")
+	rootCmd.PersistentFlags().BoolVar(&xml.ConfiguredXMLPreferences.SkipProcInst, "xml-skip-proc-inst", xml.ConfiguredXMLPreferences.SkipProcInst, "skip over process instructions (e.g. <?xml version=\"1\"?>)")
+	rootCmd.PersistentFlags().BoolVar(&xml.ConfiguredXMLPreferences.SkipDirectives, "xml-skip-directives", xml.ConfiguredXMLPreferences.SkipDirectives, "skip over directives (e.g. <!DOCTYPE thing cat>)")
 
 	rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredCsvPreferences.AutoParse, "csv-auto-parse", yqlib.ConfiguredCsvPreferences.AutoParse, "parse CSV YAML/JSON values")
 	rootCmd.PersistentFlags().Var(newRuneVar(&yqlib.ConfiguredCsvPreferences.Separator), "csv-separator", "CSV Separator character")
@@ -165,13 +157,8 @@ yq -P -oy sample.json
 	rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredLuaPreferences.UnquotedKeys, "lua-unquoted", yqlib.ConfiguredLuaPreferences.UnquotedKeys, "output unquoted string keys (e.g. {foo=\"bar\"})")
 	rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredLuaPreferences.Globals, "lua-globals", yqlib.ConfiguredLuaPreferences.Globals, "output keys as top-level global variables")
 
-	rootCmd.PersistentFlags().StringVar(&yqlib.ConfiguredPropertiesPreferences.KeyValueSeparator, "properties-separator", yqlib.ConfiguredPropertiesPreferences.KeyValueSeparator, "separator to use between keys and values")
-	rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredPropertiesPreferences.UseArrayBrackets, "properties-array-brackets", yqlib.ConfiguredPropertiesPreferences.UseArrayBrackets, "use [x] in array paths (e.g. for SpringBoot)")
-
-	rootCmd.PersistentFlags().StringVar(&yqlib.ConfiguredShellVariablesPreferences.KeySeparator, "shell-key-separator", yqlib.ConfiguredShellVariablesPreferences.KeySeparator, "separator for shell variable key paths")
-	if err = rootCmd.RegisterFlagCompletionFunc("shell-key-separator", cobra.NoFileCompletions); err != nil {
-		panic(err)
-	}
+	rootCmd.PersistentFlags().StringVar(&properties.ConfiguredPropertiesPreferences.KeyValueSeparator, "properties-separator", properties.ConfiguredPropertiesPreferences.KeyValueSeparator, "separator to use between keys and values")
+	rootCmd.PersistentFlags().BoolVar(&properties.ConfiguredPropertiesPreferences.UseArrayBrackets, "properties-array-brackets", properties.ConfiguredPropertiesPreferences.UseArrayBrackets, "use [x] in array paths (e.g. for SpringBoot)")
 
 	rootCmd.PersistentFlags().BoolVar(&yqlib.StringInterpolationEnabled, "string-interpolation", yqlib.StringInterpolationEnabled, "Toggles strings interpolation of \\(exp)")
 
@@ -192,7 +179,7 @@ yq -P -oy sample.json
 	rootCmd.PersistentFlags().BoolVarP(&exitStatus, "exit-status", "e", false, "set exit status if there are no matches or null or false is returned")
 
 	rootCmd.PersistentFlags().BoolVarP(&forceColor, "colors", "C", false, "force print with colors")
-	rootCmd.PersistentFlags().BoolVarP(&forceNoColor, "no-colors", "M", forceNoColor, "force print with no colors")
+	rootCmd.PersistentFlags().BoolVarP(&forceNoColor, "no-colors", "M", false, "force print with no colors")
 	rootCmd.PersistentFlags().StringVarP(&frontMatter, "front-matter", "f", "", "(extract|process) first input as yaml front-matter. Extract will pull out the yaml content, process will run the expression against the yaml content, leaving the remaining data intact")
 	if err = rootCmd.RegisterFlagCompletionFunc("front-matter", cobra.FixedCompletions([]string{"extract", "process"}, cobra.ShellCompDirectiveNoFileComp)); err != nil {
 		panic(err)
@@ -202,9 +189,8 @@ yq -P -oy sample.json
 		panic(err)
 	}
 	rootCmd.PersistentFlags().BoolVarP(&yqlib.ConfiguredYamlPreferences.LeadingContentPreProcessing, "header-preprocess", "", true, "Slurp any header comments and separators before processing expression.")
-	rootCmd.PersistentFlags().BoolVarP(&yqlib.ConfiguredYamlPreferences.FixMergeAnchorToSpec, "yaml-fix-merge-anchor-to-spec", "", false, "Fix merge anchor to match YAML spec. Will default to true in late 2025")
 
-	rootCmd.PersistentFlags().StringVarP(&splitFileExp, "split-exp", "s", "", "print each result (or doc) into a file named (exp). [exp] argument must return a string. You can use $index in the expression as the result counter. The necessary directories will be created.")
+	rootCmd.PersistentFlags().StringVarP(&splitFileExp, "split-exp", "s", "", "print each result (or doc) into a file named (exp). [exp] argument must return a string. You can use $index in the expression as the result counter.")
 	if err = rootCmd.RegisterFlagCompletionFunc("split-exp", cobra.NoFileCompletions); err != nil {
 		panic(err)
 	}

cmd/root_test.go

@@ -1,264 +0,0 @@
-package cmd
-
-import (
-	"strings"
-	"testing"
-)
-
-func TestNewRuneVar(t *testing.T) {
-	var r rune
-	runeVar := newRuneVar(&r)
-
-	if runeVar == nil {
-		t.Fatal("newRuneVar returned nil")
-	}
-}
-
-func TestRuneValue_String(t *testing.T) {
-	tests := []struct {
-		name     string
-		runeVal  rune
-		expected string
-	}{
-		{
-			name:     "simple character",
-			runeVal:  'a',
-			expected: "a",
-		},
-		{
-			name:     "special character",
-			runeVal:  '\n',
-			expected: "\n",
-		},
-		{
-			name:     "unicode character",
-			runeVal:  'ñ',
-			expected: "ñ",
-		},
-		{
-			name:     "zero rune",
-			runeVal:  0,
-			expected: string(rune(0)),
-		},
-	}
-
-	for _, tt := range tests {
-		t.Run(tt.name, func(t *testing.T) {
-			runeVal := runeValue(tt.runeVal)
-			result := runeVal.String()
-			if result != tt.expected {
-				t.Errorf("runeValue.String() = %q, want %q", result, tt.expected)
-			}
-		})
-	}
-}
-
-func TestRuneValue_Set(t *testing.T) {
-	tests := []struct {
-		name        string
-		input       string
-		expected    rune
-		expectError bool
-	}{
-		{
-			name:        "simple character",
-			input:       "a",
-			expected:    'a',
-			expectError: false,
-		},
-		{
-			name:        "newline escape",
-			input:       "\\n",
-			expected:    '\n',
-			expectError: false,
-		},
-		{
-			name:        "tab escape",
-			input:       "\\t",
-			expected:    '\t',
-			expectError: false,
-		},
-		{
-			name:        "carriage return escape",
-			input:       "\\r",
-			expected:    '\r',
-			expectError: false,
-		},
-		{
-			name:        "form feed escape",
-			input:       "\\f",
-			expected:    '\f',
-			expectError: false,
-		},
-		{
-			name:        "vertical tab escape",
-			input:       "\\v",
-			expected:    '\v',
-			expectError: false,
-		},
-		{
-			name:        "empty string",
-			input:       "",
-			expected:    0,
-			expectError: true,
-		},
-		{
-			name:        "multiple characters",
-			input:       "ab",
-			expected:    0,
-			expectError: true,
-		},
-		{
-			name:        "special character",
-			input:       "ñ",
-			expected:    'ñ',
-			expectError: true, // This will fail because the Set function checks len(val) != 1
-		},
-	}
-
-	for _, tt := range tests {
-		t.Run(tt.name, func(t *testing.T) {
-			var r rune
-			runeVal := newRuneVar(&r)
-
-			err := runeVal.Set(tt.input)
-
-			if tt.expectError {
-				if err == nil {
-					t.Errorf("Expected error for input %q, but got none", tt.input)
-				}
-			} else {
-				if err != nil {
-					t.Errorf("Unexpected error for input %q: %v", tt.input, err)
-				}
-				if r != tt.expected {
-					t.Errorf("Expected rune %q (%d), got %q (%d)",
-						string(tt.expected), tt.expected, string(r), r)
-				}
-			}
-		})
-	}
-}
-
-func TestRuneValue_Set_ErrorMessages(t *testing.T) {
-	tests := []struct {
-		name          string
-		input         string
-		expectedError string
-	}{
-		{
-			name:          "empty string error",
-			input:         "",
-			expectedError: "[] is not a valid character. Must be length 1 was 0",
-		},
-		{
-			name:          "multiple characters error",
-			input:         "abc",
-			expectedError: "[abc] is not a valid character. Must be length 1 was 3",
-		},
-	}
-
-	for _, tt := range tests {
-		t.Run(tt.name, func(t *testing.T) {
-			var r rune
-			runeVal := newRuneVar(&r)
-
-			err := runeVal.Set(tt.input)
-
-			if err == nil {
-				t.Errorf("Expected error for input %q, but got none", tt.input)
-				return
-			}
-
-			if !strings.Contains(err.Error(), tt.expectedError) {
-				t.Errorf("Expected error message to contain %q, got %q",
-					tt.expectedError, err.Error())
-			}
-		})
-	}
-}
-
-func TestRuneValue_Type(t *testing.T) {
-	var r rune
-	runeVal := newRuneVar(&r)
-
-	result := runeVal.Type()
-	expected := "char"
-
-	if result != expected {
-		t.Errorf("runeValue.Type() = %q, want %q", result, expected)
-	}
-}
-
-func TestNew(t *testing.T) {
-	rootCmd := New()
-
-	if rootCmd == nil {
-		t.Fatal("New() returned nil")
-	}
-
-	// Test basic command properties
-	if rootCmd.Use != "yq" {
-		t.Errorf("Expected Use to be 'yq', got %q", rootCmd.Use)
-	}
-
-	if rootCmd.Short == "" {
-		t.Error("Expected Short description to be non-empty")
-	}
-
-	if rootCmd.Long == "" {
-		t.Error("Expected Long description to be non-empty")
-	}
-
-	// Test that the command has the expected subcommands
-	expectedCommands := []string{"eval", "eval-all", "completion"}
-	actualCommands := make([]string, 0, len(rootCmd.Commands()))
-
-	for _, cmd := range rootCmd.Commands() {
-		actualCommands = append(actualCommands, cmd.Name())
-	}
-
-	for _, expected := range expectedCommands {
-		found := false
-		for _, actual := range actualCommands {
-			if actual == expected {
-				found = true
-				break
-			}
-		}
-		if !found {
-			t.Errorf("Expected command %q not found in actual commands: %v",
-				expected, actualCommands)
-		}
-	}
-}
-
-func TestNew_FlagCompletions(t *testing.T) {
-	rootCmd := New()
-
-	// Test that flag completion functions are registered
-	// This is a basic smoke test - we can't easily test the actual completion logic
-	// without more complex setup
-	flags := []string{
-		"output-format",
-		"input-format",
-		"xml-attribute-prefix",
-		"xml-content-name",
-		"xml-proc-inst-prefix",
-		"xml-directive-name",
-		"lua-prefix",
-		"lua-suffix",
-		"properties-separator",
-		"indent",
-		"front-matter",
-		"expression",
-		"split-exp",
-	}
-
-	for _, flagName := range flags {
-		flag := rootCmd.PersistentFlags().Lookup(flagName)
-		if flag == nil {
-			t.Errorf("Expected flag %q to exist", flagName)
-		}
-	}
-}

cmd/shell-completion.go

@@ -7,20 +7,19 @@ import (
 )
 
 var completionCmd = &cobra.Command{
-	Use:     "completion [bash|zsh|fish|powershell]",
-	Aliases: []string{"shell-completion"},
-	Short:   "Generate the autocompletion script for the specified shell",
+	Use:   "shell-completion [bash|zsh|fish|powershell]",
+	Short: "Generate completion script",
 	Long: `To load completions:
 
 Bash:
 
-$ source <(yq completion bash)
+$ source <(yq shell-completion bash)
 
 # To load completions for each session, execute once:
 Linux:
-  $ yq completion bash > /etc/bash_completion.d/yq
+  $ yq shell-completion bash > /etc/bash_completion.d/yq
 MacOS:
-  $ yq completion bash > /usr/local/etc/bash_completion.d/yq
+  $ yq shell-completion bash > /usr/local/etc/bash_completion.d/yq
 
 Zsh:
 
@@ -30,16 +29,16 @@ Zsh:
 $ echo "autoload -U compinit; compinit" >> ~/.zshrc
 
 # To load completions for each session, execute once:
-$ yq completion zsh > "${fpath[1]}/_yq"
+$ yq shell-completion zsh > "${fpath[1]}/_yq"
 
 # You will need to start a new shell for this setup to take effect.
 
 Fish:
 
-$ yq completion fish | source
+$ yq shell-completion fish | source
 
 # To load completions for each session, execute once:
-$ yq completion fish > ~/.config/fish/completions/yq.fish
+$ yq shell-completion fish > ~/.config/fish/completions/yq.fish
 `,
 	DisableFlagsInUseLine: true,
 	ValidArgs:             []string{"bash", "zsh", "fish", "powershell"},

cmd/utils.go

@@ -6,6 +6,8 @@ import (
 	"os"
 	"strings"
 
+	"github.com/mikefarah/yq/v4/pkg/properties"
+	"github.com/mikefarah/yq/v4/pkg/xml"
 	"github.com/mikefarah/yq/v4/pkg/yqlib"
 	"github.com/spf13/cobra"
 	"gopkg.in/op/go-logging.v1"
@@ -18,100 +20,52 @@ func isAutomaticOutputFormat() bool {
 func initCommand(cmd *cobra.Command, args []string) (string, []string, error) {
 	cmd.SilenceUsage = true
 
-	setupColors()
+	fileInfo, _ := os.Stdout.Stat()
+
+	if forceColor || (!forceNoColor && (fileInfo.Mode()&os.ModeCharDevice) != 0) {
+		colorsEnabled = true
+	}
 
 	expression, args, err := processArgs(args)
 	if err != nil {
 		return "", nil, err
 	}
 
-	if err := loadSplitFileExpression(); err != nil {
-		return "", nil, err
-	}
-
-	handleBackwardsCompatibility()
-
-	if err := validateCommandFlags(args); err != nil {
-		return "", nil, err
-	}
-
-	if err := configureFormats(args); err != nil {
-		return "", nil, err
-	}
-
-	configureUnwrapScalar()
-
-	return expression, args, nil
-}
-
-func setupColors() {
-	fileInfo, _ := os.Stdout.Stat()
-
-	if forceColor || (!forceNoColor && (fileInfo.Mode()&os.ModeCharDevice) != 0) {
-		colorsEnabled = true
-	}
-}
-
-func loadSplitFileExpression() error {
 	if splitFileExpFile != "" {
 		splitExpressionBytes, err := os.ReadFile(splitFileExpFile)
 		if err != nil {
-			return err
+			return "", nil, err
 		}
 		splitFileExp = string(splitExpressionBytes)
 	}
-	return nil
-}
 
-func handleBackwardsCompatibility() {
 	// backwards compatibility
 	if outputToJSON {
 		outputFormat = "json"
 	}
-}
 
-func validateCommandFlags(args []string) error {
 	if writeInplace && (len(args) == 0 || args[0] == "-") {
-		return fmt.Errorf("write in place flag only applicable when giving an expression and at least one file")
+		return "", nil, fmt.Errorf("write in place flag only applicable when giving an expression and at least one file")
 	}
 
 	if frontMatter != "" && len(args) == 0 {
-		return fmt.Errorf("front matter flag only applicable when giving an expression and at least one file")
+		return "", nil, fmt.Errorf("front matter flag only applicable when giving an expression and at least one file")
 	}
 
 	if writeInplace && splitFileExp != "" {
-		return fmt.Errorf("write in place cannot be used with split file")
+		return "", nil, fmt.Errorf("write in place cannot be used with split file")
 	}
 
 	if nullInput && len(args) > 0 {
-		return fmt.Errorf("cannot pass files in when using null-input flag")
+		return "", nil, fmt.Errorf("cannot pass files in when using null-input flag")
 	}
 
-	return nil
-}
-
-func configureFormats(args []string) error {
 	inputFilename := ""
 	if len(args) > 0 {
 		inputFilename = args[0]
 	}
-
-	if err := configureInputFormat(inputFilename); err != nil {
-		return err
-	}
-
-	if err := configureOutputFormat(); err != nil {
-		return err
-	}
-
-	yqlib.GetLogger().Debug("Using input format %v", inputFormat)
-	yqlib.GetLogger().Debug("Using output format %v", outputFormat)
-
-	return nil
-}
-
-func configureInputFormat(inputFilename string) error {
 	if inputFormat == "" || inputFormat == "auto" || inputFormat == "a" {
+
 		inputFormat = yqlib.FormatStringFromFilename(inputFilename)
 
 		_, err := yqlib.FormatFromString(inputFormat)
@@ -123,6 +77,10 @@ func configureInputFormat(inputFilename string) error {
 				outputFormat = "yaml"
 			}
 		} else if isAutomaticOutputFormat() {
+			// automatic input worked, we can do it for output too unless specified
+			if inputFormat == "json" {
+				yqlib.GetLogger().Warning("JSON file output is now JSON by default (instead of yaml). Use '-oy' or '--output-format=yaml' for yaml output")
+			}
 			outputFormat = inputFormat
 		}
 	} else if isAutomaticOutputFormat() {
@@ -136,27 +94,24 @@ func configureInputFormat(inputFilename string) error {
 		}
 		outputFormat = "yaml"
 	}
-	return nil
-}
 
-func configureOutputFormat() error {
 	outputFormatType, err := yqlib.FormatFromString(outputFormat)
+
 	if err != nil {
-		return err
+		return "", nil, err
 	}
+	yqlib.GetLogger().Debug("Using input format %v", inputFormat)
+	yqlib.GetLogger().Debug("Using output format %v", outputFormat)
 
 	if outputFormatType == yqlib.YamlFormat ||
-		outputFormatType == yqlib.PropertiesFormat {
+		outputFormatType == properties.PropertiesFormat {
 		unwrapScalar = true
 	}
-
-	return nil
-}
-
-func configureUnwrapScalar() {
 	if unwrapScalarFlag.IsExplicitlySet() {
 		unwrapScalar = unwrapScalarFlag.IsSet()
 	}
+
+	return expression, args, nil
 }
 
 func configureDecoder(evaluateTogether bool) (yqlib.Decoder, error) {
@@ -195,12 +150,12 @@ func configureEncoder() (yqlib.Encoder, error) {
 	if err != nil {
 		return nil, err
 	}
-	yqlib.ConfiguredXMLPreferences.Indent = indent
+	xml.ConfiguredXMLPreferences.Indent = indent
 	yqlib.ConfiguredYamlPreferences.Indent = indent
 	yqlib.ConfiguredJSONPreferences.Indent = indent
 
 	yqlib.ConfiguredYamlPreferences.UnwrapScalar = unwrapScalar
-	yqlib.ConfiguredPropertiesPreferences.UnwrapScalar = unwrapScalar
+	properties.ConfiguredPropertiesPreferences.UnwrapScalar = unwrapScalar
 	yqlib.ConfiguredJSONPreferences.UnwrapScalar = unwrapScalar
 
 	yqlib.ConfiguredYamlPreferences.ColorsEnabled = colorsEnabled

cmd/version.go

@@ -11,7 +11,7 @@ var (
 	GitDescribe string
 
 	// Version is main version number that is being run at the moment.
-	Version = "v4.48.2"
+	Version = "v4.42.1"
 
 	// VersionPrerelease is a pre-release marker for the version. If this is "" (empty string)
 	// then it means that it is a final release. Otherwise, this is a pre-release
@@ -45,5 +45,5 @@ func getHumanVersion() string {
 	}
 
 	// Strip off any single quotes added by the git information.
-	return strings.ReplaceAll(version, "'", "")
+	return strings.Replace(version, "'", "", -1)
 }

debian/changelog

@@ -0,0 +1,210 @@
+yq (4.16.2) focal; urgency=medium
+
+  * Fixed with semicolon space issue
+  * Updating with documentation
+  * Added STDIN example to the top
+  * minor readme cleanup
+  * Help text tweak
+  * Fixed docker timeout - simplify docker builds
+  * New release with docker build fixes
+  * Updating to go 1.17 to fix CVE #944
+  * Fix a typo in root.go
+  * Skip the tests if the nocheck Debian build option is specified
+  * Fixed select bug (#958)
+  * Sped up explode operator
+  * Slight performance improvement to context.ChildContext
+  * Speed up multiply
+  * Update README with recently added / changed options
+  * Make deepMatch report in linear time
+  * Removed leadingContentPreProcessing flag - header preprocessing is stable
+  * Revert "Removed leadingContentPreProcessing flag - header preprocessing is stable"
+  * Keep flag, it is needed in corner cases
+  * Updated Readme
+  * Man page
+  * Fixed expression parsing bug #970
+  * Bumping go-lang, docker versions
+  * Added test release flow
+  * Updated github action release to generate man page
+  * Bumping version
+  * Removing no longer needed github action
+  * Added decoder op
+  * Fixed newline handling when decoding/encoding
+  * Fixed newline handling in encoder/decoder
+  * Can specify indent in encode ops
+  * Added group_by operator
+  * Added flatten operator
+  * Fixed flatten error message
+  * Improving docs
+  * Split printer
+  * Refactored command logic
+  * Fix JSON encoding removing null #985
+  * Fixed acceptance tests
+  * gitbook
+  * Update document generation script
+  * Updating README
+  * Updating release instructions
+  * github action no longer uses data1.yml
+  * Create dependabot.yml
+  * Bump actions/create-release from 1.0.0 to 1.1.4
+  * Bump actions/setup-go from 1 to 2.1.4
+  * Bump github.com/goccy/go-yaml from 1.8.9 to 1.9.4
+  * Bump github.com/jinzhu/copier from 0.2.8 to 0.3.2
+  * Bump github.com/fatih/color from 1.10.0 to 1.13.0
+  * Bump github.com/spf13/cobra from 1.1.3 to 1.2.1
+  * Update dependabot.yml
+  * Update go.yml
+  * add build check to PRs
+  * Include secure as part of build process
+  * Fixing bad label in github action
+  * fixed printer test
+  * remove leading content indicator
+  * Fixed header preprocessing!
+  * lint : define golangci configuration file
+  * Update check.sh
+  * Load file acceptance test
+  * Minor improvement on handling front matter
+  * Improved load doc
+  * feature: detect MANPATh and install there
+  * Update install-man-page.sh
+  * simplify prod stage, move version label to action
+  * add labels, quote some values
+  * enable errorlint linter
+  * Added errorlint to devtools
+  * Added key operator
+  * Added more tests
+  * Fixing comments
+  * Attempt to fix golint problem
+  * Include version query for tools
+  * Clean up errored file?
+  * enable misspell linter
+  * updated readme
+  * update Golangci version to v1.43.0
+  * gci linter
+  * Better merge array by key example
+  * Added credit for merge by array example
+  * Better formatting of merge arrays example
+  * Better merge example
+  * Add accessor for the yq logger instance (#1013)
+  * Fixed collect op when working with multiple nodes
+  * Added map, map_values
+  * Add support for Podman as well as Docker (#1026)
+  * Bump github.com/jinzhu/copier from 0.3.2 to 0.3.4 (#1027)
+  * Added csv, tsv output formats
+  * Added encoder tests
+  * Cleanup test
+  * Fixed docker permission issue #1014
+  * Recording release notes for next release
+  * Assignment op no longer clobbers anchor (#1029)
+  * Added sort_by operator
+  * Improved error message
+  * Improved tips and tricks
+  * Report while filename failed to parse #1030
+  * Added script for extracting checksums
+  * Improved extract-checksum.sh
+  * Bump github.com/spf13/cobra from 1.2.1 to 1.3.0 (#1039)
+  * enable more linters (#1043)
+  * Bump Golang compiler #1037
+
+ -- Roberto Mier Escandon <rmescandon@gmail.com>  Tue, 21 Dec 2021 09:41:44 +0000
+
+yq (4.13.0) focal; urgency=medium
+
+  * New `with` operator for making multiple changes to a given path
+  * New `contains` operator, works like the `jq` equivalent
+  * Subtract operator now supports subtracting elements from arrays!
+  * Fixed Swapping values using variables #934
+  * Github Action now properly supports multiline output #936, thanks @pjxiao
+  * Fixed missing closing bracket validation #932
+  * Fix processing of hex numbers #929
+  * Fixed alternative and union operator issues #930
+  * Can now convert yaml to properties properties format (`-o=props`), See [docs](https://mikefarah.gitbook.io/yq/v/v4.x/usage/properties) for more info.
+  * Fixed document header/footer comment handling when merging (https://github.com/mikefarah/yq/issues/919)
+  * pretty print yaml 1.1 compatibility  (https://github.com/mikefarah/yq/issues/914)
+
+ -- Roberto Mier Escandon <rmescandon@gmail.com>  Thu, 16 Sep 2021 20:58:30 +0200
+
+yq (4.9.6) focal; urgency=medium
+
+  * Added darwin/arm64 build, thanks @alecthomas
+  * Incremented docker alpine base version, thanks @da6d6i7-bronga
+  * Bug fix: multiline expression
+  * Bug fix: special character
+
+ -- Roberto Mier Escandon <rmescandon@gmail.com>  Tue, 29 Jun 2021 21:32:14 +0200
+
+yq (3.3.2) focal; urgency=medium
+
+  * Bug fix: existStatus bug (#459)
+  * Automatically makes an os temp directory if it does not exist (#461)
+
+ -- Roberto Mier Escandon <rmescandon@gmail.com>  Fri, 07 Aug 2020 18:53:01 +0200
+
+yq (3.3-0) focal; urgency=medium
+
+  * You can control string styles (quotes) using the new --style flag
+  * String values now always have quotes when outputting to json
+  * Negative array indices now traverse the array backwards
+  * Added a --stripComments flag to print yaml without any comments
+  * Bumped go to version 1.14
+
+ -- Roberto Mier Escandon <rmescandon@gmail.com>  Thu, 30 Apr 2020 20:45:44 +0200
+
+yq (3.1-2) eoan; urgency=medium
+
+  * Bug fix: yq 3 was removing empty inline-style objects and arrays (#355)
+  * Bug fix: Merge option returned different output when switching order of 
+    merging files(#347)
+  * Bug fix: Add new object to existing array object was failing in 3.1.1 (#361)
+  * Bug fix: yq 3 empty keys did not allow merging of values (#356)
+  * Bug fix: keys quoted during merge (#363)
+  * Bug fix: Correct length with wc -l (#362)
+  * Bug fix: Write to empty document removed path (#359)
+
+ -- Roberto Mier Escandon <rmescandon@gmail.com>  Mon, 24 Feb 2020 20:31:58 +0100
+
+yq (3.1-1) eoan; urgency=medium
+
+  * Keeps yaml comments and formatting, can specify yaml tags when updating.
+  * Handles anchors
+  * Can print out matching paths and values when splatting
+  * JSON output works for all commands
+  * Yaml files with multiple documents are printed out as one JSON 
+    document per line.
+  * Deep splat (**) to match arbitrary paths
+  * Update scripts file format has changed to be more powerful
+  * Reading and splatting, matching results are printed once per line
+  * Bugfixing
+
+ -- Roberto Mier Escandon <rmescandon@gmail.com>  Tue, 11 Feb 2020 22:18:24 +0100
+
+yq (2.2-1) bionic; urgency=medium
+
+  * Added Windows support for the "--inplace" command flag
+  * Prefix now supports arrays
+  * Add prefix command
+  * Bump Alpine version to 3.8
+  * Improved docker build process
+  * Lint fixes
+  * Build support for all linux architectures supported by gox
+
+ -- Roberto Mier Escandon <rmescandon@gmail.com>  Sat, 19 Jan 2019 15:50:47 +0100
+
+yq (2.1-0) bionic; urgency=medium
+
+  * Ability to read multiple documents in a single file
+  * Ability to append list items instead of overwriting
+
+ -- Roberto Mier Escandón <rmescandon@gmail.com>  Tue, 10 Jul 2018 14:02:42 +0200
+
+yq (2.0-0) bionic; urgency=medium
+
+  * Release 2.0.0
+
+ -- Roberto Mier Escandón <rmescandon@gmail.com>  Wed, 20 Jun 2018 10:29:53 +0200
+
+yq (1.15-0) bionic; urgency=medium
+
+  * Release 1.15
+
+ -- Roberto Mier Escandón <rmescandon@gmail.com>  Wed, 06 Jun 2018 11:32:03 +0200
+

debian/compat

@@ -0,0 +1 @@
+10

debian/control

@@ -0,0 +1,22 @@
+Source: yq
+Section: devel
+Priority: optional
+Maintainer: Roberto Mier Escandón <rmescandon@gmail.com>
+Build-Depends: debhelper (>=10),
+               golang-1.17-go,
+               pandoc,
+               rsync
+Standards-Version: 4.1.4
+Homepage: https://github.com/mikefarah/yq.git
+Vcs-Browser: https://github.com/mikefarah/yq.git
+Vcs-Git: https://github.com/mikefarah/yq.git
+XS-Go-Import-Path: github.com/mikefarah/yq
+XSBC-Original-Maintainer: Roberto Mier Escandón <rmescandon@gmail.com>
+
+Package: yq
+Architecture: any
+Depends: ${shlibs:Depends}, ${misc:Depends}
+Description: lightweight and portable command-line data file processor
+ .
+ The aim of the project is to be the 
+ [jq](https://github.com/stedolan/jq) or sed of yaml files.

debian/copyright

@@ -0,0 +1,24 @@
+Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
+Upstream-Name: yq
+Source: https://github.com/mikefarah/yq.git
+
+Files: *
+Copyright: 2017 Mike Farah
+License: Expat
+    Permission is hereby granted, free of charge, to any person obtaining a copy
+    of this software and associated documentation files (the "Software"), to deal
+    in the Software without restriction, including without limitation the rights
+    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+    copies of the Software, and to permit persons to whom the Software is
+    furnished to do so, subject to the following conditions:
+    .
+    The above copyright notice and this permission notice shall be included in all
+    copies or substantial portions of the Software.
+    .
+    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+    SOFTWARE.

debian/gbp.conf

@@ -0,0 +1,2 @@
+[DEFAULT]
+pristine-tar = True

debian/rules

@@ -0,0 +1,74 @@
+#!/usr/bin/make -f
+#
+# Copyright (C) 2018-2021 Roberto Mier Escandón <rmescandon@gmail.com>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+PROJECT := yq
+OWNER := mikefarah
+REPO := github.com
+
+export DH_OPTIONS
+export DH_GOPKG := ${REPO}/${OWNER}/${PROJECT}
+export GOROOT := /usr/local/go
+export GOPATH := ${CURDIR}/_build
+export GOBIN := ${GOPATH}/bin
+export PATH := ${GOROOT}/bin:${GOBIN}:${PATH}
+export GOCACHE := /tmp/gocache
+export GOFLAGS := -mod=vendor
+export GO111MODULE := on
+
+SRCDIR := ${GOPATH}/src/${DH_GOPKG}
+DESTDIR := ${CURDIR}/debian/${PROJECT}
+BINDIR := /usr/bin
+MANDIR := /usr/share/man/man1/
+ASSETSDIR := /usr/share/${PROJECT}
+
+%:
+	dh $@
+
+override_dh_auto_build:
+	mkdir -p ${SRCDIR}
+	mkdir -p ${GOBIN}
+	# copy project to local srcdir to build from there
+	rsync -avz --progress --exclude=_build --exclude=debian --exclude=tmp. --exclude=go.mod --exclude=docs . $(SRCDIR)
+	# build go code
+	( \
+		cd ${SRCDIR} && \
+		go install -buildmode=pie ./... \
+	)
+
+	# build man page
+	( \
+		cd ${SRCDIR} && \
+		./scripts/generate-man-page-md.sh && \
+		./scripts/generate-man-page.sh \
+	)
+	
+override_dh_auto_test:
+ifeq (,$(filter nocheck,$(DEB_BUILD_OPTIONS)))
+	(cd ${SRCDIR} && go test -v ./...)
+endif
+
+override_dh_auto_install:
+	cp ${GOBIN}/yq ${DESTDIR}/${BINDIR}
+	cp -f ${SRCDIR}/LICENSE ${DESTDIR}/${ASSETSDIR}
+	chmod a+x ${DESTDIR}/${BINDIR}/yq
+
+	# man
+	mkdir -p "${DESTDIR}"/"${MANDIR}"
+	cp "${SRCDIR}"/yq.1 "${DESTDIR}"/"${MANDIR}" \
+
+override_dh_auto_clean:
+	dh_clean
+	rm -rf ${CURDIR}/_build

debian/source/format

@@ -0,0 +1 @@
+3.0 (native)

debian/yq.dirs

@@ -0,0 +1,3 @@
+usr/bin
+usr/share/yq
+usr/share/man/man1

examples/data1.yaml

@@ -1,8 +1,11 @@
-# 001
----
-abc:  # 001
-- 1   # one
-- 2   # two
-
----
-def   # 002
+# block comments come through
+person: # neither do comments on maps
+    name: Mike Wazowski # comments on values appear
+    pets: 
+    - cat # comments on array values appear
+    - dog # comments on array values appear
+    - things:
+        - frog
+    food: [pizza] # comments on arrays do not
+emptyArray: []
+emptyMap: []

examples/front-matter.yaml

@@ -1,6 +1,6 @@
 ---
 a: apple
-b: banana
+b: bannana
 ---
 hello there
 apples: great

examples/sample.ini

@@ -1,14 +0,0 @@
-
-; This is a INI document
-
-[owner]
-name = "Tom Preston-Werner"
-dob = 1979-05-27T07:32:00-08:00
-
-[database]
-db_host = "localhost"
-db_port = 5432
-db_user = "postgres"
-db_password = ""
-db_name = "postgres"
-

examples/sample_no_sections.ini

@@ -1,8 +0,0 @@
-
-; This is a INI document
-name = "Tom Preston-Werner"
-dob = 1979-05-27T07:32:00-08:00
-enabled = true
-ip = "10.0.0.1"
-role = "frontend"
-treads = 4

go.mod

@@ -1,35 +1,33 @@
 module github.com/mikefarah/yq/v4
 
 require (
-	github.com/a8m/envsubst v1.4.3
-	github.com/alecthomas/participle/v2 v2.1.4
-	github.com/alecthomas/repr v0.5.2
+	github.com/a8m/envsubst v1.4.2
+	github.com/alecthomas/participle/v2 v2.1.1
+	github.com/alecthomas/repr v0.4.0
 	github.com/dimchansky/utfbom v1.1.1
-	github.com/elliotchance/orderedmap v1.8.0
-	github.com/fatih/color v1.18.0
-	github.com/go-ini/ini v1.67.0
-	github.com/goccy/go-json v0.10.5
-	github.com/goccy/go-yaml v1.18.0
+	github.com/elliotchance/orderedmap v1.5.1
+	github.com/fatih/color v1.16.0
+	github.com/goccy/go-json v0.10.2
+	github.com/goccy/go-yaml v1.11.3
 	github.com/jinzhu/copier v0.4.0
-	github.com/magiconair/properties v1.8.10
-	github.com/pelletier/go-toml/v2 v2.2.4
+	github.com/magiconair/properties v1.8.7
+	github.com/pelletier/go-toml/v2 v2.1.1
 	github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e
-	github.com/spf13/cobra v1.10.1
-	github.com/spf13/pflag v1.0.10
+	github.com/spf13/cobra v1.8.0
+	github.com/spf13/pflag v1.0.5
 	github.com/yuin/gopher-lua v1.1.1
-	go.yaml.in/yaml/v4 v4.0.0-rc.3
-	golang.org/x/net v0.47.0
-	golang.org/x/text v0.31.0
+	golang.org/x/net v0.22.0
+	golang.org/x/text v0.14.0
 	gopkg.in/op/go-logging.v1 v1.0.0-20160211212156-b2cb9fa56473
+	gopkg.in/yaml.v3 v3.0.1
 )
 
 require (
 	github.com/inconshreveable/mousetrap v1.1.0 // indirect
-	github.com/mattn/go-colorable v0.1.14 // indirect
+	github.com/mattn/go-colorable v0.1.13 // indirect
 	github.com/mattn/go-isatty v0.0.20 // indirect
-	golang.org/x/sys v0.38.0 // indirect
+	golang.org/x/sys v0.18.0 // indirect
+	golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 // indirect
 )
 
-go 1.24.0
-
-toolchain go1.24.1
+go 1.21

go.sum

@@ -1,64 +1,82 @@
-github.com/a8m/envsubst v1.4.3 h1:kDF7paGK8QACWYaQo6KtyYBozY2jhQrTuNNuUxQkhJY=
-github.com/a8m/envsubst v1.4.3/go.mod h1:4jjHWQlZoaXPoLQUb7H2qT4iLkZDdmEQiOUogdUmqVU=
-github.com/alecthomas/assert/v2 v2.11.0 h1:2Q9r3ki8+JYXvGsDyBXwH3LcJ+WK5D0gc5E8vS6K3D0=
-github.com/alecthomas/assert/v2 v2.11.0/go.mod h1:Bze95FyfUr7x34QZrjL+XP+0qgp/zg8yS+TtBj1WA3k=
-github.com/alecthomas/participle/v2 v2.1.4 h1:W/H79S8Sat/krZ3el6sQMvMaahJ+XcM9WSI2naI7w2U=
-github.com/alecthomas/participle/v2 v2.1.4/go.mod h1:8tqVbpTX20Ru4NfYQgZf4mP18eXPTBViyMWiArNEgGI=
-github.com/alecthomas/repr v0.5.2 h1:SU73FTI9D1P5UNtvseffFSGmdNci/O6RsqzeXJtP0Qs=
-github.com/alecthomas/repr v0.5.2/go.mod h1:Fr0507jx4eOXV7AlPV6AVZLYrLIuIeSOWtW57eE/O/4=
-github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
-github.com/davecgh/go-spew v1.1.0 h1:ZDRjVQ15GmhC3fiQ8ni8+OwkZQO4DARzQgrnXU1Liz8=
+github.com/a8m/envsubst v1.4.2 h1:4yWIHXOLEJHQEFd4UjrWDrYeYlV7ncFWJOCBRLOZHQg=
+github.com/a8m/envsubst v1.4.2/go.mod h1:MVUTQNGQ3tsjOOtKCNd+fl8RzhsXcDvvAEzkhGtlsbY=
+github.com/alecthomas/assert/v2 v2.3.0 h1:mAsH2wmvjsuvyBvAmCtm7zFsBlb8mIHx5ySLVdDZXL0=
+github.com/alecthomas/assert/v2 v2.3.0/go.mod h1:pXcQ2Asjp247dahGEmsZ6ru0UVwnkhktn7S0bBDLxvQ=
+github.com/alecthomas/participle/v2 v2.1.1 h1:hrjKESvSqGHzRb4yW1ciisFJ4p3MGYih6icjJvbsmV8=
+github.com/alecthomas/participle/v2 v2.1.1/go.mod h1:Y1+hAs8DHPmc3YUFzqllV+eSQ9ljPTk0ZkPMtEdAx2c=
+github.com/alecthomas/repr v0.4.0 h1:GhI2A8MACjfegCPVq9f1FLvIBS+DrQ2KQBFZP1iFzXc=
+github.com/alecthomas/repr v0.4.0/go.mod h1:Fr0507jx4eOXV7AlPV6AVZLYrLIuIeSOWtW57eE/O/4=
+github.com/cpuguy83/go-md2man/v2 v2.0.3/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
+github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/dimchansky/utfbom v1.1.1 h1:vV6w1AhK4VMnhBno/TPVCoK9U/LP0PkLCS9tbxHdi/U=
 github.com/dimchansky/utfbom v1.1.1/go.mod h1:SxdoEBH5qIqFocHMyGOXVAybYJdr71b1Q/j0mACtrfE=
-github.com/elliotchance/orderedmap v1.8.0 h1:TrOREecvh3JbS+NCgwposXG5ZTFHtEsQiCGOhPElnMw=
-github.com/elliotchance/orderedmap v1.8.0/go.mod h1:wsDwEaX5jEoyhbs7x93zk2H/qv0zwuhg4inXhDkYqys=
-github.com/fatih/color v1.18.0 h1:S8gINlzdQ840/4pfAwic/ZE0djQEH3wM94VfqLTZcOM=
-github.com/fatih/color v1.18.0/go.mod h1:4FelSpRwEGDpQ12mAdzqdOukCy4u8WUtOY6lkT/6HfU=
-github.com/go-ini/ini v1.67.0 h1:z6ZrTEZqSWOTyH2FlglNbNgARyHG8oLW9gMELqKr06A=
-github.com/go-ini/ini v1.67.0/go.mod h1:ByCAeIL28uOIIG0E3PJtZPDL8WnHpFKFOtgjp+3Ies8=
-github.com/goccy/go-json v0.10.5 h1:Fq85nIqj+gXn/S5ahsiTlK3TmC85qgirsdTP/+DeaC4=
-github.com/goccy/go-json v0.10.5/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
-github.com/goccy/go-yaml v1.18.0 h1:8W7wMFS12Pcas7KU+VVkaiCng+kG8QiFeFwzFb+rwuw=
-github.com/goccy/go-yaml v1.18.0/go.mod h1:XBurs7gK8ATbW4ZPGKgcbrY1Br56PdM69F7LkFRi1kA=
+github.com/elliotchance/orderedmap v1.5.1 h1:G1X4PYlljzimbdQ3RXmtIZiQ9d6aRQ3sH1nzjq5mECE=
+github.com/elliotchance/orderedmap v1.5.1/go.mod h1:wsDwEaX5jEoyhbs7x93zk2H/qv0zwuhg4inXhDkYqys=
+github.com/fatih/color v1.16.0 h1:zmkK9Ngbjj+K0yRhTVONQh1p/HknKYSlNT+vZCzyokM=
+github.com/fatih/color v1.16.0/go.mod h1:fL2Sau1YI5c0pdGEVCbKQbLXB6edEj1ZgiY4NijnWvE=
+github.com/go-playground/locales v0.13.0 h1:HyWk6mgj5qFqCT5fjGBuRArbVDfE4hi8+e8ceBS/t7Q=
+github.com/go-playground/locales v0.13.0/go.mod h1:taPMhCMXrRLJO55olJkUXHZBHCxTMfnGwq/HNwmWNS8=
+github.com/go-playground/universal-translator v0.17.0 h1:icxd5fm+REJzpZx7ZfpaD876Lmtgy7VtROAbHHXk8no=
+github.com/go-playground/universal-translator v0.17.0/go.mod h1:UkSxE5sNxxRwHyU+Scu5vgOQjsIJAF8j9muTVoKLVtA=
+github.com/go-playground/validator/v10 v10.4.1 h1:pH2c5ADXtd66mxoE0Zm9SUhxE20r7aM3F26W0hOn+GE=
+github.com/go-playground/validator/v10 v10.4.1/go.mod h1:nlOn6nFhuKACm19sB/8EGNn9GlaMV7XkbRSipzJ0Ii4=
+github.com/goccy/go-json v0.10.2 h1:CrxCmQqYDkv1z7lO7Wbh2HN93uovUHgrECaO5ZrCXAU=
+github.com/goccy/go-json v0.10.2/go.mod h1:6MelG93GURQebXPDq3khkgXZkazVtN9CRI+MGFi0w8I=
+github.com/goccy/go-yaml v1.11.3 h1:B3W9IdWbvrUu2OYQGwvU1nZtvMQJPBKgBUuweJjLj6I=
+github.com/goccy/go-yaml v1.11.3/go.mod h1:wKnAMd44+9JAAnGQpWVEgBzGt3YuTaQ4uXoHvE4m7WU=
+github.com/google/go-cmp v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38=
+github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
 github.com/hexops/gotextdiff v1.0.3 h1:gitA9+qJrrTCsiCl7+kh75nPqQt1cx4ZkudSTLoUqJM=
 github.com/hexops/gotextdiff v1.0.3/go.mod h1:pSWU5MAI3yDq+fZBTazCSJysOMbxWL1BSow5/V2vxeg=
 github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
 github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
 github.com/jinzhu/copier v0.4.0 h1:w3ciUoD19shMCRargcpm0cm91ytaBhDvuRpz1ODO/U8=
 github.com/jinzhu/copier v0.4.0/go.mod h1:DfbEm0FYsaqBcKcFuvmOZb218JkPGtvSHsKg8S8hyyg=
-github.com/magiconair/properties v1.8.10 h1:s31yESBquKXCV9a/ScB3ESkOjUYYv+X0rg8SYxI99mE=
-github.com/magiconair/properties v1.8.10/go.mod h1:Dhd985XPs7jluiymwWYZ0G4Z61jb3vdS329zhj2hYo0=
-github.com/mattn/go-colorable v0.1.14 h1:9A9LHSqF/7dyVVX6g0U9cwm9pG3kP9gSzcuIPHPsaIE=
-github.com/mattn/go-colorable v0.1.14/go.mod h1:6LmQG8QLFO4G5z1gPvYEzlUgJ2wF+stgPZH1UqBm1s8=
+github.com/leodido/go-urn v1.2.0 h1:hpXL4XnriNwQ/ABnpepYM/1vCLWNDfUNts8dX3xTG6Y=
+github.com/leodido/go-urn v1.2.0/go.mod h1:+8+nEpDfqqsY+g338gtMEUOtuK+4dEMhiQEgxpxOKII=
+github.com/magiconair/properties v1.8.7 h1:IeQXZAiQcpL9mgcAe1Nu6cX9LLw6ExEHKjN0VQdvPDY=
+github.com/magiconair/properties v1.8.7/go.mod h1:Dhd985XPs7jluiymwWYZ0G4Z61jb3vdS329zhj2hYo0=
+github.com/mattn/go-colorable v0.1.13 h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA=
+github.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovkB8vQcUbaXHg=
+github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM=
 github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
 github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
-github.com/pelletier/go-toml/v2 v2.2.4 h1:mye9XuhQ6gvn5h28+VilKrrPoQVanw5PMw/TB0t5Ec4=
-github.com/pelletier/go-toml/v2 v2.2.4/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8t9Stt+h56mCY=
+github.com/pelletier/go-toml/v2 v2.1.1 h1:LWAJwfNvjQZCFIDKWYQaM62NcYeYViCmWIwmOStowAI=
+github.com/pelletier/go-toml/v2 v2.1.1/go.mod h1:tJU2Z3ZkXwnxa4DPO899bsyIoywizdUvyaeZurnPPDc=
 github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e h1:aoZm08cpOy4WuID//EZDgcC4zIxODThtZNPirFr42+A=
 github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e/go.mod h1:pJLUxLENpZxwdsKMEsNbx1VGcRFpLqf3715MtcvvzbA=
 github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
 github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
-github.com/spf13/cobra v1.10.1 h1:lJeBwCfmrnXthfAupyUTzJ/J4Nc1RsHC/mSRU2dll/s=
-github.com/spf13/cobra v1.10.1/go.mod h1:7SmJGaTHFVBY0jW4NXGluQoLvhqFQM+6XSKD+P4XaB0=
-github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
-github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk=
-github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
+github.com/spf13/cobra v1.8.0 h1:7aJaZx1B85qltLMc546zn58BxxfZdR/W22ej9CFoEf0=
+github.com/spf13/cobra v1.8.0/go.mod h1:WXLWApfZ71AjXPya3WOlMsY9yMs7YeiHhFVlvLyhcho=
+github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
+github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
 github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
-github.com/stretchr/testify v1.7.0 h1:nwc3DEeHmmLAfoZucVR881uASk0Mfjw8xYJ99tb5CcY=
+github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
+github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
 github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
+github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
+github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
+github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk=
+github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
 github.com/yuin/gopher-lua v1.1.1 h1:kYKnWBjvbNP4XLT3+bPEwAXJx262OhaHDWDVOPjL46M=
 github.com/yuin/gopher-lua v1.1.1/go.mod h1:GBR0iDaNXjAgGg9zfCvksxSRnQx76gclCIb7kdAd1Pw=
-go.yaml.in/yaml/v4 v4.0.0-rc.3 h1:3h1fjsh1CTAPjW7q/EMe+C8shx5d8ctzZTrLcs/j8Go=
-go.yaml.in/yaml/v4 v4.0.0-rc.3/go.mod h1:aZqd9kCMsGL7AuUv/m/PvWLdg5sjJsZ4oHDEnfPPfY0=
-golang.org/x/net v0.47.0 h1:Mx+4dIFzqraBXUugkia1OOvlD6LemFo1ALMHjrXDOhY=
-golang.org/x/net v0.47.0/go.mod h1:/jNxtkgq5yWUGYkaZGqo27cfGZ1c5Nen03aYrrKpVRU=
+golang.org/x/crypto v0.21.0 h1:X31++rzVUdKhX5sWmSOFZxx8UW/ldWx55cbf08iNAMA=
+golang.org/x/crypto v0.21.0/go.mod h1:0BP7YvVV9gBbVKyeTG0Gyn+gZm94bibOW5BjDEYAOMs=
+golang.org/x/net v0.22.0 h1:9sGLhx7iRIHEiX0oAJ3MRZMUCElJgy7Br1nO+AMN3Tc=
+golang.org/x/net v0.22.0/go.mod h1:JKghWKKOSdJwpW2GEx0Ja7fmaKnMsbu+MWVZTokSYmg=
+golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.38.0 h1:3yZWxaJjBmCWXqhN1qh02AkOnCQ1poK6oF+a7xWL6Gc=
-golang.org/x/sys v0.38.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
-golang.org/x/text v0.31.0 h1:aC8ghyu4JhP8VojJ2lEHBnochRno1sgL6nEi9WGFGMM=
-golang.org/x/text v0.31.0/go.mod h1:tKRAlv61yKIjGGHX/4tP1LTbc13YSec1pxVEWXzfoeM=
+golang.org/x/sys v0.18.0 h1:DBdB3niSjOA/O0blCZBqDefyWNYveAYMNF1Wum0DYQ4=
+golang.org/x/sys v0.18.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
+golang.org/x/text v0.14.0 h1:ScX5w1eTa3QqT8oi6+ziP7dTV1S2+ALU0bI+0zXKWiQ=
+golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
+golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 h1:H2TDz8ibqkAF6YGhCdN3jS9O0/s90v0rJh3X/OLHEUk=
+golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2/go.mod h1:K8+ghG5WaK9qNqU5K3HdILfMLy1f3aNYFI/wnl100a8=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/op/go-logging.v1 v1.0.0-20160211212156-b2cb9fa56473 h1:6D+BvnJ/j6e222UW8s2qTSe3wGBtvo0MbVQG/c5k8RE=
 gopkg.in/op/go-logging.v1 v1.0.0-20160211212156-b2cb9fa56473/go.mod h1:N1eN2tsCx0Ydtgjl4cqmbRCsY4/+z4cYDeqwZTk6zog=

pkg/properties/decoder_properties.go

@@ -1,6 +1,4 @@
-//go:build !yq_noprops
-
-package yqlib
+package properties
 
 import (
 	"bytes"
@@ -10,16 +8,17 @@ import (
 	"strings"
 
 	"github.com/magiconair/properties"
+	"github.com/mikefarah/yq/v4/pkg/yqlib"
 )
 
 type propertiesDecoder struct {
 	reader   io.Reader
 	finished bool
-	d        DataTreeNavigator
+	d        yqlib.DataTreeNavigator
 }
 
-func NewPropertiesDecoder() Decoder {
-	return &propertiesDecoder{d: NewDataTreeNavigator(), finished: false}
+func NewPropertiesDecoder() yqlib.Decoder {
+	return &propertiesDecoder{d: yqlib.NewDataTreeNavigator(), finished: false}
 }
 
 func (dec *propertiesDecoder) Init(reader io.Reader) error {
@@ -49,31 +48,18 @@ func (dec *propertiesDecoder) processComment(c string) string {
 	return "# " + c
 }
 
-func (dec *propertiesDecoder) applyPropertyComments(context Context, path []interface{}, comments []string) error {
-	assignmentOp := &Operation{OperationType: assignOpType, Preferences: assignPreferences{}}
-
-	rhsCandidateNode := &CandidateNode{
+func (dec *propertiesDecoder) applyPropertyComments(context yqlib.Context, path []interface{}, comments []string) error {
+	rhsCandidateNode := &yqlib.CandidateNode{
 		Tag:         "!!str",
 		Value:       fmt.Sprintf("%v", path[len(path)-1]),
 		HeadComment: dec.processComment(strings.Join(comments, "\n")),
-		Kind:        ScalarNode,
+		Kind:        yqlib.ScalarNode,
 	}
-
-	rhsCandidateNode.Tag = rhsCandidateNode.guessTagFromCustomType()
-
-	rhsOp := &Operation{OperationType: referenceOpType, CandidateNode: rhsCandidateNode}
-
-	assignmentOpNode := &ExpressionNode{
-		Operation: assignmentOp,
-		LHS:       createTraversalTree(path, traversePreferences{}, true),
-		RHS:       &ExpressionNode{Operation: rhsOp},
-	}
-
-	_, err := dec.d.GetMatchingNodes(context, assignmentOpNode)
-	return err
+	rhsCandidateNode.Tag = rhsCandidateNode.GuessTagFromCustomType()
+	return dec.d.DeeplyAssignKey(context, path, rhsCandidateNode)
 }
 
-func (dec *propertiesDecoder) applyProperty(context Context, properties *properties.Properties, key string) error {
+func (dec *propertiesDecoder) applyProperty(context yqlib.Context, properties *properties.Properties, key string) error {
 	value, _ := properties.Get(key)
 	path := parsePropKey(key)
 
@@ -85,13 +71,13 @@ func (dec *propertiesDecoder) applyProperty(context Context, properties *propert
 		}
 	}
 
-	rhsNode := createStringScalarNode(value)
-	rhsNode.Tag = rhsNode.guessTagFromCustomType()
+	rhsNode := yqlib.CreateStringScalarNode(value)
+	rhsNode.Tag = rhsNode.GuessTagFromCustomType()
 
 	return dec.d.DeeplyAssign(context, path, rhsNode)
 }
 
-func (dec *propertiesDecoder) Decode() (*CandidateNode, error) {
+func (dec *propertiesDecoder) Decode() (*yqlib.CandidateNode, error) {
 	if dec.finished {
 		return nil, io.EOF
 	}
@@ -110,12 +96,12 @@ func (dec *propertiesDecoder) Decode() (*CandidateNode, error) {
 	}
 	properties.DisableExpansion = true
 
-	rootMap := &CandidateNode{
-		Kind: MappingNode,
+	rootMap := &yqlib.CandidateNode{
+		Kind: yqlib.MappingNode,
 		Tag:  "!!map",
 	}
 
-	context := Context{}
+	context := yqlib.Context{}
 	context = context.SingleChildContext(rootMap)
 
 	for _, key := range properties.Keys() {

pkg/properties/encoder_properties.go

@@ -1,6 +1,4 @@
-//go:build !yq_noprops
-
-package yqlib
+package properties
 
 import (
 	"bufio"
@@ -10,13 +8,14 @@ import (
 	"strings"
 
 	"github.com/magiconair/properties"
+	"github.com/mikefarah/yq/v4/pkg/yqlib"
 )
 
 type propertiesEncoder struct {
 	prefs PropertiesPreferences
 }
 
-func NewPropertiesEncoder(prefs PropertiesPreferences) Encoder {
+func NewPropertiesEncoder(prefs PropertiesPreferences) yqlib.Encoder {
 	return &propertiesEncoder{
 		prefs: prefs,
 	}
@@ -45,7 +44,7 @@ func (pe *propertiesEncoder) PrintLeadingContent(writer io.Writer, content strin
 			}
 
 		} else {
-			if err := writeString(writer, readline); err != nil {
+			if err := yqlib.WriteString(writer, readline); err != nil {
 				return err
 			}
 		}
@@ -53,7 +52,7 @@ func (pe *propertiesEncoder) PrintLeadingContent(writer io.Writer, content strin
 		if errors.Is(errReading, io.EOF) {
 			if readline != "" {
 				// the last comment we read didn't have a newline, put one in
-				if err := writeString(writer, "\n"); err != nil {
+				if err := yqlib.WriteString(writer, "\n"); err != nil {
 					return err
 				}
 			}
@@ -63,13 +62,13 @@ func (pe *propertiesEncoder) PrintLeadingContent(writer io.Writer, content strin
 	return nil
 }
 
-func (pe *propertiesEncoder) Encode(writer io.Writer, node *CandidateNode) error {
+func (pe *propertiesEncoder) Encode(writer io.Writer, node *yqlib.CandidateNode) error {
 
-	if node.Kind == ScalarNode {
-		return writeString(writer, node.Value+"\n")
+	if node.Kind == yqlib.ScalarNode {
+		return yqlib.WriteString(writer, node.Value+"\n")
 	}
 
-	mapKeysToStrings(node)
+	node.ConvertKeysToStrings()
 	p := properties.NewProperties()
 	p.WriteSeparator = pe.prefs.KeyValueSeparator
 	err := pe.doEncode(p, node, "", nil)
@@ -81,19 +80,19 @@ func (pe *propertiesEncoder) Encode(writer io.Writer, node *CandidateNode) error
 	return err
 }
 
-func (pe *propertiesEncoder) doEncode(p *properties.Properties, node *CandidateNode, path string, keyNode *CandidateNode) error {
+func (pe *propertiesEncoder) doEncode(p *properties.Properties, node *yqlib.CandidateNode, path string, keyNode *yqlib.CandidateNode) error {
 
 	comments := ""
 	if keyNode != nil {
 		// include the key node comments if present
-		comments = headAndLineComment(keyNode)
+		comments = keyNode.CleanHeadAndLineComment()
 	}
-	comments = comments + headAndLineComment(node)
+	comments = comments + node.CleanHeadAndLineComment()
 	commentsWithSpaces := strings.ReplaceAll(comments, "\n", "\n ")
 	p.SetComments(path, strings.Split(commentsWithSpaces, "\n"))
 
 	switch node.Kind {
-	case ScalarNode:
+	case yqlib.ScalarNode:
 		var nodeValue string
 		if pe.prefs.UnwrapScalar || !strings.Contains(node.Value, " ") {
 			nodeValue = node.Value
@@ -102,14 +101,14 @@ func (pe *propertiesEncoder) doEncode(p *properties.Properties, node *CandidateN
 		}
 		_, _, err := p.Set(path, nodeValue)
 		return err
-	case SequenceNode:
+	case yqlib.SequenceNode:
 		return pe.encodeArray(p, node.Content, path)
-	case MappingNode:
+	case yqlib.MappingNode:
 		return pe.encodeMap(p, node.Content, path)
-	case AliasNode:
+	case yqlib.AliasNode:
 		return pe.doEncode(p, node.Alias, path, nil)
 	default:
-		return fmt.Errorf("unsupported node %v", node.Tag)
+		return fmt.Errorf("Unsupported node %v", node.Tag)
 	}
 }
 
@@ -127,7 +126,7 @@ func (pe *propertiesEncoder) appendPath(path string, key interface{}) string {
 	return fmt.Sprintf("%v.%v", path, key)
 }
 
-func (pe *propertiesEncoder) encodeArray(p *properties.Properties, kids []*CandidateNode, path string) error {
+func (pe *propertiesEncoder) encodeArray(p *properties.Properties, kids []*yqlib.CandidateNode, path string) error {
 	for index, child := range kids {
 		err := pe.doEncode(p, child, pe.appendPath(path, index), nil)
 		if err != nil {
@@ -137,7 +136,7 @@ func (pe *propertiesEncoder) encodeArray(p *properties.Properties, kids []*Candi
 	return nil
 }
 
-func (pe *propertiesEncoder) encodeMap(p *properties.Properties, kids []*CandidateNode, path string) error {
+func (pe *propertiesEncoder) encodeMap(p *properties.Properties, kids []*yqlib.CandidateNode, path string) error {
 	for index := 0; index < len(kids); index = index + 2 {
 		key := kids[index]
 		value := kids[index+1]

pkg/properties/encoder_properties_test.go

@@ -1,4 +1,4 @@
-package yqlib
+package properties
 
 import (
 	"bufio"

pkg/properties/properties.go

@@ -0,0 +1,45 @@
+package properties
+
+import "github.com/mikefarah/yq/v4/pkg/yqlib"
+
+type PropertiesPreferences struct {
+	UnwrapScalar      bool
+	KeyValueSeparator string
+	UseArrayBrackets  bool
+}
+
+func NewDefaultPropertiesPreferences() PropertiesPreferences {
+	return PropertiesPreferences{
+		UnwrapScalar:      true,
+		KeyValueSeparator: " = ",
+		UseArrayBrackets:  false,
+	}
+}
+
+func (p *PropertiesPreferences) Copy() PropertiesPreferences {
+	return PropertiesPreferences{
+		UnwrapScalar:      p.UnwrapScalar,
+		KeyValueSeparator: p.KeyValueSeparator,
+		UseArrayBrackets:  p.UseArrayBrackets,
+	}
+}
+
+var PropertiesFormat = &yqlib.Format{"props", []string{"p", "properties"}, "properties",
+	func() yqlib.Encoder { return NewPropertiesEncoder(ConfiguredPropertiesPreferences) },
+	func() yqlib.Decoder { return NewPropertiesDecoder() },
+	nil,
+}
+
+var propertyYqRules = []*yqlib.ParticipleYqRule{
+	{"PropertiesDecode", `from_?props|@propsd`, yqlib.CreateDecodeOpYqAction(PropertiesFormat), 0},
+	{"PropsEncode", `to_?props|@props`, yqlib.CreateEncodeOpYqAction(PropertiesFormat, 2), 0},
+	{"LoadProperties", `load_?props`, yqlib.CreateLoadOpYqAction(NewPropertiesDecoder()), 0},
+}
+
+func RegisterPropertiesFormat() {
+	yqlib.RegisterFormat(PropertiesFormat)
+	yqlib.RegisterRules(propertyYqRules)
+
+}
+
+var ConfiguredPropertiesPreferences = NewDefaultPropertiesPreferences()

pkg/xml/decoder_xml.go

@@ -1,6 +1,6 @@
 //go:build !yq_noxml
 
-package yqlib
+package xml
 
 import (
 	"encoding/xml"
@@ -11,6 +11,7 @@ import (
 	"strings"
 	"unicode"
 
+	"github.com/mikefarah/yq/v4/pkg/yqlib"
 	"golang.org/x/net/html/charset"
 )
 
@@ -21,7 +22,7 @@ type xmlDecoder struct {
 	prefs        XmlPreferences
 }
 
-func NewXMLDecoder(prefs XmlPreferences) Decoder {
+func NewXMLDecoder(prefs XmlPreferences) yqlib.Decoder {
 	return &xmlDecoder{
 		finished: false,
 		prefs:    prefs,
@@ -35,8 +36,8 @@ func (dec *xmlDecoder) Init(reader io.Reader) error {
 	return nil
 }
 
-func (dec *xmlDecoder) createSequence(nodes []*xmlNode) (*CandidateNode, error) {
-	yamlNode := &CandidateNode{Kind: SequenceNode, Tag: "!!seq"}
+func (dec *xmlDecoder) createSequence(nodes []*xmlNode) (*yqlib.CandidateNode, error) {
+	yamlNode := &yqlib.CandidateNode{Kind: yqlib.SequenceNode, Tag: "!!seq"}
 	for _, child := range nodes {
 		yamlChild, err := dec.convertToYamlNode(child)
 		if err != nil {
@@ -63,14 +64,14 @@ func (dec *xmlDecoder) processComment(c string) string {
 	return replacement
 }
 
-func (dec *xmlDecoder) createMap(n *xmlNode) (*CandidateNode, error) {
-	log.Debug("createMap: headC: %v, lineC: %v, footC: %v", n.HeadComment, n.LineComment, n.FootComment)
-	yamlNode := &CandidateNode{Kind: MappingNode, Tag: "!!map"}
+func (dec *xmlDecoder) createMap(n *xmlNode) (*yqlib.CandidateNode, error) {
+	yqlib.GetLogger().Debug("createMap: headC: %v, lineC: %v, footC: %v", n.HeadComment, n.LineComment, n.FootComment)
+	yamlNode := &yqlib.CandidateNode{Kind: yqlib.MappingNode, Tag: "!!map"}
 
 	if len(n.Data) > 0 {
-		log.Debugf("creating content node for map: %v", dec.prefs.ContentName)
+		yqlib.GetLogger().Debugf("creating content node for map: %v", dec.prefs.ContentName)
 		label := dec.prefs.ContentName
-		labelNode := createScalarNode(label, label)
+		labelNode := yqlib.CreateScalarNode(label, label)
 		labelNode.HeadComment = dec.processComment(n.HeadComment)
 		labelNode.LineComment = dec.processComment(n.LineComment)
 		labelNode.FootComment = dec.processComment(n.FootComment)
@@ -80,19 +81,19 @@ func (dec *xmlDecoder) createMap(n *xmlNode) (*CandidateNode, error) {
 	for i, keyValuePair := range n.Children {
 		label := keyValuePair.K
 		children := keyValuePair.V
-		labelNode := createScalarNode(label, label)
-		var valueNode *CandidateNode
+		labelNode := yqlib.CreateScalarNode(label, label)
+		var valueNode *yqlib.CandidateNode
 		var err error
 
 		if i == 0 {
-			log.Debugf("head comment here")
+			yqlib.GetLogger().Debugf("head comment here")
 			labelNode.HeadComment = dec.processComment(n.HeadComment)
 
 		}
-		log.Debugf("label=%v, i=%v, keyValuePair.FootComment: %v", label, i, keyValuePair.FootComment)
+		yqlib.GetLogger().Debugf("label=%v, i=%v, keyValuePair.FootComment: %v", label, i, keyValuePair.FootComment)
 		labelNode.FootComment = dec.processComment(keyValuePair.FootComment)
 
-		log.Debug("len of children in %v is %v", label, len(children))
+		yqlib.GetLogger().Debug("len of children in %v is %v", label, len(children))
 		if len(children) > 1 {
 			valueNode, err = dec.createSequence(children)
 			if err != nil {
@@ -105,7 +106,7 @@ func (dec *xmlDecoder) createMap(n *xmlNode) (*CandidateNode, error) {
 			if len(children[0].Children) == 0 && children[0].HeadComment != "" {
 				if len(children[0].Data) > 0 {
 
-					log.Debug("scalar comment hack, currentlabel [%v]", labelNode.HeadComment)
+					yqlib.GetLogger().Debug("scalar comment hack, currentlabel [%v]", labelNode.HeadComment)
 					labelNode.HeadComment = joinComments([]string{labelNode.HeadComment, strings.TrimSpace(children[0].HeadComment)}, "\n")
 					children[0].HeadComment = ""
 				} else {
@@ -125,33 +126,33 @@ func (dec *xmlDecoder) createMap(n *xmlNode) (*CandidateNode, error) {
 	return yamlNode, nil
 }
 
-func (dec *xmlDecoder) createValueNodeFromData(values []string) *CandidateNode {
+func (dec *xmlDecoder) createValueNodeFromData(values []string) *yqlib.CandidateNode {
 	switch len(values) {
 	case 0:
-		return createScalarNode(nil, "")
+		return yqlib.CreateScalarNode(nil, "")
 	case 1:
-		return createScalarNode(values[0], values[0])
+		return yqlib.CreateScalarNode(values[0], values[0])
 	default:
-		content := make([]*CandidateNode, 0)
+		content := make([]*yqlib.CandidateNode, 0)
 		for _, value := range values {
-			content = append(content, createScalarNode(value, value))
+			content = append(content, yqlib.CreateScalarNode(value, value))
 		}
-		return &CandidateNode{
-			Kind:    SequenceNode,
+		return &yqlib.CandidateNode{
+			Kind:    yqlib.SequenceNode,
 			Tag:     "!!seq",
 			Content: content,
 		}
 	}
 }
 
-func (dec *xmlDecoder) convertToYamlNode(n *xmlNode) (*CandidateNode, error) {
+func (dec *xmlDecoder) convertToYamlNode(n *xmlNode) (*yqlib.CandidateNode, error) {
 	if len(n.Children) > 0 {
 		return dec.createMap(n)
 	}
 
 	scalar := dec.createValueNodeFromData(n.Data)
 
-	log.Debug("scalar (%v), headC: %v, lineC: %v, footC: %v", scalar.Tag, n.HeadComment, n.LineComment, n.FootComment)
+	yqlib.GetLogger().Debug("scalar (%v), headC: %v, lineC: %v, footC: %v", scalar.Tag, n.HeadComment, n.LineComment, n.FootComment)
 	scalar.HeadComment = dec.processComment(n.HeadComment)
 	scalar.LineComment = dec.processComment(n.LineComment)
 	if scalar.Tag == "!!seq" {
@@ -164,7 +165,7 @@ func (dec *xmlDecoder) convertToYamlNode(n *xmlNode) (*CandidateNode, error) {
 	return scalar, nil
 }
 
-func (dec *xmlDecoder) Decode() (*CandidateNode, error) {
+func (dec *xmlDecoder) Decode() (*yqlib.CandidateNode, error) {
 	if dec.finished {
 		return nil, io.EOF
 	}
@@ -211,17 +212,17 @@ func (n *xmlNode) AddChild(s string, c *xmlNode) {
 	if n.Children == nil {
 		n.Children = make([]*xmlChildrenKv, 0)
 	}
-	log.Debug("looking for %s", s)
+	yqlib.GetLogger().Debug("looking for %s", s)
 	// see if we can find an existing entry to add to
 	for _, childEntry := range n.Children {
 		if childEntry.K == s {
-			log.Debug("found it, appending an entry%s", s)
+			yqlib.GetLogger().Debug("found it, appending an entry%s", s)
 			childEntry.V = append(childEntry.V, c)
-			log.Debug("yay len of children in %v is %v", s, len(childEntry.V))
+			yqlib.GetLogger().Debug("yay len of children in %v is %v", s, len(childEntry.V))
 			return
 		}
 	}
-	log.Debug("not there, making a new one %s", s)
+	yqlib.GetLogger().Debug("not there, making a new one %s", s)
 	n.Children = append(n.Children, &xmlChildrenKv{K: s, V: []*xmlNode{c}})
 }
 
@@ -267,19 +268,13 @@ func (dec *xmlDecoder) decodeXML(root *xmlNode) error {
 
 		switch se := t.(type) {
 		case xml.StartElement:
-			log.Debug("start element %v", se.Name.Local)
+			yqlib.GetLogger().Debug("start element %v", se.Name.Local)
 			elem.state = "started"
 			// Build new a new current element and link it to its parent
-			var label = se.Name.Local
-			if dec.prefs.KeepNamespace {
-				if se.Name.Space != "" {
-					label = se.Name.Space + ":" + se.Name.Local
-				}
-			}
 			elem = &element{
 				parent: elem,
 				n:      &xmlNode{},
-				label:  label,
+				label:  se.Name.Local,
 			}
 
 			// Extract attributes as children
@@ -302,14 +297,14 @@ func (dec *xmlDecoder) decodeXML(root *xmlNode) error {
 			if len(newBit) > 0 {
 				elem.n.Data = append(elem.n.Data, newBit)
 				elem.state = "chardata"
-				log.Debug("chardata [%v] for %v", elem.n.Data, elem.label)
+				yqlib.GetLogger().Debug("chardata [%v] for %v", elem.n.Data, elem.label)
 			}
 		case xml.EndElement:
 			if elem == nil {
-				log.Debug("no element, probably bad xml")
+				yqlib.GetLogger().Debug("no element, probably bad xml")
 				continue
 			}
-			log.Debug("end element %v", elem.label)
+			yqlib.GetLogger().Debug("end element %v", elem.label)
 			elem.state = "finished"
 			// And add it to its parent list
 			if elem.parent != nil {
@@ -321,15 +316,14 @@ func (dec *xmlDecoder) decodeXML(root *xmlNode) error {
 		case xml.Comment:
 
 			commentStr := string(xml.CharData(se))
-			switch elem.state {
-			case "started":
+			if elem.state == "started" {
 				applyFootComment(elem, commentStr)
 
-			case "chardata":
-				log.Debug("got a line comment for (%v) %v: [%v]", elem.state, elem.label, commentStr)
+			} else if elem.state == "chardata" {
+				yqlib.GetLogger().Debug("got a line comment for (%v) %v: [%v]", elem.state, elem.label, commentStr)
 				elem.n.LineComment = joinComments([]string{elem.n.LineComment, commentStr}, " ")
-			default:
-				log.Debug("got a head comment for (%v) %v: [%v]", elem.state, elem.label, commentStr)
+			} else {
+				yqlib.GetLogger().Debug("got a head comment for (%v) %v: [%v]", elem.state, elem.label, commentStr)
 				elem.n.HeadComment = joinComments([]string{elem.n.HeadComment, commentStr}, " ")
 			}
 
@@ -354,7 +348,7 @@ func applyFootComment(elem *element, commentStr string) {
 	if len(elem.n.Children) > 0 {
 		lastChildIndex := len(elem.n.Children) - 1
 		childKv := elem.n.Children[lastChildIndex]
-		log.Debug("got a foot comment, putting on last child for %v: [%v]", childKv.K, commentStr)
+		yqlib.GetLogger().Debug("got a foot comment, putting on last child for %v: [%v]", childKv.K, commentStr)
 		// if it's an array of scalars, put the foot comment on the scalar itself
 		if len(childKv.V) > 0 && len(childKv.V[0].Children) == 0 {
 			nodeToUpdate := childKv.V[len(childKv.V)-1]
@@ -363,7 +357,7 @@ func applyFootComment(elem *element, commentStr string) {
 			childKv.FootComment = joinComments([]string{elem.n.FootComment, commentStr}, " ")
 		}
 	} else {
-		log.Debug("got a foot comment for %v: [%v]", elem.label, commentStr)
+		yqlib.GetLogger().Debug("got a foot comment for %v: [%v]", elem.label, commentStr)
 		elem.n.FootComment = joinComments([]string{elem.n.FootComment, commentStr}, " ")
 	}
 }

pkg/xml/encoder_xml.go

@@ -1,6 +1,6 @@
 //go:build !yq_noxml
 
-package yqlib
+package xml
 
 import (
 	"encoding/xml"
@@ -8,6 +8,8 @@ import (
 	"io"
 	"regexp"
 	"strings"
+
+	"github.com/mikefarah/yq/v4/pkg/yqlib"
 )
 
 type xmlEncoder struct {
@@ -17,7 +19,7 @@ type xmlEncoder struct {
 	leadingContent string
 }
 
-func NewXMLEncoder(prefs XmlPreferences) Encoder {
+func NewXMLEncoder(prefs XmlPreferences) yqlib.Encoder {
 	var indentString = ""
 
 	for index := 0; index < prefs.Indent; index++ {
@@ -39,7 +41,7 @@ func (e *xmlEncoder) PrintLeadingContent(_ io.Writer, content string) error {
 	return nil
 }
 
-func (e *xmlEncoder) Encode(writer io.Writer, node *CandidateNode) error {
+func (e *xmlEncoder) Encode(writer io.Writer, node *yqlib.CandidateNode) error {
 	encoder := xml.NewEncoder(writer)
 	// hack so we can manually add newlines to procInst and directives
 	e.writer = writer
@@ -59,7 +61,7 @@ func (e *xmlEncoder) Encode(writer io.Writer, node *CandidateNode) error {
 					return err
 				}
 				if _, err := e.writer.Write([]byte("\n")); err != nil {
-					log.Warning("Unable to write newline, skipping: %w", err)
+					yqlib.GetLogger().Warning("Unable to write newline, skipping: %w", err)
 				}
 			}
 		}
@@ -79,12 +81,12 @@ func (e *xmlEncoder) Encode(writer io.Writer, node *CandidateNode) error {
 	}
 
 	switch node.Kind {
-	case MappingNode:
+	case yqlib.MappingNode:
 		err := e.encodeTopLevelMap(encoder, node)
 		if err != nil {
 			return err
 		}
-	case ScalarNode:
+	case yqlib.ScalarNode:
 		var charData xml.CharData = []byte(node.Value)
 		err := encoder.EncodeToken(charData)
 		if err != nil {
@@ -99,8 +101,8 @@ func (e *xmlEncoder) Encode(writer io.Writer, node *CandidateNode) error {
 
 }
 
-func (e *xmlEncoder) encodeTopLevelMap(encoder *xml.Encoder, node *CandidateNode) error {
-	err := e.encodeComment(encoder, headAndLineComment(node))
+func (e *xmlEncoder) encodeTopLevelMap(encoder *xml.Encoder, node *yqlib.CandidateNode) error {
+	err := e.encodeComment(encoder, node.CleanHeadAndLineComment())
 	if err != nil {
 		return err
 	}
@@ -109,12 +111,12 @@ func (e *xmlEncoder) encodeTopLevelMap(encoder *xml.Encoder, node *CandidateNode
 		value := node.Content[i+1]
 
 		start := xml.StartElement{Name: xml.Name{Local: key.Value}}
-		log.Debugf("comments of key %v", key.Value)
-		err := e.encodeComment(encoder, headAndLineComment(key))
+		yqlib.GetLogger().Debugf("comments of key %v", key.Value)
+		err := e.encodeComment(encoder, key.CleanHeadAndLineComment())
 		if err != nil {
 			return err
 		}
-		if headAndLineComment(key) != "" {
+		if key.CleanHeadAndLineComment() != "" {
 			var newLine xml.CharData = []byte("\n")
 			err = encoder.EncodeToken(newLine)
 			if err != nil {
@@ -131,7 +133,7 @@ func (e *xmlEncoder) encodeTopLevelMap(encoder *xml.Encoder, node *CandidateNode
 				return err
 			}
 			if _, err := e.writer.Write([]byte("\n")); err != nil {
-				log.Warning("Unable to write newline, skipping: %w", err)
+				yqlib.GetLogger().Warning("Unable to write newline, skipping: %w", err)
 			}
 		} else if key.Value == e.prefs.DirectiveName {
 			var directive xml.Directive = []byte(value.Value)
@@ -139,48 +141,48 @@ func (e *xmlEncoder) encodeTopLevelMap(encoder *xml.Encoder, node *CandidateNode
 				return err
 			}
 			if _, err := e.writer.Write([]byte("\n")); err != nil {
-				log.Warning("Unable to write newline, skipping: %w", err)
+				yqlib.GetLogger().Warning("Unable to write newline, skipping: %w", err)
 			}
 		} else {
 
-			log.Debugf("recursing")
+			yqlib.GetLogger().Debugf("recursing")
 
 			err = e.doEncode(encoder, value, start)
 			if err != nil {
 				return err
 			}
 		}
-		err = e.encodeComment(encoder, footComment(key))
+		err = e.encodeComment(encoder, key.CleanFootComment())
 		if err != nil {
 			return err
 		}
 	}
-	return e.encodeComment(encoder, footComment(node))
+	return e.encodeComment(encoder, node.CleanFootComment())
 }
 
-func (e *xmlEncoder) encodeStart(encoder *xml.Encoder, node *CandidateNode, start xml.StartElement) error {
+func (e *xmlEncoder) encodeStart(encoder *xml.Encoder, node *yqlib.CandidateNode, start xml.StartElement) error {
 	err := encoder.EncodeToken(start)
 	if err != nil {
 		return err
 	}
-	return e.encodeComment(encoder, headComment(node))
+	return e.encodeComment(encoder, node.CleanHeadComment())
 }
 
-func (e *xmlEncoder) encodeEnd(encoder *xml.Encoder, node *CandidateNode, start xml.StartElement) error {
+func (e *xmlEncoder) encodeEnd(encoder *xml.Encoder, node *yqlib.CandidateNode, start xml.StartElement) error {
 	err := encoder.EncodeToken(start.End())
 	if err != nil {
 		return err
 	}
-	return e.encodeComment(encoder, footComment(node))
+	return e.encodeComment(encoder, node.CleanFootComment())
 }
 
-func (e *xmlEncoder) doEncode(encoder *xml.Encoder, node *CandidateNode, start xml.StartElement) error {
+func (e *xmlEncoder) doEncode(encoder *xml.Encoder, node *yqlib.CandidateNode, start xml.StartElement) error {
 	switch node.Kind {
-	case MappingNode:
+	case yqlib.MappingNode:
 		return e.encodeMap(encoder, node, start)
-	case SequenceNode:
+	case yqlib.SequenceNode:
 		return e.encodeArray(encoder, node, start)
-	case ScalarNode:
+	case yqlib.ScalarNode:
 		err := e.encodeStart(encoder, node, start)
 		if err != nil {
 			return err
@@ -192,7 +194,7 @@ func (e *xmlEncoder) doEncode(encoder *xml.Encoder, node *CandidateNode, start x
 			return err
 		}
 
-		if err = e.encodeComment(encoder, lineComment(node)); err != nil {
+		if err = e.encodeComment(encoder, node.CleanLineComment()); err != nil {
 			return err
 		}
 
@@ -207,13 +209,13 @@ var chompRegexp = regexp.MustCompile(`\n$`)
 
 func (e *xmlEncoder) encodeComment(encoder *xml.Encoder, commentStr string) error {
 	if commentStr != "" {
-		log.Debugf("got comment [%v]", commentStr)
+		yqlib.GetLogger().Debugf("got comment [%v]", commentStr)
 		// multi line string
 		if len(commentStr) > 2 && strings.Contains(commentStr[1:len(commentStr)-1], "\n") {
 			commentStr = chompRegexp.ReplaceAllString(commentStr, "")
-			log.Debugf("chompRegexp [%v]", commentStr)
+			yqlib.GetLogger().Debugf("chompRegexp [%v]", commentStr)
 			commentStr = xmlEncodeMultilineCommentRegex.ReplaceAllString(commentStr, "$1$2")
-			log.Debugf("processed multiline [%v]", commentStr)
+			yqlib.GetLogger().Debugf("processed multiline [%v]", commentStr)
 			// if the first line is non blank, add a space
 			if commentStr[0] != '\n' && commentStr[0] != ' ' {
 				commentStr = " " + commentStr
@@ -225,9 +227,9 @@ func (e *xmlEncoder) encodeComment(encoder *xml.Encoder, commentStr string) erro
 
 		if !strings.HasSuffix(commentStr, " ") && !strings.HasSuffix(commentStr, "\n") {
 			commentStr = commentStr + " "
-			log.Debugf("added suffix [%v]", commentStr)
+			yqlib.GetLogger().Debugf("added suffix [%v]", commentStr)
 		}
-		log.Debugf("encoding comment [%v]", commentStr)
+		yqlib.GetLogger().Debugf("encoding comment [%v]", commentStr)
 
 		var comment xml.Comment = []byte(commentStr)
 		err := encoder.EncodeToken(comment)
@@ -238,9 +240,9 @@ func (e *xmlEncoder) encodeComment(encoder *xml.Encoder, commentStr string) erro
 	return nil
 }
 
-func (e *xmlEncoder) encodeArray(encoder *xml.Encoder, node *CandidateNode, start xml.StartElement) error {
+func (e *xmlEncoder) encodeArray(encoder *xml.Encoder, node *yqlib.CandidateNode, start xml.StartElement) error {
 
-	if err := e.encodeComment(encoder, headAndLineComment(node)); err != nil {
+	if err := e.encodeComment(encoder, node.CleanHeadAndLineComment()); err != nil {
 		return err
 	}
 
@@ -250,7 +252,7 @@ func (e *xmlEncoder) encodeArray(encoder *xml.Encoder, node *CandidateNode, star
 			return err
 		}
 	}
-	return e.encodeComment(encoder, footComment(node))
+	return e.encodeComment(encoder, node.CleanFootComment())
 }
 
 func (e *xmlEncoder) isAttribute(name string) bool {
@@ -260,8 +262,8 @@ func (e *xmlEncoder) isAttribute(name string) bool {
 		!strings.HasPrefix(name, e.prefs.ProcInstPrefix)
 }
 
-func (e *xmlEncoder) encodeMap(encoder *xml.Encoder, node *CandidateNode, start xml.StartElement) error {
-	log.Debug("its a map")
+func (e *xmlEncoder) encodeMap(encoder *xml.Encoder, node *yqlib.CandidateNode, start xml.StartElement) error {
+	yqlib.GetLogger().Debug("its a map")
 
 	//first find all the attributes and put them on the start token
 	for i := 0; i < len(node.Content); i += 2 {
@@ -269,7 +271,7 @@ func (e *xmlEncoder) encodeMap(encoder *xml.Encoder, node *CandidateNode, start
 		value := node.Content[i+1]
 
 		if e.isAttribute(key.Value) {
-			if value.Kind == ScalarNode {
+			if value.Kind == yqlib.ScalarNode {
 				attributeName := strings.Replace(key.Value, e.prefs.AttributePrefix, "", 1)
 				start.Attr = append(start.Attr, xml.Attr{Name: xml.Name{Local: attributeName}, Value: value.Value})
 			} else {
@@ -288,7 +290,7 @@ func (e *xmlEncoder) encodeMap(encoder *xml.Encoder, node *CandidateNode, start
 		key := node.Content[i]
 		value := node.Content[i+1]
 
-		err := e.encodeComment(encoder, headAndLineComment(key))
+		err := e.encodeComment(encoder, key.CleanHeadAndLineComment())
 		if err != nil {
 			return err
 		}
@@ -305,7 +307,7 @@ func (e *xmlEncoder) encodeMap(encoder *xml.Encoder, node *CandidateNode, start
 			}
 		} else if key.Value == e.prefs.ContentName {
 			// directly encode the contents
-			err = e.encodeComment(encoder, headAndLineComment(value))
+			err = e.encodeComment(encoder, value.CleanHeadAndLineComment())
 			if err != nil {
 				return err
 			}
@@ -314,7 +316,7 @@ func (e *xmlEncoder) encodeMap(encoder *xml.Encoder, node *CandidateNode, start
 			if err != nil {
 				return err
 			}
-			err = e.encodeComment(encoder, footComment(value))
+			err = e.encodeComment(encoder, value.CleanFootComment())
 			if err != nil {
 				return err
 			}
@@ -325,7 +327,7 @@ func (e *xmlEncoder) encodeMap(encoder *xml.Encoder, node *CandidateNode, start
 				return err
 			}
 		}
-		err = e.encodeComment(encoder, footComment(key))
+		err = e.encodeComment(encoder, key.CleanFootComment())
 		if err != nil {
 			return err
 		}

pkg/xml/xml.go

@@ -1,4 +1,6 @@
-package yqlib
+package xml
+
+import "github.com/mikefarah/yq/v4/pkg/yqlib"
 
 type XmlPreferences struct {
 	Indent          int
@@ -44,3 +46,27 @@ func (p *XmlPreferences) Copy() XmlPreferences {
 }
 
 var ConfiguredXMLPreferences = NewDefaultXmlPreferences()
+
+var XMLFormat = &yqlib.Format{"xml", []string{"x"}, "xml",
+	func() yqlib.Encoder { return NewXMLEncoder(ConfiguredXMLPreferences) },
+	func() yqlib.Decoder { return NewXMLDecoder(ConfiguredXMLPreferences) },
+	func(indent int) yqlib.Encoder {
+		prefs := ConfiguredXMLPreferences.Copy()
+		prefs.Indent = indent
+		return NewXMLEncoder(prefs)
+	},
+}
+
+var xmlYqRules = []*yqlib.ParticipleYqRule{
+	{"XMLEncodeWithIndent", `to_?xml\([0-9]+\)`, yqlib.CreateEncodeYqActionParsingIndent(XMLFormat), 0},
+	{"XmlDecode", `from_?xml|@xmld`, yqlib.CreateDecodeOpYqAction(XMLFormat), 0},
+	{"XMLEncode", `to_?xml`, yqlib.CreateEncodeOpYqAction(XMLFormat, 2), 0},
+	{"XMLEncodeNoIndent", `@xml`, yqlib.CreateEncodeOpYqAction(XMLFormat, 0), 0},
+	{"LoadXML", `load_?xml|xml_?load`, yqlib.CreateLoadOpYqAction(NewXMLDecoder(ConfiguredXMLPreferences)), 0},
+}
+
+func RegisterXmlFormat() {
+	yqlib.RegisterFormat(XMLFormat)
+	yqlib.RegisterRules(xmlYqRules)
+
+}

pkg/xml/xml_test.go

@@ -1,12 +1,13 @@
 //go:build !yq_noxml
 
-package yqlib
+package xml
 
 import (
 	"bufio"
 	"fmt"
 	"testing"
 
+	"github.com/mikefarah/yq/v4/pkg/yqlib"
 	"github.com/mikefarah/yq/v4/test"
 )
 
@@ -188,10 +189,7 @@ above_cat
 `
 
 const inputXMLWithNamespacedAttr = `<?xml version="1.0"?>
-<map xmlns="some-namespace" xmlns:xsi="some-instance" xsi:schemaLocation="some-url">
-  <item foo="bar">baz</item>
-  <xsi:item>foobar</xsi:item>
-</map>
+<map xmlns="some-namespace" xmlns:xsi="some-instance" xsi:schemaLocation="some-url"></map>
 `
 
 const expectedYAMLWithNamespacedAttr = `+p_xml: version="1.0"
@@ -199,10 +197,6 @@ map:
   +@xmlns: some-namespace
   +@xmlns:xsi: some-instance
   +@xsi:schemaLocation: some-url
-  item:
-    +content: baz
-    +@foo: bar
-  xsi:item: foobar
 `
 
 const expectedYAMLWithRawNamespacedAttr = `+p_xml: version="1.0"
@@ -210,21 +204,13 @@ map:
   +@xmlns: some-namespace
   +@xmlns:xsi: some-instance
   +@xsi:schemaLocation: some-url
-  item:
-    +content: baz
-    +@foo: bar
-  xsi:item: foobar
 `
 
 const expectedYAMLWithoutRawNamespacedAttr = `+p_xml: version="1.0"
-some-namespace:map:
+map:
   +@xmlns: some-namespace
   +@xmlns:xsi: some-instance
   +@some-instance:schemaLocation: some-url
-  some-namespace:item:
-    +content: baz
-    +@foo: bar
-  some-instance:item: foobar
 `
 
 const xmlWithCustomDtd = `
@@ -277,7 +263,7 @@ const expectedXmlWithProcInstAndDirectives = `<?xml version="1.0"?>
 </apple>
 `
 
-var xmlScenarios = []formatScenario{
+var xmlScenarios = []yqlib.FormatScenario{
 	{
 		skipDoc:     true,
 		description: "bad xml",
@@ -629,10 +615,10 @@ var xmlScenarios = []formatScenario{
 	},
 }
 
-func testXMLScenario(t *testing.T, s formatScenario) {
+func testXMLScenario(t *testing.T, s yqlib.FormatScenario) {
 	switch s.scenarioType {
 	case "", "decode":
-		yamlPrefs := ConfiguredYamlPreferences.Copy()
+		yamlPrefs := yqlib.ConfiguredYamlPreferences.Copy()
 		yamlPrefs.Indent = 4
 		test.AssertResultWithContext(t, s.expected, mustProcessFormatScenario(s, NewXMLDecoder(ConfiguredXMLPreferences), NewYamlEncoder(yamlPrefs)), s.description)
 	case "encode":
@@ -676,7 +662,7 @@ func testXMLScenario(t *testing.T, s formatScenario) {
 }
 
 func documentXMLScenario(_ *testing.T, w *bufio.Writer, i interface{}) {
-	s := i.(formatScenario)
+	s := i.(FormatScenario)
 
 	if s.skipDoc {
 		return
@@ -700,7 +686,7 @@ func documentXMLScenario(_ *testing.T, w *bufio.Writer, i interface{}) {
 	}
 }
 
-func documentXMLDecodeScenario(w *bufio.Writer, s formatScenario) {
+func documentXMLDecodeScenario(w *bufio.Writer, s FormatScenario) {
 	writeOrPanic(w, fmt.Sprintf("## %v\n", s.description))
 
 	if s.subdescription != "" {
@@ -722,7 +708,7 @@ func documentXMLDecodeScenario(w *bufio.Writer, s formatScenario) {
 	writeOrPanic(w, fmt.Sprintf("```yaml\n%v```\n\n", mustProcessFormatScenario(s, NewXMLDecoder(ConfiguredXMLPreferences), NewYamlEncoder(ConfiguredYamlPreferences))))
 }
 
-func documentXMLDecodeKeepNsScenario(w *bufio.Writer, s formatScenario) {
+func documentXMLDecodeKeepNsScenario(w *bufio.Writer, s FormatScenario) {
 	writeOrPanic(w, fmt.Sprintf("## %v\n", s.description))
 
 	if s.subdescription != "" {
@@ -746,7 +732,7 @@ func documentXMLDecodeKeepNsScenario(w *bufio.Writer, s formatScenario) {
 	writeOrPanic(w, fmt.Sprintf("```xml\n%v```\n\n", mustProcessFormatScenario(s, NewXMLDecoder(prefsWithout), NewXMLEncoder(prefsWithout))))
 }
 
-func documentXMLDecodeKeepNsRawTokenScenario(w *bufio.Writer, s formatScenario) {
+func documentXMLDecodeKeepNsRawTokenScenario(w *bufio.Writer, s FormatScenario) {
 	writeOrPanic(w, fmt.Sprintf("## %v\n", s.description))
 
 	if s.subdescription != "" {
@@ -773,7 +759,7 @@ func documentXMLDecodeKeepNsRawTokenScenario(w *bufio.Writer, s formatScenario)
 	writeOrPanic(w, fmt.Sprintf("```xml\n%v```\n\n", mustProcessFormatScenario(s, NewXMLDecoder(prefsWithout), NewXMLEncoder(prefsWithout))))
 }
 
-func documentXMLEncodeScenario(w *bufio.Writer, s formatScenario) {
+func documentXMLEncodeScenario(w *bufio.Writer, s FormatScenario) {
 	writeOrPanic(w, fmt.Sprintf("## %v\n", s.description))
 
 	if s.subdescription != "" {
@@ -791,7 +777,7 @@ func documentXMLEncodeScenario(w *bufio.Writer, s formatScenario) {
 	writeOrPanic(w, fmt.Sprintf("```xml\n%v```\n\n", mustProcessFormatScenario(s, NewYamlDecoder(ConfiguredYamlPreferences), NewXMLEncoder(ConfiguredXMLPreferences))))
 }
 
-func documentXMLRoundTripScenario(w *bufio.Writer, s formatScenario) {
+func documentXMLRoundTripScenario(w *bufio.Writer, s FormatScenario) {
 	writeOrPanic(w, fmt.Sprintf("## %v\n", s.description))
 
 	if s.subdescription != "" {
@@ -809,7 +795,7 @@ func documentXMLRoundTripScenario(w *bufio.Writer, s formatScenario) {
 	writeOrPanic(w, fmt.Sprintf("```xml\n%v```\n\n", mustProcessFormatScenario(s, NewXMLDecoder(ConfiguredXMLPreferences), NewXMLEncoder(ConfiguredXMLPreferences))))
 }
 
-func documentXMLSkipDirectivesScenario(w *bufio.Writer, s formatScenario) {
+func documentXMLSkipDirectivesScenario(w *bufio.Writer, s FormatScenario) {
 	writeOrPanic(w, fmt.Sprintf("## %v\n", s.description))
 
 	if s.subdescription != "" {

pkg/yqlib/all_at_once_evaluator.go

@@ -63,7 +63,7 @@ func (e *allAtOnceEvaluator) EvaluateFiles(expression string, filenames []string
 	}
 
 	if allDocuments.Len() == 0 {
-		candidateNode := createScalarNode(nil, "")
+		candidateNode := CreateScalarNode(nil, "")
 		allDocuments.PushBack(candidateNode)
 	}
 

pkg/yqlib/base64_test.go

@@ -1,269 +0,0 @@
-//go:build !yq_nobase64
-
-package yqlib
-
-import (
-	"bufio"
-	"fmt"
-	"testing"
-
-	"github.com/mikefarah/yq/v4/test"
-)
-
-const base64EncodedSimple = "YSBzcGVjaWFsIHN0cmluZw=="
-const base64DecodedSimpleExtraSpaces = "\n " + base64EncodedSimple + "  \n"
-const base64DecodedSimple = "a special string"
-
-const base64EncodedUTF8 = "V29ya3Mgd2l0aCBVVEYtMTYg8J+Yig=="
-const base64DecodedUTF8 = "Works with UTF-16 😊"
-
-const base64EncodedYaml = "YTogYXBwbGUK"
-const base64DecodedYaml = "a: apple\n"
-
-const base64EncodedEmpty = ""
-const base64DecodedEmpty = ""
-
-const base64MissingPadding = "Y2F0cw"
-const base64DecodedMissingPadding = "cats"
-
-const base64EncodedCats = "Y2F0cw=="
-const base64DecodedCats = "cats"
-
-var base64Scenarios = []formatScenario{
-	{
-		skipDoc:      true,
-		description:  "empty decode",
-		input:        base64EncodedEmpty,
-		expected:     base64DecodedEmpty + "\n",
-		scenarioType: "decode",
-	},
-	{
-		skipDoc:      true,
-		description:  "simple decode",
-		input:        base64EncodedSimple,
-		expected:     base64DecodedSimple + "\n",
-		scenarioType: "decode",
-	},
-	{
-		description:    "Decode base64: simple",
-		subdescription: "Decoded data is assumed to be a string.",
-		input:          base64EncodedSimple,
-		expected:       base64DecodedSimple + "\n",
-		scenarioType:   "decode",
-	},
-	{
-		description:    "Decode base64: UTF-8",
-		subdescription: "Base64 decoding supports UTF-8 encoded strings.",
-		input:          base64EncodedUTF8,
-		expected:       base64DecodedUTF8 + "\n",
-		scenarioType:   "decode",
-	},
-	{
-		skipDoc:      true,
-		description:  "decode missing padding",
-		input:        base64MissingPadding,
-		expected:     base64DecodedMissingPadding + "\n",
-		scenarioType: "decode",
-	},
-	{
-
-		description:    "Decode with extra spaces",
-		subdescription: "Extra leading/trailing whitespace is stripped",
-		input:          base64DecodedSimpleExtraSpaces,
-		expected:       base64DecodedSimple + "\n",
-		scenarioType:   "decode",
-	},
-	{
-		skipDoc:      true,
-		description:  "decode with padding",
-		input:        base64EncodedCats,
-		expected:     base64DecodedCats + "\n",
-		scenarioType: "decode",
-	},
-	{
-		skipDoc:      true,
-		description:  "decode yaml document",
-		input:        base64EncodedYaml,
-		expected:     base64DecodedYaml + "\n",
-		scenarioType: "decode",
-	},
-	{
-		description:  "Encode base64: string",
-		input:        "\"" + base64DecodedSimple + "\"",
-		expected:     base64EncodedSimple,
-		scenarioType: "encode",
-	},
-	{
-		description:    "Encode base64: string from document",
-		subdescription: "Extract a string field and encode it to base64.",
-		input:          "coolData: \"" + base64DecodedSimple + "\"",
-		expression:     ".coolData",
-		expected:       base64EncodedSimple,
-		scenarioType:   "encode",
-	},
-	{
-		skipDoc:      true,
-		description:  "encode empty string",
-		input:        "\"\"",
-		expected:     "",
-		scenarioType: "encode",
-	},
-	{
-		skipDoc:      true,
-		description:  "encode UTF-8 string",
-		input:        "\"" + base64DecodedUTF8 + "\"",
-		expected:     base64EncodedUTF8,
-		scenarioType: "encode",
-	},
-	{
-		skipDoc:      true,
-		description:  "encode cats",
-		input:        "\"" + base64DecodedCats + "\"",
-		expected:     base64EncodedCats,
-		scenarioType: "encode",
-	},
-	{
-		description:  "Roundtrip: simple",
-		skipDoc:      true,
-		input:        base64EncodedSimple,
-		expected:     base64EncodedSimple,
-		scenarioType: "roundtrip",
-	},
-	{
-		description:  "Roundtrip: UTF-8",
-		skipDoc:      true,
-		input:        base64EncodedUTF8,
-		expected:     base64EncodedUTF8,
-		scenarioType: "roundtrip",
-	},
-	{
-		description:  "Roundtrip: missing padding",
-		skipDoc:      true,
-		input:        base64MissingPadding,
-		expected:     base64EncodedCats,
-		scenarioType: "roundtrip",
-	},
-	{
-		description:  "Roundtrip: empty",
-		skipDoc:      true,
-		input:        base64EncodedEmpty,
-		expected:     base64EncodedEmpty,
-		scenarioType: "roundtrip",
-	},
-	{
-		description:   "Encode error: non-string",
-		skipDoc:       true,
-		input:         "123",
-		expectedError: "cannot encode !!int as base64, can only operate on strings",
-		scenarioType:  "encode-error",
-	},
-	{
-		description:   "Encode error: array",
-		skipDoc:       true,
-		input:         "[1, 2, 3]",
-		expectedError: "cannot encode !!seq as base64, can only operate on strings",
-		scenarioType:  "encode-error",
-	},
-	{
-		description:   "Encode error: map",
-		skipDoc:       true,
-		input:         "{b: c}",
-		expectedError: "cannot encode !!map as base64, can only operate on strings",
-		scenarioType:  "encode-error",
-	},
-}
-
-func testBase64Scenario(t *testing.T, s formatScenario) {
-	switch s.scenarioType {
-	case "", "decode":
-		yamlPrefs := ConfiguredYamlPreferences.Copy()
-		yamlPrefs.Indent = 4
-		test.AssertResultWithContext(t, s.expected, mustProcessFormatScenario(s, NewBase64Decoder(), NewYamlEncoder(yamlPrefs)), s.description)
-	case "encode":
-		test.AssertResultWithContext(t, s.expected, mustProcessFormatScenario(s, NewYamlDecoder(ConfiguredYamlPreferences), NewBase64Encoder()), s.description)
-	case "roundtrip":
-		test.AssertResultWithContext(t, s.expected, mustProcessFormatScenario(s, NewBase64Decoder(), NewBase64Encoder()), s.description)
-	case "encode-error":
-		result, err := processFormatScenario(s, NewYamlDecoder(ConfiguredYamlPreferences), NewBase64Encoder())
-		if err == nil {
-			t.Errorf("Expected error '%v' but it worked: %v", s.expectedError, result)
-		} else {
-			test.AssertResultComplexWithContext(t, s.expectedError, err.Error(), s.description)
-		}
-
-	default:
-		panic(fmt.Sprintf("unhandled scenario type %q", s.scenarioType))
-	}
-}
-
-func documentBase64Scenario(_ *testing.T, w *bufio.Writer, i interface{}) {
-	s := i.(formatScenario)
-
-	if s.skipDoc {
-		return
-	}
-	switch s.scenarioType {
-	case "", "decode":
-		documentBase64DecodeScenario(w, s)
-	case "encode":
-		documentBase64EncodeScenario(w, s)
-
-	default:
-		panic(fmt.Sprintf("unhandled scenario type %q", s.scenarioType))
-	}
-}
-
-func documentBase64DecodeScenario(w *bufio.Writer, s formatScenario) {
-	writeOrPanic(w, fmt.Sprintf("## %v\n", s.description))
-
-	if s.subdescription != "" {
-		writeOrPanic(w, s.subdescription)
-		writeOrPanic(w, "\n\n")
-	}
-
-	writeOrPanic(w, "Given a sample.txt file of:\n")
-	writeOrPanic(w, fmt.Sprintf("```\n%v\n```\n", s.input))
-
-	writeOrPanic(w, "then\n")
-	expression := s.expression
-	if expression == "" {
-		expression = "."
-	}
-	writeOrPanic(w, fmt.Sprintf("```bash\nyq -p=base64 -oy '%v' sample.txt\n```\n", expression))
-	writeOrPanic(w, "will output\n")
-
-	writeOrPanic(w, fmt.Sprintf("```yaml\n%v```\n\n", mustProcessFormatScenario(s, NewBase64Decoder(), NewYamlEncoder(ConfiguredYamlPreferences))))
-}
-
-func documentBase64EncodeScenario(w *bufio.Writer, s formatScenario) {
-	writeOrPanic(w, fmt.Sprintf("## %v\n", s.description))
-
-	if s.subdescription != "" {
-		writeOrPanic(w, s.subdescription)
-		writeOrPanic(w, "\n\n")
-	}
-
-	writeOrPanic(w, "Given a sample.yml file of:\n")
-	writeOrPanic(w, fmt.Sprintf("```yaml\n%v\n```\n", s.input))
-
-	writeOrPanic(w, "then\n")
-	expression := s.expression
-	if expression == "" {
-		expression = "."
-	}
-	writeOrPanic(w, fmt.Sprintf("```bash\nyq -o=base64 '%v' sample.yml\n```\n", expression))
-	writeOrPanic(w, "will output\n")
-
-	writeOrPanic(w, fmt.Sprintf("```\n%v```\n\n", mustProcessFormatScenario(s, NewYamlDecoder(ConfiguredYamlPreferences), NewBase64Encoder())))
-}
-
-func TestBase64Scenarios(t *testing.T) {
-	for _, tt := range base64Scenarios {
-		testBase64Scenario(t, tt)
-	}
-	genericScenarios := make([]interface{}, len(base64Scenarios))
-	for i, s := range base64Scenarios {
-		genericScenarios[i] = s
-	}
-	documentScenarios(t, "usage", "base64", genericScenarios, documentBase64Scenario)
-}

pkg/yqlib/candidate_node.go

@@ -27,14 +27,14 @@ const (
 	FlowStyle
 )
 
-func createStringScalarNode(stringValue string) *CandidateNode {
+func CreateStringScalarNode(stringValue string) *CandidateNode {
 	var node = &CandidateNode{Kind: ScalarNode}
 	node.Value = stringValue
 	node.Tag = "!!str"
 	return node
 }
 
-func createScalarNode(value interface{}, stringValue string) *CandidateNode {
+func CreateScalarNode(value interface{}, stringValue string) *CandidateNode {
 	var node = &CandidateNode{Kind: ScalarNode}
 	node.Value = stringValue
 
@@ -53,20 +53,6 @@ func createScalarNode(value interface{}, stringValue string) *CandidateNode {
 	return node
 }
 
-type NodeInfo struct {
-	Kind        string      `yaml:"kind"`
-	Style       string      `yaml:"style,omitempty"`
-	Anchor      string      `yaml:"anchor,omitempty"`
-	Tag         string      `yaml:"tag,omitempty"`
-	HeadComment string      `yaml:"headComment,omitempty"`
-	LineComment string      `yaml:"lineComment,omitempty"`
-	FootComment string      `yaml:"footComment,omitempty"`
-	Value       string      `yaml:"value,omitempty"`
-	Line        int         `yaml:"line,omitempty"`
-	Column      int         `yaml:"column,omitempty"`
-	Content     []*NodeInfo `yaml:"content,omitempty"`
-}
-
 type CandidateNode struct {
 	Kind  Kind
 	Style Style
@@ -169,18 +155,6 @@ func (n *CandidateNode) getParsedKey() interface{} {
 
 }
 
-func (n *CandidateNode) FilterMapContentByKey(keyPredicate func(*CandidateNode) bool) []*CandidateNode {
-	var result []*CandidateNode
-	for index := 0; index < len(n.Content); index = index + 2 {
-		keyNode := n.Content[index]
-		valueNode := n.Content[index+1]
-		if keyPredicate(keyNode) {
-			result = append(result, keyNode, valueNode)
-		}
-	}
-	return result
-}
-
 func (n *CandidateNode) GetPath() []interface{} {
 	key := n.getParsedKey()
 	if n.Parent != nil && key != nil {
@@ -224,30 +198,6 @@ func (n *CandidateNode) SetParent(parent *CandidateNode) {
 	n.Parent = parent
 }
 
-type ValueVisitor func(*CandidateNode) error
-
-func (n *CandidateNode) VisitValues(visitor ValueVisitor) error {
-	switch n.Kind {
-	case MappingNode:
-		for i := 1; i < len(n.Content); i = i + 2 {
-			if err := visitor(n.Content[i]); err != nil {
-				return err
-			}
-		}
-	case SequenceNode:
-		for i := 0; i < len(n.Content); i = i + 1 {
-			if err := visitor(n.Content[i]); err != nil {
-				return err
-			}
-		}
-	}
-	return nil
-}
-
-func (n *CandidateNode) CanVisitValues() bool {
-	return n.Kind == MappingNode || n.Kind == SequenceNode
-}
-
 func (n *CandidateNode) AddKeyValueChild(rawKey *CandidateNode, rawValue *CandidateNode) (*CandidateNode, *CandidateNode) {
 	key := rawKey.Copy()
 	key.SetParent(n)
@@ -265,13 +215,14 @@ func (n *CandidateNode) AddKeyValueChild(rawKey *CandidateNode, rawValue *Candid
 func (n *CandidateNode) AddChild(rawChild *CandidateNode) {
 	value := rawChild.Copy()
 	value.SetParent(n)
-	value.IsMapKey = false
-
-	index := len(n.Content)
-	keyNode := createScalarNode(index, fmt.Sprintf("%v", index))
-	keyNode.SetParent(n)
-	value.Key = keyNode
-
+	if value.Key != nil {
+		value.Key.SetParent(n)
+	} else {
+		index := len(n.Content)
+		keyNode := CreateScalarNode(index, fmt.Sprintf("%v", index))
+		keyNode.SetParent(n)
+		value.Key = keyNode
+	}
 	n.Content = append(n.Content, value)
 }
 
@@ -292,7 +243,7 @@ func (n *CandidateNode) AddChildren(children []*CandidateNode) {
 
 func (n *CandidateNode) GetValueRep() (interface{}, error) {
 	log.Debugf("GetValueRep for %v value: %v", n.GetNicePath(), n.Value)
-	realTag := n.guessTagFromCustomType()
+	realTag := n.GuessTagFromCustomType()
 
 	switch realTag {
 	case "!!int":
@@ -310,17 +261,17 @@ func (n *CandidateNode) GetValueRep() (interface{}, error) {
 	return n.Value, nil
 }
 
-func (n *CandidateNode) guessTagFromCustomType() string {
+func (n *CandidateNode) GuessTagFromCustomType() string {
 	if strings.HasPrefix(n.Tag, "!!") {
 		return n.Tag
 	} else if n.Value == "" {
-		log.Debug("guessTagFromCustomType: node has no value to guess the type with")
+		log.Debug("GuessTagFromCustomType: node has no value to guess the type with")
 		return n.Tag
 	}
 	dataBucket, errorReading := parseSnippet(n.Value)
 
 	if errorReading != nil {
-		log.Debug("guessTagFromCustomType: could not guess underlying tag type %v", errorReading)
+		log.Debug("GuessTagFromCustomType: could not guess underlying tag type %v", errorReading)
 		return n.Tag
 	}
 	guessedTag := dataBucket.Tag
@@ -425,7 +376,7 @@ func (n *CandidateNode) UpdateFrom(other *CandidateNode, prefs assignPreferences
 	// if this is an empty map or empty array, use the style of other node.
 	if (n.Kind != ScalarNode && len(n.Content) == 0) ||
 		// if the tag has changed (e.g. from str to bool)
-		(n.guessTagFromCustomType() != other.guessTagFromCustomType()) {
+		(n.GuessTagFromCustomType() != other.GuessTagFromCustomType()) {
 		n.Style = other.Style
 	}
 
@@ -478,63 +429,33 @@ func (n *CandidateNode) UpdateAttributesFrom(other *CandidateNode, prefs assignP
 	}
 }
 
-func (n *CandidateNode) ConvertToNodeInfo() *NodeInfo {
-	info := &NodeInfo{
-		Kind:        kindToString(n.Kind),
-		Style:       styleToString(n.Style),
-		Anchor:      n.Anchor,
-		Tag:         n.Tag,
-		HeadComment: n.HeadComment,
-		LineComment: n.LineComment,
-		FootComment: n.FootComment,
-		Value:       n.Value,
-		Line:        n.Line,
-		Column:      n.Column,
-	}
-	if len(n.Content) > 0 {
-		info.Content = make([]*NodeInfo, len(n.Content))
-		for i, child := range n.Content {
-			info.Content[i] = child.ConvertToNodeInfo()
+func (n *CandidateNode) CleanHeadAndLineComment() string {
+	return n.CleanHeadComment() + n.CleanLineComment()
+}
+
+func (n *CandidateNode) CleanHeadComment() string {
+	return strings.Replace(n.HeadComment, "#", "", 1)
+}
+
+func (n *CandidateNode) CleanLineComment() string {
+	return strings.Replace(n.LineComment, "#", "", 1)
+}
+
+func (n *CandidateNode) CleanFootComment() string {
+	return strings.Replace(n.FootComment, "#", "", 1)
+}
+
+func (n *CandidateNode) ConvertKeysToStrings() {
+
+	if n.Kind == MappingNode {
+		for index, child := range n.Content {
+			if index%2 == 0 { // its a map key
+				child.Tag = "!!str"
+			}
 		}
 	}
-	return info
-}
 
-// Helper functions to convert Kind and Style to string for NodeInfo
-func kindToString(k Kind) string {
-	switch k {
-	case SequenceNode:
-		return "SequenceNode"
-	case MappingNode:
-		return "MappingNode"
-	case ScalarNode:
-		return "ScalarNode"
-	case AliasNode:
-		return "AliasNode"
-	default:
-		return "Unknown"
+	for _, child := range n.Content {
+		child.ConvertKeysToStrings()
 	}
 }
-
-func styleToString(s Style) string {
-	var styles []string
-	if s&TaggedStyle != 0 {
-		styles = append(styles, "TaggedStyle")
-	}
-	if s&DoubleQuotedStyle != 0 {
-		styles = append(styles, "DoubleQuotedStyle")
-	}
-	if s&SingleQuotedStyle != 0 {
-		styles = append(styles, "SingleQuotedStyle")
-	}
-	if s&LiteralStyle != 0 {
-		styles = append(styles, "LiteralStyle")
-	}
-	if s&FoldedStyle != 0 {
-		styles = append(styles, "FoldedStyle")
-	}
-	if s&FlowStyle != 0 {
-		styles = append(styles, "FlowStyle")
-	}
-	return strings.Join(styles, ",")
-}

pkg/yqlib/candidate_node_goccy_yaml.go

@@ -9,14 +9,14 @@ import (
 	goccyToken "github.com/goccy/go-yaml/token"
 )
 
-func (o *CandidateNode) goccyDecodeIntoChild(childNode ast.Node, cm yaml.CommentMap, anchorMap map[string]*CandidateNode) (*CandidateNode, error) {
+func (o *CandidateNode) goccyDecodeIntoChild(childNode ast.Node, cm yaml.CommentMap) (*CandidateNode, error) {
 	newChild := o.CreateChild()
 
-	err := newChild.UnmarshalGoccyYAML(childNode, cm, anchorMap)
+	err := newChild.UnmarshalGoccyYAML(childNode, cm)
 	return newChild, err
 }
 
-func (o *CandidateNode) UnmarshalGoccyYAML(node ast.Node, cm yaml.CommentMap, anchorMap map[string]*CandidateNode) error {
+func (o *CandidateNode) UnmarshalGoccyYAML(node ast.Node, cm yaml.CommentMap) error {
 	log.Debugf("UnmarshalYAML %v", node)
 	log.Debugf("UnmarshalYAML %v", node.Type().String())
 	log.Debugf("UnmarshalYAML Node Value: %v", node.String())
@@ -51,9 +51,6 @@ func (o *CandidateNode) UnmarshalGoccyYAML(node ast.Node, cm yaml.CommentMap, an
 	}
 
 	o.Value = node.String()
-	o.Line = node.GetToken().Position.Line
-	o.Column = node.GetToken().Position.Column
-
 	switch node.Type() {
 	case ast.IntegerType:
 		o.Kind = ScalarNode
@@ -65,13 +62,9 @@ func (o *CandidateNode) UnmarshalGoccyYAML(node ast.Node, cm yaml.CommentMap, an
 		o.Kind = ScalarNode
 		o.Tag = "!!bool"
 	case ast.NullType:
-		log.Debugf("its a null type with value %v", node.GetToken().Value)
 		o.Kind = ScalarNode
 		o.Tag = "!!null"
 		o.Value = node.GetToken().Value
-		if node.GetToken().Type == goccyToken.ImplicitNullType {
-			o.Value = ""
-		}
 	case ast.StringType:
 		o.Kind = ScalarNode
 		o.Tag = "!!str"
@@ -99,7 +92,7 @@ func (o *CandidateNode) UnmarshalGoccyYAML(node ast.Node, cm yaml.CommentMap, an
 		// to solve the multiline > problem
 		o.Value = astLiteral.Value.Value
 	case ast.TagType:
-		if err := o.UnmarshalGoccyYAML(node.(*ast.TagNode).Value, cm, anchorMap); err != nil {
+		if err := o.UnmarshalGoccyYAML(node.(*ast.TagNode).Value, cm); err != nil {
 			return err
 		}
 		o.Tag = node.(*ast.TagNode).Start.Value
@@ -113,9 +106,9 @@ func (o *CandidateNode) UnmarshalGoccyYAML(node ast.Node, cm yaml.CommentMap, an
 			o.Style = FlowStyle
 		}
 		for _, mappingValueNode := range mappingNode.Values {
-			err := o.goccyProcessMappingValueNode(mappingValueNode, cm, anchorMap)
+			err := o.goccyProcessMappingValueNode(mappingValueNode, cm)
 			if err != nil {
-				return err
+				return ast.ErrInvalidAnchorName
 			}
 		}
 		if mappingNode.FootComment != nil {
@@ -127,9 +120,9 @@ func (o *CandidateNode) UnmarshalGoccyYAML(node ast.Node, cm yaml.CommentMap, an
 		o.Kind = MappingNode
 		o.Tag = "!!map"
 		mappingValueNode := node.(*ast.MappingValueNode)
-		err := o.goccyProcessMappingValueNode(mappingValueNode, cm, anchorMap)
+		err := o.goccyProcessMappingValueNode(mappingValueNode, cm)
 		if err != nil {
-			return err
+			return ast.ErrInvalidAnchorName
 		}
 	case ast.SequenceType:
 		log.Debugf("UnmarshalYAML -  a sequence node")
@@ -148,7 +141,7 @@ func (o *CandidateNode) UnmarshalGoccyYAML(node ast.Node, cm yaml.CommentMap, an
 			keyNode.Kind = ScalarNode
 			keyNode.Value = fmt.Sprintf("%v", i)
 
-			valueNode, err := o.goccyDecodeIntoChild(astSeq[i], cm, anchorMap)
+			valueNode, err := o.goccyDecodeIntoChild(astSeq[i], cm)
 			if err != nil {
 				return err
 			}
@@ -156,55 +149,32 @@ func (o *CandidateNode) UnmarshalGoccyYAML(node ast.Node, cm yaml.CommentMap, an
 			valueNode.Key = keyNode
 			o.Content[i] = valueNode
 		}
-	case ast.AnchorType:
-		log.Debugf("UnmarshalYAML -  an anchor node")
-		anchorNode := node.(*ast.AnchorNode)
-		err := o.UnmarshalGoccyYAML(anchorNode.Value, cm, anchorMap)
-		if err != nil {
-			return err
-		}
-		o.Anchor = anchorNode.Name.String()
-		anchorMap[o.Anchor] = o
-
-	case ast.AliasType:
-		log.Debugf("UnmarshalYAML -  an alias node")
-		aliasNode := node.(*ast.AliasNode)
-		o.Kind = AliasNode
-		o.Value = aliasNode.Value.String()
-		o.Alias = anchorMap[o.Value]
-
-	case ast.MergeKeyType:
-		log.Debugf("UnmarshalYAML -  a merge key")
-		o.Kind = ScalarNode
-		o.Tag = "!!merge" // note - I should be able to get rid of this.
-		o.Value = "<<"
 
 	default:
-		log.Debugf("UnmarshalYAML -  no idea of the type!!\n%v: %v", node.Type(), node.String())
+		log.Debugf("UnmarshalYAML -  node idea of the type!!")
 	}
 	log.Debugf("KIND: %v", o.Kind)
 	return nil
 }
 
-func (o *CandidateNode) goccyProcessMappingValueNode(mappingEntry *ast.MappingValueNode, cm yaml.CommentMap, anchorMap map[string]*CandidateNode) error {
+func (o *CandidateNode) goccyProcessMappingValueNode(mappingEntry *ast.MappingValueNode, cm yaml.CommentMap) error {
 	log.Debug("UnmarshalYAML MAP KEY entry %v", mappingEntry.Key)
-
-	// AddKeyValueFirst because it clones the nodes, and we want to have the real refs when Unmarshalling
-	// particularly for the anchorMap
-	keyNode, valueNode := o.AddKeyValueChild(&CandidateNode{}, &CandidateNode{})
-
-	if err := keyNode.UnmarshalGoccyYAML(mappingEntry.Key, cm, anchorMap); err != nil {
+	keyNode, err := o.goccyDecodeIntoChild(mappingEntry.Key, cm)
+	if err != nil {
 		return err
 	}
+	keyNode.IsMapKey = true
 
 	log.Debug("UnmarshalYAML MAP VALUE entry %v", mappingEntry.Value)
-	if err := valueNode.UnmarshalGoccyYAML(mappingEntry.Value, cm, anchorMap); err != nil {
+	valueNode, err := o.goccyDecodeIntoChild(mappingEntry.Value, cm)
+	if err != nil {
 		return err
 	}
 
 	if mappingEntry.FootComment != nil {
 		valueNode.FootComment = mappingEntry.FootComment.String()
 	}
+	o.AddKeyValueChild(keyNode, valueNode)
 
 	return nil
 }

pkg/yqlib/candidate_node_test.go

@@ -64,13 +64,13 @@ func TestCandidateNodeChildWhenParentUpdated(t *testing.T) {
 	test.AssertResultWithContext(t, uint(1), child.GetDocument(), "document index")
 }
 
-type createScalarNodeScenario struct {
+type CreateScalarNodeScenario struct {
 	value       interface{}
 	stringValue string
 	expectedTag string
 }
 
-var createScalarScenarios = []createScalarNodeScenario{
+var createScalarScenarios = []CreateScalarNodeScenario{
 	{
 		value:       "mike",
 		stringValue: "mike",
@@ -100,20 +100,20 @@ var createScalarScenarios = []createScalarNodeScenario{
 
 func TestCreateScalarNodeScenarios(t *testing.T) {
 	for _, tt := range createScalarScenarios {
-		actual := createScalarNode(tt.value, tt.stringValue)
+		actual := CreateScalarNode(tt.value, tt.stringValue)
 		test.AssertResultWithContext(t, tt.stringValue, actual.Value, fmt.Sprintf("Value for: Value: [%v], String: %v", tt.value, tt.stringValue))
 		test.AssertResultWithContext(t, tt.expectedTag, actual.Tag, fmt.Sprintf("Value for: Value: [%v], String: %v", tt.value, tt.stringValue))
 	}
 }
 
 func TestGetKeyForMapValue(t *testing.T) {
-	key := createStringScalarNode("yourKey")
+	key := CreateStringScalarNode("yourKey")
 	n := CandidateNode{Key: key, Value: "meow", document: 3}
 	test.AssertResult(t, "3 - yourKey", n.GetKey())
 }
 
 func TestGetKeyForMapKey(t *testing.T) {
-	key := createStringScalarNode("yourKey")
+	key := CreateStringScalarNode("yourKey")
 	key.IsMapKey = true
 	key.document = 3
 	test.AssertResult(t, "key-yourKey-3 - ", key.GetKey())
@@ -125,7 +125,7 @@ func TestGetKeyForValue(t *testing.T) {
 }
 
 func TestGetParsedKeyForMapKey(t *testing.T) {
-	key := createStringScalarNode("yourKey")
+	key := CreateStringScalarNode("yourKey")
 	key.IsMapKey = true
 	key.document = 3
 	test.AssertResult(t, "yourKey", key.getParsedKey())
@@ -137,13 +137,13 @@ func TestGetParsedKeyForLooseValue(t *testing.T) {
 }
 
 func TestGetParsedKeyForMapValue(t *testing.T) {
-	key := createStringScalarNode("yourKey")
+	key := CreateStringScalarNode("yourKey")
 	n := CandidateNode{Key: key, Value: "meow", document: 3}
 	test.AssertResult(t, "yourKey", n.getParsedKey())
 }
 
 func TestGetParsedKeyForArrayValue(t *testing.T) {
-	key := createScalarNode(4, "4")
+	key := CreateScalarNode(4, "4")
 	n := CandidateNode{Key: key, Value: "meow", document: 3}
 	test.AssertResult(t, 4, n.getParsedKey())
 }
@@ -152,243 +152,11 @@ func TestCandidateNodeAddKeyValueChild(t *testing.T) {
 	key := CandidateNode{Value: "cool", IsMapKey: true}
 	node := CandidateNode{}
 
+	// if we use a key in a new node as a value, it should no longer be marked as a key
+
 	_, keyIsValueNow := node.AddKeyValueChild(&CandidateNode{Value: "newKey"}, &key)
 
 	test.AssertResult(t, keyIsValueNow.IsMapKey, false)
 	test.AssertResult(t, key.IsMapKey, true)
 
 }
-
-func TestConvertToNodeInfo(t *testing.T) {
-	child := &CandidateNode{
-		Kind:   ScalarNode,
-		Style:  DoubleQuotedStyle,
-		Tag:    "!!str",
-		Value:  "childValue",
-		Line:   2,
-		Column: 3,
-	}
-	parent := &CandidateNode{
-		Kind:        MappingNode,
-		Style:       TaggedStyle,
-		Tag:         "!!map",
-		Value:       "",
-		Line:        1,
-		Column:      1,
-		Content:     []*CandidateNode{child},
-		HeadComment: "head",
-		LineComment: "line",
-		FootComment: "foot",
-		Anchor:      "anchor",
-	}
-	info := parent.ConvertToNodeInfo()
-	test.AssertResult(t, "MappingNode", info.Kind)
-	test.AssertResult(t, "TaggedStyle", info.Style)
-	test.AssertResult(t, "!!map", info.Tag)
-	test.AssertResult(t, "head", info.HeadComment)
-	test.AssertResult(t, "line", info.LineComment)
-	test.AssertResult(t, "foot", info.FootComment)
-	test.AssertResult(t, "anchor", info.Anchor)
-	test.AssertResult(t, 1, info.Line)
-	test.AssertResult(t, 1, info.Column)
-	if len(info.Content) != 1 {
-		t.Fatalf("Expected 1 child, got %d", len(info.Content))
-	}
-	childInfo := info.Content[0]
-	test.AssertResult(t, "ScalarNode", childInfo.Kind)
-	test.AssertResult(t, "DoubleQuotedStyle", childInfo.Style)
-	test.AssertResult(t, "!!str", childInfo.Tag)
-	test.AssertResult(t, "childValue", childInfo.Value)
-	test.AssertResult(t, 2, childInfo.Line)
-	test.AssertResult(t, 3, childInfo.Column)
-}
-
-func TestCandidateNodeGetPath(t *testing.T) {
-	// Test root node with no parent
-	root := CandidateNode{Value: "root"}
-	path := root.GetPath()
-	test.AssertResult(t, 0, len(path))
-
-	// Test node with key
-	key := createStringScalarNode("myKey")
-	node := CandidateNode{Key: key, Value: "myValue"}
-	path = node.GetPath()
-	test.AssertResult(t, 1, len(path))
-	test.AssertResult(t, "myKey", path[0])
-
-	// Test nested path
-	parent := CandidateNode{}
-	parentKey := createStringScalarNode("parent")
-	parent.Key = parentKey
-	node.Parent = &parent
-	path = node.GetPath()
-	test.AssertResult(t, 2, len(path))
-	test.AssertResult(t, "parent", path[0])
-	test.AssertResult(t, "myKey", path[1])
-}
-
-func TestCandidateNodeGetNicePath(t *testing.T) {
-	// Test simple key
-	key := createStringScalarNode("simple")
-	node := CandidateNode{Key: key}
-	nicePath := node.GetNicePath()
-	test.AssertResult(t, "simple", nicePath)
-
-	// Test array index
-	arrayKey := createScalarNode(0, "0")
-	arrayNode := CandidateNode{Key: arrayKey}
-	nicePath = arrayNode.GetNicePath()
-	test.AssertResult(t, "[0]", nicePath)
-
-	dotKey := createStringScalarNode("key.with.dots")
-	dotNode := CandidateNode{Key: dotKey}
-	nicePath = dotNode.GetNicePath()
-	test.AssertResult(t, "key.with.dots", nicePath)
-
-	// Test nested path
-	parentKey := createStringScalarNode("parent")
-	parent := CandidateNode{Key: parentKey}
-	childKey := createStringScalarNode("child")
-	child := CandidateNode{Key: childKey, Parent: &parent}
-	nicePath = child.GetNicePath()
-	test.AssertResult(t, "parent.child", nicePath)
-}
-
-func TestCandidateNodeFilterMapContentByKey(t *testing.T) {
-	// Create a map with multiple key-value pairs
-	key1 := createStringScalarNode("key1")
-	value1 := createStringScalarNode("value1")
-	key2 := createStringScalarNode("key2")
-	value2 := createStringScalarNode("value2")
-	key3 := createStringScalarNode("key3")
-	value3 := createStringScalarNode("value3")
-
-	mapNode := &CandidateNode{
-		Kind:    MappingNode,
-		Content: []*CandidateNode{key1, value1, key2, value2, key3, value3},
-	}
-
-	// Filter by key predicate that matches key1 and key3
-	filtered := mapNode.FilterMapContentByKey(func(key *CandidateNode) bool {
-		return key.Value == "key1" || key.Value == "key3"
-	})
-
-	// Should return key1, value1, key3, value3
-	test.AssertResult(t, 4, len(filtered))
-	test.AssertResult(t, "key1", filtered[0].Value)
-	test.AssertResult(t, "value1", filtered[1].Value)
-	test.AssertResult(t, "key3", filtered[2].Value)
-	test.AssertResult(t, "value3", filtered[3].Value)
-}
-
-func TestCandidateNodeVisitValues(t *testing.T) {
-	// Test mapping node
-	key1 := createStringScalarNode("key1")
-	value1 := createStringScalarNode("value1")
-	key2 := createStringScalarNode("key2")
-	value2 := createStringScalarNode("value2")
-
-	mapNode := &CandidateNode{
-		Kind:    MappingNode,
-		Content: []*CandidateNode{key1, value1, key2, value2},
-	}
-
-	var visited []string
-	err := mapNode.VisitValues(func(node *CandidateNode) error {
-		visited = append(visited, node.Value)
-		return nil
-	})
-
-	test.AssertResult(t, nil, err)
-	test.AssertResult(t, 2, len(visited))
-	test.AssertResult(t, "value1", visited[0])
-	test.AssertResult(t, "value2", visited[1])
-
-	// Test sequence node
-	item1 := createStringScalarNode("item1")
-	item2 := createStringScalarNode("item2")
-
-	seqNode := &CandidateNode{
-		Kind:    SequenceNode,
-		Content: []*CandidateNode{item1, item2},
-	}
-
-	visited = []string{}
-	err = seqNode.VisitValues(func(node *CandidateNode) error {
-		visited = append(visited, node.Value)
-		return nil
-	})
-
-	test.AssertResult(t, nil, err)
-	test.AssertResult(t, 2, len(visited))
-	test.AssertResult(t, "item1", visited[0])
-	test.AssertResult(t, "item2", visited[1])
-
-	// Test scalar node (should not visit anything)
-	scalarNode := &CandidateNode{
-		Kind:  ScalarNode,
-		Value: "scalar",
-	}
-
-	visited = []string{}
-	err = scalarNode.VisitValues(func(node *CandidateNode) error {
-		visited = append(visited, node.Value)
-		return nil
-	})
-
-	test.AssertResult(t, nil, err)
-	test.AssertResult(t, 0, len(visited))
-}
-
-func TestCandidateNodeCanVisitValues(t *testing.T) {
-	mapNode := &CandidateNode{Kind: MappingNode}
-	seqNode := &CandidateNode{Kind: SequenceNode}
-	scalarNode := &CandidateNode{Kind: ScalarNode}
-
-	test.AssertResult(t, true, mapNode.CanVisitValues())
-	test.AssertResult(t, true, seqNode.CanVisitValues())
-	test.AssertResult(t, false, scalarNode.CanVisitValues())
-}
-
-func TestCandidateNodeAddChild(t *testing.T) {
-	parent := &CandidateNode{Kind: SequenceNode}
-	child := createStringScalarNode("child")
-
-	parent.AddChild(child)
-
-	test.AssertResult(t, 1, len(parent.Content))
-	test.AssertResult(t, false, parent.Content[0].IsMapKey)
-	test.AssertResult(t, "0", parent.Content[0].Key.Value)
-	// Check that parent is set correctly
-	if parent.Content[0].Parent != parent {
-		t.Errorf("Expected parent to be set correctly")
-	}
-}
-
-func TestCandidateNodeAddChildren(t *testing.T) {
-	// Test sequence node
-	parent := &CandidateNode{Kind: SequenceNode}
-	child1 := createStringScalarNode("child1")
-	child2 := createStringScalarNode("child2")
-
-	parent.AddChildren([]*CandidateNode{child1, child2})
-
-	test.AssertResult(t, 2, len(parent.Content))
-	test.AssertResult(t, "child1", parent.Content[0].Value)
-	test.AssertResult(t, "child2", parent.Content[1].Value)
-
-	// Test mapping node
-	mapParent := &CandidateNode{Kind: MappingNode}
-	key1 := createStringScalarNode("key1")
-	value1 := createStringScalarNode("value1")
-	key2 := createStringScalarNode("key2")
-	value2 := createStringScalarNode("value2")
-
-	mapParent.AddChildren([]*CandidateNode{key1, value1, key2, value2})
-
-	test.AssertResult(t, 4, len(mapParent.Content))
-	test.AssertResult(t, true, mapParent.Content[0].IsMapKey)  // key1
-	test.AssertResult(t, false, mapParent.Content[1].IsMapKey) // value1
-	test.AssertResult(t, true, mapParent.Content[2].IsMapKey)  // key2
-	test.AssertResult(t, false, mapParent.Content[3].IsMapKey) // value2
-}

pkg/yqlib/candidate_node_yaml.go

@@ -3,7 +3,7 @@ package yqlib
 import (
 	"fmt"
 
-	yaml "go.yaml.in/yaml/v4"
+	yaml "gopkg.in/yaml.v3"
 )
 
 func MapYamlStyle(original yaml.Style) Style {
@@ -78,6 +78,9 @@ func (o *CandidateNode) copyToYamlNode(node *yaml.Node) {
 	node.Value = o.Value
 	node.Anchor = o.Anchor
 
+	// node.Alias = TODO - find Alias in our own structure
+	// might need to be a post process thing
+
 	node.HeadComment = o.HeadComment
 
 	node.LineComment = o.LineComment

pkg/yqlib/candidiate_node_json.go

@@ -1,5 +1,3 @@
-//go:build !yq_nojson
-
 package yqlib
 
 import (
@@ -98,7 +96,7 @@ func (o *CandidateNode) UnmarshalJSON(data []byte) error {
 
 			if child == nil {
 				// need to represent it as a null scalar
-				child = createScalarNode(nil, "null")
+				child = CreateScalarNode(nil, "null")
 			}
 			childKey := o.CreateChild()
 			childKey.Kind = ScalarNode

pkg/yqlib/chown_linux_test.go

@@ -1,139 +0,0 @@
-//go:build linux
-
-package yqlib
-
-import (
-	"os"
-	"path/filepath"
-	"testing"
-	"time"
-)
-
-func TestChangeOwner(t *testing.T) {
-	// Create a temporary file for testing
-	tempDir := t.TempDir()
-	testFile := filepath.Join(tempDir, "testfile.txt")
-
-	// Create a test file
-	err := os.WriteFile(testFile, []byte("test content"), 0600)
-	if err != nil {
-		t.Fatalf("Failed to create test file: %v", err)
-	}
-
-	// Get file info
-	info, err := os.Stat(testFile)
-	if err != nil {
-		t.Fatalf("Failed to stat test file: %v", err)
-	}
-
-	// Create another temporary file to change ownership of
-	tempFile, err := os.CreateTemp(tempDir, "chown_test_*.txt")
-	if err != nil {
-		t.Fatalf("Failed to create temp file: %v", err)
-	}
-	defer os.Remove(tempFile.Name())
-	tempFile.Close()
-
-	// Test changeOwner function
-	err = changeOwner(info, tempFile)
-	if err != nil {
-		t.Errorf("changeOwner failed: %v", err)
-	}
-
-	// Verify that the function doesn't panic with valid input
-	tempFile2, err := os.CreateTemp(tempDir, "chown_test2_*.txt")
-	if err != nil {
-		t.Fatalf("Failed to create second temp file: %v", err)
-	}
-	defer os.Remove(tempFile2.Name())
-	tempFile2.Close()
-
-	// Test with the second file
-	err = changeOwner(info, tempFile2)
-	if err != nil {
-		t.Errorf("changeOwner failed on second file: %v", err)
-	}
-}
-
-func TestChangeOwnerWithInvalidFileInfo(t *testing.T) {
-	// Create a mock file info that doesn't have syscall.Stat_t
-	mockInfo := &mockFileInfo{
-		name: "mock",
-		size: 0,
-		mode: 0600,
-	}
-
-	// Create a temporary file
-	tempFile, err := os.CreateTemp(t.TempDir(), "chown_test_*.txt")
-	if err != nil {
-		t.Fatalf("Failed to create temp file: %v", err)
-	}
-	defer os.Remove(tempFile.Name())
-	tempFile.Close()
-
-	// Test changeOwner with mock file info (should not panic)
-	err = changeOwner(mockInfo, tempFile)
-	if err != nil {
-		t.Errorf("changeOwner failed with mock file info: %v", err)
-	}
-}
-
-func TestChangeOwnerWithNonExistentFile(t *testing.T) {
-	// Create a temporary file
-	tempFile, err := os.CreateTemp(t.TempDir(), "chown_test_*.txt")
-	if err != nil {
-		t.Fatalf("Failed to create temp file: %v", err)
-	}
-	defer os.Remove(tempFile.Name())
-	tempFile.Close()
-
-	// Get file info
-	info, err := os.Stat(tempFile.Name())
-	if err != nil {
-		t.Fatalf("Failed to stat temp file: %v", err)
-	}
-
-	// Remove the file
-	os.Remove(tempFile.Name())
-
-	err = changeOwner(info, tempFile)
-	// The function should not panic even if the file doesn't exist
-	if err != nil {
-		t.Logf("Expected error when changing owner of non-existent file: %v", err)
-	}
-}
-
-// mockFileInfo implements fs.FileInfo but doesn't have syscall.Stat_t
-type mockFileInfo struct {
-	name string
-	size int64
-	mode os.FileMode
-}
-
-func (m *mockFileInfo) Name() string       { return m.name }
-func (m *mockFileInfo) Size() int64        { return m.size }
-func (m *mockFileInfo) Mode() os.FileMode  { return m.mode }
-func (m *mockFileInfo) ModTime() time.Time { return time.Time{} }
-func (m *mockFileInfo) IsDir() bool        { return false }
-func (m *mockFileInfo) Sys() interface{}   { return nil } // This will cause the type assertion to fail
-
-func TestChangeOwnerWithSyscallStatT(t *testing.T) {
-	// Create a temporary file
-	tempFile, err := os.CreateTemp(t.TempDir(), "chown_test_*.txt")
-	if err != nil {
-		t.Fatalf("Failed to create temp file: %v", err)
-	}
-	defer os.Remove(tempFile.Name())
-	tempFile.Close()
-
-	// Get file info
-	info, err := os.Stat(tempFile.Name())
-	if err != nil {
-		t.Fatalf("Failed to stat temp file: %v", err)
-	}
-
-	err = changeOwner(info, tempFile)
-	if err != nil {
-		t.Logf("changeOwner returned error (this might be expected in some environments): %v", err)
-	}
-}

pkg/yqlib/chown_not_linux_os.go

@@ -7,6 +7,6 @@ import (
 	"os"
 )
 
-func changeOwner(_ fs.FileInfo, _ *os.File) error {
+func changeOwner(info fs.FileInfo, file *os.File) error {
 	return nil
 }

pkg/yqlib/color_print.go

@@ -56,12 +56,6 @@ func colorizeAndPrint(yamlBytes []byte, writer io.Writer) error {
 			Suffix: format(color.Reset),
 		}
 	}
-	p.Comment = func() *printer.Property {
-		return &printer.Property{
-			Prefix: format(color.FgHiBlack),
-			Suffix: format(color.Reset),
-		}
-	}
 	_, err := writer.Write([]byte(p.PrintTokens(tokens) + "\n"))
 	return err
 }

pkg/yqlib/color_print_test.go

@@ -1,153 +0,0 @@
-package yqlib
-
-import (
-	"bytes"
-	"strings"
-	"testing"
-
-	"github.com/fatih/color"
-)
-
-func TestFormat(t *testing.T) {
-	tests := []struct {
-		name     string
-		attr     color.Attribute
-		expected string
-	}{
-		{
-			name:     "reset color",
-			attr:     color.Reset,
-			expected: "\x1b[0m",
-		},
-		{
-			name:     "red color",
-			attr:     color.FgRed,
-			expected: "\x1b[31m",
-		},
-		{
-			name:     "green color",
-			attr:     color.FgGreen,
-			expected: "\x1b[32m",
-		},
-	}
-
-	for _, tt := range tests {
-		t.Run(tt.name, func(t *testing.T) {
-			result := format(tt.attr)
-			if result != tt.expected {
-				t.Errorf("format(%d) = %q, want %q", tt.attr, result, tt.expected)
-			}
-		})
-	}
-}
-
-func TestColorizeAndPrint(t *testing.T) {
-	tests := []struct {
-		name      string
-		yamlBytes []byte
-		expectErr bool
-	}{
-		{
-			name:      "simple yaml",
-			yamlBytes: []byte("name: test\nage: 25\n"),
-			expectErr: false,
-		},
-		{
-			name:      "yaml with strings",
-			yamlBytes: []byte("name: \"hello world\"\nactive: true\ncount: 42\n"),
-			expectErr: false,
-		},
-		{
-			name:      "yaml with anchors and aliases",
-			yamlBytes: []byte("default: &default\n  name: test\nuser: *default\n"),
-			expectErr: false,
-		},
-		{
-			name:      "yaml with comments",
-			yamlBytes: []byte("# This is a comment\nname: test\n"),
-			expectErr: false,
-		},
-		{
-			name:      "empty yaml",
-			yamlBytes: []byte(""),
-			expectErr: false,
-		},
-	}
-
-	for _, tt := range tests {
-		t.Run(tt.name, func(t *testing.T) {
-			var buf bytes.Buffer
-			err := colorizeAndPrint(tt.yamlBytes, &buf)
-
-			if tt.expectErr && err == nil {
-				t.Error("Expected error but got none")
-			}
-			if !tt.expectErr && err != nil {
-				t.Errorf("Unexpected error: %v", err)
-			}
-
-			// Check that output contains escape sequences (color codes)
-			if !tt.expectErr && len(tt.yamlBytes) > 0 {
-				output := buf.String()
-				if !strings.Contains(output, "\x1b[") {
-					t.Error("Expected output to contain color escape sequences")
-				}
-			}
-		})
-	}
-}
-
-func TestColorizeAndPrintWithDifferentYamlTypes(t *testing.T) {
-	testCases := []struct {
-		name      string
-		yaml      string
-		expectErr bool
-	}{
-		{
-			name: "boolean values",
-			yaml: "active: true\ninactive: false\n",
-		},
-		{
-			name: "numeric values",
-			yaml: "integer: 42\nfloat: 3.14\nnegative: -10\n",
-		},
-		{
-			name: "map keys",
-			yaml: "user:\n  name: john\n  age: 30\n",
-		},
-		{
-			name: "string values",
-			yaml: "message: \"hello world\"\ndescription: 'single quotes'\n",
-		},
-		{
-			name: "mixed types",
-			yaml: "config:\n  debug: true\n  port: 8080\n  host: \"localhost\"\n",
-		},
-	}
-
-	for _, tc := range testCases {
-		t.Run(tc.name, func(t *testing.T) {
-			var buf bytes.Buffer
-			err := colorizeAndPrint([]byte(tc.yaml), &buf)
-
-			if tc.expectErr && err == nil {
-				t.Error("Expected error but got none")
-			}
-			if !tc.expectErr && err != nil {
-				t.Errorf("Unexpected error: %v", err)
-			}
-
-			// Verify output contains color codes
-			if !tc.expectErr {
-				output := buf.String()
-				if !strings.Contains(output, "\x1b[") {
-					t.Error("Expected output to contain color escape sequences")
-				}
-				// Should end with newline
-				if !strings.HasSuffix(output, "\n") {
-					t.Error("Expected output to end with newline")
-				}
-			}
-		})
-	}
-}

pkg/yqlib/context_test.go

@@ -2,11 +2,9 @@ package yqlib
 
 import (
 	"container/list"
-	"strings"
 	"testing"
 
 	"github.com/mikefarah/yq/v4/test"
-	logging "gopkg.in/op/go-logging.v1"
 )
 
 func TestChildContext(t *testing.T) {
@@ -51,211 +49,3 @@ func TestChildContextNoVariables(t *testing.T) {
 	test.AssertResultComplex(t, make(map[string]*list.List), clone.Variables)
 
 }
-
-func TestSingleReadonlyChildContext(t *testing.T) {
-	original := Context{
-		DontAutoCreate: false,
-		datetimeLayout: "2006-01-02",
-	}
-
-	candidate := &CandidateNode{Value: "test"}
-	clone := original.SingleReadonlyChildContext(candidate)
-
-	// Should have DontAutoCreate set to true
-	test.AssertResultComplex(t, true, clone.DontAutoCreate)
-
-	// Should have the candidate node in MatchingNodes
-	test.AssertResultComplex(t, 1, clone.MatchingNodes.Len())
-	test.AssertResultComplex(t, candidate, clone.MatchingNodes.Front().Value)
-}
-
-func TestSingleChildContext(t *testing.T) {
-	original := Context{
-		DontAutoCreate: true,
-		datetimeLayout: "2006-01-02",
-	}
-
-	candidate := &CandidateNode{Value: "test"}
-	clone := original.SingleChildContext(candidate)
-
-	// Should preserve DontAutoCreate
-	test.AssertResultComplex(t, true, clone.DontAutoCreate)
-
-	// Should have the candidate node in MatchingNodes
-	test.AssertResultComplex(t, 1, clone.MatchingNodes.Len())
-	test.AssertResultComplex(t, candidate, clone.MatchingNodes.Front().Value)
-}
-
-func TestSetDateTimeLayout(t *testing.T) {
-	context := Context{}
-
-	// Test setting datetime layout
-	context.SetDateTimeLayout("2006-01-02T15:04:05Z07:00")
-	test.AssertResultComplex(t, "2006-01-02T15:04:05Z07:00", context.datetimeLayout)
-}
-
-func TestGetDateTimeLayout(t *testing.T) {
-	// Test with custom layout
-	context := Context{datetimeLayout: "2006-01-02"}
-	result := context.GetDateTimeLayout()
-	test.AssertResultComplex(t, "2006-01-02", result)
-
-	// Test with empty layout (should return default)
-	context = Context{}
-	result = context.GetDateTimeLayout()
-	test.AssertResultComplex(t, "2006-01-02T15:04:05Z07:00", result)
-}
-
-func TestGetVariable(t *testing.T) {
-	// Test with nil Variables
-	context := Context{}
-	result := context.GetVariable("nonexistent")
-	test.AssertResultComplex(t, (*list.List)(nil), result)
-
-	// Test with existing variable
-	variables := make(map[string]*list.List)
-	variables["test"] = list.New()
-	variables["test"].PushBack(&CandidateNode{Value: "value"})
-
-	context = Context{Variables: variables}
-	result = context.GetVariable("test")
-	test.AssertResultComplex(t, variables["test"], result)
-
-	// Test with non-existent variable
-	result = context.GetVariable("nonexistent")
-	test.AssertResultComplex(t, (*list.List)(nil), result)
-}
-
-func TestSetVariable(t *testing.T) {
-	// Test setting variable when Variables is nil
-	context := Context{}
-	value := list.New()
-	value.PushBack(&CandidateNode{Value: "test"})
-
-	context.SetVariable("key", value)
-	test.AssertResultComplex(t, value, context.Variables["key"])
-
-	// Test setting variable when Variables already exists
-	context.SetVariable("key2", value)
-	test.AssertResultComplex(t, value, context.Variables["key2"])
-}
-
-func TestToString(t *testing.T) {
-	context := Context{
-		DontAutoCreate: true,
-		MatchingNodes:  list.New(),
-	}
-
-	// Add a node to test the full string representation
-	node := &CandidateNode{Value: "test"}
-	context.MatchingNodes.PushBack(node)
-
-	// Test with debug logging disabled (default)
-	result := context.ToString()
-	test.AssertResultComplex(t, "", result)
-
-	// Test with debug logging enabled
-	logging.SetLevel(logging.DEBUG, "")
-	defer logging.SetLevel(logging.INFO, "") // Reset to default
-
-	result2 := context.ToString()
-	test.AssertResultComplex(t, true, len(result2) > 0)
-	test.AssertResultComplex(t, true, strings.Contains(result2, "Context"))
-	test.AssertResultComplex(t, true, strings.Contains(result2, "DontAutoCreate: true"))
-}
-
-func TestDeepClone(t *testing.T) {
-	// Create original context with variables and matching nodes
-	originalVariables := make(map[string]*list.List)
-	originalVariables["test"] = list.New()
-	originalVariables["test"].PushBack(&CandidateNode{Value: "original"})
-
-	original := Context{
-		DontAutoCreate: true,
-		datetimeLayout: "2006-01-02",
-		Variables:      originalVariables,
-		MatchingNodes:  list.New(),
-	}
-
-	// Add a node to MatchingNodes
-	node := &CandidateNode{Value: "test"}
-	original.MatchingNodes.PushBack(node)
-
-	clone := original.DeepClone()
-
-	// Should preserve DontAutoCreate and datetimeLayout
-	test.AssertResultComplex(t, original.DontAutoCreate, clone.DontAutoCreate)
-	test.AssertResultComplex(t, original.datetimeLayout, clone.datetimeLayout)
-
-	// Should have copied variables
-	test.AssertResultComplex(t, 1, len(clone.Variables))
-	test.AssertResultComplex(t, "original", clone.Variables["test"].Front().Value.(*CandidateNode).Value)
-
-	// Should have deep copied MatchingNodes
-	test.AssertResultComplex(t, 1, clone.MatchingNodes.Len())
-
-	// Verify it's a deep copy by modifying the original
-	original.MatchingNodes.Front().Value.(*CandidateNode).Value = "modified"
-	test.AssertResultComplex(t, "test", clone.MatchingNodes.Front().Value.(*CandidateNode).Value)
-}
-
-func TestClone(t *testing.T) {
-	// Create original context
-	original := Context{
-		DontAutoCreate: true,
-		datetimeLayout: "2006-01-02",
-		MatchingNodes:  list.New(),
-	}
-
-	node := &CandidateNode{Value: "test"}
-	original.MatchingNodes.PushBack(node)
-
-	clone := original.Clone()
-
-	// Should preserve DontAutoCreate and datetimeLayout
-	test.AssertResultComplex(t, original.DontAutoCreate, clone.DontAutoCreate)
-	test.AssertResultComplex(t, original.datetimeLayout, clone.datetimeLayout)
-
-	// Should have the same MatchingNodes reference
-	test.AssertResultComplex(t, original.MatchingNodes, clone.MatchingNodes)
-}
-
-func TestReadOnlyClone(t *testing.T) {
-	original := Context{
-		DontAutoCreate: false,
-		datetimeLayout: "2006-01-02",
-		MatchingNodes:  list.New(),
-	}
-
-	node := &CandidateNode{Value: "test"}
-	original.MatchingNodes.PushBack(node)
-
-	clone := original.ReadOnlyClone()
-
-	// Should set DontAutoCreate to true
-	test.AssertResultComplex(t, true, clone.DontAutoCreate)
-
-	// Should preserve other fields
-	test.AssertResultComplex(t, original.datetimeLayout, clone.datetimeLayout)
-	test.AssertResultComplex(t, original.MatchingNodes, clone.MatchingNodes)
-}
-
-func TestWritableClone(t *testing.T) {
-	original := Context{
-		DontAutoCreate: true,
-		datetimeLayout: "2006-01-02",
-		MatchingNodes:  list.New(),
-	}
-
-	node := &CandidateNode{Value: "test"}
-	original.MatchingNodes.PushBack(node)
-
-	clone := original.WritableClone()
-
-	// Should set DontAutoCreate to false
-	test.AssertResultComplex(t, false, clone.DontAutoCreate)
-
-	// Should preserve other fields
-	test.AssertResultComplex(t, original.datetimeLayout, clone.datetimeLayout)
-	test.AssertResultComplex(t, original.MatchingNodes, clone.MatchingNodes)
-}

pkg/yqlib/csv_test.go

@@ -94,7 +94,7 @@ const tsvTestExpectedSimpleCsv = `i	like	csv
 because	excel	is	cool
 `
 
-var csvScenarios = []formatScenario{
+var csvScenarios = []FormatScenario{
 	{
 		description:  "Encode CSV simple",
 		input:        csvTestSimpleYaml,
@@ -186,20 +186,6 @@ var csvScenarios = []formatScenario{
 		expected:       expectedYamlFromCSVNoParsing,
 		scenarioType:   "decode-csv-no-auto",
 	},
-	{
-		description:  "values starting with #, no auto parse",
-		skipDoc:      true,
-		input:        "value\n#ffff",
-		expected:     "- value: '#ffff'\n",
-		scenarioType: "decode-csv-no-auto",
-	},
-	{
-		description:  "values starting with #",
-		skipDoc:      true,
-		input:        "value\n#ffff",
-		expected:     "- value: #ffff\n",
-		scenarioType: "decode-csv",
-	},
 	{
 		description:  "Scalar roundtrip",
 		skipDoc:      true,
@@ -224,7 +210,7 @@ var csvScenarios = []formatScenario{
 	},
 }
 
-func testCSVScenario(t *testing.T, s formatScenario) {
+func testCSVScenario(t *testing.T, s FormatScenario) {
 	switch s.scenarioType {
 	case "encode-csv":
 		test.AssertResultWithContext(t, s.expected, mustProcessFormatScenario(s, NewYamlDecoder(ConfiguredYamlPreferences), NewCsvEncoder(ConfiguredCsvPreferences)), s.description)
@@ -243,7 +229,7 @@ func testCSVScenario(t *testing.T, s formatScenario) {
 	}
 }
 
-func documentCSVDecodeObjectScenario(w *bufio.Writer, s formatScenario, formatType string) {
+func documentCSVDecodeObjectScenario(w *bufio.Writer, s FormatScenario, formatType string) {
 	writeOrPanic(w, fmt.Sprintf("## %v\n", s.description))
 
 	if s.subdescription != "" {
@@ -268,7 +254,7 @@ func documentCSVDecodeObjectScenario(w *bufio.Writer, s formatScenario, formatTy
 	)
 }
 
-func documentCSVDecodeObjectNoAutoScenario(w *bufio.Writer, s formatScenario, formatType string) {
+func documentCSVDecodeObjectNoAutoScenario(w *bufio.Writer, s FormatScenario, formatType string) {
 	writeOrPanic(w, fmt.Sprintf("## %v\n", s.description))
 
 	if s.subdescription != "" {
@@ -293,7 +279,7 @@ func documentCSVDecodeObjectNoAutoScenario(w *bufio.Writer, s formatScenario, fo
 	)
 }
 
-func documentCSVEncodeScenario(w *bufio.Writer, s formatScenario, formatType string) {
+func documentCSVEncodeScenario(w *bufio.Writer, s FormatScenario, formatType string) {
 	writeOrPanic(w, fmt.Sprintf("## %v\n", s.description))
 
 	if s.subdescription != "" {
@@ -326,7 +312,7 @@ func documentCSVEncodeScenario(w *bufio.Writer, s formatScenario, formatType str
 	)
 }
 
-func documentCSVRoundTripScenario(w *bufio.Writer, s formatScenario, formatType string) {
+func documentCSVRoundTripScenario(w *bufio.Writer, s FormatScenario, formatType string) {
 	writeOrPanic(w, fmt.Sprintf("## %v\n", s.description))
 
 	if s.subdescription != "" {
@@ -362,7 +348,7 @@ func documentCSVRoundTripScenario(w *bufio.Writer, s formatScenario, formatType
 }
 
 func documentCSVScenario(_ *testing.T, w *bufio.Writer, i interface{}) {
-	s := i.(formatScenario)
+	s := i.(FormatScenario)
 	if s.skipDoc {
 		return
 	}

pkg/yqlib/data_tree_navigator.go

@@ -13,6 +13,7 @@ type DataTreeNavigator interface {
 	GetMatchingNodes(context Context, expressionNode *ExpressionNode) (Context, error)
 
 	DeeplyAssign(context Context, path []interface{}, rhsNode *CandidateNode) error
+	DeeplyAssignKey(context Context, path []interface{}, rhsNode *CandidateNode) error
 }
 
 type dataTreeNavigator struct {
@@ -23,6 +24,14 @@ func NewDataTreeNavigator() DataTreeNavigator {
 }
 
 func (d *dataTreeNavigator) DeeplyAssign(context Context, path []interface{}, rhsCandidateNode *CandidateNode) error {
+	return d.deeplyAssign(context, path, rhsCandidateNode, false)
+}
+
+func (d *dataTreeNavigator) DeeplyAssignKey(context Context, path []interface{}, rhsCandidateNode *CandidateNode) error {
+	return d.deeplyAssign(context, path, rhsCandidateNode, true)
+}
+
+func (d *dataTreeNavigator) deeplyAssign(context Context, path []interface{}, rhsCandidateNode *CandidateNode, targetKey bool) error {
 
 	assignmentOp := &Operation{OperationType: assignOpType, Preferences: assignPreferences{}}
 
@@ -41,7 +50,7 @@ func (d *dataTreeNavigator) DeeplyAssign(context Context, path []interface{}, rh
 
 	assignmentOpNode := &ExpressionNode{
 		Operation: assignmentOp,
-		LHS:       createTraversalTree(path, traversePreferences{}, false),
+		LHS:       createTraversalTree(path, traversePreferences{}, targetKey),
 		RHS:       &ExpressionNode{Operation: rhsOp},
 	}
 
@@ -64,6 +73,6 @@ func (d *dataTreeNavigator) GetMatchingNodes(context Context, expressionNode *Ex
 	if handler != nil {
 		return handler(d, context, expressionNode)
 	}
-	return Context{}, fmt.Errorf("unknown operator %v", expressionNode.Operation.OperationType.Type)
+	return Context{}, fmt.Errorf("Unknown operator %v", expressionNode.Operation.OperationType)
 
 }

pkg/yqlib/data_tree_navigator_test.go

@@ -1,435 +0,0 @@
-package yqlib
-
-import (
-	"container/list"
-	"testing"
-
-	"github.com/mikefarah/yq/v4/test"
-)
-
-func TestGetMatchingNodes_NilExpressionNode(t *testing.T) {
-	navigator := NewDataTreeNavigator()
-	context := Context{
-		MatchingNodes: list.New(),
-	}
-
-	result, err := navigator.GetMatchingNodes(context, nil)
-
-	test.AssertResult(t, nil, err)
-	test.AssertResultComplex(t, context, result)
-}
-
-func TestGetMatchingNodes_UnknownOperator(t *testing.T) {
-	navigator := NewDataTreeNavigator()
-	context := Context{
-		MatchingNodes: list.New(),
-	}
-
-	// Create an expression node with an unknown operation type
-	unknownOpType := &operationType{Type: "UNKNOWN", Handler: nil}
-	expressionNode := &ExpressionNode{
-		Operation: &Operation{OperationType: unknownOpType},
-	}
-
-	result, err := navigator.GetMatchingNodes(context, expressionNode)
-
-	test.AssertResult(t, "unknown operator UNKNOWN", err.Error())
-	test.AssertResultComplex(t, Context{}, result)
-}
-
-func TestGetMatchingNodes_ValidOperator(t *testing.T) {
-	navigator := NewDataTreeNavigator()
-
-	// Create a simple context with a scalar node
-	scalarNode := &CandidateNode{
-		Kind:  ScalarNode,
-		Tag:   "!!str",
-		Value: "test",
-	}
-	context := Context{
-		MatchingNodes: list.New(),
-	}
-	context.MatchingNodes.PushBack(scalarNode)
-
-	// Create an expression node with a valid operation (self reference)
-	expressionNode := &ExpressionNode{
-		Operation: &Operation{OperationType: selfReferenceOpType},
-	}
-
-	result, err := navigator.GetMatchingNodes(context, expressionNode)
-
-	test.AssertResult(t, nil, err)
-	test.AssertResult(t, 1, result.MatchingNodes.Len())
-
-	// Verify the result contains the same node
-	resultNode := result.MatchingNodes.Front().Value.(*CandidateNode)
-	test.AssertResult(t, scalarNode, resultNode)
-}
-
-func TestDeeplyAssign_ScalarNode(t *testing.T) {
-	navigator := NewDataTreeNavigator()
-
-	// Create a context with a root mapping node
-	rootNode := &CandidateNode{
-		Kind: MappingNode,
-		Tag:  "!!map",
-		Content: []*CandidateNode{
-			{Kind: ScalarNode, Tag: "!!str", Value: "existing", IsMapKey: true},
-			{Kind: ScalarNode, Tag: "!!str", Value: "old_value"},
-		},
-	}
-	context := Context{
-		MatchingNodes: list.New(),
-	}
-	context.MatchingNodes.PushBack(rootNode)
-
-	// Create a scalar node to assign
-	scalarNode := &CandidateNode{
-		Kind:  ScalarNode,
-		Tag:   "!!str",
-		Value: "new_value",
-	}
-
-	// Assign to path ["new_key"]
-	path := []interface{}{"new_key"}
-	err := navigator.DeeplyAssign(context, path, scalarNode)
-
-	test.AssertResult(t, nil, err)
-
-	// Verify the assignment was made
-	// The root node should now have the new key-value pair
-	test.AssertResult(t, 4, len(rootNode.Content)) // 2 original + 2 new
-
-	// Find the new key-value pair
-	found := false
-	for i := 0; i < len(rootNode.Content)-1; i += 2 {
-		key := rootNode.Content[i]
-		value := rootNode.Content[i+1]
-		if key.Value == "new_key" && value.Value == "new_value" {
-			found = true
-			break
-		}
-	}
-	test.AssertResult(t, true, found)
-}
-
-func TestDeeplyAssign_MappingNode(t *testing.T) {
-	navigator := NewDataTreeNavigator()
-
-	// Create a context with a root mapping node
-	rootNode := &CandidateNode{
-		Kind: MappingNode,
-		Tag:  "!!map",
-		Content: []*CandidateNode{
-			{Kind: ScalarNode, Tag: "!!str", Value: "existing", IsMapKey: true},
-			{Kind: ScalarNode, Tag: "!!str", Value: "old_value"},
-		},
-	}
-	context := Context{
-		MatchingNodes: list.New(),
-	}
-	context.MatchingNodes.PushBack(rootNode)
-
-	// Create a mapping node to assign (this should trigger deep merge)
-	mappingNode := &CandidateNode{
-		Kind: MappingNode,
-		Tag:  "!!map",
-		Content: []*CandidateNode{
-			{Kind: ScalarNode, Tag: "!!str", Value: "nested_key", IsMapKey: true},
-			{Kind: ScalarNode, Tag: "!!str", Value: "nested_value"},
-		},
-	}
-
-	// Assign to path ["new_map"]
-	path := []interface{}{"new_map"}
-	err := navigator.DeeplyAssign(context, path, mappingNode)
-
-	test.AssertResult(t, nil, err)
-
-	// Verify the assignment was made
-	// The root node should now have the new mapping
-	test.AssertResult(t, 4, len(rootNode.Content)) // 2 original + 2 new
-
-	// Find the new mapping
-	found := false
-	for i := 0; i < len(rootNode.Content); i += 2 {
-		if i+1 < len(rootNode.Content) {
-			key := rootNode.Content[i]
-			value := rootNode.Content[i+1]
-			if key.Value == "new_map" && value.Kind == MappingNode {
-				found = true
-				// Verify the nested content
-				test.AssertResult(t, 2, len(value.Content))
-				test.AssertResult(t, "nested_key", value.Content[0].Value)
-				test.AssertResult(t, "nested_value", value.Content[1].Value)
-				break
-			}
-		}
-	}
-	test.AssertResult(t, true, found)
-}
-
-func TestDeeplyAssign_DeepPath(t *testing.T) {
-	navigator := NewDataTreeNavigator()
-
-	// Create a context with a root mapping node
-	rootNode := &CandidateNode{
-		Kind: MappingNode,
-		Tag:  "!!map",
-		Content: []*CandidateNode{
-			{Kind: ScalarNode, Tag: "!!str", Value: "level1", IsMapKey: true},
-			{Kind: MappingNode, Tag: "!!map", Content: []*CandidateNode{}},
-		},
-	}
-	context := Context{
-		MatchingNodes: list.New(),
-	}
-	context.MatchingNodes.PushBack(rootNode)
-
-	// Create a scalar node to assign
-	scalarNode := &CandidateNode{
-		Kind:  ScalarNode,
-		Tag:   "!!str",
-		Value: "deep_value",
-	}
-
-	// Assign to deep path ["level1", "level2", "level3"]
-	path := []interface{}{"level1", "level2", "level3"}
-	err := navigator.DeeplyAssign(context, path, scalarNode)
-
-	test.AssertResult(t, nil, err)
-
-	// Verify the deep assignment was made
-	level1Node := rootNode.Content[1]                // The mapping node
-	test.AssertResult(t, 2, len(level1Node.Content)) // Should have level2 key-value
-
-	level2Key := level1Node.Content[0]
-	level2Value := level1Node.Content[1]
-	test.AssertResult(t, "level2", level2Key.Value)
-	test.AssertResult(t, MappingNode, level2Value.Kind)
-
-	level3Key := level2Value.Content[0]
-	level3Value := level2Value.Content[1]
-	test.AssertResult(t, "level3", level3Key.Value)
-	test.AssertResult(t, "deep_value", level3Value.Value)
-}
-
-func TestDeeplyAssign_ArrayPath(t *testing.T) {
-	navigator := NewDataTreeNavigator()
-
-	// Create a context with a root mapping node containing an array
-	rootNode := &CandidateNode{
-		Kind: MappingNode,
-		Tag:  "!!map",
-		Content: []*CandidateNode{
-			{Kind: ScalarNode, Tag: "!!str", Value: "array", IsMapKey: true},
-			{Kind: SequenceNode, Tag: "!!seq", Content: []*CandidateNode{}},
-		},
-	}
-	context := Context{
-		MatchingNodes: list.New(),
-	}
-	context.MatchingNodes.PushBack(rootNode)
-
-	// Create a scalar node to assign
-	scalarNode := &CandidateNode{
-		Kind:  ScalarNode,
-		Tag:   "!!str",
-		Value: "array_value",
-	}
-
-	// Assign to array path ["array", 0]
-	path := []interface{}{"array", 0}
-	err := navigator.DeeplyAssign(context, path, scalarNode)
-
-	test.AssertResult(t, nil, err)
-
-	// Verify the array assignment was made
-	arrayNode := rootNode.Content[1]                // The sequence node
-	test.AssertResult(t, 1, len(arrayNode.Content)) // Should have one element
-
-	arrayElement := arrayNode.Content[0]
-	test.AssertResult(t, "array_value", arrayElement.Value)
-}
-
-func TestDeeplyAssign_OverwriteExisting(t *testing.T) {
-	navigator := NewDataTreeNavigator()
-
-	// Create a context with a root mapping node
-	rootNode := &CandidateNode{
-		Kind: MappingNode,
-		Tag:  "!!map",
-		Content: []*CandidateNode{
-			{Kind: ScalarNode, Tag: "!!str", Value: "key", IsMapKey: true},
-			{Kind: ScalarNode, Tag: "!!str", Value: "old_value"},
-		},
-	}
-	context := Context{
-		MatchingNodes: list.New(),
-	}
-	context.MatchingNodes.PushBack(rootNode)
-
-	// Create a scalar node to assign
-	scalarNode := &CandidateNode{
-		Kind:  ScalarNode,
-		Tag:   "!!str",
-		Value: "new_value",
-	}
-
-	// Assign to existing path ["key"]
-	path := []interface{}{"key"}
-	err := navigator.DeeplyAssign(context, path, scalarNode)
-
-	test.AssertResult(t, nil, err)
-
-	// Verify the value was overwritten
-	test.AssertResult(t, 2, len(rootNode.Content)) // Should still have 2 elements
-
-	key := rootNode.Content[0]
-	value := rootNode.Content[1]
-	test.AssertResult(t, "key", key.Value)
-	test.AssertResult(t, "new_value", value.Value) // Should be overwritten
-}
-
-func TestDeeplyAssign_ErrorHandling(t *testing.T) {
-	navigator := NewDataTreeNavigator()
-
-	// Create a context with a scalar node (not a mapping)
-	scalarNode := &CandidateNode{
-		Kind:  ScalarNode,
-		Tag:   "!!str",
-		Value: "not_a_map",
-	}
-	context := Context{
-		MatchingNodes: list.New(),
-	}
-	context.MatchingNodes.PushBack(scalarNode)
-
-	// Create a scalar node to assign
-	assignNode := &CandidateNode{
-		Kind:  ScalarNode,
-		Tag:   "!!str",
-		Value: "value",
-	}
-
-	path := []interface{}{"key"}
-	err := navigator.DeeplyAssign(context, path, assignNode)
-
-	// Print the actual error for debugging
-	if err != nil {
-		t.Logf("Actual error: %v", err)
-	}
-
-	test.AssertResult(t, nil, err)
-}
-
-func TestGetMatchingNodes_WithVariables(t *testing.T) {
-	navigator := NewDataTreeNavigator()
-
-	// Create a context with variables
-	variables := make(map[string]*list.List)
-	varList := list.New()
-	varList.PushBack(&CandidateNode{Kind: ScalarNode, Tag: "!!str", Value: "var_value"})
-	variables["test_var"] = varList
-
-	context := Context{
-		MatchingNodes: list.New(),
-		Variables:     variables,
-	}
-
-	// Create an expression node that gets a variable
-	expressionNode := &ExpressionNode{
-		Operation: &Operation{OperationType: getVariableOpType, StringValue: "test_var"},
-	}
-
-	result, err := navigator.GetMatchingNodes(context, expressionNode)
-
-	test.AssertResult(t, nil, err)
-	test.AssertResult(t, 1, result.MatchingNodes.Len())
-
-	// Verify the variable was retrieved
-	resultNode := result.MatchingNodes.Front().Value.(*CandidateNode)
-	test.AssertResult(t, "var_value", resultNode.Value)
-}
-
-func TestGetMatchingNodes_EmptyContext(t *testing.T) {
-	navigator := NewDataTreeNavigator()
-
-	// Create an empty context
-	context := Context{
-		MatchingNodes: list.New(),
-	}
-
-	// Create an expression node with self reference
-	expressionNode := &ExpressionNode{
-		Operation: &Operation{OperationType: selfReferenceOpType},
-	}
-
-	result, err := navigator.GetMatchingNodes(context, expressionNode)
-
-	test.AssertResult(t, nil, err)
-	test.AssertResult(t, 0, result.MatchingNodes.Len())
-}
-
-func TestDeeplyAssign_ComplexMappingMerge(t *testing.T) {
-	navigator := NewDataTreeNavigator()
-
-	// Create a context with a root mapping node containing nested data
-	rootNode := &CandidateNode{
-		Kind: MappingNode,
-		Tag:  "!!map",
-		Content: []*CandidateNode{
-			{Kind: ScalarNode, Tag: "!!str", Value: "config", IsMapKey: true},
-			{Kind: MappingNode, Tag: "!!map", Content: []*CandidateNode{
-				{Kind: ScalarNode, Tag: "!!str", Value: "existing_key", IsMapKey: true},
-				{Kind: ScalarNode, Tag: "!!str", Value: "existing_value"},
-			}},
-		},
-	}
-	context := Context{
-		MatchingNodes: list.New(),
-	}
-	context.MatchingNodes.PushBack(rootNode)
-
-	// Create a mapping node to merge
-	mappingNode := &CandidateNode{
-		Kind: MappingNode,
-		Tag:  "!!map",
-		Content: []*CandidateNode{
-			{Kind: ScalarNode, Tag: "!!str", Value: "new_key", IsMapKey: true},
-			{Kind: ScalarNode, Tag: "!!str", Value: "new_value"},
-			{Kind: ScalarNode, Tag: "!!str", Value: "existing_key", IsMapKey: true},
-			{Kind: ScalarNode, Tag: "!!str", Value: "updated_value"},
-		},
-	}
-
-	// Assign to path ["config"] (should merge with existing mapping)
-	path := []interface{}{"config"}
-	err := navigator.DeeplyAssign(context, path, mappingNode)
-
-	test.AssertResult(t, nil, err)
-
-	// Verify the merge was successful
-	configNode := rootNode.Content[1]                // The config mapping node
-	test.AssertResult(t, 4, len(configNode.Content)) // Should have 2 key-value pairs
-
-	// Check that both existing and new keys are present
-	foundExisting := false
-	foundNew := false
-	for i := 0; i < len(configNode.Content); i += 2 {
-		if i+1 < len(configNode.Content) {
-			key := configNode.Content[i]
-			value := configNode.Content[i+1]
-			switch key.Value {
-			case "existing_key":
-				foundExisting = true
-				test.AssertResult(t, "updated_value", value.Value) // Should be updated
-			case "new_key":
-				foundNew = true
-				test.AssertResult(t, "new_value", value.Value)
-			}
-		}
-	}
-	test.AssertResult(t, true, foundExisting)
-	test.AssertResult(t, true, foundNew)
-}

pkg/yqlib/decoder_base64.go

@@ -1,5 +1,3 @@
-//go:build !yq_nobase64
-
 package yqlib
 
 import (
@@ -9,6 +7,28 @@ import (
 	"strings"
 )
 
+type base64Padder struct {
+	count uint64
+	io.Reader
+}
+
+func (c *base64Padder) pad(buf []byte) (int, error) {
+	pad := strings.Repeat("=", int(4-c.count%4))
+	n, err := strings.NewReader(pad).Read(buf)
+	c.count += uint64(n)
+	return n, err
+}
+
+func (c *base64Padder) Read(buf []byte) (int, error) {
+	n, err := c.Reader.Read(buf)
+	c.count += uint64(n)
+
+	if err == io.EOF && c.count%4 != 0 {
+		return c.pad(buf)
+	}
+	return n, err
+}
+
 type base64Decoder struct {
 	reader       io.Reader
 	finished     bool
@@ -21,25 +41,7 @@ func NewBase64Decoder() Decoder {
 }
 
 func (dec *base64Decoder) Init(reader io.Reader) error {
-	// Read all data from the reader and strip leading/trailing whitespace
-	// This is necessary because base64 decoding needs to see the complete input
-	// to handle padding correctly, and we need to strip whitespace before decoding.
-	buf := new(bytes.Buffer)
-	if _, err := buf.ReadFrom(reader); err != nil {
-		return err
-	}
-
-	// Strip leading and trailing whitespace
-	stripped := strings.TrimSpace(buf.String())
-
-	// Add padding if needed (base64 strings should be a multiple of 4 characters)
-	padLen := len(stripped) % 4
-	if padLen > 0 {
-		stripped += strings.Repeat("=", 4-padLen)
-	}
-
-	// Create a new reader from the stripped and padded data
-	dec.reader = strings.NewReader(stripped)
+	dec.reader = &base64Padder{Reader: reader}
 	dec.readAnything = false
 	dec.finished = false
 	return nil
@@ -66,5 +68,5 @@ func (dec *base64Decoder) Decode() (*CandidateNode, error) {
 		}
 	}
 	dec.readAnything = true
-	return createStringScalarNode(buf.String()), nil
+	return CreateStringScalarNode(buf.String()), nil
 }

pkg/yqlib/decoder_csv_object.go

@@ -1,5 +1,3 @@
-//go:build !yq_nocsv
-
 package yqlib
 
 import (
@@ -33,8 +31,8 @@ func (dec *csvObjectDecoder) convertToNode(content string) *CandidateNode {
 	node, err := parseSnippet(content)
 	// if we're not auto-parsing, then we wont put in parsed objects or arrays
 	// but we still parse scalars
-	if err != nil || (!dec.prefs.AutoParse && (node.Kind != ScalarNode || node.Value != content)) {
-		return createScalarNode(content, content)
+	if err != nil || (!dec.prefs.AutoParse && node.Kind != ScalarNode) {
+		return CreateScalarNode(content, content)
 	}
 	return node
 }
@@ -43,7 +41,7 @@ func (dec *csvObjectDecoder) createObject(headerRow []string, contentRow []strin
 	objectNode := &CandidateNode{Kind: MappingNode, Tag: "!!map"}
 
 	for i, header := range headerRow {
-		objectNode.AddKeyValueChild(createScalarNode(header, header), dec.convertToNode(contentRow[i]))
+		objectNode.AddKeyValueChild(CreateScalarNode(header, header), dec.convertToNode(contentRow[i]))
 	}
 	return objectNode
 }

pkg/yqlib/decoder_goccy_yaml.go

@@ -16,8 +16,6 @@ import (
 type goccyYamlDecoder struct {
 	decoder yaml.Decoder
 	cm      yaml.CommentMap
-	// anchor map persists over multiple documents for convenience.
-	anchorMap map[string]*CandidateNode
 }
 
 func NewGoccyYAMLDecoder() Decoder {
@@ -27,7 +25,6 @@ func NewGoccyYAMLDecoder() Decoder {
 func (dec *goccyYamlDecoder) Init(reader io.Reader) error {
 	dec.cm = yaml.CommentMap{}
 	dec.decoder = *yaml.NewDecoder(reader, yaml.CommentToMap(dec.cm), yaml.UseOrderedMap())
-	dec.anchorMap = make(map[string]*CandidateNode)
 	return nil
 }
 
@@ -41,7 +38,7 @@ func (dec *goccyYamlDecoder) Decode() (*CandidateNode, error) {
 	}
 
 	candidateNode := &CandidateNode{}
-	if err := candidateNode.UnmarshalGoccyYAML(ast, dec.cm, dec.anchorMap); err != nil {
+	if err := candidateNode.UnmarshalGoccyYAML(ast, dec.cm); err != nil {
 		return nil, err
 	}
 

pkg/yqlib/decoder_ini.go

@@ -1,106 +0,0 @@
-//go:build !yq_noini
-
-package yqlib
-
-import (
-	"fmt"
-	"io"
-
-	"github.com/go-ini/ini"
-)
-
-type iniDecoder struct {
-	reader   io.Reader
-	finished bool // Flag to signal completion of processing
-}
-
-func NewINIDecoder() Decoder {
-	return &iniDecoder{
-		finished: false, // Initialize the flag as false
-	}
-}
-
-func (dec *iniDecoder) Init(reader io.Reader) error {
-	// Store the reader for use in Decode
-	dec.reader = reader
-	return nil
-}
-
-func (dec *iniDecoder) Decode() (*CandidateNode, error) {
-	// If processing is already finished, return io.EOF
-	if dec.finished {
-		return nil, io.EOF
-	}
-
-	// Read all content from the stored reader
-	content, err := io.ReadAll(dec.reader)
-	if err != nil {
-		return nil, fmt.Errorf("failed to read INI content: %w", err)
-	}
-
-	// Parse the INI content
-	cfg, err := ini.Load(content)
-	if err != nil {
-		return nil, fmt.Errorf("failed to parse INI content: %w", err)
-	}
-
-	// Create a root CandidateNode as a MappingNode (since INI is key-value based)
-	root := &CandidateNode{
-		Kind:  MappingNode,
-		Tag:   "!!map",
-		Value: "",
-	}
-
-	// Process each section in the INI file
-	for _, section := range cfg.Sections() {
-		sectionName := section.Name()
-
-		if sectionName == ini.DefaultSection {
-			// For the default section, add key-value pairs directly to the root node
-			for _, key := range section.Keys() {
-				keyName := key.Name()
-				keyValue := key.String()
-
-				// Create a key node (scalar for the key name)
-				keyNode := createStringScalarNode(keyName)
-				// Create a value node (scalar for the value)
-				valueNode := createStringScalarNode(keyValue)
-
-				// Add key-value pair to the root node
-				root.AddKeyValueChild(keyNode, valueNode)
-			}
-		} else {
-			// For named sections, create a nested map
-			sectionNode := &CandidateNode{
-				Kind:  MappingNode,
-				Tag:   "!!map",
-				Value: "",
-			}
-
-			// Add key-value pairs to the section node
-			for _, key := range section.Keys() {
-				keyName := key.Name()
-				keyValue := key.String()
-
-				// Create a key node (scalar for the key name)
-				keyNode := createStringScalarNode(keyName)
-				// Create a value node (scalar for the value)
-				valueNode := createStringScalarNode(keyValue)
-
-				// Add key-value pair to the section node
-				sectionNode.AddKeyValueChild(keyNode, valueNode)
-			}
-
-			// Create a key node for the section name
-			sectionKeyNode := createStringScalarNode(sectionName)
-			// Add the section as a nested map to the root node
-			root.AddKeyValueChild(sectionKeyNode, sectionNode)
-		}
-	}
-
-	// Set the finished flag to true to prevent further Decode calls
-	dec.finished = true
-
-	// Return the root node
-	return root, nil
-}

pkg/yqlib/decoder_test.go

@@ -7,7 +7,7 @@ import (
 	"strings"
 )
 
-type formatScenario struct {
+type FormatScenario struct {
 	input          string
 	indent         int
 	expression     string
@@ -19,7 +19,7 @@ type formatScenario struct {
 	expectedError  string
 }
 
-func processFormatScenario(s formatScenario, decoder Decoder, encoder Encoder) (string, error) {
+func processFormatScenario(s FormatScenario, decoder Decoder, encoder Encoder) (string, error) {
 	var output bytes.Buffer
 	writer := bufio.NewWriter(&output)
 
@@ -64,12 +64,11 @@ func processFormatScenario(s formatScenario, decoder Decoder, encoder Encoder) (
 	return output.String(), nil
 }
 
-func mustProcessFormatScenario(s formatScenario, decoder Decoder, encoder Encoder) string {
+func mustProcessFormatScenario(s FormatScenario, decoder Decoder, encoder Encoder) string {
 
 	result, err := processFormatScenario(s, decoder, encoder)
 	if err != nil {
-		log.Error("Bad scenario %v: %w", s.description, err)
-		return fmt.Sprintf("Bad scenario %v: %v", s.description, err.Error())
+		panic(fmt.Errorf("Bad scenario %v: %w", s.description, err))
 	}
 	return result
 

pkg/yqlib/decoder_toml.go

@@ -145,30 +145,30 @@ func (dec *tomlDecoder) createArray(tomlNode *toml.Node) (*CandidateNode, error)
 
 func (dec *tomlDecoder) createStringScalar(tomlNode *toml.Node) (*CandidateNode, error) {
 	content := string(tomlNode.Data)
-	return createScalarNode(content, content), nil
+	return CreateScalarNode(content, content), nil
 }
 
 func (dec *tomlDecoder) createBoolScalar(tomlNode *toml.Node) (*CandidateNode, error) {
 	content := string(tomlNode.Data)
-	return createScalarNode(content == "true", content), nil
+	return CreateScalarNode(content == "true", content), nil
 }
 
 func (dec *tomlDecoder) createIntegerScalar(tomlNode *toml.Node) (*CandidateNode, error) {
 	content := string(tomlNode.Data)
 	_, num, err := parseInt64(content)
-	return createScalarNode(num, content), err
+	return CreateScalarNode(num, content), err
 }
 
 func (dec *tomlDecoder) createDateTimeScalar(tomlNode *toml.Node) (*CandidateNode, error) {
 	content := string(tomlNode.Data)
 	val, err := parseDateTime(time.RFC3339, content)
-	return createScalarNode(val, content), err
+	return CreateScalarNode(val, content), err
 }
 
 func (dec *tomlDecoder) createFloatScalar(tomlNode *toml.Node) (*CandidateNode, error) {
 	content := string(tomlNode.Data)
 	num, err := strconv.ParseFloat(content, 64)
-	return createScalarNode(num, content), err
+	return CreateScalarNode(num, content), err
 }
 
 func (dec *tomlDecoder) decodeNode(tomlNode *toml.Node) (*CandidateNode, error) {
@@ -249,12 +249,11 @@ func (dec *tomlDecoder) processTopLevelNode(currentNode *toml.Node) (bool, error
 	var runAgainstCurrentExp bool
 	var err error
 	log.Debug("processTopLevelNode: Going to process %v state is current %v", currentNode.Kind, NodeToString(dec.rootMap))
-	switch currentNode.Kind {
-	case toml.Table:
+	if currentNode.Kind == toml.Table {
 		runAgainstCurrentExp, err = dec.processTable(currentNode)
-	case toml.ArrayTable:
+	} else if currentNode.Kind == toml.ArrayTable {
 		runAgainstCurrentExp, err = dec.processArrayTable(currentNode)
-	default:
+	} else {
 		runAgainstCurrentExp, err = dec.decodeKeyValuesIntoMap(dec.rootMap, currentNode)
 	}
 
@@ -282,13 +281,13 @@ func (dec *tomlDecoder) processTable(currentNode *toml.Node) (bool, error) {
 		tableValue = dec.parser.Expression()
 		// next expression is not table data, so we are done
 		if tableValue.Kind != toml.KeyValue {
-			log.Debug("got an empty table")
-			runAgainstCurrentExp = true
-		} else {
-			runAgainstCurrentExp, err = dec.decodeKeyValuesIntoMap(tableNodeValue, tableValue)
-			if err != nil && !errors.Is(err, io.EOF) {
-				return false, err
-			}
+			log.Debug("got an empty table, returning")
+			return true, nil
+		}
+
+		runAgainstCurrentExp, err = dec.decodeKeyValuesIntoMap(tableNodeValue, tableValue)
+		if err != nil && !errors.Is(io.EOF, err) {
+			return false, err
 		}
 	}
 
@@ -344,7 +343,7 @@ func (dec *tomlDecoder) processArrayTable(currentNode *toml.Node) (bool, error)
 	tableValue := dec.parser.Expression()
 	runAgainstCurrentExp, err := dec.decodeKeyValuesIntoMap(tableNodeValue, tableValue)
 	log.Debugf("table node err: %w", err)
-	if err != nil && !errors.Is(err, io.EOF) {
+	if err != nil && !errors.Is(io.EOF, err) {
 		return false, err
 	}
 	c := Context{}

pkg/yqlib/decoder_uri.go

@@ -1,5 +1,3 @@
-//go:build !yq_nouri
-
 package yqlib
 
 import (
@@ -50,5 +48,5 @@ func (dec *uriDecoder) Decode() (*CandidateNode, error) {
 		return nil, err
 	}
 	dec.readAnything = true
-	return createStringScalarNode(newValue), nil
+	return CreateStringScalarNode(newValue), nil
 }

pkg/yqlib/decoder_yaml.go

@@ -8,7 +8,7 @@ import (
 	"regexp"
 	"strings"
 
-	yaml "go.yaml.in/yaml/v4"
+	yaml "gopkg.in/yaml.v3"
 )
 
 type yamlDecoder struct {
@@ -129,8 +129,6 @@ func (dec *yamlDecoder) Decode() (*CandidateNode, error) {
 		return nil, err
 	} else if err != nil {
 		return nil, err
-	} else if len(yamlNode.Content) == 0 {
-		return nil, errors.New("yaml node has no content")
 	}
 
 	candidateNode := CandidateNode{document: dec.documentIndex}
@@ -153,7 +151,7 @@ func (dec *yamlDecoder) Decode() (*CandidateNode, error) {
 }
 
 func (dec *yamlDecoder) blankNodeWithComment() *CandidateNode {
-	node := createScalarNode(nil, "")
+	node := CreateScalarNode(nil, "")
 	node.LeadingContent = dec.leadingContent
 	return node
 }

pkg/yqlib/doc/operators/anchor-and-alias-operators.md

@@ -5,18 +5,6 @@ Use the `alias` and `anchor` operators to read and write yaml aliases and anchor
 `yq` supports merge aliases (like `<<: *blah`) however this is no longer in the standard yaml spec (1.2) and so `yq` will automatically add the `!!merge` tag to these nodes as it is effectively a custom tag.
 
 
-## NOTE --yaml-fix-merge-anchor-to-spec flag
-`yq` doesn't merge anchors `<<:` to spec, in some circumstances it incorrectly overrides existing keys when the spec documents not to do that.
-
-To minimise disruption while still fixing the issue, a flag has been added to toggle this behaviour. This will first default to false; and log warnings to users. Then it will default to true (and still allow users to specify false if needed).
-
-This flag also enables advanced merging, like inline maps, as well as fixes to ensure when exploding a particular path, neighbours are not affect ed.
-
-Long story short, you should be setting this flag to true.
-
-See examples of the flag differences below, where LEGACY is with the flag off; and FIXED is with the flag on.
-
-
 ## Merge one map
 see https://yaml.org/type/merge.html
 
@@ -46,6 +34,68 @@ y: 2
 r: 10
 ```
 
+## Merge multiple maps
+see https://yaml.org/type/merge.html
+
+Given a sample.yml file of:
+```yaml
+- &CENTER
+  x: 1
+  y: 2
+- &LEFT
+  x: 0
+  y: 2
+- &BIG
+  r: 10
+- &SMALL
+  r: 1
+- !!merge <<:
+    - *CENTER
+    - *BIG
+```
+then
+```bash
+yq '.[4] | explode(.)' sample.yml
+```
+will output
+```yaml
+r: 10
+x: 1
+y: 2
+```
+
+## Override
+see https://yaml.org/type/merge.html
+
+Given a sample.yml file of:
+```yaml
+- &CENTER
+  x: 1
+  y: 2
+- &LEFT
+  x: 0
+  y: 2
+- &BIG
+  r: 10
+- &SMALL
+  r: 1
+- !!merge <<:
+    - *BIG
+    - *LEFT
+    - *SMALL
+  x: 1
+```
+then
+```bash
+yq '.[4] | explode(.)' sample.yml
+```
+will output
+```yaml
+r: 10
+x: 1
+y: 2
+```
+
 ## Get anchor
 Given a sample.yml file of:
 ```yaml
@@ -191,7 +241,7 @@ Given a sample.yml file of:
 ```yaml
 f:
   a: &a cat
-  *a : b
+  *a: b
 ```
 then
 ```bash
@@ -204,6 +254,53 @@ f:
   cat: b
 ```
 
+## Explode with merge anchors
+Given a sample.yml file of:
+```yaml
+foo: &foo
+  a: foo_a
+  thing: foo_thing
+  c: foo_c
+bar: &bar
+  b: bar_b
+  thing: bar_thing
+  c: bar_c
+foobarList:
+  b: foobarList_b
+  !!merge <<:
+    - *foo
+    - *bar
+  c: foobarList_c
+foobar:
+  c: foobar_c
+  !!merge <<: *foo
+  thing: foobar_thing
+```
+then
+```bash
+yq 'explode(.)' sample.yml
+```
+will output
+```yaml
+foo:
+  a: foo_a
+  thing: foo_thing
+  c: foo_c
+bar:
+  b: bar_b
+  thing: bar_thing
+  c: bar_c
+foobarList:
+  b: bar_b
+  thing: foo_thing
+  c: foobarList_c
+  a: foo_a
+foobar:
+  c: foo_c
+  a: foo_a
+  thing: foobar_thing
+```
+
 ## Dereference and update a field
 Use explode with multiply to dereference an object
 
@@ -220,7 +317,7 @@ thingTwo:
 ```
 then
 ```bash
-yq '.thingOne |= (explode(.) | sort_keys(.)) * {"value": false}' sample.yml
+yq '.thingOne |= explode(.) * {"value": false}' sample.yml
 ```
 will output
 ```yaml
@@ -234,250 +331,3 @@ thingTwo:
   !!merge <<: *item_value
 ```
 
-## LEGACY: Explode with merge anchors
-Caution: this is for when --yaml-fix-merge-anchor-to-spec=false; it's not to YAML spec because the merge anchors incorrectly override the object values (foobarList.b is set to bar_b when it should still be foobarList_b). Flag will default to true in late 2025
-
-Given a sample.yml file of:
-```yaml
-foo: &foo
-  a: foo_a
-  thing: foo_thing
-  c: foo_c
-bar: &bar
-  b: bar_b
-  thing: bar_thing
-  c: bar_c
-foobarList:
-  b: foobarList_b
-  !!merge <<:
-    - *foo
-    - *bar
-  c: foobarList_c
-foobar:
-  c: foobar_c
-  !!merge <<: *foo
-  thing: foobar_thing
-```
-then
-```bash
-yq 'explode(.)' sample.yml
-```
-will output
-```yaml
-foo:
-  a: foo_a
-  thing: foo_thing
-  c: foo_c
-bar:
-  b: bar_b
-  thing: bar_thing
-  c: bar_c
-foobarList:
-  b: bar_b
-  thing: foo_thing
-  c: foobarList_c
-  a: foo_a
-foobar:
-  c: foo_c
-  a: foo_a
-  thing: foobar_thing
-```
-
-## LEGACY: Merge multiple maps
-see https://yaml.org/type/merge.html. This has the correct data, but the wrong key order; set --yaml-fix-merge-anchor-to-spec=true to fix the key order.
-
-Given a sample.yml file of:
-```yaml
-- &CENTER
-  x: 1
-  y: 2
-- &LEFT
-  x: 0
-  y: 2
-- &BIG
-  r: 10
-- &SMALL
-  r: 1
-- !!merge <<:
-    - *CENTER
-    - *BIG
-```
-then
-```bash
-yq '.[4] | explode(.)' sample.yml
-```
-will output
-```yaml
-r: 10
-x: 1
-y: 2
-```
-
-## LEGACY: Override
-see https://yaml.org/type/merge.html. This has the correct data, but the wrong key order; set --yaml-fix-merge-anchor-to-spec=true to fix the key order.
-
-Given a sample.yml file of:
-```yaml
-- &CENTER
-  x: 1
-  y: 2
-- &LEFT
-  x: 0
-  y: 2
-- &BIG
-  r: 10
-- &SMALL
-  r: 1
-- !!merge <<:
-    - *BIG
-    - *LEFT
-    - *SMALL
-  x: 1
-```
-then
-```bash
-yq '.[4] | explode(.)' sample.yml
-```
-will output
-```yaml
-r: 10
-x: 1
-y: 2
-```
-
-## FIXED: Explode with merge anchors
-Set `--yaml-fix-merge-anchor-to-spec=true` to get this correct merge behaviour (flag will default to true in late 2025).
-Observe that foobarList.b property is still foobarList_b.
-
-Given a sample.yml file of:
-```yaml
-foo: &foo
-  a: foo_a
-  thing: foo_thing
-  c: foo_c
-bar: &bar
-  b: bar_b
-  thing: bar_thing
-  c: bar_c
-foobarList:
-  b: foobarList_b
-  !!merge <<:
-    - *foo
-    - *bar
-  c: foobarList_c
-foobar:
-  c: foobar_c
-  !!merge <<: *foo
-  thing: foobar_thing
-```
-then
-```bash
-yq 'explode(.)' sample.yml
-```
-will output
-```yaml
-foo:
-  a: foo_a
-  thing: foo_thing
-  c: foo_c
-bar:
-  b: bar_b
-  thing: bar_thing
-  c: bar_c
-foobarList:
-  b: foobarList_b
-  a: foo_a
-  thing: foo_thing
-  c: foobarList_c
-foobar:
-  c: foobar_c
-  a: foo_a
-  thing: foobar_thing
-```
-
-## FIXED: Merge multiple maps
-Set `--yaml-fix-merge-anchor-to-spec=true` to get this correct merge behaviour (flag will default to true in late 2025).
-Taken from https://yaml.org/type/merge.html. Same values as legacy, but with the correct key order.
-
-Given a sample.yml file of:
-```yaml
-- &CENTER
-  x: 1
-  y: 2
-- &LEFT
-  x: 0
-  y: 2
-- &BIG
-  r: 10
-- &SMALL
-  r: 1
-- !!merge <<:
-    - *CENTER
-    - *BIG
-```
-then
-```bash
-yq '.[4] | explode(.)' sample.yml
-```
-will output
-```yaml
-x: 1
-y: 2
-r: 10
-```
-
-## FIXED: Override
-Set `--yaml-fix-merge-anchor-to-spec=true` to get this correct merge behaviour (flag will default to true in late 2025).
-Taken from https://yaml.org/type/merge.html. Same values as legacy, but with the correct key order.
-
-Given a sample.yml file of:
-```yaml
-- &CENTER
-  x: 1
-  y: 2
-- &LEFT
-  x: 0
-  y: 2
-- &BIG
-  r: 10
-- &SMALL
-  r: 1
-- !!merge <<:
-    - *BIG
-    - *LEFT
-    - *SMALL
-  x: 1
-```
-then
-```bash
-yq '.[4] | explode(.)' sample.yml
-```
-will output
-```yaml
-r: 10
-y: 2
-x: 1
-```
-
-## Exploding inline merge anchor
-Set `--yaml-fix-merge-anchor-to-spec=true` to get this correct merge behaviour (flag will default to true in late 2025).
-
-
-Given a sample.yml file of:
-```yaml
-a:
-  b: &b 42
-!!merge <<:
-  c: *b
-```
-then
-```bash
-yq 'explode(.) | sort_keys(.)' sample.yml
-```
-will output
-```yaml
-a:
-  b: 42
-c: 42
-```
-

pkg/yqlib/doc/operators/entries.md

@@ -2,8 +2,6 @@
 
 Similar to the same named functions in `jq` these functions convert to/from an object and an array of key-value pairs. This is most useful for performing operations on keys of maps.
 
-Use `with_entries(op)` as a syntactic sugar for doing `to_entries | op | from_entries`.
-
 ## to_entries Map
 Given a sample.yml file of:
 ```yaml
@@ -103,28 +101,6 @@ KEY_a: 1
 KEY_b: 2
 ```
 
-## Use with_entries to update keys recursively
-We use (.. | select(tag="map")) to find all the maps in the doc, then |= to update each one of those maps. In the update, with_entries is used.
-
-Given a sample.yml file of:
-```yaml
-a: 1
-b:
-  b_a: nested
-  b_b: thing
-```
-then
-```bash
-yq '(.. | select(tag=="!!map")) |= with_entries(.key |= "KEY_" + .)' sample.yml
-```
-will output
-```yaml
-KEY_a: 1
-KEY_b:
-  KEY_b_a: nested
-  KEY_b_b: thing
-```
-
 ## Custom sort map keys
 Use to_entries to convert to an array of key/value pairs, sort the array using sort/sort_by/etc, and convert it back.
 

pkg/yqlib/doc/operators/first.md

@@ -1,319 +0,0 @@
-# First
-
-Returns the first matching element in an array, or first matching value in a map.
-
-Can be given an expression to match with, otherwise will just return the first.
-
-## First matching element from array
-Given a sample.yml file of:
-```yaml
-- a: banana
-- a: cat
-- a: apple
-```
-then
-```bash
-yq 'first(.a == "cat")' sample.yml
-```
-will output
-```yaml
-a: cat
-```
-
-## First matching element from array with multiple matches
-Given a sample.yml file of:
-```yaml
-- a: banana
-- a: cat
-  b: firstCat
-- a: apple
-- a: cat
-  b: secondCat
-```
-then
-```bash
-yq 'first(.a == "cat")' sample.yml
-```
-will output
-```yaml
-a: cat
-b: firstCat
-```
-
-## First matching element from array with numeric condition
-Given a sample.yml file of:
-```yaml
-- a: 10
-- a: 100
-- a: 1
-- a: 101
-```
-then
-```bash
-yq 'first(.a > 50)' sample.yml
-```
-will output
-```yaml
-a: 100
-```
-
-## First matching element from array with boolean condition
-Given a sample.yml file of:
-```yaml
-- a: false
-- a: true
-  b: firstTrue
-- a: false
-- a: true
-  b: secondTrue
-```
-then
-```bash
-yq 'first(.a == true)' sample.yml
-```
-will output
-```yaml
-a: true
-b: firstTrue
-```
-
-## First matching element from array with null values
-Given a sample.yml file of:
-```yaml
-- a: null
-- a: cat
-- a: apple
-```
-then
-```bash
-yq 'first(.a != null)' sample.yml
-```
-will output
-```yaml
-a: cat
-```
-
-## First matching element from array with complex condition
-Given a sample.yml file of:
-```yaml
-- a: dog
-  b: 7
-- a: cat
-  b: 3
-- a: apple
-  b: 5
-```
-then
-```bash
-yq 'first(.b > 4 and .b < 6)' sample.yml
-```
-will output
-```yaml
-a: apple
-b: 5
-```
-
-## First matching element from map
-Given a sample.yml file of:
-```yaml
-x:
-  a: banana
-y:
-  a: cat
-z:
-  a: apple
-```
-then
-```bash
-yq 'first(.a == "cat")' sample.yml
-```
-will output
-```yaml
-a: cat
-```
-
-## First matching element from map with numeric condition
-Given a sample.yml file of:
-```yaml
-x:
-  a: 10
-y:
-  a: 100
-z:
-  a: 101
-```
-then
-```bash
-yq 'first(.a > 50)' sample.yml
-```
-will output
-```yaml
-a: 100
-```
-
-## First matching element from nested structure
-Given a sample.yml file of:
-```yaml
-items:
-  - a: banana
-  - a: cat
-  - a: apple
-```
-then
-```bash
-yq '.items | first(.a == "cat")' sample.yml
-```
-will output
-```yaml
-a: cat
-```
-
-## First matching element with no matches
-Given a sample.yml file of:
-```yaml
-- a: banana
-- a: cat
-- a: apple
-```
-then
-```bash
-yq 'first(.a == "dog")' sample.yml
-```
-will output
-```yaml
-```
-
-## First matching element from empty array
-Given a sample.yml file of:
-```yaml
-[]
-```
-then
-```bash
-yq 'first(.a == "cat")' sample.yml
-```
-will output
-```yaml
-```
-
-## First matching element from scalar node
-Given a sample.yml file of:
-```yaml
-hello
-```
-then
-```bash
-yq 'first(. == "hello")' sample.yml
-```
-will output
-```yaml
-```
-
-## First matching element from null node
-Given a sample.yml file of:
-```yaml
-null
-```
-then
-```bash
-yq 'first(. == "hello")' sample.yml
-```
-will output
-```yaml
-```
-
-## First matching element with string condition
-Given a sample.yml file of:
-```yaml
-- a: banana
-- a: cat
-- a: apple
-```
-then
-```bash
-yq 'first(.a | test("^c"))' sample.yml
-```
-will output
-```yaml
-a: cat
-```
-
-## First matching element with length condition
-Given a sample.yml file of:
-```yaml
-- a: hi
-- a: hello
-- a: world
-```
-then
-```bash
-yq 'first(.a | length > 4)' sample.yml
-```
-will output
-```yaml
-a: hello
-```
-
-## First matching element from array of strings
-Given a sample.yml file of:
-```yaml
-- banana
-- cat
-- apple
-```
-then
-```bash
-yq 'first(. == "cat")' sample.yml
-```
-will output
-```yaml
-cat
-```
-
-## First matching element from array of numbers
-Given a sample.yml file of:
-```yaml
-- 10
-- 100
-- 1
-```
-then
-```bash
-yq 'first(. > 50)' sample.yml
-```
-will output
-```yaml
-100
-```
-
-## First element with no filter from array
-Given a sample.yml file of:
-```yaml
-- 10
-- 100
-- 1
-```
-then
-```bash
-yq 'first' sample.yml
-```
-will output
-```yaml
-10
-```
-
-## First element with no filter from array of maps
-Given a sample.yml file of:
-```yaml
-- a: 10
-- a: 100
-```
-then
-```bash
-yq 'first' sample.yml
-```
-will output
-```yaml
-a: 10
-```
-

pkg/yqlib/doc/operators/headers/anchor-and-alias-operators.md

@@ -4,15 +4,3 @@ Use the `alias` and `anchor` operators to read and write yaml aliases and anchor
 
 `yq` supports merge aliases (like `<<: *blah`) however this is no longer in the standard yaml spec (1.2) and so `yq` will automatically add the `!!merge` tag to these nodes as it is effectively a custom tag.
 
-
-## NOTE --yaml-fix-merge-anchor-to-spec flag
-`yq` doesn't merge anchors `<<:` to spec, in some circumstances it incorrectly overrides existing keys when the spec documents not to do that.
-
-To minimise disruption while still fixing the issue, a flag has been added to toggle this behaviour. This will first default to false; and log warnings to users. Then it will default to true (and still allow users to specify false if needed).
-
-This flag also enables advanced merging, like inline maps, as well as fixes to ensure when exploding a particular path, neighbours are not affect ed.
-
-Long story short, you should be setting this flag to true.
-
-See examples of the flag differences below, where LEGACY is with the flag off; and FIXED is with the flag on.
-

pkg/yqlib/doc/operators/headers/entries.md

@@ -1,5 +1,3 @@
 # Entries
 
 Similar to the same named functions in `jq` these functions convert to/from an object and an array of key-value pairs. This is most useful for performing operations on keys of maps.
-
-Use `with_entries(op)` as a syntactic sugar for doing `to_entries | op | from_entries`.

pkg/yqlib/doc/operators/headers/first.md

@@ -1,5 +0,0 @@
-# First
-
-Returns the first matching element in an array, or first matching value in a map.
-
-Can be given an expression to match with, otherwise will just return the first.

pkg/yqlib/doc/operators/headers/kind.md

@@ -1,7 +0,0 @@
-# Kind
-
-The `kind` operator identifies the type of a node as either `scalar`, `map`, or `seq`.
-
-This can be used for filtering or transforming nodes based on their type.
-
-Note that `null` values are treated as `scalar`.

pkg/yqlib/doc/operators/headers/max.md

@@ -1,3 +0,0 @@
-# Max
-
-Computes the maximum among an incoming sequence of scalar values.

pkg/yqlib/doc/operators/headers/min.md

@@ -1,3 +0,0 @@
-# Min
-
-Computes the minimum among an incoming sequence of scalar values.

pkg/yqlib/doc/operators/headers/omit.md

@@ -1,3 +0,0 @@
-# Omit
-
-Works like `pick`, but instead you specify the keys/indices that you _don't_ want included.

pkg/yqlib/doc/operators/headers/pivot.md

@@ -1,3 +0,0 @@
-# Pivot
-
-Emulates the `PIVOT` function supported by several popular RDBMS systems.

pkg/yqlib/doc/operators/headers/slice-array.md

@@ -2,4 +2,4 @@
 
 The slice array operator takes an array as input and returns a subarray. Like the `jq` equivalent, `.[10:15]` will return an array of length 5, starting from index 10 inclusive, up to index 15 exclusive. Negative numbers count backwards from the end of the array.
 
-You may leave out the first or second number, which will refer to the start or end of the array respectively.
+You may leave out the first or second number, which will will refer to the start or end of the array respectively.

pkg/yqlib/doc/operators/headers/sort-keys.md

@@ -12,5 +12,5 @@ diff file1.yml file2.yml
 
 Note that `yq` does not yet consider anchors when sorting by keys - this may result in invalid yaml documents if you are using merge anchors.
 
-For more advanced sorting, you can use the [sort_by](https://mikefarah.gitbook.io/yq/operators/sort) function on a map, and give it a custom function like `sort_by(key | downcase)`.
-
+For more advanced sorting, using `to_entries` to convert the map to an array, then sort/process the array as you like (e.g. using `sort_by`) and convert back to a map using `from_entries`.
+See [here](https://mikefarah.gitbook.io/yq/operators/entries#custom-sort-map-keys) for an example. 

pkg/yqlib/doc/operators/headers/string-operators.md

@@ -3,7 +3,7 @@
 ## RegEx
 This uses Golang's native regex functions under the hood - See their [docs](https://github.com/google/re2/wiki/Syntax) for the supported syntax.
 
-Case insensitive tip: prefix the regex with `(?i)` - e.g. `test("(?i)cats")`.
+Case insensitive tip: prefix the regex with `(?i)` - e.g. `test("(?i)cats)"`.
 
 ### match(regEx)
 This operator returns the substring match details of the given regEx.

pkg/yqlib/doc/operators/headers/traverse-read.md

@@ -1,12 +1,3 @@
 # Traverse (Read)
 
 This is the simplest (and perhaps most used) operator. It is used to navigate deeply into yaml structures.
-
-
-## NOTE --yaml-fix-merge-anchor-to-spec flag
-`yq` doesn't merge anchors `<<:` to spec, in some circumstances it incorrectly overrides existing keys when the spec documents not to do that.
-
-To minimise disruption while still fixing the issue, a flag has been added to toggle this behaviour. This will first default to false; and log warnings to users. Then it will default to true (and still allow users to specify false if needed)
-
-See examples of the flag differences below, where LEGACY is the flag off; and FIXED is with the flag on.
-

pkg/yqlib/doc/operators/kind.md

@@ -1,10 +1,3 @@
-# Kind
-
-The `kind` operator identifies the type of a node as either `scalar`, `map`, or `seq`.
-
-This can be used for filtering or transforming nodes based on their type.
-
-Note that `null` values are treated as `scalar`.
 
 ## Get kind
 Given a sample.yml file of:

pkg/yqlib/doc/operators/max.md

@@ -1,51 +0,0 @@
-# Max
-
-Computes the maximum among an incoming sequence of scalar values.
-
-## Maximum int
-Given a sample.yml file of:
-```yaml
-- 99
-- 16
-- 12
-- 6
-- 66
-```
-then
-```bash
-yq 'max' sample.yml
-```
-will output
-```yaml
-99
-```
-
-## Maximum string
-Given a sample.yml file of:
-```yaml
-- foo
-- bar
-- baz
-```
-then
-```bash
-yq 'max' sample.yml
-```
-will output
-```yaml
-foo
-```
-
-## Maximum of empty
-Given a sample.yml file of:
-```yaml
-[]
-```
-then
-```bash
-yq 'max' sample.yml
-```
-will output
-```yaml
-```
-

pkg/yqlib/doc/operators/min.md

@@ -1,51 +0,0 @@
-# Min
-
-Computes the minimum among an incoming sequence of scalar values.
-
-## Minimum int
-Given a sample.yml file of:
-```yaml
-- 99
-- 16
-- 12
-- 6
-- 66
-```
-then
-```bash
-yq 'min' sample.yml
-```
-will output
-```yaml
-6
-```
-
-## Minimum string
-Given a sample.yml file of:
-```yaml
-- foo
-- bar
-- baz
-```
-then
-```bash
-yq 'min' sample.yml
-```
-will output
-```yaml
-bar
-```
-
-## Minimum of empty
-Given a sample.yml file of:
-```yaml
-[]
-```
-then
-```bash
-yq 'min' sample.yml
-```
-will output
-```yaml
-```
-

pkg/yqlib/doc/operators/multiply-merge.md

@@ -55,62 +55,6 @@ a: 12
 b: 4
 ```
 
-## Multiply string node X int
-Given a sample.yml file of:
-```yaml
-b: banana
-```
-then
-```bash
-yq '.b * 4' sample.yml
-```
-will output
-```yaml
-bananabananabananabanana
-```
-
-## Multiply int X string node
-Given a sample.yml file of:
-```yaml
-b: banana
-```
-then
-```bash
-yq '4 * .b' sample.yml
-```
-will output
-```yaml
-bananabananabananabanana
-```
-
-## Multiply string X int node
-Given a sample.yml file of:
-```yaml
-n: 4
-```
-then
-```bash
-yq '"banana" * .n' sample.yml
-```
-will output
-```yaml
-bananabananabananabanana
-```
-
-## Multiply int node X string
-Given a sample.yml file of:
-```yaml
-n: 4
-```
-then
-```bash
-yq '.n * "banana"' sample.yml
-```
-will output
-```yaml
-bananabananabananabanana
-```
-
 ## Merge objects together, returning merged result only
 Given a sample.yml file of:
 ```yaml

pkg/yqlib/doc/operators/omit.md

@@ -1,44 +0,0 @@
-# Omit
-
-Works like `pick`, but instead you specify the keys/indices that you _don't_ want included.
-
-## Omit keys from map
-Note that non existent keys are skipped.
-
-Given a sample.yml file of:
-```yaml
-myMap:
-  cat: meow
-  dog: bark
-  thing: hamster
-  hamster: squeak
-```
-then
-```bash
-yq '.myMap |= omit(["hamster", "cat", "goat"])' sample.yml
-```
-will output
-```yaml
-myMap:
-  dog: bark
-  thing: hamster
-```
-
-## Omit indices from array
-Note that non existent indices are skipped.
-
-Given a sample.yml file of:
-```yaml
-- cat
-- leopard
-- lion
-```
-then
-```bash
-yq 'omit([2, 0, 734, -5])' sample.yml
-```
-will output
-```yaml
-- leopard
-```
-

pkg/yqlib/doc/operators/parent.md

@@ -37,48 +37,6 @@ fruit: banana
 name: sam
 ```
 
-## Get parent attribute
-Given a sample.yml file of:
-```yaml
-a:
-  fruit: apple
-  name: bob
-b:
-  fruit: banana
-  name: sam
-```
-then
-```bash
-yq '.. | select(. == "banana") | parent.name' sample.yml
-```
-will output
-```yaml
-sam
-```
-
-## Get parents
-Match all parents
-
-Given a sample.yml file of:
-```yaml
-a:
-  b:
-    c: cat
-```
-then
-```bash
-yq '.a.b.c | parents' sample.yml
-```
-will output
-```yaml
-- c: cat
-- b:
-    c: cat
-- a:
-    b:
-      c: cat
-```
-
 ## N-th parent
 You can optionally supply the number of levels to go up for the parent, the default being 1.
 

pkg/yqlib/doc/operators/pick.md

@@ -26,30 +26,6 @@ myMap:
   cat: meow
 ```
 
-## Pick keys from map, included all the keys
-We create a map of the picked keys plus all the current keys, and run that through unique
-
-Given a sample.yml file of:
-```yaml
-myMap:
-  cat: meow
-  dog: bark
-  thing: hamster
-  hamster: squeak
-```
-then
-```bash
-yq '.myMap |= pick( (["thing"] + keys) | unique)' sample.yml
-```
-will output
-```yaml
-myMap:
-  thing: hamster
-  cat: meow
-  dog: bark
-  hamster: squeak
-```
-
 ## Pick indices from array
 Note that the order of the indices matches the pick order and non existent indices are skipped.
 

pkg/yqlib/doc/operators/pivot.md

@@ -1,117 +0,0 @@
-# Pivot
-
-Emulates the `PIVOT` function supported by several popular RDBMS systems.
-
-## Pivot a sequence of sequences
-Given a sample.yml file of:
-```yaml
-- - foo
-  - bar
-  - baz
-- - sis
-  - boom
-  - bah
-```
-then
-```bash
-yq 'pivot' sample.yml
-```
-will output
-```yaml
-- - foo
-  - sis
-- - bar
-  - boom
-- - baz
-  - bah
-```
-
-## Pivot sequence of heterogeneous sequences
-Missing values are "padded" to null.
-
-Given a sample.yml file of:
-```yaml
-- - foo
-  - bar
-  - baz
-- - sis
-  - boom
-  - bah
-  - blah
-```
-then
-```bash
-yq 'pivot' sample.yml
-```
-will output
-```yaml
-- - foo
-  - sis
-- - bar
-  - boom
-- - baz
-  - bah
-- -
-  - blah
-```
-
-## Pivot sequence of maps
-Given a sample.yml file of:
-```yaml
-- foo: a
-  bar: b
-  baz: c
-- foo: x
-  bar: y
-  baz: z
-```
-then
-```bash
-yq 'pivot' sample.yml
-```
-will output
-```yaml
-foo:
-  - a
-  - x
-bar:
-  - b
-  - y
-baz:
-  - c
-  - z
-```
-
-## Pivot sequence of heterogeneous maps
-Missing values are "padded" to null.
-
-Given a sample.yml file of:
-```yaml
-- foo: a
-  bar: b
-  baz: c
-- foo: x
-  bar: y
-  baz: z
-  what: ever
-```
-then
-```bash
-yq 'pivot' sample.yml
-```
-will output
-```yaml
-foo:
-  - a
-  - x
-bar:
-  - b
-  - y
-baz:
-  - c
-  - z
-what:
-  -
-  - ever
-```
-

pkg/yqlib/doc/operators/slice-array.md

@@ -2,7 +2,7 @@
 
 The slice array operator takes an array as input and returns a subarray. Like the `jq` equivalent, `.[10:15]` will return an array of length 5, starting from index 10 inclusive, up to index 15 exclusive. Negative numbers count backwards from the end of the array.
 
-You may leave out the first or second number, which will refer to the start or end of the array respectively.
+You may leave out the first or second number, which will will refer to the start or end of the array respectively.
 
 ## Slicing arrays
 Given a sample.yml file of: