Fixing merge anchor key order

also add a unit test for func FormatStringFromFilename to cover such case

.dockerignore

@@ -1 +1 @@
-bin
+bin/*

.github/FUNDING.yml

@@ -0,0 +1 @@
+github: mikefarah

.github/ISSUE_TEMPLATE/bug_report_v4.md

@@ -1,8 +1,8 @@
 ---
-name: Bug report
+name: Bug report - V4
 about: Create a report to help us improve
 title: ''
-labels: bug
+labels: bug, v4
 assignees: ''
 
 ---
@@ -10,8 +10,14 @@ assignees: ''
 **Describe the bug**
 A clear and concise description of what the bug is.
 
+Note that any how to questions should be posted in the discussion board and not raised as an issue.
+
+Version of yq: 4.X.X
+Operating system: mac/linux/windows/....
+Installed via: docker/binary release/homebrew/snap/...
+
 **Input Yaml**
-Concise yaml document(s) (as simple as possible to show the bug)
+Concise yaml document(s) (as simple as possible to show the bug, please keep it to 10 lines or less)
 data1.yml:
 ```yaml
 this: should really work
@@ -25,7 +31,7 @@ but: it strangely didn't
 **Command**
 The command you ran:
 ```
-yq merge data1.yml data2.yml
+yq eval-all 'select(fileIndex==0) | .a.b.c' data1.yml data2.yml
 ```
 
 **Actual behavior**

.github/ISSUE_TEMPLATE/feature_request.md

@@ -1,17 +1,23 @@
 ---
-name: Feature request
+name: Feature request - V4
 about: Suggest an idea for this project
 title: ''
-labels: enhancement
+labels: enhancement, v4
 assignees: ''
 
 ---
 
-**Is your feature request related to a problem? Please describe.**
-A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+**Please describe your feature request.**
+A clear and concise description of what the request is and what it would solve. 
+Eg. I wish I could use yq to [...]
+
+Note:
+- how to questions should be posted in the discussion board and not raised as an issue.
+- V3 will no longer have any enhancements.
 
 **Describe the solution you'd like**
 If we have data1.yml like:
+(please keep to around 10 lines )
 
 ```yaml
 country: Australia
@@ -20,7 +26,7 @@ country: Australia
 And we run a command:
 
 ```bash
-yq predictWeather data1.yml
+yq 'predictWeatherOf(.country)'
 ```
 
 it could output

.github/dependabot.yml

@@ -0,0 +1,22 @@
+# To get started with Dependabot version updates, you'll need to specify which
+# package ecosystems to update and where the package manifests are located.
+# Please see the documentation for all configuration options:
+# https://help.github.com/github/administering-a-repository/configuration-options-for-dependency-updates
+
+version: 2
+updates:
+  - package-ecosystem: docker
+    directory: /
+    schedule:
+      day: thursday
+      interval: weekly
+  - package-ecosystem: github-actions
+    directory: /
+    schedule:
+      day: thursday
+      interval: weekly
+  - package-ecosystem: gomod
+    directory: /
+    schedule:
+      day: thursday
+      interval: weekly

.github/workflows/codeql.yml

@@ -0,0 +1,70 @@
+# For most projects, this workflow file will not need changing; you simply need
+# to commit it to your repository.
+#
+# You may wish to alter this file to override the set of languages analyzed,
+# or to provide custom queries or build logic.
+#
+# ******** NOTE ********
+# We have attempted to detect the languages in your repository. Please check
+# the `language` matrix defined below to confirm you have the correct set of
+# supported CodeQL languages.
+#
+name: "CodeQL"
+
+on:
+  push:
+    branches: [ master ]
+  pull_request:
+    # The branches below must be a subset of the branches above
+    branches: [ master ]
+  schedule:
+    - cron: '24 3 * * 1'
+
+jobs:
+  analyze:
+    name: Analyze
+    runs-on: ubuntu-latest
+    permissions:
+      actions: read
+      contents: read
+      security-events: write
+
+    strategy:
+      fail-fast: false
+      matrix:
+        language: [ 'go' ]
+        # CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python', 'ruby' ]
+        # Learn more about CodeQL language support at https://aka.ms/codeql-docs/language-support
+
+    steps:
+    - name: Checkout repository
+      uses: actions/checkout@v4
+
+    # Initializes the CodeQL tools for scanning.
+    - name: Initialize CodeQL
+      uses: github/codeql-action/init@v3
+      with:
+        languages: ${{ matrix.language }}
+        # If you wish to specify custom queries, you can do so here or in a config file.
+        # By default, queries listed here will override any specified in a config file.
+        # Prefix the list here with "+" to use these queries and those in the config file.
+        # queries: ./path/to/local/query, your-org/your-repo/queries@main
+
+    # Autobuild attempts to build any compiled languages  (C/C++, C#, or Java).
+    # If this step fails, then you should remove it and run the build manually (see below)
+    - name: Autobuild
+      uses: github/codeql-action/autobuild@v3
+
+    # ℹ️ Command-line programs to run using the OS shell.
+    # 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun
+
+    # ✏️ If the Autobuild fails above, remove it and uncomment the following three lines
+    #    and modify them (or add more) to build your code if your project
+    #    uses a compiled language
+
+    #- run: |
+    #   make bootstrap
+    #   make release
+
+    - name: Perform CodeQL Analysis
+      uses: github/codeql-action/analyze@v3

.github/workflows/docker-release.yml

@@ -0,0 +1,100 @@
+name: Release Docker
+
+on:
+  release:
+    types: [released]
+
+  # Allows you to run this workflow manually from the Actions tab
+  workflow_dispatch:
+
+jobs:
+  publishDocker:
+    environment: dockerhub
+    env:
+      IMAGE_NAME: mikefarah/yq
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@v3
+        with:
+          platforms: all
+
+      - name: Set up Docker Buildx
+        id: buildx
+        uses: docker/setup-buildx-action@v3
+        with:
+          version: latest
+
+      - name: Available platforms
+        run: echo ${{ steps.buildx.outputs.platforms }} && docker version
+
+      - name: Login to Docker Hub
+        uses: docker/login-action@v3
+        with:
+          username: ${{ secrets.DOCKER_USERNAME }}
+          password: ${{ secrets.DOCKER_PASSWORD }}
+
+      - name: Login to GitHub Container Registry
+        uses: docker/login-action@v3
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Build and push image
+        run: |
+          echo "GithubRef: ${GITHUB_REF}"
+          VERSION=${GITHUB_REF##*/}
+          echo "VERSION: ${VERSION}"
+          IMAGE_VERSION=${VERSION:1}
+          echo "IMAGE_VERSION: ${IMAGE_VERSION}"
+
+          PLATFORMS="linux/amd64,linux/ppc64le,linux/arm64,linux/arm/v7,linux/s390x"
+
+          echo "Building and pushing version ${IMAGE_VERSION} of image ${IMAGE_NAME}"
+          docker buildx build \
+            --label "org.opencontainers.image.authors=https://github.com/mikefarah/yq/graphs/contributors" \
+            --label "org.opencontainers.image.created=$(date --rfc-3339=seconds)" \
+            --label "org.opencontainers.image.description=yq is a portable command-line data file processor" \
+            --label "org.opencontainers.image.documentation=https://mikefarah.gitbook.io/yq/" \
+            --label "org.opencontainers.image.licenses=MIT" \
+            --label "org.opencontainers.image.revision=$(git rev-parse HEAD)" \
+            --label "org.opencontainers.image.source=https://github.com/mikefarah/yq" \
+            --label "org.opencontainers.image.title=yq" \
+            --label "org.opencontainers.image.url=https://mikefarah.gitbook.io/yq/" \
+            --label "org.opencontainers.image.version=${IMAGE_VERSION}" \
+            --platform "${PLATFORMS}" \
+            --pull \
+            --push \
+            -t "${IMAGE_NAME}:${IMAGE_VERSION}" \
+            -t "${IMAGE_NAME}:4" \
+            -t "${IMAGE_NAME}:latest" \
+            -t "ghcr.io/${IMAGE_NAME}:${IMAGE_VERSION}" \
+            -t "ghcr.io/${IMAGE_NAME}:4" \
+            -t "ghcr.io/${IMAGE_NAME}:latest" \
+            .
+
+          cd github-action
+          docker buildx build \
+            --label "org.opencontainers.image.authors=https://github.com/mikefarah/yq/graphs/contributors" \
+            --label "org.opencontainers.image.created=$(date --rfc-3339=seconds)" \
+            --label "org.opencontainers.image.description=yq is a portable command-line data file processor" \
+            --label "org.opencontainers.image.documentation=https://mikefarah.gitbook.io/yq/" \
+            --label "org.opencontainers.image.licenses=MIT" \
+            --label "org.opencontainers.image.revision=$(git rev-parse HEAD)" \
+            --label "org.opencontainers.image.source=https://github.com/mikefarah/yq" \
+            --label "org.opencontainers.image.title=yq" \
+            --label "org.opencontainers.image.url=https://mikefarah.gitbook.io/yq/" \
+            --label "org.opencontainers.image.version=${IMAGE_VERSION}" \
+            --platform "${PLATFORMS}" \
+            --pull \
+            --push \
+            -t "${IMAGE_NAME}:${IMAGE_VERSION}-githubaction" \
+            -t "${IMAGE_NAME}:4-githubaction" \
+            -t "${IMAGE_NAME}:latest-githubaction" \
+            -t "ghcr.io/${IMAGE_NAME}:${IMAGE_VERSION}-githubaction" \
+            -t "ghrc.io/${IMAGE_NAME}:4-githubaction" \
+            -t "ghcr.io/${IMAGE_NAME}:latest-githubaction" \
+            .

.github/workflows/go.yml

@@ -1,5 +1,8 @@
 name: Build
-on: [push]
+on: [push, pull_request]
+permissions:
+  contents: read
+
 jobs:
 
   build:
@@ -7,14 +10,14 @@ jobs:
     runs-on: ubuntu-latest
     steps:
 
-    - name: Set up Go 1.14
-      uses: actions/setup-go@v1
+    - name: Set up Go
+      uses: actions/setup-go@v5
       with:
-        go-version: 1.14
+        go-version: '^1.20'
       id: go
 
     - name: Check out code into the Go module directory
-      uses: actions/checkout@v2
+      uses: actions/checkout@v4
 
     - name: Get dependencies
       run: |
@@ -24,11 +27,9 @@ jobs:
             dep ensure
         fi
 
-    - name: Download deps
+    - name: Check the build
+      shell: bash -l {0}
       run: |
         export PATH=${PATH}:`go env GOPATH`/bin
         scripts/devtools.sh
-    - name: Build
-      run: |
-        export PATH=${PATH}:`go env GOPATH`/bin
-        make local build
+        make local build

.github/workflows/release.yml

@@ -0,0 +1,51 @@
+name: Release YQ
+on:
+  push:
+    tags:
+      - 'v4.*'
+      - 'draft-*'
+
+jobs:
+  publishGitRelease:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-go@v5
+        with:
+          go-version: '^1.20'
+          check-latest: true
+      - name: Compile man page markup
+        id: gen-man-page-md
+        run: |
+          ./scripts/generate-man-page-md.sh
+
+      - name: Get the version
+        id: get_version
+        run: echo "VERSION=${GITHUB_REF##*/}" >> "${GITHUB_OUTPUT}"
+
+      - name: Generate man page
+        uses: docker://pandoc/core:2.14.2
+        id: gen-man-page
+        with:
+          args: >-
+            --standalone
+            --to man
+            --variable=title:"YQ"
+            --variable=section:"1"
+            --variable=header:"yq (https://github.com/mikefarah/yq/) version ${{ steps.get_version.outputs.VERSION }}"
+            --variable=author:"Mike Farah"
+            --output=yq.1
+            man.md
+
+      - name: Cross compile
+        run: |
+          sudo apt-get install rhash -y
+          go install github.com/goreleaser/goreleaser/v2@latest
+          ./scripts/xcompile.sh
+
+      - name: Release
+        uses: softprops/action-gh-release@v1
+        with:
+          files: build/*
+          draft: true
+          fail_on_unmatched_files: true

.github/workflows/snap-release.yml

@@ -0,0 +1,23 @@
+name: Release Snap
+
+on:
+  release:
+    types: [released]
+
+  # Allows you to run this workflow manually from the Actions tab
+  workflow_dispatch:
+
+jobs:
+    buildSnap:
+      environment: snap
+      runs-on: ubuntu-latest
+      steps:
+        - uses: actions/checkout@v4
+        - uses: snapcore/action-build@v1
+          id: build
+        - uses: snapcore/action-publish@v1
+          env:
+            SNAPCRAFT_STORE_CREDENTIALS: ${{ secrets.STORE_LOGIN }}
+          with:
+            snap: ${{ steps.build.outputs.snap }}
+            release: stable

.github/workflows/test-yq.yml

@@ -0,0 +1,28 @@
+name: Test Yq Action
+on: 
+  push:
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+
+  build:
+    name: Build
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v4
+      - name: Get test
+        id: get_value
+        uses: mikefarah/yq@master
+        with:
+          cmd: yq '.test' examples/multiline-text.yaml
+      - name: Multiline test
+        run: echo "### It was [${{ steps.get_value.outputs.result }}]" >> $GITHUB_STEP_SUMMARY
+      - name: Write inplace test
+        id: lookupSdkVersion
+        uses: mikefarah/yq@master
+        with:
+          cmd: yq -i '.b.c = 5' examples/sample.yaml

.gitignore

@@ -22,6 +22,7 @@ _cgo_gotypes.go
 _cgo_export.*
 
 _testmain.go
+cover.out
 coverage.out
 coverage.html
 *.exe
@@ -38,3 +39,31 @@ parts/
 prime/
 .snapcraft/
 yq*.snap
+
+test.yml
+test*.yml
+test*.xml
+test*.toml
+test*.yaml
+test_dir1/
+test_dir2/
+0.yml
+1.yml
+2.yml
+
+
+# man page
+man.md
+yq.1
+
+# debian pkg
+_build
+debian/files
+
+# intellij
+/.idea
+
+# vscode
+.vscode
+
+yq3

.golangci.bck.yml

@@ -0,0 +1,38 @@
+run:
+  timeout: 5m
+linters:
+  enable:
+    - asciicheck
+    - depguard
+    - errorlint
+    - gci
+    - gochecknoinits
+    - gofmt
+    - goimports
+    - gosec
+    - gosimple
+    - staticcheck
+    - unused
+    - misspell
+    - nakedret
+    - nolintlint
+    - predeclared
+    - revive
+    - unconvert
+    - unparam
+linters-settings:
+  depguard:
+    rules:
+      prevent_unmaintained_packages:
+        list-mode: lax
+        files:
+          - $all
+          - "!$test"
+        deny:
+          - pkg: io/ioutil
+            desc: "replaced by io and os packages since Go 1.16: https://tip.golang.org/doc/go1.16#ioutil"
+issues:
+  exclude-rules:
+  - linters:
+    - revive
+    text: "var-naming"

.golangci.yml

@@ -0,0 +1,52 @@
+version: "2"
+linters:
+  enable:
+    - asciicheck
+    - depguard
+    - errorlint
+    - gochecknoinits
+    - gosec
+    - misspell
+    - nakedret
+    - nolintlint
+    - predeclared
+    - revive
+    - unconvert
+    - unparam
+  settings:
+    depguard:
+      rules:
+        prevent_unmaintained_packages:
+          list-mode: lax
+          files:
+            - $all
+            - '!$test'
+          deny:
+            - pkg: io/ioutil
+              desc: 'replaced by io and os packages since Go 1.16: https://tip.golang.org/doc/go1.16#ioutil'
+  exclusions:
+    generated: lax
+    presets:
+      - comments
+      - common-false-positives
+      - legacy
+      - std-error-handling
+    rules:
+      - linters:
+          - revive
+        text: var-naming
+    paths:
+      - third_party$
+      - builtin$
+      - examples$
+formatters:
+  enable:
+    - gci
+    - gofmt
+    - goimports
+  exclusions:
+    generated: lax
+    paths:
+      - third_party$
+      - builtin$
+      - examples$

.goreleaser.yaml

@@ -0,0 +1,48 @@
+version: 2
+
+dist: build
+
+builds:
+  - id: yq
+
+    binary: yq_{{ .Os }}_{{ .Arch }}
+
+    ldflags:
+      - -s -w
+
+    env:
+      - CGO_ENABLED=0
+
+    targets:
+      - darwin_amd64
+      - darwin_arm64
+      - freebsd_386
+      - freebsd_amd64
+      - freebsd_arm
+      - linux_386
+      - linux_amd64
+      - linux_arm
+      - linux_arm64
+      - linux_mips
+      - linux_mips64
+      - linux_mips64le
+      - linux_mipsle
+      - linux_ppc64
+      - linux_ppc64le
+      - linux_riscv64
+      - linux_s390x
+      - netbsd_386
+      - netbsd_amd64
+      - netbsd_arm
+      - openbsd_386
+      - openbsd_amd64
+      - windows_386
+      - windows_amd64
+      - windows_arm
+      - windows_arm64
+
+    no_unique_dist_dir: true
+
+release:
+  disable: true
+  skip_upload: true

CONTRIBUTING.md

@@ -1,4 +1,6 @@
-1. Install (golang)[https://golang.org/]
+.# Development
+
+1. Install (Golang)[https://golang.org/]
 1. Run  `scripts/devtools.sh` to install the required devtools
 2. Run `make [local] vendor` to install the vendor dependencies
 2. Run `make [local] test` to ensure you can run the existing tests
@@ -6,3 +8,35 @@
 4. Make the code changes.
 5. `make [local] test` to lint code and run tests
 6. Profit! ok no profit, but raise a PR and get kudos :)
+
+
+# Documentation
+
+The documentation is a bit of a mixed bag (sorry in advance, I do plan on simplifying it...) - with some parts automatically generated and stiched together and some statically defined.
+
+Documentation is written in markdown, and is published in the 'gitbook' branch.
+
+The various operator documentation (e.g. 'strings') are generated from the 'master' branch, and have a statically defined header (e.g. `pkg/yqlib/doc/operators/headers/add.md`) and the bulk of the docs are generated from the unit tests e.g. `pkg/yqlib/operator_add_test.go`.
+
+The pipeline will run the tests and automatically concatenate the files together, and put them under 
+`pkg/qylib/doc/add.md`. These files are checked in the master branch (and are copied to the gitbook branch as part of the release process).
+
+## How to contribute
+
+The first step is to find if what you want is automatically generated or not - start by looking in the master branch. 
+
+Note that PRs with small changes (e.g. minor typos) may not be merged (see https://joel.net/how-one-guy-ruined-hacktoberfest2020-drama).
+
+### Updating dynamic documentation from master
+- Search for the documentation you want to update. If you find matches in a `*_test.go` file - update that, as that will automatically update the matching `*.md` file 
+- Assuming you are updating a `*_test.go` file, once updated, run the test to regenerated the docs. E.g. for the 'Add' test generated docs, from the pkg/yqlib folder run:
+`go test -run TestAddOperatorScenarios` which will run that test defined in the `operator_add_test.go` file.
+- Ensure the tests still pass, and check the generated documentation have your update.
+- Note: If the documentation is only in a `headers/*.md` file, then just update that directly
+- Raise a PR to merge the changes into master!
+
+### Updating static documentation from the gitbook branch
+If you haven't found what you want to update in the master branch, then check the gitbook branch directly as there are a few pages in there that are not in master.
+
+- Update the `*.md` files
+- Raise a PR to merge the changes into gitbook.

Dockerfile

@@ -1,23 +1,28 @@
-FROM golang:1.14 as builder
+FROM golang:1.24.5 AS builder
 
 WORKDIR /go/src/mikefarah/yq
 
-# cache devtools
-COPY ./scripts/devtools.sh /go/src/mikefarah/yq/scripts/devtools.sh
-RUN ./scripts/devtools.sh
+COPY . .
 
-COPY . /go/src/mikefarah/yq
-
-RUN CGO_ENABLED=0 make local build
+RUN CGO_ENABLED=0 go build -ldflags "-s -w" .
+# RUN ./scripts/test.sh -- this too often times out in the github pipeline.
+RUN ./scripts/acceptance.sh
 
 # Choose alpine as a base image to make this useful for CI, as many
 # CI tools expect an interactive shell inside the container
-FROM alpine:3.8 as production
+FROM alpine:3 AS production
+LABEL maintainer="Mike Farah <mikefarah@users.noreply.github.com>"
 
 COPY --from=builder /go/src/mikefarah/yq/yq /usr/bin/yq
-RUN chmod +x /usr/bin/yq
-
-ARG VERSION=none
-LABEL version=${VERSION}
 
 WORKDIR /workdir
+
+RUN set -eux; \
+  addgroup -g 1000 yq; \
+  adduser -u 1000 -G yq -s /bin/sh -h /home/yq -D yq
+
+RUN chown -R yq:yq /workdir
+
+USER yq
+
+ENTRYPOINT ["/usr/bin/yq"]

Dockerfile.dev

@@ -1,30 +1,15 @@
-FROM golang:1.14
+FROM golang:1.24.5
+
+RUN apt-get update && \
+    apt-get install -y npm && \
+    npm install -g npx cspell@latest
 
 COPY scripts/devtools.sh /opt/devtools.sh
 
-RUN set -e -x \
-    && /opt/devtools.sh
+RUN set -e -x && \
+    /opt/devtools.sh
 
-# install mkdocs
-RUN set -ex \
-    && buildDeps=' \
-        build-essential \
-        python3-dev \
-    ' \
-    && apt-get update && apt-get install -y --no-install-recommends \
-        $buildDeps \
-        python3 \
-        python3-setuptools \
-        python3-wheel \
-        python3-pip \
-    && pip3 install --upgrade \
-        pip \
-        'Markdown>=2.6.9' \
-        'mkdocs>=0.16.3' \
-        'mkdocs-material>=1.10.1' \
-        'markdown-include>=0.5.1' \
-    && apt-get purge -y --auto-remove $buildDeps \
-    && rm -rf /var/lib/apt/lists/*
+ENV PATH=/go/bin:$PATH
 
 ENV CGO_ENABLED 0
 ENV GOPATH /go:/yq

LICENSE

@@ -1,3 +1,5 @@
+The MIT License (MIT)
+
 Copyright (c) 2017 Mike Farah
 
 Permission is hereby granted, free of charge, to any person obtaining a copy

Makefile

@@ -2,6 +2,7 @@ MAKEFLAGS += --warn-undefined-variables
 SHELL := /bin/bash
 .SHELLFLAGS := -o pipefail -euc
 .DEFAULT_GOAL := install
+ENGINE := $(shell { (podman version > /dev/null 2>&1 && command -v podman) || command -v docker; } 2>/dev/null)
 
 include Makefile.variables
 
@@ -17,6 +18,7 @@ help:
 	@echo '    make vendor          Install dependencies to vendor directory.'
 	@echo '    make format          Run code formatter.'
 	@echo '    make check           Run static code analysis (lint).'
+	@echo '    make secure          Run gosec.'
 	@echo '    make test            Run tests on project.'
 	@echo '    make cover           Run tests and capture code coverage metrics on project.'
 	@echo '    make clean           Clean the directory tree of produced artifacts.'
@@ -32,7 +34,7 @@ clean:
 
 ## prefix before other make targets to run in your local dev environment
 local: | quiet
-	@$(eval DOCKRUN= )
+	@$(eval ENGINERUN= )
 	@mkdir -p tmp
 	@touch tmp/dev_image_id
 quiet: # this is silly but shuts up 'Nothing to be done for `local`'
@@ -41,9 +43,9 @@ quiet: # this is silly but shuts up 'Nothing to be done for `local`'
 prepare: tmp/dev_image_id
 tmp/dev_image_id: Dockerfile.dev scripts/devtools.sh
 	@mkdir -p tmp
-	@docker rmi -f ${DEV_IMAGE} > /dev/null 2>&1 || true
-	@docker build -t ${DEV_IMAGE} -f Dockerfile.dev .
-	@docker inspect -f "{{ .ID }}" ${DEV_IMAGE} > tmp/dev_image_id
+	@${ENGINE} rmi -f ${DEV_IMAGE} > /dev/null 2>&1 || true
+	@${ENGINE} build -t ${DEV_IMAGE} -f Dockerfile.dev .
+	@${ENGINE} inspect -f "{{ .ID }}" ${DEV_IMAGE} > tmp/dev_image_id
 
 # ----------------------------------------------
 # build
@@ -53,58 +55,65 @@ build: build/dev
 .PHONY: build/dev
 build/dev: test *.go
 	@mkdir -p bin/
-	${DOCKRUN} go build --ldflags "$(LDFLAGS)"
-	${DOCKRUN} bash ./scripts/acceptance.sh
+	${ENGINERUN} go build --ldflags "$(LDFLAGS)"
+	${ENGINERUN} bash ./scripts/acceptance.sh
 
 ## Compile the project for multiple OS and Architectures.
 xcompile: check
 	@rm -rf build/
 	@mkdir -p build
-	${DOCKRUN} bash ./scripts/xcompile.sh
+	${ENGINERUN} bash ./scripts/xcompile.sh
 	@find build -type d -exec chmod 755 {} \; || :
 	@find build -type f -exec chmod 755 {} \; || :
 
 .PHONY: install
 install: build
-	${DOCKRUN} go install
+	${ENGINERUN} go install
 
 # Each of the fetch should be an entry within vendor.json; not currently included within project
 .PHONY: vendor
 vendor: tmp/dev_image_id
-	${DOCKRUN} go mod vendor
+	@mkdir -p vendor
+	${ENGINERUN} go mod vendor
 
 # ----------------------------------------------
 # develop and test
 
 .PHONY: format
 format: vendor
-	${DOCKRUN} bash ./scripts/format.sh
+	${ENGINERUN} bash ./scripts/format.sh
+
+
+.PHONY: spelling
+spelling: format
+	${ENGINERUN} bash ./scripts/spelling.sh
+
+.PHONY: secure
+secure: spelling
+	${ENGINERUN} bash ./scripts/secure.sh
 
 .PHONY: check
-check: format
-	${DOCKRUN} bash ./scripts/check.sh
+check: secure
+	${ENGINERUN} bash ./scripts/check.sh
+
+
 
 .PHONY: test
 test: check
-	${DOCKRUN} bash ./scripts/test.sh
+	${ENGINERUN} bash ./scripts/test.sh
 
 .PHONY: cover
 cover: check
 	@rm -rf cover/
 	@mkdir -p cover
-	${DOCKRUN} bash ./scripts/coverage.sh
+	${ENGINERUN} bash ./scripts/coverage.sh
 	@find cover -type d -exec chmod 755 {} \; || :
 	@find cover -type f -exec chmod 644 {} \; || :
 
-.PHONY: build-docs
-build-docs: prepare mkdocs.yml mkdocs/*
-	${DOCKRUN} mkdocs build
-	@find docs -type d -exec chmod 755 {} \; || :
-	@find docs -type f -exec chmod 644 {} \; || :
 
 .PHONY: release
 release: xcompile
-	${DOCKRUN} bash ./scripts/publish.sh
+	${ENGINERUN} bash ./scripts/publish.sh
 
 # ----------------------------------------------
 # utilities

Makefile.variables

@@ -7,6 +7,7 @@ export GIT_DESCRIBE = $(shell git describe --tags --always)
 LDFLAGS :=
 LDFLAGS += -X main.GitCommit=${GIT_COMMIT}${GIT_DIRTY}
 LDFLAGS += -X main.GitDescribe=${GIT_DESCRIBE}
+LDFLAGS += -w -s
 
 GITHUB_TOKEN ?=
 
@@ -25,14 +26,15 @@ ifeq ($(CYG_CHECK),1)
 else
 	# all non-windows environments
 	ROOT := $(shell pwd)
+	SELINUX := $(shell which getenforce 2>&1 >/dev/null && echo :z)
 endif
 
 DEV_IMAGE := ${PROJECT}_dev
 
-DOCKRUN := docker run --rm \
+ENGINERUN := ${ENGINE} run --rm \
 	-e LDFLAGS="${LDFLAGS}" \
 	-e GITHUB_TOKEN="${GITHUB_TOKEN}" \
-	-v ${ROOT}/vendor:/go/src \
-	-v ${ROOT}:/${PROJECT}/src/${IMPORT_PATH} \
+	-v ${ROOT}/vendor:/go/src${SELINUX} \
+	-v ${ROOT}:/${PROJECT}/src/${IMPORT_PATH}${SELINUX} \
 	-w /${PROJECT}/src/${IMPORT_PATH} \
 	${DEV_IMAGE}

README.md

@@ -1,31 +1,105 @@
 # yq
 
-![Build](https://github.com/mikefarah/yq/workflows/Build/badge.svg)  ![Docker Pulls](https://img.shields.io/docker/pulls/mikefarah/yq.svg) ![Github Releases (by Release)](https://img.shields.io/github/downloads/mikefarah/yq/total.svg) ![Go Report](https://goreportcard.com/badge/github.com/mikefarah/yq)
+![Build](https://github.com/mikefarah/yq/workflows/Build/badge.svg)  ![Docker Pulls](https://img.shields.io/docker/pulls/mikefarah/yq.svg) ![Github Releases (by Release)](https://img.shields.io/github/downloads/mikefarah/yq/total.svg) ![Go Report](https://goreportcard.com/badge/github.com/mikefarah/yq) ![CodeQL](https://github.com/mikefarah/yq/workflows/CodeQL/badge.svg)
 
 
-a lightweight and portable command-line YAML processor
+a lightweight and portable command-line YAML, JSON, INI and XML processor. `yq` uses [jq](https://github.com/stedolan/jq) like syntax but works with yaml files as well as json, xml, ini, properties, csv and tsv. It doesn't yet support everything `jq` does - but it does support the most common operations and functions, and more is being added continuously.
 
-The aim of the project is to be the [jq](https://github.com/stedolan/jq) or sed of yaml files.
+yq is written in go - so you can download a dependency free binary for your platform and you are good to go! If you prefer there are a variety of package managers that can be used as well as Docker and Podman, all listed below.
 
-## New version!
-V3 is officially out - if you've been using v2 and want/need to upgrade, checkout the [upgrade guide](https://mikefarah.gitbook.io/yq/upgrading-from-v2).
+## Quick Usage Guide
+
+Read a value:
+```bash
+yq '.a.b[0].c' file.yaml
+```
+
+Pipe from STDIN:
+```bash
+yq '.a.b[0].c' < file.yaml
+```
+
+Update a yaml file, in place
+```bash
+yq -i '.a.b[0].c = "cool"' file.yaml
+```
+
+Update using environment variables
+```bash
+NAME=mike yq -i '.a.b[0].c = strenv(NAME)' file.yaml
+```
+
+Merge multiple files
+```bash
+# merge two files
+yq -n 'load("file1.yaml") * load("file2.yaml")'
+
+# merge using globs:
+# note the use of `ea` to evaluate all the files at once
+# instead of in sequence
+yq ea '. as $item ireduce ({}; . * $item )' path/to/*.yml
+```
+
+Multiple updates to a yaml file
+```bash
+yq -i '
+  .a.b[0].c = "cool" |
+  .x.y.z = "foobar" |
+  .person.name = strenv(NAME)
+' file.yaml
+```
+
+Find and update an item in an array:
+```bash
+yq '(.[] | select(.name == "foo") | .address) = "12 cat st"'
+```
+
+Convert JSON to YAML
+```bash
+yq -Poy sample.json
+```
+
+See [recipes](https://mikefarah.gitbook.io/yq/recipes) for more examples and the [documentation](https://mikefarah.gitbook.io/yq/) for more information.
+
+Take a look at the discussions for [common questions](https://github.com/mikefarah/yq/discussions/categories/q-a), and [cool ideas](https://github.com/mikefarah/yq/discussions/categories/show-and-tell)
 
 ## Install
 
 ### [Download the latest binary](https://github.com/mikefarah/yq/releases/latest)
 
-### MacOS:
+### wget
+Use wget to download, gzipped pre-compiled binaries:
+
+
+For instance, VERSION=v4.2.0 and BINARY=yq_linux_amd64
+
+#### Compressed via tar.gz
+```bash
+wget https://github.com/mikefarah/yq/releases/download/${VERSION}/${BINARY}.tar.gz -O - |\
+  tar xz && mv ${BINARY} /usr/local/bin/yq
+```
+
+#### Plain binary
+
+```bash
+wget https://github.com/mikefarah/yq/releases/download/${VERSION}/${BINARY} -O /usr/local/bin/yq &&\
+    chmod +x /usr/local/bin/yq
+```
+
+#### Latest version
+
+```bash
+wget https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -O /usr/local/bin/yq &&\
+    chmod +x /usr/local/bin/yq
+```
+
+### MacOS / Linux via Homebrew:
+Using [Homebrew](https://brew.sh/)
 ```
 brew install yq
 ```
 
-### Windows:
-```
-choco install yq
-```
-Supported by @chillum
-
-### Ubuntu and other Linux distros supporting `snap` packages:
+### Linux via snap:
 ```
 snap install yq
 ```
@@ -34,74 +108,245 @@ snap install yq
 `yq` installs with [_strict confinement_](https://docs.snapcraft.io/snap-confinement/6233) in snap, this means it doesn't have direct access to root files. To read root files you can:
 
 ```
-sudo cat /etc/myfile | yq r - a.path
+sudo cat /etc/myfile | yq '.a.path'
 ```
 
 And to write to a root file you can either use [sponge](https://linux.die.net/man/1/sponge):
 ```
-sudo cat /etc/myfile | yq w - a.path value | sudo sponge /etc/myfile
+sudo cat /etc/myfile | yq '.a.path = "value"' | sudo sponge /etc/myfile
 ```
 or write to a temporary file:
 ```
-sudo cat /etc/myfile | yq w - a.path value | sudo tee /etc/myfile.tmp
+sudo cat /etc/myfile | yq '.a.path = "value"' | sudo tee /etc/myfile.tmp
 sudo mv /etc/myfile.tmp /etc/myfile
 rm /etc/myfile.tmp
 ```
 
-### On Ubuntu 16.04 or higher from Debian package:
-```sh
-sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys CC86BB64
-sudo add-apt-repository ppa:rmescandon/yq
-sudo apt update
-sudo apt install yq -y
-```
-Supported by @rmescandon
-
-### Go Get:
-```
-GO111MODULE=on go get github.com/mikefarah/yq/v3
-```
-
-## Run with Docker
-
-Oneshot use:
+### Run with Docker or Podman
+#### Oneshot use:
 
 ```bash
-docker run --rm -v "${PWD}":/workdir mikefarah/yq yq [flags] <command> FILE...
+docker run --rm -v "${PWD}":/workdir mikefarah/yq [command] [flags] [expression ]FILE...
 ```
 
-Run commands interactively:
+Note that you can run `yq` in docker without network access and other privileges if you desire,
+namely `--security-opt=no-new-privileges --cap-drop all --network none`.
 
 ```bash
-docker run --rm -it -v "${PWD}":/workdir mikefarah/yq sh
+podman run --rm -v "${PWD}":/workdir mikefarah/yq [command] [flags] [expression ]FILE...
+```
+
+#### Pipe in via STDIN:
+
+You'll need to pass the `-i\--interactive` flag to docker:
+
+```bash
+docker run -i --rm mikefarah/yq '.this.thing' < myfile.yml
+```
+
+```bash
+podman run -i --rm mikefarah/yq '.this.thing' < myfile.yml
+```
+
+#### Run commands interactively:
+
+```bash
+docker run --rm -it -v "${PWD}":/workdir --entrypoint sh mikefarah/yq
+```
+
+```bash
+podman run --rm -it -v "${PWD}":/workdir --entrypoint sh mikefarah/yq
 ```
 
 It can be useful to have a bash function to avoid typing the whole docker command:
 
 ```bash
 yq() {
-  docker run --rm -i -v "${PWD}":/workdir mikefarah/yq yq "$@"
+  docker run --rm -i -v "${PWD}":/workdir mikefarah/yq "$@"
 }
 ```
 
+```bash
+yq() {
+  podman run --rm -i -v "${PWD}":/workdir mikefarah/yq "$@"
+}
+```
+#### Running as root:
+
+`yq`'s container image no longer runs under root (https://github.com/mikefarah/yq/pull/860). If you'd like to install more things in the container image, or you're having permissions issues when attempting to read/write files you'll need to either:
+
+
+```
+docker run --user="root" -it --entrypoint sh mikefarah/yq
+```
+
+```
+podman run --user="root" -it --entrypoint sh mikefarah/yq
+```
+
+Or, in your Dockerfile:
+
+```
+FROM mikefarah/yq
+
+USER root
+RUN apk add --no-cache bash
+USER yq
+```
+
+#### Missing timezone data
+By default, the alpine image yq uses does not include timezone data. If you'd like to use the `tz` operator, you'll need to include this data:
+
+```
+FROM mikefarah/yq
+
+USER root
+RUN apk add --no-cache tzdata
+USER yq
+```
+
+#### Podman with SELinux
+
+If you are using podman with SELinux, you will need to set the shared volume flag `:z` on the volume mount:
+
+```
+-v "${PWD}":/workdir:z
+```
+
+### GitHub Action
+```
+  - name: Set foobar to cool
+    uses: mikefarah/yq@master
+    with:
+      cmd: yq -i '.foo.bar = "cool"' 'config.yml'
+  - name: Get an entry with a variable that might contain dots or spaces
+    id: get_username
+    uses: mikefarah/yq@master
+    with:
+      cmd: yq '.all.children.["${{ matrix.ip_address }}"].username' ops/inventories/production.yml
+  - name: Reuse a variable obtained in another step
+    run: echo ${{ steps.get_username.outputs.result }}
+```
+
+See https://mikefarah.gitbook.io/yq/usage/github-action for more.
+
+### Go Install:
+```
+go install github.com/mikefarah/yq/v4@latest
+```
+
+## Community Supported Installation methods
+As these are supported by the community :heart: - however, they may be out of date with the officially supported releases.
+
+_Please note that the Debian package (previously supported by @rmescandon) is no longer maintained. Please use an alternative installation method._
+
+
+### X-CMD
+Checkout `yq` on x-cmd: https://x-cmd.com/mod/yq
+
+- Instant Results: See the output of your yq filter in real-time.
+- Error Handling: Encounter a syntax error? It will display the error message and the results of the closest valid filter
+
+Thanks @edwinjhlee!
+
+### Nix
+
+```
+nix profile install nixpkgs#yq-go
+```
+
+See [here](https://search.nixos.org/packages?channel=unstable&show=yq-go&from=0&size=50&sort=relevance&type=packages&query=yq-go)
+
+
+### Webi
+
+```
+webi yq
+```
+
+See [webi](https://webinstall.dev/)
+Supported by @adithyasunil26 (https://github.com/webinstall/webi-installers/tree/master/yq)
+
+### Arch Linux
+
+```
+pacman -S go-yq
+```
+
+### Windows:
+
+Using [Chocolatey](https://chocolatey.org)
+
+[![Chocolatey](https://img.shields.io/chocolatey/v/yq.svg)](https://chocolatey.org/packages/yq)
+[![Chocolatey](https://img.shields.io/chocolatey/dt/yq.svg)](https://chocolatey.org/packages/yq)
+```
+choco install yq
+```
+Supported by @chillum (https://chocolatey.org/packages/yq)
+
+Using [scoop](https://scoop.sh/)
+```
+scoop install main/yq
+```
+
+Using [winget](https://learn.microsoft.com/en-us/windows/package-manager/)
+```
+winget install --id MikeFarah.yq
+```
+
+### Mac:
+Using [MacPorts](https://www.macports.org/)
+```
+sudo port selfupdate
+sudo port install yq
+```
+Supported by @herbygillot (https://ports.macports.org/maintainer/github/herbygillot)
+
+### Alpine Linux
+
+Alpine Linux v3.20+ (and Edge):
+```
+apk add yq-go
+```
+
+Alpine Linux up to v3.19:
+```
+apk add yq
+```
+
+Supported by Tuan Hoang (https://pkgs.alpinelinux.org/packages?name=yq-go)
+
+### Flox:
+
+Flox can be used to install yq on Linux, MacOS, and Windows through WSL.
+
+```
+flox install yq
+```
+
 ## Features
+- [Detailed documentation with many examples](https://mikefarah.gitbook.io/yq/)
 - Written in portable go, so you can download a lovely dependency free binary
-- [Colorize the output](https://mikefarah.gitbook.io/yq/usage/output-format#colorize-output)
-- [Deep read a yaml file with a given path expression](https://mikefarah.gitbook.io/yq/commands/read#basic)
-- [List matching paths of a given path expression](https://mikefarah.gitbook.io/yq/commands/read#path-only)
-- [Return the lengths of arrays/object/scalars](https://mikefarah.gitbook.io/yq/commands/read#printing-length-of-the-results)
-- Update a yaml file given a [path expression](https://mikefarah.gitbook.io/yq/commands/write-update#basic) or [script file](https://mikefarah.gitbook.io/yq/commands/write-update#basic)
-- Update creates any missing entries in the path on the fly
-- Deeply [compare](https://mikefarah.gitbook.io/yq/commands/compare) yaml files
-- Keeps yaml formatting and comments when updating
-- [Validate a yaml file](https://mikefarah.gitbook.io/yq/commands/validate)
-- Create a yaml file given a [deep path and value](https://mikefarah.gitbook.io/yq/commands/create#creating-a-simple-yaml-file) or a [script file](https://mikefarah.gitbook.io/yq/commands/create#creating-using-a-create-script)
-- [Prefix a path to a yaml file](https://mikefarah.gitbook.io/yq/commands/prefix)
-- [Convert to/from json to yaml](https://mikefarah.gitbook.io/yq/usage/convert)
-- [Pipe data in by using '-'](https://mikefarah.gitbook.io/yq/commands/read#from-stdin)
-- [Merge](https://mikefarah.gitbook.io/yq/commands/merge) multiple yaml files with various options for [overriding](https://mikefarah.gitbook.io/yq/commands/merge#overwrite-values) and [appending](https://mikefarah.gitbook.io/yq/commands/merge#append-values-with-arrays)
-- Supports multiple documents in a single yaml file for [reading](https://mikefarah.gitbook.io/yq/commands/read#multiple-documents), [writing](https://mikefarah.gitbook.io/yq/commands/write-update#multiple-documents) and [merging](https://mikefarah.gitbook.io/yq/commands/merge#multiple-documents)
-- General shell completion scripts (bash/zsh/fish/powershell) (https://mikefarah.gitbook.io/yq/commands/shell-completion)
+- Uses similar syntax as `jq` but works with YAML, INI, [JSON](https://mikefarah.gitbook.io/yq/usage/convert) and [XML](https://mikefarah.gitbook.io/yq/usage/xml) files
+- Fully supports multi document yaml files
+- Supports yaml [front matter](https://mikefarah.gitbook.io/yq/usage/front-matter) blocks (e.g. jekyll/assemble)
+- Colorized yaml output
+- [Date/Time manipulation and formatting with TZ](https://mikefarah.gitbook.io/yq/operators/datetime)
+- [Deeply data structures](https://mikefarah.gitbook.io/yq/operators/traverse-read)
+- [Sort keys](https://mikefarah.gitbook.io/yq/operators/sort-keys)
+- Manipulate yaml [comments](https://mikefarah.gitbook.io/yq/operators/comment-operators), [styling](https://mikefarah.gitbook.io/yq/operators/style), [tags](https://mikefarah.gitbook.io/yq/operators/tag) and [anchors and aliases](https://mikefarah.gitbook.io/yq/operators/anchor-and-alias-operators).
+- [Update in place](https://mikefarah.gitbook.io/yq/v/v4.x/commands/evaluate#flags)
+- [Complex expressions to select and update](https://mikefarah.gitbook.io/yq/operators/select#select-and-update-matching-values-in-map)
+- Keeps yaml formatting and comments when updating (though there are issues with whitespace)
+- [Decode/Encode base64 data](https://mikefarah.gitbook.io/yq/operators/encode-decode)
+- [Load content from other files](https://mikefarah.gitbook.io/yq/operators/load)
+- [Convert to/from json/ndjson](https://mikefarah.gitbook.io/yq/v/v4.x/usage/convert)
+- [Convert to/from xml](https://mikefarah.gitbook.io/yq/v/v4.x/usage/xml)
+- [Convert to/from properties](https://mikefarah.gitbook.io/yq/v/v4.x/usage/properties)
+- [Convert to/from csv/tsv](https://mikefarah.gitbook.io/yq/usage/csv-tsv)
+- [General shell completion scripts (bash/zsh/fish/powershell)](https://mikefarah.gitbook.io/yq/v/v4.x/commands/shell-completion)
+- [Reduce](https://mikefarah.gitbook.io/yq/operators/reduce) to merge multiple files or sum an array or other fancy things.
+- [Github Action](https://mikefarah.gitbook.io/yq/usage/github-action) to use in your automated pipeline (thanks @devorbitus)
 
 ## [Usage](https://mikefarah.gitbook.io/yq/)
 
@@ -112,26 +357,67 @@ Usage:
   yq [flags]
   yq [command]
 
+Examples:
+
+# yq defaults to 'eval' command if no command is specified. See "yq eval --help" for more examples.
+yq '.stuff' < myfile.yml # outputs the data at the "stuff" node from "myfile.yml"
+
+yq -i '.stuff = "foo"' myfile.yml # update myfile.yml in place
+
+
 Available Commands:
-  compare          yq x [--prettyPrint/-P] dataA.yaml dataB.yaml 'b.e(name==fr*).value'
-  delete           yq d [--inplace/-i] [--doc/-d index] sample.yaml 'b.e(name==fred)'
-  help             Help about any command
-  merge            yq m [--inplace/-i] [--doc/-d index] [--overwrite/-x] [--append/-a] sample.yaml sample2.yaml
-  new              yq n [--script/-s script_file] a.b.c newValue
-  prefix           yq p [--inplace/-i] [--doc/-d index] sample.yaml a.b.c
-  read             yq r [--printMode/-p pv] sample.yaml 'b.e(name==fr*).value'
-  shell-completion Generates shell completion scripts
-  validate         yq v sample.yaml
-  write            yq w [--inplace/-i] [--script/-s script_file] [--doc/-d index] sample.yaml 'b.e(name==fr*).value' newValue
+  completion  Generate the autocompletion script for the specified shell
+  eval        (default) Apply the expression to each document in each yaml file in sequence
+  eval-all    Loads _all_ yaml documents of _all_ yaml files and runs expression once
+  help        Help about any command
 
 Flags:
-  -C, --colors        print with colors
-  -h, --help          help for yq
-  -I, --indent int    sets indent level for output (default 2)
-  -P, --prettyPrint   pretty print
-  -j, --tojson        output as json. By default it prints a json document in one line, use the prettyPrint flag to print a formatted doc.
-  -v, --verbose       verbose mode
-  -V, --version       Print version information and quit
+  -C, --colors                        force print with colors
+      --csv-auto-parse                parse CSV YAML/JSON values (default true)
+      --csv-separator char            CSV Separator character (default ,)
+  -e, --exit-status                   set exit status if there are no matches or null or false is returned
+      --expression string             forcibly set the expression argument. Useful when yq argument detection thinks your expression is a file.
+      --from-file string              Load expression from specified file.
+  -f, --front-matter string           (extract|process) first input as yaml front-matter. Extract will pull out the yaml content, process will run the expression against the yaml content, leaving the remaining data intact
+      --header-preprocess             Slurp any header comments and separators before processing expression. (default true)
+  -h, --help                          help for yq
+  -I, --indent int                    sets indent level for output (default 2)
+  -i, --inplace                       update the file in place of first file given.
+  -p, --input-format string           [auto|a|yaml|y|json|j|props|p|csv|c|tsv|t|xml|x|base64|uri|toml|lua|l|ini|i] parse format for input. (default "auto")
+      --lua-globals                   output keys as top-level global variables
+      --lua-prefix string             prefix (default "return ")
+      --lua-suffix string             suffix (default ";\n")
+      --lua-unquoted                  output unquoted string keys (e.g. {foo="bar"})
+  -M, --no-colors                     force print with no colors
+  -N, --no-doc                        Don't print document separators (---)
+  -0, --nul-output                    Use NUL char to separate values. If unwrap scalar is also set, fail if unwrapped scalar contains NUL char.
+  -n, --null-input                    Don't read input, simply evaluate the expression given. Useful for creating docs from scratch.
+  -o, --output-format string          [auto|a|yaml|y|json|j|props|p|csv|c|tsv|t|xml|x|base64|uri|toml|shell|s|lua|l|ini|i] output format type. (default "auto")
+  -P, --prettyPrint                   pretty print, shorthand for '... style = ""'
+      --properties-array-brackets     use [x] in array paths (e.g. for SpringBoot)
+      --properties-separator string   separator to use between keys and values (default " = ")
+  -s, --split-exp string              print each result (or doc) into a file named (exp). [exp] argument must return a string. You can use $index in the expression as the result counter. The necessary directories will be created.
+      --split-exp-file string         Use a file to specify the split-exp expression.
+      --string-interpolation          Toggles strings interpolation of \(exp) (default true)
+      --tsv-auto-parse                parse TSV YAML/JSON values (default true)
+  -r, --unwrapScalar                  unwrap scalar, print the value with no quotes, colors or comments. Defaults to true for yaml (default true)
+  -v, --verbose                       verbose mode
+  -V, --version                       Print version information and quit
+      --xml-attribute-prefix string   prefix for xml attributes (default "+@")
+      --xml-content-name string       name for xml content (if no attribute name is present). (default "+content")
+      --xml-directive-name string     name for xml directives (e.g. <!DOCTYPE thing cat>) (default "+directive")
+      --xml-keep-namespace            enables keeping namespace after parsing attributes (default true)
+      --xml-proc-inst-prefix string   prefix for xml processing instructions (e.g. <?xml version="1"?>) (default "+p_")
+      --xml-raw-token                 enables using RawToken method instead Token. Commonly disables namespace translations. See https://pkg.go.dev/encoding/xml#Decoder.RawToken for details. (default true)
+      --xml-skip-directives           skip over directives (e.g. <!DOCTYPE thing cat>)
+      --xml-skip-proc-inst            skip over process instructions (e.g. <?xml version="1"?>)
+      --xml-strict-mode               enables strict parsing of XML. See https://pkg.go.dev/encoding/xml for more details.
 
 Use "yq [command] --help" for more information about a command.
 ```
+## Known Issues / Missing Features
+- `yq` attempts to preserve comment positions and whitespace as much as possible, but it does not handle all scenarios (see https://github.com/go-yaml/yaml/tree/v3 for details)
+- Powershell has its own...[opinions on quoting yq](https://mikefarah.gitbook.io/yq/usage/tips-and-tricks#quotes-in-windows-powershell)
+- "yes", "no" were dropped as boolean values in the yaml 1.2 standard - which is the standard yq assumes.
+
+See [tips and tricks](https://mikefarah.gitbook.io/yq/usage/tips-and-tricks) for more common problems and solutions.

acceptance_tests/bad_args.sh

@@ -0,0 +1,41 @@
+#!/bin/bash
+
+testWriteInPlacePipeIn() {
+  result=$(./yq e -i -n '.a' 2>&1)
+  assertEquals 1 $?
+  assertEquals "Error: write in place flag only applicable when giving an expression and at least one file" "$result"
+}
+
+testWriteInPlacePipeInEvalall() {
+  result=$(./yq ea -i -n '.a' 2>&1)
+  assertEquals 1 $?
+  assertEquals "Error: write in place flag only applicable when giving an expression and at least one file" "$result"
+}
+
+testWriteInPlaceWithSplit() {
+  result=$(./yq e -s "cat" -i '.a = "thing"' test.yml 2>&1)
+  assertEquals 1 $?
+  assertEquals "Error: write in place cannot be used with split file" "$result"
+}
+
+testWriteInPlaceWithSplitEvalAll() {
+  result=$(./yq ea -s "cat" -i '.a = "thing"' test.yml 2>&1)
+  assertEquals 1 $?
+  assertEquals "Error: write in place cannot be used with split file" "$result"
+}
+
+testNullWithFiles() {
+  result=$(./yq e -n '.a = "thing"' test.yml 2>&1)
+  assertEquals 1 $?
+  assertEquals "Error: cannot pass files in when using null-input flag" "$result"
+}
+
+testNullWithFilesEvalAll() {
+  result=$(./yq ea -n '.a = "thing"' test.yml 2>&1)
+  assertEquals 1 $?
+  assertEquals "Error: cannot pass files in when using null-input flag" "$result"
+}
+
+
+
+source ./scripts/shunit2

acceptance_tests/basic.sh

@@ -0,0 +1,373 @@
+#!/bin/bash
+
+setUp() {
+  rm test*.yml 2>/dev/null || true
+  rm .xyz 2>/dev/null || true
+  rm instructions.txt 2>/dev/null || true
+}
+
+testBasicEvalRoundTrip() {
+  ./yq -n ".a = 123" > test.yml
+  X=$(./yq '.a' test.yml)
+  assertEquals 123 "$X"
+}
+
+testBasicTrailingContent() {
+  cat >test-trailing.yml <<EOL
+test:
+# this comment will be removed
+EOL
+
+  read -r -d '' expected << EOM
+test:
+# this comment will be removed
+EOM
+  X=$(./yq test-trailing.yml -P)
+  assertEquals "$expected" "$X"
+}
+
+testBasicTrailingContent() {
+  cat >test-trailing.yml <<EOL
+test:
+# this comment will be removed
+EOL
+
+  read -r -d '' expected << EOM
+test:
+# hi
+EOM
+  X=$(./yq '. footComment = "hi"' test-trailing.yml)
+  assertEquals "$expected" "$X"
+}
+
+testBasicTrailingContentEvalAll() {
+  cat >test-trailing.yml <<EOL
+test:
+# this comment will be removed
+EOL
+
+  read -r -d '' expected << EOM
+test:
+# this comment will be removed
+EOM
+  X=$(./yq ea test-trailing.yml -P)
+  assertEquals "$expected" "$X"
+}
+
+testBasicTrailingContentEvalAll() {
+  cat >test-trailing.yml <<EOL
+test:
+# this comment will be removed
+EOL
+
+  read -r -d '' expected << EOM
+test:
+# hi
+EOM
+  X=$(./yq ea '. footComment = "hi"' test-trailing.yml)
+  assertEquals "$expected" "$X"
+}
+
+testBasicPipeWithDot() {
+  ./yq -n ".a = 123" > test.yml
+  X=$(cat test.yml | ./yq '.')
+  assertEquals "a: 123" "$X"
+}
+
+testBasicExpressionMatchesFileName() {
+  ./yq -n ".xyz = 123" > test.yml
+  touch .xyz
+  
+  X=$(./yq --expression '.xyz' test.yml)
+  assertEquals "123" "$X"
+
+  X=$(./yq ea --expression '.xyz' test.yml)
+  assertEquals "123" "$X"
+}
+
+testBasicExpressionFromFile() {
+  ./yq -n ".xyz = 123" > test.yml
+  echo '.xyz = "meow" | .cool = "frog"' > instructions.txt
+  
+  X=$(./yq --from-file instructions.txt test.yml -o=j -I=0)
+  assertEquals '{"xyz":"meow","cool":"frog"}' "$X"
+
+  X=$(./yq ea --from-file instructions.txt test.yml -o=j -I=0)
+  assertEquals '{"xyz":"meow","cool":"frog"}' "$X"
+}
+
+testBasicExpressionFromFileDos() {
+  ./yq -n ".xyz = 123" > test.yml
+  echo '.xyz = "meow" | .cool = "frog"' | sed 's/$'"/`echo \\\r`/" > instructions.txt
+  
+  X=$(./yq --from-file instructions.txt test.yml -o=j -I=0)
+  assertEquals '{"xyz":"meow","cool":"frog"}' "$X"
+
+  X=$(./yq ea --from-file instructions.txt test.yml -o=j -I=0)
+  assertEquals '{"xyz":"meow","cool":"frog"}' "$X"
+}
+
+testBasicGitHubAction() {
+  ./yq -n ".a = 123" > test.yml
+  X=$(cat /dev/null | ./yq test.yml)
+  assertEquals "a: 123" "$X"
+
+  X=$(cat /dev/null | ./yq e test.yml)
+  assertEquals "a: 123" "$X"
+
+  X=$(cat /dev/null | ./yq ea test.yml)
+  assertEquals "a: 123" "$X"
+}
+
+testBasicGitHubActionWithExpression() {
+  ./yq -n ".a = 123" > test.yml
+  X=$(cat /dev/null | ./yq '.a' test.yml)
+  assertEquals "123" "$X"
+
+  X=$(cat /dev/null | ./yq e '.a' test.yml)
+  assertEquals "123" "$X"
+
+  X=$(cat /dev/null | ./yq ea '.a' test.yml)
+  assertEquals "123" "$X"
+}
+
+
+testBasicEvalAllAllFiles() {
+  ./yq -n ".a = 123" > test.yml
+  ./yq -n ".a = 124" > test2.yml
+  X=$(./yq ea test.yml test2.yml)
+  Y=$(./yq e '.' test.yml test2.yml)
+  assertEquals "$Y" "$X"
+}
+
+# when given a file, don't read STDIN
+# otherwise strange things start happening
+# in scripts
+# https://github.com/mikefarah/yq/issues/1115
+
+testBasicCatWithFilesNoDash() {
+  ./yq -n ".a = 123" > test.yml
+  ./yq -n ".a = 124" > test2.yml
+  X=$(cat test.yml | ./yq test2.yml)
+  Y=$(./yq e '.' test2.yml)
+  assertEquals "$Y" "$X"
+}
+
+# when the nullinput flag is used
+# don't automatically read STDIN (this breaks github actions)
+testBasicCreateFileGithubAction() {
+  cat /dev/null | ./yq -n ".a = 123" > test.yml
+}
+
+testBasicEvalAllCatWithFilesNoDash() {
+  ./yq -n ".a = 123" > test.yml
+  ./yq -n ".a = 124" > test2.yml
+  X=$(cat test.yml | ./yq ea test2.yml)
+  Y=$(./yq e '.' test2.yml)
+  assertEquals "$Y" "$X"
+}
+
+testBasicCatWithFilesNoDashWithExp() {
+  ./yq -n ".a = 123" > test.yml
+  ./yq -n ".a = 124" > test2.yml
+  X=$(cat test.yml | ./yq '.a' test2.yml)
+  Y=$(./yq e '.a' test2.yml)
+  assertEquals "$Y" "$X"
+}
+
+testBasicEvalAllCatWithFilesNoDashWithExp() {
+  ./yq -n ".a = 123" > test.yml
+  ./yq -n ".a = 124" > test2.yml
+  X=$(cat test.yml | ./yq ea '.a' test2.yml)
+  Y=$(./yq e '.a' test2.yml)
+  assertEquals "$Y" "$X"
+}
+
+
+testBasicStdInWithFiles() {
+  ./yq -n ".a = 123" > test.yml
+  ./yq -n ".a = 124" > test2.yml
+  X=$(cat test.yml | ./yq - test2.yml)
+  Y=$(./yq e '.' test.yml test2.yml)
+  assertEquals "$Y" "$X"
+}
+
+testBasicEvalAllStdInWithFiles() {
+  ./yq -n ".a = 123" > test.yml
+  ./yq -n ".a = 124" > test2.yml
+  X=$(cat test.yml | ./yq ea - test2.yml)
+  Y=$(./yq e '.' test.yml test2.yml)
+  assertEquals "$Y" "$X"
+}
+
+testBasicStdInWithFilesReverse() {
+  ./yq -n ".a = 123" > test.yml
+  ./yq -n ".a = 124" > test2.yml
+  X=$(cat test.yml | ./yq test2.yml -)
+  Y=$(./yq e '.' test2.yml test.yml)
+  assertEquals "$Y" "$X"
+}
+
+testBasicEvalAllStdInWithFilesReverse() {
+  ./yq -n ".a = 123" > test.yml
+  ./yq -n ".a = 124" > test2.yml
+  X=$(cat test.yml | ./yq ea test2.yml -)
+  Y=$(./yq e '.' test2.yml test.yml)
+  assertEquals "$Y" "$X"
+}
+
+testBasicEvalRoundTripNoEval() {
+  ./yq -n ".a = 123" > test.yml
+  X=$(./yq '.a' test.yml)
+  assertEquals 123 "$X"
+}
+
+testBasicStdInWithOneArg() {
+  ./yq e -n ".a = 123" > test.yml
+  X=$(cat test.yml | ./yq e ".a")
+  assertEquals 123 "$X"
+
+  X=$(cat test.yml | ./yq ea ".a")
+  assertEquals 123 "$X"
+
+  X=$(cat test.yml | ./yq ".a")
+  assertEquals 123 "$X"
+}
+
+testBasicUpdateInPlaceSequence() {
+  cat >test.yml <<EOL
+a: 0
+EOL
+  ./yq e -i ".a = 10" test.yml
+  X=$(./yq e '.a' test.yml)
+  assertEquals "10" "$X"
+}
+
+testBasicUpdateInPlaceSequenceNoEval() {
+  cat >test.yml <<EOL
+a: 0
+EOL
+  ./yq -i ".a = 10" test.yml
+  X=$(./yq '.a' test.yml)
+  assertEquals "10" "$X"
+}
+
+testBasicUpdateInPlaceSequenceEvalAll() {
+  cat >test.yml <<EOL
+a: 0
+EOL
+  ./yq ea -i ".a = 10" test.yml
+  X=$(./yq e '.a' test.yml)
+  assertEquals "10" "$X"
+}
+
+testBasicUpdateInPlaceMultipleFilesNoExpressionEval() {
+  cat >test.yml <<EOL
+a: 0
+EOL
+  cat >test2.yml <<EOL
+a: 1
+EOL
+read -r -d '' expected << EOM
+0
+---
+1
+EOM
+  ./yq -i test.yml test2.yml
+  X=$(./yq e '.a' test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testBasicUpdateInPlaceMultipleFilesNoExpressionEvalAll() {
+  cat >test.yml <<EOL
+a: 0
+EOL
+  cat >test2.yml <<EOL
+a: 1
+EOL
+read -r -d '' expected << EOM
+0
+---
+1
+EOM
+  ./yq -i ea test.yml test2.yml
+  X=$(./yq e '.a' test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testBasicNoExitStatus() {
+  echo "a: cat" > test.yml
+  X=$(./yq e '.z' test.yml)
+  assertEquals "null" "$X"
+}
+
+testBasicExitStatus() {
+  echo "a: cat" > test.yml
+  X=$(./yq e -e '.z' test.yml 2&>/dev/null)
+  assertEquals 1 "$?"
+}
+
+testBasicExitStatusNoEval() {
+  echo "a: cat" > test.yml
+  X=$(./yq -e '.z' test.yml 2&>/dev/null)
+  assertEquals 1 "$?"
+}
+
+testBasicExtractFieldWithSeparator() {
+    cat >test.yml <<EOL
+---
+name: chart-name
+version: 1.2.3
+EOL
+  X=$(./yq e '.name' test.yml)
+  assertEquals "chart-name" "$X"
+}
+
+testBasicExtractMultipleFieldWithSeparator() {
+    cat >test.yml <<EOL
+---
+name: chart-name
+version: 1.2.3
+---
+name: thing
+version: 1.2.3
+EOL
+
+read -r -d '' expected << EOM
+chart-name
+---
+thing
+EOM
+  X=$(./yq e '.name' test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testBasicMultiplyAssignMultiDoc() {
+      cat >test.yml <<EOL
+a: 1
+---
+b: 2
+EOL
+
+read -r -d '' expected << EOM
+a: 1
+c: 3
+---
+b: 2
+c: 3
+EOM
+
+
+  X=$(./yq '. *= {"c":3}' test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testBasicClosedStdIn() {
+  cat >test.yml <<EOL
+a: 1
+EOL
+  X=$(./yq e '.a' test.yml <&-)
+  assertEquals "1" "$X"
+}
+
+source ./scripts/shunit2

acceptance_tests/completion.sh

@@ -0,0 +1,9 @@
+#!/bin/bash
+
+testCompletionRuns() {
+    result=$(./yq __complete "" 2>&1)
+    assertEquals 0 $?
+    assertContains "$result" "Completion ended with directive:"
+}
+
+source ./scripts/shunit2

acceptance_tests/empty.sh

@@ -0,0 +1,83 @@
+#!/bin/bash
+
+setUp() {
+  rm test*.yml || true
+  cat >test.yml <<EOL
+# comment
+EOL
+}
+
+testEmptyEval() {
+  X=$(./yq e test.yml)
+  expected="# comment"
+  assertEquals 0 $?
+  assertEquals "$expected" "$X"
+}
+
+testEmptyEvalNoNewLine() {
+  echo -n "#comment" >test.yml
+  X=$(./yq e test.yml)
+  expected=$(cat test.yml)
+  assertEquals 0 $?
+  assertEquals "$expected" "$X"
+}
+
+testEmptyEvalNoNewLineWithExpression() {
+  echo -n "# comment" >test.yml
+  X=$(./yq e '.apple = "tree"' test.yml)
+  read -r -d '' expected << EOM
+# comment
+apple: tree
+EOM
+  assertEquals "$expected" "$X"
+}
+
+testEmptyEvalPipe() {
+  X=$(./yq e - < test.yml)
+  assertEquals 0 $?
+}
+
+testEmptyCommentsWithExpressionEval() {
+  read -r -d '' expected << EOM
+# comment
+apple: tree
+EOM
+
+  X=$(./yq e '.apple="tree"' test.yml)
+
+  assertEquals "$expected" "$X"
+}
+
+testEmptyCommentsWithExpressionEvalAll() {
+  read -r -d '' expected << EOM
+# comment
+apple: tree
+EOM
+
+  X=$(./yq ea '.apple="tree"' test.yml)
+
+  assertEquals "$expected" "$X"
+}
+
+testEmptyWithExpressionEval() {
+  rm test.yml
+  touch test.yml
+  expected="apple: tree"
+
+  X=$(./yq e '.apple="tree"' test.yml)
+
+  assertEquals "$expected" "$X"
+}
+
+testEmptyWithExpressionEvalAll() {
+  rm test.yml
+  touch test.yml
+  expected="apple: tree"
+
+  X=$(./yq ea '.apple="tree"' test.yml)
+
+  assertEquals "$expected" "$X"
+}
+
+
+source ./scripts/shunit2

acceptance_tests/flags.sh

@@ -0,0 +1,21 @@
+#!/bin/bash
+
+setUp() {
+  rm test*.yml || true
+  cat >test.yml <<EOL
+# comment
+EOL
+}
+
+testStringInterpolation() {
+    X=$(./yq -n '"Mike \(3 + 4)"')
+    assertEquals "Mike 7" "$X"
+}
+
+testNoStringInterpolation() {
+    X=$(./yq --string-interpolation=f -n '"Mike \(3 + 4)"')
+    assertEquals "Mike \(3 + 4)" "$X"
+}
+
+
+source ./scripts/shunit2

acceptance_tests/front-matter.sh

@@ -0,0 +1,76 @@
+#!/bin/bash
+
+setUp() {
+  rm test*.yml || true
+  cat >test.yml <<EOL
+---
+a: apple
+b: cat
+---
+not yaml
+c: at
+EOL
+}
+
+testFrontMatterProcessEval() {
+  read -r -d '' expected << EOM
+---
+a: apple
+b: dog
+---
+not yaml
+c: at
+EOM
+  ./yq e --front-matter="process" '.b = "dog"' test.yml -i
+  assertEquals "$expected" "$(cat test.yml)"
+}
+
+testFrontMatterProcessEvalAll() {
+  read -r -d '' expected << EOM
+---
+a: apple
+b: dog
+---
+not yaml
+c: at
+EOM
+  ./yq ea --front-matter="process" '.b = "dog"' test.yml -i
+  assertEquals "$expected" "$(cat test.yml)"
+}
+
+testFrontMatterExtractEval() {
+    cat >test.yml <<EOL
+a: apple
+b: cat
+---
+not yaml
+c: at
+EOL
+
+  read -r -d '' expected << EOM
+a: apple
+b: dog
+EOM
+  ./yq e --front-matter="extract" '.b = "dog"' test.yml -i
+  assertEquals "$expected" "$(cat test.yml)"
+}
+
+testFrontMatterExtractEvalAll() {
+  cat >test.yml <<EOL
+a: apple
+b: cat
+---
+not yaml
+c: at
+EOL
+
+  read -r -d '' expected << EOM
+a: apple
+b: dog
+EOM
+  ./yq ea --front-matter="extract" '.b = "dog"' test.yml -i
+  assertEquals "$expected" "$(cat test.yml)"
+}
+
+
+source ./scripts/shunit2

acceptance_tests/header-processing-off.sh

@@ -0,0 +1,43 @@
+#!/bin/bash
+
+setUp() {
+  rm test*.yml || true
+  
+}
+
+testLineCountFirstLineComment() {
+  cat >test.yml <<EOL
+#test123 
+abc: 123 
+test123: 123123 
+#comment 
+lalilu: lalilu
+EOL
+
+  X=$(./yq '.lalilu | line' --header-preprocess=false < test.yml)
+  assertEquals "5" "$X"
+}
+
+testArrayOfDocs() {
+  cat >test.yml <<EOL
+---
+# leading comment doc 1
+a: 1
+---
+# leading comment doc 2
+a: 2
+EOL
+
+read -r -d '' expected << EOM
+- # leading comment doc 1
+  a: 1
+- # leading comment doc 2
+  a: 2
+EOM
+
+  X=$(./yq ea '[.]' --header-preprocess=false < test.yml)
+  assertEquals "$expected" "$X"
+
+}
+
+source ./scripts/shunit2

acceptance_tests/inputs-format-auto.sh

@@ -0,0 +1,249 @@
+#!/bin/bash
+
+setUp() {
+  rm test*.yml 2>/dev/null || true
+  rm test*.toml 2>/dev/null || true
+  rm test*.tfstate 2>/dev/null || true
+  rm test*.json 2>/dev/null || true
+  rm test*.properties 2>/dev/null || true
+  rm test*.csv 2>/dev/null || true
+  rm test*.tsv 2>/dev/null || true
+  rm test*.xml 2>/dev/null || true
+}
+
+testInputJson() {
+  cat >test.json <<EOL
+{ "mike" : { "things": "cool" } }
+EOL
+
+  read -r -d '' expected << EOM
+{
+  "mike": {
+    "things": "cool"
+  }
+}
+EOM
+
+  X=$(./yq test.json)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea test.json)
+  assertEquals "$expected" "$X"
+}
+
+testInputToml() {
+  cat >test.toml <<EOL
+[owner]
+name = "Tom Preston-Werner"
+dob = 1979-05-27T07:32:00-08:00
+EOL
+
+  read -r -d '' expected << EOM
+owner:
+  name: Tom Preston-Werner
+  dob: 1979-05-27T07:32:00-08:00
+EOM
+
+  X=$(./yq -oy test.toml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea -oy test.toml)
+  assertEquals "$expected" "$X"
+}
+
+testInputTfstate() {
+  cat >test.tfstate <<EOL
+{ "mike" : { "things": "cool" } }
+EOL
+
+  read -r -d '' expected << EOM
+{"mike": {"things": "cool"}}
+EOM
+
+  X=$(./yq test.tfstate)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea test.tfstate)
+  assertEquals "$expected" "$X"
+}
+
+testInputJsonOutputYaml() {
+  cat >test.json <<EOL
+{ "mike" : { "things": "cool" } }
+EOL
+
+  read -r -d '' expected << EOM
+mike:
+  things: cool
+EOM
+
+  X=$(./yq test.json -oy)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea test.json -oy)
+  assertEquals "$expected" "$X"
+}
+
+testInputProperties() {
+  cat >test.properties <<EOL
+mike.things = hello
+EOL
+
+  read -r -d '' expected << EOM
+mike.things = hello
+EOM
+
+  X=$(./yq e test.properties)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq test.properties)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea test.properties)
+  assertEquals "$expected" "$X"
+}
+
+testInputPropertiesGitHubAction() {
+  cat >test.properties <<EOL
+mike.things = hello
+EOL
+
+  read -r -d '' expected << EOM
+mike.things = hello
+EOM
+
+  X=$(cat /dev/null | ./yq e test.properties)
+  assertEquals "$expected" "$X"
+
+  X=$(cat /dev/null | ./yq ea test.properties)
+  assertEquals "$expected" "$X"
+}
+
+testInputCSV() {
+  cat >test.csv <<EOL
+fruit,yumLevel
+apple,5
+banana,4
+EOL
+
+  read -r -d '' expected << EOM
+fruit,yumLevel
+apple,5
+banana,4
+EOM
+
+  X=$(./yq e test.csv)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea test.csv)
+  assertEquals "$expected" "$X"
+}
+
+testInputCSVUTF8() {
+  read -r -d '' expected << EOM
+id,first,last
+1,john,smith
+1,jane,smith
+EOM
+
+  X=$(./yq utf8.csv)
+  assertEquals "$expected" "$X"
+}
+
+testInputTSV() {
+  cat >test.tsv <<EOL
+fruit	yumLevel
+apple	5
+banana	4
+EOL
+
+  read -r -d '' expected << EOM
+fruit	yumLevel
+apple	5
+banana	4
+EOM
+
+  X=$(./yq e test.tsv)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea test.tsv)
+  assertEquals "$expected" "$X"
+}
+
+
+
+
+testInputXml() {
+  cat >test.xml <<EOL
+<cat legs="4">BiBi</cat>
+EOL
+
+  read -r -d '' expected << EOM
+<cat legs="4">BiBi</cat>
+EOM
+
+  X=$(./yq e test.xml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea test.xml)
+  assertEquals "$expected" "$X"
+}
+
+testInputXmlNamespaces() {
+  cat >test.xml <<EOL
+<?xml version="1.0"?>
+<map xmlns="some-namespace" xmlns:xsi="some-instance" xsi:schemaLocation="some-url">
+</map>
+EOL
+
+  read -r -d '' expected << EOM
+<?xml version="1.0"?>
+<map xmlns="some-namespace" xmlns:xsi="some-instance" xsi:schemaLocation="some-url"></map>
+EOM
+
+  X=$(./yq e test.xml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea test.xml)
+  assertEquals "$expected" "$X"
+}
+
+
+
+testInputXmlStrict() {
+  cat >test.xml <<EOL
+<?xml version="1.0"?>
+<!DOCTYPE root [
+<!ENTITY writer "Catherine.">
+<!ENTITY copyright "(r) Great">
+]>
+<root>
+    <item>&writer;&copyright;</item>
+</root>
+EOL
+
+  X=$(./yq --xml-strict-mode test.xml  2>&1)
+  assertEquals 1 $?
+  assertEquals "Error: bad file 'test.xml': XML syntax error on line 7: invalid character entity &writer;" "$X"
+
+  X=$(./yq ea --xml-strict-mode test.xml  2>&1)
+  assertEquals "Error: bad file 'test.xml': XML syntax error on line 7: invalid character entity &writer;" "$X"
+}
+
+testInputXmlGithubAction() {
+  cat >test.xml <<EOL
+<cat legs="4">BiBi</cat>
+EOL
+
+  read -r -d '' expected << EOM
+<cat legs="4">BiBi</cat>
+EOM
+
+  X=$(cat /dev/null | ./yq e test.xml)
+  assertEquals "$expected" "$X"
+
+  X=$(cat /dev/null | ./yq ea test.xml)
+  assertEquals "$expected" "$X"
+}
+
+source ./scripts/shunit2

acceptance_tests/inputs-format.sh

@@ -0,0 +1,258 @@
+#!/bin/bash
+
+setUp() {
+  rm test*.yml 2>/dev/null || true
+  rm test*.properties 2>/dev/null || true
+  rm test*.csv 2>/dev/null || true
+  rm test*.tsv 2>/dev/null || true
+  rm test*.xml 2>/dev/null || true
+}
+
+testInputProperties() {
+  cat >test.properties <<EOL
+mike.things = hello
+EOL
+
+  read -r -d '' expected << EOM
+mike:
+  things: hello
+EOM
+
+  X=$(./yq e -p=props test.properties)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea -p=props test.properties)
+  assertEquals "$expected" "$X"
+}
+
+testInputPropertiesGitHubAction() {
+  cat >test.properties <<EOL
+mike.things = hello
+EOL
+
+  read -r -d '' expected << EOM
+mike:
+  things: hello
+EOM
+
+  X=$(cat /dev/null | ./yq e -p=props test.properties)
+  assertEquals "$expected" "$X"
+
+  X=$(cat /dev/null | ./yq ea -p=props test.properties)
+  assertEquals "$expected" "$X"
+}
+
+testInputCSV() {
+  cat >test.csv <<EOL
+fruit,yumLevel
+apple,5
+banana,4
+EOL
+
+  read -r -d '' expected << EOM
+- fruit: apple
+  yumLevel: 5
+- fruit: banana
+  yumLevel: 4
+EOM
+
+  X=$(./yq e -p=csv test.csv)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea -p=csv test.csv)
+  assertEquals "$expected" "$X"
+}
+
+testInputCSVCustomSeparator() {
+  cat >test.csv <<EOL
+fruit;yumLevel
+apple;5
+banana;4
+EOL
+
+  read -r -d '' expected << EOM
+- fruit: apple
+  yumLevel: 5
+- fruit: banana
+  yumLevel: 4
+EOM
+
+  X=$(./yq -p=csv --csv-separator ";" test.csv)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea -p=csv --csv-separator ";" test.csv)
+  assertEquals "$expected" "$X"
+}
+
+testInputCSVNoAuto() {
+  cat >test.csv <<EOL
+thing1
+name: cat
+EOL
+
+  read -r -d '' expected << EOM
+- thing1: 'name: cat'
+EOM
+
+  X=$(./yq --csv-auto-parse=f test.csv -oy)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea --csv-auto-parse=f test.csv -oy)
+  assertEquals "$expected" "$X"
+}
+
+testInputTSVNoAuto() {
+  cat >test.tsv <<EOL
+thing1
+name: cat
+EOL
+
+  read -r -d '' expected << EOM
+- thing1: 'name: cat'
+EOM
+
+  X=$(./yq --tsv-auto-parse=f test.tsv -oy)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea --tsv-auto-parse=f test.tsv -oy)
+  assertEquals "$expected" "$X"
+}
+
+testInputCSVUTF8() {
+  read -r -d '' expected << EOM
+- id: 1
+  first: john
+  last: smith
+- id: 1
+  first: jane
+  last: smith
+EOM
+
+  X=$(./yq -p=csv utf8.csv)
+  assertEquals "$expected" "$X"
+}
+
+testInputTSV() {
+  cat >test.tsv <<EOL
+fruit	yumLevel
+apple	5
+banana	4
+EOL
+
+  read -r -d '' expected << EOM
+- fruit: apple
+  yumLevel: 5
+- fruit: banana
+  yumLevel: 4
+EOM
+
+  X=$(./yq e -p=t test.tsv)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea -p=t test.tsv)
+  assertEquals "$expected" "$X"
+}
+
+
+
+
+testInputXml() {
+  cat >test.yml <<EOL
+<cat legs="4">BiBi</cat>
+EOL
+
+  read -r -d '' expected << EOM
+cat:
+  +content: BiBi
+  +@legs: "4"
+EOM
+
+  X=$(./yq e -p=xml test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea -p=xml test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testInputXmlNamespaces() {
+  cat >test.xml <<EOL
+<?xml version="1.0"?>
+<map xmlns="some-namespace" xmlns:xsi="some-instance" xsi:schemaLocation="some-url">
+</map>
+EOL
+
+  read -r -d '' expected << EOM
++p_xml: version="1.0"
+map:
+  +@xmlns: some-namespace
+  +@xmlns:xsi: some-instance
+  +@xsi:schemaLocation: some-url
+EOM
+
+  X=$(./yq e -p=xml test.xml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea -p=xml test.xml)
+  assertEquals "$expected" "$X"
+}
+
+testInputXmlRoundtrip() {
+  cat >test.yml <<EOL
+<?xml version="1.0"?>
+<!DOCTYPE config SYSTEM "/etc/iwatch/iwatch.dtd" >
+<map xmlns="some-namespace" xmlns:xsi="some-instance" xsi:schemaLocation="some-url">Meow</map>
+EOL
+
+  read -r -d '' expected << EOM
+<?xml version="1.0"?>
+<!DOCTYPE config SYSTEM "/etc/iwatch/iwatch.dtd" >
+<map xmlns="some-namespace" xmlns:xsi="some-instance" xsi:schemaLocation="some-url">Meow</map>
+EOM
+
+  X=$(./yq -p=xml -o=xml test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea -p=xml -o=xml test.yml)
+  assertEquals "$expected" "$X"
+}
+
+
+testInputXmlStrict() {
+  cat >test.yml <<EOL
+<?xml version="1.0"?>
+<!DOCTYPE root [
+<!ENTITY writer "Catherine.">
+<!ENTITY copyright "(r) Great">
+]>
+<root>
+    <item>&writer;&copyright;</item>
+</root>
+EOL
+
+  X=$(./yq -p=xml --xml-strict-mode test.yml -o=xml 2>&1)
+  assertEquals 1 $?
+  assertEquals "Error: bad file 'test.yml': XML syntax error on line 7: invalid character entity &writer;" "$X"
+
+  X=$(./yq ea -p=xml --xml-strict-mode test.yml -o=xml 2>&1)
+  assertEquals "Error: bad file 'test.yml': XML syntax error on line 7: invalid character entity &writer;" "$X"
+}
+
+testInputXmlGithubAction() {
+  cat >test.yml <<EOL
+<cat legs="4">BiBi</cat>
+EOL
+
+  read -r -d '' expected << EOM
+cat:
+  +content: BiBi
+  +@legs: "4"
+EOM
+
+  X=$(cat /dev/null | ./yq e -p=xml test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(cat /dev/null | ./yq ea -p=xml test.yml)
+  assertEquals "$expected" "$X"
+}
+
+source ./scripts/shunit2

acceptance_tests/leading-separator.sh

@@ -0,0 +1,529 @@
+#!/bin/bash
+
+
+# examples where header-preprocess is required
+
+setUp() {
+  rm test*.yml || true
+  cat >test.yml <<EOL
+---
+a: test
+EOL
+}
+
+testLeadingSeparatorWithDoc() {
+  cat >test.yml <<EOL
+# hi peeps
+# cool
+---
+a: test
+---
+b: cool
+EOL
+
+  read -r -d '' expected << EOM
+# hi peeps
+# cool
+---
+a: thing
+---
+b: cool
+EOM
+
+  X=$(./yq e '(select(di == 0) | .a) = "thing"' - < test.yml)
+  assertEquals "$expected" "$X"
+}
+
+
+testLeadingSeparatorWithNewlinesNewDoc() {
+  cat >test.yml <<EOL
+# hi peeps
+# cool
+
+
+---
+a: test
+---
+b: cool
+EOL
+
+  read -r -d '' expected << EOM
+# hi peeps
+# cool
+
+
+---
+a: thing
+---
+b: cool
+EOM
+
+  X=$(./yq e '(select(di == 0) | .a) = "thing"' - < test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testLeadingSeparatorWithNewlinesMoreComments() {
+  cat >test.yml <<EOL
+# hi peeps
+# cool
+
+---
+# great
+
+a: test
+---
+b: cool
+EOL
+
+  read -r -d '' expected << EOM
+# hi peeps
+# cool
+
+---
+# great
+
+a: thing
+---
+b: cool
+EOM
+
+  X=$(./yq e '(select(di == 0) | .a) = "thing"' - < test.yml)
+  assertEquals "$expected" "$X"
+}
+
+
+testLeadingSeparatorWithDirective() {
+  cat >test.yml <<EOL
+%YAML 1.1
+---
+this: should really work
+EOL
+
+  read -r -d '' expected << EOM
+%YAML 1.1
+---
+this: should really work
+EOM
+
+  X=$(./yq < test.yml)
+  assertEquals "$expected" "$X"
+}
+
+
+testLeadingSeparatorPipeIntoEvalSeq() {
+  X=$(./yq e - < test.yml)
+  expected=$(cat test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testLeadingSeparatorExtractField() {
+  X=$(./yq e '.a' - < test.yml)
+  assertEquals "test" "$X"
+}
+
+testLeadingSeparatorExtractFieldWithCommentsAfterSep() {
+  cat >test.yml <<EOL
+---
+# hi peeps
+# cool
+a: test
+EOL
+  X=$(./yq e '.a' test.yml)
+  assertEquals "test" "$X"
+}
+
+testLeadingSeparatorExtractFieldWithCommentsBeforeSep() {
+  cat >test.yml <<EOL
+# hi peeps
+# cool
+---
+a: test
+EOL
+  X=$(./yq e '.a' test.yml)
+  assertEquals "test" "$X"
+}
+
+
+testLeadingSeparatorExtractFieldMultiDoc() {
+  cat >test.yml <<EOL
+---
+a: test
+---
+a: test2
+EOL
+
+  read -r -d '' expected << EOM
+test
+---
+test2
+EOM
+  X=$(./yq e '.a' test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testLeadingSeparatorExtractFieldMultiDocWithComments() {
+  cat >test.yml <<EOL
+# here
+---
+# there
+a: test
+# wherever
+---
+# you are
+a: test2
+# woop
+EOL
+
+  read -r -d '' expected << EOM
+test
+---
+test2
+EOM
+  X=$(./yq e '.a' test.yml)
+  assertEquals "$expected" "$X"
+}
+
+
+testLeadingSeparatorEvalSeq() {
+  X=$(./yq e test.yml)
+  expected=$(cat test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testLeadingSeparatorPipeIntoEvalAll() {
+  X=$(./yq ea - < test.yml)
+  expected=$(cat test.yml)
+  assertEquals "$expected" "$X"
+}
+
+
+testLeadingSeparatorEvalAll() {
+  X=$(./yq ea test.yml)
+  expected=$(cat test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testLeadingSeparatorMultiDocEvalSimple() {
+  read -r -d '' expected << EOM
+---
+a: test
+---
+version: 3
+application: MyApp
+EOM
+
+
+  X=$(./yq e '.' test.yml examples/order.yaml)
+  assertEquals "$expected" "$X"
+}
+
+testLeadingSeparatorMultiDocInOneFile() {
+  cat >test.yml <<EOL
+---
+# hi peeps
+# cool
+a: test
+---
+b: things
+EOL
+  expected=$(cat test.yml)
+  X=$(./yq e '.' test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testLeadingSeparatorMultiDocInOneFileEvalAll() {
+  cat >test.yml <<EOL
+---
+# hi peeps
+# cool
+a: test
+---
+b: things
+EOL
+  expected=$(cat test.yml)
+  X=$(./yq ea '.' test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testLeadingSeparatorMultiDocEvalComments() {
+  cat >test.yml <<EOL
+# hi peeps
+# cool
+a: test
+EOL
+
+cat >test2.yml <<EOL
+# this is another doc
+# great
+b: sane
+EOL
+
+  read -r -d '' expected << EOM
+# hi peeps
+# cool
+a: test
+---
+# this is another doc
+# great
+b: sane
+EOM
+
+
+  X=$(./yq e '.' test.yml test2.yml)
+  assertEquals "$expected" "$X"
+}
+
+testLeadingSeparatorMultiDocEvalCommentsTrailingSep() {
+  cat >test.yml <<EOL
+# hi peeps
+# cool
+---
+a: test
+EOL
+
+cat >test2.yml <<EOL
+# this is another doc
+# great
+---
+b: sane
+EOL
+
+  read -r -d '' expected << EOM
+# hi peeps
+# cool
+---
+a: test
+---
+# this is another doc
+# great
+---
+b: sane
+EOM
+
+
+  X=$(./yq e '.' test.yml test2.yml)
+  assertEquals "$expected" "$X"
+}
+
+testLeadingSeparatorMultiMultiDocEvalCommentsTrailingSep() {
+  cat >test.yml <<EOL
+# hi peeps
+# cool
+---
+a: test
+---
+a1: test2
+EOL
+
+cat >test2.yml <<EOL
+# this is another doc
+# great
+---
+b: sane
+---
+b2: cool
+EOL
+
+  read -r -d '' expected << EOM
+# hi peeps
+# cool
+---
+a: test
+---
+a1: test2
+---
+# this is another doc
+# great
+---
+b: sane
+---
+b2: cool
+EOM
+
+
+  X=$(./yq e '.' test.yml test2.yml)
+  assertEquals "$expected" "$X"
+}
+
+testLeadingSeparatorMultiDocEvalCommentsLeadingSep() {
+  cat >test.yml <<EOL
+---
+# hi peeps
+# cool
+a: test
+EOL
+
+cat >test2.yml <<EOL
+---
+# this is another doc
+# great
+b: sane
+EOL
+
+  read -r -d '' expected << EOM
+---
+# hi peeps
+# cool
+a: test
+---
+# this is another doc
+# great
+b: sane
+EOM
+
+
+  X=$(./yq e '.' test.yml test2.yml)
+  assertEquals "$expected" "$X"
+}
+
+# https://github.com/mikefarah/yq/issues/919
+testLeadingSeparatorDoesNotBreakCommentsOnOtherFiles() {
+  cat >test.yml <<EOL
+# a1
+a: 1
+# a2
+EOL
+
+cat >test2.yml <<EOL
+# b1
+b: 2
+# b2
+EOL
+
+  read -r -d '' expected << EOM
+# a1
+a: 1
+# a2
+
+# b1
+b: 2
+# b2
+EOM
+
+
+  X=$(./yq ea 'select(fi == 0) * select(fi == 1)' test.yml test2.yml)
+  assertEquals "$expected" "$X"
+}
+
+testLeadingSeparatorMultiDocEvalCommentsStripComments() {
+  cat >test.yml <<EOL
+---
+# hi peeps
+# cool
+a: test
+---
+# this is another doc
+# great
+b: sane
+EOL
+
+  # it will be hard to remove that top level separator
+  read -r -d '' expected << EOM
+a: test
+---
+b: sane
+EOM
+
+  X=$(./yq e '... comments=""'  test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testLeadingSeparatorMultiDocEvalCommentsLeadingSepNoDocFlag() {
+  cat >test.yml <<EOL
+---
+# hi peeps
+# cool
+a: test
+---
+# this is another doc
+# great
+b: sane
+EOL
+
+  read -r -d '' expected << EOM
+# hi peeps
+# cool
+a: test
+# this is another doc
+# great
+b: sane
+EOM
+
+
+  X=$(./yq e '.' --no-doc test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testLeadingSeparatorMultiDocEvalJsonFlag() {
+  cat >test.yml <<EOL
+---
+# hi peeps
+# cool
+a: test
+EOL
+
+cat >test2.yml <<EOL
+---
+# this is another doc
+# great
+b: sane
+EOL
+
+  read -r -d '' expected << EOM
+{
+  "a": "test"
+}
+{
+  "b": "sane"
+}
+EOM
+
+
+  X=$(./yq e '.' -j test.yml test2.yml)
+  assertEquals "$expected" "$X"
+}
+
+testLeadingSeparatorMultiDocEvalAllJsonFlag() {
+  cat >test.yml <<EOL
+---
+# hi peeps
+# cool
+a: test
+EOL
+
+cat >test2.yml <<EOL
+---
+# this is another doc
+# great
+b: sane
+EOL
+
+  read -r -d '' expected << EOM
+{
+  "a": "test"
+}
+{
+  "b": "sane"
+}
+EOM
+
+
+  X=$(./yq ea '.' -j test.yml test2.yml)
+  assertEquals "$expected" "$X"
+}
+
+testLeadingSeparatorMultiDocEvalAll() {
+  read -r -d '' expected << EOM
+---
+a: test
+---
+version: 3
+application: MyApp
+EOM
+
+
+  X=$(./yq ea '.' test.yml examples/order.yaml)
+  assertEquals "$expected" "$X"
+}
+
+source ./scripts/shunit2

acceptance_tests/load-file.sh

@@ -0,0 +1,27 @@
+#!/bin/bash
+
+testLoadFileNotExist() {
+  result=$(./yq e -n 'load("cat.yml")' 2>&1)
+  assertEquals 1 $?
+  assertEquals "Error: failed to load cat.yml: open cat.yml: no such file or directory" "$result"
+}
+
+testLoadFileExpNotExist() {
+  result=$(./yq e -n 'load(.a)' 2>&1)
+  assertEquals 1 $?
+  assertEquals "Error: filename expression returned nil" "$result"
+}
+
+testStrLoadFileNotExist() {
+  result=$(./yq e -n 'strload("cat.yml")' 2>&1)
+  assertEquals 1 $?
+  assertEquals "Error: failed to load cat.yml: open cat.yml: no such file or directory" "$result"
+}
+
+testStrLoadFileExpNotExist() {
+  result=$(./yq e -n 'strload(.a)' 2>&1)
+  assertEquals 1 $?
+  assertEquals "Error: filename expression returned nil" "$result"
+}
+
+source ./scripts/shunit2

acceptance_tests/nul-separator.sh

@@ -0,0 +1,286 @@
+#!/bin/bash
+
+setUp() {
+    rm test*.yml || true
+}
+
+## Convenient bash shortcut to read records of NUL separated values
+## from stdin the safe way. See example usage in the next tests.
+read-0() {
+    local eof="" IFS=''
+    while [ "$1" ]; do
+        ## - The `-r` avoids bad surprise with '\n' and other interpreted
+        ##   sequences that can be read.
+        ## - The `-d ''` is the (strange?) way to refer to NUL delimiter.
+        ## - The `--` is how to avoid unpleasant surprises if your
+        ##   "$1" starts with "-" (minus) sign. This protection also
+        ##   will produce a readable error if you want to try to start
+        ##   your variable names with a "-".
+        read -r -d '' -- "$1" || eof=1
+        shift
+    done
+    [ -z "$eof" ] ## fail on EOF
+}
+
+## Convenient bash shortcut to be used with the next function `p-err`
+## to read NUL separated values the safe way AND catch any errors from
+## the process creating the stream of NUL separated data.  See example
+## usage in the tests.
+read-0-err() {
+    local ret="$1" eof="" idx=0 last=
+    read -r -- "${ret?}" <<<"0"
+    shift
+    while [ "$1" ]; do
+        last=$idx
+        read -r -d '' -- "$1" || {
+            ## Put this last value in ${!ret}
+            eof="$1"
+            read -r -- "$ret" <<<"${!eof}"
+            break
+        }
+        ((idx++))
+        shift
+    done
+    [ -z "$eof" ] || {
+        if [ "$last" != 0 ]; then
+            ## Uhoh, we have no idea if the errorlevel of the internal
+            ## command was properly delimited with a NUL char, and
+            ## anyway something went really wrong at least about the
+            ## number of fields separated by NUL char and the one
+            ## expected.
+            echo "Error: read-0-err couldn't fill all value $ret = '${!ret}', '$eof', '${!eof}'" >&2
+            read -r -- "$ret" <<<"not-enough-values"
+        else
+            if ! [[ "${!ret}" =~ ^[0-9]+$ && "${!ret}" -ge 0 && "${!ret}" -le 127 ]]; then
+                ## This could happen if you don't use `p-err` wrapper,
+                ## or used stdout in unexpected ways in your inner
+                ## command.
+                echo "Error: last value is not a number, did you finish with an errorlevel ?" >&2
+                read -r -- "$ret" <<<"last-value-not-a-number"
+            fi
+        fi
+        false
+    }
+}
+
+## Simply runs command given as argument and adds errorlevel in the
+## standard output. Is expected to be used in tandem with
+## `read-0-err`.
+p-err() {
+    local exp="$1"
+    "$@"
+    printf "%s" "$?"
+}
+
+wyq-r() {
+    local exp="$1"
+    ./yq e -0 -r=false "$1"
+    printf "%s" "$?"
+}
+
+testBasicUsageRaw() {
+  cat >test.yml <<EOL
+a: foo
+b: bar
+EOL
+
+  printf "foo\0bar\0" > expected.out
+
+  ## We need to compare binary content here. We have to filter the compared
+  ## content through a representation that gets rid of NUL chars but accurately
+  ## transcribe the content.
+  ## Also as it would be nice to have a pretty output in case the test fails,
+  ## we use here 'hd': a widely available shortcut to 'hexdump' that will
+  ## pretty-print any binary to it's hexadecimal representation.
+  ##
+  ## Note that the standard `assertEquals` compare its arguments
+  ## value, but they can't hold NUL characters (this comes from the
+  ## limitation of the C API of `exec*(..)` functions that requires
+  ## `const char *arv[]`). And these are NUL terminated strings.  As a
+  ## consequence, the NUL characters gets removed in bash arguments.
+  assertEquals "$(hd expected.out)" \
+               "$(./yq e -0 '.a, .b' test.yml | hd)"
+
+  rm expected.out
+}
+
+testBasicUsage() {
+  local a b
+  cat >test.yml <<EOL
+a: foo
+b: bar
+EOL
+
+  ## We provide 2 values, and ask to fill 2 variables.
+  read-0 a b < <(./yq e -0 '.a, .b' test.yml)
+  assertEquals "$?" "0"      ## Everything is fine
+  assertEquals "foo" "$a"    ## Values are correctly parsed
+  assertEquals "bar" "$b"
+
+  a=YYY ; b=XXX
+  ## Not enough values provided to fill `a` and `b`.
+  read-0 a b < <(./yq e -0 '.a' test.yml)
+  assertEquals "$?" "1"      ## An error was emitted
+  assertEquals "foo" "$a"    ## First value was correctly parsed
+  assertEquals "" "$b"       ## Second was still reset
+
+  ## Error from inner command are not catchable !. Use
+  ## `read-0-err`/`p-err` for that.
+  read-0 a < <(printf "\0"; ./yq e -0 'xxx' test.yml; )
+  assertEquals "$?" "0"
+
+}
+
+testBasicUsageJson() {
+  cat >test.yml <<EOL
+a:
+  x: foo
+b: bar
+EOL
+
+  read-0 a b < <(./yq e -0 -o=json '.a, .b' test.yml)
+
+  assertEquals '{
+  "x": "foo"
+}' "$a"
+  assertEquals '"bar"' "$b"
+
+}
+
+testFailWithValueContainingNUL() {
+  local a b c
+  ## Note that value of field 'a' actually contains a NUL char !
+  cat >test.yml <<EOL
+a: "foo\u0000bar"
+b: 1
+c: |
+  wiz
+  boom
+EOL
+
+  ## We are looking for trouble with asking to separated fields with NUL
+  ## char and requested value `.a` actually contains itself a NUL char !
+  read-0 a b c < <(./yq e -0 '.a, .b, .c' test.yml)
+  assertNotEquals "0" "$?"   ## read-0 failed to fill all values
+
+  ## But here, we can request for one value, even if `./yq` fails
+  read-0 b < <(./yq e -0 '.b, .a' test.yml)
+  assertEquals "0" "$?"   ## read-0 succeeds at feeding the first value
+  ## Note: to catch the failure of `yq`, see in the next tests the usage
+  ## of `read-0-err`.
+
+  ## using -r=false solves any NUL containing value issues, but keeps
+  ## all in YAML representation:
+  read-0 a b c < <(./yq e -0 -r=false '.a, .b, .c' test.yml)
+  assertEquals "0" "$?"    ## All goes well despite asking for `a` value
+
+  assertEquals '"foo\0bar"' "$a"   ## This is a YAML string representation
+  assertEquals '1' "$b"
+  assertEquals '|
+  wiz
+  boom' "$c"
+}
+
+testStandardLoop() {
+    local E a b res
+
+    ## Here everything is normal: 4 values, that will be paired
+    ## in key/values.
+    cat >test.yml <<EOL
+- yay
+- wiz
+- hop
+- pow
+EOL
+
+    res=""
+    while read-0-err E a b; do
+        res+="$a: $b;"
+    done < <(p-err ./yq -0 '.[]' test.yml)
+
+    assertEquals "0" "$E"                     ## errorlevel of internal command
+    assertEquals "yay: wiz;hop: pow;" "$res"  ## expected result
+}
+
+testStandardLoopWithoutEnoughValues() {
+    local E a b res
+
+    ## Here 5 values, there will be a missing value when reading
+    ## pairs of value.
+    cat >test.yml <<EOL
+- yay
+- wiz
+- hop
+- pow
+- kwak
+EOL
+
+    res=""
+    ## The loop will succeed 2 times then fail
+    while read-0-err E a b; do
+        res+="$a: $b;"
+    done < <(p-err ./yq -0 '.[]' test.yml)
+
+    assertEquals "not-enough-values" "$E"     ## Not enough value error
+    assertEquals "yay: wiz;hop: pow;" "$res"  ## the 2 full key/value pairs
+
+}
+
+testStandardLoopWithInternalCmdError() {
+    local E a b res
+
+    ## Note the third value contains a NUL char !
+    cat >test.yml <<EOL
+- yay
+- wiz
+- "foo\0bar"
+- hop
+- pow
+EOL
+
+    res=""
+    ## It should be only upon the second pass in the loop that
+    ## read-0-err will catch the fact that there is an error !
+    while read-0-err E a b; do
+        res+="$a: $b;"
+    done < <(p-err ./yq -0 '.[]' test.yml)
+    assertEquals "1" "$E"            ## Internal command errorlevel (from `./yq`)
+    assertEquals "yay: wiz;" "$res"  ## first 2 values were ok at least
+
+}
+
+testStandardLoopNotEnoughErrorEatsCmdError() {
+    local E a b res
+
+    ## Because of possible edge cases where the internal errorlevel
+    ## reported by `p-err` in the standard output might be mangled
+    ## with the unfinished record, `read-0-err E ...` will NOT report
+    ## the internal command error in the variable E and instead will
+    ## store the value 'not-enough-values'. In real world, anyway, you
+    ## will want to react the same if the internal command failed
+    ## and/or you didn't get as much values as expected while
+    ## reading. Keep in mind also that standard error is not
+    ## swallowed, so you can read reports from the inner command AND
+    ## from `read-0-err`.
+
+    ## Here, note that the fourth value contains a NUL char !
+    cat >test.yml <<EOL
+- yay
+- wiz
+- hop
+- "foo\0bar"
+- pow
+EOL
+
+    res=""
+    ## It should be only upon the second loop that read-0-err will catch
+    ## the fact that there are not enough data to fill the requested variables
+    while read-0-err E a b; do
+        res+="$a: $b;"
+    done < <(p-err ./yq -0 '.[]' test.yml)
+    assertEquals "not-enough-values" "$E"          ## Not enough values error eats internal error !
+    assertEquals "yay: wiz;" "$res"  ## first 2 values were ok at least
+}
+
+
+source ./scripts/shunit2

acceptance_tests/output-format.sh

@@ -0,0 +1,343 @@
+#!/bin/bash
+
+setUp() {
+  rm test*.yml || true
+}
+
+testOutputJsonDeprecated() {
+  cat >test.yml <<EOL
+a: {b: ["cat"]}
+EOL
+
+  read -r -d '' expected << EOM
+{
+  "a": {
+    "b": [
+      "cat"
+    ]
+  }
+}
+EOM
+
+  X=$(./yq e -j test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea -j test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testOutputJson() {
+  cat >test.yml <<EOL
+a: {b: ["cat"]}
+EOL
+
+  read -r -d '' expected << EOM
+{
+  "a": {
+    "b": [
+      "cat"
+    ]
+  }
+}
+EOM
+
+  X=$(./yq e --output-format=json test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea --output-format=json test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testOutputYamlRawDefault() {
+  cat >test.yml <<EOL
+a: "cat"
+EOL
+
+  X=$(./yq e  '.a' test.yml)
+  assertEquals "cat" "$X"
+
+  X=$(./yq ea '.a' test.yml)
+  assertEquals "cat" "$X"
+}
+
+testOutputYamlRawOff() {
+  cat >test.yml <<EOL
+a: "cat"
+EOL
+
+  X=$(./yq e -r=false '.a' test.yml)
+  assertEquals "\"cat\"" "$X"
+
+  X=$(./yq ea -r=false '.a' test.yml)
+  assertEquals "\"cat\"" "$X"
+}
+
+testOutputYamlRawOnRoot() {
+  cat >test.yml <<EOL
+'a'
+EOL
+
+  X=$(./yq e -r '.' test.yml)
+  assertEquals "a" "$X"
+}
+
+testOutputJsonRaw() {
+  cat >test.yml <<EOL
+a: cat
+EOL
+
+  X=$(./yq e -r --output-format=json '.a' test.yml)
+  assertEquals "cat" "$X"
+
+  X=$(./yq ea -r --output-format=json '.a' test.yml)
+  assertEquals "cat" "$X"
+}
+
+testOutputJsonDefault() {
+  cat >test.yml <<EOL
+a: cat
+EOL
+
+  X=$(./yq e --output-format=json '.a' test.yml)
+  assertEquals "\"cat\"" "$X"
+
+  X=$(./yq ea --output-format=json '.a' test.yml)
+  assertEquals "\"cat\"" "$X"
+}
+
+
+testOutputJsonShort() {
+  cat >test.yml <<EOL
+a: {b: ["cat"]}
+EOL
+
+  read -r -d '' expected << EOM
+{
+  "a": {
+    "b": [
+      "cat"
+    ]
+  }
+}
+EOM
+
+  X=$(./yq e -o=j test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea -o=j test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testOutputProperties() {
+  cat >test.yml <<EOL
+a: {b: {c: ["cat cat"]}}
+EOL
+
+  read -r -d '' expected << EOM
+a.b.c.0 = cat cat
+EOM
+
+  X=$(./yq e --output-format=props test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea --output-format=props test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testOutputPropertiesDontUnwrap() {
+  cat >test.yml <<EOL
+a: {b: {c: ["cat cat"]}}
+EOL
+
+  read -r -d '' expected << EOM
+a.b.c.0 = "cat cat"
+EOM
+
+  X=$(./yq e -r=false --output-format=props test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea -r=false --output-format=props test.yml)
+  assertEquals "$expected" "$X"
+}
+
+
+testOutputPropertiesShort() {
+  cat >test.yml <<EOL
+a: {b: {c: ["cat cat"]}}
+EOL
+
+  read -r -d '' expected << EOM
+a.b.c.0 = cat cat
+EOM
+
+  X=$(./yq e -o=p test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea -o=p test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testOutputCSV() {
+  cat >test.yml <<EOL
+- fruit: apple
+  yumLevel: 5
+- fruit: banana
+  yumLevel: 4
+EOL
+
+  read -r -d '' expected << EOM
+fruit,yumLevel
+apple,5
+banana,4
+EOM
+
+  X=$(./yq -o=c test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea -o=csv test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testOutputCSVCustomSeparator() {
+  cat >test.yml <<EOL
+- fruit: apple
+  yumLevel: 5
+- fruit: banana
+  yumLevel: 4
+EOL
+
+  read -r -d '' expected << EOM
+fruit;yumLevel
+apple;5
+banana;4
+EOM
+
+  X=$(./yq -oc --csv-separator ";" test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea -o=csv --csv-separator ";" test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testOutputTSV() {
+  cat >test.yml <<EOL
+- fruit: apple
+  yumLevel: 5
+- fruit: banana
+  yumLevel: 4
+EOL
+
+  read -r -d '' expected << EOM
+fruit	yumLevel
+apple	5
+banana	4
+EOM
+
+  X=$(./yq -o=t test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea -o=tsv test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testOutputXml() {
+  cat >test.yml <<EOL
+a: {b: {c: ["cat"]}}
+EOL
+
+  read -r -d '' expected << EOM
+<a>
+  <b>
+    <c>cat</c>
+  </b>
+</a>
+EOM
+
+  X=$(./yq e --output-format=xml test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea --output-format=xml test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testOutputXmlShort() {
+  cat >test.yml <<EOL
+a: {b: {c: ["cat"]}}
+EOL
+
+  read -r -d '' expected << EOM
+<a>
+  <b>
+    <c>cat</c>
+  </b>
+</a>
+EOM
+
+  X=$(./yq e --output-format=x test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea --output-format=x test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testOutputXmComplex() {
+  cat >test.yml <<EOL
+a: {b: {c: ["cat", "dog"], +@f: meow}}
+EOL
+
+  read -r -d '' expected << EOM
+<a>
+  <b f="meow">
+    <c>cat</c>
+    <c>dog</c>
+  </b>
+</a>
+EOM
+
+  X=$(./yq e --output-format=x test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea --output-format=x test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testLuaOutputPretty() {
+  cat >test.yml <<EOL
+animals:
+  cat: meow
+EOL
+
+  read -r -d '' expected << EOM
+return {
+	["animals"] = {
+		["cat"] = "meow";
+	};
+};
+EOM
+
+  X=$(./yq e --output-format=lua test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq e --output-format=lua --prettyPrint test.yml)
+  assertEquals "$expected" "$X"
+
+}
+
+testLuaOutputSubset() {
+  cat >test.yml <<EOL
+animals:
+  cat: meow
+EOL
+
+  read -r -d '' expected << EOM
+return {
+	["cat"] = "meow";
+};
+EOM
+
+  X=$(./yq e --output-format=lua '.animals' test.yml)
+  assertEquals "$expected" "$X"
+
+}
+
+source ./scripts/shunit2

acceptance_tests/pipe.sh

@@ -0,0 +1,70 @@
+#!/bin/bash
+
+setUp() {
+  rm test*.yml || true
+  cat >test.yml <<EOL
+a: frog
+EOL
+}
+
+testPipeViaCatWithParam() {
+  X=$(cat test.yml | ./yq '.a')
+  assertEquals "frog" "$X"
+}
+
+testPipeViaCatWithParamEval() {
+  X=$(cat test.yml | ./yq e '.a')
+  assertEquals "frog" "$X"
+}
+
+testPipeViaCatWithParamEvalAll() {
+  X=$(cat test.yml | ./yq ea '.a')
+  assertEquals "frog" "$X"
+}
+
+testPipeViaCatNoParam() {
+  X=$(cat test.yml | ./yq)
+  assertEquals "a: frog" "$X"
+}
+
+testPipeViaCatNoParamEval() {
+  X=$(cat test.yml | ./yq e)
+  assertEquals "a: frog" "$X"
+}
+
+testPipeViaCatNoParamEvalAll() {
+  X=$(cat test.yml | ./yq ea)
+  assertEquals "a: frog" "$X"
+}
+
+testPipeViaFileishWithParam() {
+  X=$(./yq '.a' < test.yml)
+  assertEquals "frog" "$X"
+}
+
+testPipeViaFileishWithParamEval() {
+  X=$(./yq e '.a' < test.yml)
+  assertEquals "frog" "$X"
+}
+
+testPipeViaFileishWithParamEvalAll() {
+  X=$(./yq ea '.a' < test.yml)
+  assertEquals "frog" "$X"
+}
+
+testPipeViaFileishNoParam() {
+  X=$(./yq < test.yml)
+  assertEquals "a: frog" "$X"
+}
+
+testPipeViaFileishNoParamEval() {
+  X=$(./yq e < test.yml)
+  assertEquals "a: frog" "$X"
+}
+
+testPipeViaFileishNoParamEvalAll() {
+  X=$(./yq ea < test.yml)
+  assertEquals "a: frog" "$X"
+}
+
+source ./scripts/shunit2

acceptance_tests/pretty-print.sh

@@ -0,0 +1,176 @@
+#!/bin/bash
+
+setUp() {
+  rm test*.yml || true
+}
+
+testPrettyPrintWithBooleans() {
+  cat >test.yml <<EOL
+leaveUnquoted: [yes, no, on, off, y, n, true, false]
+leaveQuoted: ["yes", "no", "on", "off", "y", "n", "true", "false"]
+
+EOL
+
+  read -r -d '' expected << EOM
+leaveUnquoted:
+  - yes
+  - no
+  - on
+  - off
+  - y
+  - n
+  - true
+  - false
+leaveQuoted:
+  - "yes"
+  - "no"
+  - "on"
+  - "off"
+  - "y"
+  - "n"
+  - "true"
+  - "false"
+EOM
+
+  X=$(./yq e --prettyPrint test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea --prettyPrint test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testPrettyPrintWithBooleansCapitals() {
+  cat >test.yml <<EOL
+leaveUnquoted: [YES, NO, ON, OFF, Y, N, TRUE, FALSE]
+leaveQuoted: ["YES", "NO", "ON", "OFF", "Y", "N", "TRUE", "FALSE"]
+EOL
+
+  read -r -d '' expected << EOM
+leaveUnquoted:
+  - YES
+  - NO
+  - ON
+  - OFF
+  - Y
+  - N
+  - TRUE
+  - FALSE
+leaveQuoted:
+  - "YES"
+  - "NO"
+  - "ON"
+  - "OFF"
+  - "Y"
+  - "N"
+  - "TRUE"
+  - "FALSE"
+EOM
+
+  X=$(./yq e --prettyPrint test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea --prettyPrint test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testPrettyPrintOtherStringValues() {
+  cat >test.yml <<EOL
+leaveUnquoted: [yesSir, hellno, bonapite]
+makeUnquoted: ["yesSir", "hellno", "bonapite"]
+EOL
+
+  read -r -d '' expected << EOM
+leaveUnquoted:
+  - yesSir
+  - hellno
+  - bonapite
+makeUnquoted:
+  - yesSir
+  - hellno
+  - bonapite
+EOM
+
+  X=$(./yq e --prettyPrint test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea --prettyPrint test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testPrettyPrintKeys() {
+  cat >test.yml <<EOL
+"removeQuotes": "please"
+EOL
+
+  read -r -d '' expected << EOM
+removeQuotes: please
+EOM
+
+  X=$(./yq e --prettyPrint test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea --prettyPrint test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testPrettyPrintOtherStringValues() {
+  cat >test.yml <<EOL
+leaveUnquoted: [yesSir, hellno, bonapite]
+makeUnquoted: ["yesSir", "hellno", "bonapite"]
+EOL
+
+  read -r -d '' expected << EOM
+leaveUnquoted:
+  - yesSir
+  - hellno
+  - bonapite
+makeUnquoted:
+  - yesSir
+  - hellno
+  - bonapite
+EOM
+
+  X=$(./yq e --prettyPrint test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea --prettyPrint test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testPrettyPrintStringBlocks() {
+  cat >test.yml <<EOL
+"removeQuotes": | 
+  "please"
+EOL
+
+  read -r -d '' expected << EOM
+removeQuotes: |
+  "please"
+EOM
+
+  X=$(./yq e --prettyPrint test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea --prettyPrint test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testPrettyPrintWithExpression() {
+  cat >test.yml <<EOL
+a: {b: {c: ["cat"]}}
+EOL
+
+  read -r -d '' expected << EOM
+b:
+  c:
+    - cat
+EOM
+
+  X=$(./yq e '.a' --prettyPrint test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea '.a' --prettyPrint test.yml)
+  assertEquals "$expected" "$X"
+}
+
+source ./scripts/shunit2

acceptance_tests/shebang.sh

@@ -0,0 +1,26 @@
+#!/bin/bash
+
+setUp() {
+  rm test*.yq || true
+  cat >test.yq <<EOL
+#!./yq
+.a.b
+EOL
+chmod +x test.yq
+
+rm test*.yml || true
+  cat >test.yml <<EOL
+a: {b: apple}
+EOL
+}
+
+testCanExecYqFile() {
+  read -r -d '' expected << EOM
+apple
+EOM
+   X=$(./test.yq test.yml)
+  assertEquals "$expected" "$X"
+}
+
+source ./scripts/shunit2
+

acceptance_tests/split-printer.sh

@@ -0,0 +1,227 @@
+#!/bin/bash
+
+setUp() {
+  rm test*.yml || true
+  rm -rf test_dir* || true
+}
+
+testBasicSplitWithName() {
+  cat >test.yml <<EOL
+a: test_doc1
+--- 
+a: test_doc2
+EOL
+
+  ./yq e test.yml -s ".a"
+
+  doc1=$(cat test_doc1.yml)
+  
+  assertEquals "a: test_doc1" "$doc1"
+
+  doc2=$(cat test_doc2.yml)
+  read -r -d '' expectedDoc2 << EOM
+---
+a: test_doc2
+EOM
+  assertEquals "$expectedDoc2" "$doc2"
+}
+
+testBasicSplitWithNameCustomExtension() {
+  rm test*.yaml || true
+  cat >test.yml <<EOL
+a: test_doc1
+--- 
+a: test_doc2
+EOL
+
+  ./yq e test.yml -s '.a + ".yaml"'
+
+  doc1=$(cat test_doc1.yaml)
+  
+  assertEquals "a: test_doc1" "$doc1"
+
+  doc2=$(cat test_doc2.yaml)
+  read -r -d '' expectedDoc2 << EOM
+---
+a: test_doc2
+EOM
+  assertEquals "$expectedDoc2" "$doc2"
+}
+
+
+
+
+testSplitFromFile() {
+  cat >test.yml <<EOL
+a: test_doc1
+--- 
+a: test_doc2
+EOL
+
+cat >test_splitExp.yml <<EOL
+.a
+EOL
+
+  ./yq test.yml --split-exp-file test_splitExp.yml
+
+  doc1=$(cat test_doc1.yml)
+  
+  assertEquals "a: test_doc1" "$doc1"
+
+  doc2=$(cat test_doc2.yml)
+  read -r -d '' expectedDoc2 << EOM
+---
+a: test_doc2
+EOM
+  assertEquals "$expectedDoc2" "$doc2"
+}
+
+testBasicSplitWithNameEvalAll() {
+  cat >test.yml <<EOL
+a: test_doc1
+--- 
+a: test_doc2
+EOL
+
+  ./yq ea test.yml -s ".a"
+
+  doc1=$(cat test_doc1.yml)
+  
+  assertEquals "a: test_doc1" "$doc1"
+
+  doc2=$(cat test_doc2.yml)
+  read -r -d '' expectedDoc2 << EOM
+---
+a: test_doc2
+EOM
+  assertEquals "$expectedDoc2" "$doc2"
+}
+
+testBasicSplitWithIndex() {
+  cat >test.yml <<EOL
+a: test_doc1
+--- 
+a: test_doc2
+EOL
+
+  ./yq e test.yml -s '"test_" + $index'
+
+  doc1=$(cat test_0.yml)
+  
+  assertEquals "a: test_doc1" "$doc1"
+
+  doc2=$(cat test_1.yml)
+  read -r -d '' expectedDoc2 << EOM
+---
+a: test_doc2
+EOM
+  assertEquals "$expectedDoc2" "$doc2"
+}
+
+testBasicSplitWithIndexEvalAll() {
+  cat >test.yml <<EOL
+a: test_doc1
+--- 
+a: test_doc2
+EOL
+
+  ./yq ea test.yml -s '"test_" + $index'
+
+  doc1=$(cat test_0.yml)
+  
+  assertEquals "a: test_doc1" "$doc1"
+
+  doc2=$(cat test_1.yml)
+  read -r -d '' expectedDoc2 << EOM
+---
+a: test_doc2
+EOM
+  assertEquals "$expectedDoc2" "$doc2"
+}
+
+
+testArraySplitWithNameNoSeparators() {
+  cat >test.yml <<EOL
+- name: test_fred
+  age: 35
+- name: test_catherine
+  age: 37
+EOL
+
+  ./yq e --no-doc -s ".name"  ".[]" test.yml 
+
+  doc1=$(cat test_fred.yml)
+  read -r -d '' expectedDoc1 << EOM
+name: test_fred
+age: 35
+EOM
+
+  assertEquals "$expectedDoc1" "$doc1"
+
+  doc2=$(cat test_catherine.yml)
+  read -r -d '' expectedDoc2 << EOM
+name: test_catherine
+age: 37
+EOM
+  assertEquals "$expectedDoc2" "$doc2"
+}
+
+testArraySplitWithNameNoSeparatorsEvalAll() {
+  cat >test.yml <<EOL
+- name: test_fred
+  age: 35
+- name: test_catherine
+  age: 37
+EOL
+
+cat >test2.yml <<EOL
+- name: test_mike
+  age: 564
+EOL
+
+  ./yq ea --no-doc -s ".name"  ".[]" test.yml test2.yml
+
+  doc1=$(cat test_fred.yml)
+  read -r -d '' expectedDoc1 << EOM
+name: test_fred
+age: 35
+EOM
+
+  assertEquals "$expectedDoc1" "$doc1"
+
+  doc2=$(cat test_catherine.yml)
+  read -r -d '' expectedDoc2 << EOM
+name: test_catherine
+age: 37
+EOM
+  assertEquals "$expectedDoc2" "$doc2"
+
+
+  doc3=$(cat test_mike.yml)
+  read -r -d '' expectedDoc3 << EOM
+name: test_mike
+age: 564
+EOM
+  assertEquals "$expectedDoc3" "$doc3"
+}
+
+testSplitWithDirectories() {
+  cat >test.yml <<EOL
+f: test_dir1/test_file1
+---
+f: test_dir2/dir22/test_file2
+---
+f: test_file3
+EOL
+
+  ./yq e --no-doc -s ".f" test.yml
+
+  doc1=$(cat test_dir1/test_file1.yml)
+  assertEquals "f: test_dir1/test_file1" "$doc1"
+  doc2=$(cat test_dir2/dir22/test_file2.yml)
+  assertEquals "f: test_dir2/dir22/test_file2" "$doc2"
+  doc3=$(cat test_file3.yml)
+  assertEquals "f: test_file3" "$doc3"
+}
+
+source ./scripts/shunit2

action.yml

@@ -1,13 +1,17 @@
 name: 'yq - portable yaml processor'
 description: 'create, read, update, delete, merge, validate and do more with yaml'
-icon: command
-color: gray-dark
+branding:
+  icon: command
+  color: gray-dark
 inputs:
   cmd:
     description: 'The Command which should be run'
     required: true
+outputs:
+  result:
+    description: "The complete result from the yq command being run"
 runs:
   using: 'docker'
-  image: 'github-action/Dockerfile'
+  image: 'docker://mikefarah/yq:4-githubaction'
   args:
     - ${{ inputs.cmd }}

cmd/commands_test.go

@@ -1,83 +0,0 @@
-package cmd
-
-import (
-	"strings"
-	"testing"
-
-	"github.com/mikefarah/yq/v3/test"
-	"github.com/spf13/cobra"
-)
-
-func getRootCommand() *cobra.Command {
-	return New()
-}
-
-func TestRootCmd(t *testing.T) {
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "")
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-
-	if !strings.Contains(result.Output, "Usage:") {
-		t.Error("Expected usage message to be printed out, but the usage message was not found.")
-	}
-}
-
-func TestRootCmd_Help(t *testing.T) {
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "--help")
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-
-	if !strings.Contains(result.Output, "yq is a lightweight and portable command-line YAML processor. It aims to be the jq or sed of yaml files.") {
-		t.Error("Expected usage message to be printed out, but the usage message was not found.")
-	}
-}
-
-func TestRootCmd_VerboseLong(t *testing.T) {
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "--verbose")
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-
-	if !verbose {
-		t.Error("Expected verbose to be true")
-	}
-}
-
-func TestRootCmd_VerboseShort(t *testing.T) {
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "-v")
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-
-	if !verbose {
-		t.Error("Expected verbose to be true")
-	}
-}
-
-func TestRootCmd_VersionShort(t *testing.T) {
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "-V")
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	if !strings.Contains(result.Output, "yq version") {
-		t.Error("expected version message to be printed out, but the message was not found.")
-	}
-}
-
-func TestRootCmd_VersionLong(t *testing.T) {
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "--version")
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	if !strings.Contains(result.Output, "yq version") {
-		t.Error("expected version message to be printed out, but the message was not found.")
-	}
-}

cmd/compare.go

@@ -1,89 +0,0 @@
-package cmd
-
-import (
-	"bufio"
-	"bytes"
-	"os"
-	"strings"
-
-	"github.com/kylelemons/godebug/diff"
-	"github.com/mikefarah/yq/v3/pkg/yqlib"
-	errors "github.com/pkg/errors"
-	"github.com/spf13/cobra"
-)
-
-// turn off for unit tests :(
-var forceOsExit = true
-
-func createCompareCmd() *cobra.Command {
-	var cmdCompare = &cobra.Command{
-		Use:     "compare [yaml_file_a] [yaml_file_b]",
-		Aliases: []string{"x"},
-		Short:   "yq x [--prettyPrint/-P] dataA.yaml dataB.yaml 'b.e(name==fr*).value'",
-		Example: `
-yq x - data2.yml # reads from stdin
-yq x -pp dataA.yaml dataB.yaml '**' # compare paths
-yq x -d1 dataA.yaml dataB.yaml 'a.b.c'
-`,
-		Long: "Deeply compares two yaml files, prints the difference. Use with prettyPrint flag to ignore formatting differences.",
-		RunE: compareDocuments,
-	}
-	cmdCompare.PersistentFlags().StringVarP(&docIndex, "doc", "d", "0", "process document index number (0 based, * for all documents)")
-	cmdCompare.PersistentFlags().StringVarP(&printMode, "printMode", "p", "v", "print mode (v (values, default), p (paths), pv (path and value pairs)")
-	cmdCompare.PersistentFlags().StringVarP(&defaultValue, "defaultValue", "D", "", "default value printed when there are no results")
-	cmdCompare.PersistentFlags().BoolVarP(&stripComments, "stripComments", "", false, "strip comments out before comparing")
-	cmdCompare.PersistentFlags().BoolVarP(&explodeAnchors, "explodeAnchors", "X", false, "explode anchors")
-	return cmdCompare
-}
-
-func compareDocuments(cmd *cobra.Command, args []string) error {
-	var path = ""
-
-	if len(args) < 2 {
-		return errors.New("Must provide at 2 yaml files")
-	} else if len(args) > 2 {
-		path = args[2]
-	}
-
-	var updateAll, docIndexInt, errorParsingDocIndex = parseDocumentIndex()
-	if errorParsingDocIndex != nil {
-		return errorParsingDocIndex
-	}
-
-	var matchingNodesA []*yqlib.NodeContext
-	var matchingNodesB []*yqlib.NodeContext
-	var errorDoingThings error
-
-	matchingNodesA, errorDoingThings = readYamlFile(args[0], path, updateAll, docIndexInt)
-
-	if errorDoingThings != nil {
-		return errorDoingThings
-	}
-
-	matchingNodesB, errorDoingThings = readYamlFile(args[1], path, updateAll, docIndexInt)
-	if errorDoingThings != nil {
-		return errorDoingThings
-	}
-
-	var dataBufferA bytes.Buffer
-	var dataBufferB bytes.Buffer
-	errorDoingThings = printResults(matchingNodesA, bufio.NewWriter(&dataBufferA))
-	if errorDoingThings != nil {
-		return errorDoingThings
-	}
-	errorDoingThings = printResults(matchingNodesB, bufio.NewWriter(&dataBufferB))
-	if errorDoingThings != nil {
-		return errorDoingThings
-	}
-
-	diffString := diff.Diff(strings.TrimSuffix(dataBufferA.String(), "\n"), strings.TrimSuffix(dataBufferB.String(), "\n"))
-
-	if len(diffString) > 1 {
-		cmd.Print(diffString)
-		cmd.Print("\n")
-		if forceOsExit {
-			os.Exit(1)
-		}
-	}
-	return nil
-}

cmd/compare_test.go

@@ -1,115 +0,0 @@
-package cmd
-
-import (
-	"testing"
-
-	"github.com/mikefarah/yq/v3/test"
-)
-
-func TestCompareSameCmd(t *testing.T) {
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "compare ../examples/data1.yaml ../examples/data1.yaml")
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := ``
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestCompareIgnoreCommentsCmd(t *testing.T) {
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "compare --stripComments ../examples/data1.yaml ../examples/data1-no-comments.yaml")
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := ``
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestCompareDontIgnoreCommentsCmd(t *testing.T) {
-	forceOsExit = false
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "compare ../examples/data1.yaml ../examples/data1-no-comments.yaml")
-
-	expectedOutput := `-a: simple # just the best
-+a: simple
- b: [1, 2]
- c:
-   test: 1
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestCompareExplodeAnchorsCommentsCmd(t *testing.T) {
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "compare --explodeAnchors ../examples/simple-anchor.yaml ../examples/simple-anchor-exploded.yaml")
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := ``
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestCompareDontExplodeAnchorsCmd(t *testing.T) {
-	forceOsExit = false
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "compare ../examples/simple-anchor.yaml ../examples/simple-anchor-exploded.yaml")
-
-	expectedOutput := `-foo: &foo
-+foo:
-   a: 1
- foobar:
--  !!merge <<: *foo
-+  a: 1
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestCompareDifferentCmd(t *testing.T) {
-	forceOsExit = false
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "compare ../examples/data1.yaml ../examples/data3.yaml")
-
-	expectedOutput := `-a: simple # just the best
--b: [1, 2]
-+a: "simple" # just the best
-+b: [1, 3]
- c:
-   test: 1
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestComparePrettyCmd(t *testing.T) {
-	forceOsExit = false
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "compare -P ../examples/data1.yaml ../examples/data3.yaml")
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := ` a: simple # just the best
- b:
-   - 1
--  - 2
-+  - 3
- c:
-   test: 1
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestComparePathsCmd(t *testing.T) {
-	forceOsExit = false
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "compare -P -ppv ../examples/data1.yaml ../examples/data3.yaml **")
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := ` a: simple # just the best
- b.[0]: 1
--b.[1]: 2
-+b.[1]: 3
- c.test: 1
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}

cmd/completion.go

@@ -0,0 +1,62 @@
+package cmd
+
+import (
+	"os"
+
+	"github.com/spf13/cobra"
+)
+
+var completionCmd = &cobra.Command{
+	Use:     "completion [bash|zsh|fish|powershell]",
+	Aliases: []string{"shell-completion"},
+	Short:   "Generate the autocompletion script for the specified shell",
+	Long: `To load completions:
+
+Bash:
+
+$ source <(yq completion bash)
+
+# To load completions for each session, execute once:
+Linux:
+  $ yq completion bash > /etc/bash_completion.d/yq
+MacOS:
+  $ yq completion bash > /usr/local/etc/bash_completion.d/yq
+
+Zsh:
+
+# If shell completion is not already enabled in your environment you will need
+# to enable it.  You can execute the following once:
+
+$ echo "autoload -U compinit; compinit" >> ~/.zshrc
+
+# To load completions for each session, execute once:
+$ yq completion zsh > "${fpath[1]}/_yq"
+
+# You will need to start a new shell for this setup to take effect.
+
+Fish:
+
+$ yq completion fish | source
+
+# To load completions for each session, execute once:
+$ yq completion fish > ~/.config/fish/completions/yq.fish
+`,
+	DisableFlagsInUseLine: true,
+	ValidArgs:             []string{"bash", "zsh", "fish", "powershell"},
+	Args:                  cobra.MatchAll(cobra.ExactArgs(1), cobra.OnlyValidArgs),
+	RunE: func(cmd *cobra.Command, args []string) error {
+		var err error = nil
+		switch args[0] {
+		case "bash":
+			err = cmd.Root().GenBashCompletionV2(os.Stdout, true)
+		case "zsh":
+			err = cmd.Root().GenZshCompletion(os.Stdout)
+		case "fish":
+			err = cmd.Root().GenFishCompletion(os.Stdout, true)
+		case "powershell":
+			err = cmd.Root().GenPowerShellCompletion(os.Stdout)
+		}
+		return err
+
+	},
+}

cmd/constant.go

@@ -1,35 +1,39 @@
 package cmd
 
-import (
-	"github.com/mikefarah/yq/v3/pkg/yqlib"
-	logging "gopkg.in/op/go-logging.v1"
-)
+var unwrapScalarFlag = newUnwrapFlag()
+
+var printNodeInfo = false
+
+var unwrapScalar = false
 
-var customTag = ""
-var printMode = "v"
-var printLength = false
-var unwrapScalar = true
-var customStyle = ""
-var anchorName = ""
-var makeAlias = false
-var stripComments = false
-var collectIntoArray = false
 var writeInplace = false
-var writeScript = ""
-var sourceYamlFile = ""
 var outputToJSON = false
+
+var outputFormat = ""
+
+var inputFormat = ""
+
 var exitStatus = false
-var prettyPrint = false
-var explodeAnchors = false
-var colorsEnabled = false
-var defaultValue = ""
 var indent = 2
-var overwriteFlag = false
-var autoCreateFlag = true
-var appendFlag = false
+var noDocSeparators = false
+var nullInput = false
+var nulSepOutput = false
 var verbose = false
 var version = false
-var docIndex = "0"
-var log = logging.MustGetLogger("yq")
-var lib = yqlib.NewYqLib()
-var valueParser = yqlib.NewValueParser()
+var prettyPrint = false
+
+var forceColor = false
+var forceNoColor = false
+var colorsEnabled = false
+
+// can be either "" (off), "extract" or "process"
+var frontMatter = ""
+
+var splitFileExp = ""
+var splitFileExpFile = ""
+
+var completedSuccessfully = false
+
+var forceExpression = ""
+
+var expressionFile = ""

cmd/delete.go

@@ -1,41 +0,0 @@
-package cmd
-
-import (
-	"github.com/mikefarah/yq/v3/pkg/yqlib"
-	errors "github.com/pkg/errors"
-	"github.com/spf13/cobra"
-)
-
-func createDeleteCmd() *cobra.Command {
-	var cmdDelete = &cobra.Command{
-		Use:     "delete [yaml_file] [path_expression]",
-		Aliases: []string{"d"},
-		Short:   "yq d [--inplace/-i] [--doc/-d index] sample.yaml 'b.e(name==fred)'",
-		Example: `
-yq delete things.yaml 'a.b.c'
-yq delete things.yaml 'a.*.c'
-yq delete things.yaml 'a.(child.subchild==co*).c'
-yq delete things.yaml 'a.**'
-yq delete --inplace things.yaml 'a.b.c'
-yq delete --inplace -- things.yaml '--key-starting-with-dash' # need to use '--' to stop processing arguments as flags
-yq d -i things.yaml 'a.b.c'
-	`,
-		Long: `Deletes the nodes matching the given path expression from the YAML file.
-Outputs to STDOUT unless the inplace flag is used, in which case the file is updated instead.
-`,
-		RunE: deleteProperty,
-	}
-	cmdDelete.PersistentFlags().BoolVarP(&writeInplace, "inplace", "i", false, "update the yaml file inplace")
-	cmdDelete.PersistentFlags().StringVarP(&docIndex, "doc", "d", "0", "process document index number (0 based, * for all documents)")
-	return cmdDelete
-}
-
-func deleteProperty(cmd *cobra.Command, args []string) error {
-	if len(args) < 2 {
-		return errors.New("Must provide <filename> <path_to_delete>")
-	}
-	var updateCommands []yqlib.UpdateCommand = make([]yqlib.UpdateCommand, 1)
-	updateCommands[0] = yqlib.UpdateCommand{Command: "delete", Path: args[1]}
-
-	return updateDoc(args[0], updateCommands, cmd.OutOrStdout())
-}

cmd/delete_test.go

@@ -1,246 +0,0 @@
-package cmd
-
-import (
-	"fmt"
-	"strings"
-	"testing"
-
-	"github.com/mikefarah/yq/v3/test"
-)
-
-func TestDeleteYamlCmd(t *testing.T) {
-	content := `a: 2
-b:
-  c: things
-  d: something else
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("delete %s b.c", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-
-	expectedOutput := `a: 2
-b:
-  d: something else
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestDeleteDeepDoesNotExistCmd(t *testing.T) {
-	content := `a: 2`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("delete %s b.c", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-
-	expectedOutput := `a: 2
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestDeleteSplatYaml(t *testing.T) {
-	content := `a: other
-b: [3, 4]
-c:
-  toast: leave
-  test: 1
-  tell: 1
-  tasty.taco: cool
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("delete %s c.te*", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-
-	expectedOutput := `a: other
-b: [3, 4]
-c:
-  toast: leave
-  tasty.taco: cool
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestDeleteSplatArrayYaml(t *testing.T) {
-	content := `a: 2
-b:
- hi:
-  - thing: item1
-    name: fred
-  - thing: item2
-    name: sam
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("delete %s b.hi[*].thing", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-
-	expectedOutput := `a: 2
-b:
-  hi:
-    - name: fred
-    - name: sam
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestDeleteDeepSplatArrayYaml(t *testing.T) {
-	content := `thing: 123
-b:
- hi:
-  - thing: item1
-    name: fred
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("delete %s **.thing", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-
-	expectedOutput := `b:
-  hi:
-    - name: fred
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestDeleteSplatPrefixYaml(t *testing.T) {
-	content := `a: 2
-b:
- hi:
-   c: things
-   d: something else
- there:
-   c: more things
-   d: more something else
- there2:
-   c: more things also
-   d: more something else also
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("delete %s b.there*.c", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-
-	expectedOutput := `a: 2
-b:
-  hi:
-    c: things
-    d: something else
-  there:
-    d: more something else
-  there2:
-    d: more something else also
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestDeleteYamlArrayCmd(t *testing.T) {
-	content := `- 1
-- 2
-- 3
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("delete %s [1]", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-
-	expectedOutput := `- 1
-- 3
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestDeleteYamlArrayExpressionCmd(t *testing.T) {
-	content := `- name: fred
-- name: cat
-- name: thing
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("delete %s (name==cat)", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-
-	expectedOutput := `- name: fred
-- name: thing
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestDeleteYamlMulti(t *testing.T) {
-	content := `apples: great
----
-- 1
-- 2
-- 3
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("delete -d 1 %s [1]", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-
-	expectedOutput := `apples: great
----
-- 1
-- 3
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestDeleteYamlMultiAllCmd(t *testing.T) {
-	content := `b:
-  c: 3
-apples: great
----
-apples: great
-something: else
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("delete %s -d * apples", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `b:
-  c: 3
----
-something: else`
-	test.AssertResult(t, expectedOutput, strings.Trim(result.Output, "\n "))
-}

cmd/evaluate_all_command.go

@@ -0,0 +1,140 @@
+package cmd
+
+import (
+	"errors"
+
+	"github.com/mikefarah/yq/v4/pkg/yqlib"
+	"github.com/spf13/cobra"
+)
+
+func createEvaluateAllCommand() *cobra.Command {
+	var cmdEvalAll = &cobra.Command{
+		Use:     "eval-all [expression] [yaml_file1]...",
+		Aliases: []string{"ea"},
+		Short:   "Loads _all_ yaml documents of _all_ yaml files and runs expression once",
+		ValidArgsFunction: func(_ *cobra.Command, args []string, _ string) ([]string, cobra.ShellCompDirective) {
+			if len(args) == 0 {
+				return nil, cobra.ShellCompDirectiveNoFileComp
+			}
+			return nil, cobra.ShellCompDirectiveDefault
+		},
+		Example: `
+# Merge f2.yml into f1.yml (in place)
+yq eval-all --inplace 'select(fileIndex == 0) * select(fileIndex == 1)' f1.yml f2.yml
+## the same command and expression using shortened names:
+yq ea -i 'select(fi == 0) * select(fi == 1)' f1.yml f2.yml
+
+
+# Merge all given files
+yq ea '. as $item ireduce ({}; . * $item )' file1.yml file2.yml ...
+
+# Pipe from STDIN
+## use '-' as a filename to pipe from STDIN
+cat file2.yml | yq ea '.a.b' file1.yml - file3.yml
+`,
+		Long: `yq is a portable command-line data file processor (https://github.com/mikefarah/yq/) 
+See https://mikefarah.gitbook.io/yq/ for detailed documentation and examples.
+
+## Evaluate All ##
+This command loads _all_ yaml documents of _all_ yaml files and runs expression once
+Useful when you need to run an expression across several yaml documents or files (like merge).
+Note that it consumes more memory than eval.
+`,
+		RunE: evaluateAll,
+	}
+	return cmdEvalAll
+}
+func evaluateAll(cmd *cobra.Command, args []string) (cmdError error) {
+	// 0 args, read std in
+	// 1 arg, null input, process expression
+	// 1 arg, read file in sequence
+	// 2+ args, [0] = expression, file the rest
+
+	var err error
+
+	expression, args, err := initCommand(cmd, args)
+	if err != nil {
+		return err
+	}
+
+	out := cmd.OutOrStdout()
+
+	if writeInplace {
+		// only use colors if its forced
+		colorsEnabled = forceColor
+		writeInPlaceHandler := yqlib.NewWriteInPlaceHandler(args[0])
+		out, err = writeInPlaceHandler.CreateTempFile()
+		if err != nil {
+			return err
+		}
+		// need to indirectly call the function so  that completedSuccessfully is
+		// passed when we finish execution as opposed to now
+		defer func() {
+			if cmdError == nil {
+				cmdError = writeInPlaceHandler.FinishWriteInPlace(completedSuccessfully)
+			}
+		}()
+	}
+
+	format, err := yqlib.FormatFromString(outputFormat)
+	if err != nil {
+		return err
+	}
+
+	decoder, err := configureDecoder(true)
+	if err != nil {
+		return err
+	}
+
+	printerWriter, err := configurePrinterWriter(format, out)
+	if err != nil {
+		return err
+	}
+	encoder, err := configureEncoder()
+	if err != nil {
+		return err
+	}
+
+	printer := yqlib.NewPrinter(encoder, printerWriter)
+	if nulSepOutput {
+		printer.SetNulSepOutput(true)
+	}
+
+	if frontMatter != "" {
+		frontMatterHandler := yqlib.NewFrontMatterHandler(args[0])
+		err = frontMatterHandler.Split()
+		if err != nil {
+			return err
+		}
+		args[0] = frontMatterHandler.GetYamlFrontMatterFilename()
+
+		if frontMatter == "process" {
+			reader := frontMatterHandler.GetContentReader()
+			printer.SetAppendix(reader)
+			defer yqlib.SafelyCloseReader(reader)
+		}
+		defer frontMatterHandler.CleanUp()
+	}
+
+	allAtOnceEvaluator := yqlib.NewAllAtOnceEvaluator()
+
+	switch len(args) {
+	case 0:
+		if nullInput {
+			err = yqlib.NewStreamEvaluator().EvaluateNew(processExpression(expression), printer)
+		} else {
+			cmd.Println(cmd.UsageString())
+			return nil
+		}
+	default:
+		err = allAtOnceEvaluator.EvaluateFiles(processExpression(expression), args, printer, decoder)
+	}
+
+	completedSuccessfully = err == nil
+
+	if err == nil && exitStatus && !printer.PrintedAnything() {
+		return errors.New("no matches found")
+	}
+
+	return err
+}

cmd/evaluate_sequence_command.go

@@ -0,0 +1,158 @@
+package cmd
+
+import (
+	"errors"
+	"fmt"
+
+	"github.com/mikefarah/yq/v4/pkg/yqlib"
+	"github.com/spf13/cobra"
+)
+
+func createEvaluateSequenceCommand() *cobra.Command {
+	var cmdEvalSequence = &cobra.Command{
+		Use:     "eval [expression] [yaml_file1]...",
+		Aliases: []string{"e"},
+		Short:   "(default) Apply the expression to each document in each yaml file in sequence",
+		ValidArgsFunction: func(_ *cobra.Command, args []string, _ string) ([]string, cobra.ShellCompDirective) {
+			if len(args) == 0 {
+				return nil, cobra.ShellCompDirectiveNoFileComp
+			}
+			return nil, cobra.ShellCompDirectiveDefault
+		},
+		Example: `
+# Reads field under the given path for each file
+yq e '.a.b' f1.yml f2.yml 
+
+# Prints out the file
+yq e sample.yaml 
+
+# Pipe from STDIN
+## use '-' as a filename to pipe from STDIN
+cat file2.yml | yq e '.a.b' file1.yml - file3.yml
+
+# Creates a new yaml document
+## Note that editing an empty file does not work.
+yq e -n '.a.b.c = "cat"' 
+
+# Update a file in place
+yq e '.a.b = "cool"' -i file.yaml 
+`,
+		Long: `yq is a portable command-line data file processor (https://github.com/mikefarah/yq/) 
+See https://mikefarah.gitbook.io/yq/ for detailed documentation and examples.
+
+## Evaluate Sequence ##
+This command iterates over each yaml document from each given file, applies the 
+expression and prints the result in sequence.`,
+		RunE: evaluateSequence,
+	}
+	return cmdEvalSequence
+}
+
+func processExpression(expression string) string {
+
+	if prettyPrint && expression == "" {
+		return yqlib.PrettyPrintExp
+	} else if prettyPrint {
+		return fmt.Sprintf("%v | %v", expression, yqlib.PrettyPrintExp)
+	}
+	return expression
+}
+
+func evaluateSequence(cmd *cobra.Command, args []string) (cmdError error) {
+	// 0 args, read std in
+	// 1 arg, null input, process expression
+	// 1 arg, read file in sequence
+	// 2+ args, [0] = expression, file the rest
+
+	out := cmd.OutOrStdout()
+
+	var err error
+
+	expression, args, err := initCommand(cmd, args)
+	if err != nil {
+		return err
+	}
+
+	if writeInplace {
+		// only use colors if its forced
+		colorsEnabled = forceColor
+		writeInPlaceHandler := yqlib.NewWriteInPlaceHandler(args[0])
+		out, err = writeInPlaceHandler.CreateTempFile()
+		if err != nil {
+			return err
+		}
+		// need to indirectly call the function so  that completedSuccessfully is
+		// passed when we finish execution as opposed to now
+		defer func() {
+			if cmdError == nil {
+				cmdError = writeInPlaceHandler.FinishWriteInPlace(completedSuccessfully)
+			}
+		}()
+	}
+
+	format, err := yqlib.FormatFromString(outputFormat)
+	if err != nil {
+		return err
+	}
+
+	printerWriter, err := configurePrinterWriter(format, out)
+	if err != nil {
+		return err
+	}
+	encoder, err := configureEncoder()
+	if err != nil {
+		return err
+	}
+
+	printer := yqlib.NewPrinter(encoder, printerWriter)
+
+	if printNodeInfo {
+		printer = yqlib.NewNodeInfoPrinter(printerWriter)
+	}
+
+	if nulSepOutput {
+		printer.SetNulSepOutput(true)
+	}
+
+	decoder, err := configureDecoder(false)
+	if err != nil {
+		return err
+	}
+	streamEvaluator := yqlib.NewStreamEvaluator()
+
+	if frontMatter != "" {
+		yqlib.GetLogger().Debug("using front matter handler")
+		frontMatterHandler := yqlib.NewFrontMatterHandler(args[0])
+		err = frontMatterHandler.Split()
+		if err != nil {
+			return err
+		}
+		args[0] = frontMatterHandler.GetYamlFrontMatterFilename()
+
+		if frontMatter == "process" {
+			reader := frontMatterHandler.GetContentReader()
+			printer.SetAppendix(reader)
+			defer yqlib.SafelyCloseReader(reader)
+		}
+		defer frontMatterHandler.CleanUp()
+	}
+
+	switch len(args) {
+	case 0:
+		if nullInput {
+			err = streamEvaluator.EvaluateNew(processExpression(expression), printer)
+		} else {
+			cmd.Println(cmd.UsageString())
+			return nil
+		}
+	default:
+		err = streamEvaluator.EvaluateFiles(processExpression(expression), args, printer, decoder)
+	}
+	completedSuccessfully = err == nil
+
+	if err == nil && exitStatus && !printer.PrintedAnything() {
+		return errors.New("no matches found")
+	}
+
+	return err
+}

cmd/merge.go

@@ -1,74 +0,0 @@
-package cmd
-
-import (
-	"github.com/mikefarah/yq/v3/pkg/yqlib"
-	errors "github.com/pkg/errors"
-	"github.com/spf13/cobra"
-	yaml "gopkg.in/yaml.v3"
-)
-
-func createMergeCmd() *cobra.Command {
-	var cmdMerge = &cobra.Command{
-		Use:     "merge [initial_yaml_file] [additional_yaml_file]...",
-		Aliases: []string{"m"},
-		Short:   "yq m [--inplace/-i] [--doc/-d index] [--overwrite/-x] [--append/-a] sample.yaml sample2.yaml",
-		Example: `
-yq merge things.yaml other.yaml
-yq merge --inplace things.yaml other.yaml
-yq m -i things.yaml other.yaml
-yq m --overwrite things.yaml other.yaml
-yq m -i -x things.yaml other.yaml
-yq m -i -a things.yaml other.yaml
-yq m -i --autocreate=false things.yaml other.yaml
-      `,
-		Long: `Updates the yaml file by adding/updating the path(s) and value(s) from additional yaml file(s).
-Outputs to STDOUT unless the inplace flag is used, in which case the file is updated instead.
-
-If overwrite flag is set then existing values will be overwritten using the values from each additional yaml file.
-If append flag is set then existing arrays will be merged with the arrays from each additional yaml file.
-`,
-		RunE: mergeProperties,
-	}
-	cmdMerge.PersistentFlags().BoolVarP(&writeInplace, "inplace", "i", false, "update the yaml file inplace")
-	cmdMerge.PersistentFlags().BoolVarP(&overwriteFlag, "overwrite", "x", false, "update the yaml file by overwriting existing values")
-	cmdMerge.PersistentFlags().BoolVarP(&autoCreateFlag, "autocreate", "c", true, "automatically create any missing entries")
-	cmdMerge.PersistentFlags().BoolVarP(&appendFlag, "append", "a", false, "update the yaml file by appending array values")
-	cmdMerge.PersistentFlags().StringVarP(&docIndex, "doc", "d", "0", "process document index number (0 based, * for all documents)")
-	return cmdMerge
-}
-
-/*
-* We don't deeply traverse arrays when appending a merge, instead we want to
-* append the entire array element.
- */
-func createReadFunctionForMerge() func(*yaml.Node) ([]*yqlib.NodeContext, error) {
-	return func(dataBucket *yaml.Node) ([]*yqlib.NodeContext, error) {
-		return lib.Get(dataBucket, "**", !appendFlag)
-	}
-}
-
-func mergeProperties(cmd *cobra.Command, args []string) error {
-	var updateCommands []yqlib.UpdateCommand = make([]yqlib.UpdateCommand, 0)
-
-	if len(args) < 1 {
-		return errors.New("Must provide at least 1 yaml file")
-	}
-
-	if len(args) > 1 {
-		// first generate update commands from the file
-		var filesToMerge = args[1:]
-
-		for _, fileToMerge := range filesToMerge {
-			matchingNodes, errorProcessingFile := doReadYamlFile(fileToMerge, createReadFunctionForMerge(), false, 0)
-			if errorProcessingFile != nil {
-				return errorProcessingFile
-			}
-			for _, matchingNode := range matchingNodes {
-				mergePath := lib.MergePathStackToString(matchingNode.PathStack, appendFlag)
-				updateCommands = append(updateCommands, yqlib.UpdateCommand{Command: "update", Path: mergePath, Value: matchingNode.Node, Overwrite: overwriteFlag})
-			}
-		}
-	}
-
-	return updateDoc(args[0], updateCommands, cmd.OutOrStdout())
-}

cmd/merge_test.go

@@ -1,337 +0,0 @@
-package cmd
-
-import (
-	"fmt"
-	"os"
-	"runtime"
-	"testing"
-
-	"github.com/mikefarah/yq/v3/test"
-)
-
-func TestMergeCmd(t *testing.T) {
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "merge ../examples/data1.yaml ../examples/data2.yaml")
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `a: simple # just the best
-b: [1, 2]
-c:
-  test: 1
-  toast: leave
-  tell: 1
-  tasty.taco: cool
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestMergeOneFileCmd(t *testing.T) {
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "merge ../examples/data1.yaml")
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `a: simple # just the best
-b: [1, 2]
-c:
-  test: 1
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestMergeNoAutoCreateCmd(t *testing.T) {
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "merge -c=false ../examples/data1.yaml ../examples/data2.yaml")
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `a: simple # just the best
-b: [1, 2]
-c:
-  test: 1
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestMergeOverwriteCmd(t *testing.T) {
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "merge -c=false --overwrite ../examples/data1.yaml ../examples/data2.yaml")
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `a: other # better than the original
-b: [3, 4]
-c:
-  test: 1
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestMergeAppendCmd(t *testing.T) {
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "merge --autocreate=false --append ../examples/data1.yaml ../examples/data2.yaml")
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `a: simple # just the best
-b: [1, 2, 3, 4]
-c:
-  test: 1
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestMergeAppendArraysCmd(t *testing.T) {
-	content := `people:
-  - name: Barry
-    age: 21`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	mergeContent := `people:
-  - name: Roger
-    age: 44`
-	mergeFilename := test.WriteTempYamlFile(mergeContent)
-	defer test.RemoveTempYamlFile(mergeFilename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("merge --append -d* %s %s", filename, mergeFilename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `people:
-  - name: Barry
-    age: 21
-  - name: Roger
-    age: 44
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestMergeOverwriteAndAppendCmd(t *testing.T) {
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "merge --autocreate=false --append --overwrite ../examples/data1.yaml ../examples/data2.yaml")
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `a: other # better than the original
-b: [1, 2, 3, 4]
-c:
-  test: 1
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestMergeArraysCmd(t *testing.T) {
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "merge --append ../examples/sample_array.yaml ../examples/sample_array_2.yaml")
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `[1, 2, 3, 4, 5]
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestMergeCmd_Multi(t *testing.T) {
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "merge -d1 ../examples/multiple_docs_small.yaml ../examples/data1.yaml")
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `a: Easy! as one two three
----
-another:
-  document: here
-a: simple # just the best
-b:
-  - 1
-  - 2
-c:
-  test: 1
----
-- 1
-- 2
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestMergeYamlMultiAllCmd(t *testing.T) {
-	content := `b:
-  c: 3
-apples: green
----
-something: else`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	mergeContent := `apples: red
-something: good`
-	mergeFilename := test.WriteTempYamlFile(mergeContent)
-	defer test.RemoveTempYamlFile(mergeFilename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("merge -d* %s %s", filename, mergeFilename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `b:
-  c: 3
-apples: green
-something: good
----
-something: else
-apples: red
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestMergeSpecialCharacterKeysCmd(t *testing.T) {
-	content := ``
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	mergeContent := `key[bracket]: value
-key.bracket: value
-key"value": value
-key'value': value
-`
-	mergeFilename := test.WriteTempYamlFile(mergeContent)
-	defer test.RemoveTempYamlFile(mergeFilename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("merge %s %s", filename, mergeFilename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	test.AssertResult(t, mergeContent, result.Output)
-}
-
-func TestMergeYamlMultiAllOverwriteCmd(t *testing.T) {
-	content := `b:
-  c: 3
-apples: green
----
-something: else`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	mergeContent := `apples: red
-something: good`
-	mergeFilename := test.WriteTempYamlFile(mergeContent)
-	defer test.RemoveTempYamlFile(mergeFilename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("merge --overwrite -d* %s %s", filename, mergeFilename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `b:
-  c: 3
-apples: red
-something: good
----
-something: good
-apples: red
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestMergeYamlNullMapCmd(t *testing.T) {
-	content := `b:`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	mergeContent := `b:
-  thing: a frog
-`
-	mergeFilename := test.WriteTempYamlFile(mergeContent)
-	defer test.RemoveTempYamlFile(mergeFilename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("merge %s %s", filename, mergeFilename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	test.AssertResult(t, mergeContent, result.Output)
-}
-
-func TestMergeCmd_Error(t *testing.T) {
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "merge")
-	if result.Error == nil {
-		t.Error("Expected command to fail due to missing arg")
-	}
-	expectedOutput := `Must provide at least 1 yaml file`
-	test.AssertResult(t, expectedOutput, result.Error.Error())
-}
-
-func TestMergeCmd_ErrorUnreadableFile(t *testing.T) {
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "merge ../examples/data1.yaml fake-unknown")
-	if result.Error == nil {
-		t.Error("Expected command to fail due to unknown file")
-	}
-	var expectedOutput string
-	if runtime.GOOS == "windows" {
-		expectedOutput = `open fake-unknown: The system cannot find the file specified.`
-	} else {
-		expectedOutput = `open fake-unknown: no such file or directory`
-	}
-	test.AssertResult(t, expectedOutput, result.Error.Error())
-}
-
-func TestMergeCmd_Inplace(t *testing.T) {
-	filename := test.WriteTempYamlFile(test.ReadTempYamlFile("../examples/data1.yaml"))
-	err := os.Chmod(filename, os.FileMode(int(0666)))
-	if err != nil {
-		t.Error(err)
-	}
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("merge -i %s ../examples/data2.yaml", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	info, _ := os.Stat(filename)
-	gotOutput := test.ReadTempYamlFile(filename)
-	expectedOutput := `a: simple # just the best
-b: [1, 2]
-c:
-  test: 1
-  toast: leave
-  tell: 1
-  tasty.taco: cool
-`
-	test.AssertResult(t, expectedOutput, gotOutput)
-	test.AssertResult(t, os.FileMode(int(0666)), info.Mode())
-}
-
-func TestMergeAllowEmptyTargetCmd(t *testing.T) {
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "merge ../examples/empty.yaml ../examples/data1.yaml")
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `a: simple # just the best
-b:
-  - 1
-  - 2
-c:
-  test: 1
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestMergeAllowEmptyMergeCmd(t *testing.T) {
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "merge ../examples/data1.yaml ../examples/empty.yaml")
-	expectedOutput := `a: simple # just the best
-b: [1, 2]
-c:
-  test: 1
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}

cmd/new.go

@@ -1,60 +0,0 @@
-package cmd
-
-import (
-	"github.com/mikefarah/yq/v3/pkg/yqlib"
-	errors "github.com/pkg/errors"
-	"github.com/spf13/cobra"
-)
-
-func createNewCmd() *cobra.Command {
-	var cmdNew = &cobra.Command{
-		Use:     "new [path] [value]",
-		Aliases: []string{"n"},
-		Short:   "yq n [--script/-s script_file] a.b.c newValue",
-		Example: `
-yq new 'a.b.c' cat
-yq n 'a.b.c' --tag '!!str' true # force 'true' to be interpreted as a string instead of bool
-yq n 'a.b[+]' cat
-yq n -- '--key-starting-with-dash' cat # need to use '--' to stop processing arguments as flags
-yq n --script create_script.yaml
-      `,
-		Long: `Creates a new yaml w.r.t the given path and value.
-Outputs to STDOUT
-
-Create Scripts:
-Note that you can give a create script to perform more sophisticated yaml. This follows the same format as the update script.
-`,
-		RunE: newProperty,
-	}
-	cmdNew.PersistentFlags().StringVarP(&writeScript, "script", "s", "", "yaml script for creating yaml")
-	cmdNew.PersistentFlags().StringVarP(&customTag, "tag", "t", "", "set yaml tag (e.g. !!int)")
-	cmdNew.PersistentFlags().StringVarP(&customStyle, "style", "", "", "formatting style of the value: single, double, folded, flow, literal, tagged")
-	cmdNew.PersistentFlags().StringVarP(&anchorName, "anchorName", "", "", "anchor name")
-	cmdNew.PersistentFlags().BoolVarP(&makeAlias, "makeAlias", "", false, "create an alias using the value as the anchor name")
-	return cmdNew
-}
-
-func newProperty(cmd *cobra.Command, args []string) error {
-	var badArgsMessage = "Must provide <path_to_update> <value>"
-	if len(args) != 2 {
-		return errors.New(badArgsMessage)
-	}
-
-	var updateCommands, updateCommandsError = readUpdateCommands(args, 2, badArgsMessage)
-	if updateCommandsError != nil {
-		return updateCommandsError
-	}
-	newNode := lib.New(updateCommands[0].Path)
-
-	for _, updateCommand := range updateCommands {
-
-		errorUpdating := lib.Update(&newNode, updateCommand, true)
-
-		if errorUpdating != nil {
-			return errorUpdating
-		}
-	}
-
-	var encoder = yqlib.NewYamlEncoder(cmd.OutOrStdout(), indent, colorsEnabled)
-	return encoder.Encode(&newNode)
-}

cmd/new_test.go

@@ -1,101 +0,0 @@
-package cmd
-
-import (
-	"testing"
-
-	"github.com/mikefarah/yq/v3/test"
-)
-
-func TestNewCmd(t *testing.T) {
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "new b.c 3")
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `b:
-  c: 3
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestNewAnchorCmd(t *testing.T) {
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "new b.c 3 --anchorName=fred")
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `b:
-  c: &fred 3
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestNewAliasCmd(t *testing.T) {
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "new b.c foo --makeAlias")
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `b:
-  c: *foo
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestNewArrayCmd(t *testing.T) {
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "new b[0] 3")
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `b:
-  - 3
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestNewCmd_Error(t *testing.T) {
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "new b.c")
-	if result.Error == nil {
-		t.Error("Expected command to fail due to missing arg")
-	}
-	expectedOutput := `Must provide <path_to_update> <value>`
-	test.AssertResult(t, expectedOutput, result.Error.Error())
-}
-
-func TestNewWithTaggedStyleCmd(t *testing.T) {
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "new b.c cat --tag=!!str --style=tagged")
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `b:
-  c: !!str cat
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestNewWithDoubleQuotedStyleCmd(t *testing.T) {
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "new b.c cat --style=double")
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `b:
-  c: "cat"
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestNewWithSingleQuotedStyleCmd(t *testing.T) {
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "new b.c cat --style=single")
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `b:
-  c: 'cat'
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}

cmd/prefix.go

@@ -1,50 +0,0 @@
-package cmd
-
-import (
-	"github.com/mikefarah/yq/v3/pkg/yqlib"
-	errors "github.com/pkg/errors"
-	"github.com/spf13/cobra"
-	yaml "gopkg.in/yaml.v3"
-)
-
-func createPrefixCmd() *cobra.Command {
-	var cmdPrefix = &cobra.Command{
-		Use:     "prefix [yaml_file] [path]",
-		Aliases: []string{"p"},
-		Short:   "yq p [--inplace/-i] [--doc/-d index] sample.yaml a.b.c",
-		Example: `
-yq prefix things.yaml 'a.b.c'
-yq prefix --inplace things.yaml 'a.b.c'
-yq prefix --inplace -- things.yaml '--key-starting-with-dash' # need to use '--' to stop processing arguments as flags
-yq p -i things.yaml 'a.b.c'
-yq p --doc 2 things.yaml 'a.b.d'
-yq p -d2 things.yaml 'a.b.d'
-      `,
-		Long: `Prefixes w.r.t to the yaml file at the given path.
-Outputs to STDOUT unless the inplace flag is used, in which case the file is updated instead.
-`,
-		RunE: prefixProperty,
-	}
-	cmdPrefix.PersistentFlags().BoolVarP(&writeInplace, "inplace", "i", false, "update the yaml file inplace")
-	cmdPrefix.PersistentFlags().StringVarP(&docIndex, "doc", "d", "0", "process document index number (0 based, * for all documents)")
-	return cmdPrefix
-}
-
-func prefixProperty(cmd *cobra.Command, args []string) error {
-
-	if len(args) < 2 {
-		return errors.New("Must provide <filename> <prefixed_path>")
-	}
-	updateCommand := yqlib.UpdateCommand{Command: "update", Path: args[1]}
-	log.Debugf("args %v", args)
-
-	var updateAll, docIndexInt, errorParsingDocIndex = parseDocumentIndex()
-	if errorParsingDocIndex != nil {
-		return errorParsingDocIndex
-	}
-
-	var updateData = func(dataBucket *yaml.Node, currentIndex int) error {
-		return prefixDocument(updateAll, docIndexInt, currentIndex, dataBucket, updateCommand)
-	}
-	return readAndUpdate(cmd.OutOrStdout(), args[0], updateData)
-}

cmd/prefix_test.go

@@ -1,189 +0,0 @@
-package cmd
-
-import (
-	"fmt"
-	"runtime"
-	"strings"
-	"testing"
-
-	"github.com/mikefarah/yq/v3/test"
-)
-
-func TestPrefixCmd(t *testing.T) {
-	content := `b:
-  c: 3
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("prefix %s d", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `d:
-  b:
-    c: 3
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestPrefixCmdArray(t *testing.T) {
-	content := `b:
-  c: 3
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("prefix %s [+].d.[+]", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `- d:
-    - b:
-        c: 3
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestPrefixCmd_MultiLayer(t *testing.T) {
-	content := `b:
-  c: 3
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("prefix %s d.e.f", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `d:
-  e:
-    f:
-      b:
-        c: 3
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestPrefixMultiCmd(t *testing.T) {
-	content := `b:
-  c: 3
----
-apples: great
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("prefix %s -d 1 d", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `b:
-  c: 3
----
-d:
-  apples: great
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-func TestPrefixInvalidDocumentIndexCmd(t *testing.T) {
-	content := `b:
-  c: 3
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("prefix %s -df d", filename))
-	if result.Error == nil {
-		t.Error("Expected command to fail due to invalid path")
-	}
-	expectedOutput := `Document index f is not a integer or *: strconv.ParseInt: parsing "f": invalid syntax`
-	test.AssertResult(t, expectedOutput, result.Error.Error())
-}
-
-func TestPrefixBadDocumentIndexCmd(t *testing.T) {
-	content := `b:
-  c: 3
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("prefix %s -d 1 d", filename))
-	if result.Error == nil {
-		t.Error("Expected command to fail due to invalid path")
-	}
-	expectedOutput := `asked to process document index 1 but there are only 1 document(s)`
-	test.AssertResult(t, expectedOutput, result.Error.Error())
-}
-func TestPrefixMultiAllCmd(t *testing.T) {
-	content := `b:
-  c: 3
----
-apples: great
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("prefix %s -d * d", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `d:
-  b:
-    c: 3
----
-d:
-  apples: great`
-	test.AssertResult(t, expectedOutput, strings.Trim(result.Output, "\n "))
-}
-
-func TestPrefixCmd_Error(t *testing.T) {
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "prefix")
-	if result.Error == nil {
-		t.Error("Expected command to fail due to missing arg")
-	}
-	expectedOutput := `Must provide <filename> <prefixed_path>`
-	test.AssertResult(t, expectedOutput, result.Error.Error())
-}
-
-func TestPrefixCmd_ErrorUnreadableFile(t *testing.T) {
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "prefix fake-unknown a.b")
-	if result.Error == nil {
-		t.Error("Expected command to fail due to unknown file")
-	}
-	var expectedOutput string
-	if runtime.GOOS == "windows" {
-		expectedOutput = `open fake-unknown: The system cannot find the file specified.`
-	} else {
-		expectedOutput = `open fake-unknown: no such file or directory`
-	}
-	test.AssertResult(t, expectedOutput, result.Error.Error())
-}
-
-func TestPrefixCmd_Inplace(t *testing.T) {
-	content := `b:
-  c: 3
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("prefix -i %s d", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	gotOutput := test.ReadTempYamlFile(filename)
-	expectedOutput := `d:
-  b:
-    c: 3`
-	test.AssertResult(t, expectedOutput, strings.Trim(gotOutput, "\n "))
-}

cmd/read.go

@@ -1,66 +0,0 @@
-package cmd
-
-import (
-	errors "github.com/pkg/errors"
-	"github.com/spf13/cobra"
-)
-
-func createReadCmd() *cobra.Command {
-	var cmdRead = &cobra.Command{
-		Use:     "read [yaml_file] [path_expression]",
-		Aliases: []string{"r"},
-		Short:   "yq r [--printMode/-p pv] sample.yaml 'b.e(name==fr*).value'",
-		Example: `
-yq read things.yaml 'a.b.c'
-yq r - 'a.b.c' # reads from stdin
-yq r things.yaml 'a.*.c'
-yq r things.yaml 'a.**.c' # deep splat
-yq r things.yaml 'a.(child.subchild==co*).c'
-yq r -d1 things.yaml 'a.array[0].blah'
-yq r things.yaml 'a.array[*].blah'
-yq r -- things.yaml '--key-starting-with-dashes.blah'
-      `,
-		Long: "Outputs the value of the given path in the yaml file to STDOUT",
-		RunE: readProperty,
-	}
-	cmdRead.PersistentFlags().StringVarP(&docIndex, "doc", "d", "0", "process document index number (0 based, * for all documents)")
-	cmdRead.PersistentFlags().StringVarP(&printMode, "printMode", "p", "v", "print mode (v (values, default), p (paths), pv (path and value pairs)")
-	cmdRead.PersistentFlags().StringVarP(&defaultValue, "defaultValue", "D", "", "default value printed when there are no results")
-	cmdRead.PersistentFlags().BoolVarP(&printLength, "length", "l", false, "print length of results")
-	cmdRead.PersistentFlags().BoolVarP(&collectIntoArray, "collect", "c", false, "collect results into array")
-	cmdRead.PersistentFlags().BoolVarP(&unwrapScalar, "unwrapScalar", "", true, "unwrap scalar, print the value with no quotes, colors or comments")
-	cmdRead.PersistentFlags().BoolVarP(&stripComments, "stripComments", "", false, "print yaml without any comments")
-	cmdRead.PersistentFlags().BoolVarP(&explodeAnchors, "explodeAnchors", "X", false, "explode anchors")
-	cmdRead.PersistentFlags().BoolVarP(&exitStatus, "exitStatus", "e", false, "set exit status if no matches are found")
-	return cmdRead
-}
-
-func readProperty(cmd *cobra.Command, args []string) error {
-	var path = ""
-
-	if len(args) < 1 {
-		return errors.New("Must provide filename")
-	} else if len(args) > 1 {
-		path = args[1]
-	}
-
-	var updateAll, docIndexInt, errorParsingDocIndex = parseDocumentIndex()
-	if errorParsingDocIndex != nil {
-		return errorParsingDocIndex
-	}
-
-	matchingNodes, errorReadingStream := readYamlFile(args[0], path, updateAll, docIndexInt)
-
-	if exitStatus && len(matchingNodes) == 0 {
-		cmd.SilenceUsage = true
-		return errors.New("No matches found")
-	}
-
-	if errorReadingStream != nil {
-		cmd.SilenceUsage = true
-		return errorReadingStream
-	}
-	out := cmd.OutOrStdout()
-
-	return printResults(matchingNodes, out)
-}

cmd/root.go

@@ -1,60 +1,222 @@
 package cmd
 
 import (
+	"fmt"
 	"os"
+	"strings"
 
+	"github.com/mikefarah/yq/v4/pkg/yqlib"
 	"github.com/spf13/cobra"
 	logging "gopkg.in/op/go-logging.v1"
 )
 
+type runeValue rune
+
+func newRuneVar(p *rune) *runeValue {
+	return (*runeValue)(p)
+}
+
+func (r *runeValue) String() string {
+	return string(*r)
+}
+
+func (r *runeValue) Set(rawVal string) error {
+	val := strings.ReplaceAll(rawVal, "\\n", "\n")
+	val = strings.ReplaceAll(val, "\\t", "\t")
+	val = strings.ReplaceAll(val, "\\r", "\r")
+	val = strings.ReplaceAll(val, "\\f", "\f")
+	val = strings.ReplaceAll(val, "\\v", "\v")
+	if len(val) != 1 {
+		return fmt.Errorf("[%v] is not a valid character. Must be length 1 was %v", val, len(val))
+	}
+	*r = runeValue(rune(val[0]))
+	return nil
+}
+
+func (r *runeValue) Type() string {
+	return "char"
+}
+
 func New() *cobra.Command {
 	var rootCmd = &cobra.Command{
 		Use:   "yq",
-		Short: "yq is a lightweight and portable command-line YAML processor.",
-		Long:  `yq is a lightweight and portable command-line YAML processor. It aims to be the jq or sed of yaml files.`,
+		Short: "yq is a lightweight and portable command-line data file processor.",
+		Long: `yq is a portable command-line data file processor (https://github.com/mikefarah/yq/) 
+See https://mikefarah.gitbook.io/yq/ for detailed documentation and examples.`,
+		Example: `
+# yq tries to auto-detect the file format based off the extension, and defaults to YAML if it's unknown (or piping through STDIN)
+# Use the '-p/--input-format' flag to specify a format type.
+cat file.xml | yq -p xml
+
+# read the "stuff" node from "myfile.yml"
+yq '.stuff' < myfile.yml
+
+# update myfile.yml in place
+yq -i '.stuff = "foo"' myfile.yml
+
+# print contents of sample.json as idiomatic YAML
+yq -P -oy sample.json
+`,
+
 		RunE: func(cmd *cobra.Command, args []string) error {
 			if version {
 				cmd.Print(GetVersionDisplay())
 				return nil
 			}
-			cmd.Println(cmd.UsageString())
+			return evaluateSequence(cmd, args)
 
-			return nil
 		},
-		PersistentPreRun: func(cmd *cobra.Command, args []string) {
-			var format = logging.MustStringFormatter(
-				`%{color}%{time:15:04:05} %{shortfunc} [%{level:.4s}]%{color:reset} %{message}`,
-			)
+		PersistentPreRunE: func(cmd *cobra.Command, _ []string) error {
+			cmd.SetOut(cmd.OutOrStdout())
+			level := logging.WARNING
+			stringFormat := `[%{level}] %{color}%{time:15:04:05}%{color:reset} %{message}`
+
+			// when NO_COLOR environment variable presents and not an empty string the coloured output should be disabled;
+			// refer to no-color.org
+			forceNoColor = forceNoColor || os.Getenv("NO_COLOR") != ""
+
+			if verbose && forceNoColor {
+				level = logging.DEBUG
+				stringFormat = `[%{level:5.5s}] %{time:15:04:05} %{shortfile:-33s} %{shortfunc:-25s} %{message}`
+			} else if verbose {
+				level = logging.DEBUG
+				stringFormat = `[%{level:5.5s}] %{color}%{time:15:04:05}%{color:bold} %{shortfile:-33s} %{shortfunc:-25s}%{color:reset} %{message}`
+			} else if forceNoColor {
+				stringFormat = `[%{level}] %{time:15:04:05} %{message}`
+			}
+
+			var format = logging.MustStringFormatter(stringFormat)
 			var backend = logging.AddModuleLevel(
 				logging.NewBackendFormatter(logging.NewLogBackend(os.Stderr, "", 0), format))
 
-			if verbose {
-				backend.SetLevel(logging.DEBUG, "")
-			} else {
-				backend.SetLevel(logging.ERROR, "")
-			}
+			backend.SetLevel(level, "")
 
 			logging.SetBackend(backend)
+			yqlib.InitExpressionParser()
+
+			return nil
 		},
 	}
 
 	rootCmd.PersistentFlags().BoolVarP(&verbose, "verbose", "v", false, "verbose mode")
-	rootCmd.PersistentFlags().BoolVarP(&outputToJSON, "tojson", "j", false, "output as json. By default it prints a json document in one line, use the prettyPrint flag to print a formatted doc.")
-	rootCmd.PersistentFlags().BoolVarP(&prettyPrint, "prettyPrint", "P", false, "pretty print")
+	rootCmd.PersistentFlags().BoolVarP(&printNodeInfo, "debug-node-info", "", false, "debug node info")
+
+	rootCmd.PersistentFlags().BoolVarP(&outputToJSON, "tojson", "j", false, "(deprecated) output as json. Set indent to 0 to print json in one line.")
+	err := rootCmd.PersistentFlags().MarkDeprecated("tojson", "please use -o=json instead")
+	if err != nil {
+		panic(err)
+	}
+
+	rootCmd.PersistentFlags().StringVarP(&outputFormat, "output-format", "o", "auto", fmt.Sprintf("[auto|a|%v] output format type.", yqlib.GetAvailableOutputFormatString()))
+	var outputCompletions = []string{"auto"}
+	for _, formats := range yqlib.GetAvailableOutputFormats() {
+		outputCompletions = append(outputCompletions, formats.FormalName)
+	}
+
+	if err = rootCmd.RegisterFlagCompletionFunc("output-format", cobra.FixedCompletions(outputCompletions, cobra.ShellCompDirectiveNoFileComp)); err != nil {
+		panic(err)
+	}
+	rootCmd.PersistentFlags().StringVarP(&inputFormat, "input-format", "p", "auto", fmt.Sprintf("[auto|a|%v] parse format for input.", yqlib.GetAvailableInputFormatString()))
+
+	var inputCompletions = []string{"auto"}
+	for _, formats := range yqlib.GetAvailableInputFormats() {
+		inputCompletions = append(inputCompletions, formats.FormalName)
+	}
+
+	if err = rootCmd.RegisterFlagCompletionFunc("input-format", cobra.FixedCompletions(inputCompletions, cobra.ShellCompDirectiveNoFileComp)); err != nil {
+		panic(err)
+	}
+
+	rootCmd.PersistentFlags().StringVar(&yqlib.ConfiguredXMLPreferences.AttributePrefix, "xml-attribute-prefix", yqlib.ConfiguredXMLPreferences.AttributePrefix, "prefix for xml attributes")
+	if err = rootCmd.RegisterFlagCompletionFunc("xml-attribute-prefix", cobra.NoFileCompletions); err != nil {
+		panic(err)
+	}
+	rootCmd.PersistentFlags().StringVar(&yqlib.ConfiguredXMLPreferences.ContentName, "xml-content-name", yqlib.ConfiguredXMLPreferences.ContentName, "name for xml content (if no attribute name is present).")
+	if err = rootCmd.RegisterFlagCompletionFunc("xml-content-name", cobra.NoFileCompletions); err != nil {
+		panic(err)
+	}
+	rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredXMLPreferences.StrictMode, "xml-strict-mode", yqlib.ConfiguredXMLPreferences.StrictMode, "enables strict parsing of XML. See https://pkg.go.dev/encoding/xml for more details.")
+	rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredXMLPreferences.KeepNamespace, "xml-keep-namespace", yqlib.ConfiguredXMLPreferences.KeepNamespace, "enables keeping namespace after parsing attributes")
+	rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredXMLPreferences.UseRawToken, "xml-raw-token", yqlib.ConfiguredXMLPreferences.UseRawToken, "enables using RawToken method instead Token. Commonly disables namespace translations. See https://pkg.go.dev/encoding/xml#Decoder.RawToken for details.")
+	rootCmd.PersistentFlags().StringVar(&yqlib.ConfiguredXMLPreferences.ProcInstPrefix, "xml-proc-inst-prefix", yqlib.ConfiguredXMLPreferences.ProcInstPrefix, "prefix for xml processing instructions (e.g. <?xml version=\"1\"?>)")
+	if err = rootCmd.RegisterFlagCompletionFunc("xml-proc-inst-prefix", cobra.NoFileCompletions); err != nil {
+		panic(err)
+	}
+	rootCmd.PersistentFlags().StringVar(&yqlib.ConfiguredXMLPreferences.DirectiveName, "xml-directive-name", yqlib.ConfiguredXMLPreferences.DirectiveName, "name for xml directives (e.g. <!DOCTYPE thing cat>)")
+	if err = rootCmd.RegisterFlagCompletionFunc("xml-directive-name", cobra.NoFileCompletions); err != nil {
+		panic(err)
+	}
+	rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredXMLPreferences.SkipProcInst, "xml-skip-proc-inst", yqlib.ConfiguredXMLPreferences.SkipProcInst, "skip over process instructions (e.g. <?xml version=\"1\"?>)")
+	rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredXMLPreferences.SkipDirectives, "xml-skip-directives", yqlib.ConfiguredXMLPreferences.SkipDirectives, "skip over directives (e.g. <!DOCTYPE thing cat>)")
+
+	rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredCsvPreferences.AutoParse, "csv-auto-parse", yqlib.ConfiguredCsvPreferences.AutoParse, "parse CSV YAML/JSON values")
+	rootCmd.PersistentFlags().Var(newRuneVar(&yqlib.ConfiguredCsvPreferences.Separator), "csv-separator", "CSV Separator character")
+
+	rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredTsvPreferences.AutoParse, "tsv-auto-parse", yqlib.ConfiguredTsvPreferences.AutoParse, "parse TSV YAML/JSON values")
+
+	rootCmd.PersistentFlags().StringVar(&yqlib.ConfiguredLuaPreferences.DocPrefix, "lua-prefix", yqlib.ConfiguredLuaPreferences.DocPrefix, "prefix")
+	if err = rootCmd.RegisterFlagCompletionFunc("lua-prefix", cobra.NoFileCompletions); err != nil {
+		panic(err)
+	}
+	rootCmd.PersistentFlags().StringVar(&yqlib.ConfiguredLuaPreferences.DocSuffix, "lua-suffix", yqlib.ConfiguredLuaPreferences.DocSuffix, "suffix")
+	if err = rootCmd.RegisterFlagCompletionFunc("lua-suffix", cobra.NoFileCompletions); err != nil {
+		panic(err)
+	}
+	rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredLuaPreferences.UnquotedKeys, "lua-unquoted", yqlib.ConfiguredLuaPreferences.UnquotedKeys, "output unquoted string keys (e.g. {foo=\"bar\"})")
+	rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredLuaPreferences.Globals, "lua-globals", yqlib.ConfiguredLuaPreferences.Globals, "output keys as top-level global variables")
+
+	rootCmd.PersistentFlags().StringVar(&yqlib.ConfiguredPropertiesPreferences.KeyValueSeparator, "properties-separator", yqlib.ConfiguredPropertiesPreferences.KeyValueSeparator, "separator to use between keys and values")
+	rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredPropertiesPreferences.UseArrayBrackets, "properties-array-brackets", yqlib.ConfiguredPropertiesPreferences.UseArrayBrackets, "use [x] in array paths (e.g. for SpringBoot)")
+
+	rootCmd.PersistentFlags().BoolVar(&yqlib.StringInterpolationEnabled, "string-interpolation", yqlib.StringInterpolationEnabled, "Toggles strings interpolation of \\(exp)")
+
+	rootCmd.PersistentFlags().BoolVarP(&nullInput, "null-input", "n", false, "Don't read input, simply evaluate the expression given. Useful for creating docs from scratch.")
+	rootCmd.PersistentFlags().BoolVarP(&noDocSeparators, "no-doc", "N", false, "Don't print document separators (---)")
+
 	rootCmd.PersistentFlags().IntVarP(&indent, "indent", "I", 2, "sets indent level for output")
+	if err = rootCmd.RegisterFlagCompletionFunc("indent", cobra.NoFileCompletions); err != nil {
+		panic(err)
+	}
 	rootCmd.Flags().BoolVarP(&version, "version", "V", false, "Print version information and quit")
-	rootCmd.PersistentFlags().BoolVarP(&colorsEnabled, "colors", "C", false, "print with colors")
+	rootCmd.PersistentFlags().BoolVarP(&writeInplace, "inplace", "i", false, "update the file in place of first file given.")
+	rootCmd.PersistentFlags().VarP(unwrapScalarFlag, "unwrapScalar", "r", "unwrap scalar, print the value with no quotes, colors or comments. Defaults to true for yaml")
+	rootCmd.PersistentFlags().Lookup("unwrapScalar").NoOptDefVal = "true"
+	rootCmd.PersistentFlags().BoolVarP(&nulSepOutput, "nul-output", "0", false, "Use NUL char to separate values. If unwrap scalar is also set, fail if unwrapped scalar contains NUL char.")
+
+	rootCmd.PersistentFlags().BoolVarP(&prettyPrint, "prettyPrint", "P", false, "pretty print, shorthand for '... style = \"\"'")
+	rootCmd.PersistentFlags().BoolVarP(&exitStatus, "exit-status", "e", false, "set exit status if there are no matches or null or false is returned")
+
+	rootCmd.PersistentFlags().BoolVarP(&forceColor, "colors", "C", false, "force print with colors")
+	rootCmd.PersistentFlags().BoolVarP(&forceNoColor, "no-colors", "M", forceNoColor, "force print with no colors")
+	rootCmd.PersistentFlags().StringVarP(&frontMatter, "front-matter", "f", "", "(extract|process) first input as yaml front-matter. Extract will pull out the yaml content, process will run the expression against the yaml content, leaving the remaining data intact")
+	if err = rootCmd.RegisterFlagCompletionFunc("front-matter", cobra.FixedCompletions([]string{"extract", "process"}, cobra.ShellCompDirectiveNoFileComp)); err != nil {
+		panic(err)
+	}
+	rootCmd.PersistentFlags().StringVarP(&forceExpression, "expression", "", "", "forcibly set the expression argument. Useful when yq argument detection thinks your expression is a file.")
+	if err = rootCmd.RegisterFlagCompletionFunc("expression", cobra.NoFileCompletions); err != nil {
+		panic(err)
+	}
+	rootCmd.PersistentFlags().BoolVarP(&yqlib.ConfiguredYamlPreferences.LeadingContentPreProcessing, "header-preprocess", "", true, "Slurp any header comments and separators before processing expression.")
+	rootCmd.PersistentFlags().BoolVarP(&yqlib.ConfiguredYamlPreferences.FixMergeAnchorToSpec, "yaml-fix-merge-anchor-to-spec", "", false, "Fix merge anchor to match YAML spec. Will default to true in late 2025")
+
+	rootCmd.PersistentFlags().StringVarP(&splitFileExp, "split-exp", "s", "", "print each result (or doc) into a file named (exp). [exp] argument must return a string. You can use $index in the expression as the result counter. The necessary directories will be created.")
+	if err = rootCmd.RegisterFlagCompletionFunc("split-exp", cobra.NoFileCompletions); err != nil {
+		panic(err)
+	}
+	rootCmd.PersistentFlags().StringVarP(&splitFileExpFile, "split-exp-file", "", "", "Use a file to specify the split-exp expression.")
+	if err = rootCmd.MarkPersistentFlagFilename("split-exp-file"); err != nil {
+		panic(err)
+	}
+
+	rootCmd.PersistentFlags().StringVarP(&expressionFile, "from-file", "", "", "Load expression from specified file.")
+	if err = rootCmd.MarkPersistentFlagFilename("from-file"); err != nil {
+		panic(err)
+	}
 
 	rootCmd.AddCommand(
-		createReadCmd(),
-		createCompareCmd(),
-		createValidateCmd(),
-		createWriteCmd(),
-		createPrefixCmd(),
-		createDeleteCmd(),
-		createNewCmd(),
-		createMergeCmd(),
-		createBashCompletionCmd(rootCmd),
+		createEvaluateSequenceCommand(),
+		createEvaluateAllCommand(),
+		completionCmd,
 	)
 	return rootCmd
 }

cmd/shell_completion.go

@@ -1,57 +0,0 @@
-package cmd
-
-import (
-	"fmt"
-	"os"
-
-	"github.com/spf13/cobra"
-)
-
-var shellVariant = "bash"
-
-func createBashCompletionCmd(rootCmd *cobra.Command) *cobra.Command {
-	var completionCmd = &cobra.Command{
-		Use:   "shell-completion",
-		Short: "Generates shell completion scripts",
-		Long: `To load completion for:
-bash:
-	Run	
-	. <(yq shell-completion)
-	
-	To configure your bash shell to load completions for each session add to
-	your bashrc
-	
-	# ~/.bashrc or ~/.profile
-	. <(yq shell-completion)
-
-zsh:
-	The generated completion script should be put somewhere in your $fpath named _yq
-
-powershell:
-	Users need PowerShell version 5.0 or above, which comes with Windows 10 and 
-	can be downloaded separately for Windows 7 or 8.1. They can then write the 
-	completions to a file and source this file from their PowerShell profile, 
-	which is referenced by the $Profile environment variable.
-
-fish:
-	Save the output to a fish file and add it to your completions directory.
-
-	`,
-		RunE: func(cmd *cobra.Command, args []string) error {
-			switch shellVariant {
-			case "bash", "":
-				return rootCmd.GenBashCompletion(os.Stdout)
-			case "zsh":
-				return rootCmd.GenZshCompletion(os.Stdout)
-			case "fish":
-				return rootCmd.GenFishCompletion(os.Stdout, true)
-			case "powershell":
-				return rootCmd.GenPowerShellCompletion(os.Stdout)
-			default:
-				return fmt.Errorf("Unknown variant %v", shellVariant)
-			}
-		},
-	}
-	completionCmd.PersistentFlags().StringVarP(&shellVariant, "variation", "V", "", "shell variation: bash (default), zsh, fish, powershell")
-	return completionCmd
-}

cmd/unwrap_flag.go

@@ -0,0 +1,46 @@
+package cmd
+
+import (
+	"strconv"
+
+	"github.com/spf13/pflag"
+)
+
+type boolFlag interface {
+	pflag.Value
+	IsExplicitlySet() bool
+	IsSet() bool
+}
+
+type unwrapScalarFlagStrc struct {
+	explicitlySet bool
+	value         bool
+}
+
+func newUnwrapFlag() boolFlag {
+	return &unwrapScalarFlagStrc{value: true}
+}
+
+func (f *unwrapScalarFlagStrc) IsExplicitlySet() bool {
+	return f.explicitlySet
+}
+
+func (f *unwrapScalarFlagStrc) IsSet() bool {
+	return f.value
+}
+
+func (f *unwrapScalarFlagStrc) String() string {
+	return strconv.FormatBool(f.value)
+}
+
+func (f *unwrapScalarFlagStrc) Set(value string) error {
+
+	v, err := strconv.ParseBool(value)
+	f.value = v
+	f.explicitlySet = true
+	return err
+}
+
+func (*unwrapScalarFlagStrc) Type() string {
+	return "bool"
+}

cmd/utils.go

@@ -1,609 +1,294 @@
 package cmd
 
 import (
-	"bufio"
 	"fmt"
 	"io"
-	"io/ioutil"
 	"os"
-	"strconv"
+	"strings"
 
-	"github.com/mikefarah/yq/v3/pkg/yqlib"
-	errors "github.com/pkg/errors"
-	yaml "gopkg.in/yaml.v3"
+	"github.com/mikefarah/yq/v4/pkg/yqlib"
+	"github.com/spf13/cobra"
+	"gopkg.in/op/go-logging.v1"
 )
 
-type readDataFn func(dataBucket *yaml.Node) ([]*yqlib.NodeContext, error)
-
-func createReadFunction(path string) func(*yaml.Node) ([]*yqlib.NodeContext, error) {
-	return func(dataBucket *yaml.Node) ([]*yqlib.NodeContext, error) {
-		return lib.Get(dataBucket, path, true)
-	}
+func isAutomaticOutputFormat() bool {
+	return outputFormat == "" || outputFormat == "auto" || outputFormat == "a"
 }
 
-func readYamlFile(filename string, path string, updateAll bool, docIndexInt int) ([]*yqlib.NodeContext, error) {
-	return doReadYamlFile(filename, createReadFunction(path), updateAll, docIndexInt)
-}
+func initCommand(cmd *cobra.Command, args []string) (string, []string, error) {
+	cmd.SilenceUsage = true
 
-func doReadYamlFile(filename string, readFn readDataFn, updateAll bool, docIndexInt int) ([]*yqlib.NodeContext, error) {
-	var matchingNodes []*yqlib.NodeContext
+	setupColors()
 
-	var currentIndex = 0
-	var errorReadingStream = readStream(filename, func(decoder *yaml.Decoder) error {
-		for {
-			var dataBucket yaml.Node
-			errorReading := decoder.Decode(&dataBucket)
-
-			if errorReading == io.EOF {
-				return handleEOF(updateAll, docIndexInt, currentIndex)
-			} else if errorReading != nil {
-				return errorReading
-			}
-
-			var errorParsing error
-			matchingNodes, errorParsing = appendDocument(matchingNodes, dataBucket, readFn, updateAll, docIndexInt, currentIndex)
-			if errorParsing != nil {
-				return errorParsing
-			}
-			if !updateAll && currentIndex == docIndexInt {
-				log.Debug("all done")
-				return nil
-			}
-			currentIndex = currentIndex + 1
-		}
-	})
-	return matchingNodes, errorReadingStream
-}
-
-func handleEOF(updateAll bool, docIndexInt int, currentIndex int) error {
-	log.Debugf("done %v / %v", currentIndex, docIndexInt)
-	if !updateAll && currentIndex <= docIndexInt && docIndexInt != 0 {
-		return fmt.Errorf("Could not process document index %v as there are only %v document(s)", docIndex, currentIndex)
-	}
-	return nil
-}
-
-func appendDocument(originalMatchingNodes []*yqlib.NodeContext, dataBucket yaml.Node, readFn readDataFn, updateAll bool, docIndexInt int, currentIndex int) ([]*yqlib.NodeContext, error) {
-	log.Debugf("processing document %v - requested index %v", currentIndex, docIndexInt)
-	yqlib.DebugNode(&dataBucket)
-	if !updateAll && currentIndex != docIndexInt {
-		return originalMatchingNodes, nil
-	}
-	log.Debugf("reading in document %v", currentIndex)
-	matchingNodes, errorParsing := readFn(&dataBucket)
-	if errorParsing != nil {
-		return nil, errors.Wrapf(errorParsing, "Error reading path in document index %v", currentIndex)
-	}
-	return append(originalMatchingNodes, matchingNodes...), nil
-}
-
-func lengthOf(node *yaml.Node) int {
-	kindToCheck := node.Kind
-	if node.Kind == yaml.DocumentNode && len(node.Content) == 1 {
-		log.Debugf("length of document node, calculating length of child")
-		kindToCheck = node.Content[0].Kind
-	}
-	switch kindToCheck {
-	case yaml.ScalarNode:
-		return len(node.Value)
-	case yaml.MappingNode:
-		return len(node.Content) / 2
-	default:
-		return len(node.Content)
-	}
-}
-
-// transforms node before printing, if required
-func transformNode(node *yaml.Node) *yaml.Node {
-	if printLength {
-		return &yaml.Node{Kind: yaml.ScalarNode, Value: fmt.Sprintf("%v", lengthOf(node))}
-	}
-	return node
-}
-
-func printNode(node *yaml.Node, writer io.Writer) error {
-	var encoder yqlib.Encoder
-	if node.Kind == yaml.ScalarNode && unwrapScalar && !outputToJSON {
-		return writeString(writer, node.Value+"\n")
-	}
-	if outputToJSON {
-		encoder = yqlib.NewJsonEncoder(writer, prettyPrint, indent)
-	} else {
-		encoder = yqlib.NewYamlEncoder(writer, indent, colorsEnabled)
-	}
-	return encoder.Encode(node)
-}
-
-func removeComments(matchingNodes []*yqlib.NodeContext) {
-	for _, nodeContext := range matchingNodes {
-		removeCommentOfNode(nodeContext.Node)
-	}
-}
-
-func removeCommentOfNode(node *yaml.Node) {
-	node.HeadComment = ""
-	node.LineComment = ""
-	node.FootComment = ""
-
-	for _, child := range node.Content {
-		removeCommentOfNode(child)
-	}
-}
-
-func setStyle(matchingNodes []*yqlib.NodeContext, style yaml.Style) {
-	for _, nodeContext := range matchingNodes {
-		updateStyleOfNode(nodeContext.Node, style)
-	}
-}
-
-func updateStyleOfNode(node *yaml.Node, style yaml.Style) {
-	node.Style = style
-
-	for _, child := range node.Content {
-		updateStyleOfNode(child, style)
-	}
-}
-
-func writeString(writer io.Writer, txt string) error {
-	_, errorWriting := writer.Write([]byte(txt))
-	return errorWriting
-}
-
-func setIfNotThere(node *yaml.Node, key string, value *yaml.Node) {
-	for index := 0; index < len(node.Content); index = index + 2 {
-		keyNode := node.Content[index]
-		if keyNode.Value == key {
-			return
-		}
-	}
-	// need to add it to the map
-	mapEntryKey := yaml.Node{Value: key, Kind: yaml.ScalarNode}
-	node.Content = append(node.Content, &mapEntryKey)
-	node.Content = append(node.Content, value)
-}
-
-func applyAlias(node *yaml.Node, alias *yaml.Node) {
-	for index := 0; index < len(alias.Content); index = index + 2 {
-		keyNode := alias.Content[index]
-		log.Debugf("applying alias key %v", keyNode.Value)
-		valueNode := alias.Content[index+1]
-		setIfNotThere(node, keyNode.Value, valueNode)
-	}
-}
-
-func explodeNode(node *yaml.Node) error {
-	node.Anchor = ""
-	switch node.Kind {
-	case yaml.SequenceNode, yaml.DocumentNode:
-		for index, contentNode := range node.Content {
-			log.Debugf("exploding index %v", index)
-			errorInContent := explodeNode(contentNode)
-			if errorInContent != nil {
-				return errorInContent
-			}
-		}
-		return nil
-	case yaml.AliasNode:
-		log.Debugf("its an alias!")
-		node.Kind = node.Alias.Kind
-		node.Style = node.Alias.Style
-		node.Tag = node.Alias.Tag
-		node.Content = node.Alias.Content
-		node.Value = node.Alias.Value
-		node.Alias = nil
-		return nil
-	case yaml.MappingNode:
-		for index := 0; index < len(node.Content); index = index + 2 {
-			keyNode := node.Content[index]
-			valueNode := node.Content[index+1]
-			log.Debugf("traversing %v", keyNode.Value)
-			if keyNode.Value != "<<" {
-				errorInContent := explodeNode(valueNode)
-				if errorInContent != nil {
-					return errorInContent
-				}
-			} else {
-				if valueNode.Kind == yaml.SequenceNode {
-					log.Debugf("an alias merge list!")
-					for index := len(valueNode.Content) - 1; index >= 0; index = index - 1 {
-						aliasNode := valueNode.Content[index]
-						applyAlias(node, aliasNode.Alias)
-					}
-				} else {
-					log.Debugf("an alias merge!")
-					applyAlias(node, valueNode.Alias)
-				}
-				node.Content = append(node.Content[:index], node.Content[index+2:]...)
-				//replay that index, since the array is shorter now.
-				index = index - 2
-			}
-		}
-
-		return nil
-	default:
-		return nil
-	}
-}
-
-func explode(matchingNodes []*yqlib.NodeContext) error {
-	log.Debug("exploding nodes")
-	for _, nodeContext := range matchingNodes {
-		log.Debugf("exploding %v", nodeContext.Head)
-		errorExplodingNode := explodeNode(nodeContext.Node)
-		if errorExplodingNode != nil {
-			return errorExplodingNode
-		}
-	}
-	return nil
-}
-
-func printResults(matchingNodes []*yqlib.NodeContext, writer io.Writer) error {
-	if prettyPrint {
-		setStyle(matchingNodes, 0)
-	}
-
-	if stripComments {
-		removeComments(matchingNodes)
-	}
-
-	//always explode anchors when printing json
-	if explodeAnchors || outputToJSON {
-		errorExploding := explode(matchingNodes)
-		if errorExploding != nil {
-			return errorExploding
-		}
-	}
-
-	bufferedWriter := bufio.NewWriter(writer)
-	defer safelyFlush(bufferedWriter)
-
-	if len(matchingNodes) == 0 {
-		log.Debug("no matching results, nothing to print")
-		if defaultValue != "" {
-			return writeString(bufferedWriter, defaultValue)
-		}
-		return nil
-	}
-	var errorWriting error
-
-	var arrayCollection = yaml.Node{Kind: yaml.SequenceNode}
-
-	for _, mappedDoc := range matchingNodes {
-		switch printMode {
-		case "p":
-			errorWriting = writeString(bufferedWriter, lib.PathStackToString(mappedDoc.PathStack)+"\n")
-			if errorWriting != nil {
-				return errorWriting
-			}
-		case "pv", "vp":
-			// put it into a node and print that.
-			var parentNode = yaml.Node{Kind: yaml.MappingNode}
-			parentNode.Content = make([]*yaml.Node, 2)
-			parentNode.Content[0] = &yaml.Node{Kind: yaml.ScalarNode, Value: lib.PathStackToString(mappedDoc.PathStack)}
-			parentNode.Content[1] = transformNode(mappedDoc.Node)
-			if collectIntoArray {
-				arrayCollection.Content = append(arrayCollection.Content, &parentNode)
-			} else if err := printNode(&parentNode, bufferedWriter); err != nil {
-				return err
-			}
-		default:
-			if collectIntoArray {
-				arrayCollection.Content = append(arrayCollection.Content, mappedDoc.Node)
-			} else if err := printNode(transformNode(mappedDoc.Node), bufferedWriter); err != nil {
-				return err
-			}
-		}
-	}
-
-	if collectIntoArray {
-		if err := printNode(transformNode(&arrayCollection), bufferedWriter); err != nil {
-			return err
-		}
-	}
-
-	return nil
-}
-
-func parseDocumentIndex() (bool, int, error) {
-	if docIndex == "*" {
-		return true, -1, nil
-	}
-	docIndexInt64, err := strconv.ParseInt(docIndex, 10, 32)
+	expression, args, err := processArgs(args)
 	if err != nil {
-		return false, -1, errors.Wrapf(err, "Document index %v is not a integer or *", docIndex)
+		return "", nil, err
 	}
-	return false, int(docIndexInt64), nil
+
+	if err := loadSplitFileExpression(); err != nil {
+		return "", nil, err
+	}
+
+	handleBackwardsCompatibility()
+
+	if err := validateCommandFlags(args); err != nil {
+		return "", nil, err
+	}
+
+	if err := configureFormats(args); err != nil {
+		return "", nil, err
+	}
+
+	configureUnwrapScalar()
+
+	return expression, args, nil
 }
 
-type updateDataFn func(dataBucket *yaml.Node, currentIndex int) error
+func setupColors() {
+	fileInfo, _ := os.Stdout.Stat()
 
-func isNullDocument(dataBucket *yaml.Node) bool {
-	return dataBucket.Kind == yaml.DocumentNode && (len(dataBucket.Content) == 0 ||
-		dataBucket.Content[0].Kind == yaml.ScalarNode && dataBucket.Content[0].Tag == "!!null")
-}
-
-func mapYamlDecoder(updateData updateDataFn, encoder yqlib.Encoder) yamlDecoderFn {
-	return func(decoder *yaml.Decoder) error {
-		var dataBucket yaml.Node
-		var errorReading error
-		var errorWriting error
-		var errorUpdating error
-		var currentIndex = 0
-
-		var updateAll, docIndexInt, errorParsingDocIndex = parseDocumentIndex()
-		if errorParsingDocIndex != nil {
-			return errorParsingDocIndex
-		}
-
-		for {
-			log.Debugf("Read doc %v", currentIndex)
-			errorReading = decoder.Decode(&dataBucket)
-
-			if errorReading == io.EOF && docIndexInt == 0 && currentIndex == 0 {
-				//empty document, lets just make one
-				dataBucket = yaml.Node{Kind: yaml.DocumentNode, Content: make([]*yaml.Node, 1)}
-				child := yaml.Node{Kind: yaml.MappingNode}
-				dataBucket.Content[0] = &child
-			} else if isNullDocument(&dataBucket) && (updateAll || docIndexInt == currentIndex) {
-				child := yaml.Node{Kind: yaml.MappingNode}
-				dataBucket.Content[0] = &child
-			} else if errorReading == io.EOF {
-				if !updateAll && currentIndex <= docIndexInt {
-					return fmt.Errorf("asked to process document index %v but there are only %v document(s)", docIndex, currentIndex)
-				}
-				return nil
-			} else if errorReading != nil {
-				return errors.Wrapf(errorReading, "Error reading document at index %v, %v", currentIndex, errorReading)
-			}
-			errorUpdating = updateData(&dataBucket, currentIndex)
-			if errorUpdating != nil {
-				return errors.Wrapf(errorUpdating, "Error updating document at index %v", currentIndex)
-			}
-
-			if prettyPrint {
-				updateStyleOfNode(&dataBucket, 0)
-			}
-
-			errorWriting = encoder.Encode(&dataBucket)
-
-			if errorWriting != nil {
-				return errors.Wrapf(errorWriting, "Error writing document at index %v, %v", currentIndex, errorWriting)
-			}
-			currentIndex = currentIndex + 1
-		}
+	if forceColor || (!forceNoColor && (fileInfo.Mode()&os.ModeCharDevice) != 0) {
+		colorsEnabled = true
 	}
 }
 
-func prefixDocument(updateAll bool, docIndexInt int, currentIndex int, dataBucket *yaml.Node, updateCommand yqlib.UpdateCommand) error {
-	if updateAll || currentIndex == docIndexInt {
-		log.Debugf("Prefixing document %v", currentIndex)
-		yqlib.DebugNode(dataBucket)
-		updateCommand.Value = dataBucket.Content[0]
-		dataBucket.Content = make([]*yaml.Node, 1)
-
-		newNode := lib.New(updateCommand.Path)
-		dataBucket.Content[0] = &newNode
-
-		errorUpdating := lib.Update(dataBucket, updateCommand, true)
-		if errorUpdating != nil {
-			return errorUpdating
+func loadSplitFileExpression() error {
+	if splitFileExpFile != "" {
+		splitExpressionBytes, err := os.ReadFile(splitFileExpFile)
+		if err != nil {
+			return err
 		}
+		splitFileExp = string(splitExpressionBytes)
 	}
 	return nil
 }
 
-func updateDoc(inputFile string, updateCommands []yqlib.UpdateCommand, writer io.Writer) error {
-	var updateAll, docIndexInt, errorParsingDocIndex = parseDocumentIndex()
-	if errorParsingDocIndex != nil {
-		return errorParsingDocIndex
-	}
-
-	var updateData = func(dataBucket *yaml.Node, currentIndex int) error {
-		if updateAll || currentIndex == docIndexInt {
-			log.Debugf("Updating doc %v", currentIndex)
-			for _, updateCommand := range updateCommands {
-				log.Debugf("Processing update to Path %v", updateCommand.Path)
-				errorUpdating := lib.Update(dataBucket, updateCommand, autoCreateFlag)
-				if errorUpdating != nil {
-					return errorUpdating
-				}
-			}
-		}
-		return nil
-	}
-	return readAndUpdate(writer, inputFile, updateData)
-}
-
-func readAndUpdate(stdOut io.Writer, inputFile string, updateData updateDataFn) error {
-	var destination io.Writer
-	var destinationName string
-	var completedSuccessfully = false
-	if writeInplace {
-		info, err := os.Stat(inputFile)
-		if err != nil {
-			return err
-		}
-		// mkdir temp dir as some docker images does not have temp dir
-		_, err = os.Stat(os.TempDir())
-		if os.IsNotExist(err) {
-			err = os.Mkdir(os.TempDir(), 0700)
-			if err != nil {
-				return err
-			}
-		} else if err != nil {
-			return err
-		}
-		tempFile, err := ioutil.TempFile("", "temp")
-		if err != nil {
-			return err
-		}
-		destinationName = tempFile.Name()
-		err = os.Chmod(destinationName, info.Mode())
-		if err != nil {
-			return err
-		}
-		destination = tempFile
-		defer func() {
-			safelyCloseFile(tempFile)
-			if completedSuccessfully {
-				safelyRenameFile(tempFile.Name(), inputFile)
-			}
-		}()
-	} else {
-		destination = stdOut
-		destinationName = "Stdout"
-	}
-
-	log.Debugf("Writing to %v from %v", destinationName, inputFile)
-
-	bufferedWriter := bufio.NewWriter(destination)
-	defer safelyFlush(bufferedWriter)
-
-	var encoder yqlib.Encoder
+func handleBackwardsCompatibility() {
+	// backwards compatibility
 	if outputToJSON {
-		encoder = yqlib.NewJsonEncoder(bufferedWriter, prettyPrint, indent)
-	} else {
-		encoder = yqlib.NewYamlEncoder(bufferedWriter, indent, colorsEnabled)
+		outputFormat = "json"
+	}
+}
+
+func validateCommandFlags(args []string) error {
+	if writeInplace && (len(args) == 0 || args[0] == "-") {
+		return fmt.Errorf("write in place flag only applicable when giving an expression and at least one file")
 	}
 
-	var errorProcessing = readStream(inputFile, mapYamlDecoder(updateData, encoder))
-	completedSuccessfully = errorProcessing == nil
-	return errorProcessing
-}
-
-type updateCommandParsed struct {
-	Command string
-	Path    string
-	Value   yaml.Node
-}
-
-func readUpdateCommands(args []string, expectedArgs int, badArgsMessage string) ([]yqlib.UpdateCommand, error) {
-	var updateCommands []yqlib.UpdateCommand = make([]yqlib.UpdateCommand, 0)
-	if writeScript != "" {
-		var parsedCommands = make([]updateCommandParsed, 0)
-
-		err := readData(writeScript, 0, &parsedCommands)
-
-		if err != nil && err != io.EOF {
-			return nil, err
-		}
-
-		log.Debugf("Read write commands file '%v'", parsedCommands)
-		for index := range parsedCommands {
-			parsedCommand := parsedCommands[index]
-			updateCommand := yqlib.UpdateCommand{Command: parsedCommand.Command, Path: parsedCommand.Path, Value: &parsedCommand.Value, Overwrite: true}
-			updateCommands = append(updateCommands, updateCommand)
-		}
-
-		log.Debugf("Read write commands file '%v'", updateCommands)
-	} else if sourceYamlFile != "" && len(args) == expectedArgs-1 {
-		log.Debugf("Reading value from %v", sourceYamlFile)
-		var value yaml.Node
-		err := readData(sourceYamlFile, 0, &value)
-		if err != nil && err != io.EOF {
-			return nil, err
-		}
-		log.Debug("args %v", args[expectedArgs-2])
-		updateCommands = make([]yqlib.UpdateCommand, 1)
-		updateCommands[0] = yqlib.UpdateCommand{Command: "update", Path: args[expectedArgs-2], Value: value.Content[0], Overwrite: true}
-	} else if len(args) == expectedArgs {
-		updateCommands = make([]yqlib.UpdateCommand, 1)
-		log.Debug("args %v", args)
-		log.Debug("path %v", args[expectedArgs-2])
-		log.Debug("Value %v", args[expectedArgs-1])
-		updateCommands[0] = yqlib.UpdateCommand{Command: "update", Path: args[expectedArgs-2], Value: valueParser.Parse(args[expectedArgs-1], customTag, customStyle, anchorName, makeAlias), Overwrite: true}
-	} else if len(args) == expectedArgs-1 {
-		// don't update the value
-		updateCommands = make([]yqlib.UpdateCommand, 1)
-		log.Debug("args %v", args)
-		log.Debug("path %v", args[expectedArgs-2])
-		updateCommands[0] = yqlib.UpdateCommand{Command: "update", Path: args[expectedArgs-2], Value: valueParser.Parse("", customTag, customStyle, anchorName, makeAlias), Overwrite: true, DontUpdateNodeValue: true}
-	} else {
-		return nil, errors.New(badArgsMessage)
+	if frontMatter != "" && len(args) == 0 {
+		return fmt.Errorf("front matter flag only applicable when giving an expression and at least one file")
 	}
-	return updateCommands, nil
+
+	if writeInplace && splitFileExp != "" {
+		return fmt.Errorf("write in place cannot be used with split file")
+	}
+
+	if nullInput && len(args) > 0 {
+		return fmt.Errorf("cannot pass files in when using null-input flag")
+	}
+
+	return nil
 }
 
-func safelyRenameFile(from string, to string) {
-	if renameError := os.Rename(from, to); renameError != nil {
-		log.Debugf("Error renaming from %v to %v, attempting to copy contents", from, to)
-		log.Debug(renameError.Error())
-		// can't do this rename when running in docker to a file targeted in a mounted volume,
-		// so gracefully degrade to copying the entire contents.
-		if copyError := copyFileContents(from, to); copyError != nil {
-			log.Errorf("Failed copying from %v to %v", from, to)
-			log.Error(copyError.Error())
+func configureFormats(args []string) error {
+	inputFilename := ""
+	if len(args) > 0 {
+		inputFilename = args[0]
+	}
+
+	if err := configureInputFormat(inputFilename); err != nil {
+		return err
+	}
+
+	if err := configureOutputFormat(); err != nil {
+		return err
+	}
+
+	yqlib.GetLogger().Debug("Using input format %v", inputFormat)
+	yqlib.GetLogger().Debug("Using output format %v", outputFormat)
+
+	return nil
+}
+
+func configureInputFormat(inputFilename string) error {
+	if inputFormat == "" || inputFormat == "auto" || inputFormat == "a" {
+		inputFormat = yqlib.FormatStringFromFilename(inputFilename)
+
+		_, err := yqlib.FormatFromString(inputFormat)
+		if err != nil {
+			// unknown file type, default to yaml
+			yqlib.GetLogger().Debug("Unknown file format extension '%v', defaulting to yaml", inputFormat)
+			inputFormat = "yaml"
+			if isAutomaticOutputFormat() {
+				outputFormat = "yaml"
+			}
+		} else if isAutomaticOutputFormat() {
+			outputFormat = inputFormat
+		}
+	} else if isAutomaticOutputFormat() {
+		// backwards compatibility -
+		// before this was introduced, `yq -pcsv things.csv`
+		// would produce *yaml* output.
+		//
+		outputFormat = yqlib.FormatStringFromFilename(inputFilename)
+		if inputFilename != "-" {
+			yqlib.GetLogger().Warning("yq default output is now 'auto' (based on the filename extension). Normally yq would output '%v', but for backwards compatibility 'yaml' has been set. Please use -oy to specify yaml, or drop the -p flag.", outputFormat)
+		}
+		outputFormat = "yaml"
+	}
+	return nil
+}
+
+func configureOutputFormat() error {
+	outputFormatType, err := yqlib.FormatFromString(outputFormat)
+	if err != nil {
+		return err
+	}
+
+	if outputFormatType == yqlib.YamlFormat ||
+		outputFormatType == yqlib.PropertiesFormat {
+		unwrapScalar = true
+	}
+
+	return nil
+}
+
+func configureUnwrapScalar() {
+	if unwrapScalarFlag.IsExplicitlySet() {
+		unwrapScalar = unwrapScalarFlag.IsSet()
+	}
+}
+
+func configureDecoder(evaluateTogether bool) (yqlib.Decoder, error) {
+	format, err := yqlib.FormatFromString(inputFormat)
+	if err != nil {
+		return nil, err
+	}
+	yqlib.ConfiguredYamlPreferences.EvaluateTogether = evaluateTogether
+
+	yqlibDecoder := format.DecoderFactory()
+	if yqlibDecoder == nil {
+		return nil, fmt.Errorf("no support for %s input format", inputFormat)
+	}
+	return yqlibDecoder, nil
+}
+
+func configurePrinterWriter(format *yqlib.Format, out io.Writer) (yqlib.PrinterWriter, error) {
+
+	var printerWriter yqlib.PrinterWriter
+
+	if splitFileExp != "" {
+		colorsEnabled = forceColor
+		splitExp, err := yqlib.ExpressionParser.ParseExpression(splitFileExp)
+		if err != nil {
+			return nil, fmt.Errorf("bad split document expression: %w", err)
+		}
+		printerWriter = yqlib.NewMultiPrinterWriter(splitExp, format)
+	} else {
+		printerWriter = yqlib.NewSinglePrinterWriter(out)
+	}
+	return printerWriter, nil
+}
+
+func configureEncoder() (yqlib.Encoder, error) {
+	yqlibOutputFormat, err := yqlib.FormatFromString(outputFormat)
+	if err != nil {
+		return nil, err
+	}
+	yqlib.ConfiguredXMLPreferences.Indent = indent
+	yqlib.ConfiguredYamlPreferences.Indent = indent
+	yqlib.ConfiguredJSONPreferences.Indent = indent
+
+	yqlib.ConfiguredYamlPreferences.UnwrapScalar = unwrapScalar
+	yqlib.ConfiguredPropertiesPreferences.UnwrapScalar = unwrapScalar
+	yqlib.ConfiguredJSONPreferences.UnwrapScalar = unwrapScalar
+
+	yqlib.ConfiguredYamlPreferences.ColorsEnabled = colorsEnabled
+	yqlib.ConfiguredJSONPreferences.ColorsEnabled = colorsEnabled
+
+	yqlib.ConfiguredYamlPreferences.PrintDocSeparators = !noDocSeparators
+
+	encoder := yqlibOutputFormat.EncoderFactory()
+
+	if encoder == nil {
+		return nil, fmt.Errorf("no support for %s output format", outputFormat)
+	}
+	return encoder, err
+}
+
+// this is a hack to enable backwards compatibility with githubactions (which pipe /dev/null into everything)
+// and being able to call yq with the filename as a single parameter
+//
+// without this - yq detects there is stdin (thanks githubactions),
+// then tries to parse the filename as an expression
+func maybeFile(str string) bool {
+	yqlib.GetLogger().Debugf("checking '%v' is a file", str)
+	stat, err := os.Stat(str) // #nosec
+	result := err == nil && !stat.IsDir()
+	if yqlib.GetLogger().IsEnabledFor(logging.DEBUG) {
+		if err != nil {
+			yqlib.GetLogger().Debugf("error: %v", err)
 		} else {
-			removeErr := os.Remove(from)
-			if removeErr != nil {
-				log.Errorf("failed removing original file: %s", from)
-			}
+			yqlib.GetLogger().Debugf("error: %v, dir: %v", err, stat.IsDir())
+		}
+		yqlib.GetLogger().Debugf("result: %v", result)
+	}
+	return result
+}
+
+func processStdInArgs(args []string) []string {
+	stat, err := os.Stdin.Stat()
+	if err != nil {
+		yqlib.GetLogger().Debugf("error getting stdin: %v", err)
+	}
+	pipingStdin := stat != nil && (stat.Mode()&os.ModeCharDevice) == 0
+
+	// if we've been given a file, don't automatically
+	// read from stdin.
+	// this happens if there is more than one argument
+	// or only one argument and its a file
+	if nullInput || !pipingStdin || len(args) > 1 || (len(args) > 0 && maybeFile(args[0])) {
+		return args
+	}
+
+	for _, arg := range args {
+		if arg == "-" {
+			return args
 		}
 	}
+	yqlib.GetLogger().Debugf("missing '-', adding it to the end")
+
+	// we're piping from stdin, but there's no '-' arg
+	// lets add one to the end
+	return append(args, "-")
 }
 
-// thanks https://stackoverflow.com/questions/21060945/simple-way-to-copy-a-file-in-golang
-func copyFileContents(src, dst string) (err error) {
-	in, err := os.Open(src) // nolint gosec
-	if err != nil {
-		return err
-	}
-	defer safelyCloseFile(in)
-	out, err := os.Create(dst)
-	if err != nil {
-		return err
-	}
-	defer safelyCloseFile(out)
-	if _, err = io.Copy(out, in); err != nil {
-		return err
-	}
-	return out.Sync()
-}
+func processArgs(originalArgs []string) (string, []string, error) {
+	expression := forceExpression
+	args := processStdInArgs(originalArgs)
+	maybeFirstArgIsAFile := len(args) > 0 && maybeFile(args[0])
 
-func safelyFlush(writer *bufio.Writer) {
-	if err := writer.Flush(); err != nil {
-		log.Error("Error flushing writer!")
-		log.Error(err.Error())
+	if expressionFile == "" && maybeFirstArgIsAFile && strings.HasSuffix(args[0], ".yq") {
+		// lets check if an expression file was given
+		yqlib.GetLogger().Debug("Assuming arg %v is an expression file", args[0])
+		expressionFile = args[0]
+		args = args[1:]
 	}
 
-}
-func safelyCloseFile(file *os.File) {
-	err := file.Close()
-	if err != nil {
-		log.Error("Error closing file!")
-		log.Error(err.Error())
-	}
-}
-
-type yamlDecoderFn func(*yaml.Decoder) error
-
-func readStream(filename string, yamlDecoder yamlDecoderFn) error {
-	if filename == "" {
-		return errors.New("Must provide filename")
-	}
-
-	var stream io.Reader
-	if filename == "-" {
-		stream = bufio.NewReader(os.Stdin)
-	} else {
-		file, err := os.Open(filename) // nolint gosec
+	if expressionFile != "" {
+		expressionBytes, err := os.ReadFile(expressionFile)
 		if err != nil {
-			return err
+			return "", nil, err
 		}
-		defer safelyCloseFile(file)
-		stream = file
+		//replace \r\n (windows) with good ol' unix file endings.
+		expression = strings.ReplaceAll(string(expressionBytes), "\r\n", "\n")
 	}
-	return yamlDecoder(yaml.NewDecoder(stream))
-}
 
-func readData(filename string, indexToRead int, parsedData interface{}) error {
-	return readStream(filename, func(decoder *yaml.Decoder) error {
-		for currentIndex := 0; currentIndex < indexToRead; currentIndex++ {
-			errorSkipping := decoder.Decode(parsedData)
-			if errorSkipping != nil {
-				return errors.Wrapf(errorSkipping, "Error processing document at index %v, %v", currentIndex, errorSkipping)
-			}
-		}
-		return decoder.Decode(parsedData)
-	})
+	yqlib.GetLogger().Debugf("processed args: %v", args)
+	if expression == "" && len(args) > 0 && args[0] != "-" && !maybeFile(args[0]) {
+		yqlib.GetLogger().Debug("assuming expression is '%v'", args[0])
+		expression = args[0]
+		args = args[1:]
+	}
+	return expression, args, nil
 }

cmd/validate.go

@@ -1,37 +0,0 @@
-package cmd
-
-import (
-	errors "github.com/pkg/errors"
-	"github.com/spf13/cobra"
-)
-
-func createValidateCmd() *cobra.Command {
-	var cmdRead = &cobra.Command{
-		Use:     "validate [yaml_file]",
-		Aliases: []string{"v"},
-		Short:   "yq v sample.yaml",
-		Example: `
-yq v - # reads from stdin
-`,
-		RunE:          validateProperty,
-		SilenceUsage:  true,
-		SilenceErrors: true,
-	}
-	cmdRead.PersistentFlags().StringVarP(&docIndex, "doc", "d", "0", "process document index number (0 based, * for all documents)")
-	return cmdRead
-}
-
-func validateProperty(cmd *cobra.Command, args []string) error {
-	if len(args) < 1 {
-		return errors.New("Must provide filename")
-	}
-
-	var updateAll, docIndexInt, errorParsingDocIndex = parseDocumentIndex()
-	if errorParsingDocIndex != nil {
-		return errorParsingDocIndex
-	}
-
-	_, errorReadingStream := readYamlFile(args[0], "", updateAll, docIndexInt)
-
-	return errorReadingStream
-}

cmd/validate_test.go

@@ -1,31 +0,0 @@
-package cmd
-
-import (
-	"fmt"
-	"testing"
-
-	"github.com/mikefarah/yq/v3/test"
-)
-
-func TestValidateCmd(t *testing.T) {
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "validate ../examples/sample.yaml b.c")
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	test.AssertResult(t, "", result.Output)
-}
-
-func TestValidateBadDataCmd(t *testing.T) {
-	content := `[!Whatever]`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("validate %s", filename))
-	if result.Error == nil {
-		t.Error("Expected command to fail")
-	}
-	expectedOutput := `yaml: line 1: did not find expected ',' or ']'`
-	test.AssertResult(t, expectedOutput, result.Error.Error())
-}

cmd/version.go

@@ -11,7 +11,7 @@ var (
 	GitDescribe string
 
 	// Version is main version number that is being run at the moment.
-	Version = "3.3.2"
+	Version = "v4.46.1"
 
 	// VersionPrerelease is a pre-release marker for the version. If this is "" (empty string)
 	// then it means that it is a final release. Otherwise, this is a pre-release
@@ -25,7 +25,7 @@ const ProductName = "yq"
 // GetVersionDisplay composes the parts of the version in a way that's suitable
 // for displaying to humans.
 func GetVersionDisplay() string {
-	return fmt.Sprintf("%s version %s\n", ProductName, getHumanVersion())
+	return fmt.Sprintf("yq (https://github.com/mikefarah/yq/) version %s\n", getHumanVersion())
 }
 
 func getHumanVersion() string {
@@ -45,5 +45,5 @@ func getHumanVersion() string {
 	}
 
 	// Strip off any single quotes added by the git information.
-	return strings.Replace(version, "'", "", -1)
+	return strings.ReplaceAll(version, "'", "")
 }

cmd/version_test.go

@@ -3,7 +3,7 @@ package cmd
 import "testing"
 
 func TestGetVersionDisplay(t *testing.T) {
-	var expectedVersion = ProductName + " version " + Version
+	var expectedVersion = ProductName + " (https://github.com/mikefarah/yq/) version " + Version
 	if VersionPrerelease != "" {
 		expectedVersion = expectedVersion + "-" + VersionPrerelease
 	}

cmd/write.go

@@ -1,61 +0,0 @@
-package cmd
-
-import (
-	"github.com/spf13/cobra"
-)
-
-func createWriteCmd() *cobra.Command {
-	var cmdWrite = &cobra.Command{
-		Use:     "write [yaml_file] [path_expression] [value]",
-		Aliases: []string{"w"},
-		Short:   "yq w [--inplace/-i] [--script/-s script_file] [--doc/-d index] sample.yaml 'b.e(name==fr*).value' newValue",
-		Example: `
-yq write things.yaml 'a.b.c' true
-yq write things.yaml 'a.*.c' true
-yq write things.yaml 'a.**' true
-yq write things.yaml 'a.(child.subchild==co*).c' true
-yq write things.yaml 'a.b.c' --tag '!!str' true # force 'true' to be interpreted as a string instead of bool
-yq write things.yaml 'a.b.c' --tag '!!float' 3
-yq write --inplace -- things.yaml 'a.b.c' '--cat' # need to use '--' to stop processing arguments as flags
-yq w -i things.yaml 'a.b.c' cat
-yq w -i -s update_script.yaml things.yaml
-yq w things.yaml 'a.b.d[+]' foo # appends a new node to the 'd' array
-yq w --doc 2 things.yaml 'a.b.d[+]' foo # updates the 3rd document of the yaml file
-      `,
-		Long: `Updates the yaml file w.r.t the given path and value.
-Outputs to STDOUT unless the inplace flag is used, in which case the file is updated instead.
-
-Append value to array adds the value to the end of array.
-
-Update Scripts:
-Note that you can give an update script to perform more sophisticated update. Update script
-format is list of update commands (update or delete) like so:
----
-- command: update
-  path: b.c
-  value:
-    #great
-    things: frog # wow!
-- command: delete
-  path: b.d
-`,
-		RunE: writeProperty,
-	}
-	cmdWrite.PersistentFlags().BoolVarP(&writeInplace, "inplace", "i", false, "update the yaml file inplace")
-	cmdWrite.PersistentFlags().StringVarP(&writeScript, "script", "s", "", "yaml script for updating yaml")
-	cmdWrite.PersistentFlags().StringVarP(&sourceYamlFile, "from", "f", "", "yaml file for updating yaml (as-is)")
-	cmdWrite.PersistentFlags().StringVarP(&customTag, "tag", "t", "", "set yaml tag (e.g. !!int)")
-	cmdWrite.PersistentFlags().StringVarP(&docIndex, "doc", "d", "0", "process document index number (0 based, * for all documents)")
-	cmdWrite.PersistentFlags().StringVarP(&customStyle, "style", "", "", "formatting style of the value: single, double, folded, flow, literal, tagged")
-	cmdWrite.PersistentFlags().StringVarP(&anchorName, "anchorName", "", "", "anchor name")
-	cmdWrite.PersistentFlags().BoolVarP(&makeAlias, "makeAlias", "", false, "create an alias using the value as the anchor name")
-	return cmdWrite
-}
-
-func writeProperty(cmd *cobra.Command, args []string) error {
-	var updateCommands, updateCommandsError = readUpdateCommands(args, 3, "Must provide <filename> <path_to_update> <value>")
-	if updateCommandsError != nil {
-		return updateCommandsError
-	}
-	return updateDoc(args[0], updateCommands, cmd.OutOrStdout())
-}

cmd/write_test.go

@@ -1,593 +0,0 @@
-package cmd
-
-import (
-	"fmt"
-	"runtime"
-	"strings"
-	"testing"
-
-	"github.com/mikefarah/yq/v3/test"
-)
-
-func TestWriteCmd(t *testing.T) {
-	content := `b:
-  c: 3
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("write %s b.c 7", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `b:
-  c: 7
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestWriteWithTaggedStyleCmd(t *testing.T) {
-	content := `b:
-  c: dog
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("write %s b.c cat --tag=!!str --style=tagged", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `b:
-  c: !!str cat
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestWriteWithDoubleQuotedStyleCmd(t *testing.T) {
-	content := `b:
-  c: dog
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("write %s b.c cat --style=double", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `b:
-  c: "cat"
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestWriteUpdateStyleOnlyCmd(t *testing.T) {
-	content := `b:
-  c: dog
-  d: things
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("write %s b.* --style=single", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `b:
-  c: 'dog'
-  d: 'things'
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestWriteUpdateTagOnlyCmd(t *testing.T) {
-	content := `b:
-  c: true
-  d: false
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("write %s b.* --tag=!!str", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `b:
-  c: "true"
-  d: "false"
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestWriteWithSingleQuotedStyleCmd(t *testing.T) {
-	content := `b:
-  c: dog
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("write %s b.c cat --style=single", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `b:
-  c: 'cat'
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestWriteWithLiteralStyleCmd(t *testing.T) {
-	content := `b:
-  c: dog
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("write %s b.c cat --style=literal", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `b:
-  c: |-
-    cat
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestWriteWithFoldedStyleCmd(t *testing.T) {
-	content := `b:
-  c: dog
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("write %s b.c cat --style=folded", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `b:
-  c: >-
-    cat
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestWriteEmptyMultiDocCmd(t *testing.T) {
-	content := `# this is empty
----
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("write %s c 7", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `c: 7
-
-# this is empty
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestWriteSurroundingEmptyMultiDocCmd(t *testing.T) {
-	content := `---
-# empty
----
-cat: frog
----
-
-# empty
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("write %s -d1 c 7", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `
-
-# empty
----
-cat: frog
-c: 7
----
-
-
-# empty
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestWriteFromFileCmd(t *testing.T) {
-	content := `b:
-  c: 3
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	source := `kittens: are cute # sure are!`
-	fromFilename := test.WriteTempYamlFile(source)
-	defer test.RemoveTempYamlFile(fromFilename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("write %s b.c -f %s", filename, fromFilename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `b:
-  c:
-    kittens: are cute # sure are!
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestWriteEmptyCmd(t *testing.T) {
-	content := ``
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("write %s b.c 7", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `b:
-  c: 7
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestWriteAutoCreateCmd(t *testing.T) {
-	content := `applications:
-  - name: app
-    env:`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("write %s applications[0].env.hello world", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `applications:
-  - name: app
-    env:
-      hello: world
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestWriteCmdScript(t *testing.T) {
-	content := `b:
-  c: 3
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	updateScript := `- command: update
-  path: b.c
-  value: 7`
-	scriptFilename := test.WriteTempYamlFile(updateScript)
-	defer test.RemoveTempYamlFile(scriptFilename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("write --script %s %s", scriptFilename, filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `b:
-  c: 7
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestWriteCmdEmptyScript(t *testing.T) {
-	content := `b:
-  c: 3
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	updateScript := ``
-	scriptFilename := test.WriteTempYamlFile(updateScript)
-	defer test.RemoveTempYamlFile(scriptFilename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("write --script %s %s", scriptFilename, filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `b:
-  c: 3
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestWriteMultiCmd(t *testing.T) {
-	content := `b:
-  c: 3
----
-apples: great
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("write %s -d 1 apples ok", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `b:
-  c: 3
----
-apples: ok
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-func TestWriteInvalidDocumentIndexCmd(t *testing.T) {
-	content := `b:
-  c: 3
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("write %s -df apples ok", filename))
-	if result.Error == nil {
-		t.Error("Expected command to fail due to invalid path")
-	}
-	expectedOutput := `Document index f is not a integer or *: strconv.ParseInt: parsing "f": invalid syntax`
-	test.AssertResult(t, expectedOutput, result.Error.Error())
-}
-
-func TestWriteBadDocumentIndexCmd(t *testing.T) {
-	content := `b:
-  c: 3
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("write %s -d 1 apples ok", filename))
-	if result.Error == nil {
-		t.Error("Expected command to fail due to invalid path")
-	}
-	expectedOutput := `asked to process document index 1 but there are only 1 document(s)`
-	test.AssertResult(t, expectedOutput, result.Error.Error())
-}
-func TestWriteMultiAllCmd(t *testing.T) {
-	content := `b:
-  c: 3
----
-apples: great
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("write %s -d * apples ok", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `b:
-  c: 3
-apples: ok
----
-apples: ok`
-	test.AssertResult(t, expectedOutput, strings.Trim(result.Output, "\n "))
-}
-
-func TestWriteCmd_EmptyArray(t *testing.T) {
-	content := `b: 3`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("write %s a []", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `b: 3
-a: []
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestWriteCmd_Error(t *testing.T) {
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "write")
-	if result.Error == nil {
-		t.Error("Expected command to fail due to missing arg")
-	}
-	expectedOutput := `Must provide <filename> <path_to_update> <value>`
-	test.AssertResult(t, expectedOutput, result.Error.Error())
-}
-
-func TestWriteCmd_ErrorUnreadableFile(t *testing.T) {
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "write fake-unknown a.b 3")
-	if result.Error == nil {
-		t.Error("Expected command to fail due to unknown file")
-	}
-	var expectedOutput string
-	if runtime.GOOS == "windows" {
-		expectedOutput = `open fake-unknown: The system cannot find the file specified.`
-	} else {
-		expectedOutput = `open fake-unknown: no such file or directory`
-	}
-	test.AssertResult(t, expectedOutput, result.Error.Error())
-}
-
-func TestWriteCmd_Inplace(t *testing.T) {
-	content := `b:
-  c: 3
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("write -i %s b.c 7", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	gotOutput := test.ReadTempYamlFile(filename)
-	expectedOutput := `b:
-  c: 7`
-	test.AssertResult(t, expectedOutput, strings.Trim(gotOutput, "\n "))
-}
-
-func TestWriteCmd_InplaceError(t *testing.T) {
-	content := `b: cat
-  c: 3
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("write -i %s b.c 7", filename))
-	if result.Error == nil {
-		t.Error("Expected Error to occur!")
-	}
-	gotOutput := test.ReadTempYamlFile(filename)
-	test.AssertResult(t, content, gotOutput)
-}
-
-func TestWriteCmd_Append(t *testing.T) {
-	content := `b:
-  - foo
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("write %s b[+] 7", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `b:
-  - foo
-  - 7
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestWriteCmd_AppendInline(t *testing.T) {
-	content := `b: [foo]`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("write %s b[+] 7", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `b: [foo, 7]
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestWriteCmd_AppendInlinePretty(t *testing.T) {
-	content := `b: [foo]`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("write %s -P b[+] 7", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `b:
-  - foo
-  - 7
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestWriteCmd_AppendEmptyArray(t *testing.T) {
-	content := `a: 2
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("write %s b[+] v", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `a: 2
-b:
-  - v
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestWriteCmd_SplatArray(t *testing.T) {
-	content := `b:
-- c: thing
-- c: another thing
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("write %s b[*].c new", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `b:
-  - c: new
-  - c: new
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestWriteCmd_SplatMap(t *testing.T) {
-	content := `b:
-  c: thing
-  d: another thing
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("write %s b.* new", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `b:
-  c: new
-  d: new
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
-func TestWriteCmd_SplatMapEmpty(t *testing.T) {
-	content := `b:
-  c: thing
-  d: another thing
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("write %s b.c.* new", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `b:
-  c: {}
-  d: another thing
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}

compare.sh

@@ -1,13 +0,0 @@
-GREEN='\033[0;32m'
-NC='\033[0m'
-
-echo "${GREEN}---Old---${NC}"
-yq $@ > /tmp/yq-old-output
-cat /tmp/yq-old-output
-
-echo "${GREEN}---New---${NC}"
-./yq $@ > /tmp/yq-new-output
-cat /tmp/yq-new-output
-
-echo "${GREEN}---Diff---${NC}"
-colordiff /tmp/yq-old-output /tmp/yq-new-output

cspell.config.yaml

@@ -0,0 +1,14 @@
+---
+$schema: https://raw.githubusercontent.com/streetsidesoftware/cspell/main/cspell.schema.json
+version: '0.2'
+language: en-GB
+dictionaryDefinitions:
+  - name: project-words
+    path: './project-words.txt'
+    addWords: true
+dictionaries:
+  - project-words
+ignorePaths:
+  - 'vendor'
+  - 'bin'
+  - '/project-words.txt'

debian/changelog

@@ -1,69 +0,0 @@
-yq (3.3-0) focal; urgency=medium
-
-  * You can control string styles (quotes) using the new --style flag
-  * String values now always have quotes when outputting to json
-  * Negative array indices now traverse the array backwards
-  * Added a --stripComments flag to print yaml without any comments
-  * Bumped go to version 1.14
-
- -- Roberto Mier Escandon <rmescandon@gmail.com>  Thu, 30 Apr 2020 20:45:44 +0200
-
-yq (3.1-2) eoan; urgency=medium
-
-  * Bug fix: yq 3 was removing empty inline-style objects and arrays (#355)
-  * Bug fix: Merge option returned different output when switching order of 
-    merging files(#347)
-  * Bug fix: Add new object to existing array object was failing in 3.1.1 (#361)
-  * Bug fix: yq 3 empty keys did not allow merging of values (#356)
-  * Bug fix: keys quoted during merge (#363)
-  * Bug fix: Correct length with wc -l (#362)
-  * Bug fix: Write to empty document removed path (#359)
-
- -- Roberto Mier Escandon <rmescandon@gmail.com>  Mon, 24 Feb 2020 20:31:58 +0100
-
-yq (3.1-1) eoan; urgency=medium
-
-  * Keeps yaml comments and formatting, can specify yaml tags when updating.
-  * Handles anchors
-  * Can print out matching paths and values when splatting
-  * JSON output works for all commands
-  * Yaml files with multiple documents are printed out as one JSON 
-    document per line.
-  * Deep splat (**) to match arbitrary paths
-  * Update scripts file format has changed to be more powerful
-  * Reading and splatting, matching results are printed once per line
-  * Bugfixing
-
- -- Roberto Mier Escandon <rmescandon@gmail.com>  Tue, 11 Feb 2020 22:18:24 +0100
-
-yq (2.2-1) bionic; urgency=medium
-
-  * Added Windows support for the "--inplace" command flag
-  * Prefix now supports arrays
-  * Add prefix command
-  * Bump Alpine version to 3.8
-  * Improved docker build process
-  * Lint fixes
-  * Build support for all linux architectures supported by gox
-
- -- Roberto Mier Escandon <rmescandon@gmail.com>  Sat, 19 Jan 2019 15:50:47 +0100
-
-yq (2.1-0) bionic; urgency=medium
-
-  * Ability to read multiple documents in a single file
-  * Ability to append list items instead of overwriting
-
- -- Roberto Mier Escandón <rmescandon@gmail.com>  Tue, 10 Jul 2018 14:02:42 +0200
-
-yq (2.0-0) bionic; urgency=medium
-
-  * Release 2.0.0
-
- -- Roberto Mier Escandón <rmescandon@gmail.com>  Wed, 20 Jun 2018 10:29:53 +0200
-
-yq (1.15-0) bionic; urgency=medium
-
-  * Release 1.15
-
- -- Roberto Mier Escandón <rmescandon@gmail.com>  Wed, 06 Jun 2018 11:32:03 +0200
-

debian/compat

@@ -1 +0,0 @@
-10

debian/control

@@ -1,22 +0,0 @@
-Source: yq
-Section: devel
-Priority: optional
-Maintainer: Roberto Mier Escandón <rmescandon@gmail.com>
-Build-Depends: debhelper (>=10),
-               dh-golang (>=1.34),
-               golang-1.13,
-               rsync
-Standards-Version: 4.1.4
-Homepage: https://github.com/mikefarah/yq.git
-Vcs-Browser: https://github.com/mikefarah/yq.git
-Vcs-Git: https://github.com/mikefarah/yq.git
-XS-Go-Import-Path: github.com/mikefarah/yq
-XSBC-Original-Maintainer: Roberto Mier Escandón <rmescandon@gmail.com>
-
-Package: yq
-Architecture: any
-Depends: ${shlibs:Depends}, ${misc:Depends}
-Description: lightweight and portable command-line YAML processor
- .
- The aim of the project is to be the 
- [jq](https://github.com/stedolan/jq) or sed of yaml files.

debian/copyright

@@ -1,24 +0,0 @@
-Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
-Upstream-Name: yq
-Source: https://github.com/mikefarah/yq.git
-
-Files: *
-Copyright: 2017 Mike Farah
-License: Expat
-    Permission is hereby granted, free of charge, to any person obtaining a copy
-    of this software and associated documentation files (the "Software"), to deal
-    in the Software without restriction, including without limitation the rights
-    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-    copies of the Software, and to permit persons to whom the Software is
-    furnished to do so, subject to the following conditions:
-    .
-    The above copyright notice and this permission notice shall be included in all
-    copies or substantial portions of the Software.
-    .
-    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-    SOFTWARE.

debian/gbp.conf

@@ -1,2 +0,0 @@
-[DEFAULT]
-pristine-tar = True

debian/rules

@@ -1,57 +0,0 @@
-#!/usr/bin/make -f
-#
-# Copyright (C) 2018 Roberto Mier Escandón <rmescandon@gmail.com>
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 3 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-PROJECT := yq
-OWNER := mikefarah
-REPO := github.com
-GOVERSION := 1.13
-
-export DH_OPTIONS
-export DH_GOPKG := ${REPO}/${OWNER}/${PROJECT}
-export GOROOT := /usr/lib/go-${GOVERSION}
-export GOPATH := ${CURDIR}/_build
-export GOBIN := ${GOPATH}/bin
-export PATH := ${GOROOT}/bin:${GOBIN}:${PATH}
-export GOCACHE := /tmp/gocache
-export GOFLAGS := -mod=vendor
-
-SRCDIR := ${GOPATH}/src/${DH_GOPKG}
-DESTDIR := ${CURDIR}/debian/${PROJECT}
-BINDIR := /usr/bin
-ASSETSDIR := /usr/share/${PROJECT}
-
-%:
-	dh $@ --builddirectory=${GOPATH} --buildsystem=golang
-
-override_dh_auto_build:
-	mkdir -p ${SRCDIR}
-	mkdir -p ${GOBIN}
-	# copy project to local srcdir to build from there
-	rsync -avz --progress --exclude=_build --exclude=debian --exclude=tmp. --exclude=go.mod --exclude=docs . $(SRCDIR)
-	# build go code
-	(cd ${SRCDIR} && go install -buildmode=pie ./...)
-	
-override_dh_auto_test:
-	(cd ${SRCDIR} && go test -v ./...)
-
-override_dh_auto_install:
-	cp ${GOBIN}/yq ${DESTDIR}/${BINDIR}
-	cp -f ${SRCDIR}/LICENSE ${DESTDIR}/${ASSETSDIR}
-	chmod a+x ${DESTDIR}/${BINDIR}/yq
-
-override_dh_auto_clean:
-	dh_clean
-	rm -rf ${CURDIR}/_build

debian/source/format

@@ -1 +0,0 @@
-3.0 (native)

debian/yq.dirs

@@ -1,3 +0,0 @@
-usr/bin
-usr/share/yq
-usr/share/man/man1

docs/404.html

@@ -1,268 +0,0 @@
-
-
-
-
-<!doctype html>
-<html lang="en" class="no-js">
-  <head>
-    
-      <meta charset="utf-8">
-      <meta name="viewport" content="width=device-width,initial-scale=1">
-      <meta http-equiv="x-ua-compatible" content="ie=edge">
-      
-      
-      
-      
-        <meta name="lang:clipboard.copy" content="Copy to clipboard">
-      
-        <meta name="lang:clipboard.copied" content="Copied to clipboard">
-      
-        <meta name="lang:search.language" content="en">
-      
-        <meta name="lang:search.pipeline.stopwords" content="True">
-      
-        <meta name="lang:search.pipeline.trimmer" content="True">
-      
-        <meta name="lang:search.result.none" content="No matching documents">
-      
-        <meta name="lang:search.result.one" content="1 matching document">
-      
-        <meta name="lang:search.result.other" content="# matching documents">
-      
-        <meta name="lang:search.tokenizer" content="[\s\-]+">
-      
-      <link rel="shortcut icon" href="/assets/images/favicon.png">
-      <meta name="generator" content="mkdocs-1.0.4, mkdocs-material-4.6.0">
-    
-    
-      
-        <title>Yq</title>
-      
-    
-    
-      <link rel="stylesheet" href="/assets/stylesheets/application.1b62728e.css">
-      
-      
-    
-    
-      <script src="/assets/javascripts/modernizr.268332fc.js"></script>
-    
-    
-      
-        <link href="https://fonts.gstatic.com" rel="preconnect" crossorigin>
-        <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,400,400i,700%7CRoboto+Mono&display=fallback">
-        <style>body,input{font-family:"Roboto","Helvetica Neue",Helvetica,Arial,sans-serif}code,kbd,pre{font-family:"Roboto Mono","Courier New",Courier,monospace}</style>
-      
-    
-    <link rel="stylesheet" href="/assets/fonts/material-icons.css">
-    
-    
-    
-      
-    
-    
-  </head>
-  
-    <body dir="ltr">
-  
-    <svg class="md-svg">
-      <defs>
-        
-        
-          <svg xmlns="http://www.w3.org/2000/svg" width="416" height="448" viewBox="0 0 416 448" id="__github"><path fill="currentColor" d="M160 304q0 10-3.125 20.5t-10.75 19T128 352t-18.125-8.5-10.75-19T96 304t3.125-20.5 10.75-19T128 256t18.125 8.5 10.75 19T160 304zm160 0q0 10-3.125 20.5t-10.75 19T288 352t-18.125-8.5-10.75-19T256 304t3.125-20.5 10.75-19T288 256t18.125 8.5 10.75 19T320 304zm40 0q0-30-17.25-51T296 232q-10.25 0-48.75 5.25Q229.5 240 208 240t-39.25-2.75Q130.75 232 120 232q-29.5 0-46.75 21T56 304q0 22 8 38.375t20.25 25.75 30.5 15 35 7.375 37.25 1.75h42q20.5 0 37.25-1.75t35-7.375 30.5-15 20.25-25.75T360 304zm56-44q0 51.75-15.25 82.75-9.5 19.25-26.375 33.25t-35.25 21.5-42.5 11.875-42.875 5.5T212 416q-19.5 0-35.5-.75t-36.875-3.125-38.125-7.5-34.25-12.875T37 371.5t-21.5-28.75Q0 312 0 260q0-59.25 34-99-6.75-20.5-6.75-42.5 0-29 12.75-54.5 27 0 47.5 9.875t47.25 30.875Q171.5 96 212 96q37 0 70 8 26.25-20.5 46.75-30.25T376 64q12.75 25.5 12.75 54.5 0 21.75-6.75 42 34 40 34 99.5z"/></svg>
-        
-      </defs>
-    </svg>
-    <input class="md-toggle" data-md-toggle="drawer" type="checkbox" id="__drawer" autocomplete="off">
-    <input class="md-toggle" data-md-toggle="search" type="checkbox" id="__search" autocomplete="off">
-    <label class="md-overlay" data-md-component="overlay" for="__drawer"></label>
-    
-    
-      <header class="md-header" data-md-component="header">
-  <nav class="md-header-nav md-grid">
-    <div class="md-flex">
-      <div class="md-flex__cell md-flex__cell--shrink">
-        <a href="/." title="Yq" class="md-header-nav__button md-logo">
-          
-            <i class="md-icon"></i>
-          
-        </a>
-      </div>
-      <div class="md-flex__cell md-flex__cell--shrink">
-        <label class="md-icon md-icon--menu md-header-nav__button" for="__drawer"></label>
-      </div>
-      <div class="md-flex__cell md-flex__cell--stretch">
-        <div class="md-flex__ellipsis md-header-nav__title" data-md-component="title">
-          
-            <span class="md-header-nav__topic">
-              Yq
-            </span>
-            <span class="md-header-nav__topic">
-              
-                
-              
-            </span>
-          
-        </div>
-      </div>
-      <div class="md-flex__cell md-flex__cell--shrink">
-        
-          <label class="md-icon md-icon--search md-header-nav__button" for="__search"></label>
-          
-<div class="md-search" data-md-component="search" role="dialog">
-  <label class="md-search__overlay" for="__search"></label>
-  <div class="md-search__inner" role="search">
-    <form class="md-search__form" name="search">
-      <input type="text" class="md-search__input" name="query" placeholder="Search" autocapitalize="off" autocorrect="off" autocomplete="off" spellcheck="false" data-md-component="query" data-md-state="active">
-      <label class="md-icon md-search__icon" for="__search"></label>
-      <button type="reset" class="md-icon md-search__icon" data-md-component="reset" tabindex="-1">
-        &#xE5CD;
-      </button>
-    </form>
-    <div class="md-search__output">
-      <div class="md-search__scrollwrap" data-md-scrollfix>
-        <div class="md-search-result" data-md-component="result">
-          <div class="md-search-result__meta">
-            Type to start searching
-          </div>
-          <ol class="md-search-result__list"></ol>
-        </div>
-      </div>
-    </div>
-  </div>
-</div>
-        
-      </div>
-      
-        <div class="md-flex__cell md-flex__cell--shrink">
-          <div class="md-header-nav__source">
-            
-
-
-  
-
-<a href="https://github.com/mikefarah/yq/" title="Go to repository" class="md-source" data-md-source="github">
-  
-    <div class="md-source__icon">
-      <svg viewBox="0 0 24 24" width="24" height="24">
-        <use xlink:href="#__github" width="24" height="24"></use>
-      </svg>
-    </div>
-  
-  <div class="md-source__repository">
-    mikefarah/yq
-  </div>
-</a>
-          </div>
-        </div>
-      
-    </div>
-  </nav>
-</header>
-    
-    <div class="md-container">
-      
-        
-      
-      
-      <main class="md-main" role="main">
-        <div class="md-main__inner md-grid" data-md-component="container">
-          
-            
-              <div class="md-sidebar md-sidebar--primary" data-md-component="navigation">
-                <div class="md-sidebar__scrollwrap">
-                  <div class="md-sidebar__inner">
-                    <nav class="md-nav md-nav--primary" data-md-level="0">
-  <label class="md-nav__title md-nav__title--site" for="__drawer">
-    <a href="/." title="Yq" class="md-nav__button md-logo">
-      
-        <i class="md-icon"></i>
-      
-    </a>
-    Yq
-  </label>
-  
-    <div class="md-nav__source">
-      
-
-
-  
-
-<a href="https://github.com/mikefarah/yq/" title="Go to repository" class="md-source" data-md-source="github">
-  
-    <div class="md-source__icon">
-      <svg viewBox="0 0 24 24" width="24" height="24">
-        <use xlink:href="#__github" width="24" height="24"></use>
-      </svg>
-    </div>
-  
-  <div class="md-source__repository">
-    mikefarah/yq
-  </div>
-</a>
-    </div>
-  
-  <ul class="md-nav__list" data-md-scrollfix>
-    
-      
-      
-      
-
-
-  <li class="md-nav__item">
-    <a href="/." title="Home" class="md-nav__link">
-      Home
-    </a>
-  </li>
-
-    
-  </ul>
-</nav>
-                  </div>
-                </div>
-              </div>
-            
-            
-          
-          <div class="md-content">
-            <article class="md-content__inner md-typeset">
-              
-  <h1>404 - Not found</h1>
-
-              
-                
-
-
-              
-            </article>
-          </div>
-        </div>
-      </main>
-      
-        
-<footer class="md-footer">
-  
-  <div class="md-footer-meta md-typeset">
-    <div class="md-footer-meta__inner md-grid">
-      <div class="md-footer-copyright">
-        
-        powered by
-        <a href="https://www.mkdocs.org">MkDocs</a>
-        and
-        <a href="https://squidfunk.github.io/mkdocs-material/">
-          Material for MkDocs</a>
-      </div>
-      
-    </div>
-  </div>
-</footer>
-      
-    </div>
-    
-      <script src="/assets/javascripts/application.808e90bb.js"></script>
-      
-      <script>app.initialize({version:"1.0.4",url:{base:"/"}})</script>
-      
-    
-  </body>
-</html>

docs/assets/fonts/material-icons.css

@@ -1,13 +0,0 @@
-/*!
- * Licensed under the Apache License, Version 2.0 (the "License"); you may not
- * use this file except in compliance with the License. You may obtain a copy
- * of the License at:
- *
- *   http://www.apache.org/licenses/LICENSE-2.0
- *
- * UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING, SOFTWARE
- * DISTRIBUTED UNDER THE LICENSE IS DISTRIBUTED ON AN "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED.
- * SEE THE LICENSE FOR THE SPECIFIC LANGUAGE GOVERNING PERMISSIONS AND
- * LIMITATIONS UNDER THE LICENSE.
- */@font-face{font-family:"Material Icons";font-style:normal;font-weight:400;src:local("Material Icons"),local("MaterialIcons-Regular"),url("specimen/MaterialIcons-Regular.woff2") format("woff2"),url("specimen/MaterialIcons-Regular.woff") format("woff"),url("specimen/MaterialIcons-Regular.ttf") format("truetype")}

docs/assets/images/icons/bitbucket.1b09e088.svg

@@ -1 +0,0 @@
-<svg xmlns="http://www.w3.org/2000/svg" width="352" height="448" viewBox="0 0 352 448" id="__bitbucket"><path fill="currentColor" d="M203.75 214.75q2 15.75-12.625 25.25t-27.875 1.5q-9.75-4.25-13.375-14.5t-.125-20.5 13-14.5q9-4.5 18.125-3t16 8.875 6.875 16.875zm27.75-5.25q-3.5-26.75-28.25-41T154 165.25q-15.75 7-25.125 22.125t-8.625 32.375q1 22.75 19.375 38.75t41.375 14q22.75-2 38-21t12.5-42zM291.25 74q-5-6.75-14-11.125t-14.5-5.5T245 54.25q-72.75-11.75-141.5.5-10.75 1.75-16.5 3t-13.75 5.5T60.75 74q7.5 7 19 11.375t18.375 5.5T120 93.75Q177 101 232 94q15.75-2 22.375-3t18.125-5.375T291.25 74zm14.25 258.75q-2 6.5-3.875 19.125t-3.5 21-7.125 17.5-14.5 14.125q-21.5 12-47.375 17.875t-50.5 5.5-50.375-4.625q-11.5-2-20.375-4.5T88.75 412 70.5 401.125t-13-15.375q-6.25-24-14.25-73l1.5-4 4.5-2.25q55.75 37 126.625 37t126.875-37q5.25 1.5 6 5.75t-1.25 11.25-2 9.25zM350.75 92.5q-6.5 41.75-27.75 163.75-1.25 7.5-6.75 14t-10.875 10T291.75 288q-63 31.5-152.5 22-62-6.75-98.5-34.75-3.75-3-6.375-6.625t-4.25-8.75-2.25-8.5-1.5-9.875T25 232.75q-2.25-12.5-6.625-37.5t-7-40.375T5.5 118 0 78.5Q.75 72 4.375 66.375T12.25 57t11.25-7.5T35 43.875t12-4.625q31.25-11.5 78.25-16 94.75-9.25 169 12.5Q333 47.25 348 66.25q4 5 4.125 12.75t-1.375 13.5z"/></svg>

docs/assets/images/icons/github.f0b8504a.svg

@@ -1 +0,0 @@
-<svg xmlns="http://www.w3.org/2000/svg" width="416" height="448" viewBox="0 0 416 448" id="__github"><path fill="currentColor" d="M160 304q0 10-3.125 20.5t-10.75 19T128 352t-18.125-8.5-10.75-19T96 304t3.125-20.5 10.75-19T128 256t18.125 8.5 10.75 19T160 304zm160 0q0 10-3.125 20.5t-10.75 19T288 352t-18.125-8.5-10.75-19T256 304t3.125-20.5 10.75-19T288 256t18.125 8.5 10.75 19T320 304zm40 0q0-30-17.25-51T296 232q-10.25 0-48.75 5.25Q229.5 240 208 240t-39.25-2.75Q130.75 232 120 232q-29.5 0-46.75 21T56 304q0 22 8 38.375t20.25 25.75 30.5 15 35 7.375 37.25 1.75h42q20.5 0 37.25-1.75t35-7.375 30.5-15 20.25-25.75T360 304zm56-44q0 51.75-15.25 82.75-9.5 19.25-26.375 33.25t-35.25 21.5-42.5 11.875-42.875 5.5T212 416q-19.5 0-35.5-.75t-36.875-3.125-38.125-7.5-34.25-12.875T37 371.5t-21.5-28.75Q0 312 0 260q0-59.25 34-99-6.75-20.5-6.75-42.5 0-29 12.75-54.5 27 0 47.5 9.875t47.25 30.875Q171.5 96 212 96q37 0 70 8 26.25-20.5 46.75-30.25T376 64q12.75 25.5 12.75 54.5 0 21.75-6.75 42 34 40 34 99.5z"/></svg>

docs/assets/images/icons/gitlab.6dd19c00.svg

@@ -1 +0,0 @@
-<svg xmlns="http://www.w3.org/2000/svg" width="500" height="500" viewBox="0 0 500 500" id="__gitlab"><path fill="currentColor" d="M93.667 473.347l90.684-279.097H2.983l90.684 279.097z" transform="translate(156.198 1.16)"/><path fill="currentColor" d="M221.333 473.345L130.649 194.25H3.557l217.776 279.095z" transform="translate(28.531 1.16)" opacity=".7"/><path fill="currentColor" d="M32 195.155L4.441 279.97a18.773 18.773 0 0 0 6.821 20.99l238.514 173.29L32 195.155z" transform="translate(.089 .256)" opacity=".5"/><path fill="currentColor" d="M2.667-84.844h127.092L75.14-252.942c-2.811-8.649-15.047-8.649-17.856 0L2.667-84.844z" transform="translate(29.422 280.256)"/><path fill="currentColor" d="M2.667 473.345L93.351 194.25h127.092L2.667 473.345z" transform="translate(247.198 1.16)" opacity=".7"/><path fill="currentColor" d="M221.334 195.155l27.559 84.815a18.772 18.772 0 0 1-6.821 20.99L3.557 474.25l217.777-279.095z" transform="translate(246.307 .256)" opacity=".5"/><path fill="currentColor" d="M130.667-84.844H3.575l54.618-168.098c2.811-8.649 15.047-8.649 17.856 0l54.618 168.098z" transform="translate(336.974 280.256)"/></svg>

docs/assets/javascripts/lunr/lunr.da.js

@@ -1,17 +0,0 @@
-/*!
- * Lunr languages, `Danish` language
- * https://github.com/MihaiValentin/lunr-languages
- *
- * Copyright 2014, Mihai Valentin
- * http://www.mozilla.org/MPL/
- */
-/*!
- * based on
- * Snowball JavaScript Library v0.3
- * http://code.google.com/p/urim/
- * http://snowball.tartarus.org/
- *
- * Copyright 2010, Oleg Mazko
- * http://www.mozilla.org/MPL/
- */
-!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");var r,m,i;e.da=function(){this.pipeline.reset(),this.pipeline.add(e.da.trimmer,e.da.stopWordFilter,e.da.stemmer),this.searchPipeline&&(this.searchPipeline.reset(),this.searchPipeline.add(e.da.stemmer))},e.da.wordCharacters="A-Za-zªºÀ-ÖØ-öø-ʸˠ-ˤᴀ-ᴥᴬ-ᵜᵢ-ᵥᵫ-ᵷᵹ-ᶾḀ-ỿⁱⁿₐ-ₜKÅℲⅎⅠ-ↈⱠ-ⱿꜢ-ꞇꞋ-ꞭꞰ-ꞷꟷ-ꟿꬰ-ꭚꭜ-ꭤff-stＡ-Ｚａ-ｚ",e.da.trimmer=e.trimmerSupport.generateTrimmer(e.da.wordCharacters),e.Pipeline.registerFunction(e.da.trimmer,"trimmer-da"),e.da.stemmer=(r=e.stemmerSupport.Among,m=e.stemmerSupport.SnowballProgram,i=new function(){var i,t,n,s=[new r("hed",-1,1),new r("ethed",0,1),new r("ered",-1,1),new r("e",-1,1),new r("erede",3,1),new r("ende",3,1),new r("erende",5,1),new r("ene",3,1),new r("erne",3,1),new r("ere",3,1),new r("en",-1,1),new r("heden",10,1),new r("eren",10,1),new r("er",-1,1),new r("heder",13,1),new r("erer",13,1),new r("s",-1,2),new r("heds",16,1),new r("es",16,1),new r("endes",18,1),new r("erendes",19,1),new r("enes",18,1),new r("ernes",18,1),new r("eres",18,1),new r("ens",16,1),new r("hedens",24,1),new r("erens",24,1),new r("ers",16,1),new r("ets",16,1),new r("erets",28,1),new r("et",-1,1),new r("eret",30,1)],o=[new r("gd",-1,-1),new r("dt",-1,-1),new r("gt",-1,-1),new r("kt",-1,-1)],a=[new r("ig",-1,1),new r("lig",0,1),new r("elig",1,1),new r("els",-1,1),new r("løst",-1,2)],d=[17,65,16,1,0,0,0,0,0,0,0,0,0,0,0,0,48,0,128],u=[239,254,42,3,0,0,0,0,0,0,0,0,0,0,0,0,16],c=new m;function l(){var e,r=c.limit-c.cursor;c.cursor>=t&&(e=c.limit_backward,c.limit_backward=t,c.ket=c.cursor,c.find_among_b(o,4)?(c.bra=c.cursor,c.limit_backward=e,c.cursor=c.limit-r,c.cursor>c.limit_backward&&(c.cursor--,c.bra=c.cursor,c.slice_del())):c.limit_backward=e)}this.setCurrent=function(e){c.setCurrent(e)},this.getCurrent=function(){return c.getCurrent()},this.stem=function(){var e,r=c.cursor;return function(){var e,r=c.cursor+3;if(t=c.limit,0<=r&&r<=c.limit){for(i=r;;){if(e=c.cursor,c.in_grouping(d,97,248)){c.cursor=e;break}if((c.cursor=e)>=c.limit)return;c.cursor++}for(;!c.out_grouping(d,97,248);){if(c.cursor>=c.limit)return;c.cursor++}(t=c.cursor)<i&&(t=i)}}(),c.limit_backward=r,c.cursor=c.limit,function(){var e,r;if(c.cursor>=t&&(r=c.limit_backward,c.limit_backward=t,c.ket=c.cursor,e=c.find_among_b(s,32),c.limit_backward=r,e))switch(c.bra=c.cursor,e){case 1:c.slice_del();break;case 2:c.in_grouping_b(u,97,229)&&c.slice_del()}}(),c.cursor=c.limit,l(),c.cursor=c.limit,function(){var e,r,i,n=c.limit-c.cursor;if(c.ket=c.cursor,c.eq_s_b(2,"st")&&(c.bra=c.cursor,c.eq_s_b(2,"ig")&&c.slice_del()),c.cursor=c.limit-n,c.cursor>=t&&(r=c.limit_backward,c.limit_backward=t,c.ket=c.cursor,e=c.find_among_b(a,5),c.limit_backward=r,e))switch(c.bra=c.cursor,e){case 1:c.slice_del(),i=c.limit-c.cursor,l(),c.cursor=c.limit-i;break;case 2:c.slice_from("løs")}}(),c.cursor=c.limit,c.cursor>=t&&(e=c.limit_backward,c.limit_backward=t,c.ket=c.cursor,c.out_grouping_b(d,97,248)?(c.bra=c.cursor,n=c.slice_to(n),c.limit_backward=e,c.eq_v_b(n)&&c.slice_del()):c.limit_backward=e),!0}},function(e){return"function"==typeof e.update?e.update(function(e){return i.setCurrent(e),i.stem(),i.getCurrent()}):(i.setCurrent(e),i.stem(),i.getCurrent())}),e.Pipeline.registerFunction(e.da.stemmer,"stemmer-da"),e.da.stopWordFilter=e.generateStopWordFilter("ad af alle alt anden at blev blive bliver da de dem den denne der deres det dette dig din disse dog du efter eller en end er et for fra ham han hans har havde have hende hendes her hos hun hvad hvis hvor i ikke ind jeg jer jo kunne man mange med meget men mig min mine mit mod ned noget nogle nu når og også om op os over på selv sig sin sine sit skal skulle som sådan thi til ud under var vi vil ville vor være været".split(" ")),e.Pipeline.registerFunction(e.da.stopWordFilter,"stopWordFilter-da")}});