From fc8a3fc3ce2bc794d476dcedfe9bec06652dd323 Mon Sep 17 00:00:00 2001 From: Tommy Malmqvist Date: Tue, 16 Sep 2025 15:56:56 +0200 Subject: [PATCH 1/4] Enhance GitHub Action with custom registry support and input parameters. Updated action.yml to include optional inputs for image, registry, and authentication credentials. Refactored to use a composite action for improved container management. Updated README.md to reflect new features and usage examples. --- README.md | 212 +++++++++++++++++++++++++++++++++++++++++------------ action.yml | 71 ++++++++++++++++-- 2 files changed, 227 insertions(+), 56 deletions(-) diff --git a/README.md b/README.md index 21604ddb..543983f8 100644 --- a/README.md +++ b/README.md @@ -1,35 +1,49 @@ # yq -![Build](https://github.com/mikefarah/yq/workflows/Build/badge.svg) ![Docker Pulls](https://img.shields.io/docker/pulls/mikefarah/yq.svg) ![Github Releases (by Release)](https://img.shields.io/github/downloads/mikefarah/yq/total.svg) ![Go Report](https://goreportcard.com/badge/github.com/mikefarah/yq) ![CodeQL](https://github.com/mikefarah/yq/workflows/CodeQL/badge.svg) +![Build](https://github.com/mikefarah/yq/workflows/Build/badge.svg) +![Docker Pulls](https://img.shields.io/docker/pulls/mikefarah/yq.svg) +![Github Releases (by Release)](https://img.shields.io/github/downloads/mikefarah/yq/total.svg) +![Go Report](https://goreportcard.com/badge/github.com/mikefarah/yq) +![CodeQL](https://github.com/mikefarah/yq/workflows/CodeQL/badge.svg) +a lightweight and portable command-line YAML, JSON, INI and XML processor. `yq` +uses [jq](https://github.com/stedolan/jq) like syntax but works with yaml files +as well as json, xml, ini, properties, csv and tsv. It doesn't yet support +everything `jq` does - but it does support the most common operations and +functions, and more is being added continuously. -a lightweight and portable command-line YAML, JSON, INI and XML processor. `yq` uses [jq](https://github.com/stedolan/jq) like syntax but works with yaml files as well as json, xml, ini, properties, csv and tsv. It doesn't yet support everything `jq` does - but it does support the most common operations and functions, and more is being added continuously. - -yq is written in go - so you can download a dependency free binary for your platform and you are good to go! If you prefer there are a variety of package managers that can be used as well as Docker and Podman, all listed below. +yq is written in go - so you can download a dependency free binary for your +platform and you are good to go! If you prefer there are a variety of package +managers that can be used as well as Docker and Podman, all listed below. ## Quick Usage Guide Read a value: + ```bash yq '.a.b[0].c' file.yaml ``` Pipe from STDIN: + ```bash yq '.a.b[0].c' < file.yaml ``` Update a yaml file, in place + ```bash yq -i '.a.b[0].c = "cool"' file.yaml ``` Update using environment variables + ```bash NAME=mike yq -i '.a.b[0].c = strenv(NAME)' file.yaml ``` Merge multiple files + ```bash # merge two files yq -n 'load("file1.yaml") * load("file2.yaml")' @@ -41,6 +55,7 @@ yq ea '. as $item ireduce ({}; . * $item )' path/to/*.yml ``` Multiple updates to a yaml file + ```bash yq -i ' .a.b[0].c = "cool" | @@ -50,30 +65,37 @@ yq -i ' ``` Find and update an item in an array: + ```bash yq '(.[] | select(.name == "foo") | .address) = "12 cat st"' ``` Convert JSON to YAML + ```bash yq -Poy sample.json ``` -See [recipes](https://mikefarah.gitbook.io/yq/recipes) for more examples and the [documentation](https://mikefarah.gitbook.io/yq/) for more information. +See [recipes](https://mikefarah.gitbook.io/yq/recipes) for more examples and the +[documentation](https://mikefarah.gitbook.io/yq/) for more information. -Take a look at the discussions for [common questions](https://github.com/mikefarah/yq/discussions/categories/q-a), and [cool ideas](https://github.com/mikefarah/yq/discussions/categories/show-and-tell) +Take a look at the discussions for +[common questions](https://github.com/mikefarah/yq/discussions/categories/q-a), +and +[cool ideas](https://github.com/mikefarah/yq/discussions/categories/show-and-tell) ## Install ### [Download the latest binary](https://github.com/mikefarah/yq/releases/latest) ### wget -Use wget to download, gzipped pre-compiled binaries: +Use wget to download, gzipped pre-compiled binaries: For instance, VERSION=v4.2.0 and BINARY=yq_linux_amd64 #### Compressed via tar.gz + ```bash wget https://github.com/mikefarah/yq/releases/download/${VERSION}/${BINARY}.tar.gz -O - |\ tar xz && mv ${BINARY} /usr/local/bin/yq @@ -94,28 +116,39 @@ wget https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -O ``` ### MacOS / Linux via Homebrew: + Using [Homebrew](https://brew.sh/) + ``` brew install yq ``` ### Linux via snap: + ``` snap install yq ``` #### Snap notes -`yq` installs with [_strict confinement_](https://docs.snapcraft.io/snap-confinement/6233) in snap, this means it doesn't have direct access to root files. To read root files you can: + +`yq` installs with +[_strict confinement_](https://docs.snapcraft.io/snap-confinement/6233) in snap, +this means it doesn't have direct access to root files. To read root files you +can: ``` sudo cat /etc/myfile | yq '.a.path' ``` -And to write to a root file you can either use [sponge](https://linux.die.net/man/1/sponge): +And to write to a root file you can either use +[sponge](https://linux.die.net/man/1/sponge): + ``` sudo cat /etc/myfile | yq '.a.path = "value"' | sudo sponge /etc/myfile ``` + or write to a temporary file: + ``` sudo cat /etc/myfile | yq '.a.path = "value"' | sudo tee /etc/myfile.tmp sudo mv /etc/myfile.tmp /etc/myfile @@ -123,14 +156,16 @@ rm /etc/myfile.tmp ``` ### Run with Docker or Podman + #### Oneshot use: ```bash docker run --rm -v "${PWD}":/workdir mikefarah/yq [command] [flags] [expression ]FILE... ``` -Note that you can run `yq` in docker without network access and other privileges if you desire, -namely `--security-opt=no-new-privileges --cap-drop all --network none`. +Note that you can run `yq` in docker without network access and other privileges +if you desire, namely +`--security-opt=no-new-privileges --cap-drop all --network none`. ```bash podman run --rm -v "${PWD}":/workdir mikefarah/yq [command] [flags] [expression ]FILE... @@ -158,7 +193,8 @@ docker run --rm -it -v "${PWD}":/workdir --entrypoint sh mikefarah/yq podman run --rm -it -v "${PWD}":/workdir --entrypoint sh mikefarah/yq ``` -It can be useful to have a bash function to avoid typing the whole docker command: +It can be useful to have a bash function to avoid typing the whole docker +command: ```bash yq() { @@ -171,10 +207,13 @@ yq() { podman run --rm -i -v "${PWD}":/workdir mikefarah/yq "$@" } ``` + #### Running as root: -`yq`'s container image no longer runs under root (https://github.com/mikefarah/yq/pull/860). If you'd like to install more things in the container image, or you're having permissions issues when attempting to read/write files you'll need to either: - +`yq`'s container image no longer runs under root +(https://github.com/mikefarah/yq/pull/860). If you'd like to install more things +in the container image, or you're having permissions issues when attempting to +read/write files you'll need to either: ``` docker run --user="root" -it --entrypoint sh mikefarah/yq @@ -195,7 +234,9 @@ USER yq ``` #### Missing timezone data -By default, the alpine image yq uses does not include timezone data. If you'd like to use the `tz` operator, you'll need to include this data: + +By default, the alpine image yq uses does not include timezone data. If you'd +like to use the `tz` operator, you'll need to include this data: ``` FROM mikefarah/yq @@ -207,45 +248,90 @@ USER yq #### Podman with SELinux -If you are using podman with SELinux, you will need to set the shared volume flag `:z` on the volume mount: +If you are using podman with SELinux, you will need to set the shared volume +flag `:z` on the volume mount: ``` -v "${PWD}":/workdir:z ``` ### GitHub Action + ``` - - name: Set foobar to cool - uses: mikefarah/yq@master - with: - cmd: yq -i '.foo.bar = "cool"' 'config.yml' - - name: Get an entry with a variable that might contain dots or spaces - id: get_username - uses: mikefarah/yq@master - with: - cmd: yq '.all.children.["${{ matrix.ip_address }}"].username' ops/inventories/production.yml - - name: Reuse a variable obtained in another step - run: echo ${{ steps.get_username.outputs.result }} +- name: Set foobar to cool + uses: mikefarah/yq@master + with: + cmd: yq -i '.foo.bar = "cool"' 'config.yml' +- name: Get an entry with a variable that might contain dots or spaces + id: get_username + uses: mikefarah/yq@master + with: + cmd: yq '.all.children.["${{ matrix.ip_address }}"].username' ops/inventories/production.yml +- name: Reuse a variable obtained in another step + run: echo ${{ steps.get_username.outputs.result }} ``` See https://mikefarah.gitbook.io/yq/usage/github-action for more. +### Action: custom registry support + +The action now accepts optional inputs to control which container image is used +and (optionally) authenticate to a registry: + +- `image`: the image name to run (default `mikefarah/yq:4-githubaction`). +- `registry`: optional artifact repository hostname/prefix to be prepended to + `image` (for example `artifacts.example.com/repository/docker`). +- `registry_username` / `registry_password`: optional credentials to login + before pulling the image. + +Important caveat: GitHub-hosted runners pull action container images (when using +`runs.using: docker`) before any workflow steps run. To make +registry-authenticated pulls work on GitHub-hosted runners, either: + +- Use the composite action (this repository) which performs a `docker login` + inside the action before running the container (works when credentials are + provided via action inputs). Note that some registries block pulls even with + login if the runner cannot reach them. +- Use self-hosted runners configured to use your artifact repository as a + pull-through cache or pre-pull the image on the runner. +- Alternatively, avoid using the action container image and instead + `docker login` and `docker run` from a regular `run:` step in your workflow + (see example below). + +Example using the composite action (local action reference): + +``` +- name: Run yq from artifact repository + uses: . + with: + registry: artifacts.example.com/repository/docker + image: mikefarah/yq:4-githubaction + registry_username: ${{ secrets.ARTIFACT_REPO_USER }} + registry_password: ${{ secrets.ARTIFACT_REPO_PASS }} + cmd: yq --version +``` + ### Go Install: + ``` go install github.com/mikefarah/yq/v4@latest ``` ## Community Supported Installation methods -As these are supported by the community :heart: - however, they may be out of date with the officially supported releases. -_Please note that the Debian package (previously supported by @rmescandon) is no longer maintained. Please use an alternative installation method._ +As these are supported by the community :heart: - however, they may be out of +date with the officially supported releases. +_Please note that the Debian package (previously supported by @rmescandon) is no +longer maintained. Please use an alternative installation method._ ### X-CMD + Checkout `yq` on x-cmd: https://x-cmd.com/mod/yq - Instant Results: See the output of your yq filter in real-time. -- Error Handling: Encounter a syntax error? It will display the error message and the results of the closest valid filter +- Error Handling: Encounter a syntax error? It will display the error message + and the results of the closest valid filter Thanks @edwinjhlee! @@ -255,8 +341,8 @@ Thanks @edwinjhlee! nix profile install nixpkgs#yq-go ``` -See [here](https://search.nixos.org/packages?channel=unstable&show=yq-go&from=0&size=50&sort=relevance&type=packages&query=yq-go) - +See +[here](https://search.nixos.org/packages?channel=unstable&show=yq-go&from=0&size=50&sort=relevance&type=packages&query=yq-go) ### Webi @@ -264,8 +350,8 @@ See [here](https://search.nixos.org/packages?channel=unstable&show=yq-go&from=0& webi yq ``` -See [webi](https://webinstall.dev/) -Supported by @adithyasunil26 (https://github.com/webinstall/webi-installers/tree/master/yq) +See [webi](https://webinstall.dev/) Supported by @adithyasunil26 +(https://github.com/webinstall/webi-installers/tree/master/yq) ### Arch Linux @@ -279,37 +365,47 @@ Using [Chocolatey](https://chocolatey.org) [![Chocolatey](https://img.shields.io/chocolatey/v/yq.svg)](https://chocolatey.org/packages/yq) [![Chocolatey](https://img.shields.io/chocolatey/dt/yq.svg)](https://chocolatey.org/packages/yq) + ``` choco install yq ``` + Supported by @chillum (https://chocolatey.org/packages/yq) Using [scoop](https://scoop.sh/) + ``` scoop install main/yq ``` Using [winget](https://learn.microsoft.com/en-us/windows/package-manager/) + ``` winget install --id MikeFarah.yq ``` ### MacPorts: + Using [MacPorts](https://www.macports.org/) + ``` sudo port selfupdate sudo port install yq ``` -Supported by @herbygillot (https://ports.macports.org/maintainer/github/herbygillot) + +Supported by @herbygillot +(https://ports.macports.org/maintainer/github/herbygillot) ### Alpine Linux Alpine Linux v3.20+ (and Edge): + ``` apk add yq-go ``` Alpine Linux up to v3.19: + ``` apk add yq ``` @@ -324,8 +420,8 @@ Flox can be used to install yq on Linux, MacOS, and Windows through WSL. flox install yq ``` - ### MacOS / Linux via gah: + Using [gah](https://github.com/marverix/gah) ``` @@ -333,19 +429,29 @@ gah install yq ``` ## Features + - [Detailed documentation with many examples](https://mikefarah.gitbook.io/yq/) - Written in portable go, so you can download a lovely dependency free binary -- Uses similar syntax as `jq` but works with YAML, INI, [JSON](https://mikefarah.gitbook.io/yq/usage/convert) and [XML](https://mikefarah.gitbook.io/yq/usage/xml) files +- Uses similar syntax as `jq` but works with YAML, INI, + [JSON](https://mikefarah.gitbook.io/yq/usage/convert) and + [XML](https://mikefarah.gitbook.io/yq/usage/xml) files - Fully supports multi document yaml files -- Supports yaml [front matter](https://mikefarah.gitbook.io/yq/usage/front-matter) blocks (e.g. jekyll/assemble) +- Supports yaml + [front matter](https://mikefarah.gitbook.io/yq/usage/front-matter) blocks + (e.g. jekyll/assemble) - Colorized yaml output - [Date/Time manipulation and formatting with TZ](https://mikefarah.gitbook.io/yq/operators/datetime) - [Deeply data structures](https://mikefarah.gitbook.io/yq/operators/traverse-read) - [Sort keys](https://mikefarah.gitbook.io/yq/operators/sort-keys) -- Manipulate yaml [comments](https://mikefarah.gitbook.io/yq/operators/comment-operators), [styling](https://mikefarah.gitbook.io/yq/operators/style), [tags](https://mikefarah.gitbook.io/yq/operators/tag) and [anchors and aliases](https://mikefarah.gitbook.io/yq/operators/anchor-and-alias-operators). +- Manipulate yaml + [comments](https://mikefarah.gitbook.io/yq/operators/comment-operators), + [styling](https://mikefarah.gitbook.io/yq/operators/style), + [tags](https://mikefarah.gitbook.io/yq/operators/tag) and + [anchors and aliases](https://mikefarah.gitbook.io/yq/operators/anchor-and-alias-operators). - [Update in place](https://mikefarah.gitbook.io/yq/v/v4.x/commands/evaluate#flags) - [Complex expressions to select and update](https://mikefarah.gitbook.io/yq/operators/select#select-and-update-matching-values-in-map) -- Keeps yaml formatting and comments when updating (though there are issues with whitespace) +- Keeps yaml formatting and comments when updating (though there are issues with + whitespace) - [Decode/Encode base64 data](https://mikefarah.gitbook.io/yq/operators/encode-decode) - [Load content from other files](https://mikefarah.gitbook.io/yq/operators/load) - [Convert to/from json/ndjson](https://mikefarah.gitbook.io/yq/v/v4.x/usage/convert) @@ -353,12 +459,15 @@ gah install yq - [Convert to/from properties](https://mikefarah.gitbook.io/yq/v/v4.x/usage/properties) - [Convert to/from csv/tsv](https://mikefarah.gitbook.io/yq/usage/csv-tsv) - [General shell completion scripts (bash/zsh/fish/powershell)](https://mikefarah.gitbook.io/yq/v/v4.x/commands/shell-completion) -- [Reduce](https://mikefarah.gitbook.io/yq/operators/reduce) to merge multiple files or sum an array or other fancy things. -- [Github Action](https://mikefarah.gitbook.io/yq/usage/github-action) to use in your automated pipeline (thanks @devorbitus) +- [Reduce](https://mikefarah.gitbook.io/yq/operators/reduce) to merge multiple + files or sum an array or other fancy things. +- [Github Action](https://mikefarah.gitbook.io/yq/usage/github-action) to use in + your automated pipeline (thanks @devorbitus) ## [Usage](https://mikefarah.gitbook.io/yq/) -Check out the [documentation](https://mikefarah.gitbook.io/yq/) for more detailed and advanced usage. +Check out the [documentation](https://mikefarah.gitbook.io/yq/) for more +detailed and advanced usage. ``` Usage: @@ -423,9 +532,16 @@ Flags: Use "yq [command] --help" for more information about a command. ``` -## Known Issues / Missing Features -- `yq` attempts to preserve comment positions and whitespace as much as possible, but it does not handle all scenarios (see https://github.com/go-yaml/yaml/tree/v3 for details) -- Powershell has its own...[opinions on quoting yq](https://mikefarah.gitbook.io/yq/usage/tips-and-tricks#quotes-in-windows-powershell) -- "yes", "no" were dropped as boolean values in the yaml 1.2 standard - which is the standard yq assumes. -See [tips and tricks](https://mikefarah.gitbook.io/yq/usage/tips-and-tricks) for more common problems and solutions. +## Known Issues / Missing Features + +- `yq` attempts to preserve comment positions and whitespace as much as + possible, but it does not handle all scenarios (see + https://github.com/go-yaml/yaml/tree/v3 for details) +- Powershell has its + own...[opinions on quoting yq](https://mikefarah.gitbook.io/yq/usage/tips-and-tricks#quotes-in-windows-powershell) +- "yes", "no" were dropped as boolean values in the yaml 1.2 standard - which is + the standard yq assumes. + +See [tips and tricks](https://mikefarah.gitbook.io/yq/usage/tips-and-tricks) for +more common problems and solutions. diff --git a/action.yml b/action.yml index 3ff32f57..efc03136 100644 --- a/action.yml +++ b/action.yml @@ -1,17 +1,72 @@ -name: 'yq - portable yaml processor' -description: 'create, read, update, delete, merge, validate and do more with yaml' +name: "yq - portable yaml processor" +description: "create, read, update, delete, merge, validate and do more with yaml" branding: icon: command color: gray-dark inputs: + image: + description: 'Container image to run. Example: "mikefarah/yq:4-githubaction" or fully qualified "artifacts.example.com/repo/mikefarah/yq:4-githubaction".' + required: false + default: "mikefarah/yq:4-githubaction" + registry: + description: "Optional artifact repository hostname to prefix the `image`. Leave empty if your `image` already includes a registry." + required: false + default: "" + registry_username: + description: "Optional registry username for `docker login` (use with `registry_password`)." + required: false + default: "" + registry_password: + description: "Optional registry password for `docker login` (use with `registry_username`). Pass secrets via workflow `with:` from secrets." + required: false + default: "" cmd: - description: 'The Command which should be run' + description: "The Command which should be run" required: true +runs: + using: "composite" + steps: + - id: login + name: Login to registry (if credentials provided) + if: ${{ inputs.registry_username && inputs.registry_password && inputs.registry }} + shell: bash + env: + REGISTRY: ${{ inputs.registry }} + REG_USER: ${{ inputs.registry_username }} + REG_PASS: ${{ inputs.registry_password }} + run: | + set -euo pipefail + if [ -z "$REGISTRY" ] || [ -z "$REG_USER" ] || [ -z "$REG_PASS" ]; then + echo "Missing registry or credentials; skipping login" + exit 0 + fi + echo "Logging into registry: $REGISTRY" + echo "$REG_PASS" | docker login "$REGISTRY" --username "$REG_USER" --password-stdin + + - id: run + name: Run yq container + shell: bash + env: + IMAGE_INPUT: ${{ inputs.image }} + REGISTRY: ${{ inputs.registry }} + CMD_INPUT: ${{ inputs.cmd }} + run: | + set -euo pipefail + IMAGE="$IMAGE_INPUT" + if [ -n "$REGISTRY" ]; then + REG="${REGISTRY%/}" + IMAGE="$REG/$IMAGE" + fi + echo "Using image: $IMAGE" + RC=0 + OUTPUT=$(docker run --rm -v "$GITHUB_WORKSPACE":/work -w /work "$IMAGE" sh -lc "$CMD_INPUT" 2>&1) || RC=$? + echo "result<> $GITHUB_OUTPUT + echo "$OUTPUT" >> $GITHUB_OUTPUT + echo "EOF" >> $GITHUB_OUTPUT + if [ "$RC" -ne 0 ]; then + exit "$RC" + fi outputs: result: description: "The complete result from the yq command being run" -runs: - using: 'docker' - image: 'docker://mikefarah/yq:4-githubaction' - args: - - ${{ inputs.cmd }} + value: ${{ steps.run.outputs.result }} From c8efd595fc0b8a8c4138229cb922327b4cce1063 Mon Sep 17 00:00:00 2001 From: Tommy Malmqvist Date: Tue, 16 Sep 2025 16:12:45 +0200 Subject: [PATCH 2/4] Update README.md --- README.md | 209 ++++++++++++------------------------------------------ 1 file changed, 47 insertions(+), 162 deletions(-) diff --git a/README.md b/README.md index 543983f8..f134d1d7 100644 --- a/README.md +++ b/README.md @@ -1,49 +1,35 @@ # yq -![Build](https://github.com/mikefarah/yq/workflows/Build/badge.svg) -![Docker Pulls](https://img.shields.io/docker/pulls/mikefarah/yq.svg) -![Github Releases (by Release)](https://img.shields.io/github/downloads/mikefarah/yq/total.svg) -![Go Report](https://goreportcard.com/badge/github.com/mikefarah/yq) -![CodeQL](https://github.com/mikefarah/yq/workflows/CodeQL/badge.svg) +![Build](https://github.com/mikefarah/yq/workflows/Build/badge.svg) ![Docker Pulls](https://img.shields.io/docker/pulls/mikefarah/yq.svg) ![Github Releases (by Release)](https://img.shields.io/github/downloads/mikefarah/yq/total.svg) ![Go Report](https://goreportcard.com/badge/github.com/mikefarah/yq) ![CodeQL](https://github.com/mikefarah/yq/workflows/CodeQL/badge.svg) -a lightweight and portable command-line YAML, JSON, INI and XML processor. `yq` -uses [jq](https://github.com/stedolan/jq) like syntax but works with yaml files -as well as json, xml, ini, properties, csv and tsv. It doesn't yet support -everything `jq` does - but it does support the most common operations and -functions, and more is being added continuously. -yq is written in go - so you can download a dependency free binary for your -platform and you are good to go! If you prefer there are a variety of package -managers that can be used as well as Docker and Podman, all listed below. +a lightweight and portable command-line YAML, JSON, INI and XML processor. `yq` uses [jq](https://github.com/stedolan/jq) like syntax but works with yaml files as well as json, xml, ini, properties, csv and tsv. It doesn't yet support everything `jq` does - but it does support the most common operations and functions, and more is being added continuously. + +yq is written in go - so you can download a dependency free binary for your platform and you are good to go! If you prefer there are a variety of package managers that can be used as well as Docker and Podman, all listed below. ## Quick Usage Guide Read a value: - ```bash yq '.a.b[0].c' file.yaml ``` Pipe from STDIN: - ```bash yq '.a.b[0].c' < file.yaml ``` Update a yaml file, in place - ```bash yq -i '.a.b[0].c = "cool"' file.yaml ``` Update using environment variables - ```bash NAME=mike yq -i '.a.b[0].c = strenv(NAME)' file.yaml ``` Merge multiple files - ```bash # merge two files yq -n 'load("file1.yaml") * load("file2.yaml")' @@ -55,7 +41,6 @@ yq ea '. as $item ireduce ({}; . * $item )' path/to/*.yml ``` Multiple updates to a yaml file - ```bash yq -i ' .a.b[0].c = "cool" | @@ -65,37 +50,30 @@ yq -i ' ``` Find and update an item in an array: - ```bash yq '(.[] | select(.name == "foo") | .address) = "12 cat st"' ``` Convert JSON to YAML - ```bash yq -Poy sample.json ``` -See [recipes](https://mikefarah.gitbook.io/yq/recipes) for more examples and the -[documentation](https://mikefarah.gitbook.io/yq/) for more information. +See [recipes](https://mikefarah.gitbook.io/yq/recipes) for more examples and the [documentation](https://mikefarah.gitbook.io/yq/) for more information. -Take a look at the discussions for -[common questions](https://github.com/mikefarah/yq/discussions/categories/q-a), -and -[cool ideas](https://github.com/mikefarah/yq/discussions/categories/show-and-tell) +Take a look at the discussions for [common questions](https://github.com/mikefarah/yq/discussions/categories/q-a), and [cool ideas](https://github.com/mikefarah/yq/discussions/categories/show-and-tell) ## Install ### [Download the latest binary](https://github.com/mikefarah/yq/releases/latest) ### wget - Use wget to download, gzipped pre-compiled binaries: + For instance, VERSION=v4.2.0 and BINARY=yq_linux_amd64 #### Compressed via tar.gz - ```bash wget https://github.com/mikefarah/yq/releases/download/${VERSION}/${BINARY}.tar.gz -O - |\ tar xz && mv ${BINARY} /usr/local/bin/yq @@ -116,39 +94,28 @@ wget https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -O ``` ### MacOS / Linux via Homebrew: - Using [Homebrew](https://brew.sh/) - ``` brew install yq ``` ### Linux via snap: - ``` snap install yq ``` #### Snap notes - -`yq` installs with -[_strict confinement_](https://docs.snapcraft.io/snap-confinement/6233) in snap, -this means it doesn't have direct access to root files. To read root files you -can: +`yq` installs with [_strict confinement_](https://docs.snapcraft.io/snap-confinement/6233) in snap, this means it doesn't have direct access to root files. To read root files you can: ``` sudo cat /etc/myfile | yq '.a.path' ``` -And to write to a root file you can either use -[sponge](https://linux.die.net/man/1/sponge): - +And to write to a root file you can either use [sponge](https://linux.die.net/man/1/sponge): ``` sudo cat /etc/myfile | yq '.a.path = "value"' | sudo sponge /etc/myfile ``` - or write to a temporary file: - ``` sudo cat /etc/myfile | yq '.a.path = "value"' | sudo tee /etc/myfile.tmp sudo mv /etc/myfile.tmp /etc/myfile @@ -156,16 +123,14 @@ rm /etc/myfile.tmp ``` ### Run with Docker or Podman - #### Oneshot use: ```bash docker run --rm -v "${PWD}":/workdir mikefarah/yq [command] [flags] [expression ]FILE... ``` -Note that you can run `yq` in docker without network access and other privileges -if you desire, namely -`--security-opt=no-new-privileges --cap-drop all --network none`. +Note that you can run `yq` in docker without network access and other privileges if you desire, +namely `--security-opt=no-new-privileges --cap-drop all --network none`. ```bash podman run --rm -v "${PWD}":/workdir mikefarah/yq [command] [flags] [expression ]FILE... @@ -193,8 +158,7 @@ docker run --rm -it -v "${PWD}":/workdir --entrypoint sh mikefarah/yq podman run --rm -it -v "${PWD}":/workdir --entrypoint sh mikefarah/yq ``` -It can be useful to have a bash function to avoid typing the whole docker -command: +It can be useful to have a bash function to avoid typing the whole docker command: ```bash yq() { @@ -207,13 +171,10 @@ yq() { podman run --rm -i -v "${PWD}":/workdir mikefarah/yq "$@" } ``` - #### Running as root: -`yq`'s container image no longer runs under root -(https://github.com/mikefarah/yq/pull/860). If you'd like to install more things -in the container image, or you're having permissions issues when attempting to -read/write files you'll need to either: +`yq`'s container image no longer runs under root (https://github.com/mikefarah/yq/pull/860). If you'd like to install more things in the container image, or you're having permissions issues when attempting to read/write files you'll need to either: + ``` docker run --user="root" -it --entrypoint sh mikefarah/yq @@ -234,9 +195,7 @@ USER yq ``` #### Missing timezone data - -By default, the alpine image yq uses does not include timezone data. If you'd -like to use the `tz` operator, you'll need to include this data: +By default, the alpine image yq uses does not include timezone data. If you'd like to use the `tz` operator, you'll need to include this data: ``` FROM mikefarah/yq @@ -248,90 +207,45 @@ USER yq #### Podman with SELinux -If you are using podman with SELinux, you will need to set the shared volume -flag `:z` on the volume mount: +If you are using podman with SELinux, you will need to set the shared volume flag `:z` on the volume mount: ``` -v "${PWD}":/workdir:z ``` ### GitHub Action - ``` -- name: Set foobar to cool - uses: mikefarah/yq@master - with: - cmd: yq -i '.foo.bar = "cool"' 'config.yml' -- name: Get an entry with a variable that might contain dots or spaces - id: get_username - uses: mikefarah/yq@master - with: - cmd: yq '.all.children.["${{ matrix.ip_address }}"].username' ops/inventories/production.yml -- name: Reuse a variable obtained in another step - run: echo ${{ steps.get_username.outputs.result }} + - name: Set foobar to cool + uses: mikefarah/yq@master + with: + cmd: yq -i '.foo.bar = "cool"' 'config.yml' + - name: Get an entry with a variable that might contain dots or spaces + id: get_username + uses: mikefarah/yq@master + with: + cmd: yq '.all.children.["${{ matrix.ip_address }}"].username' ops/inventories/production.yml + - name: Reuse a variable obtained in another step + run: echo ${{ steps.get_username.outputs.result }} ``` See https://mikefarah.gitbook.io/yq/usage/github-action for more. -### Action: custom registry support - -The action now accepts optional inputs to control which container image is used -and (optionally) authenticate to a registry: - -- `image`: the image name to run (default `mikefarah/yq:4-githubaction`). -- `registry`: optional artifact repository hostname/prefix to be prepended to - `image` (for example `artifacts.example.com/repository/docker`). -- `registry_username` / `registry_password`: optional credentials to login - before pulling the image. - -Important caveat: GitHub-hosted runners pull action container images (when using -`runs.using: docker`) before any workflow steps run. To make -registry-authenticated pulls work on GitHub-hosted runners, either: - -- Use the composite action (this repository) which performs a `docker login` - inside the action before running the container (works when credentials are - provided via action inputs). Note that some registries block pulls even with - login if the runner cannot reach them. -- Use self-hosted runners configured to use your artifact repository as a - pull-through cache or pre-pull the image on the runner. -- Alternatively, avoid using the action container image and instead - `docker login` and `docker run` from a regular `run:` step in your workflow - (see example below). - -Example using the composite action (local action reference): - -``` -- name: Run yq from artifact repository - uses: . - with: - registry: artifacts.example.com/repository/docker - image: mikefarah/yq:4-githubaction - registry_username: ${{ secrets.ARTIFACT_REPO_USER }} - registry_password: ${{ secrets.ARTIFACT_REPO_PASS }} - cmd: yq --version -``` - ### Go Install: - ``` go install github.com/mikefarah/yq/v4@latest ``` ## Community Supported Installation methods +As these are supported by the community :heart: - however, they may be out of date with the officially supported releases. -As these are supported by the community :heart: - however, they may be out of -date with the officially supported releases. +_Please note that the Debian package (previously supported by @rmescandon) is no longer maintained. Please use an alternative installation method._ -_Please note that the Debian package (previously supported by @rmescandon) is no -longer maintained. Please use an alternative installation method._ ### X-CMD - Checkout `yq` on x-cmd: https://x-cmd.com/mod/yq - Instant Results: See the output of your yq filter in real-time. -- Error Handling: Encounter a syntax error? It will display the error message - and the results of the closest valid filter +- Error Handling: Encounter a syntax error? It will display the error message and the results of the closest valid filter Thanks @edwinjhlee! @@ -341,8 +255,8 @@ Thanks @edwinjhlee! nix profile install nixpkgs#yq-go ``` -See -[here](https://search.nixos.org/packages?channel=unstable&show=yq-go&from=0&size=50&sort=relevance&type=packages&query=yq-go) +See [here](https://search.nixos.org/packages?channel=unstable&show=yq-go&from=0&size=50&sort=relevance&type=packages&query=yq-go) + ### Webi @@ -350,8 +264,8 @@ See webi yq ``` -See [webi](https://webinstall.dev/) Supported by @adithyasunil26 -(https://github.com/webinstall/webi-installers/tree/master/yq) +See [webi](https://webinstall.dev/) +Supported by @adithyasunil26 (https://github.com/webinstall/webi-installers/tree/master/yq) ### Arch Linux @@ -365,47 +279,37 @@ Using [Chocolatey](https://chocolatey.org) [![Chocolatey](https://img.shields.io/chocolatey/v/yq.svg)](https://chocolatey.org/packages/yq) [![Chocolatey](https://img.shields.io/chocolatey/dt/yq.svg)](https://chocolatey.org/packages/yq) - ``` choco install yq ``` - Supported by @chillum (https://chocolatey.org/packages/yq) Using [scoop](https://scoop.sh/) - ``` scoop install main/yq ``` Using [winget](https://learn.microsoft.com/en-us/windows/package-manager/) - ``` winget install --id MikeFarah.yq ``` ### MacPorts: - Using [MacPorts](https://www.macports.org/) - ``` sudo port selfupdate sudo port install yq ``` - -Supported by @herbygillot -(https://ports.macports.org/maintainer/github/herbygillot) +Supported by @herbygillot (https://ports.macports.org/maintainer/github/herbygillot) ### Alpine Linux Alpine Linux v3.20+ (and Edge): - ``` apk add yq-go ``` Alpine Linux up to v3.19: - ``` apk add yq ``` @@ -420,8 +324,8 @@ Flox can be used to install yq on Linux, MacOS, and Windows through WSL. flox install yq ``` -### MacOS / Linux via gah: +### MacOS / Linux via gah: Using [gah](https://github.com/marverix/gah) ``` @@ -429,29 +333,19 @@ gah install yq ``` ## Features - - [Detailed documentation with many examples](https://mikefarah.gitbook.io/yq/) - Written in portable go, so you can download a lovely dependency free binary -- Uses similar syntax as `jq` but works with YAML, INI, - [JSON](https://mikefarah.gitbook.io/yq/usage/convert) and - [XML](https://mikefarah.gitbook.io/yq/usage/xml) files +- Uses similar syntax as `jq` but works with YAML, INI, [JSON](https://mikefarah.gitbook.io/yq/usage/convert) and [XML](https://mikefarah.gitbook.io/yq/usage/xml) files - Fully supports multi document yaml files -- Supports yaml - [front matter](https://mikefarah.gitbook.io/yq/usage/front-matter) blocks - (e.g. jekyll/assemble) +- Supports yaml [front matter](https://mikefarah.gitbook.io/yq/usage/front-matter) blocks (e.g. jekyll/assemble) - Colorized yaml output - [Date/Time manipulation and formatting with TZ](https://mikefarah.gitbook.io/yq/operators/datetime) - [Deeply data structures](https://mikefarah.gitbook.io/yq/operators/traverse-read) - [Sort keys](https://mikefarah.gitbook.io/yq/operators/sort-keys) -- Manipulate yaml - [comments](https://mikefarah.gitbook.io/yq/operators/comment-operators), - [styling](https://mikefarah.gitbook.io/yq/operators/style), - [tags](https://mikefarah.gitbook.io/yq/operators/tag) and - [anchors and aliases](https://mikefarah.gitbook.io/yq/operators/anchor-and-alias-operators). +- Manipulate yaml [comments](https://mikefarah.gitbook.io/yq/operators/comment-operators), [styling](https://mikefarah.gitbook.io/yq/operators/style), [tags](https://mikefarah.gitbook.io/yq/operators/tag) and [anchors and aliases](https://mikefarah.gitbook.io/yq/operators/anchor-and-alias-operators). - [Update in place](https://mikefarah.gitbook.io/yq/v/v4.x/commands/evaluate#flags) - [Complex expressions to select and update](https://mikefarah.gitbook.io/yq/operators/select#select-and-update-matching-values-in-map) -- Keeps yaml formatting and comments when updating (though there are issues with - whitespace) +- Keeps yaml formatting and comments when updating (though there are issues with whitespace) - [Decode/Encode base64 data](https://mikefarah.gitbook.io/yq/operators/encode-decode) - [Load content from other files](https://mikefarah.gitbook.io/yq/operators/load) - [Convert to/from json/ndjson](https://mikefarah.gitbook.io/yq/v/v4.x/usage/convert) @@ -459,15 +353,12 @@ gah install yq - [Convert to/from properties](https://mikefarah.gitbook.io/yq/v/v4.x/usage/properties) - [Convert to/from csv/tsv](https://mikefarah.gitbook.io/yq/usage/csv-tsv) - [General shell completion scripts (bash/zsh/fish/powershell)](https://mikefarah.gitbook.io/yq/v/v4.x/commands/shell-completion) -- [Reduce](https://mikefarah.gitbook.io/yq/operators/reduce) to merge multiple - files or sum an array or other fancy things. -- [Github Action](https://mikefarah.gitbook.io/yq/usage/github-action) to use in - your automated pipeline (thanks @devorbitus) +- [Reduce](https://mikefarah.gitbook.io/yq/operators/reduce) to merge multiple files or sum an array or other fancy things. +- [Github Action](https://mikefarah.gitbook.io/yq/usage/github-action) to use in your automated pipeline (thanks @devorbitus) ## [Usage](https://mikefarah.gitbook.io/yq/) -Check out the [documentation](https://mikefarah.gitbook.io/yq/) for more -detailed and advanced usage. +Check out the [documentation](https://mikefarah.gitbook.io/yq/) for more detailed and advanced usage. ``` Usage: @@ -532,16 +423,10 @@ Flags: Use "yq [command] --help" for more information about a command. ``` - ## Known Issues / Missing Features +- `yq` attempts to preserve comment positions and whitespace as much as possible, but it does not handle all scenarios (see https://github.com/go-yaml/yaml/tree/v3 for details) +- Powershell has its own...[opinions on quoting yq](https://mikefarah.gitbook.io/yq/usage/tips-and-tricks#quotes-in-windows-powershell) +- "yes", "no" were dropped as boolean values in the yaml 1.2 standard - which is the standard yq assumes. -- `yq` attempts to preserve comment positions and whitespace as much as - possible, but it does not handle all scenarios (see - https://github.com/go-yaml/yaml/tree/v3 for details) -- Powershell has its - own...[opinions on quoting yq](https://mikefarah.gitbook.io/yq/usage/tips-and-tricks#quotes-in-windows-powershell) -- "yes", "no" were dropped as boolean values in the yaml 1.2 standard - which is - the standard yq assumes. +See [tips and tricks](https://mikefarah.gitbook.io/yq/usage/tips-and-tricks) for more common problems and solutions. -See [tips and tricks](https://mikefarah.gitbook.io/yq/usage/tips-and-tricks) for -more common problems and solutions. From f20f287d5b28a3cb6c3f387035a9d03a7054c28e Mon Sep 17 00:00:00 2001 From: Tommy Malmqvist Date: Wed, 17 Sep 2025 10:16:56 +0200 Subject: [PATCH 3/4] Mask sensitive values during Docker login in action.yml --- action.yml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/action.yml b/action.yml index efc03136..18715933 100644 --- a/action.yml +++ b/action.yml @@ -40,6 +40,10 @@ runs: echo "Missing registry or credentials; skipping login" exit 0 fi + # Mask sensitive values + if [ -n "$REG_PASS" ]; then + echo "::add-mask::$REG_PASS" + fi echo "Logging into registry: $REGISTRY" echo "$REG_PASS" | docker login "$REGISTRY" --username "$REG_USER" --password-stdin From 9ce5c8afeea5b38409b391d47ffb9818d23ce447 Mon Sep 17 00:00:00 2001 From: Tommy Malmqvist Date: Thu, 18 Sep 2025 07:31:47 +0200 Subject: [PATCH 4/4] Refactor Docker login process in action.yml to enhance image pulling logic. Added handling for successful and failed pulls with credentials and anonymous access. Mask sensitive values during login. --- action.yml | 43 ++++++++++++++++++++++++++++++++++++------- 1 file changed, 36 insertions(+), 7 deletions(-) diff --git a/action.yml b/action.yml index 18715933..bca3a606 100644 --- a/action.yml +++ b/action.yml @@ -26,26 +26,55 @@ inputs: runs: using: "composite" steps: - - id: login - name: Login to registry (if credentials provided) + - id: pull-with-credentials + name: Pull image using provided credentials if: ${{ inputs.registry_username && inputs.registry_password && inputs.registry }} shell: bash env: + IMAGE_INPUT: ${{ inputs.image }} REGISTRY: ${{ inputs.registry }} REG_USER: ${{ inputs.registry_username }} REG_PASS: ${{ inputs.registry_password }} run: | set -euo pipefail - if [ -z "$REGISTRY" ] || [ -z "$REG_USER" ] || [ -z "$REG_PASS" ]; then - echo "Missing registry or credentials; skipping login" - exit 0 + IMAGE="$IMAGE_INPUT" + if [ -n "$REGISTRY" ]; then + REG="${REGISTRY%/}" + IMAGE="$REG/$IMAGE" fi - # Mask sensitive values + echo "Using image: $IMAGE" + echo "Credentials provided; attempting docker login to $REGISTRY" if [ -n "$REG_PASS" ]; then echo "::add-mask::$REG_PASS" fi - echo "Logging into registry: $REGISTRY" echo "$REG_PASS" | docker login "$REGISTRY" --username "$REG_USER" --password-stdin + if docker pull "$IMAGE" >/dev/null 2>&1; then + echo "Image pulled successfully after login." + else + echo "Failed to pull image after login; proceeding to run (docker run may fail)." + fi + + - id: pull-anonymous + name: Pull image anonymously + if: ${{ !(inputs.registry_username && inputs.registry_password && inputs.registry) }} + shell: bash + env: + IMAGE_INPUT: ${{ inputs.image }} + REGISTRY: ${{ inputs.registry }} + run: | + set -euo pipefail + IMAGE="$IMAGE_INPUT" + if [ -n "$REGISTRY" ]; then + REG="${REGISTRY%/}" + IMAGE="$REG/$IMAGE" + fi + echo "Using image: $IMAGE" + echo "No credentials provided (or registry not set); attempting anonymous pull" + if docker pull "$IMAGE" >/dev/null 2>&1; then + echo "Anonymous pull succeeded." + else + echo "Anonymous pull failed; proceeding to run (docker run may fail if auth required)." + fi - id: run name: Run yq container