yq/README.md

yq

a lightweight and portable command-line YAML, JSON and XML processor. yq uses jq like syntax but works with yaml files as well as json and xml. It doesn't yet support everything jq does - but it does support the most common operations and functions, and more is being added continuously.

yq is written in go - so you can download a dependency free binary for your platform and you are good to go! If you prefer there are a variety of package managers that can be used as well as Docker and Podman, all listed below.

Notice for v4.x versions prior to 4.18.1

Since 4.18.1, yq's 'eval/e' command is the default command and no longers needs to be specified.

Older versions will still need to specify 'eval/e'.

Similarly, '-' is no longer required as a filename to read from STDIN (unless reading from multiple files).

TLDR:

Prior to 4.18.1

cat file.yaml | yq e '.cool' -

4.18+

cat file.yaml | yq '.cool'

When merging multiple files together, eval-all/ea is still required to tell yq to run the expression against all the document at once.

Quick Usage Guide

Read a value:

yq '.a.b[0].c' file.yaml

Pipe from STDIN:

cat file.yaml | yq '.a.b[0].c'

Update a yaml file, inplace

yq -i '.a.b[0].c = "cool"' file.yaml

Update using environment variables

NAME=mike yq -i '.a.b[0].c = strenv(NAME)' file.yaml

Merge multiple files

# note the use of `ea` to evaluate all the files at once
# instead of in sequence
yq ea '. as $item ireduce ({}; . * $item )' path/to/*.yml

Multiple updates to a yaml file

yq -i '
  .a.b[0].c = "cool" |
  .x.y.z = "foobar" |
  .person.name = strenv(NAME)
' file.yaml

See the documentation for more.

Install

Download the latest binary

wget

Use wget to download the pre-compiled binaries:

Compressed via tar.gz

wget https://github.com/mikefarah/yq/releases/download/${VERSION}/${BINARY}.tar.gz -O - |\
  tar xz && mv ${BINARY} /usr/bin/yq

Plain binary

wget https://github.com/mikefarah/yq/releases/download/${VERSION}/${BINARY} -O /usr/bin/yq &&\
    chmod +x /usr/bin/yq

For instance, VERSION=v4.2.0 and BINARY=yq_linux_amd64

MacOS / Linux via Homebrew:

Using Homebrew

brew install yq

Linux via snap:

snap install yq

Snap notes

yq installs with strict confinement in snap, this means it doesn't have direct access to root files. To read root files you can:

sudo cat /etc/myfile | yq '.a.path'

And to write to a root file you can either use sponge:

sudo cat /etc/myfile | yq '.a.path = "value"' | sudo sponge /etc/myfile

or write to a temporary file:

sudo cat /etc/myfile | yq '.a.path = "value"' | sudo tee /etc/myfile.tmp
sudo mv /etc/myfile.tmp /etc/myfile
rm /etc/myfile.tmp

Run with Docker or Podman

Oneshot use:

docker run --rm -v "${PWD}":/workdir mikefarah/yq [command] [flags] [expression ]FILE...

Note that you can run yq in docker without network access and other privileges if you desire, namely --security-opt=no-new-privileges --cap-drop all --network none.

podman run --rm -v "${PWD}":/workdir mikefarah/yq [command] [flags] [expression ]FILE...

Pipe in via STDIN:

You'll need to pass the -i\--interactive flag to docker:

cat myfile.yml | docker run -i --rm mikefarah/yq '.this.thing'

cat myfile.yml | podman run -i --rm mikefarah/yq '.this.thing'

Run commands interactively:

docker run --rm -it -v "${PWD}":/workdir --entrypoint sh mikefarah/yq

podman run --rm -it -v "${PWD}":/workdir --entrypoint sh mikefarah/yq

It can be useful to have a bash function to avoid typing the whole docker command:

yq() {
  docker run --rm -i -v "${PWD}":/workdir mikefarah/yq "$@"
}

yq() {
  podman run --rm -i -v "${PWD}":/workdir mikefarah/yq "$@"
}

Running as root:

yq's container image no longer runs under root (https://github.com/mikefarah/yq/pull/860). If you'd like to install more things in the container image, or you're having permissions issues when attempting to read/write files you'll need to either:

docker run --user="root" -it --entrypoint sh mikefarah/yq

podman run --user="root" -it --entrypoint sh mikefarah/yq

Or, in your Dockerfile:

FROM mikefarah/yq

USER root
RUN apk add --no-cache bash
USER yq

GitHub Action

  - name: Set foobar to cool
    uses: mikefarah/yq@master
    with:
      cmd: yq -i '.foo.bar = "cool"' 'config.yml'

See https://mikefarah.gitbook.io/yq/usage/github-action for more.

Go Get:

GO111MODULE=on go get github.com/mikefarah/yq/v4

Community Supported Installation methods

As these are supported by the community ❤️ - however, they may be out of date with the officially supported releases.

Webi

webi yq

See webi Supported by @adithyasunil26 (https://github.com/webinstall/webi-installers/tree/master/yq)

Arch Linux

pacman -S go-yq

Windows:

choco install yq

Supported by @chillum (https://chocolatey.org/packages/yq)

Mac:

Using MacPorts

sudo port selfupdate
sudo port install yq

Supported by @herbygillot (https://ports.macports.org/maintainer/github/herbygillot)

Alpine Linux

• Enable edge/community repo by adding $MIRROR/alpine/edge/community to /etc/apk/repositories
• Update database index with apk update
• Install yq with apk add yq

Supported by Tuan Hoang https://pkgs.alpinelinux.org/package/edge/community/x86/yq

On Ubuntu 16.04 or higher from Debian package:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys CC86BB64
sudo add-apt-repository ppa:rmescandon/yq
sudo apt update
sudo apt install yq -y

Supported by @rmescandon (https://launchpad.net/~rmescandon/+archive/ubuntu/yq)

Features

• Detailed documentation with many examples
• Written in portable go, so you can download a lovely dependency free binary
• Uses similar syntax as jq but works with YAML, JSON and XML files
• Fully supports multi document yaml files
• Supports yaml front matter blocks (e.g. jekyll/assemble)
• Colorized yaml output
• Deeply data structures
• Sort keys
• Manipulate yaml comments, styling, tags and anchors and aliases.
• Update inplace
• Complex expressions to select and update
• Keeps yaml formatting and comments when updating (though there are issues with whitespace)
• Load content from other files
• Convert to/from json
• Convert to/from xml
• Convert to properties
• Convert to csv/tsv
• Pipe data in by using '-'
• General shell completion scripts (bash/zsh/fish/powershell)
• Reduce to merge multiple files or sum an array or other fancy things.
• Github Action to use in your automated pipeline (thanks @devorbitus)

Usage

Check out the documentation for more detailed and advanced usage.

Usage:
  yq [flags]
  yq [command]

Examples:

# yq defaults to 'eval' command if no command is specified. See "yq eval --help" for more examples.
cat myfile.yml | yq '.stuff' # outputs the data at the "stuff" node from "myfile.yml"

yq -i '.stuff = "foo"' myfile.yml # update myfile.yml inplace


Available Commands:
  completion       Generate the autocompletion script for the specified shell
  eval             (default) Apply the expression to each document in each yaml file in sequence
  eval-all         Loads _all_ yaml documents of _all_ yaml files and runs expression once
  help             Help about any command
  shell-completion Generate completion script

Flags:
  -C, --colors                        force print with colors
  -e, --exit-status                   set exit status if there are no matches or null or false is returned
  -f, --front-matter string           (extract|process) first input as yaml front-matter. Extract will pull out the yaml content, process will run the expression against the yaml content, leaving the remaining data intact
      --header-preprocess             Slurp any header comments and separators before processing expression. (default true)
  -h, --help                          help for yq
  -I, --indent int                    sets indent level for output (default 2)
  -i, --inplace                       update the file inplace of first file given.
  -p, --input-format string           [yaml|y|xml|x] parse format for input. Note that json is a subset of yaml. (default "yaml")
  -M, --no-colors                     force print with no colors
  -N, --no-doc                        Don't print document separators (---)
  -n, --null-input                    Don't read input, simply evaluate the expression given. Useful for creating docs from scratch.
  -o, --output-format string          [yaml|y|json|j|props|p|xml|x] output format type. (default "yaml")
  -P, --prettyPrint                   pretty print, shorthand for '... style = ""'
  -s, --split-exp string              print each result (or doc) into a file named (exp). [exp] argument must return a string. You can use $index in the expression as the result counter.
      --unwrapScalar                  unwrap scalar, print the value with no quotes, colors or comments (default true)
  -v, --verbose                       verbose mode
  -V, --version                       Print version information and quit
      --xml-attribute-prefix string   prefix for xml attributes (default "+")
      --xml-content-name string       name for xml content (if no attribute name is present). (default "+content")

Use "yq [command] --help" for more information about a command.

Known Issues / Missing Features

• yq attempts to preserve comment positions and whitespace as much as possible, but it does not handle all scenarios (see https://github.com/go-yaml/yaml/tree/v3 for details)
• Powershell has its own...opinions on quoting yq

See tips and tricks for more common problems and solutions.