yq/README.md

383 lines
13 KiB
Markdown
Raw Normal View History

2019-05-14 17:28:06 +00:00
# yq
2018-07-31 09:04:34 +00:00
2022-05-06 03:54:49 +00:00
![Build](https://github.com/mikefarah/yq/workflows/Build/badge.svg) ![Docker Pulls](https://img.shields.io/docker/pulls/mikefarah/yq.svg) ![Github Releases (by Release)](https://img.shields.io/github/downloads/mikefarah/yq/total.svg) ![Go Report](https://goreportcard.com/badge/github.com/mikefarah/yq) ![CodeQL](https://github.com/mikefarah/yq/workflows/CodeQL/badge.svg)
2018-07-10 11:46:36 +00:00
2022-08-01 03:47:16 +00:00
a lightweight and portable command-line YAML, JSON and XML processor. `yq` uses [jq](https://github.com/stedolan/jq) like syntax but works with yaml files as well as json, xml, properties, csv and tsv. It doesn't yet support everything `jq` does - but it does support the most common operations and functions, and more is being added continuously.
2015-09-27 03:29:20 +00:00
yq is written in go - so you can download a dependency free binary for your platform and you are good to go! If you prefer there are a variety of package managers that can be used as well as Docker and Podman, all listed below.
2015-09-27 03:29:20 +00:00
2021-07-25 00:47:53 +00:00
## Quick Usage Guide
2020-12-20 02:34:58 +00:00
2021-07-25 00:47:53 +00:00
Read a value:
```bash
2022-01-27 06:21:10 +00:00
yq '.a.b[0].c' file.yaml
2021-09-19 23:55:35 +00:00
```
Pipe from STDIN:
```bash
2022-03-29 02:15:27 +00:00
yq '.a.b[0].c' < file.yaml
2021-07-25 00:47:53 +00:00
```
2023-09-18 23:52:36 +00:00
Update a yaml file, in place
2021-07-25 00:47:53 +00:00
```bash
2022-01-27 06:21:10 +00:00
yq -i '.a.b[0].c = "cool"' file.yaml
2021-07-25 00:47:53 +00:00
```
Update using environment variables
```bash
2022-01-27 06:21:10 +00:00
NAME=mike yq -i '.a.b[0].c = strenv(NAME)' file.yaml
2021-07-25 00:47:53 +00:00
```
Merge multiple files
2022-01-27 06:21:10 +00:00
```bash
2024-02-09 03:54:22 +00:00
# merge two files
yq -n 'load("file1.yaml") * load("file2.yaml")'
# merge using globs:
2022-01-27 06:21:10 +00:00
# note the use of `ea` to evaluate all the files at once
# instead of in sequence
2021-08-27 02:30:22 +00:00
yq ea '. as $item ireduce ({}; . * $item )' path/to/*.yml
2021-07-25 00:47:53 +00:00
```
Multiple updates to a yaml file
```bash
2022-01-27 06:21:10 +00:00
yq -i '
2021-07-25 00:47:53 +00:00
.a.b[0].c = "cool" |
.x.y.z = "foobar" |
.person.name = strenv(NAME)
' file.yaml
```
2023-09-01 02:17:54 +00:00
Find and update an item in an array:
```bash
yq '(.[] | select(.name == "foo") | .address) = "12 cat st"'
```
2022-03-19 23:55:24 +00:00
Convert JSON to YAML
```bash
yq -Poy sample.json
2022-03-19 23:55:24 +00:00
```
2023-09-01 04:43:38 +00:00
See [recipes](https://mikefarah.gitbook.io/yq/recipes) for more examples and the [documentation](https://mikefarah.gitbook.io/yq/) for more information.
2021-01-06 04:33:20 +00:00
2022-08-01 06:55:55 +00:00
Take a look at the discussions for [common questions](https://github.com/mikefarah/yq/discussions/categories/q-a), and [cool ideas](https://github.com/mikefarah/yq/discussions/categories/show-and-tell)
2015-11-20 10:42:45 +00:00
## Install
2020-01-11 00:44:02 +00:00
2020-02-21 10:07:37 +00:00
### [Download the latest binary](https://github.com/mikefarah/yq/releases/latest)
2020-01-30 23:37:27 +00:00
2020-12-30 22:22:18 +00:00
### wget
2022-10-26 02:12:20 +00:00
Use wget to download, gzipped pre-compiled binaries:
2020-12-30 22:22:18 +00:00
For instance, VERSION=v4.2.0 and BINARY=yq_linux_amd64
2020-12-30 22:33:15 +00:00
#### Compressed via tar.gz
2020-12-30 22:30:59 +00:00
```bash
wget https://github.com/mikefarah/yq/releases/download/${VERSION}/${BINARY}.tar.gz -O - |\
tar xz && mv ${BINARY} /usr/bin/yq
```
2020-12-30 22:33:15 +00:00
#### Plain binary
2020-12-30 22:30:59 +00:00
2020-12-30 22:22:18 +00:00
```bash
wget https://github.com/mikefarah/yq/releases/download/${VERSION}/${BINARY} -O /usr/bin/yq &&\
chmod +x /usr/bin/yq
```
#### Latest version
```bash
wget https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -O /usr/bin/yq &&\
chmod +x /usr/bin/yq
```
2020-12-30 22:22:18 +00:00
### MacOS / Linux via Homebrew:
Using [Homebrew](https://brew.sh/)
2017-12-24 19:30:49 +00:00
```
brew install yq
```
2020-04-22 13:29:44 +00:00
2020-12-30 22:22:18 +00:00
### Linux via snap:
```
snap install yq
```
#### Snap notes
2020-05-23 14:30:00 +00:00
`yq` installs with [_strict confinement_](https://docs.snapcraft.io/snap-confinement/6233) in snap, this means it doesn't have direct access to root files. To read root files you can:
```
2022-01-27 06:21:10 +00:00
sudo cat /etc/myfile | yq '.a.path'
```
And to write to a root file you can either use [sponge](https://linux.die.net/man/1/sponge):
```
2022-01-27 06:21:10 +00:00
sudo cat /etc/myfile | yq '.a.path = "value"' | sudo sponge /etc/myfile
```
or write to a temporary file:
```
2022-01-27 06:21:10 +00:00
sudo cat /etc/myfile | yq '.a.path = "value"' | sudo tee /etc/myfile.tmp
sudo mv /etc/myfile.tmp /etc/myfile
rm /etc/myfile.tmp
```
### Run with Docker or Podman
#### Oneshot use:
```bash
2022-01-27 06:21:10 +00:00
docker run --rm -v "${PWD}":/workdir mikefarah/yq [command] [flags] [expression ]FILE...
```
Note that you can run `yq` in docker without network access and other privileges if you desire,
namely `--security-opt=no-new-privileges --cap-drop all --network none`.
```bash
2022-01-27 06:21:10 +00:00
podman run --rm -v "${PWD}":/workdir mikefarah/yq [command] [flags] [expression ]FILE...
```
#### Pipe in via STDIN:
You'll need to pass the `-i\--interactive` flag to docker:
```bash
2022-03-29 02:15:27 +00:00
docker run -i --rm mikefarah/yq '.this.thing' < myfile.yml
```
```bash
2022-03-29 02:15:27 +00:00
podman run -i --rm mikefarah/yq '.this.thing' < myfile.yml
```
#### Run commands interactively:
```bash
2021-01-14 22:44:02 +00:00
docker run --rm -it -v "${PWD}":/workdir --entrypoint sh mikefarah/yq
```
```bash
podman run --rm -it -v "${PWD}":/workdir --entrypoint sh mikefarah/yq
```
It can be useful to have a bash function to avoid typing the whole docker command:
```bash
yq() {
docker run --rm -i -v "${PWD}":/workdir mikefarah/yq "$@"
}
```
```bash
yq() {
podman run --rm -i -v "${PWD}":/workdir mikefarah/yq "$@"
}
```
2021-07-08 00:02:23 +00:00
#### Running as root:
`yq`'s container image no longer runs under root (https://github.com/mikefarah/yq/pull/860). If you'd like to install more things in the container image, or you're having permissions issues when attempting to read/write files you'll need to either:
2021-07-08 00:02:23 +00:00
```
docker run --user="root" -it --entrypoint sh mikefarah/yq
```
```
podman run --user="root" -it --entrypoint sh mikefarah/yq
```
Or, in your Dockerfile:
2021-07-08 00:02:23 +00:00
```
FROM mikefarah/yq
USER root
RUN apk add --no-cache bash
2021-07-08 00:02:23 +00:00
USER yq
```
2022-02-15 02:38:05 +00:00
#### Missing timezone data
By default, the alpine image yq uses does not include timezone data. If you'd like to use the `tz` operator, you'll need to include this data:
```
FROM mikefarah/yq
USER root
RUN apk add --no-cache tzdata
USER yq
```
2022-08-01 03:47:16 +00:00
#### Podman with SELinux
If you are using podman with SELinux, you will need to set the shared volume flag `:z` on the volume mount:
```
-v "${PWD}":/workdir:z
```
2021-07-25 00:51:13 +00:00
### GitHub Action
```
- name: Set foobar to cool
uses: mikefarah/yq@master
with:
2022-01-27 06:21:10 +00:00
cmd: yq -i '.foo.bar = "cool"' 'config.yml'
- name: Get an entry with a variable that might contain dots or spaces
id: get_username
uses: mikefarah/yq@master
with:
cmd: yq '.all.children.["${{ matrix.ip_address }}"].username' ops/inventories/production.yml
- name: Reuse a variable obtained in another step
run: echo ${{ steps.get_username.outputs.result }}
2021-07-25 00:51:13 +00:00
```
See https://mikefarah.gitbook.io/yq/usage/github-action for more.
2022-03-28 03:36:27 +00:00
### Go Install:
```
2022-03-28 03:36:27 +00:00
go install github.com/mikefarah/yq/v4@latest
```
## Community Supported Installation methods
As these are supported by the community :heart: - however, they may be out of date with the officially supported releases.
2023-03-15 10:21:15 +00:00
_Please note that the Debian package (previously supported by @rmescandon) is no longer maintained. Please use an alternative installation method._
2022-10-23 22:42:22 +00:00
### Nix
```
nix profile install nixpkgs#yq-go
```
See [here](https://search.nixos.org/packages?channel=unstable&show=yq-go&from=0&size=50&sort=relevance&type=packages&query=yq-go)
### Webi
2021-01-08 01:14:36 +00:00
```
webi yq
```
See [webi](https://webinstall.dev/)
Supported by @adithyasunil26 (https://github.com/webinstall/webi-installers/tree/master/yq)
### Arch Linux
```
pacman -S go-yq
```
### Windows:
Using [Chocolatey](https://chocolatey.org)
2021-03-23 09:40:39 +00:00
[![Chocolatey](https://img.shields.io/chocolatey/v/yq.svg)](https://chocolatey.org/packages/yq)
[![Chocolatey](https://img.shields.io/chocolatey/dt/yq.svg)](https://chocolatey.org/packages/yq)
```
choco install yq
```
Supported by @chillum (https://chocolatey.org/packages/yq)
Using [scoop](https://scoop.sh/)
```
scoop install main/yq
```
2022-10-05 09:51:48 +00:00
Using [winget](https://learn.microsoft.com/en-us/windows/package-manager/)
```
winget install --id MikeFarah.yq
```
2022-10-05 09:51:48 +00:00
### Mac:
Using [MacPorts](https://www.macports.org/)
```
sudo port selfupdate
sudo port install yq
```
Supported by @herbygillot (https://ports.macports.org/maintainer/github/herbygillot)
### Alpine Linux
- Enable edge/community repo by adding ```$MIRROR/alpine/edge/community``` to ```/etc/apk/repositories```
- Update database index with ```apk update```
- Install yq with ```apk add yq```
Supported by Tuan Hoang
https://pkgs.alpinelinux.org/package/edge/community/x86/yq
2015-10-13 10:31:02 +00:00
## Features
2021-03-19 00:52:13 +00:00
- [Detailed documentation with many examples](https://mikefarah.gitbook.io/yq/)
2015-10-13 10:31:02 +00:00
- Written in portable go, so you can download a lovely dependency free binary
2022-01-22 01:35:33 +00:00
- Uses similar syntax as `jq` but works with YAML, [JSON](https://mikefarah.gitbook.io/yq/usage/convert) and [XML](https://mikefarah.gitbook.io/yq/usage/xml) files
- Fully supports multi document yaml files
2021-07-18 03:44:56 +00:00
- Supports yaml [front matter](https://mikefarah.gitbook.io/yq/usage/front-matter) blocks (e.g. jekyll/assemble)
- Colorized yaml output
2022-02-27 01:23:36 +00:00
- [Date/Time manipulation and formatting with TZ](https://mikefarah.gitbook.io/yq/operators/datetime)
2022-01-22 01:35:33 +00:00
- [Deeply data structures](https://mikefarah.gitbook.io/yq/operators/traverse-read)
- [Sort keys](https://mikefarah.gitbook.io/yq/operators/sort-keys)
2021-03-19 00:52:13 +00:00
- Manipulate yaml [comments](https://mikefarah.gitbook.io/yq/operators/comment-operators), [styling](https://mikefarah.gitbook.io/yq/operators/style), [tags](https://mikefarah.gitbook.io/yq/operators/tag) and [anchors and aliases](https://mikefarah.gitbook.io/yq/operators/anchor-and-alias-operators).
2023-09-18 23:52:36 +00:00
- [Update in place](https://mikefarah.gitbook.io/yq/v/v4.x/commands/evaluate#flags)
2021-03-19 00:52:13 +00:00
- [Complex expressions to select and update](https://mikefarah.gitbook.io/yq/operators/select#select-and-update-matching-values-in-map)
- Keeps yaml formatting and comments when updating (though there are issues with whitespace)
2022-02-27 01:23:36 +00:00
- [Decode/Encode base64 data](https://mikefarah.gitbook.io/yq/operators/encode-decode)
2021-11-25 22:54:01 +00:00
- [Load content from other files](https://mikefarah.gitbook.io/yq/operators/load)
2022-08-01 03:47:16 +00:00
- [Convert to/from json/ndjson](https://mikefarah.gitbook.io/yq/v/v4.x/usage/convert)
2022-01-22 01:35:33 +00:00
- [Convert to/from xml](https://mikefarah.gitbook.io/yq/v/v4.x/usage/xml)
2022-02-15 03:13:55 +00:00
- [Convert to/from properties](https://mikefarah.gitbook.io/yq/v/v4.x/usage/properties)
2022-08-01 03:47:16 +00:00
- [Convert to/from csv/tsv](https://mikefarah.gitbook.io/yq/usage/csv-tsv)
2020-12-20 02:34:58 +00:00
- [General shell completion scripts (bash/zsh/fish/powershell)](https://mikefarah.gitbook.io/yq/v/v4.x/commands/shell-completion)
2021-02-25 06:04:08 +00:00
- [Reduce](https://mikefarah.gitbook.io/yq/operators/reduce) to merge multiple files or sum an array or other fancy things.
2021-06-08 22:57:42 +00:00
- [Github Action](https://mikefarah.gitbook.io/yq/usage/github-action) to use in your automated pipeline (thanks @devorbitus)
2015-09-29 00:20:31 +00:00
2020-01-29 22:58:21 +00:00
## [Usage](https://mikefarah.gitbook.io/yq/)
2017-04-13 05:44:31 +00:00
2020-01-29 22:58:21 +00:00
Check out the [documentation](https://mikefarah.gitbook.io/yq/) for more detailed and advanced usage.
2017-04-13 05:44:31 +00:00
2015-09-27 03:29:20 +00:00
```
2017-04-14 02:40:03 +00:00
Usage:
2017-12-17 22:11:08 +00:00
yq [flags]
yq [command]
2015-09-27 03:40:09 +00:00
2022-01-27 06:21:10 +00:00
Examples:
# yq defaults to 'eval' command if no command is specified. See "yq eval --help" for more examples.
2022-03-29 02:15:27 +00:00
yq '.stuff' < myfile.yml # outputs the data at the "stuff" node from "myfile.yml"
2022-01-27 06:21:10 +00:00
2023-09-18 23:52:36 +00:00
yq -i '.stuff = "foo"' myfile.yml # update myfile.yml in place
2022-01-27 06:21:10 +00:00
2017-04-14 02:40:03 +00:00
Available Commands:
2022-01-22 01:35:33 +00:00
completion Generate the autocompletion script for the specified shell
2022-01-27 06:21:10 +00:00
eval (default) Apply the expression to each document in each yaml file in sequence
eval-all Loads _all_ yaml documents of _all_ yaml files and runs expression once
2020-06-11 23:30:05 +00:00
help Help about any command
shell-completion Generate completion script
2015-10-05 03:41:01 +00:00
2017-04-14 02:40:03 +00:00
Flags:
2022-01-22 01:35:33 +00:00
-C, --colors force print with colors
-e, --exit-status set exit status if there are no matches or null or false is returned
-f, --front-matter string (extract|process) first input as yaml front-matter. Extract will pull out the yaml content, process will run the expression against the yaml content, leaving the remaining data intact
--header-preprocess Slurp any header comments and separators before processing expression. (default true)
-h, --help help for yq
-I, --indent int sets indent level for output (default 2)
2023-09-18 23:52:36 +00:00
-i, --inplace update the file in place of first file given.
2022-01-22 01:35:33 +00:00
-p, --input-format string [yaml|y|xml|x] parse format for input. Note that json is a subset of yaml. (default "yaml")
-M, --no-colors force print with no colors
-N, --no-doc Don't print document separators (---)
-n, --null-input Don't read input, simply evaluate the expression given. Useful for creating docs from scratch.
-o, --output-format string [yaml|y|json|j|props|p|xml|x] output format type. (default "yaml")
-P, --prettyPrint pretty print, shorthand for '... style = ""'
-s, --split-exp string print each result (or doc) into a file named (exp). [exp] argument must return a string. You can use $index in the expression as the result counter.
--unwrapScalar unwrap scalar, print the value with no quotes, colors or comments (default true)
-v, --verbose verbose mode
-V, --version Print version information and quit
--xml-attribute-prefix string prefix for xml attributes (default "+")
--xml-content-name string name for xml content (if no attribute name is present). (default "+content")
2015-10-05 23:01:33 +00:00
2017-12-17 22:11:08 +00:00
Use "yq [command] --help" for more information about a command.
2015-09-29 06:29:32 +00:00
```
2020-09-16 05:42:09 +00:00
## Known Issues / Missing Features
2020-09-03 00:58:00 +00:00
- `yq` attempts to preserve comment positions and whitespace as much as possible, but it does not handle all scenarios (see https://github.com/go-yaml/yaml/tree/v3 for details)
2021-11-03 10:51:43 +00:00
- Powershell has its own...[opinions on quoting yq](https://mikefarah.gitbook.io/yq/usage/tips-and-tricks#quotes-in-windows-powershell)
- "yes", "no" were dropped as boolean values in the yaml 1.2 standard - which is the standard yq assumes.
2021-07-16 00:06:28 +00:00
See [tips and tricks](https://mikefarah.gitbook.io/yq/usage/tips-and-tricks) for more common problems and solutions.