readStream opened an *os.File but returned a *bufio.Reader, so the
*os.File type switch in streamEvaluator.EvaluateFiles never matched and
no input file was ever closed. allAtOnceEvaluator.EvaluateFiles did not
attempt to close them at all. Descriptors were only released at process
exit or by the os.File finaliser, so a multi-file run could exhaust the
file descriptor limit.
readStream now returns an explicit cleanup function alongside the reader,
which both evaluators call once the file has been processed.
Co-authored-by: Pablo Garcia <pablito@MacBook-Neo-de-Pablo.local>
Co-authored-by: Mike Farah <mikefarah@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Resolve conflict in decoder_hcl.go by keeping typed object keys via
convertCtyValueToNode (preserves int/bool/float keys in YAML) rather than
coercing all keys to strings. Retain PR test case for integer key with
empty object value, adapted to typed key output.
Co-authored-by: Cursor <cursoragent@cursor.com>
writeString converted every string to []byte before calling
io.Writer.Write, which Go escape analysis reports as escaping to the
heap. io.WriteString uses the io.StringWriter fast path when available
(the standard printer writer is a *bufio.Writer) and falls back to
Write([]byte(txt)) otherwise.
Fixes#2807
Co-authored-by: Pablo Garcia <pablito@MacBook-Neo-de-Pablo.local>
HCL object keys may be non-string literals (e.g. a number like 1). The
decoder called cty.Value.AsString on the evaluated key, which panics with
'not a string' for any non-string key type. Mirror OpenTofu/Terragrunt by
silently converting non-string keys to their string representation via
cty/convert. Add decode scenarios covering numeric and boolean keys.
The sequence branch only checked the upper bound (len > index), so has(-1) returned true even on an empty array. Guard index >= 0.
Co-authored-by: max <max@example.com>
The int/int comparator returned int(lhs - rhs); the int64 subtraction overflows for large-magnitude values, giving the wrong sign. Compare directly instead.
Co-authored-by: max <max@example.com>
When explode resolves merge anchors (<<:), items copied from the
alias target retained the original parent pointer instead of being
set to the enclosing node being reconstructed. This caused GetPath()
to return wrong paths for children of merge-anchored nodes, making
subsequent merge operations target the wrong LHS keys.
In fixedReconstructAliasedMap, set copied.Parent = node after copy.
In reconstructAliasedMap (non-spec path), replace AddChild (which
creates sequence-style numeric-index entries) with AddKeyValueChild
to properly reconstruct mapping key-value pairs. AddKeyValueChild
also correctly sets parent references via SetParent.
* Guard ExpressionParser initialization with sync.Once
The evaluators call InitExpressionParser on every Evaluate, so two goroutines
could construct a parser at the same time. newParticipleLexer also fills in the
ParticipleTokenType field on each shared participleYqRules entry as it builds
the lexer, which another goroutine could be reading through getYqDefinition.
These are pointer and struct field writes rather than map writes, so the
runtime does not stop the process; it corrupts quietly instead.
The nil check stays inside the Once so a caller that assigns ExpressionParser
itself is still respected.
Fixes#2788
* docs: use UK spelling for initialised in new comments
Matches the existing prose convention in pkg/yqlib (operator_reduce.go,
decoder_ini.go, operator_alternative_test.go). Review feedback on #2789.
* fix: skip UTF-8 BOM when processing front matter
--front-matter=process wrapped the file in a bufio.Reader without
skipping a leading UTF-8 BOM, so a BOM before the opening --- meant the
separator was not recognised and the opening --- was dropped from the
output (data loss). Skip the BOM with utfbom.Skip before wrapping the
reader, matching the CSV object decoder.
* refactor: inline UTF-8 BOM skip instead of utfbom dependency
per ccoVeille's review, front matter only needs UTF-8 BOM handling, not
the full utfbom package. replace utfbom.Skip with a small unexported
stripUTF8BOM helper and add a stdin-path regression test alongside the
existing file-path one. utfbom stays in go.mod since decoder_csv_object.go
still uses it for CSV decoding.
Add INIPreferences.PreserveSurroundedQuote option that wires through to
go-ini/ini's LoadOptions.PreserveSurroundedQuote. When enabled, existing
surrounding quotes on INI values are preserved during decode/encode
round-trips.
Fixes#2456
Co-authored-by: toller892 <toller892@gmail.com>
YAML scalars tagged `!!float` were round-tripped through `float64` and
re-serialized by Go's JSON encoder, which strips the decimal part of
whole-number floats. As a result, `50.0` came out as `50` and a
sequence like `[50.0, 95.0, 99.0, 99.9]` became `[50,95,99,99.9]`,
turning a uniform array of floats into a mixed int/float array that
downstream consumers (Horreum, JSON Schema validators, jq, etc.)
reject.
The JSON spec does not distinguish ints from floats, but every common
JSON library (Go's `encoding/json`, Python's `json`, jq) preserves the
fractional form of values that came in as floats. yq's YAML decoder
already parses these as `!!float` with the original text intact, so we
can emit them verbatim instead of round-tripping.
`MarshalJSON` for `ScalarNode` now special-cases `!!float`:
- if `Value` is already a JSON-shaped number literal containing a `.`
or exponent, emit it verbatim (e.g. `50.0`, `99.9`, `1.5e-3`, `-7.0`);
- if `Value` is an integer-shaped string tagged `!!float` (e.g.
`!!float 5`), format the parsed float and append `.0` so it stays a
JSON number with a fractional part;
- otherwise (empty value, parse error, or non-finite result), fall back
to the existing encoding path so behaviour for `.inf` / `.nan` and
anything unusual is unchanged.
`!!int` nodes still encode as JSON integers.
Closes#2683
Signed-off-by: ChrisJr404 <chris@hacknow.com>
Ensure root-level TOML attributes are emitted before table sections so fields like sort remain root-scoped. Skip null-valued object fields during TOML encoding
instead of converting them to empty strings.
* Initial plan
* fix: TOML encoder no longer treats YAML FlowStyle as inline tables
Remove FlowStyle checks from the TOML encoder. YAML flow-style mappings
are a YAML-specific rendering hint and should not influence TOML output.
Only nodes explicitly marked with EncodeHintInline (set by the TOML
decoder for actual TOML inline tables) will produce TOML inline table
syntax.
This fixes the bug where JSON auto-detected via the YAML parser (which
parses {} as flow-style mappings) would produce inline TOML tables
instead of readable table sections, while explicitly parsing with
-p json produced correct table sections.
Updated tests: YAML flow mappings now produce table sections (same as
block mappings), consistent with the fix. Added new test cases for the
JSON → TOML conversion via both YAML decoder (auto-detection) and JSON
decoder.
Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/3e504870-b585-4998-af9c-a451e2f6a6a3
Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>
The test literal "ab" * 4611686018427387904 (2^62) exceeds MaxInt32,
so parseInt rejects it before the size guard runs. Compute the count
with 1 << (bits.UintSize - 2) to yield 2^30 on 32-bit and 2^62 on
64-bit. Both values, when doubled by len("ab"), wrap past MaxInt and
bypass a naive len*count guard, exercising the division-safe check
added in #2644.
Signed-off-by: Jan Dubois <jan@jandubois.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Initial plan
* Fix TOML encoder to prefer readable table sections over inline tables
When converting from YAML/JSON to TOML, the encoder now always uses
readable TOML table section syntax ([section]) instead of compact inline
hash table syntax (key = { ... }), which better matches TOML's goal as
a human-focused configuration format.
Changes:
- decoder_toml.go: Mark inline TOML tables with FlowStyle so round-trips
can be distinguished from YAML flow mappings
- encoder_toml.go:
- encodeTopLevelEntry: use FlowStyle check instead of EncodeSeparate to
decide inline vs table section (all block mappings now become tables)
- encodeSeparateMapping: count FlowStyle children as attributes; use
recursive encodeSeparateMapping for nested non-flow mappings
- encodeMappingBodyWithPath: emit non-flow child mappings as sub-table
sections instead of inline tables
- toml_test.go: add encode (YAML→TOML) test scenarios, update roundtrip
expectations for inline tables (now expanded to table sections)
Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/4824a219-6d5e-42e7-bca1-a8a277bf8c6a
Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>
* Fix TOML roundtrip: use TomlInline flag instead of FlowStyle to preserve inline tables
FlowStyle affected YAML decode output (causing inline tables to appear as
YAML flow mappings). Replace it with a new TOML-specific TomlInline bool
on CandidateNode that:
- Is set by the TOML decoder for inline tables (not FlowStyle)
- Is copied by UpdateAttributesFrom so it survives DeeplyAssign merges
- Is checked by the TOML encoder alongside FlowStyle (for YAML flow maps)
- Has no effect on the YAML encoder, preserving existing TOML→YAML output
TOML roundtrip tests are restored to their original expected values (inline
tables stay inline, table sections stay as sections).
Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/f59bdf62-6d16-4664-991b-38eb87c9d81c
Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>
* Refactor EncodeSeparate+TomlInline into a single EncodeHint enum
Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/24db9a8f-601d-4ccf-ada7-129ed3226bb6
Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>
* Fix stale comment in hasStructuralChildren
Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/24db9a8f-601d-4ccf-ada7-129ed3226bb6
Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>
* Remove unused hasStructuralChildren method from tomlEncoder
Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/2c234b77-28e9-4995-ba6f-9d213ec551a0
Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>
recurseNodeObjectEqual and containsObject both used findInArray to
locate keys in a MappingNode's Content array. findInArray steps by 1,
so it matches against both keys (even indices) and values (odd indices).
In recurseNodeObjectEqual, when a null key in the LHS matched a null
value in the RHS at the last position, rhs.Content[indexInRHS+1]
accessed an out-of-bounds index, causing a panic.
In containsObject, a %2 guard prevented the panic but introduced false
negatives: when a null value appeared before the actual null key,
findInArray returned the value's odd index, the guard rejected it, and
the function reported the key as missing.
Both functions now use findKeyInMap, which steps by 2 and compares only
key positions. The %2 guard in containsObject is removed.
Reproducer for the panic (recurseNodeObjectEqual):
echo '? [{~: ~}]
: v1
? [{2: ~}]
: v2' | yq '. += .'
Reproducer for the false negative (containsObject):
printf '? 1\n: ~\n? ~\n: x\n' | yq 'contains({~: "x"})'
Found by OSS-Fuzz via the lima project's FuzzEvaluateExpression target.
https://issues.oss-fuzz.com/issues/383860504
Signed-off-by: Jan Dubois <jan@jandubois.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
sliceArrayOperator adjusts negative indices by adding Content length,
but does not clamp the result. When the absolute value of a negative
index exceeds Content length (e.g. .[-99999:3] on a 3-element array),
the adjusted index remains negative and causes an out-of-bounds access
in the Content slice loop.
Extract the adjust-and-clamp logic into clampSliceIndex and use it for
both index positions.
Reproducer (panics before this fix, returns full array after):
echo '[a, b, c]' | yq '.[-99999:3]'
Found by OSS-Fuzz via the lima project's FuzzEvaluateExpression target.
https://issues.oss-fuzz.com/issues/438776028
Signed-off-by: Jan Dubois <jan@jandubois.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
go-yaml accepts cross-document alias references, which the YAML spec
forbids (anchors are scoped to a single document). When a nested
assignment targets such an alias, UpdateFrom copies the Alias field
between nodes, creating a self-referencing AliasNode. Both traverse()
and traverseArrayIndices() then follow this cycle indefinitely.
Extract resolveAliasChain(), which follows aliases iteratively with a
visited set and returns an error on cycles. Both traverse() and
traverseArrayIndices() now call it, eliminating the recursive alias
handling in both code paths.
Note: traverseMergeAnchor() also dereferences aliases (lines 358 and
371) but with single-step assignment, not recursion. A self-referencing
alias there falls through the kind switch silently rather than
crashing. Using resolveAliasChain() in that function would produce a
clear error instead of silently dropping the node.
Reproducer (stack overflow before this fix, returns error after):
echo '&-- a
---
*--' | yq eval-all '. = (.x = 1)'
Found by OSS-Fuzz via the lima project's FuzzEvaluateExpression target.
https://issues.oss-fuzz.com/issues/390467412
Signed-off-by: Jan Dubois <jan@jandubois.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The existing check (count > 10 million) does not account for string
length. A 68-byte string repeated 35 trillion times passes the count
check but panics in strings.Repeat with "makeslice: len out of range".
Smaller counts (e.g. 10 million * 6-byte string = 60 MB) cause OOM on
memory-constrained environments like OSS-Fuzz (2560 MB limit).
Replace the count-only check with a result size check: the product of
string length and repeat count must not exceed 10 MiB. Use division
(len > limit/count) instead of multiplication (len*count > limit) to
avoid integer overflow — a large count can wrap the product to a
negative value, bypassing the guard entirely.
Fixes at least four OSS-Fuzz bugs found via Lima's FuzzEvaluateExpression:
https://issues.oss-fuzz.com/issues/418818862 (makeslice overflow)
https://issues.oss-fuzz.com/issues/422001683 (timeout from huge alloc)
https://issues.oss-fuzz.com/issues/383195001 (OOM, 3 GB allocation)
https://issues.oss-fuzz.com/issues/385180606 (OOM, 97 TB allocation)
Signed-off-by: Jan Dubois <jan@jandubois.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When using --front-matter, yq creates a temporary file for the
extracted YAML content but replaces the original filename in args
with the temp file path. This caused the 'filename' operator to
return the temp file path instead of the original filename.
Added a filename alias mechanism: when front matter processing
replaces the file path, it registers the original filename as an
alias. The readDocuments and stream evaluator functions resolve
aliases before setting candidateNode.filename.
Fixes#2538
Co-authored-by: cobyfrombrooklyn-bot <cobyfrombrooklyn@gmail.com>
Standalone TOML comments immediately inside a table/array-table no longer end the table scope, preventing subsequent keys from being flattened to the document root.
Adds a new CLI flag that enables compact sequence indentation where '- ' is
considered part of the indentation. This leverages the CompactSeqIndent()
method from the underlying go.yaml.in/yaml/v4 library.
Example output with --yaml-compact-seq-indent:
parent:
items:
- one
- two
Instead of the default:
parent:
items:
- one
- two
Closes#1841
The colorizeToml function intended to force colors by setting
color.NoColor = false, but SprintFunc() still respects the NO_COLOR
environment variable. This caused TestTomlColourization to fail in
CI environments where NO_COLOR=1 is set.
Fixed by calling EnableColor() on each color object, which explicitly
forces colors regardless of environment settings.
Vibe-coded with Cursor (Claude Opus 4)
* feat: Add UnwrapScalar to ShellVariablesPreferences
- Add UnwrapScalar boolean field to ShellVariablesPreferences struct.
- Initialize UnwrapScalar to false in NewDefaultShellVariablesPreferences.
- This preference will control whether shell output should be quoted or raw.
* feat: Propagate unwrapScalar to ShellVariablesPreferences
- In configureEncoder function, set UnwrapScalar in ConfiguredShellVariablesPreferences.
- This ensures the -r flag's state is passed to the shell encoder for raw output control.
* feat: Implement conditional quoting in shellVariablesEncoder
- Modify doEncode method to check pe.prefs.UnwrapScalar.
- If UnwrapScalar is true, output raw node.Value.
- Otherwise, use quoteValue for shell-safe quoting.
- This enables quote-free output for Kubernetes workflows when -r is used.
* test: Add tests for UnwrapScalar in shell encoder
- Introduce assertEncodesToUnwrapped helper function.
- Add TestShellVariablesEncoderUnwrapScalar to verify quote-free output with -r.
- Add TestShellVariablesEncoderDefaultQuoting to confirm default quoting behavior without -r.
- Ensure comprehensive testing of conditional quoting logic for shell output.
* remove redundant test
* feat: K8S KYAML output format support
Reference: https://github.com/kubernetes/enhancements/blob/master/keps/sig-cli/5295-kyaml/README.md
Co-authored-by: Codex <codex@openai.com>
Generated-with: OpenAI Codex CLI (partial)
Signed-off-by: Robin H. Johnson <rjohnson@coreweave.com>
* build: gomodcache/gocache should not be committed
Signed-off-by: Robin H. Johnson <rjohnson@coreweave.com>
* chore: fix spelling of behaviour
Signed-off-by: Robin H. Johnson <robbat2@gentoo.org>
* build: pass GOFLAGS to docker to support buildvcs=false
In trying to develop the KYAML support, various tests gave false
positive results because they made assumptions about Git functionality
Make it possible to avoid that by passing GOFLAGS='-buildvcs=false' to
to Makefile.
Signed-off-by: Robin H. Johnson <robbat2@gentoo.org>
* doc: cover documentScenarios for tests
Signed-off-by: Robin H. Johnson <rjohnson@coreweave.com>
* build: exclude go caches from gosec
Without tuning, gosec scans all of the vendor/gocache/gomodcache, taking
several minutes (3m35 here), whereas the core of the yq takes only 15
seconds to scan.
If we intend to remediate upstream issues in future; add a seperate
target to scan those.
Signed-off-by: Robin H. Johnson <rjohnson@coreweave.com>
---------
Signed-off-by: Robin H. Johnson <rjohnson@coreweave.com>
Signed-off-by: Robin H. Johnson <robbat2@gentoo.org>
Co-authored-by: Codex <codex@openai.com>
- Add ShellVariablesPreferences struct with KeySeparator field (default: '_')
- Update shellVariablesEncoder to use configurable separator
- Add --shell-key-separator CLI flag
- Add comprehensive tests for custom separator functionality
- Update documentation with example usage for custom separator
This feature allows users to specify a custom separator (e.g. '__') when
outputting shell variables, which helps disambiguate nested keys from
keys that contain underscores in their names.
Example:
yq -o=shell --shell-key-separator='__' file.yaml
Fixes ambiguity when original YAML keys contain underscores.
- Allow inline maps instead of just aliases
- Allow aliased sequences
- Disallow other types
- Use tag `!!merge` instead of key `<<`
- Fix insertion index for sequence merge
Closes#2386
Updated the list of supported platforms in the GitHub Actions
Docker release workflow to include linux/s390x.
Signed-off-by: Ashok Pariya <ashok.pariya@ibm.com>
With docker-hub starting to introduce a much stricter limit on anonymous
pulls, it makes sense to also host the image in ghcr.io as well,
allowing users flexibility in where they pull the docker images from.
Also with the github action it makes more sense hosting the docker image
on github infrastructure.
I've introduced a github action for logging into registries as well.
Problem: When --split-exp is used and produces filenames with slashes in
them, the target directories must already exist otherwise yq fails.
Fix/feature: Create the necessary directories with os.MkdirAll().
The permissions 0750 were chosen to satisfy the vulnerability checker.
Solves
```
❯ make test
Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM Error: unable to connect to Podman socket: failed to read identity "/Users/tiger/.local/share/containers/podman/machine/machine": open /Users/tiger/.local/share/containers/podman/machine/machine: no such file or directory /opt/homebrew/bin/docker run --rm -e LDFLAGS=" -X main.GitCommit=bbdd974+CHANGES -X main.GitDescribe=v4 -w -s" -e GITHUB_TOKEN="" -v /Users/tiger/git/yq-k8s/vendor:/go/src -v /Users/tiger/git/yq-k8s:/yq/src/github.com/mikefarah/yq -w /yq/src/github.com/mikefarah/yq yq_dev go mod vendor
Error: podman-machine-default: VM already exists
```
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
* build: use goreleaser to cross-compile binaries for all platforms and archs
The main motivation behind it is because "gox" is unmaintained and archived
and it does not support "linux/riscv64" as a target.
Right now, goreleaser is only building the binaries, and I've tried to replicate
the exact same way the old script does.
In the future, if so desired, goreleaser could be used to further automate the
build and release pipeline by creating the release in GitHub.
* build: create binaries and OCI image for linux/riscv64
* convert file ext to lowercase for format detection
To ensure proper file format detection with case-insensitive file
systems.
* use filepath.Ext for more reliable file ext detection
especially for paths like index.js/foo
* add a test for file ext based format detection
This commit adds the --properties-separator option, which lets users
specify the separator used between keys and values in the properties
output format. This is done by adjusting the value of
github.com/magiconair/properties#Properties.WriteSeparator at encode
time.
Some refactoring of the properties encoder unit tests was done to make
it easier to write unit tests that include different separator values.
Fixes: #1864
Signed-off-by: Ryan Drew <ryan.drew@isovalent.com>
When the shell executing yq has no open stdin, os.Stdin.Stat() return nil and yq fails to continue. This commit fixes a missing verification on the result of os.Stdin.Stat() in the utils.processStdInArgs function and adds an acceptance test to cover this scenario in the future. This bug affects yq since version 4.26.1.
* encoder_lua: Handle explicitly positive infinity
* encoder_lua: Fix inclusion of pre-/suffix when prettyPrinted
It seems certain operations like --prettyPrint or subset selections does
not produce a DocumentNode, which is where the lua pre- and suffix was
printed, causing those to be omitted.
* encoder_lua: Improve Tag handling robustness
Using the method call seems more reliable in case the input parser
forgets to set the tag.
- Add missing package manager (scoop)
- Improve winget installationn example & add link to reference documentation
- Add link to missing reference documentation for chocolatey
* Implement basic Lua output support
Ref #1700
Basic but working serialization to Lua tables.
* Escape larger set of characters in Lua output
Started with a minimum of replacements, this should be more complete,
tho not all substitutions are strictly required in Lua.
* Print simple keys unquoted in Lua output
String keys that satisfy the requirements for variable names can be used
as keys without quotes in tables.
* Quote Lua keywords in table keys
Keywords are not valid as unquoted keys, thus must be quoted
* Make output of unquoted Lua table keys optional
Generally safer and simpler to not do it.
* Hook up settings for Lua output
* Allow special characters in Lua prefix and suffix
--lua-suffix='});^M' didn't work, so taking this approach instead
* Panic on unhandled YAML Kind in Lua encoder
* Handle YAML case varied booleans in Lua encoder
* Handle special-case numbers in Lua encoder
* Reject unhandled scalar Tags in Lua encoder
* Add note about how Lua nil is unsuitable as table key
Could add some context tracking in the future to allow rejecting nil in
a table key context.
* Return error instead of panic in Lua encoder
* Add initial test for Lua encoder
Boilerplate mostly copied from toml_test.go
* Additional Lua output tests
* Generate Lua encoder documentation
Mostly just for the boilerplate
* Convert octal for Lua output
Lua doesn't have the 0oNNN syntax for octal integers, only decimal and
hexadecimal, hence those can be passed trough as is while octal needs
special treatment.
* Implement indentation in in Lua output
* Respect string Style in Lua encoder
Lua has 'single', "double" and [[ long ]] strings.
* Expand Lua examples
* Output line comments in Lua output
* Implement Lua globals output mode
* fix typo in a comment
* implement shell output format
* fix a typo
* add two test cases, have source uses ascii only
* add integration tests and documentation
* add fixes after code revieew
This is to ensure solid parsing of complex data (with any binary
content except NUL chars) by separating the `yq` root collection
member's output with NUL char. As a safe-guard, an error will be cast
if trying to use NUL character with content that contains itself NUL
characters inside.
Allow snap users to process YAML documents under /media or /mnt paths after connecting to the `removable-media` snapd slot:
```
sudo snap connect yq :removable-media
```
Fixes#1618.
* Remove extra backtick
* Reword explanation of update
* Reword explanation of relative update
* Change "remaple" to "remain"
* Change "clovver" to "clobber"
* Reword explanation of update for comment operators
* Reword explanation of relative update for comment operators
* Change "array" to "expression"
* Change "the golangs" to "Golang's"
* Change "golangs" to "Golang's"
* Change "can durations" to "can add durations"
* Change "array scalars" to "arrays"
* Change "beit" to "be it"
* Fix typo in `eval` tip
* Fix typo in header for `has` operation
* Add space before pipe in `line` operator example
* Fix typos in explanation of deep array merges
* Change "is now used" to "is now used."
* Change "object," to "object."
* Changes "indexes" to "indices"
* Remove extraneous copied text from `..` article
* Reword explanation of `...` operator
* Change "your are" to "you are"
* Add link to `string` operator docs in `select` article
* Change "is a" to "parameter specifies" in `string` operators article
* Change "new line" to "newline"
* Change "golang regex" to "Golang's regex"
* Change "golang" to "Golang"
* Add period
* Remove comma in `subtract` article
* Remove duplicate number subtraction example
* Remove comma in `traverse` operator article
* Clarify use of brackets when `read`ing with special characters
* detect inputFormat from filename
* refactor and extract func InputFormatFromFilename
* detect inputFormat only when file is provided
* add test for automatic input format detection
* Refactor ordered_map into separate files
Separate json and xml, from the regular yaml.
Makes it possible to compile, without those...
* Refactor encoder and decoder creation
Use more consistent parameters vs globals
Return errors instead of calling panic()
* Allow build without json and xml support
* Build constraint not working for non linux
* Go Build Constraint: Fix Non-Linux Filename (#1494)
Correct the filename pkg/yqlib/chown_not_linux.go to escape the
default OS detection in filename format `*_GOOS[_test].go`.
Add an extra word after `linux` to resolve the issue.
pkg/yqlib/chown_not_linux_os.go
Signed-off-by: Bhargav Ravuri <vaguecoder0to.n@gmail.com>
Signed-off-by: Bhargav Ravuri <vaguecoder0to.n@gmail.com>
Signed-off-by: Bhargav Ravuri <vaguecoder0to.n@gmail.com>
Co-authored-by: Bhargav Ravuri <saibhargavravuri@gmail.com>
1. Escaping behavior changed as a result of set-output deprecation.
- Otherwise you'll encounter breaking changes for example "appVersion: "1.0.0"%0Aapplica..."
where %0A is escaped newline \n but shouldn't be escaped.
2. Delimeter is required for multiline strings.
- See here: https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#multiline-strings
- Delimeter must be random. UUID is a solid choice.
I verified that /proc/sys/kernel/random/uuid does indeed exist on the yq image.
* adding support for --wrapScalar=false in properties encoder
* altering encoder and decoder tests somewhat
* adding .idea
* Revert "altering encoder and decoder tests somewhat"
This reverts commit e3655130e2.
* adding test scenario for encoding with wrapped scalars
4ec533b introduced a bug which causes the 'hidden' cli parameter
`__complete` to be ignored.
Add a check for this parameter, so that it can pass to `cobra`
Signed-off-by: Tom Whitwell <tom@whi.tw>
* Decoder Properties
* Added properties round trip test
* Fixed property decode for github actions
* Refactored XML test to use common functions
* Switched formatScenario parameter order for more readablity
* refactor: move from io/ioutil to io and os packages
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* build: update snapcraft `go-channel` to 1.17
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* chore: script to build the deb sources
now it includes the man files. This script makes only sense executing
on the machine having the gpg key needed to sign and push to ppa
* fix: dput dependency and proper configuration
This put everything concerning golangci following [this](https://golangci-lint.run/usage/configuration/).
This way, anyone running golangci with a command line or with it's favorite Ide will use the same configuration for the current project.
Whenever you perform **any** GitHub action on behalf of the user, you **must** disclose that an AI agent (Cursor) wrote the content and is acting on the user's behalf — **not the user personally**. Do this **before** submitting; never post first and add the disclosure later.
Applies to **all** GitHub interactions, including:
- Pull requests (titles, descriptions, and reviews)
- PR comments and inline review comments
- Issues (new issues, comments, and updates)
- Any other post or reply on GitHub
**How to disclose:** Put it prominently at the **top** of every PR description, review body, comment, or issue. Use wording like:
Inline review comments must include a short disclosure too (e.g. `> Generated by Cursor acting on the user's behalf, not the user personally.`).
**Never** submit a GitHub action without this disclosure.
---
Always run the spellcheck before raising a PR:
```bash
bash scripts/spelling.sh
```
This is also included in the full CI pipeline via `make local test`.
## Cursor Cloud specific instructions
### Overview
**yq** is a Go CLI for querying and transforming YAML, JSON, XML, INI, and other structured formats. There are no long-running services — development is build-and-test against a local `./yq` binary.
### Prerequisites
- **Go ≥ 1.25** (see `go.mod`)
- **Bash** (acceptance tests)
- **Docker/Podman** is optional; use `make local <target>` to run natively when containers are unavailable
### PATH
After `scripts/devtools.sh`, add Go tool binaries to PATH:
```bash
exportPATH="$HOME/go/bin:$PATH"
```
`golangci-lint` and `typos` install to `$HOME/go/bin`; `gosec` installs to `./bin/gosec` in the repo root.
### Common commands (local, no Docker)
| Task | Command |
|------|---------|
| Install dev tools | `bash scripts/devtools.sh` |
| Vendor dependencies | `make local vendor` |
| Build binary | `go build -o yq .` or `make local build` |
| Format | `make local format` |
| Lint | `make local check` |
| Unit tests | `make local test` or `bash scripts/test.sh` |
`make local build` runs the full CI chain (format → spelling → gosec → lint → unit tests → build → acceptance). For a faster loop, build with `go build -o yq .` and run `bash scripts/acceptance.sh`.
### Caveats
- **`make` without `local`** tries Docker/Podman (`Dockerfile.dev`). In Cloud Agent VMs without Docker, always prefix with `make local`.
- **Spelling step** uses `typos` (installed by `scripts/devtools.sh`).
- **`make local test` / `scripts/check.sh`** require `golangci-lint` on PATH (`devtools.sh`).
---
# General rules
✅ **DO:**
- You can use ./yq with the `--debug-node-info` flag to get a deeper understanding of the ast.
- run ./scripts/format.sh to format the code; then ./scripts/check.sh lint and finally ./scripts/spelling.sh to check spelling.
- Add comprehensive tests to cover the changes
- Run test suite to ensure there is no regression
- Use UK english spelling
- **Follow the mandatory GitHub agent disclosure rule above** on every GitHub action — no exceptions
❌ **DON'T:**
- Git add or commit
- Add comments to functions that are self-explanatory
- **Post to GitHub without the mandatory agent disclosure** (PRs, reviews, comments, issues, or any other GitHub interaction)
# Adding a New Encoder/Decoder
This guide explains how to add support for a new format (encoder/decoder) to yq without modifying `candidate_node.go`.
## Overview
The encoder/decoder architecture in yq is based on two main interfaces:
- **Encoder**: Converts a `CandidateNode` to output in a specific format
- **Decoder**: Reads input in a specific format and creates a `CandidateNode`
Each format is registered in `pkg/yqlib/format.go` and made available through factory functions.
## Architecture
### Key Files
-`pkg/yqlib/encoder.go` - Defines the `Encoder` interface
-`pkg/yqlib/decoder.go` - Defines the `Decoder` interface
-`pkg/yqlib/format.go` - Format registry and factory functions
If your format has preferences/configuration options:
1. Create a preferences struct with your configuration fields
2. Update the encoder to accept preferences in its factory function
3. Update `format.go` to pass the configured preferences
4. Update `operator_encoder_decoder.go` if special indent handling is needed (see existing formats like JSON and YAML for the pattern)
This pattern is optional and only needed if your format has user-configurable options.
## Build Tags
Use build tags to allow optional compilation of formats:
- Add `//go:build !yq_no<format>` at the top of your encoder and decoder files
- Create a no-build version in `pkg/yqlib/no_<format>.go` that returns nil for encoder/decoder factories
This allows users to compile yq without certain formats using: `go build -tags yq_no<format>`
## Working with CandidateNode
The `CandidateNode` struct represents a YAML node with:
-`Kind`: The node type (ScalarNode, SequenceNode, MappingNode)
-`Tag`: The YAML tag (e.g., "!!str", "!!int", "!!map")
-`Value`: The scalar value (for ScalarNode only)
-`Content`: Child nodes (for SequenceNode and MappingNode)
Key methods:
-`node.guessTagFromCustomType()` - Infer the tag from Go type
-`node.AsList()` - Convert to a list for processing
-`node.CreateReplacement()` - Create a new replacement node
-`NewCandidate()` - Create a new CandidateNode
## Key Points
✅ **DO:**
- Implement only the `Encoder` and `Decoder` interfaces
- Register your format in `format.go` only
- Keep format-specific logic in your encoder/decoder files
- Use the candidate_node style attribute to store style information for round-trip. Ask if this needs to be updated with new styles.
- Use build tags for optional compilation
- Add comprehensive tests
- Run the specific encoder/decoder test (e.g. <format>_test.go) whenever you make ay changes to the encoder_<format> or decoder_<format>
- Handle errors gracefully
- Add the no build directive, like the xml encoder and decoder, that enables a minimal yq builds. e.g. `//go:build !yq_<format>`. Be sure to also update the build_small-yq.sh and build-tinygo-yq.sh to not include the new format.
❌ **DON'T:**
- Modify `candidate_node.go` to add format-specific logic
- Add format-specific fields to `CandidateNode`
- Create special cases in core navigation or evaluation logic
- Bypass the encoder/decoder interfaces
- Use candidate_node tag attribute for anything other than indicate the data type
## Examples
Refer to existing format implementations for patterns:
This guide explains how to add a new operator to yq. Operators are the core of yq's expression language and process `CandidateNode` objects without requiring modifications to `candidate_node.go` itself.
## Overview
Operators transform data by implementing a handler function that processes a `Context` containing `CandidateNode` objects. Each operator is:
1. Defined as an `operationType` in `operation.go`
2. Registered in the lexer in `lexer_participle.go`
3. Implemented in its own `operator_<type>.go` file
4. Tested in `operator_<type>_test.go`
5. Documented in `pkg/yqlib/doc/operators/headers/<type>.md`
## Architecture
### Key Files
-`pkg/yqlib/operation.go` - Defines `operationType` and operator registry
-`pkg/yqlib/lexer_participle.go` - Registers operators with their syntax patterns
1. Run `scripts/devtools.sh` to install the required devtools
2. Run `make [local] vendor` to install the vendor dependencies
2. Run `make [local] test` to ensure you can run the existing tests
3. Write unit tests - (see existing examples). Changes will not be accepted without corresponding unit tests.
4. Make the code changes.
5.`make [local] test` to lint code and run tests
6. Profit! ok no profit, but raise a PR and get kudos :)
# Before you begin
Not all new PRs will be merged in
It's recommended to check with the owner first (e.g. raise an issue) to discuss a new feature before developing, to ensure your hard efforts don't go to waste.
PRs to fix bugs and issues are almost always welcome :pray: please ensure you write tests as well.
The following types of PRs will _not_ be accepted:
- **Significant refactors** take a lot of time to understand and can have all sorts of unintended side effects. If you think there's a better way to do things (that requires significant changes) raise an issue for discussion first :)
- **Release pipeline PRs** are a security risk - it's too easy for a serious vulnerability to sneak in (either intended or not). If there is a new cool way of releasing things, raise an issue for discussion first - it will need to be gone over with a fine tooth comb.
- **Version bumps** are handled by dependabot, the bot will auto-raise PRs and they will be regularly merged in.
- **New release platforms** At this stage, yq is not going to maintain any other release platforms other than GitHub and Docker - that said, I'm more than happy to put in other community maintained methods in the README for visibility :heart:
# Development
## Initial Setup
1. Install [Golang](https://golang.org/) (version 1.24.0 or later)
2. Run `scripts/devtools.sh` to install required development tools:
- golangci-lint for code linting
- gosec for security analysis
3. Run `make [local] vendor` to install vendor dependencies
4. Run `make [local] test` to ensure you can run the existing tests
## Development Workflow
1.**Write unit tests first** - Changes will not be accepted without corresponding unit tests (see Testing section below)
2.**Make your code changes**
3.**Run tests and linting**: `make [local] test` (this runs formatting, linting, security checks, and tests)
4.**Create your PR** and get kudos! :)
## Make Commands
- Use `make [local] <command>` for local development (runs in Docker container)
- Use `make <command>` for CI/CD environments
- Common commands:
-`make [local] vendor` - Install dependencies
-`make [local] test` - Run all checks and tests
-`make [local] build` - Build the yq binary
-`make [local] format` - Format code
-`make [local] check` - Run linting and security checks
# Code Quality
## Linting and Formatting
The project uses strict linting rules defined in `.golangci.yml`. All code must pass:
- **Code formatting**: gofmt, goimports, gci
- **Linting**: revive, errorlint, gosec, misspell, and others
- **Security checks**: gosec security analysis
- **Spelling checks**: misspell detection
Run `make [local] check` to verify your code meets all quality standards.
## Code Style Guidelines
- Follow standard Go conventions
- Use meaningful variable names
- Add comments for public functions and complex logic
- Keep functions focused and reasonably sized
- Use the project's existing patterns and conventions
# Testing
## Test Structure
Tests in yq use the `expressionScenario` pattern. Each test scenario includes:
-`expression`: The yq expression to test
-`document`: Input YAML/JSON (optional)
-`expected`: Expected output
-`skipDoc`: Whether to skip documentation generation
## Writing Tests
1.**Find the appropriate test file** (e.g., `operator_add_test.go` for addition operations)
2.**Add your test scenario** to the `*OperatorScenarios` slice
3.**Run the specific test**: `go test -run TestAddOperatorScenarios` (replace with appropriate test name)
4.**Verify documentation generation** (see Documentation section)
## Test Examples
```go
varaddOperatorScenarios=[]expressionScenario{
{
skipDoc:true,
expression:`"foo" + "bar"`,
expected:[]string{
"D0, P[], (!!str)::foobar\n",
},
},
{
document:"apples: 3",
expression:`.apples + 3`,
expected:[]string{
"D0, P[apples], (!!int)::6\n",
},
},
}
```
## Running Tests
- **All tests**: `make [local] test`
- **Specific test**: `go test -run TestName`
- **With coverage**: `make [local] cover`
# Documentation
## Documentation Generation
The project uses a documentation system that combines static headers with dynamically generated content from tests.
### How It Works
1.**Static headers** are defined in `pkg/yqlib/doc/operators/headers/*.md`
2.**Dynamic content** is generated from test scenarios in `*_test.go` files
3.**Generated docs** are created in `pkg/yqlib/doc/*.md` by concatenating headers with test-generated content
4.**Documentation is synced** to the gitbook branch for the website
### Updating Operator Documentation
#### For Test-Generated Documentation
Most operator documentation is generated from tests. To update:
1.**Find the test file** (e.g., `operator_add_test.go`)
2.**Update test scenarios** - each `expressionScenario` with `skipDoc: false` becomes documentation
3.**Run the test** to regenerate docs:
```bash
cd pkg/yqlib
go test -run TestAddOperatorScenarios
```
4. **Verify the generated documentation** in `pkg/yqlib/doc/add.md`
5. **Create a PR** with your changes
#### For Header-Only Documentation
If documentation exists only in `headers/*.md` files:
1. **Update the header file directly** (e.g., `pkg/yqlib/doc/operators/headers/add.md`)
2. **Create a PR** with your changes
### Updating Static Documentation
For documentation not in the master branch:
1. **Check the gitbook branch** for additional pages
2. **Update the `*.md` files** directly
3. **Create a PR** to the gitbook branch
### Documentation Best Practices
- **Write clear, concise examples** in test scenarios
- **Use meaningful variable names** in examples
- **Include edge cases** and error conditions
- **Test your documentation changes** by running the specific test
a lightweight and portable command-line YAML processor. `yq` uses [jq](https://github.com/stedolan/jq) like syntax but works with yaml files as well as json. It doesn't yet support everything `jq` does - but it does support the most common operations and functions, and more is being added continuously.
A lightweight and portable command-line YAML, JSON, INI and XML processor. `yq` uses [jq](https://github.com/stedolan/jq) (a popular JSON processor) like syntax but works with yaml files as well as json, kyaml, xml, ini, properties, csv and tsv. It doesn't yet support everything `jq` does - but it does support the most common operations and functions, and more is being added continuously.
yq is written in go - so you can download a dependency free binary for your platform and you are good to go! If you prefer there are a variety of package managers that can be used as well as docker, all listed below.
yq is written in Go - so you can download a dependency free binary for your platform and you are good to go! If you prefer there are a variety of package managers that can be used as well as Docker and Podman, all listed below.
## Quick Usage Guide
Read a value:
### Basic Operations
**Read a value:**
```bash
yq e '.a.b[0].c' file.yaml
yq '.a.b[0].c' file.yaml
```
Pipe from STDIN:
**Pipe from STDIN:**
```bash
cat file.yaml | yq e'.a.b[0].c'-
yq'.a.b[0].c'< file.yaml
```
Update a yaml file, inplace
**Update a yaml file inplace:**
```bash
yq e -i '.a.b[0].c = "cool"' file.yaml
yq -i '.a.b[0].c = "cool"' file.yaml
```
Update using environment variables
**Update using environment variables:**
```bash
NAME=mike yq e -i '.a.b[0].c = strenv(NAME)' file.yaml
See [recipes](https://mikefarah.gitbook.io/yq/recipes) for more examples and the [documentation](https://mikefarah.gitbook.io/yq/) for more information.
Take a look at the discussions for [common questions](https://github.com/mikefarah/yq/discussions/categories/q-a), and [cool ideas](https://github.com/mikefarah/yq/discussions/categories/show-and-tell)
## Install
### [Download the latest binary](https://github.com/mikefarah/yq/releases/latest)
### wget
Use wget to download the pre-compiled binaries:
Use wget to download pre-compiled binaries. Choose your platform and architecture:
For instance, VERSION=v4.2.0 and BINARY=yq_linux_amd64
**Available platforms:**`linux_amd64`, `linux_arm64`, `linux_arm`, `linux_386`, `darwin_amd64`, `darwin_arm64`, `windows_amd64`, `windows_386`, etc.
### MacOS / Linux via Homebrew:
Using [Homebrew](https://brew.sh/)
@@ -73,57 +109,56 @@ Using [Homebrew](https://brew.sh/)
brew install yq
```
or, for the (deprecated) v3 version:
```
brew install yq@3
```
Note that for v3, as it is a versioned brew it will not add the `yq` command to your path automatically. Please follow the instructions given by brew upon installation.
### Linux via snap:
```
snap install yq
```
or, for the (deprecated) v3 version:
```
snap install yq --channel=v3/stable
```
#### Snap notes
`yq` installs with [_strict confinement_](https://docs.snapcraft.io/snap-confinement/6233) in snap, this means it doesn't have direct access to root files. To read root files you can:
```
sudo cat /etc/myfile | yq e '.a.path' -
sudo cat /etc/myfile | yq '.a.path'
```
And to write to a root file you can either use [sponge](https://linux.die.net/man/1/sponge):
You'll need to pass the `-i\--interactive` flag to docker:
#### Pipe data via STDIN:
You'll need to pass the `-i --interactive` flag to Docker/Podman:
```bash
cat myfile.yml | docker run -i --rm mikefarah/yq e . -
# Process piped data
docker run -i --rm mikefarah/yq '.this.thing' < myfile.yml
# Same with Podman
podman run -i --rm mikefarah/yq '.this.thing' < myfile.yml
```
#### Run commands interactively:
@@ -132,6 +167,10 @@ cat myfile.yml | docker run -i --rm mikefarah/yq e . -
docker run --rm -it -v "${PWD}":/workdir --entrypoint sh mikefarah/yq
```
```bash
podman run --rm -it -v "${PWD}":/workdir --entrypoint sh mikefarah/yq
```
It can be useful to have a bash function to avoid typing the whole docker command:
```bash
@@ -140,45 +179,99 @@ yq() {
}
```
```bash
yq(){
podman run --rm -i -v "${PWD}":/workdir mikefarah/yq "$@"
}
```
#### Running as root:
`yq`'s docker image no longer runs under root (https://github.com/mikefarah/yq/pull/860). If you'd like to install more things in the docker image, or you're having permissions issues when attempting to read/write files you'll need to either:
`yq`'s container image no longer runs under root (https://github.com/mikefarah/yq/pull/860). If you'd like to install more things in the container image, or you're having permissions issues when attempting to read/write files you'll need to either:
```
docker run --user="root" -it --entrypoint sh mikefarah/yq
```
Or, in your docker file:
```
podman run --user="root" -it --entrypoint sh mikefarah/yq
```
Or, in your Dockerfile:
```
FROM mikefarah/yq
USER root
RUN apk add bash
RUN apk add --no-cache bash
USER yq
```
#### Missing timezone data
By default, the alpine image yq uses does not include timezone data. If you'd like to use the `tz` operator, you'll need to include this data:
```
FROM mikefarah/yq
USER root
RUN apk add --no-cache tzdata
USER yq
```
#### Podman with SELinux
If you are using podman with SELinux, you will need to set the shared volume flag `:z` on the volume mount:
```
-v "${PWD}":/workdir:z
```
### GitHub Action
```
- name: Set foobar to cool
uses: mikefarah/yq@master
with:
cmd: yq eval -i '.foo.bar = "cool"' 'config.yml'
cmd: yq -i '.foo.bar = "cool"' 'config.yml'
- name: Get an entry with a variable that might contain dots or spaces
Supported by Tuan Hoang (https://pkgs.alpinelinux.org/packages?name=yq-go)
### Flox:
Flox can be used to install yq on Linux, MacOS, and Windows through WSL.
```
flox install yq
```
### MacOS / Linux via gah:
Using [gah](https://github.com/marverix/gah)
```
gah install yq
```
Supported by @rmescandon (https://launchpad.net/~rmescandon/+archive/ubuntu/yq)
## Features
- [Detailed documentation with many examples](https://mikefarah.gitbook.io/yq/)
- Written in portable go, so you can download a lovely dependency free binary
- Uses similar syntax as `jq` but works with YAML and JSON files
- Uses similar syntax as `jq` but works with YAML, INI, [JSON](https://mikefarah.gitbook.io/yq/usage/convert) and [XML](https://mikefarah.gitbook.io/yq/usage/xml) files
- Manipulate yaml [comments](https://mikefarah.gitbook.io/yq/operators/comment-operators), [styling](https://mikefarah.gitbook.io/yq/operators/style), [tags](https://mikefarah.gitbook.io/yq/operators/tag) and [anchors and aliases](https://mikefarah.gitbook.io/yq/operators/anchor-and-alias-operators).
- [Reduce](https://mikefarah.gitbook.io/yq/operators/reduce) to merge multiple files or sum an array or other fancy things.
- [Github Action](https://mikefarah.gitbook.io/yq/usage/github-action) to use in your automated pipeline (thanks @devorbitus)
@@ -250,33 +380,101 @@ Usage:
yq [flags]
yq [command]
Examples:
# yq tries to auto-detect the file format based off the extension, and defaults to YAML if it's unknown (or piping through STDIN)
# Use the '-p/--input-format' flag to specify a format type.
cat file.xml | yq -p xml
# read the "stuff" node from "myfile.yml"
yq '.stuff' < myfile.yml
# update myfile.yml in place
yq -i '.stuff = "foo"' myfile.yml
# print contents of sample.json as idiomatic YAML
yq -P -oy sample.json
Available Commands:
eval Apply the expression to each document in each yaml file in sequence
eval-all Loads _all_ yaml documents of _all_ yaml files and runs expression once
help Help about any command
shell-completion Generate completion script
completion Generate the autocompletion script for the specified shell
eval (default) Apply the expression to each document in each yaml file in sequence
eval-all Loads _all_ yaml documents of _all_ yaml files and runs expression once
help Help about any command
Flags:
-C, --colors force print with colors
-e, --exit-status set exit status if there are no matches or null or false is returned
-f, --front-matter string (extract|process) first input as yaml front-matter. Extract will pull out the yaml content, process will run the expression against the yaml content, leaving the remaining data intact
--header-preprocess Slurp any header comments and seperators before processing expression. This is a workaround for go-yaml to persist header content (default true)
-h, --helphelp for yq
-I, --indent int sets indent level for output (default 2)
-i, --inplace update the yaml file inplace of first yaml file given.
--csv-separator char CSV Separator character (default ,)
--debug-node-info debug node info
-e, --exit-status set exit status if there are no matches or null or false is returned
--expression string forcibly set the expression argument. Useful when yq argument detection thinks your expression is a file.
--from-file string Load expression from specified file.
-f, --front-matter string (extract|process) first input as yaml front-matter. Extract will pull out the yaml content, process will run the expression against the yaml content, leaving the remaining data intact
--header-preprocess Slurp any header comments and separators before processing expression. (default true)
-h, --help help for yq
-I, --indent int sets indent level for output (default 2)
-i, --inplace update the file in place of first file given.
-p, --input-format string [auto|a|yaml|y|json|j|kyaml|ky|props|p|csv|c|tsv|t|xml|x|base64|uri|toml|hcl|h|lua|l|ini|i] parse format for input. (default "auto")
--lua-globals output keys as top-level global variables
-0, --nul-output Use NUL char to separate values. If unwrap scalar is also set, fail if unwrapped scalar contains NUL char.
-n, --null-input Don't read input, simply evaluate the expression given. Useful for creating docs from scratch.
-o, --output-format string [auto|a|yaml|y|json|j|kyaml|ky|props|p|csv|c|tsv|t|xml|x|base64|uri|toml|hcl|h|shell|s|lua|l|ini|i] output format type. (default "auto")
-P, --prettyPrint pretty print, shorthand for '... style = ""'
--properties-array-brackets use [x] in array paths (e.g. for SpringBoot)
--properties-separator string separator to use between keys and values (default " = ")
--security-disable-env-ops Disable env related operations.
--security-disable-file-ops Disable file related operations (e.g. load)
--shell-key-separator string separator for shell variable key paths (default "_")
-s, --split-exp string print each result (or doc) into a file named (exp). [exp] argument must return a string. You can use $index in the expression as the result counter. The necessary directories will be created.
--split-exp-file string Use a file to specify the split-exp expression.
--string-interpolation Toggles strings interpolation of \(exp) (default true)
- **Examples**: [Recipes and examples](https://mikefarah.gitbook.io/yq/recipes)
## Known Issues / Missing Features
-`yq` attempts to preserve comment positions and whitespace as much as possible, but it does not handle all scenarios (see https://github.com/go-yaml/yaml/tree/v3 for details)
- Powershell has its own...opinions: https://mikefarah.gitbook.io/yq/usage/tips-and-tricks#quotes-in-windows-powershell
- Powershell has its own...[opinions on quoting yq](https://mikefarah.gitbook.io/yq/usage/tips-and-tricks#quotes-in-windows-powershell)
- "yes", "no" were dropped as boolean values in the yaml 1.2 standard - which is the standard yq assumes.
See [tips and tricks](https://mikefarah.gitbook.io/yq/usage/tips-and-tricks) for more common problems and solutions.
See [tips and tricks](https://mikefarah.gitbook.io/yq/usage/tips-and-tricks) for more common problems and solutions.
Please **do not** report security vulnerabilities through public GitHub issues.
Instead, use GitHub's private vulnerability reporting feature:
👉 https://github.com/mikefarah/yq/security
This allows vulnerabilities to be triaged and addressed confidentially before any public disclosure.
## Scope
### HTTP / TLS / Network vulnerabilities
yq is a command-line YAML/JSON/TOML processor that reads from files or standard input and writes to standard output. **yq does not include any HTTP or network libraries** and makes no network connections at runtime. CVEs related to HTTP, TLS, or networking are therefore **not applicable** to yq.
### Dependency version bumps
yq uses [Dependabot](https://docs.github.com/en/code-security/dependabot) to automatically raise pull requests for:
- Go module dependencies
- Go toolchain version
- Docker base images
Please **do not** raise pull requests or issues solely to bump dependency or Go versions — Dependabot handles this automatically and the maintainers merge those PRs regularly.
rootCmd.PersistentFlags().BoolVarP(&outputToJSON,"tojson","j",false,"(deprecated) output as json. Set indent to 0 to print json in one line.")
err:=rootCmd.PersistentFlags().MarkDeprecated("tojson","please use -o=json instead")
@@ -48,21 +98,124 @@ See https://mikefarah.gitbook.io/yq/ for detailed documentation and examples.`,
panic(err)
}
rootCmd.PersistentFlags().StringVarP(&outputFormat,"output-format","o","yaml","[yaml|y|json|j|props|p] output format type.")
rootCmd.PersistentFlags().BoolVarP(&nullInput,"null-input","n",false,"Don't read input, simply evaluate the expression given. Useful for creating yaml docs from scratch.")
rootCmd.PersistentFlags().StringVarP(&outputFormat,"output-format","o","auto",fmt.Sprintf("[auto|a|%v] output format type.",yqlib.GetAvailableOutputFormatString()))
rootCmd.PersistentFlags().StringVarP(&inputFormat,"input-format","p","auto",fmt.Sprintf("[auto|a|%v] parse format for input.",yqlib.GetAvailableInputFormatString()))
rootCmd.PersistentFlags().StringVar(&yqlib.ConfiguredXMLPreferences.AttributePrefix,"xml-attribute-prefix",yqlib.ConfiguredXMLPreferences.AttributePrefix,"prefix for xml attributes")
rootCmd.PersistentFlags().StringVar(&yqlib.ConfiguredXMLPreferences.ContentName,"xml-content-name",yqlib.ConfiguredXMLPreferences.ContentName,"name for xml content (if no attribute name is present).")
rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredXMLPreferences.StrictMode,"xml-strict-mode",yqlib.ConfiguredXMLPreferences.StrictMode,"enables strict parsing of XML. See https://pkg.go.dev/encoding/xml for more details.")
rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredXMLPreferences.KeepNamespace,"xml-keep-namespace",yqlib.ConfiguredXMLPreferences.KeepNamespace,"enables keeping namespace after parsing attributes")
rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredXMLPreferences.UseRawToken,"xml-raw-token",yqlib.ConfiguredXMLPreferences.UseRawToken,"enables using RawToken method instead Token. Commonly disables namespace translations. See https://pkg.go.dev/encoding/xml#Decoder.RawToken for details.")
rootCmd.PersistentFlags().StringVar(&yqlib.ConfiguredXMLPreferences.ProcInstPrefix,"xml-proc-inst-prefix",yqlib.ConfiguredXMLPreferences.ProcInstPrefix,"prefix for xml processing instructions (e.g. <?xml version=\"1\"?>)")
rootCmd.PersistentFlags().StringVar(&yqlib.ConfiguredXMLPreferences.DirectiveName,"xml-directive-name",yqlib.ConfiguredXMLPreferences.DirectiveName,"name for xml directives (e.g. <!DOCTYPE thing cat>)")
rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredXMLPreferences.SkipProcInst,"xml-skip-proc-inst",yqlib.ConfiguredXMLPreferences.SkipProcInst,"skip over process instructions (e.g. <?xml version=\"1\"?>)")
rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredXMLPreferences.SkipDirectives,"xml-skip-directives",yqlib.ConfiguredXMLPreferences.SkipDirectives,"skip over directives (e.g. <!DOCTYPE thing cat>)")
rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredLuaPreferences.Globals,"lua-globals",yqlib.ConfiguredLuaPreferences.Globals,"output keys as top-level global variables")
rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredINIPreferences.PreserveSurroundedQuote,"ini-preserve-quotes",yqlib.ConfiguredINIPreferences.PreserveSurroundedQuote,"preserve surrounding quotes on INI values during round-trip")
rootCmd.PersistentFlags().StringVar(&yqlib.ConfiguredPropertiesPreferences.KeyValueSeparator,"properties-separator",yqlib.ConfiguredPropertiesPreferences.KeyValueSeparator,"separator to use between keys and values")
rootCmd.PersistentFlags().BoolVar(&yqlib.ConfiguredPropertiesPreferences.UseArrayBrackets,"properties-array-brackets",yqlib.ConfiguredPropertiesPreferences.UseArrayBrackets,"use [x] in array paths (e.g. for SpringBoot)")
rootCmd.PersistentFlags().StringVar(&yqlib.ConfiguredShellVariablesPreferences.KeySeparator,"shell-key-separator",yqlib.ConfiguredShellVariablesPreferences.KeySeparator,"separator for shell variable key paths")
rootCmd.PersistentFlags().BoolVar(&yqlib.StringInterpolationEnabled,"string-interpolation",yqlib.StringInterpolationEnabled,"Toggles strings interpolation of \\(exp)")
rootCmd.PersistentFlags().BoolVarP(&nullInput,"null-input","n",false,"Don't read input, simply evaluate the expression given. Useful for creating docs from scratch.")
rootCmd.Flags().BoolVarP(&version,"version","V",false,"Print version information and quit")
rootCmd.PersistentFlags().BoolVarP(&writeInplace,"inplace","i",false,"update the yaml file inplace of first yaml file given.")
rootCmd.PersistentFlags().BoolVarP(&unwrapScalar,"unwrapScalar","",true,"unwrap scalar, print the value with no quotes, colors or comments")
rootCmd.PersistentFlags().BoolVarP(&writeInplace,"inplace","i",false,"update the file inplace of first file given.")
rootCmd.PersistentFlags().VarP(unwrapScalarFlag,"unwrapScalar","r","unwrap scalar, print the value with no quotes, colours or comments. Defaults to true for yaml")
rootCmd.PersistentFlags().BoolVarP(&nulSepOutput,"nul-output","0",false,"Use NUL char to separate values. If unwrap scalar is also set, fail if unwrapped scalar contains NUL char.")
rootCmd.PersistentFlags().BoolVarP(&prettyPrint,"prettyPrint","P",false,"pretty print, shorthand for '... style = \"\"'")
rootCmd.PersistentFlags().BoolVarP(&exitStatus,"exit-status","e",false,"set exit status if there are no matches or null or false is returned")
rootCmd.PersistentFlags().BoolVarP(&forceColor,"colors","C",false,"force print with colors")
rootCmd.PersistentFlags().BoolVarP(&forceNoColor,"no-colors","M",false,"force print with no colors")
rootCmd.PersistentFlags().BoolVarP(&forceNoColor,"no-colors","M",forceNoColor,"force print with no colors")
rootCmd.PersistentFlags().StringVarP(&frontMatter,"front-matter","f","","(extract|process) first input as yaml front-matter. Extract will pull out the yaml content, process will run the expression against the yaml content, leaving the remaining data intact")
rootCmd.PersistentFlags().BoolVarP(&leadingContentPreProcessing,"header-preprocess","",true,"Slurp any header comments and separators before processing expression. This is a workaround for go-yaml to persist header content properly.")
rootCmd.PersistentFlags().StringVarP(&forceExpression,"expression","","","forcibly set the expression argument. Useful when yq argument detection thinks your expression is a file.")
rootCmd.PersistentFlags().BoolVarP(&yqlib.ConfiguredYamlPreferences.LeadingContentPreProcessing,"header-preprocess","",true,"Slurp any header comments and separators before processing expression.")
rootCmd.PersistentFlags().BoolVarP(&yqlib.ConfiguredYamlPreferences.FixMergeAnchorToSpec,"yaml-fix-merge-anchor-to-spec","",false,"Fix merge anchor to match YAML spec. Will default to true in late 2025")
rootCmd.PersistentFlags().BoolVarP(&yqlib.ConfiguredYamlPreferences.CompactSequenceIndent,"yaml-compact-seq-indent","c",false,"Use compact sequence indentation where '- ' is considered part of the indentation.")
rootCmd.PersistentFlags().StringVarP(&splitFileExp,"split-exp","s","","print each result (or doc) into a file named (exp). [exp] argument must return a string. You can use $index in the expression as the result counter. The necessary directories will be created.")
rootCmd.PersistentFlags().BoolVarP(&yqlib.ConfiguredSecurityPreferences.DisableEnvOps,"security-disable-env-ops","",false,"Disable env related operations.")
rootCmd.PersistentFlags().BoolVarP(&yqlib.ConfiguredSecurityPreferences.DisableFileOps,"security-disable-file-ops","",false,"Disable file related operations (e.g. load)")
rootCmd.PersistentFlags().BoolVarP(&yqlib.ConfiguredSecurityPreferences.EnableSystemOps,"security-enable-system-operator","",false,"Enable system operator to allow execution of external commands.")
yqlib.GetLogger().Warningf("yq default output is now 'auto' (based on the filename extension). Normally yq would output '%v', but for backwards compatibility 'yaml' has been set. Please use -oy to specify yaml, or drop the -p flag.",outputFormat)
* Fixed alternative and union operator issues #930
* Can now convert yaml to properties properties format (`-o=props`), See [docs](https://mikefarah.gitbook.io/yq/v/v4.x/usage/properties) for more info.
* Fixed document header/footer comment handling when merging (https://github.com/mikefarah/yq/issues/919)
In `yq`, expressions are made up of operators and pipes. A context of nodes is passed through the expression, and each operation takes the context as input and returns a new context as output. That output is piped in as input for the next operation in the expression.
Let's break down the process step by step using a diagram. We'll start with a single YAML document, apply an expression, and observe how the context changes at each step.
Given a document like:
```yaml
root:
items:
- name:apple
type:fruit
- name:carrot
type:vegetable
- name:banana
type:fruit
```
You can use dot notation to access nested structures. For example, to access the `name` of the first item, you would use the expression `.root.items[0].name`, which would return `apple`.
But lets see how we could find all the fruit under `items`
## Step 1: Initial Context
The context starts at the root of the YAML document. In this case, the entire document is the initial context.
```
root
└── items
├── name: apple
│ type: fruit
├── name: carrot
│ type: vegetable
└── name: banana
type: fruit
```
## Step 2: Splatting the Array
Using the expression `.root.items[]`, we "splat" the items array. This means each element of the array becomes its own node in the context:
```
Node 1: { name: apple, type: fruit }
Node 2: { name: carrot, type: vegetable }
Node 3: { name: banana, type: fruit }
```
## Step 3: Filtering the Nodes
Next, we apply a filter to select only the nodes where type is fruit. The expression `.root.items[] | select(.type == "fruit")` filters the nodes:
```
Filtered Node 1: { name: apple, type: fruit }
Filtered Node 2: { name: banana, type: fruit }
```
## Step 4: Extracting a Field
Finally, we extract the name field from the filtered nodes using `.root.items[] | select(.type == "fruit") | .name` This results in:
```
apple
banana
```
## Simple assignment example
Given a document like:
```yaml
a:cat
b:dog
```
with an expression:
```
.a = .b
```
Like math expressions - operator precedence is important.
The `=` operator takes two arguments, a `lhs` expression, which in this case is `.a` and `rhs` expression which is `.b`.
It pipes the current, lets call it 'root' context through the `lhs` expression of `.a` to return the node
```yaml
cat
```
Side note: this node holds not only its value 'cat', but comments and metadata too, including path and parent information.
The `=` operator then pipes the 'root' context through the `rhs` expression of `.b` to return the node
```yaml
dog
```
Both sides have now been evaluated, so now the operator copies across the value from the RHS (`.b`) to the LHS (`.a`), and it returns the now updated context:
```yaml
a:dog
b:dog
```
## Complex assignment, operator precedence rules
Just like math expressions - `yq` expressions have an order of precedence. The pipe `|` operator has a low order of precedence, so operators with higher precedence will get evaluated first.
Most of the time, this is intuitively what you'd want, for instance `.a = "cat" | .b = "dog"` is effectively: `(.a = "cat") | (.b = "dog")`.
However, this is not always the case, particularly if you have a complex LHS or RHS expression, for instance if you want to select particular nodes to update.
Lets say you had:
```yaml
- name:bob
fruit:apple
- name:sally
fruit:orange
```
Lets say you wanted to update the `sally` entry to have fruit: 'mango'. The _incorrect_ way to do that is:
Because `|` has a low operator precedence, this will be evaluated (_incorrectly_) as : `(.[]) | (select(.name == "sally")) | (.fruit = "mango")`. What you'll see is only the updated segment returned:
```yaml
name:sally
fruit:mango
```
**Important**: To properly update this YAML, you must wrap the entire LHS in parentheses. Think of it like using brackets in math to ensure the correct order of operations.
Now that entire LHS expression is passed to the 'assign' (`=`) operator, and the yaml is correctly updated and returned:
```yaml
- name:bob
fruit:apple
- name:sally
fruit:mango
```
## Relative update (e.g. `|=`)
There is another form of the `=` operator which we call the relative form. It's very similar to `=` but with one key difference when evaluating the RHS expression.
In the plain form, we pass in the 'root' level context to the RHS expression. In relative form, we pass in _each result of the LHS_ to the RHS expression. Let's go through an example.
Given a document like:
```yaml
a:1
b:thing
```
with an expression:
```
.a |= . + 1
```
Similar to the `=` operator, `|=` takes two operands, the LHS and RHS.
It pipes the current context (the whole document) through the LHS expression of `.a` to get the node value:
```
1
```
Now it pipes _that LHS context_ into the RHS expression `. + 1` (whereas in the `=` plain form it piped the original document context into the RHS) to yield:
```
2
```
The assignment operator then copies across the value from the RHS to the value on the LHS, and it returns the now updated 'root' context:
```yaml
a:2
b:thing
```
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
