Add INIPreferences.PreserveSurroundedQuote option that wires through to
go-ini/ini's LoadOptions.PreserveSurroundedQuote. When enabled, existing
surrounding quotes on INI values are preserved during decode/encode
round-trips.
Fixes#2456
Co-authored-by: toller892 <toller892@gmail.com>
YAML scalars tagged `!!float` were round-tripped through `float64` and
re-serialized by Go's JSON encoder, which strips the decimal part of
whole-number floats. As a result, `50.0` came out as `50` and a
sequence like `[50.0, 95.0, 99.0, 99.9]` became `[50,95,99,99.9]`,
turning a uniform array of floats into a mixed int/float array that
downstream consumers (Horreum, JSON Schema validators, jq, etc.)
reject.
The JSON spec does not distinguish ints from floats, but every common
JSON library (Go's `encoding/json`, Python's `json`, jq) preserves the
fractional form of values that came in as floats. yq's YAML decoder
already parses these as `!!float` with the original text intact, so we
can emit them verbatim instead of round-tripping.
`MarshalJSON` for `ScalarNode` now special-cases `!!float`:
- if `Value` is already a JSON-shaped number literal containing a `.`
or exponent, emit it verbatim (e.g. `50.0`, `99.9`, `1.5e-3`, `-7.0`);
- if `Value` is an integer-shaped string tagged `!!float` (e.g.
`!!float 5`), format the parsed float and append `.0` so it stays a
JSON number with a fractional part;
- otherwise (empty value, parse error, or non-finite result), fall back
to the existing encoding path so behaviour for `.inf` / `.nan` and
anything unusual is unchanged.
`!!int` nodes still encode as JSON integers.
Closes#2683
Signed-off-by: ChrisJr404 <chris@hacknow.com>
Ensure root-level TOML attributes are emitted before table sections so fields like sort remain root-scoped. Skip null-valued object fields during TOML encoding
instead of converting them to empty strings.
* Initial plan
* fix: TOML encoder no longer treats YAML FlowStyle as inline tables
Remove FlowStyle checks from the TOML encoder. YAML flow-style mappings
are a YAML-specific rendering hint and should not influence TOML output.
Only nodes explicitly marked with EncodeHintInline (set by the TOML
decoder for actual TOML inline tables) will produce TOML inline table
syntax.
This fixes the bug where JSON auto-detected via the YAML parser (which
parses {} as flow-style mappings) would produce inline TOML tables
instead of readable table sections, while explicitly parsing with
-p json produced correct table sections.
Updated tests: YAML flow mappings now produce table sections (same as
block mappings), consistent with the fix. Added new test cases for the
JSON → TOML conversion via both YAML decoder (auto-detection) and JSON
decoder.
Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/3e504870-b585-4998-af9c-a451e2f6a6a3
Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>
The test literal "ab" * 4611686018427387904 (2^62) exceeds MaxInt32,
so parseInt rejects it before the size guard runs. Compute the count
with 1 << (bits.UintSize - 2) to yield 2^30 on 32-bit and 2^62 on
64-bit. Both values, when doubled by len("ab"), wrap past MaxInt and
bypass a naive len*count guard, exercising the division-safe check
added in #2644.
Signed-off-by: Jan Dubois <jan@jandubois.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Initial plan
* Fix TOML encoder to prefer readable table sections over inline tables
When converting from YAML/JSON to TOML, the encoder now always uses
readable TOML table section syntax ([section]) instead of compact inline
hash table syntax (key = { ... }), which better matches TOML's goal as
a human-focused configuration format.
Changes:
- decoder_toml.go: Mark inline TOML tables with FlowStyle so round-trips
can be distinguished from YAML flow mappings
- encoder_toml.go:
- encodeTopLevelEntry: use FlowStyle check instead of EncodeSeparate to
decide inline vs table section (all block mappings now become tables)
- encodeSeparateMapping: count FlowStyle children as attributes; use
recursive encodeSeparateMapping for nested non-flow mappings
- encodeMappingBodyWithPath: emit non-flow child mappings as sub-table
sections instead of inline tables
- toml_test.go: add encode (YAML→TOML) test scenarios, update roundtrip
expectations for inline tables (now expanded to table sections)
Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/4824a219-6d5e-42e7-bca1-a8a277bf8c6a
Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>
* Fix TOML roundtrip: use TomlInline flag instead of FlowStyle to preserve inline tables
FlowStyle affected YAML decode output (causing inline tables to appear as
YAML flow mappings). Replace it with a new TOML-specific TomlInline bool
on CandidateNode that:
- Is set by the TOML decoder for inline tables (not FlowStyle)
- Is copied by UpdateAttributesFrom so it survives DeeplyAssign merges
- Is checked by the TOML encoder alongside FlowStyle (for YAML flow maps)
- Has no effect on the YAML encoder, preserving existing TOML→YAML output
TOML roundtrip tests are restored to their original expected values (inline
tables stay inline, table sections stay as sections).
Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/f59bdf62-6d16-4664-991b-38eb87c9d81c
Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>
* Refactor EncodeSeparate+TomlInline into a single EncodeHint enum
Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/24db9a8f-601d-4ccf-ada7-129ed3226bb6
Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>
* Fix stale comment in hasStructuralChildren
Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/24db9a8f-601d-4ccf-ada7-129ed3226bb6
Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>
* Remove unused hasStructuralChildren method from tomlEncoder
Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/2c234b77-28e9-4995-ba6f-9d213ec551a0
Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>
recurseNodeObjectEqual and containsObject both used findInArray to
locate keys in a MappingNode's Content array. findInArray steps by 1,
so it matches against both keys (even indices) and values (odd indices).
In recurseNodeObjectEqual, when a null key in the LHS matched a null
value in the RHS at the last position, rhs.Content[indexInRHS+1]
accessed an out-of-bounds index, causing a panic.
In containsObject, a %2 guard prevented the panic but introduced false
negatives: when a null value appeared before the actual null key,
findInArray returned the value's odd index, the guard rejected it, and
the function reported the key as missing.
Both functions now use findKeyInMap, which steps by 2 and compares only
key positions. The %2 guard in containsObject is removed.
Reproducer for the panic (recurseNodeObjectEqual):
echo '? [{~: ~}]
: v1
? [{2: ~}]
: v2' | yq '. += .'
Reproducer for the false negative (containsObject):
printf '? 1\n: ~\n? ~\n: x\n' | yq 'contains({~: "x"})'
Found by OSS-Fuzz via the lima project's FuzzEvaluateExpression target.
https://issues.oss-fuzz.com/issues/383860504
Signed-off-by: Jan Dubois <jan@jandubois.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
sliceArrayOperator adjusts negative indices by adding Content length,
but does not clamp the result. When the absolute value of a negative
index exceeds Content length (e.g. .[-99999:3] on a 3-element array),
the adjusted index remains negative and causes an out-of-bounds access
in the Content slice loop.
Extract the adjust-and-clamp logic into clampSliceIndex and use it for
both index positions.
Reproducer (panics before this fix, returns full array after):
echo '[a, b, c]' | yq '.[-99999:3]'
Found by OSS-Fuzz via the lima project's FuzzEvaluateExpression target.
https://issues.oss-fuzz.com/issues/438776028
Signed-off-by: Jan Dubois <jan@jandubois.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
go-yaml accepts cross-document alias references, which the YAML spec
forbids (anchors are scoped to a single document). When a nested
assignment targets such an alias, UpdateFrom copies the Alias field
between nodes, creating a self-referencing AliasNode. Both traverse()
and traverseArrayIndices() then follow this cycle indefinitely.
Extract resolveAliasChain(), which follows aliases iteratively with a
visited set and returns an error on cycles. Both traverse() and
traverseArrayIndices() now call it, eliminating the recursive alias
handling in both code paths.
Note: traverseMergeAnchor() also dereferences aliases (lines 358 and
371) but with single-step assignment, not recursion. A self-referencing
alias there falls through the kind switch silently rather than
crashing. Using resolveAliasChain() in that function would produce a
clear error instead of silently dropping the node.
Reproducer (stack overflow before this fix, returns error after):
echo '&-- a
---
*--' | yq eval-all '. = (.x = 1)'
Found by OSS-Fuzz via the lima project's FuzzEvaluateExpression target.
https://issues.oss-fuzz.com/issues/390467412
Signed-off-by: Jan Dubois <jan@jandubois.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The existing check (count > 10 million) does not account for string
length. A 68-byte string repeated 35 trillion times passes the count
check but panics in strings.Repeat with "makeslice: len out of range".
Smaller counts (e.g. 10 million * 6-byte string = 60 MB) cause OOM on
memory-constrained environments like OSS-Fuzz (2560 MB limit).
Replace the count-only check with a result size check: the product of
string length and repeat count must not exceed 10 MiB. Use division
(len > limit/count) instead of multiplication (len*count > limit) to
avoid integer overflow — a large count can wrap the product to a
negative value, bypassing the guard entirely.
Fixes at least four OSS-Fuzz bugs found via Lima's FuzzEvaluateExpression:
https://issues.oss-fuzz.com/issues/418818862 (makeslice overflow)
https://issues.oss-fuzz.com/issues/422001683 (timeout from huge alloc)
https://issues.oss-fuzz.com/issues/383195001 (OOM, 3 GB allocation)
https://issues.oss-fuzz.com/issues/385180606 (OOM, 97 TB allocation)
Signed-off-by: Jan Dubois <jan@jandubois.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When using --front-matter, yq creates a temporary file for the
extracted YAML content but replaces the original filename in args
with the temp file path. This caused the 'filename' operator to
return the temp file path instead of the original filename.
Added a filename alias mechanism: when front matter processing
replaces the file path, it registers the original filename as an
alias. The readDocuments and stream evaluator functions resolve
aliases before setting candidateNode.filename.
Fixes#2538
Co-authored-by: cobyfrombrooklyn-bot <cobyfrombrooklyn@gmail.com>
Standalone TOML comments immediately inside a table/array-table no longer end the table scope, preventing subsequent keys from being flattened to the document root.
Adds a new CLI flag that enables compact sequence indentation where '- ' is
considered part of the indentation. This leverages the CompactSeqIndent()
method from the underlying go.yaml.in/yaml/v4 library.
Example output with --yaml-compact-seq-indent:
parent:
items:
- one
- two
Instead of the default:
parent:
items:
- one
- two
Closes#1841
The colorizeToml function intended to force colors by setting
color.NoColor = false, but SprintFunc() still respects the NO_COLOR
environment variable. This caused TestTomlColourization to fail in
CI environments where NO_COLOR=1 is set.
Fixed by calling EnableColor() on each color object, which explicitly
forces colors regardless of environment settings.
Vibe-coded with Cursor (Claude Opus 4)
* feat: Add UnwrapScalar to ShellVariablesPreferences
- Add UnwrapScalar boolean field to ShellVariablesPreferences struct.
- Initialize UnwrapScalar to false in NewDefaultShellVariablesPreferences.
- This preference will control whether shell output should be quoted or raw.
* feat: Propagate unwrapScalar to ShellVariablesPreferences
- In configureEncoder function, set UnwrapScalar in ConfiguredShellVariablesPreferences.
- This ensures the -r flag's state is passed to the shell encoder for raw output control.
* feat: Implement conditional quoting in shellVariablesEncoder
- Modify doEncode method to check pe.prefs.UnwrapScalar.
- If UnwrapScalar is true, output raw node.Value.
- Otherwise, use quoteValue for shell-safe quoting.
- This enables quote-free output for Kubernetes workflows when -r is used.
* test: Add tests for UnwrapScalar in shell encoder
- Introduce assertEncodesToUnwrapped helper function.
- Add TestShellVariablesEncoderUnwrapScalar to verify quote-free output with -r.
- Add TestShellVariablesEncoderDefaultQuoting to confirm default quoting behavior without -r.
- Ensure comprehensive testing of conditional quoting logic for shell output.
* remove redundant test
* feat: K8S KYAML output format support
Reference: https://github.com/kubernetes/enhancements/blob/master/keps/sig-cli/5295-kyaml/README.md
Co-authored-by: Codex <codex@openai.com>
Generated-with: OpenAI Codex CLI (partial)
Signed-off-by: Robin H. Johnson <rjohnson@coreweave.com>
* build: gomodcache/gocache should not be committed
Signed-off-by: Robin H. Johnson <rjohnson@coreweave.com>
* chore: fix spelling of behaviour
Signed-off-by: Robin H. Johnson <robbat2@gentoo.org>
* build: pass GOFLAGS to docker to support buildvcs=false
In trying to develop the KYAML support, various tests gave false
positive results because they made assumptions about Git functionality
Make it possible to avoid that by passing GOFLAGS='-buildvcs=false' to
to Makefile.
Signed-off-by: Robin H. Johnson <robbat2@gentoo.org>
* doc: cover documentScenarios for tests
Signed-off-by: Robin H. Johnson <rjohnson@coreweave.com>
* build: exclude go caches from gosec
Without tuning, gosec scans all of the vendor/gocache/gomodcache, taking
several minutes (3m35 here), whereas the core of the yq takes only 15
seconds to scan.
If we intend to remediate upstream issues in future; add a seperate
target to scan those.
Signed-off-by: Robin H. Johnson <rjohnson@coreweave.com>
---------
Signed-off-by: Robin H. Johnson <rjohnson@coreweave.com>
Signed-off-by: Robin H. Johnson <robbat2@gentoo.org>
Co-authored-by: Codex <codex@openai.com>
