Compare commits

...
Author SHA1 Message Date
Mike Farah c8f6c1a042 Updating release to sign checksums 2026-04-12 19:39:01 +10:00
CopilotGitHubmikefarahcopilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
0e803833fb chore: pin GitHub Actions and Docker base images to full-length hashes (OSSF scorecard) (#2658)
* Initial plan

* chore: pin GitHub Actions dependencies to specific commit SHAs (OSSF)

Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/cbd03f0a-f2dc-4da4-b01c-7dd06ad83ee9

Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>

* chore: pin Dockerfile base images to specific SHA digests (OSSF)

Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/7a8f6690-37fb-42ab-b3dc-0dd23c270fbe

Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>

* chore: revert yq pins in test-yq.yml; add release note for github-action/Dockerfile SHA

Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/e1b35d79-92a3-47d5-b4ac-a2efe2fd58ce

Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>
2026-04-12 19:31:32 +10:00
CopilotGitHubmikefarahcopilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
30ca9ffde7 Add SECURITY.md security policy (#2660)
* Initial plan

* Add SECURITY.md with security policy

Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/f9ff8a4c-addc-485b-abb8-4103394851a4

Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>
2026-04-12 18:57:11 +10:00
CopilotGitHubmikefarahcopilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
2927a28283 TOML encoder: prefer readable table sections over inline tables (#2649)
* Initial plan

* Fix TOML encoder to prefer readable table sections over inline tables

When converting from YAML/JSON to TOML, the encoder now always uses
readable TOML table section syntax ([section]) instead of compact inline
hash table syntax (key = { ... }), which better matches TOML's goal as
a human-focused configuration format.

Changes:
- decoder_toml.go: Mark inline TOML tables with FlowStyle so round-trips
  can be distinguished from YAML flow mappings
- encoder_toml.go:
  - encodeTopLevelEntry: use FlowStyle check instead of EncodeSeparate to
    decide inline vs table section (all block mappings now become tables)
  - encodeSeparateMapping: count FlowStyle children as attributes; use
    recursive encodeSeparateMapping for nested non-flow mappings
  - encodeMappingBodyWithPath: emit non-flow child mappings as sub-table
    sections instead of inline tables
- toml_test.go: add encode (YAML→TOML) test scenarios, update roundtrip
  expectations for inline tables (now expanded to table sections)

Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/4824a219-6d5e-42e7-bca1-a8a277bf8c6a

Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>

* Fix TOML roundtrip: use TomlInline flag instead of FlowStyle to preserve inline tables

FlowStyle affected YAML decode output (causing inline tables to appear as
YAML flow mappings). Replace it with a new TOML-specific TomlInline bool
on CandidateNode that:
- Is set by the TOML decoder for inline tables (not FlowStyle)
- Is copied by UpdateAttributesFrom so it survives DeeplyAssign merges
- Is checked by the TOML encoder alongside FlowStyle (for YAML flow maps)
- Has no effect on the YAML encoder, preserving existing TOML→YAML output

TOML roundtrip tests are restored to their original expected values (inline
tables stay inline, table sections stay as sections).

Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/f59bdf62-6d16-4664-991b-38eb87c9d81c

Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>

* Refactor EncodeSeparate+TomlInline into a single EncodeHint enum

Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/24db9a8f-601d-4ccf-ada7-129ed3226bb6

Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>

* Fix stale comment in hasStructuralChildren

Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/24db9a8f-601d-4ccf-ada7-129ed3226bb6

Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>

* Remove unused hasStructuralChildren method from tomlEncoder

Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/2c234b77-28e9-4995-ba6f-9d213ec551a0

Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>
2026-04-12 18:36:43 +10:00
CopilotGitHubmikefarahcopilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
c47fe40a30 Fix TOML encoder to quote keys containing special characters (#2648)
* Initial plan

* Fix TOML encoder to quote keys with special characters

Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/b2b52954-d13f-4e67-831a-16fdd3378de5

Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>

* Add test for dotted table section header with special character key

Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/12c783dd-8b7f-43bf-b71a-e7a0b5e55fea

Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>

* Apply De Morgan's law to tomlKey condition to fix staticcheck QF1001

Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/eeab0316-309f-418f-b357-11bbacffb471

Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>
2026-04-12 14:27:20 +10:00
dependabot[bot]GitHubmikefarahdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
8c018da9c9 Bump go.yaml.in/yaml/v4 from 4.0.0-rc.3 to 4.0.0-rc.4 (#2579)
* Bump go.yaml.in/yaml/v4 from 4.0.0-rc.3 to 4.0.0-rc.4

Bumps [go.yaml.in/yaml/v4](https://github.com/yaml/go-yaml) from 4.0.0-rc.3 to 4.0.0-rc.4.
- [Commits](https://github.com/yaml/go-yaml/compare/v4.0.0-rc.3...v4.0.0-rc.4)

---
updated-dependencies:
- dependency-name: go.yaml.in/yaml/v4
  dependency-version: 4.0.0-rc.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Fix test expectations for go.yaml.in/yaml/v4 rc.4 error message changes

Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/e172bcc4-f547-4c9f-bcc5-ba61849d37e5

Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>
2026-04-12 14:26:15 +10:00
CopilotGitHubmikefarahCopilotcopilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>Mike Farah
44c55c8a54 Add system(command; args) operator (disabled by default) (#2640)
* Initial plan

* Add system(command; args) operator with --enable-system-operator flag

Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/8a11e9a0-10d2-4f2a-ae29-4e9d0bfc266f

Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>

* Update pkg/yqlib/operator_system.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Evaluate system command/args per matched node using SingleReadonlyChildContext

Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/dca841eb-3f63-4f23-adeb-556431560420

Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>

* Add yqFlags to expressionScenario for doc command snippets; fix system op docs

Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/3f8a5375-25fd-4428-a8e6-b630194c36b2

Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>

* Update pkg/yqlib/doc/operators/headers/system-operators.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update pkg/yqlib/doc/operators/system-operators.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Validate command node type and handle multiple results with debug log

Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/928aabc5-ad71-41d8-94ab-403942e3f92d

Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Remove deprecated --enable-system-operator alias; use --security-enable-system-operator consistently

Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/286b95e9-b6d7-4ab8-b401-2d7a03853922

Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Address deep review feedback: error on disabled, strict arg/cmd validation, debug logs, docs

Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/fbfba2db-60ea-4c20-a4c2-0fd396b80c81

Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>
Co-authored-by: Mike Farah <mikefarah@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-04-10 20:06:46 +10:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
22e609b2d9 Bump golang from 1.26.1 to 1.26.2 (#2654)
Bumps golang from 1.26.1 to 1.26.2.

---
updated-dependencies:
- dependency-name: golang
  dependency-version: 1.26.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-10 18:57:30 +10:00
CopilotGitHubmikefarahCopilotcopilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>Mike Farah
3b2423e871 Add string slicing support (#2639)
* Initial plan

* Add string slicing support to yq

Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/a8525fbb-77a7-4bb0-a3a7-b24f99ae8710

Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>

* Fix sliceStringNode signature and fix test descriptions/expressions

Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/58726b13-68ae-4f93-971f-eb70459edcf4

Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>

* Update pkg/yqlib/operator_slice.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Fix array slice out-of-bounds panic with very negative indices

Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/7c146762-d251-45fd-8555-2488f59fc57b

Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>

* S2-S4: tighten lexer condition, fix doc header, add Unicode example

Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/ec06083e-e20a-45d2-bf7e-4e1fa7be1073

Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>

* Fix spelling: multibyte -> multi-byte in Unicode test subdescription

Agent-Logs-Url: https://github.com/mikefarah/yq/sessions/6e7b304b-5b52-4e89-8bad-ba22813305c7

Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: mikefarah <1151925+mikefarah@users.noreply.github.com>
Co-authored-by: Mike Farah <mikefarah@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-04-06 19:29:07 +10:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
68f0322ba3 Bump softprops/action-gh-release from 1 to 2 (#1978)
Bumps [softprops/action-gh-release](https://github.com/softprops/action-gh-release) from 1 to 2.
- [Release notes](https://github.com/softprops/action-gh-release/releases)
- [Changelog](https://github.com/softprops/action-gh-release/blob/master/CHANGELOG.md)
- [Commits](https://github.com/softprops/action-gh-release/compare/v1...v2)

---
updated-dependencies:
- dependency-name: softprops/action-gh-release
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-06 18:53:44 +10:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
d69c7d1a36 Bump github.com/yuin/gopher-lua from 1.1.1 to 1.1.2 (#2642)
Bumps [github.com/yuin/gopher-lua](https://github.com/yuin/gopher-lua) from 1.1.1 to 1.1.2.
- [Release notes](https://github.com/yuin/gopher-lua/releases)
- [Commits](https://github.com/yuin/gopher-lua/compare/v1.1.1...v1.1.2)

---
updated-dependencies:
- dependency-name: github.com/yuin/gopher-lua
  dependency-version: 1.1.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-06 18:45:44 +10:00
b0ba9589d7 Fix findInArray misuse on MappingNodes in equality and contains (#2645)
recurseNodeObjectEqual and containsObject both used findInArray to
locate keys in a MappingNode's Content array. findInArray steps by 1,
so it matches against both keys (even indices) and values (odd indices).

In recurseNodeObjectEqual, when a null key in the LHS matched a null
value in the RHS at the last position, rhs.Content[indexInRHS+1]
accessed an out-of-bounds index, causing a panic.

In containsObject, a %2 guard prevented the panic but introduced false
negatives: when a null value appeared before the actual null key,
findInArray returned the value's odd index, the guard rejected it, and
the function reported the key as missing.

Both functions now use findKeyInMap, which steps by 2 and compares only
key positions. The %2 guard in containsObject is removed.

Reproducer for the panic (recurseNodeObjectEqual):

    echo '? [{~: ~}]
    : v1
    ? [{2: ~}]
    : v2' | yq '. += .'

Reproducer for the false negative (containsObject):

    printf '? 1\n: ~\n? ~\n: x\n' | yq 'contains({~: "x"})'

Found by OSS-Fuzz via the lima project's FuzzEvaluateExpression target.
https://issues.oss-fuzz.com/issues/383860504

Signed-off-by: Jan Dubois <jan@jandubois.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 18:30:44 +10:00
80139ae1cc Fix panic on negative slice indices that underflow after adjustment (#2646)
sliceArrayOperator adjusts negative indices by adding Content length,
but does not clamp the result. When the absolute value of a negative
index exceeds Content length (e.g. .[-99999:3] on a 3-element array),
the adjusted index remains negative and causes an out-of-bounds access
in the Content slice loop.

Extract the adjust-and-clamp logic into clampSliceIndex and use it for
both index positions.

Reproducer (panics before this fix, returns full array after):

    echo '[a, b, c]' | yq '.[-99999:3]'

Found by OSS-Fuzz via the lima project's FuzzEvaluateExpression target.
https://issues.oss-fuzz.com/issues/438776028

Signed-off-by: Jan Dubois <jan@jandubois.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 18:27:02 +10:00
0374ad6b4b Fix stack overflow from circular alias in traverse (#2647)
go-yaml accepts cross-document alias references, which the YAML spec
forbids (anchors are scoped to a single document). When a nested
assignment targets such an alias, UpdateFrom copies the Alias field
between nodes, creating a self-referencing AliasNode. Both traverse()
and traverseArrayIndices() then follow this cycle indefinitely.

Extract resolveAliasChain(), which follows aliases iteratively with a
visited set and returns an error on cycles. Both traverse() and
traverseArrayIndices() now call it, eliminating the recursive alias
handling in both code paths.

Note: traverseMergeAnchor() also dereferences aliases (lines 358 and
371) but with single-step assignment, not recursion. A self-referencing
alias there falls through the kind switch silently rather than
crashing. Using resolveAliasChain() in that function would produce a
clear error instead of silently dropping the node.

Reproducer (stack overflow before this fix, returns error after):

    echo '&-- a
    ---
    *--' | yq eval-all '. = (.x = 1)'

Found by OSS-Fuzz via the lima project's FuzzEvaluateExpression target.
https://issues.oss-fuzz.com/issues/390467412

Signed-off-by: Jan Dubois <jan@jandubois.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 18:25:13 +10:00
2ef934281e Fix panic and OOM in repeatString for large repeat counts (#2644)
The existing check (count > 10 million) does not account for string
length. A 68-byte string repeated 35 trillion times passes the count
check but panics in strings.Repeat with "makeslice: len out of range".
Smaller counts (e.g. 10 million * 6-byte string = 60 MB) cause OOM on
memory-constrained environments like OSS-Fuzz (2560 MB limit).

Replace the count-only check with a result size check: the product of
string length and repeat count must not exceed 10 MiB. Use division
(len > limit/count) instead of multiplication (len*count > limit) to
avoid integer overflow — a large count can wrap the product to a
negative value, bypassing the guard entirely.

Fixes at least four OSS-Fuzz bugs found via Lima's FuzzEvaluateExpression:
  https://issues.oss-fuzz.com/issues/418818862 (makeslice overflow)
  https://issues.oss-fuzz.com/issues/422001683 (timeout from huge alloc)
  https://issues.oss-fuzz.com/issues/383195001 (OOM, 3 GB allocation)
  https://issues.oss-fuzz.com/issues/385180606 (OOM, 97 TB allocation)

Signed-off-by: Jan Dubois <jan@jandubois.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 18:22:46 +10:00
44 changed files with 1087 additions and 154 deletions
+4 -4
View File
@@ -38,11 +38,11 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@v6
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@v4
uses: github/codeql-action/init@c10b8064de6f491fea524254123dbe5e09572f13 # v4.35.1
with:
languages: ${{ matrix.language }}
# If you wish to specify custom queries, you can do so here or in a config file.
@@ -53,7 +53,7 @@ jobs:
# Autobuild attempts to build any compiled languages (C/C++, C#, or Java).
# If this step fails, then you should remove it and run the build manually (see below)
- name: Autobuild
uses: github/codeql-action/autobuild@v4
uses: github/codeql-action/autobuild@c10b8064de6f491fea524254123dbe5e09572f13 # v4.35.1
# ️ Command-line programs to run using the OS shell.
# 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun
@@ -67,4 +67,4 @@ jobs:
# make release
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v4
uses: github/codeql-action/analyze@c10b8064de6f491fea524254123dbe5e09572f13 # v4.35.1
+5 -5
View File
@@ -14,16 +14,16 @@ jobs:
IMAGE_NAME: mikefarah/yq
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Set up QEMU
uses: docker/setup-qemu-action@v4
uses: docker/setup-qemu-action@ce360397dd3f832beb865e1373c09c0e9f86d70a # v4.0.0
with:
platforms: all
- name: Set up Docker Buildx
id: buildx
uses: docker/setup-buildx-action@v4
uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0
with:
version: latest
@@ -31,13 +31,13 @@ jobs:
run: echo ${{ steps.buildx.outputs.platforms }} && docker version
- name: Login to Docker Hub
uses: docker/login-action@v4
uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Login to GitHub Container Registry
uses: docker/login-action@v4
uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
with:
registry: ghcr.io
username: ${{ github.actor }}
+2 -2
View File
@@ -11,13 +11,13 @@ jobs:
steps:
- name: Set up Go
uses: actions/setup-go@v6
uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6.4.0
with:
go-version: '^1.20'
id: go
- name: Check out code into the Go module directory
uses: actions/checkout@v6
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Get dependencies
run: |
+14 -3
View File
@@ -8,9 +8,12 @@ on:
jobs:
publishGitRelease:
runs-on: ubuntu-latest
permissions:
contents: write
id-token: write
steps:
- uses: actions/checkout@v6
- uses: actions/setup-go@v6
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6.4.0
with:
go-version: '^1.20'
check-latest: true
@@ -37,14 +40,22 @@ jobs:
--output=yq.1
man.md
- name: Install cosign
uses: sigstore/cosign-installer@v3
- name: Cross compile
run: |
sudo apt-get install rhash -y
go install github.com/goreleaser/goreleaser/v2@latest
./scripts/xcompile.sh
- name: Sign checksums
run: |
cosign sign-blob --yes --output-bundle build/checksums.bundle build/checksums
cosign sign-blob --yes --output-bundle build/checksums-bsd.bundle build/checksums-bsd
- name: Release
uses: softprops/action-gh-release@v1
uses: softprops/action-gh-release@3bb12739c298aeb8a4eeaf626c5b8d85266b0e65 # v2.6.2
with:
files: build/*
draft: true
+3 -3
View File
@@ -12,14 +12,14 @@ jobs:
environment: snap
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- uses: snapcore/action-build@v1
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: snapcore/action-build@3bdaa03e1ba6bf59a65f84a751d943d549a54e79 # v1.3.0
id: build
env:
SNAPCRAFT_STORE_CREDENTIALS: ${{ secrets.STORE_LOGIN }}
with:
snapcraft-args: "remote-build --launchpad-accept-public-upload"
- uses: snapcore/action-publish@v1
- uses: snapcore/action-publish@214b86e5ca036ead1668c79afb81e550e6c54d40 # v1.2.0
env:
SNAPCRAFT_STORE_CREDENTIALS: ${{ secrets.STORE_LOGIN }}
with:
+1 -1
View File
@@ -13,7 +13,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Get test
id: get_value
uses: mikefarah/yq@master
+2 -2
View File
@@ -1,4 +1,4 @@
FROM golang:1.26.1 AS builder
FROM golang:1.26.2@sha256:2a2b4b5791cea8ae09caecba7bad0bd9631def96e5fe362e4a5e67009fe4ae61 AS builder
WORKDIR /go/src/mikefarah/yq
@@ -10,7 +10,7 @@ RUN ./scripts/acceptance.sh
# Choose alpine as a base image to make this useful for CI, as many
# CI tools expect an interactive shell inside the container
FROM alpine:3 AS production
FROM alpine:3@sha256:25109184c71bdad752c8312a8623239686a9a2071e8825f20acb8f2198c3f659 AS production
LABEL maintainer="Mike Farah <mikefarah@users.noreply.github.com>"
COPY --from=builder /go/src/mikefarah/yq/yq /usr/bin/yq
+1 -1
View File
@@ -1,4 +1,4 @@
FROM golang:1.26.1
FROM golang:1.26.2@sha256:2a2b4b5791cea8ae09caecba7bad0bd9631def96e5fe362e4a5e67009fe4ae61
RUN apt-get update && \
apt-get install -y npm && \
+26
View File
@@ -0,0 +1,26 @@
# Security Policy
## Reporting a Vulnerability
Please **do not** report security vulnerabilities through public GitHub issues.
Instead, use GitHub's private vulnerability reporting feature:
👉 https://github.com/mikefarah/yq/security
This allows vulnerabilities to be triaged and addressed confidentially before any public disclosure.
## Scope
### HTTP / TLS / Network vulnerabilities
yq is a command-line YAML/JSON/TOML processor that reads from files or standard input and writes to standard output. **yq does not include any HTTP or network libraries** and makes no network connections at runtime. CVEs related to HTTP, TLS, or networking are therefore **not applicable** to yq.
### Dependency version bumps
yq uses [Dependabot](https://docs.github.com/en/code-security/dependabot) to automatically raise pull requests for:
- Go module dependencies
- Go toolchain version
- Docker base images
Please **do not** raise pull requests or issues solely to bump dependency or Go versions — Dependabot handles this automatically and the maintainers merge those PRs regularly.
+1
View File
@@ -212,6 +212,7 @@ yq -P -oy sample.json
rootCmd.PersistentFlags().BoolVarP(&yqlib.ConfiguredSecurityPreferences.DisableEnvOps, "security-disable-env-ops", "", false, "Disable env related operations.")
rootCmd.PersistentFlags().BoolVarP(&yqlib.ConfiguredSecurityPreferences.DisableFileOps, "security-disable-file-ops", "", false, "Disable file related operations (e.g. load)")
rootCmd.PersistentFlags().BoolVarP(&yqlib.ConfiguredSecurityPreferences.EnableSystemOps, "security-enable-system-operator", "", false, "Enable system operator to allow execution of external commands.")
rootCmd.AddCommand(
createEvaluateSequenceCommand(),
+1 -1
View File
@@ -1,4 +1,4 @@
FROM mikefarah/yq:4
FROM mikefarah/yq:4@sha256:603ebff15eb308a05f1c5b8b7613179cad859aed3ec9fdd04f2ef5d32345950e
COPY entrypoint.sh /entrypoint.sh
+2 -2
View File
@@ -17,9 +17,9 @@ require (
github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e
github.com/spf13/cobra v1.10.2
github.com/spf13/pflag v1.0.10
github.com/yuin/gopher-lua v1.1.1
github.com/yuin/gopher-lua v1.1.2
github.com/zclconf/go-cty v1.18.0
go.yaml.in/yaml/v4 v4.0.0-rc.3
go.yaml.in/yaml/v4 v4.0.0-rc.4
golang.org/x/mod v0.34.0
golang.org/x/net v0.52.0
golang.org/x/text v0.35.0
+4 -4
View File
@@ -61,15 +61,15 @@ github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3A
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.7.0 h1:nwc3DEeHmmLAfoZucVR881uASk0Mfjw8xYJ99tb5CcY=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/yuin/gopher-lua v1.1.1 h1:kYKnWBjvbNP4XLT3+bPEwAXJx262OhaHDWDVOPjL46M=
github.com/yuin/gopher-lua v1.1.1/go.mod h1:GBR0iDaNXjAgGg9zfCvksxSRnQx76gclCIb7kdAd1Pw=
github.com/yuin/gopher-lua v1.1.2 h1:yF/FjE3hD65tBbt0VXLE13HWS9h34fdzJmrWRXwobGA=
github.com/yuin/gopher-lua v1.1.2/go.mod h1:7aRmXIWl37SqRf0koeyylBEzJ+aPt8A+mmkQ4f1ntR8=
github.com/zclconf/go-cty v1.18.0 h1:pJ8+HNI4gFoyRNqVE37wWbJWVw43BZczFo7KUoRczaA=
github.com/zclconf/go-cty v1.18.0/go.mod h1:qpnV6EDNgC1sns/AleL1fvatHw72j+S+nS+MJ+T2CSg=
github.com/zclconf/go-cty-debug v0.0.0-20240509010212-0d6042c53940 h1:4r45xpDWB6ZMSMNJFMOjqrGHynW3DIBuR2H9j0ug+Mo=
github.com/zclconf/go-cty-debug v0.0.0-20240509010212-0d6042c53940/go.mod h1:CmBdvvj3nqzfzJ6nTCIwDTPZ56aVGvDrmztiO5g3qrM=
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
go.yaml.in/yaml/v4 v4.0.0-rc.3 h1:3h1fjsh1CTAPjW7q/EMe+C8shx5d8ctzZTrLcs/j8Go=
go.yaml.in/yaml/v4 v4.0.0-rc.3/go.mod h1:aZqd9kCMsGL7AuUv/m/PvWLdg5sjJsZ4oHDEnfPPfY0=
go.yaml.in/yaml/v4 v4.0.0-rc.4 h1:UP4+v6fFrBIb1l934bDl//mmnoIZEDK0idg1+AIvX5U=
go.yaml.in/yaml/v4 v4.0.0-rc.4/go.mod h1:aZqd9kCMsGL7AuUv/m/PvWLdg5sjJsZ4oHDEnfPPfY0=
golang.org/x/mod v0.34.0 h1:xIHgNUUnW6sYkcM5Jleh05DvLOtwc6RitGHbDk4akRI=
golang.org/x/mod v0.34.0/go.mod h1:ykgH52iCZe79kzLLMhyCUzhMci+nQj+0XkbXpNYtVjY=
golang.org/x/net v0.52.0 h1:He/TN1l0e4mmR3QqHMT2Xab3Aj3L9qjbhRm78/6jrW0=
+22 -6
View File
@@ -27,6 +27,22 @@ const (
FlowStyle
)
// EncodeHint controls how a mapping node is serialised by format-specific encoders
// that distinguish between inline and block/section representations (e.g. TOML, HCL).
type EncodeHint int
const (
// EncodeHintDefault lets the encoder choose the representation (e.g. TOML block
// mappings default to [section] headers).
EncodeHintDefault EncodeHint = iota
// EncodeHintSeparateBlock forces the node to be emitted as a separate block or
// table-section header (used by TOML [section] and HCL block decoders).
EncodeHintSeparateBlock
// EncodeHintInline forces the node to be emitted as an inline / flow table
// (used by TOML inline-table decoder and TOML encoder).
EncodeHintInline
)
func createStringScalarNode(stringValue string) *CandidateNode {
var node = &CandidateNode{Kind: ScalarNode}
node.Value = stringValue
@@ -97,9 +113,9 @@ type CandidateNode struct {
// (e.g. top level cross document merge). This property does not propagate to child nodes.
EvaluateTogether bool
IsMapKey bool
// For formats like HCL and TOML: indicates that child entries should be emitted as separate blocks/tables
// rather than consolidated into nested mappings (default behaviour)
EncodeSeparate bool
// EncodeHint controls how a mapping node is serialised by format-specific encoders
// (e.g. TOML, HCL) that support both inline and block/section representations.
EncodeHint EncodeHint
}
func (n *CandidateNode) CreateChild() *CandidateNode {
@@ -411,7 +427,7 @@ func (n *CandidateNode) doCopy(cloneContent bool) *CandidateNode {
EvaluateTogether: n.EvaluateTogether,
IsMapKey: n.IsMapKey,
EncodeSeparate: n.EncodeSeparate,
EncodeHint: n.EncodeHint,
}
if cloneContent {
@@ -465,8 +481,8 @@ func (n *CandidateNode) UpdateAttributesFrom(other *CandidateNode, prefs assignP
n.Anchor = other.Anchor
}
// Preserve EncodeSeparate flag for format-specific encoding hints
n.EncodeSeparate = other.EncodeSeparate
// Preserve EncodeHint for format-specific encoding hints
n.EncodeHint = other.EncodeHint
// merge will pickup the style of the new thing
// when autocreating nodes
+1 -1
View File
@@ -226,7 +226,7 @@ func addBlockToMapping(parent *CandidateNode, block *hclsyntax.Block, src []byte
// Mark the type node if there are multiple blocks of this type at this level
// This tells the encoder to emit them as separate blocks rather than consolidating them
if isMultipleBlocksOfType {
typeNode.EncodeSeparate = true
typeNode.EncodeHint = EncodeHintSeparateBlock
}
}
current = typeNode
+11 -10
View File
@@ -150,9 +150,10 @@ func (dec *tomlDecoder) createInlineTableMap(tomlNode *toml.Node) (*CandidateNod
}
return &CandidateNode{
Kind: MappingNode,
Tag: "!!map",
Content: content,
Kind: MappingNode,
Tag: "!!map",
EncodeHint: EncodeHintInline,
Content: content,
}, nil
}
@@ -345,10 +346,10 @@ func (dec *tomlDecoder) processTable(currentNode *toml.Node) (bool, error) {
}
tableNodeValue := &CandidateNode{
Kind: MappingNode,
Tag: "!!map",
Content: make([]*CandidateNode, 0),
EncodeSeparate: true,
Kind: MappingNode,
Tag: "!!map",
Content: make([]*CandidateNode, 0),
EncodeHint: EncodeHintSeparateBlock,
}
// Attach pending head comments to the table
@@ -442,9 +443,9 @@ func (dec *tomlDecoder) processArrayTable(currentNode *toml.Node) (bool, error)
hasValue := dec.parser.NextExpression()
tableNodeValue := &CandidateNode{
Kind: MappingNode,
Tag: "!!map",
EncodeSeparate: true,
Kind: MappingNode,
Tag: "!!map",
EncodeHint: EncodeHintSeparateBlock,
}
// Attach pending head comments to the array table
@@ -1,5 +1,5 @@
# Slice/Splice Array
# Slice Array or String
The slice array operator takes an array as input and returns a subarray. Like the `jq` equivalent, `.[10:15]` will return an array of length 5, starting from index 10 inclusive, up to index 15 exclusive. Negative numbers count backwards from the end of the array.
The slice operator works on both arrays and strings. Like the `jq` equivalent, `.[10:15]` will return a subarray (or substring) of length 5, starting from index 10 inclusive, up to index 15 exclusive. Negative numbers count backwards from the end of the array or string.
You may leave out the first or second number, which will refer to the start or end of the array respectively.
You may leave out the first or second number, which will refer to the start or end of the array or string respectively.
@@ -0,0 +1,27 @@
# System Operators
The `system` operator allows you to run an external command and use its output as a value in your expression.
**Security warning**: The system operator is disabled by default. You must explicitly pass `--security-enable-system-operator` to use it.
**Note:** When enabled, the system operator can replicate the functionality of `env` and `load`
operators via external commands. Enabling it effectively overrides `--security-disable-env-ops`
and `--security-disable-file-ops`.
## Usage
```bash
yq --security-enable-system-operator --null-input '.field = system("command"; "arg1")'
```
The operator takes:
- A command string (required)
- An argument (or an array of arguments), separated from the command by `;` (optional)
The current matched node's value is serialised and piped to the command via stdin. The command's stdout (with trailing newline stripped) is returned as a string.
## Disabling the system operator
The system operator is disabled by default. When disabled, an error is returned instead of running the command, consistent with `--security-disable-env-ops` and `--security-disable-file-ops`.
Use `--security-enable-system-operator` flag to enable it.
+81 -3
View File
@@ -1,8 +1,8 @@
# Slice/Splice Array
# Slice Array or String
The slice array operator takes an array as input and returns a subarray. Like the `jq` equivalent, `.[10:15]` will return an array of length 5, starting from index 10 inclusive, up to index 15 exclusive. Negative numbers count backwards from the end of the array.
The slice operator works on both arrays and strings. Like the `jq` equivalent, `.[10:15]` will return a subarray (or substring) of length 5, starting from index 10 inclusive, up to index 15 exclusive. Negative numbers count backwards from the end of the array or string.
You may leave out the first or second number, which will refer to the start or end of the array respectively.
You may leave out the first or second number, which will refer to the start or end of the array or string respectively.
## Slicing arrays
Given a sample.yml file of:
@@ -103,3 +103,81 @@ will output
- cow
```
## Slicing strings
Given a sample.yml file of:
```yaml
country: Australia
```
then
```bash
yq '.country[0:5]' sample.yml
```
will output
```yaml
Austr
```
## Slicing strings - without the second number
Finishes at the end of the string
Given a sample.yml file of:
```yaml
country: Australia
```
then
```bash
yq '.country[5:]' sample.yml
```
will output
```yaml
alia
```
## Slicing strings - without the first number
Starts from the start of the string
Given a sample.yml file of:
```yaml
country: Australia
```
then
```bash
yq '.country[:5]' sample.yml
```
will output
```yaml
Austr
```
## Slicing strings - use negative numbers to count backwards from the end
Negative indices count from the end of the string
Given a sample.yml file of:
```yaml
country: Australia
```
then
```bash
yq '.country[-5:]' sample.yml
```
will output
```yaml
ralia
```
## Slicing strings - Unicode
Indices are rune-based, so multi-byte characters are handled correctly
Given a sample.yml file of:
```yaml
greeting: héllo
```
then
```bash
yq '.greeting[1:3]' sample.yml
```
will output
```yaml
él
```
@@ -0,0 +1,76 @@
# System Operators
The `system` operator allows you to run an external command and use its output as a value in your expression.
**Security warning**: The system operator is disabled by default. You must explicitly pass `--security-enable-system-operator` to use it.
**Note:** When enabled, the system operator can replicate the functionality of `env` and `load`
operators via external commands. Enabling it effectively overrides `--security-disable-env-ops`
and `--security-disable-file-ops`.
## Usage
```bash
yq --security-enable-system-operator --null-input '.field = system("command"; "arg1")'
```
The operator takes:
- A command string (required)
- An argument (or an array of arguments), separated from the command by `;` (optional)
The current matched node's value is serialised and piped to the command via stdin. The command's stdout (with trailing newline stripped) is returned as a string.
## Disabling the system operator
The system operator is disabled by default. When disabled, an error is returned instead of running the command, consistent with `--security-disable-env-ops` and `--security-disable-file-ops`.
Use `--security-enable-system-operator` flag to enable it.
## system operator returns error when disabled
Use `--security-enable-system-operator` to enable the system operator.
Given a sample.yml file of:
```yaml
country: Australia
```
then
```bash
yq '.country = system("/usr/bin/echo"; "test")' sample.yml
```
will output
```bash
Error: system operations are disabled, use --security-enable-system-operator to enable
```
## Run a command with an argument
Use `--security-enable-system-operator` to enable the system operator.
Given a sample.yml file of:
```yaml
country: Australia
```
then
```bash
yq --security-enable-system-operator '.country = system("/usr/bin/echo"; "test")' sample.yml
```
will output
```yaml
country: test
```
## Run a command without arguments
Omit the semicolon and args to run the command with no extra arguments.
Given a sample.yml file of:
```yaml
a: hello
```
then
```bash
yq --security-enable-system-operator '.a = system("/usr/bin/echo")' sample.yml
```
will output
```yaml
a: ""
```
+17
View File
@@ -384,3 +384,20 @@ ip = "10.0.0.2"
role = "backend"
```
## Encode: Simple mapping produces table section
Given a sample.yml file of:
```yaml
arg:
hello: foo
```
then
```bash
yq -o toml '.' sample.yml
```
will output
```toml
[arg]
hello = "foo"
```
+4 -4
View File
@@ -449,8 +449,8 @@ func (he *hclEncoder) encodeBlockIfMapping(body *hclwrite.Body, key string, valu
return false
}
// If EncodeSeparate is set, emit children as separate blocks regardless of label extraction
if valueNode.EncodeSeparate {
// If EncodeHintSeparateBlock is set, emit children as separate blocks regardless of label extraction
if valueNode.EncodeHint == EncodeHintSeparateBlock {
if handled, _ := he.encodeMappingChildrenAsBlocks(body, key, valueNode); handled {
return true
}
@@ -537,9 +537,9 @@ func (he *hclEncoder) encodeMappingChildrenAsBlocks(body *hclwrite.Body, blockTy
return false, nil
}
// Only emit as separate blocks if EncodeSeparate is true
// Only emit as separate blocks if EncodeHintSeparateBlock is set
// This allows the encoder to respect the original block structure preserved by the decoder
if !valueNode.EncodeSeparate {
if valueNode.EncodeHint != EncodeHintSeparateBlock {
return false, nil
}
+57 -62
View File
@@ -69,6 +69,27 @@ func (te *tomlEncoder) CanHandleAliases() bool {
// ---- helpers ----
// tomlKey returns the key quoted if it contains characters that are not valid
// in a TOML bare key. TOML bare keys may only contain ASCII letters, ASCII
// digits, underscores, and dashes.
func tomlKey(key string) string {
for _, r := range key {
if (r < 'A' || r > 'Z') && (r < 'a' || r > 'z') && (r < '0' || r > '9') && r != '_' && r != '-' {
return fmt.Sprintf("%q", key)
}
}
return key
}
// tomlDottedKey joins path components, quoting any that require it.
func tomlDottedKey(path []string) string {
parts := make([]string, len(path))
for i, p := range path {
parts[i] = tomlKey(p)
}
return strings.Join(parts, ".")
}
func (te *tomlEncoder) writeComment(w io.Writer, comment string) error {
if comment == "" {
return nil
@@ -148,9 +169,10 @@ func (te *tomlEncoder) encodeTopLevelEntry(w io.Writer, path []string, node *Can
}
if allMaps {
key := path[len(path)-1]
quotedKey := tomlKey(key)
for _, it := range node.Content {
// [[key]] then body
if _, err := w.Write([]byte("[[" + key + "]]\n")); err != nil {
if _, err := w.Write([]byte("[[" + quotedKey + "]]\n")); err != nil {
return err
}
if err := te.encodeMappingBodyWithPath(w, []string{key}, it); err != nil {
@@ -162,12 +184,9 @@ func (te *tomlEncoder) encodeTopLevelEntry(w io.Writer, path []string, node *Can
// Regular array attribute
return te.writeArrayAttribute(w, path[len(path)-1], node)
case MappingNode:
// Inline table if not EncodeSeparate, else emit separate tables/arrays of tables for children under this path
if !node.EncodeSeparate {
// If children contain mappings or arrays of mappings, prefer separate sections
if te.hasEncodeSeparateChild(node) || te.hasStructuralChildren(node) {
return te.encodeSeparateMapping(w, path, node)
}
// Use inline table syntax for nodes explicitly marked as TOML inline tables
// or YAML flow mappings. All other mappings become readable TOML table sections.
if node.EncodeHint == EncodeHintInline || node.Style&FlowStyle != 0 {
return te.writeInlineTableAttribute(w, path[len(path)-1], node)
}
return te.encodeSeparateMapping(w, path, node)
@@ -185,7 +204,7 @@ func (te *tomlEncoder) writeAttribute(w io.Writer, key string, value *CandidateN
}
// Write the attribute
line := key + " = " + te.formatScalar(value)
line := tomlKey(key) + " = " + te.formatScalar(value)
// Add line comment if present
if value.LineComment != "" {
@@ -210,7 +229,7 @@ func (te *tomlEncoder) writeArrayAttribute(w io.Writer, key string, seq *Candida
// Handle empty arrays
if len(seq.Content) == 0 {
line := key + " = []"
line := tomlKey(key) + " = []"
if seq.LineComment != "" {
lineComment := strings.TrimSpace(seq.LineComment)
if !strings.HasPrefix(lineComment, "#") {
@@ -233,7 +252,7 @@ func (te *tomlEncoder) writeArrayAttribute(w io.Writer, key string, seq *Candida
if hasElementComments {
// Write multiline array format with comments
if _, err := w.Write([]byte(key + " = [\n")); err != nil {
if _, err := w.Write([]byte(tomlKey(key) + " = [\n")); err != nil {
return err
}
@@ -324,7 +343,7 @@ func (te *tomlEncoder) writeArrayAttribute(w io.Writer, key string, seq *Candida
}
}
line := key + " = [" + strings.Join(items, ", ") + "]"
line := tomlKey(key) + " = [" + strings.Join(items, ", ") + "]"
// Add line comment if present
if seq.LineComment != "" {
@@ -372,21 +391,21 @@ func (te *tomlEncoder) mappingToInlineTable(m *CandidateNode) (string, error) {
v := m.Content[i+1]
switch v.Kind {
case ScalarNode:
parts = append(parts, fmt.Sprintf("%s = %s", k, te.formatScalar(v)))
parts = append(parts, fmt.Sprintf("%s = %s", tomlKey(k), te.formatScalar(v)))
case SequenceNode:
// inline array in inline table
arr, err := te.sequenceToInlineArray(v)
if err != nil {
return "", err
}
parts = append(parts, fmt.Sprintf("%s = %s", k, arr))
parts = append(parts, fmt.Sprintf("%s = %s", tomlKey(k), arr))
case MappingNode:
// nested inline table
inline, err := te.mappingToInlineTable(v)
if err != nil {
return "", err
}
parts = append(parts, fmt.Sprintf("%s = %s", k, inline))
parts = append(parts, fmt.Sprintf("%s = %s", tomlKey(k), inline))
default:
return "", fmt.Errorf("unsupported inline table value kind: %v", v.Kind)
}
@@ -399,7 +418,7 @@ func (te *tomlEncoder) writeInlineTableAttribute(w io.Writer, key string, m *Can
if err != nil {
return err
}
_, err = w.Write([]byte(key + " = " + inline + "\n"))
_, err = w.Write([]byte(tomlKey(key) + " = " + inline + "\n"))
return err
}
@@ -421,7 +440,7 @@ func (te *tomlEncoder) writeTableHeader(w io.Writer, path []string, m *Candidate
}
// Write table header [a.b.c]
header := "[" + strings.Join(path, ".") + "]\n"
header := "[" + tomlDottedKey(path) + "]\n"
_, err := w.Write([]byte(header))
return err
}
@@ -429,7 +448,8 @@ func (te *tomlEncoder) writeTableHeader(w io.Writer, path []string, m *Candidate
// encodeSeparateMapping handles a mapping that should be encoded as table sections.
// It emits the table header for this mapping if it has any content, then processes children.
func (te *tomlEncoder) encodeSeparateMapping(w io.Writer, path []string, m *CandidateNode) error {
// Check if this mapping has any non-mapping, non-array-of-tables children (i.e., attributes)
// Check if this mapping has any non-mapping, non-array-of-tables children (i.e., attributes).
// Inline mapping children also count as attributes since they render as key = { ... }.
hasAttrs := false
for i := 0; i < len(m.Content); i += 2 {
v := m.Content[i+1]
@@ -437,6 +457,10 @@ func (te *tomlEncoder) encodeSeparateMapping(w io.Writer, path []string, m *Cand
hasAttrs = true
break
}
if v.Kind == MappingNode && (v.EncodeHint == EncodeHintInline || v.Style&FlowStyle != 0) {
hasAttrs = true
break
}
if v.Kind == SequenceNode {
// Check if it's NOT an array of tables
allMaps := true
@@ -464,18 +488,14 @@ func (te *tomlEncoder) encodeSeparateMapping(w io.Writer, path []string, m *Cand
return nil
}
// No attributes, just nested structures - process children
// No attributes, just nested table structures - process children recursively
for i := 0; i < len(m.Content); i += 2 {
k := m.Content[i].Value
v := m.Content[i+1]
switch v.Kind {
case MappingNode:
// Emit [path.k]
newPath := append(append([]string{}, path...), k)
if err := te.writeTableHeader(w, newPath, v); err != nil {
return err
}
if err := te.encodeMappingBodyWithPath(w, newPath, v); err != nil {
if err := te.encodeSeparateMapping(w, newPath, v); err != nil {
return err
}
case SequenceNode:
@@ -488,7 +508,7 @@ func (te *tomlEncoder) encodeSeparateMapping(w io.Writer, path []string, m *Cand
}
}
if allMaps {
key := strings.Join(append(append([]string{}, path...), k), ".")
key := tomlDottedKey(append(append([]string{}, path...), k))
for _, it := range v.Content {
if _, err := w.Write([]byte("[[" + key + "]]\n")); err != nil {
return err
@@ -513,39 +533,6 @@ func (te *tomlEncoder) encodeSeparateMapping(w io.Writer, path []string, m *Cand
return nil
}
func (te *tomlEncoder) hasEncodeSeparateChild(m *CandidateNode) bool {
for i := 0; i < len(m.Content); i += 2 {
v := m.Content[i+1]
if v.Kind == MappingNode && v.EncodeSeparate {
return true
}
}
return false
}
func (te *tomlEncoder) hasStructuralChildren(m *CandidateNode) bool {
for i := 0; i < len(m.Content); i += 2 {
v := m.Content[i+1]
// Only consider it structural if mapping has EncodeSeparate or is non-empty
if v.Kind == MappingNode && v.EncodeSeparate {
return true
}
if v.Kind == SequenceNode {
allMaps := true
for _, it := range v.Content {
if it.Kind != MappingNode {
allMaps = false
break
}
}
if allMaps {
return true
}
}
}
return false
}
// encodeMappingBodyWithPath encodes attributes and nested arrays of tables using full dotted path context
func (te *tomlEncoder) encodeMappingBodyWithPath(w io.Writer, path []string, m *CandidateNode) error {
// First, attributes (scalars and non-map arrays)
@@ -586,7 +573,7 @@ func (te *tomlEncoder) encodeMappingBodyWithPath(w io.Writer, path []string, m *
}
}
if allMaps {
dotted := strings.Join(append(append([]string{}, path...), k), ".")
dotted := tomlDottedKey(append(append([]string{}, path...), k))
for _, it := range v.Content {
if _, err := w.Write([]byte("[[" + dotted + "]]\n")); err != nil {
return err
@@ -599,13 +586,21 @@ func (te *tomlEncoder) encodeMappingBodyWithPath(w io.Writer, path []string, m *
}
}
// Finally, child mappings that are not marked EncodeSeparate get inlined as attributes
// Finally, child mappings: inline-hint or flow-style ones become inline table attributes,
// while all others are emitted as separate sub-table sections.
for i := 0; i < len(m.Content); i += 2 {
k := m.Content[i].Value
v := m.Content[i+1]
if v.Kind == MappingNode && !v.EncodeSeparate {
if err := te.writeInlineTableAttribute(w, k, v); err != nil {
return err
if v.Kind == MappingNode {
if v.EncodeHint == EncodeHintInline || v.Style&FlowStyle != 0 {
if err := te.writeInlineTableAttribute(w, k, v); err != nil {
return err
}
} else {
subPath := append(append([]string{}, path...), k)
if err := te.encodeSeparateMapping(w, subPath, v); err != nil {
return err
}
}
}
}
+5
View File
@@ -131,6 +131,11 @@ func handleToken(tokens []*token, index int, postProcessedTokens []*token) (toke
log.Debugf("previous token is : traverseArrayOpType")
// need to put the number 0 before this token, as that is implied
postProcessedTokens = append(postProcessedTokens, &token{TokenType: operationToken, Operation: createValueOperation(0, "0")})
} else if index >= 2 && tokens[index-1].TokenType == openCollect &&
(tokens[index-2].TokenType == operationToken || tokens[index-2].TokenType == closeCollect || tokens[index-2].TokenType == closeCollectObject) {
log.Debugf("previous token is : openCollect following a traversal, implying 0 start")
// need to put the number 0 before this token, as that is implied
postProcessedTokens = append(postProcessedTokens, &token{TokenType: operationToken, Operation: createValueOperation(0, "0")})
}
}
+2
View File
@@ -96,6 +96,8 @@ var participleYqRules = []*participleYqRule{
simpleOp("load_?str|str_?load", loadStringOpType),
{"LoadYaml", `load`, loadOp(NewYamlDecoder(LoadYamlPreferences)), 0},
simpleOp("system", systemOpType),
{"SplitDocument", `splitDoc|split_?doc`, opToken(splitDocumentOpType), 0},
simpleOp("select", selectOpType),
+1 -1
View File
@@ -80,7 +80,7 @@ func recurseNodeObjectEqual(lhs *CandidateNode, rhs *CandidateNode) bool {
key := lhs.Content[index]
value := lhs.Content[index+1]
indexInRHS := findInArray(rhs, key)
indexInRHS := findKeyInMap(rhs, key)
if indexInRHS == -1 || !recursiveNodeEqual(value, rhs.Content[indexInRHS+1]) {
return false
+19 -1
View File
@@ -24,7 +24,7 @@ type parseSnippetScenario struct {
var parseSnippetScenarios = []parseSnippetScenario{
{
snippet: ":",
expectedError: "yaml: did not find expected key",
expectedError: "yaml: while parsing a block mapping at <unknown position>: did not find expected key",
},
{
snippet: "",
@@ -300,6 +300,24 @@ func TestRecurseNodeObjectEqual(t *testing.T) {
test.AssertResult(t, true, recurseNodeObjectEqual(obj1, obj2))
test.AssertResult(t, false, recurseNodeObjectEqual(obj1, obj3))
test.AssertResult(t, false, recurseNodeObjectEqual(obj1, obj4))
// A null key must not match a null value in the other map.
// Regression test for https://issues.oss-fuzz.com/issues/383860504
nullKey := &CandidateNode{Kind: ScalarNode, Tag: "!!null"}
nullVal := &CandidateNode{Kind: ScalarNode, Tag: "!!null"}
intKey := createScalarNode(2, "2")
intKey.Tag = "!!int"
intVal := &CandidateNode{Kind: ScalarNode, Tag: "!!null"}
mapWithNullKey := &CandidateNode{
Kind: MappingNode,
Content: []*CandidateNode{nullKey, nullVal},
}
mapWithIntKey := &CandidateNode{
Kind: MappingNode,
Content: []*CandidateNode{intKey, intVal},
}
test.AssertResult(t, false, recurseNodeObjectEqual(mapWithNullKey, mapWithIntKey))
}
func TestParseInt(t *testing.T) {
+2
View File
@@ -164,6 +164,8 @@ var stringInterpolationOpType = &operationType{Type: "STRING_INT", NumArgs: 0, P
var loadOpType = &operationType{Type: "LOAD", NumArgs: 1, Precedence: 52, Handler: loadOperator, CheckForPostTraverse: true}
var loadStringOpType = &operationType{Type: "LOAD_STRING", NumArgs: 1, Precedence: 52, Handler: loadStringOperator}
var systemOpType = &operationType{Type: "SYSTEM", NumArgs: 1, Precedence: 50, Handler: systemOperator}
var keysOpType = &operationType{Type: "KEYS", NumArgs: 0, Precedence: 52, Handler: keysOperator, CheckForPostTraverse: true}
var collectObjectOpType = &operationType{Type: "COLLECT_OBJECT", NumArgs: 0, Precedence: 50, Handler: collectObjectOperator}
+12
View File
@@ -527,6 +527,18 @@ var addOperatorScenarios = []expressionScenario{
expression: `.a += [2]`,
expectedError: "!!seq () cannot be added to a !!str (a)",
},
{
// Regression test for https://issues.oss-fuzz.com/issues/383860504
// Adding a map to itself must not panic when sequence keys contain
// single-entry mappings with a null key in one and a non-null key
// in the other.
skipDoc: true,
document: "? [{~: ~}]\n: v1\n? [{2: ~}]\n: v2",
expression: `. += .`,
expected: []string{
"D0, P[], (!!map)::? [{~: ~}]\n: v1\n? [{2: ~}]\n: v2\n",
},
},
}
func TestAddOperatorScenarios(t *testing.T) {
+2 -2
View File
@@ -46,9 +46,9 @@ func containsObject(lhs *CandidateNode, rhs *CandidateNode) (bool, error) {
rhsKey := rhs.Content[index]
rhsValue := rhs.Content[index+1]
log.Debugf("Looking for %v in the lhs", rhsKey.Value)
lhsKeyIndex := findInArray(lhs, rhsKey)
lhsKeyIndex := findKeyInMap(lhs, rhsKey)
log.Debugf("index is %v", lhsKeyIndex)
if lhsKeyIndex < 0 || lhsKeyIndex%2 != 0 {
if lhsKeyIndex < 0 {
return false, nil
}
lhsValue := lhs.Content[lhsKeyIndex+1]
+10
View File
@@ -65,6 +65,16 @@ var containsOperatorScenarios = []expressionScenario{
"D0, P[], (!!bool)::false\n",
},
},
{
// Regression: findInArray could match a null key against a null
// value at an earlier odd index, producing a false negative.
skipDoc: true,
document: "? 1\n: ~\n? ~\n: x",
expression: `contains({~: "x"})`,
expected: []string{
"D0, P[], (!!bool)::true\n",
},
},
{
description: "String contains substring",
document: `"foobar"`,
+4 -2
View File
@@ -155,8 +155,10 @@ func repeatString(lhs *CandidateNode, rhs *CandidateNode) (*CandidateNode, error
return nil, err
} else if count < 0 {
return nil, fmt.Errorf("cannot repeat string by a negative number (%v)", count)
} else if count > 10000000 {
return nil, fmt.Errorf("cannot repeat string by more than 100 million (%v)", count)
}
maxResultLen := 10 * 1024 * 1024 // 10 MiB
if count > 0 && len(stringNode.Value) > maxResultLen/count {
return nil, fmt.Errorf("result of repeating string (%v bytes) by %v would exceed %v bytes", len(stringNode.Value), count, maxResultLen)
}
target.Value = strings.Repeat(stringNode.Value, count)
+24 -4
View File
@@ -237,12 +237,11 @@ var multiplyOperatorScenarios = []expressionScenario{
expectedError: "cannot repeat string by a negative number (-4)",
},
{
description: "Multiply string X by more than 100 million",
// very large string.repeats causes a panic
description: "Multiply string by count that exceeds result size limit",
skipDoc: true,
document: `n: 100000001`,
expression: `"banana" * .n`,
expectedError: "cannot repeat string by more than 100 million (100000001)",
expectedError: "result of repeating string (6 bytes) by 100000001 would exceed 10485760 bytes",
},
{
description: "Multiply int node X string",
@@ -554,7 +553,7 @@ var multiplyOperatorScenarios = []expressionScenario{
document: document,
expression: `.b * .c`,
expected: []string{
"D0, P[b], (!!map)::{name: dog, <<: *cat}\n",
"D0, P[b], (!!map)::{name: dog, \"<<\": *cat}\n",
},
},
{
@@ -693,6 +692,27 @@ var multiplyOperatorScenarios = []expressionScenario{
"D0, P[], (!!null)::null\n",
},
},
{
// Regression test for https://issues.oss-fuzz.com/issues/418818862
// Large repeat count with a long string must not panic.
skipDoc: true,
expression: `"abc" * 99999999`,
expectedError: "result of repeating string (3 bytes) by 99999999 would exceed 10485760 bytes",
},
{
// Regression test for https://issues.oss-fuzz.com/issues/383195001
// Product of string length * repeat count must be bounded.
skipDoc: true,
expression: `"x" * 99999999`,
expectedError: "result of repeating string (1 bytes) by 99999999 would exceed 10485760 bytes",
},
{
// The size guard must not overflow: len * count can wrap to
// a negative or small value on 64-bit, bypassing the check.
skipDoc: true,
expression: `"ab" * 4611686018427387904`,
expectedError: "result of repeating string (2 bytes) by 4611686018427387904 would exceed 10485760 bytes",
},
}
func TestMultiplyOperatorScenarios(t *testing.T) {
+39 -10
View File
@@ -16,6 +16,39 @@ func getSliceNumber(d *dataTreeNavigator, context Context, node *CandidateNode,
return parseInt(result.MatchingNodes.Front().Value.(*CandidateNode).Value)
}
// clampSliceIndex resolves a possibly-negative slice index against
// length and clamps the result to [0, length].
func clampSliceIndex(index, length int) int {
if index < 0 {
index += length
}
if index < 0 {
return 0
}
if index > length {
return length
}
return index
}
func sliceStringNode(lhsNode *CandidateNode, firstNumber int, secondNumber int) *CandidateNode {
runes := []rune(lhsNode.Value)
length := len(runes)
relativeFirstNumber := clampSliceIndex(firstNumber, length)
relativeSecondNumber := clampSliceIndex(secondNumber, length)
if relativeSecondNumber < relativeFirstNumber {
relativeSecondNumber = relativeFirstNumber
}
log.Debugf("sliceStringNode: slice from %v to %v", relativeFirstNumber, relativeSecondNumber)
slicedString := string(runes[relativeFirstNumber:relativeSecondNumber])
replacement := lhsNode.CreateReplacement(ScalarNode, lhsNode.Tag, slicedString)
replacement.Style = lhsNode.Style
return replacement
}
func sliceArrayOperator(d *dataTreeNavigator, context Context, expressionNode *ExpressionNode) (Context, error) {
log.Debug("slice array operator!")
@@ -28,27 +61,23 @@ func sliceArrayOperator(d *dataTreeNavigator, context Context, expressionNode *E
lhsNode := el.Value.(*CandidateNode)
firstNumber, err := getSliceNumber(d, context, lhsNode, expressionNode.LHS)
if err != nil {
return Context{}, err
}
relativeFirstNumber := firstNumber
if relativeFirstNumber < 0 {
relativeFirstNumber = len(lhsNode.Content) + firstNumber
}
secondNumber, err := getSliceNumber(d, context, lhsNode, expressionNode.RHS)
if err != nil {
return Context{}, err
}
relativeSecondNumber := secondNumber
if relativeSecondNumber < 0 {
relativeSecondNumber = len(lhsNode.Content) + secondNumber
} else if relativeSecondNumber > len(lhsNode.Content) {
relativeSecondNumber = len(lhsNode.Content)
if lhsNode.Kind == ScalarNode && lhsNode.guessTagFromCustomType() == "!!str" {
results.PushBack(sliceStringNode(lhsNode, firstNumber, secondNumber))
continue
}
relativeFirstNumber := clampSliceIndex(firstNumber, len(lhsNode.Content))
relativeSecondNumber := clampSliceIndex(secondNumber, len(lhsNode.Content))
log.Debugf("calculateIndicesToTraverse: slice from %v to %v", relativeFirstNumber, relativeSecondNumber)
var newResults []*CandidateNode
+109
View File
@@ -98,6 +98,115 @@ var sliceArrayScenarios = []expressionScenario{
"D0, P[], (!!seq)::- cat1\n",
},
},
{
// Regression test for https://issues.oss-fuzz.com/issues/438776028
// Negative second index that underflows after adjustment must
// clamp to zero, yielding an empty sequence.
skipDoc: true,
document: `[a, b, c]`,
expression: `.[0:-99999]`,
expected: []string{
"D0, P[], (!!seq)::[]\n",
},
},
{
// First-index underflow: without clamping, the loop starts at a
// negative index and panics on Content access.
skipDoc: true,
document: `[a, b, c]`,
expression: `.[-99999:3]`,
expected: []string{
"D0, P[], (!!seq)::- a\n- b\n- c\n",
},
},
{
// Both indices underflow: both clamp to zero, yielding an empty
// sequence.
skipDoc: true,
document: `[a, b, c]`,
expression: `.[-99999:-99998]`,
expected: []string{
"D0, P[], (!!seq)::[]\n",
},
},
{
description: "Slicing strings",
document: `country: Australia`,
expression: `.country[0:5]`,
expected: []string{
"D0, P[country], (!!str)::Austr\n",
},
},
{
description: "Slicing strings - without the second number",
subdescription: "Finishes at the end of the string",
document: `country: Australia`,
expression: `.country[5:]`,
expected: []string{
"D0, P[country], (!!str)::alia\n",
},
},
{
description: "Slicing strings - without the first number",
subdescription: "Starts from the start of the string",
document: `country: Australia`,
expression: `.country[:5]`,
expected: []string{
"D0, P[country], (!!str)::Austr\n",
},
},
{
description: "Slicing strings - use negative numbers to count backwards from the end",
subdescription: "Negative indices count from the end of the string",
document: `country: Australia`,
expression: `.country[-5:]`,
expected: []string{
"D0, P[country], (!!str)::ralia\n",
},
},
{
skipDoc: true,
document: `country: Australia`,
expression: `.country[1:-1]`,
expected: []string{
"D0, P[country], (!!str)::ustrali\n",
},
},
{
skipDoc: true,
document: `country: Australia`,
expression: `.country[:]`,
expected: []string{
"D0, P[country], (!!str)::Australia\n",
},
},
{
skipDoc: true,
description: "second index beyond string length clamps",
document: `country: Australia`,
expression: `.country[:100]`,
expected: []string{
"D0, P[country], (!!str)::Australia\n",
},
},
{
skipDoc: true,
description: "first index beyond string length returns empty string",
document: `country: Australia`,
expression: `.country[100:]`,
expected: []string{
"D0, P[country], (!!str)::\n",
},
},
{
description: "Slicing strings - Unicode",
subdescription: "Indices are rune-based, so multi-byte characters are handled correctly",
document: `greeting: héllo`,
expression: `.greeting[1:3]`,
expected: []string{
"D0, P[greeting], (!!str)::él\n",
},
},
}
func TestSliceOperatorScenarios(t *testing.T) {
+146
View File
@@ -0,0 +1,146 @@
package yqlib
import (
"bytes"
"container/list"
"fmt"
"os/exec"
"strings"
)
func resolveSystemArgs(argsNode *CandidateNode) ([]string, error) {
if argsNode == nil {
return nil, nil
}
if argsNode.Kind == SequenceNode {
args := make([]string, 0, len(argsNode.Content))
for _, child := range argsNode.Content {
// Only non-null scalar children are valid arguments.
if child == nil {
continue
}
if child.Kind != ScalarNode || child.Tag == "!!null" {
return nil, fmt.Errorf("system operator: argument must be a non-null scalar; got kind=%v tag=%v", child.Kind, child.Tag)
}
args = append(args, child.Value)
}
if len(args) == 0 {
return nil, nil
}
return args, nil
}
// Single-argument case: only accept a non-null scalar node.
if argsNode.Tag == "!!null" {
return nil, nil
}
if argsNode.Kind != ScalarNode {
return nil, fmt.Errorf("system operator: args must be a non-null scalar or sequence of non-null scalars; got kind=%v tag=%v", argsNode.Kind, argsNode.Tag)
}
return []string{argsNode.Value}, nil
}
func resolveCommandNode(commandNodes Context) (string, error) {
if commandNodes.MatchingNodes.Front() == nil {
return "", fmt.Errorf("system operator: command expression returned no results")
}
if commandNodes.MatchingNodes.Len() > 1 {
log.Debugf("system operator: command expression returned %d results, using first", commandNodes.MatchingNodes.Len())
}
cmdNode := commandNodes.MatchingNodes.Front().Value.(*CandidateNode)
if cmdNode.Kind != ScalarNode || cmdNode.guessTagFromCustomType() != "!!str" {
return "", fmt.Errorf("system operator: command must be a string scalar")
}
if cmdNode.Value == "" {
return "", fmt.Errorf("system operator: command must be a non-empty string")
}
return cmdNode.Value, nil
}
func systemOperator(d *dataTreeNavigator, context Context, expressionNode *ExpressionNode) (Context, error) {
if !ConfiguredSecurityPreferences.EnableSystemOps {
return Context{}, fmt.Errorf("system operations are disabled, use --security-enable-system-operator to enable")
}
// determine at parse time whether we have (command; args) or just (command)
hasArgs := expressionNode.RHS.Operation.OperationType == blockOpType
var results = list.New()
for el := context.MatchingNodes.Front(); el != nil; el = el.Next() {
candidate := el.Value.(*CandidateNode)
nodeContext := context.SingleReadonlyChildContext(candidate)
var command string
var args []string
if hasArgs {
block := expressionNode.RHS
commandNodes, err := d.GetMatchingNodes(nodeContext, block.LHS)
if err != nil {
return Context{}, err
}
command, err = resolveCommandNode(commandNodes)
if err != nil {
return Context{}, err
}
argsNodes, err := d.GetMatchingNodes(nodeContext, block.RHS)
if err != nil {
return Context{}, err
}
if argsNodes.MatchingNodes.Len() > 1 {
log.Debugf("system operator: args expression returned %d results, using first", argsNodes.MatchingNodes.Len())
}
if argsNodes.MatchingNodes.Front() != nil {
args, err = resolveSystemArgs(argsNodes.MatchingNodes.Front().Value.(*CandidateNode))
if err != nil {
return Context{}, err
}
}
} else {
commandNodes, err := d.GetMatchingNodes(nodeContext, expressionNode.RHS)
if err != nil {
return Context{}, err
}
command, err = resolveCommandNode(commandNodes)
if err != nil {
return Context{}, err
}
}
var stdin bytes.Buffer
encoded, err := encodeToYamlString(candidate)
if err != nil {
return Context{}, err
}
stdin.WriteString(encoded)
// #nosec G204 - intentional: user must explicitly enable this operator
cmd := exec.Command(command, args...)
cmd.Stdin = &stdin
var stderr bytes.Buffer
cmd.Stderr = &stderr
output, err := cmd.Output()
if err != nil {
stderrStr := strings.TrimSpace(stderr.String())
if stderrStr != "" {
return Context{}, fmt.Errorf("system command '%v' failed: %w\nstderr: %v", command, err, stderrStr)
}
return Context{}, fmt.Errorf("system command '%v' failed: %w", command, err)
}
result := string(output)
if strings.HasSuffix(result, "\r\n") {
result = result[:len(result)-2]
} else if strings.HasSuffix(result, "\n") {
result = result[:len(result)-1]
}
newNode := candidate.CreateReplacement(ScalarNode, "!!str", result)
results.PushBack(newNode)
}
return context.ChildContext(results), nil
}
+123
View File
@@ -0,0 +1,123 @@
package yqlib
import (
"os/exec"
"testing"
)
func findExec(t *testing.T, name string) string {
t.Helper()
path, err := exec.LookPath(name)
if err != nil {
t.Skipf("skipping: %v not found: %v", name, err)
}
return path
}
var systemOperatorDisabledScenarios = []expressionScenario{
{
description: "system operator returns error when disabled",
subdescription: "Use `--security-enable-system-operator` to enable the system operator.",
document: "country: Australia",
expression: `.country = system("/usr/bin/echo"; "test")`,
expectedError: "system operations are disabled, use --security-enable-system-operator to enable",
},
}
func TestSystemOperatorDisabledScenarios(t *testing.T) {
originalEnableSystemOps := ConfiguredSecurityPreferences.EnableSystemOps
defer func() {
ConfiguredSecurityPreferences.EnableSystemOps = originalEnableSystemOps
}()
ConfiguredSecurityPreferences.EnableSystemOps = false
for _, tt := range systemOperatorDisabledScenarios {
testScenario(t, &tt)
}
documentOperatorScenarios(t, "system-operators", systemOperatorDisabledScenarios)
}
func TestSystemOperatorEnabledScenarios(t *testing.T) {
echoPath := findExec(t, "echo")
falsePath := findExec(t, "false")
originalEnableSystemOps := ConfiguredSecurityPreferences.EnableSystemOps
defer func() {
ConfiguredSecurityPreferences.EnableSystemOps = originalEnableSystemOps
}()
ConfiguredSecurityPreferences.EnableSystemOps = true
scenarios := []expressionScenario{
{
description: "Run a command with an argument",
subdescription: "Use `--security-enable-system-operator` to enable the system operator.",
yqFlags: "--security-enable-system-operator",
document: "country: Australia",
expression: `.country = system("` + echoPath + `"; "test")`,
expected: []string{
"D0, P[], (!!map)::country: test\n",
},
},
{
description: "Run a command without arguments",
subdescription: "Omit the semicolon and args to run the command with no extra arguments.",
yqFlags: "--security-enable-system-operator",
document: "a: hello",
expression: `.a = system("` + echoPath + `")`,
expected: []string{
"D0, P[], (!!map)::a: \"\"\n",
},
},
{
description: "Run a command with multiple arguments",
subdescription: "Pass an array of arguments.",
skipDoc: true,
document: "a: hello",
expression: `.a = system("` + echoPath + `"; ["foo", "bar"])`,
expected: []string{
"D0, P[], (!!map)::a: foo bar\n",
},
},
{
description: "Command and args are evaluated per matched node",
skipDoc: true,
document: "cmd: " + echoPath + "\narg: hello",
expression: `.result = system(.cmd; .arg)`,
expected: []string{
"D0, P[], (!!map)::cmd: " + echoPath + "\narg: hello\nresult: hello\n",
},
},
{
description: "Command failure returns error",
skipDoc: true,
document: "a: hello",
expression: `.a = system("` + falsePath + `")`,
expectedError: "system command '" + falsePath + "' failed: exit status 1",
},
{
description: "Null command returns error",
skipDoc: true,
document: "a: hello",
expression: `.a = system(null)`,
expectedError: "system operator: command must be a string scalar",
},
{
description: "System operator processes multiple matched nodes",
skipDoc: true,
document: "a: first",
document2: "a: second",
expression: `.a = system("` + echoPath + `"; "replaced")`,
expected: []string{
"D0, P[], (!!map)::a: replaced\n",
"D0, P[], (!!map)::a: replaced\n",
},
},
}
for _, tt := range scenarios {
testScenario(t, &tt)
}
appendOperatorDocumentScenario(t, "system-operators", scenarios)
}
+36 -9
View File
@@ -36,9 +36,33 @@ func traversePathOperator(_ *dataTreeNavigator, context Context, expressionNode
return context.ChildContext(matches), nil
}
// resolveAliasChain follows an alias chain iteratively, returning the
// first non-alias node. Returns an error if a cycle is detected.
func resolveAliasChain(node *CandidateNode) (*CandidateNode, error) {
if node.Kind != AliasNode {
return node, nil
}
visited := map[*CandidateNode]bool{}
for node.Kind == AliasNode {
if visited[node] {
return nil, fmt.Errorf("alias cycle detected")
}
visited[node] = true
log.Debug("its an alias!")
node = node.Alias
}
return node, nil
}
func traverse(context Context, matchingNode *CandidateNode, operation *Operation) (*list.List, error) {
log.Debugf("Traversing %v", NodeToString(matchingNode))
var err error
matchingNode, err = resolveAliasChain(matchingNode)
if err != nil {
return nil, err
}
if matchingNode.Tag == "!!null" && operation.Value != "[]" && !context.DontAutoCreate {
log.Debugf("Guessing kind")
// we must have added this automatically, lets guess what it should be now
@@ -62,10 +86,6 @@ func traverse(context Context, matchingNode *CandidateNode, operation *Operation
log.Debugf("its a sequence of %v things!", len(matchingNode.Content))
return traverseArray(matchingNode, operation, operation.Preferences.(traversePreferences))
case AliasNode:
log.Debug("its an alias!")
matchingNode = matchingNode.Alias
return traverse(context, matchingNode, operation)
default:
return list.New(), nil
}
@@ -79,7 +99,11 @@ func traverseArrayOperator(d *dataTreeNavigator, context Context, expressionNode
log.Debugf("--traverseArrayOperator")
if expressionNode.RHS != nil && expressionNode.RHS.RHS != nil && expressionNode.RHS.RHS.Operation.OperationType == createMapOpType {
return sliceArrayOperator(d, context, expressionNode.RHS.RHS)
lhsContext, err := d.GetMatchingNodes(context, expressionNode.LHS)
if err != nil {
return Context{}, err
}
return sliceArrayOperator(d, lhsContext, expressionNode.RHS.RHS)
}
lhs, err := d.GetMatchingNodes(context, expressionNode.LHS)
@@ -125,7 +149,13 @@ func traverseNodesWithArrayIndices(context Context, indicesToTraverse []*Candida
return context.ChildContext(matchingNodeMap), nil
}
func traverseArrayIndices(context Context, matchingNode *CandidateNode, indicesToTraverse []*CandidateNode, prefs traversePreferences) (*list.List, error) { // call this if doc / alias like the other traverse
func traverseArrayIndices(context Context, matchingNode *CandidateNode, indicesToTraverse []*CandidateNode, prefs traversePreferences) (*list.List, error) {
var err error
matchingNode, err = resolveAliasChain(matchingNode)
if err != nil {
return nil, err
}
if matchingNode.Tag == "!!null" {
log.Debugf("OperatorArrayTraverse got a null - turning it into an empty array")
// auto vivification
@@ -138,9 +168,6 @@ func traverseArrayIndices(context Context, matchingNode *CandidateNode, indicesT
}
switch matchingNode.Kind {
case AliasNode:
matchingNode = matchingNode.Alias
return traverseArrayIndices(context, matchingNode, indicesToTraverse, prefs)
case SequenceNode:
return traverseArrayWithIndices(matchingNode, indicesToTraverse, prefs)
case MappingNode:
+65
View File
@@ -665,6 +665,16 @@ var traversePathOperatorScenarios = []expressionScenario{
"D0, P[a], (!!null)::null\n",
},
},
{
// Regression test for https://issues.oss-fuzz.com/issues/390467412
// go-yaml accepts cross-document alias references (invalid per
// YAML spec). A nested assignment on such an alias can create a
// circular alias node, which must not cause a stack overflow.
skipDoc: true,
document: "&-- a\n---\n*--",
expression: ". = (.x = 1)",
expectedError: "alias cycle detected",
},
}
func TestTraversePathOperatorScenarios(t *testing.T) {
@@ -682,3 +692,58 @@ func TestTraversePathOperatorAlignedToSpecScenarios(t *testing.T) {
appendOperatorDocumentScenario(t, "traverse-read", fixedTraversePathOperatorScenarios)
ConfiguredYamlPreferences.FixMergeAnchorToSpec = false
}
// Regression test for https://issues.oss-fuzz.com/issues/390467412
// A circular alias (alias pointing back to itself) must not cause a
// stack overflow. resolveAliasChain should detect the cycle and return
// an error; both traverse() and traverseArrayIndices() use it.
func TestTraverseAliasCycle(t *testing.T) {
aliasNode := &CandidateNode{
Kind: AliasNode,
}
aliasNode.Alias = aliasNode // A -> A
op := &Operation{
OperationType: traversePathOpType,
Value: "key",
StringValue: "key",
Preferences: traversePreferences{},
}
_, err := traverse(Context{}, aliasNode, op)
if err == nil {
t.Fatal("expected error for alias cycle, got nil")
}
if err.Error() != "alias cycle detected" {
t.Fatalf("expected 'alias cycle detected', got %q", err.Error())
}
// Same cycle must be caught through the array traversal path.
_, err = traverseArrayIndices(Context{}, aliasNode, nil, traversePreferences{})
if err == nil {
t.Fatal("expected error for alias cycle via traverseArrayIndices, got nil")
}
if err.Error() != "alias cycle detected" {
t.Fatalf("expected 'alias cycle detected', got %q", err.Error())
}
}
func TestTraverseAliasCycleChain(t *testing.T) {
nodeA := &CandidateNode{Kind: AliasNode}
nodeB := &CandidateNode{Kind: AliasNode}
nodeA.Alias = nodeB
nodeB.Alias = nodeA // A -> B -> A
op := &Operation{
OperationType: traversePathOpType,
Value: "key",
StringValue: "key",
Preferences: traversePreferences{},
}
_, err := traverse(Context{}, nodeA, op)
if err == nil {
t.Fatal("expected error for alias cycle chain, got nil")
}
if err.Error() != "alias cycle detected" {
t.Fatalf("expected 'alias cycle detected', got %q", err.Error())
}
}
+12 -3
View File
@@ -31,6 +31,7 @@ type expressionScenario struct {
dontFormatInputForDoc bool // dont format input doc for documentation generation
requiresFormat string
skipForGoccy bool
yqFlags string // extra yq flags to include in generated doc command snippets
}
var goccyTesting = false
@@ -356,14 +357,22 @@ func documentInput(w *bufio.Writer, s expressionScenario) (string, string) {
writeOrPanic(w, "then\n")
flagsPrefix := ""
if s.yqFlags != "" {
flagsPrefix = s.yqFlags + " "
}
if s.expression != "" {
writeOrPanic(w, fmt.Sprintf("```bash\n%vyq %v'%v' %v\n```\n", envCommand, command, strings.ReplaceAll(s.expression, "'", `'\''`), files))
writeOrPanic(w, fmt.Sprintf("```bash\n%vyq %v%v'%v' %v\n```\n", envCommand, flagsPrefix, command, strings.ReplaceAll(s.expression, "'", `'\''`), files))
} else {
writeOrPanic(w, fmt.Sprintf("```bash\n%vyq %v%v\n```\n", envCommand, command, files))
writeOrPanic(w, fmt.Sprintf("```bash\n%vyq %v%v%v\n```\n", envCommand, flagsPrefix, command, files))
}
} else {
writeOrPanic(w, "Running\n")
writeOrPanic(w, fmt.Sprintf("```bash\n%vyq %v--null-input '%v'\n```\n", envCommand, command, s.expression))
flagsPrefix := ""
if s.yqFlags != "" {
flagsPrefix = s.yqFlags + " "
}
writeOrPanic(w, fmt.Sprintf("```bash\n%vyq %v%v--null-input '%v'\n```\n", envCommand, flagsPrefix, command, s.expression))
}
return formattedDoc, formattedDoc2
}
+6 -4
View File
@@ -1,11 +1,13 @@
package yqlib
type SecurityPreferences struct {
DisableEnvOps bool
DisableFileOps bool
DisableEnvOps bool
DisableFileOps bool
EnableSystemOps bool
}
var ConfiguredSecurityPreferences = SecurityPreferences{
DisableEnvOps: false,
DisableFileOps: false,
DisableEnvOps: false,
DisableFileOps: false,
EnableSystemOps: false,
}
+90
View File
@@ -287,6 +287,18 @@ var expectedSubArrays = `array:
- {}
`
// Keys with special characters that require quoting in TOML
var rtSpecialKeyInlineTable = `host = { "http://sealos.hub:5000" = { capabilities = ["pull", "resolve", "push"], skip_verify = true } }
`
var rtSpecialKeyTableSection = `["/tmp/blah"]
value = "hello"
`
var rtSpecialKeyDottedTableSection = `[servers."http://localhost:8080"]
ip = "127.0.0.1"
`
var tomlScenarios = []formatScenario{
{
skipDoc: true,
@@ -614,6 +626,58 @@ var tomlScenarios = []formatScenario{
expected: tomlTableWithComments,
scenarioType: "roundtrip",
},
// Encode (YAML → TOML) scenarios - verify readable table sections are produced
{
description: "Encode: Simple mapping produces table section",
input: "arg:\n hello: foo\n",
expected: "[arg]\nhello = \"foo\"\n",
scenarioType: "encode",
},
{
skipDoc: true,
description: "Encode: Nested mappings produce nested table sections",
input: "a:\n b:\n c: val\n",
expected: "[a.b]\nc = \"val\"\n",
scenarioType: "encode",
},
{
skipDoc: true,
description: "Encode: Mixed scalars and nested mapping",
input: "a:\n hello: foo\n nested:\n key: val\n",
expected: "[a]\nhello = \"foo\"\n\n[a.nested]\nkey = \"val\"\n",
scenarioType: "encode",
},
{
skipDoc: true,
description: "Encode: YAML flow mapping stays inline",
input: "arg: {hello: foo}\n",
expected: "arg = { hello = \"foo\" }\n",
scenarioType: "encode",
},
{
skipDoc: true,
description: "Roundtrip: key with special characters in inline table",
input: rtSpecialKeyInlineTable,
expression: ".",
expected: rtSpecialKeyInlineTable,
scenarioType: "roundtrip",
},
{
skipDoc: true,
description: "Roundtrip: key with special characters in table section",
input: rtSpecialKeyTableSection,
expression: ".",
expected: rtSpecialKeyTableSection,
scenarioType: "roundtrip",
},
{
skipDoc: true,
description: "Roundtrip: special character key in dotted table section header",
input: rtSpecialKeyDottedTableSection,
expression: ".",
expected: rtSpecialKeyDottedTableSection,
scenarioType: "roundtrip",
},
}
func testTomlScenario(t *testing.T, s formatScenario) {
@@ -629,6 +693,8 @@ func testTomlScenario(t *testing.T, s formatScenario) {
}
case "roundtrip":
test.AssertResultWithContext(t, s.expected, mustProcessFormatScenario(s, NewTomlDecoder(), NewTomlEncoder()), s.description)
case "encode":
test.AssertResultWithContext(t, s.expected, mustProcessFormatScenario(s, NewYamlDecoder(ConfiguredYamlPreferences), NewTomlEncoder()), s.description)
}
}
@@ -654,6 +720,28 @@ func documentTomlDecodeScenario(w *bufio.Writer, s formatScenario) {
writeOrPanic(w, fmt.Sprintf("```yaml\n%v```\n\n", mustProcessFormatScenario(s, NewTomlDecoder(), NewYamlEncoder(ConfiguredYamlPreferences))))
}
func documentTomlEncodeScenario(w *bufio.Writer, s formatScenario) {
writeOrPanic(w, fmt.Sprintf("## %v\n", s.description))
if s.subdescription != "" {
writeOrPanic(w, s.subdescription)
writeOrPanic(w, "\n\n")
}
writeOrPanic(w, "Given a sample.yml file of:\n")
writeOrPanic(w, fmt.Sprintf("```yaml\n%v\n```\n", s.input))
writeOrPanic(w, "then\n")
expression := s.expression
if expression == "" {
expression = "."
}
writeOrPanic(w, fmt.Sprintf("```bash\nyq -o toml '%v' sample.yml\n```\n", expression))
writeOrPanic(w, "will output\n")
writeOrPanic(w, fmt.Sprintf("```toml\n%v```\n\n", mustProcessFormatScenario(s, NewYamlDecoder(ConfiguredYamlPreferences), NewTomlEncoder())))
}
func documentTomlRoundtripScenario(w *bufio.Writer, s formatScenario) {
writeOrPanic(w, fmt.Sprintf("## %v\n", s.description))
@@ -687,6 +775,8 @@ func documentTomlScenario(_ *testing.T, w *bufio.Writer, i interface{}) {
documentTomlDecodeScenario(w, s)
case "roundtrip":
documentTomlRoundtripScenario(w, s)
case "encode":
documentTomlEncodeScenario(w, s)
default:
panic(fmt.Sprintf("unhandled scenario type %q", s.scenarioType))
+6 -1
View File
@@ -298,4 +298,9 @@ subsubarray
Ffile
Fquery
coverpkg
gsub
gsub
ralia
Austr
ustrali
héllo
alia
+9
View File
@@ -8,6 +8,15 @@
- git push --tags
- use github actions to publish docker and make github release
- check github updated yq action in marketplace
- update github-action/Dockerfile to pin the newly published docker image digest:
skopeo inspect docker://docker.io/mikefarah/yq:4 | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['Digest'])"
then update the FROM line in github-action/Dockerfile with the new digest:
FROM mikefarah/yq:4@sha256:<new-digest>
// release artifacts are signed with cosign keyless signing (Sigstore)
// users can verify with:
// cosign verify-blob --bundle checksums.bundle checksums
// install cosign: brew install cosign OR go install github.com/sigstore/cosign/v2/cmd/cosign@latest
- snapcraft