Compare commits

...
54 Commits
Author SHA1 Message Date
Mike Farah bbdd97482f Bumping version 2024-08-05 15:38:57 +10:00
Mike Farah 9940668729 Preparing release 2024-08-05 15:38:30 +10:00
95aaccee46 Add Flox to installation methods (#2117)
Co-authored-by: Bryan Honof <bryanhonof@bryans-mbp.space.flox.local>
2024-08-05 15:15:06 +10:00
ryenusandGitHub b9c3ff6f0a convert file ext to lowercase for format detection (#2121)
* convert file ext to lowercase for format detection

To ensure proper file format detection with case-insensitive file
systems.

* use filepath.Ext for more reliable file ext detection

especially for paths like index.js/foo

* add a test for file ext based format detection
2024-08-05 15:14:43 +10:00
Kian ParvinandGitHub b80e1cb35e Use static bin in snap (#2090) 2024-07-21 10:52:32 +10:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
0ff8415244 Bump golang.org/x/net from 0.26.0 to 0.27.0 (#2099)
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.26.0 to 0.27.0.
- [Commits](https://github.com/golang/net/compare/v0.26.0...v0.27.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-07-21 10:08:58 +10:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
e8dcf3f1a4 Bump github.com/goccy/go-yaml from 1.11.3 to 1.12.0 (#2108)
Bumps [github.com/goccy/go-yaml](https://github.com/goccy/go-yaml) from 1.11.3 to 1.12.0.
- [Release notes](https://github.com/goccy/go-yaml/releases)
- [Changelog](https://github.com/goccy/go-yaml/blob/master/CHANGELOG.md)
- [Commits](https://github.com/goccy/go-yaml/compare/v1.11.3...v1.12.0)

---
updated-dependencies:
- dependency-name: github.com/goccy/go-yaml
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-07-21 10:08:42 +10:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
208302cf6b Bump golang from 1.22.4 to 1.22.5 (#2091)
Bumps golang from 1.22.4 to 1.22.5.

---
updated-dependencies:
- dependency-name: golang
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-07-21 10:08:23 +10:00
Mike Farah ef6fb92e7f Log printing follow no-colors flag #2082 2024-06-29 15:26:10 +10:00
Mike Farah 28646c7b50 Skip and warn when interpolating strings and theres a unclosed bracket #2083 2024-06-29 15:20:07 +10:00
Mike Farah 0b7d4b799c Fixed CSV content starting with # issue #2076 2024-06-29 15:04:07 +10:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
7e1722510a Bump github.com/spf13/cobra from 1.8.0 to 1.8.1 (#2075)
Bumps [github.com/spf13/cobra](https://github.com/spf13/cobra) from 1.8.0 to 1.8.1.
- [Release notes](https://github.com/spf13/cobra/releases)
- [Commits](https://github.com/spf13/cobra/compare/v1.8.0...v1.8.1)

---
updated-dependencies:
- dependency-name: github.com/spf13/cobra
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-06-29 13:59:50 +10:00
Mike Farah f15500b20a Bumping version 2024-06-16 11:13:40 +10:00
Mike Farah b8aec71e66 Preparing release 2024-06-16 11:13:25 +10:00
Mike Farah 68aafb6a53 Handle numbers with underscores #2039 2024-06-16 11:09:52 +10:00
Mike Farah d58870b056 Adding splat[] short hand to explode,collect,flatten,groupby,path,pivot,select and more 2024-06-16 10:48:01 +10:00
Mike Farah f1964dece0 Adding splat[] short hand to collect,eval,map,omit,path,pick 2024-06-16 10:22:42 +10:00
Mike Farah 45eda8dbae Added support for shorthand splat[] for filter,reverse,sort,shuffle 2024-06-15 22:13:58 +10:00
Mike Farah d9af2dd976 Support env[] expression #2071 2024-06-15 21:56:32 +10:00
Mike Farah 7bad2f4f06 Adding back in commented tests :facepalm 2024-06-15 21:49:23 +10:00
Mike Farah 3ea19ad2ed clean code 2024-06-15 17:06:17 +10:00
Mike Farah 42120e1341 Unique now works on maps and arrays #2068 2024-06-15 17:04:48 +10:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
1147113fd3 Bump github.com/goccy/go-json from 0.10.2 to 0.10.3 (#2046)
Bumps [github.com/goccy/go-json](https://github.com/goccy/go-json) from 0.10.2 to 0.10.3.
- [Release notes](https://github.com/goccy/go-json/releases)
- [Changelog](https://github.com/goccy/go-json/blob/master/CHANGELOG.md)
- [Commits](https://github.com/goccy/go-json/compare/v0.10.2...v0.10.3)

---
updated-dependencies:
- dependency-name: github.com/goccy/go-json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-06-15 16:00:53 +10:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2edfa834d5 Bump github.com/elliotchance/orderedmap from 1.5.1 to 1.6.0 (#2053)
Bumps [github.com/elliotchance/orderedmap](https://github.com/elliotchance/orderedmap) from 1.5.1 to 1.6.0.
- [Release notes](https://github.com/elliotchance/orderedmap/releases)
- [Commits](https://github.com/elliotchance/orderedmap/compare/v1.5.1...v1.6.0)

---
updated-dependencies:
- dependency-name: github.com/elliotchance/orderedmap
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-06-15 16:00:38 +10:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
8c7cfb720c Bump golang.org/x/net from 0.25.0 to 0.26.0 (#2061)
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.25.0 to 0.26.0.
- [Commits](https://github.com/golang/net/compare/v0.25.0...v0.26.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-06-15 16:00:16 +10:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
356c60f40c Bump golang from 1.22.3 to 1.22.4 (#2062)
Bumps golang from 1.22.3 to 1.22.4.

---
updated-dependencies:
- dependency-name: golang
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-06-15 16:00:02 +10:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
4cc646b16b Bump github.com/fatih/color from 1.16.0 to 1.17.0 (#2043)
Bumps [github.com/fatih/color](https://github.com/fatih/color) from 1.16.0 to 1.17.0.
- [Release notes](https://github.com/fatih/color/releases)
- [Commits](https://github.com/fatih/color/compare/v1.16.0...v1.17.0)

---
updated-dependencies:
- dependency-name: github.com/fatih/color
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-06-01 16:45:11 +10:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2e44ab4512 Bump golang.org/x/net from 0.24.0 to 0.25.0 (#2035)
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.24.0 to 0.25.0.
- [Commits](https://github.com/golang/net/compare/v0.24.0...v0.25.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-06-01 16:45:01 +10:00
Mike Farah 557dcb87b8 Bumping version 2024-05-12 12:27:39 +10:00
Mike Farah 2daa39e306 Updating release notes 2024-05-12 12:27:06 +10:00
Mike Farah 0c15cf35a7 Added properties convert to numbers example #2023 2024-05-12 12:07:04 +10:00
Jakub JirutkaandGitHub d9adcdb668 Update instructions for Alpine Linux (#2034)
The package has been renamed from `yq` to `yq-go` to match other distros.
2024-05-12 11:55:06 +10:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
9010809d89 Bump github.com/pelletier/go-toml/v2 from 2.2.1 to 2.2.2 (#2024)
Bumps [github.com/pelletier/go-toml/v2](https://github.com/pelletier/go-toml) from 2.2.1 to 2.2.2.
- [Release notes](https://github.com/pelletier/go-toml/releases)
- [Changelog](https://github.com/pelletier/go-toml/blob/v2/.goreleaser.yaml)
- [Commits](https://github.com/pelletier/go-toml/compare/v2.2.1...v2.2.2)

---
updated-dependencies:
- dependency-name: github.com/pelletier/go-toml/v2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-05-12 11:53:40 +10:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
8250f0c2bd Bump golang.org/x/text from 0.14.0 to 0.15.0 (#2036)
Bumps [golang.org/x/text](https://github.com/golang/text) from 0.14.0 to 0.15.0.
- [Release notes](https://github.com/golang/text/releases)
- [Commits](https://github.com/golang/text/compare/v0.14.0...v0.15.0)

---
updated-dependencies:
- dependency-name: golang.org/x/text
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-05-12 11:53:02 +10:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
8ac9f605f6 Bump golang from 1.22.2 to 1.22.3 (#2037)
Bumps golang from 1.22.2 to 1.22.3.

---
updated-dependencies:
- dependency-name: golang
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-05-12 11:52:48 +10:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
1748171463 Bump github.com/pelletier/go-toml/v2 from 2.2.0 to 2.2.1 (#2014)
Bumps [github.com/pelletier/go-toml/v2](https://github.com/pelletier/go-toml) from 2.2.0 to 2.2.1.
- [Release notes](https://github.com/pelletier/go-toml/releases)
- [Changelog](https://github.com/pelletier/go-toml/blob/v2/.goreleaser.yaml)
- [Commits](https://github.com/pelletier/go-toml/compare/v2.2.0...v2.2.1)

---
updated-dependencies:
- dependency-name: github.com/pelletier/go-toml/v2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-04-22 14:15:45 +10:00
Cory LatschkowskiandGitHub 21ba506edb Fix: shell-completion (#2006)
* fix: shell-completion

* update: description
2024-04-22 14:15:36 +10:00
Matt BensonandGitHub b54d7eb21c handle escaped backslashes (#1997) 2024-04-14 18:52:08 +10:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
db6f2dc6a2 Bump golang from 1.22.1 to 1.22.2 (#1998)
Bumps golang from 1.22.1 to 1.22.2.

---
updated-dependencies:
- dependency-name: golang
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-04-14 18:51:39 +10:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
1411f45055 Bump golang.org/x/net from 0.22.0 to 0.24.0 (#2008)
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.22.0 to 0.24.0.
- [Commits](https://github.com/golang/net/compare/v0.22.0...v0.24.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-04-14 18:51:28 +10:00
Cory LatschkowskiandGitHub c7baa8bc34 Fix: linux (w/ selinux) build (#2004)
* add: check for selinux for vol mount

* update: install packages first

build time not increased if script changes

* update: selinux check
2024-04-14 18:51:17 +10:00
Mike Farah a4b9c3049b Removed unused debian directory #2000 2024-04-13 11:00:39 +10:00
Mike Farah fe84f62288 Docs 2024-03-30 14:01:42 +11:00
Mike Farah 28c406706a Fixes npe when given filename ending with "." #1994 2024-03-30 14:00:57 +11:00
Matt BensonandGitHub 101cf14b8c min/max operators (#1992)
* min/max operators

* min, max operator headers
2024-03-30 13:34:36 +11:00
Matt BensonandGitHub 3283c65dc4 pivot operator (#1993) 2024-03-30 13:29:35 +11:00
Mike Farah 2ee7508b76 typos 2024-03-24 11:32:02 +11:00
Mike Farah 2ecd8b6092 updated contribe guide 2024-03-24 11:29:48 +11:00
Mike Farah 021d5f05f9 Fixing omit docs 2024-03-24 10:59:29 +11:00
Mike Farah 19a0e6dd8b Bumping version 2024-03-24 10:54:35 +11:00
Mike Farah fbd4c8428a Remove old JSON warning 2024-03-23 17:17:23 +11:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
d9ef03a2d1 Bump github.com/pelletier/go-toml/v2 from 2.1.1 to 2.2.0 (#1986)
Bumps [github.com/pelletier/go-toml/v2](https://github.com/pelletier/go-toml) from 2.1.1 to 2.2.0.
- [Release notes](https://github.com/pelletier/go-toml/releases)
- [Changelog](https://github.com/pelletier/go-toml/blob/v2/.goreleaser.yaml)
- [Commits](https://github.com/pelletier/go-toml/compare/v2.1.1...v2.2.0)

---
updated-dependencies:
- dependency-name: github.com/pelletier/go-toml/v2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-03-23 17:15:56 +11:00
Matt BensonandGitHub f5bfe5a248 omit operator (#1989) 2024-03-23 17:15:45 +11:00
Matt BensonandGitHub 3b85cef340 support string X int multiplication as jq (#1988) 2024-03-22 20:44:49 +11:00
76 changed files with 2180 additions and 1144 deletions
+3 -1
View File
@@ -1,4 +1,4 @@
# Development
.# Development
1. Install (Golang)[https://golang.org/]
1. Run `scripts/devtools.sh` to install the required devtools
@@ -25,6 +25,8 @@ The pipeline will run the tests and automatically concatenate the files together
The first step is to find if what you want is automatically generated or not - start by looking in the master branch.
Note that PRs with small changes (e.g. minor typos) may not be merged (see https://joel.net/how-one-guy-ruined-hacktoberfest2020-drama).
### Updating dynamic documentation from master
- Search for the documentation you want to update. If you find matches in a `*_test.go` file - update that, as that will automatically update the matching `*.md` file
- Assuming you are updating a `*_test.go` file, once updated, run the test to regenerated the docs. E.g. for the 'Add' test generated docs, from the pkg/yqlib folder run:
+1 -1
View File
@@ -1,4 +1,4 @@
FROM golang:1.22.1 as builder
FROM golang:1.22.5 as builder
WORKDIR /go/src/mikefarah/yq
+8 -5
View File
@@ -1,12 +1,15 @@
FROM golang:1.22.1
FROM golang:1.22.5
RUN apt-get update && \
apt-get install -y npm && \
npm install -g npx cspell@latest
COPY scripts/devtools.sh /opt/devtools.sh
RUN set -e -x \
&& /opt/devtools.sh
ENV PATH=/go/bin:$PATH
RUN set -e -x && \
/opt/devtools.sh
RUN apt-get update && apt-get install -y npm && npm install -g npx cspell@latest
ENV PATH=/go/bin:$PATH
ENV CGO_ENABLED 0
ENV GOPATH /go:/yq
+3 -2
View File
@@ -26,6 +26,7 @@ ifeq ($(CYG_CHECK),1)
else
# all non-windows environments
ROOT := $(shell pwd)
SELINUX := $(shell which getenforce 2>&1 >/dev/null && echo :z)
endif
DEV_IMAGE := ${PROJECT}_dev
@@ -33,7 +34,7 @@ DEV_IMAGE := ${PROJECT}_dev
ENGINERUN := ${ENGINE} run --rm \
-e LDFLAGS="${LDFLAGS}" \
-e GITHUB_TOKEN="${GITHUB_TOKEN}" \
-v ${ROOT}/vendor:/go/src \
-v ${ROOT}:/${PROJECT}/src/${IMPORT_PATH} \
-v ${ROOT}/vendor:/go/src${SELINUX} \
-v ${ROOT}:/${PROJECT}/src/${IMPORT_PATH}${SELINUX} \
-w /${PROJECT}/src/${IMPORT_PATH} \
${DEV_IMAGE}
+18 -6
View File
@@ -295,13 +295,26 @@ sudo port install yq
Supported by @herbygillot (https://ports.macports.org/maintainer/github/herbygillot)
### Alpine Linux
- Enable edge/community repo by adding ```$MIRROR/alpine/edge/community``` to ```/etc/apk/repositories```
- Update database index with ```apk update```
- Install yq with ```apk add yq```
Supported by Tuan Hoang
https://pkgs.alpinelinux.org/package/edge/community/x86/yq
Alpine Linux v3.20+ (and Edge):
```
apk add yq-go
```
Alpine Linux up to v3.19:
```
apk add yq
```
Supported by Tuan Hoang (https://pkgs.alpinelinux.org/packages?name=yq-go)
### Flox:
Flox can be used to install yq on Linux, MacOS, and Windows through WSL.
```
flox install yq
```
## Features
- [Detailed documentation with many examples](https://mikefarah.gitbook.io/yq/)
@@ -349,7 +362,6 @@ Available Commands:
eval (default) Apply the expression to each document in each yaml file in sequence
eval-all Loads _all_ yaml documents of _all_ yaml files and runs expression once
help Help about any command
shell-completion Generate completion script
Flags:
-C, --colors force print with colors
@@ -7,19 +7,20 @@ import (
)
var completionCmd = &cobra.Command{
Use: "shell-completion [bash|zsh|fish|powershell]",
Short: "Generate completion script",
Use: "completion [bash|zsh|fish|powershell]",
Aliases: []string{"shell-completion"},
Short: "Generate the autocompletion script for the specified shell",
Long: `To load completions:
Bash:
$ source <(yq shell-completion bash)
$ source <(yq completion bash)
# To load completions for each session, execute once:
Linux:
$ yq shell-completion bash > /etc/bash_completion.d/yq
$ yq completion bash > /etc/bash_completion.d/yq
MacOS:
$ yq shell-completion bash > /usr/local/etc/bash_completion.d/yq
$ yq completion bash > /usr/local/etc/bash_completion.d/yq
Zsh:
@@ -29,16 +30,16 @@ Zsh:
$ echo "autoload -U compinit; compinit" >> ~/.zshrc
# To load completions for each session, execute once:
$ yq shell-completion zsh > "${fpath[1]}/_yq"
$ yq completion zsh > "${fpath[1]}/_yq"
# You will need to start a new shell for this setup to take effect.
Fish:
$ yq shell-completion fish | source
$ yq completion fish | source
# To load completions for each session, execute once:
$ yq shell-completion fish > ~/.config/fish/completions/yq.fish
$ yq completion fish > ~/.config/fish/completions/yq.fish
`,
DisableFlagsInUseLine: true,
ValidArgs: []string{"bash", "zsh", "fish", "powershell"},
+6 -1
View File
@@ -71,9 +71,14 @@ yq -P -oy sample.json
level := logging.WARNING
stringFormat := `[%{level}] %{color}%{time:15:04:05}%{color:reset} %{message}`
if verbose {
if verbose && forceNoColor {
level = logging.DEBUG
stringFormat = `[%{level:5.5s}] %{time:15:04:05} %{shortfile:-33s} %{shortfunc:-25s} %{message}`
} else if verbose {
level = logging.DEBUG
stringFormat = `[%{level:5.5s}] %{color}%{time:15:04:05}%{color:bold} %{shortfile:-33s} %{shortfunc:-25s}%{color:reset} %{message}`
} else if forceNoColor {
stringFormat = `[%{level}] %{time:15:04:05} %{message}`
}
var format = logging.MustStringFormatter(stringFormat)
-4
View File
@@ -75,10 +75,6 @@ func initCommand(cmd *cobra.Command, args []string) (string, []string, error) {
outputFormat = "yaml"
}
} else if isAutomaticOutputFormat() {
// automatic input worked, we can do it for output too unless specified
if inputFormat == "json" {
yqlib.GetLogger().Warning("JSON file output is now JSON by default (instead of yaml). Use '-oy' or '--output-format=yaml' for yaml output")
}
outputFormat = inputFormat
}
} else if isAutomaticOutputFormat() {
+1 -1
View File
@@ -11,7 +11,7 @@ var (
GitDescribe string
// Version is main version number that is being run at the moment.
Version = "v4.42.1"
Version = "v4.44.3"
// VersionPrerelease is a pre-release marker for the version. If this is "" (empty string)
// then it means that it is a final release. Otherwise, this is a pre-release
-210
View File
@@ -1,210 +0,0 @@
yq (4.16.2) focal; urgency=medium
* Fixed with semicolon space issue
* Updating with documentation
* Added STDIN example to the top
* minor readme cleanup
* Help text tweak
* Fixed docker timeout - simplify docker builds
* New release with docker build fixes
* Updating to go 1.17 to fix CVE #944
* Fix a typo in root.go
* Skip the tests if the nocheck Debian build option is specified
* Fixed select bug (#958)
* Sped up explode operator
* Slight performance improvement to context.ChildContext
* Speed up multiply
* Update README with recently added / changed options
* Make deepMatch report in linear time
* Removed leadingContentPreProcessing flag - header preprocessing is stable
* Revert "Removed leadingContentPreProcessing flag - header preprocessing is stable"
* Keep flag, it is needed in corner cases
* Updated Readme
* Man page
* Fixed expression parsing bug #970
* Bumping go-lang, docker versions
* Added test release flow
* Updated github action release to generate man page
* Bumping version
* Removing no longer needed github action
* Added decoder op
* Fixed newline handling when decoding/encoding
* Fixed newline handling in encoder/decoder
* Can specify indent in encode ops
* Added group_by operator
* Added flatten operator
* Fixed flatten error message
* Improving docs
* Split printer
* Refactored command logic
* Fix JSON encoding removing null #985
* Fixed acceptance tests
* gitbook
* Update document generation script
* Updating README
* Updating release instructions
* github action no longer uses data1.yml
* Create dependabot.yml
* Bump actions/create-release from 1.0.0 to 1.1.4
* Bump actions/setup-go from 1 to 2.1.4
* Bump github.com/goccy/go-yaml from 1.8.9 to 1.9.4
* Bump github.com/jinzhu/copier from 0.2.8 to 0.3.2
* Bump github.com/fatih/color from 1.10.0 to 1.13.0
* Bump github.com/spf13/cobra from 1.1.3 to 1.2.1
* Update dependabot.yml
* Update go.yml
* add build check to PRs
* Include secure as part of build process
* Fixing bad label in github action
* fixed printer test
* remove leading content indicator
* Fixed header preprocessing!
* lint : define golangci configuration file
* Update check.sh
* Load file acceptance test
* Minor improvement on handling front matter
* Improved load doc
* feature: detect MANPATh and install there
* Update install-man-page.sh
* simplify prod stage, move version label to action
* add labels, quote some values
* enable errorlint linter
* Added errorlint to devtools
* Added key operator
* Added more tests
* Fixing comments
* Attempt to fix golint problem
* Include version query for tools
* Clean up errored file?
* enable misspell linter
* updated readme
* update Golangci version to v1.43.0
* gci linter
* Better merge array by key example
* Added credit for merge by array example
* Better formatting of merge arrays example
* Better merge example
* Add accessor for the yq logger instance (#1013)
* Fixed collect op when working with multiple nodes
* Added map, map_values
* Add support for Podman as well as Docker (#1026)
* Bump github.com/jinzhu/copier from 0.3.2 to 0.3.4 (#1027)
* Added csv, tsv output formats
* Added encoder tests
* Cleanup test
* Fixed docker permission issue #1014
* Recording release notes for next release
* Assignment op no longer clobbers anchor (#1029)
* Added sort_by operator
* Improved error message
* Improved tips and tricks
* Report while filename failed to parse #1030
* Added script for extracting checksums
* Improved extract-checksum.sh
* Bump github.com/spf13/cobra from 1.2.1 to 1.3.0 (#1039)
* enable more linters (#1043)
* Bump Golang compiler #1037
-- Roberto Mier Escandon <rmescandon@gmail.com> Tue, 21 Dec 2021 09:41:44 +0000
yq (4.13.0) focal; urgency=medium
* New `with` operator for making multiple changes to a given path
* New `contains` operator, works like the `jq` equivalent
* Subtract operator now supports subtracting elements from arrays!
* Fixed Swapping values using variables #934
* Github Action now properly supports multiline output #936, thanks @pjxiao
* Fixed missing closing bracket validation #932
* Fix processing of hex numbers #929
* Fixed alternative and union operator issues #930
* Can now convert yaml to properties properties format (`-o=props`), See [docs](https://mikefarah.gitbook.io/yq/v/v4.x/usage/properties) for more info.
* Fixed document header/footer comment handling when merging (https://github.com/mikefarah/yq/issues/919)
* pretty print yaml 1.1 compatibility (https://github.com/mikefarah/yq/issues/914)
-- Roberto Mier Escandon <rmescandon@gmail.com> Thu, 16 Sep 2021 20:58:30 +0200
yq (4.9.6) focal; urgency=medium
* Added darwin/arm64 build, thanks @alecthomas
* Incremented docker alpine base version, thanks @da6d6i7-bronga
* Bug fix: multiline expression
* Bug fix: special character
-- Roberto Mier Escandon <rmescandon@gmail.com> Tue, 29 Jun 2021 21:32:14 +0200
yq (3.3.2) focal; urgency=medium
* Bug fix: existStatus bug (#459)
* Automatically makes an os temp directory if it does not exist (#461)
-- Roberto Mier Escandon <rmescandon@gmail.com> Fri, 07 Aug 2020 18:53:01 +0200
yq (3.3-0) focal; urgency=medium
* You can control string styles (quotes) using the new --style flag
* String values now always have quotes when outputting to json
* Negative array indices now traverse the array backwards
* Added a --stripComments flag to print yaml without any comments
* Bumped go to version 1.14
-- Roberto Mier Escandon <rmescandon@gmail.com> Thu, 30 Apr 2020 20:45:44 +0200
yq (3.1-2) eoan; urgency=medium
* Bug fix: yq 3 was removing empty inline-style objects and arrays (#355)
* Bug fix: Merge option returned different output when switching order of
merging files(#347)
* Bug fix: Add new object to existing array object was failing in 3.1.1 (#361)
* Bug fix: yq 3 empty keys did not allow merging of values (#356)
* Bug fix: keys quoted during merge (#363)
* Bug fix: Correct length with wc -l (#362)
* Bug fix: Write to empty document removed path (#359)
-- Roberto Mier Escandon <rmescandon@gmail.com> Mon, 24 Feb 2020 20:31:58 +0100
yq (3.1-1) eoan; urgency=medium
* Keeps yaml comments and formatting, can specify yaml tags when updating.
* Handles anchors
* Can print out matching paths and values when splatting
* JSON output works for all commands
* Yaml files with multiple documents are printed out as one JSON
document per line.
* Deep splat (**) to match arbitrary paths
* Update scripts file format has changed to be more powerful
* Reading and splatting, matching results are printed once per line
* Bugfixing
-- Roberto Mier Escandon <rmescandon@gmail.com> Tue, 11 Feb 2020 22:18:24 +0100
yq (2.2-1) bionic; urgency=medium
* Added Windows support for the "--inplace" command flag
* Prefix now supports arrays
* Add prefix command
* Bump Alpine version to 3.8
* Improved docker build process
* Lint fixes
* Build support for all linux architectures supported by gox
-- Roberto Mier Escandon <rmescandon@gmail.com> Sat, 19 Jan 2019 15:50:47 +0100
yq (2.1-0) bionic; urgency=medium
* Ability to read multiple documents in a single file
* Ability to append list items instead of overwriting
-- Roberto Mier Escandón <rmescandon@gmail.com> Tue, 10 Jul 2018 14:02:42 +0200
yq (2.0-0) bionic; urgency=medium
* Release 2.0.0
-- Roberto Mier Escandón <rmescandon@gmail.com> Wed, 20 Jun 2018 10:29:53 +0200
yq (1.15-0) bionic; urgency=medium
* Release 1.15
-- Roberto Mier Escandón <rmescandon@gmail.com> Wed, 06 Jun 2018 11:32:03 +0200
-1
View File
@@ -1 +0,0 @@
10
-22
View File
@@ -1,22 +0,0 @@
Source: yq
Section: devel
Priority: optional
Maintainer: Roberto Mier Escandón <rmescandon@gmail.com>
Build-Depends: debhelper (>=10),
golang-1.17-go,
pandoc,
rsync
Standards-Version: 4.1.4
Homepage: https://github.com/mikefarah/yq.git
Vcs-Browser: https://github.com/mikefarah/yq.git
Vcs-Git: https://github.com/mikefarah/yq.git
XS-Go-Import-Path: github.com/mikefarah/yq
XSBC-Original-Maintainer: Roberto Mier Escandón <rmescandon@gmail.com>
Package: yq
Architecture: any
Depends: ${shlibs:Depends}, ${misc:Depends}
Description: lightweight and portable command-line data file processor
.
The aim of the project is to be the
[jq](https://github.com/stedolan/jq) or sed of yaml files.
-24
View File
@@ -1,24 +0,0 @@
Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
Upstream-Name: yq
Source: https://github.com/mikefarah/yq.git
Files: *
Copyright: 2017 Mike Farah
License: Expat
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
.
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
-2
View File
@@ -1,2 +0,0 @@
[DEFAULT]
pristine-tar = True
-74
View File
@@ -1,74 +0,0 @@
#!/usr/bin/make -f
#
# Copyright (C) 2018-2021 Roberto Mier Escandón <rmescandon@gmail.com>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License version 3 as
# published by the Free Software Foundation.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
PROJECT := yq
OWNER := mikefarah
REPO := github.com
export DH_OPTIONS
export DH_GOPKG := ${REPO}/${OWNER}/${PROJECT}
export GOROOT := /usr/local/go
export GOPATH := ${CURDIR}/_build
export GOBIN := ${GOPATH}/bin
export PATH := ${GOROOT}/bin:${GOBIN}:${PATH}
export GOCACHE := /tmp/gocache
export GOFLAGS := -mod=vendor
export GO111MODULE := on
SRCDIR := ${GOPATH}/src/${DH_GOPKG}
DESTDIR := ${CURDIR}/debian/${PROJECT}
BINDIR := /usr/bin
MANDIR := /usr/share/man/man1/
ASSETSDIR := /usr/share/${PROJECT}
%:
dh $@
override_dh_auto_build:
mkdir -p ${SRCDIR}
mkdir -p ${GOBIN}
# copy project to local srcdir to build from there
rsync -avz --progress --exclude=_build --exclude=debian --exclude=tmp. --exclude=go.mod --exclude=docs . $(SRCDIR)
# build go code
( \
cd ${SRCDIR} && \
go install -buildmode=pie ./... \
)
# build man page
( \
cd ${SRCDIR} && \
./scripts/generate-man-page-md.sh && \
./scripts/generate-man-page.sh \
)
override_dh_auto_test:
ifeq (,$(filter nocheck,$(DEB_BUILD_OPTIONS)))
(cd ${SRCDIR} && go test -v ./...)
endif
override_dh_auto_install:
cp ${GOBIN}/yq ${DESTDIR}/${BINDIR}
cp -f ${SRCDIR}/LICENSE ${DESTDIR}/${ASSETSDIR}
chmod a+x ${DESTDIR}/${BINDIR}/yq
# man
mkdir -p "${DESTDIR}"/"${MANDIR}"
cp "${SRCDIR}"/yq.1 "${DESTDIR}"/"${MANDIR}" \
override_dh_auto_clean:
dh_clean
rm -rf ${CURDIR}/_build
-1
View File
@@ -1 +0,0 @@
3.0 (native)
-3
View File
@@ -1,3 +0,0 @@
usr/bin
usr/share/yq
usr/share/man/man1
+9 -9
View File
@@ -5,19 +5,19 @@ require (
github.com/alecthomas/participle/v2 v2.1.1
github.com/alecthomas/repr v0.4.0
github.com/dimchansky/utfbom v1.1.1
github.com/elliotchance/orderedmap v1.5.1
github.com/fatih/color v1.16.0
github.com/goccy/go-json v0.10.2
github.com/goccy/go-yaml v1.11.3
github.com/elliotchance/orderedmap v1.6.0
github.com/fatih/color v1.17.0
github.com/goccy/go-json v0.10.3
github.com/goccy/go-yaml v1.12.0
github.com/jinzhu/copier v0.4.0
github.com/magiconair/properties v1.8.7
github.com/pelletier/go-toml/v2 v2.1.1
github.com/pelletier/go-toml/v2 v2.2.2
github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e
github.com/spf13/cobra v1.8.0
github.com/spf13/cobra v1.8.1
github.com/spf13/pflag v1.0.5
github.com/yuin/gopher-lua v1.1.1
golang.org/x/net v0.22.0
golang.org/x/text v0.14.0
golang.org/x/net v0.27.0
golang.org/x/text v0.16.0
gopkg.in/op/go-logging.v1 v1.0.0-20160211212156-b2cb9fa56473
gopkg.in/yaml.v3 v3.0.1
)
@@ -26,7 +26,7 @@ require (
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/mattn/go-colorable v0.1.13 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
golang.org/x/sys v0.18.0 // indirect
golang.org/x/sys v0.22.0 // indirect
golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 // indirect
)
+24 -22
View File
@@ -6,26 +6,26 @@ github.com/alecthomas/participle/v2 v2.1.1 h1:hrjKESvSqGHzRb4yW1ciisFJ4p3MGYih6i
github.com/alecthomas/participle/v2 v2.1.1/go.mod h1:Y1+hAs8DHPmc3YUFzqllV+eSQ9ljPTk0ZkPMtEdAx2c=
github.com/alecthomas/repr v0.4.0 h1:GhI2A8MACjfegCPVq9f1FLvIBS+DrQ2KQBFZP1iFzXc=
github.com/alecthomas/repr v0.4.0/go.mod h1:Fr0507jx4eOXV7AlPV6AVZLYrLIuIeSOWtW57eE/O/4=
github.com/cpuguy83/go-md2man/v2 v2.0.3/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
github.com/cpuguy83/go-md2man/v2 v2.0.4/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dimchansky/utfbom v1.1.1 h1:vV6w1AhK4VMnhBno/TPVCoK9U/LP0PkLCS9tbxHdi/U=
github.com/dimchansky/utfbom v1.1.1/go.mod h1:SxdoEBH5qIqFocHMyGOXVAybYJdr71b1Q/j0mACtrfE=
github.com/elliotchance/orderedmap v1.5.1 h1:G1X4PYlljzimbdQ3RXmtIZiQ9d6aRQ3sH1nzjq5mECE=
github.com/elliotchance/orderedmap v1.5.1/go.mod h1:wsDwEaX5jEoyhbs7x93zk2H/qv0zwuhg4inXhDkYqys=
github.com/fatih/color v1.16.0 h1:zmkK9Ngbjj+K0yRhTVONQh1p/HknKYSlNT+vZCzyokM=
github.com/fatih/color v1.16.0/go.mod h1:fL2Sau1YI5c0pdGEVCbKQbLXB6edEj1ZgiY4NijnWvE=
github.com/elliotchance/orderedmap v1.6.0 h1:xjn+kbbKXeDq6v9RVE+WYwRbYfAZKvlWfcJNxM8pvEw=
github.com/elliotchance/orderedmap v1.6.0/go.mod h1:wsDwEaX5jEoyhbs7x93zk2H/qv0zwuhg4inXhDkYqys=
github.com/fatih/color v1.17.0 h1:GlRw1BRJxkpqUCBKzKOw098ed57fEsKeNjpTe3cSjK4=
github.com/fatih/color v1.17.0/go.mod h1:YZ7TlrGPkiz6ku9fK3TLD/pl3CpsiFyu8N92HLgmosI=
github.com/go-playground/locales v0.13.0 h1:HyWk6mgj5qFqCT5fjGBuRArbVDfE4hi8+e8ceBS/t7Q=
github.com/go-playground/locales v0.13.0/go.mod h1:taPMhCMXrRLJO55olJkUXHZBHCxTMfnGwq/HNwmWNS8=
github.com/go-playground/universal-translator v0.17.0 h1:icxd5fm+REJzpZx7ZfpaD876Lmtgy7VtROAbHHXk8no=
github.com/go-playground/universal-translator v0.17.0/go.mod h1:UkSxE5sNxxRwHyU+Scu5vgOQjsIJAF8j9muTVoKLVtA=
github.com/go-playground/validator/v10 v10.4.1 h1:pH2c5ADXtd66mxoE0Zm9SUhxE20r7aM3F26W0hOn+GE=
github.com/go-playground/validator/v10 v10.4.1/go.mod h1:nlOn6nFhuKACm19sB/8EGNn9GlaMV7XkbRSipzJ0Ii4=
github.com/goccy/go-json v0.10.2 h1:CrxCmQqYDkv1z7lO7Wbh2HN93uovUHgrECaO5ZrCXAU=
github.com/goccy/go-json v0.10.2/go.mod h1:6MelG93GURQebXPDq3khkgXZkazVtN9CRI+MGFi0w8I=
github.com/goccy/go-yaml v1.11.3 h1:B3W9IdWbvrUu2OYQGwvU1nZtvMQJPBKgBUuweJjLj6I=
github.com/goccy/go-yaml v1.11.3/go.mod h1:wKnAMd44+9JAAnGQpWVEgBzGt3YuTaQ4uXoHvE4m7WU=
github.com/goccy/go-json v0.10.3 h1:KZ5WoDbxAIgm2HNbYckL0se1fHD6rz5j4ywS6ebzDqA=
github.com/goccy/go-json v0.10.3/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
github.com/goccy/go-yaml v1.12.0 h1:/1WHjnMsI1dlIBQutrvSMGZRQufVO3asrHfTwfACoPM=
github.com/goccy/go-yaml v1.12.0/go.mod h1:wKnAMd44+9JAAnGQpWVEgBzGt3YuTaQ4uXoHvE4m7WU=
github.com/google/go-cmp v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38=
github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/hexops/gotextdiff v1.0.3 h1:gitA9+qJrrTCsiCl7+kh75nPqQt1cx4ZkudSTLoUqJM=
@@ -43,37 +43,39 @@ github.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovk
github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM=
github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
github.com/pelletier/go-toml/v2 v2.1.1 h1:LWAJwfNvjQZCFIDKWYQaM62NcYeYViCmWIwmOStowAI=
github.com/pelletier/go-toml/v2 v2.1.1/go.mod h1:tJU2Z3ZkXwnxa4DPO899bsyIoywizdUvyaeZurnPPDc=
github.com/pelletier/go-toml/v2 v2.2.2 h1:aYUidT7k73Pcl9nb2gScu7NSrKCSHIDE89b3+6Wq+LM=
github.com/pelletier/go-toml/v2 v2.2.2/go.mod h1:1t835xjRzz80PqgE6HHgN2JOsmgYu/h4qDAS4n929Rs=
github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e h1:aoZm08cpOy4WuID//EZDgcC4zIxODThtZNPirFr42+A=
github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e/go.mod h1:pJLUxLENpZxwdsKMEsNbx1VGcRFpLqf3715MtcvvzbA=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/spf13/cobra v1.8.0 h1:7aJaZx1B85qltLMc546zn58BxxfZdR/W22ej9CFoEf0=
github.com/spf13/cobra v1.8.0/go.mod h1:WXLWApfZ71AjXPya3WOlMsY9yMs7YeiHhFVlvLyhcho=
github.com/spf13/cobra v1.8.1 h1:e5/vxKd/rZsfSJMUX1agtjeTDf+qv1/JdBF8gg5k9ZM=
github.com/spf13/cobra v1.8.1/go.mod h1:wHxEcudfqmLYa8iTfL+OuZPbBZkmvliBWKIezN3kD9Y=
github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
github.com/stretchr/objx v0.5.2/go.mod h1:FRsXN1f5AsAjCGJKqEizvkpNtU+EGNCLh3NxZ/8L+MA=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk=
github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsTg=
github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
github.com/yuin/gopher-lua v1.1.1 h1:kYKnWBjvbNP4XLT3+bPEwAXJx262OhaHDWDVOPjL46M=
github.com/yuin/gopher-lua v1.1.1/go.mod h1:GBR0iDaNXjAgGg9zfCvksxSRnQx76gclCIb7kdAd1Pw=
golang.org/x/crypto v0.21.0 h1:X31++rzVUdKhX5sWmSOFZxx8UW/ldWx55cbf08iNAMA=
golang.org/x/crypto v0.21.0/go.mod h1:0BP7YvVV9gBbVKyeTG0Gyn+gZm94bibOW5BjDEYAOMs=
golang.org/x/net v0.22.0 h1:9sGLhx7iRIHEiX0oAJ3MRZMUCElJgy7Br1nO+AMN3Tc=
golang.org/x/net v0.22.0/go.mod h1:JKghWKKOSdJwpW2GEx0Ja7fmaKnMsbu+MWVZTokSYmg=
golang.org/x/crypto v0.25.0 h1:ypSNr+bnYL2YhwoMt2zPxHFmbAN1KZs/njMG3hxUp30=
golang.org/x/crypto v0.25.0/go.mod h1:T+wALwcMOSE0kXgUAnPAHqTLW+XHgcELELW8VaDgm/M=
golang.org/x/net v0.27.0 h1:5K3Njcw06/l2y9vpGCSdcxWOYHOUk3dVNGDXN+FvAys=
golang.org/x/net v0.27.0/go.mod h1:dDi0PyhWNoiUOrAS8uXv/vnScO4wnHQO4mj9fn/RytE=
golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.18.0 h1:DBdB3niSjOA/O0blCZBqDefyWNYveAYMNF1Wum0DYQ4=
golang.org/x/sys v0.18.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/text v0.14.0 h1:ScX5w1eTa3QqT8oi6+ziP7dTV1S2+ALU0bI+0zXKWiQ=
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/sys v0.22.0 h1:RI27ohtqKCnwULzJLqkv897zojh5/DwS/ENaMzUOaWI=
golang.org/x/sys v0.22.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/text v0.16.0 h1:a94ExnEXNtEwYLGJSIUxnWoxoRz/ZcCsV63ROupILh4=
golang.org/x/text v0.16.0/go.mod h1:GhwF1Be+LQoKShO3cGOHzqOgRrGaYc9AvblQOmPVHnI=
golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 h1:H2TDz8ibqkAF6YGhCdN3jS9O0/s90v0rJh3X/OLHEUk=
golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2/go.mod h1:K8+ghG5WaK9qNqU5K3HdILfMLy1f3aNYFI/wnl100a8=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
+1 -1
View File
@@ -7,6 +7,6 @@ import (
"os"
)
func changeOwner(info fs.FileInfo, file *os.File) error {
func changeOwner(_ fs.FileInfo, _ *os.File) error {
return nil
}
+14
View File
@@ -186,6 +186,20 @@ var csvScenarios = []formatScenario{
expected: expectedYamlFromCSVNoParsing,
scenarioType: "decode-csv-no-auto",
},
{
description: "values starting with #, no auto parse",
skipDoc: true,
input: "value\n#ffff",
expected: "- value: '#ffff'\n",
scenarioType: "decode-csv-no-auto",
},
{
description: "values starting with #",
skipDoc: true,
input: "value\n#ffff",
expected: "- value: #ffff\n",
scenarioType: "decode-csv",
},
{
description: "Scalar roundtrip",
skipDoc: true,
+1 -1
View File
@@ -31,7 +31,7 @@ func (dec *csvObjectDecoder) convertToNode(content string) *CandidateNode {
node, err := parseSnippet(content)
// if we're not auto-parsing, then we wont put in parsed objects or arrays
// but we still parse scalars
if err != nil || (!dec.prefs.AutoParse && node.Kind != ScalarNode) {
if err != nil || (!dec.prefs.AutoParse && (node.Kind != ScalarNode || node.Value != content)) {
return createScalarNode(content, content)
}
return node
+3
View File
@@ -0,0 +1,3 @@
# Max
Computes the maximum among an incoming sequence of scalar values.
+3
View File
@@ -0,0 +1,3 @@
# Min
Computes the minimum among an incoming sequence of scalar values.
+3
View File
@@ -0,0 +1,3 @@
# Omit
Works like `pick`, but instead you specify the keys/indices that you _don't_ want included.
+3
View File
@@ -0,0 +1,3 @@
# Pivot
Emulates the `PIVOT` function supported by several popular RDBMS systems.
@@ -2,4 +2,4 @@
The slice array operator takes an array as input and returns a subarray. Like the `jq` equivalent, `.[10:15]` will return an array of length 5, starting from index 10 inclusive, up to index 15 exclusive. Negative numbers count backwards from the end of the array.
You may leave out the first or second number, which will will refer to the start or end of the array respectively.
You may leave out the first or second number, which will refer to the start or end of the array respectively.
+51
View File
@@ -0,0 +1,51 @@
# Max
Computes the maximum among an incoming sequence of scalar values.
## Maximum int
Given a sample.yml file of:
```yaml
- 99
- 16
- 12
- 6
- 66
```
then
```bash
yq 'max' sample.yml
```
will output
```yaml
99
```
## Maximum string
Given a sample.yml file of:
```yaml
- foo
- bar
- baz
```
then
```bash
yq 'max' sample.yml
```
will output
```yaml
foo
```
## Maximum of empty
Given a sample.yml file of:
```yaml
[]
```
then
```bash
yq 'max' sample.yml
```
will output
```yaml
```
+51
View File
@@ -0,0 +1,51 @@
# Min
Computes the minimum among an incoming sequence of scalar values.
## Minimum int
Given a sample.yml file of:
```yaml
- 99
- 16
- 12
- 6
- 66
```
then
```bash
yq 'min' sample.yml
```
will output
```yaml
6
```
## Minimum string
Given a sample.yml file of:
```yaml
- foo
- bar
- baz
```
then
```bash
yq 'min' sample.yml
```
will output
```yaml
bar
```
## Minimum of empty
Given a sample.yml file of:
```yaml
[]
```
then
```bash
yq 'min' sample.yml
```
will output
```yaml
```
+56
View File
@@ -55,6 +55,62 @@ a: 12
b: 4
```
## Multiply string node X int
Given a sample.yml file of:
```yaml
b: banana
```
then
```bash
yq '.b * 4' sample.yml
```
will output
```yaml
bananabananabananabanana
```
## Multiply int X string node
Given a sample.yml file of:
```yaml
b: banana
```
then
```bash
yq '4 * .b' sample.yml
```
will output
```yaml
bananabananabananabanana
```
## Multiply string X int node
Given a sample.yml file of:
```yaml
n: 4
```
then
```bash
yq '"banana" * .n' sample.yml
```
will output
```yaml
bananabananabananabanana
```
## Multiply int node X string
Given a sample.yml file of:
```yaml
n: 4
```
then
```bash
yq '.n * "banana"' sample.yml
```
will output
```yaml
bananabananabananabanana
```
## Merge objects together, returning merged result only
Given a sample.yml file of:
```yaml
+44
View File
@@ -0,0 +1,44 @@
# Omit
Works like `pick`, but instead you specify the keys/indices that you _don't_ want included.
## Omit keys from map
Note that non existent keys are skipped.
Given a sample.yml file of:
```yaml
myMap:
cat: meow
dog: bark
thing: hamster
hamster: squeak
```
then
```bash
yq '.myMap |= omit(["hamster", "cat", "goat"])' sample.yml
```
will output
```yaml
myMap:
dog: bark
thing: hamster
```
## Omit indices from array
Note that non existent indices are skipped.
Given a sample.yml file of:
```yaml
- cat
- leopard
- lion
```
then
```bash
yq 'omit([2, 0, 734, -5])' sample.yml
```
will output
```yaml
- leopard
```
+117
View File
@@ -0,0 +1,117 @@
# Pivot
Emulates the `PIVOT` function supported by several popular RDBMS systems.
## Pivot a sequence of sequences
Given a sample.yml file of:
```yaml
- - foo
- bar
- baz
- - sis
- boom
- bah
```
then
```bash
yq 'pivot' sample.yml
```
will output
```yaml
- - foo
- sis
- - bar
- boom
- - baz
- bah
```
## Pivot sequence of heterogeneous sequences
Missing values are "padded" to null.
Given a sample.yml file of:
```yaml
- - foo
- bar
- baz
- - sis
- boom
- bah
- blah
```
then
```bash
yq 'pivot' sample.yml
```
will output
```yaml
- - foo
- sis
- - bar
- boom
- - baz
- bah
- -
- blah
```
## Pivot sequence of maps
Given a sample.yml file of:
```yaml
- foo: a
bar: b
baz: c
- foo: x
bar: y
baz: z
```
then
```bash
yq 'pivot' sample.yml
```
will output
```yaml
foo:
- a
- x
bar:
- b
- y
baz:
- c
- z
```
## Pivot sequence of heterogeneous maps
Missing values are "padded" to null.
Given a sample.yml file of:
```yaml
- foo: a
bar: b
baz: c
- foo: x
bar: y
baz: z
what: ever
```
then
```bash
yq 'pivot' sample.yml
```
will output
```yaml
foo:
- a
- x
bar:
- b
- y
baz:
- c
- z
what:
-
- ever
```
+1 -1
View File
@@ -2,7 +2,7 @@
The slice array operator takes an array as input and returns a subarray. Like the `jq` equivalent, `.[10:15]` will return an array of length 5, starting from index 10 inclusive, up to index 15 exclusive. Negative numbers count backwards from the end of the array.
You may leave out the first or second number, which will will refer to the start or end of the array respectively.
You may leave out the first or second number, which will refer to the start or end of the array respectively.
## Slicing arrays
Given a sample.yml file of:
+45 -1
View File
@@ -63,7 +63,29 @@ will output
- ~
```
## Unique array object fields
## Unique array objects
Given a sample.yml file of:
```yaml
- name: harry
pet: cat
- name: billy
pet: dog
- name: harry
pet: cat
```
then
```bash
yq 'unique' sample.yml
```
will output
```yaml
- name: harry
pet: cat
- name: billy
pet: dog
```
## Unique array of objects by a field
Given a sample.yml file of:
```yaml
- name: harry
@@ -85,3 +107,25 @@ will output
pet: dog
```
## Unique array of arrays
Given a sample.yml file of:
```yaml
- - cat
- dog
- - cat
- sheep
- - cat
- dog
```
then
```bash
yq 'unique' sample.yml
```
will output
```yaml
- - cat
- dog
- - cat
- sheep
```
+17
View File
@@ -230,6 +230,23 @@ person:
- pizza
```
## Decode properties: numbers
All values are assumed to be strings when parsing properties, but you can use the `from_yaml` operator on all the strings values to autoparse into the correct type.
Given a sample.properties file of:
```properties
a.b = 10
```
then
```bash
yq -p=props ' (.. | select(tag == "!!str")) |= from_yaml' sample.properties
```
will output
```yaml
a:
b: 10
```
## Decode properties - array should be a map
If you have a numeric map key in your property files, use array_to_map to convert them to maps.
+10 -8
View File
@@ -2,6 +2,7 @@ package yqlib
import (
"fmt"
"path/filepath"
"strings"
)
@@ -107,12 +108,11 @@ func (f *Format) GetConfiguredEncoder() Encoder {
}
func FormatStringFromFilename(filename string) string {
if filename != "" {
GetLogger().Debugf("checking file extension '%s' for auto format detection", filename)
nPos := strings.LastIndex(filename, ".")
if nPos > -1 {
format := filename[nPos+1:]
GetLogger().Debugf("checking filename '%s' for auto format detection", filename)
ext := filepath.Ext(filename)
if ext != "" && ext[0] == '.' {
format := strings.ToLower(ext[1:])
GetLogger().Debugf("detected format '%s'", format)
return format
}
@@ -123,9 +123,11 @@ func FormatStringFromFilename(filename string) string {
}
func FormatFromString(format string) (*Format, error) {
for _, printerFormat := range Formats {
if printerFormat.MatchesName(format) {
return printerFormat, nil
if format != "" {
for _, printerFormat := range Formats {
if printerFormat.MatchesName(format) {
return printerFormat, nil
}
}
}
return nil, fmt.Errorf("unknown format '%v' please use [%v]", format, GetAvailableOutputFormatString())
+62
View File
@@ -0,0 +1,62 @@
package yqlib
import (
"fmt"
"strings"
"testing"
"github.com/mikefarah/yq/v4/test"
)
type formatStringScenario struct {
description string
input string
expectedFormat *Format
expectedError string
}
var formatStringScenarios = []formatStringScenario{
{
description: "yaml",
input: "yaml",
expectedFormat: YamlFormat,
},
{
description: "Unknown format type",
input: "doc",
expectedError: "unknown format 'doc' please use",
},
{
description: "blank should error",
input: "",
expectedError: "unknown format '' please use",
},
}
func TestFormatFromString(t *testing.T) {
for _, tt := range formatStringScenarios {
actualFormat, actualError := FormatFromString(tt.input)
if tt.expectedError != "" {
if actualError == nil {
t.Errorf("Expected [%v] error but found none", tt.expectedError)
} else {
test.AssertResultWithContext(t, true, strings.Contains(actualError.Error(), tt.expectedError),
fmt.Sprintf("Expected [%v] to contain [%v]", actualError.Error(), tt.expectedError),
)
}
} else {
test.AssertResult(t, tt.expectedFormat, actualFormat)
}
}
}
func TestFormatStringFromFilename(t *testing.T) {
test.AssertResult(t, "yaml", FormatStringFromFilename("test.Yaml"))
test.AssertResult(t, "yaml", FormatStringFromFilename("test.index.Yaml"))
test.AssertResult(t, "yaml", FormatStringFromFilename("test"))
test.AssertResult(t, "json", FormatStringFromFilename("test.json"))
test.AssertResult(t, "json", FormatStringFromFilename("TEST.JSON"))
test.AssertResult(t, "yaml", FormatStringFromFilename("test.json/foo"))
test.AssertResult(t, "yaml", FormatStringFromFilename(""))
}
+8 -2
View File
@@ -40,6 +40,7 @@ var participleYqRules = []*participleYqRule{
simpleOp("map", mapOpType),
simpleOp("filter", filterOpType),
simpleOp("pick", pickOpType),
simpleOp("omit", omitOpType),
{"FlattenWithDepth", `flatten\([0-9]+\)`, flattenWithDepth(), 0},
{"Flatten", `flatten`, opTokenWithPrefs(flattenOpType, nil, flattenPreferences{depth: -1}), 0},
@@ -198,6 +199,9 @@ var participleYqRules = []*participleYqRule{
{"GreaterThan", `\s*>\s*`, opTokenWithPrefs(compareOpType, nil, compareTypePref{OrEqual: false, Greater: true}), 0},
{"LessThan", `\s*<\s*`, opTokenWithPrefs(compareOpType, nil, compareTypePref{OrEqual: false, Greater: false}), 0},
simpleOp("min", minOpType),
simpleOp("max", maxOpType),
{"AssignRelative", `\|=[c]*`, assignOpToken(true), 0},
{"Assign", `=[c]*`, assignOpToken(false), 0},
@@ -223,6 +227,8 @@ var participleYqRules = []*participleYqRule{
{"SubtractAssign", `\-=`, opToken(subtractAssignOpType), 0},
{"Subtract", `\-`, opToken(subtractOpType), 0},
{"Comment", `#.*`, nil, 0},
simpleOp("pivot", pivotOpType),
}
type yqAction func(lexer.Token) (*token, error)
@@ -322,7 +328,7 @@ func flattenWithDepth() yqAction {
prefs := flattenPreferences{depth: depth}
op := &Operation{OperationType: flattenOpType, Value: flattenOpType.Type, StringValue: value, Preferences: prefs}
return &token{TokenType: operationToken, Operation: op}, nil
return &token{TokenType: operationToken, Operation: op, CheckForPostTraverse: flattenOpType.CheckForPostTraverse}, nil
}
}
@@ -400,7 +406,7 @@ func envOp(strenv bool) yqAction {
envOperation.OperationType = envOpType
envOperation.Preferences = preferences
return &token{TokenType: operationToken, Operation: envOperation}, nil
return &token{TokenType: operationToken, Operation: envOperation, CheckForPostTraverse: envOpType.CheckForPostTraverse}, nil
}
}
File diff suppressed because it is too large Load Diff
+16 -2
View File
@@ -95,10 +95,20 @@ func parseSnippet(value string) (*CandidateNode, error) {
if err != nil {
return nil, err
}
if result.Kind == ScalarNode {
result.LineComment = result.LeadingContent
} else {
result.HeadComment = result.LeadingContent
}
result.LeadingContent = ""
if result.Tag == "!!str" {
// use the original string value, as
// decoding drops new lines
return createScalarNode(value, value), nil
newNode := createScalarNode(value, value)
newNode.LineComment = result.LineComment
return newNode, nil
}
result.Line = 0
result.Column = 0
@@ -135,8 +145,12 @@ func recursiveNodeEqual(lhs *CandidateNode, rhs *CandidateNode) bool {
return false
}
// yaml numbers can be hex and octal encoded...
// yaml numbers can have underscores, be hex and octal encoded...
func parseInt64(numberString string) (string, int64, error) {
if strings.Contains(numberString, "_") {
numberString = strings.ReplaceAll(numberString, "_", "")
}
if strings.HasPrefix(numberString, "0x") ||
strings.HasPrefix(numberString, "0X") {
num, err := strconv.ParseInt(numberString[2:], 16, 64)
+25 -1
View File
@@ -62,6 +62,16 @@ var parseSnippetScenarios = []parseSnippetScenario{
Column: 0,
},
},
{
snippet: "# things",
expected: &CandidateNode{
Kind: ScalarNode,
Tag: "!!null",
LineComment: "# things",
Line: 0,
Column: 0,
},
},
{
snippet: "3.1",
expected: &CandidateNode{
@@ -106,6 +116,7 @@ func TestParseSnippet(t *testing.T) {
type parseInt64Scenario struct {
numberString string
expectedParsedNumber int64
expectedFormatString string
}
var parseInt64Scenarios = []parseInt64Scenario{
@@ -113,10 +124,20 @@ var parseInt64Scenarios = []parseInt64Scenario{
numberString: "34",
expectedParsedNumber: 34,
},
{
numberString: "10_000",
expectedParsedNumber: 10000,
expectedFormatString: "10000",
},
{
numberString: "0x10",
expectedParsedNumber: 16,
},
{
numberString: "0x10_000",
expectedParsedNumber: 65536,
expectedFormatString: "0x10000",
},
{
numberString: "0o10",
expectedParsedNumber: 8,
@@ -132,7 +153,10 @@ func TestParseInt64(t *testing.T) {
t.Error(err)
}
test.AssertResultComplexWithContext(t, tt.expectedParsedNumber, actualNumber, tt.numberString)
if tt.expectedFormatString == "" {
tt.expectedFormatString = tt.numberString
}
test.AssertResultComplexWithContext(t, tt.numberString, fmt.Sprintf(format, actualNumber), fmt.Sprintf("Formatting of: %v", tt.numberString))
test.AssertResultComplexWithContext(t, tt.expectedFormatString, fmt.Sprintf(format, actualNumber), fmt.Sprintf("Formatting of: %v", tt.numberString))
}
}
+30 -24
View File
@@ -73,6 +73,8 @@ var equalsOpType = &operationType{Type: "EQUALS", NumArgs: 2, Precedence: 40, Ha
var notEqualsOpType = &operationType{Type: "NOT_EQUALS", NumArgs: 2, Precedence: 40, Handler: notEqualsOperator}
var compareOpType = &operationType{Type: "COMPARE", NumArgs: 2, Precedence: 40, Handler: compareOperator}
var minOpType = &operationType{Type: "MIN", NumArgs: 0, Precedence: 40, Handler: minOperator}
var maxOpType = &operationType{Type: "MAX", NumArgs: 0, Precedence: 40, Handler: maxOperator}
// createmap needs to be above union, as we use union to build the components of the objects
var createMapOpType = &operationType{Type: "CREATE_MAP", NumArgs: 2, Precedence: 15, Handler: createMapOperator}
@@ -83,15 +85,17 @@ var lengthOpType = &operationType{Type: "LENGTH", NumArgs: 0, Precedence: 50, Ha
var lineOpType = &operationType{Type: "LINE", NumArgs: 0, Precedence: 50, Handler: lineOperator}
var columnOpType = &operationType{Type: "LINE", NumArgs: 0, Precedence: 50, Handler: columnOperator}
// Use this expression to create alias/syntactic sugar expressions (in lexer_participle).
var expressionOpType = &operationType{Type: "EXP", NumArgs: 0, Precedence: 50, Handler: expressionOperator}
var collectOpType = &operationType{Type: "COLLECT", NumArgs: 1, Precedence: 50, Handler: collectOperator}
var mapOpType = &operationType{Type: "MAP", NumArgs: 1, Precedence: 50, Handler: mapOperator}
var filterOpType = &operationType{Type: "FILTER", NumArgs: 1, Precedence: 50, Handler: filterOperator}
var mapOpType = &operationType{Type: "MAP", NumArgs: 1, Precedence: 52, Handler: mapOperator, CheckForPostTraverse: true}
var filterOpType = &operationType{Type: "FILTER", NumArgs: 1, Precedence: 52, Handler: filterOperator, CheckForPostTraverse: true}
var errorOpType = &operationType{Type: "ERROR", NumArgs: 1, Precedence: 50, Handler: errorOperator}
var pickOpType = &operationType{Type: "PICK", NumArgs: 1, Precedence: 50, Handler: pickOperator}
var evalOpType = &operationType{Type: "EVAL", NumArgs: 1, Precedence: 50, Handler: evalOperator}
var mapValuesOpType = &operationType{Type: "MAP_VALUES", NumArgs: 1, Precedence: 50, Handler: mapValuesOperator}
var pickOpType = &operationType{Type: "PICK", NumArgs: 1, Precedence: 52, Handler: pickOperator, CheckForPostTraverse: true}
var omitOpType = &operationType{Type: "OMIT", NumArgs: 1, Precedence: 52, Handler: omitOperator, CheckForPostTraverse: true}
var evalOpType = &operationType{Type: "EVAL", NumArgs: 1, Precedence: 52, Handler: evalOperator, CheckForPostTraverse: true}
var mapValuesOpType = &operationType{Type: "MAP_VALUES", NumArgs: 1, Precedence: 52, Handler: mapValuesOperator, CheckForPostTraverse: true}
var formatDateTimeOpType = &operationType{Type: "FORMAT_DATE_TIME", NumArgs: 1, Precedence: 50, Handler: formatDateTime}
var withDtFormatOpType = &operationType{Type: "WITH_DATE_TIME_FORMAT", NumArgs: 1, Precedence: 50, Handler: withDateTimeFormat}
@@ -113,9 +117,9 @@ var toEntriesOpType = &operationType{Type: "TO_ENTRIES", NumArgs: 0, Precedence:
var fromEntriesOpType = &operationType{Type: "FROM_ENTRIES", NumArgs: 0, Precedence: 50, Handler: fromEntriesOperator}
var withEntriesOpType = &operationType{Type: "WITH_ENTRIES", NumArgs: 1, Precedence: 50, Handler: withEntriesOperator}
var withOpType = &operationType{Type: "WITH", NumArgs: 1, Precedence: 50, Handler: withOperator}
var withOpType = &operationType{Type: "WITH", NumArgs: 1, Precedence: 52, Handler: withOperator, CheckForPostTraverse: true}
var splitDocumentOpType = &operationType{Type: "SPLIT_DOC", NumArgs: 0, Precedence: 50, Handler: splitDocumentOperator}
var splitDocumentOpType = &operationType{Type: "SPLIT_DOC", NumArgs: 0, Precedence: 52, Handler: splitDocumentOperator, CheckForPostTraverse: true}
var getVariableOpType = &operationType{Type: "GET_VARIABLE", NumArgs: 0, Precedence: 55, Handler: getVariableOperator}
var getStyleOpType = &operationType{Type: "GET_STYLE", NumArgs: 0, Precedence: 50, Handler: getStyleOperator}
var getTagOpType = &operationType{Type: "GET_TAG", NumArgs: 0, Precedence: 50, Handler: getTagOperator}
@@ -132,33 +136,33 @@ var getDocumentIndexOpType = &operationType{Type: "GET_DOCUMENT_INDEX", NumArgs:
var getFilenameOpType = &operationType{Type: "GET_FILENAME", NumArgs: 0, Precedence: 50, Handler: getFilenameOperator}
var getFileIndexOpType = &operationType{Type: "GET_FILE_INDEX", NumArgs: 0, Precedence: 50, Handler: getFileIndexOperator}
var getPathOpType = &operationType{Type: "GET_PATH", NumArgs: 0, Precedence: 50, Handler: getPathOperator}
var getPathOpType = &operationType{Type: "GET_PATH", NumArgs: 0, Precedence: 52, Handler: getPathOperator, CheckForPostTraverse: true}
var setPathOpType = &operationType{Type: "SET_PATH", NumArgs: 1, Precedence: 50, Handler: setPathOperator}
var delPathsOpType = &operationType{Type: "DEL_PATHS", NumArgs: 1, Precedence: 50, Handler: delPathsOperator}
var delPathsOpType = &operationType{Type: "DEL_PATHS", NumArgs: 1, Precedence: 52, Handler: delPathsOperator, CheckForPostTraverse: true}
var explodeOpType = &operationType{Type: "EXPLODE", NumArgs: 1, Precedence: 50, Handler: explodeOperator}
var sortByOpType = &operationType{Type: "SORT_BY", NumArgs: 1, Precedence: 50, Handler: sortByOperator}
var reverseOpType = &operationType{Type: "REVERSE", NumArgs: 0, Precedence: 50, Handler: reverseOperator}
var sortOpType = &operationType{Type: "SORT", NumArgs: 0, Precedence: 50, Handler: sortOperator}
var shuffleOpType = &operationType{Type: "SHUFFLE", NumArgs: 0, Precedence: 50, Handler: shuffleOperator}
var explodeOpType = &operationType{Type: "EXPLODE", NumArgs: 1, Precedence: 52, Handler: explodeOperator, CheckForPostTraverse: true}
var sortByOpType = &operationType{Type: "SORT_BY", NumArgs: 1, Precedence: 52, Handler: sortByOperator, CheckForPostTraverse: true}
var reverseOpType = &operationType{Type: "REVERSE", NumArgs: 0, Precedence: 52, Handler: reverseOperator, CheckForPostTraverse: true}
var sortOpType = &operationType{Type: "SORT", NumArgs: 0, Precedence: 52, Handler: sortOperator, CheckForPostTraverse: true}
var shuffleOpType = &operationType{Type: "SHUFFLE", NumArgs: 0, Precedence: 52, Handler: shuffleOperator, CheckForPostTraverse: true}
var sortKeysOpType = &operationType{Type: "SORT_KEYS", NumArgs: 1, Precedence: 50, Handler: sortKeysOperator}
var sortKeysOpType = &operationType{Type: "SORT_KEYS", NumArgs: 1, Precedence: 52, Handler: sortKeysOperator, CheckForPostTraverse: true}
var joinStringOpType = &operationType{Type: "JOIN", NumArgs: 1, Precedence: 50, Handler: joinStringOperator}
var subStringOpType = &operationType{Type: "SUBSTR", NumArgs: 1, Precedence: 50, Handler: substituteStringOperator}
var matchOpType = &operationType{Type: "MATCH", NumArgs: 1, Precedence: 50, Handler: matchOperator}
var captureOpType = &operationType{Type: "CAPTURE", NumArgs: 1, Precedence: 50, Handler: captureOperator}
var testOpType = &operationType{Type: "TEST", NumArgs: 1, Precedence: 50, Handler: testOperator}
var splitStringOpType = &operationType{Type: "SPLIT", NumArgs: 1, Precedence: 50, Handler: splitStringOperator}
var splitStringOpType = &operationType{Type: "SPLIT", NumArgs: 1, Precedence: 52, Handler: splitStringOperator, CheckForPostTraverse: true}
var changeCaseOpType = &operationType{Type: "CHANGE_CASE", NumArgs: 0, Precedence: 50, Handler: changeCaseOperator}
var trimOpType = &operationType{Type: "TRIM", NumArgs: 0, Precedence: 50, Handler: trimSpaceOperator}
var toStringOpType = &operationType{Type: "TO_STRING", NumArgs: 0, Precedence: 50, Handler: toStringOperator}
var stringInterpolationOpType = &operationType{Type: "STRING_INT", NumArgs: 0, Precedence: 50, Handler: stringInterpolationOperator, ToString: valueToStringFunc}
var loadOpType = &operationType{Type: "LOAD", NumArgs: 1, Precedence: 52, Handler: loadOperator}
var loadOpType = &operationType{Type: "LOAD", NumArgs: 1, Precedence: 52, Handler: loadOperator, CheckForPostTraverse: true}
var loadStringOpType = &operationType{Type: "LOAD_STRING", NumArgs: 1, Precedence: 52, Handler: loadStringOperator}
var keysOpType = &operationType{Type: "KEYS", NumArgs: 0, Precedence: 50, Handler: keysOperator}
var keysOpType = &operationType{Type: "KEYS", NumArgs: 0, Precedence: 52, Handler: keysOperator, CheckForPostTraverse: true}
var collectObjectOpType = &operationType{Type: "COLLECT_OBJECT", NumArgs: 0, Precedence: 50, Handler: collectObjectOperator}
@@ -172,7 +176,7 @@ var traverseArrayOpType = &operationType{Type: "TRAVERSE_ARRAY", NumArgs: 2, Pre
var selfReferenceOpType = &operationType{Type: "SELF", NumArgs: 0, Precedence: 55, Handler: selfOperator}
var valueOpType = &operationType{Type: "VALUE", NumArgs: 0, Precedence: 50, Handler: valueOperator, ToString: valueToStringFunc}
var referenceOpType = &operationType{Type: "REF", NumArgs: 0, Precedence: 50, Handler: referenceOperator}
var envOpType = &operationType{Type: "ENV", NumArgs: 0, Precedence: 50, Handler: envOperator}
var envOpType = &operationType{Type: "ENV", NumArgs: 0, Precedence: 52, Handler: envOperator, CheckForPostTraverse: true}
var notOpType = &operationType{Type: "NOT", NumArgs: 0, Precedence: 50, Handler: notOperator}
var toNumberOpType = &operationType{Type: "TO_NUMBER", NumArgs: 0, Precedence: 50, Handler: toNumberOperator}
var emptyOpType = &operationType{Type: "EMPTY", Precedence: 50, Handler: emptyOperator}
@@ -181,14 +185,16 @@ var envsubstOpType = &operationType{Type: "ENVSUBST", NumArgs: 0, Precedence: 50
var recursiveDescentOpType = &operationType{Type: "RECURSIVE_DESCENT", NumArgs: 0, Precedence: 50, Handler: recursiveDescentOperator}
var selectOpType = &operationType{Type: "SELECT", NumArgs: 1, Precedence: 50, Handler: selectOperator}
var selectOpType = &operationType{Type: "SELECT", NumArgs: 1, Precedence: 52, Handler: selectOperator, CheckForPostTraverse: true}
var hasOpType = &operationType{Type: "HAS", NumArgs: 1, Precedence: 50, Handler: hasOperator}
var uniqueOpType = &operationType{Type: "UNIQUE", NumArgs: 0, Precedence: 50, Handler: unique}
var uniqueByOpType = &operationType{Type: "UNIQUE_BY", NumArgs: 1, Precedence: 50, Handler: uniqueBy}
var groupByOpType = &operationType{Type: "GROUP_BY", NumArgs: 1, Precedence: 50, Handler: groupBy}
var flattenOpType = &operationType{Type: "FLATTEN_BY", NumArgs: 0, Precedence: 50, Handler: flattenOp}
var uniqueOpType = &operationType{Type: "UNIQUE", NumArgs: 0, Precedence: 52, Handler: unique, CheckForPostTraverse: true}
var uniqueByOpType = &operationType{Type: "UNIQUE_BY", NumArgs: 1, Precedence: 52, Handler: uniqueBy, CheckForPostTraverse: true}
var groupByOpType = &operationType{Type: "GROUP_BY", NumArgs: 1, Precedence: 52, Handler: groupBy, CheckForPostTraverse: true}
var flattenOpType = &operationType{Type: "FLATTEN_BY", NumArgs: 0, Precedence: 52, Handler: flattenOp, CheckForPostTraverse: true}
var deleteChildOpType = &operationType{Type: "DELETE", NumArgs: 1, Precedence: 40, Handler: deleteChildOperator}
var pivotOpType = &operationType{Type: "PIVOT", NumArgs: 0, Precedence: 52, Handler: pivotOperator, CheckForPostTraverse: true}
// debugging purposes only
func (p *Operation) toString() string {
if p == nil {
@@ -188,6 +188,16 @@ var anchorOperatorScenarios = []expressionScenario{
"D0, P[b], (!!str)::cat\n",
},
},
{
description: "Explode splat",
skipDoc: true,
document: `{a: &a cat, b: *a}`,
expression: `explode(.)[]`,
expected: []string{
"D0, P[a], (!!str)::cat\n",
"D0, P[b], (!!str)::cat\n",
},
},
{
description: "Explode alias and anchor - check original parent",
skipDoc: true,
+10
View File
@@ -28,6 +28,16 @@ var collectObjectOperatorScenarios = []expressionScenario{
"D0, P[], (!!map)::dog: great\n",
},
},
{
description: "collect splat",
skipDoc: true,
document: `[{name: cat}, {name: dog}]`,
expression: `.[] | {.name: "great"}[]`,
expected: []string{
"D0, P[cat], (!!str)::great\n",
"D0, P[dog], (!!str)::great\n",
},
},
{
skipDoc: true,
expression: `({} + {}) | (.b = 3)`,
+8
View File
@@ -109,6 +109,14 @@ var collectOperatorScenarios = []expressionScenario{
"D0, P[], (!!seq)::- 1\n- 2\n- 3\n",
},
},
{
skipDoc: true,
expression: `[1,2][]`,
expected: []string{
"D0, P[0], (!!int)::1\n",
"D0, P[1], (!!int)::2\n",
},
},
{
document: `a: {b: [1,2,3]}`,
expression: `[.a.b.[]]`,
+38
View File
@@ -1,6 +1,7 @@
package yqlib
import (
"container/list"
"fmt"
"strconv"
)
@@ -129,3 +130,40 @@ func compareScalars(context Context, prefs compareTypePref, lhs *CandidateNode,
return false, fmt.Errorf("%v not yet supported for comparison", lhs.Tag)
}
func superlativeByComparison(d *dataTreeNavigator, context Context, prefs compareTypePref) (Context, error) {
fn := compare(prefs)
var results = list.New()
for seq := context.MatchingNodes.Front(); seq != nil; seq = seq.Next() {
splatted, err := splat(context.SingleChildContext(seq.Value.(*CandidateNode)), traversePreferences{})
if err != nil {
return Context{}, err
}
result := splatted.MatchingNodes.Front()
if result != nil {
for el := result.Next(); el != nil; el = el.Next() {
cmp, err := fn(d, context, el.Value.(*CandidateNode), result.Value.(*CandidateNode))
if err != nil {
return Context{}, err
}
if isTruthyNode(cmp) {
result = el
}
}
results.PushBack(result.Value)
}
}
return context.ChildContext(results), nil
}
func minOperator(d *dataTreeNavigator, context Context, _ *ExpressionNode) (Context, error) {
log.Debug(("Min"))
return superlativeByComparison(d, context, compareTypePref{Greater: false})
}
func maxOperator(d *dataTreeNavigator, context Context, _ *ExpressionNode) (Context, error) {
log.Debug(("Max"))
return superlativeByComparison(d, context, compareTypePref{Greater: true})
}
+10
View File
@@ -5,6 +5,16 @@ import (
)
var envOperatorScenarios = []expressionScenario{
{
description: "Read string environment variable",
skipDoc: true,
environmentVariables: map[string]string{"myenv": "[cat,dog]"},
expression: `env(myenv)[]`,
expected: []string{
"D0, P[0], (!!str)::cat\n",
"D0, P[1], (!!str)::dog\n",
},
},
{
description: "Read string environment variable",
environmentVariables: map[string]string{"myenv": "cat meow"},
+9
View File
@@ -14,6 +14,15 @@ var evalOperatorScenarios = []expressionScenario{
"D0, P[a b 1], (!!map)::{name: cat}\n",
},
},
{
description: "eval splat",
skipDoc: true,
document: `{pathExp: '.a.b[] | select(.name == "cat")', a: {b: [{name: dog}, {name: cat}]}}`,
expression: `eval(.pathExp)[]`,
expected: []string{
"D0, P[a b 1 name], (!!str)::cat\n",
},
},
{
description: "Dynamically update a path from an environment variable",
subdescription: "The env variable can be any valid yq expression.",
+10
View File
@@ -13,6 +13,16 @@ var filterOperatorScenarios = []expressionScenario{
"D0, P[], (!!seq)::[1, 2]\n",
},
},
{
description: "Filter array splat",
skipDoc: true,
document: `[1,2,3]`,
expression: `filter(. < 3)[]`,
expected: []string{
"D0, P[0], (!!int)::1\n",
"D0, P[1], (!!int)::2\n",
},
},
{
description: "Filter map values",
document: `{c: {things: cool, frog: yes}, d: {things: hot, frog: false}}`,
+22
View File
@@ -14,6 +14,17 @@ var flattenOperatorScenarios = []expressionScenario{
"D0, P[], (!!seq)::[1, 2, 3]\n",
},
},
{
description: "Flatten splat",
skipDoc: true,
document: `[1, [2], [[3]]]`,
expression: `flatten[]`,
expected: []string{
"D0, P[0], (!!int)::1\n",
"D0, P[0], (!!int)::2\n",
"D0, P[0], (!!int)::3\n",
},
},
{
description: "Flatten with depth of one",
document: `[1, [2], [[3]]]`,
@@ -22,6 +33,17 @@ var flattenOperatorScenarios = []expressionScenario{
"D0, P[], (!!seq)::[1, 2, [3]]\n",
},
},
{
description: "Flatten with depth and splat",
skipDoc: true,
document: `[1, [2], [[3]]]`,
expression: `flatten(1)[]`,
expected: []string{
"D0, P[0], (!!int)::1\n",
"D0, P[0], (!!int)::2\n",
"D0, P[0], (!!seq)::[3]\n",
},
},
{
description: "Flatten empty array",
document: `[[]]`,
+10
View File
@@ -13,6 +13,16 @@ var groupByOperatorScenarios = []expressionScenario{
"D0, P[], (!!seq)::- - {foo: 1, bar: 10}\n - {foo: 1, bar: 1}\n- - {foo: 3, bar: 100}\n",
},
},
{
description: "Group splat",
skipDoc: true,
document: `[{foo: 1, bar: 10}, {foo: 3, bar: 100}, {foo: 1, bar: 1}]`,
expression: `group_by(.foo)[]`,
expected: []string{
"D0, P[0], (!!seq)::- {foo: 1, bar: 10}\n- {foo: 1, bar: 1}\n",
"D0, P[1], (!!seq)::- {foo: 3, bar: 100}\n",
},
},
{
description: "Group by field, with nulls",
document: `[{cat: dog}, {foo: 1, bar: 10}, {foo: 3, bar: 100}, {no: foo for you}, {foo: 1, bar: 1}]`,
+10
View File
@@ -39,6 +39,16 @@ var keysOperatorScenarios = []expressionScenario{
"D0, P[], (!!seq)::- dog\n- cat\n",
},
},
{
description: "Map keys with splat",
skipDoc: true,
document: `{dog: woof, cat: meow}`,
expression: `keys[]`,
expected: []string{
"D0, P[dog], (!!str)::dog\n",
"D0, P[cat], (!!str)::cat\n",
},
},
{
skipDoc: true,
document: `{}`,
+19
View File
@@ -15,6 +15,15 @@ var mapOperatorScenarios = []expressionScenario{
"D0, P[], (!!seq)::[6, 7, 8]\n",
},
},
{
skipDoc: true,
document: `[1,2]`,
expression: `map(. + 1)[]`,
expected: []string{
"D0, P[0], (!!int)::2\n",
"D0, P[1], (!!int)::3\n",
},
},
{
description: "Map array",
document: `[1,2,3]`,
@@ -33,6 +42,16 @@ var mapOperatorScenarios = []expressionScenario{
"D0, P[], (!!map)::{x: 11, y: 21, z: 31}\n",
},
},
{
description: "map values splat",
skipDoc: true,
document: `{a: 1, b: 2}`,
expression: `map_values(. + 1)[]`,
expected: []string{
"D0, P[a], (!!int)::2\n",
"D0, P[b], (!!int)::3\n",
},
},
{
description: "Map object values",
document: `{a: 1, b: 2, c: 3}`,
+24
View File
@@ -91,6 +91,8 @@ func multiplyScalars(lhs *CandidateNode, rhs *CandidateNode) (*CandidateNode, er
return multiplyIntegers(lhs, rhs)
} else if (lhsTag == "!!int" || lhsTag == "!!float") && (rhsTag == "!!int" || rhsTag == "!!float") {
return multiplyFloats(lhs, rhs, lhsIsCustom)
} else if (lhsTag == "!!str" && rhsTag == "!!int") || (lhsTag == "!!int" && rhsTag == "!!str") {
return repeatString(lhs, rhs)
}
return nil, fmt.Errorf("cannot multiply %v with %v", lhs.Tag, rhs.Tag)
}
@@ -135,6 +137,28 @@ func multiplyIntegers(lhs *CandidateNode, rhs *CandidateNode) (*CandidateNode, e
return target, nil
}
func repeatString(lhs *CandidateNode, rhs *CandidateNode) (*CandidateNode, error) {
var stringNode *CandidateNode
var intNode *CandidateNode
if lhs.Tag == "!!str" {
stringNode = lhs
intNode = rhs
} else {
stringNode = rhs
intNode = lhs
}
target := lhs.CopyWithoutContent()
target.UpdateAttributesFrom(stringNode, assignPreferences{})
count, err := parseInt(intNode.Value)
if err != nil {
return nil, err
}
target.Value = strings.Repeat(stringNode.Value, count)
return target, nil
}
func mergeObjects(d *dataTreeNavigator, context Context, lhs *CandidateNode, rhs *CandidateNode, preferences multiplyPreferences) (*CandidateNode, error) {
var results = list.New()
+36
View File
@@ -1,6 +1,8 @@
package yqlib
import (
"fmt"
"strings"
"testing"
)
@@ -174,6 +176,40 @@ var multiplyOperatorScenarios = []expressionScenario{
"D0, P[], (!!map)::a: 12\nb: 4\n",
},
},
{
description: "Multiply string node X int",
document: docNoComments,
expression: ".b * 4",
expected: []string{
fmt.Sprintf("D0, P[b], (!!str)::%s\n", strings.Repeat("banana", 4)),
},
},
{
description: "Multiply int X string node",
document: docNoComments,
expression: "4 * .b",
expected: []string{
fmt.Sprintf("D0, P[], (!!str)::%s\n", strings.Repeat("banana", 4)),
},
},
{
description: "Multiply string X int node",
document: `n: 4
`,
expression: `"banana" * .n`,
expected: []string{
fmt.Sprintf("D0, P[], (!!str)::%s\n", strings.Repeat("banana", 4)),
},
},
{
description: "Multiply int node X string",
document: `n: 4
`,
expression: `.n * "banana"`,
expected: []string{
fmt.Sprintf("D0, P[n], (!!str)::%s\n", strings.Repeat("banana", 4)),
},
},
{
skipDoc: true,
document: doc1,
+73
View File
@@ -0,0 +1,73 @@
package yqlib
import (
"container/list"
"strconv"
)
func omitMap(original *CandidateNode, indices *CandidateNode) *CandidateNode {
filteredContent := make([]*CandidateNode, 0, max(0, len(original.Content)-len(indices.Content)*2))
for index := 0; index < len(original.Content); index += 2 {
pos := findInArray(indices, original.Content[index])
if pos < 0 {
clonedKey := original.Content[index].Copy()
clonedValue := original.Content[index+1].Copy()
filteredContent = append(filteredContent, clonedKey, clonedValue)
}
}
result := original.CopyWithoutContent()
result.AddChildren(filteredContent)
return result
}
func omitSequence(original *CandidateNode, indices *CandidateNode) *CandidateNode {
filteredContent := make([]*CandidateNode, 0, max(0, len(original.Content)-len(indices.Content)))
for index := 0; index < len(original.Content); index++ {
pos := findInArray(indices, createScalarNode(index, strconv.Itoa(index)))
if pos < 0 {
filteredContent = append(filteredContent, original.Content[index].Copy())
}
}
result := original.CopyWithoutContent()
result.AddChildren(filteredContent)
return result
}
func omitOperator(d *dataTreeNavigator, context Context, expressionNode *ExpressionNode) (Context, error) {
log.Debugf("Omit")
contextIndicesToOmit, err := d.GetMatchingNodes(context, expressionNode.RHS)
if err != nil {
return Context{}, err
}
indicesToOmit := &CandidateNode{}
if contextIndicesToOmit.MatchingNodes.Len() > 0 {
indicesToOmit = contextIndicesToOmit.MatchingNodes.Front().Value.(*CandidateNode)
}
if len(indicesToOmit.Content) == 0 {
log.Debugf("No omit indices specified")
return context, nil
}
var results = list.New()
for el := context.MatchingNodes.Front(); el != nil; el = el.Next() {
node := el.Value.(*CandidateNode)
var replacement *CandidateNode
if node.Kind == MappingNode {
replacement = omitMap(node, indicesToOmit)
} else if node.Kind == SequenceNode {
replacement = omitSequence(node, indicesToOmit)
} else {
log.Debugf("Omit from type %v (%v) is noop", node.Tag, node.GetNicePath())
return context, nil
}
replacement.LeadingContent = node.LeadingContent
results.PushBack(replacement)
}
return context.ChildContext(results), nil
}
+70
View File
@@ -0,0 +1,70 @@
package yqlib
import (
"testing"
)
var omitOperatorScenarios = []expressionScenario{
{
description: "Omit keys from map",
subdescription: "Note that non existent keys are skipped.",
document: "myMap: {cat: meow, dog: bark, thing: hamster, hamster: squeak}\n",
expression: `.myMap |= omit(["hamster", "cat", "goat"])`,
expected: []string{
"D0, P[], (!!map)::myMap: {dog: bark, thing: hamster}\n",
},
},
{
description: "Omit splat",
skipDoc: true,
document: "{cat: meow, dog: bark, hamster: squeak}\n",
expression: `omit(["dog"])[]`,
expected: []string{
"D0, P[cat], (!!str)::meow\n",
"D0, P[hamster], (!!str)::squeak\n",
},
},
{
description: "Omit keys from map",
skipDoc: true,
document: "!things myMap: {cat: meow, dog: bark, thing: hamster, hamster: squeak}\n",
expression: `.myMap |= omit(["hamster", "cat", "goat"])`,
expected: []string{
"D0, P[], (!!map)::!things myMap: {dog: bark, thing: hamster}\n",
},
},
{
description: "Omit keys from map with comments",
skipDoc: true,
document: "# abc\nmyMap: {cat: meow, dog: bark, thing: hamster, hamster: squeak}\n# xyz\n",
expression: `.myMap |= omit(["hamster", "cat", "goat"])`,
expected: []string{
"D0, P[], (!!map)::# abc\nmyMap: {dog: bark, thing: hamster}\n# xyz\n",
},
},
{
description: "Omit indices from array",
subdescription: "Note that non existent indices are skipped.",
document: `[cat, leopard, lion]`,
expression: `omit([2, 0, 734, -5])`,
expected: []string{
"D0, P[], (!!seq)::[leopard]\n",
},
},
{
description: "Omit indices from array with comments",
skipDoc: true,
document: "# abc\n[cat, leopard, lion]\n# xyz",
expression: `omit([2, 0, 734, -5])`,
expected: []string{
"D0, P[], (!!seq)::# abc\n[leopard]\n# xyz\n",
},
},
}
func TestOmitOperatorScenarios(t *testing.T) {
for _, tt := range omitOperatorScenarios {
testScenario(t, &tt)
}
documentOperatorScenarios(t, "omit", omitOperatorScenarios)
}
+19
View File
@@ -23,6 +23,16 @@ var pathOperatorScenarios = []expressionScenario{
"D0, P[a b], (!!seq)::- a\n- b\n",
},
},
{
description: "Map path",
document: `{a: {b: cat}}`,
skipDoc: true,
expression: `.a.b | path[]`,
expected: []string{
"D0, P[a b 0], (!!str)::a\n",
"D0, P[a b 1], (!!str)::b\n",
},
},
{
skipDoc: true,
document: `a:
@@ -129,6 +139,15 @@ var pathOperatorScenarios = []expressionScenario{
"D0, P[], (!!map)::a: [frog]\n",
},
},
{
description: "Delete splat",
skipDoc: true,
document: `a: [cat, frog]`,
expression: `delpaths([["a", 0]])[]`,
expected: []string{
"D0, P[a], (!!seq)::[frog]\n",
},
},
{
description: "Delete - wrong parameter",
subdescription: "delpaths does not work with a single path array",
+10
View File
@@ -14,6 +14,16 @@ var pickOperatorScenarios = []expressionScenario{
"D0, P[], (!!map)::myMap: {hamster: squeak, cat: meow}\n",
},
},
{
description: "Pick splat",
skipDoc: true,
document: "{cat: meow, dog: bark, thing: hamster, hamster: squeak}\n",
expression: `pick(["hamster", "cat"])[]`,
expected: []string{
"D0, P[hamster], (!!str)::squeak\n",
"D0, P[cat], (!!str)::meow\n",
},
},
{
description: "Pick keys from map",
skipDoc: true,
+121
View File
@@ -0,0 +1,121 @@
package yqlib
import (
"container/list"
"fmt"
)
func getUniqueElementTag(seq *CandidateNode) (string, error) {
switch l := len(seq.Content); l {
case 0:
return "", nil
default:
result := seq.Content[0].Tag
for i := 1; i < l; i++ {
t := seq.Content[i].Tag
if t != result {
return "", fmt.Errorf("sequence contains elements of %v and %v types", result, t)
}
}
return result, nil
}
}
var nullNodeFactory = func() *CandidateNode { return createScalarNode(nil, "") }
func pad[E any](array []E, length int, factory func() E) []E {
sz := len(array)
if sz >= length {
return array
}
pad := make([]E, length-sz)
for i := 0; i < len(pad); i++ {
pad[i] = factory()
}
return append(array, pad...)
}
func pivotSequences(seq *CandidateNode) *CandidateNode {
sz := len(seq.Content)
if sz == 0 {
return seq
}
m := make(map[int][]*CandidateNode)
for i := 0; i < sz; i++ {
row := seq.Content[i]
for j := 0; j < len(row.Content); j++ {
e := m[j]
if e == nil {
e = make([]*CandidateNode, 0, sz)
}
m[j] = append(pad(e, i, nullNodeFactory), row.Content[j])
}
}
result := CandidateNode{Kind: SequenceNode}
for i := 0; i < len(m); i++ {
e := CandidateNode{Kind: SequenceNode}
e.AddChildren(pad(m[i], sz, nullNodeFactory))
result.AddChild(&e)
}
return &result
}
func pivotMaps(seq *CandidateNode) *CandidateNode {
sz := len(seq.Content)
if sz == 0 {
return &CandidateNode{Kind: MappingNode}
}
m := make(map[string][]*CandidateNode)
keys := make([]string, 0)
for i := 0; i < sz; i++ {
row := seq.Content[i]
for j := 0; j < len(row.Content); j += 2 {
k := row.Content[j].Value
v := row.Content[j+1]
e := m[k]
if e == nil {
keys = append(keys, k)
e = make([]*CandidateNode, 0, sz)
}
m[k] = append(pad(e, i, nullNodeFactory), v)
}
}
result := CandidateNode{Kind: MappingNode}
for _, k := range keys {
pivotRow := CandidateNode{Kind: SequenceNode}
pivotRow.AddChildren(
pad(m[k], sz, nullNodeFactory))
result.AddKeyValueChild(createScalarNode(k, k), &pivotRow)
}
return &result
}
func pivotOperator(_ *dataTreeNavigator, context Context, _ *ExpressionNode) (Context, error) {
log.Debug("Pivot")
results := list.New()
for el := context.MatchingNodes.Front(); el != nil; el = el.Next() {
candidate := el.Value.(*CandidateNode)
if candidate.Tag != "!!seq" {
return Context{}, fmt.Errorf("cannot pivot node of type %v", candidate.Tag)
}
tag, err := getUniqueElementTag(candidate)
if err != nil {
return Context{}, err
}
var pivot *CandidateNode
switch tag {
case "!!seq":
pivot = pivotSequences(candidate)
case "!!map":
pivot = pivotMaps(candidate)
default:
return Context{}, fmt.Errorf("can only pivot elements of !!seq or !!map types, received %v", tag)
}
results.PushBack(pivot)
}
return context.ChildContext(results), nil
}
+57
View File
@@ -0,0 +1,57 @@
package yqlib
import "testing"
var pivotOperatorScenarios = []expressionScenario{
{
description: "Pivot a sequence of sequences",
document: "[[foo, bar, baz], [sis, boom, bah]]\n",
expression: `pivot`,
expected: []string{
"D0, P[], ()::- - foo\n - sis\n- - bar\n - boom\n- - baz\n - bah\n",
},
},
{
description: "Pivot splat",
skipDoc: true,
document: "[[foo, bar], [sis, boom]]\n",
expression: `pivot[]`,
expected: []string{
"D0, P[0], ()::- foo\n- sis\n",
"D0, P[1], ()::- bar\n- boom\n",
},
},
{
description: "Pivot sequence of heterogeneous sequences",
subdescription: `Missing values are "padded" to null.`,
document: "[[foo, bar, baz], [sis, boom, bah, blah]]\n",
expression: `pivot`,
expected: []string{
"D0, P[], ()::- - foo\n - sis\n- - bar\n - boom\n- - baz\n - bah\n- -\n - blah\n",
},
},
{
description: "Pivot sequence of maps",
document: "[{foo: a, bar: b, baz: c}, {foo: x, bar: y, baz: z}]\n",
expression: `pivot`,
expected: []string{
"D0, P[], ()::foo:\n - a\n - x\nbar:\n - b\n - y\nbaz:\n - c\n - z\n",
},
},
{
description: "Pivot sequence of heterogeneous maps",
subdescription: `Missing values are "padded" to null.`,
document: "[{foo: a, bar: b, baz: c}, {foo: x, bar: y, baz: z, what: ever}]\n",
expression: `pivot`,
expected: []string{
"D0, P[], ()::foo:\n - a\n - x\nbar:\n - b\n - y\nbaz:\n - c\n - z\nwhat:\n -\n - ever\n",
},
},
}
func TestPivotOperatorScenarios(t *testing.T) {
for _, tt := range pivotOperatorScenarios {
testScenario(t, &tt)
}
documentOperatorScenarios(t, "pivot", pivotOperatorScenarios)
}
+10
View File
@@ -11,6 +11,16 @@ var reverseOperatorScenarios = []expressionScenario{
"D0, P[], (!!seq)::[3, 2, 1]\n",
},
},
{
description: "Reverse",
skipDoc: true,
document: "[1, 2]",
expression: `reverse[]`,
expected: []string{
"D0, P[1], (!!int)::2\n",
"D0, P[0], (!!int)::1\n",
},
},
{
skipDoc: true,
document: "[]",
+11
View File
@@ -75,6 +75,17 @@ var selectOperatorScenarios = []expressionScenario{
"D0, P[], (!!map)::b: world\n",
},
},
{
description: "select splat",
skipDoc: true,
document: "a: hello",
document2: "b: world",
expression: `select(.a == "hello" or .b == "world")[]`,
expected: []string{
"D0, P[a], (!!str)::hello\n",
"D0, P[b], (!!str)::world\n",
},
},
{
description: "select does not update the map",
skipDoc: true,
+11
View File
@@ -11,6 +11,17 @@ var shuffleOperatorScenarios = []expressionScenario{
"D0, P[], (!!seq)::[5, 2, 4, 1, 3]\n",
},
},
{
description: "Shuffle array",
skipDoc: true,
document: "[1, 2, 3]",
expression: `shuffle[]`,
expected: []string{
"D0, P[2], (!!int)::3\n",
"D0, P[0], (!!int)::1\n",
"D0, P[1], (!!int)::2\n",
},
},
{
description: "Shuffle array in place",
+10
View File
@@ -13,6 +13,16 @@ var sortKeysOperatorScenarios = []expressionScenario{
"D0, P[], (!!map)::{a: blah, b: bing, c: frog}\n",
},
},
{
description: "Sort keys of map",
skipDoc: true,
document: `{c: frog, a: zoo}`,
expression: `sort_keys(.)[]`,
expected: []string{
"D0, P[a], (!!str)::zoo\n",
"D0, P[c], (!!str)::frog\n",
},
},
{
skipDoc: true,
document: `{c: frog}`,
+20
View File
@@ -11,6 +11,16 @@ var sortByOperatorScenarios = []expressionScenario{
"D0, P[], (!!seq)::[{a: apple}, {a: banana}, {a: cat}]\n",
},
},
{
description: "Sort by string field",
skipDoc: true,
document: "[{a: banana},{a: apple}]",
expression: `sort_by(.a)[]`,
expected: []string{
"D0, P[1], (!!map)::{a: apple}\n",
"D0, P[0], (!!map)::{a: banana}\n",
},
},
{
description: "Sort by multiple fields",
document: "[{a: dog},{a: cat, b: banana},{a: cat, b: apple}]",
@@ -104,6 +114,16 @@ var sortByOperatorScenarios = []expressionScenario{
"D0, P[], (!!seq)::[null, false, true, 3, 6, 8, cat]\n",
},
},
{
description: "Sort, nulls come first",
skipDoc: true,
document: "[8,null]",
expression: `sort[]`,
expected: []string{
"D0, P[1], (!!null)::null\n",
"D0, P[0], (!!int)::8\n",
},
},
{
skipDoc: true,
description: "false before true",
+10
View File
@@ -22,6 +22,16 @@ var splitDocOperatorScenarios = []expressionScenario{
"D1, P[1], (!!map)::{b: dog}\n",
},
},
{
description: "Split splat",
skipDoc: true,
document: `[{a: cat}, {b: dog}]`,
expression: `.[] | split_doc[]`,
expected: []string{
"D0, P[0 a], (!!str)::cat\n",
"D1, P[1 b], (!!str)::dog\n",
},
},
}
func TestSplitDocOperatorScenarios(t *testing.T) {
+39 -26
View File
@@ -54,43 +54,56 @@ func interpolate(d *dataTreeNavigator, context Context, str string) (string, err
for i := 0; i < len(runes); i++ {
char := runes[i]
if !inExpression {
if char == '\\' && i != len(runes)-1 && runes[i+1] == '(' {
inExpression = true
i = i + 1 // skip over the next open bracket
continue
if char == '\\' && i < len(runes)-1 {
switch runes[i+1] {
case '(':
inExpression = true
// skip the lparen
i++
continue
case '\\':
// skip the escaped backslash
i++
default:
log.Debugf("Ignoring non-escaping backslash @ %v[%d]", str, i)
}
}
sb.WriteRune(char)
} else { // we are in an expression
if char == ')' && nestedBracketsCounter == 0 {
// finished the expression!
log.Debugf("Expression is :%v", expSb.String())
value, err := evaluate(d, context, expSb.String())
if err != nil {
return "", err
if char == ')' {
if nestedBracketsCounter == 0 {
// finished the expression!
log.Debugf("Expression is :%v", expSb.String())
value, err := evaluate(d, context, expSb.String())
if err != nil {
return "", err
}
inExpression = false
expSb = strings.Builder{} // reset this
sb.WriteString(value)
continue
}
inExpression = false
expSb = strings.Builder{} // reset this
sb.WriteString(value)
continue
nestedBracketsCounter--
} else if char == '(' {
nestedBracketsCounter++
} else if char == '\\' && i != len(runes)-1 && runes[i+1] == ')' {
// close brackets is escaped, skip over it
expSb.WriteRune(char)
expSb.WriteRune(runes[i+1])
i = i + 1
continue
} else if char == ')' {
nestedBracketsCounter--
} else if char == '\\' && i < len(runes)-1 {
switch esc := runes[i+1]; esc {
case ')', '\\':
// write escaped character
expSb.WriteRune(esc)
i++
continue
default:
log.Debugf("Ignoring non-escaping backslash @ %v[%d]", str, i)
}
}
expSb.WriteRune(char)
}
}
if inExpression {
return "", fmt.Errorf("unclosed interpolation string \\(")
log.Warning("unclosed interpolation string, skipping interpolation")
return str, nil
}
return sb.String(), nil
}
+36 -6
View File
@@ -26,7 +26,7 @@ var stringsOperatorScenarios = []expressionScenario{
description: "Interpolation - just escape",
expression: `"\\"`,
expected: []string{
"D0, P[], (!!str)::\\\\\n",
"D0, P[], (!!str)::\\\n",
},
},
{
@@ -47,6 +47,15 @@ var stringsOperatorScenarios = []expressionScenario{
"D0, P[], (!!str)::Hi (.value)\n",
},
},
{
skipDoc: true,
description: "Interpolation - don't!",
document: `value: things`,
expression: `"Hi \\(.value)"`,
expected: []string{
"D0, P[], (!!str)::Hi \\(.value)\n",
},
},
{
skipDoc: true,
description: "Interpolation - random close bracket",
@@ -57,11 +66,22 @@ var stringsOperatorScenarios = []expressionScenario{
},
},
{
skipDoc: true,
description: "Interpolation - unclosed interpolation string",
document: `value: things`,
expression: `"Hi \("`,
expectedError: "unclosed interpolation string \\(",
skipDoc: true,
description: "Interpolation - unclosed interpolation string",
document: `value: things`,
expression: `"Hi \("`,
expected: []string{
"D0, P[], (!!str)::Hi \\(\n",
},
},
{
skipDoc: true,
description: "Interpolation - unclosed interpolation string due to escape",
document: `value: things`,
expression: `"Hi \(\)"`,
expected: []string{
"D0, P[], (!!str)::Hi \\(\\)\n",
},
},
{
description: "To up (upper) case",
@@ -304,6 +324,16 @@ var stringsOperatorScenarios = []expressionScenario{
"D0, P[], (!!seq)::- word\n",
},
},
{
description: "Split splat",
skipDoc: true,
document: `"word; cat"`,
expression: `split("; ")[]`,
expected: []string{
"D0, P[0], (!!str)::word\n",
"D0, P[1], (!!str)::cat\n",
},
},
{
skipDoc: true,
document: `!horse "word"`,
+19 -7
View File
@@ -23,7 +23,7 @@ func uniqueBy(d *dataTreeNavigator, context Context, expressionNode *ExpressionN
candidate := el.Value.(*CandidateNode)
if candidate.Kind != SequenceNode {
return Context{}, fmt.Errorf("Only arrays are supported for unique")
return Context{}, fmt.Errorf("only arrays are supported for unique")
}
var newMatches = orderedmap.NewOrderedMap()
@@ -34,12 +34,9 @@ func uniqueBy(d *dataTreeNavigator, context Context, expressionNode *ExpressionN
return Context{}, err
}
keyValue := "null"
if rhs.MatchingNodes.Len() > 0 {
first := rhs.MatchingNodes.Front()
keyCandidate := first.Value.(*CandidateNode)
keyValue = keyCandidate.Value
keyValue, err := getUniqueKeyValue(rhs)
if err != nil {
return Context{}, err
}
_, exists := newMatches.Get(keyValue)
@@ -59,3 +56,18 @@ func uniqueBy(d *dataTreeNavigator, context Context, expressionNode *ExpressionN
return context.ChildContext(results), nil
}
func getUniqueKeyValue(rhs Context) (string, error) {
keyValue := "null"
var err error
if rhs.MatchingNodes.Len() > 0 {
first := rhs.MatchingNodes.Front()
keyCandidate := first.Value.(*CandidateNode)
keyValue = keyCandidate.Value
if keyCandidate.Kind != ScalarNode {
keyValue, err = encodeToString(keyCandidate, encoderPreferences{YamlFormat, 0})
}
}
return keyValue, err
}
+37 -1
View File
@@ -14,6 +14,16 @@ var uniqueOperatorScenarios = []expressionScenario{
"D0, P[], (!!seq)::[2, 1, 3]\n",
},
},
{
description: "Unique splat",
skipDoc: true,
document: `[2,1,2]`,
expression: `unique[]`,
expected: []string{
"D0, P[0], (!!int)::2\n",
"D0, P[1], (!!int)::1\n",
},
},
{
description: "Unique nulls",
subdescription: "Unique works on the node value, so it considers different representations of nulls to be different",
@@ -33,13 +43,29 @@ var uniqueOperatorScenarios = []expressionScenario{
},
},
{
description: "Unique array object fields",
description: "Unique array objects",
document: `[{name: harry, pet: cat}, {name: billy, pet: dog}, {name: harry, pet: cat}]`,
expression: `unique`,
expected: []string{
"D0, P[], (!!seq)::[{name: harry, pet: cat}, {name: billy, pet: dog}]\n",
},
},
{
description: "Unique array of objects by a field",
document: `[{name: harry, pet: cat}, {name: billy, pet: dog}, {name: harry, pet: dog}]`,
expression: `unique_by(.name)`,
expected: []string{
"D0, P[], (!!seq)::[{name: harry, pet: cat}, {name: billy, pet: dog}]\n",
},
},
{
description: "Unique array of arrays",
document: `[[cat,dog], [cat, sheep], [cat,dog]]`,
expression: `unique`,
expected: []string{
"D0, P[], (!!seq)::[[cat, dog], [cat, sheep]]\n",
},
},
{
skipDoc: true,
document: `[{name: harry, pet: cat}, {pet: fish}, {name: harry, pet: dog}]`,
@@ -48,6 +74,16 @@ var uniqueOperatorScenarios = []expressionScenario{
"D0, P[], (!!seq)::[{name: harry, pet: cat}, {pet: fish}]\n",
},
},
{
description: "unique by splat",
skipDoc: true,
document: `[{name: harry, pet: cat}, {pet: fish}, {name: harry, pet: dog}]`,
expression: `unique_by(.name)[]`,
expected: []string{
"D0, P[0], (!!map)::{name: harry, pet: cat}\n",
"D0, P[1], (!!map)::{pet: fish}\n",
},
},
{
skipDoc: true,
document: `[{name: harry, pet: cat}, {pet: fish}, {name: harry, pet: dog}]`,
+9
View File
@@ -11,6 +11,15 @@ var withOperatorScenarios = []expressionScenario{
"D0, P[], (!!map)::a: {deeply: {nested: 'newValue'}}\n",
},
},
{
description: "with splat",
skipDoc: true,
document: `a: {deeply: {nested: value}}`,
expression: `with(.a.deeply.nested; . = "newValue" | . style="single")[]`,
expected: []string{
"D0, P[a], (!!map)::{deeply: {nested: 'newValue'}}\n",
},
},
{
description: "Update multiple deeply nested properties",
document: `a: {deeply: {nested: value, other: thing}}`,
+64
View File
@@ -383,3 +383,67 @@ func TestCompareOperatorScenarios(t *testing.T) {
}
documentOperatorScenarios(t, "compare", compareOperatorScenarios)
}
var minOperatorScenarios = []expressionScenario{
{
description: "Minimum int",
document: "[99, 16, 12, 6, 66]\n",
expression: `min`,
expected: []string{
"D0, P[3], (!!int)::6\n",
},
},
{
description: "Minimum string",
document: "[foo, bar, baz]\n",
expression: `min`,
expected: []string{
"D0, P[1], (!!str)::bar\n",
},
},
{
description: "Minimum of empty",
document: "[]\n",
expression: `min`,
expected: []string{},
},
}
func TestMinOperatorScenarios(t *testing.T) {
for _, tt := range minOperatorScenarios {
testScenario(t, &tt)
}
documentOperatorScenarios(t, "min", minOperatorScenarios)
}
var maxOperatorScenarios = []expressionScenario{
{
description: "Maximum int",
document: "[99, 16, 12, 6, 66]\n",
expression: `max`,
expected: []string{
"D0, P[0], (!!int)::99\n",
},
},
{
description: "Maximum string",
document: "[foo, bar, baz]\n",
expression: `max`,
expected: []string{
"D0, P[0], (!!str)::foo\n",
},
},
{
description: "Maximum of empty",
document: "[]\n",
expression: `max`,
expected: []string{},
},
}
func TestMaxOperatorScenarios(t *testing.T) {
for _, tt := range maxOperatorScenarios {
testScenario(t, &tt)
}
documentOperatorScenarios(t, "max", maxOperatorScenarios)
}
+8 -1
View File
@@ -227,7 +227,14 @@ var propertyScenarios = []formatScenario{
expected: "- person\n- name\n",
scenarioType: "decode",
},
{
description: "Decode properties: numbers",
subdescription: "All values are assumed to be strings when parsing properties, but you can use the `from_yaml` operator on all the strings values to autoparse into the correct type.",
input: "a.b = 10",
expression: " (.. | select(tag == \"!!str\")) |= from_yaml",
expected: "a:\n b: 10\n",
scenarioType: "decode",
},
{
description: "Decode properties - array should be a map",
subdescription: "If you have a numeric map key in your property files, use array_to_map to convert them to maps.",
+27 -1
View File
@@ -1,10 +1,36 @@
4.43.1: (not released yet)
4.44.3:
- Fixed upper-case file extension detection, Thanks @ryenus (#2121)
- Log printing follow no-colors flag #2082
- Skip and warn when interpolating strings and theres a unclosed bracket #2083
- Fixed CSV content starting with # issue #2076
- Bumped dependencies
4.44.2:
- Handle numbers with underscores #2039
- Unique now works on maps and arrays #2068
- Added support for short hand splat with env[] expression #2071,
as well as many other operators (split,select,eval,pick..)
- Bumped dependencies
4.44.1:
- Added min/max operators (#1992) Thanks @mbenson
- Added pivot oeprator (#1993) Thanks @mbenson
- Fix: shell-completion (#2006) Thanks @codekow
- Handle escaped backslashes (#1997) Thanks @mbenson
- Fix npe when given filename ending with "." (#1994)
- Fix: linux (w/ selinux) build (#2004) Thanks @codekow
- Bumped dependencies
4.43.1:
- Added omit operator #1989 thanks @mbenson!
- Added tostring #72
- Added string interpolation #1149
- Can specify parent(n) levels #1970
- Can now multiply strings by numbers #1988 thanks @mbenson!
- Fixed CSV line break issue #1974
- Adding a EvaluateAll function to StringEvaluator #1966
- yqlib, default to colors off when using yaml library #1964
- Removed JSON output warning
- Bumped dependencies
+4 -2
View File
@@ -1,5 +1,5 @@
name: yq
version: 'v4.42.1'
version: 'v4.44.3'
summary: A lightweight and portable command-line data file processor
description: |
`yq` uses [jq](https://github.com/stedolan/jq) like syntax but works with yaml, json, xml, csv, properties and TOML files.
@@ -20,7 +20,9 @@ apps:
parts:
yq:
plugin: go
build-environment:
- CGO_ENABLED: 0
source: https://github.com/mikefarah/yq.git
source-tag: v4.42.1
source-tag: v4.44.3
build-snaps:
- go/latest/stable