mirror of
https://github.com/mikefarah/yq.git
synced 2026-08-24 16:39:58 +08:00
Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Mike Farah
|
d8f8661567 | ||
|
Mike Farah
|
9628aaf8c5 | ||
|
Mike Farah
|
cb0bae128f | ||
|
Mike Farah
|
183a42c249 |
@@ -1,63 +0,0 @@
|
||||
name: Release Github action Docker
|
||||
|
||||
on:
|
||||
release:
|
||||
types: [released]
|
||||
|
||||
# Allows you to run this workflow manually from the Actions tab
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
publishDocker:
|
||||
environment: dockerhub
|
||||
env:
|
||||
IMAGE_NAME: mikefarah/yq
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
|
||||
- name: Set up QEMU
|
||||
uses: docker/setup-qemu-action@v2
|
||||
with:
|
||||
platforms: all
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
id: buildx
|
||||
uses: docker/setup-buildx-action@v2
|
||||
with:
|
||||
version: latest
|
||||
|
||||
- name: Available platforms
|
||||
run: echo ${{ steps.buildx.outputs.platforms }} && docker version
|
||||
|
||||
- name: Build and push image
|
||||
run: |
|
||||
echo "GithubRef: ${GITHUB_REF}"
|
||||
VERSION=${GITHUB_REF##*/}
|
||||
echo "VERSION: ${VERSION}"
|
||||
IMAGE_VERSION=${VERSION:1}
|
||||
echo "IMAGE_VERSION: ${IMAGE_VERSION}"
|
||||
|
||||
PLATFORMS="linux/amd64,linux/ppc64le,linux/arm64"
|
||||
|
||||
echo "Building and pushing version ${IMAGE_VERSION} of image ${IMAGE_NAME}"
|
||||
echo '${{ secrets.DOCKER_PASSWORD }}' | docker login -u '${{ secrets.DOCKER_USERNAME }}' --password-stdin
|
||||
cd github-action
|
||||
docker buildx build \
|
||||
--label "org.opencontainers.image.authors=https://github.com/mikefarah/yq/graphs/contributors" \
|
||||
--label "org.opencontainers.image.created=$(date --rfc-3339=seconds)" \
|
||||
--label "org.opencontainers.image.description=yq is a portable command-line YAML processor" \
|
||||
--label "org.opencontainers.image.documentation=https://mikefarah.gitbook.io/yq/" \
|
||||
--label "org.opencontainers.image.licenses=MIT" \
|
||||
--label "org.opencontainers.image.revision=$(git rev-parse HEAD)" \
|
||||
--label "org.opencontainers.image.source=https://github.com/mikefarah/yq" \
|
||||
--label "org.opencontainers.image.title=yq" \
|
||||
--label "org.opencontainers.image.url=https://mikefarah.gitbook.io/yq/" \
|
||||
--label "org.opencontainers.image.version=${IMAGE_VERSION}" \
|
||||
--platform "${PLATFORMS}" \
|
||||
--pull \
|
||||
--push \
|
||||
-t "${IMAGE_NAME}:${IMAGE_VERSION}-githubaction" \
|
||||
-t "${IMAGE_NAME}:4-githubaction" \
|
||||
-t "${IMAGE_NAME}:latest-githubaction" \
|
||||
.
|
||||
@@ -10,10 +10,10 @@ jobs:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
|
||||
- name: Set up Go 1.19
|
||||
- name: Set up Go 1.18
|
||||
uses: actions/setup-go@v3
|
||||
with:
|
||||
go-version: 1.19
|
||||
go-version: 1.18
|
||||
id: go
|
||||
|
||||
- name: Check out code into the Go module directory
|
||||
|
||||
@@ -12,7 +12,7 @@ jobs:
|
||||
- uses: actions/checkout@v3
|
||||
- uses: actions/setup-go@v3
|
||||
with:
|
||||
go-version: '^1.19'
|
||||
go-version: '^1.18'
|
||||
- name: Compile man page markup
|
||||
id: gen-man-page-md
|
||||
run: |
|
||||
|
||||
@@ -12,14 +12,11 @@ jobs:
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
- name: Get test
|
||||
id: get_value
|
||||
uses: mikefarah/yq@master
|
||||
with:
|
||||
cmd: yq '.test' examples/multiline-text.yaml
|
||||
- name: Multiline test
|
||||
run: echo "### It was [${{ steps.get_value.outputs.result }}]" >> $GITHUB_STEP_SUMMARY
|
||||
cmd: yq eval '.a' examples/sample.yaml
|
||||
- name: Write inplace test
|
||||
id: lookupSdkVersion
|
||||
uses: mikefarah/yq@master
|
||||
with:
|
||||
cmd: yq -i '.b.c = 5' examples/sample.yaml
|
||||
cmd: yq eval -i '.b.c = 5' examples/sample.yaml
|
||||
+1
-6
@@ -56,9 +56,4 @@ _build
|
||||
debian/files
|
||||
|
||||
# intellij
|
||||
/.idea
|
||||
|
||||
# vscode
|
||||
.vscode
|
||||
|
||||
yq3
|
||||
/.idea
|
||||
+1
-1
@@ -1,4 +1,4 @@
|
||||
FROM golang:1.19.2 as builder
|
||||
FROM golang:1.18.4 as builder
|
||||
|
||||
WORKDIR /go/src/mikefarah/yq
|
||||
|
||||
|
||||
+1
-1
@@ -1,4 +1,4 @@
|
||||
FROM golang:1.19.2
|
||||
FROM golang:1.18.4
|
||||
|
||||
COPY scripts/devtools.sh /opt/devtools.sh
|
||||
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
    
|
||||
|
||||
|
||||
a lightweight and portable command-line YAML, JSON and XML processor. `yq` uses [jq](https://github.com/stedolan/jq) like syntax but works with yaml files as well as json, xml, properties, csv and tsv. It doesn't yet support everything `jq` does - but it does support the most common operations and functions, and more is being added continuously.
|
||||
a lightweight and portable command-line YAML, JSON and XML processor. `yq` uses [jq](https://github.com/stedolan/jq) like syntax but works with yaml files as well as json and xml. It doesn't yet support everything `jq` does - but it does support the most common operations and functions, and more is being added continuously.
|
||||
|
||||
yq is written in go - so you can download a dependency free binary for your platform and you are good to go! If you prefer there are a variety of package managers that can be used as well as Docker and Podman, all listed below.
|
||||
|
||||
@@ -73,8 +73,6 @@ yq -P sample.json
|
||||
|
||||
See the [documentation](https://mikefarah.gitbook.io/yq/) for more examples.
|
||||
|
||||
Take a look at the discussions for [common questions](https://github.com/mikefarah/yq/discussions/categories/q-a), and [cool ideas](https://github.com/mikefarah/yq/discussions/categories/show-and-tell)
|
||||
|
||||
## Install
|
||||
|
||||
### [Download the latest binary](https://github.com/mikefarah/yq/releases/latest)
|
||||
@@ -209,27 +207,12 @@ RUN apk add --no-cache tzdata
|
||||
USER yq
|
||||
```
|
||||
|
||||
#### Podman with SELinux
|
||||
|
||||
If you are using podman with SELinux, you will need to set the shared volume flag `:z` on the volume mount:
|
||||
|
||||
```
|
||||
-v "${PWD}":/workdir:z
|
||||
```
|
||||
|
||||
### GitHub Action
|
||||
```
|
||||
- name: Set foobar to cool
|
||||
uses: mikefarah/yq@master
|
||||
with:
|
||||
cmd: yq -i '.foo.bar = "cool"' 'config.yml'
|
||||
- name: Get an entry with a variable that might contain dots or spaces
|
||||
id: get_username
|
||||
uses: mikefarah/yq@master
|
||||
with:
|
||||
cmd: yq '.all.children.["${{ matrix.ip_address }}"].username' ops/inventories/production.yml
|
||||
- name: Reuse a variable obtained in another step
|
||||
run: echo ${{ steps.get_username.outputs.result }}
|
||||
```
|
||||
|
||||
See https://mikefarah.gitbook.io/yq/usage/github-action for more.
|
||||
@@ -265,13 +248,6 @@ choco install yq
|
||||
```
|
||||
Supported by @chillum (https://chocolatey.org/packages/yq)
|
||||
|
||||
and
|
||||
|
||||
### Winget
|
||||
winget install yq
|
||||
|
||||
https://winget.run/pkg/MikeFarah/yq
|
||||
|
||||
### Mac:
|
||||
Using [MacPorts](https://www.macports.org/)
|
||||
```
|
||||
@@ -314,10 +290,10 @@ Supported by @rmescandon (https://launchpad.net/~rmescandon/+archive/ubuntu/yq)
|
||||
- Keeps yaml formatting and comments when updating (though there are issues with whitespace)
|
||||
- [Decode/Encode base64 data](https://mikefarah.gitbook.io/yq/operators/encode-decode)
|
||||
- [Load content from other files](https://mikefarah.gitbook.io/yq/operators/load)
|
||||
- [Convert to/from json/ndjson](https://mikefarah.gitbook.io/yq/v/v4.x/usage/convert)
|
||||
- [Convert to/from json](https://mikefarah.gitbook.io/yq/v/v4.x/usage/convert)
|
||||
- [Convert to/from xml](https://mikefarah.gitbook.io/yq/v/v4.x/usage/xml)
|
||||
- [Convert to/from properties](https://mikefarah.gitbook.io/yq/v/v4.x/usage/properties)
|
||||
- [Convert to/from csv/tsv](https://mikefarah.gitbook.io/yq/usage/csv-tsv)
|
||||
- [Convert to csv/tsv](https://mikefarah.gitbook.io/yq/usage/csv-tsv)
|
||||
- [General shell completion scripts (bash/zsh/fish/powershell)](https://mikefarah.gitbook.io/yq/v/v4.x/commands/shell-completion)
|
||||
- [Reduce](https://mikefarah.gitbook.io/yq/operators/reduce) to merge multiple files or sum an array or other fancy things.
|
||||
- [Github Action](https://mikefarah.gitbook.io/yq/usage/github-action) to use in your automated pipeline (thanks @devorbitus)
|
||||
|
||||
@@ -1,43 +0,0 @@
|
||||
#!/bin/bash
|
||||
|
||||
setUp() {
|
||||
rm test*.yml || true
|
||||
|
||||
}
|
||||
|
||||
testLineCountFirstLineComment() {
|
||||
cat >test.yml <<EOL
|
||||
#test123
|
||||
abc: 123
|
||||
test123: 123123
|
||||
#comment
|
||||
lalilu: lalilu
|
||||
EOL
|
||||
|
||||
X=$(./yq '.lalilu | line' --header-preprocess=false < test.yml)
|
||||
assertEquals "5" "$X"
|
||||
}
|
||||
|
||||
testArrayOfDocs() {
|
||||
cat >test.yml <<EOL
|
||||
---
|
||||
# leading comment doc 1
|
||||
a: 1
|
||||
---
|
||||
# leading comment doc 2
|
||||
a: 2
|
||||
EOL
|
||||
|
||||
read -r -d '' expected << EOM
|
||||
- # leading comment doc 1
|
||||
a: 1
|
||||
- # leading comment doc 2
|
||||
a: 2
|
||||
EOM
|
||||
|
||||
X=$(./yq ea '[.]' --header-preprocess=false < test.yml)
|
||||
assertEquals "$expected" "$X"
|
||||
|
||||
}
|
||||
|
||||
source ./scripts/shunit2
|
||||
@@ -63,20 +63,6 @@ EOM
|
||||
assertEquals "$expected" "$X"
|
||||
}
|
||||
|
||||
testInputCSVUTF8() {
|
||||
read -r -d '' expected << EOM
|
||||
- id: 1
|
||||
first: john
|
||||
last: smith
|
||||
- id: 1
|
||||
first: jane
|
||||
last: smith
|
||||
EOM
|
||||
|
||||
X=$(./yq -p=csv utf8.csv)
|
||||
assertEquals "$expected" "$X"
|
||||
}
|
||||
|
||||
testInputTSV() {
|
||||
cat >test.tsv <<EOL
|
||||
fruit yumLevel
|
||||
|
||||
@@ -1,8 +1,5 @@
|
||||
#!/bin/bash
|
||||
|
||||
|
||||
# examples where header-preprocess is required
|
||||
|
||||
setUp() {
|
||||
rm test*.yml || true
|
||||
cat >test.yml <<EOL
|
||||
|
||||
+1
-1
@@ -12,6 +12,6 @@ outputs:
|
||||
description: "The complete result from the yq command being run"
|
||||
runs:
|
||||
using: 'docker'
|
||||
image: 'docker://mikefarah/yq:4-githubaction'
|
||||
image: 'github-action/Dockerfile'
|
||||
args:
|
||||
- ${{ inputs.cmd }}
|
||||
|
||||
@@ -42,7 +42,7 @@ $ yq shell-completion fish > ~/.config/fish/completions/yq.fish
|
||||
`,
|
||||
DisableFlagsInUseLine: true,
|
||||
ValidArgs: []string{"bash", "zsh", "fish", "powershell"},
|
||||
Args: cobra.MatchAll(cobra.ExactArgs(1), cobra.OnlyValidArgs),
|
||||
Args: cobra.ExactValidArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
var err error = nil
|
||||
switch args[0] {
|
||||
|
||||
+1
-1
@@ -11,7 +11,7 @@ var (
|
||||
GitDescribe string
|
||||
|
||||
// Version is main version number that is being run at the moment.
|
||||
Version = "4.28.1"
|
||||
Version = "4.26.1"
|
||||
|
||||
// VersionPrerelease is a pre-release marker for the version. If this is "" (empty string)
|
||||
// then it means that it is a final release. Otherwise, this is a pre-release
|
||||
|
||||
+12
-5
@@ -1,5 +1,12 @@
|
||||
volumes:
|
||||
beehive-conf:
|
||||
driver: local
|
||||
services:
|
||||
- beehive
|
||||
# --------------------------------------------------
|
||||
# It's a test with comment
|
||||
# --------------------------------------------------
|
||||
test:
|
||||
- name: a
|
||||
- name: b
|
||||
- name: c
|
||||
groups:
|
||||
- name: a
|
||||
- name: b
|
||||
- name: c
|
||||
- name: c
|
||||
@@ -2,8 +2,7 @@ FROM mikefarah/yq:4
|
||||
|
||||
COPY entrypoint.sh /entrypoint.sh
|
||||
|
||||
# github action recommendation is to run as root.
|
||||
# https://docs.github.com/en/actions/creating-actions/dockerfile-support-for-github-actions#user
|
||||
USER root
|
||||
# this seems to be the default user in github actions
|
||||
USER 1001
|
||||
|
||||
ENTRYPOINT ["/entrypoint.sh"]
|
||||
|
||||
@@ -2,11 +2,8 @@
|
||||
set -e
|
||||
echo "::debug::\$cmd: $1"
|
||||
RESULT=$(eval "$1")
|
||||
RESULT="${RESULT//'%'/'%25'}"
|
||||
RESULT="${RESULT//$'\n'/'%0A'}"
|
||||
RESULT="${RESULT//$'\r'/'%0D'}"
|
||||
echo "::debug::\$RESULT: $RESULT"
|
||||
# updating from
|
||||
# https://github.com/orgs/community/discussions/26288#discussioncomment-3876281
|
||||
# https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#setting-an-output-parameter
|
||||
delimiter=$(cat /proc/sys/kernel/random/uuid)
|
||||
echo "result<<${delimiter}" >> "${GITHUB_OUTPUT}"
|
||||
echo "${RESULT}" >> "${GITHUB_OUTPUT}"
|
||||
echo "${delimiter}" >> "${GITHUB_OUTPUT}"
|
||||
echo ::set-output name=result::"$RESULT"
|
||||
|
||||
@@ -2,24 +2,23 @@ module github.com/mikefarah/yq/v4
|
||||
|
||||
require (
|
||||
github.com/a8m/envsubst v1.3.0
|
||||
github.com/alecthomas/participle/v2 v2.0.0-beta.5
|
||||
github.com/alecthomas/repr v0.1.1
|
||||
github.com/dimchansky/utfbom v1.1.1
|
||||
github.com/elliotchance/orderedmap v1.5.0
|
||||
github.com/alecthomas/participle/v2 v2.0.0-beta.4
|
||||
github.com/alecthomas/repr v0.1.0
|
||||
github.com/elliotchance/orderedmap v1.4.0
|
||||
github.com/fatih/color v1.13.0
|
||||
github.com/goccy/go-json v0.9.11
|
||||
github.com/goccy/go-json v0.9.10
|
||||
github.com/goccy/go-yaml v1.9.5
|
||||
github.com/jinzhu/copier v0.3.5
|
||||
github.com/magiconair/properties v1.8.6
|
||||
github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e
|
||||
github.com/spf13/cobra v1.6.0
|
||||
github.com/spf13/cobra v1.5.0
|
||||
golang.org/x/net v0.0.0-20220708220712-1185a9018129
|
||||
gopkg.in/op/go-logging.v1 v1.0.0-20160211212156-b2cb9fa56473
|
||||
gopkg.in/yaml.v3 v3.0.1
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/inconshreveable/mousetrap v1.0.1 // indirect
|
||||
github.com/inconshreveable/mousetrap v1.0.0 // indirect
|
||||
github.com/mattn/go-colorable v0.1.12 // indirect
|
||||
github.com/mattn/go-isatty v0.0.14 // indirect
|
||||
github.com/spf13/pflag v1.0.5 // indirect
|
||||
@@ -28,4 +27,4 @@ require (
|
||||
golang.org/x/xerrors v0.0.0-20220609144429-65e65417b02f // indirect
|
||||
)
|
||||
|
||||
go 1.19
|
||||
go 1.18
|
||||
|
||||
@@ -1,17 +1,15 @@
|
||||
github.com/a8m/envsubst v1.3.0 h1:GmXKmVssap0YtlU3E230W98RWtWCyIZzjtf1apWWyAg=
|
||||
github.com/a8m/envsubst v1.3.0/go.mod h1:MVUTQNGQ3tsjOOtKCNd+fl8RzhsXcDvvAEzkhGtlsbY=
|
||||
github.com/alecthomas/assert/v2 v2.0.3 h1:WKqJODfOiQG0nEJKFKzDIG3E29CN2/4zR9XGJzKIkbg=
|
||||
github.com/alecthomas/participle/v2 v2.0.0-beta.5 h1:y6dsSYVb1G5eK6mgmy+BgI3Mw35a3WghArZ/Hbebrjo=
|
||||
github.com/alecthomas/participle/v2 v2.0.0-beta.5/go.mod h1:RC764t6n4L8D8ITAJv0qdokritYSNR3wV5cVwmIEaMM=
|
||||
github.com/alecthomas/repr v0.1.1 h1:87P60cSmareLAxMc4Hro0r2RBY4ROm0dYwkJNpS4pPs=
|
||||
github.com/alecthomas/repr v0.1.1/go.mod h1:Fr0507jx4eOXV7AlPV6AVZLYrLIuIeSOWtW57eE/O/4=
|
||||
github.com/alecthomas/participle/v2 v2.0.0-beta.4 h1:ublfGBm+x+p2j7KotHhrUMbKtejT7M0Gv1Mt1u3absw=
|
||||
github.com/alecthomas/participle/v2 v2.0.0-beta.4/go.mod h1:RC764t6n4L8D8ITAJv0qdokritYSNR3wV5cVwmIEaMM=
|
||||
github.com/alecthomas/repr v0.1.0 h1:ENn2e1+J3k09gyj2shc0dHr/yjaWSHRlrJ4DPMevDqE=
|
||||
github.com/alecthomas/repr v0.1.0/go.mod h1:2kn6fqh/zIyPLmm3ugklbEi5hg5wS435eygvNfaDQL8=
|
||||
github.com/cpuguy83/go-md2man/v2 v2.0.2/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
|
||||
github.com/davecgh/go-spew v1.1.0 h1:ZDRjVQ15GmhC3fiQ8ni8+OwkZQO4DARzQgrnXU1Liz8=
|
||||
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/dimchansky/utfbom v1.1.1 h1:vV6w1AhK4VMnhBno/TPVCoK9U/LP0PkLCS9tbxHdi/U=
|
||||
github.com/dimchansky/utfbom v1.1.1/go.mod h1:SxdoEBH5qIqFocHMyGOXVAybYJdr71b1Q/j0mACtrfE=
|
||||
github.com/elliotchance/orderedmap v1.5.0 h1:1IsExUsjv5XNBD3ZdC7jkAAqLWOOKdbPTmkHx63OsBg=
|
||||
github.com/elliotchance/orderedmap v1.5.0/go.mod h1:wsDwEaX5jEoyhbs7x93zk2H/qv0zwuhg4inXhDkYqys=
|
||||
github.com/elliotchance/orderedmap v1.4.0 h1:wZtfeEONCbx6in1CZyE6bELEt/vFayMvsxqI5SgsR+A=
|
||||
github.com/elliotchance/orderedmap v1.4.0/go.mod h1:wsDwEaX5jEoyhbs7x93zk2H/qv0zwuhg4inXhDkYqys=
|
||||
github.com/fatih/color v1.10.0/go.mod h1:ELkj/draVOlAH/xkhN6mQ50Qd0MPOk5AAr3maGEBuJM=
|
||||
github.com/fatih/color v1.13.0 h1:8LOYc1KYPPmyKMuN8QV2DNRWNbLo6LZ0iLs8+mlH53w=
|
||||
github.com/fatih/color v1.13.0/go.mod h1:kLAiJbzzSOZDVNGyDpeOxJ47H46qBXwg5ILebYFFOfk=
|
||||
@@ -19,13 +17,13 @@ github.com/go-playground/assert/v2 v2.0.1/go.mod h1:VDjEfimB/XKnb+ZQfWdccd7VUvSc
|
||||
github.com/go-playground/locales v0.13.0/go.mod h1:taPMhCMXrRLJO55olJkUXHZBHCxTMfnGwq/HNwmWNS8=
|
||||
github.com/go-playground/universal-translator v0.17.0/go.mod h1:UkSxE5sNxxRwHyU+Scu5vgOQjsIJAF8j9muTVoKLVtA=
|
||||
github.com/go-playground/validator/v10 v10.4.1/go.mod h1:nlOn6nFhuKACm19sB/8EGNn9GlaMV7XkbRSipzJ0Ii4=
|
||||
github.com/goccy/go-json v0.9.11 h1:/pAaQDLHEoCq/5FFmSKBswWmK6H0e8g4159Kc/X/nqk=
|
||||
github.com/goccy/go-json v0.9.11/go.mod h1:6MelG93GURQebXPDq3khkgXZkazVtN9CRI+MGFi0w8I=
|
||||
github.com/goccy/go-json v0.9.10 h1:hCeNmprSNLB8B8vQKWl6DpuH0t60oEs+TAk9a7CScKc=
|
||||
github.com/goccy/go-json v0.9.10/go.mod h1:6MelG93GURQebXPDq3khkgXZkazVtN9CRI+MGFi0w8I=
|
||||
github.com/goccy/go-yaml v1.9.5 h1:Eh/+3uk9kLxG4koCX6lRMAPS1OaMSAi+FJcya0INdB0=
|
||||
github.com/goccy/go-yaml v1.9.5/go.mod h1:U/jl18uSupI5rdI2jmuCswEA2htH9eXfferR3KfscvA=
|
||||
github.com/hexops/gotextdiff v1.0.3 h1:gitA9+qJrrTCsiCl7+kh75nPqQt1cx4ZkudSTLoUqJM=
|
||||
github.com/inconshreveable/mousetrap v1.0.1 h1:U3uMjPSQEBMNp1lFxmllqCPM6P5u/Xq7Pgzkat/bFNc=
|
||||
github.com/inconshreveable/mousetrap v1.0.1/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
|
||||
github.com/inconshreveable/mousetrap v1.0.0 h1:Z8tu5sraLXCXIcARxBp/8cbvlwVa7Z1NHg9XEKhtSvM=
|
||||
github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
|
||||
github.com/jinzhu/copier v0.3.5 h1:GlvfUwHk62RokgqVNvYsku0TATCF7bAHVwEXoBh3iJg=
|
||||
github.com/jinzhu/copier v0.3.5/go.mod h1:DfbEm0FYsaqBcKcFuvmOZb218JkPGtvSHsKg8S8hyyg=
|
||||
github.com/leodido/go-urn v1.2.0/go.mod h1:+8+nEpDfqqsY+g338gtMEUOtuK+4dEMhiQEgxpxOKII=
|
||||
@@ -43,8 +41,8 @@ github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e/go.mod h1:pJLUxLENpZxwdsK
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
|
||||
github.com/spf13/cobra v1.6.0 h1:42a0n6jwCot1pUmomAp4T7DeMD+20LFv4Q54pxLf2LI=
|
||||
github.com/spf13/cobra v1.6.0/go.mod h1:IOw/AERYS7UzyrGinqmz6HLUo219MORXGxhbaJUqzrY=
|
||||
github.com/spf13/cobra v1.5.0 h1:X+jTBEBqF0bHN+9cSMgmfuvv2VHJ9ezmFNf9Y/XstYU=
|
||||
github.com/spf13/cobra v1.5.0/go.mod h1:dWXEIy2H428czQCjInthrTRUg7yKbok+2Qi/yBIJoUM=
|
||||
github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
|
||||
github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
|
||||
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||
@@ -77,6 +75,7 @@ gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8
|
||||
gopkg.in/op/go-logging.v1 v1.0.0-20160211212156-b2cb9fa56473 h1:6D+BvnJ/j6e222UW8s2qTSe3wGBtvo0MbVQG/c5k8RE=
|
||||
gopkg.in/op/go-logging.v1 v1.0.0-20160211212156-b2cb9fa56473/go.mod h1:N1eN2tsCx0Ydtgjl4cqmbRCsY4/+z4cYDeqwZTk6zog=
|
||||
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
|
||||
gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
|
||||
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
|
||||
+1
-1
@@ -37,7 +37,7 @@ The `=` operator then pipes the 'root' context through the `rhs` expression of `
|
||||
dog
|
||||
```
|
||||
|
||||
Both sides have now been evaluated, so now the operator copies across the value from the RHS (`.b`) to the LHS (`.a`), and it returns the now updated context:
|
||||
Both sides have now been evaluated, so now the operator copies across the value from the RHS (`.b`) to the the LHS (`.a`), and it returns the now updated context:
|
||||
|
||||
```yaml
|
||||
a: dog
|
||||
|
||||
@@ -157,7 +157,7 @@ func (n *CandidateNode) UpdateAttributesFrom(other *CandidateNode, prefs assignP
|
||||
n.Node.Kind = other.Node.Kind
|
||||
|
||||
// don't clobber custom tags...
|
||||
if prefs.ClobberCustomTags || strings.HasPrefix(n.Node.Tag, "!!") || n.Node.Tag == "" {
|
||||
if strings.HasPrefix(n.Node.Tag, "!!") || n.Node.Tag == "" {
|
||||
n.Node.Tag = other.Node.Tag
|
||||
}
|
||||
|
||||
|
||||
@@ -150,7 +150,7 @@ func testCSVScenario(t *testing.T, s formatScenario) {
|
||||
}
|
||||
}
|
||||
|
||||
func documentCSVDecodeObjectScenario(w *bufio.Writer, s formatScenario, formatType string) {
|
||||
func documentCSVDecodeObjectScenario(t *testing.T, w *bufio.Writer, s formatScenario, formatType string) {
|
||||
writeOrPanic(w, fmt.Sprintf("## %v\n", s.description))
|
||||
|
||||
if s.subdescription != "" {
|
||||
@@ -250,9 +250,9 @@ func documentCSVScenario(t *testing.T, w *bufio.Writer, i interface{}) {
|
||||
case "encode-tsv":
|
||||
documentCSVEncodeScenario(w, s, "tsv")
|
||||
case "decode-csv-object":
|
||||
documentCSVDecodeObjectScenario(w, s, "csv")
|
||||
documentCSVDecodeObjectScenario(t, w, s, "csv")
|
||||
case "decode-tsv-object":
|
||||
documentCSVDecodeObjectScenario(w, s, "tsv")
|
||||
documentCSVDecodeObjectScenario(t, w, s, "tsv")
|
||||
case "roundtrip-csv":
|
||||
documentCSVRoundTripScenario(w, s, "csv")
|
||||
|
||||
|
||||
@@ -5,7 +5,6 @@ import (
|
||||
"errors"
|
||||
"io"
|
||||
|
||||
"github.com/dimchansky/utfbom"
|
||||
yaml "gopkg.in/yaml.v3"
|
||||
)
|
||||
|
||||
@@ -20,9 +19,7 @@ func NewCSVObjectDecoder(separator rune) Decoder {
|
||||
}
|
||||
|
||||
func (dec *csvObjectDecoder) Init(reader io.Reader) {
|
||||
cleanReader, enc := utfbom.Skip(reader)
|
||||
log.Debugf("Detected encoding: %s\n", enc)
|
||||
dec.reader = *csv.NewReader(cleanReader)
|
||||
dec.reader = *csv.NewReader(reader)
|
||||
dec.reader.Comma = dec.separator
|
||||
dec.finished = false
|
||||
}
|
||||
|
||||
+25
-22
@@ -22,13 +22,13 @@ func (dec *jsonDecoder) Init(reader io.Reader) {
|
||||
|
||||
func (dec *jsonDecoder) Decode(rootYamlNode *yaml.Node) error {
|
||||
|
||||
var dataBucket orderedMap
|
||||
var dataBucket interface{}
|
||||
log.Debug("going to decode")
|
||||
err := dec.decoder.Decode(&dataBucket)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
node, err := dec.convertToYamlNode(&dataBucket)
|
||||
node, err := dec.convertToYamlNode(dataBucket)
|
||||
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -38,36 +38,39 @@ func (dec *jsonDecoder) Decode(rootYamlNode *yaml.Node) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (dec *jsonDecoder) convertToYamlNode(data *orderedMap) (*yaml.Node, error) {
|
||||
if data.kv == nil {
|
||||
switch rawData := data.altVal.(type) {
|
||||
case nil:
|
||||
return createScalarNode(nil, "null"), nil
|
||||
case float64, float32:
|
||||
// json decoder returns ints as float.
|
||||
return parseSnippet(fmt.Sprintf("%v", rawData))
|
||||
case int, int64, int32, string, bool:
|
||||
return createScalarNode(rawData, fmt.Sprintf("%v", rawData)), nil
|
||||
case []*orderedMap:
|
||||
return dec.parseArray(rawData)
|
||||
default:
|
||||
return nil, fmt.Errorf("unrecognised type :( %v", rawData)
|
||||
}
|
||||
func (dec *jsonDecoder) convertToYamlNode(data interface{}) (*yaml.Node, error) {
|
||||
switch data := data.(type) {
|
||||
case nil:
|
||||
return createScalarNode(nil, "null"), nil
|
||||
case float64, float32:
|
||||
// json decoder returns ints as float.
|
||||
return parseSnippet(fmt.Sprintf("%v", data))
|
||||
case int, int64, int32, string, bool:
|
||||
return createScalarNode(data, fmt.Sprintf("%v", data)), nil
|
||||
case map[string]interface{}:
|
||||
return dec.parseMap(data)
|
||||
case []interface{}:
|
||||
return dec.parseArray(data)
|
||||
default:
|
||||
return nil, fmt.Errorf("unrecognised type :(")
|
||||
}
|
||||
}
|
||||
|
||||
func (dec *jsonDecoder) parseMap(dataMap map[string]interface{}) (*yaml.Node, error) {
|
||||
|
||||
var yamlMap = &yaml.Node{Kind: yaml.MappingNode}
|
||||
for _, keyValuePair := range data.kv {
|
||||
yamlValue, err := dec.convertToYamlNode(&keyValuePair.V)
|
||||
|
||||
for key, value := range dataMap {
|
||||
yamlValue, err := dec.convertToYamlNode(value)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
yamlMap.Content = append(yamlMap.Content, createScalarNode(keyValuePair.K, keyValuePair.K), yamlValue)
|
||||
yamlMap.Content = append(yamlMap.Content, createScalarNode(key, fmt.Sprintf("%v", key)), yamlValue)
|
||||
}
|
||||
return yamlMap, nil
|
||||
|
||||
}
|
||||
|
||||
func (dec *jsonDecoder) parseArray(dataArray []*orderedMap) (*yaml.Node, error) {
|
||||
func (dec *jsonDecoder) parseArray(dataArray []interface{}) (*yaml.Node, error) {
|
||||
|
||||
var yamlMap = &yaml.Node{Kind: yaml.SequenceNode}
|
||||
|
||||
|
||||
@@ -79,36 +79,3 @@ will output
|
||||
cat
|
||||
```
|
||||
|
||||
## Update or create - entity exists
|
||||
This initialises `a` if it's not present
|
||||
|
||||
Given a sample.yml file of:
|
||||
```yaml
|
||||
a: 1
|
||||
```
|
||||
then
|
||||
```bash
|
||||
yq '(.a // (.a = 0)) += 1' sample.yml
|
||||
```
|
||||
will output
|
||||
```yaml
|
||||
a: 2
|
||||
```
|
||||
|
||||
## Update or create - entity does not exist
|
||||
This initialises `a` if it's not present
|
||||
|
||||
Given a sample.yml file of:
|
||||
```yaml
|
||||
b: camel
|
||||
```
|
||||
then
|
||||
```bash
|
||||
yq '(.a // (.a = 0)) += 1' sample.yml
|
||||
```
|
||||
will output
|
||||
```yaml
|
||||
b: camel
|
||||
a: 1
|
||||
```
|
||||
|
||||
|
||||
@@ -7,11 +7,6 @@ Which will assign the LHS node values to the RHS node values. The RHS expression
|
||||
|
||||
### relative form: `|=`
|
||||
This will do a similar thing to the plain form, however, the RHS expression is run against _the LHS nodes_. This is useful for updating values based on old values, e.g. increment.
|
||||
|
||||
|
||||
### Flags
|
||||
- `c` clobber custom tags
|
||||
|
||||
{% hint style="warning" %}
|
||||
Note that versions prior to 4.18 require the 'eval/e' command to be specified. 
|
||||
|
||||
@@ -240,37 +235,3 @@ a:
|
||||
- bogs
|
||||
```
|
||||
|
||||
## Custom types are maintained by default
|
||||
Given a sample.yml file of:
|
||||
```yaml
|
||||
a: !cat meow
|
||||
b: !dog woof
|
||||
```
|
||||
then
|
||||
```bash
|
||||
yq '.a = .b' sample.yml
|
||||
```
|
||||
will output
|
||||
```yaml
|
||||
a: !cat woof
|
||||
b: !dog woof
|
||||
```
|
||||
|
||||
## Custom types: clovver
|
||||
Use the `c` option to clobber custom tags
|
||||
|
||||
Given a sample.yml file of:
|
||||
```yaml
|
||||
a: !cat meow
|
||||
b: !dog woof
|
||||
```
|
||||
then
|
||||
```bash
|
||||
yq '.a =c .b' sample.yml
|
||||
```
|
||||
will output
|
||||
```yaml
|
||||
a: !dog woof
|
||||
b: !dog woof
|
||||
```
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Comment Operators
|
||||
|
||||
Use these comment operators to set or retrieve comments. Note that line comments on maps/arrays are actually set on the _key_ node as opposed to the _value_ (map/array). See below for examples.
|
||||
Use these comment operators to set or retrieve comments.
|
||||
|
||||
Like the `=` and `|=` assign operators, the same syntax applies when updating comments:
|
||||
|
||||
@@ -17,8 +17,6 @@ Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#
|
||||
{% endhint %}
|
||||
|
||||
## Set line comment
|
||||
Set the comment on the key node for more reliability (see below).
|
||||
|
||||
Given a sample.yml file of:
|
||||
```yaml
|
||||
a: cat
|
||||
@@ -32,24 +30,6 @@ will output
|
||||
a: cat # single
|
||||
```
|
||||
|
||||
## Set line comment of a maps/arrays
|
||||
For maps and arrays, you need to set the line comment on the _key_ node. This will also work for scalars.
|
||||
|
||||
Given a sample.yml file of:
|
||||
```yaml
|
||||
a:
|
||||
b: things
|
||||
```
|
||||
then
|
||||
```bash
|
||||
yq '(.a | key) line_comment="single"' sample.yml
|
||||
```
|
||||
will output
|
||||
```yaml
|
||||
a: # single
|
||||
b: things
|
||||
```
|
||||
|
||||
## Use update assign to perform relative updates
|
||||
Given a sample.yml file of:
|
||||
```yaml
|
||||
@@ -66,121 +46,6 @@ a: cat # cat
|
||||
b: dog # dog
|
||||
```
|
||||
|
||||
## Where is the comment - map key example
|
||||
The underlying yaml parser can assign comments in a document to surprising nodes. Use an expression like this to find where you comment is. 'p' indicates the path, 'isKey' is if the node is a map key (as opposed to a map value).
|
||||
From this, you can see the 'hello-world-comment' is actually on the 'hello' key
|
||||
|
||||
Given a sample.yml file of:
|
||||
```yaml
|
||||
hello: # hello-world-comment
|
||||
message: world
|
||||
```
|
||||
then
|
||||
```bash
|
||||
yq '[... | {"p": path | join("."), "isKey": is_key, "hc": headComment, "lc": lineComment, "fc": footComment}]' sample.yml
|
||||
```
|
||||
will output
|
||||
```yaml
|
||||
- p: ""
|
||||
isKey: false
|
||||
hc: ""
|
||||
lc: ""
|
||||
fc: ""
|
||||
- p: hello
|
||||
isKey: true
|
||||
hc: ""
|
||||
lc: hello-world-comment
|
||||
fc: ""
|
||||
- p: hello
|
||||
isKey: false
|
||||
hc: ""
|
||||
lc: ""
|
||||
fc: ""
|
||||
- p: hello.message
|
||||
isKey: true
|
||||
hc: ""
|
||||
lc: ""
|
||||
fc: ""
|
||||
- p: hello.message
|
||||
isKey: false
|
||||
hc: ""
|
||||
lc: ""
|
||||
fc: ""
|
||||
```
|
||||
|
||||
## Retrieve comment - map key example
|
||||
From the previous example, we know that the comment is on the 'hello' _key_ as a lineComment
|
||||
|
||||
Given a sample.yml file of:
|
||||
```yaml
|
||||
hello: # hello-world-comment
|
||||
message: world
|
||||
```
|
||||
then
|
||||
```bash
|
||||
yq '.hello | key | line_comment' sample.yml
|
||||
```
|
||||
will output
|
||||
```yaml
|
||||
hello-world-comment
|
||||
```
|
||||
|
||||
## Where is the comment - array example
|
||||
The underlying yaml parser can assign comments in a document to surprising nodes. Use an expression like this to find where you comment is. 'p' indicates the path, 'isKey' is if the node is a map key (as opposed to a map value).
|
||||
From this, you can see the 'under-name-comment' is actually on the first child
|
||||
|
||||
Given a sample.yml file of:
|
||||
```yaml
|
||||
name:
|
||||
# under-name-comment
|
||||
- first-array-child
|
||||
```
|
||||
then
|
||||
```bash
|
||||
yq '[... | {"p": path | join("."), "isKey": is_key, "hc": headComment, "lc": lineComment, "fc": footComment}]' sample.yml
|
||||
```
|
||||
will output
|
||||
```yaml
|
||||
- p: ""
|
||||
isKey: false
|
||||
hc: ""
|
||||
lc: ""
|
||||
fc: ""
|
||||
- p: name
|
||||
isKey: true
|
||||
hc: ""
|
||||
lc: ""
|
||||
fc: ""
|
||||
- p: name
|
||||
isKey: false
|
||||
hc: ""
|
||||
lc: ""
|
||||
fc: ""
|
||||
- p: name.0
|
||||
isKey: false
|
||||
hc: under-name-comment
|
||||
lc: ""
|
||||
fc: ""
|
||||
```
|
||||
|
||||
## Retrieve comment - array example
|
||||
From the previous example, we know that the comment is on the first child as a headComment
|
||||
|
||||
Given a sample.yml file of:
|
||||
```yaml
|
||||
name:
|
||||
# under-name-comment
|
||||
- first-array-child
|
||||
```
|
||||
then
|
||||
```bash
|
||||
yq '.name[0] | headComment' sample.yml
|
||||
```
|
||||
will output
|
||||
```yaml
|
||||
under-name-comment
|
||||
```
|
||||
|
||||
## Set head comment
|
||||
Given a sample.yml file of:
|
||||
```yaml
|
||||
@@ -197,25 +62,6 @@ will output
|
||||
a: cat
|
||||
```
|
||||
|
||||
## Set head comment of a map entry
|
||||
Given a sample.yml file of:
|
||||
```yaml
|
||||
f: foo
|
||||
a:
|
||||
b: cat
|
||||
```
|
||||
then
|
||||
```bash
|
||||
yq '(.a | key) head_comment="single"' sample.yml
|
||||
```
|
||||
will output
|
||||
```yaml
|
||||
f: foo
|
||||
# single
|
||||
a:
|
||||
b: cat
|
||||
```
|
||||
|
||||
## Set foot comment, using an expression
|
||||
Given a sample.yml file of:
|
||||
```yaml
|
||||
|
||||
@@ -13,11 +13,11 @@ To specify a custom parsing format, use the `with_dtf` operator. The first param
|
||||
yq 'with_dtf("myformat"; .a + "3h" | tz("Australia/Melbourne"))'
|
||||
```
|
||||
|
||||
See the [library docs](https://pkg.go.dev/time#pkg-constants) for examples of formatting options.
|
||||
See https://pkg.go.dev/time#pkg-constants for examples of formatting options.
|
||||
|
||||
|
||||
## Timezones
|
||||
This uses golangs built in LoadLocation function to parse timezones strings. See the [library docs](https://pkg.go.dev/time#LoadLocation) for more details.
|
||||
This uses golangs built in LoadLocation function to parse timezones strings. See https://pkg.go.dev/time#LoadLocation for more details.
|
||||
|
||||
|
||||
## Durations
|
||||
@@ -156,7 +156,7 @@ a: 2021-01-01T03:10:00Z
|
||||
```
|
||||
|
||||
## Date subtraction
|
||||
You can subtract durations from dates. Assumes RFC3339 date time format, see [date-time operators](https://mikefarah.gitbook.io/yq/operators/datetime#date-time-formattings) for more information.
|
||||
You can subtract durations from dates. Assumes RFC3339 date time format, see [date-time operators](https://mikefarah.gitbook.io/yq/operators/date-time-operators) for more information.
|
||||
|
||||
Given a sample.yml file of:
|
||||
```yaml
|
||||
|
||||
@@ -9,12 +9,12 @@ These operators are useful to process yaml documents that have stringified embed
|
||||
|
||||
| Format | Decode (from string) | Encode (to string) |
|
||||
| --- | -- | --|
|
||||
| Yaml | from_yaml/@yamld | to_yaml(i)/@yaml |
|
||||
| JSON | from_json/@jsond | to_json(i)/@json |
|
||||
| Yaml | from_yaml | to_yaml(i)/@yaml |
|
||||
| JSON | from_json | to_json(i)/@json |
|
||||
| Properties | from_props/@propsd | to_props/@props |
|
||||
| CSV | from_csv/@csvd | to_csv/@csv |
|
||||
| TSV | from_tsv/@tsvd | to_tsv/@tsv |
|
||||
| XML | from_xml/@xmld | to_xml(i)/@xml |
|
||||
| XML | from_xml | to_xml(i)/@xml |
|
||||
| Base64 | @base64d | @base64 |
|
||||
|
||||
|
||||
|
||||
@@ -107,26 +107,6 @@ KEY_a: 1
|
||||
KEY_b: 2
|
||||
```
|
||||
|
||||
## Custom sort map keys
|
||||
Use to_entries to convert to an array of key/value pairs, sort the array using sort/sort_by/etc, and convert it back.
|
||||
|
||||
Given a sample.yml file of:
|
||||
```yaml
|
||||
a: 1
|
||||
c: 3
|
||||
b: 2
|
||||
```
|
||||
then
|
||||
```bash
|
||||
yq 'to_entries | sort_by(.key) | reverse | from_entries' sample.yml
|
||||
```
|
||||
will output
|
||||
```yaml
|
||||
c: 3
|
||||
b: 2
|
||||
a: 1
|
||||
```
|
||||
|
||||
## Use with_entries to filter the map
|
||||
Given a sample.yml file of:
|
||||
```yaml
|
||||
|
||||
@@ -6,8 +6,4 @@ This operator is used to update node values. It can be used in either the:
|
||||
Which will assign the LHS node values to the RHS node values. The RHS expression is run against the matching nodes in the pipeline.
|
||||
|
||||
### relative form: `|=`
|
||||
This will do a similar thing to the plain form, however, the RHS expression is run against _the LHS nodes_. This is useful for updating values based on old values, e.g. increment.
|
||||
|
||||
|
||||
### Flags
|
||||
- `c` clobber custom tags
|
||||
This will do a similar thing to the plain form, however, the RHS expression is run against _the LHS nodes_. This is useful for updating values based on old values, e.g. increment.
|
||||
@@ -1,6 +1,6 @@
|
||||
# Comment Operators
|
||||
|
||||
Use these comment operators to set or retrieve comments. Note that line comments on maps/arrays are actually set on the _key_ node as opposed to the _value_ (map/array). See below for examples.
|
||||
Use these comment operators to set or retrieve comments.
|
||||
|
||||
Like the `=` and `|=` assign operators, the same syntax applies when updating comments:
|
||||
|
||||
|
||||
@@ -13,11 +13,11 @@ To specify a custom parsing format, use the `with_dtf` operator. The first param
|
||||
yq 'with_dtf("myformat"; .a + "3h" | tz("Australia/Melbourne"))'
|
||||
```
|
||||
|
||||
See the [library docs](https://pkg.go.dev/time#pkg-constants) for examples of formatting options.
|
||||
See https://pkg.go.dev/time#pkg-constants for examples of formatting options.
|
||||
|
||||
|
||||
## Timezones
|
||||
This uses golangs built in LoadLocation function to parse timezones strings. See the [library docs](https://pkg.go.dev/time#LoadLocation) for more details.
|
||||
This uses golangs built in LoadLocation function to parse timezones strings. See https://pkg.go.dev/time#LoadLocation for more details.
|
||||
|
||||
|
||||
## Durations
|
||||
|
||||
@@ -9,12 +9,12 @@ These operators are useful to process yaml documents that have stringified embed
|
||||
|
||||
| Format | Decode (from string) | Encode (to string) |
|
||||
| --- | -- | --|
|
||||
| Yaml | from_yaml/@yamld | to_yaml(i)/@yaml |
|
||||
| JSON | from_json/@jsond | to_json(i)/@json |
|
||||
| Yaml | from_yaml | to_yaml(i)/@yaml |
|
||||
| JSON | from_json | to_json(i)/@json |
|
||||
| Properties | from_props/@propsd | to_props/@props |
|
||||
| CSV | from_csv/@csvd | to_csv/@csv |
|
||||
| TSV | from_tsv/@tsvd | to_tsv/@tsv |
|
||||
| XML | from_xml/@xmld | to_xml(i)/@xml |
|
||||
| XML | from_xml | to_xml(i)/@xml |
|
||||
| Base64 | @base64d | @base64 |
|
||||
|
||||
|
||||
|
||||
@@ -14,7 +14,6 @@ You can control how objects are merged by using one or more of the following fla
|
||||
- `d` deeply merge arrays
|
||||
- `?` only merge _existing_ fields
|
||||
- `n` only merge _new_ fields
|
||||
- `c` clobber custom tags
|
||||
|
||||
|
||||
### Merge two files together
|
||||
|
||||
@@ -1,8 +1,5 @@
|
||||
# Path
|
||||
|
||||
The `path` operator can be used to get the traversal paths of matching nodes in an expression. The path is returned as an array, which if traversed in order will lead to the matching node.
|
||||
The path operator can be used to get the traversal paths of matching nodes in an expression. The path is returned as an array, which if traversed in order will lead to the matching node.
|
||||
|
||||
You can get the key/index of matching nodes by using the `path` operator to return the path array then piping that through `.[-1]` to get the last element of that array, the key.
|
||||
|
||||
Use `setpath` to set a value to the path array returned by `path`, and similarly `delpaths` for an array of path arrays.
|
||||
|
||||
|
||||
@@ -11,6 +11,3 @@ diff file1.yml file2.yml
|
||||
```
|
||||
|
||||
Note that `yq` does not yet consider anchors when sorting by keys - this may result in invalid yaml documents if your are using merge anchors.
|
||||
|
||||
For more advanced sorting, using `to_entries` to convert the map to an array, then sort/process the array as you like (e.g. using `sort_by`) and convert back to a map using `from_entries`.
|
||||
See [here](https://mikefarah.gitbook.io/yq/operators/entries#custom-sort-map-keys) for an example.
|
||||
|
||||
@@ -118,43 +118,3 @@ will output
|
||||
comment on key
|
||||
```
|
||||
|
||||
## Check node is a key
|
||||
Given a sample.yml file of:
|
||||
```yaml
|
||||
a:
|
||||
b:
|
||||
- cat
|
||||
c: frog
|
||||
```
|
||||
then
|
||||
```bash
|
||||
yq '[... | { "p": path | join("."), "isKey": is_key, "tag": tag }]' sample.yml
|
||||
```
|
||||
will output
|
||||
```yaml
|
||||
- p: ""
|
||||
isKey: false
|
||||
tag: '!!map'
|
||||
- p: a
|
||||
isKey: true
|
||||
tag: '!!str'
|
||||
- p: a
|
||||
isKey: false
|
||||
tag: '!!map'
|
||||
- p: a.b
|
||||
isKey: true
|
||||
tag: '!!str'
|
||||
- p: a.b
|
||||
isKey: false
|
||||
tag: '!!seq'
|
||||
- p: a.b.0
|
||||
isKey: false
|
||||
tag: '!!str'
|
||||
- p: a.c
|
||||
isKey: true
|
||||
tag: '!!str'
|
||||
- p: a.c
|
||||
isKey: false
|
||||
tag: '!!str'
|
||||
```
|
||||
|
||||
|
||||
@@ -14,7 +14,6 @@ You can control how objects are merged by using one or more of the following fla
|
||||
- `d` deeply merge arrays
|
||||
- `?` only merge _existing_ fields
|
||||
- `n` only merge _new_ fields
|
||||
- `c` clobber custom tags
|
||||
|
||||
|
||||
### Merge two files together
|
||||
@@ -482,26 +481,3 @@ b: !goat
|
||||
dog: woof
|
||||
```
|
||||
|
||||
## Custom types: clobber tags
|
||||
Use the `c` option to clobber custom tags. Note that the second tag is now used
|
||||
|
||||
Given a sample.yml file of:
|
||||
```yaml
|
||||
a: !horse
|
||||
cat: meow
|
||||
b: !goat
|
||||
dog: woof
|
||||
```
|
||||
then
|
||||
```bash
|
||||
yq '.a *=c .b' sample.yml
|
||||
```
|
||||
will output
|
||||
```yaml
|
||||
a: !goat
|
||||
cat: meow
|
||||
dog: woof
|
||||
b: !goat
|
||||
dog: woof
|
||||
```
|
||||
|
||||
|
||||
@@ -1,12 +1,9 @@
|
||||
# Path
|
||||
|
||||
The `path` operator can be used to get the traversal paths of matching nodes in an expression. The path is returned as an array, which if traversed in order will lead to the matching node.
|
||||
The path operator can be used to get the traversal paths of matching nodes in an expression. The path is returned as an array, which if traversed in order will lead to the matching node.
|
||||
|
||||
You can get the key/index of matching nodes by using the `path` operator to return the path array then piping that through `.[-1]` to get the last element of that array, the key.
|
||||
|
||||
Use `setpath` to set a value to the path array returned by `path`, and similarly `delpaths` for an array of path arrays.
|
||||
|
||||
|
||||
{% hint style="warning" %}
|
||||
Note that versions prior to 4.18 require the 'eval/e' command to be specified. 
|
||||
|
||||
@@ -101,114 +98,3 @@ will output
|
||||
value: frog
|
||||
```
|
||||
|
||||
## Set path
|
||||
Given a sample.yml file of:
|
||||
```yaml
|
||||
a:
|
||||
b: cat
|
||||
```
|
||||
then
|
||||
```bash
|
||||
yq 'setpath(["a", "b"]; "things")' sample.yml
|
||||
```
|
||||
will output
|
||||
```yaml
|
||||
a:
|
||||
b: things
|
||||
```
|
||||
|
||||
## Set on empty document
|
||||
Running
|
||||
```bash
|
||||
yq --null-input 'setpath(["a", "b"]; "things")'
|
||||
```
|
||||
will output
|
||||
```yaml
|
||||
a:
|
||||
b: things
|
||||
```
|
||||
|
||||
## Set array path
|
||||
Given a sample.yml file of:
|
||||
```yaml
|
||||
a:
|
||||
- cat
|
||||
- frog
|
||||
```
|
||||
then
|
||||
```bash
|
||||
yq 'setpath(["a", 0]; "things")' sample.yml
|
||||
```
|
||||
will output
|
||||
```yaml
|
||||
a:
|
||||
- things
|
||||
- frog
|
||||
```
|
||||
|
||||
## Set array path empty
|
||||
Running
|
||||
```bash
|
||||
yq --null-input 'setpath(["a", 0]; "things")'
|
||||
```
|
||||
will output
|
||||
```yaml
|
||||
a:
|
||||
- things
|
||||
```
|
||||
|
||||
## Delete path
|
||||
Notice delpaths takes an _array_ of paths.
|
||||
|
||||
Given a sample.yml file of:
|
||||
```yaml
|
||||
a:
|
||||
b: cat
|
||||
c: dog
|
||||
d: frog
|
||||
```
|
||||
then
|
||||
```bash
|
||||
yq 'delpaths([["a", "c"], ["a", "d"]])' sample.yml
|
||||
```
|
||||
will output
|
||||
```yaml
|
||||
a:
|
||||
b: cat
|
||||
```
|
||||
|
||||
## Delete array path
|
||||
Given a sample.yml file of:
|
||||
```yaml
|
||||
a:
|
||||
- cat
|
||||
- frog
|
||||
```
|
||||
then
|
||||
```bash
|
||||
yq 'delpaths([["a", 0]])' sample.yml
|
||||
```
|
||||
will output
|
||||
```yaml
|
||||
a:
|
||||
- frog
|
||||
```
|
||||
|
||||
## Delete - wrong parameter
|
||||
delpaths does not work with a single path array
|
||||
|
||||
Given a sample.yml file of:
|
||||
```yaml
|
||||
a:
|
||||
- cat
|
||||
- frog
|
||||
```
|
||||
then
|
||||
```bash
|
||||
yq 'delpaths(["a", 0])' sample.yml
|
||||
```
|
||||
will output
|
||||
```bash
|
||||
Error: DELPATHS: expected entry [0] to be a sequence, but its a !!str. Note that delpaths takes an array of path arrays, e.g. [["a", "b"]]
|
||||
```
|
||||
|
||||
|
||||
@@ -12,9 +12,6 @@ diff file1.yml file2.yml
|
||||
|
||||
Note that `yq` does not yet consider anchors when sorting by keys - this may result in invalid yaml documents if your are using merge anchors.
|
||||
|
||||
For more advanced sorting, using `to_entries` to convert the map to an array, then sort/process the array as you like (e.g. using `sort_by`) and convert back to a map using `from_entries`.
|
||||
See [here](https://mikefarah.gitbook.io/yq/operators/entries#custom-sort-map-keys) for an example.
|
||||
|
||||
{% hint style="warning" %}
|
||||
Note that versions prior to 4.18 require the 'eval/e' command to be specified. 
|
||||
|
||||
|
||||
@@ -112,26 +112,6 @@ will output
|
||||
cat; meow; 1; ; true
|
||||
```
|
||||
|
||||
## Trim strings
|
||||
Given a sample.yml file of:
|
||||
```yaml
|
||||
- ' cat'
|
||||
- 'dog '
|
||||
- ' cow cow '
|
||||
- horse
|
||||
```
|
||||
then
|
||||
```bash
|
||||
yq '.[] | trim' sample.yml
|
||||
```
|
||||
will output
|
||||
```yaml
|
||||
cat
|
||||
dog
|
||||
cow cow
|
||||
horse
|
||||
```
|
||||
|
||||
## Match string
|
||||
Given a sample.yml file of:
|
||||
```yaml
|
||||
|
||||
@@ -1,9 +1,10 @@
|
||||
# JSON
|
||||
|
||||
Encode and decode to and from JSON. Supports multiple JSON documents in a single file (e.g. NDJSON).
|
||||
Encode and decode to and from JSON. Note that, unless you have multiple JSON documents in a single file, YAML is a _superset_ of JSON - so `yq` can read any json file without doing anything special.
|
||||
|
||||
Note that YAML is a superset of (single document) JSON - so you don't have to use the JSON parser to read JSON when there is only one JSON document in the input. You will probably want to pretty print the result in this case, to get idiomatic YAML styling.
|
||||
If you do have mulitple JSON documents in a single file (e.g. NDJSON) then you will need to explicity use the json parser `-p=json`.
|
||||
|
||||
This means you don't need to 'convert' a JSON file to YAML - however if you want idiomatic YAML styling, then you can use the `-P/--prettyPrint` flag, see examples below.
|
||||
|
||||
{% hint style="warning" %}
|
||||
Note that versions prior to 4.18 require the 'eval/e' command to be specified. 
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
# JSON
|
||||
|
||||
Encode and decode to and from JSON. Supports multiple JSON documents in a single file (e.g. NDJSON).
|
||||
Encode and decode to and from JSON. Note that, unless you have multiple JSON documents in a single file, YAML is a _superset_ of JSON - so `yq` can read any json file without doing anything special.
|
||||
|
||||
Note that YAML is a superset of (single document) JSON - so you don't have to use the JSON parser to read JSON when there is only one JSON document in the input. You will probably want to pretty print the result in this case, to get idiomatic YAML styling.
|
||||
If you do have mulitple JSON documents in a single file (e.g. NDJSON) then you will need to explicity use the json parser `-p=json`.
|
||||
|
||||
This means you don't need to 'convert' a JSON file to YAML - however if you want idiomatic YAML styling, then you can use the `-P/--prettyPrint` flag, see examples below.
|
||||
|
||||
@@ -66,13 +66,8 @@ func (o *orderedMap) UnmarshalJSON(data []byte) error {
|
||||
}
|
||||
return nil
|
||||
case '[':
|
||||
var res []*orderedMap
|
||||
if err := json.Unmarshal(data, &res); err != nil {
|
||||
return err
|
||||
}
|
||||
o.altVal = res
|
||||
o.kv = nil
|
||||
return nil
|
||||
var arr []orderedMap
|
||||
return json.Unmarshal(data, &arr)
|
||||
}
|
||||
|
||||
return json.Unmarshal(data, &o.altVal)
|
||||
|
||||
@@ -70,7 +70,7 @@ func (p *expressionParserImpl) createExpressionTree(postFixPath []*Operation) (*
|
||||
stack = append(stack, &newNode)
|
||||
}
|
||||
if len(stack) != 1 {
|
||||
return nil, fmt.Errorf("bad expression, please check expression syntax")
|
||||
return nil, fmt.Errorf("Bad expression, please check expression syntax")
|
||||
}
|
||||
return stack[0], nil
|
||||
}
|
||||
|
||||
@@ -11,38 +11,33 @@ func getExpressionParser() ExpressionParserInterface {
|
||||
return ExpressionParser
|
||||
}
|
||||
|
||||
func TestParserNoMatchingCloseBracket(t *testing.T) {
|
||||
_, err := getExpressionParser().ParseExpression(".cat | with(.;.bob")
|
||||
test.AssertResultComplex(t, "bad expression - probably missing close bracket on WITH", err.Error())
|
||||
}
|
||||
|
||||
func TestParserNoMatchingCloseCollect(t *testing.T) {
|
||||
_, err := getExpressionParser().ParseExpression("[1,2")
|
||||
test.AssertResultComplex(t, "bad expression, could not find matching `]`", err.Error())
|
||||
test.AssertResultComplex(t, "Bad expression, could not find matching `]`", err.Error())
|
||||
}
|
||||
func TestParserNoMatchingCloseObjectInCollect(t *testing.T) {
|
||||
_, err := getExpressionParser().ParseExpression(`[{"b": "c"]`)
|
||||
test.AssertResultComplex(t, "bad expression, could not find matching `}`", err.Error())
|
||||
test.AssertResultComplex(t, "Bad expression, could not find matching `}`", err.Error())
|
||||
}
|
||||
|
||||
func TestParserNoMatchingCloseInCollect(t *testing.T) {
|
||||
_, err := getExpressionParser().ParseExpression(`[(.a]`)
|
||||
test.AssertResultComplex(t, "bad expression, could not find matching `)`", err.Error())
|
||||
test.AssertResultComplex(t, "Bad expression, could not find matching `)`", err.Error())
|
||||
}
|
||||
|
||||
func TestParserNoMatchingCloseCollectObject(t *testing.T) {
|
||||
_, err := getExpressionParser().ParseExpression(`{"a": "b"`)
|
||||
test.AssertResultComplex(t, "bad expression, could not find matching `}`", err.Error())
|
||||
test.AssertResultComplex(t, "Bad expression, could not find matching `}`", err.Error())
|
||||
}
|
||||
|
||||
func TestParserNoMatchingCloseCollectInCollectObject(t *testing.T) {
|
||||
_, err := getExpressionParser().ParseExpression(`{"b": [1}`)
|
||||
test.AssertResultComplex(t, "bad expression, could not find matching `]`", err.Error())
|
||||
test.AssertResultComplex(t, "Bad expression, could not find matching `]`", err.Error())
|
||||
}
|
||||
|
||||
func TestParserNoMatchingCloseBracketInCollectObject(t *testing.T) {
|
||||
_, err := getExpressionParser().ParseExpression(`{"b": (1}`)
|
||||
test.AssertResultComplex(t, "bad expression, could not find matching `)`", err.Error())
|
||||
test.AssertResultComplex(t, "Bad expression, could not find matching `)`", err.Error())
|
||||
}
|
||||
|
||||
func TestParserNoArgsForTwoArgOp(t *testing.T) {
|
||||
@@ -77,5 +72,5 @@ func TestParserOneArgForOneArgOp(t *testing.T) {
|
||||
|
||||
func TestParserExtraArgs(t *testing.T) {
|
||||
_, err := getExpressionParser().ParseExpression("sortKeys(.) explode(.)")
|
||||
test.AssertResultComplex(t, "bad expression, please check expression syntax", err.Error())
|
||||
test.AssertResultComplex(t, "Bad expression, please check expression syntax", err.Error())
|
||||
}
|
||||
|
||||
@@ -27,11 +27,11 @@ func popOpToResult(opStack []*token, result []*Operation) ([]*token, []*Operatio
|
||||
|
||||
func validateNoOpenTokens(token *token) error {
|
||||
if token.TokenType == openCollect {
|
||||
return fmt.Errorf(("bad expression, could not find matching `]`"))
|
||||
return fmt.Errorf(("Bad expression, could not find matching `]`"))
|
||||
} else if token.TokenType == openCollectObject {
|
||||
return fmt.Errorf(("bad expression, could not find matching `}`"))
|
||||
return fmt.Errorf(("Bad expression, could not find matching `}`"))
|
||||
} else if token.TokenType == openBracket {
|
||||
return fmt.Errorf(("bad expression, could not find matching `)`"))
|
||||
return fmt.Errorf(("Bad expression, could not find matching `)`"))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -104,7 +104,7 @@ func (p *expressionPostFixerImpl) ConvertToPostfix(infixTokens []*token) ([]*Ope
|
||||
opStack, result = popOpToResult(opStack, result)
|
||||
}
|
||||
if len(opStack) == 0 {
|
||||
return nil, errors.New("bad expression, got close brackets without matching opening bracket")
|
||||
return nil, errors.New("Bad path expression, got close brackets without matching opening bracket")
|
||||
}
|
||||
// now we should have ( as the last element on the opStack, get rid of it
|
||||
opStack = opStack[0 : len(opStack)-1]
|
||||
@@ -124,14 +124,6 @@ func (p *expressionPostFixerImpl) ConvertToPostfix(infixTokens []*token) ([]*Ope
|
||||
}
|
||||
|
||||
log.Debugf("opstackLen: %v", len(opStack))
|
||||
if len(opStack) > 0 {
|
||||
log.Debugf("opstack:")
|
||||
for _, token := range opStack {
|
||||
log.Debugf("- %v", token.toString(true))
|
||||
}
|
||||
|
||||
return nil, fmt.Errorf("bad expression - probably missing close bracket on %v", opStack[len(opStack)-1].toString(false))
|
||||
}
|
||||
|
||||
if log.IsEnabledFor(logging.DEBUG) {
|
||||
log.Debugf("PostFix Result:")
|
||||
|
||||
@@ -22,13 +22,6 @@ const sampleNdJson = `{"this": "is a multidoc json file"}
|
||||
{"a number": 4}
|
||||
`
|
||||
|
||||
const sampleNdJsonKey = `{"a": "first", "b": "next", "ab": "last"}`
|
||||
|
||||
const expectedJsonKeysInOrder = `a: first
|
||||
b: next
|
||||
ab: last
|
||||
`
|
||||
|
||||
const expectedNdJsonYaml = `this: is a multidoc json file
|
||||
---
|
||||
each:
|
||||
@@ -164,13 +157,6 @@ var jsonScenarios = []formatScenario{
|
||||
expected: expectedNdJsonYaml,
|
||||
scenarioType: "decode-ndjson",
|
||||
},
|
||||
{
|
||||
description: "Decode NDJSON, maintain key order",
|
||||
skipDoc: true,
|
||||
input: sampleNdJsonKey,
|
||||
expected: expectedJsonKeysInOrder,
|
||||
scenarioType: "decode-ndjson",
|
||||
},
|
||||
{
|
||||
description: "numbers",
|
||||
skipDoc: true,
|
||||
@@ -178,20 +164,6 @@ var jsonScenarios = []formatScenario{
|
||||
expected: "- 3\n- 3\n- 3.1\n- -1\n",
|
||||
scenarioType: "decode-ndjson",
|
||||
},
|
||||
{
|
||||
description: "number single",
|
||||
skipDoc: true,
|
||||
input: "3",
|
||||
expected: "3\n",
|
||||
scenarioType: "decode-ndjson",
|
||||
},
|
||||
{
|
||||
description: "empty string",
|
||||
skipDoc: true,
|
||||
input: `""`,
|
||||
expected: "\"\"\n",
|
||||
scenarioType: "decode-ndjson",
|
||||
},
|
||||
{
|
||||
description: "strings",
|
||||
skipDoc: true,
|
||||
|
||||
@@ -123,13 +123,10 @@ var participleYqRules = []*participleYqRule{
|
||||
|
||||
simpleOp("keys", keysOpType),
|
||||
simpleOp("key", getKeyOpType),
|
||||
simpleOp("is_?key", isKeyOpType),
|
||||
|
||||
simpleOp("file_?name|fileName", getFilenameOpType),
|
||||
simpleOp("file_?index|fileIndex|fi", getFileIndexOpType),
|
||||
simpleOp("path", getPathOpType),
|
||||
simpleOp("set_?path", setPathOpType),
|
||||
simpleOp("del_?paths", delPathsOpType),
|
||||
|
||||
simpleOp("to_?entries|toEntries", toEntriesOpType),
|
||||
simpleOp("from_?entries|fromEntries", fromEntriesOpType),
|
||||
@@ -155,7 +152,6 @@ var participleYqRules = []*participleYqRule{
|
||||
|
||||
{"Uppercase", `upcase|ascii_?upcase`, opTokenWithPrefs(changeCaseOpType, nil, changeCasePrefs{ToUpperCase: true}), 0},
|
||||
{"Downcase", `downcase|ascii_?downcase`, opTokenWithPrefs(changeCaseOpType, nil, changeCasePrefs{ToUpperCase: false}), 0},
|
||||
simpleOp("trim", trimOpType),
|
||||
|
||||
{"HexValue", `0[xX][0-9A-Fa-f]+`, hexValue(), 0},
|
||||
{"FloatValueScientific", `-?[1-9](\.\d+)?[Ee][-+]?\d+`, floatValue(), 0},
|
||||
@@ -185,8 +181,8 @@ var participleYqRules = []*participleYqRule{
|
||||
{"GreaterThan", `\s*>\s*`, opTokenWithPrefs(compareOpType, nil, compareTypePref{OrEqual: false, Greater: true}), 0},
|
||||
{"LessThan", `\s*<\s*`, opTokenWithPrefs(compareOpType, nil, compareTypePref{OrEqual: false, Greater: false}), 0},
|
||||
|
||||
{"AssignRelative", `\|=[c]*`, assignOpToken(true), 0},
|
||||
{"Assign", `=[c]*`, assignOpToken(false), 0},
|
||||
{"AssignRelative", `\|=`, assignOpToken(true), 0},
|
||||
{"Assign", `=`, assignOpToken(false), 0},
|
||||
|
||||
{`whitespace`, `[ \t\n]+`, nil, 0},
|
||||
|
||||
@@ -197,8 +193,8 @@ var participleYqRules = []*participleYqRule{
|
||||
|
||||
{"Union", `,`, opToken(unionOpType), 0},
|
||||
|
||||
{"MultiplyAssign", `\*=[\+|\?cdn]*`, multiplyWithPrefs(multiplyAssignOpType), 0},
|
||||
{"Multiply", `\*[\+|\?cdn]*`, multiplyWithPrefs(multiplyOpType), 0},
|
||||
{"MultiplyAssign", `\*=[\+|\?dn]*`, multiplyWithPrefs(multiplyAssignOpType), 0},
|
||||
{"Multiply", `\*[\+|\?dn]*`, multiplyWithPrefs(multiplyOpType), 0},
|
||||
|
||||
{"AddAssign", `\+=`, opToken(addAssignOpType), 0},
|
||||
{"Add", `\+`, opToken(addOpType), 0},
|
||||
@@ -320,9 +316,6 @@ func assignOpToken(updateAssign bool) yqAction {
|
||||
log.Debug("assignOpToken %v", rawToken.Value)
|
||||
value := rawToken.Value
|
||||
prefs := assignPreferences{DontOverWriteAnchor: true}
|
||||
if strings.Contains(value, "c") {
|
||||
prefs.ClobberCustomTags = true
|
||||
}
|
||||
op := &Operation{OperationType: assignOpType, Value: assignOpType.Type, StringValue: value, UpdateAssign: updateAssign, Preferences: prefs}
|
||||
return &token{TokenType: operationToken, Operation: op}, nil
|
||||
}
|
||||
@@ -393,7 +386,6 @@ func envSubstWithOptions() yqAction {
|
||||
func multiplyWithPrefs(op *operationType) yqAction {
|
||||
return func(rawToken lexer.Token) (*token, error) {
|
||||
prefs := multiplyPreferences{}
|
||||
prefs.AssignPrefs = assignPreferences{}
|
||||
options := rawToken.Value
|
||||
if strings.Contains(options, "+") {
|
||||
prefs.AppendArrays = true
|
||||
@@ -402,14 +394,11 @@ func multiplyWithPrefs(op *operationType) yqAction {
|
||||
prefs.TraversePrefs = traversePreferences{DontAutoCreate: true}
|
||||
}
|
||||
if strings.Contains(options, "n") {
|
||||
prefs.AssignPrefs.OnlyWriteNull = true
|
||||
prefs.AssignPrefs = assignPreferences{OnlyWriteNull: true}
|
||||
}
|
||||
if strings.Contains(options, "d") {
|
||||
prefs.DeepMergeArrays = true
|
||||
}
|
||||
if strings.Contains(options, "c") {
|
||||
prefs.AssignPrefs.ClobberCustomTags = true
|
||||
}
|
||||
prefs.TraversePrefs.DontFollowAlias = true
|
||||
op := &Operation{OperationType: op, Value: multiplyOpType.Type, StringValue: options, Preferences: prefs}
|
||||
return &token{TokenType: operationToken, Operation: op}, nil
|
||||
|
||||
+3
-7
@@ -1,4 +1,5 @@
|
||||
// Use the top level Evaluator or StreamEvaluator to evaluate expressions and return matches.
|
||||
//
|
||||
package yqlib
|
||||
|
||||
import (
|
||||
@@ -79,7 +80,7 @@ var notEqualsOpType = &operationType{Type: "NOT_EQUALS", NumArgs: 2, Precedence:
|
||||
|
||||
var compareOpType = &operationType{Type: "COMPARE", NumArgs: 2, Precedence: 40, Handler: compareOperator}
|
||||
|
||||
// createmap needs to be above union, as we use union to build the components of the objects
|
||||
//createmap needs to be above union, as we use union to build the components of the objects
|
||||
var createMapOpType = &operationType{Type: "CREATE_MAP", NumArgs: 2, Precedence: 15, Handler: createMapOperator}
|
||||
|
||||
var shortPipeOpType = &operationType{Type: "SHORT_PIPE", NumArgs: 2, Precedence: 45, Handler: pipeOperator}
|
||||
@@ -121,7 +122,6 @@ var getStyleOpType = &operationType{Type: "GET_STYLE", NumArgs: 0, Precedence: 5
|
||||
var getTagOpType = &operationType{Type: "GET_TAG", NumArgs: 0, Precedence: 50, Handler: getTagOperator}
|
||||
|
||||
var getKeyOpType = &operationType{Type: "GET_KEY", NumArgs: 0, Precedence: 50, Handler: getKeyOperator}
|
||||
var isKeyOpType = &operationType{Type: "IS_KEY", NumArgs: 0, Precedence: 50, Handler: isKeyOperator}
|
||||
var getParentOpType = &operationType{Type: "GET_PARENT", NumArgs: 0, Precedence: 50, Handler: getParentOperator}
|
||||
|
||||
var getCommentOpType = &operationType{Type: "GET_COMMENT", NumArgs: 0, Precedence: 50, Handler: getCommentsOperator}
|
||||
@@ -130,10 +130,7 @@ var getAliasOpType = &operationType{Type: "GET_ALIAS", NumArgs: 0, Precedence: 5
|
||||
var getDocumentIndexOpType = &operationType{Type: "GET_DOCUMENT_INDEX", NumArgs: 0, Precedence: 50, Handler: getDocumentIndexOperator}
|
||||
var getFilenameOpType = &operationType{Type: "GET_FILENAME", NumArgs: 0, Precedence: 50, Handler: getFilenameOperator}
|
||||
var getFileIndexOpType = &operationType{Type: "GET_FILE_INDEX", NumArgs: 0, Precedence: 50, Handler: getFileIndexOperator}
|
||||
|
||||
var getPathOpType = &operationType{Type: "GET_PATH", NumArgs: 0, Precedence: 50, Handler: getPathOperator}
|
||||
var setPathOpType = &operationType{Type: "SET_PATH", NumArgs: 1, Precedence: 50, Handler: setPathOperator}
|
||||
var delPathsOpType = &operationType{Type: "DEL_PATHS", NumArgs: 1, Precedence: 50, Handler: delPathsOperator}
|
||||
|
||||
var explodeOpType = &operationType{Type: "EXPLODE", NumArgs: 1, Precedence: 50, Handler: explodeOperator}
|
||||
var sortByOpType = &operationType{Type: "SORT_BY", NumArgs: 1, Precedence: 50, Handler: sortByOperator}
|
||||
@@ -149,7 +146,6 @@ var captureOpType = &operationType{Type: "CAPTURE", NumArgs: 1, Precedence: 50,
|
||||
var testOpType = &operationType{Type: "TEST", NumArgs: 1, Precedence: 50, Handler: testOperator}
|
||||
var splitStringOpType = &operationType{Type: "SPLIT", NumArgs: 1, Precedence: 50, Handler: splitStringOperator}
|
||||
var changeCaseOpType = &operationType{Type: "CHANGE_CASE", NumArgs: 0, Precedence: 50, Handler: changeCaseOperator}
|
||||
var trimOpType = &operationType{Type: "TRIM", NumArgs: 0, Precedence: 50, Handler: trimSpaceOperator}
|
||||
|
||||
var loadOpType = &operationType{Type: "LOAD", NumArgs: 1, Precedence: 50, Handler: loadYamlOperator}
|
||||
|
||||
@@ -434,7 +430,7 @@ func (p *Operation) toString() string {
|
||||
}
|
||||
}
|
||||
|
||||
// use for debugging only
|
||||
//use for debugging only
|
||||
func NodesToString(collection *list.List) string {
|
||||
if !log.IsEnabledFor(logging.DEBUG) {
|
||||
return ""
|
||||
|
||||
@@ -62,9 +62,6 @@ func add(d *dataTreeNavigator, context Context, lhs *CandidateNode, rhs *Candida
|
||||
|
||||
switch lhsNode.Kind {
|
||||
case yaml.MappingNode:
|
||||
if rhs.Node.Kind != yaml.MappingNode {
|
||||
return nil, fmt.Errorf("%v (%v) cannot be added to a %v (%v)", rhs.Node.Tag, rhs.GetNicePath(), lhsNode.Tag, lhs.GetNicePath())
|
||||
}
|
||||
addMaps(target, lhs, rhs)
|
||||
case yaml.SequenceNode:
|
||||
if err := addSequences(target, lhs, rhs); err != nil {
|
||||
@@ -73,7 +70,7 @@ func add(d *dataTreeNavigator, context Context, lhs *CandidateNode, rhs *Candida
|
||||
|
||||
case yaml.ScalarNode:
|
||||
if rhs.Node.Kind != yaml.ScalarNode {
|
||||
return nil, fmt.Errorf("%v (%v) cannot be added to a %v (%v)", rhs.Node.Tag, rhs.GetNicePath(), lhsNode.Tag, lhs.GetNicePath())
|
||||
return nil, fmt.Errorf("%v (%v) cannot be added to a %v", rhs.Node.Tag, rhs.Path, lhsNode.Tag)
|
||||
}
|
||||
target.Node.Kind = yaml.ScalarNode
|
||||
target.Node.Style = lhsNode.Style
|
||||
|
||||
@@ -350,20 +350,6 @@ var addOperatorScenarios = []expressionScenario{
|
||||
"D0, P[], (doc)::a: &horse [1, 2]\n",
|
||||
},
|
||||
},
|
||||
{
|
||||
skipDoc: true,
|
||||
description: "Add sequence to map",
|
||||
document: "a: {x: cool}",
|
||||
expression: `.a += [2]`,
|
||||
expectedError: "!!seq () cannot be added to a !!map (a)",
|
||||
},
|
||||
{
|
||||
skipDoc: true,
|
||||
description: "Add sequence to scalar",
|
||||
document: "a: cool",
|
||||
expression: `.a += [2]`,
|
||||
expectedError: "!!seq () cannot be added to a !!str (a)",
|
||||
},
|
||||
}
|
||||
|
||||
func TestAddOperatorScenarios(t *testing.T) {
|
||||
|
||||
@@ -2,24 +2,7 @@ package yqlib
|
||||
|
||||
func alternativeOperator(d *dataTreeNavigator, context Context, expressionNode *ExpressionNode) (Context, error) {
|
||||
log.Debugf("-- alternative")
|
||||
prefs := crossFunctionPreferences{
|
||||
CalcWhenEmpty: true,
|
||||
Calculation: alternativeFunc,
|
||||
LhsResultValue: func(lhs *CandidateNode) (*CandidateNode, error) {
|
||||
if lhs == nil {
|
||||
return nil, nil
|
||||
}
|
||||
truthy, err := isTruthy(lhs)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if truthy {
|
||||
return lhs, nil
|
||||
}
|
||||
return nil, nil
|
||||
},
|
||||
}
|
||||
return crossFunctionWithPrefs(d, context, expressionNode, prefs)
|
||||
return crossFunction(d, context, expressionNode, alternativeFunc, true)
|
||||
}
|
||||
|
||||
func alternativeFunc(d *dataTreeNavigator, context Context, lhs *CandidateNode, rhs *CandidateNode) (*CandidateNode, error) {
|
||||
|
||||
@@ -85,24 +85,6 @@ var alternativeOperatorScenarios = []expressionScenario{
|
||||
"D0, P[], (!!bool)::true\n",
|
||||
},
|
||||
},
|
||||
{
|
||||
description: "Update or create - entity exists",
|
||||
subdescription: "This initialises `a` if it's not present",
|
||||
expression: "(.a // (.a = 0)) += 1",
|
||||
document: `a: 1`,
|
||||
expected: []string{
|
||||
"D0, P[], (doc)::a: 2\n",
|
||||
},
|
||||
},
|
||||
{
|
||||
description: "Update or create - entity does not exist",
|
||||
subdescription: "This initialises `a` if it's not present",
|
||||
expression: "(.a // (.a = 0)) += 1",
|
||||
document: `b: camel`,
|
||||
expected: []string{
|
||||
"D0, P[], (!!map)::b: camel\na: 1\n",
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
func TestAlternativeOperatorScenarios(t *testing.T) {
|
||||
|
||||
@@ -3,7 +3,6 @@ package yqlib
|
||||
type assignPreferences struct {
|
||||
DontOverWriteAnchor bool
|
||||
OnlyWriteNull bool
|
||||
ClobberCustomTags bool
|
||||
}
|
||||
|
||||
func assignUpdateFunc(prefs assignPreferences) crossFunctionCalculation {
|
||||
@@ -16,29 +15,18 @@ func assignUpdateFunc(prefs assignPreferences) crossFunctionCalculation {
|
||||
}
|
||||
}
|
||||
|
||||
// they way *= (multipleAssign) is handled, we set the multiplePrefs
|
||||
// on the node, not assignPrefs. Long story.
|
||||
func getAssignPreferences(preferences interface{}) assignPreferences {
|
||||
prefs := assignPreferences{}
|
||||
|
||||
switch typedPref := preferences.(type) {
|
||||
case assignPreferences:
|
||||
prefs = typedPref
|
||||
case multiplyPreferences:
|
||||
prefs = typedPref.AssignPrefs
|
||||
}
|
||||
return prefs
|
||||
}
|
||||
|
||||
func assignUpdateOperator(d *dataTreeNavigator, context Context, expressionNode *ExpressionNode) (Context, error) {
|
||||
lhs, err := d.GetMatchingNodes(context, expressionNode.LHS)
|
||||
if err != nil {
|
||||
return Context{}, err
|
||||
}
|
||||
|
||||
prefs := getAssignPreferences(expressionNode.Operation.Preferences)
|
||||
|
||||
log.Debug("assignUpdateOperator prefs: %v", prefs)
|
||||
prefs := assignPreferences{}
|
||||
// they way *= (multipleAssign) is handled, we set the multiplePrefs
|
||||
// on the node, not assignPrefs. Long story.
|
||||
if p, ok := expressionNode.Operation.Preferences.(assignPreferences); ok {
|
||||
prefs = p
|
||||
}
|
||||
|
||||
if !expressionNode.Operation.UpdateAssign {
|
||||
// this works because we already ran against LHS with an editable context.
|
||||
|
||||
@@ -208,23 +208,6 @@ var assignOperatorScenarios = []expressionScenario{
|
||||
"D0, P[], (doc)::a:\n b:\n - null\n - c: bogs\n",
|
||||
},
|
||||
},
|
||||
{
|
||||
description: "Custom types are maintained by default",
|
||||
document: "a: !cat meow\nb: !dog woof",
|
||||
expression: `.a = .b`,
|
||||
expected: []string{
|
||||
"D0, P[], (doc)::a: !cat woof\nb: !dog woof\n",
|
||||
},
|
||||
},
|
||||
{
|
||||
description: "Custom types: clovver",
|
||||
subdescription: "Use the `c` option to clobber custom tags",
|
||||
document: "a: !cat meow\nb: !dog woof",
|
||||
expression: `.a =c .b`,
|
||||
expected: []string{
|
||||
"D0, P[], (doc)::a: !dog woof\nb: !dog woof\n",
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
func TestAssignOperatorScenarios(t *testing.T) {
|
||||
|
||||
@@ -63,12 +63,8 @@ func assignCommentsOperator(d *dataTreeNavigator, context Context, expressionNod
|
||||
}
|
||||
if preferences.FootComment && candidate.Node.Kind == yaml.DocumentNode && comment != "" {
|
||||
candidate.TrailingContent = "# " + comment
|
||||
} else if preferences.FootComment && candidate.Node.Kind == yaml.DocumentNode {
|
||||
candidate.TrailingContent = comment
|
||||
|
||||
} else if preferences.FootComment && candidate.Node.Kind != yaml.DocumentNode {
|
||||
candidate.Node.FootComment = comment
|
||||
candidate.TrailingContent = ""
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@@ -4,74 +4,15 @@ import (
|
||||
"testing"
|
||||
)
|
||||
|
||||
var expectedWhereIsMyCommentMapKey = `D0, P[], (!!seq)::- p: ""
|
||||
isKey: false
|
||||
hc: ""
|
||||
lc: ""
|
||||
fc: ""
|
||||
- p: hello
|
||||
isKey: true
|
||||
hc: ""
|
||||
lc: hello-world-comment
|
||||
fc: ""
|
||||
- p: hello
|
||||
isKey: false
|
||||
hc: ""
|
||||
lc: ""
|
||||
fc: ""
|
||||
- p: hello.message
|
||||
isKey: true
|
||||
hc: ""
|
||||
lc: ""
|
||||
fc: ""
|
||||
- p: hello.message
|
||||
isKey: false
|
||||
hc: ""
|
||||
lc: ""
|
||||
fc: ""
|
||||
`
|
||||
|
||||
var expectedWhereIsMyCommentArray = `D0, P[], (!!seq)::- p: ""
|
||||
isKey: false
|
||||
hc: ""
|
||||
lc: ""
|
||||
fc: ""
|
||||
- p: name
|
||||
isKey: true
|
||||
hc: ""
|
||||
lc: ""
|
||||
fc: ""
|
||||
- p: name
|
||||
isKey: false
|
||||
hc: ""
|
||||
lc: ""
|
||||
fc: ""
|
||||
- p: name.0
|
||||
isKey: false
|
||||
hc: under-name-comment
|
||||
lc: ""
|
||||
fc: ""
|
||||
`
|
||||
|
||||
var commentOperatorScenarios = []expressionScenario{
|
||||
{
|
||||
description: "Set line comment",
|
||||
subdescription: "Set the comment on the key node for more reliability (see below).",
|
||||
document: `a: cat`,
|
||||
expression: `.a line_comment="single"`,
|
||||
description: "Set line comment",
|
||||
document: `a: cat`,
|
||||
expression: `.a line_comment="single"`,
|
||||
expected: []string{
|
||||
"D0, P[], (doc)::a: cat # single\n",
|
||||
},
|
||||
},
|
||||
{
|
||||
description: "Set line comment of a maps/arrays",
|
||||
subdescription: "For maps and arrays, you need to set the line comment on the _key_ node. This will also work for scalars.",
|
||||
document: "a:\n b: things",
|
||||
expression: `(.a | key) line_comment="single"`,
|
||||
expected: []string{
|
||||
"D0, P[], (doc)::a: # single\n b: things\n",
|
||||
},
|
||||
},
|
||||
{
|
||||
skipDoc: true,
|
||||
document: "a: cat\nb: dog",
|
||||
@@ -105,42 +46,6 @@ var commentOperatorScenarios = []expressionScenario{
|
||||
"D0, P[], (!!map)::a: cat # cat\n# cat\n\n# cat\nb: dog # dog\n# dog\n\n# dog\n",
|
||||
},
|
||||
},
|
||||
{
|
||||
description: "Where is the comment - map key example",
|
||||
subdescription: "The underlying yaml parser can assign comments in a document to surprising nodes. Use an expression like this to find where you comment is. 'p' indicates the path, 'isKey' is if the node is a map key (as opposed to a map value).\nFrom this, you can see the 'hello-world-comment' is actually on the 'hello' key",
|
||||
document: "hello: # hello-world-comment\n message: world",
|
||||
expression: `[... | {"p": path | join("."), "isKey": is_key, "hc": headComment, "lc": lineComment, "fc": footComment}]`,
|
||||
expected: []string{
|
||||
expectedWhereIsMyCommentMapKey,
|
||||
},
|
||||
},
|
||||
{
|
||||
description: "Retrieve comment - map key example",
|
||||
subdescription: "From the previous example, we know that the comment is on the 'hello' _key_ as a lineComment",
|
||||
document: "hello: # hello-world-comment\n message: world",
|
||||
expression: `.hello | key | line_comment`,
|
||||
expected: []string{
|
||||
"D0, P[hello], (!!str)::hello-world-comment\n",
|
||||
},
|
||||
},
|
||||
{
|
||||
description: "Where is the comment - array example",
|
||||
subdescription: "The underlying yaml parser can assign comments in a document to surprising nodes. Use an expression like this to find where you comment is. 'p' indicates the path, 'isKey' is if the node is a map key (as opposed to a map value).\nFrom this, you can see the 'under-name-comment' is actually on the first child",
|
||||
document: "name:\n # under-name-comment\n - first-array-child",
|
||||
expression: `[... | {"p": path | join("."), "isKey": is_key, "hc": headComment, "lc": lineComment, "fc": footComment}]`,
|
||||
expected: []string{
|
||||
expectedWhereIsMyCommentArray,
|
||||
},
|
||||
},
|
||||
{
|
||||
description: "Retrieve comment - array example",
|
||||
subdescription: "From the previous example, we know that the comment is on the first child as a headComment",
|
||||
document: "name:\n # under-name-comment\n - first-array-child",
|
||||
expression: `.name[0] | headComment`,
|
||||
expected: []string{
|
||||
"D0, P[name 0], (!!str)::under-name-comment\n",
|
||||
},
|
||||
},
|
||||
{
|
||||
description: "Set head comment",
|
||||
document: `a: cat`,
|
||||
@@ -149,14 +54,6 @@ var commentOperatorScenarios = []expressionScenario{
|
||||
"D0, P[], (doc)::# single\n\na: cat\n",
|
||||
},
|
||||
},
|
||||
{
|
||||
description: "Set head comment of a map entry",
|
||||
document: "f: foo\na:\n b: cat",
|
||||
expression: `(.a | key) head_comment="single"`,
|
||||
expected: []string{
|
||||
"D0, P[], (doc)::f: foo\n# single\na:\n b: cat\n",
|
||||
},
|
||||
},
|
||||
{
|
||||
description: "Set foot comment, using an expression",
|
||||
document: `a: cat`,
|
||||
@@ -165,15 +62,6 @@ var commentOperatorScenarios = []expressionScenario{
|
||||
"D0, P[], (doc)::a: cat\n# cat\n",
|
||||
},
|
||||
},
|
||||
{
|
||||
skipDoc: true,
|
||||
description: "Set foot comment, using an expression",
|
||||
document: "a: cat\n\n# hi",
|
||||
expression: `. foot_comment=""`,
|
||||
expected: []string{
|
||||
"D0, P[], (doc)::a: cat\n",
|
||||
},
|
||||
},
|
||||
{
|
||||
skipDoc: true,
|
||||
document: `a: cat`,
|
||||
@@ -224,24 +112,6 @@ var commentOperatorScenarios = []expressionScenario{
|
||||
"D0, P[], (!!str)::welcome!\n",
|
||||
},
|
||||
},
|
||||
{
|
||||
skipDoc: true,
|
||||
description: "strip trailing comment recurse all",
|
||||
document: "a: cat\n\n# haha",
|
||||
expression: `... comments= ""`,
|
||||
expected: []string{
|
||||
"D0, P[], (!!map)::a: cat\n",
|
||||
},
|
||||
},
|
||||
{
|
||||
skipDoc: true,
|
||||
description: "strip trailing comment recurse values",
|
||||
document: "a: cat\n\n# haha",
|
||||
expression: `.. comments= ""`,
|
||||
expected: []string{
|
||||
"D0, P[], (!!map)::a: cat\n",
|
||||
},
|
||||
},
|
||||
{
|
||||
description: "Head comment with document split",
|
||||
dontFormatInputForDoc: true,
|
||||
|
||||
@@ -72,7 +72,7 @@ func sequenceFor(d *dataTreeNavigator, context Context, matchingNode *CandidateN
|
||||
return &CandidateNode{Node: innerList, Document: document, Path: path}, nil
|
||||
}
|
||||
|
||||
// NOTE: here the document index gets dropped so we
|
||||
//NOTE: here the document index gets dropped so we
|
||||
// no longer know where the node originates from.
|
||||
func listToNodeSeq(list *list.List) *yaml.Node {
|
||||
node := yaml.Node{Kind: yaml.SequenceNode, Tag: "!!seq"}
|
||||
|
||||
@@ -77,7 +77,7 @@ var dateTimeOperatorScenarios = []expressionScenario{
|
||||
},
|
||||
{
|
||||
description: "Date subtraction",
|
||||
subdescription: "You can subtract durations from dates. Assumes RFC3339 date time format, see [date-time operators](https://mikefarah.gitbook.io/yq/operators/datetime#date-time-formattings) for more information.",
|
||||
subdescription: "You can subtract durations from dates. Assumes RFC3339 date time format, see [date-time operators](https://mikefarah.gitbook.io/yq/operators/date-time-operators) for more information.",
|
||||
document: `a: 2021-01-01T03:10:00Z`,
|
||||
expression: `.a -= "3h10m"`,
|
||||
expected: []string{
|
||||
|
||||
@@ -1,7 +1,6 @@
|
||||
package yqlib
|
||||
|
||||
import (
|
||||
"container/list"
|
||||
"fmt"
|
||||
|
||||
yaml "gopkg.in/yaml.v3"
|
||||
@@ -17,22 +16,10 @@ func deleteChildOperator(d *dataTreeNavigator, context Context, expressionNode *
|
||||
for el := nodesToDelete.MatchingNodes.Back(); el != nil; el = el.Prev() {
|
||||
candidate := el.Value.(*CandidateNode)
|
||||
|
||||
if candidate.Node.Kind == yaml.DocumentNode {
|
||||
//need to delete this node from context.
|
||||
newResults := list.New()
|
||||
for item := context.MatchingNodes.Front(); item != nil; item = item.Next() {
|
||||
nodeInContext := item.Value.(*CandidateNode)
|
||||
if nodeInContext.Node != candidate.Node {
|
||||
newResults.PushBack(nodeInContext)
|
||||
} else {
|
||||
log.Info("Need to delete this %v", NodeToString(nodeInContext))
|
||||
}
|
||||
}
|
||||
return context.ChildContext(newResults), nil
|
||||
} else if candidate.Parent == nil {
|
||||
//problem: context may already be '.a' and then I pass in '.a.a2'.
|
||||
// should pass in .a2.
|
||||
log.Info("Could not find parent of %v", NodeToString(candidate))
|
||||
//problem: context may already be '.a' and then I pass in '.a.a2'.
|
||||
// should pass in .a2.
|
||||
if candidate.Parent == nil {
|
||||
log.Info("Could not find parent of %v", candidate.GetKey())
|
||||
return context, nil
|
||||
}
|
||||
|
||||
|
||||
@@ -29,16 +29,6 @@ var deleteOperatorScenarios = []expressionScenario{
|
||||
"D0, P[a], (!!map)::{a2: frood}\n",
|
||||
},
|
||||
},
|
||||
{
|
||||
skipDoc: true,
|
||||
description: "delete whole document",
|
||||
document2: `a: slow`,
|
||||
document: `a: fast`,
|
||||
expression: `del(select(.a == "fast"))`,
|
||||
expected: []string{
|
||||
"D0, P[], (doc)::a: slow\n",
|
||||
},
|
||||
},
|
||||
{
|
||||
skipDoc: true,
|
||||
document: `a: [1,2,3]`,
|
||||
|
||||
@@ -81,8 +81,8 @@ var encoderDecoderOperatorScenarios = []expressionScenario{
|
||||
},
|
||||
{
|
||||
description: "Decode tsv encoded string",
|
||||
document: `a: "cats dogs\ngreat cool as well"`,
|
||||
expression: `.a |= @tsvd`,
|
||||
document: `a: "cats dogs\ngreat cool as well"`,
|
||||
expression: `.a |= @tsvd`,
|
||||
expected: []string{
|
||||
"D0, P[], (doc)::a:\n - cats: great\n dogs: cool as well\n",
|
||||
},
|
||||
|
||||
@@ -52,15 +52,6 @@ var entriesOperatorScenarios = []expressionScenario{
|
||||
"D0, P[], (!!map)::KEY_a: 1\nKEY_b: 2\n",
|
||||
},
|
||||
},
|
||||
{
|
||||
description: "Custom sort map keys",
|
||||
subdescription: "Use to_entries to convert to an array of key/value pairs, sort the array using sort/sort_by/etc, and convert it back.",
|
||||
document: `{a: 1, c: 3, b: 2}`,
|
||||
expression: `to_entries | sort_by(.key) | reverse | from_entries`,
|
||||
expected: []string{
|
||||
"D0, P[], (!!map)::c: 3\nb: 2\na: 1\n",
|
||||
},
|
||||
},
|
||||
{
|
||||
skipDoc: true,
|
||||
document: `{a: 1, b: 2}`,
|
||||
|
||||
@@ -7,20 +7,6 @@ import (
|
||||
"gopkg.in/yaml.v3"
|
||||
)
|
||||
|
||||
func isKeyOperator(d *dataTreeNavigator, context Context, expressionNode *ExpressionNode) (Context, error) {
|
||||
log.Debugf("-- isKeyOperator")
|
||||
|
||||
var results = list.New()
|
||||
|
||||
for el := context.MatchingNodes.Front(); el != nil; el = el.Next() {
|
||||
candidate := el.Value.(*CandidateNode)
|
||||
|
||||
results.PushBack(createBooleanCandidate(candidate, candidate.IsMapKey))
|
||||
}
|
||||
|
||||
return context.ChildContext(results), nil
|
||||
}
|
||||
|
||||
func getKeyOperator(d *dataTreeNavigator, context Context, expressionNode *ExpressionNode) (Context, error) {
|
||||
log.Debugf("-- getKeyOperator")
|
||||
|
||||
|
||||
@@ -4,32 +4,6 @@ import (
|
||||
"testing"
|
||||
)
|
||||
|
||||
var expectedIsKey = `D0, P[], (!!seq)::- p: ""
|
||||
isKey: false
|
||||
tag: '!!map'
|
||||
- p: a
|
||||
isKey: true
|
||||
tag: '!!str'
|
||||
- p: a
|
||||
isKey: false
|
||||
tag: '!!map'
|
||||
- p: a.b
|
||||
isKey: true
|
||||
tag: '!!str'
|
||||
- p: a.b
|
||||
isKey: false
|
||||
tag: '!!seq'
|
||||
- p: a.b.0
|
||||
isKey: false
|
||||
tag: '!!str'
|
||||
- p: a.c
|
||||
isKey: true
|
||||
tag: '!!str'
|
||||
- p: a.c
|
||||
isKey: false
|
||||
tag: '!!str'
|
||||
`
|
||||
|
||||
var keysOperatorScenarios = []expressionScenario{
|
||||
{
|
||||
description: "Map keys",
|
||||
@@ -101,14 +75,6 @@ var keysOperatorScenarios = []expressionScenario{
|
||||
"D0, P[a x], (!!str)::comment on key\n",
|
||||
},
|
||||
},
|
||||
{
|
||||
description: "Check node is a key",
|
||||
document: "a: \n b: [cat]\n c: frog\n",
|
||||
expression: `[... | { "p": path | join("."), "isKey": is_key, "tag": tag }]`,
|
||||
expected: []string{
|
||||
expectedIsKey,
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
func TestKeysOperatorScenarios(t *testing.T) {
|
||||
|
||||
@@ -523,35 +523,6 @@ var multiplyOperatorScenarios = []expressionScenario{
|
||||
"D0, P[], (doc)::a: !horse {cat: meow, dog: woof}\nb: !goat {dog: woof}\n",
|
||||
},
|
||||
},
|
||||
{
|
||||
description: "Custom types: clobber tags",
|
||||
subdescription: "Use the `c` option to clobber custom tags. Note that the second tag is now used",
|
||||
document: "a: !horse {cat: meow}\nb: !goat {dog: woof}",
|
||||
expression: ".a *=c .b",
|
||||
expected: []string{
|
||||
"D0, P[], (doc)::a: !goat {cat: meow, dog: woof}\nb: !goat {dog: woof}\n",
|
||||
},
|
||||
},
|
||||
{
|
||||
skipDoc: true,
|
||||
description: "Custom types: clobber tags - *=",
|
||||
subdescription: "Use the `c` option to clobber custom tags - on both the `=` and `*` operator. Note that the second tag is now used",
|
||||
document: "a: !horse {cat: meow}\nb: !goat {dog: woof}",
|
||||
expression: ".a =c .a *c .b",
|
||||
expected: []string{
|
||||
"D0, P[], (doc)::a: !goat {cat: meow, dog: woof}\nb: !goat {dog: woof}\n",
|
||||
},
|
||||
},
|
||||
{
|
||||
skipDoc: true,
|
||||
description: "Custom types: dont clobber tags - *=",
|
||||
subdescription: "Use the `c` option to clobber custom tags - on both the `=` and `*` operator. Note that the second tag is now used",
|
||||
document: "a: !horse {cat: meow}\nb: !goat {dog: woof}",
|
||||
expression: ".a *= .b",
|
||||
expected: []string{
|
||||
"D0, P[], (doc)::a: !horse {cat: meow, dog: woof}\nb: !goat {dog: woof}\n",
|
||||
},
|
||||
},
|
||||
{
|
||||
skipDoc: true,
|
||||
description: "Custom types: that are really maps",
|
||||
@@ -561,24 +532,6 @@ var multiplyOperatorScenarios = []expressionScenario{
|
||||
"D0, P[], (doc)::a: {cat: !horse 5}\nb: {cat: 5}\n",
|
||||
},
|
||||
},
|
||||
{
|
||||
skipDoc: true,
|
||||
description: "Relative merge, new fields only",
|
||||
document: "a: {a: original}\n",
|
||||
expression: `.a *=n load("../../examples/thing.yml")`,
|
||||
expected: []string{
|
||||
"D0, P[], (doc)::a: {a: original, b: cool.}\n",
|
||||
},
|
||||
},
|
||||
{
|
||||
skipDoc: true,
|
||||
description: "Relative merge",
|
||||
document: "a: {a: original}\n",
|
||||
expression: `.a *= load("../../examples/thing.yml")`,
|
||||
expected: []string{
|
||||
"D0, P[], (doc)::a: {a: apple is included, b: cool.}\n",
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
func TestMultiplyOperatorScenarios(t *testing.T) {
|
||||
|
||||
@@ -16,139 +16,6 @@ func createPathNodeFor(pathElement interface{}) *yaml.Node {
|
||||
}
|
||||
}
|
||||
|
||||
func getPathArrayFromNode(funcName string, node *yaml.Node) ([]interface{}, error) {
|
||||
if node.Kind != yaml.SequenceNode {
|
||||
return nil, fmt.Errorf("%v: expected path array, but got %v instead", funcName, node.Tag)
|
||||
}
|
||||
|
||||
path := make([]interface{}, len(node.Content))
|
||||
|
||||
for i, childNode := range node.Content {
|
||||
if childNode.Tag == "!!str" {
|
||||
path[i] = childNode.Value
|
||||
} else if childNode.Tag == "!!int" {
|
||||
number, err := parseInt(childNode.Value)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%v: could not parse %v as an int: %w", funcName, childNode.Value, err)
|
||||
}
|
||||
path[i] = number
|
||||
} else {
|
||||
return nil, fmt.Errorf("%v: expected either a !!str or !!int in the path, found %v instead", funcName, childNode.Tag)
|
||||
}
|
||||
|
||||
}
|
||||
return path, nil
|
||||
}
|
||||
|
||||
// SETPATH(pathArray; value)
|
||||
func setPathOperator(d *dataTreeNavigator, context Context, expressionNode *ExpressionNode) (Context, error) {
|
||||
log.Debugf("SetPath")
|
||||
|
||||
if expressionNode.RHS.Operation.OperationType != blockOpType {
|
||||
return Context{}, fmt.Errorf("SETPATH must be given a block (;), got %v instead", expressionNode.RHS.Operation.OperationType.Type)
|
||||
}
|
||||
|
||||
lhsPathContext, err := d.GetMatchingNodes(context.ReadOnlyClone(), expressionNode.RHS.LHS)
|
||||
|
||||
if err != nil {
|
||||
return Context{}, err
|
||||
}
|
||||
|
||||
if lhsPathContext.MatchingNodes.Len() != 1 {
|
||||
return Context{}, fmt.Errorf("SETPATH: expected single path but found %v results instead", lhsPathContext.MatchingNodes.Len())
|
||||
}
|
||||
lhsValue := lhsPathContext.MatchingNodes.Front().Value.(*CandidateNode)
|
||||
|
||||
lhsPath, err := getPathArrayFromNode("SETPATH", lhsValue.Node)
|
||||
|
||||
if err != nil {
|
||||
return Context{}, err
|
||||
}
|
||||
|
||||
lhsTraversalTree := createTraversalTree(lhsPath, traversePreferences{}, false)
|
||||
|
||||
assignmentOp := &Operation{OperationType: assignOpType}
|
||||
|
||||
for el := context.MatchingNodes.Front(); el != nil; el = el.Next() {
|
||||
candidate := el.Value.(*CandidateNode)
|
||||
|
||||
targetContextValue, err := d.GetMatchingNodes(context.SingleReadonlyChildContext(candidate), expressionNode.RHS.RHS)
|
||||
if err != nil {
|
||||
return Context{}, err
|
||||
}
|
||||
|
||||
if targetContextValue.MatchingNodes.Len() != 1 {
|
||||
return Context{}, fmt.Errorf("SETPATH: expected single value on RHS but found %v", targetContextValue.MatchingNodes.Len())
|
||||
}
|
||||
|
||||
rhsOp := &Operation{OperationType: valueOpType, CandidateNode: targetContextValue.MatchingNodes.Front().Value.(*CandidateNode)}
|
||||
|
||||
assignmentOpNode := &ExpressionNode{
|
||||
Operation: assignmentOp,
|
||||
LHS: lhsTraversalTree,
|
||||
RHS: &ExpressionNode{Operation: rhsOp},
|
||||
}
|
||||
|
||||
_, err = d.GetMatchingNodes(context.SingleChildContext(candidate), assignmentOpNode)
|
||||
|
||||
if err != nil {
|
||||
return Context{}, err
|
||||
}
|
||||
|
||||
}
|
||||
return context, nil
|
||||
}
|
||||
|
||||
func delPathsOperator(d *dataTreeNavigator, context Context, expressionNode *ExpressionNode) (Context, error) {
|
||||
log.Debugf("delPaths")
|
||||
// single RHS expression that returns an array of paths (array of arrays)
|
||||
|
||||
pathArraysContext, err := d.GetMatchingNodes(context.ReadOnlyClone(), expressionNode.RHS)
|
||||
if err != nil {
|
||||
return Context{}, err
|
||||
}
|
||||
if pathArraysContext.MatchingNodes.Len() != 1 {
|
||||
return Context{}, fmt.Errorf("DELPATHS: expected single value but found %v", pathArraysContext.MatchingNodes.Len())
|
||||
}
|
||||
pathArraysNode := pathArraysContext.MatchingNodes.Front().Value.(*CandidateNode).Node
|
||||
|
||||
if pathArraysNode.Tag != "!!seq" {
|
||||
return Context{}, fmt.Errorf("DELPATHS: expected a sequence of sequences, but found %v", pathArraysNode.Tag)
|
||||
}
|
||||
|
||||
updatedContext := context
|
||||
|
||||
for i, child := range pathArraysNode.Content {
|
||||
|
||||
if child.Tag != "!!seq" {
|
||||
return Context{}, fmt.Errorf("DELPATHS: expected entry [%v] to be a sequence, but its a %v. Note that delpaths takes an array of path arrays, e.g. [[\"a\", \"b\"]]", i, child.Tag)
|
||||
}
|
||||
childPath, err := getPathArrayFromNode("DELPATHS", child)
|
||||
|
||||
if err != nil {
|
||||
return Context{}, err
|
||||
}
|
||||
|
||||
childTraversalExp := createTraversalTree(childPath, traversePreferences{}, false)
|
||||
deleteChildOp := &Operation{OperationType: deleteChildOpType}
|
||||
|
||||
deleteChildOpNode := &ExpressionNode{
|
||||
Operation: deleteChildOp,
|
||||
RHS: childTraversalExp,
|
||||
}
|
||||
|
||||
updatedContext, err = d.GetMatchingNodes(updatedContext, deleteChildOpNode)
|
||||
|
||||
if err != nil {
|
||||
return Context{}, err
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
return updatedContext, nil
|
||||
|
||||
}
|
||||
|
||||
func getPathOperator(d *dataTreeNavigator, context Context, expressionNode *ExpressionNode) (Context, error) {
|
||||
log.Debugf("GetPath")
|
||||
|
||||
|
||||
@@ -63,60 +63,6 @@ var pathOperatorScenarios = []expressionScenario{
|
||||
"D0, P[a 2], (!!seq)::- path:\n - a\n - 2\n value: frog\n",
|
||||
},
|
||||
},
|
||||
{
|
||||
description: "Set path",
|
||||
document: `{a: {b: cat}}`,
|
||||
expression: `setpath(["a", "b"]; "things")`,
|
||||
expected: []string{
|
||||
"D0, P[], (doc)::{a: {b: things}}\n",
|
||||
},
|
||||
},
|
||||
{
|
||||
description: "Set on empty document",
|
||||
expression: `setpath(["a", "b"]; "things")`,
|
||||
expected: []string{
|
||||
"D0, P[], ()::a:\n b: things\n",
|
||||
},
|
||||
},
|
||||
{
|
||||
description: "Set array path",
|
||||
document: `a: [cat, frog]`,
|
||||
expression: `setpath(["a", 0]; "things")`,
|
||||
expected: []string{
|
||||
"D0, P[], (doc)::a: [things, frog]\n",
|
||||
},
|
||||
},
|
||||
{
|
||||
description: "Set array path empty",
|
||||
expression: `setpath(["a", 0]; "things")`,
|
||||
expected: []string{
|
||||
"D0, P[], ()::a:\n - things\n",
|
||||
},
|
||||
},
|
||||
{
|
||||
description: "Delete path",
|
||||
subdescription: "Notice delpaths takes an _array_ of paths.",
|
||||
document: `{a: {b: cat, c: dog, d: frog}}`,
|
||||
expression: `delpaths([["a", "c"], ["a", "d"]])`,
|
||||
expected: []string{
|
||||
"D0, P[], (doc)::{a: {b: cat}}\n",
|
||||
},
|
||||
},
|
||||
{
|
||||
description: "Delete array path",
|
||||
document: `a: [cat, frog]`,
|
||||
expression: `delpaths([["a", 0]])`,
|
||||
expected: []string{
|
||||
"D0, P[], (doc)::a: [frog]\n",
|
||||
},
|
||||
},
|
||||
{
|
||||
description: "Delete - wrong parameter",
|
||||
subdescription: "delpaths does not work with a single path array",
|
||||
document: `a: [cat, frog]`,
|
||||
expression: `delpaths(["a", 0])`,
|
||||
expectedError: "DELPATHS: expected entry [0] to be a sequence, but its a !!str. Note that delpaths takes an array of path arrays, e.g. [[\"a\", \"b\"]]",
|
||||
},
|
||||
}
|
||||
|
||||
func TestPathOperatorsScenarios(t *testing.T) {
|
||||
|
||||
@@ -17,7 +17,7 @@ func sortOperator(d *dataTreeNavigator, context Context, expressionNode *Express
|
||||
}
|
||||
|
||||
// context represents the current matching nodes in the expression pipeline
|
||||
// expressionNode is your current expression (sort_by)
|
||||
//expressionNode is your current expression (sort_by)
|
||||
func sortByOperator(d *dataTreeNavigator, context Context, expressionNode *ExpressionNode) (Context, error) {
|
||||
|
||||
results := list.New()
|
||||
|
||||
@@ -13,26 +13,6 @@ type changeCasePrefs struct {
|
||||
ToUpperCase bool
|
||||
}
|
||||
|
||||
func trimSpaceOperator(d *dataTreeNavigator, context Context, expressionNode *ExpressionNode) (Context, error) {
|
||||
results := list.New()
|
||||
for el := context.MatchingNodes.Front(); el != nil; el = el.Next() {
|
||||
candidate := el.Value.(*CandidateNode)
|
||||
|
||||
node := unwrapDoc(candidate.Node)
|
||||
|
||||
if guessTagFromCustomType(node) != "!!str" {
|
||||
return Context{}, fmt.Errorf("cannot trim %v, can only operate on strings. ", node.Tag)
|
||||
}
|
||||
|
||||
newStringNode := &yaml.Node{Kind: yaml.ScalarNode, Tag: node.Tag, Style: node.Style}
|
||||
newStringNode.Value = strings.TrimSpace(node.Value)
|
||||
results.PushBack(candidate.CreateReplacement(newStringNode))
|
||||
|
||||
}
|
||||
|
||||
return context.ChildContext(results), nil
|
||||
}
|
||||
|
||||
func changeCaseOperator(d *dataTreeNavigator, context Context, expressionNode *ExpressionNode) (Context, error) {
|
||||
results := list.New()
|
||||
prefs := expressionNode.Operation.Preferences.(changeCasePrefs)
|
||||
|
||||
@@ -47,17 +47,6 @@ var stringsOperatorScenarios = []expressionScenario{
|
||||
"D0, P[], (!!str)::cat; meow; 1; ; true\n",
|
||||
},
|
||||
},
|
||||
{
|
||||
description: "Trim strings",
|
||||
document: `[" cat", "dog ", " cow cow ", horse]`,
|
||||
expression: `.[] | trim`,
|
||||
expected: []string{
|
||||
"D0, P[0], (!!str)::cat\n",
|
||||
"D0, P[1], (!!str)::dog\n",
|
||||
"D0, P[2], (!!str)::cow cow\n",
|
||||
"D0, P[3], (!!str)::horse\n",
|
||||
},
|
||||
},
|
||||
{
|
||||
skipDoc: true,
|
||||
document: `[!horse cat, !goat meow, !frog 1, null, true]`,
|
||||
|
||||
@@ -7,7 +7,7 @@ func withOperator(d *dataTreeNavigator, context Context, expressionNode *Express
|
||||
// with(path, exp)
|
||||
|
||||
if expressionNode.RHS.Operation.OperationType != blockOpType {
|
||||
return Context{}, fmt.Errorf("with must be given a block (;), got %v instead", expressionNode.RHS.Operation.OperationType.Type)
|
||||
return Context{}, fmt.Errorf("with must be given a block, got %v instead", expressionNode.RHS.Operation.OperationType.Type)
|
||||
}
|
||||
|
||||
pathExp := expressionNode.RHS.LHS
|
||||
|
||||
+2
-19
@@ -5,7 +5,6 @@ import (
|
||||
"fmt"
|
||||
|
||||
"github.com/jinzhu/copier"
|
||||
logging "gopkg.in/op/go-logging.v1"
|
||||
"gopkg.in/yaml.v3"
|
||||
)
|
||||
|
||||
@@ -19,21 +18,7 @@ func compoundAssignFunction(d *dataTreeNavigator, context Context, expressionNod
|
||||
return Context{}, err
|
||||
}
|
||||
|
||||
// tricky logic when we are running *= with flags.
|
||||
// we have an op like: .a *=nc .b
|
||||
// which should roughly translate to .a =c .a *nc .b
|
||||
// note that the 'n' flag only applies to the multiple op, not the assignment
|
||||
// but the clobber flag applies to both!
|
||||
|
||||
prefs := assignPreferences{}
|
||||
switch typedPref := expressionNode.Operation.Preferences.(type) {
|
||||
case assignPreferences:
|
||||
prefs = typedPref
|
||||
case multiplyPreferences:
|
||||
prefs.ClobberCustomTags = typedPref.AssignPrefs.ClobberCustomTags
|
||||
}
|
||||
|
||||
assignmentOp := &Operation{OperationType: assignOpType, Preferences: prefs}
|
||||
assignmentOp := &Operation{OperationType: assignOpType, Preferences: expressionNode.Operation.Preferences}
|
||||
|
||||
for el := lhs.MatchingNodes.Front(); el != nil; el = el.Next() {
|
||||
candidate := el.Value.(*CandidateNode)
|
||||
@@ -98,10 +83,8 @@ func resultsForRHS(d *dataTreeNavigator, context Context, lhsCandidate *Candidat
|
||||
}
|
||||
|
||||
for rightEl := rhs.MatchingNodes.Front(); rightEl != nil; rightEl = rightEl.Next() {
|
||||
log.Debugf("Applying calc")
|
||||
rhsCandidate := rightEl.Value.(*CandidateNode)
|
||||
if !log.IsEnabledFor(logging.DEBUG) {
|
||||
log.Debugf("Applying lhs: %v, rhsCandidate, %v", NodeToString(lhsCandidate), NodeToString(rhsCandidate))
|
||||
}
|
||||
resultCandidate, err := prefs.Calculation(d, context, lhsCandidate, rhsCandidate)
|
||||
if err != nil {
|
||||
return err
|
||||
|
||||
@@ -100,11 +100,7 @@ func testScenario(t *testing.T, s *expressionScenario) {
|
||||
context, err := NewDataTreeNavigator().GetMatchingNodes(Context{MatchingNodes: inputs}, node)
|
||||
|
||||
if s.expectedError != "" {
|
||||
if err == nil {
|
||||
t.Errorf("Expected error '%v' but it worked!", s.expectedError)
|
||||
} else {
|
||||
test.AssertResultComplexWithContext(t, s.expectedError, err.Error(), fmt.Sprintf("desc: %v\nexp: %v\ndoc: %v", s.description, s.expression, s.document))
|
||||
}
|
||||
test.AssertResultComplexWithContext(t, s.expectedError, err.Error(), fmt.Sprintf("desc: %v\nexp: %v\ndoc: %v", s.description, s.expression, s.document))
|
||||
return
|
||||
}
|
||||
|
||||
|
||||
@@ -110,7 +110,7 @@ func (s *streamEvaluator) Evaluate(filename string, reader io.Reader, node *Expr
|
||||
Node: &dataBucket,
|
||||
FileIndex: s.fileIndex,
|
||||
}
|
||||
// move document comments into candidate node
|
||||
//move document comments into candidate node
|
||||
// otherwise unwrap drops them.
|
||||
candidateNode.TrailingContent = dataBucket.FootComment
|
||||
dataBucket.FootComment = ""
|
||||
|
||||
@@ -1,85 +0,0 @@
|
||||
package yqlib
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"container/list"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
|
||||
yaml "gopkg.in/yaml.v3"
|
||||
)
|
||||
|
||||
type StringEvaluator interface {
|
||||
Evaluate(expression string, input string, encoder Encoder, leadingContentPreProcessing bool, decoder Decoder) (string, error)
|
||||
}
|
||||
|
||||
type stringEvaluator struct {
|
||||
treeNavigator DataTreeNavigator
|
||||
fileIndex int
|
||||
}
|
||||
|
||||
func NewStringEvaluator() StringEvaluator {
|
||||
return &stringEvaluator{
|
||||
treeNavigator: NewDataTreeNavigator(),
|
||||
}
|
||||
}
|
||||
|
||||
func (s *stringEvaluator) Evaluate(expression string, input string, encoder Encoder, leadingContentPreProcessing bool, decoder Decoder) (string, error) {
|
||||
|
||||
// Use bytes.Buffer for output of string
|
||||
out := new(bytes.Buffer)
|
||||
printer := NewPrinter(encoder, NewSinglePrinterWriter(out))
|
||||
|
||||
InitExpressionParser()
|
||||
node, err := ExpressionParser.ParseExpression(expression)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
reader, leadingContent, err := readString(input, leadingContentPreProcessing)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
var currentIndex uint
|
||||
decoder.Init(reader)
|
||||
for {
|
||||
var dataBucket yaml.Node
|
||||
errorReading := decoder.Decode(&dataBucket)
|
||||
|
||||
if errors.Is(errorReading, io.EOF) {
|
||||
s.fileIndex = s.fileIndex + 1
|
||||
return out.String(), nil
|
||||
} else if errorReading != nil {
|
||||
return "", fmt.Errorf("bad input '%v': %w", input, errorReading)
|
||||
}
|
||||
|
||||
candidateNode := &CandidateNode{
|
||||
Document: currentIndex,
|
||||
Node: &dataBucket,
|
||||
FileIndex: s.fileIndex,
|
||||
}
|
||||
// move document comments into candidate node
|
||||
// otherwise unwrap drops them.
|
||||
candidateNode.TrailingContent = dataBucket.FootComment
|
||||
dataBucket.FootComment = ""
|
||||
|
||||
if currentIndex == 0 {
|
||||
candidateNode.LeadingContent = leadingContent
|
||||
}
|
||||
inputList := list.New()
|
||||
inputList.PushBack(candidateNode)
|
||||
|
||||
result, errorParsing := s.treeNavigator.GetMatchingNodes(Context{MatchingNodes: inputList}, node)
|
||||
if errorParsing != nil {
|
||||
return "", errorParsing
|
||||
}
|
||||
err = printer.PrintResults(result.MatchingNodes)
|
||||
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
currentIndex = currentIndex + 1
|
||||
}
|
||||
}
|
||||
@@ -1,30 +0,0 @@
|
||||
package yqlib
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/mikefarah/yq/v4/test"
|
||||
)
|
||||
|
||||
func TestStringEvaluator_Evaluate_Nominal(t *testing.T) {
|
||||
expected_output := `` +
|
||||
`yq` + "\n" +
|
||||
`---` + "\n" +
|
||||
`jq` + "\n"
|
||||
expression := ".[].name"
|
||||
input := `` +
|
||||
` - name: yq` + "\n" +
|
||||
` description: yq is a portable command-line YAML, JSON and XML processor` + "\n" +
|
||||
`---` + "\n" +
|
||||
` - name: jq` + "\n" +
|
||||
` description: Command-line JSON processor` + "\n"
|
||||
encoder := NewYamlEncoder(2, true, true, true)
|
||||
decoder := NewYamlDecoder()
|
||||
|
||||
result, err := NewStringEvaluator().Evaluate(expression, input, encoder, true, decoder)
|
||||
if err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
|
||||
test.AssertResult(t, expected_output, result)
|
||||
}
|
||||
@@ -33,14 +33,6 @@ func readStream(filename string, leadingContentPreProcessing bool) (io.Reader, s
|
||||
return processReadStream(reader)
|
||||
}
|
||||
|
||||
func readString(input string, leadingContentPreProcessing bool) (io.Reader, string, error) {
|
||||
reader := bufio.NewReader(strings.NewReader(input))
|
||||
if !leadingContentPreProcessing {
|
||||
return reader, "", nil
|
||||
}
|
||||
return processReadStream(reader)
|
||||
}
|
||||
|
||||
func writeString(writer io.Writer, txt string) error {
|
||||
_, errorWriting := writer.Write([]byte(txt))
|
||||
return errorWriting
|
||||
|
||||
@@ -2,6 +2,10 @@
|
||||
- increment version in version.go
|
||||
- increment version in snapcraft.yaml
|
||||
- make sure local build passes
|
||||
- check manpage
|
||||
- ./scripts/generate-man-page-md.sh
|
||||
- ./scripts/generate-man-page.sh
|
||||
- man ./yq.1
|
||||
- run ./scripts/copy-docs.sh (and commit the changed in the yq-book branch)
|
||||
- commit version update changes
|
||||
- tag git with same version number
|
||||
|
||||
@@ -1,29 +1,5 @@
|
||||
4.28.1:
|
||||
- Added `setpath` and `delpaths` operators, like jq (#1374)
|
||||
- Added `is_key` operator, to check if a match was a key when recursing
|
||||
- Added validation when attempting to add sequences to maps (#1341)
|
||||
|
||||
4.27.5:
|
||||
- Fixed relative merge bug #1333
|
||||
|
||||
4.27.4:
|
||||
- Fixed bug in alternative (//) operator, RHS being evaluated when it didn't need to be
|
||||
- Fixed footer comment issue #1231
|
||||
- Github action now runs as root (as recommended by Github Actions doc)
|
||||
- Updated dependencies
|
||||
|
||||
4.27.3:
|
||||
- Added new 'c' merge and assign flag that clobbers custom tags
|
||||
- Bumped go dependency to fix CVE (#1316)
|
||||
- Updated dependencies
|
||||
|
||||
4.27.2:
|
||||
- Fixed JSON decoder to maintain object key order.
|
||||
|
||||
4.27.1:
|
||||
- Added 'json' decoder for support for multiple JSON documents in a single file (e.g. NDJSON)
|
||||
- Added 'csv' decoding, array of objects encoding, and round-triping
|
||||
- New StringEvaluator when using yq as a lib (thanks @leviliangtw)
|
||||
- Fixed XML decoding issue (#1284)
|
||||
|
||||
4.26.1:
|
||||
|
||||
+2
-2
@@ -1,5 +1,5 @@
|
||||
#!/bin/sh
|
||||
set -ex
|
||||
go mod download golang.org/x/tools@latest
|
||||
curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(go env GOPATH)/bin v1.49.0
|
||||
wget -O- -nv https://raw.githubusercontent.com/securego/gosec/master/install.sh | sh -s v2.13.1
|
||||
curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(go env GOPATH)/bin v1.45.2
|
||||
wget -O- -nv https://raw.githubusercontent.com/securego/gosec/master/install.sh | sh -s v2.11.0
|
||||
|
||||
+2
-2
@@ -1,5 +1,5 @@
|
||||
name: yq
|
||||
version: '4.28.1'
|
||||
version: '4.26.1'
|
||||
summary: A lightweight and portable command-line YAML processor
|
||||
description: |
|
||||
The aim of the project is to be the jq or sed of yaml files.
|
||||
@@ -16,7 +16,7 @@ apps:
|
||||
parts:
|
||||
yq:
|
||||
plugin: go
|
||||
go-channel: 1.19/stable
|
||||
go-channel: 1.18/stable
|
||||
source: .
|
||||
source-type: git
|
||||
go-importpath: github.com/mikefarah/yq
|
||||
|
||||
Reference in New Issue
Block a user