Bumping version

README.md

@@ -7,12 +7,39 @@ a lightweight and portable command-line YAML processor. `yq` uses [jq](https://g
 
 yq is written in go - so you can download a dependency free binary for your platform and you are good to go! If you prefer there are a variety of package managers that can be used as well as docker, all listed below.
 
-## V4 released!
-V4 is now officially released, it's quite different from V3 (sorry for the migration), however it is much more similar to ```jq```, using a similar expression syntax and therefore support much more complex functionality! 
+## Quick Usage Guide
 
-If you've been using v3 and want/need to upgrade, checkout the [upgrade guide](https://mikefarah.gitbook.io/yq/v/v4.x/upgrading-from-v3).
+Read a value:
 
-Support for v3 will cease August 2021, until then, critical bug and security fixes will still get applied if required.
+```bash
+yq e '.a.b[0].c' file.yaml
+```
+
+Update a yaml file, inplace
+```bash
+yq e -i '.a.b[0].c = "cool"' file.yaml
+```
+
+Update using environment variables
+```bash
+NAME=mike yq e -i '.a.b[0].c = strenv(NAME)' file.yaml
+```
+
+Merge multiple files
+```
+yq ea '. as $item ireduce ({}; . * $item )' path/to/*.yml
+```
+
+Multiple updates to a yaml file
+```bash
+yq e -i '
+  .a.b[0].c = "cool" |
+  .x.y.z = "foobar" |
+  .person.name = strenv(NAME)
+' file.yaml
+```
+
+See the [documentation](https://mikefarah.gitbook.io/yq/) for more.
 
 ## Install
 
@@ -87,6 +114,14 @@ rm /etc/myfile.tmp
 docker run --rm -v "${PWD}":/workdir mikefarah/yq <command> [flags] [expression ]FILE...
 ```
 
+#### Pipe in via STDIN:
+
+You'll need to pass the `-i\--interactive` flag to docker:
+
+```bash
+cat myfile.yml | docker run -i --rm mikefarah/yq e . -
+```
+
 #### Run commands interactively:
 
 ```bash
@@ -101,6 +136,7 @@ yq() {
 }
 ```
 
+
 #### Running as root:
 
 `yq`'s docker image no longer runs under root (https://github.com/mikefarah/yq/pull/860). If you'd like to install more things in the docker image, or you're having permissions issues when attempting to read/write files you'll need to either:
@@ -120,6 +156,16 @@ RUN apk add bash
 USER yq
 ```
 
+### GitHub Action
+```
+  - name: Set foobar to cool
+    uses: mikefarah/yq@master
+    with:
+      cmd: yq eval -i '.foo.bar = "cool"' 'config.yml'
+```
+
+See https://mikefarah.gitbook.io/yq/usage/github-action for more.
+
 ### Go Get:
 ```
 GO111MODULE=on go get github.com/mikefarah/yq/v4
@@ -184,7 +230,8 @@ Supported by @rmescandon (https://launchpad.net/~rmescandon/+archive/ubuntu/yq)
 - [Update yaml inplace](https://mikefarah.gitbook.io/yq/v/v4.x/commands/evaluate#flags)
 - [Complex expressions to select and update](https://mikefarah.gitbook.io/yq/operators/select#select-and-update-matching-values-in-map)
 - Keeps yaml formatting and comments when updating (though there are issues with whitespace)
-- [Convert to/from json to yaml](https://mikefarah.gitbook.io/yq/v/v4.x/usage/convert)
+- [Convert to/from json](https://mikefarah.gitbook.io/yq/v/v4.x/usage/convert)
+- [Convert to properties](https://mikefarah.gitbook.io/yq/v/v4.x/usage/properties)
 - [Pipe data in by using '-'](https://mikefarah.gitbook.io/yq/v/v4.x/commands/evaluate)
 - [General shell completion scripts (bash/zsh/fish/powershell)](https://mikefarah.gitbook.io/yq/v/v4.x/commands/shell-completion)
 - [Reduce](https://mikefarah.gitbook.io/yq/operators/reduce) to merge multiple files or sum an array or other fancy things.
@@ -223,13 +270,6 @@ Flags:
 
 Use "yq [command] --help" for more information about a command.
 ```
-
-Simple Example:
-
-```bash
-yq e '.a.b | length' f1.yml f2.yml 
-```
-
 ## Known Issues / Missing Features
 - `yq` attempts to preserve comment positions and whitespace as much as possible, but it does not handle all scenarios (see https://github.com/go-yaml/yaml/tree/v3 for details)
 - Powershell has its own...opinions: https://mikefarah.gitbook.io/yq/usage/tips-and-tricks#quotes-in-windows-powershell

acceptance_tests/empty.sh

@@ -8,7 +8,27 @@ EOL
 
 testEmptyEval() {
   X=$(./yq e test.yml)
+  expected=$(cat test.yml)
   assertEquals 0 $?
+  assertEquals "$expected" "$X"
+}
+
+testEmptyEvalNoNewLine() {
+  echo -n "#comment" >test.yml
+  X=$(./yq e test.yml)
+  expected=$(cat test.yml)
+  assertEquals 0 $?
+  assertEquals "$expected" "$X"
+}
+
+testEmptyEvalNoNewLineWithExpression() {
+  echo -n "# comment" >test.yml
+  X=$(./yq e '.apple = "tree"' test.yml)
+  read -r -d '' expected << EOM
+# comment
+apple: tree
+EOM
+  assertEquals "$expected" "$X"
 }
 
 testEmptyEvalPipe() {

acceptance_tests/leading-seperator.sh

@@ -36,6 +36,73 @@ testLeadingSeperatorPipeIntoEvalSeq() {
   assertEquals "$expected" "$X"
 }
 
+testLeadingSeperatorExtractField() {
+  X=$(./yq e '.a' - < test.yml)
+  assertEquals "test" "$X"
+}
+
+testLeadingSeperatorExtractFieldWithCommentsAfterSep() {
+  cat >test.yml <<EOL
+---
+# hi peeps
+# cool
+a: test
+EOL
+  X=$(./yq e '.a' test.yml)
+  assertEquals "test" "$X"
+}
+
+testLeadingSeperatorExtractFieldWithCommentsBeforeSep() {
+  cat >test.yml <<EOL
+# hi peeps
+# cool
+---
+a: test
+EOL
+  X=$(./yq e '.a' test.yml)
+  assertEquals "test" "$X"
+}
+
+
+testLeadingSeperatorExtractFieldMultiDoc() {
+  cat >test.yml <<EOL
+---
+a: test
+---
+a: test2
+EOL
+
+  read -r -d '' expected << EOM
+test
+---
+test2
+EOM
+  X=$(./yq e '.a' test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testLeadingSeperatorExtractFieldMultiDocWithComments() {
+  cat >test.yml <<EOL
+# here
+---
+# there
+a: test
+# whereever
+---
+# you are
+a: test2
+# woop
+EOL
+
+  read -r -d '' expected << EOM
+test
+---
+test2
+EOM
+  X=$(./yq e '.a' test.yml)
+  assertEquals "$expected" "$X"
+}
+
 
 testLeadingSeperatorEvalSeq() {
   X=$(./yq e test.yml)
@@ -229,6 +296,35 @@ EOM
   assertEquals "$expected" "$X"
 }
 
+# https://github.com/mikefarah/yq/issues/919
+testLeadingSeparatorDoesNotBreakCommentsOnOtherFiles() {
+  cat >test.yml <<EOL
+# a1
+a: 1
+# a2
+EOL
+
+cat >test2.yml <<EOL
+# b1
+b: 2
+# b2
+EOL
+
+  read -r -d '' expected << EOM
+# a1
+a: 1
+# a2
+
+# b1
+b: 2
+# b2
+EOM
+
+
+  X=$(./yq ea 'select(fi == 0) * select(fi == 1)' test.yml test2.yml)
+  assertEquals "$expected" "$X"
+}
+
 testLeadingSeperatorMultiDocEvalCommentsStripComments() {
   cat >test.yml <<EOL
 ---

acceptance_tests/output-format.sh

@@ -0,0 +1,101 @@
+#!/bin/bash
+
+testOutputJsonDeprecated() {
+  cat >test.yml <<EOL
+a: {b: ["cat"]}
+EOL
+
+  read -r -d '' expected << EOM
+{
+  "a": {
+    "b": [
+      "cat"
+    ]
+  }
+}
+EOM
+
+  X=$(./yq e -j test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea -j test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testOutputJson() {
+  cat >test.yml <<EOL
+a: {b: ["cat"]}
+EOL
+
+  read -r -d '' expected << EOM
+{
+  "a": {
+    "b": [
+      "cat"
+    ]
+  }
+}
+EOM
+
+  X=$(./yq e --output-format=json test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea --output-format=json test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testOutputJsonShort() {
+  cat >test.yml <<EOL
+a: {b: ["cat"]}
+EOL
+
+  read -r -d '' expected << EOM
+{
+  "a": {
+    "b": [
+      "cat"
+    ]
+  }
+}
+EOM
+
+  X=$(./yq e -o=j test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea -o=j test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testOutputProperties() {
+  cat >test.yml <<EOL
+a: {b: {c: ["cat"]}}
+EOL
+
+  read -r -d '' expected << EOM
+a.b.c.0 = cat
+EOM
+
+  X=$(./yq e --output-format=props test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea --output-format=props test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testOutputPropertiesShort() {
+  cat >test.yml <<EOL
+a: {b: {c: ["cat"]}}
+EOL
+
+  read -r -d '' expected << EOM
+a.b.c.0 = cat
+EOM
+
+  X=$(./yq e -o=p test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea -o=p test.yml)
+  assertEquals "$expected" "$X"
+}
+
+source ./scripts/shunit2

acceptance_tests/pretty-print.sh

@@ -0,0 +1,173 @@
+#!/bin/bash
+
+
+testPrettyPrintWithBooleans() {
+  cat >test.yml <<EOL
+leaveUnquoted: [yes, no, on, off, y, n, true, false]
+leaveQuoted: ["yes", "no", "on", "off", "y", "n", "true", "false"]
+
+EOL
+
+  read -r -d '' expected << EOM
+leaveUnquoted:
+  - yes
+  - no
+  - on
+  - off
+  - y
+  - n
+  - true
+  - false
+leaveQuoted:
+  - "yes"
+  - "no"
+  - "on"
+  - "off"
+  - "y"
+  - "n"
+  - "true"
+  - "false"
+EOM
+
+  X=$(./yq e --prettyPrint test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea --prettyPrint test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testPrettyPrintWithBooleansCapitals() {
+  cat >test.yml <<EOL
+leaveUnquoted: [YES, NO, ON, OFF, Y, N, TRUE, FALSE]
+leaveQuoted: ["YES", "NO", "ON", "OFF", "Y", "N", "TRUE", "FALSE"]
+EOL
+
+  read -r -d '' expected << EOM
+leaveUnquoted:
+  - YES
+  - NO
+  - ON
+  - OFF
+  - Y
+  - N
+  - TRUE
+  - FALSE
+leaveQuoted:
+  - "YES"
+  - "NO"
+  - "ON"
+  - "OFF"
+  - "Y"
+  - "N"
+  - "TRUE"
+  - "FALSE"
+EOM
+
+  X=$(./yq e --prettyPrint test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea --prettyPrint test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testPrettyPrintOtherStringValues() {
+  cat >test.yml <<EOL
+leaveUnquoted: [yesSir, hellno, bonapite]
+makeUnquoted: ["yesSir", "hellno", "bonapite"]
+EOL
+
+  read -r -d '' expected << EOM
+leaveUnquoted:
+  - yesSir
+  - hellno
+  - bonapite
+makeUnquoted:
+  - yesSir
+  - hellno
+  - bonapite
+EOM
+
+  X=$(./yq e --prettyPrint test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea --prettyPrint test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testPrettyPrintKeys() {
+  cat >test.yml <<EOL
+"removeQuotes": "please"
+EOL
+
+  read -r -d '' expected << EOM
+removeQuotes: please
+EOM
+
+  X=$(./yq e --prettyPrint test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea --prettyPrint test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testPrettyPrintOtherStringValues() {
+  cat >test.yml <<EOL
+leaveUnquoted: [yesSir, hellno, bonapite]
+makeUnquoted: ["yesSir", "hellno", "bonapite"]
+EOL
+
+  read -r -d '' expected << EOM
+leaveUnquoted:
+  - yesSir
+  - hellno
+  - bonapite
+makeUnquoted:
+  - yesSir
+  - hellno
+  - bonapite
+EOM
+
+  X=$(./yq e --prettyPrint test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea --prettyPrint test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testPrettyPrintStringBlocks() {
+  cat >test.yml <<EOL
+"removeQuotes": | 
+  "please"
+EOL
+
+  read -r -d '' expected << EOM
+removeQuotes: |
+  "please"
+EOM
+
+  X=$(./yq e --prettyPrint test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea --prettyPrint test.yml)
+  assertEquals "$expected" "$X"
+}
+
+testPrettyPrintWithExpression() {
+  cat >test.yml <<EOL
+a: {b: {c: ["cat"]}}
+EOL
+
+  read -r -d '' expected << EOM
+b:
+  c:
+    - cat
+EOM
+
+  X=$(./yq e '.a' --prettyPrint test.yml)
+  assertEquals "$expected" "$X"
+
+  X=$(./yq ea '.a' --prettyPrint test.yml)
+  assertEquals "$expected" "$X"
+}
+
+source ./scripts/shunit2

cmd/constant.go

@@ -5,6 +5,7 @@ var unwrapScalar = true
 
 var writeInplace = false
 var outputToJSON = false
+var outputFormat = "yaml"
 
 var exitStatus = false
 var forceColor = false

cmd/evaluate_all_command.go

@@ -86,8 +86,17 @@ func evaluateAll(cmd *cobra.Command, args []string) error {
 	if nullInput && len(args) > 1 {
 		return errors.New("Cannot pass files in when using null-input flag")
 	}
+	// backwards compatibilty
+	if outputToJSON {
+		outputFormat = "json"
+	}
 
-	printer := yqlib.NewPrinter(out, outputToJSON, unwrapScalar, colorsEnabled, indent, !noDocSeparators)
+	format, err := yqlib.OutputFormatFromString(outputFormat)
+	if err != nil {
+		return err
+	}
+
+	printer := yqlib.NewPrinter(out, format, unwrapScalar, colorsEnabled, indent, !noDocSeparators)
 
 	if frontMatter != "" {
 		frontMatterHandler := yqlib.NewFrontMatterHandler(args[firstFileIndex])
@@ -98,10 +107,7 @@ func evaluateAll(cmd *cobra.Command, args []string) error {
 		args[firstFileIndex] = frontMatterHandler.GetYamlFrontMatterFilename()
 
 		if frontMatter == "process" {
-			reader, err := os.Open(frontMatterHandler.GetContentFilename()) // #nosec
-			if err != nil {
-				return err
-			}
+			reader := frontMatterHandler.GetContentReader()
 			printer.SetAppendix(reader)
 			defer yqlib.SafelyCloseReader(reader)
 		}

cmd/evalute_sequence_command.go

@@ -44,10 +44,11 @@ expression and prints the result in sequence.`,
 }
 
 func processExpression(expression string) string {
+	var prettyPrintExp = `(... | (select(tag != "!!str"), select(tag == "!!str") | select(test("(?i)^(y|yes|n|no|on|off)$") | not))  ) style=""`
 	if prettyPrint && expression == "" {
-		return `... style=""`
+		return prettyPrintExp
 	} else if prettyPrint {
-		return fmt.Sprintf("%v | ... style= \"\"", expression)
+		return fmt.Sprintf("%v | %v", expression, prettyPrintExp)
 	}
 	return expression
 }
@@ -95,7 +96,17 @@ func evaluateSequence(cmd *cobra.Command, args []string) error {
 		defer func() { writeInPlaceHandler.FinishWriteInPlace(completedSuccessfully) }()
 	}
 
-	printer := yqlib.NewPrinter(out, outputToJSON, unwrapScalar, colorsEnabled, indent, !noDocSeparators)
+	// backwards compatibilty
+	if outputToJSON {
+		outputFormat = "json"
+	}
+
+	format, err := yqlib.OutputFormatFromString(outputFormat)
+	if err != nil {
+		return err
+	}
+
+	printer := yqlib.NewPrinter(out, format, unwrapScalar, colorsEnabled, indent, !noDocSeparators)
 
 	streamEvaluator := yqlib.NewStreamEvaluator()
 
@@ -112,10 +123,7 @@ func evaluateSequence(cmd *cobra.Command, args []string) error {
 		args[firstFileIndex] = frontMatterHandler.GetYamlFrontMatterFilename()
 
 		if frontMatter == "process" {
-			reader, err := os.Open(frontMatterHandler.GetContentFilename()) // #nosec
-			if err != nil {
-				return err
-			}
+			reader := frontMatterHandler.GetContentReader()
 			printer.SetAppendix(reader)
 			defer yqlib.SafelyCloseReader(reader)
 		}

cmd/root.go

@@ -41,7 +41,14 @@ See https://mikefarah.gitbook.io/yq/ for detailed documentation and examples.`,
 	}
 
 	rootCmd.PersistentFlags().BoolVarP(&verbose, "verbose", "v", false, "verbose mode")
-	rootCmd.PersistentFlags().BoolVarP(&outputToJSON, "tojson", "j", false, "output as json. Set indent to 0 to print json in one line.")
+
+	rootCmd.PersistentFlags().BoolVarP(&outputToJSON, "tojson", "j", false, "(deprecated) output as json. Set indent to 0 to print json in one line.")
+	err := rootCmd.PersistentFlags().MarkDeprecated("tojson", "please use -o=json instead")
+	if err != nil {
+		panic(err)
+	}
+
+	rootCmd.PersistentFlags().StringVarP(&outputFormat, "output-format", "o", "yaml", "[yaml|y|json|j|props|p] output format type.")
 	rootCmd.PersistentFlags().BoolVarP(&nullInput, "null-input", "n", false, "Don't read input, simply evaluate the expression given. Useful for creating yaml docs from scratch.")
 	rootCmd.PersistentFlags().BoolVarP(&noDocSeparators, "no-doc", "N", false, "Don't print document separators (---)")
 

cmd/version.go

@@ -11,7 +11,7 @@ var (
 	GitDescribe string
 
 	// Version is main version number that is being run at the moment.
-	Version = "4.11.0"
+	Version = "4.13.0"
 
 	// VersionPrerelease is a pre-release marker for the version. If this is "" (empty string)
 	// then it means that it is a final release. Otherwise, this is a pre-release

examples/data1.yaml

@@ -1,4 +1,5 @@
----
-# hi peeps
-# cool
-a: test
+Resources:
+  S3Bucket:
+    Type: AWS::CloudFormation::Stack
+    Properties:
+      BucketName: !Ref MyBucketNameA

examples/data2.yaml

@@ -1,7 +1,3 @@
-a: other # better than the original
-b: [3, 4]
-c:
-  toast: leave
-  test: 1
-  tell: 1
-  tasty.taco: cool
+#b1 
+b: 2
+#b2

examples/front-matter.yaml

@@ -2,4 +2,5 @@
 a: apple
 b: bannana
 ---
-<h1>I like {{a}} and {{b}} </h1>
+hello there
+apples: great

github-action/Dockerfile

@@ -1,4 +1,4 @@
-FROM mikefarah/yq:4.11.0
+FROM mikefarah/yq:4.13.0
 
 COPY entrypoint.sh /entrypoint.sh
 

github-action/entrypoint.sh

@@ -1,6 +1,9 @@
 #!/bin/sh -l
-
+set -e
 echo "::debug::\$cmd: $1"
 RESULT=$(eval "$1")
+RESULT="${RESULT//'%'/'%25'}"
+RESULT="${RESULT//$'\n'/'%0A'}"
+RESULT="${RESULT//$'\r'/'%0D'}"
 echo "::debug::\$RESULT: $RESULT"
 echo ::set-output name=result::"$RESULT"

go.mod

@@ -5,6 +5,7 @@ require (
 	github.com/fatih/color v1.10.0
 	github.com/goccy/go-yaml v1.8.9
 	github.com/jinzhu/copier v0.2.8
+	github.com/magiconair/properties v1.8.5
 	github.com/spf13/cobra v1.1.3
 	github.com/timtadh/data-structures v0.5.3 // indirect
 	github.com/timtadh/lexmachine v0.2.2

go.sum

@@ -119,6 +119,8 @@ github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
 github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
 github.com/leodido/go-urn v1.2.0/go.mod h1:+8+nEpDfqqsY+g338gtMEUOtuK+4dEMhiQEgxpxOKII=
 github.com/magiconair/properties v1.8.1/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
+github.com/magiconair/properties v1.8.5 h1:b6kJs+EmPFMYGkow9GiUyCyOvIwYetYJ3fSaWak/Gls=
+github.com/magiconair/properties v1.8.5/go.mod h1:y3VJvCyxH9uVvJTWEGAELF3aiYNyPKd5NZ3oSwXrF60=
 github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU=
 github.com/mattn/go-colorable v0.1.8 h1:c1ghPdyEDarC70ftn0y+A/Ee++9zz8ljHG1b13eJ0s8=
 github.com/mattn/go-colorable v0.1.8/go.mod h1:u6P/XSegPjTcexA+o6vUJrdnUu04hMope9wVRipJSqc=

pkg/yqlib/all_at_once_evaluator.go

@@ -52,7 +52,7 @@ func (e *allAtOnceEvaluator) EvaluateFiles(expression string, filenames []string
 
 	var allDocuments *list.List = list.New()
 	for _, filename := range filenames {
-		reader, leadingContent, err := readStream(filename, leadingContentPreProcessing)
+		reader, leadingContent, err := readStream(filename, fileIndex == 0 && leadingContentPreProcessing)
 		if err != nil {
 			return err
 		}

pkg/yqlib/context.go

@@ -2,8 +2,10 @@ package yqlib
 
 import (
 	"container/list"
+	"fmt"
 
 	"github.com/jinzhu/copier"
+	logging "gopkg.in/op/go-logging.v1"
 )
 
 type Context struct {
@@ -51,9 +53,39 @@ func (n *Context) ChildContext(results *list.List) Context {
 	return clone
 }
 
+func (n *Context) ToString() string {
+	if !log.IsEnabledFor(logging.DEBUG) {
+		return ""
+	}
+	result := fmt.Sprintf("Context\nDontAutoCreate: %v\n", n.DontAutoCreate)
+	return result + NodesToString(n.MatchingNodes)
+}
+
+func (n *Context) DeepClone() Context {
+	clone := Context{}
+	err := copier.Copy(&clone, n)
+	// copier doesn't do lists properly for some reason
+	clone.MatchingNodes = list.New()
+	for el := n.MatchingNodes.Front(); el != nil; el = el.Next() {
+		clonedNode, err := el.Value.(*CandidateNode).Copy()
+		if err != nil {
+			log.Error("Error cloning context :(")
+			panic(err)
+		}
+		clone.MatchingNodes.PushBack(clonedNode)
+	}
+
+	if err != nil {
+		log.Error("Error cloning context :(")
+		panic(err)
+	}
+	return clone
+}
+
 func (n *Context) Clone() Context {
 	clone := Context{}
 	err := copier.Copy(&clone, n)
+
 	if err != nil {
 		log.Error("Error cloning context :(")
 		panic(err)

pkg/yqlib/doc/Contains.md

@@ -0,0 +1,85 @@
+
+## Array contains array
+Array is equal or subset of
+
+Given a sample.yml file of:
+```yaml
+- foobar
+- foobaz
+- blarp
+```
+then
+```bash
+yq eval 'contains(["baz", "bar"])' sample.yml
+```
+will output
+```yaml
+true
+```
+
+## Object included in array
+Given a sample.yml file of:
+```yaml
+"foo": 12
+"bar":
+  - 1
+  - 2
+  - "barp": 12
+    "blip": 13
+```
+then
+```bash
+yq eval 'contains({"bar": [{"barp": 12}]})' sample.yml
+```
+will output
+```yaml
+true
+```
+
+## Object not included in array
+Given a sample.yml file of:
+```yaml
+"foo": 12
+"bar":
+  - 1
+  - 2
+  - "barp": 12
+    "blip": 13
+```
+then
+```bash
+yq eval 'contains({"foo": 12, "bar": [{"barp": 15}]})' sample.yml
+```
+will output
+```yaml
+false
+```
+
+## String contains substring
+Given a sample.yml file of:
+```yaml
+foobar
+```
+then
+```bash
+yq eval 'contains("bar")' sample.yml
+```
+will output
+```yaml
+true
+```
+
+## String equals string
+Given a sample.yml file of:
+```yaml
+meow
+```
+then
+```bash
+yq eval 'contains("meow")' sample.yml
+```
+will output
+```yaml
+true
+```
+

pkg/yqlib/doc/String Operators.md

@@ -3,6 +3,45 @@
 ## RegEx
 This uses golangs native regex functions under the hood - See https://github.com/google/re2/wiki/Syntax for the supported syntax.
 
+
+# String blocks, bash and newlines
+Bash is notorious for chomping on precious trailing newline characters, making it tricky to set strings with newlines properly. In particular, the `$( exp )` _will trim trailing newlines_.
+
+For instance to get this yaml:
+
+```
+a: |
+  cat
+```
+
+Using `$( exp )` wont work, as it will trim the trailing new line.
+
+```
+m=$(echo "cat\n") yq e -n '.a = strenv(m)'
+a: cat
+```
+
+However, using printf works:
+```
+printf -v m "cat\n" ; m="$m" yq e -n '.a = strenv(m)'
+a: |
+  cat
+```
+
+As well as having multiline expressions:
+```
+m="cat
+"  yq e -n '.a = strenv(m)'
+a: |
+  cat
+```
+
+Similarly, if you're trying to set the content from a file, and want a trailing new line:
+
+```
+IFS= read -rd '' output < <(cat my_file)
+output=$output ./yq e '.data.values = strenv(output)' first.yml
+```
 ## Join strings
 Given a sample.yml file of:
 ```yaml

pkg/yqlib/doc/Style.md

@@ -18,8 +18,6 @@ a:
 ```
 
 ## Update and set style of a particular node using path variables
-You can use a variable to re-use a path
-
 Given a sample.yml file of:
 ```yaml
 a:
@@ -28,7 +26,7 @@ a:
 ```
 then
 ```bash
-yq eval '.a.b as $x | $x = "new" | $x style="double"' sample.yml
+yq eval 'with(.a.b ; . = "new" | . style="double")' sample.yml
 ```
 will output
 ```yaml

pkg/yqlib/doc/Subtract.md

@@ -1,4 +1,43 @@
 
+## Array subtraction
+Running
+```bash
+yq eval --null-input '[1,2] - [2,3]'
+```
+will output
+```yaml
+- 1
+```
+
+## Array subtraction with nested array
+Running
+```bash
+yq eval --null-input '[[1], 1, 2] - [[1], 3]'
+```
+will output
+```yaml
+- 1
+- 2
+```
+
+## Array subtraction with nested object
+Note that order of the keys does not matter
+
+Given a sample.yml file of:
+```yaml
+- a: b
+  c: d
+- a: b
+```
+then
+```bash
+yq eval '. - [{"c": "d", "a": "b"}]' sample.yml
+```
+will output
+```yaml
+- a: b
+```
+
 ## Number subtraction - float
 If the lhs or rhs are floats then the expression will be calculated with floats.
 

pkg/yqlib/doc/Variable Operators.md

@@ -1,4 +1,6 @@
-For more complex scenarios, variables can be used to hold values of expression to be used in other expressions.
+Like the `jq` equivalents, variables are sometimes required for the more complex expressions (or swapping values between fields).
+
+Note that there is also an additional `ref` operator that holds a reference (instead of a copy) of the path, allowing you to make multiple changes to the same path.
 
 ## Single value variable
 Given a sample.yml file of:
@@ -56,3 +58,39 @@ title: A well-written article
 author: Person McPherson
 ```
 
+## Using variables to swap values
+Given a sample.yml file of:
+```yaml
+a: a_value
+b: b_value
+```
+then
+```bash
+yq eval '.a as $x  | .b as $y | .b = $x | .a = $y' sample.yml
+```
+will output
+```yaml
+a: b_value
+b: a_value
+```
+
+## Use ref to reference a path repeatedly
+Note: You may find the `with` operator more useful.
+
+Given a sample.yml file of:
+```yaml
+a:
+  b: thing
+  c: something
+```
+then
+```bash
+yq eval '.a.b ref $x | $x = "new" | $x style="double"' sample.yml
+```
+will output
+```yaml
+a:
+  b: "new"
+  c: something
+```
+

pkg/yqlib/doc/With.md

@@ -0,0 +1,60 @@
+Use the `with` operator to conveniently make multiple updates to a deeply nested path, or to update array elements relatively to each other.
+
+## Update and style
+Given a sample.yml file of:
+```yaml
+a:
+  deeply:
+    nested: value
+```
+then
+```bash
+yq eval 'with(.a.deeply.nested ; . = "newValue" | . style="single")' sample.yml
+```
+will output
+```yaml
+a:
+  deeply:
+    nested: 'newValue'
+```
+
+## Update multiple deeply nested properties
+Given a sample.yml file of:
+```yaml
+a:
+  deeply:
+    nested: value
+    other: thing
+```
+then
+```bash
+yq eval 'with(.a.deeply ; .nested = "newValue" | .other= "newThing")' sample.yml
+```
+will output
+```yaml
+a:
+  deeply:
+    nested: newValue
+    other: newThing
+```
+
+## Update array elements relatively
+Given a sample.yml file of:
+```yaml
+myArray:
+  - a: apple
+  - a: banana
+```
+then
+```bash
+yq eval 'with(.myArray[] ; .b = .a + " yum")' sample.yml
+```
+will output
+```yaml
+myArray:
+  - a: apple
+    b: apple yum
+  - a: banana
+    b: banana yum
+```
+

pkg/yqlib/doc/headers/String Operators.md

@@ -2,3 +2,43 @@
 
 ## RegEx
 This uses golangs native regex functions under the hood - See https://github.com/google/re2/wiki/Syntax for the supported syntax.
+
+
+# String blocks, bash and newlines
+Bash is notorious for chomping on precious trailing newline characters, making it tricky to set strings with newlines properly. In particular, the `$( exp )` _will trim trailing newlines_.
+
+For instance to get this yaml:
+
+```
+a: |
+  cat
+```
+
+Using `$( exp )` wont work, as it will trim the trailing new line.
+
+```
+m=$(echo "cat\n") yq e -n '.a = strenv(m)'
+a: cat
+```
+
+However, using printf works:
+```
+printf -v m "cat\n" ; m="$m" yq e -n '.a = strenv(m)'
+a: |
+  cat
+```
+
+As well as having multiline expressions:
+```
+m="cat
+"  yq e -n '.a = strenv(m)'
+a: |
+  cat
+```
+
+Similarly, if you're trying to set the content from a file, and want a trailing new line:
+
+```
+IFS= read -rd '' output < <(cat my_file)
+output=$output ./yq e '.data.values = strenv(output)' first.yml
+```

pkg/yqlib/doc/headers/Variable Operators.md

@@ -1 +1,3 @@
-For more complex scenarios, variables can be used to hold values of expression to be used in other expressions.
+Like the `jq` equivalents, variables are sometimes required for the more complex expressions (or swapping values between fields).
+
+Note that there is also an additional `ref` operator that holds a reference (instead of a copy) of the path, allowing you to make multiple changes to the same path.

pkg/yqlib/doc/headers/With.md

@@ -0,0 +1 @@
+Use the `with` operator to conveniently make multiple updates to a deeply nested path, or to update array elements relatively to each other.

pkg/yqlib/encoder_properties.go

@@ -0,0 +1,80 @@
+package yqlib
+
+import (
+	"fmt"
+	"io"
+	"strings"
+
+	"github.com/magiconair/properties"
+	yaml "gopkg.in/yaml.v3"
+)
+
+type propertiesEncoder struct {
+	destination io.Writer
+}
+
+func NewPropertiesEncoder(destination io.Writer) Encoder {
+	return &propertiesEncoder{destination}
+}
+
+func (pe *propertiesEncoder) Encode(node *yaml.Node) error {
+	mapKeysToStrings(node)
+	p := properties.NewProperties()
+	err := pe.doEncode(p, node, "")
+	if err != nil {
+		return err
+	}
+
+	_, err = p.WriteComment(pe.destination, "#", properties.UTF8)
+	return err
+}
+
+func (pe *propertiesEncoder) doEncode(p *properties.Properties, node *yaml.Node, path string) error {
+	p.SetComment(path,
+		strings.Replace(node.HeadComment, "#", "", 1)+
+			strings.Replace(node.LineComment, "#", "", 1))
+	switch node.Kind {
+	case yaml.ScalarNode:
+		_, _, err := p.Set(path, node.Value)
+		return err
+	case yaml.DocumentNode:
+		return pe.doEncode(p, node.Content[0], path)
+	case yaml.SequenceNode:
+		return pe.encodeArray(p, node.Content, path)
+	case yaml.MappingNode:
+		return pe.encodeMap(p, node.Content, path)
+	case yaml.AliasNode:
+		return pe.doEncode(p, node.Alias, path)
+	default:
+		return fmt.Errorf("Unsupported node %v", node.Tag)
+	}
+}
+
+func (pe *propertiesEncoder) appendPath(path string, key interface{}) string {
+	if path == "" {
+		return fmt.Sprintf("%v", key)
+	}
+	return fmt.Sprintf("%v.%v", path, key)
+}
+
+func (pe *propertiesEncoder) encodeArray(p *properties.Properties, kids []*yaml.Node, path string) error {
+	for index, child := range kids {
+		err := pe.doEncode(p, child, pe.appendPath(path, index))
+		if err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+func (pe *propertiesEncoder) encodeMap(p *properties.Properties, kids []*yaml.Node, path string) error {
+	for index := 0; index < len(kids); index = index + 2 {
+		key := kids[index]
+		value := kids[index+1]
+		err := pe.doEncode(p, value, pe.appendPath(path, key.Value))
+		if err != nil {
+			return err
+		}
+	}
+	return nil
+}

pkg/yqlib/encoder_properties_test.go

@@ -0,0 +1,78 @@
+package yqlib
+
+import (
+	"bufio"
+	"bytes"
+	"strings"
+	"testing"
+
+	"github.com/mikefarah/yq/v4/test"
+)
+
+func yamlToProps(sampleYaml string) string {
+	var output bytes.Buffer
+	writer := bufio.NewWriter(&output)
+
+	var propsEncoder = NewPropertiesEncoder(writer)
+	inputs, err := readDocuments(strings.NewReader(sampleYaml), "sample.yml", 0)
+	if err != nil {
+		panic(err)
+	}
+	node := inputs.Front().Value.(*CandidateNode).Node
+	err = propsEncoder.Encode(node)
+	if err != nil {
+		panic(err)
+	}
+	writer.Flush()
+
+	return strings.TrimSuffix(output.String(), "\n")
+}
+
+func TestPropertiesEncoderSimple(t *testing.T) {
+	var sampleYaml = `a: 'bob cool'`
+
+	var expectedJson = `a = bob cool`
+	var actualProps = yamlToProps(sampleYaml)
+	test.AssertResult(t, expectedJson, actualProps)
+}
+
+func TestPropertiesEncoderSimpleWithComments(t *testing.T) {
+	var sampleYaml = `a: 'bob cool' # line`
+
+	var expectedJson = `# line
+a = bob cool`
+	var actualProps = yamlToProps(sampleYaml)
+	test.AssertResult(t, expectedJson, actualProps)
+}
+
+func TestPropertiesEncoderDeep(t *testing.T) {
+	var sampleYaml = `a: 
+  b: "bob cool"
+`
+
+	var expectedJson = `a.b = bob cool`
+	var actualProps = yamlToProps(sampleYaml)
+	test.AssertResult(t, expectedJson, actualProps)
+}
+
+func TestPropertiesEncoderDeepWithComments(t *testing.T) {
+	var sampleYaml = `a:  # a thing
+  b: "bob cool" # b thing
+`
+
+	var expectedJson = `# b thing
+a.b = bob cool`
+	var actualProps = yamlToProps(sampleYaml)
+	test.AssertResult(t, expectedJson, actualProps)
+}
+
+func TestPropertiesEncoderArray(t *testing.T) {
+	var sampleYaml = `a: 
+  b: [{c: dog}, {c: cat}]
+`
+
+	var expectedJson = `a.b.0.c = dog
+a.b.1.c = cat`
+	var actualProps = yamlToProps(sampleYaml)
+	test.AssertResult(t, expectedJson, actualProps)
+}

pkg/yqlib/expression_parser_test.go

@@ -6,37 +6,66 @@ import (
 	"github.com/mikefarah/yq/v4/test"
 )
 
-func TestPathTreeNoArgsForTwoArgOp(t *testing.T) {
+func TestParserNoMatchingCloseCollect(t *testing.T) {
+	_, err := NewExpressionParser().ParseExpression("[1,2")
+	test.AssertResultComplex(t, "Bad expression, could not find matching `]`", err.Error())
+}
+func TestParserNoMatchingCloseObjectInCollect(t *testing.T) {
+	_, err := NewExpressionParser().ParseExpression(`[{"b": "c"]`)
+	test.AssertResultComplex(t, "Bad expression, could not find matching `}`", err.Error())
+}
+
+func TestParserNoMatchingCloseInCollect(t *testing.T) {
+	_, err := NewExpressionParser().ParseExpression(`[(.a]`)
+	test.AssertResultComplex(t, "Bad expression, could not find matching `)`", err.Error())
+}
+
+func TestParserNoMatchingCloseCollectObject(t *testing.T) {
+	_, err := NewExpressionParser().ParseExpression(`{"a": "b"`)
+	test.AssertResultComplex(t, "Bad expression, could not find matching `}`", err.Error())
+}
+
+func TestParserNoMatchingCloseCollectInCollectObject(t *testing.T) {
+	_, err := NewExpressionParser().ParseExpression(`{"b": [1}`)
+	test.AssertResultComplex(t, "Bad expression, could not find matching `]`", err.Error())
+}
+
+func TestParserNoMatchingCloseBracketInCollectObject(t *testing.T) {
+	_, err := NewExpressionParser().ParseExpression(`{"b": (1}`)
+	test.AssertResultComplex(t, "Bad expression, could not find matching `)`", err.Error())
+}
+
+func TestParserNoArgsForTwoArgOp(t *testing.T) {
 	_, err := NewExpressionParser().ParseExpression("=")
 	test.AssertResultComplex(t, "'=' expects 2 args but there is 0", err.Error())
 }
 
-func TestPathTreeOneLhsArgsForTwoArgOp(t *testing.T) {
+func TestParserOneLhsArgsForTwoArgOp(t *testing.T) {
 	_, err := NewExpressionParser().ParseExpression(".a =")
 	test.AssertResultComplex(t, "'=' expects 2 args but there is 1", err.Error())
 }
 
-func TestPathTreeOneRhsArgsForTwoArgOp(t *testing.T) {
+func TestParserOneRhsArgsForTwoArgOp(t *testing.T) {
 	_, err := NewExpressionParser().ParseExpression("= .a")
 	test.AssertResultComplex(t, "'=' expects 2 args but there is 1", err.Error())
 }
 
-func TestPathTreeTwoArgsForTwoArgOp(t *testing.T) {
+func TestParserTwoArgsForTwoArgOp(t *testing.T) {
 	_, err := NewExpressionParser().ParseExpression(".a = .b")
 	test.AssertResultComplex(t, nil, err)
 }
 
-func TestPathTreeNoArgsForOneArgOp(t *testing.T) {
+func TestParserNoArgsForOneArgOp(t *testing.T) {
 	_, err := NewExpressionParser().ParseExpression("explode")
 	test.AssertResultComplex(t, "'explode' expects 1 arg but received none", err.Error())
 }
 
-func TestPathTreeOneArgForOneArgOp(t *testing.T) {
+func TestParserOneArgForOneArgOp(t *testing.T) {
 	_, err := NewExpressionParser().ParseExpression("explode(.)")
 	test.AssertResultComplex(t, nil, err)
 }
 
-func TestPathTreeExtraArgs(t *testing.T) {
+func TestParserExtraArgs(t *testing.T) {
 	_, err := NewExpressionParser().ParseExpression("sortKeys(.) explode(.)")
 	test.AssertResultComplex(t, "Bad expression, please check expression syntax", err.Error())
 }

pkg/yqlib/expression_postfix.go

@@ -2,6 +2,7 @@ package yqlib
 
 import (
 	"errors"
+	"fmt"
 
 	logging "gopkg.in/op/go-logging.v1"
 )
@@ -24,6 +25,17 @@ func popOpToResult(opStack []*token, result []*Operation) ([]*token, []*Operatio
 	return opStack, append(result, newOp.Operation)
 }
 
+func validateNoOpenTokens(token *token) error {
+	if token.TokenType == openCollect {
+		return fmt.Errorf(("Bad expression, could not find matching `]`"))
+	} else if token.TokenType == openCollectObject {
+		return fmt.Errorf(("Bad expression, could not find matching `}`"))
+	} else if token.TokenType == openBracket {
+		return fmt.Errorf(("Bad expression, could not find matching `)`"))
+	}
+	return nil
+}
+
 func (p *expressionPostFixerImpl) ConvertToPostfix(infixTokens []*token) ([]*Operation, error) {
 	var result []*Operation
 	// surround the whole thing with brackets
@@ -45,6 +57,10 @@ func (p *expressionPostFixerImpl) ConvertToPostfix(infixTokens []*token) ([]*Ope
 			}
 
 			for len(opStack) > 0 && opStack[len(opStack)-1].TokenType != opener {
+				missingClosingTokenErr := validateNoOpenTokens(opStack[len(opStack)-1])
+				if missingClosingTokenErr != nil {
+					return nil, missingClosingTokenErr
+				}
 				opStack, result = popOpToResult(opStack, result)
 			}
 			if len(opStack) == 0 {
@@ -76,6 +92,11 @@ func (p *expressionPostFixerImpl) ConvertToPostfix(infixTokens []*token) ([]*Ope
 
 		case closeBracket:
 			for len(opStack) > 0 && opStack[len(opStack)-1].TokenType != openBracket {
+				missingClosingTokenErr := validateNoOpenTokens(opStack[len(opStack)-1])
+				if missingClosingTokenErr != nil {
+					return nil, missingClosingTokenErr
+				}
+
 				opStack, result = popOpToResult(opStack, result)
 			}
 			if len(opStack) == 0 {
@@ -98,6 +119,8 @@ func (p *expressionPostFixerImpl) ConvertToPostfix(infixTokens []*token) ([]*Ope
 		}
 	}
 
+	log.Debugf("opstackLen: %v", len(opStack))
+
 	if log.IsEnabledFor(logging.DEBUG) {
 		log.Debugf("PostFix Result:")
 		for _, currentToken := range result {

pkg/yqlib/expression_processing_test.go

@@ -15,6 +15,16 @@ var pathTests = []struct {
 	expectedTokens  []interface{}
 	expectedPostFix []interface{}
 }{
+	{
+		"0x12",
+		append(make([]interface{}, 0), "18 (int64)"),
+		append(make([]interface{}, 0), "18 (int64)"),
+	},
+	{
+		"0X12",
+		append(make([]interface{}, 0), "18 (int64)"),
+		append(make([]interface{}, 0), "18 (int64)"),
+	},
 	{
 		".a\n",
 		append(make([]interface{}, 0), "a"),

pkg/yqlib/expression_tokeniser.go

@@ -166,6 +166,20 @@ func numberValue() lex.Action {
 	}
 }
 
+func hexValue() lex.Action {
+	return func(s *lex.Scanner, m *machines.Match) (interface{}, error) {
+		var originalString = string(m.Bytes)
+		var numberString = originalString[2:]
+		log.Debugf("numberString: %v", numberString)
+		var number, errParsingInt = strconv.ParseInt(numberString, 16, 64) // nolint
+		if errParsingInt != nil {
+			return nil, errParsingInt
+		}
+
+		return &token{TokenType: operationToken, Operation: createValueOperation(number, originalString)}, nil
+	}
+}
+
 func floatValue() lex.Action {
 	return func(s *lex.Scanner, m *machines.Match) (interface{}, error) {
 		var numberString = string(m.Bytes)
@@ -284,6 +298,7 @@ func initLexer() (*lex.Lexer, error) {
 	lexer.Add([]byte(`any_c`), opToken(anyConditionOpType))
 	lexer.Add([]byte(`all`), opToken(allOpType))
 	lexer.Add([]byte(`all_c`), opToken(allConditionOpType))
+	lexer.Add([]byte(`contains`), opToken(containsOpType))
 
 	lexer.Add([]byte(`split`), opToken(splitStringOpType))
 	lexer.Add([]byte(`keys`), opToken(keysOpType))
@@ -301,6 +316,8 @@ func initLexer() (*lex.Lexer, error) {
 	lexer.Add([]byte(`from_entries`), opToken(fromEntriesOpType))
 	lexer.Add([]byte(`with_entries`), opToken(withEntriesOpType))
 
+	lexer.Add([]byte(`with`), opToken(withOpType))
+
 	lexer.Add([]byte(`lineComment`), opTokenWithPrefs(getCommentOpType, assignCommentOpType, commentOpPreferences{LineComment: true}))
 
 	lexer.Add([]byte(`headComment`), opTokenWithPrefs(getCommentOpType, assignCommentOpType, commentOpPreferences{HeadComment: true}))
@@ -328,6 +345,7 @@ func initLexer() (*lex.Lexer, error) {
 
 	lexer.Add([]byte(`\|`), opToken(pipeOpType))
 
+	lexer.Add([]byte(`0[xX][0-9A-Fa-f]+`), hexValue())
 	lexer.Add([]byte(`-?\d+(\.\d+)`), floatValue())
 	lexer.Add([]byte(`-?[1-9](\.\d+)?[Ee][-+]?\d+`), floatValue())
 	lexer.Add([]byte(`-?\d+`), numberValue())
@@ -352,7 +370,8 @@ func initLexer() (*lex.Lexer, error) {
 	lexer.Add([]byte(`\-`), opToken(subtractOpType))
 	lexer.Add([]byte(`\-=`), opToken(subtractAssignOpType))
 	lexer.Add([]byte(`\$[a-zA-Z_-0-9]+`), getVariableOpToken())
-	lexer.Add([]byte(`as`), opToken(assignVariableOpType))
+	lexer.Add([]byte(`as`), opTokenWithPrefs(assignVariableOpType, nil, assignVarPreferences{}))
+	lexer.Add([]byte(`ref`), opTokenWithPrefs(assignVariableOpType, nil, assignVarPreferences{IsReference: true}))
 
 	err := lexer.CompileNFA()
 	if err != nil {

pkg/yqlib/front_matter.go

@@ -9,47 +9,48 @@ import (
 type frontMatterHandler interface {
 	Split() error
 	GetYamlFrontMatterFilename() string
-	GetContentFilename() string
+	GetContentReader() io.Reader
 	CleanUp()
 }
 
 type frontMatterHandlerImpl struct {
 	originalFilename        string
 	yamlFrontMatterFilename string
-	contentFilename         string
+	contentReader           io.Reader
 }
 
 func NewFrontMatterHandler(originalFilename string) frontMatterHandler {
-	return &frontMatterHandlerImpl{originalFilename, "", ""}
+	return &frontMatterHandlerImpl{originalFilename, "", nil}
 }
 
 func (f *frontMatterHandlerImpl) GetYamlFrontMatterFilename() string {
 	return f.yamlFrontMatterFilename
 }
 
-func (f *frontMatterHandlerImpl) GetContentFilename() string {
-	return f.contentFilename
+func (f *frontMatterHandlerImpl) GetContentReader() io.Reader {
+	return f.contentReader
 }
 
 func (f *frontMatterHandlerImpl) CleanUp() {
 	tryRemoveFile(f.yamlFrontMatterFilename)
-	tryRemoveFile(f.contentFilename)
 }
 
 // Splits the given file by yaml front matter
 // yaml content will be saved to first temporary file
 // remaining content will be saved to second temporary file
 func (f *frontMatterHandlerImpl) Split() error {
-	var reader io.Reader
+	var reader *bufio.Reader
 	var err error
 	if f.originalFilename == "-" {
 		reader = bufio.NewReader(os.Stdin)
 	} else {
-		reader, err = os.Open(f.originalFilename) // #nosec
+		file, err := os.Open(f.originalFilename) // #nosec
 		if err != nil {
 			return err
 		}
+		reader = bufio.NewReader(file)
 	}
+	f.contentReader = reader
 
 	yamlTempFile, err := createTempFile()
 	if err != nil {
@@ -58,39 +59,35 @@ func (f *frontMatterHandlerImpl) Split() error {
 	f.yamlFrontMatterFilename = yamlTempFile.Name()
 	log.Debug("yamlTempFile: %v", yamlTempFile.Name())
 
-	contentTempFile, err := createTempFile()
-	if err != nil {
-		return err
-	}
-	f.contentFilename = contentTempFile.Name()
-	log.Debug("contentTempFile: %v", contentTempFile.Name())
-
-	scanner := bufio.NewScanner(reader)
-
 	lineCount := 0
-	yamlContentBlock := true
 
-	for scanner.Scan() {
-		line := scanner.Text()
-
-		if lineCount > 0 && line == "---" {
-			//we've finished reading the yaml content
-			yamlContentBlock = false
-		}
-		if yamlContentBlock {
-			_, err = yamlTempFile.Write([]byte(line + "\n"))
-		} else {
-			_, err = contentTempFile.Write([]byte(line + "\n"))
-		}
-		if err != nil {
+	for {
+		peekBytes, err := reader.Peek(3)
+		if err == io.EOF {
+			// we've finished reading the yaml content..I guess
+			break
+		} else if err != nil {
 			return err
 		}
+		if lineCount > 0 && string(peekBytes) == "---" {
+			// we've finished reading the yaml content..
+			break
+		}
+		line, errReading := reader.ReadString('\n')
 		lineCount = lineCount + 1
+		if errReading != nil && errReading != io.EOF {
+			return errReading
+		}
+
+		_, errWriting := yamlTempFile.Write([]byte(line))
+
+		if errWriting != nil {
+			return errWriting
+		}
 	}
 
 	safelyCloseFile(yamlTempFile)
-	safelyCloseFile(contentTempFile)
 
-	return scanner.Err()
+	return nil
 
 }

pkg/yqlib/front_matter_test.go

@@ -60,8 +60,11 @@ yaml: doc
 
 	test.AssertResult(t, expectedYamlFm, yamlFm)
 
-	content := readFile(fmHandler.GetContentFilename())
-	test.AssertResult(t, expectedContent, content)
+	contentBytes, err := ioutil.ReadAll(fmHandler.GetContentReader())
+	if err != nil {
+		panic(err)
+	}
+	test.AssertResult(t, expectedContent, string(contentBytes))
 
 	tryRemoveFile(file)
 	fmHandler.CleanUp()
@@ -94,8 +97,11 @@ yaml: doc
 
 	test.AssertResult(t, expectedYamlFm, yamlFm)
 
-	content := readFile(fmHandler.GetContentFilename())
-	test.AssertResult(t, expectedContent, content)
+	contentBytes, err := ioutil.ReadAll(fmHandler.GetContentReader())
+	if err != nil {
+		panic(err)
+	}
+	test.AssertResult(t, expectedContent, string(contentBytes))
 
 	tryRemoveFile(file)
 	fmHandler.CleanUp()
@@ -125,8 +131,11 @@ yaml: doc
 
 	test.AssertResult(t, expectedYamlFm, yamlFm)
 
-	content := readFile(fmHandler.GetContentFilename())
-	test.AssertResult(t, expectedContent, content)
+	contentBytes, err := ioutil.ReadAll(fmHandler.GetContentReader())
+	if err != nil {
+		panic(err)
+	}
+	test.AssertResult(t, expectedContent, string(contentBytes))
 
 	tryRemoveFile(file)
 	fmHandler.CleanUp()

pkg/yqlib/lib.go

@@ -6,6 +6,8 @@ import (
 	"bytes"
 	"container/list"
 	"fmt"
+	"strconv"
+	"strings"
 
 	logging "gopkg.in/op/go-logging.v1"
 	yaml "gopkg.in/yaml.v3"
@@ -60,6 +62,7 @@ var collectOpType = &operationType{Type: "COLLECT", NumArgs: 0, Precedence: 50,
 
 var anyOpType = &operationType{Type: "ANY", NumArgs: 0, Precedence: 50, Handler: anyOperator}
 var allOpType = &operationType{Type: "ALL", NumArgs: 0, Precedence: 50, Handler: allOperator}
+var containsOpType = &operationType{Type: "CONTAINS", NumArgs: 1, Precedence: 50, Handler: containsOperator}
 var anyConditionOpType = &operationType{Type: "ANY_CONDITION", NumArgs: 1, Precedence: 50, Handler: anyOperator}
 var allConditionOpType = &operationType{Type: "ALL_CONDITION", NumArgs: 1, Precedence: 50, Handler: allOperator}
 
@@ -67,6 +70,8 @@ var toEntriesOpType = &operationType{Type: "TO_ENTRIES", NumArgs: 0, Precedence:
 var fromEntriesOpType = &operationType{Type: "FROM_ENTRIES", NumArgs: 0, Precedence: 50, Handler: fromEntriesOperator}
 var withEntriesOpType = &operationType{Type: "WITH_ENTRIES", NumArgs: 1, Precedence: 50, Handler: withEntriesOperator}
 
+var withOpType = &operationType{Type: "WITH", NumArgs: 1, Precedence: 50, Handler: withOperator}
+
 var splitDocumentOpType = &operationType{Type: "SPLIT_DOC", NumArgs: 0, Precedence: 50, Handler: splitDocumentOperator}
 var getVariableOpType = &operationType{Type: "GET_VARIABLE", NumArgs: 0, Precedence: 55, Handler: getVariableOperator}
 var getStyleOpType = &operationType{Type: "GET_STYLE", NumArgs: 0, Precedence: 50, Handler: getStyleOperator}
@@ -117,6 +122,75 @@ type Operation struct {
 	UpdateAssign  bool // used for assign ops, when true it means we evaluate the rhs given the lhs
 }
 
+func recurseNodeArrayEqual(lhs *yaml.Node, rhs *yaml.Node) bool {
+	if len(lhs.Content) != len(rhs.Content) {
+		return false
+	}
+
+	for index := 0; index < len(lhs.Content); index = index + 1 {
+		if !recursiveNodeEqual(lhs.Content[index], rhs.Content[index]) {
+			return false
+		}
+	}
+	return true
+}
+
+func findInArray(array *yaml.Node, item *yaml.Node) int {
+
+	for index := 0; index < len(array.Content); index = index + 1 {
+		if recursiveNodeEqual(array.Content[index], item) {
+			return index
+		}
+	}
+	return -1
+}
+
+func recurseNodeObjectEqual(lhs *yaml.Node, rhs *yaml.Node) bool {
+	if len(lhs.Content) != len(rhs.Content) {
+		return false
+	}
+
+	for index := 0; index < len(lhs.Content); index = index + 2 {
+		key := lhs.Content[index]
+		value := lhs.Content[index+1]
+
+		indexInRhs := findInArray(rhs, key)
+
+		if indexInRhs == -1 || !recursiveNodeEqual(value, rhs.Content[indexInRhs+1]) {
+			return false
+		}
+	}
+	return true
+}
+
+func recursiveNodeEqual(lhs *yaml.Node, rhs *yaml.Node) bool {
+	if lhs.Kind != rhs.Kind || lhs.Tag != rhs.Tag {
+		return false
+	} else if lhs.Tag == "!!null" {
+		return true
+
+	} else if lhs.Kind == yaml.ScalarNode {
+		return lhs.Value == rhs.Value
+	} else if lhs.Kind == yaml.SequenceNode {
+		return recurseNodeArrayEqual(lhs, rhs)
+	} else if lhs.Kind == yaml.MappingNode {
+		return recurseNodeObjectEqual(lhs, rhs)
+	}
+	return false
+
+}
+
+// yaml numbers can be hex encoded...
+func parseInt(numberString string) (string, int64, error) {
+	if strings.HasPrefix(numberString, "0x") ||
+		strings.HasPrefix(numberString, "0X") {
+		num, err := strconv.ParseInt(numberString[2:], 16, 64) // nolint
+		return "0x%X", num, err
+	}
+	num, err := strconv.ParseInt(numberString, 10, 64) // nolint
+	return "%v", num, err
+}
+
 func createScalarNode(value interface{}, stringValue string) *yaml.Node {
 	var node = &yaml.Node{Kind: yaml.ScalarNode}
 	node.Value = stringValue
@@ -149,6 +223,9 @@ func createValueOperation(value interface{}, stringValue string) *Operation {
 
 // debugging purposes only
 func (p *Operation) toString() string {
+	if p == nil {
+		return "OP IS NIL"
+	}
 	if p.OperationType == traversePathOpType {
 		return fmt.Sprintf("%v", p.Value)
 	} else if p.OperationType == selfReferenceOpType {
@@ -166,7 +243,7 @@ func NodesToString(collection *list.List) string {
 		return ""
 	}
 
-	result := ""
+	result := fmt.Sprintf("%v results\n", collection.Len())
 	for el := collection.Front(); el != nil; el = el.Next() {
 		result = result + "\n" + NodeToString(el.Value.(*CandidateNode))
 	}

pkg/yqlib/operator_add.go

@@ -76,17 +76,17 @@ func addScalars(target *CandidateNode, lhs *yaml.Node, rhs *yaml.Node) (*Candida
 		target.Node.Tag = "!!str"
 		target.Node.Value = lhs.Value + rhs.Value
 	} else if lhs.Tag == "!!int" && rhs.Tag == "!!int" {
-		lhsNum, err := strconv.Atoi(lhs.Value)
+		format, lhsNum, err := parseInt(lhs.Value)
 		if err != nil {
 			return nil, err
 		}
-		rhsNum, err := strconv.Atoi(rhs.Value)
+		_, rhsNum, err := parseInt(rhs.Value)
 		if err != nil {
 			return nil, err
 		}
 		sum := lhsNum + rhsNum
 		target.Node.Tag = "!!int"
-		target.Node.Value = fmt.Sprintf("%v", sum)
+		target.Node.Value = fmt.Sprintf(format, sum)
 	} else if (lhs.Tag == "!!int" || lhs.Tag == "!!float") && (rhs.Tag == "!!int" || rhs.Tag == "!!float") {
 		lhsNum, err := strconv.ParseFloat(lhs.Value, 64)
 		if err != nil {

pkg/yqlib/operator_alternative.go

@@ -2,7 +2,7 @@ package yqlib
 
 func alternativeOperator(d *dataTreeNavigator, context Context, expressionNode *ExpressionNode) (Context, error) {
 	log.Debugf("-- alternative")
-	return crossFunction(d, context.ReadOnlyClone(), expressionNode, alternativeFunc, true)
+	return crossFunction(d, context, expressionNode, alternativeFunc, true)
 }
 
 func alternativeFunc(d *dataTreeNavigator, context Context, lhs *CandidateNode, rhs *CandidateNode) (*CandidateNode, error) {

pkg/yqlib/operator_alternative_test.go

@@ -6,10 +6,13 @@ import (
 
 var alternativeOperatorScenarios = []expressionScenario{
 	{
+		// to match jq - we do not use a readonly clone context on the LHS.
 		skipDoc:    true,
 		expression: `.b // .c`,
 		document:   `a: bridge`,
-		expected:   []string{},
+		expected: []string{
+			"D0, P[c], (!!null)::null\n",
+		},
 	},
 	{
 		skipDoc:    true,

pkg/yqlib/operator_collect_object.go

@@ -20,8 +20,7 @@ import (
 func collectObjectOperator(d *dataTreeNavigator, originalContext Context, expressionNode *ExpressionNode) (Context, error) {
 	log.Debugf("-- collectObjectOperation")
 
-	context := originalContext.Clone()
-	context.DontAutoCreate = false
+	context := originalContext.WritableClone()
 
 	if context.MatchingNodes.Len() == 0 {
 		node := &yaml.Node{Kind: yaml.MappingNode, Tag: "!!map", Value: "{}"}

pkg/yqlib/operator_contains.go

@@ -0,0 +1,111 @@
+package yqlib
+
+import (
+	"fmt"
+	"strings"
+
+	yaml "gopkg.in/yaml.v3"
+)
+
+func containsOperator(d *dataTreeNavigator, context Context, expressionNode *ExpressionNode) (Context, error) {
+	return crossFunction(d, context.ReadOnlyClone(), expressionNode, containsWithNodes, false)
+}
+
+func containsArrayElement(array *yaml.Node, item *yaml.Node) (bool, error) {
+	for index := 0; index < len(array.Content); index = index + 1 {
+		containedInArray, err := contains(array.Content[index], item)
+		if err != nil {
+			return false, err
+		}
+		if containedInArray {
+			return true, nil
+		}
+	}
+	return false, nil
+}
+
+func containsArray(lhs *yaml.Node, rhs *yaml.Node) (bool, error) {
+	if rhs.Kind != yaml.SequenceNode {
+		return containsArrayElement(lhs, rhs)
+	}
+	for index := 0; index < len(rhs.Content); index = index + 1 {
+		itemInArray, err := containsArrayElement(lhs, rhs.Content[index])
+		if err != nil {
+			return false, err
+		}
+		if !itemInArray {
+			return false, nil
+		}
+	}
+	return true, nil
+}
+
+func containsObject(lhs *yaml.Node, rhs *yaml.Node) (bool, error) {
+	if rhs.Kind != yaml.MappingNode {
+		return false, nil
+	}
+	for index := 0; index < len(rhs.Content); index = index + 2 {
+		rhsKey := rhs.Content[index]
+		rhsValue := rhs.Content[index+1]
+		log.Debugf("Looking for %v in the lhs", rhsKey.Value)
+		lhsKeyIndex := findInArray(lhs, rhsKey)
+		log.Debugf("index is %v", lhsKeyIndex)
+		if lhsKeyIndex < 0 || lhsKeyIndex%2 != 0 {
+			return false, nil
+		}
+		lhsValue := lhs.Content[lhsKeyIndex+1]
+		log.Debugf("lhsValue is %v", lhsValue.Value)
+
+		itemInArray, err := contains(lhsValue, rhsValue)
+		log.Debugf("rhsValue is %v", rhsValue.Value)
+		if err != nil {
+			return false, err
+		}
+		if !itemInArray {
+			return false, nil
+		}
+	}
+	return true, nil
+}
+
+func containsScalars(lhs *yaml.Node, rhs *yaml.Node) (bool, error) {
+	if lhs.Tag == "!!str" {
+		return strings.Contains(lhs.Value, rhs.Value), nil
+	}
+	return lhs.Value == rhs.Value, nil
+}
+
+func contains(lhs *yaml.Node, rhs *yaml.Node) (bool, error) {
+	switch lhs.Kind {
+	case yaml.MappingNode:
+		return containsObject(lhs, rhs)
+	case yaml.SequenceNode:
+		return containsArray(lhs, rhs)
+	case yaml.ScalarNode:
+		if rhs.Kind != yaml.ScalarNode || lhs.Tag != rhs.Tag {
+			return false, nil
+		}
+		if lhs.Tag == "!!null" {
+			return rhs.Tag == "!!null", nil
+		}
+		return containsScalars(lhs, rhs)
+	}
+
+	return false, fmt.Errorf("%v not yet supported for contains", lhs.Tag)
+}
+
+func containsWithNodes(d *dataTreeNavigator, context Context, lhs *CandidateNode, rhs *CandidateNode) (*CandidateNode, error) {
+	lhs.Node = unwrapDoc(lhs.Node)
+	rhs.Node = unwrapDoc(rhs.Node)
+
+	if lhs.Node.Kind != rhs.Node.Kind {
+		return nil, fmt.Errorf("%v cannot check contained in %v", rhs.Node.Tag, lhs.Node.Tag)
+	}
+
+	result, err := contains(lhs.Node, rhs.Node)
+	if err != nil {
+		return nil, err
+	}
+
+	return createBooleanCandidate(lhs, result), nil
+}

pkg/yqlib/operator_contains_test.go

@@ -0,0 +1,82 @@
+package yqlib
+
+import "testing"
+
+var containsOperatorScenarios = []expressionScenario{
+	{
+		skipDoc:    true,
+		expression: `null | contains(~)`,
+		expected: []string{
+			"D0, P[], (!!bool)::true\n",
+		},
+	},
+	{
+		skipDoc:    true,
+		expression: `3 | contains(3)`,
+		expected: []string{
+			"D0, P[], (!!bool)::true\n",
+		},
+	},
+	{
+		skipDoc:    true,
+		expression: `3 | contains(32)`,
+		expected: []string{
+			"D0, P[], (!!bool)::false\n",
+		},
+	},
+	{
+		description:    "Array contains array",
+		subdescription: "Array is equal or subset of",
+		document:       `["foobar", "foobaz", "blarp"]`,
+		expression:     `contains(["baz", "bar"])`,
+		expected: []string{
+			"D0, P[], (!!bool)::true\n",
+		},
+	},
+	{
+		skipDoc:    true,
+		expression: `["dog", "cat", "giraffe"] | contains(["camel"])`,
+		expected: []string{
+			"D0, P[], (!!bool)::false\n",
+		},
+	},
+	{
+		description: "Object included in array",
+		document:    `{"foo": 12, "bar":[1,2,{"barp":12, "blip":13}]}`,
+		expression:  `contains({"bar": [{"barp": 12}]})`,
+		expected: []string{
+			"D0, P[], (!!bool)::true\n",
+		},
+	},
+	{
+		description: "Object not included in array",
+		document:    `{"foo": 12, "bar":[1,2,{"barp":12, "blip":13}]}`,
+		expression:  `contains({"foo": 12, "bar": [{"barp": 15}]})`,
+		expected: []string{
+			"D0, P[], (!!bool)::false\n",
+		},
+	},
+	{
+		description: "String contains substring",
+		document:    `"foobar"`,
+		expression:  `contains("bar")`,
+		expected: []string{
+			"D0, P[], (!!bool)::true\n",
+		},
+	},
+	{
+		description: "String equals string",
+		document:    `"meow"`,
+		expression:  `contains("meow")`,
+		expected: []string{
+			"D0, P[], (!!bool)::true\n",
+		},
+	},
+}
+
+func TestContainsOperatorScenarios(t *testing.T) {
+	for _, tt := range containsOperatorScenarios {
+		testScenario(t, &tt)
+	}
+	documentScenarios(t, "Contains", containsOperatorScenarios)
+}

pkg/yqlib/operator_multiply.go

@@ -20,8 +20,29 @@ func multiplyOperator(d *dataTreeNavigator, context Context, expressionNode *Exp
 	return crossFunction(d, context, expressionNode, multiply(expressionNode.Operation.Preferences.(multiplyPreferences)), false)
 }
 
+func getNewBlankNode(lhs *yaml.Node, rhs *yaml.Node) *yaml.Node {
+
+	blankNode := &yaml.Node{}
+
+	if lhs.HeadComment != "" {
+		blankNode.HeadComment = lhs.HeadComment
+	} else if rhs.HeadComment != "" {
+		blankNode.HeadComment = rhs.HeadComment
+	}
+
+	if lhs.FootComment != "" {
+		blankNode.FootComment = lhs.FootComment
+	} else if rhs.FootComment != "" {
+		blankNode.FootComment = rhs.FootComment
+	}
+
+	return blankNode
+}
+
 func multiply(preferences multiplyPreferences) func(d *dataTreeNavigator, context Context, lhs *CandidateNode, rhs *CandidateNode) (*CandidateNode, error) {
 	return func(d *dataTreeNavigator, context Context, lhs *CandidateNode, rhs *CandidateNode) (*CandidateNode, error) {
+		// need to do this before unWrapping the potential document node
+		newBlankNode := getNewBlankNode(lhs.Node, rhs.Node)
 		lhs.Node = unwrapDoc(lhs.Node)
 		rhs.Node = unwrapDoc(rhs.Node)
 		log.Debugf("Multipling LHS: %v", lhs.Node.Tag)
@@ -30,8 +51,8 @@ func multiply(preferences multiplyPreferences) func(d *dataTreeNavigator, contex
 		if lhs.Node.Kind == yaml.MappingNode && rhs.Node.Kind == yaml.MappingNode ||
 			(lhs.Node.Kind == yaml.SequenceNode && rhs.Node.Kind == yaml.SequenceNode) {
 
-			var newBlank = lhs.CreateChild(nil, &yaml.Node{})
-
+			var newBlank = lhs.CreateChild(nil, newBlankNode)
+			log.Debugf("merge - merge lhs into blank")
 			var newThing, err = mergeObjects(d, context.WritableClone(), newBlank, lhs, multiplyPreferences{})
 			if err != nil {
 				return nil, err
@@ -70,25 +91,26 @@ func multiplyIntegers(lhs *CandidateNode, rhs *CandidateNode) (*CandidateNode, e
 	target.Node.Style = lhs.Node.Style
 	target.Node.Tag = "!!int"
 
-	lhsNum, err := strconv.Atoi(lhs.Node.Value)
+	format, lhsNum, err := parseInt(lhs.Node.Value)
 	if err != nil {
 		return nil, err
 	}
-	rhsNum, err := strconv.Atoi(rhs.Node.Value)
+	_, rhsNum, err := parseInt(rhs.Node.Value)
 	if err != nil {
 		return nil, err
 	}
-	target.Node.Value = fmt.Sprintf("%v", lhsNum*rhsNum)
+	target.Node.Value = fmt.Sprintf(format, lhsNum*rhsNum)
 	return target, nil
 }
 
 func mergeObjects(d *dataTreeNavigator, context Context, lhs *CandidateNode, rhs *CandidateNode, preferences multiplyPreferences) (*CandidateNode, error) {
-	shouldAppendArrays := preferences.AppendArrays
 	var results = list.New()
 
-	// shouldn't recurse arrays if appending
-	prefs := recursiveDescentPreferences{RecurseArray: !shouldAppendArrays,
+	// only need to recurse the array if we are doing a deep merge
+	prefs := recursiveDescentPreferences{RecurseArray: preferences.DeepMergeArrays,
 		TraversePreferences: traversePreferences{DontFollowAlias: true, IncludeMapKeys: true}}
+	log.Debugf("merge - preferences.DeepMergeArrays %v", preferences.DeepMergeArrays)
+	log.Debugf("merge - preferences.AppendArrays %v", preferences.AppendArrays)
 	err := recursiveDecent(d, results, context.SingleChildContext(rhs), prefs)
 	if err != nil {
 		return nil, err
@@ -118,14 +140,22 @@ func applyAssignment(d *dataTreeNavigator, context Context, pathIndexToStartFrom
 	log.Debugf("merge - applyAssignment lhs %v, rhs: %v", lhs.GetKey(), rhs.GetKey())
 
 	lhsPath := rhs.Path[pathIndexToStartFrom:]
+	log.Debugf("merge - lhsPath %v", lhsPath)
 
 	assignmentOp := &Operation{OperationType: assignAttributesOpType}
 	if shouldAppendArrays && rhs.Node.Kind == yaml.SequenceNode {
 		assignmentOp.OperationType = addAssignOpType
+		log.Debugf("merge - assignmentOp.OperationType = addAssignOpType")
 	} else if !preferences.DeepMergeArrays && rhs.Node.Kind == yaml.SequenceNode ||
 		(rhs.Node.Kind == yaml.ScalarNode || rhs.Node.Kind == yaml.AliasNode) {
 		assignmentOp.OperationType = assignOpType
 		assignmentOp.UpdateAssign = false
+		log.Debugf("merge - rhs.Node.Kind == yaml.SequenceNode: %v", rhs.Node.Kind == yaml.SequenceNode)
+		log.Debugf("merge - rhs.Node.Kind == yaml.ScalarNode: %v", rhs.Node.Kind == yaml.ScalarNode)
+		log.Debugf("merge - rhs.Node.Kind == yaml.AliasNode: %v", rhs.Node.Kind == yaml.AliasNode)
+		log.Debugf("merge - assignmentOp.OperationType = assignOpType, no updateassign")
+	} else {
+		log.Debugf("merge - assignmentOp := &Operation{OperationType: assignAttributesOpType}")
 	}
 	rhsOp := &Operation{OperationType: valueOpType, CandidateNode: rhs}
 

pkg/yqlib/operator_multiply_test.go

@@ -24,13 +24,114 @@ list2:
     - "123"
 `
 
+var mergeArrayWithAnchors = `sample:
+- &a
+- <<: *a
+`
+
 var mergeArraysObjectKeysText = `It's a complex command, the trickyness comes from needing to have the right context in the expressions.
 First we save the second array into a variable '$two' which lets us reference it later.
 We then need to update the first array. We will use the relative update (|=) because we need to update relative to the current element of the array in the LHS in the RHS expression. 
 We set the current element of the first array as $cur. Now we multiply (merge) $cur with the matching entry in $two, by passing $two through a select filter.
 `
 
+var docWithHeader = `
+# here
+
+a: apple
+`
+
+var nodeWithHeader = `
+# here
+a: apple
+`
+
+var docNoComments = `
+b: banana
+`
+
+var docWithFooter = `
+a: apple
+
+# footer
+`
+
+var nodeWithFooter = `
+a: apple
+# footer`
+
 var multiplyOperatorScenarios = []expressionScenario{
+	{
+		skipDoc:    true,
+		document:   mergeArrayWithAnchors,
+		expression: `. * .`,
+		expected: []string{
+			"D0, P[], (!!map)::sample:\n    - &a\n    - !!merge <<: *a\n",
+		},
+	},
+	{
+		skipDoc:    true,
+		document:   docWithHeader,
+		document2:  docNoComments,
+		expression: `select(fi == 0) * select(fi == 1)`,
+		expected: []string{
+			"D0, P[], (!!map)::# here\na: apple\nb: banana\n",
+		},
+	},
+	{
+		skipDoc:    true,
+		document:   nodeWithHeader,
+		document2:  docNoComments,
+		expression: `select(fi == 0) * select(fi == 1)`,
+		expected: []string{
+			"D0, P[], (!!map)::# here\na: apple\nb: banana\n",
+		},
+	},
+	{
+		skipDoc:    true,
+		document:   docNoComments,
+		document2:  docWithHeader,
+		expression: `select(fi == 0) * select(fi == 1)`,
+		expected: []string{
+			"D0, P[], (!!map)::# here\nb: banana\na: apple\n",
+		},
+	},
+	{
+		skipDoc:    true,
+		document:   docNoComments,
+		document2:  nodeWithHeader,
+		expression: `select(fi == 0) * select(fi == 1)`,
+		expected: []string{
+			"D0, P[], (!!map)::b: banana\n# here\na: apple\n",
+		},
+	},
+	{
+		skipDoc:    true,
+		document:   docWithFooter,
+		document2:  docNoComments,
+		expression: `select(fi == 0) * select(fi == 1)`,
+		expected: []string{
+			"D0, P[], (!!map)::a: apple\nb: banana\n\n# footer\n",
+		},
+	},
+	{
+		skipDoc:    true,
+		document:   nodeWithFooter,
+		document2:  docNoComments,
+		expression: `select(fi == 0) * select(fi == 1)`,
+		expected: []string{ // not sure why there's an extra newline *shrug*
+			"D0, P[], (!!map)::a: apple\n# footer\n\nb: banana\n",
+		},
+	},
+	{
+		skipDoc:    true,
+		document:   docNoComments,
+		document2:  docWithFooter,
+		expression: `select(fi == 0) * select(fi == 1)`,
+		expected: []string{
+			"D0, P[], (!!map)::b: banana\na: apple\n\n# footer\n",
+		},
+	},
 	{
 		description: "Multiply integers",
 		expression:  `3 * 4`,

pkg/yqlib/operator_style_test.go

@@ -14,10 +14,9 @@ var styleOperatorScenarios = []expressionScenario{
 		},
 	},
 	{
-		description:    "Update and set style of a particular node using path variables",
-		subdescription: "You can use a variable to re-use a path",
-		document:       `a: {b: thing, c: something}`,
-		expression:     `.a.b as $x | $x = "new" | $x style="double"`,
+		description: "Update and set style of a particular node using path variables",
+		document:    `a: {b: thing, c: something}`,
+		expression:  `with(.a.b ; . = "new" | . style="double")`,
 		expected: []string{
 			"D0, P[], (doc)::a: {b: \"new\", c: something}\n",
 		},

pkg/yqlib/operator_subtract.go

@@ -5,7 +5,7 @@ import (
 
 	"strconv"
 
-	yaml "gopkg.in/yaml.v3"
+	"gopkg.in/yaml.v3"
 )
 
 func createSubtractOp(lhs *ExpressionNode, rhs *ExpressionNode) *ExpressionNode {
@@ -24,6 +24,24 @@ func subtractOperator(d *dataTreeNavigator, context Context, expressionNode *Exp
 	return crossFunction(d, context.ReadOnlyClone(), expressionNode, subtract, false)
 }
 
+func subtractArray(d *dataTreeNavigator, context Context, lhs *CandidateNode, rhs *CandidateNode) (*CandidateNode, error) {
+	newLhsArray := make([]*yaml.Node, 0)
+
+	for lindex := 0; lindex < len(lhs.Node.Content); lindex = lindex + 1 {
+		shouldInclude := true
+		for rindex := 0; rindex < len(rhs.Node.Content) && shouldInclude; rindex = rindex + 1 {
+			if recursiveNodeEqual(lhs.Node.Content[lindex], rhs.Node.Content[rindex]) {
+				shouldInclude = false
+			}
+		}
+		if shouldInclude {
+			newLhsArray = append(newLhsArray, lhs.Node.Content[lindex])
+		}
+	}
+	lhs.Node.Content = newLhsArray
+	return lhs, nil
+}
+
 func subtract(d *dataTreeNavigator, context Context, lhs *CandidateNode, rhs *CandidateNode) (*CandidateNode, error) {
 	lhs.Node = unwrapDoc(lhs.Node)
 	rhs.Node = unwrapDoc(rhs.Node)
@@ -40,14 +58,13 @@ func subtract(d *dataTreeNavigator, context Context, lhs *CandidateNode, rhs *Ca
 	case yaml.MappingNode:
 		return nil, fmt.Errorf("Maps not yet supported for subtraction")
 	case yaml.SequenceNode:
-		return nil, fmt.Errorf("Sequences not yet supported for subtraction")
-		// target.Node.Kind = yaml.SequenceNode
-		// target.Node.Style = lhsNode.Style
-		// target.Node.Tag = "!!seq"
-		// target.Node.Content = append(lhsNode.Content, toNodes(rhs)...)
+		if rhs.Node.Kind != yaml.SequenceNode {
+			return nil, fmt.Errorf("%v (%v) cannot be subtracted from %v", rhs.Node.Tag, rhs.Path, lhsNode.Tag)
+		}
+		return subtractArray(d, context, lhs, rhs)
 	case yaml.ScalarNode:
 		if rhs.Node.Kind != yaml.ScalarNode {
-			return nil, fmt.Errorf("%v (%v) cannot be added to a %v", rhs.Node.Tag, rhs.Path, lhsNode.Tag)
+			return nil, fmt.Errorf("%v (%v) cannot be subtracted from %v", rhs.Node.Tag, rhs.Path, lhsNode.Tag)
 		}
 		target.Node.Kind = yaml.ScalarNode
 		target.Node.Style = lhsNode.Style
@@ -62,17 +79,17 @@ func subtractScalars(target *CandidateNode, lhs *yaml.Node, rhs *yaml.Node) (*Ca
 	if lhs.Tag == "!!str" {
 		return nil, fmt.Errorf("strings cannot be subtracted")
 	} else if lhs.Tag == "!!int" && rhs.Tag == "!!int" {
-		lhsNum, err := strconv.Atoi(lhs.Value)
+		format, lhsNum, err := parseInt(lhs.Value)
 		if err != nil {
 			return nil, err
 		}
-		rhsNum, err := strconv.Atoi(rhs.Value)
+		_, rhsNum, err := parseInt(rhs.Value)
 		if err != nil {
 			return nil, err
 		}
 		result := lhsNum - rhsNum
 		target.Node.Tag = "!!int"
-		target.Node.Value = fmt.Sprintf("%v", result)
+		target.Node.Value = fmt.Sprintf(format, result)
 	} else if (lhs.Tag == "!!int" || lhs.Tag == "!!float") && (rhs.Tag == "!!int" || rhs.Tag == "!!float") {
 		lhsNum, err := strconv.ParseFloat(lhs.Value, 64)
 		if err != nil {

pkg/yqlib/operator_subtract_test.go

@@ -13,6 +13,51 @@ var subtractOperatorScenarios = []expressionScenario{
 			"D0, P[], (doc)::{}\n",
 		},
 	},
+	{
+		description: "Array subtraction",
+		expression:  `[1,2] - [2,3]`,
+		expected: []string{
+			"D0, P[], (!!seq)::- 1\n",
+		},
+	},
+	{
+		skipDoc:    true,
+		expression: `[2,1,2,2] - [2,3]`,
+		expected: []string{
+			"D0, P[], (!!seq)::- 1\n",
+		},
+	},
+	{
+		description: "Array subtraction with nested array",
+		expression:  `[[1], 1, 2] - [[1], 3]`,
+		expected: []string{
+			"D0, P[], (!!seq)::- 1\n- 2\n",
+		},
+	},
+	{
+		skipDoc:    true,
+		expression: `[[1], 1, [[[2]]]] - [[1], [[[3]]]]`,
+		expected: []string{
+			"D0, P[], (!!seq)::- 1\n- - - - 2\n",
+		},
+	},
+	{
+		description:    "Array subtraction with nested object",
+		subdescription: `Note that order of the keys does not matter`,
+		document:       `[{a: b, c: d}, {a: b}]`,
+		expression:     `. - [{"c": "d", "a": "b"}]`,
+		expected: []string{
+			"D0, P[], (!!seq)::[{a: b}]\n",
+		},
+	},
+	{
+		skipDoc:    true,
+		document:   `[{a: [1], c: d}, {a: [2], c: d}, {a: b}]`,
+		expression: `. - [{"c": "d", "a": [1]}]`,
+		expected: []string{
+			"D0, P[], (!!seq)::[{a: [2], c: d}, {a: b}]\n",
+		},
+	},
 	{
 		description:    "Number subtraction - float",
 		subdescription: "If the lhs or rhs are floats then the expression will be calculated with floats.",

pkg/yqlib/operator_union.go

@@ -1,17 +1,36 @@
 package yqlib
 
 func unionOperator(d *dataTreeNavigator, context Context, expressionNode *ExpressionNode) (Context, error) {
+	log.Debug("unionOperator")
+	log.Debug("context: %v", NodesToString(context.MatchingNodes))
 	lhs, err := d.GetMatchingNodes(context, expressionNode.Lhs)
 	if err != nil {
 		return Context{}, err
 	}
+	log.Debug("lhs: %v", NodesToString(lhs.MatchingNodes))
+	log.Debug("rhs input: %v", NodesToString(context.MatchingNodes))
+	log.Debug("rhs: %v", expressionNode.Rhs.Operation.toString())
 	rhs, err := d.GetMatchingNodes(context, expressionNode.Rhs)
+
 	if err != nil {
 		return Context{}, err
 	}
-	for el := rhs.MatchingNodes.Front(); el != nil; el = el.Next() {
-		node := el.Value.(*CandidateNode)
-		lhs.MatchingNodes.PushBack(node)
+	log.Debug("lhs: %v", lhs.ToString())
+	log.Debug("rhs: %v", rhs.ToString())
+
+	// this can happen when both expressions modify the context
+	// instead of creating their own.
+	/// (.foo = "bar"), (.thing = "cat")
+
+	if rhs.MatchingNodes != lhs.MatchingNodes {
+
+		for el := rhs.MatchingNodes.Front(); el != nil; el = el.Next() {
+			node := el.Value.(*CandidateNode)
+			log.Debug("processing %v", NodeToString(node))
+
+			lhs.MatchingNodes.PushBack(node)
+		}
 	}
+	log.Debug("all together: %v", lhs.ToString())
 	return lhs, nil
 }

pkg/yqlib/operator_union_test.go

@@ -5,20 +5,21 @@ import (
 )
 
 var unionOperatorScenarios = []expressionScenario{
-	// {
-	// 	skipDoc:    true,
-	// 	document:   "{}",
-	// 	expression: `(.a, .b.c) as $x`,
-	// 	expected: []string{
-	// 		"D0, P[], (doc)::{}\n",
-	// 	},
-	// },
-	// {
-	// 	skipDoc:    true,
-	// 	document:   "{}",
-	// 	expression: `(.a, .b.c)`,
-	// 	expected:   []string{},
-	// },
+	{
+		skipDoc:    true,
+		document:   "{}",
+		expression: `(.a, .b.c) as $x`,
+		expected: []string{
+			"D0, P[], (doc)::{}\n",
+		},
+	},
+	{
+		skipDoc:    true,
+		expression: `(.foo = "bar"), (.toe = "jam")`,
+		expected: []string{
+			"D0, P[], ()::foo: bar\ntoe: jam\n",
+		},
+	},
 	{
 		description: "Combine scalars",
 		expression:  `1, true, "cat"`,

pkg/yqlib/operator_value_test.go

@@ -12,6 +12,76 @@ var valueOperatorScenarios = []expressionScenario{
 			"D0, P[], (!!int)::1\n",
 		},
 	},
+	{
+		document:   ``,
+		expression: `0x9f`,
+		expected: []string{
+			"D0, P[], (!!int)::0x9f\n",
+		},
+	},
+	{
+		document:   ``,
+		expression: `0x1A`,
+		expected: []string{
+			"D0, P[], (!!int)::0x1A\n",
+		},
+	},
+	{
+		document:   ``,
+		expression: `0x1A + 2`,
+		expected: []string{
+			"D0, P[], (!!int)::0x1C\n",
+		},
+	},
+	{
+		document:   ``,
+		expression: `0x12 * 2`,
+		expected: []string{
+			"D0, P[], (!!int)::0x24\n",
+		},
+	},
+	{
+		document:   ``,
+		expression: `0xF - 1`,
+		expected: []string{
+			"D0, P[], (!!int)::0xE\n",
+		},
+	},
+	{
+		document:   ``,
+		expression: `12`,
+		expected: []string{
+			"D0, P[], (!!int)::12\n",
+		},
+	},
+	{
+		document:   ``,
+		expression: `12 + 2`,
+		expected: []string{
+			"D0, P[], (!!int)::14\n",
+		},
+	},
+	{
+		document:   ``,
+		expression: `12 * 2`,
+		expected: []string{
+			"D0, P[], (!!int)::24\n",
+		},
+	},
+	{
+		document:   ``,
+		expression: `12 - 2`,
+		expected: []string{
+			"D0, P[], (!!int)::10\n",
+		},
+	},
+	{
+		document:   ``,
+		expression: `0X12`,
+		expected: []string{
+			"D0, P[], (!!int)::0X12\n",
+		},
+	},
 	{
 		document:   ``,
 		expression: `-1`,

pkg/yqlib/operator_variables.go

@@ -15,6 +15,10 @@ func getVariableOperator(d *dataTreeNavigator, context Context, expressionNode *
 	return context.ChildContext(result), nil
 }
 
+type assignVarPreferences struct {
+	IsReference bool
+}
+
 func assignVariableOperator(d *dataTreeNavigator, context Context, expressionNode *ExpressionNode) (Context, error) {
 	lhs, err := d.GetMatchingNodes(context.ReadOnlyClone(), expressionNode.Lhs)
 	if err != nil {
@@ -24,6 +28,15 @@ func assignVariableOperator(d *dataTreeNavigator, context Context, expressionNod
 		return Context{}, fmt.Errorf("RHS of 'as' operator must be a variable name e.g. $foo")
 	}
 	variableName := expressionNode.Rhs.Operation.StringValue
-	context.SetVariable(variableName, lhs.MatchingNodes)
+
+	prefs := expressionNode.Operation.Preferences.(assignVarPreferences)
+
+	var variableValue *list.List
+	if prefs.IsReference {
+		variableValue = lhs.MatchingNodes
+	} else {
+		variableValue = lhs.DeepClone().MatchingNodes
+	}
+	context.SetVariable(variableName, variableValue)
 	return context, nil
 }

pkg/yqlib/operator_variables_test.go

@@ -34,15 +34,32 @@ var variableOperatorScenarios = []expressionScenario{
 		description:    "Using variables as a lookup",
 		subdescription: "Example taken from [jq](https://stedolan.github.io/jq/manual/#Variable/SymbolicBindingOperator:...as$identifier|...)",
 		document: `{"posts": [{"title": "Frist psot", "author": "anon"},
-		{"title": "A well-written article", "author": "person1"}],
-"realnames": {"anon": "Anonymous Coward",
-				"person1": "Person McPherson"}}`,
+			{"title": "A well-written article", "author": "person1"}],
+	"realnames": {"anon": "Anonymous Coward",
+					"person1": "Person McPherson"}}`,
 		expression: `.realnames as $names | .posts[] | {"title":.title, "author": $names[.author]}`,
 		expected: []string{
 			"D0, P[], (!!map)::title: \"Frist psot\"\nauthor: \"Anonymous Coward\"\n",
 			"D0, P[], (!!map)::title: \"A well-written article\"\nauthor: \"Person McPherson\"\n",
 		},
 	},
+	{
+		description: "Using variables to swap values",
+		document:    "a: a_value\nb: b_value",
+		expression:  `.a as $x  | .b as $y | .b = $x | .a = $y`,
+		expected: []string{
+			"D0, P[], (doc)::a: b_value\nb: a_value\n",
+		},
+	},
+	{
+		description:    "Use ref to reference a path repeatedly",
+		subdescription: "Note: You may find the `with` operator more useful.",
+		document:       `a: {b: thing, c: something}`,
+		expression:     `.a.b ref $x | $x = "new" | $x style="double"`,
+		expected: []string{
+			"D0, P[], (doc)::a: {b: \"new\", c: something}\n",
+		},
+	},
 }
 
 func TestVariableOperatorScenarios(t *testing.T) {

pkg/yqlib/operator_with.go

@@ -0,0 +1,30 @@
+package yqlib
+
+import "fmt"
+
+func withOperator(d *dataTreeNavigator, context Context, expressionNode *ExpressionNode) (Context, error) {
+	log.Debugf("-- withOperator")
+	// with(path, exp)
+
+	if expressionNode.Rhs.Operation.OperationType != blockOpType {
+		return Context{}, fmt.Errorf("with must be given a block, got %v instead", expressionNode.Rhs.Operation.OperationType.Type)
+	}
+
+	pathExp := expressionNode.Rhs.Lhs
+
+	updateContext, err := d.GetMatchingNodes(context, pathExp)
+
+	if err != nil {
+		return Context{}, err
+	}
+
+	updateExp := expressionNode.Rhs.Rhs
+
+	_, err = d.GetMatchingNodes(updateContext, updateExp)
+	if err != nil {
+		return Context{}, err
+	}
+
+	return context, nil
+
+}

pkg/yqlib/operator_with_test.go

@@ -0,0 +1,37 @@
+package yqlib
+
+import "testing"
+
+var withOperatorScenarios = []expressionScenario{
+	{
+		description: "Update and style",
+		document:    `a: {deeply: {nested: value}}`,
+		expression:  `with(.a.deeply.nested ; . = "newValue" | . style="single")`,
+		expected: []string{
+			"D0, P[], (doc)::a: {deeply: {nested: 'newValue'}}\n",
+		},
+	},
+	{
+		description: "Update multiple deeply nested properties",
+		document:    `a: {deeply: {nested: value, other: thing}}`,
+		expression:  `with(.a.deeply ; .nested = "newValue" | .other= "newThing")`,
+		expected: []string{
+			"D0, P[], (doc)::a: {deeply: {nested: newValue, other: newThing}}\n",
+		},
+	},
+	{
+		description: "Update array elements relatively",
+		document:    `myArray: [{a: apple},{a: banana}]`,
+		expression:  `with(.myArray[] ; .b = .a + " yum")`,
+		expected: []string{
+			"D0, P[], (doc)::myArray: [{a: apple, b: apple yum}, {a: banana, b: banana yum}]\n",
+		},
+	},
+}
+
+func TestWithOperatorScenarios(t *testing.T) {
+	for _, tt := range withOperatorScenarios {
+		testScenario(t, &tt)
+	}
+	documentScenarios(t, "With", withOperatorScenarios)
+}

pkg/yqlib/operators_test.go

@@ -107,7 +107,7 @@ func copyFromHeader(title string, out *os.File) error {
 
 func formatYaml(yaml string, filename string) string {
 	var output bytes.Buffer
-	printer := NewPrinter(bufio.NewWriter(&output), false, true, false, 2, true)
+	printer := NewPrinter(bufio.NewWriter(&output), YamlOutputFormat, true, false, 2, true)
 
 	node, err := NewExpressionParser().ParseExpression(".. style= \"\"")
 	if err != nil {
@@ -216,7 +216,7 @@ func documentInput(w *bufio.Writer, s expressionScenario) (string, string) {
 func documentOutput(t *testing.T, w *bufio.Writer, s expressionScenario, formattedDoc string, formattedDoc2 string) {
 	var output bytes.Buffer
 	var err error
-	printer := NewPrinter(bufio.NewWriter(&output), false, true, false, 2, true)
+	printer := NewPrinter(bufio.NewWriter(&output), YamlOutputFormat, true, false, 2, true)
 
 	node, err := NewExpressionParser().ParseExpression(s.expression)
 	if err != nil {

pkg/yqlib/printer.go

@@ -3,6 +3,7 @@ package yqlib
 import (
 	"bufio"
 	"container/list"
+	"fmt"
 	"io"
 	"strings"
 
@@ -16,8 +17,29 @@ type Printer interface {
 	SetAppendix(reader io.Reader)
 }
 
+type PrinterOutputFormat uint32
+
+const (
+	YamlOutputFormat = 1 << iota
+	JsonOutputFormat
+	PropsOutputFormat
+)
+
+func OutputFormatFromString(format string) (PrinterOutputFormat, error) {
+	switch format {
+	case "yaml", "y":
+		return YamlOutputFormat, nil
+	case "json", "j":
+		return JsonOutputFormat, nil
+	case "props", "p":
+		return PropsOutputFormat, nil
+	default:
+		return 0, fmt.Errorf("Unknown fromat '%v' please use [yaml|json|props]", format)
+	}
+}
+
 type resultsPrinter struct {
-	outputToJSON       bool
+	outputFormat       PrinterOutputFormat
 	unwrapScalar       bool
 	colorsEnabled      bool
 	indent             int
@@ -28,27 +50,22 @@ type resultsPrinter struct {
 	previousFileIndex  int
 	printedMatches     bool
 	treeNavigator      DataTreeNavigator
-	preambleReader     io.Reader
 	appendixReader     io.Reader
 }
 
-func NewPrinter(writer io.Writer, outputToJSON bool, unwrapScalar bool, colorsEnabled bool, indent int, printDocSeparators bool) Printer {
+func NewPrinter(writer io.Writer, outputFormat PrinterOutputFormat, unwrapScalar bool, colorsEnabled bool, indent int, printDocSeparators bool) Printer {
 	return &resultsPrinter{
 		writer:             writer,
-		outputToJSON:       outputToJSON,
+		outputFormat:       outputFormat,
 		unwrapScalar:       unwrapScalar,
 		colorsEnabled:      colorsEnabled,
 		indent:             indent,
-		printDocSeparators: !outputToJSON && printDocSeparators,
+		printDocSeparators: outputFormat == YamlOutputFormat && printDocSeparators,
 		firstTimePrinting:  true,
 		treeNavigator:      NewDataTreeNavigator(),
 	}
 }
 
-func (p *resultsPrinter) SetPreamble(reader io.Reader) {
-	p.preambleReader = reader
-}
-
 func (p *resultsPrinter) SetAppendix(reader io.Reader) {
 	p.appendixReader = reader
 }
@@ -62,11 +79,14 @@ func (p *resultsPrinter) printNode(node *yaml.Node, writer io.Writer) error {
 		(node.Tag != "!!bool" || node.Value != "false"))
 
 	var encoder Encoder
-	if node.Kind == yaml.ScalarNode && p.unwrapScalar && !p.outputToJSON {
+	if node.Kind == yaml.ScalarNode && p.unwrapScalar && p.outputFormat == YamlOutputFormat {
 		return p.writeString(writer, node.Value+"\n")
 	}
-	if p.outputToJSON {
+
+	if p.outputFormat == JsonOutputFormat {
 		encoder = NewJsonEncoder(writer, p.indent)
+	} else if p.outputFormat == PropsOutputFormat {
+		encoder = NewPropertiesEncoder(writer)
 	} else {
 		encoder = NewYamlEncoder(writer, p.indent, p.colorsEnabled)
 	}
@@ -87,7 +107,7 @@ func (p *resultsPrinter) safelyFlush(writer *bufio.Writer) {
 
 func (p *resultsPrinter) PrintResults(matchingNodes *list.List) error {
 	log.Debug("PrintResults for %v matches", matchingNodes.Len())
-	if p.outputToJSON {
+	if p.outputFormat != YamlOutputFormat {
 		explodeOp := Operation{OperationType: explodeOpType}
 		explodeNode := ExpressionNode{Operation: &explodeOp}
 		context, err := p.treeNavigator.GetMatchingNodes(Context{MatchingNodes: matchingNodes}, &explodeNode)
@@ -145,18 +165,25 @@ func (p *resultsPrinter) PrintResults(matchingNodes *list.List) error {
 							return err
 						}
 					}
-				} else if !p.outputToJSON {
+				} else if p.outputFormat == YamlOutputFormat {
 					if err := p.writeString(bufferedWriter, readline); err != nil {
 						return err
 					}
 				}
 
 				if errReading == io.EOF {
+					if readline != "" {
+						// the last comment we read didn't have a new line, put one in
+						if err := p.writeString(bufferedWriter, "\n"); err != nil {
+							return err
+						}
+					}
 					break
 				}
 			}
 
 		}
+
 		if err := p.printNode(mappedDoc.Node, bufferedWriter); err != nil {
 			return err
 		}
@@ -164,7 +191,7 @@ func (p *resultsPrinter) PrintResults(matchingNodes *list.List) error {
 		p.previousDocIndex = mappedDoc.Document
 	}
 
-	if p.appendixReader != nil && !p.outputToJSON {
+	if p.appendixReader != nil && p.outputFormat == YamlOutputFormat {
 		log.Debug("Piping appendix reader...")
 		betterReader := bufio.NewReader(p.appendixReader)
 		_, err := io.Copy(bufferedWriter, betterReader)

pkg/yqlib/printer_test.go

@@ -36,7 +36,7 @@ func nodeToList(candidate *CandidateNode) *list.List {
 func TestPrinterMultipleDocsInSequence(t *testing.T) {
 	var output bytes.Buffer
 	var writer = bufio.NewWriter(&output)
-	printer := NewPrinter(writer, false, true, false, 2, true)
+	printer := NewPrinter(writer, YamlOutputFormat, true, false, 2, true)
 
 	inputs, err := readDocuments(strings.NewReader(multiDocSample), "sample.yml", 0)
 	if err != nil {
@@ -74,7 +74,7 @@ func TestPrinterMultipleDocsInSequence(t *testing.T) {
 func TestPrinterMultipleDocsInSequenceWithLeadingContent(t *testing.T) {
 	var output bytes.Buffer
 	var writer = bufio.NewWriter(&output)
-	printer := NewPrinter(writer, false, true, false, 2, true)
+	printer := NewPrinter(writer, YamlOutputFormat, true, false, 2, true)
 
 	inputs, err := readDocuments(strings.NewReader(multiDocSample), "sample.yml", 0)
 	if err != nil {
@@ -116,7 +116,7 @@ func TestPrinterMultipleDocsInSequenceWithLeadingContent(t *testing.T) {
 func TestPrinterMultipleFilesInSequence(t *testing.T) {
 	var output bytes.Buffer
 	var writer = bufio.NewWriter(&output)
-	printer := NewPrinter(writer, false, true, false, 2, true)
+	printer := NewPrinter(writer, YamlOutputFormat, true, false, 2, true)
 
 	inputs, err := readDocuments(strings.NewReader(multiDocSample), "sample.yml", 0)
 	if err != nil {
@@ -163,7 +163,7 @@ func TestPrinterMultipleFilesInSequence(t *testing.T) {
 func TestPrinterMultipleFilesInSequenceWithLeadingContent(t *testing.T) {
 	var output bytes.Buffer
 	var writer = bufio.NewWriter(&output)
-	printer := NewPrinter(writer, false, true, false, 2, true)
+	printer := NewPrinter(writer, YamlOutputFormat, true, false, 2, true)
 
 	inputs, err := readDocuments(strings.NewReader(multiDocSample), "sample.yml", 0)
 	if err != nil {
@@ -213,7 +213,7 @@ func TestPrinterMultipleFilesInSequenceWithLeadingContent(t *testing.T) {
 func TestPrinterMultipleDocsInSinglePrint(t *testing.T) {
 	var output bytes.Buffer
 	var writer = bufio.NewWriter(&output)
-	printer := NewPrinter(writer, false, true, false, 2, true)
+	printer := NewPrinter(writer, YamlOutputFormat, true, false, 2, true)
 
 	inputs, err := readDocuments(strings.NewReader(multiDocSample), "sample.yml", 0)
 	if err != nil {
@@ -232,7 +232,7 @@ func TestPrinterMultipleDocsInSinglePrint(t *testing.T) {
 func TestPrinterMultipleDocsInSinglePrintWithLeadingDoc(t *testing.T) {
 	var output bytes.Buffer
 	var writer = bufio.NewWriter(&output)
-	printer := NewPrinter(writer, false, true, false, 2, true)
+	printer := NewPrinter(writer, YamlOutputFormat, true, false, 2, true)
 
 	inputs, err := readDocuments(strings.NewReader(multiDocSample), "sample.yml", 0)
 	if err != nil {
@@ -261,7 +261,7 @@ a: coconut
 func TestPrinterMultipleDocsInSinglePrintWithLeadingDocTrailing(t *testing.T) {
 	var output bytes.Buffer
 	var writer = bufio.NewWriter(&output)
-	printer := NewPrinter(writer, false, true, false, 2, true)
+	printer := NewPrinter(writer, YamlOutputFormat, true, false, 2, true)
 
 	inputs, err := readDocuments(strings.NewReader(multiDocSample), "sample.yml", 0)
 	if err != nil {
@@ -287,7 +287,7 @@ a: coconut
 func TestPrinterScalarWithLeadingCont(t *testing.T) {
 	var output bytes.Buffer
 	var writer = bufio.NewWriter(&output)
-	printer := NewPrinter(writer, false, true, false, 2, true)
+	printer := NewPrinter(writer, YamlOutputFormat, true, false, 2, true)
 
 	node, err := NewExpressionParser().ParseExpression(".a")
 	if err != nil {
@@ -314,7 +314,7 @@ func TestPrinterMultipleDocsJson(t *testing.T) {
 	var writer = bufio.NewWriter(&output)
 	// note printDocSeparators is true, it should still not print document separators
 	// when outputing JSON.
-	printer := NewPrinter(writer, true, true, false, 0, true)
+	printer := NewPrinter(writer, JsonOutputFormat, true, false, 0, true)
 
 	inputs, err := readDocuments(strings.NewReader(multiDocSample), "sample.yml", 0)
 	if err != nil {

release_instructions.txt

@@ -1,3 +1,4 @@
+- update release notes
 - increment version in version.go
 - increment version in snapcraft.yaml
 - increment version in github-action/Dockerfile

release_notes.txt

@@ -0,0 +1,29 @@
+4.13.0:
+
+BREAKING CHANGE - the `as` variable operator (e.g. `.a as $x`) now makes a _copy_ of the node(s) at the 
+path rather than a reference. This is in order to make it work more like the `jq` equivalent. 
+
+This means any updates made to that variable do not update the original.
+
+There's a new operator `ref` that will make a reference (and allow multiple updates to the original path by referencing the variable).
+Sorry for any inconvenience caused!.
+
+
+- New `with` operator for making multiple changes to a given path
+- New `contains` operator, works like the `jq` equivalent
+- Subtract operator now supports subtracting elements from arrays!
+- Fixed Swapping values using variables #934
+- Github Action now properly supports multiline output #936, thanks @pjxiao
+- Fixed missing closing bracket validation #932
+
+4.12.2:
+- Fix processing of hex numbers #929
+- Fixed alternative and union operator issues #930
+
+4.12.1:
+ - Merge comment fix #919
+
+4.12.0:
+- Can now convert yaml to properties properties format (`-o=props`), See [docs](https://mikefarah.gitbook.io/yq/v/v4.x/usage/properties) for more info.
+- Fixed document header/footer comment handling when merging (https://github.com/mikefarah/yq/issues/919)
+- pretty print yaml 1.1 compatibility  (https://github.com/mikefarah/yq/issues/914)

snap/snapcraft.yaml

@@ -1,5 +1,5 @@
 name: yq
-version: '4.11.0'
+version: '4.13.0'
 summary: A lightweight and portable command-line YAML processor
 description: |
   The aim of the project is to be the jq or sed of yaml files.