v4.50.1 - HCL

README.md

@@ -1,6 +1,6 @@
 # yq
 
-aa lightweight and portable command-line YAML, JSON, INI and XML processor. `yq` uses [jq](https://github.com/stedolan/jq) like syntax but works with yaml files as well as json, xml, ini, properties, csv and tsv. It doesn't yet support everything `jq` does - but it does support the most common operations and functions, and more is being added continuously.
+a lightweight and portable command-line YAML, JSON, INI and XML processor. `yq` uses [jq](https://github.com/stedolan/jq) like syntax but works with yaml files as well as json, xml, ini, properties, csv and tsv. It doesn't yet support everything `jq` does - but it does support the most common operations and functions, and more is being added continuously.
 
 yq is written in go - so you can download a dependency free binary for your platform and you are good to go! If you prefer there are a variety of package managers that can be used as well as Docker and Podman, all listed below.
 

SUMMARY.md

@@ -80,10 +80,12 @@
 ## Usage
 
 * [Output format](usage/output-format.md)
+* [Working with Base64](usage/base64.md)
 * [Working with CSV, TSV](usage/csv-tsv.md)
 * [Working with JSON](usage/convert.md)
 * [Working with Properties](usage/properties.md)
 * [Working with XML](usage/xml.md)
+* [Working with HCL](usage/hcl.md)
 * [Working with LUA](usage/lua.md)
 * [Working with TOML](usage/toml.md)
 * [Working with Shell Output](usage/shellvariables.md)

operators/anchor-and-alias-operators.md

@@ -191,7 +191,7 @@ Given a sample.yml file of:
 ```yaml
 f:
   a: &a cat
-  *a: b
+  *a : b
 ```
 then
 ```bash

operators/env-variable-operators.md

@@ -29,6 +29,9 @@ as follows:
 yq '(.. | select(tag == "!!str")) |= envsubst' file.yaml
 ```
 
+## Disabling env operators
+If required, you can use the `--security-disable-env-ops` to disable env operations.
+
 
 ## Read string environment variable
 Running
@@ -254,3 +257,39 @@ will output
 Error: variable ${notThere} not set
 ```
 
+## env() operation fails when security is enabled
+Use `--security-disable-env-ops` to disable env operations for security.
+
+Running
+```bash
+yq --null-input 'env("MYENV")'
+```
+will output
+```bash
+Error: env operations have been disabled
+```
+
+## strenv() operation fails when security is enabled
+Use `--security-disable-env-ops` to disable env operations for security.
+
+Running
+```bash
+yq --null-input 'strenv("MYENV")'
+```
+will output
+```bash
+Error: env operations have been disabled
+```
+
+## envsubst() operation fails when security is enabled
+Use `--security-disable-env-ops` to disable env operations for security.
+
+Running
+```bash
+yq --null-input '"value: ${MYENV}" | envsubst'
+```
+will output
+```bash
+Error: env operations have been disabled
+```
+

operators/load.md

@@ -47,6 +47,10 @@ this.is = a properties file
 bXkgc2VjcmV0IGNoaWxsaSByZWNpcGUgaXMuLi4u
 ```
 
+## Disabling file operators
+If required, you can use the `--security-disable-file-ops` to disable file operations.
+
+
 ## Simple example
 Given a sample.yml file of:
 ```yaml
@@ -194,3 +198,63 @@ cool: things
 more_stuff: my secret chilli recipe is....
 ```
 
+## load() operation fails when security is enabled
+Use `--security-disable-file-ops` to disable file operations for security.
+
+Running
+```bash
+yq --null-input 'load("../../examples/thing.yml")'
+```
+will output
+```bash
+Error: file operations have been disabled
+```
+
+## load_str() operation fails when security is enabled
+Use `--security-disable-file-ops` to disable file operations for security.
+
+Running
+```bash
+yq --null-input 'load_str("../../examples/thing.yml")'
+```
+will output
+```bash
+Error: file operations have been disabled
+```
+
+## load_xml() operation fails when security is enabled
+Use `--security-disable-file-ops` to disable file operations for security.
+
+Running
+```bash
+yq --null-input 'load_xml("../../examples/small.xml")'
+```
+will output
+```bash
+Error: file operations have been disabled
+```
+
+## load_props() operation fails when security is enabled
+Use `--security-disable-file-ops` to disable file operations for security.
+
+Running
+```bash
+yq --null-input 'load_props("../../examples/small.properties")'
+```
+will output
+```bash
+Error: file operations have been disabled
+```
+
+## load_base64() operation fails when security is enabled
+Use `--security-disable-file-ops` to disable file operations for security.
+
+Running
+```bash
+yq --null-input 'load_base64("../../examples/base64.txt")'
+```
+will output
+```bash
+Error: file operations have been disabled
+```
+

operators/traverse-read.md

@@ -3,7 +3,7 @@
 This is the simplest (and perhaps most used) operator. It is used to navigate deeply into yaml structures.
 
 
-## NOTE --yaml-fix-merge-anchor-to-spec flag
+## NOTE --yaml-fix-merge-anchor-to-spec flag (v4.47.1 [#2110](https://github.com/mikefarah/yq/issues/2110))
 `yq` doesn't merge anchors `<<:` to spec, in some circumstances it incorrectly overrides existing keys when the spec documents not to do that.
 
 To minimise disruption while still fixing the issue, a flag has been added to toggle this behaviour. This will first default to false; and log warnings to users. Then it will default to true (and still allow users to specify false if needed)

usage/base64.md

@@ -0,0 +1,88 @@
+# Base64
+
+Encode and decode to and from Base64.
+
+Base64 assumes [RFC4648](https://rfc-editor.org/rfc/rfc4648.html) encoding. Encoding and decoding both assume that the content is a UTF-8 string and not binary content.
+
+
+See below for examples
+
+
+## Decode base64: simple
+Decoded data is assumed to be a string.
+
+Given a sample.txt file of:
+```
+YSBzcGVjaWFsIHN0cmluZw==
+```
+then
+```bash
+yq -p=base64 -oy '.' sample.txt
+```
+will output
+```yaml
+a special string
+```
+
+## Decode base64: UTF-8
+Base64 decoding supports UTF-8 encoded strings.
+
+Given a sample.txt file of:
+```
+V29ya3Mgd2l0aCBVVEYtMTYg8J+Yig==
+```
+then
+```bash
+yq -p=base64 -oy '.' sample.txt
+```
+will output
+```yaml
+Works with UTF-16 😊
+```
+
+## Decode with extra spaces
+Extra leading/trailing whitespace is stripped
+
+Given a sample.txt file of:
+```
+
+ YSBzcGVjaWFsIHN0cmluZw==  
+
+```
+then
+```bash
+yq -p=base64 -oy '.' sample.txt
+```
+will output
+```yaml
+a special string
+```
+
+## Encode base64: string
+Given a sample.yml file of:
+```yaml
+"a special string"
+```
+then
+```bash
+yq -o=base64 '.' sample.yml
+```
+will output
+```
+YSBzcGVjaWFsIHN0cmluZw==```
+
+## Encode base64: string from document
+Extract a string field and encode it to base64.
+
+Given a sample.yml file of:
+```yaml
+coolData: "a special string"
+```
+then
+```bash
+yq -o=base64 '.coolData' sample.yml
+```
+will output
+```
+YSBzcGVjaWFsIHN0cmluZw==```
+

usage/hcl.md

@@ -0,0 +1,201 @@
+# HCL
+
+Encode and decode to and from [HashiCorp Configuration Language (HCL)](https://github.com/hashicorp/hcl).
+
+HCL is commonly used in HashiCorp tools like Terraform for configuration files. The yq HCL encoder and decoder support:
+- Blocks and attributes
+- String interpolation and expressions (preserved without quotes)
+- Comments (leading, head, and line comments)
+- Nested structures (maps and lists)
+- Syntax colorization when enabled
+
+
+## Parse HCL
+Given a sample.hcl file of:
+```hcl
+io_mode = "async"
+```
+then
+```bash
+yq -oy sample.hcl
+```
+will output
+```yaml
+io_mode: "async"
+```
+
+## Roundtrip: Sample Doc
+Given a sample.hcl file of:
+```hcl
+service "cat" {
+  process "main" {
+    command = ["/usr/local/bin/awesome-app", "server"]
+  }
+
+  process "management" {
+    command = ["/usr/local/bin/awesome-app", "management"]
+  }
+}
+
+```
+then
+```bash
+yq sample.hcl
+```
+will output
+```hcl
+service "cat" {
+  process "main" {
+    command = ["/usr/local/bin/awesome-app", "server"]
+  }
+  process "management" {
+    command = ["/usr/local/bin/awesome-app", "management"]
+  }
+}
+```
+
+## Roundtrip: With an update
+Given a sample.hcl file of:
+```hcl
+service "cat" {
+  process "main" {
+    command = ["/usr/local/bin/awesome-app", "server"]
+  }
+
+  process "management" {
+    command = ["/usr/local/bin/awesome-app", "management"]
+  }
+}
+
+```
+then
+```bash
+yq '.service.cat.process.main.command += "meow"' sample.hcl
+```
+will output
+```hcl
+service "cat" {
+  process "main" {
+    command = ["/usr/local/bin/awesome-app", "server", "meow"]
+  }
+  process "management" {
+    command = ["/usr/local/bin/awesome-app", "management"]
+  }
+}
+```
+
+## Parse HCL: Sample Doc
+Given a sample.hcl file of:
+```hcl
+service "cat" {
+  process "main" {
+    command = ["/usr/local/bin/awesome-app", "server"]
+  }
+
+  process "management" {
+    command = ["/usr/local/bin/awesome-app", "management"]
+  }
+}
+
+```
+then
+```bash
+yq -oy sample.hcl
+```
+will output
+```yaml
+service:
+  cat:
+    process:
+      main:
+        command:
+          - "/usr/local/bin/awesome-app"
+          - "server"
+      management:
+        command:
+          - "/usr/local/bin/awesome-app"
+          - "management"
+```
+
+## Parse HCL: with comments
+Given a sample.hcl file of:
+```hcl
+# Configuration
+port = 8080 # server port
+```
+then
+```bash
+yq -oy sample.hcl
+```
+will output
+```yaml
+# Configuration
+port: 8080 # server port
+```
+
+## Roundtrip: with comments
+Given a sample.hcl file of:
+```hcl
+# Configuration
+port = 8080
+```
+then
+```bash
+yq sample.hcl
+```
+will output
+```hcl
+# Configuration
+port = 8080
+```
+
+## Roundtrip: With templates, functions and arithmetic
+Given a sample.hcl file of:
+```hcl
+# Arithmetic with literals and application-provided variables
+sum = 1 + addend
+
+# String interpolation and templates
+message = "Hello, ${name}!"
+
+# Application-provided functions
+shouty_message = upper(message)
+```
+then
+```bash
+yq sample.hcl
+```
+will output
+```hcl
+# Arithmetic with literals and application-provided variables
+sum = 1 + addend
+# String interpolation and templates
+message = "Hello, ${name}!"
+# Application-provided functions
+shouty_message = upper(message)
+```
+
+## Roundtrip: Separate blocks with same name.
+Given a sample.hcl file of:
+```hcl
+resource "aws_instance" "web" {
+  ami = "ami-12345"
+}
+resource "aws_instance" "db" {
+  ami = "ami-67890"
+}
+```
+then
+```bash
+yq sample.hcl
+```
+will output
+```hcl
+resource "aws_instance" "web" {
+  ami = "ami-12345"
+}
+resource "aws_instance" "db" {
+  ami = "ami-67890"
+}
+```
+

usage/shellvariables.md

@@ -84,3 +84,23 @@ will output
 name='Miles O'"'"'Brien'
 ```
 
+## Encode shell variables: custom separator
+Use --shell-key-separator to specify a custom separator between keys. This is useful when the original keys contain underscores.
+
+Given a sample.yml file of:
+```yaml
+my_app:
+  db_config:
+    host: localhost
+    port: 5432
+```
+then
+```bash
+yq -o=shell --shell-key-separator="__" sample.yml
+```
+will output
+```sh
+my_app__db_config__host=localhost
+my_app__db_config__port=5432
+```
+

usage/toml.md

@@ -104,6 +104,27 @@ owner:
       suburb: nice
 ```
 
+## Parse: Array of Array Table
+Given a sample.toml file of:
+```toml
+
+[[fruits]]
+name = "apple"
+[[fruits.varieties]]  # nested array of tables
+name = "red delicious"
+```
+then
+```bash
+yq -oy '.' sample.toml
+```
+will output
+```yaml
+fruits:
+  - name: apple
+    varieties:
+      - name: red delicious
+```
+
 ## Parse: Empty Table
 Given a sample.toml file of:
 ```toml

usage/xml.md

@@ -53,7 +53,7 @@ Given a sample.xml file of:
 ```
 then
 ```bash
-yq -oy '.' sample.xml
+yq -oy sample.xml
 ```
 will output
 ```yaml
@@ -100,7 +100,7 @@ Given a sample.xml file of:
 ```
 then
 ```bash
-yq -oy '.' sample.xml
+yq -oy sample.xml
 ```
 will output
 ```yaml
@@ -157,7 +157,7 @@ Given a sample.xml file of:
 ```
 then
 ```bash
-yq -oy '.' sample.xml
+yq -oy sample.xml
 ```
 will output
 ```yaml
@@ -177,7 +177,7 @@ Given a sample.xml file of:
 ```
 then
 ```bash
-yq -oy '.' sample.xml
+yq -oy sample.xml
 ```
 will output
 ```yaml
@@ -196,7 +196,7 @@ Given a sample.xml file of:
 ```
 then
 ```bash
-yq -oy '.' sample.xml
+yq -oy sample.xml
 ```
 will output
 ```yaml
@@ -225,7 +225,7 @@ Given a sample.xml file of:
 ```
 then
 ```bash
-yq '.' sample.xml
+yq sample.xml
 ```
 will output
 ```xml
@@ -256,7 +256,7 @@ Given a sample.xml file of:
 ```
 then
 ```bash
-yq --xml-skip-directives '.' sample.xml
+yq --xml-skip-directives sample.xml
 ```
 will output
 ```xml
@@ -292,7 +292,7 @@ for x --></x>
 ```
 then
 ```bash
-yq -oy '.' sample.xml
+yq -oy sample.xml
 ```
 will output
 ```yaml
@@ -327,7 +327,7 @@ Given a sample.xml file of:
 ```
 then
 ```bash
-yq --xml-keep-namespace=false '.' sample.xml
+yq --xml-keep-namespace=false sample.xml
 ```
 will output
 ```xml
@@ -361,7 +361,7 @@ Given a sample.xml file of:
 ```
 then
 ```bash
-yq --xml-raw-token=false '.' sample.xml
+yq --xml-raw-token=false sample.xml
 ```
 will output
 ```xml
@@ -542,7 +542,7 @@ for x --></x>
 ```
 then
 ```bash
-yq '.' sample.xml
+yq sample.xml
 ```
 will output
 ```xml
@@ -575,7 +575,7 @@ Given a sample.xml file of:
 ```
 then
 ```bash
-yq '.' sample.xml
+yq sample.xml
 ```
 will output
 ```xml