suyiiyii/yq
1
0
Fork 0
mirror of https://github.com/mikefarah/yq.git synced 2025-01-13 20:15:57 +00:00
Code Issues Packages Projects Releases Wiki Activity

Updating doc for release

Browse Source
This commit is contained in:
Mike Farah 2022-08-01 16:46:09 +10:00
parent 97c5c563e6
commit 9458fa1cf5
4 changed files with 357 additions and 56 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

52
operators/encode-decode.md
View File

@ -9,16 +9,16 @@ These operators are useful to process yaml documents that have stringified embed
| Format | Decode (from string) | Encode (to string) |
| --- | -- | --|
| Yaml | from_yaml | to_yaml(i)/@yaml |
| JSON | from_json | to_json(i)/@json |
| Properties | from_props | to_props/@props |
| CSV | | to_csv/@csv |
| TSV | | to_tsv/@tsv |
| XML | from_xml | to_xml(i)/@xml |
| Yaml | from_yaml/@yamld | to_yaml(i)/@yaml |
| JSON | from_json/@jsond | to_json(i)/@json |
| Properties | from_props/@propsd | to_props/@props |
| CSV | from_csv/@csvd | to_csv/@csv |
| TSV | from_tsv/@tsvd | to_tsv/@tsv |
| XML | from_xml/@xmld | to_xml(i)/@xml |
| Base64 | @base64d | @base64 |
CSV and TSV format both accept either a single array or scalars (representing a single row), or an array of array of scalars (representing multiple rows).
See CSV and TSV [documentation](https://mikefarah.gitbook.io/yq/usage/csv-tsv) for accepted formats.
XML uses the `--xml-attribute-prefix` and `xml-content-name` flags to identify attributes and content fields.
@ -132,7 +132,7 @@ a: |-
```
then
```bash
yq '.a |= from_props' sample.yml
yq '.a |= @propsd' sample.yml
```
will output
```yaml
@ -141,6 +141,42 @@ a:
dogs: cool as well
```
## Decode csv encoded string
Given a sample.yml file of:
```yaml
a: |-
cats,dogs
great,cool as well
```
then
```bash
yq '.a |= @csvd' sample.yml
```
will output
```yaml
a:
- cats: great
dogs: cool as well
```
## Decode tsv encoded string
Given a sample.yml file of:
```yaml
a: |-
cats dogs
great cool as well
```
then
```bash
yq '.a |= @tsvd' sample.yml
```
will output
```yaml
a:
- cats: great
dogs: cool as well
```
## Encode value as yaml string
Indent defaults to 2

131
usage/convert.md
View File

@ -1,8 +1,9 @@
# JSON
Encode and decode to and from JSON. Note that YAML is a _superset_ of JSON - so `yq` can read any json file without doing anything special.
Encode and decode to and from JSON. Supports multiple JSON documents in a single file (e.g. NDJSON).
Note that YAML is a superset of (single document) JSON - so you don't have to use the JSON parser to read JSON when there is only one JSON document in the input. You will probably want to pretty print the result in this case, to get idiomatic YAML styling.
This means you don't need to 'convert' a JSON file to YAML - however if you want idiomatic YAML styling, then you can use the `-P/--prettyPrint` flag, see examples below.
{% hint style="warning" %}
Note that versions prior to 4.18 require the 'eval/e' command to be specified. 
@ -130,3 +131,129 @@ will output
{"whatever":"cat"}
```
## Roundtrip NDJSON
Unfortunately the json encoder strips leading spaces of values.
Given a sample.json file of:
```json
{"this": "is a multidoc json file"}
{"each": ["line is a valid json document"]}
{"a number": 4}
```
then
```bash
yq -p=json -o=json -I=0 sample.json
```
will output
```yaml
{"this":"is a multidoc json file"}
{"each":["line is a valid json document"]}
{"a number":4}
```
## Roundtrip multi-document JSON
The NDJSON parser can also handle multiple multi-line json documents in a single file!
Given a sample.json file of:
```json
{
"this": "is a multidoc json file"
}
{
"it": [
"has",
"consecutive",
"json documents"
]
}
{
"a number": 4
}
```
then
```bash
yq -p=json -o=json -I=2 sample.json
```
will output
```yaml
{
"this": "is a multidoc json file"
}
{
"it": [
"has",
"consecutive",
"json documents"
]
}
{
"a number": 4
}
```
## Update a specific document in a multi-document json
Documents are indexed by the `documentIndex` or `di` operator.
Given a sample.json file of:
```json
{"this": "is a multidoc json file"}
{"each": ["line is a valid json document"]}
{"a number": 4}
```
then
```bash
yq -p=json -o=json -I=0 '(select(di == 1) | .each ) += "cool"' sample.json
```
will output
```yaml
{"this":"is a multidoc json file"}
{"each":["line is a valid json document","cool"]}
{"a number":4}
```
## Find and update a specific document in a multi-document json
Use expressions as you normally would.
Given a sample.json file of:
```json
{"this": "is a multidoc json file"}
{"each": ["line is a valid json document"]}
{"a number": 4}
```
then
```bash
yq -p=json -o=json -I=0 '(select(has("each")) | .each ) += "cool"' sample.json
```
will output
```yaml
{"this":"is a multidoc json file"}
{"each":["line is a valid json document","cool"]}
{"a number":4}
```
## Decode NDJSON
Given a sample.json file of:
```json
{"this": "is a multidoc json file"}
{"each": ["line is a valid json document"]}
{"a number": 4}
```
then
```bash
yq -p=json sample.json
```
will output
```yaml
this: is a multidoc json file
---
each:
- line is a valid json document
---
a number: 4
```

228
usage/csv-tsv.md
View File

@ -1,4 +1,33 @@
# Working with CSV, TSV
# CSV
Encode/Decode/Roundtrip CSV and TSV files.
## Encode
Currently supports arrays of homogenous flat objects, that is: no nesting and it assumes the _first_ object has all the keys required:
```yaml
- name: Bobo
type: dog
- name: Fifi
type: cat
```
As well as arrays of arrays of scalars (strings/numbers/booleans):
```yaml
- [Bobo, dog]
- [Fifi, cat]
```
## Decode
Decode assumes the first CSV/TSV row is the header row, and all rows beneath are the entries.
The data will be coded into an array of objects, using the header rows as keys.
```csv
name,type
Bobo,dog
Fifi,cat
```
{% hint style="warning" %}
Note that versions prior to 4.18 require the 'eval/e' command to be specified. 
@ -6,71 +35,180 @@ Note that versions prior to 4.18 require the 'eval/e' command to be specified.&#
`yq e <exp> <file>`
{% endhint %}
## Yaml to CSV/TSV
You can convert compatible yaml structures to CSV or TSV by using:
* `--output-format=csv` or `-o=c` for csv (comma separated values)
* `--output-format=tsv` or `-o=t` for tsv (tab separated values)
Compatible structures is either an array of scalars (strings/numbers/booleans), which is a single row; or an array of arrays of scalars (multiple rows).
## Encode CSV simple
Given a sample.yml file of:
```yaml
- [i, like, csv]
- [because, excel, is, cool]
```
then
```bash
yq '.' -o=csv sample.yaml
yq -o=csv sample.yml
```
will output:
will output
```csv
i,like,csv
because,excel,is,cool
```
Similarly, for tsv:
## Encode TSV simple
Given a sample.yml file of:
```yaml
- [i, like, csv]
- [because, excel, is, cool]
```
then
```bash
yq '.' -o=tsv sample.yaml
```
will output:
yq -o=tsv sample.yml
```
will output
```tsv
i like csv
because excel is cool
```
### Converting an array of objects to CSV
If you have a yaml document like:
## Encode array of objects to csv
Given a sample.yml file of:
```yaml
foo: bar
items:
- name: Tom
species: cat
color: blue
- name: Jim
species: dog
color: brown
- name: Gary
numberOfCats: 1
likesApples: true
height: 168.8
- name: Samantha's Rabbit
numberOfCats: 2
likesApples: false
height: -188.8
```
To convert to CSV, you need to transform this into a array of CSV rows. Assuming you also want a header, then you can do:
then
```bash
yq '[["name", "species", "color"]] + [.items[] | [.name, .species, .color]]' data.yaml -o=csv
yq -o=csv sample.yml
```
to yield:
will output
```csv
name,species,color
Tom,cat,blue
Jim,dog,brown
name,numberOfCats,likesApples,height
Gary,1,true,168.8
Samantha's Rabbit,2,false,-188.8
```
## Encode array of objects to custom csv format
Add the header row manually, then the we convert each object into an array of values - resulting in an array of arrays. Pick the columns and call the header whatever you like.
Given a sample.yml file of:
```yaml
- name: Gary
numberOfCats: 1
likesApples: true
height: 168.8
- name: Samantha's Rabbit
numberOfCats: 2
likesApples: false
height: -188.8
```
then
```bash
yq -o=csv '[["Name", "Number of Cats"]] + [.[] | [.name, .numberOfCats ]]' sample.yml
```
will output
```csv
Name,Number of Cats
Gary,1
Samantha's Rabbit,2
```
## Encode array of objects to csv - missing fields behaviour
First entry is used to determine the headers, and it is missing 'likesApples', so it is not included in the csv. Second entry does not have 'numberOfCats' so that is blank
Given a sample.yml file of:
```yaml
- name: Gary
numberOfCats: 1
height: 168.8
- name: Samantha's Rabbit
height: -188.8
likesApples: false
```
then
```bash
yq -o=csv sample.yml
```
will output
```csv
name,numberOfCats,height
Gary,1,168.8
Samantha's Rabbit,,-188.8
```
## Parse CSV into an array of objects
First row is assumed to be the header row.
Given a sample.csv file of:
```csv
name,numberOfCats,likesApples,height
Gary,1,true,168.8
Samantha's Rabbit,2,false,-188.8
```
then
```bash
yq -p=csv sample.csv
```
will output
```yaml
- name: Gary
numberOfCats: 1
likesApples: true
height: 168.8
- name: Samantha's Rabbit
numberOfCats: 2
likesApples: false
height: -188.8
```
## Parse TSV into an array of objects
First row is assumed to be the header row.
Given a sample.tsv file of:
```tsv
name numberOfCats likesApples height
Gary 1 true 168.8
Samantha's Rabbit 2 false -188.8
```
then
```bash
yq -p=tsv sample.tsv
```
will output
```yaml
- name: Gary
numberOfCats: 1
likesApples: true
height: 168.8
- name: Samantha's Rabbit
numberOfCats: 2
likesApples: false
height: -188.8
```
## Round trip
Given a sample.csv file of:
```csv
name,numberOfCats,likesApples,height
Gary,1,true,168.8
Samantha's Rabbit,2,false,-188.8
```
then
```bash
yq -p=csv -o=csv '(.[] | select(.name == "Gary") | .numberOfCats) = 3' sample.csv
```
will output
```csv
name,numberOfCats,likesApples,height
Gary,3,true,168.8
Samantha's Rabbit,2,false,-188.8
```

2
usage/properties.md
View File

@ -1,6 +1,6 @@
# Properties
Encode to a property file (decode not yet supported). Line comments on value nodes will be copied across.
Encode/Decode/Roundtrip to/from a property file. Line comments on value nodes will be copied across.
By default, empty maps and arrays are not encoded - see below for an example on how to encode a value for these.