Added date subtract support

This commit is contained in:
Mike Farah 2022-02-14 15:29:17 +11:00
parent 246d1af8d6
commit 778cbd348a
11 changed files with 181 additions and 40 deletions

View File

@ -124,7 +124,6 @@ rm /etc/myfile.tmp
``` ```
### Run with Docker or Podman ### Run with Docker or Podman
#### Oneshot use: #### Oneshot use:
```bash ```bash
@ -194,7 +193,7 @@ Or, in your Dockerfile:
FROM mikefarah/yq FROM mikefarah/yq
USER root USER root
RUN apk add bash RUN apk add --no-cache bash
USER yq USER yq
``` ```

View File

@ -231,7 +231,7 @@ a: Saturday, 15-Dec-01 at 2:59AM GMT
``` ```
then then
```bash ```bash
yq 'with_dtformat("Monday, 02-Jan-06 at 3:04PM MST", .a += "3h1m")' sample.yml yq 'with_dtf("Monday, 02-Jan-06 at 3:04PM MST", .a += "3h1m")' sample.yml
``` ```
will output will output
```yaml ```yaml

View File

@ -5,16 +5,15 @@ Various operators for parsing and manipulating dates.
## Date time formattings ## Date time formattings
This uses the golangs built in time library for parsing and formatting date times. This uses the golangs built in time library for parsing and formatting date times.
When not specified, the RFC3339 standard is assumed `2006-01-02T15:04:05Z07:00`. When not specified, the RFC3339 standard is assumed `2006-01-02T15:04:05Z07:00` for parsing.
To use a custom format, use the `with_dtformat` operator to set the formatting context. Expressions in the second parameter then assume that date format. To specify a custom parsing format, use the `with_dtf` operator. The first parameter sets the datetime parsing format for the expression in the second parameter. The expression can be any valid `yq` expression tree.
```bash ```bash
yq 'with_dtformat("myformat"; .a + "3h" | tz("Australia/Melbourne"))' yq 'with_dtf("myformat"; .a + "3h" | tz("Australia/Melbourne"))'
``` ```
See https://pkg.go.dev/time#pkg-constants for more examples. See https://pkg.go.dev/time#pkg-constants for examples of formatting options.
## Timezones ## Timezones
@ -49,7 +48,7 @@ a: Saturday, 15-Dec-01 at 2:59AM
``` ```
## Format: from custom date time ## Format: from custom date time
Use with_dtformat to set a custom datetime format for parsing. Use with_dtf to set a custom datetime format for parsing.
Given a sample.yml file of: Given a sample.yml file of:
```yaml ```yaml
@ -57,7 +56,7 @@ a: Saturday, 15-Dec-01 at 2:59AM
``` ```
then then
```bash ```bash
yq '.a |= with_dtformat("Monday, 02-Jan-06 at 3:04PM"; format_datetime("2006-01-02"))' sample.yml yq '.a |= with_dtf("Monday, 02-Jan-06 at 3:04PM"; format_datetime("2006-01-02"))' sample.yml
``` ```
will output will output
```yaml ```yaml
@ -119,7 +118,7 @@ a: Saturday, 15-Dec-01 at 2:59AM GMT
``` ```
then then
```bash ```bash
yq '.a |= with_dtformat("Monday, 02-Jan-06 at 3:04PM MST"; tz("Australia/Sydney"))' sample.yml yq '.a |= with_dtf("Monday, 02-Jan-06 at 3:04PM MST"; tz("Australia/Sydney"))' sample.yml
``` ```
will output will output
```yaml ```yaml
@ -135,7 +134,7 @@ a: Saturday, 15-Dec-01 at 2:59AM GMT
``` ```
then then
```bash ```bash
yq '.a |= with_dtformat("Monday, 02-Jan-06 at 3:04PM MST"; tz("Australia/Sydney"))' sample.yml yq '.a |= with_dtf("Monday, 02-Jan-06 at 3:04PM MST"; tz("Australia/Sydney"))' sample.yml
``` ```
will output will output
```yaml ```yaml
@ -156,6 +155,22 @@ will output
a: 2021-01-01T03:10:00Z a: 2021-01-01T03:10:00Z
``` ```
## Date subtraction
You can subtract durations from dates. Assumes RFC3339 date time format, see [date-time operators](https://mikefarah.gitbook.io/yq/operators/date-time-operators) for more information.
Given a sample.yml file of:
```yaml
a: 2021-01-01T03:10:00Z
```
then
```bash
yq '.a -= "3h10m"' sample.yml
```
will output
```yaml
a: 2021-01-01T00:00:00Z
```
## Date addition - custom format ## Date addition - custom format
Given a sample.yml file of: Given a sample.yml file of:
```yaml ```yaml
@ -163,10 +178,26 @@ a: Saturday, 15-Dec-01 at 2:59AM GMT
``` ```
then then
```bash ```bash
yq 'with_dtformat("Monday, 02-Jan-06 at 3:04PM MST"; .a += "3h1m")' sample.yml yq 'with_dtf("Monday, 02-Jan-06 at 3:04PM MST"; .a += "3h1m")' sample.yml
``` ```
will output will output
```yaml ```yaml
a: Saturday, 15-Dec-01 at 6:00AM GMT a: Saturday, 15-Dec-01 at 6:00AM GMT
``` ```
## Date script with custom format
You can embed full expressions in with_dtf if needed.
Given a sample.yml file of:
```yaml
a: Saturday, 15-Dec-01 at 2:59AM GMT
```
then
```bash
yq 'with_dtf("Monday, 02-Jan-06 at 3:04PM MST"; .a = (.a + "3h1m" | tz("Australia/Perth")))' sample.yml
```
will output
```yaml
a: Saturday, 15-Dec-01 at 2:00PM AWST
```

View File

@ -5,16 +5,15 @@ Various operators for parsing and manipulating dates.
## Date time formattings ## Date time formattings
This uses the golangs built in time library for parsing and formatting date times. This uses the golangs built in time library for parsing and formatting date times.
When not specified, the RFC3339 standard is assumed `2006-01-02T15:04:05Z07:00`. When not specified, the RFC3339 standard is assumed `2006-01-02T15:04:05Z07:00` for parsing.
To use a custom format, use the `with_dtformat` operator to set the formatting context. Expressions in the second parameter then assume that date format. To specify a custom parsing format, use the `with_dtf` operator. The first parameter sets the datetime parsing format for the expression in the second parameter. The expression can be any valid `yq` expression tree.
```bash ```bash
yq 'with_dtformat("myformat"; .a + "3h" | tz("Australia/Melbourne"))' yq 'with_dtf("myformat"; .a + "3h" | tz("Australia/Melbourne"))'
``` ```
See https://pkg.go.dev/time#pkg-constants for more examples. See https://pkg.go.dev/time#pkg-constants for examples of formatting options.
## Timezones ## Timezones

View File

@ -117,6 +117,38 @@ a: 2
b: 4 b: 4
``` ```
## Date subtraction
You can subtract durations from dates. Assumes RFC3339 date time format, see [date-time operators](https://mikefarah.gitbook.io/yq/operators/date-time-operators) for more information.
Given a sample.yml file of:
```yaml
a: 2021-01-01T03:10:00Z
```
then
```bash
yq '.a -= "3h10m"' sample.yml
```
will output
```yaml
a: 2021-01-01T00:00:00Z
```
## Date subtraction - custom format
Use with_dtf to specify your datetime format. See [date-time operators](https://mikefarah.gitbook.io/yq/operators/date-time-operators) for more information.
Given a sample.yml file of:
```yaml
a: Saturday, 15-Dec-01 at 6:00AM GMT
```
then
```bash
yq 'with_dtf("Monday, 02-Jan-06 at 3:04PM MST", .a -= "3h1m")' sample.yml
```
will output
```yaml
a: Saturday, 15-Dec-01 at 2:59AM GMT
```
## Custom types: that are really numbers ## Custom types: that are really numbers
When custom tags are encountered, yq will try to decode the underlying type. When custom tags are encountered, yq will try to decode the underlying type.

View File

@ -327,7 +327,7 @@ func initLexer() (*lex.Lexer, error) {
lexer.Add([]byte(`format_datetime`), opToken(formatDateTimeOpType)) lexer.Add([]byte(`format_datetime`), opToken(formatDateTimeOpType))
lexer.Add([]byte(`now`), opToken(nowOpType)) lexer.Add([]byte(`now`), opToken(nowOpType))
lexer.Add([]byte(`tz`), opToken(tzOpType)) lexer.Add([]byte(`tz`), opToken(tzOpType))
lexer.Add([]byte(`with_dtformat`), opToken(withDtFormatOpType)) lexer.Add([]byte(`with_dtf`), opToken(withDtFormatOpType))
lexer.Add([]byte(`toyaml\([0-9]+\)`), encodeWithIndent(YamlOutputFormat)) lexer.Add([]byte(`toyaml\([0-9]+\)`), encodeWithIndent(YamlOutputFormat))
lexer.Add([]byte(`to_yaml\([0-9]+\)`), encodeWithIndent(YamlOutputFormat)) lexer.Add([]byte(`to_yaml\([0-9]+\)`), encodeWithIndent(YamlOutputFormat))

View File

@ -221,7 +221,7 @@ var addOperatorScenarios = []expressionScenario{
description: "Date addition - custom format", description: "Date addition - custom format",
subdescription: "You can add durations to dates. See [date-time operators](https://mikefarah.gitbook.io/yq/operators/date-time-operators) for more information.", subdescription: "You can add durations to dates. See [date-time operators](https://mikefarah.gitbook.io/yq/operators/date-time-operators) for more information.",
document: `a: Saturday, 15-Dec-01 at 2:59AM GMT`, document: `a: Saturday, 15-Dec-01 at 2:59AM GMT`,
expression: `with_dtformat("Monday, 02-Jan-06 at 3:04PM MST", .a += "3h1m")`, expression: `with_dtf("Monday, 02-Jan-06 at 3:04PM MST", .a += "3h1m")`,
expected: []string{ expected: []string{
"D0, P[], (doc)::a: Saturday, 15-Dec-01 at 6:00AM GMT\n", "D0, P[], (doc)::a: Saturday, 15-Dec-01 at 6:00AM GMT\n",
}, },
@ -231,7 +231,7 @@ var addOperatorScenarios = []expressionScenario{
description: "Date addition - custom format", description: "Date addition - custom format",
subdescription: "You can add durations to dates. See [date-time operators](https://mikefarah.gitbook.io/yq/operators/date-time-operators) for more information.", subdescription: "You can add durations to dates. See [date-time operators](https://mikefarah.gitbook.io/yq/operators/date-time-operators) for more information.",
document: `a: !cat Saturday, 15-Dec-01 at 2:59AM GMT`, document: `a: !cat Saturday, 15-Dec-01 at 2:59AM GMT`,
expression: `with_dtformat("Monday, 02-Jan-06 at 3:04PM MST", .a += "3h1m")`, expression: `with_dtf("Monday, 02-Jan-06 at 3:04PM MST", .a += "3h1m")`,
expected: []string{ expected: []string{
"D0, P[], (doc)::a: !cat Saturday, 15-Dec-01 at 6:00AM GMT\n", "D0, P[], (doc)::a: !cat Saturday, 15-Dec-01 at 6:00AM GMT\n",
}, },

View File

@ -32,7 +32,7 @@ func withDateTimeFormat(d *dataTreeNavigator, context Context, expressionNode *E
return d.GetMatchingNodes(context, expressionNode.RHS.RHS) return d.GetMatchingNodes(context, expressionNode.RHS.RHS)
} }
return Context{}, errors.New(`must provide a date time format string and an expression, e.g. with_dtformat("Monday, 02-Jan-06 at 3:04PM MST"; <exp>)`) return Context{}, errors.New(`must provide a date time format string and an expression, e.g. with_dtf("Monday, 02-Jan-06 at 3:04PM MST"; <exp>)`)
} }

View File

@ -16,9 +16,9 @@ var dateTimeOperatorScenarios = []expressionScenario{
}, },
{ {
description: "Format: from custom date time", description: "Format: from custom date time",
subdescription: "Use with_dtformat to set a custom datetime format for parsing.", subdescription: "Use with_dtf to set a custom datetime format for parsing.",
document: `a: Saturday, 15-Dec-01 at 2:59AM`, document: `a: Saturday, 15-Dec-01 at 2:59AM`,
expression: `.a |= with_dtformat("Monday, 02-Jan-06 at 3:04PM"; format_datetime("2006-01-02"))`, expression: `.a |= with_dtf("Monday, 02-Jan-06 at 3:04PM"; format_datetime("2006-01-02"))`,
expected: []string{ expected: []string{
"D0, P[], (doc)::a: 2001-12-15\n", "D0, P[], (doc)::a: 2001-12-15\n",
}, },
@ -53,7 +53,7 @@ var dateTimeOperatorScenarios = []expressionScenario{
description: "Timezone: with custom format", description: "Timezone: with custom format",
subdescription: "Specify standard IANA Time Zone format or 'utc', 'local'", subdescription: "Specify standard IANA Time Zone format or 'utc', 'local'",
document: "a: Saturday, 15-Dec-01 at 2:59AM GMT", document: "a: Saturday, 15-Dec-01 at 2:59AM GMT",
expression: `.a |= with_dtformat("Monday, 02-Jan-06 at 3:04PM MST"; tz("Australia/Sydney"))`, expression: `.a |= with_dtf("Monday, 02-Jan-06 at 3:04PM MST"; tz("Australia/Sydney"))`,
expected: []string{ expected: []string{
"D0, P[], (doc)::a: Saturday, 15-Dec-01 at 1:59PM AEDT\n", "D0, P[], (doc)::a: Saturday, 15-Dec-01 at 1:59PM AEDT\n",
}, },
@ -62,7 +62,7 @@ var dateTimeOperatorScenarios = []expressionScenario{
description: "Add and tz custom format", description: "Add and tz custom format",
subdescription: "Specify standard IANA Time Zone format or 'utc', 'local'", subdescription: "Specify standard IANA Time Zone format or 'utc', 'local'",
document: "a: Saturday, 15-Dec-01 at 2:59AM GMT", document: "a: Saturday, 15-Dec-01 at 2:59AM GMT",
expression: `.a |= with_dtformat("Monday, 02-Jan-06 at 3:04PM MST"; tz("Australia/Sydney"))`, expression: `.a |= with_dtf("Monday, 02-Jan-06 at 3:04PM MST"; tz("Australia/Sydney"))`,
expected: []string{ expected: []string{
"D0, P[], (doc)::a: Saturday, 15-Dec-01 at 1:59PM AEDT\n", "D0, P[], (doc)::a: Saturday, 15-Dec-01 at 1:59PM AEDT\n",
}, },
@ -75,20 +75,37 @@ var dateTimeOperatorScenarios = []expressionScenario{
"D0, P[], (doc)::a: 2021-01-01T03:10:00Z\n", "D0, P[], (doc)::a: 2021-01-01T03:10:00Z\n",
}, },
}, },
{
description: "Date subtraction",
subdescription: "You can subtract durations from dates. Assumes RFC3339 date time format, see [date-time operators](https://mikefarah.gitbook.io/yq/operators/date-time-operators) for more information.",
document: `a: 2021-01-01T03:10:00Z`,
expression: `.a -= "3h10m"`,
expected: []string{
"D0, P[], (doc)::a: 2021-01-01T00:00:00Z\n",
},
},
{ {
description: "Date addition - custom format", description: "Date addition - custom format",
document: `a: Saturday, 15-Dec-01 at 2:59AM GMT`, document: `a: Saturday, 15-Dec-01 at 2:59AM GMT`,
expression: `with_dtformat("Monday, 02-Jan-06 at 3:04PM MST"; .a += "3h1m")`, expression: `with_dtf("Monday, 02-Jan-06 at 3:04PM MST"; .a += "3h1m")`,
expected: []string{ expected: []string{
"D0, P[], (doc)::a: Saturday, 15-Dec-01 at 6:00AM GMT\n", "D0, P[], (doc)::a: Saturday, 15-Dec-01 at 6:00AM GMT\n",
}, },
}, },
{
description: "Date script with custom format",
subdescription: "You can embed full expressions in with_dtf if needed.",
document: `a: Saturday, 15-Dec-01 at 2:59AM GMT`,
expression: `with_dtf("Monday, 02-Jan-06 at 3:04PM MST"; .a = (.a + "3h1m" | tz("Australia/Perth")))`,
expected: []string{
"D0, P[], (doc)::a: Saturday, 15-Dec-01 at 2:00PM AWST\n",
},
},
{ {
description: "allow comma", description: "allow comma",
skipDoc: true, skipDoc: true,
document: "a: Saturday, 15-Dec-01 at 2:59AM GMT", document: "a: Saturday, 15-Dec-01 at 2:59AM GMT",
expression: `.a |= with_dtformat("Monday, 02-Jan-06 at 3:04PM MST", tz("Australia/Sydney"))`, expression: `.a |= with_dtf("Monday, 02-Jan-06 at 3:04PM MST", tz("Australia/Sydney"))`,
expected: []string{ expected: []string{
"D0, P[], (doc)::a: Saturday, 15-Dec-01 at 1:59PM AEDT\n", "D0, P[], (doc)::a: Saturday, 15-Dec-01 at 1:59PM AEDT\n",
}, },

View File

@ -4,6 +4,7 @@ import (
"fmt" "fmt"
"strconv" "strconv"
"strings" "strings"
"time"
"gopkg.in/yaml.v3" "gopkg.in/yaml.v3"
) )
@ -56,7 +57,7 @@ func subtract(d *dataTreeNavigator, context Context, lhs *CandidateNode, rhs *Ca
switch lhsNode.Kind { switch lhsNode.Kind {
case yaml.MappingNode: case yaml.MappingNode:
return nil, fmt.Errorf("Maps not yet supported for subtraction") return nil, fmt.Errorf("maps not yet supported for subtraction")
case yaml.SequenceNode: case yaml.SequenceNode:
if rhs.Node.Kind != yaml.SequenceNode { if rhs.Node.Kind != yaml.SequenceNode {
return nil, fmt.Errorf("%v (%v) cannot be subtracted from %v", rhs.Node.Tag, rhs.Path, lhsNode.Tag) return nil, fmt.Errorf("%v (%v) cannot be subtracted from %v", rhs.Node.Tag, rhs.Path, lhsNode.Tag)
@ -68,13 +69,15 @@ func subtract(d *dataTreeNavigator, context Context, lhs *CandidateNode, rhs *Ca
} }
target.Node.Kind = yaml.ScalarNode target.Node.Kind = yaml.ScalarNode
target.Node.Style = lhsNode.Style target.Node.Style = lhsNode.Style
return subtractScalars(target, lhsNode, rhs.Node) if err := subtractScalars(context, target, lhsNode, rhs.Node); err != nil {
return nil, err
}
} }
return target, nil return target, nil
} }
func subtractScalars(target *CandidateNode, lhs *yaml.Node, rhs *yaml.Node) (*CandidateNode, error) { func subtractScalars(context Context, target *CandidateNode, lhs *yaml.Node, rhs *yaml.Node) error {
lhsTag := lhs.Tag lhsTag := lhs.Tag
rhsTag := rhs.Tag rhsTag := rhs.Tag
lhsIsCustom := false lhsIsCustom := false
@ -89,16 +92,25 @@ func subtractScalars(target *CandidateNode, lhs *yaml.Node, rhs *yaml.Node) (*Ca
rhsTag = guessTagFromCustomType(rhs) rhsTag = guessTagFromCustomType(rhs)
} }
if lhsTag == "!!str" { isDateTime := lhs.Tag == "!!timestamp"
return nil, fmt.Errorf("strings cannot be subtracted") // if the lhs is a string, it might be a timestamp in a custom format.
if lhsTag == "!!str" && context.GetDateTimeLayout() != time.RFC3339 {
_, err := time.Parse(context.GetDateTimeLayout(), lhs.Value)
isDateTime = err == nil
}
if isDateTime {
return subtractDateTime(context.GetDateTimeLayout(), target, lhs, rhs)
} else if lhsTag == "!!str" {
return fmt.Errorf("strings cannot be subtracted")
} else if lhsTag == "!!int" && rhsTag == "!!int" { } else if lhsTag == "!!int" && rhsTag == "!!int" {
format, lhsNum, err := parseInt(lhs.Value) format, lhsNum, err := parseInt(lhs.Value)
if err != nil { if err != nil {
return nil, err return err
} }
_, rhsNum, err := parseInt(rhs.Value) _, rhsNum, err := parseInt(rhs.Value)
if err != nil { if err != nil {
return nil, err return err
} }
result := lhsNum - rhsNum result := lhsNum - rhsNum
target.Node.Tag = lhs.Tag target.Node.Tag = lhs.Tag
@ -106,11 +118,11 @@ func subtractScalars(target *CandidateNode, lhs *yaml.Node, rhs *yaml.Node) (*Ca
} else if (lhsTag == "!!int" || lhsTag == "!!float") && (rhsTag == "!!int" || rhsTag == "!!float") { } else if (lhsTag == "!!int" || lhsTag == "!!float") && (rhsTag == "!!int" || rhsTag == "!!float") {
lhsNum, err := strconv.ParseFloat(lhs.Value, 64) lhsNum, err := strconv.ParseFloat(lhs.Value, 64)
if err != nil { if err != nil {
return nil, err return err
} }
rhsNum, err := strconv.ParseFloat(rhs.Value, 64) rhsNum, err := strconv.ParseFloat(rhs.Value, 64)
if err != nil { if err != nil {
return nil, err return err
} }
result := lhsNum - rhsNum result := lhsNum - rhsNum
if lhsIsCustom { if lhsIsCustom {
@ -120,8 +132,31 @@ func subtractScalars(target *CandidateNode, lhs *yaml.Node, rhs *yaml.Node) (*Ca
} }
target.Node.Value = fmt.Sprintf("%v", result) target.Node.Value = fmt.Sprintf("%v", result)
} else { } else {
return nil, fmt.Errorf("%v cannot be added to %v", lhs.Tag, rhs.Tag) return fmt.Errorf("%v cannot be added to %v", lhs.Tag, rhs.Tag)
} }
return target, nil return nil
}
func subtractDateTime(layout string, target *CandidateNode, lhs *yaml.Node, rhs *yaml.Node) error {
var durationStr string
if strings.HasPrefix(rhs.Value, "-") {
durationStr = rhs.Value[1:]
} else {
durationStr = "-" + rhs.Value
}
duration, err := time.ParseDuration(durationStr)
if err != nil {
return fmt.Errorf("unable to parse duration [%v]: %w", rhs.Value, err)
}
currentTime, err := time.Parse(layout, lhs.Value)
if err != nil {
return err
}
newTime := currentTime.Add(duration)
target.Node.Value = newTime.Format(layout)
return nil
} }

View File

@ -93,6 +93,34 @@ var subtractOperatorScenarios = []expressionScenario{
"D0, P[], (doc)::{a: 2, b: 4}\n", "D0, P[], (doc)::{a: 2, b: 4}\n",
}, },
}, },
{
description: "Date subtraction",
subdescription: "You can subtract durations from dates. Assumes RFC3339 date time format, see [date-time operators](https://mikefarah.gitbook.io/yq/operators/date-time-operators) for more information.",
document: `a: 2021-01-01T03:10:00Z`,
expression: `.a -= "3h10m"`,
expected: []string{
"D0, P[], (doc)::a: 2021-01-01T00:00:00Z\n",
},
},
{
description: "Date subtraction - custom format",
subdescription: "Use with_dtf to specify your datetime format. See [date-time operators](https://mikefarah.gitbook.io/yq/operators/date-time-operators) for more information.",
document: `a: Saturday, 15-Dec-01 at 6:00AM GMT`,
expression: `with_dtf("Monday, 02-Jan-06 at 3:04PM MST", .a -= "3h1m")`,
expected: []string{
"D0, P[], (doc)::a: Saturday, 15-Dec-01 at 2:59AM GMT\n",
},
},
{
skipDoc: true,
description: "Date subtraction - custom format",
subdescription: "You can subtract durations from dates. See [date-time operators](https://mikefarah.gitbook.io/yq/operators/date-time-operators) for more information.",
document: `a: !cat Saturday, 15-Dec-01 at 6:00AM GMT`,
expression: `with_dtf("Monday, 02-Jan-06 at 3:04PM MST", .a -= "3h1m")`,
expected: []string{
"D0, P[], (doc)::a: !cat Saturday, 15-Dec-01 at 2:59AM GMT\n",
},
},
{ {
description: "Custom types: that are really numbers", description: "Custom types: that are really numbers",
subdescription: "When custom tags are encountered, yq will try to decode the underlying type.", subdescription: "When custom tags are encountered, yq will try to decode the underlying type.",