Validating yaml files

Yaml files can be surprisingly lenient in what can be parsed as a yaml file. A reasonable way of validation a yaml file is to ensure the top level is a map or array (although it is valid yaml to have scalars at the top level, but often this is not what you want). This can be done by:

yq e --exit-status 'tag == "!!map" or tag== "!!seq"' file.txt > /dev/null

Set contents from another file

Use an environment variable with the strenv operator to inject the contents from an environment variable.

LICENSE=$(cat LICENSE) yq eval -n '.a = strenv(LICENSE)'

Special characters in strings

The strenv operator is a great way to handle special characters in strings:

VAL='.a |[email protected]  == "string2"' yq e '.a = strenv(VAL)' example.yaml

Quotes in Windows Powershell

Powershell has its own way of handling quotes:

PS > yq e -n '.test = ""something""'
test: something
PS >

Merge / combine all documents into one

To merge all given yaml files into one, use the reduce operator with the * (multiply) operator. Note the use of ea or eval-all to load all files into memory so that they can be merged.

yq ea '. as $item ireduce ({}; . * $item )' file1.yml file2.yml ...

Creating a new file / working with blank documents

To create a new yaml file simply:

yq e -n '.someNew="content"' > newfile.yml

Note that running expressions against blank files does not work, because the file is empty, there are no matches for yq to run through the expression pipeline and so nothing happens.

Comparing yaml files

The best way to run a diff is to use yq to normalise the yaml files and then just use diff. Here is a simple example of using pretty print -P to normalise the styling and running diff:

diff <(yq e -P examples/data1.yaml) <(yq e -P examples/data2.yaml)

This way you can use the full power of diff and normalise the yaml files as you like - for instance you may also want to remove all comments using ... comments=""

Reading multiple streams (STDINs)

Like diff and other bash commands, you can use <(exp) to pipe in multiple streams of data into yq. instance:

yq e '.apple' <(curl -s https://somewhere/data1.yaml) <(cat file.yml)

yq adds a !!merge tag automatically

The merge functionality from yaml v1.1 (e.g. <<:has actually been removed in the 1.2 spec. Thankfully, yq underlying yaml parser still supports that tag - and it's extra nice in that it explicity puts the !!merge tag on key of the map entry. This tag tells other yaml parsers that this entry is a merge entry, as opposed to a regular string key that happens to have a value of <<:. This is backwards compatible with the 1.1 spec of yaml, it's simply an explicit way of specifying the type (for instance, you can use a !!str tag to enforce a particular value to be a string.

Although this does affect the readability of the yaml to humans, it still works and processes fine with various yaml processors.