Tips, Tricks, Troubleshooting

Validating yaml files
Yaml files can be surprisingly lenient in what can be parsed as a yaml file. A reasonable way of validation a yaml file is to ensure the top level is a map or array (although it is valid yaml to have scalars at the top level, but often this is not what you want). This can be done by:
1
yq --exit-status 'tag == "!!map" or tag== "!!seq"' file.txt > /dev/null
Copied!
Split expressions over multiple lines to improve readability
Feel free to use multiple lines in your expression to improve readability.
Use with if you need to make several updates to the same path.
1
yq --inplace '
2
  with(.a.deeply.nested; 
3
    . = "newValue" | . style="single") |
4
  with(.b.another.nested; 
5
    . = "cool" | . style="folded")
6
' my_file.yaml
Copied!
Create bash array
Given a yaml file like
1
coolActions:
2
  - create
3
  - edit
4
  - delete
Copied!
You can create a bash array named actions by:
1
> readarray actions < <(yq '.coolActions[]' sample.yaml)
2
> echo "${actions[1]}"
3
edit
Copied!
Set contents from another file
Use the load operator to load contents from another file.
Special characters in strings
The strenv operator is a great way to handle special characters in strings:
1
VAL='.a |[email protected]  == "string2"' yq '.a = strenv(VAL)' example.yaml
Copied!
Quotes in Windows Powershell
Powershell has its own way of handling quotes:
1
PS > yq -n '.test = ""something""'
2
test: something
3
PS >
Copied!
See https://github.com/mikefarah/yq/issues/747 for more trickery.
Merge / combine all documents into one
To merge all given yaml files into one, use the reduce operator with the * (multiply) operator. Note the use of ea or eval-all to load all files into memory so that they can be merged.
1
yq ea '. as $item ireduce ({}; . * $item )' file1.yml file2.yml ...
Copied!
Merge - showing the source file and line
To see the original source file and line number of your merged result, you can pre-process the files and add that information in as line comments, then perform the merge.
1
yq ea '(..  lineComment |= filename + ":" + line) | select(fi==0) * select(fi==1)' data1.yaml data2.yaml
Copied!
Merge an array of objects by key
See here for a working example.
Creating a new file / working with blank documents
To create a new yaml file simply:
1
yq -n '.someNew="content"' > newfile.yml
Copied!
Comparing yaml files
The best way to run a diff is to use yq to normalise the yaml files and then just use diff. Here is a simple example of using pretty print -P to normalise the styling and running diff:
1
diff <(yq -P 'sort_keys(..)' file1.yaml) <(yq -P 'sort_keys(..)' file2.yaml)
Copied!
This way you can use the full power of diff and normalise the yaml files as you like.
You may also want to remove all comments using ... comments=""
Reading multiple streams (STDINs)
Like diff and other bash commands, you can use <(exp) to pipe in multiple streams of data into yq. instance:
1
yq '.apple' <(curl -s https://somewhere/data1.yaml) <(cat file.yml)
Copied!
Updating deeply selected paths
or why is yq only returning the updated yaml
The most important thing to remember to do is to have brackets around the LHS expression - otherwise what yq will do is first filter by the selection, and then, separately, update the filtered result and return that subset.
1
yq '(.foo.bar[] | select(name == "fred) | .apple) = "cool"'
Copied!
Combining multiple files into one
In order to combine multiple yaml files into a single file (with --- separators) you can just:
1
yq '.' somewhere/*.yaml
Copied!
Multiple updates to the same path
You can use the with operator to set a nested context:
1
yq 'with(.a.deeply ; .nested = "newValue" | .other= "newThing")' sample.yml
Copied!
The first argument expression sets the root context, and the second expression runs against that root context.
Logic without if/elif/else
yq has not yet added if expressions - however you should be able to use with and select to achieve the same outcome. Lets use an example:
1
- animal: cat
2
- animal: dog
3
- animal: frog
Copied!
Now, if you were using good ol' jq - you may have a script with ifs like so:
1
jq ' .[] |=
2
if (.animal == "cat") then
3
  .noise = "meow" |
4
  .whiskers = true
5
elif (.animal == "dog") then
6
  .noise = "woof" |
7
  .happy = true
8
else 
9
  .noise = "??"
10
end
11
' < file.yaml
Copied!
Using yq - you can get the same result by:
1
yq '.[] |= (
2
  with(select(.animal == "cat"); 
3
    .noise = "meow" | 
4
    .whiskers = true
5
  ) |
6
  with(select(.animal == "dog"); 
7
    .noise = "woof" | 
8
    .happy = true
9
  ) |
10
  with(select(.noise == null); 
11
    .noise = "???"
12
  )
13
)' < file.yml
Copied!
Note that the logic isn't quite the same, as there is no concept of 'else'. So you may need to put additional logic in the expressions, as this has for the 'else' logic.
yq adds a !!merge tag automatically
The merge functionality from yaml v1.1 (e.g. <<:has actually been removed in the 1.2 spec. Thankfully, yq underlying yaml parser still supports that tag - and it's extra nice in that it explicitly puts the !!merge tag on key of the map entry. This tag tells other yaml parsers that this entry is a merge entry, as opposed to a regular string key that happens to have a value of <<:. This is backwards compatible with the 1.1 spec of yaml, it's simply an explicit way of specifying the type (for instance, you can use a !!str tag to enforce a particular value to be a string.
Although this does affect the readability of the yaml to humans, it still works and processes fine with various yaml processors.