Tokenizing PowerShell Scripts

By turning PowerShell code into tokens and structures, you can find errors, auto-document your code, and create powerful refactoring tools.

By turning PowerShell code into tokens and structures, you can find errors, auto-document your code, and create powerful refactoring tools.

Colorful World of Tokens

Whenever you load PowerShell code into specialized editors, the code gets magically colored, and each color represents a given token type. The colors can help you understand how PowerShell interprets your code.

Generic editors without a built-in PowerShell Engine like notepad++ or VSCode use complex regular expressions to try and identify the correct tokens. A 100% precise tokenization however comes directly from the PowerShell Parser and is not the result of generic RegEx rules. In this article series we’ll look at all the goodness the PowerShell Parser is willing to share with you.

At the end of today, you get a new command: Test-PSOneScript parses one - or thousands - of PowerShell files and returns always 100% accurate tokens in no time. It is part of our PSOneTools module, so just install the latest version to get your hands on the command, or use the source code presented later in this article.

Install-Module -Name PSOneTools -Scope CurrentUser -Force

With tokens you can do a whole bunch of interesting things, for example:

  • Auto-document code and create lists of variables, commands, or method calls found in a script
  • Identify syntax errors that make the parser choke
  • Perform a security analysis and identify scripts using risky commands

PSParser Overview

The PSParser is the original parser built into the early versions of PowerShell. Even though it is old, it is still part of all PowerShell versions and very useful because of its simplicity. It distinguishes 20 different token types:

PS> [Enum]::GetNames([System.Management.Automation.PSTokenType]).Count
20

PS> [Enum]::GetNames([System.Management.Automation.PSTokenType]) | Sort-Object
Attribute
Command
CommandArgument
CommandParameter
Comment
GroupEnd
GroupStart
Keyword
LineContinuation
LoopLabel
Member
NewLine
Number
Operator
Position
StatementSeparator
String
Type
Unknown
Variable

When you use PSParser to tokenize PowerShell code, it reads your code character by character and groups the characters into meaningful words, the tokens. If the PSParser encounters characters it isn’t expecting, it generates Syntax Errors, i.e. when a string starts with double-quotes but ends with single-quotes.

Tokenizing PowerShell Code

Use Tokenize() to tokenize PowerShell code. Here is a simple example:

# the code that you want tokenized:
$code = {
  # this is some test code
  $service = Get-Service |
    Where-Object Status -eq Running
}


# create a variable to receive syntax errors:
$errors = $null
# tokenize PowerShell code:
$tokens = [System.Management.Automation.PSParser]::Tokenize($code, [ref]$errors)

# analyze errors:
if ($errors.Count -gt 0)
{
  # move the nested token up one level so we see all properties:
  $syntaxError = $errors | Select-Object -ExpandProperty Token -Property Message
  $syntaxError
}
else
{
  $tokens
}

Tokenize() expects the code you want to tokenize, plus an empty variable that it can fill with any syntax errors. Because the variable $errors is empty when Tokenize starts, and gets filled while the method parses the code, it needs to be submitted by reference (by memory pointer) which in PowerShell is done through [ref].

When Tokenize() completes, you receive all tokens as return value in $tokens, plus any syntax errors in $errors.

Looking at Tokens

This is what the first three token returned in $tokens look like:

PS> $tokens[0..2]


Content     : 
              
Type        : NewLine
Start       : 0
Length      : 2
StartLine   : 1
StartColumn : 1
EndLine     : 2
EndColumn   : 1

Content     : # this is some test code
Type        : Comment
Start       : 4
Length      : 24
StartLine   : 2
StartColumn : 3
EndLine     : 2
EndColumn   : 27

Content     : 
              
Type        : NewLine
Start       : 28
Length      : 2
StartLine   : 2
StartColumn : 27
EndLine     : 3
EndColumn   : 1

Each token is represented by a PSToken object which returns the token content as string, the token type, and the exact position where the token was found.

How Syntax Errors Work

If the parser encounters unexpected characters while parsing the code, it generates a syntax error. The parser continues parsing, so there can be multiple syntax errors returned.

Let’s create a syntax error and send a string to the parser that is missing its ending quote.

To send faulty code to the parser, you cannot use a scriptblock though because scriptblocks are smart and only accept formally correct PowerShell code. That’s why you have to send your faulty PowerShell code to the parser as a string instead of a scriptblock.

# the code that you want tokenized:
$code = "
  'Hello
"

When you run the script again, it now returns the syntax error(s):

PS> $syntaxError


Message     : The string is missing the terminator: '.
Content     : 'Hello
              
Type        : Position
Start       : 4
Length      : 8
StartLine   : 2
StartColumn : 3
EndLine     : 3
EndColumn   : 1

Improving Parser Error Objects

The parser emits a PSParseError object per syntax error which looks like this:

PS> $errors

Token                                Message                                 
-----                                -------                                 
System.Management.Automation.PSToken The string is missing the terminator: '.

Unfortunately, the token details are hidden inside the property Token. So I use a little-known trick to make all properties visible immediately:

Select-Object supports the use of -Property and -ExpandProperty at the same time. So I used -ExpandProperty to take the PSToken object out of Token, plus used -Property to attach the original property Message to the extracted token. As a result, all properties show up immediately:

PS> $errors | Select-Object -ExpandProperty Token -Property Message


Message     : The string is missing the terminator: '.
Content     : 'Hello
              
Type        : Position
Start       : 4
Length      : 8
StartLine   : 2
StartColumn : 3
EndLine     : 3
EndColumn   : 1

Examining Real Scripts

To examine real file-based scripts, simply embed the logic from above inside a pipeline-aware function. Test-PSOneScript does exactly this and makes parsing PowerShell files a snap:

function Test-PSOneScript
{
  <#
      .SYNOPSIS
      Parses a PowerShell Script (*.ps1, *.psm1, *.psd1)

      .DESCRIPTION
      Invokes the simple PSParser and returns tokens and syntax errors

      .EXAMPLE
      Test-PSOneScript -Path c:\test.ps1
      Parses the content of c:\test.ps1 and returns tokens and syntax errors

      .EXAMPLE
      Get-ChildItem -Path $home -Recurse -Include *.ps1,*.psm1,*.psd1 -File |
         Test-PSOneScript |
         Out-GridView

      parses all PowerShell files found anywhere in your user profile

      .EXAMPLE
      Get-ChildItem -Path $home -Recurse -Include *.ps1,*.psm1,*.psd1 -File |
         Test-PSOneScript |
         Where-Object Errors

      parses all PowerShell files found anywhere in your user profile
      and returns only those files that contain syntax errors

      .LINK
      https://powershell.one
  #>


  param
  (
    # Path to PowerShell script file
    # can be a string or any object that has a "Path" 
    # or "FullName" property:
    [String]
    [Parameter(Mandatory,ValueFromPipeline)]
    [Alias('FullName')]
    $Path
  )
  
  begin
  {
    $errors = $null
  }
  process
  {
    # create a variable to receive syntax errors:
    $errors = $null
    # tokenize PowerShell code:
    $code = Get-Content -Path $Path -Raw -Encoding Default
    
    # return the results as a custom object
    [PSCustomObject]@{
      Name = Split-Path -Path $Path -Leaf
      Path = $Path
      Tokens = [Management.Automation.PSParser]::Tokenize($code, [ref]$errors)
      Errors = $errors | Select-Object -ExpandProperty Token -Property Message
    }  
  }
}

Parsing Individual Files

To parse an individual file, simply submit its path to Test-PSOneScript. It immediately returns the tokens and any syntax errors (if present):

$Path = "C:\Users\tobia\test.ps1"
$result = Test-PSOneScript -Path $Path

Checking for Errors

Let’s start with checking whether the script file has syntax errors:

PS> $result.Errors.Count -gt 0
False

To get a list of all token types present in the script, try this (the output may vary depending on the actual code in your script file, of course):

PS> $result.Tokens.Type | Sort-Object -Unique
Command
CommandParameter
CommandArgument
Number
String
Variable
Member
Type
Operator
GroupStart
GroupEnd
Keyword
Comment
NewLine

Creating a List of Used Variables

To get a list of all variables used in the script, simply filter for token type Variable:

PS> $result.Tokens | 
  Where-Object Type -eq Variable | 
  Sort-Object -Property Content -Unique | 
  ForEach-Object { '${0}' -f $_.Content}

$_ldaptype
$_SortedReportProp
$AD_Capabilities
$AD_CreateDiagrams
$AD_CreateDiagramSourceFiles
$AD_DomainGPOs
...
$xlEqual
$zipPackage
$ZipReport
$ZipReportName

Creating a List of Used Commands

Likewise, if you’d like to get a list of commands used by the script, filter for the appropriate token type (Command):

PS> $result.Tokens | 
  Where-Object Type -eq Command | 
  Sort-Object -Property Content -Unique | 
  Select-Object -ExpandProperty Content

Add-Content
Add-Member
Add-Type
Add-Zip
Append-ADUserAccountControl
ConvertTo-HashArray
ConvertTo-Html
...
Start-sleep
Test-Path
Where
write-error
Write-Output
Write-Verbose
Write-Warning

You can even analyze the frequency of how often commands were used. This gets you the 10 most-often used commands:

PS> $result.Tokens | 
  Where-Object Type -eq Command | 
  Select-Object -ExpandProperty Content |
  Group-Object -NoElement |
  Sort-Object -Property Count -Descending |
  Select-Object -First 10

Count Name                     
----- ----                     
   51 Search-AD                
   49 New-Object               
   35 Write-Verbose            
   29 get-date                 
   25 %                        
   24 New-TimeSpan             
   24 Where                    
   21 select                   
   19 Sort-Object              
   17 Invoke-Method            

Analyzing Use of .NET Methods

Maybe you are interested in finding out which native .NET methods the script uses. Again, it is just a matter of token filtering:

PS> $result.Tokens | 
  Where-Object Type -eq Member | 
  Select-Object -ExpandProperty Content |
  Sort-Object -Unique
  
Accessible
ActiveSheet
Add
AdjacentSites
adminDisplayName
...
whenchanged
whencreated
Workbooks
Worksheets

At this point, you are reaching the limit of token analysis:

While it is nice to get a list of method names used by a script, it is not really useful. You’d need a bigger picture to know the object types the called methods belong to. All this is possible, too, but not with tokens alone. What’s required is a look at script structures that consist of multiple tokens - a case for the Abstract Syntax Tree (AST) which we shed light on in one of the next parts of this series.

Bulk-Analysis: Scanning Entire Folders

Test-PSOneScript can’t just examine one file at a time. It is fully pipeline-aware and knows how to deal with files returned by Get-ChildItem.

Finding Scripts With Errors

So if you want to identify scripts with syntax errors anywhere in your script library, simply runGet-ChildItem to gather the files to be tested, and pipe them to Test-PSOneScript like this:

# get all PowerShell files from your user profile...
Get-ChildItem -Path $home -Recurse -Include *.ps1, *.psd1, *.psm1 -File |
  # ...parse them...
  Test-PSOneScript |
  # ...filter those with syntax errors...
  Where-Object Errors |
  # ...expose the errors:
  ForEach-Object {
    [PSCustomObject]@{
      Name = $_.Name
      Error = $_.Errors[0].Message
      Type = $_.Errors[0].Type
      Line = $_.Errors[0].StartLine
      Column = $_.Errors[0].StartColumn
      Path = $_.Path
    }
  }

This will find any script with any syntax error. If you’d like to be more specific, you can filter on the error message.

Identifying Risky Commands

The sky is the limit, so if you’d like to identify scripts that use risky commands such as Invoke-Expression, just adjust the filter:

$blacklist = @('Invoke-Expression', 'Stop-Computer', 'Restart-Computer')


# get all PowerShell files from your user profile...
Get-ChildItem -Path $home -Recurse -Include *.ps1, *.psd1, *.psm1 -File |
# ...parse them...
Test-PSOneScript |
# ...filter those using commands in our blacklist
Foreach-Object {
  # get the first token that is a command and that is in our blacklist
  $badToken = $_.Tokens.Where{$_.Type -eq 'Command'}.Where{$_.Content -in $blacklist} | 
    Select-Object -First 1
  
  if ($badToken)
  {
    $_ | Add-Member -MemberType NoteProperty -Name BadToken -Value $badToken -PassThru
  }
  } |
  # ...expose the errors:
  ForEach-Object {
    [PSCustomObject]@{
      Name = $_.Name
      Offender = $_.BadToken.Content
      Line = $_.BadToken.StartLine
      Column = $_.BadToken.StartColumn
      Path = $_.Path
    }
  }

What’s Next

Using thePSParser is just your first step into the wonderful world of tokens and script analysis. In the next part we’ll take a look at the more sophisticated Parser object which was introduced in PowerShell 3 and differentiates 150 different token kinds plus 26 token flags.

And if that’s still not enough detail, we look into the Abstract Syntax Tree (AST) and how it forms meaningful structures from a group of token.

BTW, have you checked out PowerShell Conference EU yet? Both Call for Papers and Delegate Registration are open!