Deserializing Large JSON Payloads into a PowerShell Hashtable

When using the ConvertFrom-Json PowerShell cmdlet, you’re asking the JSON JavaScriptSerializer to deserialize a string and turn it into a PowerShell hashtable. Hidden within this class is a maxJsonLength property. If you’re working with a large enough JSON payload and manage to exceed the value, you’ll return an exception such as this:

[3,9] Exception calling "DeserializeObject" with "1" argument(s): "Error during serialization or deserialization using the JSON JavaScriptSerializer. The length of the string exceeds the value set on the maxJsonLength property. Parameter name: input"

I had a heck of a time turning that into simpler terms, but it’s really just saying that the RawContentLength of the replied content is too large. Queue the sad trombone.

Riker loves to play the sad trombone
Riker loves to play the sad trombone

It’s not possible to adjust this value using PowerShell, but I did manage to find some folks who wrote the necessary .NET code to tweak the value. I’ve adjusted the format slightly.

[void][System.Reflection.Assembly]::LoadWithPartialName("System.Web.Extensions")
$result = (New-Object -TypeName System.Web.Script.Serialization.JavaScriptSerializer -Property @{MaxJsonLength=67108864}).DeserializeObject($content)

So, you’re done, right? Well … no. Assuming that $content is the large JSON payload, the results of the newly tweaked JavaScriptSerializer will be a Generic Dictionary. It’s a bit of a mess and isn’t all that useful in this format. Running a GetType() on the returned object reveals:

System.Collections.Generic.Dictionary`2

generic-dictionary

Yuck.

Cleaning Up the Generic Dictionary

The resolution is to use a helper function to parse through and clean up the dictionary. A handy one is provided by Florian here. Again, I’ve removed some of the unwanted code from the script to filter it down to just the needed functions.

function ParseItem($jsonItem) 
{
    if($jsonItem.PSObject.TypeNames -match 'Array') 
    {
        return ParseJsonArray($jsonItem)
    }
    elseif($jsonItem.PSObject.TypeNames -match 'Dictionary') 
    {
        return ParseJsonObject([HashTable]$jsonItem)
    }
    else 
    {
        return $jsonItem
    }
}

function ParseJsonObject($jsonObj) 
{
    $result = New-Object -TypeName PSCustomObject
    foreach ($key in $jsonObj.Keys) 
    {
        $item = $jsonObj[$key]
        if ($item) 
        {
            $parsedItem = ParseItem $item
        }
        else 
        {
            $parsedItem = $null
        }
        $result | Add-Member -MemberType NoteProperty -Name $key -Value $parsedItem
    }
    return $result
}

function ParseJsonArray($jsonArray) 
{
    $result = @()
    $jsonArray | ForEach-Object -Process {
        $result += , (ParseItem $_)
    }
    return $result
}

function ParseJsonString($json) 
{
    $config = $javaScriptSerializer.DeserializeObject($json)
    return ParseJsonObject($config)
}

By feeding the .NET Serializer data into the ParseItem function, the content is properly stored into a PowerShell hashtable. It takes a little bit to iterate through the values – be patient. 🙂

$result = ParseItem ((New-Object -TypeName System.Web.Script.Serialization.JavaScriptSerializer -Property @{MaxJsonLength=67108864}).DeserializeObject($content))

proper-hashtable

Now that the data is stored in a proper hashtable, it can be used for other nefarious activities. This one stumped me a bit at first, so I hope my work helps anyone else dealing with large JSON payloads. A hat tip to those who wrote the code above so that I could piece it all together here.