Amazon AWS / .Net SDK: "MissingMethodException"

I spent way too much time digging into this, and the advice on the internet seems a little vague.

Amazon offers nice PowerShell support for working with its AWS components, such as S3 (storage) and DynamoDB (NoSQL database), and they make it really easy to prototype an idea. I've been using them on a customer project, but we needed higher integration and performance than the stock cmdlets, so I resorted to writing my own cmdlets in C#.

Writing directly in .NET requires the Amazon AWS SDK for .NET, and Visual Studio is happy to retrieve this package via nuget so you can use it directly in your project.

While attempting to use some Async methods in a cmdlet, this happens:

Update-MyStuff : Method not found: 'System.Threading.Tasks.Task`1<Amazon.S3.Model.ListObjectsV2Response>
Amazon.S3.AmazonS3Client.ListObjectsV2Async(Amazon.S3.Model.ListObjectsV2Request, System.Threading.CancellationToken)'.
At C:\Testing\Run-Test.ps1:27 char:1
+ Update-MyStuff -LocalDb $ldb -Worker $worker -Verbose
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Update-Stuff], MissingMethodException
    + FullyQualifiedErrorId : System.MissingMethodException,Unixwiz.Cmdlet.Update_MyStuff

Here, Update-MyStuff is my own cmdlet provided in a DLL, and it compiles and links just fine against the AWS SDK, but now there's a missing method.

The internet shows this happens with different random methods, not just this one. What's going on?

The problem appears to be a conflict between the "same" .NET assemblies provided by the AWS PowerShell tools and those provided by the AWS SDK for .NET.

The AWS PowerShell tools installed on my system to a version-numberred subdirectory under C:\Program Files\WindowsPowerShell\Modules\AWSPowerShell\, while the SDK components were fetched by nuget under my project's packages\AWSSDK.S3.version\lib\net45\ directory, and the net45\ part provides our first clue.

Using Microsoft's ildasm .NET disassembler tool, we compare the PowerShell version with the SDK version side by side, showing that some methods are found in the SDK version but not in PowerShell version:

The first time you use an Amazon-provided cmdlet in a PowerShell session, it automatically loads the older PowerShell-provided version of that assembly (such as AWSSDK.S3.dll), and it properly services whatever the cmdlet means to do.

When you later call your own cmdlet, even though it's depending on features of the newer assemblies, PowerShell sees that the assembly in question—AWSSDK.S3.dll—has already been loaded, so it believes that request has been satisfied. The missing-method error follows henceforth.

Awesome.

How to resolve

As noted above, the net45\ component of the SDK version suggests that the SDK is meant for modern systems with the latest .NET, while the AWS-provided PowerShell components are a bit older and can be used by a wider range of installations. It's not just the Task-based Async functions missing, it's anything that's been added to the library since the older version was created.

Don't use the Amazon-provided AWS PowerShell cmdlets

Though this would prevent the wrong DLLs from getting loaded, it's a painful step because you'd be missing out on a lot of useful facilities: you would have to re-invent the wheel for common functions, such as credential and support.

It would also make it more difficult to experiment with aspects of the AWS ecosystem you've not yet created your own code for.

Explicitly load the proper DLLs prior to invoking the AWS cmdlets

If you manually Import-Module on the AWS.Core and AWS.S3 DLL before you invoke the first AWS cmdlet, the good DLLs will already be loaded, preventing the older ones.

# start of my PowerShell test script, runs in the $(PROJECT)\testing\ directory

Import-Module ..\packages\AWSSDK.Core.3.3.31.11\lib\net45\AWSSDK.Core.dll
Import-Module ..\packages\AWSSDK.S3.3.3.31.19\lib\net45\AWSSDK.S3.dll
Import-Module ..\My.Cmdlet\bin\Debug\My.Cmdlet.dll
...

This definitely works, though it's tedious and inconvenient to keep track of where these DLLs are, especially as packages get updated.

Update the AWS PowerShell module with the newer DLLs.

This seems like a terrible idea, but it seems to work and creates the least hassle to let you get into a groove of software development without having to work around this all the time.

Be sure all PowerShell windows are closed so the AWS modules are not in use, then open a Run-As-Administrator PowerShell session:

PS> C:\Program Files\WindowsPowerShell\Modules\AWSPowerShell\3.3.462.0
PS> mkdir .old
PS> mv AWSSDK.Core.dll .old
PS> mv AWSSDK.S3.dll .old
PS> copy ...\packages\AWSSDK.Core.3.3.31.11\lib\net45\AWSSDK.Core.dll
PS> copy ...\packages\AWSSDK.S3.3.3.31.19\lib\net45\AWSSDK.S3.dll

You will have to remember you did this every time you update your modules.

Ask Amazon to provide newer cmdlets

I'm not sure how one even does this

Disclaimer: I'm not at all strong on this ecosystem with respect to mixing and matching of .NET assemblies built for different runtimes, or how much this applies broadly rather than to my own particular situation.

I have tested all this on my Windows 10 desktop system, and PowerShell reports itself as:

PS> $PSVersionTable

Name                           Value
----                           -----
PSVersion                      5.1.17134.590
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.17134.590
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

And the app I'm building targets targets .NET 4.6.1