It’s high time for the next installment in this series. I was planning on getting to it sooner, but I’ve had a bunch of crazy things going on to do with getting the .NET Framework integrated into Windows Vista and planning for the next couple versions of the .NET Framework. None of that is calming down, but it’s time to get out another installment none the less.

I took a quick and somewhat vague look at type forwarders in my previous post (on purpose). Some of you were likely satisfied by that level of information and others are likely interested in more detail. As the title of this post suggests, this is the place to get more detail. To best do this, I’ve created a simple – and not particularly useful – application to demonstrate how forwarders really work. I’ll include this app in my next post.

Here is the entire program (it is only the Main method):

using System;

using StringUtilities;

using MathUtilities;

namespace TypeForwarderApp

{

    class Program

    {

        static void Main(string[] args)

        {

            Int32 int1, int2, intSum, stringCharacterCount;

            String initialString, reversedString;

            String appMessage;

            appMessage = "This app doesn't really do anything useful";

            Console.WriteLine(appMessage);

            int1 = 16;

            int2 = 14;

            initialString = "Happy 30th Birthday Microsoft!";

            intSum = MathUtil.Add(int1, int2);

            reversedString = StringReverse.GetReversedString(initialString);

            stringCharacterCount = StringCharacterCount.GetCount(initialString);

            Console.WriteLine();

            Console.WriteLine("Initial String : {0}", initialString);

            Console.WriteLine("Reversed String: {0}", reversedString);

            Console.WriteLine("String Length  : {0}", stringCharacterCount);

            Console.WriteLine();

            Console.WriteLine("Integers: {0}, {1}", int1, int2);

            Console.WriteLine("Sum     : {0}", intSum);

            Console.WriteLine();

            Console.WriteLine(StringReverse.GetReversedString(appMessage));

        }

    }

}

Like, I said, the program doesn’t do anything useful. I included the birthday message given that two Fridays ago was the annual Microsoft company meeting at SafeCo Field in Seattle. The theme was “Beyond 30” celebrating the first 30 years of Microsoft, although the content was much more about the future than the past (which I imagine most folks there appreciated). The 25^th anniversary meeting was much more of a history lesson. I think that was the one that Sinbad was at ;)

I’m going to concentrate on one type, the StringCharacterCount type. At the moment, it is part of the StringUtilties assembly. If we look at the MSIL code within TypeForwarderApp.exe, we’ll see that clearly.

This is the MSIL code within the exe that calls the static method GetString on the StringCharacterCount type:

  IL_0030:  call       int32 [StringUtilities]StringUtilities.StringCharacterCount::GetCount(string)

As you can see, the fact that StringCharacterCount lives within the StringUtilities assembly is pretty apparent. Actually, you could go as far as saying it is hard-coded. Hence the need for forwarders …

Well, I’m a typical developer and I’ve decided in my second version of StringUtilites and MathUtilites – I happen to own them both – that StringCharacterCount is really a math utility and less of a string utility. As a result, I’m going to move this cool type to the MathUtilities assembly.

I just moved it to the MathUtilties assembly. You can see that it is indeed part of the MathUtilities assembly via the MetaInfo (CTRL-M) view in ILDasm.

TypeDef #2 (02000003)

-------------------------------------------------------

      TypDefName: StringUtilities.StringCharacterCount  (02000003)

      Flags     : [Public] [AutoLayout] [Class] [AnsiClass] [BeforeFieldInit]  (00100001)

      Extends   : 01000001 [TypeRef] System.Object

      Method #1 (06000003)

      -------------------------------------------------------

            MethodName: GetCount (06000003)

Notice that the StringCharacterCount type is still part of the StringUtilities namespace and not MathUtilties. This is for two reasons: the namespace is part of the type name and type forwarders do not change the type name in any way. Put another way, if I moved StringCharacterCount from the one assembly to the other *and* changed its namespace, then I would have a breaking change that even type forwarders could not mitigate.

This all makes sense, but we have yet to see the interesting part. What does the metadata within the new version of StringUtilties look like? Let’s take a look at the manifest (another ILDasm view) within the StringUtilties assembly. There are actually two additional directives that we didn’t have before.

.assembly extern MathUtilities

{

  .ver 1:0:0:0

}

.class extern forwarder StringUtilities.StringCharacterCount

{

  .assembly extern MathUtilities

}

StringUtilties now has a dependency on MathUtilties and there is this “.class extern forwarder” line for “StringUtilities.StringCharacterCount”. Clearly, we’ve hit our jackpot. The dependency on MathUtilities is clearly required, given that StringUtilities intends to forward to it. The second directive, the class directive, is a way to signal to the runtime that the assembly knows about the class, but that it is located at another location.

If we look further yet, we’ll see that there is an entry in ExportedType table.

ExportedType #1 (27000001)

-------------------------------------------------------

      Token: 0x27000001

      Name: StringUtilities.StringCharacterCount

      Implementation token: 0x23000002

      TypeDef token: 0x00000000

      Flags     : [NotPublic] [AutoLayout] [Class] [AnsiClass] [Forwarder]  (00200000)

We’re now looking at metadata instead of the MSIL view (the directives) above. The ExportedType table is generally used to publish all the types in child netmodules of an assembly – build a multi-module assembly and you’ll notice this first-hand. In the case of multi-module assemblies, the implementation token would point to the netmodule, not to a separate assembly. In the case of type forwarders, the implementation token points (naturally) to a separate assembly. We can easily prove that too. Look at the AssemblyRef entry for the MathUtilities Assembly below.

AssemblyRef #2 (23000002)

-------------------------------------------------------

      Token: 0x23000002

      Public Key or Token:

      Name: MathUtilities

      Version: 1.0.0.0

      Major Version: 0x00000001

      Minor Version: 0x00000000

      Build Number: 0x00000000

      Revision Number: 0x00000000

      Locale: <null>

      HashValue Blob:

      Flags: [none] (00000000)

Notice that its token (0x23000002)  matches the implementation token of our forwarded type (0x23000002). There you go. That’s it! Now you know everything I’m going to tell you about type forwarders.

Type forwarders are an interesting feature. If you’ve never needed them, you’ve probably never thought about or imagined them. They were designed to allow developers to move types from one assembly to another without breaking existing code that was dependent on types living within a particular assembly. There are a bunch of reasons why you might want to move types around; however existing type references will make that problematic.

You may be thinking “I move types between assemblies all the time and nothing breaks”. That may be true if you are building applications that happen to have dependent assemblies. At the point that you build frameworks that other developers build apps on *and* you distribute those libraries as pre-compiled binaries (not as source) *and* you promote apps built against earlier versions of your framework to newer versions (say with publisher policy or binding re-directs) without re-compiling apps, then you can easily run into this problem.

The .NET Framework hits this scenario dead-on. Everett apps, when run on Whidbey (i.e. Whidbey-only machines), for example, run against the Whidbey .NET Framework libraries. That work without issue, but those Everett apps still expect types to be in the same place as they where in Everett, which in the case of System.String is mscorlib.dll. If we moved System.String to system.dll, for example, we’d break 100% of Everett apps run on Whidbey. Why? The CLR loader would no longer be able to resolve the type reference – [mscorlib]System.String -- stored in the app’s metadata, but instead would throw a System.TypeLoadException exception. The app wouldn’t like that and neither would it’s users ;)

Before anyone gets the wrong idea, we didn’t move System.String or any other existing types in Whidbey. I’m merely using the .NET Framework as an example of how framework developers could similarly run into this problem, which would be the precursor for needing type forwarders.

Enter type forwarders. Type forwarders are a new MSIL directive that essentially say “type x used to be in this assembly, but it is now in this other one. Maybe you should go look over there”. Let me just show you.

1. Create type t1 in assembly asm1
2. Create an app that uses [asm1]t1 (that’s just short-hand for saying type t1 that lives in assembly asm1)
3. Run app.
   1. Loader loads asm1
   2. Loader loads t1
   3. Everything works
4. Close app and move back to VS 2005.
5. Move type t1 from asm1 to asm2 (these will be two different projects)
6. Create a forwarder in asm1 that points to [asm2]t1
7. Recompile asm1 and asm2. Do not recompile the app.
8. Run app
   1. Loader loads asm1
   2. Loader notices forwarder directive in asm1, pointing to asm2
   3. Loader loads asm2
   4. Loader loads t1 (this time for asm2)
   5. Everything works

You do need to realize though that forwarders are really just a temporary crutch for older apps. Notice that we didn’t re-compile the app above. That’s actually the point of the whole scenario. We’re assuming that we don’t have the option of re-compiling the app, because we don’t own it – in this scenario, we own the framework, not the app. When the owner of the app does get the chance, we get the following list of activities:

1. Re-compile app against the latest version of asm2
2. Run new version of app
   1. Loader loads asm2
   2. Loader loads t1
   3. Everything works

Notice that we no longer visit the forwarder in asm1. When the app owner re-compiled the app, the compiler found t1 in asm2. As a result, we no longer needed the mis-step in asm1. Like I said earlier, forwarders are merely a crutch for apps compiled against older versions of your framework. Once the apps are re-compiled against the new version of your framework, the forwarder is no longer needed (for that app). Unfortunately, you’ll need to keep that forwarder in place for some time, as there are likely a whole host of other apps and add-ins to those apps still reliant on the forwarder to work properly.

My next post will deal with the details of forwarders and I’ll post some source that uses the feature.

My last “Wonders of Whidbey” series seemed to be quite well received, at least in terms of aggregator traffic, so I thought that I’d do another one. This time, I’d like to talk about the new factoring features in Whidbey. This topic has been bouncing around my head now for quite some time, so it was easy to choose. I also have the next WoW series picked out, but you’ll have to wait a little longer for that one.

I’d like to give you some insight into these features first. I’m a little vague on the birth (early 2003?) of these two features, since I was not on the CLR team yet, but working in another part of Microsoft at that time. I joined the team in November 2003. In early 2003, a subset of the CLR team (and a bunch of other folks from across the company) was working on some major improvements to managed code versioning. They had a particular take on the problematic parts of the current versioning system and had come up with quite an interesting solution. As part of that, the team was going to do some serious factoring (moving types between assemblies) of the .NET Framework, but that approach would have the negative effect of breaking pretty much 100% of existing applications since type references included the assembly identity. As a result, the team developed some cool factoring features in the CLR to mitigate those problems.

We’re no longer going down the same path with versioning as we thought we were back in 2003, but we’ve kept the factoring features in the product. We do have some other uses for these features, but they are not quite as far reaching as the earlier versioning plan. I think that Microsoft platform teams are realizing, more and more, that many of the engineering problems that they face are not unique to Microsoft, but are also faced by other software companies. That realization is actually the basis of the Visual Studio Team System products in a lot of ways. That’s a long way of saying that we’ve kept these factoring features in the product, believing that they will be useful to enough managed code developers out there. That being said, I don’t expect a lot of folks out there to use these features. In fact, I’m hoping that adoption of these features is left to niche scenarios, as I can imagine some folks painting themselves into some bad corners if they do not use them judicially.

The features in question are colloquially called “Friend Assemblies” and “Type Forwarders”. They have different names within the product, but I recommend using these names since they are accurate and easier for folks to grasp than InternalsVisibleTo and TypeForwardedTo. Learn more about them in the following posts in this series.

In my two (one and two) earlier "Wonders of the Whidbey GAC" posts, I talked about processor architecture and interop with the Everett GAC. I left out one piece of interesting data because it wouldn't make as much sense until you had read the earlier posts. This data point is the binding order among the GACs. Naturally, you cannot look at them all at once, so we defined an order of probing the GACs that we always adhere to and tends to make sense.

Here is is:

1. Bitness GAC
   • "gac_32" when running on a 32-bit machine or under WoW64 on a 64-bit machine
   • "gac_64" when running natively (not WoW64) on a 64-bit machine (this is the normal case for Whidbey apps on a 64-bit machine)
2. Bitness agnostic GAC
   • "gac_msil" on all machines
   • assemblies in this GAC run equally well everywhere (X64, X86, IA64)
3. Legacy v1.x GAC
   • "gac" on all machines
   • this is where v1.0 and everett assemblies go

You can read this same information on Junfeng's blog too, as he implemented all of this. I just slow him down ;)

Junfeng referes to MSIL assemblies as "portable". I use the term "agnostic". It is the same thing.

If you write managed code in your favourite language (i.e. c#, vb) and don't do anything special, your assemblies will fall into this category. You can do the same thing with MC++ code too, if you compile it as /clrpure; otherwise, mc++ code is by design "bit-specific". Oh, there's another term, but I'm sure you can figure that one out.

If you are unsure of which of the three GAC buckets above your assembly will fall into, and your assembly is strong-name signed, install it into the GAC with gacutil /i from the Whidbey SDK. Gacutil will install the assembly into the correct GAC and then you'll know for sure. You can also open up the assembly (not required to be strong-name signed) in ildasm. That will also tell you the same answer.

I received a question from a customer a few days ago about assembly versioning. I receive questions quite often, of which >90% surprisingly enough come via Brad. I can only imagine what his inbox looks like every morning! Anyway, the question touched on the focus for this post which is the relationship between the Everett and Whidbey GACs. Before I go any further, I’m happy to report that the relationship is sensible and easy to understand. Too often, things don’t make a lot of sense and that always bugs me.

The customer’s question was actually about assembly versioning in the case that you want to support Everett and Whidbey, which I hope to cover tomorrow, since that’s when I told them I’d give them an answer ;)

First things first, on the surface it seems like interop between the GACs is what is wanted. In actual fact, it is the behaviour of the CLR loader and gacutil.exe that is really the interesting part. The GACs themselves are passive and don’t do anything other than act as a quite compelling directory structure for holding code w/o any naming conflicts.

Migrating the Everett GAC to the Whidbey GAC

We don’t do anything like this. This would have been the wrong direction for multiple reasons. Just thought I’d take care of this one up-front.

Hard linking assemblies between the Everett and Whidbey GACs as a Perfomance Enhancement

I’m glad that we didn’t do anything like this either. Ouch. OK, now onto the real answers.

Whidbey CLR Loader GAC Probing

The Whidbey CLR loader – Fusion in this case – probes for assemblies in the Everett and Whidbey GACs when an assembly request comes through the system. I’ll skip the exact algorithm and any optimizations that we might do. The high-order bit here is that we do indeed look in both GACs, so there is no need to stop using the Everett GAC for Everett assemblies since we look there. In fact, as you’ll find out below, you cannot stop using the Everett GAC.

Gacutil.exe, the Fusion APIs and installation via MSI (installers)

There are several ways that you can get assemblies installed into the GAC, with the last one being the recommended option. The big thing I wanted to know when I asked the question is what the expected behaviour was when I added an assembly to the GAC via these different methods.

Question:

Do the Whidbey tools put Everett assemblies into the Everett GAC?

Given:

Everett and Whidbey .NET Frameworks are installed on the machine

Assemblies:

1) 1 assembly compiled against Everett csc.exe (csharp compiler)

2) 1 assembly compiled against Whidbey vbc.exe (vb compiler)

Note: the compilers (vb, csharp) don’t matter for the assemblies above. I’m just trying to mix it up a little. It is the .Net Framework versions that matter.

Tools:

A) Everett Gacutil

B) Everett Fusion APIs

C) Whidbey Gacutil

D) Whidbey Fusion APIs

E) MSI installation

Scenarios + expected outcome:

1. (1) + (A) -- Everett GAC

2. (1) + (B) -- Everett GAC

3. (1) + (C) -- Everett GAC

4. (1) + (D) -- Everett GAC

5. (1) + (E) -- Everett GAC

6. (2) + (A) -- failure

7. (2) + (B) -- failure

8. (2) + (C) -- Whidbey GAC

9. (2) + (D) -- Whidbey GAC

10. (2) + (E) -- Whidbey GAC

Scenarions 6 and 7 fail since Whidbey-compiled assemblies are not loadable via the Everett CLR. Also, since processor architecture is a major part of Whidbey, it would be difficult to store Whidbey assemblies in the Everett GAC in a reasonable manner.

Why does Whidbey put Everett assemblies into the Everett GAC?

We continue putting Everett assemblies into the Everett GAC for two reasons:

-          Everett apps still require those assemblies and will only be able to use them if they are in the Everett GAC since the Everett CLR loader knows nothing about the Whidbey GAC. It is much easier and smarter for the Whidbey loader to look in the Everett GAC.

-          Everett assemblies didn’t have the concept of processor architecture, which the Whidbey GAC is heavily based on, so it doesn’t make sense to store Everett assemblies in that new structure.

-          Everett is going to live for a long time. Let’s not mess with a good thing ;)

That was actually 3 reasons, if you count the last one.

Is there an Everett GAC if there is no Everett .NET Framework on the Machine, but only Whidbey?

Yes. The Everett GAC is created as needed when Everett is not on machine. This means that the Everett GAC structure is auto-created when the first Everett-era assembly is added to the Whidbey-only machine. If Everett was also on this machine, the Everett GAC would have been created at the point that the Everett .NET Framework was installed.

The GAC -- Global Assembly Cache -- is a popular topic among managed code developers. It serves two main purposes: (A) as a catalogue of the globally publicly available assemblies available on the system, indexed by strong name, and (B) the location of those same files. It is not my intent to describe the GAC, however; look here for more info. My intent is to describe some interesting changes that have been made to the GAC for Whidbey and to promote some good practices. This is part I and is dedicated to processor architecture; we'll see how far we get with the series ;)

Whidbey is the first CLR release in which we support 64-bit execution. The CLR team has been working on porting the CLR to 64-bit for several years now and is now releasing it to the world with Whidbey. Woohoo! Not surprisingly, 64-bit has required a lot of changes all over the CLR, as it imports a bunch of new concepts onto us. My favourite is the general concept of processor architecture, of which there are four that we now support: X86, X64, IA64 and MSIL. We always supported the first and the last, but there was never a strong reason to differentiate them considerably as we only ran on a 32-bit system.

With respect to the v1.x GAC (v1.0 and Everett share the same GAC), you can see pretty clearly that processor architecture was not a design point.

V1.x GAC location
C:\Windows\assembly\GAC

V1.x System.dll location
C:\WINDOWS\assembly\GAC\System\1.0.5000.0__b77a5c561934e089\System.dll

Where would you have tagged the processor architecture? Nowhere that I can see. As a result, we changed the GAC structure in Whidbey to accommodate processor architecture, which is now an incredible important aspect of the GAC.

V2.0 GAC locations on an x86 machine
C:\Windows\assembly\GAC_MSIL
C:\Windows\assembly\GAC_32

V2.0 System.dll and mscorlib.dll locations on an x86 machine
C:\WINDOWS\assembly\GAC_MSIL\System\2.0.0.0__b77a5c561934e089\System.dll
C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll

V2.0 GAC locations on an x64 or IA64 machine
C:\Windows\assembly\GAC_MSIL
C:\Windows\assembly\GAC_32
C:\Windows\assembly\GAC_64

V2.0 System.dll and mscorlib.dll locations on an x64 machine
C:\WINDOWS\assembly\GAC_MSIL\System\2.0.0.0__b77a5c561934e089\System.dll
C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll
C:\WINDOWS\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll

You can now see that it is easy to tag assemblies with processor architecture. We’ve created three GACs to do just that! As you can see, the GAC_64 directory only shows up on 64-bit machines, which is hopefully quite intuitive.

I’ve run out of time to get any further in this post, but hopefully it gives you a good idea of how and why the GAC changed, at least with respect to processor architecture in Whidbey. I also hope I left you with the opinion that these changes were a good idea ;)

A question to ponder: If you call System.Reflection.Assembly.Load(“System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089”) and there is both a copy in the bit-specific GAC and another in the bit-agnostic (MSIL) GAC, which do you get?

I was recently reading the "remarks" section for the AppDomain.Unload() method @ http://msdn.microsoft.com/library/en-us/cpref/html/frlrfSystemAppDomainClassUnloadTopic.asp. I was doing a review of the documenation available to determine if we needed more around the CannotUnloadAppDomainException exception. I decided that we were in an OK spot for the most part, but that I would post a little more information on this exception for the small percentage of .NET programmers out there who need it. At least, I certainly hope that most .NET programmers don't need this information ;)

Note: Please mail me if this information proves particularly useful for you -- I'd like to better understand your scenario.

Upon calling the AppDomain.Unload() method, the CLR will request that all threads in the unloading domain abort. Threads do not abort instantly, so getting n threads to abort will take some time. As a result, the CLR checks after 10ms* to determine if there are any threads still running. If there are not any threads still running, then the CLR will proceed with domain shutdown. If there are threads still running, then the CLR repeats this process -- wait 10ms and check -- up to 1000* times. In the general case, it is very unlikely to need to repeat this process 1000 times (or anything close to it); eventually, all threads will be aborted and domain shutdown will occur. The purpose of this repeating "wait and see" behaviour is to avoid a deadlock due to a thread that will not abort for some reason.

If the 1000th check fails, meaning that threads are still running (not yet aborted), the CLR will throw a "CannotUnloadAppDomainException" exception. If the application, after the exception has been thrown, can do something to prevent this situation from occuring again (i.e. cleanup or manually aborting threads) then the application should perform that cleanup and call AppDomain.Unload() again. It is not expected, however, that the app should attempt to try repeat this process -- catching the "CannotUnloadAppdomainException" exception and calling AppDomain.Unload() -- an indeterminant number of times until it "works". The best approach is to remove the case where the exception is thrown at all, which means ensuring that all threads in a domain can be aborted in a timely manner. If the application, after the exception has been thrown, cannot do anything to prevent this situation from occuring again, it should not call AppDomain.Unload() again, but terminate the application or leak the domain.

* please consider these numbers (10ms and 1000 times) as implementation details. You absolutely should not take a dependency on them, for two reasons: (A) they are subject to change in a later release w/o notice, and (B) the number of running threads, processes and CPUs (+ multi-core) can very significantly change the time it takes to hit the 1000th iteration of the check due to the amount of CPU time that the unload check is given.

After logging in, be sure to visit all the options under Configuration in the Admin Menu Bar above. There are 26 themes to choose from, and you can also create your own.

Our Product Unit Manager is hiring! Jason's post is about interns, but he has a link to regular jobs too at the end of the post. The CLR team is the best team that I've worked for at Microsoft. If you want proof, read this blog entry.

We are currently closing up Whidbey and doing some initial planning for Orcas. By the time you get here, we'll likely be prototyping ideas for Orcas. In fact, I have an executive review in just over a week with Jim Allchin on the managed code versioning work we are planning for Orcas. We'll see what he thinks. If you join the team, you'll find out ;)