I have spend the best part of a day on this problem, and have not seen anyone blogging about it yet. I am really surprised this hasn’t surfaced in the community yet. So here it is…
I am always trying to work on an x64 platform if possible. Just to see what kind of problems comes up so that I am prepared to deal with the issues in the wild. This is one of those issues.
My test setup
I decided to install a VMWare based SQL 2005 cluster with a couple virtual BizTalk servers on top, to try out a few things on a multiserver installation. All OS’es are Windows Server 2008 Enterprise Edition (Not R2). It payed off as I eventually came up with a couple things that was not covered fully in the BTS 2009 documentation (I’ll document that later – nothing big) and then this issue.
The problem
When I came to the point where I should install and configure the BAM Portal, I received this error:
-----------
<Exception Message="Error configuring BAM Query Web Service user testad\BAM Management Web S and BAM Application Pool user testad\BAM Application Pool" Source="Microsoft.BizTalk.Bam.CfgExtHelper.PortalHelper" HelpID="error_ConfigureUsers">
<Exception Message="This access control list is not in canonical form and therefore cannot be modified." Source="mscorlib" HelpID=""/>
-----------
The users for the BAM Portal App Pool, and the BAM Webservice DB access were named as in the installation guide, so “BAM Management Web Service User” became “BAM Management Web S” and “BAM Application Pool Account” became “BAM Application Pool”.
I looked through the BizTalk installation error log, but could not figure out what exact registry key or folder, had its ACL messed up. Everything seemed to be ok as far as I could tell. A little earlier than the exception, the log also mentioned it was running something called bts_setreg.exe that I has never stumbled across before.
When you run this in a command prompt, it looks like this:

Looks like just another version of the aspnet_setreg.exe tool but what is this baby doing here under another name?? Hmm…
I though … what the heck … lets configure this thing manually, so I started poking around the web.config templates thats installed out of the box.
<identity impersonate="true"
userName="registry:HKLM\SOFTWARE\Microsoft\BizTalk Server\3.0\BAM\WebServices\identity\ASPNET_SETREG,userName"
password="registry:HKLM\SOFTWARE\Microsoft\BizTalk Server\3.0\BAM\WebServices\identity\ASPNET_SETREG,password" />
They contained the two lines that was somehow linked to what the bts_setreg tool was doing, but I couldn’t find those in the registry under hklm/software/Microsoft BizTalk. That didnt make much sense to me.
Anyway … in the meantime I stumbled across this great post by Derek Tan: http://blogs.msdn.com/derektan/archive/2006/01/06/510199.aspx Highly recommendable. You get a good feeling for the BAM Portal IS after doing that.
And halfway through the manual install process, Derek asks you to use ASPNET_SETREG.EXE to set the password used by the BAM Webservice account. The job of this account is to give access to the BAMPrimaryImport database on behalf of the AppPool account.
The solution
So I tried to do that manually, but nothing seemed to be created in the registry db. But the tool didn’t fail either. I ended up doing a search for the ASPNET_SETREG key, and it turned out that it had been created under the HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node key.
Then it dawned on me … off course … the ASPNET_SETREG.EXE is dated back to .net 1.0 or 1.1, so it has been compiled to only run as a 32bit process. And that explains why it lands in the 32 bit node of an 64 bit OS …
So … I ended up solving the problem by just moving the registry keys to the 64 bit location, but I could probably just have edited the web.config as well.
Afterthougths
So … why haven’t anyone else seen this? And why haven’t I seen it on my 2009 single server or development boxes??? Well .. the truth is that on all development boxes, the account for the app pool and the impersonation account have been one and the same. So during installation, the webservices have had the proper access to the BAM Primary import database. It’s only if you do it RIGHT and create two different accounts, that you run into this problem.
Anyway … thats my theory. And if that is right, there must be a LOT of people out there simplyfying the number of user accounts used when installing BizTalk …
My next thought is to change the Registry URL’s on the BAM webservices, before configuring the BAM Portal. That ought ensure that impersonation works properly during the configuration process. I’ll try that on my next x64 multiserver installation. If you have tried it and it works, post it here or email me and let me know.