Wednesday, August 6, 2014

Issues transitioning ASP.NET website to Windows Server 2012 R2

Background :


Approx. 9 year old ASP.NET website.

Originally ASP.NET 2.0 (or maybe even 1.1?), and upgraded over time to now being ASP.NET 4.0.

Running fine on a Windows Server (2003?).

We decide to virtualize our servers.  The new virtual server is Windows Server 2012 R2.

I copy the ASP.NET files across and lose DAYS trying to get it working.

It was an unexpectedly painful process, with lots of obtuse error messages.

Here are a few of the "gotchas" that got me - and hopefully will help un-got you :

Bye bye DAO & 32-bit mode


Due to its age, the system was using DAO against a Microsoft Access database for part of the system (including integration with a third-party system).

DAO only has a 32-bit version.

For no apparent reason - and I've never had this problem on any other server - but when I put the application pool in 32-bit mode, I would get extremely strange errors I could not resolve.

How did I solve it?

Switch back to 64-bit mode, and modify the database connection string to use Microsoft's "ACE" library instead of DAO.  (Scroll halfway down this article for example connection strings.)

This also requires installing ACE on the server - not installed by default.  (Go here and download AccessDatabaseEngine_x64.exe)

No HTTP Handler For The Request Type 'GET'


This is up there with the weirdest problems I've seen.

Turns out - for no apparent reason - there can somehow end up being a deficiency in the configuration of ASP.NET.  Copying web.config.default over web.config in the .NET Framework config folder magically solved the problem - thanks to these guys for saving me a huge bunch of frustration and time wastage!

Unhandled exceptions only intermittently going in to the Event Log


I wasted a lot of time trying to find details of error messages at the very start of my testing.

The web page returned to me would say that an unhandled exception had occurred and an unhandled exception had occurred in the error handler.

Not very useful, but exception details would sometimes appear in the Event Log.

Here's the big problem : it was highly unreliable.

Sometimes the .NET exception dump would appear in an Event Viewer entry pretty quickly.

Other times it seemed to take minutes.

And other times the error messages never appeared in the log at all.

I could not determine the cause of the inconsistency.

Fortunately, at some point I sufficiently adjusted file permissions to enable my web application's error handler to write its own exception dumps to text files in a log folder, and then with the detailed error messages available immediately after each page hit, I was able to solve the problems much faster.

(The error handler was quite complex and was still triggering its own unhandled exception, but after dumping the primary exception's details to text file.)

SMTP services installation was not obvious to me


I found lots of articles explaining that to use the IIS SMTP server with IIS 7.5, you need to install various things.  I started using IIS so long ago and have done sufficiently little server administration that somewhere along the line I failed to notice that with the design of Windows Server these days I had to add the SMTP server as a feature not as a role.  Why IIS is a role whilst SMTP is not, I don't know and don't intend to bother investigating, but that explains why after quite some minutes poring carefully over the list of installable items, I never saw SMTP.  I had to click "Next" to go from the Roll selection page to the Feature selection page.  That one was kinda-obvious, but it tripped me up, so I mention it in case it helps anyone else.

And I had quite a lot of other pain during the transition process - a process that I had naively imagined in advance might only take an hour or two.  But those are the main ones that stick in my mind.

HTH!

Sunday, July 20, 2014

Using vmrun on Mac OS/X (VMware Fusion)

The skinny :

To use VMware Fusion's vmrun from the command line (i.e. Terminal), you must invoke it with its FULL PATH, even if you are already in the same directory as vmrun lives in.

The fat :

Continuing to prove that VMware is great when it works and quite lousy when it doesn't, get this :

I opened Terminal.

I cd'ed to /Applications/VMware\ Fusion.app\Contents\Library.

I tried to execute "vmrun".

"Command not found"

How can that be?!!  The command is definitely in that directory!

ls -l

Yup - there it is, vmrun, and all users have execute permission.

I lost probably half an hour on this stupid bug.

Turns out that the solution is simple, but horribly inobvious :

You must include the full path to vmrun every time you invoke it, even if you are already in the same directory as it!!!

So instead of bothering to cd to the enclosing directoy, just always use :

/Applications/VMware\ Fusion.app/Contents/Library/vmrun

Note : I tried adding the directory to my PATH to see if that way I could run vmrun just with "vmrun", but it didn't work.

Thursday, July 17, 2014

VMware Fusion Converter VSS issues

VMware is simultaneously both very nice and very frustrating.

I have a 4.5 year old laptop (Sony Vaio Z) running Windows 7 Ultimate, and decided to virtualize it.  Great idea!

VMware Fusion comes with a physical-to-virtual converter.  Piece of cake!

But every time I try, I get the big bad message :

"An Error Occurred

"The VSS snapshots cannot be stored because there is not enough space on the source volumes or because the source machine does not have any NTFS volumes.  Error code: 2147754783 (0x8004231F)."

Thanks VMware!

So here are some of the things I tried that have helped not a bit :
  • I spent a long time clearing stuff off the 147GB C: drive until there was over 10% of the drive free.  Still kept complaining.
  • I disabled my R: drive RAM disk.  It was an NTFS drive, but just in case somehow it was causing trouble, I completely turned off the RAM drive system.  (Note that I had P2V'ed a Windows 8.1 machine with a RAM disk successfully, using the same tool, so this never seemed likely to be the problem.)
  • I suspended BitLocker in case that was somehow interfering (although note that I had P2V'ed a Windows 8.1 machine with BitLocker successfully, using the same tool, so this also seemed unlikely to cause the problem).
  • I adjusted a MaxTokenSize registry setting.
  • I completely disabled "previous versions".
  • I fully enabled "previous versions" (both for my files and for operating system files - i.e. the highest level) and allocated it 10% of the available space.
  • I deleted all existing previous versions.  If you're doing the maths, this means there are roughly 14GB available and allocated to VSS.  So space shortage is not the problem here!
  • I confirmed that there are no other mounted drive letters - I removed all USB sticks and external drives.
  • I confirmed that there are no unmounted volumes on the one internal drive, other than a 15MB one that appears to be a boot volume and is nearly entirely full but I doubt there's anything much I can do about it.
  • I created a new administrative user and tried using that account instead of my normal administrative account when doing the physical-to-virtual conversion.
  • I did a full checkdisk scan on a reboot - i.e. whilst the C: drive was not in use.
  • I rebooted the Mac OS/X computer that was on the receiving end of the process.
  • And of course, throughout the above I rebooted the Windows laptop many times over.
  • I discovered a "bakk" (yes - double 'k') entry in HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\ProfileList as discussed here, deleted it and rebooted, and same problem (although strangely & interestingly, laptop shutdown and boot times were much improved, or so it seemed to my subjective impression!).
  • I also verified that the other SIDs in that registry key were all valid (using psgetsid from PSTools as described here).

The VSS service was creating error log entries claiming that there is an "Unexpected error calling routine ConvertStringSidToSid" (0x80070539).  This is what led me to wonder if checkdisk might be needed - but that didn't help - and is also what led me to the MaxTokenSize trick linked above, which also didn't help.  However, after deleting that "*.bakk" registry entry, the VSS service has produced no more error messages - yet the error message from the VMware Converter remains the same, blaming VSS.

It's not inspiring that multiple other people online seem to have given up on this one.  (e.g.)

I also report with disappointment that this was not my first unfavourable experience with this supposedly easy physical-to-virtual conversion tool.  About a week ago I converted a Surface Pro 2 (Windows 8.1) physical to virtual prior to sending it in for servicing.  I also had a lot of trouble then, although finally managed to get it to work.  Two bad experiences out of two attempts.  Relatively useless documentation.  Sadly, this is my experience of VMware so far : It is awesome when it works, and an extreme pain when it doesn't, which is far too often.

VMware Fusion PC Migration Assistant requires HFS

The skinny :

The VMware Fusion PC Migration Assistant must have an HFS-formatted drive as the destination for the physical-to-virtual conversion.

The fat :

Thanks VMware for the helpful error messages - not!

Here's the error message I got trying to convert a four-and-a-half-year-old laptop to a virtual machine :

"VMware Fusion was unable to share a folder to receive your migrated pc"

Hmmm.

Retry.

Same problem.

Google.

An obtuse comment, which leads me to suspect... voila!

I was trying to convert the physical machine into a virtual machine on an exFAT drive.

Try again, converting it to an HFS drive for later copying to the exFAT destination - problem solved!

Friday, July 11, 2014

VMware Fusion on exFAT

The skinny :

VMware Fusion wastes 30 seconds of your life accomplishing apparently absolutely nothing every time you start or resume-from-disk a virtual machine on an exFAT volume.

There appears to be no workaround.

The fat :

I love virtualization.  And the more I use it, the more I love it.

It has come a long way.

I remember 12 years ago giving up in frustration when I tried to virtualize my entire software development environment.

The big issue back then was the storage system.

Now with so many so-awesome SSDs to choose from, virtualization is a pleasure.

I'm running my virtualized software development environment with I'd guess about a 10-20% performance penalty, vs the literally 200-2000% performance penalty I was experiencing 12 years ago.

And since computers are generally so fast these days, 10-20% performance penalty largely translates into no significant difference at all.

Mac & PC

I happen to be a dual-mode man.  Surface Pro 2 256GB (love it!), and Mac Mini quad-core i7 16GB RAM.  I mainly do Windows stuff, and prefer the more extensive range of keyboard shortcuts in Windows for maximum productivity (whereas with Mac you are forced to keep moving your hand back to the mouse/trackpad whether you like it or not).  But I'm very comfortable and proficient in both environments.

I wonder, could I...  Get an external SSD and put my virtual machines on it.  Then, using VMware Fusion on the Mac and VMware Player or Workstation in Windows, I can run my virtual machines on the faster more powerful Mac Mini when inside, whilst still being able to take my work on the road and run exactly the same VMs on the Surface Pro 2.

Great idea!

But so many snags.

First off, you are going to encrypt that external SSD, right?  I mean, nobody puts sensitive data on an unencrypted external storage device I hope?  (Other than government departments of course - but who expects competence from them?)

Problemo : Windows has BitLocker, and Mac has FileVault, and ne'er the twain shall meet!

And whilst you can get cool third-party utilities to read+write Mac disk format from Windows, and likewise to read Windows disk format from Mac, none of these utilities support encrypted volumes.  Major problem!  Looks like we're scuttled right at the git go!

Well, there is TrueCrypt of course.  It's discontinued, there are question marks over its actual security level, and it's open-source, which leaves one wondering whether it might just go ahead and destroy all your data mysteriously and irrecoverably due to some strange previously-unencountered bug (or worse yet a known bug that has been languishing in the support queue for years, as happens sometimes with open-source and even commercial products).

But, TrueCrypt appears to be the only strong contender.  After all, we need something cross-platform, and that alone rules out a bunch of options.  And whilst performance and reliability have to be assessed, we do know that TrueCrypt has zillions of users, so we take the punt that it'll do the job well - of course making regular VM backups just in case.

Next snag : Filesystem?  I end up opting for exFAT, because it is natively supported in read+write mode by both Windows and OS/X.

Score!  My VMs run fast, and yes, I can transfer the external SSD back & forth between Mac Mini and Surface Pro 2 and it all works!

At this point, I'm super-excited.

However, I'm bothered by something.

It seems that every time I build a VM on the Mac Mini, then run it on the SP2, then take it back to the Mac Mini, there is a strange 30 second delay at the very start of booting the VM.

I Google - no answers.

It's been frustrating me around a month now, but I finally found the answer.

It has nothing directly to do with whether the VM is in the "Shared Virtual Machines" vs "Virtual Machines" folder, and nothing directly to do with running the VM on the Windows host.

It seems to be wholly & solely the VM being on an exFAT volume.

I can take the VM that has the 30 second delay at the very start of the boot process, copy it to a native Mac partition, and it boots immediately without the 30 second delay.

I tried some configuration tweaks that seemed unlikely to help, and indeed they didn't help.

My conclusion?  Lovely conceptually as it is to be able to share VMs twixt Mac & Windows via this external SSD, it's proving all told a little on the painful side.  Not hugely painful, but having to connect the SSD, run TrueCrypt, mount the TrueCrypt volume, use VMs, unmount the TrueCrypt volume, unmount the SSD - that's a little tedious - and now add that there is an absolute waste of 30 seconds of your life at the very outset of every VM boot (and I tend to be starting & stopping VMs a lot throughout the day), and it gets a little frustrating.

The VMs once running run just fine, so it's clearly something VMware could fix.

But I found no-one else anywhere else mentioning the same problem, so I doubt it's even on VMware's radar.

I'll stick with the system for now - it does work - but I'm thinking of changing to a setup where both the Mac and the SP2 have a full copy of all VMs, on their native filesystems with their native full-disk-encryption technologies, and using any of the zillions of backup / file-replication utilities out there so that when I run the VM on one and shut it down, any changes get copied across to the other copy of the same VM.  If that works, then I'll have a truly blissful and hassle-free experience of using the same VMs on two different machines, one being an OS/X host and the other being a Windows host.

The big problem then will simply be storage space disparity.  I might end up paying the small fortune to get a 512GB Surface Pro 2 or 3, just so I can fit everything.  Or else I might use an external SSD just for the less-frequently-used VMs.  Or I might augment the SP2's storage with an external SSD encrypted with BitLocker and formatted with NTFS and used only by the SP2.

The short of it?  VMware Fusion runs VMs off exFAT partitions just fine, but for no apparent reason will waste 30 seconds of your life every time you boot a VM, at the very start of the boot process, before even the VMware Fusion logo pops up on the VM screen.  Please fix it, VMware!

P.S. The problem does not occur with VMware Workstation / Player - i.e. on the Windows host, the 30 second boot delay does not occur.  It is only a problem with VMware Fusion.

P.P.S. I'm using VMware Fusion 6 Professional (paid) and VMware Workstation 10 (trial), about to change to the latter being VMware Player Plus (using the VMware Player Plus license that comes free with VMware Fusion 6 Professional).

Saturday, April 26, 2014

Surface Pro 2 "System Interrupts" high CPU

The Surface Pro 2 is excellent hardware, but the software lets it down in various ways.

All told, it's still the best Windows machine I've ever owned, but it definitely needs a lot more work.

Here's a weird thing I hit today : I was losing around 15%-25% CPU on "System" and "System Interrupts".

I had installed c. 1.5GB of updates in the last several days - maybe that caused it?

Hmmm - there was one other thing I changed, this morning.  Maybe that caused it?

I changed it back - and the problem went away!

So : if your Surface Pro 2 runs fine after a reboot, but is wasting a lot of CPU (and hence slashing battery runtime to just an hour or two) with "System" and "System Interrupts" after putting the machine to sleep and then waking it up, there is a chance it might have something to do with this :

This morning, I went into Device Manager and disabled "Allow this device to wake the computer" in the "Power Management" tab in the Device Properties for each of the two "Mice and other pointing devices".  (I suspect one was the touchscreen and the other the Type Cover's trackpad, but they both simply showed as "HID-compliant mouse".)

I had already - weeks or months prior - done the same for the Keyboards, but that had seemed to have no effect at the time.

What I was trying to accomplish was have it that any accidental keypresses or mouse movements would not wake the device from sleep.  I wanted to know that only pressing the power button, or perhaps the Windows button, would wake it.

I at last had that behavior!  When I had only disabled wake-the-computer for the keyboards but not the mice, the keyboard would still wake the device.  I guess that because the Type Cover has both keyboard and mouse together, I must've needed to disable the wake-the-device option for both or else power would remain to both and both would remain able to wake the device.

So this morning I was very happy for a short while, because no action on the Type Cover would wake the device.  Just what I wanted!  But I quickly discovered serious side-effects.  nircmd no longer worked to turn the screen off.  And this "System" and "System Interrupts" high CPU usage bug!!!

I lost a few hours trying to get things working, and finally realized the "System" and "System Interrupts" high CPU usage might've been a strange side-effect from disabling the allow-to-wake-computer options for those two mice.  I reverted that setting, and voila - nircmd resumed working and the "System" and "System Interrupts" problem went away without even a reboot!

Don't ask me how to fix the problem in your case, and don't ask me why changing that particular setting would have such seemingly unrelated side-effects, but if you're desperately searching for clues, my case adds a few more.  I hope it helps someone!

Monday, October 14, 2013

phoneSWIPE, PanopticSecurity, myPCI, PCI compliance

phoneSWIPE is cool, so you get one.

Then you get this email from "PanopticSecurity" on behalf of "North American Bancard", telling you that you need to complete a PCI compliance questionnaire.

The very first page of the questionnaire asks you what kind of credit card processing system you're using.  At the moment, phoneSWIPE is not even listed on the page.  Anywhere.

So you take a stab at it, and end up going through a tedious and time-wasting questionnaire with loads of questions you're pretty sure don't even apply.

Finally at the end, PantopticSecurity helpfully tells you that you require quarterly external "network scans" which they can oh so helpfully do for just $60 a year.  But if you did your homework before you joined phoneSWIPE, you knew that phoneSWIPE encrypts the cardholder data on the swiper and you shouldn't need network scans.

I had a frustrating time trying to complete the questionnaire, and a frustrating time with their text chat staff, who initially sent me down the wrong track.

What they told me in the end is that if you use phoneSWIPE on a cell phone running over 3g or 4g (i.e. not running via ADSL, cable internet, or some other wired or wireless shared internet connection), then at the very start of the process you need to say you have a dial-up credit card processing system.  Dial-up.  Yeah, like an Android 4.2 phone with a sophisticated phoneSWIPE audio port dongle with built-in encryption is dial-up.  But that's what they told me.

They also told me that if you dare actually ever use your office wifi for credit card processing, you do have to go through the ridiculously long questionnaire with a zillion questions that seem entirely irrelevant and which ends with them absolutely insisting you must spend the $60 annually for external network monitoring.  I suspect it's a rip-off, but I'll end my investigations here since at least I've determined that rip-off is inapplicable in my case.

Chat transcript, to shame Panoptic Security for making such a confusing system.  (I'll give them points for a) the politeness of the text chat operator, even if they at least twice were clearly not paying attention to what I said; and b) for a few things in their questionnaire that make it clear they're trying to make it less onerous ------- but unfortunately they need to do a WHOLE LOT BETTER at least when it comes to phoneSWIPE customers.)

[Natasha] Hi, my name is Natasha on the PCI DSS Program Help Desk. How can I help you today?

[Me] This is an extremely confusing questionnaire. We have a phoneSWIPE device, but it is not in the list of options to choose from at the outset of the questionnaire.
[Me] So I tried to answer the questions as best I could, and now I'm being asked a gadzillion questions that I'm pretty sure are irrelevant.

[Natasha] is your phone swipe connected to your wifi or 4g network?

[Me] 3g Verizon network
[Me] For example, I've just come to this one : "Since your payment applications do handle sensitive authentication data, are processes in place to securely delete the data and to verify that the data is unrecoverable?" Um, no actually, I was told that the cardholder data is encrypted within the phoneSWIPE device, meaning that there is no way anything on my Android phone could ever decrypt it. It goes encrypted from the phoneSWIPE device over to the back-end processing systems. So why is this question assuming that my "payment applications DO handle sensitive authentication data"?

[Natasha] Is that the phone internet?

[Me] 3g Verizon network is the internet the phone uses, if that's what you're asking
[Me] We also have a wifi network that we leave off all the time, except that if Verizon coverage is not good enough, we turn it on just for the phone. There are no other devices on that wireless network.
[Me] It's extremely confusing trying to answer the questions in this questionnaire - they seem to presuppose a setup entirely different from our own.

[Natasha] As you use the Verizon you need to complete the questionnaire that has been assigned to you today.  If you find the question difficult you can call our help desk we do have agents that are here to assist you with completing the process today

[Me] So when I come across questions that presuppose things that simply aren't true, am I supposed to just flip a coin which way I answer?
[Me] Also, is there any reason why phoneSWIPE wasn't listed on the very first screen, where you select the technology you're using? phoneSWIPE wants us to believe they're big and important - but their non-existence on that list implies YOU think they're basically irrelevant. Are they really that small a player? If so, I should probably ditch them. If not, then list them on the main page so people like me can have more confidence the questionnaire is appropriately tailored to our situation.

[Natasha] You can answer yes to them questions.
[Natasha] They are in the process of updating that onto the portal but as for at the moment if your phone swipe is connected to your wireless then you need to click on standard internet terminal and if it is not the you need to click on standard dial up terminal .

[Me] Thanks for your comments. I'll keep muddling through the questionnaire. Have a great day!

[Natasha] Your welcome, same to you.

(long time passes)

[Me] If you happen to still be there, I've gotten to the end of the process and it's told me I "need" a network scan, costing $60 a year. I believe this conclusion to be false.

[Natasha] If you are connected to the internet then yes you will need to run a scan of the yearly cost of $59.

[Me] If I understand it correctly, phoneSWIPE represented that since the cardholder data is encrypted on the swiper and only passes through the computing device encrypted, and is only able to be decrypted on the phoneSWIPE servers which do pass PCI compliance, that there is no way that cardholder data could be compromoised on the computing device (Android phone in this case).
[Me] Further, any IP address the phone is assigned by Verizon changes without notice.

[Natasha] I understand that but as you are connected to your wireless or wifi in the business we need to scan your router.

[Me] Hmmm, but again here I'm not convinced myPCI's questionnaire anticipates our environment. The wifi is OFF ALL OF THE TIME. We turn it on when our device needs internet access _and doesn't have Verizon coverage_. The thing is, we only do credit card swiping in locations we have Verizon coverage. So the wifi is NEVER on when processing credit cards.
[Me] Also, suppose we wanted this "service", what IP address are we supposed to put in the Panoptic sign-up form, given that the we have a dynamic IP address?

[Natasha] Which is why you need to run a scan, I understand it is only turned on when you need to process a payments but you still turn it on and use it in your business for when you process a payments.  We can the firewall built within your router to make sure the firewall has no vulnerabilities in it.  So we need to make sure no one can gain access to the card holder data from the net and no information can be leaked out.

[Me] You misread - it is NEVER turned on when we need to process a payment.
[Me] We only process payments in locations we have Verizon coverage, and in such cases we do not need wifi and our wifi is OFF.

[Natasha] Every internet has a public IP address.  If you Google whatsmyip.org it will provide you with that information you require.
[Natasha] So you only use you phone internet not the business one?

[Me] For credit card processing, we NEVER use the wifi/broadband. We ALWAYS use the Verizon 3g.
[Me] (When we are selling goods & services we are out & about and in Verizon coverage areas.)

[Natasha] In the pre saq you need to click on standard dial up terminal and you will be assigned the correct questionnaire and no scan will be required form you.

[Me] So I just wasted half an hour after you told me "As you use the Verizon you need to complete the questionnaire that has been assigned to you today." I told you the questionnaire seemed inapplicable, and I told you that I use Verizon, and you told me that I had the right questionnaire. Now you tell me otherwise.
[Me] I'd also like to know how any phoneSWIPE user is supposed to figure out that their high-tech card reader plug-in for their Android cell phone is actually a "dial-up terminal". You tell me they're updating the front menu to make that more obvious. I hope they get that updated finished real soon, 'coz otherwise they're wasting a lot of time for a lot of people like me, who end up wasting a lot of time from people like you.

[Natasha] I am sorry you confused me by talking about your wifi in your business.

(Editors' note : I only mentioned wifi because the questionnaire was asking about it.  I knew it was irrelevant!!!)

[Natasha] As we are not based in America I am not familiar with the networks

[Me] Ah - that makes sense.
[Me] ok, thanks for the pointer to SAQ B and hopefully it'll be smooth sailing from here!

[Natasha] I do apologies. It is a less complex questionnaire and you should have no problem with completing the process form here.

[Me] Done - thanks again & have a great day. Over & out.