Some time back, I wrote a post on file system filter drivers and load altitude which included discussion of  kernel mode filtering and how "filter altitude" affects interaction of App Streaming file system filtering and anti-virus.  Today's post discusses application level hooking of registry operations and how this can cause collisions between application level entities, specifically, Citrix App Streaming with AppSense Environment Manager (EM). 

Application Streaming implements the layers of glass for registry operations at "application layer".  AppSense EM implements its registry filtering also at "application layer".  With the release of Citrix Application Streaming 5.2 in XenApp 5.0 Feature Pack 2, we cratered them; returning the isolated view of the registry to the application even if AppSense EM were trying to provide a view from their cached database backend.

Since then, AppSense have released version 8.x of EM which works with Citrix App Streaming "offline plugin" version 5.2.  The supporting version of EM is available now as a download from MyAppsense.com.  With the EM 8.x release, things are again working and more than working, it's worth noting that AppSense have done extra work to ensure that EM works effectively with Citrix App Streaming based applications just as it does with locally installed applications. This works XenApp server side, end user client side as well as on XenDesktop hosted desktops.

Environment Manager introduction

AppSense Environment Manager is an application set which ensures that users receive a consistent view of their desktop and application settings regardless to the computer that they use to logon or how the desktop applications are delivered.  Similar to Microsoft Roaming Profiles and Citrix Profile Manager, EM gathers configuration information from applications and retains that information across logons, sessions and across machines where it can later be provided back to the application as if it came from the same machine where it was stored.  From a technology view, EM uses a database back-end and local machine cache to store information and uses application level "hooking" to provide configuration information to the applications on-demand at runtime.  For example, if an application stores a setting at a specific place in the registry, that operation is backed up by the database rather than the local registry and the EM hook code answers the question bypassing the registry.

The interesting part is that since the database is used for central storage, this can occur with or without actually roaming the user's profile from session to session.  The backing store for the information is the database and this remains true no matter what happens on HKCU registry or file system.

In the collision case, App Streaming returned the isolated registry view of the setting, which could and often would be different than the "correct" setting from the AppSense EM central store.

How hooking works
Back in the beginning, there was DOS...

DOS was ("is"?) a single user, single tasking operating system.  In theory there could be only one application running at a time.  Clever programmers quickly figured out that they could simulate a multi-processing machine by hooking the execution of the single application.  Borland's SideKick was the first commercial success here.  It ran at at machine startup (autoexec.bat), then went away and hung out in the background until the user typed a special key, which was pressing ctrl-alt at the same time.  On seeing the hot-key, the program would come to foreground and take over the machine screen and keyboard.  When done, execution would return to the application that was interrupted as if nothing had happened.

Two things to observe

1. The single tasking OS was now running more than one app at the same time
2. Citrix XenCenter has a hot-key collision with a 27 year old application.

How did SideKick accomplish this?  Answer, Terminate and Stay Resident API into DOS to leave a portion of application in memory after the application had officially terminated.  The resident portion was "HOOKED" into the BIOS IRQ-9 keyboard hardware interrupt processing so that when both Control and Alt were pressed, the program would save away the screen, bring itself to foreground, do it's work and eventually put things back.  Impressive stuff for the day and the TSR was here to stay.

The concepts of hooking application activity have not changed significantly since the TSR days of DOS.  The Windows operating system and Windows based applications have a set of dispatch points and these dispatch points implement program or Win32 SDK functions.  To "hook" a function, the function dispatch points get "messed with" so that the hook get called instead of what the system/application really wants to call and then the hook has a chance to do things early and make intelligent decisions on whether or not to call the "real" software.  The concept is "unchanged" in 27 years and if someone is older than me, this probably goes back even further.

In the days of DOS, everyone soon wanted to climb onto interrupt 9 to watch the keyboard button press and release messages and pretty soon, dependencies arose on the "order" of placement onto the TSR stack.  Hook collisions were born!

Everyone wants to be on top

View the hook code as a stack of hook modules.  The application calls what it believes to be the registry API, but which in reality is the hook code of the first hooker.  That hooker does it's work and calls what it believes to be the real system API, but which is in reality the next hooker in the chain.  Add more hook modules and the stack gets higher, more layers between the application and the real implementation of the registry API in the Win32 system.

The "top" hook module is the first one to see the API call.  In the DOS days, the "last" TSR to get installed was generally the first code to get called in the hook.   Today, that's unchanged, the last hook module to hook is usually the FIRST entry called when the hooked API is called from the application.

If you'll recall from my post on FSFD and altitude, "everyone overrates their own importance".   Everyone is worried about their own hooking and everyone wants it to be successful in their own testing, but when you step back and look at a full system and then ask, "who should be at the top of the hooking stack", the answer is that it usually shouldn't be you.  The challenge is that when placing yourself onto the API hook stack, you can't really see the hookers below, they are indistinguishable from the real API.  You have little choice but to place yourself at the top of the stack and this means that you will get called before the people lower on the stack.

Application Streaming absolutely MUST be able to hook the registry to accomplish Application Virualization and AppSense Environment Manager absolutely MUST hook the registry to accomplish it's mission of replacing the "registry" with a local cache and database back end.  Who should be "higher".  Answer: AppSense.  Whether the application is run isolated or locally installed, the application settings must come from EM and getting the hookers out of order can prevent this from occurring.

With Application level hooking, altitude discussions are harder than with kernel mode, because there is no easy way for a programmer to control altitude for application level hooks. In the kernel, you tell the filter manager, "put me here".  At application level, chaos is supreme.

Where it broke

When an isolated application calls RegQueryValueEx, the registry API is hooked by both Citrix App Streaming and AppSense Environment Manager.  Being on the top, the Citrix code gets first look.  For a registry key that is isolated, the registry query is converted from it's before isolation location to its sandboxed location according to the layers of glass.  With the location converted, the operation is sent "down the stack" to have the Windows registry system fetch the value from the isolated registry.  To shorten the story, in the both case of App Streaming + Environment Manager, for isolated spaces, Citrix 5.2 streaming client would BYPASS the AppSense hook, preventing it from doing it's job of looking up the value in the central database.

BUSTED - And no easy fix.

With Environment Manager 8.x,  AppSense have changed their hooking code to get themselves in "higher" in the registry stack than is Citrix App Streaming.  The end result is that EM starts working again, even for streamed applications.  To get here, AppSense had to be aware of Citrix App Streaming in their code and had to take programmatic action to work past some roadblocks that we put in their way. 

What it comes down to is that AppSense have hooked the Citrix Hook code code.  Isolating the isolation system if you will, which is a pretty neat idea.

The present is a happy place

AppSense has Environment Manager working with Citrix App Streaming version 5.2.  

The future is daunting

Soon, a new release of Citrix App Streaming will ship with XenApp 6.0.  Working with AppSense, we are 100% positive that changes in App Streaming hooking implemented for 6.0 will interfere with Environment Manager and will require an update to EM to address.   I'll go on record that the fault for the break here is the Citrix end.  We changed the method of hooking and this has no choice but to break the EM hooking of the Citrix Hooks.

What did we change?  Well, AppInit_Dlls is no longer vogue, so it's time to move off.  Also, implementing isolation of NT services requires some creative programming for filtering the Windows Service Control Manager.  Add it all up and we changed the hooking from AppInit_Dlls style application hooking to kernel implemented hooking of application APIs. The result is that "everything is different" and AppSense gets broken again.

I suppose this is the nature of the business.  Citrix climbs in deep into the operating system to do neat things and becomes dependent on the OS innards for function.  Microsoft changes the innards, and AppInit_Dlls, and we have to change our software to work with it.  The same analogy applies in the Citrix - AppSense case.  They are doing neat things - but these things require climbing in deep, and when we change the deep things, they get impacted.  The good news is that early tech reviews allow them to know of the problem sooner rather than later and we have already had conversations to discuss how to best resolve.  This is a partner relationship and we're all motivated to make both successful.

So to my friends at AppSense.  Congratulations on your success with App Streaming 5.2 and ... my apologies in advance for breaking you on 6.0.

Joe Nord

Citrix Systems

Product Architect for Application Streaming and User Profile Manager

I'm debugging a good one lately, Discretionary Access Control List (DACL) programming with Citrix App Streaming.  Recall from a previous post that Citrix App Streaming dynamically adjusts user access rights for the execution cache as a function of running applications in sandbox and as a function of terminating application isolation spaces.  This is very cool stuff - but it is presently creating some headaches.  This post describes the headaches.

The quick review of the earlier post is that users on a XenApp Server are allowed to see the application installation content ONLY for the applications that they are presently running.  Applications that they are not running, they cannot see, even if those applications are presently running on that machine, right now, supporting other users.

From the layers of glass, I'm talking about the middle layer here.

 
The middle layer is SHARED across all of the isolation spaces.  If multiple users are on the machine, only one copy of the installation image exists for all.  This space is stored below \Program files\Citrix\RadeCache.  It SHOULD be below the streaming client subdirectory, but it isn't - I digress.

At installation, the streaming client installer sets permissions on the RadeCache space so that the Streaming Service user account (Ctx_StreamingSvc) has "full rights" and "users" have no rights, not even file scan.   At runtime, the streaming client grants the specific user Read+Execute rights to the GUID_V subdirectory that holds the execution content for this specific application.  This means that if a XenApp server has  100 users on it, but right now, you are the single user running "Toontown", then ONLY you will be able to see the Toontown bits.

It's even more restrictive than not being able to see the bits, you can't even see what folders exist below the RadeCache because you never have file scan for that space.  Interestingly, you CAN "CD" into the subdirectory if you know the GUID_V, though this works only while the application is "running".

Compare this to locally installed apps.  On a XenApp Server, a USER can "file open" and browse to \program files and then view all of the applications that are installed on that server.  For App Streaming delivered apps though, things are more locked down, the user cannot "see" application bits for applications that they themselves are not running.

This is very elegant.  Elegant can be problematic.  The user rights are granted using the Windows API SetNamedSecurityInfo.  This is the Windows API version of ICACLS.exe.  The headache starts in how this API implements ACL assignment.

Around 4 years into this App Streaming thing and all this DACL stuff has existed since the beginning - 4 years clean, I have seen it break down twice in the last 6 months.  

Problem #1: Customer has some really massive servers and when the individual user count hit 900, the DACL set failed and the Application Launch failed. Hum.  Why failed?  DACL size limited to roughly 64KB and the SIDs of all the authorized users started to add up.  Customer didn't have 900 users on the server, they had many hundreds, but also had a few "blowed up" sessions where the service didn't have a chance to remove rights so the DACLs built up - over months until hit a full DACL space.  The streaming client didn't notice the failure on the DACL set, but it did fail to CreateProcess the application, launch failed.  Solution: Purge the DACLs on regular basis.  This isn't today's conversation.

Problem #2:  Citrix internal IT folks implementing layers of cake; observing "slow" streamed application launch for the first App Launch, while 2nd time launch is fast as they would expect.  WHY is first slow?  It shouldn't be slow, the RadeCache is "mounted" into the space making all apps close to a 2nd time launch.  Sure the registry would need population, but that shouldn't be too bad.  Good news on that one by the way, on 5.2, the registry needs population.  On not yet released Mako, the registry load is a hive MOUNT which is much more efficient.

The suspect configuration has Citrix Provisioning Server, XenServer, XenDesktop, App Streaming and Microsoft Office 2007.  Variants of this profile have been used for stream to physical machines for 1000 users for a year or so now, including the stuff I'm using to write this post.  We know the profile is "good".

First time app launch slow

Launch debugger: Observe source code and binaries as it steps through, see a few neat things.

1) Streaming client is doing a DIR /S on the RadeCache for each sandbox create.   Wow!  That sucks and shouldn't be there, but it doesn't explain things being slow.  For those wanting to know more, it is this code that calculates the cache utilization and decides when to dispatch the cache reaper.  Ignore this - more digression.

2)  Streaming client is setting DACL to grant user access to the cache.  Wholly crap, how long did that take!

In WinDBG, you hit "F10", "F10", "F10" to step over code.  It usually takes about as long to get to the next line of code as it does to release the F10 key.  In the case of the API to set the DACL, the machine "froze".  I mean, NOTHING - for many seconds!  Wassup?

I left the room all happy with myself telling the IT guys that something is completely hosed in your enterprise disk stuff because that should be "instantaneous".  The DACL addition is a SINGLE DACL addition for a SINGLE user, for a SINGLE directory, there's no way that should have taken SECONDS.

Recall that the RadeCache space was already populated, the streaming system was merely granting a user access; but easier than that, it was only setting a single DACL on a GUID_V sub-directory, which would propagate into the Device\C\Program files\... spaces of the execution image.  SECONDS!  Your joking.

Being smart people, the IT folks didn't let it go.

One of my favorite books

One of my favorite books is "The Zen of Code Optimization", by Michael Abrash.  Yes, that's "Zen" with a 'Z'.  I haven't had the chance to meet Michael, but when I do, well - Beer will be on me, this dude knows his stuff.  He is behind a bunch of neat things like the graphics libraries for Doom and generally, he knows how to make a computer do things efficiently. 

One of the best pieces of the book is "Chapter 3 - Assume Nothing".  Yes, this is a whole chapter.  The gist of this is that just because you THINK it will be fast doesn't mean that it is fast, you must MEASURE IT.  The corollary is that just because you have 4 years of a product in the field saying that it is fast, doesn't mean that it really is.  In this case, I've paid the penalty for "The Costs of Ignorance".  This outlined in excruciating detail on page 27. 

DACLs and Inheritance  

The ICACLS command and the streaming code of reference have the same behavior.  It comes down to this:

• icacls GUID_V /grant domain\username:(CI)(RX)

In THEORY, this sets a SINGLE inherited right at the top of the execution cache to permit the named user the ability to read files in that space, directory scan and execute content.  This is SUPPOSED to be what happens.  In theory, any CreateFile access to a file/directory below that space will then be influenced by the inherited rights from the higher directory, where we set the DACL.  This is what inheritance means - the higher directories have it - and this means that the subdirs do too.  Start at the root and work your way down, and permissions follow!  SIMPLE!.   Four years of hindsight now says that this isn't actually what happens! 

I'm amazed - chapter 3 coming to bear.

What really happens 

First - I checked with a bunch of certified smart people and none could find a hole in the programming of the DACL set.   The code is "perfect" as coded - but it's slow...

Instead of doing what you tell it to do, the Windows API gets "elegant".  I didn't ask it to inspect the subfolders and files, but it does!  That is stuff that should exist in the Windows Explorer shell and maybe in icacls.exe, but it dag gone tootin shouldn't happen automatically in the API version!  I already KNOW that the DAClS for the sub-directories and files are good, don't help me out by fixing stuff that ain't broke!!!

Instead of adjusting the rights of a single subdirectory, the system RECURSES to all files and subdirectories.

In the ITDev case, the subdirectories of the indicated space contain the mounted, FULL installation image of MS Offfice 2007, over 7000 files!
On a local machine, as in true physical hardware with true physical disks, you hardly notice; it's "short" time.  On a Virtual Machine, ... it matters!  I mean, it doesn't matter for one file, but if you multiply by 7000, it adds up!

Recall that the execution machine is Provisioning Server booted, XenServer execution, some fancy disk system, everything is virtual to the end.  The files that "exist" as local in the RadeCache don't really exist, they are merely described to the virtual machine as local, indeed, there's no such thing as "local", ever for virtual machines.

For each of these files, the Windows API performed it's stuff, and recursed into all the subdirectories.  It read the DACLs for each file/directory, compared them to the DACLs for the directory that we were setting up via the commanded API, determined that "all was good" and then made no adjustment.  BUT - merely reading DACL information from the files, caused disk blocks to get "paged in" and CPU wise, this is painfully "slow", even for virtual CPUs!

For small apps, nobody will ever notice.  For a monster like 2GB MS Office 2007 at 7000 files, they notice!

Recall - this only happens on sandbox create and destroy, so it effects FIRST time app launch and not second time app launch.

Solution 

Step 1: Prevent the streaming service from adjusting rights to the GUID_V directories.  I did this with a code change, but the same thing can be simulated by removing a "right" from the streaming service user on install.  Not for the feint of heart. To know, the service will "push on" should the DACL adjustment "fail".  You can read that as "it doesn't check the return code".

Step 2: Grant "everyone" R+X access, (CI) for the whole RadeCache directory.  This allows users to see and execute the contents of the RadeCache - eliminating the security advantage of hiding this space, but it also allows users to RUN THINGSs if the DACLs aren't adjusted. 

Conclusion

Getting the layers of cake going on XenDesktop takes work.  It also takes experiment and "doing it" yourself to experience some of the problems that arrise at scale.  We're doing that work and doing it with some pretty massive applications. 

For small apps, you'll never notice that this delay exists, but for large apps, it's a noticeable delay.  We're attacking it.  Look for changes in this area in a Parra/Mako + 1 streaming client.  Notice I didn't say Mako.

The good news near term is that it only comes up on a FIRST time app launch.  Second app launches are "fast", like on real hardware.  The bad news is that on pooled desktops, the first app launch of each logon is a first time app launch.  Populating the RadeCache makes most of that time disappear, but parts still exist.  Sandbox reuse helps, but isn't the only part of this puzzle.

The other thing to observe is that some of the delay in first time app launch is loading the installation image registry content.  This is a registry load operation on 5.2, which was the test environment.  In Mako/Parra, if you load/save the profile targets, this will be a registry MOUNT - which is much more efficient - even more importantly, it doesn't involve registry "writes", which is a winner.

My compliments to the Citrix IT folks for not letting go.  

Fun for your experiments

Create a directory with no files.   Time this command

• icacls pathtodir /grant localmachinename\username:(CI)(RX)

It will be fast.
Add 7000 files to the directory by copying \program files.  Repeat the experiment.  Amazing results...

Offline, I have received some compliments for sharing some of the "faults" in the streaming system.  I hope this post contributes positively to your XenDesktop implementations and to your understanding of exactly how things are running.  I find it helps to really know what's going on. 

Joe Nord

Citrix Systems Product Architect - Application Streaming

As Architect of Application Streaming, one of the things I do along with the other Citrix architects is worry about how to build the layers of cake.  Operating system on the bottom, Applications in the middle and user profile on the top.  We gathered a few months back to work on this problem and some pretty cool stuff came out of it.  The concepts apply to XenApp and XenDesktop.  Here's the high level view:
I'm going to focus primarily on XenDesktop.  The same ideas apply to XenApp hosted on Virtual Machines as they do to XenDesktop running virtual desktops.  One way or another, the "system" has to be put together and that means assembling an operating system, applications and user data.

Starting at the top: User data.  With Roaming Profiles or Citrix Profile Manager, the application settings are already roamed as a common occurance.  This is the least interesting of the layers in this post.

The quick version is that if a user visits machine A today and machine Q tomorrow, the profile manager will move the applications settings from A to Q.  At logon/logoff the user profile is synchronized to some central machine, where it is later copied from to the new execution machine.  This kind of technology has been around for a very long time, though there still exists some art to "doing it properly".  The "properly" part says that you cannot ASSUME that a user will not be logged on to multiple machines at once.  This is the classic "last logoff wins" problem.  Thankfully, the Citrix Profile Manager does not suffer these problems, so this top layer of the layers of cake is already baked and ready for icing!

A side note is "user data" is over stated, the user profile is really worrying about Application Settings.  The user's data, as in DOCUMENTS, will normally be redirected to some network server. 

Next, the "bottom", the operating system:

Notice I skipped apps for the moment.  Apps are the point of this post, so I'm saving them for last.

Every machine, virtual or real, needs an operating system.  For real machines, no problem, INSTALL IT locally and it will still be there the next time you power the machine on.  For Virtual Machines, this same INSTALL IT solution could be used with one disk image per machine.  Store the image away at "power down" and bring it back at "power up".  This will RUN, but it will sure be a PIG on disk usage.  It will also be harder to maintain as there will be a separate disk image for every virtual machine.  The whole appeal of bringing things into the data center is that it makes things easier to maintain.  If all you do is convert physical hard disks to virtual hard disks, you haven't really worked on the problem of simplifying maintenance!

A side note: With real machines, you can also get to the ONE to Many maintenance aspects, Provisioning Services does this for real hardware just as it can for virtual hardware.

Back to the virtual world:

To get this "right", we need ONE Operating System image and ONE image for each application to go along with ONE user profile.  In this world, the operating system maintainer, maintains the OS and each application owner maintains their respective application.  In some environments, these are the same person, but in concept, they are all separate and even if they are the same person, we do not want application content to pollute the OS image.  If that happens, then every time ANY application needs update, the OS image must be updated and versioned => inefficient.

Cashes, caches and more caches!

With a virtual machine, the virtual machine manager maintains a "write back cache" for the "machine".  Pooled desktop here, there is ONE image for the operating system and each VM as it runs has disk space that holds the changes to the disk as the machine runs.  When the user logs off, the machine powers down and the virtual machine manager THROWS AWAY everything that was written to the base machine.

This "throw away" is necessary as the disk is a "block" entity and the only way that the single base image can be updated is if it is updated in ONE place.  On a future logon, the pool manager will have queued up a pristine machine, with the LATEST AND GREATEST version of the operating system image, which will bring the virtual machine to life - while implementing yet another write back cache for the machine. 

Applications, the point of this post

Application Streaming in the graphic above is the Application Virtualization system bringing in application content.  This is the artist correctly now titled the "plugin for offline applications".  I'm getting used to it.

The App Virtualization system could also be Microsoft App-V or potentially even some other solutions though I do find it easier to manage the caches for systems that have installed agents.  As Architect of App Streaming, I'm working the Citrix side today; the concepts are the same.

Given a virtual machine is running and you want to get apps onto it, what are your choices?

1) Install them into the base image.  We covered this already - it's a losing gig.

2) Presentation delivery.  Publish from XenApp and deliver to XenDesktop; HDX gets it to the user.

3) App Virtualize the applications to get them "into" the hosted desktop.

Today, we're worring about "3".

Given we have a virtual machine, anything we populate into the virtual machine will be a DISK WRITE.

The fundamental guiding principle: Disk writes are BAD!  We want everything to be a "read".

When the OS runs, we want the system to populate pieces of the OS as needed via block based fills from the virtual machine manager or disk subsystem.  When we run applications, we want the virtual machine manager to READ data from the application store and ... (finally getting to the meat of this post) ... NEVER perform a disk write.

Application Streaming has a long history of delivering application content to physical machines.  On physical machines, disk writes are fine, they are a one time cost and the up front cost actually has long term benefit to minimize network activity on future executions.  With virtual machines, disk writes are evil and they are a recurring "every logon" cost. 

We need disk writes to go away, which leads me to one of my favorite statements related to layers of cake.

• We gotta get Application Streaming out of the STREAMING business.

This task much better to hand off to the virtual machine manager or to Provisioning Services, or to the enterprise disk subsystem.  Whatever the the technique of mounting disk space, all of these folks will do a more efficient job, if the streaming system somehow makes everything a "read".  

In the layers of glass, the streaming system populates stuff into the execution cache, which is called RadeCache, named after the directory that holds the application execution content.  I have also previously described that deploy should never happen on a virtual machine based execution, so thankfully we can avoid all of that disk activity.

How do we get application content into a virtual machine?  Answer: MOUNT IT!

The Application Hub is the file server that holds the execution content.  In a physical world, the streaming part of application streaming deals with COPYING content from the Application Hub onto the physical execution machine.  In a Virtual World, we want everything to already be in place, so that application execution is based on disk reads and the disk write generating "streaming" aspects of Application Streaming, never occur.

One complication to date has been that the streaming system stores stuff on the Application Hub in compresses container files, CAB files.  These are going away ([link|http://community.citrix.com/display/ocb/2009/12/04/App Streaming - CAB to DIR]) in a future release, to be replaced with decompressed directory images of the same content.  

Putting it all together

Once we know that we don't want the streaming system to "stream", it can instead focus on application isolation and application delivery; delivering "installed" applications to single instance machines, but letting other pieces of the puzzle take care of the streaming aspects.

The Deploy space is not used; the RadeCache - is necessary.  Stick with me on how we make it go away too.

Given that the RadeCache space and the streaming profile execution content on the App Hub have the same format, there are things we can do to "mount" the Application Hub directly into the execution space.  With the "next" streaming client, this is easier because the App Hub storage format is changed to match the execution format commonly found in the RadeCache.

This can STILL be accomplished NOW using the released 5.2 version of the streaming client.  

The example App I will use is Textpad.  To follow along, profile this up and store it some place convenient.  I'm using Windows 7 machine with the 5.2 level offline plugin and skipping all the publishing aspects.

Start up a command prompt (run as administrator).  I follow with a cut / pasting of a bunch of commands which I entered to prove this can work with 5.2.  After that, we'll talk about what each of them do.

c:
md \apphub & cd \apphub
icacls . /grant:r Ctx_StreamingSvc:(R)
icacls . /grant:r Ctx_StreamingSvc:(OI)(CI)(IO)(R)
mkdir textpad
xcopy \\ConvenientPlaceFromEarlier\textpad textpad\. /s /e
cd textpad
md guid_v
cd guid_v
extract ..\guid_v.cab *.*   (Extract is from Microsoft CAB SDK and requires *.* rather than *)

cd \program files\citrix\radecache
mklink /d guid_v c:\apphub\textpad\guid_v   (Creates a JUNCTION to the mounted App Hub)
exit (close the command prompt window)
Finally run the thing (a really short version of publishing the the AMC).

start "" "%PROGFILES%\Citrix\Streaming Client\RadeRun.exe" /package:"c:\apphub\textpad\textpad.profile" /app:"Textpad"

WOW!  That was alot of stuff.

Given you did the start above, you've already seen the application run and come to life, with minimal disk writes!  Drum roll.

Here's what happened.  

The MKDIR of C:\AppHub simulated the MOUNTING of the App Hub onto the local machine.

The icacls commands gave the streaming service READ privilege to this space.  The streaming service runs on a named user account and by default it can only modify a few places on disk, like RadeCache and Deploy.  I think this is unnecessary as all users on the machine will automatically get read access to this space due to inherritance from the root.  Still, I stuck it in there in case it is a required element.  If so, you'll need this in the system configuration when the mount point is created.

The streaming service will "believe" that this space is local when it is really backed up by some XenServer, Provisioning Server, Hyper-V, ESX, Disk magic system.  One way or another, it's a MOUNT POINT and the streaming service has privilege to read it.

Why local?  The isolation system looks at all disk activity at the start and if it is "network", it jumpts out of the way and doesn't mess with it.  So, the streaming system MUST believe that this space is "local".  For all these virtual worlds, this is "easy".

Next trick is that the streaming sytem will want to populate the RadeCache at runtime, we don't want this to occur.  Instead, we want the streaming system to BELIEVE that the RadeCache is "fully populated".

Creating the JUNCTIOND at the RadeCache space, the streaming system will believe that the RadeCache space is already there, when it looks inside, it will see that the files are already in place and it won't have any reason to want to write more to this space.  Notice that with the "linkd" commands in place, when the streaming system looks into the RadeCache space, we will have LIED to it to instead sent it to the C:\AppHub space.

Lying to the Lying system - wonderful!

Deep down in the streaming streaming, the device driver that does the file system filtering isn't fooled by any of this application level nonsense; it sees the "real" location, C:\AppHub.  The isolation system insists that the RadeCache space for the application content be on local storage, and it is, as far as it knows.

C:\AppHub is "local" as far as this virtual machine is concerend.

With the call to RadeRun, the application is launched and comes to life; skipping publishing.

The application comes to life and things are happy.

Publishing

When publishing applications in the Access Management Console, one of the fields is the UNC path to the App Hub.  This can be a UNC path or a HTTP path, but either way, the AMC is told where the App Hub is located and when applications are enumerated via PNAgent/Web Interface, this information is relayed to the streaming launcher for app launch.  This means that the publishing has to reference C:\AppHub or possibly a local host based reference of local machine based UNC reference.  The AMC and the execution machine both have to believe that C:\AppHub is local storage.  You can also create a LINKD based junction on the AMC machine that reflects to the "real" app hub on the network.  The AMC will believe that this is the local machine space and will relay this to the execution machine.

I dream of a future where the streaming client could override what the AMC says to use and instead use a fixed location that the administrator says to use.  Just dreaming... 

House cleaning 

One of the things that the streaming system does is clean house in the RadeCache.  If it exceeds high water mark in size, the streaming system will launch a reaper thread that deletes things until the house is clean.  Here's a link that describes how this works.  Since we only granted the streaming service "(R)" Read access to the local machine AppHub, the streaming services' attempts to house clean in RadeCache space that is really the AppHub space will be unsuccessful - it will have no choice but to "push on" to look for some other file to erase.  

For this reason, in a XenDesktop space, we should either put everything into the read-only space where the Streaming Service can't delete it or we should set the cache high-water mark sufficiently high that the house cleaning thread will never trigger; in reality, we should do both.   Leaving the RadeCache space writable by the streamign service allows it to run applciations that haven't been published or managed into the App Hub, so this is good - though that space would be loaded on each logon to the virtual machine.

Hope this is useful.

Joe Nord

Product Architect - Application Streaming and User Profile Manager 

Citrix Systems 

When transferring execution content with Citrix Application Streaming, the source of the copy for execution image comes from the Application Hub, which is a fancy name for a file server or web server.  There is no Citrix code on the App Hub which makes it even more mysterious to give it a fancy name.  The name though is growing on me.

In the beginning, there was only UNCaccess to the Application Hub contents and this made authentication pretty simple, the LAN requester takes care of it.  In streaming client 1.2, we added web server support (http streaming), which engages a bunch of new use cases as administrators have lots of tools for managing web servers.

Here's the big picture view.  

The piece we care about here is the upper right circled item.  Notice that all the publishing information comes from the XenApp publishing infrastructure; this information is ultimately delivered to the streaming client via the Web Interface and a slight detour through the online plug-in, the artist formerly known as PNAgent.
Eventually, the streaming client is told to "run something" and when this happens, the application execution material is "sucked down" from the Application Hub and placed into the RadeCache for execution.

Authentication
There are multiple aspects to authentication. 

In a UNC / LAN Manager configuration, this question of access is easily tied to the publishing.  The network file sharing client software takes care of validating file servers and validating user access to file servers. 

Administrators generally put all the applications onto a single App Hub and each profile gets its own sub-directory. 

Since profile directories are "one directory" per profile, access lists can be maintained on the directory and everything associated with the profile is controlled in a single spot.  In the XenApp Access Management Console, administrators publish the application to a "user group", the XenApp IMA, WI, PNAgent will deliver icons to the user based on their membership in the group.  At the same time, the DACLs on the directories on the file share can also be tied to the same GROUP. 

Add a user to the group, they get the app and all access concerns are taken care of automatically. Awesome stuff.

Admins are free to not lock down the App Hub content at all, but XenApp admins are not known for being "information free" type of people and they tend to lock down everything and then open up where needed. 

What happens at runtime
When the user runs the application using a file server based App Hub, the streaming client will attempt to open the streaming target files on the file server.  This is the profilename.profile file and the GUID_version.cab file that is the streaming target.  On the first access, if the user does not already have a connection to the server, the file sharing client code will create one and authenticate the user to the server. 

Here's the cool part - the LAN code takes care of this and even prompts the user for credentials, presumably securely.  The streaming system is completely oblivious that any of this occurs, but eventually, the connection is established and the application is run.  Remember the prompt for credentials...

Web Server App Hub

If the App Hub is web based, the streaming system does the same thing to open the .profile file and open the CAB file that represents the execution target.  BUT, the authentication steps are different.  The streaming system never ever has the user credentials; to be clear, we never have the user credentials.  It also never triggers a prompt for those credentials, BUT the web libraries could still have credentials and can still use these credentials to establish a connection with the web server.

Should the user's credentials be transferred?  What does that even mean?  Stick with me.

When the streaming client establishes a connection with the web server, authentication may be required.  I note that it is optional, the web server COULD be wide open for world access, but again, XenApp admins tend to be "lock it down" kind of people.  The streaming client will make this connection using the "user context", but there's still an authentication.  Does the user have access?  How'd you decide that?

Here, the authentication falls onto WinHTTP library from Microsoft.  The streaming system though is still involved to set up parameters for the connection.  If the environment is Active Directory and the Web Server is IIS and authentication is based on logged in user, then ... no problem.  Everything happens based on the user logon TOKEN and all access is securely controlled and no user secrets were transferred in the clear.  Happy days.

What if other techniques are used?  What if Apache web server?  What if no AD?  Answer: The streaming system does not prompt for credentials and this can mean that authentication will fail.  It also means more things.

HTTPS connection

Do you need to authenticate that the web server is "legit"?  Given the web server has SSL certificates, the validity of the web server can be authenticated.  This is good and it is automatic if you use "HTTPS" as the prefix for the App Hub location.  It still doesn't validate that this user has access to the application bits that are stored on the web server (app hub).  But at least you know that the web server is legit.

Questions I have

Do you lock down your HTTP based App Hubs?  Or is this reserved just for file server based execution?

If all your users are allowed to see all your profiles, then no worries.  Authenticate the web server and you're done.  This though means that all the application profiles are available in vision to all users, so likely not the right answer. 

How bad is the cost of the HTTPS connection vs. HTTP?

If you DO prevent users from seeing profiles to which they are not authenticated, then user authentication is required.  If this is not tied to the user authentication, web authentication system, then it's more complicated.  The streaming system never has the credentials, but it can trigger the WinHTTP library to send them.  Most of this is automatic from the HTTP libraries in the OS, but there's still work to get it good.

Credentials in the clear

I have received a request that the streaming system should PREVENT any web authentication that would EVER send user credentials "in the clear".   It seems to be a reasonable request and we can/could do this by configuring the connection when it is created, a relatively small amount of programming. 

All we need is a release vehicle, which conveniently, is coming up.

The downside is that if you're using methods which would require credentials in the clear, you'll be ... broken after the upgrade.  I mean broken as in "nothing streams".  I expect we'll include a registry override to say that "yes, I want to shoot myself in the foot", but most admins though will likely say "wow!, thanks for helping me find that mistake!".

The proposal is to require at least SPNEGO user authentication for any web server that requires authentication.

I appreciate any Jedi Web-Master feedback on how this should be "properly done".  THANKS!

NOTE: This post originally posted January 19th, but it disappeared.

NOTE 2: Since writing this post, streaming development team have implemented the change and it will occur in the "next" streaming client.  This could break some people; if it does and you're okay with credentials in the clear, there will be a registry item to turn off the improved behavior.  Reg item not yet named.

Joe Nord

Citrix Systems Product Architect - App Streaming and User Profile Manager

The XenApp tech preview made available yesterday includes a technical preview of Citrix Profile Manager enhanced to provide just in time population of the User Profile space.  I alluded to this before, but now it's "real", or almost real... 

Please download and play and send feedback through the forums set up for the Beta, these are the places where the developers will see the comments.  The XenApp tech preview requires a separate sign up compared to normal mycitrix credentials. 

Many questions remain for me

Is Profile Manager more important for XenApp or XenDesktop?  To which is just in time more of a benefit?  XenApp folks have long gotten used to blowing profiles away as a function of logoff or otherwise populating a small portion of space to support a specific application.  XenDesktop clearly has a big need for user profile's well done, and fast profiles; well both have a big need for fast profiles.  

What I'm getting to is "much work" went into building the just in time capabilities; now it's in technical preview and getting ready for prime time.  How much benefit will be seen?  How much will XenApp benefit compared to XenDesktop? 

All fascinating questions - I need real world statistics and appreciate your feedback.   My previous post on just in time brings some clarity to the difficulty of just in time calculations.  Real world is much better.

Joe Nord

Citrix Systems

Product Architect User Profile Manager and Application Streaming

The App Streaming "next" release will support isolation of NT services.   A tech preview is available for download now as part of the larger XenApp Parra tech preview, download here.  To download, you are required to register for the technical preview - this separate from normal mycitrix credentials. 

App Streaming "big items" in the preview:
1) Isolation of NT Services

2) Depreciation of CAB files in favor of unzipped directories, link.

What does isolation of services mean?

First, what is a NT Service?  The long version is this fine write up from the Microsoft MSDN.  The short version is that a service is a program that runs outside the context of any user session, normally once for the whole machine.  Services "generally" run automatically at system startup, but can also be configured to run on application demand.  In concept, services are "high power" and user applications are "low power".  Services are "one", user applications are "many".  Services run with no user I/O. 

When a user application needs to do something that requires more power than a user application has, the low power app can send a request to the high power service and request that the service perform the work.  Since the high power service is trusted, this can be accomplished without violating the system integrity.

Services are also commonly used to implement a central point of control, where multiple user applications contact the service and the service ensures single access to the thing that is single use.  There are many ways to accomplish this exclusive access, but using a service is one solution.

Licensing services

The perhaps biggest use of NT services in App Virtualization space, is the implementation of licensing services.  Here, a service exists to control permitted use of the application software.  When the application runs, it contacts its service to see if execution is allowed.  These can be implemented and "gated" on a number of factors including vision to a dongle, local machine restrictions and the more common, network implemented application licensing where a central licensing server is used to limit execution of the application throughout the enterprise.  The classic problem for App virtualization is that if you cannot run the service, the application will refuse to run.  This has been a headache for Citrix App Streaming since it's first release, but will be solved with this tech preview and ultimately with the next release of XenApp.

Above said, one should understand the limitations.  Just because you run a service isolated does not necessarily mean that the application will work.  For example, consider a software licensing service that TIES the application execution to the same MAC address to which the application was "installed" or profiled.  App Streaming isolation of services does not LIE to the application to change the MAC, so even if the licensing service is run, if the execution machine is different than the profiling machine, the application will still refuse to run.  In a corporate world, the common scenario is a central licensing server which the licensing service contacts over the network - these will work; well, they SHOULD work.

Privilege

In conversations with many customers, I have been asked, many times, about isolation of services.  My question back to the customer is usually the same, what about privilege?  Running services under App Isolation is conceptually not that hard.  Running Services with PRIVILEGE under isolation and keeping the user space and system spaces separate, is hard.  Put differently, deciding to NOT run services when you otherwise view it as an app is hard and preventing apps from escalating due to the availability of the service, is hard.  This is why it has taken 4 years...

Services need to be "powerful", or don't bother running them; to run services in "user space" is an incomplete solution.  Without privilege, you'll have some services that work; but others that don't.

Services exist often because the application programmer has privileged needs.  If they could get it done from user space, there would be many fewer reasons to go through the hassle of writing a service and setting up all the communication between the user space application and the privileged service.  This takes lots of work and for many cases, the programmer would not bother if they didn't really need privilege.

The consistent customer answer:

• I do not mind running services, just make sure you run only the services I approve.

The tech preview will load and isolate services, but the code to "white-list" the services isn't done yet, so for the moment, everything is white listed.  This will be fixed for the production release where an admin only space will be used to limit the services that can be executed.  Until then, don't put the tech preview into production.

Isolating applications compared to "privilege"

Just because the service is run "with power" does not mean that it is run outside of isolation.  Isolation and privilege are very different things, conveniently described here last month.  The key is that the privileged service is still run isolated, so it's runtime needs for profile content can be provided by the isolation system in the same manner that the isolation system supports user mode applications.

The App Streaming system will run the user space applications in their own sandboxes (plural) and execute the  supporting NT services in a separate sandbox (singular - one per profile for the whole machine).  This makes it possible to maintain the ONE service for the "machine" view that the applications may expect.   

For the "layers of glass", the service and the user space applications share the application image, but they will have SEPARATE user spaces, or the top layer of of the layers of glass.   This means that cache fills to benefit the user application will also benefit the service in the same manner that two users support each other in a multi-user execution environment.

Isolated Services DO have network access

Some services install to run as user LOCAL_SYSTEM.  Others may install as NETWORK_SERVICE or even to run on install time created user accounts.  The install credentials of the service are reflected to runtime machine for the standard identifiers, but the user accounts installed services will be converted to run on one of the standard identifiers, which can be adjusted in the streaming profiler.  Notice that LOCAL_SYSTEM has very limited abilities to talk on the network, so there are places here where running on non system credentials and the ability to reflect service run privilege can be advantageous.  The particulars of these capabilities will depend on the application; the account used to run the services is configurable in the streaming profiler.

What does all of this mean?

In theory, it should bring a whole class of applications to the Application Streaming "App Compat" front.  We have tested with the Beta of Microsoft Office 2010, which includes 3 NT Services, and it seems to work.   Hopefully the tech preview of the Parra XenApp will show many other applications that will now "work".

Please give it some exercise and let us know how it goes.  Best place to comment is via feedback page and associated forum on the tech preview. 

Enjoy, 

Joe Nord

Citrix Systems Product Architect - Application Streaming and User Profile Manager.

The rage in the modern programming world is using Managed Code to execute your stuff.   This is "clearly" much better than using legacy systems, or so the world says is true.  While deep down, I know the managed world is the way to go, I am still not so quick to give up on compiled code.  The managed folks call compiled code, "legacy code", with the intended push-down that "managed code" is better.  I feel insulted!

Run "managed code" and your world is more secure, all the ducks swim in a straight line, cats and dogs play together and you don't have to worry about things like pointers referencing memory to which they are not supposed to be pointing, or even pointers at all.  Focus on the "what" and let the Common Language Runtime (CLR) handle the details of running stuff.  Oh, life is happy in a managed world.

Errors in managed code

I write this post because I was using some program the other day that bombed out with an "Unhandled exception" on access a string.something.another.eight.more.levels class which failed to check for NULL and ... BLEW UP!  Then, it kept blowing up because the exception still wasn't cleared.  Click the message box "okay" button and ... you get another message box two seconds later, also demanding top layer window in the center of the screen!  Only way to get the keyboard away from this top level menu was to task manager and kill the offending CITRIX application - Yes, I said it!  CITRIX APPLICATION!

Exceptions are SO MUCH BETTER than an Application Trap in Dr. Watson! 

No, they are not!

There's a theory in managed code that you don't have to check the assignments because if it doesn't work, the exception handler will catch it!  BUT, what if the person calling you forgot to catch one of the 18 different exceptions that you generated?  Notice I didn't say that you will throw, they also have to catch all the exceptions that you will GENERATE based on all of the people that you call and all of the people that they call.

Your code calls somebody to discuss Faberge Shampoo, that person calls two other people, who each call two other people, and so, and so on and so on, until pretty soon Heather Locklear has to protect against every possible exception in the programming world with a "catch all" exception handler! 

Clearly, I have gone WAY off topic, or have I?  

We have just eliminated the advantage of throwing exceptions, instead of checking a RC, the calling code has to register to catch a bunch of exceptions! Miss one and the application blows up.  Catch too many and you mask errors!

The person writing the code I was running the other day, missed one!

I'm perhaps old school, but if you reference NULL with your code, an App Level trap is a really nice way of letting you know that you screwed up.  Fix it and don't let that happen again!  But, programmers tend to be lazy about checking return codes and this causes bugs to manifest up a few layers, which becomes harder to track down.  The managed theory is that the code causing the exception, eight layers deep, the programmer will know exactly who threw that managed exception and be straight away to the answer. 

Is this any better?  No, as a user, I'm still hosed!

Everything has it's place...

Have you ever received a kernel mode interrupt dispatch in managed code?  It ain't gonna happen!
So, managed code has a place and complied code has a place.  The higher up the programming stack, the more incentive for using interpreted language.  This through gets us into a space where subsystems or back ends are implemented in compiled code and the GUI is implemented in managed code.  It can also go the other way. 

Have you considered calling PowerShell from compiled code?   In managed code, easy - in compiled code, harder.

History: UCSD P-Code.

Dating myself, but among the first programming classes I took in college was a Pascal class using a UCSD P-Code interpreter.  This was the rage of the day.  Today we call it "managed code", but back then we gave it the stigma of "interpreted".  The machines (8088) were fast enough and the display was slow enough, that on the display, it didn't matter that the back end was running kind of slow.  P-CODE was WAY faster than interpreted languages like Basic, but also had many of the same benefits. 

P-CODE was run in a controlled environment where the program could not break out to do things that it wasn't supposed to do and the runtime would catch the errors and shut the program down gracefully.  P-CODE interpreter was language and machine independent.

Notice that P-CODE had the same capabilities of Java or .net to move P-CODE from one machine to another without source and the ability to run independent of the actual CPU on the platform.  It even included preparsing of the P-CODE to machine code to get a machine native equivalent for faster execution.  Awesome stuff!
DOES ANY OF THIS SOUND FAMILIAR?

About 1 month into the class, Borland released Turbo Pascal and the use of UCSD P-CODE ceased in about 10 minutes.  I mean it went to ZERO, immediately - worldwide!  Here, the compiled code was SO MUCH BETTER and SO MUCH FASTER that the use of the interpreted system just flat out vanished.  ( Yes, someone is probably still running it - on Citrix MetaFrame XP ).

Paying a compliment

GUI programming is much faster in the managed world.  Draw a picture of what you want it to look like and magical quantities of highly tested classes are available to help you pull it off.  These are all really good things and I need to make sure I'm not seen as refusing to embrace cars just because horses work just fine.

My point is that just because you are "managed code" does not get you out of the business of writing code that isn't crap.  As a user, if I see an exception, you messed up!

The corollary is that if as a user, I see a BSOD from a kernel driver that forgot that it should check for NULL before using a pointer then there's a programmer that messed up here too and while I'd like to pretend that is isn't me, occasionally, this is me.  

Joe Nord

Citrix Systems Product Architect - Application Streaming and User Profile Manager.

A few weeks ago, I wrote a post on why you can't use App Streaming as a solution to deliver Windows applications to a Mac Operating System.  The foundation statement is that isolation "changes" the execution of the Windows application, but the application remains a Windows app, so the execution machine operating system has to support running the app, or running the app under isolation is literally, a non-starter.

I have received a follow-up question that discusses running Windows 16-bit applications on Windows 64-bit systems.  Before we all have the knee jerk reaction that says NO!, we need to think this through.  The Win 64 platform cannot run 16 bit apps, any 16-bit apps, ever.  So no matter how much you isolate them, they still don't run on the 64-bit system.   Does this mean that it won't work?  Probably.

The premise is that the application is really a 32-bit application that happens to be installed with an installation program that is a 16-bit program.

Also, before we all react and say that App Streaming doesn't isolate 16-bit applications, in some cases it DOES isolate them, just not Windows 16-bit applications.  App Streaming can isolate 16-bit DOS Apps, so the statement that only 32-bit apps are isolated is ... incomplete.

Question to me recently is that yes, I understand that the application has to be 32-bit to run isolated on Win 64, but what if the INSTALLER is 16-bit and the application is 32?  Notice that the installer can't be run on Win 64, but the application itself actually has a chance!  A small chance, whose odds approach zero, but I'm getting ahead.

Theory:

1. Profile installation of 32 bit app using 16 bit installer on 32-bit operating system.
2. RUN the 32-bit application on 64 bit operating system

Conceptually, it could work.  It would depend heavily on the application, and I'll say in the normal case, it won't work.  The problem is that the PROGRAMFILES location is different for the two environments and it is "likely", that the application at runtime will make references to the wrong program files location and not find some things that it is looking for.  App Streaming doesn't map these across, at least not automatically.

To take it further though, there's no reason that an admin during profiling can't add a "redirect" rule to the isolation target so that application references to %PROGRAMFILES% which convert into the 64-bit space, could be REDIRECTED into the %PROGRAMFILES(X86)% space.  Here, the "file" stuff would PROBABLY work.

The registry has similar issues.  When profiling on 32-bit, the registry store that the app does will be redirected into isolation space and become part of the profile.  At runtime, the 32-bit app will reference the 32-bit registry, which - head starting to hurt now - will really be in the SYSWOW64 space.  How this plays out to completion gets complex and I'm willing to make a strong wager that it won't work. 

Bottom line, this isn't a situation that has been looked at, whether from a dev, test or customer direction.  I stood up for years and told people that DOS applications couldn't be isolated under App Streaming and then found out that it works fine.  On this one, I'm highly suspicious, but if people could prove me wrong again, it would be a happy day.

Enjoy, 

Joe Nord

Citrix Systems Product Architect - App Streaming

In the "next" release of Citrix Application Streaming, the execution content that is today stored in a CAB file will instead be stored in a decompressed "directory" format.   This post discusses the rationale for the change and how this helps deliver applications to XenDesktop in a layers of cake model.

Some background:

• What is the Application Hub?
• What is an execution target and how does it differ from a profile? ( link )

In the beginning, ...

During the early development of Project Tarpon we were writing neat things like the streaming profiler, the streaming client, the NT service that governs it all and the file system filter driver that does the disk layers.  When discussing ideas like "streaming" and "offline", interesting questions come up on how application content is accessed and copied from the Application Hub.

• How can you enable offline execution by copying the App Hub contents to the execution machine?
• How do you know when the pre-deploy activity has finished copying?
• How do you make the execution "Target" easy to transport between App Hubs?

Using a SINGLE file to represent a "Target" makes this all easier.  At least that was the theory; it makes for an atomic test of presence.  If the file exists - it's complete!  If the file does not exist, the copy isn't finished and the "Deploy" has not finished.  In reality, the streaming client has other means of knowing the answer to these questions, but we still stuck with the container file as a design goal.

We considered ZIP files, CAB files, RAR files, unzipped directories and even UNIX Tar Balls. 

We CODED solutions for ZIP, CAB and unzipped directory storage, tested them a bit and discussed the advantages of each and ... settled on Microsoft CAB files as the storage format for Application Streaming execution targets.  The primarily winning reason for CAB was universal availability of the CAB file libraries on all versions of the OS that we had to support and Windows explorer ability to open CAB files and show the administrator what's inside without us having to write any code.  I should note that while those other formats were coded, they were not productized and were not present in the App Streaming releases to date.

Interestingly, we DID NOT store CAB files in the "default form".  With guidance from our compression expert friends at Citrix WanScaler, we optimized the storage of files into the cab in a manner that would make them efficient for "random access" single file removal while still letting them be accessed from standard cab extraction and viewer utilities.  That's a good topic for another post, so I'll get back to this one.

The winner was CAB files and ... CAB files have been a love/hate relationship ever since.

Deploy space vs. RadeCache

When you want to take a streamed application "offline", it is necessary to COPY the Application Hub contents onto the local machine to support "streaming" at runtime even when the App Hub is not available. 

Here's the Deploy directory for streamed MS Office 2007 on the notebook I am using to write this post.

Notice that the Deploy location on the execution machine looks "really similar" to the Application Hub contents on the file share.  The deploy space just has fewer files, one CAB file - it will always have only ONE execution target by GUID.  It may have multiple versions, but it will only ONE execution target regardless of how many are present on the App Hub and described in the profile.

Here's the App Hub location for the same profile.

 On the App Hub, notice that there are a BUNCH more CAB files than there are in the local deploy space.  There are multiple versions of each of the cab files on the Application Hub and only the latest is really needed for running the application or performing "Deploy", so the streaming client only deploys the "latest".  There can and in this case ARE multiple execution targets for the single profile.  One is correct for 32-bit Windows Vista and Windows 7 and another is correct for 64-bit systems.  Only ONE execution target is "correct" for any given execution machine, so only THAT execution target CAB file is copied onto the execution machine's deploy space.  In my case, it's the one that starts with "1B3".  I've learned to decode these by looking, but for better conversion of GUIDs to profile names, see this previous post.

Once the streaming Target is present in the Deploy space, the Application Hub is no longer needed.  It is consulted on App Launch to look for updates to the application.  This is RADE = Rapid Application DElivery,  but other than that, all streaming is from the Deploy space to the RadeCache execution land and this happens whether the machine is presently "online" or "offline". 

If the application is published to the user to run offline, then it is deployed and run from the deploy space - or more accurately, it is deployed and run from the RadeCache space in the same manner as for "online", but for "offline", the source for cache fill operations is the Deploy space rather than the App Hub.  I call this "streaming from one side of the hard disk to the other".

Notice that the execution target is stored twice for the offline case, once in the deploy space where it is compressed and then again a PORTION of the target is stored in the RadeCache space where it is in de-compressed form.  For a detailed look at the numbers behind this, see my earlier post on RadeCache execution space

The execution layer of the layers of cake

The layers of glass has execution occurring from the execution layer of the layer of glass.  This is the "middle" layer in the 3 layer view of isolation.  This layer is present on the execution machine in the RadeCache space, which can be moved, but which is normally located close to the installation directory for the streaming client.   It should be a directory level deeper than it is, but that's a different subject.

Here's the RadeCache space on my notebook, again for MS Office 2007 from the Citrix showcase farm.

 

Notice that there are no CAB files here.  The execution space is similar to the App Hub space, but the execution content has to exist in decompressed form so that it be be layered into the layers of glass.

The "Device" item above is a DIRECTORY and below there is a C directory for drive C: and below that are a bunch of other directories that are the backing material for the isolation layers of glass that represent their equivalent location on the real C: drive.  I was going to show an expanded listing of all of the directories, but it's 127 directories and I think I'd be lost before I found the top again.

THE EXECUTION IS FROM THE RADECACHE SPACE!!!
The Deploy space just serves as backing material to fill in the RadeCache space when needed just as the App Hub serves as backing material to fill the RadeCache in an "online" scenario.

Okay, good introduction - why are you moving away from CAB files?  Finally onto the meat of this post...

When taking application streaming from "stream to server" or "stream to client" and starting to look at "stream to desktop", the needs of deploy space and the central Application Hub start to change. 

Some definitions are needed. 

• stream to client" equals stream to a real physical machine such as this notebook I carry to and from work every day. 
• "stream to desktop" refers to streaming to a XenDesktop provided machine, which generally speaking will be a virtual desktop, hosted in the data center.

In the stream to desktop case, CAB files start to become a hindrance rather than an assist and they need to go away to help solve the layers of cake for virtual machine application delivery.

Consider the "layers of cake", the execution content is brought into the execution environment at runtime.

Wearing your App Streaming developer hat, how do you EFFICIENTLY move the application content from the Application Hub into the execution machine?

Answer: You don't.  You don't copy anything, EVER!

Instead, you replace the CAB files on the network server (Application Hub) with an unzipped representation of what is inside the CAB files and then DIRECTLY MOUNT the Application Hub contents into the execution environment.

This provides a mechanism for executing the applications in a RADE environment where the applications are delivered to the machine at runtime, but it also prevents most disk WRITES that would occur in a normal ONE to MANY shared image mode XenDesktop farm. 

Disk writes in a virtualized world are maintained by the virtualization back-end on a per-machine basis.  This occurs even though the write back cache is thrown away when the machine shuts down.  Without "mounting", the RadeCache starts empty and gets written to at application launch, which is a first time launch from a streaming view, which is slower than a second time launch.   Once full, execution is happy, but when the machine is powered off (logoff) and then powered on again (logon), the whole process would repeat and writing large amounts of data to per-machine write back cache is inefficient and needs to be avoided.

The solution is to NEVER copy onto the machine; instead MOUNT the content so that the execution machine views everything as "already present" and the streaming system has no need to perform a "write".  This keeps the per-machine write back store to a small size, permitting large machine count implementations at the minimum size of write back cache.

With mounted space, ONE to MANY is achieved.  ONE image of the Operating System is maintained and ONE image of the application image is maintained, while running with a minimal write back cache.

You have your "cake" and get optimal performance too!

Step 1 in achieving a MOUNT of the Application Hub is eliminating container files.

Steps

1. Get rid of the container files and replace with directories
2. Directly MOUNT the App Hub contents into the execution space.
3. Tell the Streaming Client to NEVER cache fill and never do cache house cleaning

The Streaming aspects of Application Streaming are overstated
In this scenario, the "streaming" aspects of Application Streaming are relegated to the disk system, where "App Streaming" then benefits from BLOCK based fills rather than file based and where the streaming client can then focus on application delivery and isolation and completely ignore cache fills / streaming.

There are several steps to get there, step 1 was getting rid of the container files and this is covered in the "next" release of App Streaming.   It wasn't a dislike of CABs to make it go away, the content just needs to be easily accessible and getting rid of the container format makes that easier to achieve.

Let's get into steps 2 and 3.  Some of these are administrator things and some are future client things.

End goal is to MOUNT the Application Hub into the execution machine and have the streaming client "notice" that the App Hub is local storage.  Given that the App Hub and the RadeCache space are now "consistently" formatted (cab to dir), with the App Hub "Local", the streaming system could "skip" the RadeCache space in it's entirety and instead mount the App Hub space into the layers of glass.   This will NOT happen in this release!

Until the App Hub can be mounted, the optimal answer is to have the RadeCache space directories be administrator defined junctions which point into the mounted App Hub.  There are numerous methods to pull this off and based on the tools that the administrator has available with disk tools, the optimal answer may vary.  Since the cab to dir is "fresh" as in not yet released, the later steps are still being worked.  I hope this post brings some vision to the goal and helps administrators efficiently solve the layers of cake.

As for having the streaming client "notice" that the App Hub is local, there are some product level headaches to achieving this goal.  Example is ensuring digital signatures are verified before execution content gets into execution space.  This is only needed for digitally signed profiles and ... not everyone is using those, but they exist and customers are using them, so we have to sort this through before the RadeCache space can be bypassed. 

How about some alternate thoughts...

VHD Files

Why not store the execution targets in a VHD file on the Application Hub and then MOUNT the execution target VHD directly.  Here, you retain the ability to have a container, but you still get unzipped via of the contents.  Answer: This would work. 

We went with the simple approach of using decompressed DIRECTORY format on the theory that this gives administrators the most flexibility on how to deliver the contents.  Notice that just because the streaming and isolation system BELIEVES that the App Hub contents are in de-compressed directory form does not mean that the contents are not REALLY stored inside of a VHD file mounted into it's view as a directory form.

Lying to the the Lying system!  Gotta love it.

What about backward compatibility

The "next" streaming client will still load things from streaming profiles created by earlier versions of the streaming profiler.  Here, cab files are the only storage and where the unzipped directory contents do not exist, the "new" streaming client will load the contents from the existing form.   In a "Deploy" operation, the CAB file on the App Hub will be moved to unzipped structure in the deploy space.  In theory, administrators don't need to care about the format of the data in the DEPLOY space, but if you've been doing Deploy using your own techniques outside of RadeDeploy.exe, then you should be aware of this change. 

When looking at the App Hub, the streaming client will look first for directory - and if found, use it.  If not found, it will look for cab file and if found, use it.   This will happen for both Application Hubs on network servers as well as Application Hubs on web servers. 

When will CAB to DIR happen?
In the "next" streaming client; the one that will accompany the next major XenApp release.  The Streaming Client version number will be larger than 5.2.

Joe Nord
Citrix Systems Product Architect - Application Streaming.

I get this question less often recently than I used to but it still shows up.  

• Can Application Streaming be ported to Apple Mac or Linux? 

The question is usually based on the idea of wanting to run XenApp published streamed applications in an isolation system on the foreign operating system.  That is, to bring streamed Windows applications to the other system. 

You can insert your favorite operating system on the above list, but the answer remains the same, no.

APPLICATION ISOLATION is about changing things and lying to applications so that they think they are doing one thing when they are really doing another.  Fundamentally though, the executed application is still a "native" application for the operating system.  The executed Windows based application is still a Windows based application and it will not run unless something exists below to satisfy the Windows APIs.  The application won't even load unless the Windows loader brings it into memory.

Can you use App Streaming on Mac?  SURE! 

Insert your favorite MACHINE virtualization system such as Parallels, install Windows into the virtual machine, install the streaming client (aka: offline plug-in) and then run all the applications streamed that you want.  This works fine! 

Is it "streaming" to the Mac?  No! 

I see people around Citrix doing this all the time.  They run streamed MS Outlook 2007 and happily check their email and do many things of their job, all day long with lots of apps.  Many of them spend most of their day inside the Windows environment of the Mac machine.

In this usage, I call the MAC the ...

• THE WORLD'S LARGEST WINDOWS LOADER!

For the non programmers in the room, the "loader" is the component of the operating system that is responsible for bringing the operating system to life.  The quick version goes something like this:

The machine powers up and and a whole bunch of things happen, but eventually the hardware kicks off the machine loader from ROM in "real mode" at address CS:IP FFFF:0000, this kicks starts the BIOS.  The BIOS h has the job of finding a 512 byte sector of disk, loading it into memory and "jumping" to it.  From the BIOS perspective, at this point the machine is "booted".  The 512 byte initial loader, brings in a bigger loader, which brings in a bit more, which brings in a primitive part of the operating system, which brings in some "boot" device drivers such as "disk" boot load device drivers, which brings in more of the operating system, which loads more device drivers, like NTFS, enables paging and does a bunch more stuff until you eventually get a machine, running and ready to do useful work.  You can make a career out of any of these activities.

In my mac example without machine isolation, the Mac must boot first and once it's done, it loads the virtual machine thingie which "powers on" the x86 box, which does a bunch of things, which then runs from "ROM", which is really "RAM" and jumpts to a "real mode" address FFFF:0000 and then boots the Windows machine.

This continues on until the Windows box is ready to do work => ergo, the Mac is the worlds largest Windows loader.  While boot sequences are fun, I am way off topic.  

Can you run App Streaming based apps on a non-Windows platform?

Answer the question with a question:

• Can you run WINDOWS based applications on a non-Windows platform?  Answer no.

Sometimes this answer receives a follow up: Have you considered adding this capability?

Now, a white-board is needed.  We use a white-board because nobody has chalk-boards anymore.  Frankly, I prefer the old style because they could be readily and reliably erased, but I'm digressing away from the topic.

How much slower does a streamed app run compared to a locally installed app?

Answer: They are the same!  CPU wise, it's the same.  A process is a process is a process and program code is program code.  The isolated app runs NATIVE on the machine.  It is loaded by Windows and the app uses Windows to do things that apps do with Windows. 

Eventually, the program may call a Windows API, such as RegOpenKeyEx or CreateFile.  When this happens, the program execution takes a brief side journey through the isolation system where the parameters to the API are "adjusted" to make the application run inside of an "isolated environment".  This is how the layers of glass are implemented.

The application is still an application and it is still dependent on the Windows machine for running the application.  Things do get a bit more complicated because even DOS apps running on the Windows machine can be isolated (link), but fundamentally, Application Isolation "adjusts" the execution of applications that are running native on the Windows machine.

Finally, the question can be answered: You can't run "isolated" Windows apps on a non-Windows machine, so there is no point is worrying about running App Streaming under MAC or Linux or others.

What about App Streaming to Windows XP Embedded?

Sure, that will work and this has been done. 

What about App Streaming to Linux under Wine?

Sounds like an interesting activity.  I'm quite sure it won't work, but there could be other neat things.

Enjoy!

Joe Nord

Citrix Product Architect - Application Streaming and User Profile Manager

As the Citrix Architect of Application Streaming AND Architect of Citrix Profile Manager, you might infer that I'm interested in leveraging one technology to help the other. 

Background on roaming profiles and Citrix Profile Manager

First, background on Windows "roaming profiles" and similar.  Consider that when a user logs onto a machine, the logon activity must "roam" or "copy" the network stored version of the user's profile onto the execution machine.  In the general sense, everything on disk beneath %USERPROFILE% or C:\Users\usename, will be copied onto the execution machine at logon and then copied back to central store at logoff. 

During logon, this is a "large" consumer of logon time where it consumes perhaps the largest portion of the overall logon clock.  With roaming profiles, this full copy happens every time, but with efficient systems such as Citrix Profile Manager, the "copy" is actually a "sync", so the copy happens really fast and the copy back is limited to only the files that changed.   While this also speeds logoff time, let's stick with the value of logon time because ... nobody cares how long it takes to logoff.

Where all of this stuff gets more interesting is when you consider a user logging on to XenApp hosted session or logging onto a hosted XenDesktop session where a common disk image is used for the base operating system.  Notice that in each of these hosted cases, the user's profile on the execution machine is initially "empty" and it will be initially "empty" on every logon.  This means that the glorious logon sync that the Citrix Profile Manager does at logon will actually be a "full copy" and here, it starts to behave with the same inefficiency as the base operating system profile solution because it will be a full copy at EVERY logon.  We like to do better than this.

For a more detailed introduction to Citrix Profile Manager, consult this Sepago white paper.  Recall that Citrix Profile Manager is based upon the Sepago Profile technology that Citrix acquired some time back.

Use "streaming" to solve profile population
Logical move: Instead of copying stuff onto the machine at logon, use isolation technology to LIE to the system to tell it everything is copied local when it is really still on the central store.  Eventually, when the system or an application references stuff in the user profile, go fetch it and make it present.  This is "just in time" population and it has the promise to greatly improve logon time in a hosted environment.

For JUST IN TIME population, the bet goes, some large portion of the user profile will never be referenced, so you save big on the logon speed and you save big on the runtime because much of what exists in the user profile will NEVER be copied to/from the central store.  This means that using a just in time profile solution will save LOTS of time for logon, and this is a great benefit!

Great - How much quicker?

The answer: LOTS QUICKER!

Yes, but do you have a number?

I'd like quote: Just in time Profile Manager speeds XenApp logon by 100%  

My gut says that the number is closer to 40% - 50%, but I don't have any hard evidence and thus the premise of this blog post...

Getting a "number" is harder because the answer is that "it depends".  Marketing people and customers prefer hard integers.  The integer number is hard to dream up because the answer depends on the size of the user's profile and the efficiency of network activity to/from the central profile store to the execution physical machine or virtual machine.  The BIGGER the profile, the more efficient.  If the profile is zero size, then JIT doesn't do anything and if the profile size if infinite the the JIT logon benefit is also without limit.

So, the answer for the logon value of just in time is is somewhere between a 100% benefit and 0%.  This doesn't help.

Let's go with an example:  The profile on my primary computer is 11GB, yes Gigabytes.   I could be a rare case.  This is pretty close to "infinite" so I will save plenty in an average logon.  

It turns out that 10 GB of my 11 GB profile is a TrueCrypt encrypted hard disk container.  I'm sure glad I'm not copying that down from a central store on each logon!  In a hosted VDI, I would be.  Technically, I'd store stuff differently, but in concept I'd be copying this down.  In a hosted XenApp execution with just in time, I would never copy down this file so Joe's benefit of just in time will be either 0% or 100% and nothing in the middle.  This still isn't helping me come up with a number.

For my normal machine, I am not connected to profile manager or roaming solution or even to a domain so my system may not be the perfect example.   As XenDesktop becomes more and more prevelant though, the strange things that users do to populate their user profile will make examples of users doing stupid things like placing 10GB files into the user profile more and more common.

If you are using the same profile for the primary hosted desktop as well as numerous XenApp server based app executions, you experience the victory!  Only ONE of them will be accessing that really big file.

In my case, the primary machine will access the really big file, but all the "vacation request" and similar applications that I run will run on another computer, where the really big file will never be referenced.  Using just in time population of the user profile, the majority of my logons and I'll say that ALL of my quick in/out sessions will have a HUGE benefit to not copying down that 10GB file!  This will make my logon time benefit near 100% on these other sesions and near 0% on the machine where I do access that single file that is 90% of my user profile!  

It is much better to quote percentages on something like this, so the time saved will be some percentage of the overall logon time and the LARGER the user profile, the HIGHER the savings!  Okay, we're getting closer.

Right - what's the number to quote?

Let's start with a formula:

• TimeSaved = TotalTimeWithouJIT - TotalTimeWithJIT;
• PercentFaster = (TimeSaved / TotalTimeWithoutJIT) * 100%;
  

How to calculate "TotalTime"?  This number will be the sum of the entire logon, nobody cares how much more efficient the roaming profile copy is, they want to know how many SECONDS this will save on logon time and how much of a percentage faster the logon time is. 

This requires breaking down the logon time of a "NORMAL" logon.  What is a "normal" logon?

Need to have: Computers that are representative of a "normal IT shop".  Need networks that are also representative of "normal world" and network servers and end user machiens that are "normal".  Must simulate some kind of load on these machines or just take it as a given that the load during the test will be similar to all the other stuff going on with the test network at the time of the measurement.

The key ingredients are:

1. Size of the user profile.
2. Speed of the network.
3. Overall logon time 
4. Logon time used to copy the full user profile

Given the above, we tigger the measurement to figure out how much time is profile population and poof!  Take the total logon time, subtract out the portion spent copying the user profile without JIT and ... We have a number!

What's that number again?

What is the SIZE of an "average" user profile?  What is the average file size?  How many files are "normal". 

Do normal users have giant files inside their user profile?  Yes, they do!  If you have have you ever copied a .MPG file or .MP3 onto your desktop, then you're as guilty as I am.  The PROFILE WILL GROW and will be large.

How large?

We need to exclude some files.  What about the files that will NEVER copy onto the execution machine even ignoring just in time.   Some stuff like "My Documents" will not be roamed, but will instead be accessed straight off the network via folder redirection.  This is "standard procedure" for setting up profile environments and here, "just in time" doesn't have any effect.

Let's get to statistcs.

Start with the initial 11GB and take out that 10GB file that is an anomaly and I'm left with 390MB.  The missing 610 MB is round off error.

Administrators usually redirect "My Documents".  Take out Joe's "My Documents" = 208,055,865 bytes and I'm left with 182,450,081 bytes.

Okay, I wonder what I have inside my USERPROFILE that could possibly constitute 182MB?   Dig deeper.  I have 24 MB of pictures!  While I am sure that they are lovely - I am also sure that I haven't looked at them in months.  If I were "server side" my admin would probably redirect "My Pictures" too.  Now I'm down to 158MB.

Keep looking....  BING BING BING BING BING!!  We have a winner.  I have 149MB of "Downloads".

First - before anyone starts, "Downloads" have ZERO relation to the 24 MB of pictures!

Something is wrong here because after you subtract all this out and I'm down to 9MB of stuff that wouldn't normally be "redirected" and I KNOW that NTUSER.DAT on my machine is 8.9 MB.  This leaves me with 100KB of stuff that is candidate for JIT value.  There's a number breakdown here someplace, but let's keep it going.

Pretty soon it's obvious that I don't have ANYTHING in the user profile that matters.  I store it all in that huge the container file and in "other places" on the hard disk.  In a hosted case, these "others places" would find their way into the user profile, so all my utilities would be a plus for the profile.  Go looking...

What are "other places".

Utilities.  I have lots of them and store them off the root.  In a hosted desktop model, they will be in the user profile.  Add in 137 MB.  I have 77 MB of sound .wav files left over from my days of writing audio device drivers.  These would almost never be accessed, but they would live in my user profile.  Batch files.  They are kept separate from executable utilities, so add in another 9 MB and utilities and 33 MB of Windows SYMBOL files for debugging stuff.  137 + 9 + 77 + 33 = 256 MB of additional stuff for the user profile.

I love it when numbers come out to a power of 2!

One number:  "Average" user profile size is 256MB!

Yes, I left the 10GB file out of this mix.  That quantity of storage just has to kind of go away from the calculation.   I hear numbers of 20-30 seconds of XenApp logon time being required for copying down user profile content?  If we can make this number be "zero", then there can be real value in just in time profile solutions.

Add in some stuff that would be moved from my container file onto the user profile and I propose that the real size could easily double. 

Joe's proposal: The Average size of user profile is 512MB!

If any of this math makes sense, then I have an example number set that can be used to construct a measurement.  Is 256MB the right number?  Is 512MB the right number?  How about 1GB?

Real world statistics are the elusive number.  If you happen to have a couple hundred profiles representing a years worth of regular hosted desktop usage and wouldn't mind sharing, please send me an email or comment below.  

THANKS.

Joe Nord
Product Architect of Application Streaming, Profile Manager and a few side projects
Citrix Systems - Fort Lauderdale, FL

I have heard some rumors of the production level App Streaming service (radesvc.exe) dying at runtime.  In the reported failure, the administrator has configured the service for automatic restart to work past the issue and I have suggested that this is only masking the problem, don't do that!  The streaming service, like most NT services, should never die and I'd much rather cure the root cause than work around the issue. 

The realities of "real users" and "production use" sometimes necessitate doing things that aren't ideal in a theoretical sense so this advice cannot always be followed, which brings us to this post where I will bring vision to the perils and values of configuring the streaming service for automatic restart.

Put your FSFD programmer hat on

You wear this hat when you're writing kernel mode code.  You write the file system filter code for the App Streaming isolation system and this code has two primary purposes; file system filtering and process monitor for sandbox management.

As a FSFD writer, you are never allowed to die or the entire machine will turn blue.  Today's post is not about kernel mode things dying, its about application level things dying.

Put your NT Service programmer hat on

You wear this hat, you think you're powerful because you run with "higher privilege"; higher than mere apps.  You may even be considered part of the "system", but from the perspective of the kernel code, you're a mere app too and as a class, all of you are untrustworthy. When a service dies, the machine does not turn blue, but it is still bad!

What does the service do?
Among other things, it is responsible for launching all isolation sandboxes and placing applications into the sandbox for execution.  Here's a chart that brings some color to this description.  What isn't drawn in the below is that the service talks to the FSFD to define sandboxes and launch applications into sandboxes.

What does the File System Filter Driver do

The FSFD hangs out and implements file system redirection - the layers of glass for the file system.  It is also responsible for managing which applications are in the isolation spaces; yes, that's plural on purpose. On a given machine, especially on a XenApp server, the FSFD can easily be tracking 500 isolation spaces.  Consider that there is state data for each of these.  It isn't large, but it exists and the code that keeps track of this actually does it in a balanced binary tree, which seems like overkill until you get large number of isolation spaces.

In the service, you also have state data for each sandbox.  Here though the state data is allocated per-thread.  Put differently, each sandbox gets a thread and this thread and only this thread is used for communication with the kernel mode code.  In this way, a few things are achieved.

1. The streaming service doesn't have to have complicated logic to manage its sandbox state
2. The kernel code can gate who it's willing to talk to based on the thread of the creator
3. When the FSFD has work for the service to do, the service "always" wakes up in the right state.

For computer science stuff, these are all positive actions. 

The negative actions

The service isn't supposed to die without a graceful shutdown and it should only close gracefully if it isn't managing any sandboxes.  In practice, "non scheduled" terminate happens all the time during development and recent reports show, it can also happen during production. 

The FSFD tolerates service death.  Why?  Primarily it does this because it doesn't have any other choice. 

If the service dies, the kernel code, being all powerful isn't surprised by this action - it "observes" that the service has died, but there isn't a whole bunch it can do about it.

Consider an example

You have isolated applications up.  Let's say you have 10 of them running, from 5 different profiles.  This means that you have 10 applications running in 5 different sandboxes.

The service dies...

The applications are still running, but they have lost their support network.

Let's say that the application now issues a DIRECTORY ENUMERATION on stuff in the isolated space.  Normally, the FSFD gathers information from the service to satisfy this request.  This is how the FSFD "LIES" to the application to tell it that things are present that aren't really present.   In this case though, the service is "gone", so what does the FSFD do?  Answer: It does the best it can and "falls back" to AIE style N layer directory merge.  The directory enumeration is satisfied, but the files that are there via a lie will not be included in the directory enumeration results?  What effect does this have on the application?  Don't know - depends on the app, but in general the results are bad.  

If the application issues a file open, you'll satisfy it based on the things you can answer without the help of the streaming service.  This means that if the file is really present in the cache, the file open will succeed and if it isn't, it won't, or execution will drop down to a lower layer in the layers of glass to answer the file operation.

Will this work for the application?  Maybe.  Ideally, you'd like to terminate the applications, but terminating applications when users have stuff running and haven't saved their work is considered bad form.

New sandboxes are launched

Recall that new sandboxes cannot be created without the help of the streaming service, so here it is a given that the service has been restarted.  When the service loads, it contacts the FSFD to register itself.  The kernel code says "nice to have you back" - but there isn't a dag gone thing it can do to help the orphaned sandboxes from the previous run of the service.  All the "app level" state data is "gone" and there's no way to put it back together again.

New launches though can be handled.  When created, the FSFD notes who the service is and will communicate with this "new" instance of the streaming service to manage the "new" sandboxes.

During development this is cool!

When developing the code, if you are the NT Service writer, this is really really cool because you can write code, debug it, terminate the debugger (which unloads the service), change the code, compile it again, run it (which loads the service) and the FSFD will just plain deal with all of this.  Very fast for development; no reboots needed and you can even do all this stuff from a visual development environment like MS Visual Studio.

During PRODUCTION this is not as cool!

Being willing to take on new sandboxes means that auto-restarting the service can seem like a good idea.  The thing this overlooks is that the orphaned sandboxes are, well they don't have their support network and without the streaming service, directory enumeration and file opens are not going to occur correctly unless the streaming cache is completely full.

Put your ADMINISTRATOR hat on

What should you do?  Answer: Treat death of the streaming service with caring detail.  It should be investigated and fixed.  The Citrix support team will love this - Joe said we should report service death rather than restarting the service.  My response, the service should not DIE unless you kill it!  I'm pretty sure service team already has the report, so I'm really writing for the next person and hopefully by the time you read this, we'll already have it fixed....

How to work around.

Above said, if you get in this situation, run one app from each profile with "-e" populated RadeRunSwitches.  This will fully populate the streaming cache and will minimize cases where the application will fail a file open or directory enumeration.  Next - Turn "-e" off as it will command a full extract on EVERY App Launch and you don't want that.  Next step - get the service fixed.  In the mean time, you can auto-restart the service to get new sandboxes created, but just be sure you aren't using the auto-restart to hide a problem that really needs to be investigated.

Before people ask, I already have feelers out to the people that have seen the service die.  Hate to have this happen with production code, but the correct answer is to research the problem and make the fix.  Hopefully readers of this post will appreciate the open nature to acknowledge a bug that isn't widely seen.

Joe Nord
Citrix Systems Product Architect - Application Streaming.

Administrators are used to the idea, that running applications under Application Streaming will permit poorly written applications to run in a multi-user terminal services environment.   For example, if the application wants to write to the \Windows directory, no problem; the application will believe that it wrote there and later if it reads the same stuff, it will see what it put there and generally, the application will work. What is less known is that that Application Streaming and XenApp publishing can be used to reduce the rights of the application at execution so that it has a reduced chance of hurting the machine.

Privilege vs. Isolation

Isolation and "privilege" are different things. Running the application "isolated" does not mean that the application can't do powerful things.   An administrator privilege ISOLATED application CAN still perform privileged operations such as adding new users to the machine, marking them as administrators and adding them to the remote desktop group where the evil doer can then remotely login, as a non-isolated administrator and easily do evil things. 

Not a problem for XenApp hosted execution

To be clear, none of this is important for XenApp hosted execution.  Here, the user is already a user and stripping power from the user to get them to user power is a "nop" because they were a "user" to start with.  This discussion of "privilege" reduction is more of a Windows XP client side, or hosted desktop statement where "admin" power users are the norm.   On Windows XP, unless you're very good at locking down the machine the end user will be running as an "Administrator" and this is not desired.  How can you make this happen as little as possible?  How can you get MOST of the applications to run with the least privilege possible?

Brain damaged applications

Some applications even CHECK to see if they are admins and refuse to run if they are not.   Awesome!  If you can't figure out how to code it, demand admin rights machine wide!    You can easily hit a situation where 90% of your desktop applications will run fine without admin rights, yet you have no choice but to make the user a full blown administrator because some small subset of the applications demand admin rights; or perhaps, even really need them.

What about the "normal" applications that don't need admin rights, or at least don't need admin rights when run under isolation?  It would sure help if we could at least make the "all powerful" user be a "lowly user" for the purposes of the majority of application execution, even if the user is really an administrator.  You can, and XenApp makes this easy.  First, some history.

DropMyRigthts

Go back in time and take a look at this 2006 technet article from Microsoft on Least User Access and a description of the DropMyRights utility by Michael Howard.   Excellent stuff and here is a related set of blogs from Aaron Margolis of Microsoft who seems to have a passion for running apps as a user!   The output of this early work was a command line utility called DropMyRights which would duplicate the user's logon token, strip the powerful rights - and then use the modified token to launch the application.  Good stuff.  As an example, here is the .BAT file I used to use to launch MS Outlook.

• dropmyrights "%PROGFILES%\Microsoft Office\OFFICE11\OUTLOOK.EXE"

The idea of running apps on forced user privilege on Windows XP was not unique to App Streaming, but we did wrap pretty GUI around it and wrapped application publishing around it to make it easy to use - and then we didn't tell anyone it was there.  To be fair, most of the usage was server side, so it wasn't as important, but hosted desktops are changing this.

The XenApp publishing system makes this dropping of user rights accessible via easy to use GUI.

Access Management Console

Here's the AMC screen that controls this setting.  Notice that this "stripping of rights" is controlled in the AMC - not in the streaming profiler.  Could it be controlled in the profiler?   Sure.  Both of these tools are nice GUIs which could accomplish the same goal, so yes, it could be controlled in the profiler, but it isn't.  One could even make a really good argument that it is in the wrong place and SHOULD be in the profiler because this is where the admin is that knows more about the application.  I would agree, but it wouldn't matter, it's still in the publishing console whether or not this seems like the right place.  

 
When I wrote the draft for this post, I did it in a place without internet access, so I couldn't easily check the default.  I wrote that SURELY! the default is that we strip the rights before launching the app.  Surely, Shirley, what ever you call it, the default is the other way; by default, the launch leaves the user's token alone and launches the app using what ever power the user has according to logon.  If you CHECK the box, then the Access Management Console tells the Citrix IMA to tell the Citrix Web Interface to tell PNAgent to tell the Streaming Client that it should strip power from the user for the purposes of running this stream to client application.  Where the application will permit it, You should set the checkbox.

XenApp server side, it won't change anything;XenApp Client side, it will ensure that the application is launched using a user token that has "lower power".  Lower power is better...

Here are some other writings on Application Streaming related to this:

•  Enhancing the Security of Application Streamingfor Desktops

Enjoy!

Joe Nord

Citrix Systems Product Architect - Application Streaming

One of the first screens you will see in the Streaming Profiler wizard is a screen about "Enable User Updates" or in the earlier profilers, this was called "Enhanced security" or "Relaxed security".   Wow!  Mysterious terms!  The first thing we do in the profiler is hit the admin with a question that they don't know the answer to.  Hum.

Steps:

1. Describe the panels
2. Describe what the settings do
3. Examples of how this effects application execution
4. Guidance on how to configure the setting

Here's the panel in the streaming profiler version 5.2 (XenApp 5 Feature Pack 2):  Hot off the presses, released GA to the web download last night.

Here's the same panel in the previous streaming profiler (1.3)

What does this setting do?

Under the profiler, it doesn't do a whole lot.  It just sets a BOOLEAN that accompanies the streaming profile.  You can see via nice visual form in this streaming profiler, but if you dig down, you'll find that all this does is set a boolean in the profile XML data; at the PROFILE layer.   Changing this setting actually does more work, but I'll get to that in a minute.

Going back to the Layers of Glass, there are conceptually 3 layers of isolation.  Here's an abbreviated version.

At runtime, the applications in the isolation sandbox see a multi-layer merge of the true machine at the bottom, masked by the installation image and at the top, a per-user layer.  The per-user layer is seen "first", followed by the lower layers of isolation and finally the true disk or true registry of the machine.

The normal action is that the machine starts out pretty much clean, the streaming profiler captures the installation activity of an "installer" that writes stuff to the file system and registry.  These are packaged up to become the "blue" layer above, the installation image.  

At end user execution, the installation image is laid down on top of the execution machine and as far as the isolated applications are concerned, they are installed.  It's all a lie - they aren't really installed.

The top layer is initially "clear" or "blank".  As the programs run, they may store documents and similar, but these would generally not be in isolated space, so they don't really show up in this picture.  The application though may WRITE things to "off-limits" locations which would be caught by the isolation system and end up with storage of stuff to the per-user layer of isolation.  These land in the top layer of the isolation stack which is set up as one per-user.  This is what allows ill-behaved application to run happily under isolation on a multi-user machine when they won't happily run without isolation.  As an example, consider an application that stores settings to the program installation directory in a .INI file.  Under isolation, this will be captured and land in per user space and the application becomes runnable in a XenApp Terminal Services world where otherwise it would not work successfully.

Back to this post

If the application updates itself at runtime, the update will land in the per-user layer of isolation and this is bad.  Standard procedure when profiling application installations is to TURN OFF all automatic updates.  The application should not update itself - this should only be done in the profiling scenario where the administrator commands the update.  Recall that the isolation space is ONE and the per-user space are MANY, so we only want application content to be updated in a single place.

What does "Enable User Updates" do?

If the user downloads application updates such as .DLL updates or .EXE updates, should this be permitted?

The general answer is "NO!".  Some administrators may have a scenario where this is desired.  The common ones are users wishing to install their own plugins for isolated web browsers or install their own addons for things like Microsoft Office.

How does it work?

Put your file system filter driver writer hat on.  For isolated applications, EVERY TIME the application opens a file or tries to open a file, you get first look.  If the file open is for executable content, should this be permitted?  If "enable user updates" is "off", then file opens for RUNNING executable content from the user layer will be denied.  

The neat part here is that the isolation system distinguishes this behavior based on WHO the caller is.

If the caller is vanilla application wanting to read or write content, no problem - do what you want.  If the caller is the Windows LOADER, then this setting comes into play.  If the LOADER is trying to load executable content from the per-user layer of isolation, then the isolation system can be told to FAIL that operation, and this is what this setting controls.  Pretty neato.

One headache

The setting while stored as a profile level single property (a boolean) is implemented in the isolation system as an attribute of EACH of the isolation rules for EACH execution target of the profile.  If you set the profile level property, the streaming profiler must modify the isolation rules (many) for each Target of the profile.  This means that if you have a profile with 4 execution targets and you're editing one of them - and you set the profile level property, behind the scenes, the profiler brings the other 3 execution targets into "edit state" to make the change and will eventually write all 4 targets back to the application hub.  Going to edit state to change the rules requires unzip of the can file from the network server onto the profiler machine.  If the profile/targets are large, this can be a very painful operation to accomplish a single boolean set, but this is how it is.  If you make this change, be aware of the large behind the scenes work that the profiler is doing.  Grummble yell a bit and then it will be done.

Fun with streaming - Great entertainment in isolation circles

Turn on the -x RadeRunSwitch so you can an isolated command prompt when you launch your next favorite streamed application.  This assumes you have user updates disabled, which is the default.

cd c:\windows\system32
c:\Windows\System32>notepad.exe
< it runs >

c:\Windows\System32>type notepad.exe
< see textual giberish - the file open succeeded for read access from CMD.exe >

c:\Windows\System32>copy notepad.exe n.exe
        1 file(s) copied.
< file copy was successful - n.exe is at the per-user layer of isolation >

c:\Windows\System32>type n.exe
< see textual giberish - the file open succeeded for read access from CMD.exe >

c:\Windows\System32>n.exe
The system cannot find the file c:\Windows\System32\n.exe.

FIREWORKS HERE!

The isolation system LIED to the Windows Loader - returning ERROR_FILE_NOT_FOUND (2) rather than completing the loaders request to run this file from user layer of isolation.  This is what this setting does!

But wait, there's more!

c:\Windows\System32>copy n.exe notepad.exe
        1 file(s) copied.

c:\Windows\System32>notepad.exe
< it runs!! >

Why does notepad.exe succeed in the final case?  Easy, there are two notepad.exes.  At the per-user layer, there's a notepad.exe which was written on the file copy from n.exe.  We don't care what this file is, but it is executable content and it exists at the per-user layer of isolation and therefore it doesn't exist for purposes of running programs.

Since the "Enable user updates" setting is set to disable user updates, executable content at the per-user layer of isolation does not exist from the perspective of the Windows loader.  BUT - at the physical layer, there does exist a file with that name and this can satisfy the file open, without violating the isolation rules.  There could also be a file with that name at the application installation image layer.  In this example there wasn't, but there could be.  The isolation system starts at the top and goes down until it finds a hit.  If "Enhanced security" is enabled, then the per-user layer is "off-limits" for execution of executable content.

The grand result

The application "update" applied by the user may have been applied as far as the user or application is concerned, but in reality, it was not applied.  The version of the application that is running is the version that the administrator profiled.  Pretty cool stuff.

Why did we rename the setting?

Putting "security" in the title implies that this will somehow prevent users from doing things to run content that they download and this is not what it does.  If the program updates itself, then this setting will block that content from being executed.  The setting can also block user installed additions to the program (plugins), depending on the location to which they were installed - was it included as an isolation rule during profiling?

Take a web browser for example, if the user downloads executable updates to the browser, this will be captured and the user installed stuff won't run, but if the user downloads evil.exe and places it on their desktop, and then double clicks it - this will be outside of isolation so the layers here do not apply.   This is also true if the user downloads evil stuff to locations outside of isolation and launches it from the isolated application.  It will still run isolated, but it will run!  Describing this activity as "disable user updates" is more accurate than the previous words, so we've made the change.  I also hope that it removes confusion in the streaming profiler wizard.  "Enable user updates" is pretty easy to understand.

How should you create your profiles

1) Enable user updates should generally be "off".  Plugins are a rare need and where there is a real need for users to add plugins, start asking yourself if you can add those plugins at profiling to the common layer.  OR, if the use of user installed executable content is large, should this application be locally installed rather than isolated?

2) Always tell the application to NEVER update itself at runtime.

 A look to the future

Streaming dev team are discussing removing this option from a future release.  That is, "Enable user updates" will always be OFF.  I'm not sure of all the ramifications of this yet.  The question really is "how many admins are profiling their applications with user installed updates permitted"?  I hope the number is "few".

Joe Nord

Product Architect - Application Streaming

Citrix Systems

To use agent-less streaming or installed agent streaming; that is the question.

I have received inquiries recently - lots of inquiries on the same question actually - that all imply that "user mode" streaming and isolation is "better" than evil kernel driver streaming and isolation, because it doesn't have any kernel components.  This sounds exciting on first glance, but when you dig down the problem is more complex.  There are advantages to installed isolation systems and there are advantages to all user mode.  This blog shows some examples for both categories. 

To get it started, consider an isolation system's ability to load and isolate a Windows NT Service.  How can you isolate the execution of an NT Service if the isolation system itself is "user privilege".   At user rights, the isolation system lacks the privilege to start a service that isn't already installed, so it can't run and isolate services unless the service is already installed outside of isolation.  

Scalability is also a big concern.  It doesn't matter for a small system, but if you load the machine up with lots of users and lots of applications, having a separate copy of the installation system for each application becomes problematic.  

On the plus side for agent-less, it is really nice to NEVER need admin rights on the execution machine and this can help engage work from home or kiosk environments.    Another plus for agentless, the "attack surface" is smaller if there are fewer privileged components; this is the point that usually starts the conversation that "agentless is better".   Keep reading...

First, some definitions:

Agent-less streaming

Nothing is "installed" into the machine.  The isolation engine can run the profiled software without ANY components themselves being installed.  "Admin" rights are never required and the system usually embeds the isolation system and profiled application into a single .EXE image that is "executed" to run the isolation system and the isolated application.

Examples of agent-less isolation systems include ThinApp, InstallFree, XenoCode.

Installed agent streaming

With an "installed agent", parts of the isolation system are "privileged", they require an install time execution which usually includes the installation of at least one kernel mode device driver and generally also includes the installation of a NT Service that supervises the sandboxes that are active on the machine. 

Examples of installed agent isolation systems include Citrix Application Streaming and Microsoft App-V.  Even this is a simplification as the Citrix system uses kernel mode for process monitoring and file system filtering and USER MODE for registry filtering, which makes it kind of a hybrid.  App-V uses kernel mode for both.   The real key is that an installation step was required.  If you require an "install", you are a member of the installed agent camp!

Which is better?

The answer is simple: Mine! 

The real answer is that you have to ask what "better" means.  I mean, better at what?  Each of the systems have their advantages.

Consider installation

Not needing to install an agent is handy for some environments especially where the execution machine is not company managed.  USB Thumb Drives were the original player here; you can take you application anywhere!  By contrast, if you're really running in a company environment, either stream to client, running on a XenApp Server or even on a XenDesktop hosted world, the installation of an agent is not a gate - the admin controls the base image and can install any agents that they want.   For more on this, see "security" below.

Consider scalability

If you are running a heavily loaded XenApp Server.  Let's say 64-bit machines, with 64-bit Operating System, a handful of high end CPUs and enough RAM memory to run 100 concurrent users, with each user running 5 distinct isolated applications.   How many isolation spaces is that?  100 * 5 = 500 isolation spaces.  If the isolation system is 100MB, that's 500 * 100MB = 50GB of allocated virtual memory just to load the isolation engines.   I made these numbers up, but stick with me on the concept.

How many separate copies of the isolation system do you want to load?  Answer: One would be nice.  

With kernel mode or NT Service isolation systems, you'll have ONE single code space for all of the isolation sandboxes, a bit of instance memory for each sandbox and other than that, you're off and going.

This too is a over simplification.  For starters, the Citrix App Streaaming case is a hybrid.  I haven't checked the memory footprint lately, but the kernel pices are ONE and the registry and named object pieces are MANY as these are implemented inside the isolation space.  To get back on subject for this post, think of it as ONE installation system because all "installed agent" systems are in the same camp.

Does agentless mean that there is really a separate isolation engine for each application?  yes and no.   We tend to generalize this to say that the isolation engine is carried along with the to be run application.

If the agentless isolation system uses DLL references to get to the isolation engine, then the agentless system's memory load will be page table shared across the images.  This a function of the Windows NT PE Loader and memory manager.  With DLL load of the runtime, many pages of the 500 instances of the isolation system will be shared.   I don't know if the clientless systems use DLLs for the agent runtime, but if they don't, they could and this will share the load.  

Citrix App Streaming uses DLLs for the registry filtering, so the shared model here still applies (ONE).  

How to update the agent

If the isolation agent needs to be updated, how many application images do you have to update?  This is a plus and a minus for both again.  With installed agents, you update one and all the profiles/sequences benefit.

With agentless, you have to touch each of the isolation images.  Or, back to the DLL approach, you could achieve the same ONE update in agentless if you're using DLL based runtime.   

If go with the "one app, one executable to distribute" model, then the isolation system is not shared and the memory usage on a XenApp hosted model will totally suck.  Prototypes will be great, but actual performance under load will be a heavy hit to single server scalibiltiy.  When it comes time to update the isolation agent, you'll have to touch all of the profiles to get things updated; or you can skip this as a plus toward not having to maintain anything once you profile it.

Consider security

There is a perception that if kernel mode components are involved, then it's less secure.  This is mostly a statement of "attack surface".  There are many privileged components in the machine and they are all candidates for attack.  The real headache with privileged components is that they have to be "installed", but the security aspects still apply and are real.  

If you're installed and you have power, then you can do powerful things.  The corollary is that if you are not installed and you don't have power, then you CANNOT do powerful things and this is as much a plus as a minus.

Consider that many agent-less streaming systems receive "yes" check-boxes in comparison matrices when discussing isolation of NT Services.  

Either you're USER MODE or you're not user mode, you can't claim to be both.  

A step back: NT Services are installed applications, with no GUI and no direct user interaction.  Services run on either a powerful system defined account or a named user account of install time specified rights. "LOCAL_SYSTEM" and "LOCAL_SERVICE" are the common installation configurations and these equal "powerful".

In many cases, the reason that the programmer went through the pain to write a service is that they needed to do something with privilege and that was impossible from the user privilege application space.  This is why they are done in the service rather than in the application itself.

If the isolation system is agentless (user mode), and if that isolation system can load and isolate NT Services - and have that service work - then a boundary has been passed and the isolation system is no longer user mode.  Given that the agent itself would not be privileged, how can the applications that it runs be privileged?

Answer: The agentless streaming system requires the applicaiton services to be installed outside of isolation, and by installed, I mean REALLY INSTALLED.   This pretty much deletes the check-box for isolation of services.

Some NT Services likely CAN tollerate user mode execution under isolation, but for the general discussion, the answer will be that this breaks down and the service requires installation to the local machine - breaking the isolation boundary between the application and the local machine.
An "installed agent" isolation system CAN have the power to start services itself, and in this environment CAN run the service isolated.  This is only possible because the agent itself is INSTALLED and has power.  If the isolation agent is not installed, then the user cannot start isolated services and there's a pretty big gap in the claimed support of isolated services.  The services have to be outside of isolation and installed - they aren't isolated and the access to that service is not governed by the published application set.

Consider application launch performance 

Now that Application Streaming 5.2 client is out (real-soon), I can describe the greatness of 2nd time application launch.  If a sandbox is running that will support this app, there is no longer ANY need to create an isolation space for a second, third or fourth application launch.   Creating an isolation space/sandbox/bubble is an expensive operation.  The 5.2 client's ability to skip this expensive operation will provide great benefits in launch speed.  I describe this briefly here, but need to write more.   

Can agentless do things to equal this application launch performance?  Can it toss execution over the wall to already running isolation spaces?  Probably, but as an big executable based execution, by the time they run to make this decision, they will already have the isolation system mapped into memory and it's big and I'm betting, slow.  Bottom line, I'm looking foward to a new round of statistics with the 5.2 client - we should kick some major butt!  "Agent based" here has advantages, but this alone will not sway a discussion.

Consider central management
Both Citrix Application Streaming and Microsoft App-V are heavily predicated on the concept of communication with a back-end infrastructure and administrator driven management of the applications available to users and even preventing the execution of non-approved applications. 

The applications are published to users and in App-V case, users or machines and both systems communicate information back to the central authority to decribe the use of applications.   App-V can even block the launch of applications based on license metering.  All of this is enabled because of communication with back end, from an installed agent.  

Can that communication be done from an agentless system?  Probably.  Is this done?  I'm not sure.  If it is done, is that something you really want happening from a user privilege component?

What applications are available to the user and how can I trigger the update of application content?  What applications have been actively used across my whole organization and which ones are published that the users really don't care about?  What applications should I focus my support on and which ones should I deprovision without telling anyone?  Do I have enough application licenses?  

All of these questions can be answered with back end information.  Agent-based makes this easier.   Having a NT Service hanging out to collect this information and centrally report back statistics is an opportunity for central management, control and monitoring of applications.

Then again - agent-less can get most of this too. 

For example, Citrix EdgeSight monitors application usage on a machine and reports this back to make truly beautiful reports that tell the admin what has been happening on their machines.  It doesn't matter if the applications are isolated or not isolated, the EdgeSight monitoring system still sees them and can report on usage.   This happens for Citrix Application Streaming and can just as easily occur for clientless based isolation system.   

Conclusion 

Is agent-less better or is agent based better?  The answer really depends on how the whole system will be architected and what control the administrator has on where the agent will be installed.   Both have advantages.

Joe Nord

Product Architect of Citrix Application Streaming (An agent based isolation system)

Citrix Systems - Fort Lauderdale, FL

Have you ever looked into the App Streaming execution cache and said, "good grief!  What are all those GUIDs!"?  GUIDs are great, for the programmer.  For the administrator, they can be a bit more mysterious. 

I recently wrote a utility to decode the GUID data of the Application Streaming execution cache, displaying the decoded information to the console, but neater than TTY, it also updates the Windows Explorer display of that space so the "secret" information is exposed in permanent fashion and via the GUI, which is good stuff.   The utility is released to the Citrix Developer Network, in source code form so you can tweak it if you'd like. 

Download from here, look for the "Rade Cache Decode" link at the bottom of the page.

For background, I have written about the GUIDs before and how they are key to the streaming system maintaining the separation between "profiles" that you can publish and execution "targets" that the streaming system uses to actually use to run stuff.   There is more on this in this post.

Public credit

The idea for this tool actually came from very smart people at Sepago, Sascha Juch and Helge Klein, who I work with on Citrix Profile Management (the artist formerly known as Sepago Profile).   They also clarified for me that despite MSDN documentation claiming "system" attribute is necessary for this explorer magic to work, marking the directories read-only gets the job done.  Tested and true!  Thank you guys.

Update: Dieter Schmitz at Sepago is the gentleman who came up with the idea.  Thank you Dieter.

What does RadeCacheDecode do?

Application Streaming can store one or more execution Targets for each App Streaming profile. This level of indirection allows a single set of "Applications" and icons to be published even if more than one execution image is required.  This is common for a profile that supports both 32-bit and 64-bit operating systems. While the application is profiled twice to create two targets, there is only ONE profile to publish and only one set of "applications" from an Access Management Console perspective.  

A necessary consequence of this is that the execution targets have names that don't necessarily have any resemblance to the name of the profile from which they came.   Targets can be associated with execution systems for a variety of configurable options and these can change from version to version of the profile. 

That is, just because a specific target was right for Windows XP German yesterday doesn't mean that it won't also be right for English AND German tomorrow.   Even ignoring "one to many" advantages of execution targets to profiles, the streaming system STILL would use some abstract term to keep track of the execution targets. 

If you're editing a profile in the Streaming Profiler and you SAVE, all the identifiers of the execution units remain unchanged.  BUT - If you Save-As, you'll get new identifiers.   "By spec", the thing that was "saved as" is a New profile and will mature separately from the original - even if the before and after profiles have the same name.  Putting it a bit differently: Two profiles with the same name are not necessarily the same profile, the unique identifiers in the profile define the uniqueness of the execution units and in theory, or the profile itself.  The studious reader will see below where at least parts of this break down.The short version: Thanks to the use of GUIDs, the streaming cache manager will never confuse two execution targets, even if they come from different profiles that have the same name.

*What about the administrator?*GUIDs are there just to make the RadeCache space hard to digest.  After enough time looking at the Citrix internal showcase farm, you get used to the idea that the execution Target that starts with "1B3" is the MS Office 2007 profile execution target that is used for Windows Vista and Windows 7.  Type 1b3, hit tab and you get the right one.  Not ideal.  Here's what it looks like in the Windows explorer.

Okay, we know that 1B3* is the MS Office 2007 execution target for Windows Vista and Windows 7.  What are the other 5 execution targets? 

What you need is something to look up each GUID and to TELL YOU which profile it came from.  Yes, it would be BETTER if the streaming client itself had left you a clue, but it doesn't.  Notice that when the streaming client created these directories it KNEW what profile it came from and there can be at most ONE profile in the entire globe that has that GUID.  Did it leave you a hint, a small hint?  NOPE!.

Ahh, the fun life of being a XenApp adminstrator. 

HELP IS ON THE WAY!

The location of the application hub is generally "known" to the admin, but to know which GUID is associated with which profile, takes some work.  Once the running application terminates, not even the streaming client knows where the GUID came from.  It's just a directory that exists in the right place, but the link that says that the "1b3" target is somehow related to MS Office is gone, there's nothing there.  

Making it easier

For offline execution, the streaming system stores information that provides the secret decoder information required to convert GUIDs to profiles.  The Deploy space holds a copy of the Application Hub content and stores it in a place that has a fixed location (registry configurable and readable from the utility).  The RadeCacheDecode utility looks at the Deploy space, sees what execution targets exist for which profiles and uses this information to "decode" the GUID entries in the RadeCache.  This should make more sense by example.

Command line execution (readonly)

If RadeCacheDecode is run with the -r (readonly) switch, the utility processes the subdirectories of the RadeCache space, and for each tries to figure out what Deployed profile goes with that execution target.

Example output:RadeCacheDecode -r092b3450-e543-4541-837a-c374cc4e73cc_1 (GoView1067)

1b345768-25fd-45d4-b689-f3984f9221ee_5 (Office20071067)

274bd686-a305-45d9-a479-a127338586b1_1 (project20071067)

4d971270-cbac-4366-8c4a-b11f6f867a58_1 (AdditionalOffice1067)

76dc5736-0c5d-4600-a7e9-a3a41c3b5c51_6 (-not-deployed-)

a9984581-0e41-44b6-861f-b6d748dacb93_1 (Visio20071067)

Okay, we have now decoded the 4 of the 5 remaining execution Targets on my notebook.

Making it really useful!

Consider the technique that the Windows Explorer uses change each user's "Documents" directory into "My Documents".  This same activity can be used to permanently improve the readability of the RadeCache space. 

When thinking this was a good idea, I tried it out by manually creating files and while I was initially intrigued, my first conclusion was that it had negatives that outweighed the positives.  If you read the MSDN docs on the desktop.ini, it leads you to believe that the directories have to be marked "System" for this Desktop.ini stuff to work.  It isn't so.  Marking the directory readonly will do the job and with that, we have a solution!  Why is "system" bad?  Answer: it makes directories disappear from normal directory searches 

"readonly" works, so we have a solution.

This time, run the utility without the -r (readonly) switch and it will WRITE to the RadeCache space to mark all of it's subdirectories with "decoded" versions of the GUIDs.  The directories are marked readonly so the desktop.ini stuff comes into play with Windows Explorer and a file named "desktop.ini" is added to each execution Target's directory to create a "friendly name" to go with the mysterious true GUID_ver name.

The end result is that the next time the Windows Explorer browses to this space, you will see a nice version of the GUIDs - with the name of the profile placed in parens after the GUID. Sometimes it is necessary to force a refresh in Windows explorer to get it to reload the friendly names.

Here's a picture of the output

We're making good progress.  It is much easier to see these profile hinting entries than just plain GUIDs.

How does it work?

The entry above that says "not deployed" is the hint.  Since the Deploy space is at a known location, the decoder utility KNOWS where to look up the execution target GUIDs.  While the streaming client doesn't know what guid goes with what once the applications terminate, the DEPLOY space knows!  The Application Hub also knows, but unfortunately, the decoder utility doesn't know where THE (think plural) Application Hubs are located.  The Deploy space is fixed and given large offline streaming usage, this becomes pretty easy to inspect.

The utility loops through the subdirectory space of RadeCache.  For all directories that look like a GUID_v entry, it considers them an execution target and then goes looking for GUID_v content in the Deploy space.  When it finds a hit, the name of the containing directory in the Deploy space tells the decoder utility the name of the profile from which the GUID_v came.

Awesome!  Works good.

What's wrong with this picture

The studious reader will notice that the Deploy space is named based on the NAME of the profile that is deployed.  The Decode utility leverages this fact to figure out what profile a given GUID came from.

This naming in the Deploy space violates the premise that two profiles with the same name, but with different GUIDs are different profiles!  Dig into the internals of a .profile file (it is XML) and you will see that the profile also has a GUID.  GUIDs aren't just for guaranteeing uniqueness of the execution targets, they also guarantee the uniqueness of the profile layer.

Good news: the caching system is correctly using GUIDs and will "never" get confused.

The Deploy space SHOULD be done based on the GUID of the profile and all the sub-directories below the Deploy space SHOULD be GUIDs (with no version).  They SHOULD be, but they AREN'T and this means that the premise of two profiles with the same name, but different GUIDs are different is somewhat not precisely correct when considering offline content. 

In reality, this doesn't come up because admins tend to stick all of their profiles onto a single Application Hub and this means that the name of the profile is a unique key.  Still, know that this isn't right and know that I've been chasing a "fix" for this for over 2 years - so far, with no fruit.  The App Hub side is fine based on name.  The Deploy space should be GUID based.  Okay, I'm off my soapbox.

Enhancements

Good enhancements to RadeCacheDecode would expand the profile search to go beyond the Deploy space and also consult the network Application Hub.   The utility should also use the COM based SDK whose webpage it is posted in.  That is, the utility looks in subdirectories of the Deploy space to figure out which profile holds a file with the right name.  This breaks the separation of the profile as a programatic thing from looking behind the curtain.  I could have coded it that way, but it just seemed like a lot of work to open up a bunch of XML files to peek for GUIDs, when all I wanted to know was presence (is it there, or not?).

The "ultimate" enhancement to RadeCacheDecode is to change the streaming service itself so that when it creates RadeCache directories, it creates the desktop.ini file directly.  As of Streming Client version 5.2 (XenApp 5 Feature Pack 2), this does not happen - but now we know how...

To achieve almost the same thing, you could run RadeCacheDecode on a timer via some automated means.  It needs to be run with power so it can see and write to the RadeCache space, but if you do this, your desktop.ini entries should always be pretty much correct.  I coded the utility to WRITE only if the existing stuff is not "correct", so you can run it repeatably and the desktop.ini files will only be written occasionally.  This was probably overkill.

If you find it useful, let me know...

Joe Nord

Citrix Systems Product Architect - Application Streaming

The first releases of Citrix Application Streaming had isolation on the brain.  If you don't know what else to do, ISOLATE!  Don't let the app hit anything that might be important.  If you don't know if it's important or not, then it's important!  Protect it! 

This over protective behavior is being relaxed, the first step was in the XenApp XenApp 5.0 (Delaware) release where streaming client 1.2 was released which changed file system isolation to auto-ignore non-boot disk drives and soon, in the XenApp 5 Feature Pack 2 release where the version 5.2 streaming profiler and client change the default file system rule from "isolate" to "ignore".

Isolation rules review

When a user stores documents to disk under Application Streaming, the isolation system inspects the file operation to decide whether that operation should be isolated, ignored or redirected.  Details on each of these can be found here in the Citrix documentation library and here in my previous blogs.  All rules are named from the perspective of the streaming system, so "ignore" means don't change it - let it go to the file system without modification.

What isn't obvious, is what the isolation system does with disk volumes beyond the boot volume.  This behavior changed in streaming profiler version 1.2 (XenApp Delaware) so that the over paranoid behavior of the original Tarpon technology would instead leave users data alone.  Notice that I'm assuming users put data on drive D: while the OS is on drive C:.  This change has been beneficial to most customers and really helps engage the desktop, though I have seen a few that needed this to not occur and this oversight will soon be corrected to be backwardly consistent - for profiles created at the 1.1 level. 

To be clear, space beneath the user profile is and always has been "ignored", which means the normal places for users to store documents have no isolation collisions.  It's the non-normal places that require attention.

In App Streaming 5.2 which isn't out yet, the isolation will be further relaxed.

In addition to ignoring non-boot disk volumes, the isolation system will change it's default rule from "isolate" to "ignore".  Notice that this will happen only for NEW profiles.  If you have existing profiles, their behavior is already defined.  

Example user of Citrix showcase internal farm

Consider an example user, we will call him "Nabeel", because - well, that is his real name.  Nabeel is a Citrix executive and he travels a lot.  When he travels, he does lots of presentations and ... uses Citrix Application Streaming to run MS Office 2007.

In the example case, he was on a 3 week trip to Asia where he visited lots of Citrix sites and lots of customers.  With the magic of Application Streaming, he was able to use MS PowerPoint "offline" on the airplane to refine his presentations and also update the PPTs throughout the journey.  All of this worked super! 

Presentations updated all over the world, presented all over the world.  Everything was well received, and then he came home....

When he returned to Fort Lauderdale, he used Windows explorer to zip up all the files from the 3 weeks of presentations and e-mailed them out to the people he had met.  The receiving people noted that the presentations were mere skeletons.  CONTENT FREE if you will.  Everything worked on for many weeks was "as it was" 5 weeks earlier before he put Application Streaming onto the notebook.  OOOPS!

My phone rings...

What happened?  Answer: the files were all stored to the \Citrix2007 directory and since this is not a space that is "known safe", the isolation system isolated it.  From the view of the powerpoint application under isolation, it saw the correct and current version of the document.  But look at the real disk and the files were their original selves.   BAD.

Step 1: Where did the good files go?

That one is easy:

Consider the layers of glass.  The answer is that the file is in the top layer of the layers of isolation.  Find it in  %APPDATA%\Citrix\RadeCache\GUID\Device\C\Citrix2007

Step 2: How to make this not happen again...

Harder.  Insufficient answer is to add an ignore rule for Citrix2007 to the profile.  No good.

Not a problem for stream to server

Before going deeper, it should be said that this problem never comes up for stream to server.  In a server environment, "users" do not have the power to create directories off of the ROOT.  Users can store documents to %USERPROFILE% and that's about it.  Server side, this problem doesn't exist.

Stream to desktop

Client side, "users" tend to be "administrators" and this creates new problems.  "Users" like storing stuff to folders off of the root.  This makes it really easy to know what you need to backup.  

The isolation system not knowing how to handle the \Citrix2007 directory isolated the operations and this is far too over protective.

App Streaming 5.2 (Yellowtail)

In the upcoming release, the default file system rule for new profiles is changed from "isolate" to "ignore".  The default rule set then includes \Windows, \Program files and similar "important" directories as places that should be isolated at runtime.  I'll add that the profiling time rules and the execution time rules have to vary a bit to make this occur.  The streaming system takes care of this automatically.  Profiling time remains paranoid by default, isolating most everything.  Runtime becomes much more relaxed - making the isolated application execution more consistent in behavior with locally installed, while maintaining "protection" of spaces that the application should not be allowed to write per the layers of glass.

File system permissions still apply

If the user tries to write to a space that they aren't allowed, the file system permissions remain in place.  A user will only be able to write to or read from the real \Citrix2007 if they have DACLs in place that make this permissible.  This is a file system statement, not a isolation system.  In the "ignore" case, what happens is that the isolation system sends the I/O operation down to the file system without change and the file system will then decide if the user is allowed to access those files. 

This change greatly improves the offline streaming experience.

Joe Nord - Citrix Systems Product Architect - Application Streaming

Learn more about Citrix XenApp 5 Feature Pack 2

• Official Press Release - http://citrix.com/English/NE/news/news.asp?newsID=1857726
• XenApp 5 Feature Pack 2 Web Site - http://citrix.com/xenapp/featurepack2
• XenApp 5 Feature Pack 2 Executive Video - http://citrix.com/xenapp/fp2/video
• XenApp 5 Feature Pack 2 Webinar - http://citrix.com/xenapp/fp2/techtalk
• XenApp feature matrix by platform, version and edition - http://citrix.com/xenapp/comparativematrix
• XenApp Expert Series videos for this release - http://citrix.com/xenapp/fp2/expertseries
• XenApp 5 Feature Pack 2 Blogs- http://community.citrix.com/blogs/tag/xa5fp2
• Download XenApp technology previews - http://citrix.com/xenapp/techpreviews
• XenApp Product Page - http://citrix.com/xenapp/

Follow XenApp on | | |

I received an interesting inquiry recently.  Given my post on Never Deploy to XenApp Servers, how do you stop deploy from happening for hosted desktop?   Excellent question!

Here's the situation.  Happy XenApp Customer (they are all happy) who uses Application Streaming to deliver applications to their XenApp servers as well as stream to client, primarily for offline access.  The applications are published to the user as offline enabled and this works swell to trigger the deploy operation that is required to execute the application when disconnected from the corporate network and also works swell to not trigger Deploy in the XenApp hosted case.

Quick refresher on Deploy

The Deploy space is a COPY of all the needed stuff from the Application Hub, but it is local to the execution machine.  The Deploy space makes it possible for the streaming client to support application usage even when disconnected from the network.  The Deploy space is generally in \Program files\Citrix\Deploy and the execution cache space is generally in \Program files\Citrix\RadeCache. 

At runtime, the streaming client runtime fills the execution space as needed by copying content from the Application Hub - or if Deployed, from the Deploy space.  Other than one being remote and the other being local, the activity of the client is the same whether "offline" or "online".  Technically, "Deploy" enables streaming from one side of the hard disk to the other and this keeps life pretty simple from a programming standpoint.   I thought it up back in the Tarpon development and I'm still pretty happy with myself.

Deploy is automatically not done on XenApp Servers

It may seem like a good idea to Deploy to the XenApp Servers, but I have already squashed that myth and apparently did a good job there because the corollary, "Never Deploy to XenDesktop" has also been heard.

Back to the customer.  They have XenApp running good and are engaging XenDesktop.  Application Streaming is delivering the applications to the XenApp servers as well as to the end user machines (notebooks). 

Here's the thing that's not always obvious.  The streaming client KNOWS that Deploying to XenApp Servers is a bad idea.  If you publish an application to a bunch of users and offline enable the applications AND if you publish the applications to hosted execution, a single user can use both without triggering a Deploy on the XenApp server.  

If the user is running the application server side, the streaming client is still involved but it knows that it is running ON a XenApp Server, so it SKIPS the Deploy.  This is the right thing to do.  If the admin doesn't like that, they can still command a Deploy to occur on the server by directly calling the deploy utility, RadeDeploy.exe, but again, you shouldn't do that.

The problem

The customer is now engaging XenDesktop and have connected the hosted desktops to the App Streaming infrastructure from the existing XenApp configuration.  Here, the streaming client is running on the hosted desktop and ... when performing its app launch concludes that this is a client side execution and background copies down all the bits for offline execution. This is bad.

Here's a good write up on the percentage of space that is used for online vs. offline execution.

The point: Lots of disk writes start happening to copy the Deploy content down from the Application Hub to place them in the Deploy space where they will exist and get copied again into the RadeCache.  All of this is bad to the layers of cake.  We want to minimize disk WRITES.  Disk writes are backed up either by Xen/Other Virtual Machine manager or by Provisioning Services.  Either way, large writes by large numbers or users are bad.  Beyond being slow, these writes will occupy space in the per-user write back cache; which is discarded on logoff, so this whole thing will repeat on the next logon.  

The solution

The Deploy should not occur!  But it does.  In a "correct" world, the streaming client would be XenDestop aware as it is XenApp aware.  If running "hosted", don't Deploy!  We do not live in a correct world.  My initial guidance to the folks that asked me about this was "erase \program files\citrix\streaming client\radedeploy.exe".  All happy with myself because I KNOW that the streaming client shells to the utility to perform the deploy.

With the utility "missing", the streaming client will have no choice but to push on without the deploy.
Survey says.... didn't work. 

Instead of "pushing on", the streaming client declared failure and aborted the app launch.  Aargh.

Beautiful part of this is that I didn't have to actually try it.  Smart folks on the other side, all I had to do was keep coming up with ideas.  

Second round: Replace RadeDeploy.exe with a program that does nothing, but returns "success" return code.

Survey says.... Success!

The only utility I had handy to do this is one I wrote a very long time ago to use for remarking out RUN= statements from the Windows registry.  Source is included - please compile it yourself before putting in production.  For the programmers in the room, said program is "Hello world" minus the printf.  In the common image where the streaming client is installed, copy nop.exe to RadeDeploy.exe. 

Back to the Deploy activity

The streaming client "shells" to RadeDeploy.exe which is really "nop.exe", which returns 0 as RC.  With the Deploy "successful", the launch proceeds with no error, AND the Deploy never happens.  Perfect!

I see a future where the streaming client will be XenDesktop aware and skip the Deploy.  Until then, this technique may prove useful.  If you have other creative solutions to this problem, please share them here.

Next step is getting the RadeCache space available to everyone from a single source, read-only and fully populated at the get-go.

Joe Nord
Product Architect Application Streaming

Citrix Systems

The Citrix Application Streaming client and profiler version 5.2 are available in Beta form.  Here is a link.    MyCitrix credentials required to download.  The Streaming Client is "Offline Plug-in version 5.2". The Streaming Profiler is "Streaming Profiler version 5.2".  

These are not production, but they are pretty close to what will ship with the next XenApp release.  I note that the Streaming Client/Profiler 5.2 can be used with everything from Presentation Server 4.5 to XenApp next release.

Please give it a whirl.  The normal tech support forums should be used to share your positive experiences.

The 5.2 client will "consume" profiles created by all previous streaming profilers and the performance improvements noted below will be seen with no need to reprofile.  That said, there has been a years worth of changes in the profiler, so there can be other reasons to move up.

Things you will see:

• Second time application performance vastly improved.  Feature: "Sandbox reuse".  In the past, launching two applications from a single profile actually created two sandboxes even though those sandboxes could be considered one as they both had the same definition of isolation scope.  Starting a sandbox is a long activity, that is now skipped for second application launches, making second app launch very similar in time to locally installed.  This change was the focus of the release and should produce very happy users. 
• Sandboxes are left alive for 5 minutes after the last application terminates, improving odds for 2nd time launch compared to first.  Registry configurable period, in minutes.  Might change this to Ms.
• Integration with the Citrix Receiver to operate as a plug-in, or as a separately installed program.  This function first shipped in 1.3 which was a streaming client shipped asynchronous to XenApp releases.
• AIE* environment variables are now visible from pre-launch/post-exit scripts run outside of isolation.  This enables things like purging the per-user isolation space before/after application usage, without using RadeRunSwitches.  Some other issues with scripts were also fixed.
• A years worth of App Compatibility improvements.

The version number will be 5.2.  If you've seen my posts before, you know that I never intended to move away from 1.x, including shipping major new function in point release, like Inter-Isolation Communication at release 1.2.  Long story as to why, but the bottom line is that the streaming client after 1.3 will be 5.2.

Enjoy, 

Joe Nord

Product Architect - Application Streaming

Citrix Systems

I have previously written about how the Application Streaming client prunes the file system cache.  What was not discussed is how the streaming system manages the REGISTRY portion of the execution content.   This post brings attention to that need and provides a reference to a source code application that can assist with pruning the registry portion of the execution cache.

When the streaming system executes an application, it can run more than one version of the application at the same time.  New versions of the app can be used by user B when user A is still using the old version.  This way, the admin can update an application "hot" and when users closes and restart the application, they get the new version.  This covered in some details in the discussion of RADE.

What is not obvious is that while the streaming system file system cache has a high water/low water algorithm for managing the cache, it has no such algorithm for the REGISTRY content.  Peek into HKLM at the RadeCache space and you can potentially see registry content supporting MANY versions of old applications.   

Here's the execution space on the machine I am using to write this post.
Notice in my case, I'm on version "5" of the execution Target 1b3... which happens to be Microsoft Office 2007, the Vista/Win 7 execution target as published in the internal "showcase" at Citrix.  Also notice that there is no "4" because I have a strange habbit of formatting my primary development box once a month whether it needs it or not and the admin hasn't updated the profile since my last reload.  I also know that our showcase admin occasionally does admin magic to delete these old spaces, but that is getting away from the purpose of this post.

Real customers

Heard from the field a couple months ago of a customer who had taken RADE to true heart; updating applications almost daily and then streaming them to server.  After a few months, they were in the hundreds in the versioning, and while this worked fine, the registry had become polluted with all the old versions.  Notice that the file system was fine, the old versions had aged themselves out of the cache and been erased, but the registry was littered with stuff that should not be there.

Utopia

One can effectively and easily argue that the Streaming Service should observe application updates and when nobody remains using the old versions, it could happily perform house cleaning to purge out the not needed information from the installation image registry.  This would be a very good argument and I'd stand up and say, you are right, which is what happened with this customer.  Things like Rollback to earlier versions makes this more complicated, but in the general sense, managing this registry space should be no different than managing file system space.  It is more difficult though because for file system, there's an advantage that anything you erase by mistake, will automatically be put back if it is needed.   For registry, it's populated once and left alone.

To "solve" the problem, I coded a little utility to prune the registry cache.  It runs outside the confines of the streaming service and defaults to "keep 2", to enable rollback.  The SOURCE code for this utility has been posted to the Citrix Developer Network, MyCitrix account required to get in.

One can imagine a day where this function is part of the streaming service.  I imagine that day too.  Until utopia arrives, if you need to programatically prune the registry cache, this utiltiy can help.  Set it up to run once a day/week and it will keep your registry space clean.  The UI is pretty primitive, it shows to stdout a before and after record of all of the space in the RadeCache along with indicator of stuff that it erased.

What it doesn't do

The utility scans the registry content, observes all of the cache space and then recursively deletes the "old" entries.  What it doesn't do is know which GUIDs are orphans, supporting applications that are no longer published.  Interestingly, the streaming service also doesn't know which entries are no longer used.  It would be a worthy addition to also reference the "last access time" information and purge items that are over a limit.  Here, shelling to the RadeCache utility to perform the erase would be the correct action so that the official tool could be used to purge our the execution cache, both file system and registry.

Enjoy, 

Joe Nord

Product Architect - Application Streaming

Citrix Systems