As part of the load testing for an audio streaming solution, I’ve tested out Web Gardening. My candidate is a beefy quad-core Xenon based server.
The audio streaming services run under IIS. Their job is to locate an audio file and deliver it in an acceptable format to the user. Some of our audio is in a compressed wave format, but most is in a proprietary encoding.
Our approach is to deliver this as a WMA format to the user, which requires two conversions. One from the proprietary format to PCM Wave format. Then we pass it through a windows media encoder to deliver it as a WMA format. As you can imagine, this is quite CPU intensive.
I started by load testing the services on the test server using a tool called TinyGet. I set up some batch files that simultaneously downloaded files from the server. Initial tests showed the server peaking at about 40 streams.
Next, I configured IIS to use “web gardening”. In IIS 6.0, this is managed at the App Pool level by increasing the max number of worker processes. For my testing, I increased this to 3 processes. This allows IIS and ASP.NET start with one process for that App Pool, but increase to 3 processes as the load increased.
The, I configured the App Pool to have a specific processor affinity. This requires editing the IIS metabase via the command line, or with the IIS resource kit’s Metabase Explorer. I set the App Pool’s SMPAffinitized property to true. Then I set the App Pool’s SMPProcessorAffinityMask to 0xE. This tells the app pool that all processes are to run on processors 1, 2, and 3. But not on processor 0.
After these settings changes (and restarting IIS), I re-ran the tests. The server responded very nicely. I was able to submit more simultaneous requests, even though the server was only using 3 of the 4 cores. Max load on this app pool was around 100 streams.
Multiple Service Levels and Isolating Apps:
The next thing I did was to create a new App Pool in IIS and a new Web Site that used that App Pool. These were exact copies of the existing App Pool and Site with two differences. It listened on a different TCP port, and it had a different processor affinity setting.
- The alternate port lets me determine which site to hit.
- The processor affinity on this second app pool was set to use processors 0, 1, and 2. But not processor 3.
So, what does this enable? Great stuff!
The second app pool has processor 0 all to itself. This means that users could max out the first app pool to the point that request are queuing up and playback is buffering. But, the second site is still very responsive. In the case of the audio testing, I was able to point Media Player at a file via the second site and have it start immediately. If I were to point it at the first site, while it’s under load, I’d have to wait minutes before it started.
So, there are some side-affects of using web gardening (multiple processes):
- Each process maintains it’s own copy of cached data.
- You must move to an out-of-process session state store.
- Each process has a little overhead, so it does consume additional resources.
From what I’m seeing so far, web gardening can be really useful in cases where you have a CPU intensive application running on a multi-core or multi-processor box. In these cases, it seems that the OS is able to support more load than running all of that in a single process.
I haven’t found many drawbacks to processor affinity yet. In cases of heavy load, it seems to make the server much more responsive by keeping one or more CPUs free for OS tasks and other applications.
Mileage will vary depending on the number or cores / processors in the machine. If you only have two, then you’ll cut your resource in half.
Setting affinity does require changes to the metabase, which must be done with the admin scripts, or with the Metabase Explorer.
Overall, web gardening and processor affinity are very nice “knobs” to adjust the way a processor intensive application runs on the server. It can:
- Isolate a resource intensive app away from other applications.
- Allow for multiple service levels on the same machine.
- This makes it possible to guarantee that certain users will have priority over other users. This could be important in the case of an application that is shared by external clients, account staff, call centers and verifications. Each group might require a different service level.
- Increase the throughput on a server where the application is CPU intensive.