IIS Insider: September 2006
By Bernard Cheah,
IIS Insider is a monthly column designed to answer your questions on how to troubleshoot and make the most of Microsoft Internet Information Services (IIS).
The example companies, organizations, products, domain names, e-mail addresses, logos, people, places, and events depicted herein are fictitious. No association with any real companies, organizations, products, domain names, e-mail addresses, logos, persons, places, or events are intended or should be inferred.
|
|
Smart Host Authentication
|
|
|
FTP Login Error - Home Directory Inaccessible
|
|
|
Finding URL with Long Processing Time
|
Smart Host Authentication
Q: I'm using Smart Host to relay all messages to a dedicated Exchange server. The Exchange server is outside of the DMZ and it required authentication before accepting any mail for relay. What can I do?
A: Good question. Before we look at the solution, let's understand more about Smart Host. what is Smart Host?
As per Microsoft documentation: Smart Host - You can route all outgoing messages for remote domains through a smart host instead of sending them directly to the domain. This enables you to route messages over a connection that may be more direct or less costly than other routes. The smart host is similar to the route domain option for remote domains. The difference is that once a smart host is designated, all outgoing messages are routed to that server. With a route domain, only messages for the remote domain are routed to a specific server.
Next, It is also important to understand that there are two types of connection authentications when it comes to SMTP requests: inbound and outbound. Inbound refers to when a client authenticates itself to the local SMTP server before the client is able to relay mail messages to local SMTP server. While outbound authentication refers to the local SMTP server authenticating with the remote SMTP server before the remote SMTP server accepts any of its message. Relaying message via Smart Host is considered an outbound connection - and that's why you would need to configure outbound authentication.
To correctly set this up on IIS 6.0, here's what you do:
- 1. Open IIS MMC.
- 2. Right click on the SMTP Virtual Server (the one that forwards to Smart Host), select Properties.
- 3. Click on ‘Delivery' Tab, then click on the ‘Outbound Security' button
- 4. What you do now depends on your Smart Host setup. If it's Microsoft Exchange and within an Active Directory domain, you can configure ‘Integrated Windows Authentication' to prevent clear text authentication. In my case, it was to iMail server, hence I can only depend on ‘Basic Authentication'. Of course, to secure the communication, you can apply TLS encryption with SSL certificate.
- 5. You select the relevant radio button for desired authentication scheme and enter the logon user credentials. In my case - as shown in Figure 1.1 - that will be ‘relayuser@mydomain.com'.
- 6. Click ‘OK' twice to apply the changes.
- 7. Restart IIS SMTP service.
To monitor whether or not the authentication takes place, look at the IIS SMTP log file. The default is ‘Anonymous' access, meaning no authentication is needed for outbound connection. With outbound authentication enabled, IIS SMTP will authenticate with the Smart Host before forwarding mail messages to the server.
FTP Login Error - Home Directory Inaccessible
Q: I have been running FTP user isolation with AD (Active Directory) integration for more than a year, and now suddenly all users can't login and they get ‘home directory inaccessible' error message. Please help!
A: Typically, this error is related to NTFS permissions not being configured properly. This is common - as the error message appears after the logon event (which technically means that the user is logged on but can not proceed to its home directory). Again, please verify that the user directory has the correct ACLs and that the user does have permissions on the folder. You can also get Filemon from www.sysinternals.com to help trace the permission-related issue.
If you have triple-verified the permissions setting and found nothing wrong, I would suggest you review your server log book (if you have one) regarding recent server changes, You can check against the IIS FTP log file to find out when the problem first occurred. If you have enabled security auditing to capture the logon event, you can check the event log and see if you can find anything related event error and then further troubleshoot from there. Enabling logon security auditing is part good security practices, more detail at ‘Enabling Security Auditing'.
Finally, I remembered once I helped a poster with same error message like yours. We exchanged couple of posts, and figure that it was due to the AD integration connection user account - this is the account FTP uses to access Active Directory information. You need to specify this account when using the FTP site creation wizard.
In the above case, the FTP site was running, but the FTP AD user account password had been changed. In this case, you would still see the same exact error message, but it makes little sense because it gives you no hints about the error. How we found out is that I asked if there were any recent changes, and he recalled that the AD account password has been changed! That's how I learned about this - and I believed you would see more information in the event logs indicating that user failed to log on, could not access home directory as well as logon failed due to bad username or password.
Before I could give the poster some advice, he came up with his own solution: delete the FTP site and recreate a new site!
Ok, what he did was drastic - there was no need to recreate the entire site. It didn't seem that way to him,.
However, since once the ftp site was created, there was no way he could change the AD connection account or password.
Well, that's true. The IIS MMC - FTP UI does not support such change, but remember that every setting in IIS is stored in a metabase. So you can change it at metabase level. Here are two metabase keys which control the FTP AD configuration:
ADConnectionsUserName and ADConnectionsPassword
As usual, you will configure the keys via ADSUtil.vbs
When you read the value:
C:\Inetpub\AdminScripts>adsutil.vbs get msftpsvc/XX/adconnectionsusername
Where XX is the FTP site ID and you get back the value as:
Adconnectionsusername : (String) "myAD\ftpuser"
Now, the actual steps to reset the AD connection user password:
- 1) Go to the command prompt.
- 2) Navigate to AdminScripts folder, for example "cd\Inetpub\AdminScripts"
- 3) Enter the following command:
C:\Inetpub\AdminScripts>adsutil.vbs set msftpsvc/1735/adconnectionspassword myftppwd2111
Adconnectionspassword : (String) "**********"
The above command will reset the FTP Site id 1735 with new AD user password as "myftppwd2111".
- 4) Next, enter "net stop msftpsvc" to stop FTP service
- 5) And "net start msftpsvc" to restart the FTP service.
- 6) Exit the command prompt window.
After the above steps, try logon to FTP with the AD user account, the user should be able to logon and redirected to his/her home folder. To summarize, the error is due to FTP service unable to get information from Active directory, in my scenario it was due to the AD account password changed issue. There are many other possible reasons as well, for example - the configured account being deleted/locked out, or FTP service is unable to connect to the Domain controller and etc. Enabling the security logon auditing in this case would help narrow down the possible cause of this issue.
Finding URL with Long Processing Time
Q: I am managing servers with around 100 websites in each of them. A few of those websites are so poorly written that some pages almost consume more than 60% of the CPU. I developed a script that monitors worker processes on the server and, once one of them consumes more that a specific limit (say 50%) it can figure out which site (using the IISApp.vbs command) and then send an email notification to the webmaster. Is there a way to figure which website's page was taking a long time to process?
A: Excellent question! I myself had been haunted by this problem since IIS 3.0 - and I learned it the hard way. I'm sure the developers will say that "it is a hardware problem, if not your server configuration". Why? They always felt that there's no way that their application is the culprit, because it runs fine in the development server. To pinpoint what requests take the most resource time is not hard - the hardest part is to prove to them that "this is the stack trace and your component is hooking the CPU" or something like that. Of course, this would need some knowledge in debugging the web application and understanding Windows internal architecture.
Anyway, assume you are using IIS5 or IIS 6. To dive in and do a debug via Debug Diagnostic, you need to know which dllhost.exe or w3wp.exe to attach the debugger. And to answer your question, how do you figure out the URL or request with the highest resource time? This is simple, and you can do it without any tool.
Start by enabling the ‘Time-taken' field in the W3C extended log file. This field time-stamps the processing time of a particular request. For example:
#Fields: date time s-sitename s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs-version cs(User-Agent) cs(Cookie) cs(Referer) cs-host sc-status sc-substatus sc-win32-status sc-bytes cs-bytes time-taken
2006-07-25 05:45:21 W3SVC1 127.0.0.1 GET /test.asp - 80 - 127.0.0.1 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.2;+SV1;+.NET+CLR+1.1.4322;+.NET+CLR+2.0.50727) - - localhost 200 0 0 246 254 13310
In the above example, the request to test.asp took 13310ms, which is 13 seconds plus. Now, what if you have a very complicated web application with a large amount of dynamic content? How do you parse the log with hundred of megabytes?
No worries. In my previous setup, I had websites that generated 1GB of log files each day. It would take me days if I had to parse it manually. You can count on an automated log analyzer to help you - and there are many commercial or open source log analyzers available. While most of them give you fancy analysis and nice diagrams, I don't like any of those (only the folks in marketing departments seem to love those reports). For me - and most other system administrators - the key is to have a fast, precise and tiny tool that I can carry anywhere I go. I don't want to wait for 3 hours for the analyzer to complete it reports before I can interpret them. So long story short - I used Log Parser from Microsoft and I love it!
This is one tiny tool that does magic for IIS administrator like you. I'm not going to tell you all the amazing stories about this tool, you can read it over the net. So how do we use Log parser to find the top 10 requests that consume most of the resource time? Simple, you run this query:
---Ch02Top10WebRequests.sql---
SELECT
TOP 10
STRCAT(EXTRACT_PATH(cs-uri-stem),'/') AS RequestPath,
EXTRACT_FILENAME(cs-uri-stem) AS RequestedFile,
COUNT(*) AS Hits,
MAX(time-taken) AS MaxTime,
AVG(time-taken) AS AvgTime,
AVG(sc-bytes) AS AvgBytesSent
FROM %source% TO %destination%
GROUP BY cs-uri-stem
ORDER BY MaxTime, TotalHits DESC
---Ch02Top10WebRequests.sql---
<Shameless plug>the above is actually from the chapter 2 of Microsoft Log Parser book (to which I have contributed). </Shameless plug>
So what does it query? Quite straight forward, if you understand standard SQL queries. Anyway, the script looks for the top 10 long running queries from a particular ‘source', And gives you the number of hits, requests the URL and file name, the maximum and average time spent as well as average bytes sent. Pretty cool huh! How do run it? You do the following:
C:\LogParser\Samples>Logparser.exe file:Ch02Top10WebRequests.sql?source="<//localhost/w3svc/1>"+destination="Top10WebRequests.txt" -o:NAT
This script is designed to let you query against any source! In the example, we parse all log files from website ID 1, you can change it to a folder with your log files, other log source, etc. And I redirect the output to Top10Webrequest.txt file. See Figure 3.1 for the output.
RequestedPath RequestedFile Hits MaxTime AvgTime AvgBytesSent
------------- ------------- ----- ------- ------- ------------
/reg/ reg.asp 821 80212 40212 1200
/expand/ incoming.asp 4095 39322 29322 20322
/processing/ cust_up.asp 3900 33293 30233 2932
/kiv/ stock.html 8032 32002 31922 370921
/expand/ detail.asp 6293 30092 29392 39223
/processing/ cust_add.asp 200 15082 13978 2011
/processing/ inv_tune.asp 2099 13021 12911 8232
/kiv/ elite.aspx 5822 11929 9218 932
/ news.asp 10003 8922 6832 2111
/html/ abs.html 4022 7990 5820 29201
Figure 3.1 - Log Parser Sample Output
What can you tell from the above? Simple - you have a ‘reg.asp' file with average processing time of 40secs and maximum time of 80secs! Wow! That's more than a minute. From here, you can then review the coding in the script to see if you can fine tune the script, by changing its logic, database queries and etc.
To learn more about the amazing Log Parser, I encourage you to read this recent blog posting from Microsoft and don't forget to get the book - this is the only book available for the tool, so visit www.logparser.com and find out more.
Ok, back to business. What's next! What if you find nothing wrong with the ‘reg.asp'. Everything is straight forward inside this ‘registration' page. It captures user input and updates it to database via a DB component.
What the above script does is just identify the long running web requests you have, but it is unable to give you what is really going on inside the process thread. To go in deeper, you might need to debug the application if you found nothing wrong with the script (and you have reviewed the script thoroughly). Hence, let me introduce our next ‘star' from Microsoft - Debug Diagnostic! This is the debugging tool that will hook on to your worker process and capture its process threads and memory dump in order to give you a complete look of what's going on inside the IIS worker process.
I hope I have answered your question, and it is way beyond the scope of this answer to give you more detail about DebugDiag here. Nevertheless, if you are interested, you can read it in the previous issue of IIS Insider, watch the webcast at IIS.Net, recent knowledge based articles of DebugDiag, and etc.
Oh well, time to say bye bye again! Ciao!