Rexiology... : Readings

Blog moved to MSDN Blogs site...

Rex Tang — Sun, 18 Mar 2007 18:33:00 GMT

This post announces the depreciation of Rexiology MSMVP blog (this site).

for new post updates please update your RSS reader to http://blogs.msdn.com/rextang ,

or you can also point to my main personal work blog site at http://rextang.net/blogs/work/ .

Thanks for reading...

Rex

Happy New Year 2007...

Rex Tang — Mon, 01 Jan 2007 13:24:00 GMT

crosspost from http://rextang.net/blogs/work/

First 2007 sunrise from Tokyo, Japan.

Happy New Year 2007!!

Clock in a Linux Guest Runs More Slowly or Quickly Than Real Time...

Rex Tang — Wed, 20 Dec 2006 19:11:00 GMT

crosspost from http://rextang.net/blogs/work/

The title came from VMWare KB and just said exactly the problems I've faced those 2 days!

As I've written in the Windows X Server Client post, I used VMWare 5.5 to setup a Fedora Core 6 Linux VM to be the linux test server. the linux kernel version is 2.6.18 and after I finished that post soon I found that my server's system time is little bit strange. it's not sync with the host time.

Since correct time is very important for a server, I think it should be more important in a linux server since there were lots of logs and crontabs that's based by time. The very first thoughts to fix this problem is to use NTP protocol to sync with world time servers. in linux kernel 2.6 there is ntpd for ntp server, in 2.4 it should be xntpd. so I quickly modified "/etc/ntp.conf" file and started the ntpd.

the time is still not sync with real time after I started ntpd. also I found that the ntpd installed is not able to sync to outside server, it will only sync to LOCAL server, even if I turned on udp port 123 in iptables (both in and out), just like what it was written in this maillist I found. also, the system time is slow, it takes more than one second to pass a second (inside this linux guest system). there seems also having problems related to VM hardware's clock settings.

This tooks me one day and didn't find any solution on the internet. finally I decided that maybe it's only happened on linux kernel 2.6.18 and decide to build up another distribution with different kernel version to test again.

I choosed CentOS 4.4 with linux kernel 2.6.9 to build up another linux vm. and after the installation, the system time is still slow like the one inside Fedora vm. this concluded that it will almost happened with 2.6 kernel in VMWare and won't happened in 2.4 kernel (since my colleague had a RedHat 7.2 with kernel 2.4.18 running without the time slow problem). some posts I read indicated that this should have something related to system clock. so I started dig into that direction.

A search to Google finally found the answer. From VMWare KB. It's exactly the answer of this problem if you are running kernel 2.6 inside VMWare. the reason that caused the slowing time inside guest system is because of that the guest clock frequency is setting too high than the host OS can offer. in 2.4 kernel the clock rate is set to 100HZ and after 2.6 kernel the clock is set to 1000HZ on compiling time, thus cause the timer in guest OS slower than host OS. refer to this KB for more detail explaination.

Just follow what the KB said to tune bootup kernel options for both slow time and quick time problem, like the following (GRUB case):

image=/boot/vmlinuz label="linux"
root=/dev/hda1 initrd=/boot/initrd.img
append="resume=/dev/hda6 splash=silent clock=pit nosmp noapic nolapic"
read-only

"clock=pit" is to fix quick clock problem, "nosmp noapic nolapic" is to fix slow clock problem.

after a reboot of guest linux (both my Fedora one and CentOS one) , the clock is back to normal and strangely that the ntpd also back to normal work and can query outside time server without problems. cool!

As also provided in another VMWare KB talking about Timekeeping in VMWare Virtual Machine, it is also able to use VMWare Tools to sync guest OS time with host one. by installing VMWare Tools in Linux server, there will be a vmware-guestd running and by setting parameter "tools.syncTime = TRUE" inside VMWare .vmx setting file, time will be sync.ed with host OS using VMWare Tools and will be not necessary to run ntpd or ntpdate in crontab to sync your time inside guest system, it will auto sync with host OS and it will now be only the host OS that needs to sync with world time server thus save the bandwith also the loading of outside time servers.

finally solved the time problem, and got to learn one more linux distribution install, also get deeper understanding of ntpd during this problem solving, nice (tough) learning.

actually Fedora Core distribution is pretty different from CentOS installation, CentOS is pretty much like RedHat distribution and many daemon names are different from Fedora and CentOS (RedHat). another tough learning to remember those difference! (should be more differences if touching FreeBSD systems!)

anyway, now I can also setup my xdmcp and vnc server env on my CentOS guest system and now have 2 guest linux systems to start my learning of those network services installation. keep walking...

Technorati Tags: linux , redhat, fedora, CentOS, ntp, clock, kernel , vmware

Working VISA in Japan...

Rex Tang — Tue, 21 Nov 2006 17:19:00 GMT

crosspost from http://rextang.net/blogs/work/

Just a memo of what actually happened in my case.

Actually recently I was busying trying to change and get a "Microsoft" technology related job in Japan. I did pass the interviews with managers and dept. director and confirmed that I am capable to work for the department. finally the job agent told me that the company is going to start the offer process and HR is starting to check my working VISA status since I am a foreigner and need to fit certain criteria.

The problem occured. Since I was graduated from Information Management dept. of National Taiwan University, this dept. issues BBA (Bachelor of Business Administration) degree to students, not BS (Bachelor of Science) degree like most other Computer Science related dept.s, although most of the classes the dept. offerred are IT related stuffs. I do think that Japan Immigration Office only saw my BBA degree and gave me "Specialist in Humanities/International Services" working VISA, rather than an Engineer VISA in my current job. and this caused the problem that the new company's HR and Law dept. felt that it should be diffecult to change the status of my visa and decide to decline my offer.

In Japan, one has to have certain working VISA to be able to work in certain job area. there are several job categories according to immigration law of Japan. the most usual working VISA type is "Specialist in Humanities/International Services" and "Engineer" , "Specialist in Humanities/International Services" visa type is normally given to people who graduated from business / management / culture / language related universities (English techers in this category), while Engineer visa type is mainly given to people who study nature science including IT / Computer fields. others like doctor or lawyer also have their own working visa type, things like that.

recently to encourage IT / Computer related businesses in Japan, actually Japan goverment had loose the definition of Engineer type visa to allow those who passed certain IT certificates to get Engineer visa regardless what university diploma one may have. those IT certificate are mainly held by Japan's institutes and anyone can take the exam without restrictions, including foreigners (exams should be in Japanese, not English). the most fundamental IT/Computer exam is held every year at April and October, only twice in an year. be sure to know the exactly date if one needs those certificates to get a valid Engineer VISA regardless of diplomas one may have.

It should be considering illegal to have actual working content that's different from the working VISA a foreigner get by the proof of company who employ this person. Thus make it important for foreigners to get proper VISA in order to work in Japan without troubles.

As my graduated dept.did have ambiguity on the judge of an Engineer visa or a Specialist in Humanities/International Services visa, since it's "Information Management" dept. and issuing BBA degree, I do think that other universities in the world do have departments like this and some offer BBA degrees but others offer BS degrees. It should be important for Japan Immigration office to check the detail of academic transcripts and decide what visa type is proper to give out, not just see if one had BBA or BS to decide it, that will cause inconvenience for foreigners that want to live and work here to evolve IT environment in Japan.

I guess the only way I can do is to take the IT certificate exam here in Japan in order to bypass the consideration of my university diploma, although all my records all showed that I've been always doing IT related jobs since I graduated. and I do own some worldwide IT certificates (which is not listed in the immigration law for now) but I guess it's hard to persuade those law and HR people or even Japan Immigration Officers.

I also wish the interviewer, the director of that dept. of new company which I want to get the job, can see this post and see if he can also persuade HR and Law dept. to reconsider my case. the issue to decline my case is not related to my abilities, but the Japan Immigration problem, and got ways to solve out, It's really disappointed to decline my case in this reason, sort of not acceptable. just like that foreigners are hard to get a Japan credit card just because we are foreigners, without reasons...

anyway, just a reference for those who want to work in Japan to know some immigration issues here, wish this help.

FYI.

Technorati Tags: work , japan, immigration, diploma, certificate, visa, working visa

Software Architecture: Past, Present and Future...

Rex Tang — Fri, 10 Nov 2006 09:30:00 GMT

crosspost from http://rextang.net/blogs/work/

[via DiegumZone]

Emerging Architect Roles

The considerations of economical changes like globalization and technological achievements like the Internet’s impact 0n the digital economy, pressed for formalizing software architecture as a discipline.

Although there is not yet a definite agreement in the distinct roles, we can sketch three major personas:

· Infrastructure Architect. These define the platform and other environments (hardware, basic software) to provide for business applications’ high availability. They must also work with developers to define mechanisms and standards that allow applications to achieve the security, reliability, manageability, transparency, and policy compliance essential to the modern business. It’s expected that the natural evolution of a senior IT professional is an Infrastructure Architect.

· Solutions Architect. These are responsible for the design of one or more applications or services within an organization, usually within the scope of a division (and for that reason also known as Application Architect). Examples of such applications are: Internet banking, companywide knowledge sharing portal, and distributed point of sales applications. A senior developer is a good candidate to become Solutions Architect.

· Enterprise Architect. Their job is to keep the business and its IT systems in alignment. They strive to maximize the return on IT investment by making sure that IT spending is prioritized towards business opportunity, and by optimizing the impact of investments across the organization’s portfolios of services, resources, projects, and processes. They must be a bridge between business leaders, development, and operations to ensure that mutual understanding is achieved, goals are realistic, and expectations are properly managed. Enterprise Architecture is about the big picture — how people and technology work together to produce world-class, long-term results. For that reason, this persona is also referred as Strategic Architect. What is expected is that a Solutions Architect or Infrastructure Architect becomes Enterprise Architect.

---------

Just had a reading of this article which provided an overall view of Software Architecture progress, nice one to have a read also it introduced an architecture site at http://www.skyscrapr.net , full of resources and discussion related to architectures, don't miss the link...

Technorati Tags: software , solution, programming, architecture, pattern

LINQ / DLINQ resource...

Rex Tang — Tue, 31 Oct 2006 08:11:00 GMT

crosspost from http://rextang.net/blogs/work/

Just read from ScottGu's blog post that included useful links to LINQ / DLINQ .

just leave a record here and will post more after I read them.

Technorati Tags: linq , dlinq , orm , programming

Test Driven Development (TDD) in Team System...

Rex Tang — Sun, 26 Mar 2006 18:01:00 GMT

crosspost from http://rextang.net/blogs/work/

[via Rob Caron]

Doug Seven got a simple and easy to understand article for developers to understand the concepts of Test Driven Development using Visual Studio 2005 Team System.

Test Driven Development (TDD) is not a new concept. In fact, the idea of test-first, code-second has been around for many years. In the latest release of Microsoft's premier developer tool, Visual Studio 2005 Team System many new features have been added, including features for testing software. What does this mean to you, the serious developer? It means you now have integrated unit testing that can be leveraged for Test Driven Development.

Technorati Tags: microsoft , vstudio, architecture

Something new about MSF - Microsoft Solution Framework...

Rex Tang — Sun, 26 Mar 2006 16:11:00 GMT

crosspost from http://rextang.net/blogs/work/

[via rohanthomas]

Microsoft had released new MSF templates at 2006/03/17, which "are" version 8.0 templates (should serve as MSF v4), including those 2:

MSF for Agile Software Development

MSF for CMMI® Process Improvement

For more information please go to Microsoft MSF Website. I've downloaded those related files and demos and will find time to dig into them.

Technorati Tags: microsoft , architecture, agile, cmmi

DotNetKicks.com - The Digg for .NET...

Rex Tang — Tue, 28 Feb 2006 06:10:00 GMT

crosspost from http://rex.la/blogs/work/

hmm... it's really like a .Net version of Digg...

[via Will Asrari] [via DotNetKicks.com]

dotnetkicks.com is a community based news site edited by our members. It specializes in Microsoft development techniques , technologies and tools including ASP.NET, C#, VB.NET, C++, Visual Studio, Team System, SQL Server and Microsoft Vista.

Individual users of the site submit and review stories, the most popular of which make it to the homepage. Users are encouraged to 'kick' stories that they would like to appear on the homepage. If a story receives enough kicks, it will be promoted.

Technorati Tags: microsoft , digg, dotnet

New Taiwan Microsoft Technology Forums Announced...

Rex Tang — Thu, 16 Feb 2006 16:00:00 GMT

crosspost from http://rex.la/blogs/work/

The official Microsoft Technology Community at Taiwan used to be running web-NNTP based forums to communicate with the public, answering questions and sharing information with each others. This year Microsoft Taiwan announced new forum based community system, established new way to interact with the public. Using a forum system (should be based on CommunityServer forum framework) enables all the content be searched by popular searching engines and thus enables information be exchanged and shared more efficient and wide.

This should be Chinese based Microsoft Technology Forums hosted by Taiwan Microsoft MVPs and I'll also help on building a good Microsoft Technology related content base to help more Chinese-reading people know more about Microsoft Technologies.

[This post in Chinese]

Technorati Tags: microsoft , community , technology , forum , chinese

Observation of BitComet DHT network and BlackIce 0.0.0.0 Tcp_Probe_Other intruder...

Rex Tang — Fri, 03 Feb 2006 23:21:00 GMT

crosspost from http://rex.la/blogs/work/

Time for sniffing the network!

It's been some time that I've noticed my server been surrounding huge packets from source IP 0.0.0.0 , noticed from my BlackIce PC Protection firewall, both TCP and UDP packets from 0.0.0.0 were there marked as TCP_Probe_Other and UDP_Probe_Other. UDP packets grew slow but TCP packets grew very fast.

Due to the lack of information provided by BlackIce (Orz), I had little clues to find out what's going on those weeks also, busy on writing code for my work; The Chinese-New-Year 9 days vacation (well, it's like X'mas in western world) just gave me the chance to fully find out what's going on.

I've kept monitoring my server during the vacation and finally it reached over 450,000 TCP packets from 0.0.0.0 in about 3 days, constantly increasing about 15-20 packets per second. I couldn't stands for it and thought I should really dig into what's going on.

A search to Google gave me a hint, according to Steve Gibson's comprehensive articles (here and here), I was originally thought that I might under someones' DRDoS attacks (the pattern is similar to what Steve described in his article). After knowing this I was wondering if there will be good tools to help me investigate my network. I used to use IRIS, but felt that it's sort of huge and seems not very compatible in Windows 2003 server, so I decided to find others.

BlackIce PC Protection is the little software firewall I am using to defend my system from opening a wild door to other systems around the internet.

SystemInternals' freeware TCPView is a good tool to monitor the server's TCP/UDP usage. (thanks to MSMVP Li Jung Chiu's info). It shows how all the real-time TCP connections are going on inside a server and can easily see what application / process is using what port to connect to others. but I can only see what process is using what tcp/udp ports to communicating with someone else; I can not see the details of packets that's going inside the wires. I'll still need a small but useful packet sniffer for that.

I did find Snort site (free, got windows version), but I had no time and mood finishing the whole user manual to start get control of this cool tool. in the search of it's related underlying socket handler package WinPcap (free), I found it's cool Links page that's full of other networking tools that might be useful to my investigation. for a quick and random pick, I got Ethereal (free) for sniffing my network.

3 tools above in a row to start peeking my WAN network!

I've seen those 0.0.0.0 packets keep coming in, no clues in TCPView about who is using them since there were only bunch of LISTENING ports with Remote Address 0.0.0.0:0 , hard to tell what's going on.

Started Ethereal for sniffing the traffic of WAN, before doing this either close most of other known net daemons (mail server, web server ,etc) or set the filter to only capture UDP packets (since there were some volume of packets of them also) to minimize the range of investigation. quickly found that there were suspicious udp packets kept coming in. it's all from random IPs with random port numbers but all coming in to my server ip with the same udp port.

BlackIce didn't mark those 0.0.0.0 udp packets as blocked status, it's allowing accepting those packets. so my instinct told me that it should be something I've set in my firewall rules. after checking the settings of BlackIce, found that it's the port number I've opened for BitComet!

I've open the same port numbers for both tcp and udp to allow BitComet communicating with its distributed network. so I just let Ethereal filter both tcp and udp packet for the same port number for BitComet. the result was just like what I am guessing, those 0.0.0.0 tcp and udp packet were all coming from BitComet's distributed network.

Knowing that I might not been suffering from some kind of DoS attacks was really a relieve (although this should be classified as one kind of attack, if not knowing it). It's now the time to find out why and how to stop this.

I closed BitComet to see if the traffic would stop. also blocked tcp and udp port for BitComet in BlackIce to see what will happen. the block of ports quickly reflected in BlackIce monitor, those 0.0.0.0 packets were now marked as blocked, although still keep coming in. there is a decay time for TCP sockets to finally reach it's end of communication, so I just wait for couple minutes. after seen from TCPView that BitComet had not using any connection, I noticed that the 0.0.0.0 packets still coming in, in the same speed.

The sniffing form Ethereal told me that under the circumstance I've made for the system (firewall port block , kill the listening and processing app), there were now only coming TCP SYN packets from various sources and no SYN/ACK packets returned (I've blocked the port!), the packets still coming in like a flood. re-open the listening app won't have any difference since it won't know that someone is contacting him. but strangely that BitComet will still uploading stuffs out to peers while the ports were blocked (incoming / download also work!).

Later I just closed BitComet also kept port blocked, waiting for days to see if the traffic will just decay by time. while in the waiting time I took some reading about BitComet's adoption of DHT (Distributed Hash Table) network. The explaination from BitComet wiki here and little bit in-depth explaination from Wikipedia here. After knowing the working of a DHT network, I was starting to imagine how it would like if my server becoming a node inside one of such a network.

If my server is being as a node inside BitComet's DHT network, then my server would act as a routing node for keys. note as BitComet wiki said that DHT network used TCP port with the same UDP port number for its overlay network, also from what Ethereal got of packets from the same port, it should be an evidence of a match. The cancellation of using DHT network seems not work at start of this experiment, but later found that it "should" work as what it stated in BitComet's settings dialog.

The blocking of the ports to prevent incoming BitComet packets seems not be a good idea. the DHT network looks like it will continue retrying the node that originally inside its network (my server) if the node didn't respose anything with other peers TCP SYN handshake, as proven above. so I just turn on the ports (both tcp and udp) from BlackIce again, and seen from Ethereal that now every incoming TCP SYN will accompany with an outgoing TCP RST/ACK to tell peers that this node is cancelling their connection and discarding their packets (cause the listening app was not started). This seemed to work! after several half-hour monitoring, the 0.0.0.0 packets were decaying in some sort of speed and constantly slowing the speed. finally only several packets from 0.0.0.0 were happened in a minute hours later. I assume that waiting for worldwide participating BitComet peers in its DHT network to notice a node (my server) is dis-joining the network would take up to half-day to complete (well, sort of faster then a DNS propagation!).

The final question would be that, why 0.0.0.0 in BlackIce?

Ethereal showed every packet with clear source and destination IPs, but it all appeared as 0.0.0.0 in BlackIce. I can only assume that it should be caused by that now BitComet network had the ability to encrypt the headers to hide peers' identifications (see BitComet settings dialog). thus it will appear to firewalls like BlackIce to only determine a fake IP 0.0.0.0 , which later in turn to see similar and familiar as a DRDoS attack.

The Conclusion as follows:

The Concepts of DDoS and DRDoS (Steve Gibson's articles here and here)
Various Network Monitoring Tools (BlackIce , TCPView , and Ethereal)
BitComet DHT network and how it works

In the last, if you too find that you've had this kind of 0.0.0.0 huge incoming packets, check if your servers or administrating networks had applications or network nodes now on any of those DHT or BT networks!

Technorati Tags: firewall , DHT, bitcomet, blackice, ethereal, p2p