Jon Skeet: Coding Blog : Benchmarking, Parallelisation

Revisiting randomness

skeet — Wed, 04 Nov 2009 07:40:15 GMT

Almost every Stack Overflow question which includes the words "random" and "repeated" has the same basic answer. It's one of the most common "gotchas" in .NET, Java, and no doubt other platforms: creating a new random number generator without specifying a seed will depend on the current instant of time. The current time as measured by the computer doesn't change very often compared with how often you can create and use a random number generator – so code which repeatedly creates a new instance of Random and uses it once will end up showing a lot of repetition.

One common solution is to use a static field to store a single instance of Random and reuse it. That's okay in Java (where Random is thread-safe) but it's not so good in .NET – if you use the same instance repeatedly from .NET, you can corrupt the internal data structures.

A long time ago, I created a StaticRandom class in MiscUtil – essentially, it was just a bunch of static methods (to mirror the instance methods found in Random) wrapping a single instance of Random and locking appropriately. This allows you to just call StaticRandom.Next(1, 7) to roll a die, for example. However, it has a couple of problems:

It doesn't scale well in a multi-threaded environment. When I originally wrote it, I benchmarked an alternative approach using [ThreadStatic] and at the time, locking won (at least on my computer, which may well have only had a single core).
It doesn't provide any way of getting at an instance of Random, other than by using new Random(StaticRandom.Next()).

The latter point is mostly a problem because it encourages a style of coding where you just use StaticRandom.Next(…) any time you want a random number. This is undoubtedly convenient in some situations, but it goes against the idea of treating a source of randomness as a service or dependency. It makes it harder to get repeatability and to see what needs that dependency.

I could have just added a method generating a new instance into the existing class, but I decided to play with a different approach – going back to per-thread instances, but this time using the ThreadLocal<T> class introduced in .NET 4.0. Here's the resulting code:

using System;
using System.Threading;

namespace RandomDemo
{
    /// <summary>
    /// Convenience class for dealing with randomness.
    /// </summary>
    public static class ThreadLocalRandom
    {
        /// <summary>
        /// Random number generator used to generate seeds,
        /// which are then used to create new random number
        /// generators on a per-thread basis.
        /// </summary>
        private static readonly Random globalRandom = new Random();
        private static readonly object globalLock = new object();

        /// <summary>
        /// Random number generator
        /// </summary>
        private static readonly ThreadLocal<Random> threadRandom = new ThreadLocal<Random>(NewRandom);

        /// <summary>
        /// Creates a new instance of Random. The seed is derived
        /// from a global (static) instance of Random, rather
        /// than time.
        /// </summary>
        public static Random NewRandom()
        {
            lock (globalLock)
            {
                return new Random(globalRandom.Next());
            }
        }

        /// <summary>
        /// Returns an instance of Random which can be used freely
        /// within the current thread.
        /// </summary>
        public static Random Instance { get { return threadRandom.Value; } }

        /// <summary>See <see cref="Random.Next()" /></summary>
        public static int Next()
        {
            return Instance.Next();
        }

        /// <summary>See <see cref="Random.Next(int)" /></summary>
        public static int Next(int maxValue)
        {
            return Instance.Next(maxValue);
        }

        /// <summary>See <see cref="Random.Next(int, int)" /></summary>
        public static int Next(int minValue, int maxValue)
        {
            return Instance.Next(minValue, maxValue);
        }

        /// <summary>See <see cref="Random.NextDouble()" /></summary>
        public static double NextDouble()
        {
            return Instance.NextDouble();
        }

        /// <summary>See <see cref="Random.NextBytes(byte[])" /></summary>
        public static void NextBytes(byte[] buffer)
        {
            Instance.NextBytes(buffer);
        }
    }
}

The user can still call the static Next(…) methods if they want, but they can also get at the thread-local instance of Random by calling ThreadLocalRandom.Instance – or easily create a new instance with ThreadLocalRandom.NewRandom(). (The fact that NewRandom uses the global instance rather than the thread-local one is an implementation detail really; it happens to be convenient from the point of view of the ThreadLocal<T> constructor. It wouldn't be terribly hard to change this.)

Now it's easy to write a method which needs randomness (e.g. to shuffle a deck of cards) and give it a Random parameter, then call it using the thread-local instance:

public void Shuffle(Random rng)
{
...
}
...
deck.Shuffle(ThreadLocalRandom.Instance);

The Shuffle method is then easier to test and debug, and expresses its dependency explicitly.

Performance

I tested the "old" and "new" implementations in a very simple way – for varying numbers of threads, I called Next() a fixed number of times (from each thread) and timed how long it took for all the threads to finish. I've also tried a .NET-3.5-compatible version using ThreadStatic instead of ThreadLocal<T>, and running the original code and the ThreadStatic version on .NET 3.5 as well.

Threads	StaticRandom (4.0b2)	ThreadLocalRandom (4.0b2)	ThreadStaticRandom (4.0b2)	StaticRandom(3.5)	ThreadStaticRandom (3.5)
1	11582	6016	10150	10373	16049
2	24667	7214	9043	25062	17257
3	38095	10295	14771	36827	25625
4	49402	13435	19116	47882	34415

A few things to take away from this:

The numbers were slightly erratic; somehow it was quicker to do twice the work with ThreadStaticRandom on .NET 4.0b2! This isn't the perfect benchmarking machine; we're interested in trends rather than absolute figures.
Locking hasn't changed much in performance between framework versions
ThreadStatic has improved massively between .NET 3.5 and 4.0
Even on 3.5, ThreadStatic wins over a global lock as soon as there's contention

The only downside to the ThreadLocal<T> version is that it requires .NET 4.0 - but the ThreadStatic version works pretty well too.

It's worth bearing in mind that of course this is testing the highly unusual situation where there's a lot of contention in the global lock version. The performance difference in the single-threaded version where the lock is always uncontended is still present, but very small.

Update

After reading the comments and thinking further, I would indeed get rid of the static methods elsewhere. Also, for the purposes of dependency injection, I agree that it's a good idea to have a factory interface where that's not overkill. The factory implementation could use either the ThreadLocal or ThreadStatic implementations, or effectively use the global lock version (by having its own instance of Random and a lock). In many cases I'd regard that as overkill, however.

One other interesting option would be to create a thread-safe instance of Random to start with, which delegated to thread-local "normal" implementations. That would be very useful from a DI standpoint.

Buffering vs streaming benchmark: first results

skeet — Tue, 24 Mar 2009 01:34:00 GMT

My poor laptop's had a busy weekend. It's run 72 tests, rebooting between each test. Most of these tests have kept both the CPU and disk busy for a lot of the time. I expect to update this blog post with more numbers - and possibly more strategies - as time goes on, but I wanted to get the numbers out as quickly as possible. Graphs should be coming when I've got a decent network to experiment with Google graphing.

Source code and setup

While I don't really expect anyone else to sacrifice their computer for the best part of 24 hours, it doesn't take very much effort to get the tests running. This zip file contains the source code for three programs, a batch file to generate the test data, and a command file.

The CreateFiles executable creates input files based on its command line arguments. (This is run multiple times by the batch file.) EncryptFiles "encrypts" (I use the term loosely) all the files in a given directory, writing the results to another directory. The class design of EncryptFiles is a little bit funny, but it's really tailored for these tests. It should be easy to write other strategies in the same way too.

ExecuteAndReboot is used to automate the testing. Basically it reads a command file (from a hard-coded directory; simplicity trumps flexibility here). If the file isn't empty, it waits five minutes (so that after a reboot the computer has time to settle down after loading all its services) and runs the command specified in the first line of the file, appending the output to a results file. When the command has finished executing, ExecuteAndReboot rewrites the command file, removing the first line, and schedules a reboot. The idea is to drop ExecuteAndReboot into a Startup folder, set Windows to log in automatically, reboot once and then just let it go. If you want to use the computer, just wait until it's in the "waiting five minutes" bit at the start of the cycle and kill ExecuteAndReboot. Next time you reboot it will start testing again.

That's the general principle - if anyone does want to run the tests on their mail and needs a bit of help setting it up, just let me know.

Test environment and workload

I expect the details of the execution environment to affect the outcome significantly. For the sake of the record then, my laptop specs are:

Dell Inspiron 1720 (unlikely to be relevant, but...)
Intel Core 2 Duo T7250 2.0GHz
Vista Home Premium 32 bit
3 GB main memory, 32K L1 cache, 2048K L2 cache
2 disks, both 120GB (Fujitsu MHY2120BH, 5400RPM, 8MB cache)

Even though there are two drives in the system, the tests only ever read from and wrote to a single disk.

I created four sets of input data; I've presented the results for each one separately below, and included the description of the files along with the results. Basically in each case there's a set of files where each file is the same size, consisting of a fixed number of fixed-length lines.

The "encryption" that is performed in each case (on the encoded version of the line) is just to XOR each byte with a value. This is repeated a specified number of times, and the XOR value on each pass is the current index, i.e. it first XORs with 0, then 1, then 2 etc. I've used work levels of 0 (the loop never runs), 100 and 1000. These were picked somewhat arbitrarily, but they give interesting results. At 0 the process is clearly IO-bound. At 1000 it's clearly CPU-bound. At 100 it's not pegging the CPU, but there's significant activity. I may try 200 as well at some point.

For each scenario I tried using 1, 2 and 3 threads (and 4 for the big tests; I'll go back and repeat earlier ones with 4 threads soon). The two strategies used are "streaming": read a line, encrypt, write the line, repeat and "buffering": read a whole file, encrypt it all, write it all out. After a thread has started, it needs no shared data at all - it's given the complete list of files to encrypt at the very start.

Results

For every table, the number of threads is shown by the leftmost column, and the work level is shown by the topmost row. The cells are of the form "x/y" where "x" is the time taken for the streaming strategy, and "y" is the time taken for the buffering strategy. All times are in seconds. For each column, I've highlighted the optimal test.

Small tests

There are 10000 test files. Each is 100 lines long, with 100 characters per line.

Threads/Work	0	100	1000
1	47/90	94/79	163/158
2	95/112	113/157	129/146
3	179/128	130/144	152/160
4	144/139	163/193	156/204

Medium tests (short lines)

There are 100 test files. Each is 500,000 lines long, with 20 characters per line.

Threads/Work	0	100	1000
1	138/157	219/267	1125/1205
2	196/238	259/263	604/691
3	292/250	312/236	624/676
4	302/291	321/281	602/666

Medium tests (long lines)

There are 100 test files. Each is 100 lines long, with 100,000 characters per line.

Threads/Work	0	100	1000
1	203/213	249/286	1027/1107
2	283/264	311/296	602/696
3	319/284	314/278	608/618
4	360/300	336/297	607/598

Big tests

There are 20 files. Each file is 100,000 lines long, with 1000 characters per line.

Threads/Work	0	100	1000
1	190/246	349/441	2047/2165
2	365/375	392/473	1097/1387
3	456/379	484/399	1161/1296
4	517/442	521/431	1113/1214

Conclusions

I'm loathe to draw too many conclusions so far. After all, we've only got data for a single test environment. It would be interesting to see what would happen on a quad core machine. However, I think a few things can be said with reasonable confidence:

Sometimes the streaming strategy wins, sometimes the buffering strategy wins. The difference can be quite significant either way. For a given input and workload, the streaming solution almost always wins on my machine.
In ideal scenario, the bottlenecked resource should be busy for as much of the time as possible. For example, in a high-CPU task it makes little sense to have idle cores when you've already loaded data. Likewise in a disk-bound task we should always be fetching (or writing) data even if all our cores are momentarily busy. Personally I think this is the operating system's pre-fetch systems's job. We could do it ourselves, but it's nicer if we don't have to.
If we're disk-bound, it makes sense to read (or write) one file at a time, to avoid seeking. This is obvious from the results - for work levels of 0 and 100, a single-thread solution is consistently best (and streaming usually works better buffering). Based on this assumption, I suspect a cleaner solution would be to have a single thread reading largish chunks (but not the whole file) and feeding the other threads - or to use async IO. However, this all gets very complicated. One nice aspect of both of the current solutions is that they're really simple to implement safely. Reading line-by-line in an async manner is a pain. I'd quite like to write an async "execute for every line" helper at some time, but not just yet.
If we're CPU bound, the buffering solution's sweet spot is 3 threads (remember we're using 2 cores) and the streaming solution works better with just 2 threads - introducing more threads will make both the IO and context switching less efficient.
The streaming strategy always uses less application memory than the buffering strategy. When running the buffering strategy against the "big" data set with 4 threads, the memory varied between 800MB and 1.8GB (as measured by the "Total bytes in all heaps" performance counter), vs around 165KB with the streaming strategy. (Obviously the process itself took more than 165KB, but that was the memory used in "normal" managed heap memory.) Using more memory to improve performance is a valid technique, but I'd say it's only worth it when the improvement is very significant. Additionally, the streaming technique instantly scales to huge data files. While they're not in the current requirements, it doesn't hurt to get that ability for free.

Next steps...

I've included quite a few "I'm going to try to [...]" points in this post. Just for reference, my plans are:

Turn off my virus checker! If this makes a significant difference, I'll rerun all the current tests
Try to visualise the results with Google graphs - although I think it may get cluttered
Rerun big/buffering/1000 test - odd results
Rerun tests with a work level of 200 and keep an eye on CPU usage - I'd like to spot where we tip between CPU and IO being the bottleneck
Try to optimize the Stream/StreamReader being used - we should be able to skip the FileStream buffer completely, and specifying FileOptions.SequentialScan should give more hints to Windows about how we're intending to read the data. I'll try a few different buffer sizes, probably 4K, 8K and 64K, just to see what happens.

This is a really interesting little example problem, because it sounds so simple (and it is simple in terms of the implementation) but it has a lot of environment variables. Of course it would be nicer if we didn't have to reboot the machine between test runs in order to get a fair result, but never mind...

Benchmarking IO: buffering vs streaming

skeet — Fri, 20 Mar 2009 11:13:31 GMT

I mentioned in my recent book review that I was concerned about a recommendation to load all of the data from an input file before processing all of it. This seems to me to be a bad idea in an age where Windows prefetch will anticipate what data you need next, etc - allowing you to process efficiently in a streaming fashion.

However, without any benchmarks I'm just guessing. I'd like to set up a benchmark to test this - it's an interesting problem which I suspect has lots of nuances. This isn't about trying to prove the book wrong - it's about investigating a problem which sounds relatively simple, but could well not be. I wouldn't be at all surprised to see that in some cases the streaming solution is faster, and in other cases the buffered solution is faster.

The Task

The situation presented is like this:

We have a bunch of input files, either locally or on the network (I'm probably just going to test locally for now)
Each file is less than 100MB
We need to encrypt each line of text in each input file, writing it to a corresponding output file

The method suggested in the book is for each thread to:

Load a file into a List<string>
Encrypt every line (replacing it in the list)
Save to a new file

My alternative option is:

Open a TextReader and a TextWriter for the input/output
Repeatedly read a line, encrypt, write the encrypted line until we've exhausted the input file
Close both the reader and the writer

These are the two implementations I want to test. I strongly suspect that the optimal solution would involve async IO, but doing an async version of ReadLine is a real pain for various reasons. I'm going to keep it simple - using plain threading, no TPL etc.

I haven't written any code yet. This is where you come in - not to write the code for me, but to make sure I test in a useful way.

Environmental variations

My plan of attack is to first write a small program to generate the input files. These will just be random text files, and the program will have a few command line parameters:

Directory to put files under (one per test variation, basically)
Number of files to create
Number of lines per file
Number of characters per line

I'll probably test a high and a low number for each of the last three parameters, possibly omitting a few variations for practical reasons.

In an ideal world I'd test on several different computers, locally and networked, but that just isn't practical. In particular I'd be interested to see how much difference an SSD (low seek time) makes to this test. I'll be using my normal laptop, which is a dual core Core Duo with two normal laptop disks. I may well try using different drives for reading and writing to see how much difference that makes.

Benchmarking

The benchmark program will also have a few command line parameters:

Directory to read files from
Directory to write files to
Number of threads to use (in some cases I suspect that more threads than cores will be useful, to avoid cores idling while data is read for a blocking thread)
Strategy to use (buffered or streaming)
Encryption work level

The first three parameters here are pretty self-explanatory, but the encryption work level isn't. Basically I want to be able to vary the difficulty of the task, which will vary whether it ends up being CPU-bound or IO-bound (I expect). So, for a particular line I will:

Convert to binary (using Encoding.ASCII - I'll generate just ASCII files)
Encrypt the binary data
Encrypt the encrypted binary data
Encrypt the encrypted encrypted [...] etc until we've hit the number given by the encryption work level
Base64 encode the result - this will be the output line

So with an encryption work level of 1 I'll just encrypt once. With a work level of 2 I'll encrypt twice, etc. This is purely for the sake of giving the computer something to do. I'll use AES unless anyone has a better suggestion. (Another option would be to just use an XOR or something else incredibly simple.) The key/IV will be fixed for all tests, just in case that has a bearing on anything.

The benchmarking program is going to be as simple as I can possibly make it:

Start a stopwatch
Read the names of all the files in the directory
Create a list of files for each thread to encrypt
Create and start the threads
Use Thread.Join on all the threads
Stop the stopwatch and report the time taken

No rendezvous required at all, which certainly simplifies things. By creating the work list before the thread, I don't need to worry about memory model issues. It should all just be fine.

In the absence of a better way of emptying all the file read caches (at the Windows and disk levels) I plan to reboot my computer between test runs (which makes it pretty expensive in terms of time spent - hence omitting some variations). I wasn't planning on shutting services etc down: I really hope that Vista won't do anything silly like trying to index the disk while I've got a heavy load going. Obviously I won't run any other applications at the same time.

If anyone has any suggested changes, I'd be very glad to hear them. Have I missed anything? Should I run a test where the file sizes vary? Is there a better way of flushing all caches than rebooting?

I don't know exactly when I'm going to find time to do all of this, but I'll get there eventually :)