Cluebat-man to the rescue : cplusplus

Practical ATL: Solving the race condition in CAtlExeModuleT

vanDooren — Wed, 15 Apr 2009 12:40:00 GMT

When I was spelunking through the ATL header for my previous articles (the class object plumbing) I discovered that there is a serious race condition in the server lifetime management.

I contacted the Microsoft C++ folks, and someone from the libraries group told me that this problem has already been fixed in the next release of Visual C++.

But since the issue still exists in all current ATL projects, I decided to write something about it and publish the fix which will circumvent the problem in the recent versions of ATL (7.0 and upwards if I am correct).

The problem

The module lock count of an ATL COM server is managed in the CAtlExeModuleT<T> class. It is this lock count that keeps the server alive as long as it is nonzero.

The CAtlExeModuleT<T> class manages its reference count via the CComGlobalsThreadModel class increment and decrement methods. CComGlobalsThreadModel is a typedef for CComMultiThreadModel so these methods map to InterlockedIncrement and InterlockedDecrement

The last module lock is released if
- the class object itself has no more external connections (IExternalConnection), and
- there are no more object instances.

As soon as the final module lock is released (module refcount is 0), the shutdown of the server is initiated, which will happen through either a WM_QUIT message or a win32 event object. In both cases there is a time window between the triggering of the shutdown, and the actual call to CoRevokeClassObject. This time window can be several seconds.

This time window causes a race condition between the server (which is already dying at that point) trying to revoke the class object, and a hypothetical client, requesting an interface pointer to the class object. Depending on how the race turns out, the client might end up with an S_OK HRESULT from CoGetClassObject, and an invalid interface pointer. This will cause a critical problem in the client, and may cause an application crash.

The solution

The proper way to manage the server lifetime is to use CoAddRefServerProcess and CoReleaseRefServerProcess to manage the module lockcount. Those functions will (according to MSDN http://msdn.microsoft.com/en-us/library/aa910552.aspx) suspend all class objects when the reference count drops to 0, in an atomic operation.

In that case, the server starts dying AFTER all class objects are suspended. Any incoming client request for the class object will spawn a new server process after the dying server has finished revoking its class objects.

The fix

The code for fixing this is simple. I simply created a new class that inherits from CAtlExeModuleT<T> and then overrides the Lock and Unlock methods.

template <typename T>

class CAtlExeModuleT_NoRace : public CAtlExeModuleT<T>

{

public:

typedef CAtlExeModuleT<T> base;

#if _ATL_VER < 0x0A00

LONG Lock(void)

{

CoAddRefServerProcess();

return base::Lock();

}

LONG Unlock(void)

{

CoReleaseServerProcess();

return base::Unlock();

}

#else

//removed compiler messages

#endif

};

I use CoxxxxServerProcess functions to explicitly manage the class object registration, and then I delegate to the base class which maintains its own reference count.

The reason I do that is that the base::Unlock function is the one which triggers the module shutdown, and I don’t want to mess with that. And Unlock works only when paired with Lock, so the only thing I really need to do is to call the CoxxxxServerProcess routines to prevent possible activation requests. The rest is still done by CAtlExeModuleT<T>.

I put a compiler condition around my implementation of the new Lock and Unlock methods, because they are only needed for current versions of ATL. The next version of ATL won’t have this problem, so it would be dangerous to meddle with the reference counting, especially since I cannot predict what side effects this may have.

The conditional compilation allows you to safely upgrade the project to the next version of ATL. The compiler messages which show up in the compiler output inform you that it is no longer needed to inherit from CAtlExeModuleT_NoRace as shown below:

class CStuff_ServerModule : public CAtlExeModuleT_NoRace< CStuff_ServerModule >

{

public :

DECLARE_LIBID(LIBID_Stuff_ServerLib)

DECLARE_REGISTRY_APPID_RESOURCEID(IDR_STUFF_SERVER, "{597B5DFF-DACE-42B0-9E49-54A09C10B2BB}")

};

The only thing you have to do to use my new module class is to include the header and change CAtlExeModuleT to CAtlExeModuleT_NoRace.

And if you upgrade the project to VC10, then you can simply change it back as indicated by the compiler output messages that are printed out when the compiler parses my class.

Conclusion

I was surprised that this bug was still in the ATL headers after all these years. When I contacted the VC++ team I was half expecting them to tell me that I had missed something. This issue is what caused CoAddRefServerProcess to be created after all, over a decade ago, so a COM programmer implementing lifetime management should be aware of it.

However, the modular and hierarchical nature of ATL made it easy to fix this problem, so I guess that is a plus for ATL. And mistakes happen. It is always a bit painful if it happens in widely used libraries, but I too have had my share of whoopsies over the years. It is sad but it happens.

The header file with the fix is attached to this blog post under the MIT license.

Practial ATL: Implementing a non-standard class object

vanDooren — Fri, 20 Feb 2009 21:03:00 GMT

In my previous article in this series, I explained how class objects are associated with COM objects, and how it all fits together.

Armed with that knowledge, I can now demonstrate how to provide a custom class object that will allow us to support parameterized construction.

If you haven’t already read my previous article on class objects, please do so first because I am going to assume you did J

The non-standard class object interface: IStuffCreator

A class object can have any interface you want, just like the COM object we have created earlier. It is just another COM object after all. My custom interface looks like this:

[
      object,
      uuid(6DD69CDB-3128-432b-B335-773A287E6F06),
      oleautomation,
      helpstring("IStuffCreator Interface"),
      pointer_default(unique)
]
interface IStuffCreator : IUnknown {
[id(1), helpstring("method MakeMeAStuff")]
    HRESULT MakeMeAStuff(
      [in] BSTR name,
      [in] REFIID riid,
      [out, iid_is(riid)] IUnknown** ppStuff);
};

The bits between the square brackets are the attributes that are applied to this interface. ‘Object’ specifies that it is a COM interface. The uuid specifies the GUID that identifies the interface. ‘oleautomation’ specifies that the arguments being used are all automation compatible. The meaning of ‘helpstring’ is pretty obvious, and pointer_default specifies the assumptions that the marshaller can make about pointers.

If you want to know the details about it, then have a look in MSDN for MIDL attributes.

The interface itself is very simple. Instead of deriving from IDispatch like the IStuff interface, this interface derives from IUnknown. Deriving from IDispatch would only complicate matters without any benefit.

The interface has only one method, which is responsible for handing out new stuffs.

The parameter that will be used for constructing the CStuff instance itself is the ‘name’ parameter. If the CStuff would need multiple parameters, they would have to be part of the parameter list so that they can be used at the time of construction. The riid and ppStuff parameters have the same meaning as those of CoCreateInstance.

The declaration of CStuffCreator

If we want to implement this interface, we add a new C++ class with the name ‘CStuffCreator’ to our project, and implement our IStuffCreator interface

class CStuffCreator:
public IStuffCreator,
public CComObjectRootEx<CComGlobalsThreadModel>
{
public:

CStuffCreator(void){}
virtual ~CStuffCreator(void){}

BEGIN_COM_MAP(CStuffCreator)
    COM_INTERFACE_ENTRY(IStuffCreator)
END_COM_MAP()

//IStuffCreator methods
HRESULT STDMETHODCALLTYPE MakeMeAStuff(
    BSTR name,
    REFIID riid,
    IUnknown **ppStuff);

// helper method
void SetVoid(void* pv);
_ATL_CREATORFUNC* m_pfnCreateInstance;
};

As you can see, it is pretty simple. The CComObjectRootEx base class helps with reference counting, and is the same that is used by the other COM objects in our project.

We add IStuffCreator to the interface map using the ATL macros, and then we simply declare the single MakeMeAStuff method that we need to implement.

The SetVoid method and the function pointer are part of the default C++ interface for class objects in ATL, and are explained in my previous article.

The implementation

The implementation of CStuffCreator is even simpler than the declaration:

HRESULT STDMETHODCALLTYPE CStuffCreator::MakeMeAStuff(

BSTR name,

REFIID riid,

IUnknown **ppStuff)

{

CComPtr<IUnknown> newStuff;

HRESULT hr = m_pfnCreateInstance(

NULL, __uuidof(IUnknown), (void**)&newStuff);

if(FAILED(hr))

return hr;

return newStuff->QueryInterface(riid, (void**)ppStuff);

}

void CStuffCreator::SetVoid(void* pv)

{

m_pfnCreateInstance = (_ATL_CREATORFUNC*)pv;

}

Ignore the ‘name’ parameter for now. That will be used when we are going to initialize the new instance. The bare functionality of our class object is to use the function pointer to create a new instance. I could have done this in 1 step instead of 2 (the QueryInterface isn’t necessary at this point) but this way I can slide in some additional functionality later on.

And you can see now that the class object will receive the creator function pointer via the ‘SetVoid’ method. This mechanism was explained in my previous article.

Parameterized construction

The point of using a non-standard class object in this demonstration was to be able to create objects that need some sort of initialization before the client has access to them. Normally, ATL supports only default constructors using, because IClassFactory is a generic interface.

For parameterized constructors, we are going to have to do something special.

There are essentially 2 ways we can do this. The first is to change CStuff to have a parameterized constructor. But ATL can’t handle that, and it would mean we have to abandon the incredible convenience of being able to use the ATL::Creator<T> class, ATL::CComObject class, and other helper classes that take the pain out of COM.

Don Box wrote an article about parameterized construction, many years ago, and this was the approach he took. Of course, ATL was much simpler in those days, and abandoning it was perhaps less of an inconvenience.

The approach I am taking is much simpler: I cheat!

That’s right. I cheat. I am not going to implement parameterized constructors. I am going to fake it. But from the client’s point of view, it’s going to look exactly the same, so he won’t mind. And from the server’s point of view, it will be much simpler to implement, so everybody wins.

The second approach to implementing parameterized construction is to implement an internal interface on CStuff, which will be used for initializing it. The new CStuff instance will not be accessible to anyone until the first interface pointer is handed out by the class object. And we are the ones implementing the class interface, so we can make sure that the new instance is initialized properly before it is released into the wild.

The initialization interface

The interface for initializing a CStuff is pretty simple (I left out the header for clarity):

interface IStuffInit : IUnknown {

[id(1), helpstring("method InitStuffInstance")]

HRESULT InitStuffInstance(

[in] BSTR Name);

};

Our CStuff instances are supposed to have a name, so that’s what needs to be in the init interface. If they would also need an address, a hat, and a pair of pants, we would have to put those in the parameter list of the InitStuffInstance method as well.

Implementing the initialization interface

Implementing the interface in CStuff is easy I highlighted the changes that have to be made to the declaration.

// CStuff

class ATL_NO_VTABLE CStuff :

public CComObjectRootEx<CComMultiThreadModel>,

public CComCoClass<CStuff, &CLSID_Stuff>,

public IDispatchImpl<IStuff, &IID_IStuff, &LIBID_Stuff_ServerLib, /*wMajor =*/ 1, /*wMinor =*/ 0>,

public IStuffInit

{

private:

CComBSTR m_Name;

public:

DECLARE_CLASSFACTORY_EX(CStuffCreator)

// removed irrelevant stuff

BEGIN_COM_MAP(CStuff)

COM_INTERFACE_ENTRY(IStuff)

COM_INTERFACE_ENTRY(IDispatch)

COM_INTERFACE_ENTRY(IStuffInit)

END_COM_MAP()

// removed irrelevant stuff

HRESULT STDMETHODCALLTYPE InitStuffInstance(BSTR Name);

};

First we add IStuffInit to the inheritance list, we add it to the COM map, and then we add the declaration for InitStuffInstance.

Of course, the name has to be part of the objects state, so we add a CComBSTR variable to hold it.

And finally, we use the DECLARE_CLASSFACTORY_EX macro to change the class object for CStuff from CComClassFactory to CStuffCreator.

The actual implementation of the interface consists of the simple code

STDMETHODIMP CStuff::InitStuffInstance(BSTR Name)

{

m_Name = Name;

return S_OK;

}

Note the ease with which CComBSTR allows us to perform simple string operations.

Initializing the new CStuff object

Now that there is a dedicated function to handle the initialization, the changes to CStuffCreator are trivial:

HRESULT STDMETHODCALLTYPE CStuffCreator::MakeMeAStuff(

BSTR name,

REFIID riid,

IUnknown **ppStuff)

{

CComPtr<IStuffInit> newStuff;

HRESULT hr = m_pfnCreateInstance(

NULL,__uuidof(IStuffInit), (void**)&newStuff);

if(FAILED(hr))

return hr;

hr = newStuff->InitStuffInstance(name);

if(FAILED(hr))

return hr;

return newStuff->QueryInterface(riid, (void**)ppStuff);

}

What about double initialization

By now you might be wondering: But what if the client application retrieves this interface and performs another initialization?

However, this will not be possible.

It’s true that I put the interface in our IDL file, and that I implemented it in the CStuff object. But I did not put that interface in the library declaration, nor did I put it in the interface list of the coclass.

So for all practical purposes, this interface is not accessible to the client application. If you are really paranoid, you could argue that an attacker might know the interface GUID, and guess the interface definition, and with some trial and error make it work.

Even then, the fix is quite simple. You can add a boolean to your class that indicates if the object was initialized or not. Set it to false in the constructor, and then the InitStuffInstance can either perform initialization if it is false and set it to true, or return E_FAIL if it is already true.

You don’t even have to care about race conditions, because there aren’t any. Only the class object can perform the real initialization. And since no one else has an interface pointer yet, there can’t be a race.

The client side

Creating a new CStuff has changed a little bit, because now we can’t use CoCreateInstance anymore. So construction is now done in 2 stages:

CComPtr<IStuffCreator> stuffClass;

hr = CoGetClassObject(

__uuidof(Stuff), CLSCTX_ALL, NULL,

__uuidof(IStuffCreator), (void**)&stuffClass);

if(FAILED(hr))

return hr;

//make a new stuff

CComBSTR stuffName = L"Kato";

CComPtr<IStuff> newStuffUnk;

hr = stuffClass->MakeMeAStuff(

stuffName.m_str,

(GUID*)&__uuidof(IStuff),

(IUnknown**)&newStuffUnk);

if(FAILED(hr))

return hr;

It’s still very simple though. First we get the class object, and then we use the IStuffCreator interface to create the new CStuff.

Server lifetime management

Everything works, hip hip hurray… until you destroy the first CStuff instance, and then try to create another one. That is where it all goes pear shaped.

The problem is that we haven’t taken care of managing the lifetime of the server (the exe). The key issue here is the module lock reference count.

Each COM module (exe or dll) has a global reference count to keep track of how many objects are ‘alive’. COM objects exist physically inside a DLL or EXE. So the EXE has to stay alive as long as there are living objects inside. Conversely, as soon as there are no living objects anymore, the EXE should shut down.

What happens in our example is that CStuff raises the module ref count when it is created. And when the CStuff instance destroys itself, it lowers the module ref count. If you remember correctly, the class object itself uses CComObjectNoLock as its COM core. This means that its existence does not influence the module lifetime.

So when the CStuff instance destroys itself and lowers the module ref count, the ref count reaches 0 and the server EXE shuts down.

The client is unaware of this and tries to use the class object again. It still thinks it has a valid interface pointer, but the underlying class object is long gone and there will be a critical error in the client application.

You might say ‘well duh, then don’t use CComObjectNoLock, doofus’ but that has the opposite effect. The COM runtime will always have the interface pointer that was used for registering the class object with CoRegisterClassObject. This means that the class object would not destroy as long as the server lives. And the server would keep living as long as the class object was around. This is clearly not the answer.

IExternalConnection to the rescue

The key to solving this problem is a rarely used standard interface. IExternalConnection is an interface that can be used by an object to track how many external connections there are to itself. This is basically an alternative to the standard object reference count.

There are 2 methods in this interface:

STDMETHODIMP_(DWORD) AddConnection(

DWORD extconn, DWORD dwReserved);

STDMETHODIMP_(DWORD) ReleaseConnection(

DWORD extconn,

DWORD dwReserved,

BOOL bLastUnlockReleases);

If a client requests any class object, the COM runtime will retrieve it. But before the interface pointer is released to the client, it will request some interfaces itself for housekeeping purposes. One of those interfaces is IExternalConnection.

If a client requests an interface pointer to a class object, the marshaller will call AddConnection when it hands out the pointer, and ReleaseConnection when the pointer is destroyed.

So all we have to do is to make sure that a call to AddConnection will increment the module lock count, and a call to ReleaseConnection releases the module lock count.

Implementing IExternalConnection

If you look in MSDN, you won’t find it. It is undocumented, but the ATL headers contain an implementation for IExternalConnection. It’s called IExternalConnectionImpl<T>.

To use it, we only have to change CStuffCreator so that it inherits from it, and add it to the COM interface map.

class CStuffCreator:

public IStuffCreator,

public IExternalConnectionImpl<CStuffCreator>,

public CComObjectRootEx<CComGlobalsThreadModel>

{

public:

//removed irrelevant stuff

BEGIN_COM_MAP(CStuffCreator)

COM_INTERFACE_ENTRY(IStuffCreator)

COM_INTERFACE_ENTRY(IExternalConnection)

END_COM_MAP()

//IExeternalConnectionImpl<T> extension

void STDMETHODCALLTYPE OnReleaseConnection(

bool bThisIsLastUnlock, bool bLastUnlockReleases);

void STDMETHODCALLTYPE OnAddConnection(

bool bThisIsFirstLock);

//removed irrelevant stuff

};

That is really all there is to it. The default implementation does not modify the module lock count. But it has 2 extension points that we can override. So that’s what we do to make sure the module lock count is managed properly.

void STDMETHODCALLTYPE CStuffCreator::OnAddConnection(

bool bThisIsFirstLock)

{

_pAtlModule->Lock();

}

void STDMETHODCALLTYPE CStuffCreator::OnReleaseConnection(

bool bThisIsLastUnlock, bool bLastUnlockReleases)

{

_pAtlModule->Unlock();

IExternalConnectionImpl<CStuffCreator>::OnReleaseConnection(

bThisIsLastUnlock, bLastUnlockReleases);

}

The only special thing that still needs mentioning is that OnReleaseConnection delegates back to the default implementation in IExternalConnectionImpl. It does this because there is something that needs to be done when the last external connection is closed.

void OnReleaseConnection(

bool bThisIsLastUnlock, bool bLastUnlockReleases)

{

if (bThisIsLastUnlock && bLastUnlockReleases)

CoDisconnectObject(static_cast<T*>(this)->GetUnknown(), 0);

}

If the last connection closes, the class object needs to call CoDisConnectObject to destroy its marshalling stub. Now I could copy paste this code into my implementation of OnReleaseConnection, but it is simpler to just delegate back to the base class for stuff that needs doing anyway.

The result

Now that we have provided this implementation of IExternalConnection, the server EXE will stay alive until all CStuff instances are gone, and noone is using the class object anymore. Only then will the EXE terminate.

CComClassFactory revisited

If you look at the interface implemented by CComClassFactory, you’ll notice that IExternalConnection is missing. And yet, server lifetime of CComClassFactory is managed correctly.

The reason for this is that the COM runtime treats IClassFactory class objects differently from other class objects. It will check if the class object supports IClassFactory. And if it does, it calls IClassFactory::LockServer to indicate that someone is using the class object. And it unlocks the class object when the class object reference is released again.

In a way, it uses the IClassFactory in almost the same way as it would use IExternalConnection. And in a way it is too bad that this functionality was included in IClassFactory directly, instead of making IClassFactory inherit from IExternalConnection. The reasons for this are lost in the mists of time. But whatever the reasons, this is why IClassFactory doesn’t need IExternalConnection.

Conclusion

As you can see, implementing a non standard class object is not so hard. As long as you have a basic understanding of the underlying COM principles, ATL takes care of all the dirty work.

And by using these techniques, we can implement COM objects that need initialization before they can safely be used by the client. The source code for this article is available under the MIT license as usual.

And again I’d like to add a thank you for Microsoft’s Tim Springfield and Amit Mohindra

Practical ATL: Understanding the class object

vanDooren — Tue, 10 Feb 2009 07:46:00 GMT

In my previous article in this series, I explained how to create a simple COM server and implement a method that returns an enumerator object.

It was all done very easily, but the ATL wizards hide some of the things that you really ought to know for my next article. For starters: where is the class object?

The class object is the COM object responsible for creating new instances of the CStuff COM object that I demonstrated in a previous article. But we didn’t have to write any code for it, nor is it even visible… so what’s up with that?

Some more on class objects

The class object of a COM server is the object that is actually responsible handing out instances of the COM object. The class object is a COM object itself, but only 1 instance of it normally exists in a module (DLL or EXE) per defined COM object type.

For example, the ‘Stuff’ COM object that I implemented in my previous article uses the standard class factory with IClassFactory as the interface for handing out new CStuff instances.

The most important method in IClassFactory is ‘CreateInstance’ which simply creates a new instance of the COM object and then passes an interface pointer back to the caller.

Normally, this process is invisible to the client application, because ‘CoCreateInstance’ uses IClassFactory behind the scenes so that the client doesn’t have to bother with it.

First it calls ‘CoGetClassObject’ to get the class object for the requested COM object, and then it uses the IClassFactory interface of that class object to create a new instance. That instance is then passed to the caller, as an interface pointer of the type that was requested in the call to ‘CoCreateInstance’.

I think it is safe to say that ‘IClassFactory’ (or one of its relatives) is the interface on the class object of 99.99% of all COM objects that have a class object.

The default class factory

By default, CStuff uses CComClassFactory as its default class object. But looking at the declaration of CStuff, it is not immediately apparent how it does this.

CStuff derives from CCoClass<T>, which has the macro DECLARE_CLASSFACTORY in its implementation.

For EXE servers, this macro expands to DECLARE_CLASSFACTORY_EX(ATL::CComClassFactory), which in turn expands to

typedef ATL::CComCreator< ATL::CComObjectNoLock< ATL::CComClassFactory > > _ClassFactoryCreatorClass;

DLL servers use CComObjectCached instead of CComObjectNoLock, but I don’t cover DLL servers in this article.

As you can remember from the first article that featured CStuff, the C++ COM classes are missing the IUnknown core by default, because the implementation of that core depends on how we want to the object to behave.

This means we have to shove our class into a COM object to make it whole. For class objects, the CComObjectNoLock is most appropriate, because the lifetime of the object should not contribute to the lifetime of the server executable. Otherwise there would be a deadlock.

The server would start, and create the class object. And then the server would continue to live until the class object was destroyed. But since the COM runtime will always have a reference to the class object, the reference count would never reach zero, the class object would never be destroyed, and the server would never be able to shut down.

And finally, this class definition is shoved into a creator object which will help us by creating a properly initialized instance of the class object itself via its static CreateInstance function.

Associating the COM object C++ class with the class factory

So the COM object C++ class has a typedef that identifies the class object type. But how does the executable use this fact?.

This is where things get a little hairy, and possibly start to make ominous squishy sounds J

After the class declaration of CStuff, there is the innocuous line OBJECT_ENTRY_AUTO(__uuidof(Stuff), CStuff)

The OBJECT_ENTRY_AUTO is a macro that expands to this:

#define OBJECT_ENTRY_AUTO(clsid, class) \
      __declspec(selectany) ATL::_ATL_OBJMAP_ENTRY __objMap_##class = {&clsid, class::UpdateRegistry, class::_ClassFactoryCreatorClass::CreateInstance, class::_CreatorClass::CreateInstance, NULL, 0, class::GetObjectDescription, class::GetCategoryMap, class::ObjectMain }; \
      extern "C" __declspec(allocate("ATL$__m")) __declspec(selectany) ATL::_ATL_OBJMAP_ENTRY* const __pobjMap_##class = &__objMap_##class; \
      OBJECT_ENTRY_PRAGMA(class)

Don’t worry if the sight of this code makes you nauseous. That’s normal. I highlighted the important parts

What it does is simple. It’s just how it does it that is ugly. It creates a map entry for the ATL innards of our server executable. In this map entry, it associates the static method for creating the class object with the static method for creating an object instance.

Creating the class object

Now that the class object and the COM object are associated with each other, the next bit of code in the ATL headers is where the magic happens:

_ATL_OBJMAP_ENTRY* pEntry;
if (m_pObjMap != NULL)
{
    pEntry = m_pObjMap;
    while (pEntry->pclsid != NULL && hr == S_OK)
    {
      hr = pEntry->RegisterClassObject(dwClsContext, dwFlags);
      pEntry++;
    }
}

When the ATL module initializes, it will iterate over all elements in the object map and register each one. And this registration process entails the following:

    IUnknown* p = NULL;
    if (pfnGetClassObject == NULL)
      return S_OK;
    HRESULT hRes = pfnGetClassObject(
                         pfnCreateInstance,
                         __uuidof(IUnknown),
                         (LPVOID*) &p);
    if (SUCCEEDED(hRes))
      hRes = CoRegisterClassObject(
               *pclsid, p, dwClsContext, dwFlags, &dwRegister);
    if (p != NULL)
      p->Release();
    return hRes;

If you do a bit of spelunking in the headers, you’ll see that pfnGetClassObject is a function pointer that maps to _ClassFactoryCreatorClass::CreateInstance. So this code simply creates a new class object and registers it.

Again, this is specific to EXE servers. In DLL servers, the class object is created on demand in the DllGetClassObject function, which is called by CoGetClassObject if the COM server is a DLL.

Creating CStuff instances

If you look at the _ClassFactoryCreatorClass::CreateInstance, you’ll see that its first parameter is a void *. And the value that is passed in this parameter by the ATL module is the address of the static method for creating new object instances.

_ClassFactoryCreatorClass::CreateInstance simply creates a new class object ‘p’, and then calls p->SetVoid(pv); to pass the aforementioned function pointer to the class object.

And the SetVoid method is no more complicated then this:

void SetVoid(void* pv)
{
m_pfnCreateInstance = (_ATL_CREATORFUNC*)pv;
}
_ATL_CREATORFUNC* m_pfnCreateInstance;

I guess the original developer hadn’t had his morning coffee yet when he came up with this name. It manages to be 100% factually correct, while being 100% opaque at the same time. But it does the job.

When the class object has to create a new instance, it simply does the following:

STDMETHOD(CreateInstance)(LPUNKNOWN pUnkOuter, REFIID riid, void** ppvObj)
{
    // snip irrelevant code
        hRes = m_pfnCreateInstance(pUnkOuter, riid, ppvObj);
    // snip irrelevant code
    return hRes;
}

Providing a new class factory

Now that you understand how the class object plumbing is hooked up, it is easy to see how you can implement your own class object.

Remember how DECLARE_CLASSFACTORY_EX maps a class object to the _ClassFactoryCreatorClass typedef?

In C++, a typedef in a derived class can override a typedef in a base class. So even though the CCoClass<T> base class says that the _ClassFactoryCreatorClass maps to ATL::CComCreator< ATL::CComObjectNoLock< ATL::CComClassFactory > >,

We can override this by simply putting DECLARE_CLASSFACTORY_EX(CStuffCreator) In the declaration for CStuff. That’s really all there is to it. The macro magic takes care of the rest.

Conclusion

If changing the class object for a COM server is as trivial as adding a single line to your COM server C++ class, then why did I just write 4 pages of stuff?

I wrote all this because if I tell someone to use a macro to do something non-trivial, I want him or her to understand what is going on. There are already far too many programmers on this planet who do things without understanding what it is they do, or how it works. And I am not going to add some more by glossing over the details.

Of course, judging by the number of visitors on this corner of the internet that I call ‘home’, I stand little risk of doing so no matter how crappy I would write J

This article was originally part of my article on implementing a custom class object. But after I spent more than 3 pages on the previous explanation, I thought it would be best to put it in a separate article.

A big thank-you goes to Jim Springfield of the Architects team within the developers division (I think) of Microsoft for taking the time to review this article for correctness, and to Amit Mohindra for helping my questions and articles reach the right people.

And you can find the demo project attached to this article under the MIT license.

Practical ATL: recognizing marshalling problems

vanDooren — Tue, 27 Jan 2009 11:30:00 GMT

Before I publish my next article, I should explain something else about the black magic aspect of ATL / COM: Marshalling.

The Stuff server we made in the previous article is an out-of-proc server. This means that the arguments being passed to / from a stuff instance have to be somehow moved back and forth between different process contexts (or different threading apartments).

In the case of IStuff, we don’t really have to bother, because IStuff derives from IDispatch. And in that case, there is a marshaller (aptly named Universal Marshaller) that can be used to do all this for you.

If IStuff wouldn’t derive from IDispatch, then we would need to provide our own proxy – stub interface that knows how to marshal the interface. With ATL, this is not difficult. You’ll notice that the Stuff solution of my previous article had a ‘Stuff_ServerPS’ project that I didn’t mention before.

That solution will build the proxy stub dll that will perform the marshaling. It is disabled by default, so you have to enable it in the build configuration before it is built. Also remember to rebuild the proxy, every time the IDL files of your project change.

So for the remainder of this article I will show what can happen if there is a marshalling problem. Don’t worry if it all seems ugly, gory, and incomprehensible. Normally you won’t have to deal with issues like this. But if you should ever do, you should at least recognize the signs that you have a marshaling problem.

Confused? It’s the marshaller

If you find yourself in a situation where very weird things happen, suspect a marshalling problem.

If you get the error code E_NOINTERFACE when you know for sure that your object implements it properly, suspect a marshalling problem.

If you get a crash while using an interface pointer that was properly created without getting a HRESULT that indicated a problem,… that’s right: you can bet it’s a marshaling problem.

Example

The article I am currently writing discusses implementing a non standard class object, for the purpose of parameterized construction of the Stuff object. Never mind for the moment about what any of that means.

For all practical purposes: it just means that I defined the following interface via copy / paste / edit:

[

      object,
      local,
      uuid(6DD69CDB-3128-432b-B335-773A287E6F06),
      dual,
      helpstring("IStuffCreator Interface"),
      pointer_default(unique)
]
interface IStuffCreator : IUnknown {
[id(1), helpstring("method MakeMeAStuff")]
    HRESULT MakeMeAStuff(
      [in] BSTR name,
      [in] REFIID riid,
      [out] void ** ppStuff);
};

There were 4 things wrong with this interface (showing just why you shouldn’t define IDL interfaces in a hurry). This has to be some sort of record.

‘local’ shouldn’t be there.
‘dual’ shouldn’t be there.
‘void’ should be ‘IUnknown’
‘ppStuff’ should be attributed with riid_is(riid).

So let’s have a look at the result of these different issues.

Issue 1: local

‘local’ simply shouldn’t be there. ‘local’ is a MIDL attribute that –when use in an interface header- causes MIDL not to generate stubs for that interface.

The effect on our interface is that the IStuff specific methods are not forwarded to the remote server.

You might wonder: why does it exist. The answer is: in case you want to write custom marshalling code, or in case the calls never have to be remoted. In that case you can have a local interface and a remote interface. And usage and marshalling could be optimized.

IUnknown is such an interface. But it has custom stub code to deal with things like remoting

The result of using ‘local’ on my IStuffCreator interface is that trying to call GetEnum on the returned IStuff interface pointer will result in a debug assertion: Run-Time Check Failure #0 - The value of ESP was not properly saved across a function call. This is usually a result of calling a function declared with one calling convention with a function pointer declared with a different calling convention.

And if you have this problem with COM, it is usually indeed an indication that you have a calling convention mismatch. In this case, it is a side effect of the fact that we are trying to execute a method that isn’t remoted.

Issue 2: dual

This one shouldn’t be here either. Dual indicates that the interface implements IDispatch, and this could have an effect on marshalling.

In this case, it worked just fine. Perhaps it is because of the fact that I use my object as a class object, or perhaps because my interface itself uses only arguments that are automation compatible, or perhaps because the stars are aligned in the proper configuration.

I don’t know. And anything you don’t know for certain will hurt you when dealing with DCOM. It has to be correct.

Issue 3: void

This one is interesting.

If I want to pass an interface pointer, it has to be passed as an IUnknown**, instead of a void**. This is because Universal Marshaller has no idea on how to marshal something which has only contextual meaning, nor does the IDL compiler know what you mean with it.

With this I mean that my server knows it will be an interface pointer, and my client will know it is an interface pointer, but the stuff in the middle doesn’t. It sees just a memory address.

I used void** because that is what QueryInterface itself uses. And what’s good enough for QI should be good enough for me. Except QI has custom code for marshalling its arguments across, which I don’t. So QI can use void** because both the proxy and the stub know what is going on.

So what happens when you use void**?

I request the class object like this:

hr = CoGetClassObject(
__uuidof(Stuff),

CLSCTX_ALL,

NULL,

__uuidof(IStuffCreator),

(void**)&stuffClass);

And the resulting hr is 0x80020008: Bad variable type. There is absolutely nothing wrong with this line of code. But because there is something in the interface itself that confuses the marshaller, it tells me to sod off when I want that specific interface.

Credit where it is due: I figured this one out thanks to someone whose name I can’t figure out J. But his blog post is here.

Issue 4: iid_is(riid)

This one is also a good one.

The iid_is informs the marshaller that riid points to the identifier which identifies the interface that is being marshaled.

This way, the marshaller knows how to marshal the interface that was specified with riid. Without that part, it would only marshal the IUnknown part of the interface, and not the IStuff specific part.

As a result, you can use the IUnknown part of the interface without a problem, but if you use the IStuff specific part, you will get an access violation, stack corruption or other interesting phenomenon.

Conclusion

A couple of things can be concluded at this point.

First of all: Marshaling problems are hard to debug, because the problems occur outside of your source code.

If you use any interfaces that don’t derive from IDispatch, then you have to enable the ‘xxxPS’ project in your solution so that the ProxyStub dll will be built and registered. And of course, this dll has to be part of your install procedure. Without this stub, you could get access violations, or you get an E_NOINTERFACE error if the marshaler doesn't know one of those interface exist.

As I have shown, you need to be careful and pay attention when you manually define COM interfaces. If you don’t, then you can have all sorts of hard to diagnose problems.

And if you ever find yourself in a situation where nothing makes sense, then suspect marshalling issues and have a long hard look at your IDL files.

File a bug report and make Visual Studio a better product

vanDooren — Thu, 22 Jan 2009 06:50:00 GMT

If you find a bug in Visual Studio, report it on http://connect.microsoft.com If the bug affects the version of VS that is in development at that time, there is a significant chance it will get fixed before the next release. Just this week I got confirmation that all 3 bugs that I filed in the last months have been resolved.

The first one is a nasty one. If you add an ATL project to an existing, non-empty solution, the ATL wizards will either hang the IDE or fail to work when adding an ATL object. I found a workaround for it, but it involved hand editing the solution file.

The second one is a small bug in the TR1 regex parser. No workaround, other than to restructure the regex.

And the last one was an error in the MSDN documentation about open modes with the fopen family, which counts as a bug because it triggered runtime errors.

All 3 have been fixed. I especially appreciate the first one. I am working on ATL projects, and I develop my COM server in the same solution as a couple of related non-ATL projects. So every time I work with ATL projects, I can have a warm and fuzzy feeling that because of me, VS is now a better product. :)

Practical ATL: Implementing an enumerator object

vanDooren — Wed, 21 Jan 2009 19:55:00 GMT

This is my first article about practical ATL examples, which I already mentioned here.

In this example I will create an ATL Server which implements 1 custom interface: IStuff. Hardly an inspiring name, but I couldn’t think of anything better before I had my first coffee.

This interface will have 1 method which will return an enumerator to the caller. Specifically, this enumerator will implement IEnumString.

Enumerators are part of the interface of many real-life COM servers, but creating your own is an arduous task. ATL makes this much easier, but an example makes things so much easier still.

Since client code will have to deal with this enumerator (and to have a demo app) I will also create a client application that creates an instance of the server, consumes the IStuff interface and then enumerate the items contained in the returned enumerator.

Before we get started

If you don’t own a copy of ‘ATL Internals, Second Edition’ yet, I strongly advise you to buy it. It is worth the money, and apart from leading you through all the interesting and gory stuff, it also starts by explaining the actual process of creating an ATL server through the IDE, as well as implementing methods, properties and events.

I am only going to cover that aspect very lightly. It should be detailed enough for anyone who has done C++ development using VS:

Create a new ATL Server project with project name ‘Stuff_Server’ and a solution name ‘Stuff’.
Choose the EXE project type and click ‘finish’
Add a new class to the project, and specify ATL Simple Object. Use ‘Stuff’ as the name of the object.
Switch to classview, and add a new method to the IStuff interface with the prototype ‘HRESULT GetEnum(IUnknown ** ppEnum). ppEnum should be marked as an ‘out’ parameter.

Build the project, and you should be ready for the actual work. The ATL wizards will have implemented all the IDL and DCOM stuff for you, and you only have to implement the functionality.

Finally, I always end by changing the project configuration to use static CRT and ATL runtimes instead of dynamic runtimes. This makes it infinitely more easy to distribute the resulting executables without having to bother with registration and installation of the runtime dlls and their manifests.

Creating the server

Now that the ATL wizards have generated the function skeleton for us, it is time to do something with it:

STDMETHODIMP CStuff::GetEnum(IUnknown** ppEnum)
{

}

The enumerator

Before we can do anything else, we have to think about the actual enumerator for a bit. In our case we implement IEnumString. This is an interface for enumerating string (who’d have thought, eh?).

Enum interfaces all look alike, and ATL helpfully has a stock implementation for generic enumerators that allows you to specify the Interface, GUID, datatype and data type copy traits as template parameters. In order to facilitate coding, we make a typedef for this class:

typedef CComEnum<

IEnumString,

&IID_IEnumString,

LPOLESTR,

_Copy<LPOLESTR> > CComEnumStringAbstract;

This will specialize the enumerator for our purposes. The use of the first 3 parameters is obvious. Don’t worry about the _Copy parameter. I will cover that later.

The enumerator itself is a normal COM object, so it needs lifetime management, interface managements, and all other things you really don’t want to care about for something so simple. At this point, CComEnumStringAbstract will still be an abstract class because it can’t decide how to do these things for you.

In order to make our enumerator a full fledged COM object, we have to shove it into a bare COM object. This is very easy:

typedef CComObject<CComEnumStringAbstract> CComEnumString;

And that is all there is to creating an enumerator! Anyone who has ever implemented one manually will agree that this is infinitely more convenient

Implementing GetEnum

The implementation for GetEnum is also fairly easy (comments removed for clarity)

STDMETHODIMP CStuff::GetEnum(IUnknown** ppEnum)
{
CComEnumString * thEnumObj = NULL;
HRESULT hr = CComEnumString::CreateInstance(&thEnumObj);
if(FAILED(hr))
return hr;

CComPtr<CComEnumString> thEnum(thEnumObj);

LPOLESTR strings[] = {L"One", L"Two", L"Three", NULL};
hr = thEnum->Init(&strings[0], &strings[3], NULL, AtlFlagCopy);
if(FAILED(hr))
return hr;

hr = thEnum->QueryInterface( __uuidof(IUnknown), (void**)ppEnum);
return hr;
}

First we create a new CComEnumString object. It is worth noting that the object returned by CreateInstance has a refcount of 0. Before we do anything with it we should AddRef it, but instead of doing that manually, I do it via assignment to the CComPtr. smart pointer. That way the object also gets released automatically at the end.

Then we have to make sure the enumerator has some data to enumerate. We do this by populating an array, and then telling the enumerator to copy the data out of it into its own internal buffer. This is done via the AtlFlagCopy flag.

It might seem weird if you come from a C background, but the ‘End’ pointer has to point to the first position beyond the last element. This is normal with collections in C++. That is also why I populate the array with a NULL pointer at the end; So that there is something in the array after the last ‘real’ element. This way there is no buffer overflow.

Using AtlFlagCopy has the overhead of having to copy the data, but otoh the enumerator does not have to care about the lifetime of the array. We could also use new[] to create a new array and then use the AtlFlagTakeOwnership flag to indicate that it is up to the enumerator to call delete[] and clean up. But in this example we can’t do that (read below).

When the enumerator is initialized, we return an IUnknown interface pointer to the caller. Since this will have increased the enumerator ref count, the enumerator object will keep on living when the CComPtr goes out of scope and releases its own reference to the enumerator.

There. That is all there is to it on the server side. No vast amounts of copy pasted boilerplate code, but just a couple of lines of powerful ATL incantations J

_Copy<LPOLESTR> revisited

If an enumerator has to copy data to a client, or if it has to copy data into its internal buffers in the ‘Init’ function, it has to know how to do this.

It’s all good and well to do a simple memcpy of an ULONG or a double, but for strings (LPOLESTR) and interface pointers (IUnknown) this won’t work. There’s ref counting and memory ownership to deal with.

Of course, the template class, not knowing about this, can’t anticipate what you want, so instead it needs a policy class to delegate to. That is where the _Copy<> template class comes in. It has 3 methods: Init, Destroy, and Copy. Those methods perform the actual operations needed to copy an instance of T to another instance of T, to initialize a new one, and to destroy one.

There are several stock specializations for _Copy<>, one of which works for LPOLESTR. So rather than copying the LPOLESTR pointer itself, it calls CoTaskMemAlloc to create a buffer for the copy of the string, and then copies over the data.

If you look at my example more closely, you will also see why I cannot simply new up an array of string literals and pass those to the Init function with the AtlFlagTakeOwnership flag.

LPOLESTR is superficially identical to wchar_t*. So using a string literal instead of an LPOLESTR (which is what I do in my example) will work without a hitch. But if I’d new up an array and tell the enumerator to manage it, that is when it would all go pear shaped.

Because the enumerator would not only call delete[] on the array, but it would also call _Copy::Destroy for each of the items. And trying to free up a string literal with CoTaskMemFree will result in a hard to diagnose crash.

Installing the server

On a development station this is done automatically by Visual Studio. But if you need to do this on another PC, you just execute the server from the command line with the /RegServer argument.

Creating the Client

Creating the client application is fairly simple, but I wanted to show how to do it anyway, because then I could demonstrate that the COM server works as intended.

I create a simple win32 console application so that I only had to care about the actual functionality.

Importing the server type library

In the olden days or yore, working with COM servers in a C++ project was a bit tedious. You had to include the IDL files in your project and compile them, then include the resulting header files, add the .c files to your project for the GUID definitions… Not rocket science by any means, but still… a bit messy and ugly.

Now, I have very few kind words for Visual Basic, but to give credit where it is due: Visual Basic is (in large part) responsible for the existence of tlb files.

Visual Basic programmers wanted to use COM too, and the people who made Visual Basic wanted to prevent the application programmers to have anything to do with IDL. They wanted Visual Basic to take care of all that mess behind the scenes.

And thus the type library (tlb) was born.

The type library contains the same stuff as the IDL files, but in a machine readable, programming language independent format.

To use a COM server, you need to do nothing more than add the following line to the StdAfx.h header file of the client program:

#import "Stuff_Server.tlb" no_namespace

You also need to make sure that the compiler knows where to find Stuff_Server.tlb. This is easily done by adding the following line to the ‘Additional include directories’ setting:

..\Stuff_Server\$(ConfigurationName)

If you then compile StdAfx.cpp, the compiler will pull in the type library and generate the appropriate headers which are then automatically included.

The no_namespace attribute tells the compiler that it shouldn’t put the declarations in a specific namespace.

Using the COM server

The client application code itself is fairly trivial:

int _tmain(int argc, _TCHAR* argv[])

{

HRESULT hr = CoInitialize(NULL);

if(SUCCEEDED(hr))

{

CComPtr<IStuff> stuff;

hr = CoCreateInstance(

__uuidof(Stuff), NULL, CLSCTX_ALL,

__uuidof(IStuff), (void**)&stuff );

if(FAILED(hr))

return hr;

CComPtr<IUnknown> thEnumUnk;

hr = stuff->GetEnum((IUnknown **)&thEnumUnk);

if(FAILED(hr))

return hr;

CComPtr<IEnumString> thEnum;

thEnumUnk->QueryInterface(__uuidof(IEnumString), (void**) &thEnum);

{

LPOLESTR str;

hr = thEnum->Next(1, &str, NULL);

if(hr == S_OK )

wprintf(L"%s\n", str);

CoTaskMemFree(str);

}while (hr == S_OK );

}

CoUninitialize();

return 0;

}

First I initialize the COM runtime and then create a new instance of the Stuff object. I request IStuff as the initial interface. I also use the __uuidof keyword to get the GUID for an interface / object instead of using the CLSID_ and IID_ parameters. In my opinion, it is more convenient.

The COM smart pointer takes care of the reference counting for me. This is most convenient, because then I am certain that it will get released properly without me having to worry about it.

Btw, this is also the reason that all the code is placed into a separate scope block after the call to CoInitialize. The smart pointers clean up their interface pointers when they go out of scope. If CoUninitialize would have been called before that, it would lead to weird problems / crashes.

Via the IStuff interface I request an enumerator object. This gets returned as an IUnknow interface, so I have to get the IEnumString interface via the IUnknown::QueryInterface method. So that’s 3 interfaces already, and I still don’t have to care about reference counting J

After that I use the ‘Next’ method of the IEnumString interface to retrieve the items contained in the enumerator. This the only place where I have to perform a manual cleanup action.

The IEnumString interface does not work with BSTRs, it works with LPOLESTRs. While these are code compatible because they are the same type of data, their semantics are not. The _Copy<LPOLESTR> mentioned earlier allocates data for the string using CoTaskMemAlloc. It has to be freed by CoTaskMemFree.

CComBSTR would use SysAllocString and SysFreeString. And while the code would compile and execute perfectly, there would be an crash as soon as the CComBSTR would try to free the string it got from the call to ‘Next’.

Conclusion

As you can see, implementing COM servers and clients can be really easy with the proper use of ATL. And you’ve also seen that implementing an enumerator object is made really easy.

The code demo code is up for download with this article under the MIT license. Have fun.

I will probably write some more ATL / COM stuff soon. Stay tuned for more DCOM archaeology from the mists of time ;-)

Getting started with ATL

vanDooren — Tue, 20 Jan 2009 10:53:00 GMT

I am currently working on some projects where I have to program a DCOM server. There are several reasons why it has to be DCOM and C++, instead of e.g. .NET remoting and C#.

The biggest pain (imo) about DCOM is that the technology stems from an era when the internet was no ubiquitous, and there were no huge forums filled with experts, or blogs by programming gurus and sites like codeproject.com There were newsgroups, but those were generally only used for specific problems.

There is a vast amount of concise information about everything you can do with the .NET framework. For DCOM there is … a big dark void.

Granted, there are several resources out there. But if you look on sites like codeproject, you will find that most of the articles about DCOM and ATL are written before 2003. Most of the information out there will not cover the improvements that have been made to ATL (or DCOM) since the last 5 years.

And when looking into the more obscure corners of DCOM, there are some things that noone really likes to talk about because the documented semantics are so vague, and if you look into things like custom IMoniker implementation… well… that is a story for another day.

Understanding DCOM

The most important thing with DCOM is that you understand the basics really well. I guess this is true for most things, but unlike other things like .NET, DCOM does not cut you any slack, and does not provide you with tons of helpful diagnostic information.

When I started with DCOM a couple of years ago, I first read ‘Essential COM’ by Don Box. This is really the most excellent book on COM ever.

Of course, this information becomes vitally important in the real world when you have to debug a problem, because no tool is going to tell you anything useful with DCOM related crashes.

Understanding ATL

In the real world (unless maintaining legacy code) you use ATL to write COM clients or servers. Personally, I don’t like books that just tell me how to use technology XYZ. I like to understand the ‘why’ and ‘how’ too.

This is where I have to give big thanks to ‘ATL Internals, Second Edition’. It is one of the very few books on ATL that were written for VS2005, and it is still valid for VS2008. It is also an extremely well written book.

If you are starting with ATL (or if you want to know how it works) buy this book. This book is very, very good, and will teach you a lot.

What happens next?

Over the next weeks / months I will probably write a couple of articles describing how to do specific things with ATL. One article that is nearly finished is about implementing and using IEnumString using ATL.

Not rocket science by any measure, but ATL is severely lacking in the area of easy-to-find, up-to-date and to-the-point examples. So I figured that there is still value in writing about the ancient art of circles and lollipops, even if many developers probably think COM and ATL are no longer worth bothering with.

Oh and I released some win32 DCOM demo projects some time ago. They were interesting for me because making them taught me a lot about the raw DCOM stuff that you normally don’t see with ATL. But I’ve always found that it is always useful to know what makes the motor hum.

So while they are not terribly useful for practical purposes, they are well documented and they might be useful if you are working your way through ‘Essential COM’

Found an interesting bug in the Visual Studio IDE

vanDooren — Thu, 08 Jan 2009 07:18:00 GMT

I know, I know, ... .NET is all the rage, and DCOM is legacy technology, best not touched by up and coming programmers with good hair and sharp look, lest they appear 'uncool' or even worse: 'obsolete'.

But some of us still use DCOM for a number of good reasons. And ATL takes much of the pain out of DCOM development. Unfortunately, the Visual Studio IDE doesn't always cooperate. I made a simple MFC dialog client application, and then tried to add the ATL Server project that the client would connect to. VS was clearly concerned about my attempts to use DCOM instead of .NET, and decided to sabotage my attempts to do anything useful with this new ATL project.

https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=391506

But luckily, I was able to figure it out and outsmart VS :)

Backing up event logs in their normal binary format

vanDooren — Fri, 19 Dec 2008 13:15:00 GMT

Some time ago I was looking for a way to make backups of the eventlogs of our server, preferably without me ever having to do anything anymore J

This was not success. There are a couple of tools to make text exports from eventlogs. These have the disadvantage that they make the logs very large and unsuitable for looking at with the even viewer. As an added bonus, these are slow too (since every item has to be read individually).

EventQuery.vbs is one of these tools and it is so dog slow that it is almost a crime that it has been included with Windows. Ever wanted to bring a domain controller slow to a crawl? Simple: Use EventQuery.vbs to export the security event log. That’ll do it.

Anyway, I checked with my fellow MVPs, and there is no built-in (or free) tool to save event logs as binary event log files, so I gave up.

But then I accidentally discovered that there is a win32 API that exposes the event log functionality. And as luck would have it, I am a programmer and thus perfectly suited to make such a thing myself. It would have been trivial to make it at work, but I decided to do this at home so that I could share the code.

It is of course perfectly possible that such a free tool exists and I simply didn't find it.

The code

The code is simple, though the line count was increased by the desire to implement error handling.

int _tmain(int argc, _TCHAR* argv[])
{
_TCHAR* computerName = NULL;
_TCHAR* logName = NULL;
_TCHAR* file = NULL;

//populate the variables, based on the input arguments
if(argc == 3)
{
    computerName = NULL;
    logName = argv[1];
    file = argv[2];
}
else if(argc == 4)
{
    computerName = argv[1];
    logName = argv[2];
    file = argv[3];
}
else
{
    Usage();
    return -1;
}

//Open the eventlog
HANDLE hEventLog = OpenEventLog(computerName, logName);
if(NULL == hEventLog)
{
    _tprintf_s(TEXT("Could not open eventlog %s on computer %s\n"),
      logName, computerName);
    PrintError();
    return -1;
}

//backup the event log to file
if(FALSE == BackupEventLog(hEventLog, file))
{
    _tprintf_s(TEXT("Could not save eventlog to file %s\n"), file);
    PrintError();
    CloseEventLog(hEventLog);
    return -1;
}

CloseEventLog(hEventLog);
return 0;
}

There is little to tell about the code. It can be invoked with either 2 or 3 command line arguments:

The name of the computer of which the logs have to be backed up. (optional)
The name of the event lot to back up
The path of the file to which the event log needs to be saved.

The application opens the event log, and then makes the backup and closes the eventlog. If something goes wrong, the error code and error message are printed to the command line.

I thought about making the error handing a little more elegant (so that PrintError and CloseEventlog were not used multiple times in the code) but with only 2 function calls, the amount of error handling code would not diminish or be any better.

I left the PrintError function out of this blog because it doesn’t do anything exciting. It retrieves the error code and prints out the corresponding error message.

What you need to know before using this tool

I’ve been using this tool for a week now, and it is working splendidly, but there are a couple of things that aren’t mentioned in the MSDN documentation of the API.

The first is that the supplied path is absolute for the computer of which you are making an even log backup. So if the path is D:\app.evt, it will make the backup to D:\app.evt on the remote computer.

The second thing you need to know is that providing a UNC name for the target file doesn’t really improve things all that much. Because what is happening under the hood is that it is not you who is making the backup.

The event log service is in charge of the event log files, and you are asking it to make a backup. And that is where it all goes pear shaped. The event log service runs with LOCAL_SYSTEM credentials. This means that by default, it has no credentials that are valid on a remote computer. So you cannot make backup files in network locations, because the event log service of the remote computer has no privileges to access the location.

There are 2 exceptions to this rule. 1) the remote location is on the remote computer itself. 2) the remote computer is a backup domain controller.

It is a little known fact (I didn’t know it either) that LOCAL_SYSTEM on a domain controller is a domain administrative account that can pretty much do anything and everything on any domain computer, including the things that even a domain admin cannot.

Conclusion

I needed a convenient tool for making backups of event log files, and now I have one. Both the source and the binary are included as attachments. The source is available under the MIT license, and the exes are available for free to do with them what you want, as long as it is understood that whatever you do with them is your responsibility, not mine.

The eventtool_clr exe is something I made to create and delete event logs. I don’t think I blogged about it yet, but what it does is so simple it’s hardly worth mentioning. Have a look at the sources for more info.

The binary is for those who just want to use it and don’t care. The source is for those who’d rather not run an untrusted binary from some guy on the internet on their internal servers J The sources are simple enough to audit and compile.

On a side note, The API to backup event logs is not exposed on the .NET api for event logs. I don’t know exactly why, but I suspect that it is because of the problems I mentioned. The API is confusing enough that it is really not fit for generic use. After all, you can’t backup to remote systems, and the path is absolute for the remote computer.

I also didn’t want a .NET program, because for this type of thing, nothing beats the convenience and performance of a compiled binary that needs no runtime DLLs or external frameworks.

Extending a native C++ project with managed code

vanDooren — Wed, 03 Sep 2008 09:59:00 GMT

One question that comes up from time to time in the newsgroups is ‘I have a native C++ project and I want to extend it with Managed code (e.g. Windows Forms). What do I do?’

The answer is not so complex. It is fairly easy to extend native projects with managed code. In this article I’ll explain how.

What NOT to do

To quote Kate Gregory ‘In the name of all that is good and right: Do not set /CLR for the entire project’.

/CLR tells the compiler that it should compile a source file as C++/CLI code in which it finds both native and managed C++ code. Theoretically you could specify this for the whole project, but this has some serious consequences.

Apart from the fact that it is possible for the compiled output size to explode, there can be significant problems with COM and CRT initialization. I am no interop expert in the matter like Marcus Heege or Kate, but I am willing to take their word for it.

Furthermore, your existing project is possibly validated by QA en unit tested 7 days from Sunday. You don’t want to travel that road again if you can prevent it.

What to do

You want to add managed stuff to your project while impacting the rest of the code as little as humanly possible. To achieve this you have to compartmentalize the managed stuff and give it a native interface that you can use in the native parts of you projects without any hassle.

I think I can best explain this with a practical example.

For the sake of this example I have a native DLL project that exports a function that returns an integer value. Perhaps this function got that integer value previously through an MFC interface, and now that value has to come from a Windows Forms interface.

Add the windows form to your project

This is the first and also the easiest step of the process. Rightclick your project and select ‘Add…->New Item’ and select ‘UI->Windows Form’ in the Visual C++ dialog.

You will then get a message box, telling you that your project will be converted to a managed project. Here you click ‘yes’. Even though you don’t want to compile your project with /clr, Visual Studio itself needs to know that you are doing managed stuff in your project. This is not necessary for compiling your project, but if you want to have the benefit of working with the forms designer and other .NET related things, then it would be a good idea. And you cannot add the form if you choose ‘No’ so it’s not like you have much choice anyway.

Now you have a .NET Windows Form in your code. You can verify the file specific settings if you want, but the cpp file will be compiled with the /clr flag set, and without the use of precompiled headers. Not using precompiled headers is important, because the default precompiled header is compiled without /clr set, and this would lead to conflicts.

Of course, if you need to, you can create a copy of StdAfx.h/cpp cpp files called ‘StdAfxClr.h/cpp’ add them to your project, and configure them to create a managed precompiled header, which you could then use for all files that are compiled with /clr. This is not necessary for small projects, but it could be a useful optimization in large projects.

The interface layer

At this point you can compile and link the entire project, but your native code isn’t yet using the new managed functionality. And because the native code will not be compiled with /clr, it will never do so. In order to make that happen, you will need a thin layer between the native and managed code. This layer will have a native C or C++ interface which can be called by the native code, and a managed implementation that will do all the .NET stuff.

In our example, the interface is 1 simple C style function in Interface.h:

int __stdcall DoManagedStuff(void);

And this function has a simple implementation in Interface.cpp:

int __stdcall DoManagedStuff(void)
{
      DemoForm ^df = gcnew DemoForm();
      df->ShowDialog();
      return df->Value;
}

Interface.h and Interface.cpp are 2 new files that you have to add to your project. You have to manually configure the cpp file to be compiled with /clr, and to not use precompiled headers.

As you can see, the interface layer provides a clean native interface for the managed stuff that you want your code to perform. Of course, you are not limited to C style interfaces. You can also work with classes and make classes with a native interface and a managed implementation. But there are a couple of issues that you need to be aware of. If you want to go there, then it would be a good idea to read the paper written by Herb Sutter which explains the rationale behind C++/CLI, as well as some of the limitations and pitfalls.

Actually, reading that paper is a good idea for anyone working with C++/CLI who also cares about understanding what is actually going on behind the scenes.

Using the interface layer

All the hard work is done. Now you can simply include Interface.h in your native code files, and call ‘DoManagedStuff’ where appropriate.

#include "stdafx.h"
#include "InterfaceLayer.h"
int __stdcall DoFoo(void)
{
return DoManagedStuff();
}

Conclusion

As you can see, extending native code with managed code is not so hard. At least, it isn’t if you maintain a clean break between managed and native code. If you cannot do this, things might become more difficult. For example, using .NET controls on an MFC dialog, or using .NET remoting in a native COM project are things that can be much more tricky.

Another thing you should be aware of is that not all compiler switches can be used in conjunction with the /clr switch. The compiler will inform you if it detects this. The combinations which are not allowed are documented, and can be found in the documentation of the /clr switch itself.

This post was not meant to be an exhaustive how-to to those complex scenarios, but instead an explanation behind the basic ideas.

The demo project for this article can be downloaded under the MIT license as usual.

Fun with templates: parsing command line arguments

vanDooren — Thu, 14 Aug 2008 10:17:00 GMT

One of the things that everyone has to do sooner or later is parsing command line arguments into program variables. Even the most trivial command line application needs some input variables to tell it what to do.

One of the things in which C and C++ are lacking is a unified approach to command line arguments. This allows anyone to do what he wants, but unfortunately it also forces everybody to figure it out for himself.

I have done this several times, and tried several approaches. My last projects all needed moderately complex command line configurations, so I finally solved this problem in a reusable way. Originally I used the boost template library for this (www.boost.org). Specifically, I used regular expressions. But now that VS2008 SP1 is out, I can use the TR1 library to do this for me.

What does a command line argument look like

The exact syntax doesn’t matter because we are using regular expressions. What is important is that the identifier and the value are recognized by the shell to belong together.

The only way to do this is of course to make sure that they are in the same string. For example, such command line arguments can look like this:

-val:12.5
/text=blabla
-file=”C:\Documents and Settings\me\desktop\article.doc”

It really doesn’t matter. As long as all the information is recognized as 1 argument, it’s fine. If the value contains spaces, you need to use quotes so that the shell will not treat

-file=”C:\Documents and Settings\me\desktop\article.doc”

as 3 arguments.

Requirements

The requirements for my solution are as follows:

1 function which can take a command parameter and a regular expression, and which can parse any given type (bool, int, string, …) from the parameter, according to the specified regex.
The function should be able to handle string and wstring parameters.
It should be able to put values into regular data types, as well as into TriStateValue template types (see here http://msmvps.com/blogs/vandooren/archive/2007/10/11/fun-with-templates-implementing-a-tri-state-value.aspx for more info).
The function should return a bool to notify the user if the argument was parsed.
The part of the regex that identifies the actual value has to be captured. I.e. it has to be enclosed between ( ) symbols.

First attempt

The first version of such a function was ready pretty rapidly.

template <typename T>
bool ParseArg(
    std::tr1::regex const& pattern,
    std::string const& arg,
    T& value)
{
    std::tr1::smatch matches;
    if( !std::tr1::regex_search(arg, matches, pattern))
      return false;

    std::istringstream iss((string) matches[1]);
    return ! (iss >> std::dec >> value).fail();
}

The function takes a regex and an argument. The value type is a template type which allows the user to use it for any type. The type itself can be inferred at compile time. I.e. the user does not have to explicitly specify the template type using brackets.

A string stream is used for getting the value out of the stream. This is very handy because this way, the text to value conversion has no dependency on the value type. By using the string stream, the function remains generic.

This function has an overload that uses a TriStateVal type for the resulting value.

template <typename T>
bool ParseArg(
    std::tr1::regex const& pattern,
    std::string const& arg,
    TriStateVal<T>& value)
{
    std::tr1::smatch matches;
    if( !std::tr1::regex_search(arg, matches, pattern))
      return false;

    std::istringstream iss((string) matches[1]);
    T val;
    if((iss >> std::dec >> val).fail())
      return false;

    value = val;
    return true;
}

The TriStateVal type allows the user of that value to know whether it had been assigned or not. Unfortunately, there is no support yet for streaming directly into a TriStateVal type, so I have to use a temporary value first.

Of course, these 2 functions work for strings. What about wstring? Well, for wstring we need another 2 overloads.

template <typename T>
bool ParseArg(
    std::tr1::wregex const& pattern,
    std::wstring const& arg,
    T& value)
{
    //
}

template <typename T>
bool ParseArg(
    std::tr1::wregex const& pattern,
    std::wstring const& arg,
    TriStateVal<T>& value)
{
    //
}

These are almost identical, except that they use wregex, wstring and wistringstream instead of regex, string and istringstream.

The overload mechanism

By now you may have wondered about the function overloading, if you are into template laws.

There are 4 overloads (removed template specifier template <typename T>

for clarity):

bool ParseArg(regex const&, string const&, T&);
bool ParseArg(regex const&, string const&, TriStateVal<T>&);
bool ParseArg(wregex const&, wstring const&, T&);
bool ParseArg(wregex const&, wstring const&, TriStateVal<T>&);

It is simple enough to see how the compiler chooses based on the types of the pattern and the argument. That is via normal overloading rules.

But how does it know when to pick the generic T template function or the more specific TriStateVal<T> template? Both can be valid.

As it turns out, the compiler can follow the rules laid out in the C++ ISO standard section 14.5.5.2, which states that if multiple template functions are valid for the supplied template types, the most specific one will be selected.

The wording is of course more complex and formal than the previous paragraph, but that is what it boils down to. Section 14.5.5.2.5 contains a number of helpful examples, and one of them is equivalent with the ParseArg function.

TriStateVal<T> is more specific than T, so the function with that parameter type will be used whenever possible.

Evaluation of the first attempt

The first implementation works very well. You use the ParseArg function like this:

wregex reFile(L"^[-/]file[:=]\"{0,1}([^\\?\\*/<>\\\"]*)\"{0,1}$");
wregex reRows(L"^[-/]rows[:=]([0-9]*)$");
wregex reCols(L"^[-/]cols[:=]([0-9]*)$");

int _tmain(int argc, _TCHAR* argv[])
{
int numRows = 0;
TriStateVal<int> numCols;
wstring file;

for(int i=0; i< argc; i++)
{
    wstring arg(argv);
    if(ParseArg(reRows, arg, numRows))
      continue;
    if(ParseArg(reCols, arg, numCols))
      continue;
    if(ParseArg(reFile, arg, file))
      continue;
}

wcout << "numRows : " << numRows << endl;;
wcout << "numCols : " << numRows << endl;;
wcout << "file : " << file << endl;;

return 0;
}

As you can see, no matter how many command parameters there are and how complex their formatting is, parsing them remains trivial if you can correctly specify the regular expressions.

The regular expressions in my example also take care of the fact that users can use either the – or the / to precede identifiers, and that the value delimiter can be a = or : sign.

The great benefit of the TriStateVal is that you don’t have to keep track of Boolean variables, indicating the status of a variable. I.e. you don’t have to manually keep track of which variables were assigned.

If your application can do several things, based on a ‘command’ parameter for example, then you can check the TriStateVal variables for that command to see if those have been assigned to. If will make your whole program easier to understand.

Problems with the first attempt

One issue is that it sadly doesn’t work if T is of type ‘char’ or ‘wchar_t’. The reason is that these types are the character types which make up ‘string’ and ‘wstring’, and streaming from a string stream into a char is not supported.

Another possible issue is that in the case that T is of type bool, the text value has to be 1 or 0. Perhaps we would also like to be able to use ‘on’ and ‘off’ as valid value texts, or ‘high’ and ‘low’.

Both problems can be solved by providing more specific template functions, like (removed template specifier

for clarity):

template <> bool ParseArg(regex const&, string const&, bool &);
template <> bool ParseArg(regex const&, string const&, char &);

But if we want to implement those functions, we not only have to implement the string-regex function, but also the wstring-wregex function, and the functions that use TriStateVal parameters.

So instead of having to write 2 additional functions, we would have to implement 8 functions.

We could partially fix that by making the regex and string parameters templates instead of qualified types, but that does not fix the problem of the types that are used internally (smatch and istringstream).

Of course, we could make those template arguments as well, but that would break automatic type inference, and force the programmer to specialize the template function explicitly, which would be ugly and confusing.

So instead we use a trick that is also used in the STL itself: we use a helper class that deduces those internal types for us.

Second attempt

The key to the second attempt is to use a helper class that –through specialization- will specify the dependent types.

The coat hanger for this mechanism is the empty class

template <typename strType>
struct __pca_typehelper{};

which has 2 specializations:

template <>
struct __pca_typehelper<std::string>
{
    typedef std::tr1::smatch MatchType;
    typedef std::istringstream SStreamType;
    typedef std::tr1::regex RegexType;
};

template <>
struct __pca_typehelper<std::wstring>
{
    typedef std::tr1::wsmatch MatchType;
    typedef std::wistringstream SStreamType;
    typedef std::tr1::wregex RegexType;
};

Depending on which specialization is picked, MatchType, SStreamType and RegexType are typedef’ed to the correct STL and TR1 types.

And the nice thing is that is the user supplies something other than string or wstring, compilation will fail at this stage already, because those typedefs don’t exist.

Using that helper class, the ParseArg function can be written like this:

template <typename Targ, typename Tresult>
bool ParseArg(
    typename __pca_typehelper<Targ>::RegexType const& pattern,
    Targ const& arg,
    Tresult& value)
{
    typename __pca_typehelper<Targ>::MatchType matches;
    if( !std::tr1::regex_search(arg, matches, pattern))
      return false;

    typename __pca_typehelper<Targ>::SStreamType
      iss((Targ) matches[1]);
    return ! (iss >> std::dec >> value).fail();
}

As you can see, there are now 2 template arguments: Targ and Tresult.

The meaning of Tresult is still the same as in the previous implementation.

Targ is new, and is the type of the argument that needs to be parsed. In our example it is either string or wstring. This type is then used to specialize __pca_typehelper so that the type of the regular expression can be deduced:

typename __pca_typehelper<Targ>::RegexType

The same goes for the internal variables ‘matches’ and ‘iss’

The implementation for the TriStateVal<Tresult> specialization is very similar to the one for Tresult so I am not going to repeat that here.

Evaluation of the second attempt

We got rid of the code duplication (which is good) and managed to have some fun in the process of doing so. At least I did. J

The code still behaves as it did previously, so using the ParseArg function hasn’t changed at all.

We still have the issues about bool and char, but at least we can now solve this problem with a lesser number of specializations (4 instead of 8).

Conclusion

My implementation of ParseArg works good, and is easy to use. I’ve been using it for some time, but used boost for the regexes.

I would like to try and get the TriStateVal<T> and T scenarios to be handled by the same function. That would require providing a >> stream operator for TriStateVal<T> and I don’t know how much work that is.

I feel like giving this a try, but I am not going to wait with publishing this article. For one thing, it is already lengthy enough, but it also won’t matter from the user’s point of view. The usage of ParseArg will not change, so I feel it is perfectly acceptable to leave this optimization for another article.

The code of this article is available for download as always.

And as always, if you have any comments or feedback, please leave a comment at the bottom of the page.

Creating a thread safe producer consumer queue in C++ without using locks

vanDooren — Fri, 05 Jan 2007 09:11:00 GMT

Yesterday, someone in the newsgroups asked a question about synchronization problems he was having. I explained that –if properly programmed– a consumer producer queue does not need synchronization even on a multiprocessor machine.

A Microsoft employee then asked me to explain exactly how this was done. Ie. To put my money where my mouth is.

The idea

A queue is a chunk of memory that can be divided into equal sized pieces. To simplify this you can think of a queue simply as a traditional 1 dimensional C style array.

You should also note that there is 1 reader, and 1 writer. Each will work in his own thread. This means that reads can happen concurrent with writes, but there will be no multiple concurrent reads or multiple concurent writes. It is possible to provide that level of concurrency without locks (I did, once) but that is significantly more complex, and not the purpose of this example.

To manage the producer and consumer part, we need to have 2 additional variables: a read index and a write index.

The read index is the index of the element right after the element that was last read by the consumer.

The write index is the index of the element right after the element that was last written by the producer.

A writer can always add elements in the queue as long as it can move the write pointer without overtaking the read pointer. An overtake would mean that the queue would overflow.

There are 3 possible cases:

The read and write pointers are equal. This is only possible if the queue is in the initialized state. Any read will result in a fail, any write will succeed.
The read pointer is pointing to the first element after the write pointer. This means that the queue is full. If one more element would be added, the read and write pointers would be the same, and the reader would assume that the queue was empty. Any read will succeed, and write will fail.
The read and write pointers are not equal, and the previous case was not true. Any read will succeed. Any write will succeed.

The implementation

Adding elements

bool PushElement(T &Element)

{

int nextElement = (m_Write + 1) % Size;

if(nextElement != m_Read)

{

m_Data[m_Write] = Element;

m_Write = nextElement;

return true;

}

else

return false;

}

As long as the first element after the write pointer is not the read pointer, it is not occupied. The element can be added and the write pointer can be updated.

Otherwise the queue is full and the write operation will fail.

Removing elements

bool PopElement(T &Element)

{

if(m_Read == m_Write)

return false;

int nextElement = (m_Read + 1) % Size;

Element = m_Data[m_Read];

m_Read = nextElement;

return true;

}

If the read pointer is equal to the write pointer, there is nothing to read. In any other case we can read the first element and update the read pointer.

Why this is safe

The advantage of this scheme is that no locking is needed. Each of the pointers (indices really) is updated only by one thread.

This means there is no write concurrency as far as the pointers are concerned.

Furthermore – and this is important – the read and write pointers are updated only after the element concerned is read or written.

This means that if the read or write pointer changes, the elements are guaranteed to reflect that situation. The gap between the element change and the pointer change is invisible to the other threads.

So logically, this code is perfectly safe. However, one final issue must be highlighted, and that is the declaration of the pointers and the data array.

volatile int m_Read;

volatile int m_Write;

static const int Size = 10;

volatile T m_Data[Size];

If we take no precautions, the optimizer can reorder the actual memory operations on the read and write pointer.

Or it could assume that variables do not change between calls and use cached values instead of what is actually in memory. This could be especially problematic on multi CPU machines where each CPU has its own cache.

See the documentation for ‘volatile’ in MSDN for a more detailed explanation.

How this could be unsafe

There is one thing that is critical in this code, and that is that memory updates are not re-ordered. You can easily understand that if the write pointer gets updated before the actual data element is updated, there is a gap that creates a race condition.

Therefore it is critical that the platform this code runs on (CPU, chipset, memory controller) does not re-order the memory write operations. Otherwise you’d have this code that is logically thread-safe, but thread-unsafe due to CPU peculiarities. You’d have all sorts of impossible to debug problems.

Extending functionality

The design as it is works reasonably well, but here are a couple of tips to make this into powerful code:

Foresee an array of read pointers instead of using only one. That way you could have a broadcast queue in which each consumer has the opportunity to receive elements. You have to check each of the read pointers before writing to make sure that there is room enough in the queue.
Allow the constructor to take a variable sized piece of memory that was previously constructed. Make the read and write pointers the first items in that memory. This allows you to put a queue on shared memory and communicate with other processes.
…

Conclusion

Making a multi thread safe consumer producer queue without synchronization is perfectly possible. If you want to play with the code, it is attached to this article and licensed under the MIT license.

I whipped this code together in a very short time for the purpose of this demo. It should be bug free, but if you think there is something specifically wrong, let me know and I’ll see if there is a problem with it.

Fun with templates: implementing policy based error handling

vanDooren — Mon, 02 Oct 2006 08:01:00 GMT

This article is a follow up of my third article on interoperability for LabVIEW arrays in C++. That article provided a method for allowing programmers to index into multi dimensional arrays.

As far as functionality went, the solution was OK, but the error handling left a lot to be desired. I also wanted the error handling strategy to be policy based. I.e. you should be able to choose at compile time whether the classes use return value error handling or exceptions, or maybe something else entirely, like structured exceptions.

After some experimenting, I now have a decent solution that is elegant and maintainable.

The general idea

After writing my previous article in this series, I was kicking around with a few ideas, but I got a great suggestion from my fellow MVP Eugene Gershnik.

Basically, it boils down to the following idea:

struct EBThrowing

{

typedef void RetVal;

static RetVal ReturnError(int ErrorCode)

{

if(0 != ErrorCode)

throw Lvtl_Exception(ErrorCode);

}

};

struct EBRetVal

{

typedef int RetVal;

static RetVal ReturnError(int ErrorCode)

{

return ErrorCode;

}

};

template<class EB> class CClass()

{

typename EB::RetVal foo(void)

{

int errorCode = 0;

//...

return EB::ReturnError(errorCode);

}

};

This example defines 2 error handling policies. One policy specifies that error handling constitutes returning error codes. The other policy specifies that errors will be reported as exceptions.

Each of those policies has a typedef to define the ‘RetVal’ type, as well as a function called ‘ReturnError’.

Each class that wants to implement policy based error handling has to be a template class that takes a policy class as input.

Return value types

This is where the magic happens: all methods that could trigger an error have to have a return value type that is defined by the policy class. Since these types are resolved at compile time, the function

typename EB::RetVal foo(void);

is equivalent to

int foo(void);

if the error handling policy class is EBRetval. If the policy class is EBThrowing, the function is equivalent to

void foo(void);

As a result, it is hard to accidentally misuse an instance of CClass, as far as error handling goes.

Suppose you try to use an instance of CClass that throws exceptions, in a situation where you expect return values will generate a compiler error because

int ret = foo();

cannot compile if foo is of type void.

Because the type identity of CClass is determined in part by the error policy, it is impossible to pass an instance of CClass<EBRetVal> where an instance of CClass<EBThrowing> is expected or vice versa.

Raising errors

Part of the policy contract is that the consuming classes never raise an error directly. I.e. they never return an actual value or throw an exception directly.

Instead, all errors will be raised by calling RaiseError.

return EB::ReturnError(errorCode);

The ReturnError function has a return value of type EB::RetVal. It depends on the policy. This is necessary because the methods of the consuming class also have this return type, and they have to match.

If the error policy prescribes error handling via return values, ReturnError simply does this:

return ErrorCode;

If the policy prescribes error handling via C++ exceptions, ReturnError simply does this:

if(0 != ErrorCode)

throw Lvtl_Exception(ErrorCode);

Not raising false errors

Consider the following snippet:

typename EB::RetVal foo(void)

{

int errorCode = 0;

//...

return EB::ReturnError(errorCode);

}

The error code will be determined by what happens in foo. However, if there was no error, no error should be raised either. With standard exception handling, this would not be a problem, because foo would simply not raise an error if there was no error.

But if return value error handling is used, foo always has to return a value, even if there was no error. In that case the general behavior is to return 0.

CClass should never return a value directly, since this would break the policy contract. Instead it must always use ReturnError. This means that ReturnError will be called with 0 as the error code if there was no error. ReturnError should then check if the error code really indicates an error before doing anything.

Providing a default policy

Template classes allow template parameters to be specified with default values. This makes it possible to use the template class without having to specify the default parameters. This makes the code more readable, and the program easier to write.

However, in this case the code is meant to be used as a toolkit by other developers. If we provide a default policy, it will be wrong for half of the users.

template<class EB=EBThrowing> class CClass();

If I want to use the EBRetVal policy, I would still have to provide a value for the EB parameter.

To solve this, I have changed CClass.

#ifndef LVTL_OVERRIDE_EB_DEFAULT

typedef EBThrowing EBDefault;

#endif

template<class EB=EBDefault> class CClass();

This construct allows me to specify a default ‘default’ policy, while still allowing the user to override it with another default by supplying a custom typedef.

Usually, only 1 policy is used throughout the whole dll or application, so allowing the user to provide a default policy should in practice mean that they never have to supply one explicitly.

Limitation 1

If you design classes that implement a policy based error handling strategy, you should take care that cleanup actions are policy agnostic.

I.e. you must make sure that cleanup is done before you raise the error. For example, if you would like to enable the user to provide his own policy class that uses structured exceptions, you have to be aware that you cannot guarantee that objects will be properly destructed when an error is raised

Limitation 2

You should also be aware that it is difficult to layer policy based classes on top of each other while retaining the policy scheme. This is demonstrated by the this implementation of foo, where bar is an instance of a CBar class that implements policy based error handling.

typename EB::RetVal foo(void)

{

int errorCode = 0;

//...

bar->SomeMethod();

//...

return EB::ReturnError(errorCode);

}

The problem is obvious. How would you know that you need to cleanup and bail out if bar raised an error?

You cannot check its return value, nor catch an exception, because you don’t know which of those you will get. Remember, using policies means that you don’t know what bar will do in advance.

If you want to use CBar, you have to fix its policy in order to be able to use it reliably.

Differences between C# and C++/CLI, part 2

vanDooren — Thu, 31 Aug 2006 20:58:00 GMT

One question that comes up from time to time in the C++ newsgroups is that functions or constants that exist in a C++/CLI class library are not accessible in a C# or VB.NET project.

The reason for this is nearly always that that constants or functions are defined outside of a class scope.

Consider the following - perfectly valid - C++/CLI:

namespace Bla

{

const int CONST = 42;

int func(void)

{

return CONST;

}

};

You can use CONST and func in other C++/CLI code.

What you can’t do is to use those symbols in other .NET languages. The reason is that those languages do not have a concept of things existing outside of a class or struct scope.

In other words, even though the CLR supports it, languages other than C++/CLI do not support free standing functions, variables and constants.

A simple workaround is to put them in a dummy class. For example, you could put all constants in a class called ‘Consts’, and functions in a class called ‘Functions’.

Differences between C# and C++/CLI, part 1

vanDooren — Thu, 31 Aug 2006 20:56:00 GMT

Learning C# if you already know C++ is relatively easy. The syntax is similar, and the semantics are definitely a lot easier to understand.

Since C# looks like C++, most of the concepts are easy to learn, but here and there are a few details that might cause problems later on.

On of these things is the difference between the meaning of classes and structures in C# and C++.

C++

With C++, a distinction needs to be made between native C++ and C++/CLI

Native C++

Classes and structures are virtually identical in native C++. Instances can be located on the stack or on the heap. It all depends on how they are instantiated.

The only difference between the two is that in a class, the default access of a class member is private, while the default access of a structure member is public.

That’s all there is to it.

C++/CLI

With the rise of C++/CLI, the compiler needs to know

a) whether a type is managed or unmanaged, and

b) whether the type (if managed) is a reference or value type.

To solve this, the new keywords ‘ref’ and ‘value’ are used. The term ‘ref type’ indicated a reference type. The type is managed, and is located on the managed heap. The term ‘value type’ indicates a managed value type. It is also managed, but located on the stack

Type can be either a class or a structure. As a result, it is possible for classes to be value types, and structures to be reference types in C++.

Semantics

In C++, the way in which you instantiate – and not the declaration – of a type determines whether it lives on the heap or on the stack.

2 important consequences of type location in C++ are that stack based instances are fixed in memory during their lifetime, and that stack based instances are guaranteed to be cleaned up when the go out of scope.

In reality, the C++ compiler is cheating you. It does some things behind your back to fool you into thinking that things live on the stack, while in reality they live on the managed heap.

Native types are either allocated on the stack or the native heap, based on whether you use stack semantics or operator new.
Value types can be created on the stack, on the native heap and on the managed heap, based on whether you instantiate them using stack semantics, operator new, or operator gcnew. But still, what you see is what you get.
Reference types are always created on the managed heap. Even if you use stack semantics for instantiating a reference type, the compiler still allocates the object on the managed heap. The compiler then takes care to insure that the object behavior (like deterministic cleanup etc…) mimics what you would expect of a stack based object.

If you are programming C++ in a mixed mode environment (both native and managed code in the same module) it is important to be aware of these possibilities and their impact.

C#

C# has a type system that is a lot easier to grasp.

Classes are reference types. They are always created on the managed heap.

Structures are value types. They are always created on the stack. C# -like C++/CLI- also supports the concept of boxing, where a copy of the value type is saved on the managed heap.

Conclusion

C# and C++/CLI share some type traits. The cases that are identical in C++/CLI and C# are in the following table:

	C#	C++/CLI
Value type	struct	value class
Reference type	class	ref struct

All other types in C++ and C++/CLI do not have a 100% equivalent in C#.

When building class libraries for use in other .NET enabled languages, you have to watch out for incompatibilities that can be caused by using C++/CLI specific types.

Fun with template classes: Manipulating LabVIEW arrays in C++, part 3

vanDooren — Thu, 10 Aug 2006 20:45:00 GMT

This article is a follow up of my second article on interoperability for LabVIEW arrays in C++. That article provided a method for allowing programmers to index into multi dimensional arrays like this:

But that solution was a bit clunky. It involved 4 classes which had 5 [] operators and 2 implementations of the Resize function. There was also no support for memory access via memcpy or something else.

After a bit of redesign, I now have an improved implementation.

The general idea

Functionality is now split in 2 classes. There is a CLvArrayHandlePtr class that represents the array itself, and a CLvArrayIndexer class to provide the [] operator.

You can no longer index in the array object directly. The CLvArrayHandlePtr class is now meant solely for managing the array itself. This makes the array simpler.

The indexing functionality has been put in the CLvArrayIndexer class. Since all that class does is to provide a [] operator and an accessor for the data pointer, it is a very lightweight class that is easy to maintain.

This solution also makes it easier to add extra functionality later on as separate lightweight classes that are tailored to a specific purpose, instead of having to perform all modifications in a class that includes everything and the kitchen sink.

The Revised Array Class

The CLvArrayHandlePtr class is still a template class with a specialization for 1 dimensional arrays. The general principles behind the array has not been changed, so I am not going to reiterate all explanations over here. Instead, I will only explain the changes.

template <class T, int Q=1> class CLvArrayHandlePtr

{

public:

#pragma pack(push)

#pragma pack(1)

typedef struct

{

size dimSize[Q];

T data[1];

} t_LvArray, *p_LvArray, **h_LvArray;

#pragma pack(pop)

h_LvArray* m_HandlePtr;

int Resize(size DimSize[ Q]){ … }

size_t GetSize(void){ … }

void GetDimSizes(size_t DimSizes[ Q]){ … }

T* GetDataPtr(void){ … }

};

template <class T> class CLvArrayHandlePtr<T, 1>

{

public:

#pragma pack(push)

#pragma pack(1)

typedef struct

{

size_t dimSize;

T data[1];

} t_LvArray, *p_LvArray, **h_LvArray;

#pragma pack(pop)

h_LvArray* m_HandlePtr;

int Resize(size_t DimSize){ … }

size_t GetSize(void){ … }

void GetDimSizes(size_t &DimSize){ … }

T* GetDataPtr(void){ … }

};

These classes have roughly the same methods as their previous versions. The most notable differences are:

They now use size_t for dimension sizes.
There is a function for returning the pointer to the first element in the array.
There is a function to retrieve the dimension sizes.

The last 2 points are necessary in order to cleanly work with the 2 new classes.

The array indexer class

The 2 new classes are used solely to allow indexing operations on the LabVIEW array. They have no other added functionality, so they are fairly simple.

template <class T, int Q=1> class CLvArrayIndexer

{

private:

size_t m_TotalSize;

size_t m_DimSizes[ Q];

T *m_Data;

public:

//create a new instance, based on an actual array.

CLvArrayIndexer(CLvArrayHandlePtr<T, Q> LvArray)

{

m_TotalSize = LvArray.GetSize();

LvArray.GetDimSizes(m_DimSizes);

m_Data = LvArray.GetDataPtr();

}

//create a new instance, based on a data pointer and dimension sizes

CLvArrayIndexer(size_t DimSizes[ Q], T* DataPtr)

{

for(int i=0; i<Q; i++)

m_DimSizes[ i] = DimSizes[ i];

m_Data = DataPtr;

}

//return a new indexer with 1 dimension less.

CLvArrayIndexer<T, Q-1> operator[](size_t Index)

{

size_t subDimSize = 1;

for(int i=1; i<Q; i++)

subDimSize *= m_DimSizes[ i];

return CLvArrayIndexer<T, Q-1>(

m_DimSizes + 1,

m_Data + Index * subDimSize);

}

//Get the size of the data segment that represents this subdimension

size_t GetSize(void)

{

return m_TotalSize;

}

//Get the pointer to the first element of this subdimension

T* GetDataPtr(void)

{

return m_Data;

}

};

template <class T> class CLvArrayIndexer<T, 1>

{

private:

size_t m_TotalSize;

T *m_Data;

public:

//create a new instance, based on an actual array.

CLvArrayIndexer(CLvArrayHandlePtr<T, 1> LvArray)

{

m_TotalSize = LvArray.GetSize();

m_Data = LvArray.GetDataPtr();

}

//create a new instance, based on a data pointer and dimension sizes

CLvArrayIndexer(size_t DimSizes[1], T* DataPtr)

{

m_TotalSize = DimSizes[1];

m_Data = DataPtr;

}

//return a reference to the element at the specified index.

T& operator[](size_t Index)

{

return m_Data[Index];

}

//Get the size of the data segment that represents this subdimension

size_t GetSize(void)

{

return m_TotalSize;

}

//Get the pointer to the first element of this subdimension

T* GetDataPtr(void)

{

return m_Data;

}

};

As you can see, there are 2 constructors for creating an indexer. The first takes the original array as an argument. It uses the array methods for getting the data size and data pointers.

The second constructor takes the size and pointer directly. This is used byt CLvArrayIndexer classes that create another CLvArrayIndexer class with a lower dimension.

The [] operator itself is also pretty simple. For multi dimensional indexers, the [] operator returns a new indexer that has 1 less dimension than itself. For 1 dimensional array indexers, it returns a reference to an element inside the original array.

The trick here is that each next [] operator points to a more specific location inside the original array until 1 specific element is referenced.

Using the array and indexer classes

To use the array and indexer classes in not very different from the previous solution.

Given an array with the following declaration:

CLvArrayHandlePtr<double, 3> Array;

Using it is done like this:

size_t size[3] = {

requestCountPages,

requestCountRows,

requestCountColumns};

Array.Resize(size);

CLvArrayIndexer<double, 3> indexer(Array);

double i=0;

for(int page=0; page<requestCountPages; page++)

for(int row=0; row<requestCountRows; row++)

for(int column=0; column<requestCountColumns; column++)

indexer[page][row][column] = i++;

Using the data pointer and size of a sub segment are of course as easy as

Memcpy(Target, indexer.GetDataPtr(), indexer.GetSize());

Afterthoughts

This solution is clean, and easy to use and maintain. I am fairly happy with it.

Of course, what I am not yet happy about is the lack of error handling. For example, the indexer class does not yet check if the array is actually allocated or not. If it isn’t, then the current solution will dereference a NULL pointer.

There is also no size checking to prevent indexers from going out of bounds.

But as before, this code is proof of concept code. I am toying with the idea of implementing those error handlers, but I want to do it in a way that allows you to use either C style return value error handling or C++ style exception handling.

One possibility is to use error handling template classes. However, That will have to wait for next time.

The full code for the classes is available for download, together with the dll code and some LabVIEW vis for testing. Everything is licensed under the MIT license as usual.

Yet another contribution to the VCFAQ

vanDooren — Tue, 01 Aug 2006 20:43:00 GMT

Did you ever encounter problems to do with the size of structures in C or C++? In that case, chances are that you had to deal with structure layout and byte packing.

You can read my explanation over here.

The joy of receiving an MVP award

vanDooren — Tue, 04 Jul 2006 20:39:00 GMT

Yaaaaaaay.

Yesterday I received a message to tell me that I received the Microsoft MVP award in the category 'Visual C++'.

I knew I was nominated 2 months ago, which I found very cool. Of course I had to wait a bit because the anouncement of the new awardees was going to be made on the 1st of July.

When I didn't get a mail by sunday evening (the 2nd of July) I figured I did not get the award.

But then I got the mail with happy news on the 3d of July.

Sometime soon I will receive a package with information on all things that have to do with being an MVP. I do not yet know the details, but apparently I can receive a free MSDN subscription. On top of that it is possible to get access to private Microsoft newsgroups or even to windows source code.

I'll have to see what it all means, but it sure looks exiting.

My MVP profile can be found here.

Another contribution to the VCFAC

vanDooren — Sun, 18 Jun 2006 20:30:00 GMT

My second contribution to the VCFAQ is now online. It discusses 3 common causes of segmentation faults after upgrading a project to a newer version of Visual C++.

You can read it over here:

http://vcfaq.mvps.org/lang/9.htm

Is it possible to create (USB) device drivers with Visual C++?

vanDooren — Thu, 15 Jun 2006 09:20:00 GMT

This question pops up in the Visual C++ newsgroups from time to time.

You can’t. At least, you cannot directly perform USB communications. USB simply doesn’t work that way. They same is true for PCI, Firewire, SCSI, ...

All USB IO has to be performed by a USB device driver. User space applications send IO requests to the device driver. The device driver will then perform some processing and complete the request.

The first possibility is that your USB device is of a standard device class in which case it doesn’t need a special device driver, or it needs its own device driver, but exports a standardized device interface.

A good example of this is a USB to serial converter. The device driver registers a standardized serial interface with the system.

Any user mode application can then use the normal .NET or win32 APIs for serial communications without having to care about the fact that the serial line is in fact connected to a USB port instead of being soldered to the motherboard.

The next possibility of course is that the device is a non-standard device. In that case you cannot access it through a standard windows high level API. Normally the device manufacturer will ship a high level API (e.g. a dll or ActiveX object) with the device driver to allow you to interact with the devices.

If no API is available, you will have to use the CreateFile, DeviceIoControl, ReadFile, WriteFile and CloseHandle functions (among others) to directly communicate with the device driver.

Lastly, it is possible that you have a custom device of which you only have the interface documentation, but no device driver or API. In that case you need to create your own device driver. For more information on writing device drivers you can read my articles (and others on www.codeproject.com) as a starting point:

http://www.codeproject.com/system/wdf_kmdf_basic.asp

http://www.codeproject.com/system/kmdf_osr_usb_fx2.asp

The first article explains all the steps involved in actually creating a device driver. The second article explains all the issues in programming a fully functional USB device driver. Both articles use the Microsoft’s new KMDF framework. It is relatively easy to learn if you have C/C++ experience.

There are some other toolkits available from 3d parties, but they cost money, and a bigger problem is that they are non-standard so you cannot ask other developers for support unless they use the same toolkit. Another problem is that the vendor might discontinue the toolkit, and leave you with nothing but unsupported software.

In any case, as soon as you need to develop a device driver you should not use VC++. Use the DDK build tools instead. They are the standard when it comes to building device drivers, and you can get good support on microsoft.public.development.device.drivers.

You can download the latest KMDF DDK here:

http://www.microsoft.com/whdc/driver/wdf/KMDF.mspx