Cluebat-man to the rescue : C++

Practical ATL: Solving the race condition in CAtlExeModuleT

vanDooren — Wed, 15 Apr 2009 12:40:00 GMT

When I was spelunking through the ATL header for my previous articles (the class object plumbing) I discovered that there is a serious race condition in the server lifetime management.

I contacted the Microsoft C++ folks, and someone from the libraries group told me that this problem has already been fixed in the next release of Visual C++.

But since the issue still exists in all current ATL projects, I decided to write something about it and publish the fix which will circumvent the problem in the recent versions of ATL (7.0 and upwards if I am correct).

The problem

The module lock count of an ATL COM server is managed in the CAtlExeModuleT<T> class. It is this lock count that keeps the server alive as long as it is nonzero.

The CAtlExeModuleT<T> class manages its reference count via the CComGlobalsThreadModel class increment and decrement methods. CComGlobalsThreadModel is a typedef for CComMultiThreadModel so these methods map to InterlockedIncrement and InterlockedDecrement

The last module lock is released if
- the class object itself has no more external connections (IExternalConnection), and
- there are no more object instances.

As soon as the final module lock is released (module refcount is 0), the shutdown of the server is initiated, which will happen through either a WM_QUIT message or a win32 event object. In both cases there is a time window between the triggering of the shutdown, and the actual call to CoRevokeClassObject. This time window can be several seconds.

This time window causes a race condition between the server (which is already dying at that point) trying to revoke the class object, and a hypothetical client, requesting an interface pointer to the class object. Depending on how the race turns out, the client might end up with an S_OK HRESULT from CoGetClassObject, and an invalid interface pointer. This will cause a critical problem in the client, and may cause an application crash.

The solution

The proper way to manage the server lifetime is to use CoAddRefServerProcess and CoReleaseRefServerProcess to manage the module lockcount. Those functions will (according to MSDN http://msdn.microsoft.com/en-us/library/aa910552.aspx) suspend all class objects when the reference count drops to 0, in an atomic operation.

In that case, the server starts dying AFTER all class objects are suspended. Any incoming client request for the class object will spawn a new server process after the dying server has finished revoking its class objects.

The fix

The code for fixing this is simple. I simply created a new class that inherits from CAtlExeModuleT<T> and then overrides the Lock and Unlock methods.

template <typename T>

class CAtlExeModuleT_NoRace : public CAtlExeModuleT<T>

{

public:

typedef CAtlExeModuleT<T> base;

#if _ATL_VER < 0x0A00

LONG Lock(void)

{

CoAddRefServerProcess();

return base::Lock();

}

LONG Unlock(void)

{

CoReleaseServerProcess();

return base::Unlock();

}

#else

//removed compiler messages

#endif

};

I use CoxxxxServerProcess functions to explicitly manage the class object registration, and then I delegate to the base class which maintains its own reference count.

The reason I do that is that the base::Unlock function is the one which triggers the module shutdown, and I don’t want to mess with that. And Unlock works only when paired with Lock, so the only thing I really need to do is to call the CoxxxxServerProcess routines to prevent possible activation requests. The rest is still done by CAtlExeModuleT<T>.

I put a compiler condition around my implementation of the new Lock and Unlock methods, because they are only needed for current versions of ATL. The next version of ATL won’t have this problem, so it would be dangerous to meddle with the reference counting, especially since I cannot predict what side effects this may have.

The conditional compilation allows you to safely upgrade the project to the next version of ATL. The compiler messages which show up in the compiler output inform you that it is no longer needed to inherit from CAtlExeModuleT_NoRace as shown below:

class CStuff_ServerModule : public CAtlExeModuleT_NoRace< CStuff_ServerModule >

{

public :

DECLARE_LIBID(LIBID_Stuff_ServerLib)

DECLARE_REGISTRY_APPID_RESOURCEID(IDR_STUFF_SERVER, "{597B5DFF-DACE-42B0-9E49-54A09C10B2BB}")

};

The only thing you have to do to use my new module class is to include the header and change CAtlExeModuleT to CAtlExeModuleT_NoRace.

And if you upgrade the project to VC10, then you can simply change it back as indicated by the compiler output messages that are printed out when the compiler parses my class.

Conclusion

I was surprised that this bug was still in the ATL headers after all these years. When I contacted the VC++ team I was half expecting them to tell me that I had missed something. This issue is what caused CoAddRefServerProcess to be created after all, over a decade ago, so a COM programmer implementing lifetime management should be aware of it.

However, the modular and hierarchical nature of ATL made it easy to fix this problem, so I guess that is a plus for ATL. And mistakes happen. It is always a bit painful if it happens in widely used libraries, but I too have had my share of whoopsies over the years. It is sad but it happens.

The header file with the fix is attached to this blog post under the MIT license.

Practial ATL: Implementing a non-standard class object

vanDooren — Fri, 20 Feb 2009 21:03:00 GMT

In my previous article in this series, I explained how class objects are associated with COM objects, and how it all fits together.

Armed with that knowledge, I can now demonstrate how to provide a custom class object that will allow us to support parameterized construction.

If you haven’t already read my previous article on class objects, please do so first because I am going to assume you did J

The non-standard class object interface: IStuffCreator

A class object can have any interface you want, just like the COM object we have created earlier. It is just another COM object after all. My custom interface looks like this:

[
      object,
      uuid(6DD69CDB-3128-432b-B335-773A287E6F06),
      oleautomation,
      helpstring("IStuffCreator Interface"),
      pointer_default(unique)
]
interface IStuffCreator : IUnknown {
[id(1), helpstring("method MakeMeAStuff")]
    HRESULT MakeMeAStuff(
      [in] BSTR name,
      [in] REFIID riid,
      [out, iid_is(riid)] IUnknown** ppStuff);
};

The bits between the square brackets are the attributes that are applied to this interface. ‘Object’ specifies that it is a COM interface. The uuid specifies the GUID that identifies the interface. ‘oleautomation’ specifies that the arguments being used are all automation compatible. The meaning of ‘helpstring’ is pretty obvious, and pointer_default specifies the assumptions that the marshaller can make about pointers.

If you want to know the details about it, then have a look in MSDN for MIDL attributes.

The interface itself is very simple. Instead of deriving from IDispatch like the IStuff interface, this interface derives from IUnknown. Deriving from IDispatch would only complicate matters without any benefit.

The interface has only one method, which is responsible for handing out new stuffs.

The parameter that will be used for constructing the CStuff instance itself is the ‘name’ parameter. If the CStuff would need multiple parameters, they would have to be part of the parameter list so that they can be used at the time of construction. The riid and ppStuff parameters have the same meaning as those of CoCreateInstance.

The declaration of CStuffCreator

If we want to implement this interface, we add a new C++ class with the name ‘CStuffCreator’ to our project, and implement our IStuffCreator interface

class CStuffCreator:
public IStuffCreator,
public CComObjectRootEx<CComGlobalsThreadModel>
{
public:

CStuffCreator(void){}
virtual ~CStuffCreator(void){}

BEGIN_COM_MAP(CStuffCreator)
    COM_INTERFACE_ENTRY(IStuffCreator)
END_COM_MAP()

//IStuffCreator methods
HRESULT STDMETHODCALLTYPE MakeMeAStuff(
    BSTR name,
    REFIID riid,
    IUnknown **ppStuff);

// helper method
void SetVoid(void* pv);
_ATL_CREATORFUNC* m_pfnCreateInstance;
};

As you can see, it is pretty simple. The CComObjectRootEx base class helps with reference counting, and is the same that is used by the other COM objects in our project.

We add IStuffCreator to the interface map using the ATL macros, and then we simply declare the single MakeMeAStuff method that we need to implement.

The SetVoid method and the function pointer are part of the default C++ interface for class objects in ATL, and are explained in my previous article.

The implementation

The implementation of CStuffCreator is even simpler than the declaration:

HRESULT STDMETHODCALLTYPE CStuffCreator::MakeMeAStuff(

BSTR name,

REFIID riid,

IUnknown **ppStuff)

{

CComPtr<IUnknown> newStuff;

HRESULT hr = m_pfnCreateInstance(

NULL, __uuidof(IUnknown), (void**)&newStuff);

if(FAILED(hr))

return hr;

return newStuff->QueryInterface(riid, (void**)ppStuff);

}

void CStuffCreator::SetVoid(void* pv)

{

m_pfnCreateInstance = (_ATL_CREATORFUNC*)pv;

}

Ignore the ‘name’ parameter for now. That will be used when we are going to initialize the new instance. The bare functionality of our class object is to use the function pointer to create a new instance. I could have done this in 1 step instead of 2 (the QueryInterface isn’t necessary at this point) but this way I can slide in some additional functionality later on.

And you can see now that the class object will receive the creator function pointer via the ‘SetVoid’ method. This mechanism was explained in my previous article.

Parameterized construction

The point of using a non-standard class object in this demonstration was to be able to create objects that need some sort of initialization before the client has access to them. Normally, ATL supports only default constructors using, because IClassFactory is a generic interface.

For parameterized constructors, we are going to have to do something special.

There are essentially 2 ways we can do this. The first is to change CStuff to have a parameterized constructor. But ATL can’t handle that, and it would mean we have to abandon the incredible convenience of being able to use the ATL::Creator<T> class, ATL::CComObject class, and other helper classes that take the pain out of COM.

Don Box wrote an article about parameterized construction, many years ago, and this was the approach he took. Of course, ATL was much simpler in those days, and abandoning it was perhaps less of an inconvenience.

The approach I am taking is much simpler: I cheat!

That’s right. I cheat. I am not going to implement parameterized constructors. I am going to fake it. But from the client’s point of view, it’s going to look exactly the same, so he won’t mind. And from the server’s point of view, it will be much simpler to implement, so everybody wins.

The second approach to implementing parameterized construction is to implement an internal interface on CStuff, which will be used for initializing it. The new CStuff instance will not be accessible to anyone until the first interface pointer is handed out by the class object. And we are the ones implementing the class interface, so we can make sure that the new instance is initialized properly before it is released into the wild.

The initialization interface

The interface for initializing a CStuff is pretty simple (I left out the header for clarity):

interface IStuffInit : IUnknown {

[id(1), helpstring("method InitStuffInstance")]

HRESULT InitStuffInstance(

[in] BSTR Name);

};

Our CStuff instances are supposed to have a name, so that’s what needs to be in the init interface. If they would also need an address, a hat, and a pair of pants, we would have to put those in the parameter list of the InitStuffInstance method as well.

Implementing the initialization interface

Implementing the interface in CStuff is easy I highlighted the changes that have to be made to the declaration.

// CStuff

class ATL_NO_VTABLE CStuff :

public CComObjectRootEx<CComMultiThreadModel>,

public CComCoClass<CStuff, &CLSID_Stuff>,

public IDispatchImpl<IStuff, &IID_IStuff, &LIBID_Stuff_ServerLib, /*wMajor =*/ 1, /*wMinor =*/ 0>,

public IStuffInit

{

private:

CComBSTR m_Name;

public:

DECLARE_CLASSFACTORY_EX(CStuffCreator)

// removed irrelevant stuff

BEGIN_COM_MAP(CStuff)

COM_INTERFACE_ENTRY(IStuff)

COM_INTERFACE_ENTRY(IDispatch)

COM_INTERFACE_ENTRY(IStuffInit)

END_COM_MAP()

// removed irrelevant stuff

HRESULT STDMETHODCALLTYPE InitStuffInstance(BSTR Name);

};

First we add IStuffInit to the inheritance list, we add it to the COM map, and then we add the declaration for InitStuffInstance.

Of course, the name has to be part of the objects state, so we add a CComBSTR variable to hold it.

And finally, we use the DECLARE_CLASSFACTORY_EX macro to change the class object for CStuff from CComClassFactory to CStuffCreator.

The actual implementation of the interface consists of the simple code

STDMETHODIMP CStuff::InitStuffInstance(BSTR Name)

{

m_Name = Name;

return S_OK;

}

Note the ease with which CComBSTR allows us to perform simple string operations.

Initializing the new CStuff object

Now that there is a dedicated function to handle the initialization, the changes to CStuffCreator are trivial:

HRESULT STDMETHODCALLTYPE CStuffCreator::MakeMeAStuff(

BSTR name,

REFIID riid,

IUnknown **ppStuff)

{

CComPtr<IStuffInit> newStuff;

HRESULT hr = m_pfnCreateInstance(

NULL,__uuidof(IStuffInit), (void**)&newStuff);

if(FAILED(hr))

return hr;

hr = newStuff->InitStuffInstance(name);

if(FAILED(hr))

return hr;

return newStuff->QueryInterface(riid, (void**)ppStuff);

}

What about double initialization

By now you might be wondering: But what if the client application retrieves this interface and performs another initialization?

However, this will not be possible.

It’s true that I put the interface in our IDL file, and that I implemented it in the CStuff object. But I did not put that interface in the library declaration, nor did I put it in the interface list of the coclass.

So for all practical purposes, this interface is not accessible to the client application. If you are really paranoid, you could argue that an attacker might know the interface GUID, and guess the interface definition, and with some trial and error make it work.

Even then, the fix is quite simple. You can add a boolean to your class that indicates if the object was initialized or not. Set it to false in the constructor, and then the InitStuffInstance can either perform initialization if it is false and set it to true, or return E_FAIL if it is already true.

You don’t even have to care about race conditions, because there aren’t any. Only the class object can perform the real initialization. And since no one else has an interface pointer yet, there can’t be a race.

The client side

Creating a new CStuff has changed a little bit, because now we can’t use CoCreateInstance anymore. So construction is now done in 2 stages:

CComPtr<IStuffCreator> stuffClass;

hr = CoGetClassObject(

__uuidof(Stuff), CLSCTX_ALL, NULL,

__uuidof(IStuffCreator), (void**)&stuffClass);

if(FAILED(hr))

return hr;

//make a new stuff

CComBSTR stuffName = L"Kato";

CComPtr<IStuff> newStuffUnk;

hr = stuffClass->MakeMeAStuff(

stuffName.m_str,

(GUID*)&__uuidof(IStuff),

(IUnknown**)&newStuffUnk);

if(FAILED(hr))

return hr;

It’s still very simple though. First we get the class object, and then we use the IStuffCreator interface to create the new CStuff.

Server lifetime management

Everything works, hip hip hurray… until you destroy the first CStuff instance, and then try to create another one. That is where it all goes pear shaped.

The problem is that we haven’t taken care of managing the lifetime of the server (the exe). The key issue here is the module lock reference count.

Each COM module (exe or dll) has a global reference count to keep track of how many objects are ‘alive’. COM objects exist physically inside a DLL or EXE. So the EXE has to stay alive as long as there are living objects inside. Conversely, as soon as there are no living objects anymore, the EXE should shut down.

What happens in our example is that CStuff raises the module ref count when it is created. And when the CStuff instance destroys itself, it lowers the module ref count. If you remember correctly, the class object itself uses CComObjectNoLock as its COM core. This means that its existence does not influence the module lifetime.

So when the CStuff instance destroys itself and lowers the module ref count, the ref count reaches 0 and the server EXE shuts down.

The client is unaware of this and tries to use the class object again. It still thinks it has a valid interface pointer, but the underlying class object is long gone and there will be a critical error in the client application.

You might say ‘well duh, then don’t use CComObjectNoLock, doofus’ but that has the opposite effect. The COM runtime will always have the interface pointer that was used for registering the class object with CoRegisterClassObject. This means that the class object would not destroy as long as the server lives. And the server would keep living as long as the class object was around. This is clearly not the answer.

IExternalConnection to the rescue

The key to solving this problem is a rarely used standard interface. IExternalConnection is an interface that can be used by an object to track how many external connections there are to itself. This is basically an alternative to the standard object reference count.

There are 2 methods in this interface:

STDMETHODIMP_(DWORD) AddConnection(

DWORD extconn, DWORD dwReserved);

STDMETHODIMP_(DWORD) ReleaseConnection(

DWORD extconn,

DWORD dwReserved,

BOOL bLastUnlockReleases);

If a client requests any class object, the COM runtime will retrieve it. But before the interface pointer is released to the client, it will request some interfaces itself for housekeeping purposes. One of those interfaces is IExternalConnection.

If a client requests an interface pointer to a class object, the marshaller will call AddConnection when it hands out the pointer, and ReleaseConnection when the pointer is destroyed.

So all we have to do is to make sure that a call to AddConnection will increment the module lock count, and a call to ReleaseConnection releases the module lock count.

Implementing IExternalConnection

If you look in MSDN, you won’t find it. It is undocumented, but the ATL headers contain an implementation for IExternalConnection. It’s called IExternalConnectionImpl<T>.

To use it, we only have to change CStuffCreator so that it inherits from it, and add it to the COM interface map.

class CStuffCreator:

public IStuffCreator,

public IExternalConnectionImpl<CStuffCreator>,

public CComObjectRootEx<CComGlobalsThreadModel>

{

public:

//removed irrelevant stuff

BEGIN_COM_MAP(CStuffCreator)

COM_INTERFACE_ENTRY(IStuffCreator)

COM_INTERFACE_ENTRY(IExternalConnection)

END_COM_MAP()

//IExeternalConnectionImpl<T> extension

void STDMETHODCALLTYPE OnReleaseConnection(

bool bThisIsLastUnlock, bool bLastUnlockReleases);

void STDMETHODCALLTYPE OnAddConnection(

bool bThisIsFirstLock);

//removed irrelevant stuff

};

That is really all there is to it. The default implementation does not modify the module lock count. But it has 2 extension points that we can override. So that’s what we do to make sure the module lock count is managed properly.

void STDMETHODCALLTYPE CStuffCreator::OnAddConnection(

bool bThisIsFirstLock)

{

_pAtlModule->Lock();

}

void STDMETHODCALLTYPE CStuffCreator::OnReleaseConnection(

bool bThisIsLastUnlock, bool bLastUnlockReleases)

{

_pAtlModule->Unlock();

IExternalConnectionImpl<CStuffCreator>::OnReleaseConnection(

bThisIsLastUnlock, bLastUnlockReleases);

}

The only special thing that still needs mentioning is that OnReleaseConnection delegates back to the default implementation in IExternalConnectionImpl. It does this because there is something that needs to be done when the last external connection is closed.

void OnReleaseConnection(

bool bThisIsLastUnlock, bool bLastUnlockReleases)

{

if (bThisIsLastUnlock && bLastUnlockReleases)

CoDisconnectObject(static_cast<T*>(this)->GetUnknown(), 0);

}

If the last connection closes, the class object needs to call CoDisConnectObject to destroy its marshalling stub. Now I could copy paste this code into my implementation of OnReleaseConnection, but it is simpler to just delegate back to the base class for stuff that needs doing anyway.

The result

Now that we have provided this implementation of IExternalConnection, the server EXE will stay alive until all CStuff instances are gone, and noone is using the class object anymore. Only then will the EXE terminate.

CComClassFactory revisited

If you look at the interface implemented by CComClassFactory, you’ll notice that IExternalConnection is missing. And yet, server lifetime of CComClassFactory is managed correctly.

The reason for this is that the COM runtime treats IClassFactory class objects differently from other class objects. It will check if the class object supports IClassFactory. And if it does, it calls IClassFactory::LockServer to indicate that someone is using the class object. And it unlocks the class object when the class object reference is released again.

In a way, it uses the IClassFactory in almost the same way as it would use IExternalConnection. And in a way it is too bad that this functionality was included in IClassFactory directly, instead of making IClassFactory inherit from IExternalConnection. The reasons for this are lost in the mists of time. But whatever the reasons, this is why IClassFactory doesn’t need IExternalConnection.

Conclusion

As you can see, implementing a non standard class object is not so hard. As long as you have a basic understanding of the underlying COM principles, ATL takes care of all the dirty work.

And by using these techniques, we can implement COM objects that need initialization before they can safely be used by the client. The source code for this article is available under the MIT license as usual.

And again I’d like to add a thank you for Microsoft’s Tim Springfield and Amit Mohindra

Practical ATL: Understanding the class object

vanDooren — Tue, 10 Feb 2009 07:46:00 GMT

In my previous article in this series, I explained how to create a simple COM server and implement a method that returns an enumerator object.

It was all done very easily, but the ATL wizards hide some of the things that you really ought to know for my next article. For starters: where is the class object?

The class object is the COM object responsible for creating new instances of the CStuff COM object that I demonstrated in a previous article. But we didn’t have to write any code for it, nor is it even visible… so what’s up with that?

Some more on class objects

The class object of a COM server is the object that is actually responsible handing out instances of the COM object. The class object is a COM object itself, but only 1 instance of it normally exists in a module (DLL or EXE) per defined COM object type.

For example, the ‘Stuff’ COM object that I implemented in my previous article uses the standard class factory with IClassFactory as the interface for handing out new CStuff instances.

The most important method in IClassFactory is ‘CreateInstance’ which simply creates a new instance of the COM object and then passes an interface pointer back to the caller.

Normally, this process is invisible to the client application, because ‘CoCreateInstance’ uses IClassFactory behind the scenes so that the client doesn’t have to bother with it.

First it calls ‘CoGetClassObject’ to get the class object for the requested COM object, and then it uses the IClassFactory interface of that class object to create a new instance. That instance is then passed to the caller, as an interface pointer of the type that was requested in the call to ‘CoCreateInstance’.

I think it is safe to say that ‘IClassFactory’ (or one of its relatives) is the interface on the class object of 99.99% of all COM objects that have a class object.

The default class factory

By default, CStuff uses CComClassFactory as its default class object. But looking at the declaration of CStuff, it is not immediately apparent how it does this.

CStuff derives from CCoClass<T>, which has the macro DECLARE_CLASSFACTORY in its implementation.

For EXE servers, this macro expands to DECLARE_CLASSFACTORY_EX(ATL::CComClassFactory), which in turn expands to

typedef ATL::CComCreator< ATL::CComObjectNoLock< ATL::CComClassFactory > > _ClassFactoryCreatorClass;

DLL servers use CComObjectCached instead of CComObjectNoLock, but I don’t cover DLL servers in this article.

As you can remember from the first article that featured CStuff, the C++ COM classes are missing the IUnknown core by default, because the implementation of that core depends on how we want to the object to behave.

This means we have to shove our class into a COM object to make it whole. For class objects, the CComObjectNoLock is most appropriate, because the lifetime of the object should not contribute to the lifetime of the server executable. Otherwise there would be a deadlock.

The server would start, and create the class object. And then the server would continue to live until the class object was destroyed. But since the COM runtime will always have a reference to the class object, the reference count would never reach zero, the class object would never be destroyed, and the server would never be able to shut down.

And finally, this class definition is shoved into a creator object which will help us by creating a properly initialized instance of the class object itself via its static CreateInstance function.

Associating the COM object C++ class with the class factory

So the COM object C++ class has a typedef that identifies the class object type. But how does the executable use this fact?.

This is where things get a little hairy, and possibly start to make ominous squishy sounds J

After the class declaration of CStuff, there is the innocuous line OBJECT_ENTRY_AUTO(__uuidof(Stuff), CStuff)

The OBJECT_ENTRY_AUTO is a macro that expands to this:

#define OBJECT_ENTRY_AUTO(clsid, class) \
      __declspec(selectany) ATL::_ATL_OBJMAP_ENTRY __objMap_##class = {&clsid, class::UpdateRegistry, class::_ClassFactoryCreatorClass::CreateInstance, class::_CreatorClass::CreateInstance, NULL, 0, class::GetObjectDescription, class::GetCategoryMap, class::ObjectMain }; \
      extern "C" __declspec(allocate("ATL$__m")) __declspec(selectany) ATL::_ATL_OBJMAP_ENTRY* const __pobjMap_##class = &__objMap_##class; \
      OBJECT_ENTRY_PRAGMA(class)

Don’t worry if the sight of this code makes you nauseous. That’s normal. I highlighted the important parts

What it does is simple. It’s just how it does it that is ugly. It creates a map entry for the ATL innards of our server executable. In this map entry, it associates the static method for creating the class object with the static method for creating an object instance.

Creating the class object

Now that the class object and the COM object are associated with each other, the next bit of code in the ATL headers is where the magic happens:

_ATL_OBJMAP_ENTRY* pEntry;
if (m_pObjMap != NULL)
{
    pEntry = m_pObjMap;
    while (pEntry->pclsid != NULL && hr == S_OK)
    {
      hr = pEntry->RegisterClassObject(dwClsContext, dwFlags);
      pEntry++;
    }
}

When the ATL module initializes, it will iterate over all elements in the object map and register each one. And this registration process entails the following:

    IUnknown* p = NULL;
    if (pfnGetClassObject == NULL)
      return S_OK;
    HRESULT hRes = pfnGetClassObject(
                         pfnCreateInstance,
                         __uuidof(IUnknown),
                         (LPVOID*) &p);
    if (SUCCEEDED(hRes))
      hRes = CoRegisterClassObject(
               *pclsid, p, dwClsContext, dwFlags, &dwRegister);
    if (p != NULL)
      p->Release();
    return hRes;

If you do a bit of spelunking in the headers, you’ll see that pfnGetClassObject is a function pointer that maps to _ClassFactoryCreatorClass::CreateInstance. So this code simply creates a new class object and registers it.

Again, this is specific to EXE servers. In DLL servers, the class object is created on demand in the DllGetClassObject function, which is called by CoGetClassObject if the COM server is a DLL.

Creating CStuff instances

If you look at the _ClassFactoryCreatorClass::CreateInstance, you’ll see that its first parameter is a void *. And the value that is passed in this parameter by the ATL module is the address of the static method for creating new object instances.

_ClassFactoryCreatorClass::CreateInstance simply creates a new class object ‘p’, and then calls p->SetVoid(pv); to pass the aforementioned function pointer to the class object.

And the SetVoid method is no more complicated then this:

void SetVoid(void* pv)
{
m_pfnCreateInstance = (_ATL_CREATORFUNC*)pv;
}
_ATL_CREATORFUNC* m_pfnCreateInstance;

I guess the original developer hadn’t had his morning coffee yet when he came up with this name. It manages to be 100% factually correct, while being 100% opaque at the same time. But it does the job.

When the class object has to create a new instance, it simply does the following:

STDMETHOD(CreateInstance)(LPUNKNOWN pUnkOuter, REFIID riid, void** ppvObj)
{
    // snip irrelevant code
        hRes = m_pfnCreateInstance(pUnkOuter, riid, ppvObj);
    // snip irrelevant code
    return hRes;
}

Providing a new class factory

Now that you understand how the class object plumbing is hooked up, it is easy to see how you can implement your own class object.

Remember how DECLARE_CLASSFACTORY_EX maps a class object to the _ClassFactoryCreatorClass typedef?

In C++, a typedef in a derived class can override a typedef in a base class. So even though the CCoClass<T> base class says that the _ClassFactoryCreatorClass maps to ATL::CComCreator< ATL::CComObjectNoLock< ATL::CComClassFactory > >,

We can override this by simply putting DECLARE_CLASSFACTORY_EX(CStuffCreator) In the declaration for CStuff. That’s really all there is to it. The macro magic takes care of the rest.

Conclusion

If changing the class object for a COM server is as trivial as adding a single line to your COM server C++ class, then why did I just write 4 pages of stuff?

I wrote all this because if I tell someone to use a macro to do something non-trivial, I want him or her to understand what is going on. There are already far too many programmers on this planet who do things without understanding what it is they do, or how it works. And I am not going to add some more by glossing over the details.

Of course, judging by the number of visitors on this corner of the internet that I call ‘home’, I stand little risk of doing so no matter how crappy I would write J

This article was originally part of my article on implementing a custom class object. But after I spent more than 3 pages on the previous explanation, I thought it would be best to put it in a separate article.

A big thank-you goes to Jim Springfield of the Architects team within the developers division (I think) of Microsoft for taking the time to review this article for correctness, and to Amit Mohindra for helping my questions and articles reach the right people.

And you can find the demo project attached to this article under the MIT license.

Getting started with ATL

vanDooren — Tue, 20 Jan 2009 10:53:00 GMT

I am currently working on some projects where I have to program a DCOM server. There are several reasons why it has to be DCOM and C++, instead of e.g. .NET remoting and C#.

The biggest pain (imo) about DCOM is that the technology stems from an era when the internet was no ubiquitous, and there were no huge forums filled with experts, or blogs by programming gurus and sites like codeproject.com There were newsgroups, but those were generally only used for specific problems.

There is a vast amount of concise information about everything you can do with the .NET framework. For DCOM there is … a big dark void.

Granted, there are several resources out there. But if you look on sites like codeproject, you will find that most of the articles about DCOM and ATL are written before 2003. Most of the information out there will not cover the improvements that have been made to ATL (or DCOM) since the last 5 years.

And when looking into the more obscure corners of DCOM, there are some things that noone really likes to talk about because the documented semantics are so vague, and if you look into things like custom IMoniker implementation… well… that is a story for another day.

Understanding DCOM

The most important thing with DCOM is that you understand the basics really well. I guess this is true for most things, but unlike other things like .NET, DCOM does not cut you any slack, and does not provide you with tons of helpful diagnostic information.

When I started with DCOM a couple of years ago, I first read ‘Essential COM’ by Don Box. This is really the most excellent book on COM ever.

Of course, this information becomes vitally important in the real world when you have to debug a problem, because no tool is going to tell you anything useful with DCOM related crashes.

Understanding ATL

In the real world (unless maintaining legacy code) you use ATL to write COM clients or servers. Personally, I don’t like books that just tell me how to use technology XYZ. I like to understand the ‘why’ and ‘how’ too.

This is where I have to give big thanks to ‘ATL Internals, Second Edition’. It is one of the very few books on ATL that were written for VS2005, and it is still valid for VS2008. It is also an extremely well written book.

If you are starting with ATL (or if you want to know how it works) buy this book. This book is very, very good, and will teach you a lot.

What happens next?

Over the next weeks / months I will probably write a couple of articles describing how to do specific things with ATL. One article that is nearly finished is about implementing and using IEnumString using ATL.

Not rocket science by any measure, but ATL is severely lacking in the area of easy-to-find, up-to-date and to-the-point examples. So I figured that there is still value in writing about the ancient art of circles and lollipops, even if many developers probably think COM and ATL are no longer worth bothering with.

Oh and I released some win32 DCOM demo projects some time ago. They were interesting for me because making them taught me a lot about the raw DCOM stuff that you normally don’t see with ATL. But I’ve always found that it is always useful to know what makes the motor hum.

So while they are not terribly useful for practical purposes, they are well documented and they might be useful if you are working your way through ‘Essential COM’

Found an interesting bug in the Visual Studio IDE

vanDooren — Thu, 08 Jan 2009 07:18:00 GMT

I know, I know, ... .NET is all the rage, and DCOM is legacy technology, best not touched by up and coming programmers with good hair and sharp look, lest they appear 'uncool' or even worse: 'obsolete'.

But some of us still use DCOM for a number of good reasons. And ATL takes much of the pain out of DCOM development. Unfortunately, the Visual Studio IDE doesn't always cooperate. I made a simple MFC dialog client application, and then tried to add the ATL Server project that the client would connect to. VS was clearly concerned about my attempts to use DCOM instead of .NET, and decided to sabotage my attempts to do anything useful with this new ATL project.

https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=391506

But luckily, I was able to figure it out and outsmart VS :)

Backing up event logs in their normal binary format

vanDooren — Fri, 19 Dec 2008 13:15:00 GMT

Some time ago I was looking for a way to make backups of the eventlogs of our server, preferably without me ever having to do anything anymore J

This was not success. There are a couple of tools to make text exports from eventlogs. These have the disadvantage that they make the logs very large and unsuitable for looking at with the even viewer. As an added bonus, these are slow too (since every item has to be read individually).

EventQuery.vbs is one of these tools and it is so dog slow that it is almost a crime that it has been included with Windows. Ever wanted to bring a domain controller slow to a crawl? Simple: Use EventQuery.vbs to export the security event log. That’ll do it.

Anyway, I checked with my fellow MVPs, and there is no built-in (or free) tool to save event logs as binary event log files, so I gave up.

But then I accidentally discovered that there is a win32 API that exposes the event log functionality. And as luck would have it, I am a programmer and thus perfectly suited to make such a thing myself. It would have been trivial to make it at work, but I decided to do this at home so that I could share the code.

It is of course perfectly possible that such a free tool exists and I simply didn't find it.

The code

The code is simple, though the line count was increased by the desire to implement error handling.

int _tmain(int argc, _TCHAR* argv[])
{
_TCHAR* computerName = NULL;
_TCHAR* logName = NULL;
_TCHAR* file = NULL;

//populate the variables, based on the input arguments
if(argc == 3)
{
    computerName = NULL;
    logName = argv[1];
    file = argv[2];
}
else if(argc == 4)
{
    computerName = argv[1];
    logName = argv[2];
    file = argv[3];
}
else
{
    Usage();
    return -1;
}

//Open the eventlog
HANDLE hEventLog = OpenEventLog(computerName, logName);
if(NULL == hEventLog)
{
    _tprintf_s(TEXT("Could not open eventlog %s on computer %s\n"),
      logName, computerName);
    PrintError();
    return -1;
}

//backup the event log to file
if(FALSE == BackupEventLog(hEventLog, file))
{
    _tprintf_s(TEXT("Could not save eventlog to file %s\n"), file);
    PrintError();
    CloseEventLog(hEventLog);
    return -1;
}

CloseEventLog(hEventLog);
return 0;
}

There is little to tell about the code. It can be invoked with either 2 or 3 command line arguments:

The name of the computer of which the logs have to be backed up. (optional)
The name of the event lot to back up
The path of the file to which the event log needs to be saved.

The application opens the event log, and then makes the backup and closes the eventlog. If something goes wrong, the error code and error message are printed to the command line.

I thought about making the error handing a little more elegant (so that PrintError and CloseEventlog were not used multiple times in the code) but with only 2 function calls, the amount of error handling code would not diminish or be any better.

I left the PrintError function out of this blog because it doesn’t do anything exciting. It retrieves the error code and prints out the corresponding error message.

What you need to know before using this tool

I’ve been using this tool for a week now, and it is working splendidly, but there are a couple of things that aren’t mentioned in the MSDN documentation of the API.

The first is that the supplied path is absolute for the computer of which you are making an even log backup. So if the path is D:\app.evt, it will make the backup to D:\app.evt on the remote computer.

The second thing you need to know is that providing a UNC name for the target file doesn’t really improve things all that much. Because what is happening under the hood is that it is not you who is making the backup.

The event log service is in charge of the event log files, and you are asking it to make a backup. And that is where it all goes pear shaped. The event log service runs with LOCAL_SYSTEM credentials. This means that by default, it has no credentials that are valid on a remote computer. So you cannot make backup files in network locations, because the event log service of the remote computer has no privileges to access the location.

There are 2 exceptions to this rule. 1) the remote location is on the remote computer itself. 2) the remote computer is a backup domain controller.

It is a little known fact (I didn’t know it either) that LOCAL_SYSTEM on a domain controller is a domain administrative account that can pretty much do anything and everything on any domain computer, including the things that even a domain admin cannot.

Conclusion

I needed a convenient tool for making backups of event log files, and now I have one. Both the source and the binary are included as attachments. The source is available under the MIT license, and the exes are available for free to do with them what you want, as long as it is understood that whatever you do with them is your responsibility, not mine.

The eventtool_clr exe is something I made to create and delete event logs. I don’t think I blogged about it yet, but what it does is so simple it’s hardly worth mentioning. Have a look at the sources for more info.

The binary is for those who just want to use it and don’t care. The source is for those who’d rather not run an untrusted binary from some guy on the internet on their internal servers J The sources are simple enough to audit and compile.

On a side note, The API to backup event logs is not exposed on the .NET api for event logs. I don’t know exactly why, but I suspect that it is because of the problems I mentioned. The API is confusing enough that it is really not fit for generic use. After all, you can’t backup to remote systems, and the path is absolute for the remote computer.

I also didn’t want a .NET program, because for this type of thing, nothing beats the convenience and performance of a compiled binary that needs no runtime DLLs or external frameworks.

Interesting blog post on the VC blog

vanDooren — Thu, 30 Oct 2008 11:09:00 GMT

http://blogs.msdn.com/vcblog/archive/2008/10/28/lambdas-auto-and-static-assert-c-0x-features-in-vc10-part-1.aspx

Stephan blogs about new stuff in the upcoming C++ 0x standard. It's quite length, but well worth a read.

Small bug in the MSDN documentation

vanDooren — Thu, 09 Oct 2008 06:02:00 GMT

Hardly worth mentioning, but there is a minor bug in the documentation of fopen and its friends. The remarks section lists the open mode encoding for 16 bit unicode as UTF16-LE instead of UTF-16LE.

As I said it is very minor, but if you use fopen, care about the documentation and have 5 minutes to spare, you can hop over to my bug report on connect and add a vote.

How not to ask question on a forum

vanDooren — Mon, 06 Oct 2008 06:29:00 GMT

Here is another person who needs to learn some netiquette (emphasis mine).

Hey,
well, I will not describe my situation because it is not part of my question and has nothing directly to do with it and would only lead to false assumptions as it already did...
My question is ... <snip>
Not that I am implying anything, but seeing that the discussion may drift to an entirely different topic directly, please do not return the question of why I am wanting to do this. I do know what I am doing and why I would need this.

Any help would be greatly appreciated.
Regards,
pd

His question would have a much better chance of being answered if he'd recognised that the people answering questions are actual people. We are volunteers, doing this to help and because some questions are actually very interesting. Treating us like helpdesk personnel is not something that we generally appreciate.

We sometimes spend a lot of effort to answer a complex question if the situation is very interesting. Or perhaps because the issue is something advanced that might be helpful to ourselves one day. Denying us the interesting bits and treating us like a bad manager treats his serfs is more likely to cause the reaction 'well, in that case figure it out on your own'.

And apart from the fact that the motivation to answer has left me already, the thing with complex questions is that the devil is in the details. If you don't tell us exactly what you are trying to do, and which problem you are trying to solve, then how are we supposed to know a) what causes the problem, and b) how to solve it.

EDIT: I discovered after some digging that he had asked this question before, and the original thread was closed by a moderator because it could be useful to malware authors. Personally I don't agree with this decision. But the OP should have mentioned this in his new thread. That way we would have known something more. And it doesn't change the fact that he should have described his problem more clearly.

Programmer discovers that floating point numbers have a floating point

vanDooren — Thu, 02 Oct 2008 05:05:00 GMT

I hang out in the MSDN forums on a regular basis, to see if there’re any questions that need answering. Usualy this is pretty unexciting, but every now and again, the asker really needs a reality check.

This thread came up some time ago, and it refuses to die.

The first thing that stands out is the title itself : ‘A very serious bug in MS Visual C++’. Sensationalist headlines like this almost always indicate a serious misunderstanding on the part of the author. It also indicates someone who is very, very sure of himself, because it excludes any possibility that he himself might be wrong.

The initial message itself was the classical example of ‘newbie discovers floating point numbers, it’s the end of the world as we know it’

If you haven’t bothered following the link, this is his gripe:

            double a=111.567,b=111,c;
            c=a-b;

            // and you will receive
            //
            //a=111.56699999999999
            //b=111.00000000000000
            //c=0.56699999999999307
            //
            //instead => a=111.567, b=111, c=0.567;

Yawn… another programmer who discovers that the floating point format does not guarantee fixed decimal correctness. The results are correct within the required precision of the floating point format, so all is well.

A is no 111.567 because it is a floating point number, and not a fixed point number. In any case, googling for ‘what every computer programmer should know about floating point’ will get you a paper that explains this issue in detail.

The original poster also explains that he has tested with every compiler from VC6 to VC2008, and they are all wrong. That in itself should have been a clue to think twice before using such a preposterous title.

Various people tried to explain the problem, but to no avail. Here are a handful of quotes from the OP (who is building an ERP system...) in the course of the discussion:

· Industry standard ??? I am not agreeing with you. There are laws of mathematics which must be respected by all. I can not agree that 111,567 is equal to 111.56699999999999 because of simple reason that it is not equal.

You cover me with theory. Thank you. But things are much simpler.
I think that something fundamental such as a declaration of fractional numbers and their actions should not be in so surrounded way. I think that the variables with floating point ('double') are unusable at this time, because they do not always give accurate results. And I would like experts from Microsoft, which deal with these issues, in some way to offer basic solution to this problem.
All indicate the standard IEEE as a dogma. I am not familiar with the IEEE simply because I do not have time. But once the standard makes it impossible to use a certain type of fundamental variables and actions with them, maybe it is better to consider changes in the standard

But it gets even better! A second poster enters the discussion with the claim

‘I am absolutely astounded by two things here:
1. That this math bug is still floating around (I remember when it was a CPU issue)
2. That people here claim that it is not a bug in c++ in VS2003 or later’

Someone then explains that his SUN stations give the same results, just to indicate that this problem has nothing to do with compiler errors (or CPU dependency) whatsoever.

But it wasn’t meant to be. What did the second poster reply?

‘Thanks for testing it for me on the SPARC system.
Perhaps it is some bizarre feature of hardware-based floating point present in modern CPUs? This may account for why similar code compiled on older compilers (VS6) (who don't have modern FCPU knowledge) yields expected results.

…

It's a bit amusing when the system can't even handle a result with one decimal place!’

Ah well.

If people are misguided, you can do your best to make them see. But if they want nothing to do with ‘reality’, then you can’t force them to understand.

After all, the powers of reason are futile in face of the powers of persuasion that allow someone to think every CPU, compiler and engineer / programmer / scientist in existence is wrong.

My latest article in NTInsider

vanDooren — Fri, 26 Sep 2008 06:00:00 GMT

A bit late perhaps, but my latest article on API development got published in the NTInsider (owned by OSR Online).

The online version can be found here. It requires free registration.

I know that at least 2 people have read it completely, because that is the number of mails I got to tell me I missed something.

In the code example at the bottom, I acidentally switched the names of fooA and fooB in the declaration. Doh!
That same code, compiled for a 64 bit platform, gives a linker warning. I don't know how I missed that. This code was made using VS2005, so one of two things could have happened: VC2005 didn't report the warning (unlikely), or I only checked the compilation in detail, and then only checked if the function was exported without checking if there were linker warnings. It's probably the latter.

I'll see if I can get these errata published. But at least now I know that people actually read my articles and use the code. Well, at least one person is using the code :) I'll have to make an extra effort next time to verify that a) I didn't make a stupid type, and b) There are no warnings anywhere.

C++ keyword of the day: export

vanDooren — Wed, 24 Sep 2008 10:48:00 GMT

The export keyword is a bit like the Higgs boson of C++. Theoretically it exists, it is described by the standard, and noone has seen it in the wild. :)

Before I get flamed to hell and back: that last part is not entirely true. There is 1 C++ compiler front-end in the world which actually supports it. That is the one made by EDG (Edison Design Group). This is used by Comeau which claim to have the only compiler that is 100% standards compliant and Borland (for which the support of export is a bit vague). I heard that the Intel compiler shipped with it, but I don’t know if this is true or not.

What’s all the fuss about?

It seems a bit strange that a language should contain a keyword that all compiler makers refuse to implement. Comeau has some interesting content here as does Wikipedia.

The reason that I believe 'export' has never really taken off is that is seems to be an significant effort to implement it, combined with the fact that is doesn't achieve all that much. The export keyword allows a C++ programmer to declare a template class in a header, and then provide the implementation separately in a cpp file. This is the normal C++ way of doing things. Unfortunately, there's a snag.

Template class implementations cannot be compiled on their own. Template code itself is meaningless without specification of template arguments. Thus it is that a compiler doesn't know what to do with an implementation until the template class is used somewhere. At that point the template arguments will be known, and the compiler can compile the template class.

Since the template is only compiled then, the easiest thing to do would be to do what most compilers do now: demand that the implementation is in scope when it is used somewhere. I.e. the implementation has to be put in a header file, and that header file has to be included by the source file that is being compiled. the reason for this is that the compiler would simply not know where to look for it if the implementation was in a cpp file somewhere.

The export keyword would solve this by telling the compiler 'Look somewhere else for the implementation'. The compiler would then have to compile the implementation for that class, and save the object code somewhere for that combination of template arguments.

The export keyword sounds like a great thing, but it doesn't solve that many problems.

Separation of interface and implementation

The separation of interface and implementation is one of the cornerstones of C++ philosophy. The traditional way is to have the declaration in a header file, and then put the implementation in a cpp file.

Since a template implementation cannot be compiled anyway, we could just as well put it in a header file.

And then we could also make 1 header file with just the declarations, and underneath that include the header file(s) containing the implementation. The nice thing is that you can then put specializations in different headers to keep the code comprehensible.

This way we keep true to the principle of separation without needing the ‘export’ keyword

Distribution

You can’t distribute template classes in a binary form, no matter whether export is supported or not.

Because templates cannot be compiled on their own, you cannot link them into a static or dynamic library. The only thing you can do is to distribute the source code.

Templates are distributed as source code. The export keyword cannot do anything about that. And if you want to distribute templates, what are you going to do? You can develop your classes using ‘export’, but then they will be usable only by a very limited subset of compilers.

If you develop your classes as if ‘export’ doesn’t exist, then they can be used by all compilers.

Of course you could do both, and use ugly macro magic to construct your code in such a way that would leave the choice to the client programmer. Some people do this. But that is kind of ugly and time consuming. Furthermore, it’s not like it gets you any more clients, because they will be able to use your code in any case.

Compiler performance

It is true that our scheme of using header files is less efficient than using the export keyword. Without ‘export’, a template is compiled whenever it is used. You can mitigate this somewhat by including commonly used templates in a precompiled header file. But the impact will only be somewhat lessened.

And whenever an implementation detail changes, every source file which includes the template has to be recompiled. This can be a real pain in large projects.

Conclusion

‘export’ is facing the chicken or the egg dilemma.

‘export’ is neglected by most compiler vendors because it is not an important language feature -> You cannot use it in code that has to be portable (which is ironic, since it is a standard keyword) -> it is not used in many scenarios -> there is no large codebase depending on support for ‘export’ -> there is little incentive for compiler vendors to start supporting it.

Make no mistake: I would appreciate it if I could use ‘export’, if only because it allows me to organize my template implementations in a standard manner, but I don’t think it’ll happen soon.

References in C++ are not necessarily safe

vanDooren — Wed, 17 Sep 2008 12:46:00 GMT

People who are new to C++ sometimes have the mistaken idea that using references instead of pointers makes your code safe. People who have been programming a bit longer know this is anything but the case. References are just semantic sugar coated pointers.

I'll explain in more detail with a couple of examples. They use this pretty simple class A.

struct A
{
int &I;
A(int &i) : I(i)
{
}
void print(void)
{
cout << I << endl;
}
};

When A is instantiated, the constructor takes a reference to an integer and uses it to initialize an internal integer reference. The print method simply prints the value of that integer.

Case 1

int * i= new int(32);
A a(*i);
*i = 43;
a.print();
delete i;
a.print();

I create a new integer value on the heap and I initialize a pointer with its address. I then create a new instance of A, and pass the newly created integer by reference. a initializes its own internal reference with the address of that integer. And indeed, if I assign a value to that integer and then print a's internal reference, the values match.

But what happens if that integer is deleted from the heap? It depends. The behavior is undefined. The next read from a.I could result in a bad value or an application crash. But in any case, it is not safe.

You might argue that I am purposely using pointers to cause problems, but in a large application, you won't know how and where your objects will be created. This could happen.

Case 2

A second case is much more innocuous. It uses no visible pointers.

vector<int> v;
v.push_back(42);
A b(v[0]);
v[0] = 43;
b.print();
v.pop_back(); //remove that element from the container
b.print();
v.push_back(2); //put something else in the same location
b.print();

Instead of dynamically allocation memory myself, I put an integer in a vector and then feed a reference to that element into an instance of A. This new instance (b) is now using the location of an element in a vector as an internal reference.

The thing with vectors is that they normally have more capacity than elements. This is because you don't want the vector to allocate more space for every element that gets inserted. Likewise, you are not going to do a deallocation with each element that gets removed.

Because of this, when that element is removed, the physical space is still there. A read from that location is not immediately going to trigger a crash. Instead, the program will keep on running, but with bad data instead. Or maybe not, if that element was at the threshold of triggering a deallocation. With bugs like this, you never know what will happen.

Case 3

There is an even more interesting option. I am not going to write a repro case for it, but consider the following: I create multiple instances of A, giving them each a reference to an integer that is located on the stack, and I pass those instances of A as pointers to newly created threads, to be used there.

Meanwhile, the stack on which the integers are located is unwound. What happens next?

Again, it depends. Probably the stack gets stomped. Or perhaps only the data will be corrupted. Again, what happens is unknown. And I don't say that this design pattern is a good one. It isn't. But given certain constraints, it might just be an approach for solving a specific problem (though perhaps not the best one).

Conclusion

By now, anyone should be convinced that using references do not protect your application against design mistakes.

The only time your references are really safe is if:
a) your application is single threaded, and there is no asynchronous code executed anywhere, and
b) your classes don't store references anywhere.

Outside of those constraints, references don't guarantee anything more than pointers would. The only guarantee you have with references is that at some point in time, there was an object of the correct type at the specified location, and it was not NULL. Probably.

If someone is actively trying to subvert your code, you can't even be sure of that. But since there is nothing you can do about that, you might as well use references because they make your life easier, and are better than raw pointers when it comes to preventing honest mistakes.

How DLL exports really work

vanDooren — Fri, 12 Sep 2008 10:42:00 GMT

I found this list of article on Raymond's blog. Raymond's blog is one of the more interesting for programmers who use native APIs because he often touchs on things that are not documented, but interesting to know if you care about how things really work under the hood.

These links all point to information that goes into DLL importing and exporting, and how it is implemented under the hood. Very interesting stuff.

How were DLL functions exported in 16-bit Windows?

How were DLL functions imported in 16-bit Windows?

How are DLL functions exported in 32-bit Windows?

Exported functions that are really forwarders

Rethinking the way DLL exports are resolved for 32-bit Windows

Calling an imported function, the naive way

How a less naive compiler calls an imported function

What happens when you get dllimport wrong?

Names in the import library are decorated for a reason

The dangers of dllexport

Another thing which may be useful if you haven't read it yet are Michael Grier's articles on the NT DLL loader over here. I've blogged about those articles before. Michael is one of the project leaders who maintained the NT DLL loader for some time.

Stuck on stupid

vanDooren — Wed, 10 Sep 2008 20:31:00 GMT

It happens to every programmer. It doesn't happen often, but it does happen to everyone. You are looking at a piece of code or some debugging output, scratching your head and thinking 'This is impossible'. It's probably only 10 or 20 lines of code or text, and you've looked carefully at every line at least a dozen times over the last hour.

Every line makes sense and seems to mean exactly what you think it means, every word is in place, every semicolon is accounted for, and yet the sum of those lines is something completely unlike what you think it should be.

What makes matters even worse is that you just know that you are missing something obvious; something that anyone else would see after looking at the issue for less than 5 seconds. Maybe someone is even explaining the issue to you and you still don't get it.

Well, I had such a moment just this weekend.

I was working on some C++ code that involved friend functions and streams. I've never done much with streams in C++, and never done anything with friends. Basically, friend declarations allow e.g. class A to specify that class B is allowed to access its private variables. Hence the joke: in C++, your friends can see your privates. I've never used it in a design, because if other classes need access to private data, the design is probably wrong.

There are some exceptions, where it can be extremely useful. One of these cases is if you have to create a serialization function that can serialize and deserialize an object. That function would need access to the private data to efficiently reconstruct the object, but you don't want any other functions or classes to have the same privilege.

In my case, I had a template class that I wanted to reconstruct from a generic iostream. I implemented a global function that performed the reconstruction, but despite my friend declaration that gave istream access, the compiler didn't agree. This is what my code boiled down to:

class A
{
friend istream;
int i;
};

istream& operator >>(istream& is, A &a)
{
is >> a.i; //error C2248: 'A::i' :
//cannot access private member declared in class 'A'
return is;
}

I looked at it for a long time, read the 'friend' documentation in MSDN, consulted my copy of 'The C++ programming language, 3ed', asked a fellow MVP to explain it to me, and I still didn't get it. I reasoned: 'I just gave istream access to A::I, so WTF is the problem'.

In this case, the solution was really a 'duh' moment. Yes, istreams have access to A. But – and this is the whole issue – it is not the istream doing the accessing. The istream accesses an integer by reference; an integer that was retrieved by the global operator >> function. My code was functionally equivalent to:

istream& operator >>(istream& is, A &a)
{
int &temp = a.i; //error C2248
is >> temp;
return is;
}

And that is pretty obviously wrong. By writing is >> a.i, I lulled myself into believing it was 'is' who did the access, while in reality it was someone else.

As I said already: this was one of those 'Aaargh' moments where I just seemed to be stuck on 'stupid'. Ah well. Live, learn, and have more coffee next time J Glad this doesn't happen every day, week, or month.

Regex bug in VC2008 SP1 TR1 library

vanDooren — Thu, 04 Sep 2008 09:46:00 GMT

Yesterday I tracked down a bug in the TR1 regex library that is shipped with VS2008 SP1.

If you use regexes, then the regex a| will cause an exception to be thrown because of supposedly illegal syntax. For example you could use (a|) in a regular expression to indicate that a string contains at a certain point either a or nothing.

There are better ways to do this of course, and I've changed the problematic regex already, but it should have worked.

I checked the .NET regexes, and they accept this syntax. .NET regex follows the ECMAScript syntax, which specifies pretty unambiguously in section 15.10.1 (Thanks Igor for helping me identify the correct section) that the alternative operator should accept empty sections.
The TR1 proposal also specifies that ECMAScript is used, so it should accept this syntax.

If you use TR1 regexes, take a minute of your time to vote for this bugreport on connect so that they fix it in the next SP / release.

Extending a native C++ project with managed code

vanDooren — Wed, 03 Sep 2008 09:59:00 GMT

One question that comes up from time to time in the newsgroups is ‘I have a native C++ project and I want to extend it with Managed code (e.g. Windows Forms). What do I do?’

The answer is not so complex. It is fairly easy to extend native projects with managed code. In this article I’ll explain how.

What NOT to do

To quote Kate Gregory ‘In the name of all that is good and right: Do not set /CLR for the entire project’.

/CLR tells the compiler that it should compile a source file as C++/CLI code in which it finds both native and managed C++ code. Theoretically you could specify this for the whole project, but this has some serious consequences.

Apart from the fact that it is possible for the compiled output size to explode, there can be significant problems with COM and CRT initialization. I am no interop expert in the matter like Marcus Heege or Kate, but I am willing to take their word for it.

Furthermore, your existing project is possibly validated by QA en unit tested 7 days from Sunday. You don’t want to travel that road again if you can prevent it.

What to do

You want to add managed stuff to your project while impacting the rest of the code as little as humanly possible. To achieve this you have to compartmentalize the managed stuff and give it a native interface that you can use in the native parts of you projects without any hassle.

I think I can best explain this with a practical example.

For the sake of this example I have a native DLL project that exports a function that returns an integer value. Perhaps this function got that integer value previously through an MFC interface, and now that value has to come from a Windows Forms interface.

Add the windows form to your project

This is the first and also the easiest step of the process. Rightclick your project and select ‘Add…->New Item’ and select ‘UI->Windows Form’ in the Visual C++ dialog.

You will then get a message box, telling you that your project will be converted to a managed project. Here you click ‘yes’. Even though you don’t want to compile your project with /clr, Visual Studio itself needs to know that you are doing managed stuff in your project. This is not necessary for compiling your project, but if you want to have the benefit of working with the forms designer and other .NET related things, then it would be a good idea. And you cannot add the form if you choose ‘No’ so it’s not like you have much choice anyway.

Now you have a .NET Windows Form in your code. You can verify the file specific settings if you want, but the cpp file will be compiled with the /clr flag set, and without the use of precompiled headers. Not using precompiled headers is important, because the default precompiled header is compiled without /clr set, and this would lead to conflicts.

Of course, if you need to, you can create a copy of StdAfx.h/cpp cpp files called ‘StdAfxClr.h/cpp’ add them to your project, and configure them to create a managed precompiled header, which you could then use for all files that are compiled with /clr. This is not necessary for small projects, but it could be a useful optimization in large projects.

The interface layer

At this point you can compile and link the entire project, but your native code isn’t yet using the new managed functionality. And because the native code will not be compiled with /clr, it will never do so. In order to make that happen, you will need a thin layer between the native and managed code. This layer will have a native C or C++ interface which can be called by the native code, and a managed implementation that will do all the .NET stuff.

In our example, the interface is 1 simple C style function in Interface.h:

int __stdcall DoManagedStuff(void);

And this function has a simple implementation in Interface.cpp:

int __stdcall DoManagedStuff(void)
{
      DemoForm ^df = gcnew DemoForm();
      df->ShowDialog();
      return df->Value;
}

Interface.h and Interface.cpp are 2 new files that you have to add to your project. You have to manually configure the cpp file to be compiled with /clr, and to not use precompiled headers.

As you can see, the interface layer provides a clean native interface for the managed stuff that you want your code to perform. Of course, you are not limited to C style interfaces. You can also work with classes and make classes with a native interface and a managed implementation. But there are a couple of issues that you need to be aware of. If you want to go there, then it would be a good idea to read the paper written by Herb Sutter which explains the rationale behind C++/CLI, as well as some of the limitations and pitfalls.

Actually, reading that paper is a good idea for anyone working with C++/CLI who also cares about understanding what is actually going on behind the scenes.

Using the interface layer

All the hard work is done. Now you can simply include Interface.h in your native code files, and call ‘DoManagedStuff’ where appropriate.

#include "stdafx.h"
#include "InterfaceLayer.h"
int __stdcall DoFoo(void)
{
return DoManagedStuff();
}

Conclusion

As you can see, extending native code with managed code is not so hard. At least, it isn’t if you maintain a clean break between managed and native code. If you cannot do this, things might become more difficult. For example, using .NET controls on an MFC dialog, or using .NET remoting in a native COM project are things that can be much more tricky.

Another thing you should be aware of is that not all compiler switches can be used in conjunction with the /clr switch. The compiler will inform you if it detects this. The combinations which are not allowed are documented, and can be found in the documentation of the /clr switch itself.

This post was not meant to be an exhaustive how-to to those complex scenarios, but instead an explanation behind the basic ideas.

The demo project for this article can be downloaded under the MIT license as usual.

Fun with templates: parsing command line arguments

vanDooren — Thu, 14 Aug 2008 10:17:00 GMT

One of the things that everyone has to do sooner or later is parsing command line arguments into program variables. Even the most trivial command line application needs some input variables to tell it what to do.

One of the things in which C and C++ are lacking is a unified approach to command line arguments. This allows anyone to do what he wants, but unfortunately it also forces everybody to figure it out for himself.

I have done this several times, and tried several approaches. My last projects all needed moderately complex command line configurations, so I finally solved this problem in a reusable way. Originally I used the boost template library for this (www.boost.org). Specifically, I used regular expressions. But now that VS2008 SP1 is out, I can use the TR1 library to do this for me.

What does a command line argument look like

The exact syntax doesn’t matter because we are using regular expressions. What is important is that the identifier and the value are recognized by the shell to belong together.

The only way to do this is of course to make sure that they are in the same string. For example, such command line arguments can look like this:

-val:12.5
/text=blabla
-file=”C:\Documents and Settings\me\desktop\article.doc”

It really doesn’t matter. As long as all the information is recognized as 1 argument, it’s fine. If the value contains spaces, you need to use quotes so that the shell will not treat

-file=”C:\Documents and Settings\me\desktop\article.doc”

as 3 arguments.

Requirements

The requirements for my solution are as follows:

1 function which can take a command parameter and a regular expression, and which can parse any given type (bool, int, string, …) from the parameter, according to the specified regex.
The function should be able to handle string and wstring parameters.
It should be able to put values into regular data types, as well as into TriStateValue template types (see here http://msmvps.com/blogs/vandooren/archive/2007/10/11/fun-with-templates-implementing-a-tri-state-value.aspx for more info).
The function should return a bool to notify the user if the argument was parsed.
The part of the regex that identifies the actual value has to be captured. I.e. it has to be enclosed between ( ) symbols.

First attempt

The first version of such a function was ready pretty rapidly.

template <typename T>
bool ParseArg(
    std::tr1::regex const& pattern,
    std::string const& arg,
    T& value)
{
    std::tr1::smatch matches;
    if( !std::tr1::regex_search(arg, matches, pattern))
      return false;

    std::istringstream iss((string) matches[1]);
    return ! (iss >> std::dec >> value).fail();
}

The function takes a regex and an argument. The value type is a template type which allows the user to use it for any type. The type itself can be inferred at compile time. I.e. the user does not have to explicitly specify the template type using brackets.

A string stream is used for getting the value out of the stream. This is very handy because this way, the text to value conversion has no dependency on the value type. By using the string stream, the function remains generic.

This function has an overload that uses a TriStateVal type for the resulting value.

template <typename T>
bool ParseArg(
    std::tr1::regex const& pattern,
    std::string const& arg,
    TriStateVal<T>& value)
{
    std::tr1::smatch matches;
    if( !std::tr1::regex_search(arg, matches, pattern))
      return false;

    std::istringstream iss((string) matches[1]);
    T val;
    if((iss >> std::dec >> val).fail())
      return false;

    value = val;
    return true;
}

The TriStateVal type allows the user of that value to know whether it had been assigned or not. Unfortunately, there is no support yet for streaming directly into a TriStateVal type, so I have to use a temporary value first.

Of course, these 2 functions work for strings. What about wstring? Well, for wstring we need another 2 overloads.

template <typename T>
bool ParseArg(
    std::tr1::wregex const& pattern,
    std::wstring const& arg,
    T& value)
{
    //
}

template <typename T>
bool ParseArg(
    std::tr1::wregex const& pattern,
    std::wstring const& arg,
    TriStateVal<T>& value)
{
    //
}

These are almost identical, except that they use wregex, wstring and wistringstream instead of regex, string and istringstream.

The overload mechanism

By now you may have wondered about the function overloading, if you are into template laws.

There are 4 overloads (removed template specifier template <typename T>

for clarity):

bool ParseArg(regex const&, string const&, T&);
bool ParseArg(regex const&, string const&, TriStateVal<T>&);
bool ParseArg(wregex const&, wstring const&, T&);
bool ParseArg(wregex const&, wstring const&, TriStateVal<T>&);

It is simple enough to see how the compiler chooses based on the types of the pattern and the argument. That is via normal overloading rules.

But how does it know when to pick the generic T template function or the more specific TriStateVal<T> template? Both can be valid.

As it turns out, the compiler can follow the rules laid out in the C++ ISO standard section 14.5.5.2, which states that if multiple template functions are valid for the supplied template types, the most specific one will be selected.

The wording is of course more complex and formal than the previous paragraph, but that is what it boils down to. Section 14.5.5.2.5 contains a number of helpful examples, and one of them is equivalent with the ParseArg function.

TriStateVal<T> is more specific than T, so the function with that parameter type will be used whenever possible.

Evaluation of the first attempt

The first implementation works very well. You use the ParseArg function like this:

wregex reFile(L"^[-/]file[:=]\"{0,1}([^\\?\\*/<>\\\"]*)\"{0,1}$");
wregex reRows(L"^[-/]rows[:=]([0-9]*)$");
wregex reCols(L"^[-/]cols[:=]([0-9]*)$");

int _tmain(int argc, _TCHAR* argv[])
{
int numRows = 0;
TriStateVal<int> numCols;
wstring file;

for(int i=0; i< argc; i++)
{
    wstring arg(argv);
    if(ParseArg(reRows, arg, numRows))
      continue;
    if(ParseArg(reCols, arg, numCols))
      continue;
    if(ParseArg(reFile, arg, file))
      continue;
}

wcout << "numRows : " << numRows << endl;;
wcout << "numCols : " << numRows << endl;;
wcout << "file : " << file << endl;;

return 0;
}

As you can see, no matter how many command parameters there are and how complex their formatting is, parsing them remains trivial if you can correctly specify the regular expressions.

The regular expressions in my example also take care of the fact that users can use either the – or the / to precede identifiers, and that the value delimiter can be a = or : sign.

The great benefit of the TriStateVal is that you don’t have to keep track of Boolean variables, indicating the status of a variable. I.e. you don’t have to manually keep track of which variables were assigned.

If your application can do several things, based on a ‘command’ parameter for example, then you can check the TriStateVal variables for that command to see if those have been assigned to. If will make your whole program easier to understand.

Problems with the first attempt

One issue is that it sadly doesn’t work if T is of type ‘char’ or ‘wchar_t’. The reason is that these types are the character types which make up ‘string’ and ‘wstring’, and streaming from a string stream into a char is not supported.

Another possible issue is that in the case that T is of type bool, the text value has to be 1 or 0. Perhaps we would also like to be able to use ‘on’ and ‘off’ as valid value texts, or ‘high’ and ‘low’.

Both problems can be solved by providing more specific template functions, like (removed template specifier

for clarity):

template <> bool ParseArg(regex const&, string const&, bool &);
template <> bool ParseArg(regex const&, string const&, char &);

But if we want to implement those functions, we not only have to implement the string-regex function, but also the wstring-wregex function, and the functions that use TriStateVal parameters.

So instead of having to write 2 additional functions, we would have to implement 8 functions.

We could partially fix that by making the regex and string parameters templates instead of qualified types, but that does not fix the problem of the types that are used internally (smatch and istringstream).

Of course, we could make those template arguments as well, but that would break automatic type inference, and force the programmer to specialize the template function explicitly, which would be ugly and confusing.

So instead we use a trick that is also used in the STL itself: we use a helper class that deduces those internal types for us.

Second attempt

The key to the second attempt is to use a helper class that –through specialization- will specify the dependent types.

The coat hanger for this mechanism is the empty class

template <typename strType>
struct __pca_typehelper{};

which has 2 specializations:

template <>
struct __pca_typehelper<std::string>
{
    typedef std::tr1::smatch MatchType;
    typedef std::istringstream SStreamType;
    typedef std::tr1::regex RegexType;
};

template <>
struct __pca_typehelper<std::wstring>
{
    typedef std::tr1::wsmatch MatchType;
    typedef std::wistringstream SStreamType;
    typedef std::tr1::wregex RegexType;
};

Depending on which specialization is picked, MatchType, SStreamType and RegexType are typedef’ed to the correct STL and TR1 types.

And the nice thing is that is the user supplies something other than string or wstring, compilation will fail at this stage already, because those typedefs don’t exist.

Using that helper class, the ParseArg function can be written like this:

template <typename Targ, typename Tresult>
bool ParseArg(
    typename __pca_typehelper<Targ>::RegexType const& pattern,
    Targ const& arg,
    Tresult& value)
{
    typename __pca_typehelper<Targ>::MatchType matches;
    if( !std::tr1::regex_search(arg, matches, pattern))
      return false;

    typename __pca_typehelper<Targ>::SStreamType
      iss((Targ) matches[1]);
    return ! (iss >> std::dec >> value).fail();
}

As you can see, there are now 2 template arguments: Targ and Tresult.

The meaning of Tresult is still the same as in the previous implementation.

Targ is new, and is the type of the argument that needs to be parsed. In our example it is either string or wstring. This type is then used to specialize __pca_typehelper so that the type of the regular expression can be deduced:

typename __pca_typehelper<Targ>::RegexType

The same goes for the internal variables ‘matches’ and ‘iss’

The implementation for the TriStateVal<Tresult> specialization is very similar to the one for Tresult so I am not going to repeat that here.

Evaluation of the second attempt

We got rid of the code duplication (which is good) and managed to have some fun in the process of doing so. At least I did. J

The code still behaves as it did previously, so using the ParseArg function hasn’t changed at all.

We still have the issues about bool and char, but at least we can now solve this problem with a lesser number of specializations (4 instead of 8).

Conclusion

My implementation of ParseArg works good, and is easy to use. I’ve been using it for some time, but used boost for the regexes.

I would like to try and get the TriStateVal<T> and T scenarios to be handled by the same function. That would require providing a >> stream operator for TriStateVal<T> and I don’t know how much work that is.

I feel like giving this a try, but I am not going to wait with publishing this article. For one thing, it is already lengthy enough, but it also won’t matter from the user’s point of view. The usage of ParseArg will not change, so I feel it is perfectly acceptable to leave this optimization for another article.

The code of this article is available for download as always.

And as always, if you have any comments or feedback, please leave a comment at the bottom of the page.

VC2008 feature pack released

vanDooren — Tue, 08 Apr 2008 20:50:00 GMT

http://blogs.msdn.com/vcblog/archive/2008/04/07/visual-c-2008-feature-pack-released.aspx

Finally we get TR1 with VC2008. Get it while it's hot. This allows me to move a codebase to VC2008 and removing the dependency on boost.

Differences between C++ and C#, part 3

vanDooren — Wed, 27 Feb 2008 11:52:00 GMT

One of the things that C# (and VB.NET) programmers have to live with is non-deterministic destruction of their objects.

The garbage collector was introduced in .NET to take care of memory leaks and remove the need for manual memory management. It does this by keeping track of references, and as soon as an object is no longer referenced, it ‘can’ be collected by the garbage collector.

The ‘can’ is important here, because cleanup happens only when the GC thinks is should happen. On systems with lots of memory, and few memory allocations and releases, it is possible that this takes minutes or hours, or even doesn’t happen at all.

This can be a real problem, because now you don’t have memory leaks anymore, but resource leaks.

Suppose your app opens a file for writing, and does not explicitly dispose of the stream reference.

FileStream stream =
new FileStream(Path.GetTempFileName(), FileMode.Open);
StreamWriter writer = new StreamWriter(stream);writer.WriteLine("Hello, world.");

writer.Close();

The file is written and flushed to disk, but as long as the garbage collector does not collect the ‘stream’ object, it will hold an open file handle that will prevent you or anyone else from accessing that file. To overcome this you have to make sure that the file is released. On way of doing that would be like this:

FileStream stream = new FileStream(Path.GetTempFileName(), FileMode.Open);
try
{
StreamWriter writer = new StreamWriter(stream);
writer.WriteLine("Hello, world.");
writer.Close();
}
finally
{
stream.Dispose();

}

You could also use the ‘using’ keyword, but the same holds true. It is easy to make mistakes, and if you have several of such objects in the same scope, then your code will become a convoluted mess of nested try/finally or using constructs.

In C++/CLI, this problem can be solved differently.

In native C++, they way you create an object determines whether it will live on the stack or in the heap. The location implies the way the object will be cleaned up. If it is on the stack, it will be released automatically when it goes out of scope.

C++/CLI does not have the ability to create managed objects on the stack, but it supports stack like semantics. I.e. even if you use the declaration that would put the object in the stack, it is still created in the managed heap. But the syntax and the semantics are stack based.

This means that in the following example, the stream object will automatically be released when it goes out of scope.

FileStream stream(Path::GetTempFileName(), FileMode::Open);
StreamWriter writer(%stream);
writer.WriteLine("Hello, world.");

writer.Close();

And regardless of how many such objects we need to create or use locally, they will all be cleaned up automatically without needing ugly nested structures, and without needing extra lines of code.

Of course, you still have to determine if this is something you want, because sometimes you need objects to live beyond the scope in which they were created. In that case you can use the gcnew operator to use heap semantics.

FileStream^ stream = gcnew FileStream(Path::GetTempFileName(), FileMode::Open);

C++ gives you the ability to choose the best approach, which is nice if you want to write correct code without wanting to increase the number of lines of code, and structural complexity.