Validation Information in Metadata

Published Tue, Feb 19 2008 8:17

Mike asks:

Just curious if your metadata also contains validation rules or not?  Things like property is required or range of valid values.

It could include them in three possible ways – it currently uses one and I’ve had two others working in the past that may be resurrected.

The metadata that the database inherently knows is automatically transferred - this would be nulls and string length. How well nulls are handled is up to the architecture, but the metadata definitely knows what's nullable.

I've experimented with two additional approaches: using extended properties and parsing the TSQL of check constraints. The first would work for simple ranges and other predictable data sets, but it puts information in an unexpected place. I currently can’t justify it over placing validation in known places in the handcrafted code.

Using check constraints leverages existing information so is a "good" thing. Unfortunately, no one ever seemed to care about the months of work I put into that five years ago so I let it stagnate. Since I know more now, I could resurrect that work, but honestly I don't think I'll get time soon.

The problem is that most people just don't put check constraints in the database very often. I find that unfortunate for many reasons, but it becomes a chicken and the egg problem. People don’t put the constraints in the database because they’ll have to restate them in the business layer for decent usability. This initiative doesn’t get attention to solve that problem, because the check constraints aren’t already there. Perhaps the time is ripe now. I would love to include check constraint based validation in the Open Source version that we plan to start up on Code Plex this week or next (public within thirty days after) – at least a framework for it.

Check constraints are closely related to defaults because both require parsing TSQL. Turns out, over the years folks have been primarily interested in defaults of “now”, new guids, and raw values. Today or Now are pretty easy because it’s just a straight up translation between a SQL function and a .NET function. Any straight up translations like that can be defined in sort of a metametadata (hate that phrase) layer. I handle all three of these scenarios in my metadata extraction tool (a metadata extraction tool will be part of the CodePlex project).

I think validation should be stated in the business layer in rules. I wasn’t doing this five years ago so the whole process of incorporating validation from check constraints will be vastly simpler. Instead of code to code, you need to recognize a category – such as a bound range (the most important) and parse out the bounds into a structure usability by a specific rule. Then another rule is “there’s a check constraint and I think you need to validate based on it, but I can’t write it so you need to.” The architecture could enforce some code being written in response to that rule. To state the change from five years ago, the metadata wouldn’t contain code but the statement of which rule and its parameters.

Validation in triggers would seem, at least to my weak TSQL mind, to be exceedingly difficult.

So, the basic answer to Mike’s question is “some, but not all of the really important scenarios are covered, and I don’t think you’ll ever cover all scenarios”

Filed under:

Comments

# Baxter Lane said on Tuesday, February 19, 2008 9:04 AM

Have you considered applying the validation in reverse? Having the generator write both the validation for the database (check constraints) and the validation for the business rules?

This might only wok well if you are writing a project from scratch.

I have been considering the same thing and have thought about how I can centralize the logic, in the business layer, to provide the same validation features to validators in the UI that the business layer would use. One idea I had would be to place regular expressions in an extended field property on my SQL Server tables as metadata. This would work quite well but I’m not sure it is ideal. While regular expressions are wonderfully powerful I’ve recently heard them compared to “cartoon characters swearing”.

# Kathleen said on Tuesday, February 19, 2008 11:04 AM

In the long run, yes. A model is the place to start. It's just a hard jump to make. It would of course make the logic very simple. But are you ready to convince your DBA not to changt the database?

You're on the list of folks working toward the open source tool, and one of the other guys on there has already done work on model first devleopment so I think this will resurface. I think it would be great to have that as a realistic model, but its hard, and harder to convince people to do it. Entity Frameworks has announced that it will not make their V1.

# AllanN said on Tuesday, February 19, 2008 4:24 PM

Validation Rules can and should be broken into two seperate groups as stated by Baxter above, this is because the database contains constraints which should not be confused with "Business Rules".  Constraints can't and shouldn't be overwritten by user descisins as they will break the application and/or corrupt data.  Business Rules on the other hand can and normally change on a very regular basis according the the business (e.g. tax rates and who they apply to, and when.  whether to apply a surcharge to a rental customer based on age, etc), hard-coding these rules into an application template is wrong.  However, I am not stating they shouldn't be in a rules template, but that's a different story.

I pose this question "How do you define 'If obj1.field1 > obj2.field1 then apply obj3.field1 to obj4.field1' to an application template?  It would also mean the business that put that rule in place can not change it without recompiling the application code.  

I think this question belongs in a rules engine blog and not the code gen blog as they are VERY different beasts.  

I am very keen to hear thoughts on this as there are now a few choices;

1.  Creating Workflow objects from templates, this is my prefered option, with a designer for businesses allowing them to change the rules,

2.  Implement a third party rules engine like ILOG, Visual Rules or BRE.

3.  For simple csla projects, expose hand-crafted class for rules, derived from XML based file.

There are many others, but you get the idea.

My rule, "Rule are not developer code, they belong to the business, controlled by the business, written by the business".

# Kathleen said on Thursday, February 21, 2008 2:02 PM

Alan,

I agree that rules engines are good for certain types of validation, but we're constantly in a dance between what our applications are enforcing on the business and those the business should be able to configure. Whether a value is required is absolutely a business rule, but one we'll rarely allow the business to control. For certain types of data, I also wouldn't let anyone enter negative numbers, dates in the last century, etc.

We engineer businesses, and have certain responsibilities in response to that.

I think rules engines are great where they apply. But for most businesses, adding a rules engine just to manage valid ranges seems overkill and many organizations don't have anyone who cares enough to keep these configurations accurate and understand the implications of changing them. Users have a very different perspective - and I do when I'm using other programs. How do I do what I want to do now, often not what I really want to do. So rules engines have a lot of potential for chaos if they are used for validation other than in organization that understands how and why its running on rules.

For simple rules - such as ranges - my hope is we'll have an engine that parses simple check constraints, and encourage people to record non-complex contraints their. Regardless of whether its in the database, I'd like to see all rules in metadata or runtime rules engines, but I'm not sure 100% is achievable there. That makes them very easy to update, although it does require redployment.

And much as I like workflow, its a darn shame the rules engine is locked inside.

(just random thoughts from a tired person)

Kathleen

Leave a Comment

(required) 
(required) 
(optional)
(required) 
If you can't read this number refresh your screen
Enter the numbers above: