Accepting status quo – when intellectual (diminished?) egos discuss the finer points of obsolete data

I got a message today, via FaceBook, that my esteemed x-colleague, Colin Scott (www.abstractcode.com) wants to challenge me on, what he terms, a blog duel.

Rather than using such a harsh and aggressive term, I’d call it a battle between two intellectually challenged egos – or, what our women normally call “discussion”.

Colin states that we should learn to live with the reality of obsolete data. This was in response to me arguing that RSS feeds weren’t necessarily good source of information. Or, more importantly the point i was trying to make, which was that the way we deal with it isn’t good.

Now, before we enter into the debate about data being obsolete purely because time has passed, there’s a few aspects i think needs to be considered first.

  • Would the data be in a state of constant change – not merely inconsistently updated/changed.
    • Is the data we’re observing the distance a person has travelled while he’s in motion, or
    • is the data we’re observing constant enough, within our own acceptable parameters, for it to be measurable
  • Would the data be historical data, which time has no effect on.

I think for some data, you have to accept that it can, or most likely will be obsolete by the time you receive it. For others, i feel that accepting status quo is a reflection of ignorance (or laziness?).

Colin makes a lot of good arguments as to why obsolete data is the reality of the world. You make decisions each day based on data, be it $1,000,000,000 business decisions or whether or not you need to step out on the road when you get to the curb.

However, Colin is off the rails and wrong – as per usual – not because his arguments are invalid or because he’s…well..Colin. No, he’s wrong because i wasn’t speaking about business decisions, nor was I discussing physical mail address changes that can confuse the post office.

I was speaking about the validity of the information you get from RSS feeds and the way our feed readers handle that information – Colin states that we make these decisions based on information – correct we do. However, the feed reader doesn’t give me the option to make a decision.

No, it makes one for me, which inadvertedly leaves me with obsolete data. It does so by design – it doesn’t give me the option of updating the current post OR for me to store the amended post – or even view it. it simply discards it.

Going back to the evaluation we had to do early on, data from an RSS feed is not in a constant state of change. It’s inconsistently updated and as such we can make a decision based on what to do when it’s updated.

Colin seems to think (being that he is, well, Colin, and wrong as usual) we should realise that we make decisions based on knowing that our data is obsolete.

I believe that we should set our acceptable measures and make a decision based on that.

As Colin did, i’ll also shown an example where we make a decision, which will be wrong, based on the fact that we accepted that our data is obsolete.

Say, I’ve read an article from a news agency (on my mobile phone that is, utilising RSS as a means of distribution), which states that there’s a fire 50km from my house – I believe that 50km was indeed a long way away so did not feel that i should do anything about the situation. I mean, 50km is a long distance away and it’ll hardly affect me. However, i decide to check the status of this fire once per hour, because i want to just keep an eye on where it’s at so i can at least get early warning. For 3-4 hours, the distance is the same..50km – suddenly i smell smoke and it’s actually in my backyard (or close enough for me to smell it). The news article has been updated 3-4 times over the last couple of hours, even new items has been published. However, because of the way RSS is dealt with i didn’t get any of the updates.

Would my decision to stay have been the same if the information had been updated with the amended data? Should i accept the data being obsolete if this was what was at stake?

This is where you have to evaluate the criteria for how important the data is – granted, somebody getting caught in a fire would probably put a lot more weight on the importance of the data being correct than somebody who’s just reading some gibberish on a joke web site.

Anyways, the examples Colin displays all validate his perception and statements. However, they have nothing to do with the points i was making.

in Colin’s post he mentions that SOA has advantages in that we’re accepting that the data is not the latest – due to the decoupled nature of services. i absolutely agree and the architecture in itself is meant to support this – but, you’ll always build in the acceptable criteria for when and how exceptions are dealt with. If data is older than X timeframe we request new data or we discard it. Applications has to rely on the data it gets, so we set those criteria. The business will set the acceptance criteria for our application to ensure that they make their decisions based on factual data – how accurate that data is, depends on the business.

We do that with all the decisions we make and the applications we create. We set the acceptance criteria. If you’ve arrived at the corner of the street you don’t just blindly step out on the road because last time you looked, 100 meters back, it was green – no, you look up again. hence we’re making a decision about stepping out onto the road or not, based on the status of the traffic signal. The acceptance there and then is that we absolutely need to see that the signal is green, otherwise chances are we wont cross the road.

I say there’s a big difference in saying that data is obsolete and we should learn to accept it, to how we set these criteria.

In summary:

  • Colin is wrong. this is obvious as he reads things only in black and white, not in reality and perspective.
  • The decision we make, and the realisation we “may” come to, is based on the acceptance criteria we make. Not blindly accepting that data is obsolete just because it is.
  • There is no guarantees in the world Colin, that’s what life is all about.

disclaimer: please note that both Colin and myself are *** stirrers and thoroughly enjoy pestering each other. we have done so for years and will most likely always continue to do so. We’ve known each other for quite a while, and while i’m still better looking than Colin…well, smarter AND better looking, he has age on his side as he’s far younger than i am (see, Colin, i do concede some wins to you). Over to you Colin..

Published Sat, Feb 14 2009 14:58 by Brian Madsen
Filed under:

Comments

# re: Accepting status quo – when intellectual (diminished?) egos discuss the finer points of obsolete data

Sunday, February 15, 2009 6:26 AM by Will Hughes

Interestingly we're struggling with this problem, in a different space, on a rather large project at work.

Specifically: How do we go about informing users about how recent the data is, in order for them to interpret it themselves in an appropriate manner.

Specifically about RSS though - you mention that there's no way of updating the post or storing the ammended post. This seems to be more of an implementation problem - either on the consumer or producer side.

I know Google do update RSS articles with their newer versions. Wordpress (at least my version) also pushes out the latest version of the article whenever checked.

So, as far as I can see there's no technical problem with the format.

Extending it to checking when writing comments, this can be fixed easily too. Most common forum software solved this years ago with a hidden field with the last-updated timestamp of the thread.

Upon submission the forums are checking that value and if it's different, they display a "Hey, that content changed" message letting you have a chance to review your post.  

Yes, this comes with the complication that if you're on particularly active thread, and keep revising your answers you may have to repeat the process several times.  

# re: Accepting status quo – when intellectual (diminished?) egos discuss the finer points of obsolete data

Sunday, February 15, 2009 4:16 PM by Brian Madsen

Hey Will - as Colin states, in general application use, mechanisms would be in place (or need to be put in place) to eliviate this problem - however, as he stresses, this cannot completely accomplish getting rid of the problem.

My idea of this is that you, as a user, would have the option available to you to decide WHEN and HOW this is implemented. Giving you the option of imposing your own acceptable criteria for how to deal with obsolete data.

As you mention, most forum/bulletin board software these days, has found means to ensure that this problem is diminished as much as possible.

Colin mentions mechanisms for dealing with these scenarios - I say that these mechanisms has to be reflections of the acceptable criteria. Eg. When data is newer by x minutes, do Y and when data is newer by x hour(s) do Z.

This is a simple reflection of how business intelligence deals with the two aspects of data:

- data which is updated at an increasingly frequent rate

and

- data which is changing at a consistently low rate.

eg. Slowly changing datasets and fast changing datasets.

Within BI, the business decides how it wishes to deal with the data being obsolete - eg. it reflects this by setting it's acceptable criteria and then rules which deals with what to do.

My argument isn't that data isn't obsolete - it is in many cases the state it is in, how we deal with it differs from person to person - or business to business.

Accepting that data is obsolete, in this example, by RSS feeds displaying "aged" data, and not setting these criteria and giving the user options on how to deal with it, is accepting status quo - what will be is what is, that's life or simply put, that's the way it is.

Will, it's not an uncommon aspect of application or service development - as a matter of fact it's something we deal with on a day to day basis so it's not a new concept.

Why is it then that RSS, or it's implementation or how it is handled, is left so far behind on an issue that's dealt with so very frequent?

as you mention, it could very well be that it's not RSS itself that's the cause of this, but how the readers interpret "existing" data versus "new" data. IMHO, it's a fundamental flaw in ensuring that users has the most relevant and up-to-date information.

I think on the whole that this comes down to complacency - again, accepting status quo, because that's just how it is.

As my original article questioned, was this a flaw in RSS or in the way it's implemented? From what you say, re: Google and Wordpress, it seems that it's not RSS itself, but most likely the readers.

I do believe, as Colin pointed out to me in a later discussion, that it would increase the complexity in which RSS was implemented - but why can't this choice be mine rather than not giving me a choice at all?

# re: Accepting status quo – when intellectual (diminished?) egos discuss the finer points of obsolete data

Monday, February 16, 2009 1:56 AM by Colin Scott

My comments and posts were more addressed at the lack of understanding I've sometimes see that all data has the potential to be out-of-date. Getting hung up on the word obsolete is not entirely helpful here. It's about not assuming perfect or complete knowledge and making decisions anyway, and the ways we protect ourselves from this (like compensation mechanisms).

Part of my objection to Brian's response to my comment in the previous thread was the suggestion that accepting data may be out-of-date is somehow going with the status quo. Frankly I'd love us to be in a situation where it was but I've not seen nearly as widespread an understanding of data currency issues as I'd like (your mileage may vary).

I'm not sure we're really that far apart, although I'm still declaring myself the winner. Especially as Brian is now trying to steal all my points. :)

# re: Accepting status quo – when intellectual (diminished?) egos discuss the finer points of obsolete data

Monday, February 16, 2009 4:42 AM by Brian Madsen

Hola Colin - uhmmm..i haven't changed my point of view at all, i clarified that pretty much in our MSN convo if i recall correctly.

My reference to status quo isn't about accepting data may be out of day, but the nonchalant display i often see which revolves around the mindset that says "well, that's just how things are" or "that's how it is, because that's how it is".

i've never been very good at accepting that things are at a standstill - especially not around technology.

My reference in that earlier comment was towards the fundamental topic's state - namely RSS..

anyways, you can be the winner if you like...i'm still far better looking :)

# re: Accepting status quo – when intellectual (diminished?) egos discuss the finer points of obsolete data

Tuesday, February 17, 2009 12:07 AM by Tiang

A point about the example with the fire. You wouldn't use RSS feeds to get real-time updates on an event. You'ld use Twitter, SMS, or other timely mechanisms.

RSS Feeds are an aggregation of data at a certain point in time. That allows feeds to be "out of date".

# re: Accepting status quo – when intellectual (diminished?) egos discuss the finer points of obsolete data

Tuesday, February 17, 2009 4:20 AM by Brian Madsen

@Tiang,

you're missing the point of that example...

Regardless, you're right, in cases such as that one you wouldn't want to rely on RSS to update data...IF you were aware of the drawbacks of it, otherwise you may actually think that it would be a completely legitimate source of data.

RE: RSS feeds are an aggregation of a certain point in time - i don't disagree, but that's also beside the point. The issue here is that the way RSS is handled doesn't give you an option of choosing if you want your current data to be updated.

hence the nature of my issue is that RSS isn't necessarily good...there's a lot of limitations with it, which could very easily be improved, so that it would move with the times.