Never Trust Vendors or Consultants

especially your database technology vendor

Issues of trust are fundamental to most relationships. Certainly, business relationships are no exception. The discourse usually revolves around who trusts who. With the implication being that if you trust me, you will do what I will advise you.

I never liked this idea. As a client, I always feel that my business decisions should only be based on facts, and nothing but the facts. I don’t care how trusted the vendor, he or she should substantiate their claims.

As a vendor, I understand that this approach requires a certain rigor that is above and beyond what is normally expected, but that’s fine because at BitWise, “above and beyond” is the norm.

So far, simple enough. Things get interesting when I don’t have enough facts. For example, we had an architecture review with a client. They wanted to use a debizium.io product for MySql. Sounds great. What does it do? It’s a CDC tool that is designed to read changes from a binlog and transfer them elsewhere. They created an “out-box pattern” whereby they would write to a specific table just so that debizium would pick it up.

Besides the problem of not using the database correctly (this should have really been written to a queue), we raised the issue of reliability. The CDC tool vendor of course claims that their product is production ready and tested. I know from my experience that a binlog can not always be trusted. It works flawlessly most of the time, but for production we usually need more than that. Moreover, it’s not designed to be a reliable source of information.

All that, however doesn’t constitute a fact based argument against debizium. What could I have done? Unfortunately, nothing. I gave them whatever information I had, and advised them against it, but they used it anyway.

Of course, DBA’s gut feelings should never be ignored. After a month they came back with a data problem caused by a binlog problem interfering with the debizium sync (still investigating the exact root cause).

A takeaway from this? Respecting others sometimes means letting them fail. As much it hurt me to see them make a wrong decision, my respect for their intelligence was more important and I didn’t make a “I’m a DBA! Trust me” scene at that meeting.

At a meeting with a different client, I was able to use this experience to say — look, I don’t have a full proof that this way won’t work, but it’s likely to fail, and in fact, I have a client that didn’t listen to me and had such and such problem.

In case you’re wondering, the original client is fixing the situation. We’re going to go with an intelligent DALC layer that is going to do all this in a reliable and scalable way.