That's an amazing article, thanks for sharing, Pat. 👏🏻👏🏻
Unfortunately, I've seen some big companies applying this mess where each application accesses the other application's data directly because it's easier for the developer building a new application since he/she doesn't have to ask another team to expose a (new) API. And I agree with you here, it is not easy to change that in the short-medium term.
By the way, this kind of scenario where everyone puts their hands on everyone's data without a public API to intermediate the integration is what I call "The Orgy of Data" 😬
Although nowadays it is controversial to talk about building APIs to integrate applications through databases, there are many approaches to enforce an API to communicate and integrate via databases, such as:
Thanks for the article, with some clear reccommendations in the conclusion. This was exactly what we were thinking when we created Alan ( https://alan-platform.com ). It makes this the only way of working, taking away the need for discipline. And at the same time makes it very easy and safe to do so. In this way it keeps your application landscape flexible at all times.
It is the greatest of many great tragedies of the world's tech stacks that we are stuck with SQL. Most of the problems you describe are problems with SQL, not with the relational model.
I'm not going to argue one way or the other. IMHO, most of these problems came ironically from databases being so damned good at taking care of data for the app. If IMS or one of the network databases of yore were around and doing excellent archive, operations, etc, people would have thrown too much crap into them.
SQL is wonderful and very annoying. What I love about it is the relational model, not the COBOL-esque syntax. Just being ONE way to do data helped a lot. Then, it made the transition to other models (e.g. key-value) harder.
Most standards have an intrinsic duality where success causes problems.
All in all, I think it's done more good than harm. I can totally respect other opinions, though.
I consider myself a database expert. The relational model is so great it is actually worth putting up with SQL to use it.
Still, aside from the godawful syntax, it has an appalling programming model, rigid storage assumptions (tables are top-level, very heavyweight things), it assumes you want ACID all the time (with all of its costs), you have to do a great many things manually (eg indexing) that ought to be automatic, and on and on.
A key-value store ought to be a trivial type of table, not a whole separate product.
Look at Datalog if you want just a glimpse into the world we ought to be inhabiting.
That's an amazing article, thanks for sharing, Pat. 👏🏻👏🏻
Unfortunately, I've seen some big companies applying this mess where each application accesses the other application's data directly because it's easier for the developer building a new application since he/she doesn't have to ask another team to expose a (new) API. And I agree with you here, it is not easy to change that in the short-medium term.
By the way, this kind of scenario where everyone puts their hands on everyone's data without a public API to intermediate the integration is what I call "The Orgy of Data" 😬
Although nowadays it is controversial to talk about building APIs to integrate applications through databases, there are many approaches to enforce an API to communicate and integrate via databases, such as:
- views
- stored procedures
- tables for integration-only
- events and signals
- grants and schemas
- replication and CDC
But exposing the schema is not one of them 👊🏻
At least a database is structured(ish) data. Too much of my career was weaning 'power users' off spreadsheets
Thanks for the article, with some clear reccommendations in the conclusion. This was exactly what we were thinking when we created Alan ( https://alan-platform.com ). It makes this the only way of working, taking away the need for discipline. And at the same time makes it very easy and safe to do so. In this way it keeps your application landscape flexible at all times.
Great article! Minor point: it's nigh-on, not neigh on.
It is the greatest of many great tragedies of the world's tech stacks that we are stuck with SQL. Most of the problems you describe are problems with SQL, not with the relational model.
I'm not going to argue one way or the other. IMHO, most of these problems came ironically from databases being so damned good at taking care of data for the app. If IMS or one of the network databases of yore were around and doing excellent archive, operations, etc, people would have thrown too much crap into them.
SQL is wonderful and very annoying. What I love about it is the relational model, not the COBOL-esque syntax. Just being ONE way to do data helped a lot. Then, it made the transition to other models (e.g. key-value) harder.
Most standards have an intrinsic duality where success causes problems.
All in all, I think it's done more good than harm. I can totally respect other opinions, though.
I consider myself a database expert. The relational model is so great it is actually worth putting up with SQL to use it.
Still, aside from the godawful syntax, it has an appalling programming model, rigid storage assumptions (tables are top-level, very heavyweight things), it assumes you want ACID all the time (with all of its costs), you have to do a great many things manually (eg indexing) that ought to be automatic, and on and on.
A key-value store ought to be a trivial type of table, not a whole separate product.
Look at Datalog if you want just a glimpse into the world we ought to be inhabiting.
I completely agree.
Still, the world would not be as good (in MY opinion) had we not gotten the consolidation that SQL brought.
Still:
-- I believe every criticism you have listed.
-- I believe that EVERY successful standard impedes innovation, SQL included.
Just sayin... It sucks but the consolidation of the industry around it has been powerful.
I love and encourage the debate and welcome pushback.
Oh, one more thing. The problems you listed (while real) are independent of the concerns I raised in my blog post.
Still, keep pushing on me if you disagree.