Long ago, at Company S, there was the Comm Engine. This was a powerful piece of software that mediated communication between our system and humans.
It sent a lot of email, both HTML and plain text. Something like 5 million a week just in newsletters, plus who knows how many emails related to product interactions, billing notices, sales campaigns, and so on. It could also send SMS, faxes (lots of faxes), and make phone calls that connected to a TTS/IVR system. Comms could be sent immediately, or scheduled for any arbitrary time in the future.
Emails in particular were enormously sophisticated. The content of a message was produced by rendering a specified template. Data could be passed to the template in the comm record -- the comm engine used database tables as its input -- which was highest performance, but each template was arbitrary code running in our application, and so it could do anything, including look up additional data in our database as it rendered. Not only did this allow us to customize each message to its recipient, but to do so using the most current data possible, since it was directly connected to our single production database which contained all our authoritative customer data.
If you've used a commercial ESP (email service provider), you probably understand how powerful this is. You probably push data to your ESP in a daily batch, or perhaps you update certain data points in real time. However you do it, the data there always gets stale at best and de-synced at worst. And no matter how well you manage it, it will only ever hold a subset of what your system knows about your customers, and juggling the set of fields you upload is a permanent maintenance headache.
We injected our own tracking into these emails. Tracking links are now perfectly common, and no one is surprised to see an email link pointing to Substack, Tinyletterapp, Mailchimp, Marketo, and so on. But way back in the early 2000s, the comm engine was adding URL parameters to links that let us track clicks not just to a template, but to the specific message (which again was a database row) the link was embedded in.
Again if you've used a commercial ESP, and in particular dealt with their analytics data, this ought to blow your mind. While its common to be able to get files of standard email metrics -- sends, opens, clicks, bounces, unsubscribes, spam reports -- that can get you to some useful aggregates, being able to trace the behavior of individual messages is unheard of (outside of transactional email, i.e. Sendgrid). With a commercial ESP you typically can't even connect opens or clicks to a specific send. The comm engine could do not only that, links in messages were themselves database entities so you could look those up and see where they pointed. You could even request a rendered copy of a particular message, even one sent years prior.
No ESP on the market today, that I know of, comes close to offering the features that Company S' comm engine had seventeen years ago. Of course they offer different features that we didn't have, like WYSIWYG template editors, integration with your favorite cloud CRM, stale data, and 6-figure price tags. (Less sarcastically, the commercial ESPs all handle email deliverability for you, which you definitely want, and which Company S handled themselves at great expense.) Tellingly, everyone I know that spent time around the comm engine, and subsequently used commercial ESPs, wishes they had the comm engine. Often they contemplate building their own, but this is invariably shot down as being too costly.
How is this possible? How can there be multiple billion-dollar companies offering email-sending-as-a-service that lack feature parity with an in-house system build by line-of-business Java developers in a flyover state nearly twenty years ago?
A boring part of the answer is that the software market just sucks. Folks building and running software businesses don't really know what their customers want. Finding out is not as simple as "asking them", much as we would like it to be, because customers lie. And even when being truthful, customers often don't really even know what they want, or are mistaken about their own needs. Internal software systems can have these problems too, but they tend to be mitigated somewhat in startups that are trying desperately to survive in tough markets. The developers and the users of the comm engine not only worked under the same roof, but to some extent were the same people, which even further mitigates the difficulty of figuring out the right thing to build.
A slightly less boring part is that the features Company S built into the comm engine were, to a large extent, fairly specific to its idiosyncratic needs. The comm engine could send faxes because Company S' customers wanted faxes. The comm engine had TTS/IVR because smartphones (as we know them) didn't exist yet. The comm engine used JSPs for templating because Company S was a Java shop.
A much less boring part is security. Can you imagine an ESP offering "upload your own code that can do literally anything" as a templating system? Seems unlikely.
But I will assert that the main reason is that the comm engine wasn't subject to the data integration needs that every cloud SaaS product is, and that allowed it to achieve functionality that cloud SaaS cannot match. By this I mean that the comm engine didn't require us to move data to it, because all of our data was always within its reach, nestled safely in our single central database. Any cloud SaaS product, whether it's an ESP, a CRM, a help desk system, a ticket tracker, doesn't have your data in it.
Your data lives in Your System, whatever that is, presumably something home-grown that runs your business and serves its unique needs. Its where users get created when they register, where orders get created when a customer checks out, where subscription records get created when a customer starts the free trial, and so on. Because data must be physically adjacent to a CPU in order to be processed, making the data in Your System available to be used by cloud SaaS software requires transferring a copy from here to there. There is no way around this. It is a fundamental fact of how computers work.
Moving data is expensive and slow! An ESP has to pay to store whatever you upload to them, and will generally place arbitrary limits on the size and complexity of such data. Those uploads will probably hit API servers that impose harsh rate limits (some older ESPs accept file uploads via FTP, which is actually better!). The more data you want to be able to use in your comms, the more you will have to upload, and the slower this will all get.
Even once you've uploaded your data, the ESP will provide only limited capability to use it in message templates. They may offer a standard templating language like FreeMarker or Handlebars, or they may have made up their own. In either case you're stuck with it, and won't be able to extend it or substitute your own.
So the fundamental advantage that the comm engine had was that it was part of our system, which let it leverage the capabilities of that system. All the data in our database and all the functionality in our application code was available to it. And since we were both its designers and its users, we were able to make it do exactly what we needed it to.
It might be possible to match some of this by building software that customers install and run, like in the old days. I have my doubts, because integrating into someone else's system with no a priori knowledge is a daunting task, but it is strictly possible and I can imagine ways that it could work. But a cloud SaaS product could never do so. And since cloud SaaS is how approximately all software is now built, I predict that Company S' legendary comm engine will never be equaled by commercial software.
