Is it possible that even with today's excitement and real achievement in software and technology, especially around mobile, M2M ("machine-to-machine"), the IOT ("Internet of Things") and so-called "big data", just to name a few hot topics du jour, that there is a major roadblock to further easy progress in technology?
After decades of achievement in the development of software technologies and software engineering, the software industry is rightly acknowledged as having contributed enormously to every aspect of business, social and personal life. It is a general belief, fostered by both science and culture that a "long revolution" based on IT will continue on, bringing ever more amazing, delightful and useful innovations.
This expectation of progress can probably be depicted as a linear function with a nice upward slope. While "Whiggish" expectations of continual progress are nice, the reality of software engineering is less rosy. The realization of future progress based on software technology may not be so easily achieved and the immediate future of software development may be disappointing.
Why is there a potential for disappointment?
The current state of software engineering and data management is characterized by what could be called a "semantic ceiling". On the software engineering side, the newest software products and software development are, while often quite wonderful, still rather limited in what they accomplish: mashups, social applications, situational applications, modeling tools, more SOA, point business applications etc. The scope of these new applications is typically either siloed or trivial in some sense.
Especially on the data management side, the growth of data resources has exacerbated the data chaos that confronts both business and individual trying to make use of technology. For this reason, it is not surprising that master data management (MDM) is a hot area in the software business.
The idea of a semantic ceiling is the idea that further progress in software engineering will only be possible with the development and deployment of a new layer of semantic technology.
The most widely known example of this new layer is the "Semantic Web"; less well known is the associated, intense (mostly) academic effort to develop the ideas of ontology for application to software.
Why is there a need for technology of a "semantic layer"? Why is the lack of such a layer a "ceiling" which creates a difficult step function on the road to software progress?
(In the graph at the left, the purple line showing benefits of technology actually declines over time, reflecting your host's suspicion that the situation described in this post is becoming worse.)
A short answer to the question of the semantic ceiling, has three parts: "work", "complexity" and "language".
The argument could be summarized as "software is about work; but the work, including the work of software itself, is becoming increasingly complex and then prohibitively costly; and lastly the the solution to complexity is the evolution of new complexity-management languages."
First, the original purpose of software is assistance with "work". From the first insurance companies which automated claims management to word processing today, software is a force multiplier which enhances human productivity. Software has been very successful in delivering productivity benefits -- although interestingly, it is only in the last decade that economists have conceded that there may in fact have been competitive and/or societal benefits from automation. The fact that there has been a debate is in large part due to the cost of software technology, which is substantially a cost in complexity. So, the question of a "semantic ceiling" is first of all a question of work and the expectation that software will help in the performance of work.
The second part of the question on a semantic ceiling then relates to complexity. Software is not only a "force multiplier", especially in white collar environments, that is to say assisting with the performance of work, but software also contributes its own complexity. And that inherent software complexity is growing, as more and more of human life is dependent on software, as that software must more and more "speak the language of life", or, at least the "language of work". And given that life and work are rich and complex, the contents of the software systems technologies that encompass life and work are likewise rich and complex. For example, a typical corporate ERP software system may deploy 20 or 30 thousand data tables, each table with multiple elements -- and all this is for a rigidly prescribed command and control system deployment in a corporation. Even more dauntingly, many Fortune 1000 corporations have a dozen or more individual ERP systems deployed. The possibility that such organizations, given the state of today's technologies, can act nimbly in a dynamic marketplace, is low. It is not for nothing that acquiring an ERP system has been described as "pouring cement over an organization".
While life and work is rich and complex, current software systems only model life in the most basic of ways. But as software becomes more and more a part of organizational and individual life, demand increases for higher-fidelity modeling of that reality. Current software tools are just that, "software tools", and are not suited for or even intended for these new demands.
One can imagine a new generation of powerful modeling tools that will meet the complexity and coverage requirements of both individuals and organizations. But, these explicitly higher-level modeling technologies do not yet exist as productized, engineered software available for common use. In the meantime, the dragooning of existing software technology in support of new organizational demands produces brittle results, exponentially escalating costs -- and failure. Existing software is technologically incapable of being used for anything other than "pouring more cement".
The new technologies of ontological engineering, coming out of at least a decade or two of exciting academic research, are the answer to the problems of software complexity and the need for higher-domain languages capable of addressing a new generation of organizational requirements.
Semantic Technologies: Investment Opportunity Or Stumbling Block?
In summary, the idea of a semantic ceiling is a reference to a break in further progress in software technological achievement. The ceiling is a joint product of the escalating complexity of software and the richness of needs for which the software is constructed. And the proposed solution to this challenge is an embrace of semantic and ontological technologies. Such an embrace implies new investment beyond existing technology investments, being an entirely new category of software. While failure to make this investment spells decreasing returns to existing investments, justifying investments in powerful but unproven technology will likely be difficult.
This observation about a need for investment in a new category of software technology is not a criticism of existing technology, merely an acknowledgement that the development of a layer of ontology-based software engineering is a precondition for mastering the step function.
There will be many business opportunities for evangelization of semantic technologies and ontological knowledge and systems. Most waves of technology change are disruptive and the introduction of semantic technologies and ontological engineering into mainstream software development will be disruptive. This introduction of new technologies may also be difficult in itself, as semantic and ontological engineering is rather different, and even possibly more difficult, than most existing software engineering. It will be a long revolution indeed.
Draft V 0.7 - December, 2012
Draft V 0.6 - May, 2011