Blogs

Smarter Is: Out of Chaos, Order

Looking back at the birth of the modern database

In the beginning, there was the data, and the data was good. But the data was alone, with neither hierarchy nor relation nor object. “Let there be a team,” IBM said, and there was, and it was good. And IBM said, “Go forth, and produceth a system that will shepherd rocket parts, and truck parts, and every other thing that rolleth on the land, or zoometh through the air.”

Well, maybe it didn’t exactly happen that way, but IBM has been involved in some of the biggest database developments of the past 50 years. It all began in the era of punch cards and disk platters. In the 1960s, the space program needed a way to track millions of parts. IBM’s answer was IMS—a joint project with North American Rockwell—and a hierarchical structure that was perfect for representing the huge bills of material required.

“We had a model and a general idea of what we wanted to do,” says Carl Chamberlin, one of the early team managers. “But I had no vision of where this was all going to go,” he says. “We never expected to see multi-terabyte databases, or the phenomenal success of IMS, or that some IMS code from the 1970s would still be running unchanged.”

What was it like to be present at the birth of modern databases? Back then, data wasn’t considered an asset that could be mined—or even reused very much. “A file was created for a particular program and was rarely used for another program,” says Pete Sadler, who first started teaching IMS in 1970 and who developed some features later incorporated into IMS.

Interactivity was also pretty rare. “IMS used to have an interactive query facility; it sort of came and went. There weren’t that many people who thought they’d like to do dynamic queries,” says Sadler. 

And IBM saw that IMS was good. But it lacked a relational partner. Thus IBM created System R, System R begat DB2 and RDBMS, and IBM saw that all was good.

Why was System R chosen as the model for IBM’s first relational database? “System R was fast, at a time when relational database advocates felt very vulnerable to the claim that SQL was never going to be fast enough,” says Michael Blasgen, who did some of the early work on the System R project. 

And new technology and new databases were fruitful and multiplied. Informix and solidDB took form and spread out upon the land, eventually to enter the realm of IBM.

Jonathan Leffler, a longtime Informix developer, remembers how different the technology world was then, and says that relational databases are now in an ideal position to exploit recent hardware improvements. “Now you have eight-core chips, and relational databases can really take advantage of that,” says Leffler.

An important recent development has been databases that are optimized for near-real-time and embedded environments, where microsecond response times and very high availability are required. Ari Valtanen, one of the founders of Solid Information Technology—acquired by IBM in 2008—says that the availability of huge amounts of RAM will further transform databases. 

And the data multiplied on the land, in the sea, and in the air. And the users of the end variety saw that it was good.