How did places like Bell Labs know how to ask the right questions?

Alternative Title: Why all applied research organizations should hire a Bell-style systems engineer

From Good Science Project fellow Eric Gilliam:

Many new science orgs are looking to pursue research that has the positive aspects of both “applied” research and “basic” research. To me, this is a very reasonable approach. After all, the “applied vs. basic research” distinction has always been a rather arbitrary one.

Some research projects feel like they are squarely in one bucket or the other, but it’s not always that clear. Applied research is meant to be research with immediate applications in mind. But, of course, applied research could stumble upon something that leads to a fundamental insight. Basic research is meant to be curiosity-driven research without immediate applications in mind. But, of course, it could quickly lead to a killer application.

In the universe of possible courses of research, there exist many questions that, in the end, will satisfy the spirit of both applied and basic research.

The natural follow-up question is: is finding this subset of golden problems really feasible?

One’s knee-jerk reaction might be that it is not replicable in any kind of systematic way; it is a matter of unreliable personal taste. The history tells a different story. The mid-20th century’s great American R&D labs show us that selecting profitable courses of research that satisfy the spirit of basic research has been done at a high level within large research organizations over the course of several decades.

In this piece, I dive into exactly how Bell Labs ensured that their researchers were working on the right problems. This piece will be the first in a series examining what modern applied research orgs can learn from the great dragons of industrial R&D — places like Bell Labs, GE Research Laboratory, and DuPont’s research department.

Institutions like these not only had Nobel prizes to their names, but each — even though they’ve diminished for various reasons — was quite profitable too. They had their differences, but they all stumbled upon many aspects of managing their research operations that were rather similar. The managers of these organizations — and other researchers at the time — often felt the management decisions they made were common sense rather than some great discovery.

However, these common-sense decisions are things we often don’t do today – but almost certainly should.

Image pulled from a 1922 issue of Bell Telephone Magazine. The image, portraying the switching development department, was used in an article explaining how “systems engineering has played a dominant part in every aspect of Bell Laboratories work” and why it was no surprise that the approach worked equally well in facilitating Bell Lab’s successful World War I work.

---

Bell Labs has become legendary in many tech circles. It’s no secret why. Famous ideas and technology like information theory, communications satellites, solar batteries, transistors, and countless other communications-related innovations trace their origins back to Bell Labs in one way or another.

Idolizing Bell Labs for its outcomes is very fair because its outcomes were extraordinary. Many, in recent years, have also begun to idolize Bell Labs’ for its processes. And there’s nothing conceptually wrong with that. If a place has consistently fantastic outcomes and seems to have some secret sauce that is super-additive to the productivity of its researchers, why would we not seek to replicate it?

We should. The issue is that many who idolize Bell’s processes seem to fundamentally misunderstand how the operation worked. The most notable misconception is that many put Bell forward as the poster child of how idle curiosity and the purest kind of basic research can have a role in industry. Looking at the historical sources, that’s not exactly an accurate takeaway.

Bell did a lot of astonishing basic research. But its research, while “basic” for an applied R&D lab, was not nearly as free as many imagine — or, rather, it embodied a different kind of freedom. Equating the freedom of a Fine Hall mathematician at the Princeton Institute for Advanced Study and a physicist at Bell Labs in the 1950s is not an accurate way of looking at it.

Bell was an industrial R&D lab. To an industrial R&D lab, the mission is everything. Frank Jewett, the founding Director of Bell Labs, once said that his new industrial R&D lab was to be:

An instrument capable of avoiding many of the mistakes of a blind cut-and-try experimentation. It is likewise an instrument which can bring to bear an aggregate of creative force on any particular problem which is infinitely greater than any force which can be conceived of as residing in the intellectual capacity of an individual.

This focus on applications leading the research is not one that faded over the course of Bell Labs’ lifetime.

John Pierce — whose 35-year career (1936-1971) at Bell Labs as researcher and manager encompassed most of Bell Labs’ existence — said this of what made Bell Labs a success:

Someone depended on them for something, and was anxious to get it. They were really needed, and they rose to the need.

The people who see Bell Labs as a bastion of freedom in private sector research are not entirely mistaken. Bell was pretty damn free for a private-sector lab. It’s just that there was a balancing act.

Jim Fisk was one of the “Young Turks” at Bell Labs — along with Pierce — who helped shepherd in its famous balance of deep research with careful problem selection. He said the following of Bell’s philosophy on problem selection when he was managing Bell Labs:

Our fundamental belief is that there is no difference between good science and good science relevant to our business. Among a thousand scientific problems, a hundred or so will be interesting, but only one or two will be truly rewarding — both to the world of science and to us. What we try to provide is the atmosphere that will make selecting the one or two in a thousand a matter of individual responsibility and essentially automatic.

This is not Fisk saying that the only relevant problems in electrical communication were those that served Bell’s business interests. But it was him saying that:

• Many scientific problems are kind of a bore or derivative. He ballparked it arbitrarily at 90%.

• Some minority of problems are interesting. He ballparked it arbitrarily at 10%.

• 1%-2% of the interesting problems — that is .1%-.2% of the total problems — would turn out to be worthwhile and relevant to Bell’s work.

The picture this paints of Bell’s preferences for its basic researchers is twofold:

1. The universe of possible problems is very large. Bell would like its more fundamental researchers to feel free to work on interesting ones.

2. Among those interesting problems, Bell Labs management would implement systems to make sure researchers identified problems that had a high probability of turning into profitable answers for the Bell Telephone system.

Over the years, Bell Labs management developed a small but coherent set of constraints and rules of thumb to ensure that its researchers internalized that, as Jim Fisk put it, it was “a matter of individual responsibility” to choose the right problems and that Bell Lab’s success in doing this at scale, across thousands of individuals, was “essentially automatic.”

Bell Labs research problem selection: rules of thumb, systems, and constraints

The majority of Bell Labs was made up of applied researchers, development engineers, and other staff — not basic researchers. And these groups usually had a normal boss and projects assigned to them – while Bell’s basic researchers did not. Nevertheless, even though the basic researchers did not have bosses in the traditional sense, they were still nudged to Bell-relevant problems in various ways.

The three key ways Bell Labs nudged its basic researchers toward the right problems were:

1. Granting researchers what I’ll call a “long leash, but a narrow fence” in which to conduct their explorations.

2. Facilitating very regular interactions between the basic researchers and Bell’s fundamental development researchers, engineers, manufacturing facilities, and implementation staff.

3. To top it off, Bell had a corps of what they called systems engineers who ensured that the integration of its best researchers and most pressing problems was not left to chance.

Let’s explore each of these, in turn.

1) Long leash, narrow fence

Bell didn’t exactly tell its basic researchers what they could and could not work on. Not usually at least. A basic researcher’s boss was more of a mentor or advisor than an actual boss.

(Note: the basic researchers made up anywhere from 7% to 18% of Bell Lab's headcount depending on the year and which Young Turk you quote.)

These individuals were guided toward the right problems in other ways. Firstly, it was made clear that the projects should have some obvious bearing on the Bell system and future business. And one was allowed to roam around, so to speak, for a bit before working out exactly what they would be spending their time on, but they should be looking to spend their time on something quite relevant to the business.

The following excerpt fromJohn Pierce’s oral history briefly describes his reflection on his time immediately after joining Bell. He had a particularly high level of freedom in feeling his way around for work, but still found his way into the Bell Labs groove all the same:

Pierce: I was told to do research on vacuum tubes. People sort of just left me alone. They did suggest that I go and see Philo Farnsworth, who was working on electron multipliers and television pick-up tubes, but I was left pretty much to myself. This was very, very confusing to me. I didn’t know what to do.

Interviewer: Were you doing it alone?

Pierce: Yes

Interviewer: Did they say, “So-and-so has been doing this and this is where he left off”?

Pierce: No. I was just supposed to plan something to do and do it. I think that is close to cruel and unusual punishment.

Pierce, who was giving this interview after his retirement from Bell Labs while he was at CalTech, then continues his reflection:

Pierce: Too much freedom is horrible. It’s like telling a young child, “Do whatever you want to.” You’ve heard this story. There are various outcomes. One is, “Do I have to do what I want to?” Complete freedom is not very helpful to a person who is inexperienced in the world. It’s certainly bad to be directed to do things very, very narrowly and with no freedom. It’s my guess that for every person who needs more freedom, there are ten people who need more help in finding their way.

Interviewer: So, did they tell you why they wanted the vacuum tubes, when you started off?

Pierce: Not really. I found out some way, inadvertently. Some people were working on electron multipliers, and I made some improvements on them. It became clear that people needed better vacuum tubes for building negative feedback amplifiers, and I worked on that. I don’t think I was told this formally; I just found out by talking to people. Then, as the war approached and we got into war, it became apparent that microwave radar was very, very important, and I worked on tubes for radar. It was a process of osmosis rather than direction that led me into these things, as I remember it.

This Pierce story is an example of things working exactly as they should. An extremely talented young researcher with a background obviously relevant to Bell — multiple electrical engineering degrees from CalTech — came to understand exactly what development work was ongoing at Labs, what it looked like for a basic researcher to be useful to that work, and came up with a course of work to suit those needs.

Morry Tannenbaum, a long-time Bell Labs chemist, famously described this patented level of freedom as “circumscribed freedom.”

Pierce’s story leads us into the second way in which Bell Labs nudged its basic researchers toward the right problems.

2) Frequent interactions with Bell’s development researchers, engineers, manufacturing facilities, and implementation staff

Relationships with the folks who might eventually deploy your research – from those who modified cutting-edge engineering equipment to those that worked in Bell’s factories – ensured researchers were hyper-aware of the problems happening throughout Bell’s massive operation.

The one hard and fast rule Labs did seem to have was that you could not say no to any request for help from any of the applied folks — or other researchers for that matter. Your day-to-day tasks often pertained to your own courses of research, but one’s role as in-house expertise was equally important. These consultations, unsurprisingly, led to all sorts of new ideas for basic research projects.

Another excerpt in Pierce’s oral history is just one of many data points that speaks to the importance of this relationship:

I remember that during the war we saw a good deal of people from Western Electric [Bell’s manufacturing arm], who were going to manufacture the things that we devised. Because all of these people were engaged in telephony, or during the war because they were all engaged in radar and other military things, you got to talk to people who were engaged in the operation of things, who were engaged in the manufacture of things, and you got a picture of the rest of the world which certainly influenced what research you did.

He continues, diving into how this type of interaction was a natural partner to great basic researchers:

I can understand a university, which does teaching and research. But the idea of a research institute without ties to either teaching or to manufacturing or operational organization seems a terribly sterile idea. You see that in the Soviet Union; there’s a lot of good activity that never results in anything. When they want to build automobiles, they hire Fiat to build an automobile plant, instead of relying on what they have learned.

(Note: Pierce also believed the university model of research to be an inferior model — for him at least — for reasons I’ll discuss later.)

And, since Bell, as a business and research operation, was far too vast and complex to rely on serendipity to match researchers and problems, it had an entire class of engineers dedicated to ensuring problems and solutions found each other.

3) Bell systems engineers tied this whole process together

Systems engineers – 10% of Bell Labs’ total headcount – were usually technically-trained individuals who spent all of their time, as Jon Gertner put it, keeping:

One eye on the reservoir of new knowledge and another on the existing phone system and analyzed how to integrate the two. In other words, the systems engineers considered whether new applications were possible, plausible, necessary, and economical.

The existence of systems engineers was Bell acknowledging that a culture of openness and helpfulness was just not sufficient when it came to finding the best problems possible. Of course, implementation/manufacturing/applied research staff often can identify which of their problems are ripe for the research team’s eyes, but a lot of the time they can’t. And yeah, sometimes researchers can come up with great research questions in their area that are ripe for helping a specific group’s work, but a lot of the time they don’t do a great job.

Even if a basic researcher does find a good applied problem, who’s to say that the problem is the best use of their time? There is almost surely a better problem out there. That’s life. The question is, can someone reliably find it?

If a systems engineer does their job well, they can.

Mervin Kelly, long-time Bell Labs manager, described the background of his systems engineers as follows:

[Systems Engineering] staff members must supply a proper blending of competence and background in each of the three areas that it contacts: research and fundamental development, specific systems and facilities development, and operations. It is, therefore, largely made up of men drawn from these areas who have exhibited unusual talents in analysis and the objectivity so essential to their appraisal responsibility.

Of course, there is a tradeoff here. Instead of using a systems engineer’s skills for normal scientific research or engineering tasks, these individuals were doing other kinds of work. That’s not a small tradeoff. At points, Bell’s systems engineering team was about the same size as, maybe larger than, Bell’s basic research group. But, in a large and complex organization like Bell, this tradeoff was well worth it.

The systems engineers made it their business to be extremely aware of the happenings of the research portions of Bell Labs as well as the minutiae of the industrial portions of Bell Telephone. This included details like:

• Manufacturing processes to produce electrical parts

• Which materials or parts tended to degrade and needed to be replaced in the telephone system

• The cost and time of various repairs and maintenance

• What portions of Bell’s service were currently bottlenecked by specific technical problems

Of course, no individual systems engineer could know everything and everyone at Bell. But the department as a whole attempted to account for everything, ensuring as little as possible fell through the cracks. And this process worked both ways. These systems engineers, in addition to bringing problems to researchers, also found ways to deploy researchers’ findings in ways that could solve problems in the field or improve Bell’s products and processes.

Mundane systems engineering problem spotting happens when a systems engineer identifies that Bell is spending the equivalent of $1 billion yearly on telephone wire maintenance in certain climates. This engineer can then notify the metallurgy researchers that this problem exists and is begging to be solved.

One Bell Labs executive, a former chemist, loved to point out that a synthetic plastic created by Bell chemists to replace the existing telephone cable sheathing saved Bell “more than the total research budget of Bell Labs for the decade in which the innovation was worked out.” This was an operationally boring problem that could have been hidden in some maintenance budget, away from the eyes of normal researchers. A system engineer exists to identify opportunities just like that.

The ROI of an applied research operation can obviously go up if researchers have problems like this regularly brought to them. And, it should be remembered, these problems were being brought to researchers not just as a veritable gold mine, but with many research-relevant constraints worked out from the beginning. It was the job of the systems researchers to work out as much of this as possible beforehand.

Mervin Kelly believed that the Bell’s continuity was what made it special. If the first two rules of thumb established the continuity, the systems engineers were what ensured it. Kelly spoke about the importance everything connecting the two endpoints of manufacture and basic research in a speech to the Royal Society, saying:

There has been so much emphasis on industrial research and mass-production methods in my country, that even our well-informed public is not sufficiently aware of the necessary and most important chain of events that lies between the initial step of basic research and the terminal operation of manufacture. In order to stress the continuity of procedures from research to engineering of product into manufacture and to emphasize their real unity, I speak of them as the single entity ‘organized creative technology.’

(Please see the bonus piece to learn more about this speech.)

Systems engineers did not technically do anything that you wouldn’t hope would happen with a culture of collaboration, but they were the people that made finding the “one or two in a thousand” problems relevant to Bell and converting them into successful business applications “essentially automatic.”

A Bell systems engineering sketch of Bell’s experimental rollout of a mobile telephone system in Chicago

Freedom comes in many forms

These rules of thumb are contrary to what many believe of Bell Labs’s culture, but most accounts I’ve read from key Labs members point to these methods as a secret sauce that made Bell Labs effective.

The famous stories of Claude Shannon frittering away his days at Bell Labs is a special case. While most researchers did not have the kinds of freedoms Shannon — or many university professors at the time — had, they were usually happy with the tradeoffs. In other ways, many felt life at Bell Labs had a different kind of freedom.

Claude Shannon’s frittering

Stories of Claude Shannon frittering away his time playing games in the Bell lunchroom have become legendary. And I get why. The stories of Shannon building chess machines, juggling, and riding around on unicycles or pogo sticks are great stories. But these stories were not the norm at Bell Labs.

The bulk of these Shannon stories, at least the most “fritter-y” ones, come from after he became a celebrity in the communications world with his discovery — or “invention” depending on your philosophical view of mathematics — of information theory.

On the heels of that discovery, instead of some big financial reward for Shannon, Bell Labs management seems to have paid Shannon with a kind of emeritus status in which they gave the genius the freedom to do whatever he wished. This came with freedom to do unique things like unicycle down hallways or work with the door to his office closed.

Prior to this emeritus lifestyle, Shannon had lived a much more applied, standard life at Labs in which he was often roped into advising the applied folks on their issues and undertook more pressing courses of basic research — fire control and cryptography for the war effort, the mathematics of carrying calls along wires more efficiently, etc.

(Also, it’s worth noting, he used this emeritus-style freedom to accomplish little-to-nothing either for Bell Labs or his personal research agenda in comparison to his earlier life when he was a normal Bell Labs employee or a graduate student on Vannevar Bush’s differential analyzer project.)

In deciding to eventually leave Bell Labs for MIT in 1956, Shannon wrote the following in a letter to his supervisor, Hendrik Bode, on how he was spending his time at Labs:

It always seemed to me that the freedom I took [at the Labs] was something of a special favor.

He knew that the way he was spending his time at Bell Labs was not in line with its culture. At a university, he felt the freedom he took in terms of focus and hours of work would be less unusual.

Claude Shannon being Claude Shannon

Did the university-trained researchers yearn for the freedom of the university?

The lifestyle of a researcher at Bell Labs, on the face of it, does not seem to have the level of personal freedom university professors had.

Some researchers, like Shannon, did leave to work at universities. Their research often became smaller-scale in terms of resources, but they were more free to do as they wished. A Bell researcher leaving to join a university often viewed it as a lateral move. Shannon wrote in his letter to Bode:

With regard to personnel, I feel Bell Labs is at least equal in caliber to the general level in academic circles. In some of their specialties Bell Labs is certainly stronger.

University life surely had freedoms that Bell didn’t, but most Bell researchers seemed rather content with the tradeoffs.

Jon Gertner’s book has a great excerpt that recounts what John Pierce thought of the freedoms of his post-Bell home, CalTech, compared to his life at Bell. The excerpt helps one see how, in its own way, the circumscribed freedom of a researcher at Bell was much freer than a professor’s — even in a less bureaucratic era of university life. Gertner writes:

Pierce went back home to Caltech. For six years he had been doing research and advising graduate students, but he was finding the adjustment difficult. At Bell Labs he had spent his days doing whatever suited him. The brunt of his management work there had consisted of dropping in, unannounced, on colleagues in their labs to ask how work was progressing. But at Caltech he had to give lectures at a prearranged time and then had to spend hours explaining complex ideas to grad students. At Bell Labs, as he recalled it, the same conversation with his colleagues would usually take minutes. (Whether his colleagues actually understood his explanations, or whether he simply walked away before he could field their questions, was a matter for debate.) “I didn’t adapt well to Cal Tech,” he later admitted. “Not that there was anything wrong. For years and years I’d had it too easy. There were very few times when it mattered where I was. I had very few obligations to be at a particular place at a particular time to do a particular thing at Bell Labs.” Pierce obviously seemed to favor the Bell Labs arrangement. As he saw it, the work at the Labs was vital; it was required to improve the network. “People cared about everything,” he said of colleagues there. On the contrary, he noted, in the university “no one can tell a professor what to do, on the one hand. But in any deep sense, nobody cares what he’s doing, either.”

To Pierce, being genuinely needed was its own kind of freedom.

Karl Compton, who would eventually become President of MIT, wrote a Science article in 1927 dedicated to all the things a university physics department could learn from how industry R&D labs worked at the time. Some of his reasons for supporting this directly address Pierce’s “nobody ever actually depends on you for anything” conundrum. Compton writes:

Much can…be done to promote cooperation and coordination through actual methods of organization. This has been strikingly demonstrated in some of the big industrial research laboratories, from which the output has greatly exceeded the individual capacities of the research workers and has been achieved only by coordination of effort.

To Compton, the project coordination of places like Bell Labs or GE Research with a clear and limited set of goals — the narrow fence we speak of — was super-additive. The minds and hands, in this setting, added up to far more than they would’ve in a university setting.

It should also be noted that when Compton did eventually take over as the head of a physics department — at Princeton — he was not able to implement any of these lessons. I’m not even sure he tried. Then, as now, changing the structure of an old university was probably a non-starter.

Luckily, newer science organizations like the ones being started today are not so tradition-laden.

John Pierce, in high school, attempting to build his own glider (via CalTech archives)

Teenage John Pierce mid-flight testing one of his gliders. (via CalTech archives)

The mobile phone system: a case study in phenomenal systems engineering

Before concluding, I’d like to paint a picture of what fantastic systems engineering work can look like.

To some, the concept of systems engineers keeping one eye on the reservoir of new knowledge and the other on the details of the phone system does not leave a lot of room for personal glory. Those who feel this way might liken systems engineers to “system quarterbacks” — a mostly derisive word for American football quarterbacks who simply try to facilitate the careful running of the offense instead of attempting to make any big plays themselves.

I don’t think that’s accurate. Great systems engineering work, like great scientific research, has an element of glamor to it. The story of how a trio of Bell systems engineers helped make the mobile phone system a reality should help you see why.

In January 1966, there were rumors that the FCC was thinking about granting Bell Labs access to a larger portion of the limited radio spectrum — a finite resource that the US government decides how to allocate. The range of spectrum being allocated — which twenty years earlier had been allocated to television broadcasters — could be used to make the dream of widespread telephony a reality…probably…somehow.

There were many open questions, but Bell had a kernel of an idea on how to do this. The writeup of the idea was submitted to the FCC by Bell engineers two decades prior. Dick Frenkiel and Phil Porter now found themselves in charge of making the idea a reality. Frenkiel and Porter were both systems engineers at Bell’s rural Holmdel outpost — Frenkiel a mechanical engineer by training and Porter an electrical engineer with a master's in physics.

This was far more exciting work than their previous assignments — at least for Frenkiel who was previously working on machines that could read off pre-recorded messages such as the day’s date. The duo excitedly took to the work.

To start, the major question was, “What would a major course of development even look like for this primitive technology?”

Luckily, they were not starting from scratch. The first step was to dust off Doug Ring and Rae Young’s short 1947 memo to the FCC — two decades before — in which they proposed a non-obvious, very efficient (albeit hypothetical) system in which Bell could use the radio spectrum. Instead of providing coverage to some large circle of users with a cell antenna at the center of it, Ring and Young proposed Bell lay out the system as a honeycomb of hexagons with small antennas at the point of each hexagon and neighboring hexagons could use different frequencies. This would make the limited spectrum allocated to Bell go much further than it otherwise could.

To give the reader an idea, the following images were pulled from Ring and Young’s initial 1947 report. These images show what the layout would look like if the FCC granted Bell either three or four frequencies.

I was not able to figure out if Ring and Young were systems engineers or simply engineers doing systems engineering style work. Regardless, it was fantastic work and a great start to the project. But the most impressive aspects of the project, in my opinion, were almost entirely ahead of this point. After all, the concept of a hexagonal honeycomb is not unfamiliar to many engineers as an efficient way to cover 100% of a space with a circle-like shape that still has vertices.

The honeycomb idea was still a great idea, it just wasn’t going to win anybody any awards on its own. It was the fantastic planning, troubleshooting, and engineering development work of Bell, all started by these two systems engineers, that would turn this idea from a document of eight short pages plus an appendix into a mass-scale engineering reality.

Gertner writes on the (varied) early work of Frenkiel and Porter on this project:

Neither Frenkiel nor Porter knew precisely how this would be achieved. “It was just two of us,” Frenkiel says. “Nothing important.”

They spent most of 1966 working on the problem — or rather, the problems. The two men covered the walls of their offices with maps and climbed on ladders in various parts of the country to count hills. There were thousands of questions they would need to answer eventually. Many of these were extremely technical, regarding reception and transmission. They talked about signal strength and interference and channel width. They knew every cell would need to be served by what they called “base station” antennas. These antennas would (1) transmit and receive the signals from the mobile phones and (2) feed those signals, by cable, into a switching center that was connected to the nationwide Bell System. Still, several big conceptual problems stood out.

The first was, How large should a hexagonal cell be? Base station antennas would be expensive. How few could they install and still have a high-functioning system?

The second was, How could you “split” a cell? The system would almost certainly start with just a few users—meaning big cells. But as the number of users grew, those cells would subdivide to accommodate the traffic. And more, smaller cells would require more base stations. What was the best and cheapest way?

The third was, How would you “hand off” a call from one cell to another? It had never been done. But it would be the system’s essential characteristic. As a mobile telephone user moved around, how could you switch the call from one antenna to another — from cell to cell, in other words — without causing great distraction to the caller?

Question by question, they persisted.

A year or so after this work began a third systems engineer, Joel Engel joined the project. Engel — who had a Ph.D. in electrical engineering — was currently assigned to a project on the Bell Boy — a kind of beeper — and was excited to use his spare time on this project because the beeper was largely uninteresting to him. He noted that to get ahead at Bell Labs, “you were supposed to work on more than you were asked to work on.” Still a bit of a newcomer to Bell Labs, he was right on this point. Mervin Kelly used to often tell new hires at Labs, “You get paid for the seven and a half hours a day you put in here, but you get your raises and promotions on what you do in the other sixteen and a half hours.” 

So, Engel joined the duo. The now-trio would huddle into conference rooms to draw hexagons on the blackboard and figure out how the technical pieces of this thing could work.

They were neither true engineers nor business visionaries. The three would surely admit to this second point. They might be more reluctant on the first point — but the proper development engineers would sometimes mention this offhand. The three did not see the true scope of what their project could become. Engel once noted:

We were not visionaries,” Engel says of the early cellular meetings. “We were techies. If there was a vision it was primarily as a business service. Real estate agents. Doctors who made house calls.

Even as a specialized service for particular businesses, the economics worked. They’d done the math. They’d worked that out along with approximate answers to hundreds of technical questions that needed to be thought through before significant engineering time and resources should be invested in developing the project.

The trio was young and unafraid to work hard, diving into the open-ended, behemoth of a project. The trio’s in-depth planning leveraged:

1. Heavy fieldwork — to understand issues involving the terrain and weather

2. The more conceptual side of their degrees — to understand and extend the initial hexagonal idea

3. (Most of all) Their knowledge of recent developments in various engineering fields — to facilitate the storage and passing of signals

Working out a high-level, implementable, profitable system that could locate a user moving through a honeycomb cell, monitor the strength of the call, and pass the call between channels and towers was the job of the systems engineers. The task required deep scientific, engineering, and operational knowledge of Bell’s installation capacities.

In the end, the trio successfully navigated all of these difficulties. By any measurement, their individual effects on the massive field of mobile telephony are giant — even if their names are not.

Of course, the trio had their limitations. The win was a team win and required the skills of far more than just those three.

While the project did not require any brand-new, Nobel-level academic accomplishments, it did require hundreds of engineering innovations and improvements in the understanding of many pieces of technology. That is where the great development engineers at Bell came in — people like Bill Jakes and Gerry DiPiazza. These were some of “the guys who made cellular real,” as Frenkiel once said.

Jakes was the lead engineer on the fundamental development end of things — this group often carried out less open-ended research projects on things like engineering equipment. Jakes was known to lead crews of engineers out, piling into vans with recording equipment and headphones, to study the effects of forces like obstructions and distances on transmission and reception. They drove thousands of miles, over many months, working through problem after problem related to things like why some particular wave or piece of equipment behaved a certain way in a certain kind of terrain.

As was always the case, the systems engineers were there every step of the way helping coordinate this development work. This was exceedingly necessary as it was not only people like Jakes carrying out this sort of work. It grew into a very large operation — as projects like this always do — across many Bell Labs research locations and teams.

(Check out the bonus piece if you’d like to know more about how this work was coordinated)

One of these research teams operating in parallel, as an example, was led by another great Bell development engineer, Gerry DiPiazza — who I’ve seen called an “engineer’s engineer” in several places. His group was carrying out similar work learning how to build better signal hardware in a stripped trailer home outside of Philadelphia.

DiPiazza reflected on his team and what they were working on when they took their midnight research road trips — when they could test signal strengths and tinker with hardware in a more “noiseless” environment:

You had to find out, what is the noise level in a suburban environment? How far would a signal go if the antenna was at ten feet, twenty feet, fifty feet? Would it go one mile, two miles, four miles? How many antennas do you need? How do you build an antenna? What are you going to put the antenna on?

There were a haunting number of problems like the ones DiPiazza describes, worked through by many Bell engineers over many months. And, of course, there’s never any substitute for great fundamental development and engineering work. But, thanks to the likes of Frenkiel, Porter, and Engel, one could be confident that all this money and effort was being spent on the right sub-problems. More importantly, one could be confident that no wicked, unsolvable problems seemed to be awaiting them.

The whole thing had been planned exceedingly diligently. The planners were engineers and Ph.D.s. They already had deep familiarity with the phone system and all its details. They’d brought all the relevant scientific questions to top research minds, consulted engineers who work with relevant tech every day, and coordinated with Bell field staff. This is what they did. When you’re allocating this much money and research time to something, the confidence that systems engineers can provide is invaluable.

They kept a research organization like Bell Labs working on the best problems possible and helped ensure that as few resources as possible were wasted on research that would not turn out to be usable.

The tools of the trade. Equipment testing vans and trailers used by Bell’s development researchers and engineers.

Applied science orgs need a systems engineer

In applied scientific work, great systems engineering work can rival the importance of great research minds.

It was not only Bell that stumbled upon a position like systems engineer to help facilitate its operation. GE Research and Dupont’s research arm — also applied science orgs that left some room for idle curiosity — had similar positions in their own operations that went by other names.

Frankly, it just makes sense. Any new science org attempting to pair usable applied science with some kind of fundamental research should think long and hard before they decide that a full-time systems engineer is not for them.

I’m not saying that, like Bell, a young applied science org needs as many systems engineers as basic researchers. I imagine the ratio of systems engineers to basic researchers should grow as the complexity of either the research operation or the system in which discoveries are going to be applied grows. No new science org has a research operation as large and varied as the mature Bell Labs. So, naturally, the ratio should be smaller.

But many new science orgs do hope to plug into complex systems and operations. Several of the orgs I’ve spoken with have problem scopes that, more or less, pertain to the needs/problems/holes present in all of life sciences research.

Finding a problem in these systems is not so hard for those familiar with the systems. That’s why many researchers and engineers do not feel the need to bring in help. But finding a set of good problems is not finding the best problems. Finding the best problems is a profession in and of itself. A systems engineer is worth it when, under the right scrutiny, it might turn out that the best problem is 10X as financially valuable, does 50X the social good, or is 2X as likely to work as just some run-of-the-mill good problem.

This can be left to researchers. But it shouldn’t.

For every ten or so basic researchers in an applied science org, it feels safe to say there should be at least one systems engineer. Bell Labs had about a 1:1 ratio of basic researchers to systems engineers and a 9:1 ratio of total research, engineering, and facilities development staff to systems engineering staff.

Different orgs will have different needs. But zero is almost surely the wrong number. Even some fraction of one, say 1/4, feels like it’s playing it needlessly cavalierly with such an important piece of an applied research organization. One person who is spending a day or two a week but most of their time on other things feels like a half-measure.

Bell had a massive sample size of engineers, researchers, and problems on their side, and even they didn’t rely on pure probability — serendipity — to do their problem identification for them.

There’s a reason: great problem selection is too important to be left to chance.

Thanks so much for reading! For those interested, I’ve put together a document breaking a speech from longtime Bell Labs leader, Mervin Kelly, on:

• How Bell Labs was structured

• What kinds of individuals were hired for each portion of Labs

• How different sections of Bell Labs were integrated

• And what day-to-day life looked like for the different roles.

It is great information for those who run their own research operations or are just hardcore hobbyists/massive Bell Labs nerds.

Bonus: More detail on how Bell Labs operated