Experience in Autonomous Vehicles (AV)
Uber Advanced Technologies Group (Uber ATG)
(Linked to publicly available YouTube video)
Section I - Coming Up to Speed
Building on decades of Aerospace & Defense experience to quickly come up to speed on Autonomous Vehicles
Baselines I Brought to Uber ATG
A deep understanding of the linkages between Systems, Design, Test, Quality, Safety, and other groups in an engineering firm
Experience working safety-critical systems (aircraft, rockets, missiles, torpedoes)
Experience building and testing highly complex hardware / software / communications systems
Experience with large scale programming efforts
Experience with RADAR, LASER, and RF, as well as other sensing technologies
Experience leading large technical teams during early stage technical uncertainties
Initial Training & Then Organizing Directorates
Executive tours of all design, manufacturing, test track, and calibration facilities
Extensive ATG|versity sessions
Artificial Intelligence (AI) applications in Autonomous Vehicles
- Industry and ATG specific terminology
- Major areas of the Self Driving Vehicle (SDB) briefed by each executive (Autonomy, Platform, Scale, Systems, Safety, etc.)
Immediate immersion into the technology planning meetings at every level ​
Studying the "as is" so I could begin to organize the requirements and architectural design IP
Standing up the Systems & Tooling Directorate, and leading the new team to harvest, de-duplicate, and begin Requirements Quality Assurance on over 10,000 requirements
Deep evaluation of the Test Directorate, and then reorganizing for flow and accountability
Deep Dives
I co-led approximately 50 "deep dive" weekly 1.5 hour sessions of autonomous technologies at Uber ATG.
​
These covered current state, recent progress, obstacles, and future plans, and we explored weaknesses and options in a safe place, and provided a technical spin-up unparalleled in my earlier career.
​
Sensing & Sensor Fusion​
Perception, Tracking, Prediction, & Planning
Deterministic Logic
Localization
Control
Platform & Hardware Stacks
Calibration
ODD Mapping
Data Management & Data Triage
Simulation, Track, & Road Testing
Engineering Processes
Ticket Flows
Business Intelligence Dashboards
Issue Identification Efforts
Because I have held many roles in past engineering firms, I was able to quickly understand the fundamentals of each system, and see the cross-functional weaknesses I could help with.​
​
Organizational ambiguity on milestones and ownership
No separation of product versions... one continuous changing stream
Calibration heavily dependent on non-scalable infrastructure
Conflicting tooling for optimizing hardware
MVP tooling never completed; needed tooling abandoned
Lack of a formal Requirements Database
Insufficient Technology Tracks, Roadmap, and TRLs
Insufficient definition of design levels and ownership
Tooling for performing post-trip analysis
Fault Management System (FMS) insufficient for real time diagnosis
No troubleshooting Tree to speed diagnostics at technician level
Insufficient track automation
Data management didn't scale to sensor or resolution changes
Branch & Burn implications on introducing latent issues on road
Conflated test results due to branching approach
Failure to "lock in" gains with heightened CM on known good
Product Roadmap not in lockstep with any Technology Roadmap or actual TRLs of hardware or software
Section II - Defining "Good Enough"
Instantiating a robust Systems Engineering Directorate after 5 years in development
Uber ATG | Systems Directorate
Stand-Up Systems Engineering for Self Driving Vehicles
Uber's Advanced Technology Group (ATG) experienced a fatal accident in Tempe, Arizona in March of 2018. One of the outcomes of the investigation was the need for a strong Systems Engineering capability.
​
Nearly two years later, three attempts to stand up a proper Systems Engineering Directorate had failed.
My first temporary assignment as Director of Systems Engineering & Tooling was to build out our Systems Engineering capacity and increase the flow of formally managed requirements to the Autonomy team.
​
By the time I handed off to my successor five months later, we had ingested over 10,000 requirements and were continuing to refine and generate more.
Broad Mandate & Authority
In this first part of a two-part assignment, I would stand up a formal Systems Engineering capability.
My responsibility was to help the Head of Systems Engineering & Test to build and integrate a new Systems Engineering Directorate into the existing Product, Safety, Autonomy Software, Program Management, and Quality teams, and prepare Systems Engineering for a full integration with the Test Directorate.
​
As the Director of Systems Engineering & Tooling, I would have full authority to reshape and build the team, tools, and processes required for success in preparation for the selection of and onboarding of a new Director before moving to my next role.
Physical Unity
SE personnel were scattered across multiple floors of multiple buildings. They were continually drawn into their "old" jobs by mere proximity, and seasoned SEs weren't sitting with the new recruits.
​
My first act was to negotiate a single space where all of SE could sit together and support each other, and work near the Product and Safety groups, our key partners.
Philosophical Unity
Some of our engineers had never worked formal SE, and the only requirements they had ever written were for Scenario based tests (not the product itself). Some had Aerospace, Automotive, or INCOSE knowledge, but would talk past each other without common and ATG specific definitions.
​
My second act was to pull the entire SE team (all sites) into one room for a few days and baseline them on SE and my expectations.
Identify Top Talent and Reorg
The team was missing command and control structures. Generic "pods" could be put on any task, but nobody formally owned any piece of the design. It was frustrating the experienced SEs and confusing the new grads.
My third act was to interview every team member, evaluate their skills, and then reorganize the team into to functional areas of responsibility with clear leads to manage different aspects of the design.
Standardize & Mandate a Requirements Database
Informal requirements were stored in all manner of formats across over 50 locations. Requirements weren't traced, or even formally reviewed. More importantly, Systems Engineering didn't own the requirements!
​
I quickly mandated JAMA as the only legitimate database for requirements, and that all requirements be migrated to JAMA and be deprecated elsewhere. Over 10,000 requirements were harvested!
Standardize and Staff MBSE
Soon after my arrival, several engineers approached me regarding Model Based Systems Engineering (MBSE). They had tried to get system modeling going, but they kept being pulled from the work for other "higher" priorities.
​
They had me at "MBSE". There is no way we can manually identify all the system's iterations and interactions. We needed scalable solutions. I assigned three engineers to begin immediately.
Formalize Requirements Reviews
With over 10,000 newly harvested requirements, the work was just beginning. Duplicates upon duplicates existed, with slight variations of wording that could be critical differences in the end design. Deduplicating and organizing would take months. But to what standards would we review?
​
I created a Requirements Quality Assurance standard and implemented in our formal processes. All requirements would be automatically reviewed in tooling where possible, and manually reviewed for everything else.
Formally Instantiate the V Model
While several engineers had tried to champion the V Model for Systems Engineering & Test, it had never been instantiated specifically for the ATG environment and product line.
​
I created a V Model for the Enterprise (Uber Rides + Self Driving Vehicle + AV System), a second one for the Self Driving Vehicle, and a third for the AV system, and then trained the entire organization how to select, discuss, and plan for each.
Systems Integration with Product and Safety
Product and Safety had each taken on roles that extended into the Systems Engineering area in the absence of a formal Systems Engineering team. But we couldn't just transfer functions; SE had to be built up first!
​
Over the 5 months I build out SE, our team took increasing load off of Product and Safety as we came up the curve on formal requirements management, system modeling, traceability, and system architecture. This required (sometimes painful) changes to everyone's RACI.
Systems Integration with Test
As ATG's Software group grew, so did the number of scenario based tests. But the tests were too brittle. What happened on the track under tight controls wasn't telling us enough about probable behaviors on public roads. What would happen when scenarios came together in complex, real-world environments?
​
I began working with the top Test managers of Development Test, Structured Test, and Operational Performance Testing, to create a flow of requirements that would
Systems Integration with Software
The Product team was accustomed to taking their visions directly to the Software Engineering group. But as the complexity of the systems, subsystems, and software interactions increased, a lack of Systems Architecture was killing morale. What does "good enough" look like?
​
Over the course of 2020, Systems began to define and pass thousands of meaningful requirements to Software, and help Test build meaningful Systems and Subsystems testing.
Systems Integration with Partners
Uber ATG started with a Suburban and a bunch of sensors on a steel rack! But as new partnerships and investors came on board, and vehicle platforms evolved, more sophisticated AV/Platform integration was required.
​
In 2020 I applied dedicated Systems Engineers to be the full-time liaison to both Volvo, our core platform partner, and Toyota, a second platform. The goal was to ensure all interfaces were well managed, and that next-generation platforms would meet our redundancy needs.
Training the Org on SE
Incorporating formal Systems Engineering is one of the most difficult growing pains and cultural shifts ATG had to go through. Some aspects of "design authority" and "architecture" were moving. Many of the SE terms and paradigms were unfamiliar to employees with a purely software company focus.
​
During the first 5 months of my assignment, and then continuing on as I stitched together Systems outputs as Test inputs from the Test side, I created and presented training, RFC's, and All Hands briefs to drive the transition and help all stakeholders understand what was changing, and why.
Results Matter
Systems Engineering & Tooling Directorate
At the 5-month mark, I was able to successfully do a warm handoff to the incoming Director with:
​
A logically and physically organized staff
10,000 formally managed requirements in the JAMA database
Detailed traceability, modeling, ticketing, and tooling all well underway
The broader organization part-way through the transition from many R&D activities to a formal Systems Architecture and Systems Requirements
​
My successor would still have a lot of work ahead of him for an organizational change of this magnitude. But he had been set-up for success in every possible way. The real benefits of the new Systems Engineering & Tooling Directorate wouldn't be fully realized until the Test Directorate could be realigned to take advantage of these changes, some 5 months down the road.
​
And so began my next task, as the Director of Test, well positioned and partnered with the new Director of Systems Engineering to start the process of aligning the Test group to the new Systems group, and consolidate and solidify all the gains for Autonomy and the rest of the Software group.
Section III - Align Test to Systems
Drive 10X test velocity improvements while aligning to Systems & improving signal
Uber ATG | Test Directorate
Increase Throughput to Autonomy Software
Uber's Advanced Technology Group (ATG) had a large Test Directorate with three senior managers. The group was already pivoting to more simulation for Test.
But our Test Track was a limited resource, COVID-19 was impacting our ability to work in proximity, our organizations ​had evolved without a plan, and we weren't aligned to the new Systems Engineering Directorate.
​
The assignment was to step in as the Director of Test, identify every waste, bottleneck, and opportunity possible, and get 10X the capacity out of Test within months.
​
​
Excessive Handoffs
I began mapping a single test from conception through track testing, and my Sr. Manager counted a whopping 53 handoffs going back and forth between different teams, and even between the three test groups.
​
This was one of the major justifications that led me to performing a Test Directorate Reorg, one that would make ownership clear, handoffs fewer, and reduce the number of people required to execute or make decisions. This was part of my 10X velocity improvement approach.
Unplanned Organic Growth
The Test Directorate was split into three groups, Advanced Testing, Structured Testing, and Test Operations. As the company grew from 5 to 1500 people, the organizations evolved more because of who had hired who, or where people and teams preferred, rather than where they optimized the business.
​
My three Sr. Managers were all painfully aware of the discontinuities. They just needed me to make the hard calls and drive the changes, while respecting all parties involved. We reorganized ~200 people without drama.
​
​
Un-Instrumented Processes
Test already had a Business Intelligence (BI) team, heavily leveraging Jira and other database backends. It already had key managers mapping processes. What it lacked was "Instrumented Processes".
​
Once I saw what the BI team was capable of, I pushed them to make forward looking projections we could make decision with, rather than just historical reviews. By plotting Ticket Progression, Turnbacks, and Mean Ticket Times against every step in a process, we could SEE why our professionals were losing nights and weekends on rework, and make changes!
Role Confusion
Across the Test Directorate, as in many startups, people picked up whatever needed done, whether their formal role or not. But now at ~1,500 people, it became harder to know everyone, anticipate who would do what, or even if they should.
​
This was another area of focus during our reorganization efforts, dubbed "The Great Untanglement". By creating clarity in roles under appropriate groups, we simplified the entire flow, reduced steps, and the need for dozens of people on every call.
Track Automation
Our track and road testing were our most expensive operations. Track testing required exact conditions to provide repeatable feedback to the Autonomy Software teams. However, manual setups, manual tracking of AV's, roamers, other vehicles, and triggers were fraught with opportunities for human error.
​
I learned that Uber ATG had previously met with University of Michigan's "Mcity" about their track automation software IP, and led a 4 month negotiation that secured the IP for ATG. More details on this in the Mcity section!
Test Tooling Deprecated or Abandoned
In a new industry like Autonomous Vehicles, much of the tooling you need must be locally created. The challenge I found was that there wasn't a "Tool Czar" to coordinate across them all, and that each tended to be a person's pet project. When that person moved on to another project or left the company, the tendency was to restart a new tool.
​
I began working the organization toward centralizing the tool management, building for reuse and scalability, and getting beyond MVP to feature complete.
Tooling Not Scalable to Common Design Changes
Hard coded or index based programming had to be reworked every time there was a change in sensor count, resolution, or sample rate was extremely costly. Design changes (add two LIDARS, or upgrade cameras, or speed up spin rate) invalidated existing versions of storage, troubleshooting tools, triage tools, simulation tools...
​
This was a fundamental problem that I was working to change systemically. Building scalable, future-proofed tooling requires architecture and forethought beyond MVP.
Excessive Turnbacks in Workflow
Once my teams had instrumented our processes, we could see instances of high frequency, high distance, and high mean process step time. The next step was to Pareto the risks and impact of each, and create projects to go after them!
​
The trick here is to recognize that even small misses can lead to disproportionate impacts. I taught the teams how to identify waste, and how to apply Lean thinking and Mistake Proofing to their areas of responsibility. And we recovered hundreds of person-hours each time.
No Continuous Improvement Program
Uber ATG didn't have a formal CI program of any kind. No branding, no intake, no repository, no leadership review, no incentive to participate, and no champion.
​
As a Lean Six Sigma Black Belt that had done this before, I established a CI program called "Velocity", in line with our 2020 objective to create a 10X velocity improvement through the Test Directorate. I built the tools, presented weekly at the All Hands, drove engagement, stood up the Leadership reviews, and assigned administrators to keep it going.
Align Test to Match Systems
Directorate Level (L7) Role
At Uber’s Advanced Technology Group (ATG), I worked as Director-level deployable staff member for the Head of Systems Engineering & Test.
In this second part of a two-part assignment, I would work to restructure and streamline Uber ATG's Test Directorate to align with the newly formed Systems Engineering & Tooling Directorate I just finished standing up. My Test Directorate consisted of 180 people across:
Development Test - R&D, Prototypes, New Feature Testing, Interactive Work with Development
Structured Test - Offline Simulation, Triage, Test Plans,
Test Operations - Track and Public Roads, Operational Safety, Business Intelligence
​
As the Director of Test I would reorganize the Testing Directorate so that it could align with the work I just completed as the Director of Systems Engineering & Tooling. My primary goal was to set up the three Test groups to operate at scale - and with sufficient velocity - to meet 2021 demands. Initial challenges included a limited track resource, unclear spans of control, lack of testing to formal Systems and Subsystems requirements, and operating in a Covid-19 constrained environment.
​
With three very capable senior managers at hand, this assignment would involve mentoring those managers and their subordinates, pathfinding to solve their common challenges, streamlining the testing flow, and improving our testing approaches to give better answers to Autonomy, Hardware, Systems, and other stakeholders. Most importantly, I worked with the managers to institute new paradigms - beyond simple scenario based testing - that would improve the fidelity and robustness of the test results in edge cases and complex situations.
​
Steps I personally took:
UNDERSTAND THE AS-IS
Extensively interviewed personnel at every level to build up a profile of operational capabilities, measurements, and challenges
Engaged in both strategic and tactical meetings to watch how processes execute or struggle
Worked with teams to document process flows and Pareto excessive handoffs and wastes
Directed the Business Intelligence team to instrument processes with data from Jira ticketing, Jama requirements, and internal tooling data to identify bottlenecks, turnbacks, and resource limitations
Evaluated what was performed at track that could be done earlier/later to preserve resource
REORGANIZE THE TEAM
Reorganized the Test Directorate over several months working closely with the top managers of DTE, STE, and VO, laying out new structures for Subsystem Test, System Test, and Vehicle Test, aligned with the new Systems Engineering
​
BUILD INFRASTRUCTURE
Led negotiations and licensed University of Michigan's Mcity automated track source code
Identified core weaknesses between Product planning and Technology readiness
Launched and led Project Framework to define and manage AV technology tracks, tech roadmaps, and TRL measures
Worked with Platform Triage to restore Fault Management System capability
Expanded Business Intelligence measures to the entire Test Directorate
​
Uber ATG | Mcity IP Negotiation
Securing New Tech to Increase Development Velocity
Real-world engineering development and testing turnaround times were too slow. Our track was our most precious resource, and it was bottlenecking Autonomy's ability to get timely feedback.
​
University of Michigan's "Mcity" had developed software that could help Uber ATG make leapfrog forward, combining our large track and automation with Mcity's track control and routing software.
​
Others before me had identified Mcity as a potential opportunity, but it had fallen off the hardware initiatives list.
​
The assignment was to renew the team's passion for this project, engage with Mcity leadership, and negotiate an IP agreement that would be a long-term win-win for the university and ATG.
Results Matter
Test Directorate
At the 12-month mark, the Test Directorate had undergone a tremendous metamorphosis.
The testing paradigm had shifted from testing narrowly defined and brittle scenarios - to testing based on systems and subsystems requirements that would persist across any scenario or combination of influences.
The three senior managers had agreed to a new testing flow, had agreed to ownership each piece of the Test Directorate (RACI matrix), and had agreed to where people needed to move to accomplish the next years objectives.
The Software and Hardware teams were seeing end to end linkages from Systems through Design to Testing for the first time.
My new Continuous Improvement program, "Velocity", had taken hold. It was generating both solutions to known barriers, and esprit de corps amongst the entire Systems Engineering & Test group.
​
​