Mobileye In New York City, Annotated

Mobileye recently announced that it has expanded its autonomous vehicle testing to New York City. The Intel subsidiary released “an unedited 40-minute drive” of its camera-only vehicle traveling autonomously through the Big Apple.

Previously, I annotated a similar Mobileye drive through Jerusalem, so I did that again for New York.

My main takeaway is just how much time vehicles spend stopped at traffic lights in New York City. No wonder everyone takes the subway.

[0:35] At CES 2021, Mobileye emphasized how quickly they can spin up testing in a new city. Now they are testing in Detroit, New York, Munich, Israel, Tokyo, and China. That’s an interesting mix of countries and cities.

[0:50] Mobileye’s “trinity” consists of redunancy, crowd-sources maps, and the “RSS formal model for safety.”

[1:00] This video will feature a camera-only car, although Mobileye is building two entirely redundant systems — camera, and lidar plus radar.

[1:25] Mobileye refers to “Vidar” on their camera-only vehicles, without explaining what this is.

[3:00] The route starts at a stop light alongside Central Park. I noticed in the Jerusalem ride that the human driver pulled the vehicle into and out of traffic. Interesting they start the vehicle in the middle of the lane here.

[3:21] The vehicle is doing very accurate semantic segmentation to paint each pixel of free driving space in the camera image.

[3:53] Really nice job navigating around a double-parked UPS van, and then pedestrians using the vehicle’s lane to exit a parked car.

[4:09] The vehicle seems to get a little confused by a car pulling around a stopped mail truck. The Mobileye vehicle basically parks itself on the lane line and takes up two lanes.

[4:45] An awful lot of driving time in New York City is spent stopped a traffic lights.

[8:00] OMG so much time spent waiting at traffic lights! The case for congestion pricing makes itself.

[8:30] The side cameras pretty well detect and track pedestrians and vehicles that are across the median from the Mobileye vehicle. This is both impressive, and also a possible waster of computational power.

[10:45] The system classifies eight types of pixels: road, road edge, elevated barrier / curb, road under car, semi-drivable area, vehicle, pedestrians, general objects. Construction barriers are classified as purple “general objects.”

[11:45] Great job navigating the tunnel under (I think) Grand Central Station. Interestingly, the in-vehicle passenger display shuts off here.

[13:55] Another car cuts in aggressively and Mobileye acts appropriately to avoid a collision.

[13:55] The images switch from pixel segmentation to paint segmentation. The different painted road markings are highlighted. Presumably this is an input into Mobileye’s driving model.

[16:09] The paint segmentation video appears choppier than the pixel segmentation video did. I wonder if they devote less computation power to this.

[16:23] The steering wheel rocks quite a bit when stopped. It’s hard to tell how stable the ride is from a passenger perspective — the video certainly seems stable.

[17:11] Now the video switches to “Vidar.” It’s a little hard to tell what this is, other than that the video blacks out everything that’s occluded. Bounding boxes appear, although those also appeared with pixel segmentation.

[17:29] The car made a right turn on red, which is an interesting policy choice in and of it self. Also, it rolled through the stop line while making the turn, which is a traffic violation.

[17:48] Narrow street with parked vehicles on both sides, but the Mobileye vehicle seems awfully tentative here.

[18:26] The dog is not captured by the bounding box.

[18:58] Nice turn through a tough intersection with lots of pedestrians in the crosswalks.

[19:19] The vehicle stops well short of the stop line, seemingly intimidated by the giant SUV in the next lane.

[20:05] The vehicle leaves a lot more space before the next vehicle stopped in traffic, compared to human drivers, but frankly it seems safer.

[21:00] Nice job navigating metal construction plates on the road.

[21:30] Tough driving, wedged between traffic on the left and parked cars on the right. Great job.

[27:05] Nice driving in a lane demarcated by traffic posts.

[27:39] Tunnel! This would freak me out as a human driver. Mobileye handles it flawlessly.

[31:09] The vehicle must be going pretty fast through the tunnel. Maybe 45mph? That’s a testament to the range of the cameras.

[32:14] We’re in Long Island City now, and everything seems just a little more open and manageable.

[34:35] They still have traffic lights on Long Island.

[35:36] Absolutely brilliant navigation around a double-parked moving truck.

[36:10] The driver of a parked car opened his door and stepped into the lane. The Mobileye vehicle didn’t even flinch.

[36:42] The Mobileye vehicle was a bit tentative getting around this double-parked vehicle, and wound up in a stick situation. The car behind the Mobileye vehicle tries to go around at the same time as the Mobileye vehicle itself tries to go around, so now we’ve got two cars driving into the opposing lane.

[41:30] The vehicle is a bit tentative navigating past pedestrians as they clear a crosswalk. As a result, another pedestrian asserts himself, and the vehicle winds up completing a right turn on red.

[42:00] The route ends behind a stopped mail truck, but pretty clearly in a lane of moving traffic. Once again interesting that vehicle doesn’t really seem to handle pullovers.

Originally published at http://davidsilver.blog on August 5, 2021.