Whilst strolling around the internet, like you do, we discovered a very interesting automotive project that used a Subaru automobile as an Xbox controller for racing sims!
https://www.notion.so/Play-Racing-Sims-with-Your-Car-3ef501bde46e4a8b959...
Right away we knew he wanted to meet this crazy genius but then we found a funny surprise...
[Don] "We saw your use a car as an Xbox controller project and thought it was great, and
then we get halfway down the page and we find out that it's based on our product! [both laugh]
We're like "this is great!"
...so we knew we had to know more about Nishanth and how he came to such a novel project
as turning a stationary car into a game controller.
[Nishanth] I think what drew me to what you guys are doing is the fact that the product is open, so before I started using Macchina I was using comma.ai's Panda but the documentation isn't exactly the greatest…
[Don smiles and Nishanth begins to laugh]
[Nishanth] That was a very knowing look you had on your face!
[Don] No, no! I deny that! That was not "knowing" that was just a smile, this is how my face looks! [laughs]
[Don] Let's jump into something about you then. What did you take in school?
[Nishanth] That changed about three times. Basically at the end of it I was like I want to get out of here and really try something in the real world, but I started off doing aerospace engineering, did a year of that, realized I didn't like math, switched to computer engineering for some reason, realized there was more math, and a computer science and an entrepreneurship degree by the time I left school, and I'm just ready to get out, but I mean the entire time I was in school I worked with our schools design school as well, so for me I got a lot of experience mixing computer science and engineering with design and user strategy and also with business so I try to get experience in all three of those areas and that's kind of how I went out into the world.
Oh, yeah, I ended up graduating from the school but I was just trying to get out of there
as quick as possible! [laughs]
[Don] So what did you ultimately end up graduating in?
[Nishanth] Computer science as a major and a minor of entrepreneurship.
[Don] What would you say your background is that kind of led you to where you are?
[Nishanth] I guess when I first got out of school I was working at Here Maps. If you're not familiar with them they used to be called Navteq back in the day, so it's maps in your car or via Mapquest or…
[Don] Yeah, I'm an old man. [both laugh]
[Nishanth] Hey, I used to print out maps when I was little, so I'm in the same boat, but they got bought by Nokia at the time, and then Nokia sold them to a conglomerate of German auto manufacturers, so what we were doing throughout the entire time was basically taking GPS data and trying to use that to understand if we had maps that are out of date or there are attributes that map that didn't exist in the real world so that kind of got me my first real experience working with large amount of data, and then to figuring out how to do stuff like machine learning and go to huge cloud infrastructures to possible setup things that experience really helped me kind of going forward so I ended up working at a small start-up after that called AirMap, so they do drone airspace management and safety order, so we built up stuff from scratch and now I think
it's used by companies like DJI and some of the bigger drone manufacturers in the world, and that was fun for me but I think there was still something missing at that time so I was all this fun software engineering, but I knew it deep down that I wanted to do something a little bit more entrepreneurial than just writing code, so I tried my hands at a couple other companies, so I ended up doing a little bit more architecture for code, I was in product management and even after all of those experiences I even ended up working with self-driving cars which was really fun, but in the end I was like, you know what? I just need to set up by myself and try my hand at running my own company, so fast forword now five years and I just started my own company about a year ago and learning the ins and out of doing a little bit of everything as I build up
the team up eventually.
I started my own company about a year ago so pretty much all my time just kind of been devoted to that. So I quit my full-time job, I kind of decided to go my own route, and still trying to muddle through the waters of figuring out how to do that.
[Don] Wow, that's great. So what did you start your company based around? What do you do?
[Nishanth] Basically I'm trying to help bicyclists and scooter riders and skateboarders stay safe on city roads, so it's smart lights that react to the way that you move around cities, and then there's like a data modeling part so that now if you know people's intent... so say both of us like riding bicycles through a busy intersection…
[Don] Mmm-hmm...
[Nishanth] ...we want to turn left; you might turn left by going down the crosswalk, I might try following traffic unwisely...
[Don] Mmm.
[Nishanth] ...and it's hard for cities to model that type of thing, but now if both are saying "hey, I am turning left here" but we go on two different paths, you now have a data source to then model the way people do this around cities, and hopefully in the future when we have self-driving cars or whatever, we have ways of predicting where people may behave in dangerous situations.
[Don] So do you have a product that is up and running right now?
[Nishanth] Almost! I'm almost there... So, I've actually got one of the prototypes right here {pulls small PCB out of his backpack}, so it's a little bit of hardware...you can mount it onto your bicycle, onto your skateboard, onto your longboard, and then connect it to do like interior lighting strips, and it'll give you basically brake lights, turn lights, and you can activate them either using your motion, or we can use gestures, so if you have a smart watch on your hand you can hold it out, and you can gesture that I'm turning left, turning right, braking... simple stuff like that.
The hope is the background collecting of information from people using these devices, and we can make roads safer for people once self-driving cars do come out in the future.
[Don] Very neat! That's a really interesting prediction,particularly how it applies to self-driving cars. I saw a story the other day about accidents and self-driving cars; every time they have an accident it propagates to every other car, so in theory it won't have exactly the same accident ever again. They were talking about how everything in their vehicles is just another data point that they can continue to build on so while accidents aren't an ideal thing it's actually kind of nice
all of these beginning accidents are making the car that much harder to get into trouble in the future. I can see how that would tie in with the idea of you tracking bicycling data; if the cars
could understand what the bicyclists are going to do that would be great.
[Nishanth] Fingers crossed! [both laugh]
[Don] It's been a dream since I was a child to take drivers out of the equation. They're not good at managing the vehicle. Well, then obviously you're coming to to do the the Xbox controller hack is just out of pure hobbyist love, this isn't a research project, this isn't a vocation per se
[Nishanth shakes head, smiling] ...no...
[Don] So how does one come to that? I mean, I read your blog post, and you said... you made it sound like it was the most natural thing that a person could do! It's like, "well, my car died so obviously I needed to do something", and most people would probably just let it sit there while they were waiting for their parts to arrive, so how did you come to that? Was it something that had been simmering in your mind for a long time and this presented an opportunity? Or was it the other way around; the opportunity when you went "hey, you know what, I could do a thing"?
[Nishanth] That's a good question... so, what I didn't tell you was all the while I have been doing these different jobs on the side I have been doing, just as a hobby, I've been doing racing. So, I do time attack, and drifting, and I just got through several motorsports and there's a couple fun events to go to around, and it's one of those hobbies that once you get into it you really
get sucked into it, and you're like, "I need my fix, I need my fix!", and when my car finally popped I was like "I still want to practice on a race track but I can't do it with a car that doesn't work, and hey, I know how to mess with CAN signals from cars because I used to work at a self-driving car company, and even before I'd been playing around with that stuff, so I was like, wait, what if you could just use the CAN signals from your car and use that to pretend it's a joystick for a racing sim, so that's kind of how the idea came about, and when I started the
project I didn't really have like a gaming computer or anything like that so the next step for those, like I have this XBOX here, how do I get the data from my car to my XBOX? So that was how the project came about.
[Don] How long overall did the project take from the point when you said "heyyyyyyyy", till the point where you are actually sitting there going "I'll be damned, this works"
[Nishanth] That's a good question too… I wanna say it was... two weeks? And the biggest challenge with that two weeks… so, when I say it was two weeks that was two weeks for the first iteration, which was wires going every which way in the car! I remember taking my car to a DIY garage that my friends around up in Chicago, and we were going to try running it on one of their lifts, and I remember that I had forgotten about half the cables that I needed for this thing to work, and I was like, "I drove three hours up here, and none of this works because I have this huge spaghetti mess of hardware to get it to run", so from there it took maybe another two to three weeks for me to make my own PCB, get the components, and solder it up, so that you can basically pop a little device on top of the M2 {into the XBEE socket} and then you have analog controls that go to an XBOX as well.
[Don] I would have thought that one of the hardest parts of this project would have actually been running down the specific CANbus messages involved in everything. How did you go about doing that?
[Nishanth] That part I kind of cheated in that I had some prior background trying to reverse-engineer the signals for my car, so for a while now I've always been trying to mess with CANbus stuff, and a crazy hare-brained idea I wanted to do after an 8-hour drive to Atlanta was "I wish I could make my own version of Tesla's Autopilot for my manual Subaru BRZ!"
[Don] YES!
[Nishanth] So, I know with the different signals available on the car, there's a lot of open-source tools out there, so you can plug any CANbus computer into your car. There's a way to filter out signals, like, for example, SavvyCAN is a pretty popular open-source tool.
[Don] Yeah!
[Nishanth] At the time before I had the Macchina I had the comma.ai Panda, and I used their Cabana web UI tool and that was really easy to figure out which signals were steering, brakes, accelerator… you can have clutch input, and your brake input, so you can basically read all those signals, and luckily Subaru was kind enough to not be like certain other auto manufacturers that filter out the data at the OBD2 port, which they do for either security purposes or just because they don't really want you to be fiddling around with that data.
[Don] Right...
[Nishanth] So, I guess I got lucky with my car, and the fact that it was relatively to find those signals from, just, maybe, I don't know... spending an hour or two in the car and looking to make sure that what I was doing in real-life on the car, like turning the steering wheel, corresponded to seeing little bits going up and down on the CAN bus.
[Don] So that's how you isolated it, you just kept turning the wheel until you found the appropriate packets?
[Nishanth] Yeah, one of the nice things about the Cabana tool that comma.ai put out is it will color bits that have changed, according to how recently it changed...
[Don] Sure.
[Nishanth] So you can be like, oh, that bit over there just changed as I did that, so maybe that's something I can look at. So, that's how I came up with that info, but then loading it on to the Macchina I was like "oh, this works pretty easily", and I did some quick debug statements saying, OK, steering input is coming in, or brake input is coming in, and it seemed to match what I was doing so I was like "OK, I think I'm on the right track".
[Don] That sounds great!
[Nishanth] Yeah!
[Don] Have you considered the exact opposite approach, which is using an Xbox controller to drive a car?
[Nishanth] Yeah!
[Don] I would think that that would have been like one of the first things to occur to you when you were done.
[Nishanth] Oh my gosh! [laughs] In fact, I don't think I'm allowed to show this video...
[Don bursts out laughing]
[Nishanth] ...but when I used to work for the self-driving car company… so we used to build control systems for research companies to develop self-driving cars on...
[Don] Mmm-hmm.
[Nishanth] Our job was to take cars that you could buy on the market right now and reverse engineer them so that we'd put our own so we'd put our own electronics into them and make them self-driving. So we'd used to have setups in some of our cars where you'd have a steering wheel, like a video game steering wheel, and throttle pedal, and brake pedal, in the passenger's seat and I got to drive that from the passenger seat while someone else tried to make sure we didn't kill ourselves on the tracks!
[Don] That's nice, that's nice!
[Nishanth] One of the most eerie things in the world is seeing the actual steering wheel turn...
[Don bursts out laughing]
[Nishanth] ...when you're in the passenger seat...
[both laughing]
[Don] That's great! That's fun! So, do you have any intention yourself of having a an "Xbox to car controller" so you can sit in the back seat with a joypad and just move yourself around?
[Nishanth] I could get away with that if I didn't have a manual transmission in my car, unless I built some crazy apparatus to change gears for me too...
[Don] Ah, sure!
[Nishanth] ...but I'm telling myself that's for the next car that I get. I'll make sure it's automatic so I can change that electronically!
[Don] Well, let me put an idea in your head: rental cars!
[both laugh]
[Don] I mean, you don't want to mess up your own car!
[Nishanth] Yeah! And get that insurance waiver you're good to go, right?
[Don] That's exactly right! Yeah, and when you're using something as wonderful as Macchina's M2 that plugs right into the OBD port, you remove the evidence, there's no problem!
[Nishanth] Exactly! [laughs] I think we'll have a couple of phone calls after this from various government organizations!
[Don] Oh, we'll be put on a couple of watch lists, but...
[Nishanth] Already there, I guess.
[Don] I'm quite certain I was there. I'm wondering if you've got any other kind of projects for cars that you have been considering?
[Nishanth] Yeah! So one of the top things that comes to mind is, I mean, everyone loves lights… and I've already been working with lights on my actual product... and one of the cool things about drifting is it is as much an art form as it is a very much technical way of driving. So if you're not in the middle of drifting you're basically going around sideways everywhere on a racetrack, as opposed to going in a straight line, and people have cars that are decked out in all sorts of crazy, fun paint and livery and even lights, but I think that a cool extension to that using the CAN bus data to have "reactive" lighting to the way you're drifting around, right? You can incorporate not just the CAN bus data from here I mean, you have motion sensors on the car, and you can also plug motion sensors into this device. And you can use that to influence the way that lights turn on and off. So, if you wanted to do a sweet late-night or evening drift session at the racetrack, and you have, say for example the amount of steering input you put in, and the angle you have it into the corner dictates certain lights reacting to the way your car... I think it's just a more advanced version of the stuff we saw back in, like, "Fast & Furious" in the early 2000's.
[Don] That's a really interesting idea! I can just see it! I can just see a car going around a turn completely sideways with all of its lighting a nice, angry Darth Vader red, as it's just burning up smoke!
[Nishanth] Exactly! And the cool thing is it now diffuses through the smoke too, so..
[Don] Yeaaah!
[Nishanth] ...you can get some really cool effects. So yeah, I think that will be a lot of fun not just from a nerdy perspective but also from a fun artsy perspective too.
[Don] Absolutely, absolutely!
[Nishanth] I think one thing that, I guess, would be nice is if you had the option to have a full wire harness and electrical system setup you can put on a bench. That is ideal, obviously you can't really always do that, so you end up being like what I was doing, which is hunched over your computer inside of your car when it's freezing outside, and trying to understand the signals coming off of your car. And I feel like I've done that more times than I would care to admit!
[Don] YEAH!
[Nishanth] But I'm sure that's something that you guys at Macchina have also experienced. Maybe not you in Nashville, but maybe the rest of the guys...
[Don] Yeah, the guys up there {Minnesota} have definitely frozen their ass off in their car a few times, but that was also very much the point behind the SuperB {our WiFi/Bluetooth adapter for the M2’s XBEE socket}. We were sick of freezing our asses off and being in… I mean, if your car is sitting in the driveway then why not just turn on the Wi-Fi and talk to your tablet or your laptop, why be tethered by a microUSB cable, which is fine, we've got the microUSB, but...
[Nishanth] I wish I'd had the presence of mind to think that when I was doing the reverse-engineering part and testing. That would have been much smarter.
[Don] Yeah, and since you're familiar with the ESP32 don't forget that that's what the SuperB is.
[Nishanth] Yeah.
[Don] It could be an accessory processor even, for someone like you.
That was pretty much it for the interview, but Nishanth had some kind words for us...
[Nishanth] I appreciate you guys taking the time to do this type of spotlight. Obviously it's, I think, a mutually beneficial thing, but on top of that you guys just being super supportive of me building stuff. I really appreciate the support, it's been really awesome having the type of interaction with both your engineering team and also with you as well. The back-and-forth makes me want to build stuff on your guy's platform even more.
[Don] Well, I appreciate it very much, and we certainly like to engender that kind of loyalty, but more fundamentally we're hoping that our platform is open enough that anybody can embrace it
and decide "hey I like what that guy did with the Xbox controller, I'm gonna do my own project!" and...
[Nishanth] There you go!
[Don] There you go, yeah!
[Nishanth] Sounds good Don, thank you so much, super nice to meet you!
[Don] All right, nice to meet you too! Talk to you later!
[Nishanth] Bye!