Video Mark and Joel Make Autonomous Drones in Their Spare Time (Video) 17
Back in the present, dronemaking is still a hobby for Mark and Joel, something they do for fun after spending their workdays as software engineers at Intel. Joel says there is 'remarkably little' crossover between their jobs and their hobby, and that (so far) Intel has contributed little beyond some Edison modules (which you can buy for less than $50) and travel to the Embedded Linux Conference, where they gave a talk accompanied by these slides. NOTE: We have a little bonus for you today. We try to keep videos to 10 minutes or less, but we have no such constraints on transcript length. So if you want the 'full' version of this interview, please read the transcript.
Timothy Lord for Slashdot: So, Mark and Joel you both work at Intel but you’re also right now engaged in an interesting side project does it fit into your day job at all?
Joel: It turns out that it has remarkably little to do with our day job, however it does use microprocessor technology that we had some familiarity with from the actual day job.
Slashdot: Any some funding from Intel as well?
Joel: Other than being sent to the Embedded Linux Conference there was no funding from Intel. And we are actually working on getting some of the project expenses reimbursed at this time.
Slashdot: So mostly labor of love then.
Joel: Absolute labor of love.
Slashdot: Now, before we get into what you’ve done with, I would like you to briefly explain about the hardware platform... What you’re using for your airframe, what did you buy that’s off the shelf?
Joel: Okay. So, off the shelf, we’re using a DJI hexacopter airframe with DJI motors with speed controls and we’re using a low level flight controller which is the DJI Naza. So that is obviously a closed source platform and it represents in our view the hardest platform to use at the low level for this sort of project because we don’t have the ability to open up software and make modifications to the platform itself.
Slashdot: So in actually trying to adjust that issue, you’ve added some things to it, could you start out with the sensors you’ve added, what sort of constellation of sensors do you have?
Joel: Sure. So, because we started with a closed source platform and we couldn’t use the sensors natively that are built into the DJI Naza, we had to add our own GPS, we added a u-blox 6m GPS and then we added a Adafruit 10 degree of freedom, for measurement sensor which has the 3-axis accelerometer, the 3-axis gyroscope, the 3-axis magnetometer, and a barometer.
Slashdot: And a little bit further on the hardware, you obviously have a good reason to use some Edison hardware, I suspect you get a good discount on it?
Joel: Right, so, Intel did provide us with an EDISON and in fact the project started when they handed us an Edison and said go make something wonderful that we can actually use. So we were looking at this and we grabbed the Edison block, and really the whole purpose behind this was to see well, can we make a chassis that we could then build applications on top off so that regular people, experimenters who are not necessarily skilled in every level of the vertical stack of building a drone could still contribute something toward making a drone fly. So if you think about a problem of building a drone, it has all the low level pieces and hardware and all the software to run the flight controller, the entire stack and then of course this middle ware and the applications, and not everyone has the ability to make every single piece in that continuum so we said, well what if you take a low level flight controller and make that with very high horsepower machine that runs embedded Linux. Now you could run some applications of interest with this incredible compute machine that can be internet connected and what can you do with that when you are not concerned with actually stabilizing the aircraft yourself. So the growth of that was, the outgrowth of that was making this extensible platform. So we worked on how would we actually connect an Edison to an arbitrary low level flight controller. And In our case, we picked the Naza which had its own input format using pulse width modulated signals and so forth. Just like all the other autopilots that are out there. We didn’t get the benefit from the fact that some of the open source controllers use protocol such as MAVlink which is a serial protocol which ultimately makes it a much easier exercise if you want to just natively connect a UART from a high level companion controller to the low level flight controller.
Slashdot: Why you aren’t using that?
Joel: Right, well, so reality wise... I already owned the Naza and so it was just cheaper to get started to go ahead and do that but what’s nice about it is, we had to come up with a general purpose solution that would work with literally everybody’s flight controller that’s already compatible with connecting to a radio control receiver. So that spreads 10s and 10s of 1000s of flight controllers as opposed to only those that are MAVLink compatible.
Slashdot: Now, you described this concept. You called it an extensible drone platform, I'd like to know what kind of capabilities does that mean, what you are thinking about. Why don’t you start with the big picture of what that means and then maybe get into what capabilities are already in place?
Joel: Alright, Mark you want to take that?
Mark: Sure. So, we are basing our extensible platform on the Edison compute module and Edison compute modules like a dual 4 or 500 megahertz Silvermont and it has
Slashdot: And that’s a current gen right?
Mark: Yes.
Slashdot: I mean,that’s kind of a moving target too.
Mark: Yeah, that’s right. Possibly in the future there will be different chipsets based on that platform, I can’t speak about that, not really, but it is possible. So the extensibility of the platform is you can add, like USB cameras, you could use Spy, you could use I2C, it has storage onboard, so 4 gigs of data storage, it has about a gig of DDR. So you can really put powerful software on there. And our software platform is the Yocto platform, so you could extend that with Open Embedded or you could write your own recipes and add on top of that.
Joel: I have a little bit to add to that, if I can.
Mark: Sure.
Joel: So really what was interesting from my perspective about it is that, if you consider what a low level flight controller is capable of doing today on the continuum of really on an autonomous platform, they are capable of certainly manual control as well as basic waypoint flying, and that’s why you are shifted to with a typical low level flight controller, including the Naza and including all the PX4 and autopilot derivatives. So the really interesting question is so what can you do when you made that with high horsepower. And so we’re looking at things like maybe a long range mission where you don’t already know what the path from Point A to B and all of a sudden you need to query a topographic map so that the robot can go ahead and avoid buildings and avoid terrain features that it doesn’t actually know about. Maybe it overcomes that with vision so we are looking at using say Intel’s real sense technology to do vision with a different compute platform so it can do real time sensing of its environment. We consider what would happen if the mission is long enough and you need to take a look at the weather and maybe reroute around that. So coupled with a modem, you do a database lookup to find real time weather conditions and can reroute around the sky. So it opens up an avenue for a robot to fly in an unknown environment with more data than it could have possibly had before and that we think just expands the envelope of the utility of what such an aerial machine could actually do in the long run. So we thought we would take a stab at that and because we haven’t dreamed everything that somebody might want to do, it allows other people that are skilled for this, to go ahead and try their hand at it, without having to reinvent the entire wheel.
Slashdot: Now, Joel, you are still on one of my question, which is what we are going to possibly deal with later. Mark you were talking a little bit about what the capabilities are now, can you talk more to that? What things can you now do? You gave a demo of a flight control path and what other things do you have now that you’ve been able to actually hold on this way?
Mark: So we can add things like ultrasonic sensors so that we can – we could see how close you are to a certain object.
Slashdot: So Joel has talked a little bit about some of the things that are going to be added. And I think
Mark: Are you talking about software capability or hardware?
Slashdot: Really both.
Joel: Just how it flies maybe?
Slashdot: The fact you can assign a flight path to it.
Mark: So built into the drone platform that we – sorry – so added on the drone platform where you put our GPS, so now we could do waypoint navigation. So we could support an arbitrary amount of waypoints, we didn’t really set a limit on how many waypoints you can do. We have compass directions, we also have a gyro, so we could do – we can see the rate and angle that you are turning and so forth.
Slashdot: One thing in your talk, you devoted a quite a bit to some of the difficulties you faced, the challenges both hardware and software. Can you give people who might still be willing to see this as a project for themselves, what are some of the issues that you found that they have to overcome and what things are you still working on?
Mark: Well, the sensors are complicated and depending on certain conditions you may not get accurate information, for example, if you want to get GPS positioning, you have to be moving. So if you are in a hovering position you are not really moving – you are not going to get real accurate information from the GPS. You will get the latitude and longitude, but you won’t get the angle the bearing that you are at. Another issue is the compass. We were in an Intel parking lot when we started and we had lots of metal objects like light poles and cars, and even the parking lot was probably laid with some rebar, so a lot of iron and that caused a lot of confusion with our compass. So when we moved out to a soccer field, it improved a little bit, but there are some things we needed to tune in our magnetometer calibration.
Slashdot: That seems like a really widespread problem, there is lot of ferrous material out there, I mean even the commercial drones have to face the same – they have to face the same issue.
Mark: Yeah, so we could leverage other sensors that aid in the directions, so we can use a gyro to help assist the magnetometer or the GPS in terms of finding the varying position.
Joel: So, I have something to add to that.
Mark: Sure.
Joel: Certainly we did a little bit of overlap , with what the low level flight controller can already do, so the Naza itself is able to maintain its own heading as is all the open source devices at this point, what we did by adding our own magnetometer and trying to do our own yaw based on the headings we were getting we were sort of reinventing the wheel and it wasn’t the point for us to actually do that, we added the sensor because we couldn’t natively rebed information off the Naza system that we had; of course if we had a PX4, or an autopilot we would go ahead and read that. but in our case we didn’t and so were experimenting in an area that’s already a solved problem but neither of us were experts at sensors and so forth we were just hacking around and making it work, but we enjoyed the stuff, so we thought we would give it a try
END of VIDEO 1
We learned that using the sensor data raw--that’s bad, it’s hard to do, but folks have developed algorithms for smoothing that data and for doing state estimation, so when you look at the state-of-the-art you’ve learned about the common filters or extended common filters that can be used to do full state estimation for the pitch, roll and yaw and actual position on the earth for the aircraft at all times. We weren’t going ahead to remake that, but we wanted to do a simple version that would give us some basic heading information and interestingly enough that was actually sufficient errors in all between the GPS heading a magnetometer heading to go ahead and do simple way point navigation so but certainly for anyone starting out, I wouldn’t recommend that they to go ahead and build that sensor hub again necessarily because we can do better than that by using the onboard sensors certainly if they really wanted to do their own, and that’s their goal to experiment with making a sensor hub and doing it then it is a fantastic platform, because stabilizing an aircraft and getting the right heading and getting the right altitude those are multiple problems that all have to be worked out in sequence and you don’t necessarily want to focus on but we can’t focus on everything at the same time, so it’s great to use an underlying platform an existing autopilot system that can already hover the aircraft for you . So that’s great.
Slashdot: Anything like deciding when to invent the wheel or discovering that you have, it’s a big thing anything embedded it.
Joel: Yeah, exactly, we did what we needed to so that we could get to the point which we wanted, which was we wanted to be able to do waypoint navigation and it’s not that that was inventing anything; really what it was the goal of the project was showing that you could make a high horsepower Linux machine, and connect that to a low level controller so you could do whatever you want. The proof of concept was we can autonomously take off and fly to a waypoint at a given altitude, and go to multiple way points and do that properly without crashing the aircraft, so what we’ve demonstrated is that we could connect a high level computer to a low level flight controller; what we did with it is somewhat on the irrelevant side because anyone who actually wants to experiment at that level can then choose to use a low level flight controller of their choice with the limitations or lack of limitations that are there and so something interesting with that. So in our case we could look at it as a few guys experimenting with a platform seeing what they could do which is ultimately no different from anyone else, we just had the particular constraints of oh we are using a closed source platform, so we are going to have to interface it with some sensors that don’t exist on the current platform and we’re going to have reinvent some algorithms to go ahead and smooth the data and do something useful with it. Certainly to start fresh it all depends on what problem it is that you go want to solve, if it’s goal to figure out how to do way point navigation then this is a great opportunity to go ahead and do that, because the flight controller is not doing it for you.
Slashdot: But the extensible seems to be that the keyword in what you are doing?
Joel: Absolutely, it’s really, it's getting behind the concept that you start with a low level piece that is now becoming commodity in many senses of the word, and bringing into a level where other software engineers, other developers, other makers can take something off the shelf -- really a computing system of their choice, in our case, we like Edison for what it gives with the level of integration with the Bluetooth and the Wi-Fi which we leveraged for our mission planning and Mark can tell you about it; that was a fantastic solution for us, because it gave us all the components that we needed in a very tiny low power mobile package.
Slashdot: Mark, why don’t you talk about that, talk about the mission planning aspect?
Mark: Great. So on the Yocto Platform that we have on it, on the Yocto distribution that we have on it, it has a node.js solution and we are able to build custom web applications and we implemented a navigation planner where you could set waypoints, you could get telemetry information from the drone, and you can visualize the position of the drone after the mission or during the mission, you could – it’s leveraging Google Maps API so that it gets the map in any format that you want, if you want a topo map, if you want a street map or if you want a satellite view it’s all there. It also leverages some of the polyline or the marker capabilities of Google maps. Also we could connect to the cloud as well, with the Google cloud and save information about the mission that you are planning and so that you could go to other drones if you have multiple drones leveraging Edison with the same software. You could load those configurations.
Slashdot: I think you are actually running a server?
Mark: Yeah, so Edison is a server, as well as high level flight controller.
Slashdot: You switched from a hard platform to a soft platform at some point for testing is that right, have you ever moved to a soccer field or anything?
Joel: Well, the latter set of our flight tests we did move to a soccer field because we, there happens to be a nice open field but still covered in snow at the same time, but it didn’t have any light poles in it. So we were able to do another set of flight tests with the magnetometer to see if we got any better results, we did get slightly better but then we had the idea of simply using the GPS and its heading, which led to a whole series of other problems, because we knew for a fact that we’re not going to get heading information unless the aircraft is actually moving but we figured at the right speed at the right rate of yaw, we might actually be able to navigate the aircraft and then it actually worked fairly successfully. The soccer field of course was great because there was nothing else to crash into and it was away from further populated areas. So, it worked well. So we’re lucky to have a no populated field to go play with.
Slashdot: Did you consider a golf course?
Mark: There is no golf course nearby, unfortunately. That would have been a nice space, but we also added a geofence, around the soccer field, so that if the drone flew away that it would be able to stop and hover, in a stable position and we can go close to it and use our bail out switch and retrieve it.
Slashdot: You mentioned your parking lot actually had a highway nearby.
Mark: Yeah.
Slashdot: Is that true?
Mark: It’s not really a highway, but it had homes nearby and it had a street and we didn’t want the drone to fly out of eyesightand then we couldn’t it’d be hard to capture it back and have it fly back.
Slashdot: That’s the hard kind of tech forgiveness that comes with your drone?
Mark: Also in Boston I remember we had these really high snow mounds. So just climbing over those mounds to get to the drone would be really troublesome. So the soccer field was actually a great help.
Why is this news? (Score:3)
I build fly and program quads there's really no mystery to them. Seeing how most of the flight controllers are built off flyduino you can add all the juicy stuff to them same as arduino, like object detection/avoidance and of course most fc's come with a gps module of some sort and are capable of flying a predetermined pattern using way points programmed into the fc software.
How is this different? (Score:1)
I'm in the planning phase of building my first uav (skywalker x-8) that will have autonomous flight capabilities. How does this differ from some of the other autonomous platforms that are already on the market like openpilot and the stuff 3d robotics offer?
Re: (Score:1)
941A? (Score:2)
The allusion in 451 Bradbury St is obvious to me, so now I'm wondering whether there is some significance to #941A? Have I just had my geek card revoked for not knowing it?
(Most /. readers will be aware that the allusion is to the novel "Farenheit 451" by Ray Bradbury.)
Anyone else read that as... (Score:2)
... Mike and Joel Make Autonomous Drones in Their Spare Time?
Well duh, Cambot! Joel, alone, must be a robotics and AI genius to create Gypsy, Tom Servo, and Crowwwww!