*36C3 preroll music*
Herald: So Sasha is a doctor with a weak spot for LEDs and, completely abstains
from HDMI adapters these days. He wanted to share with us the experience of
attempting to build a delivery robots in the past 2.5 years in the Bay Area. And
so, yeah, let's give him a big welcome.
Sasha: Thank you, Mikael. So just a show of hands who here has built robots before?
Well, it's quite a few people. What about autonomous robots, anybody built
autonomous robots? Still quite a few people. Well, today I'm gonna be sharing
with you the story of how not to build autonomous robots. Over the course of the
past two and a half years together with my team, we built the world's largest robotic
delivery infrastructure. We went from a concept sketch to a commercially viable
service running in three cities. We've had lots of successes and one or two failures.
So over the course of the next 45 minutes, I'm going to be sharing with you a couple
of different stories. First of all, I'm going to briefly introduce myself and I'm
going to share the story of how we built robots, the different prototypes we had,
the different iterations that we tried. Then I'm going to jump on manufacturing.
We actually went to China and scaled up our manufacturing, our production line.
I'm gonna share with you the story of how he did that. And finally, I'm gonna talk
about A.I. and all the magic that is artificial intelligence. So we'll be able
to see how we were able to crack that puzzle. So without further ado, let's do
the introduction. This is me. Right here. I like to build things. I built my first
website when I was eleven and I built my first business when I was 13. I was the
iPhone repair business that I was running in my bedroom. I've been really, really
passionate about building things and over the course of many years I built a couple
of different startups. One of them was a food delivery platform. We ended up
running three different cities and doing hundreds of deliveries a day. By the time
I was 19. So I got to experience startups pretty early on. I've been really enjoying
that time. After this food delivery startup failed I went to some
cryptocurrency startups and then went to work for big corporations. And that's
actually very boring. I dorned my office with some supplementary graphics. After a
while, I got a little bit bored of this corporate life. It wasn't really for me.
So I decided to get a one way ticket to San Francisco. So I ended up in San
Francisco staying on a friend's couch, not really knowing anybody. And I was really
fortunate to be introduced to an incredible group of people. And over the
course of about two and a half years, we started to take a concepts. A scatch that
we had and we built up a robot. At first it was something that barely even worked.
But then we gradually got to something that worked a little bit better and better
and better. After a while, we actually managed to build a whole fleet of robots.
I think at the peak we had 150 robots. So it was a really, really cool experience.
And during that time, I got to meet the lieutenant governor of California, how to
figure out how to do manufacturing in China and most importantly, work with an
incredible team which who I had a lot of fun with building these robots. So, yeah,
it's a little bit about me and what we were building. And maybe now we can jump
in to how not to build robots. So this is our very first robot,this is a really
small prototype. We built is basically a shopping basket on wheels. There is a RC
car there below, there's a shopping basket and Arduino Raspberry Pi. The thing barely
work. Honestly, it was really, really hacky. And what ended up happening is that
most of the time we just dropped off the robot in front of the customer that
literally just dropped it in front of the door just to see if they would like order
food with robots. The answer was overwhelmingly yes. So we decided to spend
some more time building our technology. There is a small - I don't know if you can
see it here. Yeah, there you go. There's a small orange holder, that's actually a
phone holder. So our very first prototype, it had a phone sitting on top of it doing
a video call so that somebody can remotely control it from Colombia. So we really
started out small, really humble just to see if it would work. And that's something
that we did a lot of this being really resourceful in terms of trying out things.
For about a year of this, we moved on to something that looks a little bit more
like this. So we started playing around with the shape. We start playing around
with the design. We noticed that people responded really positively to faces and
to like things that looked like people. So we actually built in a face. So we took
this little animation that we built and we put it onto the robot and there's actually
really, really positive. We had a lot of good responses from the community, a lot
of great feedback. And what we've seen is that people really love to have robots
that are kind of friendly. There was another company that deployed robots that
looked like vending machines or almost like tanks in San Francisco, and they got
banned really, really quickly. So we decided that we would do our best to make
sure our robots were as friendly as possible instead of threatening and scary.
So that was a very important part of it. After another year, we ended up scaling up
our production and we went to China to manufacture robots. And here this is what
we ended up doing. *music*
It's actually, a cool robot. We built it entirely from scratch. We got our own
chassis, our cabin, our own compute module. Basically just about everything.
That was a really cool experience. That was me. So yeah, that's a robot. That's
the one we were rolling around the past six months. And we also had some failures
in between, as you saw previously, this one. So we actually tried a couple of
different concepts. So this was one of them. This was a Kiwi trike. We thought
that maybe we can figure out how to have robots do part of the delivery and then
trikes to another part of the delivery. We also tried to do restaurant robots. We had
like robots that sit in the restaurant and bring food out from the counter to your
doorstep. But what ended up happening is that it was actually pretty inefficient
and people would wait a really long time for their deliveries. So it was very
important for us to try a lot of different things. We tried this robot, the kiwi
TRIKE that not quite worked out as we expected. We tried a restaurant robot. We
tried a box that would sit behind our robot. We tryed a hub that would have like
of a bunch of different robots inside of it. So we really, really tried a lot. And
with every iteration, we constantly tried new techniques we costantly tried new
manufacturing methods. We really tried just about everything to see if we can
make it work. And what we ended up building is a platform that was really
loved by people. We built a platform that students adored. That was our primary
demographic we're delivering to college campuses and students really loved our
products. We actually had people dressed up as Halloween costumes. We had entire
classes go for Halloween in like kiwi bot costumes. So that was really, really cool
stuff. Had a lot of great support. A lot of trust from the community as well. And
that's like coming back to the design. That aspect of having a friendly robot,
that meshes seamlessly within the fabric of a community is like super, super
important. We've seen other robots around and they were maybe not as friendly, maybe
they looked a little scary. Maybe they had something that was a bit off or maybe a
little too industrial. But having like a friendly robot that could become a meme
that was something truly revolutionary, something that really changed the
landscape. And as a matter of fact, like these cute bots are the only robots that
are deployed somewhere in the world where they coexist. Day to day with a community,
with people. Like you have some limited deployments of robots here and there,
maybe have a Roomba at home or some like that. But you don't have any large scale
deployment. We have robots and people living in the same city all the time. So,
of course, it took us a while to figure out what to do and how to do it. At first
one of our models was to have robots deliver the entire meal, like go from the
restaurant all the way to the customer and we would have a robot do that delivery.
Turns out, it was pretty inefficient. People would wait like 60 minutes, 90
minutes for their delivery. And we realized that maybe automating all of that
was not the most efficient approach. So what we instead did is a multi-modal
approach where we had people and robots. This is actually a really cool
visualization that my team came up with. The blue lines are robots. So these are
robots roaming around our Berkeley coverage area and the yellow lines are
people. So how this would work is that people would go to restaurants, they pick
up the food and they take you to a cluster. They take it to a cluster where
you had a bunch of robots, they loaded into the robot and then the robot would
actually do the last few hundred meters to your doorstep. And because we were able to
do this, we were able to go and build a platform, that handled hundreds of orders a
day with very, very few people. I mean, labor costs are really high for delivery.
You'd be paying somewhere between five and thirteen dollars to get a meal delivered
in the U.S. And as a student, that's like super expensive. That's not something that
you can afford do every day. And also there is a pretty big shortage of people
who want to do this job in the first place. The trend is really high. People
are leaving all the time because they don't like to like sit in a car all day
and deliver food. So that's why we have this parallel like this multi-modal
approach where the people are biking around, they're enjoying their time outside and
the robots are actually doing other boring stuff like the waiting. So the robot would
go up to your doorstep and would wait for you to put on your pants, your shoes and
actually walk outside. So that way we were able to change the dynamic. We're able to
change our deficiency from one or two deliveries an hour, as you would have with
like a traditional delivery service to as much as 15 deliveries an hour per person.
So it made the delivery far more affordable and we were able to offer
delivery at just one dollar a delivery, which is a cost that changes completely
the way people approach delivery. In fact, if we look our top 20 percent of users,
they were ordering over 14 times a week. So they were very, very happy that they
could get whatever they wanted very quickly. Of course, not everybody was
super happy. So we did have some people that didn't fully appreciate the magic
that is kiwi bot. So we did have one person try to steal it, but they didn't
get away with it. We found them pretty quickly. They hid it in the trunk. Not a
very smart move. We ended up finding it with G.P.S. and also triangulating the Wi-Fi.
So this guy decided to steal it because he doesn't like robots. I don't know why, but
he was clearly very passionate about that topic. And he stole it and now he's in
jail. So, yeah, don't steal robots. So maybe some conclusions from our robot
part, like from building robots, from figuring out like what to do and what not
to do. Really important thing that we do a lot in software and maybe not as much
hardware is iteration. Like we iterated through three major revisions and like
lots of small revisions during a really small period of time. It was really
interesting to see like that transition. Every single time we try something new, we
try it maybe for like 20 robots at a time, like not our whole fleet. We just try for
a small portion of our fleet and that we were able to iterate really quickly and
see what sensors worked or cameras worked. And just to see what we could do in order
to grow the products, it was very important to iterate. Communication.
Communication is absolutely fundamental. And not only communication like inside the
company or anything, but more importantly, communication with your community. Because
we weren't just building a product in isolation. We were building a product for
people who live in a city, who have an established life. And we're kind of
intruding into their lives by bringing in a new product that takes the sidewalks. So
communicating what we're doing, showing them what this is and what this robot does
is super important. Actually, very early on our designs have no text on it. They
had like no information it was just like a basket case on RC car. And people were
like really confused. The police were like: "Hey, what is this?", so we had to
add a lot of communication, we had to put food delivery on the robots really
clearly, we had to add a license plate with like a phone number that somebody
could reach out to us. So communication is very, very, very important when it comes
to robots. Also: Scaling hardware is hard, super hard. I mean, it was crazy. When we
first started it was just Arduinos and Raspberry Pis and that did not scale
really well. Like, sure, we could have maybe 10 or 20 units at once. But then how
do you handle updates? How do you handle those weird things that happen all the
time? So it was really challenging to do this. We actually killed a bunch of SD
cards. Didn't really know you could destroy SD cards, but you can. And we
learned a lot of things about hardware pushing it beyond its normal boundaries.
So yeah, iteration, super important. Communication is key, like getting buy in
from the community and scaling hardware is super, super hard. That's something we
actually figured out how to solve by going into China. So how to do or how not to do
manufacturing? So as every China story goes, I hopped on a plane and I ended up
in China. And it's really interesting to see because like you have this perception
of China from the media, you have this idea of what it would look like. But the
reality is it doesn't look anything like what you would expect. It was a completely
different world. It was at the same time Bladerunner and like the most modern city
in the world and it was truly an awesome experience. I highly recommend anybody who
has the opportunity to go in and explore the world. But of course, the culture is a
little bit different. We were surprised to see some things happening there. Was a
weird dichotomy between communism and consumerism. This is kind of interesting
to see that sometimes. But the reason why we came to China is for manufacturing and
there is no better place for that than Shenzhen. In Shenzhen, you have
Huaqiangbei. This huge market. It's a market that spans several city blocks and
you can actually find anything and everything you want. We were able to get
components super quickly, super easily. And you could spend days just walking to a
single building finding different things. There were entire city blocks dedicated to
like just LEDs or just connectors or just processors. It was absolutely crazy. You
could really, really, really get lost inside of these mazes. And what was really
incredible to see and something I've never seen anywhere else in the world is just
how easy it is to get hardware, to get things, to get parts. It was super easy.
Just go in and get something and you could get it at one piece, two pieces, a
thousand pieces like instantly. If you're anywhere else in the world, that's super
hard to do. So just by this virtue, you're actually able to prototype things. You're
able to build things incredibly fast. You're able to go in, you're able to
comission a PCB and get all the parts almost instantly, which is not something
you see anywhere else in the world. And also a lot of the manufacturers have their
booths here so these would be direct booths from the manufacturers so you could
say go up to them, start talking to them and ask, hey, can you make this product
this specific way? Can you do it how I want it? And they'll be like, sure, why
not? They'll do it for you. So it was really, really valuable to just learn from
these people, from the vendors here, from manufacturers about how to build things.
And it was actually really surprising to see everything they had in stock. Two
years ago, we built an art installation here that covered a tunnel with LEDs. We
covered one of the tunnels at 34C3 with LEDs. And we used this tiny, tiny chip. It
was a five dollar ESP 8266 chip that basically was able to control all your
LEDs. And over the course of five years, up to that point, I spent a lot of time
figuring out how to build it myself. I played with Raspberry Pis I played with
PCA controllers over serial and like I finally managed to get a prototype to
work, but it was super clunky, it was super expensive and it wasn't very
reliable and I go to China and I find that it's available there and much better
quality, much cheaper, much faster, so it was a really, really interesting shift in
perspective. It's something you can't appreciate when you're abroad. Even if
you're browsing like eBay or Ali Express it's kind of hard to appreciate just how
much selection you have and how you can find just about any tool, anything you
need to find. So it's really, really incredible. But these markets were cool
but was even cooler are the factories and during a course in China, we were able to
visit a lot of factories. All these factories there are super, super
welcoming. They always love having you over. They invited you to really, really
luxurioius dinners. We had way too much food. And it was a feast of celebration
every time. Actually, relationships are super, super important in China. Like a
lot of people in the West, like they have contracts and they say, OK, this is the
terms of the contracts. Well, China, you do sort of have contracts, but they don't
matter as much as relationships. Like when you have a relationship with the
manufacturer, you have to like always go to dinner with them, drink beer, smoke, go
to KTV like it's a really evolved relationship. And you're only able to have
good communication based on that relationship, because if you don't have a
relationship they kind of forget about you. We actually had a couple of instances
where manufacturers ghosted us. Like they had a critical component and they just
stopped answering our emails, they stop answering our weechats, they just
completely ignored us. And for some pieces they were completely irreplaceable, we
could not just go out and find another factory to produce a specific part the way
we wanted. And the only way you can ensure that this doesn't happen is by really
explicitly making sure that you have good communication, a good relationship with
that manufacturers. It's super, super important. This is one of the factories we
worked with. It's really crazy. I mean, we went there and we're just absolutely blown
away by the scale of everything and also blown away by how manual everything is.
There's actually audio here. Everything was super manual. People were just like
there with minimal or no protective equipment whatsoever. Just like building
things that look like they were made by robots or machines, but they were in
reality built by people with their hands, which is super crazy to see. And there
were a lot of Blade Runner esque designs, really bizarre contraptions there in this
factory. This is our fiberglass factory. The way we build our casing, was actually
prototyping it first in carbon fiber, sorry, fiberglass and then moving onto a
mold in carbon fiber. And actually Scotty, he made a really cool video on YouTube. So
if you search for hockey stick factory on YouTube, you can see a huge video. My
buddy Scotty actually goes and meets his factory, discover how they make this mold
and how they make these carbon fiber things. It was actually really crazy to
see it. It was cheaper to make a carbon fiber mold than it was to make a plastic
mold. So since the tolerances were a little bit different, since like the
process was a little bit simpler, you were able to make a mold, that was very, very
strong and very indestructible without necessarily having to have all of that
expense upfront for like a plastic mold. So, yeah, that our a fiberglass factory.
Really exciting stuff. Really crazy scale. These folks like the first night we came
there, we arrived at like 8 pm and there was 100 people in the factory just like
working at 8pm. Really crazy to see. This is another factory worked with. So this
was a metal factory. It was actually really, really, really interesting to see
how they built all these things and at one hand, you can build super complex things,
you can build a super complex designs. But on the other hand, we got surprised a
couple of times by being unable to manufacture really simple designs. And it
took us a while to get a grasp was like, oh, OK, so we can make really complex
metal that's bent, but as soon as we add a well to aluminum, you start to have a big,
big problem. So we had to like change a lot of our designs. We had to really adapt
to the way things were being made in China. And sometimes you could adapt
yourself, but like at an insane cost. So it was better to adapt to the way things
were being done there. So, again, very, very interesting to see how things are
done. No protective equipment, this is like a two ton press and his hands are
millimeters away from it. So, yeah, it's a different world out there. Very, very
different. Another factory we visited was a PCB factory. So this one has a really
interesting story. This factory is not in Shenzhen. It's just across the border from
Shenzhen. The city actually passed a law a couple of years ago that has very, very
strict environmental policies. So you're no longer able to do PCB manufacturing
inside the city anymore. So we actually had to drive for a couple of hours outside
of the city and over there was a huge plant. And this plant was kind of semi
automated, semi handmade. Were part of the process were done by hand, as you see
here. But then parts of the process were done with machines. So they had this giant
machine, which is basically a black box we can't really see inside of it. But you had
a bunch of chemicals and it's like take a PCB and just like move it forward through
a chain. This is really intresting to see. And this factory also had a really quick
turnaround, they had a three hour turnaround if you paid a premium and the
standard was 24 hours. You could also ask them to do PCBA so you can actually get
them to assemble the PCB for you. And we ended up doing that for some of our PCBs.
We'd give them build materials and we'd give them our designs and then they
manifacture it. We actually got in a little bit of a situation with that
because we sent them some designs, we sent them some parts that we wanted to put in
our PCB and it turns out that one of these parts was unavailable and they didn't tell
that to us until it was almost Chinese New Year. So we had to scramble all that to
find another solution. Was very exciting to see how you would deal with these
factories. There are some even cooler factories. I think the coolest factory I
visited was a battery factory where they made lithium ion and lithium polymer
batteries and it was almost entirely automated. It had giant films of things
going into a machine and then you had all sorts of liquids and powders it was all
combined together. It was super, super cool. Didn't allow us to film it
unfortunately, there may be only a dozen or so such factories in the world. They're
very protective about their technology, but the scale of how quickly they're
manufacturing these batteries was just incredible. They would manufacture them at
a crazy, crazy scale. So all these factors are cool, but actually building things is
even cooler. So we ended up partnering with a contract manufacturer. I was really
fortunate to find one through my network. Otherwise, I would have been totally lost.
A couple of days before I ended up going to China I found a contract manufacturer
that liked to work with start ups and small scale people and we ended up working
with them to build our first batch of 50 robots. It was really interesting to see
how different our designs were to what they expected. So they expected things are
really ready. They are very explicit, very clearly specified. But we didn't have
that. The difference between manufacturing in the U.S., for example, against China is
that in the U.S., like it's a super long process and the back and forth takes super
long just to get an idea of what kind of files they need. Whereas in China, you're
able to sit down directly with the engineer, with the person in charge, and
you can figure out what they need and they can help you out instantly. Actually, just
here, I just want to show you one thing. So this is my designer, Alehandrew, and he was
translating from English to Chinese with his phone with Google Translate, and it
worked surprisingly well. Google Translate actually is not blocked in China for some
reason. We were able to communicate almost all the time with that. Also, Weechat has
a built in translate feature. So Weechat is like the universal app that everyone in
China uses and has this built in translation feature that can translate
your text automatically. So it's really, really cool to see how that worked. One
question that we get commonly asked is like how do we find our manufacturers? How
do we get this relationship? So about 20 percent of that was through Alibaba. So
our fiberglass manufacturer. We quoted like 30 different manufacturers and went
with the cheapest one. Of course, it was far more expensive than we expected and we
ended up producing with them. 20 percent that, for example, our chassey, it was
built with companies that we already had a relationship with. So we were just able to
continue working with them. And then 60 percent was through just references so
select networks or getting to know people and talking to them saying, "Oh, hey, who
did you use for this or this" or "How did you make these PCBs?" or just getting a
conversation going. So having that kind of network was really, really helpful in
order to build these robots. So as you can actually see right here, our design, this
is what we had when we came into China. When we left, we had our own computer
module like super sophisticated. But this was like a Raspberry Pi, a pix hock and a
voltage converter like a DC to DC converter. That was pretty much it. As you
can see, it was not very reliable. It would break a lot. So it took us quite a
while to translate this into something that was manufacturing well. So thanks to
the dedication of my incredible team that we're able to do that. And we kind of did
not know what we were doing so, we ended up having all of our parts and all of the
components ready just days before Chinese New Year. So we actually had to do all of
that someday ourselves. We didn't have any Chinese workers who could help us do that.
So there's our team just assembling things in the factory like we wanted two days
before Chinese New Year. So that was very, very interesting. We kind of hacked or
tried to hack Chinese New Year. We assembled all the robots literally days if
not hours before Chinese New Year and we shipped them out and everything was great,
except our robots got stuck in customs. We had a trademark on our box and the customs
agents, they open the box and saw more trademarks on some parts. We had 3D
printed parts and they were like, no, this is not going to go through without the
proper paperwork. So our robots got stuck for three weeks in China, which was really
fun. Little problematic. So, yeah, those kind of things happen you have to be ready
for it. After we received our robots in California, we had to spend another like
maybe one or two months refinishing them, redoing some parts, tweaking them,
flashing them. So there's still a lot of work to get them to work. The pieces we
shipped out to China was maybe just like a case with most of the electronics in, but
not all of it. So we still had to do a lot of tweaking over back home. And of course,
all this going to be impossible without an incredible team so I was really fortunate
to be with some really, really passionate people who would work four months in a row
continuously without virtually taking any breaks. We had plenty of opportunities to
go and take the high speed rail or go to Shanghai or even Tokyo, but we all stayed
in Shenzhen and spend a lot of time together building these robots. It was a
really, really arduous journey. So maybe some conclusions for scaling,
manufacturing, some of the failures we've had in relationships. I mean,
relationships are super important, like super, super important, in China far more
important then contracts. If you're able to have a good line of communication with
your manufacturer, that really, really helps out. Because if you don't, things go
bad. We've had manufacturers that ghosted us. We have had manufacturers that
completely ignored us or manufacturers that just replaced components because they
just felt like it. So relationships, super important. Don't hack Chinese New Year. We
tried it, doesn't work. It's a thing. China just sit's down for like two or
three weeks. So it's really, really important to respect that. People buy
tickets to go to their hometowns like months in advance and they're not going to
move it for just like some pesky thing that you're building, especially for like
some small scale thing. So, yeah, don't try to hack Chinese New Year, it did not
work out well for us. Also, do it with a team . While I was in China, I saw a
couple of sole entrepreneurs try to build their own thing and it was super, super
hard, super stressful, having a team is really great, especially if like a foreign
place where you don't really know anybody. Having that team there together, to
support you is super, super important, especially since you can multitask, you
can split responsibilities and do something together. So it's really, really
important aspect. So that's how we manufactured and some of the failures
we've had. Now let's talk about how not to build A.I.. So as we all know, A.I. is
magic, right? Just as Blockchain and IoT and the cloud. It's absolutely magic,
right? Well, the reality is it's it's not that magic. So we decided to have a very
pragmatic approach to A.I.. We said, let's not do anything crazy. Let's just make
something that works. So our very first iteration of a robot was this. This is
like the control panel for a robot. It was super simple. We had a video call coming
in from the robot, on the left over there is literally an iframe, super simple
stuff. And on the right, we had a map, on the bottom we had some controls so you can
move the robot forwards, backwards. It was very, very simple. It barely worked. On
the robot we had our Arduino, Raspberry Pi all running in python and the server was
Java communicating over web sockets. But this barely worked. So we decided, OK,
what can we do? Maybe we can build an autonomous robot and we can both say that
would work entirely by itself. We actually did that. So we built a robot that could
go entirely by itself. It was fully autonomous. And it was actually really
cool. The way we built it is we had pretty beefy computer inside. We had it a Nvidia
Jetson TX2. On that, we were running ROSS and inside of ROSS we were running
TensorFlow and a couple of other technologies. We had YODA for object
detection and some other cool tech that I am not entirely familiar with it since I
didn't write that code, but over here what the robot did is it looked at objects. So
it was detecting objects. It was also measuring the distance to the objects. And
it also had an inference neural network. And you can see that on the top left of
the screen here. Basically, based on trained data, it would know where not to
drive into and it would try to plot a path based on 12 different directions it could
go into. So it had 12 directions and it would go in the direction which had the
highest probability of not colliding with somebody or something. And this worked,
OK. We were able to get like 99 percent autonomy. But the problem is, since we're
doing a commercially viable delivery service, that's like offering deliveries
to regular people and not something in the lab, it really had to do something that
worked all the time. And the challenge with this is we still needed to have
people in the loop. We still had to have people who looked at the robot to make
sure it would actually not crash. And what happens if you have something that's fully
autonomous and people assume it works well, when it doesn't work well instead
of looking at the screen and being ready to take over, they're just looking at the
phone and Instagram. So this approach wasn't the best one. And instead, we
decided to use a supervision approach. So we spent a lot of time building this. So
this is our supervisors console and it's actually really, really cool platform.
It's a platform that allows you to connect to a robot and the robot streams to you
video over Web RTC or like the 4G network and you're able to control it over web
sockets. So the way to work is you'd have a supervisor that sets waypoints for the
robot to follow. So the supervisor would click on the image and he or she would
tell the robot to move 10 meters at a time. So typically they'd set waypoints
every 5 to 10 seconds. It was a very interesting approach. We tried a couple of
different approaches. We tried to do slam, that really did not work out for us. It
took too much resources and it didn't give us a significant gain. We tried other
things as well. We tried traffic light detection. So we tried traffic light
detection. There are some amazing models available online, some great Github repos.
The problem is, yes, they do work on a very clean data set. But when you actually
have data, we actually have a real life scenario where we have like glare, you
have rain, you have weird situations, you have homeless people. It doesn't really
translate that well in the real world. So we kind of struggled with that. Instead,
we actually had a more middle ground approach. So we are able to detect traffic
lights really well, but we're not able to detect the color really well or the which
kind of signal it's giving. So instead, all we do over here, this automatically
zooms in to traffic lights. So it's very easy to see. This video actually that
you're seeing is transmitted over very low frame rate, very low bit rate as well. I
think we're doing 480p at 100 kilobytes a second. So it's very, very low bit rate.
And when the robot isn't moving, we actually make it go black and white and
even lower rate frame rate so that it doesn't waste resources. So yeah, it's
pretty cool stuff. Over here on the top left we actualy have our latency. So we
managed to build the infrastructure that allowed us to supervise this robots from
Columbia for 200 milliseconds and less than 20 milliseconds. So it's like a blink of an
eye. It was a really, really cool technology, it worked or 4G and we did a
lot to optimize that. We had also a map over here. So this map is really, really
cool. A lot of people ask us like, hey, did you do mapping? Did you map out your
environment? Did you need to have something there before you came into a new
place? And well the answer is no. But what we do instead is we actually map out the
network conditions. So we would map out the network conditions of a city and we'd
say, OK, these areas like over here. This is like high latency. We should avoid
those areas because the robot could get stuck there. It is actually very
interesting to see the network conditions change continuously. You didn't have the
same network conditions every day, all day, all year. They'd actually change
every few hours. So it was something that took us a while to figure out.
*Takes a sip of Mate* So, of course, the way this works is we
had two or three people supervising, sorry, two or three robots for a
supervisor in Colombia, and we have just a bunch of people. Typically, students who
would just be working part time and they were sitting in an office in Colombia
doing this. Of course, the press found out about this and they wrote a very small bit
of text in the article saying like, oh, Kiwi hires Colombians and pays them two
dollars an hour. And people were really frustrated about that. We had a lot of
interesting feedback about that. But what was interesting to see is that this
technology actually helps people in Colombia. If you're there, it's a third
world country it's a developing country. You can get a job at a factory. You can
get a job at a textile shop, you can get a job maybe McDonald's. But there aren't
that many tech jobs per say. The biggest employer in the country is a phone support
company. So like when you call in to support line, you get connected to
Colombia sometimes. And that's the biggest employer in the country. So in order to
get like a tech job, it's really, really hard and giving people the ability to, go
and supervise robots it's something that helped them get something on their CV and
help them step up It helped them learn a little bit more about the technology and
helped them progress in terms of their careers. Our lead A.I. guy, he actually
started off as a supervisor and he went up through the ranks and then he ended up
leading the A.I. and robotics team. So it was really interesting, really inspiring
to see how that transition happened. And we managed to get our technology to work
so well that we can do this. *Video of the inside of an airplane is shown*
So we were able to get it to work with up to eight seconds latency, which meant that
you can control it literally from anywhere in the world. So even from an airplane
above the Pacific Ocean. So it was a really interesting experience. And we
really try to make it simple. So in conclusion, for A.I., we realized that the
best approach was to keep it simple. We tried a lot, a lot of different
approaches, like we tried the traffic light detection. We tried a yellow pad
detection. I didn't mention that. So in Berkeley, you have these accessibility
ramps and you have yellow pads that blind people can actually feel them and see them
easier. So we built the algorithm to detect that and we thought, OK, maybe if
the robot is stuck in the middle of the intersection, you can automatically detect
this yellow pattern and navigate to it. It's an approach that worked in theory, in
practice it did not quite work. We tried segmentation. So that was an approach that
worked OK. But some weird things broke it. So for example, any lamp posts or bicycle
posts would crash the robot because it didn't see it. So yeah, keeping it simple
was the best approach, really not going too crazy. And the approach we ended up
going in the end was to have it more of like a driver assist type, like a parallel
approach, parallel autonomy approach, where our robots would help people the
same way that cars would help people stay in lanes or have cruise control or like
with parking assistance. That's kind of the approach we're having. I think long
term it is gonna be possible to build robots more autonomous, it could be
Starship that have some interesting ideas about how to solve that. But I don't think
it's quite something I could be scaled to every city just yet. Another really
important thing is, the lab does not equal the real world. So there are many, many
great examples of fantastic research papers from some great groups and they
were great with very polished, very clean datasets. But they did not work when you
deployed them on 100 robots, there were all different. They all had slightly
different camera calibration that all had slightly different hardware, it all had
slightly different chassis. It did not really translate as well. So these
algorithms, these lab best case scenarios, will need to be modified a little bit.
What else? Yeah, one thing, maybe jumping back to the keep it simple. We decided to
put in a very simple safety mechanism. So the robot actually breaks if it sees
something within 50 centimeters in front of it. So as kind of like a last measure,
precaution, as you saw before, there is a video like you can supervise the robot
from anywhere in the world, but a lot of latency. But having this 50 centimeter
like hard break, actually saves us in case the robot loses connectivity or the
supervisor is no longer able to supervise the robot. So it's always breaking 50
centimeters away from any collision with like a baby or a car or whatever. So the
approach we really thought about is, how can we expand human potential? There is a
lot of talk about A.I. taking jobs or A.I. replacing people's roles, but we sort of
kind of try to do that and it didn't work. Like we try to build robots that were
fully autonomous that went from the restaurant to your door and that didn't
work. People were waiting a very long time. These robots required an obscene
amount of maintenance. So we ended up going for an approach that was far more
parallel autonomy where these robots were like helping people to do more. Same way
the supervisors are getting these assistive technologies where they able to
set a waypoint to do the path finding and the robot does the motion planning on
board. We also had the couriers who would just load food into the robots instead of
the robots picking up food from the restaurant directly, so really expanding
human potential. I think that's where it's at. And over the course of the past
century, we've seen a lot of examples of this. Like we've seen operators of
elevators. Like before, elevators had operators who would make it go up or down.
And now they're fully automated. We had switchboard operators who were there to
connect phone calls. Now we can make a phone call to anywhere in the world
instantly for free. So we're seeing this transformation of work and transformation
of the way things are done. And I think this is just the start. The way I see
these robots is really meshing into the fabric of our societies and solving
physical transportation. Like, sure, you can move bits from anywhere to anywhere in
the world, but can you move atoms? It's really expensive to do that. It's really
hard to do that. That's why I see robots expanding human potential. So.
Conclusions. What we did was really cool and I think it was a cool experience. One
thing that we realized is that tech isn't the hardest part, right? We spent a lot of
time thinking how to build something, but figuring what to build is sometimes very
important as well. And I don't think we spent enough time asking ourselves that
question. We kind of went in all sorts of directions we didn't focus as much on
making the best product possible. We kind of tried things that were really weird and
not well thought out. So like having that more long term thinking, like thinking
what should we build is very important because like how, you can just look up a
tutorial on Google and figure out how to build robots. It's not the end of the
world. One really important thing for us was interaction. So interacting with
people, figuring out how to make the door open, when you actually received your food
was super hard, super, super challenging to do. Actually, the only robot that opens
the door for you. Other companies like Starship, for example, they have a button
that unlocks a solenoid. So it's like the experience is not quite there yet to bend
it down. You have to figure out how the door actually opens. So we spent a lot of
time, a lot of effort in order to optimize that experience to make it as smooth as
possible for people. Also, one thing we didn't figure out is financing. I'll come
back to that in a second. That was really, really hard to do as well. So like tech,
you know, not the hardest, financing figuring out like how to manage cash flow,
super important. But I think the most important thing is to work with a great
team. If you're going to be spending a lot of time with people who you eat, live and
breathe with, it's really important to choose a team that you really connect with
and then share the same passion as you do, because you could be miserable making an
amazing amount of money, but if your with a really crappy team with a high turnover,
it's really boring. I was really fortunate to work with one of the best teams in the
world and over the course of the past two and a half years we managed to do quite a
lot. And just last month we actually got an article in The New York Times. So that
was a really big accomplishment for our team and we got to share it with our
families. My mom was really proud. So a lot of great traction and a lot of great
coverage. But unfortunately, we actually ran out of money, so we kind of ran out of
money last month and we are no longer delivering things. So I decided to leave
and start my own thing instead of doing robots. I decided to do data. So now I'm
actually focusing more on building a tool that helps you tell stories with data. So
this is Glint. This is a data storytelling tool. You're able to drag in some files
and it tells you the story of your data without you having to write any code. So
my hope for this is to allow anybody in the world without any knowledge about how
to wrangle data, how to clean data, how to analyze data, to be able to tell stories
with their data directly from their computers. I'm imagining a tool where you
can say, oh, "In December there were X X visitors to Congress" or "Last summer we
had X X sales" and automatically filled out for you. That's kind of what I'm
thinking about. If you want to join the effort, there is a Github. I'm more than
happy to have any contributors. And if you have any questions or comments we're happy
to answer on Twitter or here in person. Thank you.
*Applause*
H: So, as usual, feel free to line up in front of the microphones or write your
question to the signal angel over there. That already has one. Um, it's all the way
down. Go ahead. Signal Angel: OK. Here is a user of your
service who apparently got an e-mail from you that announced some changes. So he's
wondering, what's up? What you're going, what you're planning to do there, whether
you're continuing your service or closing shop?
S: Yeah, it's unclear. We ran out of funds, so I think the CEO is still trying
to figure out what to do with that. I wish him the best of luck, but I ended up
leaving with a lot of other people. So we have like 50 people in November, now we
have like 10 people left in the country. So it's very ambiguous what's happening,
but yeah, I left. H: Yeah, microphone?
Audience Member 1: No audio. Okay, now it works. I'm a little bit confused because
you are presenting a 1970s concept of a manipulator, because a robot is something
that works by itself, a manipulator is somebody who has some joysticks and moves
things. So it's nothing special. You just have a interlinked Internet link for
manipulator and in the 70s, there were cables. So what's the special thing?
S: Yeah, that's a good question. I think the magic here is connecting everything
together, figuring out for us how to build these robots, how to build a reliable
connection, and how about a platform that works. And as I mentioned, like the how
that's not that interesting. It's more of the what you build. It's that experience
where you're able to order anything you want at any time and get it delivered in
under 30 minutes virtually for free. So that's good.
AM1: So, so far, so good, but evil people could just buy a remote control car, put a
bomb in it, drive under a police car and make boom. And so it's the same use case.
You deliver something by remote control. Audience Member 2: Yeah. You talked about
iterating quickly and rapidly and that's very good model for conceptual stage and
software. Were you in the stage where you were leasing your hardware with your
iterations? Because usually a thick stack of certification has to come in between.
S:So I'm not entirely sure. Are you asking if we got certified at every single
release? AM2: I suppose. Yeah. What level of like
recertification was it totally released. So you had to meet like regulations for
each iteration of that? S: Yeah, absolutely. We didn't really get
certified because we're not building hardware product for consumers. So we're
not selling it to anybody. We're operating it ourselves. So we don't fit under the
same kind of requirements. However, we did have to have some permits. And part of the
conditions that these permits was that we had to meet some expectations. But they're
very, very basic. And there were rigid like an FCC or a CE certification, for
example. AM2: That was the question. Yeah. Thanks.
S: Thank you. SA: Another question from the Internet.
"Why did you develop different applications for Android and iOS?"
S: For the consumer application? SA: I haven't got any more details.
S: We just did. I mean, we had first an iOS application. I mean, 80 percent of our
customers are using iOS. So we really spent a lot of effort like polishing that
iOS experience, making sure that worked. And at one point, our Android app was
working super badly. So we decided to kill it. And everybody was really, really
pissed off, extremely pissed off. So we actually reintroduced it and we started
catching up with features to the iOS version. Internally, all of our apps are
built in React and React native. So we had a common framework for all of our internal
apps, but we didn't have that experience. Where you're expecting the quality of
experience, that we're expecting from a consumer app using React. That's why we
had two different code bases. H: Have tried different methods regarding
perception? For example, lidar, radar and what are your conclusions from that?
S: Yeah, we tried lidar, we tried the cheap lidar we didn't try the really high
end lidar. So the challenge with having like point clouds is that you have to
compute, spent a lot of time competing. We were using a relatively low power device
and it was running from batteries. So we didn't have the luxury of having like 10
GPUs is in the trunk of a car, for example. So that was one approach. One
question. Another question is how much does it cost? So lidars, they can cost ten
thousand, hundred thousand dollars. Our bill of materials was 'round two and a
half thousand. The last versions are two and a half thousand. So all of our sensors
were very minimal. In terms of what sensors we tried? We tried a lot of
different sensors. We tried ultrasonic sensors. We tried near field infrared
sensors. We tried other sensors. Yeah, we tried a lot of different sensors. We are
ended up just going with cameras. So we have cameras. We have six cameras onboard,
all of them full HD. We stitch them into an image on our compute module and then
the supervisor decides which portion of the image they want streams. They can
manipulate with the keyboard to see which portion of the image is streamed. So we
don't stream the whole image. We just stream a part of it. The really important
part for us was to make something that's viable, that can be used commercially. I'm
sure lidar is really cool, but I'm not seeing any commercial deployments of lidar
based autonomous vehicles or robots yet. Audience Member 3: Thank you.
Audience Member 4: You've tried out many different concepts how to do it. And you
saw that your company ran out of money. Do you still believe in the business concept
of robots delivering packages of food? S: Who knows? I think I think it was a
great learning experience. We learned a lot. We had a great team. And I think
we'll see some concept of robots. Maybe not exactly what we were building, maybe
something a little bit different, but I think it's a little bit inevitable,
especially with the rise of self-driving cars. Maybe we'll have cars delivering
packages instead of robots. Not entirely sure what it would look like. I could tell
you, Amazon, they bought one of our competitors dispatch labs. So they're
making a big bet on this. There are two delivery companies in the US, Postmates
and DoorDash, that are building products internally also for... with delivery
robots. And also companies like FedEx are also building delivery robots. And then
we have companies as Starship, for example, which are building robots and
doing B2B with companies all over the world. So, yeah, I think we'll see some
form of delivery robots. I don't know if it's going to be what we had or what
somebody else is going to have. Audience Member 5: Were there any safety
certifications you had to satisfy in order to operate around people?
S: No. So. *laughter* Well, the thing is, in the US, like, it's kind of just do
whatever you want. It's very different from Germany. You can kind of just do
things and you can do them until you get in trouble. So we kind of had that
approach don't ask for permission, don't ask for forgiveness. We ended up having to have a
permit in the cities we operate in. But it was very simple. It was like, OK, you have
to have lights. You have to have a phone number and you can not go in these areas.
That was essentially all the authorization, all the permitting and
certification that we had. H: Yeah?
Audience Member 6: I wanted to ask, did you try other markets? Like, autonomous
driving is very hard, even way more than... manage it fully. So like perhaps
elderly care, like you could use this robots in elder care where you have a
controlled environment where everything is the same. Did you search after other
markets where it's less... S: Yeah, that's a great question. Yeah,
there is a lot of potential for markets like elderly care, for example, also for
mail delivery, for applications inside of factories. We had a couple different
medical companies that reached out to us and like, hey, we want to move items, move
packages inside of our facilities. So we did have a lot of interest. We tried to
keep a focus on the consumer space, like really building a consumer experience that
worked out before branching out into these more B2B approaches. Where elderly care
could be one of them. I think one important thing about elderly care and
services like Meals on Wheels, for example, is that human contact. So I think
people who are maybe not seeing as much of their family, of their relatives, they
really cherish that connection they get from people who deliver them food. So I
think it's a multifaceted approach they have to have. You have a couple of
different considerations with these kind of services for the elderly, for example.
AM 6: Thank you. Audience Member 7: What kind of
personality do Chinese entrepreneurs have? S: *laughs* I think, as I mentioned like,
it's really important to have relationships. So they were very
interesting. They were very deeply in belief of their government. They had
nothing bad to say about it. They believe they would bring them everything... the
best possible, even though they still try to access Facebook and Twitter with VPNs.
So they were very, very loyal to their governments. They were very, very
diligent. If they committed to something, they would usually deliver on that. They
really wanted to make sure you had a good experience. And also what we saw, for
example, with building up these relationships, like the first few times we
talk, they would try everything to impress us. So we got taken to these ridiculously
expensive restaurants to make sure that we were welcomed well and make sure
everything was right. I actually had an interesting episode earlier this year. I
was going to go to Burning Man and then all of a sudden one of my colleagues had
an argument with my manufacturer about whether Hong Kong is another country or
not. And I ended up having to go to China to deal with our manufacturer instead of
going to Burning Man to make sure we're aligned in terms of our beliefs. So,
sometimes it's really delicate. You cannot, like talk too much about the
government there. You can't talk too much about politics. It's best to just stick to
business and, yeah, focus on building a product.
H: I guess this was it. Thank you. S: Thank you so much. *Applause*
subtitles created by c3subtitles.de in the year 2020. Join, and help us!