Disgusting Secrets of Real Hardware
9:52PM Aug 2, 2020
Good afternoon, everyone. Welcome back for more of the last day of hope 2020. Before we jump into the next talk, we want to send a big thank you to all the attendees presenters and volunteers, whose commitment to hope and to our community have made this event possible in these difficult and trying times. So next up we have Zach Friedman, Zach is a freelance prototype developer in New York City building electronics in the name of void star lab, and has a passion for wearable technology, his talk today is titled disgusting secrets of real hardware, and it's on how to, how to clean up the filth and improve your own designs by closing root shells plugging backdoors walking down firmware designing defensively provisioning secrets and more. He says, you'll walk away a bit savvy or a bit more prepared and a bit more disgusted with your favorite electronics. Remember, we'll have a q&a session with Zack after his talk so be sure to submit your questions via the metrics chat window, and we'll feed them to Zach so that we can hear what he has to say. Now over to us Zach,
take it away Ground Control. Good evening hope attendees and good whenever it is YouTube people, I'm Zack Friedman prototype developer, and today we're making fun of some electronics. Welcome to my workshop in the beautiful West Village of New York, New York. Don't listen to the President. The city is not being sacked by barbarians. That's me wearing an eye tracking headset I built, and this is my workshop at the fat cat Fab Lab. The original plan was to throw an all night party here at the fat cat but no this viral bastard just has to invade our alveolar cells and ruin everything. I am digging this hole like hope over the internet thing. I definitely miss all the late night shenanigans but damn there are so many excellent speakers and we've got all the time in the world to listen to them. For maximum immersion I've brewed my own latte and ran it through the soda stream. It tastes like crap which means I succeeded. Alright, let's talk about filthy hacks, how many of you have ever busted open a piece of electronics to do a hardware hack. How many of you have built electronics yourself from the ground up. How many of you know that I actually recorded this a week ago. Alright, I've been building electronics for about a decade and I've been a freelance prototype developer for around eight years, I do a lot of engineering but I'm not an engineer, I got my degree in Business and Technology at Stevens and I taught myself all the programming and circuits and stuff. I build a lot of projects for clients and myself as my data glove my eye tracking headset and a PC casemod that endlessly generates fake boot text as a professional prototype developer, I build things like concept hardware art pieces custom projects and even the occasional product. My clients are satisfied, but I used to get awfully insecure about my work, my projects were made of copied and pasted reference designs Arduinos running off the shelf libraries modified example code. I just didn't think that real engineers like the ones that look like this would consider my designs to be professional quality. I was a little terrified that one day a client would send my work to a real engineer, and they shoot them an email back and say like did you pay that jabroni, or like maybe I'd forget to close some crazy security hole and some Russian hackers would pull in the crap out of it, it ends up on the news and somebody does some sort of presentation at a hacker conference about it. I really did stay up late tossing and turning. Because I left a label debug header in the release product, or I forgot to set a security fuse or I made so many mistakes and none of my footprints fit the first time. And my heart would sink whenever a client sent me a picture like this, and I panicked like how did I let this happen how could I have caused this time went on, and I got more experience. I saw other people's projects inside and out warts and all I tore open products to modify and reverse them, I read news stories I sat through talks I've listened to podcasts swap stories. And it turns out many of those bonehead design decisions that I bumbled into are not unique. Many of them are actually very common or even industry standard. So today we're going to rip into real hardware with it's been really released into the real world and discover filthy engineering. We're going to see just eye rolling levels of sloppiness laziness corner cutting short sightedness and create creativity. Before we begin, we're going to do three just three points of contacts. A, just because something is disgusting doesn't mean it's bad. Most of what I'm going to show you is deliberate and came from calculated decisions with reasoning behind it. The law these things were done on purpose with intent engineers are generally diligent and good at their jobs, and it's usually the business realities that solely the product, not you know anyone being a smooth brain point be I'm, I really wanted to use my own projects here, but
my clients are not okay with me walking randos through the security flaws and mistakes I put in there. Most of the examples are going to be authentic mass market hardware with picks from sites that are okay with with me doing such a thing. Finally I'm not a cybersecurity expert, I am a prototype developer My job is to implement the security flaws and I don't really know that much about exploiting them. I'm not purporting to be any kind of security expert. So I'm going to take the academic approach, specifically I leave the postage as an exercise to the viewer. Okay, so all that said, let's, let's get rolling. When you think of product design you might think of a racially and culturally diverse group uncomfortable but sensible attire, writing on whiteboards annotating tablets and rotating things in 3d. But okay, you probably don't but it is safe to assume that at least, you know, have meetings, or at least the guy designing the enclosure to hold the circuit board and the lady designing the circuit board itself would work together well know, pretty often they work for different companies. Most companies don't have in house industrial designers, or mechanical engineers, they engage a design firm and mechanical engineers to sculpt the physical stuff, or pick a model off the shelf. And then the engineers designed the PCB to fit the E. In fact, might be an independent contractor themselves, like, you know, like myself, electrical engineers are often left out of the high level design process instead they get an email with a fiddle workstation PC into this. Hey thanks bye, and you got to make something that looks like this. weirdly shaped boards are awkward, and they're ugly, they're harder to design, they're harder to produce, and they're ugly. They also like. They also affect performance they emit more noise, the time goes from improving the design in the schematic into cramming is all those components into that bizarre profile, really thin tiny and irregular boards flex more and they need special programming fixtures and production steps but they, I guess they make the design look sexy and your boss doesn't have to pay the engineers and the designers simultaneously.
Hi, I'm Baka and I set up our matrix and element server for hope. This year, and I'd like to share with you some
good evening hope attendees.
In your finished product. I mean, it's not even like you're gonna take your prototype straight to production. Right. Well, the boss sees that the prototype works fine, and DFM ends up being cut 25% off the bill of materials and go straight to the contract manufacturer So, things that you put in early in the process to speed up your development, end up making it all the way into the finished product. Here's an example. This is my desk, and it features a fifth.
Just look at this board, just look at.
Here's another example. This is a speaker from IKEA, just like, take a look at this board. Here's the Bluetooth audio bit on a daughter board, and the audio it receives needs to make it all the way across the, the, the audio it receives needs to make it all the way across the board around that cutout to the speaker. Why does it need to do all that so that you can ram a cork through this hole and stand it up. aesthetics. You know where ness thinks the best place to put a battery is literally in the middle of the sensor. Just look at this board, just look at it. Last example cameras cameras are the absolute worst because people expect them to look and feel like 35 millimeter film cameras from the 1960s. So boards are jammed in wherever they'll fit with ribbon cables sneaking around components everywhere.
Just like look at this.
Imagine designing electronics for this, you have to cram your super advanced image processing and digital signal processing chips onto a board that looks like a subway map. I also like how they put this heavy duty insulating boot around this massive capacitor, but they use the super fine pitch connector with like no creepage so like a speck of anything conductive with shorted out nice. So every electronic component has a datasheet, this is a document written by the manufacturer that explains how the chip works its specs tables have stats what the package looks like, etc. But it also includes this a reference design, this is sort of like a ready to go schematic. It's a schematic that implements the part that the manufacturer is tested, where each of its features are ready to use. Manufacturers sell evaluation boards and developer kits and third parties so breakout boards and all of these usually implement those reference designs. These are extremely useful in the early stages of product development because you know you're starting with known working vanilla schematics, you don't need to engineer all that yourself. You can buy them pre implemented and just chain the boards together for testing you combine that with some open hardware that's easy to program Arduinos etc. And you know off the shelf modules for complex stuff like wireless, and you can jump right into prototyping without having to start from scratch, the product development process, ideally looks something like this. First you slap a bunch of premade schematics together to sanity check your idea, refine the specs and really focus in on your business logic what the what the thing does. You then put everything on a big roadkill board there's lots of open room, so you can make adjustments and really, you know, perfect that schematic. Finally you take that and get it into the correct form factor, and that's your prototype, you trim unnecessary features, you add protective components you tighten up your design your place these expensive modules you solve problems and you'd think that by the time you're done very little of those off the shelf modules and pre made reference designs would remain in your finished product. I mean, it's not even like you're going to take your prototype straight to production. Right. Well, the boss sees that the prototype works fine, and DFM ends up being cut 25% off the bill of materials and go straight to the contract manufacturer So, things that you put in early in the process to speed up your development, end up making it all the way into the finished product. Here's an example.
This is my desk, and it features a fifth generation, ergo docks mechanical keyboard because I'm a giant nerd. The left side is. left side is suspiciously similar to an MCP 20 3018 io expander reference design for driving button matrices, such as the ones in mechanical keyboards, the right half of the keyboard is literally in Arduino Leonardo that design I was showing you earlier, that was a car stereo mode that I designed for client A while ago. It's an lm 358 amp reference design, combined with an AP 6502 switching regulator reference design and literally an Arduino Uno, it's a bunch of off the shelf circuits and goddamnit I am proud of how well it did the job. It's tempting to think that only small time outfits would build projects by copying and pasting, but that's that is just straight not true. This is an FBI tracking device that Kathy Thomas environmental activist, found stuck to her bumper. It's made of two stacked boards, the top is a ublox all in one GPS transponder module. This was cutting edge in 1999 and has a very helpful datasheet, the bottom board it has a z mix three to 500 megahertz transceiver chip and in RFM rF 1172 filter, and both implementations look an awful lot like the reference design. Actually, it looks like the FBI did change the design a bit on the top side, it looks like they added some capacitors by hand. I just love the idea of some lantern shot FBI agent black suit sunglasses, your piece. Just delicately blowing away flux fumes to see solder this up. Implementing reference designs is a lot like copying code from Stack Overflow we pretend we understand it we hit Ctrl C we hit Ctrl V, and we make pre recorded conference talks about how good we are at engineering. Implementing reference designs, or at least starting out with them is usually a good idea. These their design engineers of these companies like real humans with personalities and families that make these things, these guys will or will are often friendly and will help diagnose issues in your implementation, some data sheets have all though have sleazy reference designs that recommend specific parts from companies that they have relationships with, or they even recommend their own parts, especially the expensive ones for some reason why sell a chip, when you can sell a solution. The point is, reference designs and breakout boards are starting points. As a rule, they're not ready for production, and it's up to the engineer to find out why good engineering should require rigorous testing on the bench and in real life, I mean after all when it comes to breaking stuff, the user outranks the engineer, it's up to me it's up to everyone to find out where those reference designs fall short. And don't get too over committed to them. It's always makes sense to use the right part, even if it's harder to develop over the long term. Speaking of testing, as of 2020 electrical engineers are still incapable of jacking into their electronics projects, I consider this grave oversight and instead they add debugging interfaces to upload and diagnose software and to snoop on critical signal lines and other other critical
By the way, this right here is the best slide in the deck, I wanted a picture of a guy in a VR headset messing with a circuit board, but the best I could find was this was this guy messing with a whiteboard, and he's drawing a cat. I know because he wrote cat. Anyways, the next best thing to jacking into your circuit board, besides jacking into what into a whiteboard is adding a debug interface. So these things like these spy headers just expose the lines of communication between the microcontroller and its peripherals. These make it really easy to connect logic analyzers and scopes and they can also be used to upload firmware other ports are dedicated specifically to the software to diagnostics and programming. This is a j tag header Don't worry about the acronym, because it's just it's basically a programming interface for a microcontroller. It also provides access to debugging tools like setting breakpoints which doesn't like sound impressive if you're used to writing real programs but trust me, this is, this is hot stuff on a microcontroller. Anyways using this port you can burn new firmware onto the device extract the existing firmware Snoop the memory and just generally do all kinds of handy low level debugging functions. You can order microcontrollers pre programmed and this functionality is extremely dangerous right so it makes sense to remove this header when you go to market, especially if it's one of these fancy headers that just accepts this convenient spring loaded programmer, otherwise somebody could say, take your commercial off the shelf hardware Jabba convenient spring loaded programmer into into that header suck out your firmware steal your secrets, modify or firmware to make the device catch fire and re upload it. In the past 70 years of electrical engineering real engineers trademark have made many powerful anti tampering countermeasures. You have access prevention right once fuses code signing and code integrity checks, they're increasingly common and easy to use. That said, I don't think I've ever used them and I don't think I've ever met anyone who used them. You might you might think that it doesn't get any more nauseating than a convenient unsecured port with firmware access. So allow me to direct your attention to this little bastard. This is a serial port and it's even more common than a CI tag header and in my opinion even more dangerous. All serial ports have at least three pins you have a receive transmit and a ground. If you've ever used an Arduino, where you had a computer in 1995. You might have heard of a serial port and that's exactly what these are. What's neat about a serial port is that they're often used to communicate with humans, not complicated devices and they communicate in plain text. All you got to do is get a USB to serial adapter, and you could barge right in. This is an ftdi basic from sparkfun and I highly recommend it but there really are like a bazillion USB to serial adapters out there and they do the same thing. I keep all mine in an altoids tin so they stay fresh, a synchronous data, collect the whole set, it's really important by the way to to first use a multimeter to figure out what voltage it is because you know you connect a five volt, you five volt serial adapter to a three volt device and you're going to need a second device. Anyways, you just look for three to six parallel holes on the board and that's probably a serial port. You'll often find multiple headers on a board, if the device has multiple microcontrollers which which is increasingly common one of these holes will have a little Starburst around it, or like an extra thick trace where it connects to the ground plane. And that's, that's your ground, you might need some guesswork to find the receive and transmit pins you really that just comes down to try out the various combinations until something works but sometimes you get lucky and a friendly engineer will do you a solid and label it right there on the board.
What do you think we'll find on this cereal for it to stream maybe some debug logs, some core dumps maybe some printf style status strings leftover from development. Could it be a little bare bones command line to access hidden functions and factory diagnostics. Maybe it's an encoded channel for technician to connect some super secret maintenance tool. Or maybe it's a bash shell with root access and default username and password and you bet your ass that's what it is. Every embedded Linux distro I know exposes a shell on its dedicated on that dedicated serial port. Usually that command line is root because, creating, you know, separate accounts with reduced privileges is hard. Usually the username is root and the password is root or its admin or its password, some fancy devs change the password and, like, that said, I've never seen a manufacturer provision unique passwords onto the devices, even though they totally should, at best, they'll derive it from the serial number, or MAC address, and if you find that algorithm like you're in like Flynn. Many manufacturers will change every one of their passwords to the same password and then put that password into a manual, that's available for download NDA free from their site. This is especially common for routers and other devices that should really be secured better than this. You might have been smart and like set your SSH server on the device used to not use past keys and only certificates, but you should know that this has no effect on the hardware serial port. I don't actually think you can set that to require a certificate and not a passkey. Anyways, like you just take that device with crazy hard security login install a new private key and you now no longer need to sneak into the boiler room to mess with the HV AC. This brings us to the filthiest crevasse of the embedded world. I must stop showing you pictures of circuit boards because this fellow miasma is invisible. It's firmware, and it's the libraries that these systems are built on trigger warnings unpatched vulnerabilities decade old kernels unencrypted networks and literal fire. You guys are pretty plugged into security penetration and you're probably aware that embedded devices are notoriously insecure. Now much has changed in preparation for this talk, I went to showed in ramie Mel Rami Malik's search engine of choice, and I did a search for dropbear choppier is a super lightweight SSH server that's included with BusyBox which is in turn included with most embedded distros. The overwhelming majority of these devices are running Linux 2.6, the latest version is 5.7, and I can't imagine how many of these have been how few of these have been patched for shell shock and and all those other vulnerabilities we've seen lately. Yeah, this one was my favorite this poor Canadian bastards cable box is running dropbear SSH version 20 1255, as in version 55 of the year 2012, the last time this guy's cable box was updated Barack Obama was beginning his second term. There have been some security up there have been some security improvements since then. Here are some iomega NASS devices that backup people sensitive files. I omega was kind enough to announce to the open web that these are NASS devices presumably full of sensitive files. These things support only TLS 1.2 which I have read leaves them vulnerable to the golden doodle variant of the poodle attack. I don't know what the hell you guys are smoking when you name these, like, when you name these vulnerabilities. But this is a device designed to keep your files safe and up to date, and I omegas devs have made them unsafe by not updating them. But surely it's, it's better in industry than it is in consumer products. I mean, these are devices that are going to be in the, they're going to be in service for decades, not something that you're going to like throw out when I don't know when Steve Jobs, I guess, back in the day when Steve Jobs got on stage. Anyways, there are a lot of industrial control protocols used to coordinate heavy duty equipment. These things have failover as backups and, like other useful properties that come in handy when your software can kill people,
or if you've hacked a car and come on like it's 2020 who hasn't hacked a car. You're familiar with the CAN bus. There's also BACnet, which is used to connect like HVDC and building safety equipment, and you have the mod bus which is used to connect heavy equipment and factories and processing plants and stuff. As a rule, these protocols are unencrypted, which, which isn't as bad as it sounds like if you're connecting a conveyor belt to the control console. Like it's your own factory you don't you don't need to. You don't need to secure it that bad it's not like you're hooking your industrial control equipment into the open web right. If you have dubious morals and you want to control 40 relays in Romania, then you're in luck my dude. See the mistake wasn't putting the Ethernet port on the controller. The mistake was putting the other end of the ethernet cable into a router. Here's another intriguing device in clustal novo De Soto Italia, Italy. What's neat about this thing besides the fact that is probably an Italian temperature controller that you know shows up in a search engine is that invensense supplied the safety devices that were hacked in Saudi Arabia. With that super advanced Triton malware. I'm not sure if this is the exact device we're talking about here but you get the idea. You connect the interest industrial equipment to one end and you connect the other end to a direct link to everyone on earth with a command line. treyton was this malware that disabled gas detection safety systems and a chemical refinery. presumably with the intention of flooding the facility with toxins, through pure chance. It was caught and forwarded but hadn't succeeded trading would have been the first malware with casualties, they really should have kept a better eye on these key switches. It was what's interesting is that in the Triton attack, they actually did prevent these things from connecting online but they had poor internal controls. So, somebody who got access to the facility was was easily able to do a firmware attack and break that security wide open. Moving on, let's set some stuff on fire embedded devices increasingly have consequences, like not only are we sticking processors and everything but we're hooking these processors up to more and more subsystems this fire here was caused by a firmware attack this USB power bank was connected to a USBC quick charger, there's a microcontroller in charge, so to speak, of the charging subsystems so that it can so that the device can demand as much current as it can handle this attack is launched on the wall or by using it to charge an infected phone. The phone puts the charger into device firmware update through that USB port, and then it DFU is on a hacked firmware that instructs the power management chip to wait for the target device, and then run 20 volts through it. This means that a malware attack can blow your phone up, neat. It's a good thing that these, these wall wart manufacturers aren't just copying and pasting pre written firmware onto an unmodified reference design. If they did that. The attack would hit tons of different devices from different manufacturers and it would be a total nightmare to patch out of existence. Anyways, this is where we are. We cargo cult schematics we leave all access debug ports open to the public, we deploy outdated software we never update it, we open it up to the, to the internet. We connect more and more of the circuit to the hackable part, and we make ugly boards. These are just some of the disgusting secrets of real hardware. So, what do we do about it. I mean, as a rule, designed defensively. The bad guys are gonna buy some units and then they have all the time in the world to bust them open, so don't make it easy clean up after yourself. Get rid of those debug ports prevent one part of the circuit from damaging another use fuses and thermistors to limit hackers to limit hackers reach. Don't blindly trust reference designs force those first any bad actors to like have to start desoldering pump parts, get yourself involved earlier in the process so you have more influence on the look, feel and configuration of the prod of the project. Don't put something online or even even add provisions to bring it online, unless it's absolutely necessary. There are tons of counters there tons of measures like intranet private IP ns and just straight, not putting a conductivity on it, that can provide a lifetime of protection and finally use the anti temporary devices that are already available to you, like it's kind of a pain in the butt but you know like life sucks wear a hat. As for the hackers the makers and the modders, like, don't be discouraged by how sophisticated and polished other engineers work looks like that cat that Gizmo might look really slick on the outside but on the inside, it's just full of disgusting secrets, and they're waiting for you to unearth them.
Thanks for watching. I'm Zack Friedman, and I really hope you've enjoyed my presentation, and the entire virtual hope conference this has been a blast of research and and record and I really, really hope you guys have had a great time over this past week. Huge thanks to the organizers, especially Bernie for letting me do this talk, and thanks Brooke for being a most excellent camera person, and of course thanks to you. You keep you keep hope alive through 2020 and far into the future, and I'll see you on the other end of this pandemic. And when we do I got a cold beer with your name on it.
You take care of folks. Oh,
now it's time for QA, I am going to now time travel into the future and switch to live mode. Thanks
Zach, those are some great tips as a great presentation. So tell us about what got you started in the hardware design development space in the first place.
call me stuck what got me started building electronics.
So I had an internship.
And I had a lot of free time in the internship.
And I saw this hackaday article about this guy made a heads up display and like a wearable computer like this is the coolest thing ever. How hard could it possibly be.
It was hard,
but like I just really liked finishing, I just really like building stuff and really discovered
that I was, I was pretty good at it.
So I started a hackerspace, which, but for the record, you should probably like, you know, build hardware for more than a few months before you start a hackerspace, but
I just got so deep into it enough people are coming up and asked me to build stuff
that you know became a became a job. So, did you start the hackerspace When did you join when that was already there.
It was so, by the way, the hackerspace, they're still open. It's the Hoboken maker bar, no longer remember because I moved to New York City. But I was one of the founding members, I wore this you know janky heads up display made of a coat hanger and a video headset to a meetup and someone's
like, hey, like
you'd be interested in hacker space and I'm like, I don't
know what that is but
And we started it.
Excellent. So some questions from the audience here. Are there any manufacturers you found are better than others and not attempting to lock you into a company centric reference design. Oh, this is an excellent question.
This is a great question generally about the more specific, I guess the more specialty the manufacturer is, the more likely they are to make it play nice with others simply because they don't manufacture a huge variety of products. Personally I like anything made by linear technology like they make power. Power like regulators and various power electronics and they're really good about recommending high quality and low cost components. That's one of the ones that springs to mind. Generally the closest, the more specific the part is, the more
more the more open they're they're more open flexible that their stuff is I think at the end of the day though like
it's hard to fault t like I believe ti a little, it's hard to fault ti just because they make like literally everything. I think one particular offender is probably my crew, it could like maybe microchip if I have to sling a little bud because they recommend parts that
they recommend parts of companies they've recently acquired I'm gonna leave it at that.
All right, another question for you. What is the most common component or circuit you've been told to cut from a design you thought it was a huge mistake to draw. So
it depends on which depends on which
stage of the design it is the most.
The one that was missing. The ones that are mistake.
They're often what they're often components to like sort of protect against bad behavior right like.
Pull it you know reverse polarity diode to keep you from putting batteries in backwards or protect you like protective components on,
like, what are some,
like Zener diodes and
stuff on. Unlike plugs in case somebody like puts voltage into an output. And like the thought process is like no one's ever gonna do this, but
and we find out about it very quickly. Generally when I think something like this will happen. I will add a provision so like for the diode, for instance, I'll add a way for it to be like for it to be bypassed like with a short. I just do whatever I can to make sure that they can then add that part later once they come to their senses, the parts that should be added early, that are not is a bit of a trickier question and that is probably probably the right component for the job. A lot of the poor designs I end up working on have come that way because the oftentimes a founder or someone early in the project will be using parts off the shelf, like you know pre made boards and stuff and they will insist on using those exact parts through to the prototype or production. So, ended up with like a bunch of Arduinos and a bunch of breakout boards, a bunch of stuff on the breakout boards things end up with way too many LEDs which drain a lot of battery and I have to explain that. What let's see one other other thing I can think of is when they downgrade the processor.
Never let your clients realize that there are cheaper processors that have the same footprint that they can just drop in.
As you design around you know having a certain amount of memory and peripherals available and it was a while I get a. I'll get a problem and I'll be like, Wait a second. Why is my. Why is AVR dude saying that the signature of this chip is different.
Alright so we've got one question, I know the technical question for the, for you there. I'm not.
Well let's hope so. So how do you know the setting simple user password combos on cereal are a human error or human laziness and not an attempt at something different.
Oh, I guess like.
I mean, like, think of the attack surface here like the nefarious thing is did this person basically did this person leave themselves a backdoor.
If I mean if you want to leave yourself a backdoor, like,
oh my batteries or my cameras running a battery Oh no.
If you want to leave yourself a backdoor like what
you would probably
set a password that doesn't allow everyone in their dog to get in, like, you know, change it to 26 random alphanumeric characters and write them down on a post it note and take the post it note if you don't change everything to root. At the end of the day, though, you got like the golden rule never attribute to malice we can, we can attribute to to laziness or stupidity.
So from a manufacturer security standpoint, do you have any suggestions on how to ensure machines in the wild can stay current or is it too often simply seen as an externalized cost.
Keeping, how to keep how to actually solve this problem and keep the devices. Keep the devices correct.
So this is actually a double edged sword one thing that I was considering putting the talk, but kind of cuts muddies the waters, is that the method that you use to update the firmware is itself
an attack vector,
like anything you can use to update the firmware,
a bad guy can use to update your firmware.
But that said, like,
this is better to have the spare to have a need. I mean,
some sort of cook. You're gonna need some sort of some sort of device firmware update system, but you want one that can do an atomic update, and you want one that does that has signed code, even better. Basically what these what these two mean is, if you want to do if you want to you want a firmware update system that downloads the whole package of firmware attempts to upgrade and if it fails it rolls back. Otherwise it's very easy to get your device bricked.
And you want to use sign code.
You want to use something cool because it provides a level of protection against somebody compromising your server. That said, once you like you need a mechanism and the mechanism has to be hands off, it has to do it on its own because users are not going to push users are not going to push the Update button. It has to do
on its own. I would recommend if possible, have us
build yourself in a channel that can be used to push emergency patches. One solution I've seen that was very clever is at the time of perfect at the. When you're provisioning at the factory you provision a number of keys less, let's say five private keys onto the device and those can be used to sign high priority updates,
which I thought was pretty interesting,
but you do need a mechanism to provide it personally. The way I like to get around this is, I like using Python to write stuff and Python is just a bunch of scripts, and just copy in a new script.
So what sort of tamper resistance mechanisms, would you say add the most protection.
I mean, the big biggest, biggest thing is just take the damn debug port off,
Often, a lot of these attacks are simply like somebody like is opening it up because they're curious they see. Oh, I see four holes in a row I wonder what I can get out of this, or you know you're looking or they're looking for just a way to do damage, they just want to get into a router or something of the sort. This goes double if you're deploying the device on mass and institutional roll. Like, if you're variety like let's say you're Verizon and you're selecting the router that you're going to give to all of your customers like Please don't make a science easy to reverse engineer,
a pass, pass
pass this just
not having internet connection is another good one with cutting the attack surface down forcing someone to actually spend money.
There are other very, there are other more effective and more exotic systems like they're there, there are high security chips that have like Ingress, and it's not really Ingress detection I forget the industry term basically they detect if someone's trying to suck the code off. And they blow themselves up, but that's just that's massive overkill for for community or for a for like a consumer product. On the whole, though, make that make life harder for bad guys, and just encourage you to move on to target.
We have a couple
of folks that
make them harder for bad guys, and some question came in asking about what about the open source model right What about open source software and hardware, and make it easy for people to interact and tinker with things, how do you make that balance between, you know security avoiding attack vectors and encouraging tinkering and improvement.
This is, this is like,
I don't care.
It pains me to admit it but this is what makes hitching open hardware to clients difficult sometimes they think it'll cut their security, getting more eyes on the project, almost always a good thing. The main difference is that when you're when you're making a project hackable, you want to make it easy to work with on a bench, hard to work with in Romania, like, you want a person who has the device on their workbench to be able to mess around with it really easily, but you don't want me sitting in Manhattan, to be able to do those same do the all that messing around remotely. So this comes down to like, provide like honestly biggest thing you do is just make the dangerous stuff opt in raspberry, the Raspberry Pi foundation to their infinite credit has disabled telnet and SSH it forces you to change the password by, by default, they're on their, their largest distro, which I am sure has plugged secure hundreds of thousands of security holes across the world.
Honestly, that honestly like best thing to do is
another another thing you
can do is honestly like don't license, the thing commercial, because you don't want people to take a system built for developers is very easy to hack and just copy and paste it into production because they'll take all of those flexibilities with them. So, forcing them to do a little forcing someone to do a little work is one possibility, like
a hard copyleft license.
What about but
that's it at the end of the day, at the end of the day, open, I gotta make this absolutely clear open sourcing and open hardware is always a net positive you get more eyes on it, it massively increases the chance that a friendly person is the first one to break in. So,
what about dipping the board in epoxy someone
potting and confiding in conformal are ways to deter all but the most serious access. You often you often see these in military you often see these like military settings, automotive, and things like that. This is all right so don't at the end of the day, nothing is going to stop someone who's really dedicated from getting in there. They can find solvent that dissolves that they can mill their way in their their ways then what what that where that works best is for devices where it's not practical for them to get another copy of, like, it's great for an aircraft blackbox because if someone's trying to reverse engineer the aircraft, it's unlikely another one's going to crash in their airspace. It's a little less useful for your for your device. It's one of the challenges with those is that it makes testing and provisioning harder and it makes the whole process more expensive. I would say that if you already have a use for the. If you already have a use for the other benefits of doing so such as like weather proofing. It's not a terrible idea. It is it's an expensive production process and it also means that you can't do any sort of rework or refurbishing that
that said though like
in small numbers I've
planted projects before and I've done some conformal before.
Some of these devices, by the way. Some of these devices have another failsafe in the form of some sort of light sensor, or even, even a, an explosive charge that detects when the epoxy is being removed and it destroys key parts which is really cool. Anti tampering mechanisms are incredibly cool and there's very little reason for any one of our caliber to
worry too much about them.
Tell us more how the explosive charge piece works.
I, I am not I'm definitely not an expert at this, but I'm definitely not an expert at this but at the end of the day, it's just about you want to destroy the element that has the secrets in it. Sometimes it's just sometimes just in the form of literally like a blast like a tiny blasting cap. But other times there's a circuit that charges a capacitor, if generally if the light sensor is triggered and then it sends high voltage through the through the part that you don't want to survive the process also like back in the back in the day you'd have back containers just make had these like light sensitivity from, I don't want to talk out of my ass. So,
can you tell us something about commercial devices with cots components like fine Arduino or an ESP 8266 new device is that common who
Often, so if you find these,
like, sort of hacker devices carrying over into, into like a mass manufacturing crowd project.
I mean you don't see that quite so often because just because of economies of scale but like if someone's making like a few 10s of thousands of something, you know the difference might not be that crazy, you see that a lot of like Kickstarter type projects that are basically hardware that's put together with a small team and funded through bootstrapping and not through like not through like a huge like huge investments, just because like they just had no point Do they ever have enough money to do this massive redesign effort for a low cost low cost part.
Honestly though now that Noah's cortex and risk five processor is basically like. You can buy the same process, sort of the same core for multiple different manufacturers, you see this less and less because it is coming more affordable and processors are becoming a little more interchangeable. Back in the day, though, you saw a whole bunch of stuff, ESP 266 is is a notable exception for the NRF 52 as well you find
fun of Nordic parts that are very hackable all over the place just because they are both cheap and easy to work with.
you don't really see what we notice though cuz I mean the parts are just kind of expensive. I
wish I saw it more often though.
So tell us, tell us about the worst piece of hardware you've ever had the pleasure or pain to design.
I need to filter this by the ones that I think I can actually talk about
because of because of non disclose.
Honestly the funniest ones are the ones that I turned down. Just i
get i get approached.
I don't I don't want to sound me because like I get approached by a lot of folks with big ideas who just want to just want
to verify whether what they think of is possible.
lot of folks want to make like flexible electronics and I have to explain that a battery you can't find a flexible or transparent battery. People, folks. I had somebody who wanted to make like poker chips that, like, they wanted to make poker chips that could be tracked using RFID and I had to explain that like you can't read a stack of RFID chips.
Some things are just some people like,
I don't know, I, I guess
I could have prepared better by suing people would ask me this question but I definitely have some goofy idea, I definitely have some goofy ones.
I was approached early on
I was approached by dance troupe who had a dancing robot that performed in their troop and the guy who knew how to program a robot had left. And they needed me to repair and fix this dancing robot. And I ended up having to program new choreography onto it so I guess I've sort of made a dance.
feel some projects with electron guns, those are a little. Those are a little dangerous.
There are some sex toys
that one of the challenges of being out when the challenges of being like a consultant
is that a lot of your stuff is under non disclose.
All right, what about your favorite one. Now, even when you're most proud of it you can talk about
favorite project. So, I mean the favorite, my favorite project that I've ever built.
It's probably this is probably this data glove that I built.
I built this I built this data glove and it lets me draw letters in midair and it uses a machine learning. Neural Network thing to recognize it and type it out so I can control a wearable computer with it.
that was kind of cool. But honestly, I
mean, favorite one was probably probably a Nerf blaster, just because there's, there's just
nothing more satisfying than tagging somebody 150 feet away with something you've built yourself and you see that dirt ceiling over the horizon smacking zone being like, I printed that thing.
So, we're coming up to time but I want to ask you before we finish up. Do you have any recommended resource for people just getting started in hardware. But what are those key things you pointed to
ah biggest, biggest thing to do. Alright so, two things that I recommend the first is jump right into a project.
Get into a project, hit a wall, do a lot of work, figure out how to get past it hit another wall.
You will not learn faster or figure what questions to ask faster. The other thing don't trade time for money. When you're just getting started or when you're just making idea happen. I don't like order parts that take a month to get delivered and just do whatever it takes to get the project done now I do see some fades but a cool project lasts forever. And finally, look at other people's projects and don't be too insecure to ask folks for help.
Excellent. Well, thank you so much for your talk today Zach, it was just amazing and all the answers to the questions. On behalf of hope 2020 and all our attendees and presenters volunteers. Thank you. So, with that come back at the top of the hour for our next talk borders and biometrics boundaries of computer vision with Charlie Myers. Until then, keep on hacking. Take
it away hope ground control.