Tracking pigeons with Python and OpenCV - Neslihan Wittek

9:46AM Sep 26, 2020

Speakers:

Üstün Özgür

Keywords:

pigeon

experiment

behaviors

animal

mirror

delays

track

python

point

extract

humans

cv

data

feeder

questions

head

recording

videos

picking

analyze

All right. So, I guess we can start right in Islam.

Yes.

All right so hello everyone with another session at pi con Turkey, 2020. In this session, Miss Leon witek will give a talk on AI and deep learning. mezlan is a doctoral researcher at Ruhr University Bochum. The title of the talk is birds of a feather flock together tracking pigeons in Python, and open CV. So without further ado, stages. You're welcome.

So, thank you for nice introduction, I'm sharing my screen right now. Great. Okay. So I'm starting my presentation, and I will talk about how Python and open CV helping me to conduct my experiments and also analyze my data, but to go in further details, I would like to introduce, sir. Okay, I would like to introduce myself first. So I'm Leslie Hannah witek, I studied biology at some University, and after a bachelor I studied Environmental Sciences at Wallace University Turkey. Then, during my master's thesis, I was more into animal behavior and neuroscience, but it was not possible back then in my department therefore I decided to move to Germany to conduct my master's thesis. Then I started to work as a like research researcher at bias high culture department for my master's thesis, but later on they offered me a PhD position. And now I'm doing my PhD advisor psychology department there. And right now, I define myself as a behavioral scientist. So the question is what do behavioral scientists do if you will check the past few years like 8070s, what you would see you would see an experimental and experiment area, then experimental would sit and observe the animal behavior and write down which behavior is observed, at which point and then like writing down how many animals are showing this behavior, etc. Here you're seeing an experiment, from cow horn is a rodent universe experiment that the water and.

Okay.

Mr. Maybe if you could, refresh your page.

All right, so we'll wait a few minutes. Okay,

you can hear me right now. Exactly. Okay. Okay, so I'm not using this was no just turning effectful Okay.

So, then I'm just going on.

Yeah sure, could just share the screen as well.

Yes. All right, so you.

Okay, then just.

Okay, so I will go on from this part, I hope you'll hurt the other ones, like what do behavioral scientists do back then you would have an experimental then there would be an experiment area, then the experimental would like write down which behaviors are observed here you're seeing an example from a call home. The experiment name was brought on to university Rodin's had the full access to food, and the water, and about the population was not controlled so like, then the observer was checking Is there any particular behavior is changing from day one to day 20. And what they found here the, the aggressive behavior increase with the population increase month, and also the mother care decrease actually the day the mothers are carrying the offsprings, it was kind of a population control. But in this experimental case that he had to, like, check the experimental area all the time also removed the animals and sometimes recording sometimes striking down which here is for absorbed. But if you're a little bit luck here, like me, what you would do you would record the experiment then later you would analyze your data you would watch your data and write down which behaviors occurred at which point, and how long it took. So here what you see, you'll see that I have different page numbers, also different conditions like I will explain come to this point later, then I'm writing down which behaviors I observe for example shaking behavior or head shape behavior or here, the face crouching with the feet. And I also wrote down like the starting point and ending point of all this, like, single behaviors. So, what I'm doing here I'm actually analyzing the data manually, but analyzing the data, these behaviors many are the slow all, and labors. And you might do some subjective decisions, and also for the others searches other experimenters, it will be difficult to reproduce the experiment plus the experiments results, and also in the former example that I showed you, animal's behavior might be affected by the experimental. Then, if you're watching like, I don't know, like more than one hour to videos, or if you're in the experiment area. There will be some Miss detection because because of the attention, decrease. And that I told you that I moved to Germany for doing my master's thesis in my master's thesis I use this kind of an experimental design this is this name is Skinner box, what I was doing I was putting the pigeon in the Skinner box. And here you will see the packing keys, and I will present difference to me on the packing keys, and depending depending on the pitch and pecs on the smallest and pigeon will be rewarded. So in my experiment design, I gave them their arrival and the fixed late for the stimulus, which means that the stimulus. After picking the variable stimulus pigeon would have, I would get a foot in one second or seven seconds, but the fix Today was a speech and text, after four seconds to put the release. So in human literature, you would say that the variable option is the gambling related option and the fixed option is not risk taking option. So, what I was thinking if I will increase their dopamine level, like, increase in human during the gambling, then they would prefer more the arrival option compared to the fixed option. Then what we did, we did upon morphine injections so morphine is a dopamine agonist that also used in different animal groups, so it doesn't have a harmful effect and after like some hours, it loses its effect but it increases the dopamine level for like four to five minutes, then you would observe different behaviors, on humans also on different animal groups, but of course in my experiment design what I was expecting I injected the pigeon then pigeon will take the option more than I would just show that the humans, the pigeons are also behaving like humans as I expected they're having this gaming, like behavior since they have also a similar dopamine pathway, like humans, but what it happened, like pigeons were picking everywhere but not to picking keys so then

I was watching via the webcam that like they were picking the surface, or the floor of the experiment box not on the picking key. Then I was the experiment was getting the zero for picking on the picking key so normally just pinching on the one to eight to pick when they have the reward motivation, but they didn't, but then we didn't have option to put the touchscreen on all the walls, we have to do the smallest possible experiment designed to get the results. They only had a design like this we had cart box, then the cart box we use carbon paper. Then, on the surface there was a paper then behind there was a white paper. If the pigeon would Peck the wall of the cart box. Then, thanks to the cardboard paper we would get the pegs on the white paper. Yes, we got to pay sexually so you see that life over 2000 pegs was observed for example, in one single paper, like in one condition. But as a master student, what they told me, okay, you had the data now we have over 500 papers, you have to count all this things by hand, then, like, we have to report this results in a paper in a publication. But, of course, it will take over one year and I didn't have that much time then one friend helped me, and she made a treshold for the pigeon pecking, and we analyze via the program and game show that the how the upper morphine injections, increase the like picking grades from day one to, like, a eight, and when you inject, just the normal line which doesn't have any dopamine agonist effect that they are picking directly decreased so this was the first time that I was aware of the contribution and I was decided to go on using like all this metals, for my third experiment. And I started my PhD, and for the PhD the first pilot experiment that I did was about the body environment. So, like what is body environment just think that the pigeon has the wings, but they don't have something attached on their body. And the idea, if I would attach something on their body, what they would do how they will change their trajectory if I would put them on a mace or here you see at the entrance of the experiment box and the end of the experiment box there was a foot to the board, and they have to walk through the maze to reach the second feeder second feeder actually was getting more food than the first feeder so they had the motivation to reach this feeder. But in the pilot experiments we wanted to find out which arm length, are the most suitable for them so we went from zero to 14 centimeter. But, um, so what we had like we try this with temperatures then we had like over 200 videos, again I had to watch all the videos, but then, like when we talk. It's just a part of experiment instead of watching and losing time let's do like color. Color tracking with open CV and trick them and to see like, if they're comfortable, or if they are like suffering, after or stop doing the experiment after like particular length of arm. Then we did open CV color tricking what we did here by we turn to our frames to HSV color dimension. Then we create a mask for color range between the boundaries, then this marks is applied on the image then all the controls are selected. Then we define the biggest contour then we will define this big biggest contract as the track object that I would like

to show you,

as an example, right now, what we did. So here, you're seeing one image from the experiment, then we turn it to HSV color dimension. Then on the mask, the biggest concert is selected as the defined object that it were perfectly actually, like, at the beginning we didn't think about the color of the pigeon cloth that we put on the pigeon. To attach this art, but it worked perfectly, then we track the object and what we found, actually, like we tracked the animal, then we found out that the the most suitable arm length is 10 centimeter for the pigeon. To switch to the presentation. Right now another student is working on a project to let them to use this to teach them to use this 510 centimeter arm to solve the tasks, or to grab a food from the fire position so right now, different people are working on this project. This was also the second time that I was coming at this computer vision and especially open CV will be helpful for us of my PhD.

So, one of the my PhD project is about the newer cipher combination. And so, like, I will show you the animals that actually having this mirrors have a combination. So, for mirror cycle coordination, you will mark test what you would do, you would mark the animal on points that they wouldn't normally like observe without having good reflection, and they you will put them in front of a mirror and watch them and try to find out if they are showing any mirror oriented behavior and then you would conclude that yes they have the neuro cyclical patient ability, and then some scientists conclude this like this is characteristic of higher intelligence and also it's limited to a few animal species so, like, it was like this before but then make pies, and more. Elephants than the chimps and this is a cleaner rest fish than also and they passed is like a mirror marking test. But in my case people never passed this marking test, but it was not failure to use the same methodology for all the animal groups like some had the hands too oriented to the marking but some heads, like on the dislike feet, and sometimes the wings and the ends example is totally crazy that, like, a custom working post but it's literally doesn't make it doesn't make sense to use the same test for all these animals group strike now, scientists are working on this area discussing your block this. And in so since I was aware that pigeons are not passing this marking test. I decided to use different design, I let them to eat in front of a mirror or like in front of their own email, or in front of another patient that is standing in the next compartment and at the same time eating. Also, if you will, so I was recording this so again if I would only watch the videos which behaviors I will alter or which behaviors I would extract was by how long it took for a pigeon from the entrance to reach the foot, how long they forage. If they shop any like aggressive behavior etc but of course these are not sufficient in scientific birth to publish so you have to actually, like, show more data. And then we decided to track the pages so, but for this site we wanted to do markerless tracking because in front of neither we didn't want to be doesn't want them to have this like colorful cloth, because just, we wanted them to be like own to be on their own body shape without having any distraction and just eating so therefore we discovered this depok program is a software package for animals estimation. And what this project does actually. So you create a project and you extract frames from your videos, then you label your data. So, for example, in my case, I label the hat, and I label the bunny like you can also label the like tail or the feet and etc. Then you train your network on this label data. So later, you have to emulate your network performs what you do. So you had to extract the frames, but also the club cup gives you more labelled frames that are already labeled by the train a neural network, if you see that the labeled point from the program is accurate, that it's not like shifting or it so if you track the head, then the point is on the head not on the shoulder or somewhere else on the box, then you can be sure that your network performance is sufficient enough to go further, then of course you don't use all the videos from your experiment to train your neural network so just maybe 20 videos if you are having 10 pages let's say two videos from each page and then you like label 20 frames from each video will be sufficient. But at the end of the experiment you will probably have 200 videos. If you're less for performance is sufficient, and accurate then you will apply this net for on your other meals, then what you will get actually you will get to coordinate data from all this track points. Then later, like, you can plot your results depending on your research interest so here like this was the kind of a video from the dip cup that this is important for your old and how it's grasping the lever. So, therefore, like they track the different like points on the hand, then they extracted the like check points. Also, then they analyze, like how the relation change depending how they're progressing. So in my newer experiment, as I told you I like a track the head, the shoulders and the tail and the beak, what I was interested in in different condition if the pigeon is changing its activity, right, like, because my assumption is, if it will like see another person in the next compartment also at the same time eating, they would get a little bit excited and they would move more than normal so their activity rate would increase then not having any future and the next column part one, then also the head orientation was important for me because I wanted to see, like when the food is over, if the head is too oriented to the food, or when the meter is here, is there so are they still oriented to the mirror their self image, or if they have the other pigeon in the next compartment, are they like oriented to the next compartment or not, this was the like questions that I was interested in what we found out. So, like, when the feeder was in front of the plexiglass, which means that in front of the other, stranger pigeon. There were more oriented to the like pigeon. But in the mirror conditions they were like turning they are behind and trying to not catch them you're late, and also what changed the time the kinds of reach the feeder was increased when the feeder was praised in front of the mirror rather than in front of the stranger. And also, as I, as my assumption, like the activity rate of the pigeons decrease, when the, like, when they have this mirror related condition so they were more seemed more comfortable when they have another patient in the other compartment. Then what I conclude from here. They seem to perceive their mirror image as a different kind of individual rather than a positive. So like they don't pass the mirror, and Mark test similar to humans great x and the magpies, but also they don't perceive them your images are for danger. So, like, then we kind of concluded like the mirror psychoeducation is not a binary test you can't just say that there is a neuro psycho cognition, or not, there's actually a transition zone that, like, again the researcher has to think about the experimental methodology and find out what what are the other steps is between this like there is a motorcycle coefficient or there is another neurosurgical patient. So in the other experiment what I did, I was dreaming pigeon on the monitor.

But just think about it, if I will record you and at the same time, present your own recording on the monitor everything would be okay we are all getting used to school and everything. But if I would give you some delays,

like if I would present your own

image on the monitor let's say after 400 millisecond 500 milliseconds, you will be disturbed, you will think that your brain actually think that something is wrong, but as a human, you will get used to this. But what was my interest, what pigeon. That's like if I'm giving his or her, its own recording on the monitor. But then, increasing delays or decreasing delays, and just like, which kind of behavior might change between the condition. So, in the previous experiment that I showed, I only track the head peak and also one point on the body and the tail and also the feeder but here. Since I wanted to do more complex analyzes, I actually track more body points like the neck the shoulder again, but different parts on the wings also the body, then also for the having a better orientation. Because of the rotation of the recording. I was also like tracking the corner of the boxes, but also I had the GoPro from the OB recording but also the security camera that you see here. I also had a request, because I wanted to see that if the pigeon is directly looking to do, and monitor why it's eating or why it's sending or do they just again like in the mirror experiment turning data behind and like not interested in or kind of afraid and hiding themselves on the corner of experiment back so therefore, this time I checked, I labeled many points on the pigeon bunny. So, we came to this picture again, as I told you, like, I'm watching all the videos, and extracting the single behaviors from the pigeons. Again, like I will be doing this activity right hand orientation, the basic things that I could also be certain, we get get from the get cut but this time, my interest, it is that possible. Also, like, since I have all the trick points all the coordinate is it possible actually extract this from the cups, then. So here actually you're seeing the wise visualization of the track points for identified behaviors, for example, this is the front preening pigeon head is getting closer to the shoulder. Here you see that the pigeon is shaking the tail and here you're seeing that the pigeon is shaking the head.

So, like, um,

based on this we will work on extracting feature vectors by, let's say that area of the head shoulder polygon is getting smaller. when we have this like from printing or like tail polygon plating when there is a tail class tail shaking. Then, we will use as the base for supervised machine learning so this is our

next

phase. So right now, again, like, since there is not really categorization of the pigeon or the bird behavior is detailed. Again, I have to watch all the videos, but then.

Thanks to machine learning and the Diplock caught

on hopefully, hopefully we will extract also those behaviors and in the future. We don't need all this like behavioral single observations directly we can run this pattern on the data then like everything will be easier for at least for me for the like bird people, I hope. And so, first of all, I'm taking to all the pages in my life that they are making everything possible. And also this is a big coloration that we are working all together if is our university and one group from Newsome. So, I mean, Thank you for your attention.

Thank you very much Nestle on. That was a great talk and now we'll take questions from the audience. First, I have a question like, is this still in progress like what are the next steps. So

that the.

So the mirror experiment actually finished now we are working on the publication but the second experiment is like, um, as I told you, we are extracting the

patterns, and then the

machine learning will be applied. I think like this is arm to you for months. Like period of planning. Then, we are planning to write a methodology paper,

showing that this classification coming from duplicate

data for coordinate data is possible because right now, like you would find many a source for different animal groups like rodents, or flies, but for the birds you don't have actually this like categorization that you would actually apply on this coordinate data coming from the flock so we want to be the first but doing this, then the other scientists also might apply.

Like I said, not only like

tree for us because, like, I'm also not coming from the computer science background for me everything. Start like learning from the beginning. Of course, I'm getting help but for me also for a PhD students, if I'm different. if I'm presenting all this in my dissertation I have to understand every single point what I'm doing.

So maybe it could you also elaborate on like that journey of yours like learning programming learning open CV. And, you know, pi Python is a very diverse community, but usually it's like, mostly these days, most people do web programming. Although the, the other major strength of Python is AI and things like open CV so there so maybe you could, could you share a bit how you started

actually before maybe you saw all the tests that I have like all the Python related things so the thing I'm i right now as a behavioral scientist, I'm still not aware of which questions I correct to ask to Google to find out what I'm looking for. But since many different backgrounds people are using Python, that there is a higher chance compared to the other, like approaches that I would find the answer I will find someone actually asking the similar questions, but similar works then someone is answering this like the question so this was the biggest actually like positive side of using Python for me. Yes, for me, yes I have actually split questions, but there is someone else having the stupid questions coming from big problems, and like, but still hoping that person might be helpful for answering all this.

Alright great so you meet arrow asks, he's asking, Are you planning to use open CV for the future projects, or what are the alternatives. Do you have an, if you have anything in mind.

So like, it depends on the future projects like like whenever it's possible yes of course I will use but right now I don't have particular, in my mind, so it's like my last PhD project is this like future reality streaming delay, but in the future we might apply a different paradigm on

the experiment,

maybe I might need some open CV for, like, changing the streaming features of the camera but right now of course I'm not sure. Okay. Okay, I would just my husband is saying that like Python is the new muffler.

Yes, I also agree. Okay, thank you, thank you, Kevin by the way for the help. During the audio problem. Okay, Daniela has a question. He's saying that was really interesting. Thank you very much. He's asking what lengths of delay prevent different species of animal from recognizing themselves. Okay. Did you find very consistent results in the case of pigeons.

So still I'm doing the recording but since I have, I did this mirror experiment that they are not aware of themselves on the mirror.

One of the questions coming from the middle experiment that is

sequency between the pigeon and its own image, might be disturbing for the animal that's why that they're afraid of this like perfect secrecy. Why not, you're not trying to use more interest and giving some delays, like. So, I'm again just. So like,

I'm sure that they won't actually again,

recognize themselves, but I'm thinking they might actually start to the thing that is another pigeon on the monitor so giving some delay will actually decrease the horrible effects of the mirror, and with giving some delays they might be a little bit more comfortable compared to having like the perfect synchronous synchronization. But, yes, but like for humans, I saw for humans, as I told you, 400 500 milliseconds, they started to have some disturbance, for I think monkeys, some space that like 800 milliseconds, they started to get disturbed but there's not really many things are

like going on on this research

on this topic.

Yeah. I have another question. Yep. So with this Jupiter notebooks. How is the development workflow like do you use a ID or do you write in the Jupiter notebook and iterate on there.

Yes it is the Jupiter notebook so like I,

I have them on a condom every day to redirect to the Jupiter notebooks then like again I'm searching, what features I need for my analyzing the data or experimental design then I'm checking which libraries I have to import and importing them then I'm having some tailors because I'm using sometimes they're not sometimes, even though sometimes make book. So it's like a mixture of everything, but yes just basic searching and then typing to Jupiter notebooks.

Alright, so that was a great and very interesting talk, thank you very much for joining the event. So now we'll have a one hour lunch break. And after that, we'll be continuing with a keynote session. This time, we'll be in the stage. Oh, see you everyone in one hour by thanking Islam, once.