Monitoring the respiratory behavior of multiple cows based on computer vision and deep learning

Automatic respiration monitoring of dairy cows in modern farming not only helps to reduce manual labor but also increases the automation of health assessment. It is common for cows to congregate on farms, which poses a challenge for manual observation of cow status because they physically occlude each other. In this study, we propose a method that can monitor the respiratory behavior of multiple cows. Initially, 4,000 manually labeled images were used to fine-tune the YOLACT (You Only Look At CoefficienTs) model for recognition and segmentation of multiple cows. Respiratory behavior in the resting state could better reflect their health status. Then, the specific resting states (lying resting, standing resting) of different cows were identified by fusing the convolutional neural network and bidirectional long and short-term memory algorithms. Finally, the corresponding detection algorithms (lying and standing resting) were used for respiratory behavior monitoring. The test results of 60 videos containing different interference factors indicated that the accuracy of respiratory behavior monitoring of multiple cows in 54 videos was >90.00%, and that of 4 videos was 100.00%. The average accuracy of the proposed method was 93.56%, and the mean absolute error and root mean square error were 3.42 and 3.74, respectively. Furthermore, the effectiveness of the method was analyzed for simultaneous monitoring of respiratory behavior of multiple cows under movement, occlusion disturbance, and behavioral changes. It was feasible to monitor the respiratory behavior of multiple cows based on the proposed algorithm. This study could provide an a priori technical basis for respiratory behavior monitoring and automatic diagnosis of respiratory-related diseases of multiple dairy cows based on biomedical engineering technology. In addition, it may stimulate researchers to develop robots with health-sensing functions that are oriented toward precision livestock farming.


INTRODUCTION
Respiration is an important indicator of animal homeostasis and one of the most basic vital signs of animals, along with electrocardiogram, blood pressure, blood oxygen, and body temperature measurements (Dias and Paulo Silva Cunha, 2018).Its most intuitive manifestation is the regular reciprocal movement of the animal's abdomen that accompanies the change of respiratory state, hereafter "respiratory behavior."Monitoring a cow's respiratory behavior is not only the basis for analyzing respiratory rate but also a technical prerequisite for the automatic diagnosis of respiratoryrelated diseases, such as heat stress, based on biomedical engineering techniques (He et al., 2016;Fournel et al., 2017;Gargiulo et al., 2018;Ferraro et al., 2021).The activity area of dairy cows on the farm is large and the environment is complex and changeable.As social animals, cows tend to stay with their companions.Today, the proportion of standardized and large-scale dairy farming is growing steadily.How to monitor the respiratory behavior of multiple cows is a problem that must be considered in the information perception process of modern unmanned dairy farms (He et al., 2016).
Automatic monitoring of a cow's respiratory behavior is not only the basis of a health assessment but it also helps optimize the feeding scheme and improve animal welfare (Buczinski and Pardon, 2020).Monitoring methods that rely on visual appraisal are time consuming and labor intensive and easily cause visual fatigue (Milan et al., 2016).Previously proposed solutions focus on contact sensors, which have drawbacks such as animal stress response, short component life, and more restrictions when deployed on a large scale (Milan et al., 2016).Noncontact respiratory monitoring methods could effectively avoid these shortcomings and represent an important development trend (Stewart et al., 2017).
Compared with other noncontact monitoring technologies, computer vision has been widely used for livestock and poultry size measurement, target detection, segmentation, behavior recognition, and animal product quality inspection because of its advantages of low cost, high efficiency, and rich and stable information (Bezen et al., 2020;Cominotte et al., 2020;De La Iglesia et al., 2020;Kang et al., 2020;Liu et al., 2020;Sarwar et al., 2020;Tsai et al., 2020;Weng et al., 2020;Baek et al., 2021;Jia et al., 2021;Tassinari et al., 2021;Dohmen et al., 2022).These studies promoted the application of deep learning and computer vision in agricultural engineering and improved the automation level of information perception in livestock and poultry farming.However, the monitoring of respiratory behavior of multiple cows is still challenging, which limits the development of automatic monitoring robots in precision farming (Borchers and Bewley, 2015;Britt et al., 2018;Asseng and Asche, 2019;Lovarelli et al., 2020;Ren et al., 2020).
Respiratory behavior monitoring is difficult in dairy cows because they have poor obedience and are easily interfered with by less controllable interference factors (Jagadev and Giri, 2020;Jorquera-Chavez et al., 2020;Wu et al., 2020;Larsen et al., 2021).With the continuous development of computer vision and deep learning, researchers have reported a series of achievements in livestock respiratory behavior monitoring (Tang et al., 2015;Stewart et al., 2017;Jorquera-Chavez et al., 2019).At present, these methods are mainly based on 2 physical characteristics caused by breathing: (1) using visible light imaging to monitor abdominal fluctuations, and (2) sensing temperature changes around the nose based on thermal imaging technology (Zhao et al., 2014;Tao et al., 2017;Barbosa Pereira et al., 2019;Lowe et al., 2019;Song et al., 2019;Wu et al., 2020;Kim and Hidaka, 2021).
Among the above-mentioned respiratory behavior monitoring methods based on computer vision, methods based on infrared thermal imaging technology had better performance but were expensive, which greatly limits their large-scale deployment on dairy farms (Mateu-Mateus et al., 2019).Moreover, most focused on monitoring a single cow.The rapid development of multiple object monitoring technologies in industrial tasks, such as size measurement (Okarma 2020), yield monitoring (Ribeiro et al., 2021), defect detection (Laucka et al., 2016), and automatic driving (Kanchana et al., 2021) in recent years could provide important technical references for multiple object information monitoring in livestock farming based on computer vision.For dairy breeding, multiple object monitoring research mainly focuses on individual detection (Qiao et al., 2019;Tassinari et al., 2021), enumeration (Xu et al., 2020b), and identification (Andrew et al., 2021) and other information perceptions of population size.In contrast, health information acquisition, such as body inflammation (Cernek et al., 2020;Zhang et al., 2020), limping (Jiang et al., 2019;Kang et al., 2020), and digestive disease (Atkinson et al., 2020) still depend to a large extent on specific image acquisition space or equipment for single-target monitoring.Compared with other locomotor behaviors such as limping, breathing in cows is weaker and thus harder to observe.In addition, the more cows captured on video, the fewer pixels a single cow will occupy, which makes respiratory monitoring more difficult in settings with multiple cows.
To solve the above problems, in this study we propose a method for monitoring respiratory behavior of multiple cows based on computer vision and deep learning.Based on recognition and segmentation of multiple cows by the You Only Look at CoefficienTs (YOLACT) algorithm, each cow's resting state (lying resting and standing resting) was further recognized by integrating the convolutional neural network (CNN) and bidirectional long-and short-term memory (Bi-LSTM) algorithms.The corresponding respiratory behavior monitoring algorithm (lying or standing resting) was applied to monitor respiration according to the behavior recognition results.

MATERIALS AND METHODS
In the resting state (lying resting and standing resting being most prevalent), respiratory behavior is less affected by movement disturbances, so it could be a better representation of actual physical health than the movement state and more instructive for subsequent physiological health assessment.The resting state means that the cow's position remains relatively constant and there are no body actions with large amplitude and movement such as walking, running, and drinking.The test videos in this study were collected from July to August 2020 at Keyuan Dairy Farm (Yangling, China).The farm is equipped with an automatic cooling system and the cowshed temperature is kept within a reasonable range (24-26°C).During the data collection period, the average minimum and maximum temperatures were 20°C and 28°C, respectively, and the relative humidity was 60% to 70%.The subjects were Holstein cows in the mid-lactation period (101 to 210 d after delivery) in an outdoor exercise area.Figure 1 shows a schematic of the test video acquisition pro- cess.An HDR-CX290 camera (Sony Co.) was fixed at a vertical height of about 4.0 to 5.0 m in the outdoor exercise area.A total of 94 videos (approximately 72 min in total) were collected, each of which lasted for 6 to 70 s and included various illumination, targets, and states (lying resting, standing resting).This study was an observational experiment that did not involve experimental animal procedures, so Animal Care and Use Committee approval was not required.
Thirty-four videos were randomly selected and decomposed into sequence frames, from which 4,000 images were chosen, and each cow target in the images was manually annotated using Labelme software (https: / / github .com/wkentaro/ labelme).In this case, the images were usually zoomed in during labeling to ensure that the cow's outline could be labeled more accurately.To recognize and segment multiple cows, these labeled images were divided in a ratio of 7:3 for fine-tuning and testing of the YOLACT cow segmentation network in this study.
Table 1 shows detailed information about the 60 test videos.Although changes in the weather and wind speed increase the difficulty of respiratory behavior monitoring, this variability is useful to verify the robustness of the proposed algorithm.The recorded wind speeds under windless, breezy, and gale condi-tions were 0.3 to 1.5, 1.6 to 3.3, and 3.4 to 5.4 m/s, respectively.
The video processing platform used in this study was an Intel Core E5-1620, with a 3.50-GHz processor, 32 GB of RAM, 500 GB hard disk, and an 11-GB NVIDIA RTX 2080Ti graphics processing unit.The algorithm was developed in Python 3.6.

Overall Technical Route
As social animals, cows tend to stay with their companions.Monitoring multiple cows' respiratory behavior is an essential aspect of information perception processing on modern dairy farms.The technical process adopted in this study is shown in Figure 2 and consists of 3 main sections: (1) Recognition and segmentation of multiple dairy cows: The YOLACT deep learning algorithm was fine-tuned to segment and recognize multiple cows from irrelevant backgrounds to avoid interference from backgrounds.(2) Resting state recognition: The CNN and Bi-LSTM algorithms were fused to recognize each cow's resting behavior.The objective of this section was to lay a foundation for the respiratory behavior monitoring of each cow under different resting states (lying resting, standing resting).
(3) Respiratory behavior monitoring: According to the resting state recognition results, the corresponding monitoring algorithm (lying resting, standing resting) was applied to monitor the respiration of each individual cow.

Recognition and Segmentation of Cows Based on YOLACT Deep Learning Algorithm
Compared with the segmentation of a single cow, when the video contains multiple cows, each individual occupies fewer pixels, making it more difficult to monitor their respiratory behavior.In addition, complex backgrounds, occlusions, behavior changes, and other disturbances can affect respiratory behavior monitoring.Instance segmentation not only effectively segments the foreground and background in an image, but also identifies different instances (different individuals of the same categories).It has the advantages of both object detection and image segmentation and is an important means of eliminating background interference and extracting foreground objects.In this study, multiple cows were recognized and segmented by finetuning the YOLACT algorithm to prevent interference from irrelevant backgrounds and to facilitate resting state recognition.
Unlike instance segmentation models such as Mask R-CNN, which can only generate a mask for a candidate target region after obtaining that target (He et al., 2020;Xu et al., 2020a), the YOLACT algorithm performs both tasks simultaneously, with better performance and speed, and has been widely used for image segmentation tasks (Bolya et al., 2019).The main principles of the algorithm are as follows.
The algorithm divides the process of object instance segmentation into 2 parallel tasks and generates a "mask coefficient" and a "prototype mask" by a fully connected layer and a convolutional layer (Bolya et al., 2019), respectively.The 2 layers are combined linearly, and then extracts the masks in combination with a positioned detection box.Finally, the masks (positions and colors) of different objects could be obtained.Here, the prototype mask is the mask of the segmentation target generated by the fully convolutional network, and the mask coefficient is the corresponding confidence rate.The specific calculation method is shown in Eq. [1]: where M is the combined result of the prototype mask and the mask coefficient branch; σ is the sigmoid activation function; P is the prototype mask set of size W × H × k, where W, H, and k represent the width and height of the feature map and the mask coefficient generated by the prototype network, respectively; and C is the mask coefficient of size n × k, where n is the coefficient of the retained instance after non-maximum suppression (NMS) and threshold segmentation.The T represents the transpose operation of the matrix.
The YOLACT structure is shown in Figure 3.In the head networks used for image feature extraction (blue boxes), although these low-level features near the network input contain less information, the feature map size is larger, which is more suitable for detecting small objects.Advanced features far from the network input, on the other hand, are rich in information but coarsely localized (Zhao and Wu, 2019).To improve performance, the head network was based on the fully convolutional network structure adopted in Mask R-CNN, which was a connecting structure consisting of layers C1, C2, C3, C4, and C5 from bottom to top and layers  and their respiratory behavior was monitored by respiratory detection algorithms under standing resting (Wu et al., 2020) and resting standing (Song et al., 2019).In YOLACT, C1, C2, C3, C4, C5 and P3, P4, P5, P6, and P7 are bottom-up and top-down features, respectively, while masks for identifying cows were generated after prediction head, NMS (non-maximum suppression), and Protonet, Crop, and Threshold.
P7, P6, P5, P4, and P3 from top to bottom.Displaying more detailed network structure parameters may help the reader to understand the model more realistically (Caffarini et al., 2022); specific layer parameters about the network are listed in Supplemental Table S1 (https: / / doi .org/ 10 .17632/w2r9frh25w .1;Wu, 2022).In this study, the constructed data set was first used to fine-tune the YOLACT algorithm to recognize and segment multiple cows, and then the logic "AND" operation was performed between the mask and the original image to segment dairy cows.The parameters of this YOLACT-based cow target segmentation and recognition model were set as shown in Table 2.
Figure 4 shows a schematic diagram of multiple cows' segmentation in test video 37. Figure 4b shows that this method generated recognition and segmentation masks of multiple cows whose edges were clear and integral.The segmentation avoided interference from irrelevant backgrounds and permitted respiratory behavior monitoring of multiple dairy cows.

Resting State Recognition Based on CNN and Bi-LSTM
We selected the resting state for respiratory monitoring in dairy cows because it is relatively undisturbed by movement (lying resting, standing resting) and it better reflects the cows' physiological health status and is thus more instructive for disease diagnosis (Zhao et al., 2014;Song et al., 2019).Therefore, resting state recognition is the basis for respiratory behavior monitoring after segmentation and identification of multiple cows.
Considering that cow behavior is closely related to time, it is difficult for general neural networks to handle temporal data, whereas Bi-LSTM effectively combines video frame sequence contextual information and has a strong ability to extract features from videos.Therefore, to recognize the cow's resting state, CNN and Bi-LSTM were combined in this study (Wu et al., 2021).As shown in Figure 5, the model is composed of 2 parts.A classic visual geometry group 16 network (VGG16) was introduced as the skeleton network to extract the feature sequence of each video, so as to avoid the shortcomings of manual feature engineering such as complex feature design process and sensitivity to environmental changes.Moreover, resting state (lying resting and standing resting) recognition was conducted by classifying the feature vector sequence with the designed Bi-LSTM classification model; specific layer parameters  The cow video feature vector sequence constructed in this study was extracted from the Fc7 layer of the VGG16 network and converted into a row vector sequence.Because the size of the Fc7 layer is 4,096, the input to the Bi-LSTM model was set to 4,096.The model parameters were set as shown in Table 3.
The test results of resting state recognition showed that the precision of this method for standing resting and lying resting was 0.973 and 0.964, recall was 0.976 and 0.978, and the recognition class probability was 98.49% and 97.88%, respectively.The results indicated that the proposed method could recognize resting state effectively and laid a foundation for respiratory behavior monitoring of multiple dairy cows.

Respiratory Behavior Monitoring of Multiple Dairy Cows
Optical flow is the change in pixel movement due to object motion on an image, and it is one of the key technologies for motion perception (Shukla and Patel, 2013).We found that as the respiratory state changed   during breathing, the cow's body moved periodically and macroscopically in the horizontal direction, which is reflected in the image as a horizontal reciprocal change in the direction of the body's optical flow.Therefore, to monitor the respiratory behavior of multiple cows, the average optical flow direction was calculated for each frame with the help of the fast and efficient Lucas-Kanade (LK) algorithm based on cow identification, segmentation, and behavior recognition (Lucas and Kanade, 1981;Song et al., 2019;Wu et al., 2020).
The LK optical flow model is based on 3 assumptions: (1) constant luminance: the brightness of the same pixel does not change over time; (2) small motion: changes with time do not cause dramatic changes in position; and (3) consistent spatial position: neighboring points are projected onto the image at the same speed (Lucas and Kanade, 1981).Based on the first hypothesis, it is known that which can, in turn, be abbreviated as Eq.[4]: where I x = (αI/αx), I y = (αI/αy), u = (dx/dt), v = (dy/dt), and u, v denotes the optical flow of the pixel point (x, y) in the horizontal and vertical directions, respectively.Combined with the third hypothesis, it is possible to create constraint equations in the neighborhood of the target pixel (x, y) and to calculate u and v.Then, θ, the angle between the target pixel optical flow and the horizontal direction, can be obtained from Eq.
[5], and the average optical flow direction α of the current video frame is shown in Eq. [6]: where α is the angle between the mean optical flow and the horizontal direction, and θ n is the direction of the optical flow at the nth pixel in the neighborhood of the target pixel.
As the respiratory state changes, the cow's body moves horizontally and periodically back and forth, which causes a horizontal reciprocal change in the direction of the optical flow in the image.Because a complete respiratory cycle includes 2 processes-exhalation and inhalation-the cow would have a horizontal reversal in the direction of its optical flow when it completes a breath.The method to calculate the number of cow respiratory cycles in the video is shown in Eq. [7], which could detect a change in sign of the cosine value of the average optical flow direction of video frames to detect respiratory cycles: Increases by remains the same   , [7] where P n n = cos α and which represent the respiratory directions of the nth and n + 1th frames, respectively.Specifically, the respiratory direction sign is positive when the average optical flow direction is in a positive direction (right), and negative when the average optical flow direction is in a negative direction (left).
We observed that the cow's respiratory behavior was weaker when standing resting compared with lying resting, making the former behavior more difficult to monitor.Considering the previous experience of respiratory behavior monitoring methods for a single cow in a resting (lying resting and standing resting) state, we proposed using the corresponding monitoring method (lying resting and standing resting) to monitor the respiratory behavior of multiple cows based on cow identification, segmentation, and resting behavior recognition.The main processes for monitoring the respiratory behavior of cows in lying resting and standing resting states are shown in Figure 6.The first method (lying resting; Song et al., 2019) was mainly used to extract speckle boundaries on the body through the Vchannel in the HSV (hue, saturation, value) color space, and thus perceive the change of its respiratory direction with the help of the LK optical flow algorithm.With the second method (standing resting; Wu et al., 2020), the video was first decomposed, filtered, and motion reconstructed in the time and space domains with the help of the phase-based video magnification (PBVM) model (Wadhwa et al., 2013) to amplify the cow's weak respiratory amplitude.Then, its respiratory behavior was monitored with the LK algorithm.The main parameter settings involved in the above 2 monitoring processes are shown in Table 4, and specific function parameters about the algorithm are listed in Supple-  S3 and S4 (https: / / doi .org/ 10 .17632/w2r9frh25w .1;Wu 2022).Based on the above technical route, the respiratory behavior monitoring processes of cow 1 (standing resting) and cow 2 (lying resting) in video 37 are shown in Figure 7 and Figure 8, respectively.The red dotted line in the figures represents the reference line when the respiratory direction is 0. The curve above it indicates that the respiratory direction is positive, whereas the curve below it is negative.
When monitoring the respiratory behavior of a standing cow, the proposed method can detect significant changes in optical flow and respiratory direction based on accurate segmentation of the cow and identification of its behavior.For a cow in a lying resting state, our method can fully extract the speckle boundaries and detect the cow's respiratory behavior.In addition, the curve of the respiratory direction has obvious peaks and troughs, and changes in the breathing state can be distinguished.

Evaluation of Model Performance
In this study, we aimed to achieve more accurate monitoring of multiple cows' respiratory behavior.Considering the respiratory amplitude characteristics of cows in different resting states (the amplitude of lying resting is smaller than that of standing resting), 2 different body regions (speckle boundary and whole body) were monitored in the proposed monitoring method; ultimately their respiratory direction was monitored and the respiratory monitoring performance was evaluated based on the sign fluctuation of the respiratory direction (respiratory cycle).
To verify the performance of the method for monitoring the respiratory behavior of dairy cows, the accuracy (ACC), mean absolute error (MAE), and root mean square error (RMSE) were assessed.The definitions of ACC, MAE, and RMSE for monitoring of respiratory behavior of multiple cows are shown in Eq. [8] to [10] and are calculated based on the monitored and actual respiratory cycles: Based on the identification and segmentation of dairy cow individuals using the YOLACT (You Only Look At CoefficenTs) model, the CNN (convolutional neural network) and Bi-LSTM (bidirectional long-and short-term memory) algorithms were fused to identify their resting state.Finally, according to the resting state, the respiratory behavior was monitored using the monitoring algorithm under lying resting (Song et al., 2019) and standing resting (Wu et al., 2020).
where n r and n p represent the actual respiration cycles and the respiration cycles monitored by the proposed method, respectively, and s is the number of test videos.Specifically, the ground truth of respiratory cycles was obtained by manually counting the number of horizontal reciprocal movements of the cow's body in a slow-processed video, and monitoring was calculated using Eq.[7].

RESULTS
To verify the effectiveness of the proposed method, 60 test videos (Table 1) were used to verify the method, and the results are shown in Figure 9.Among the 60 videos, the average ACC of respiratory behavior monitoring in 44 videos collected on sunny days was 93.49%, and that of 16 videos collected on cloudy days was 93.23%, which indicated that weather changes had little effect on the proposed method.
Unlike single-cow respiratory behavior monitoring, in multiple cow monitoring tasks, each individual occupies fewer pixels in the video, making respiratory monitoring more difficult.Furthermore, wind can cause some disturbance of the cow's hair, which complicates the  In this study, 16, 36, and 8 test videos were recorded under different wind speed conditions; namely, windless, breezy, and gale.Considering the test results in Figure 8, the ACC of respiratory behavior monitoring was 93.35, 93.95, and 93.50% under windless, breezy, and gale, respectively.The results of 60 experimental videos showed that the respiratory behavior monitoring accuracy of 54 videos was >90.00%, and that of 4 videos was 100.00%.The overall mean ACC, MAE, and RMSE of the proposed method were 93.56%, 3.42, and 3.74, respectively.Thus, the proposed method was feasible to monitor respiratory behavior of multiple cows under different wind conditions (windless, breezy, and gale).

DISCUSSION
For the monitoring of respiratory behavior of multiple cows, compared with individual animals, each cow occupied fewer pixels in the video frame and breathing amplitude was smaller, which made it more difficult to monitor their respiratory behavior.Moreover, the dairy farming environment was complicated, with many interference factors, which further increased monitoring difficulty.To verify the robustness of the proposed algorithm, we analyzed the main factors that may affect the respiratory behavior monitoring of multiple dairy cows.

Comparison of the Proposed Algorithm with State-of-the-Art Studies
As shown in Table 5, we compared our proposed algorithm with state-of-the-art research.Tao et al. (2017) used a Kinect camera (Microsoft Corp.) to detect fluctuations in the abdomen of the lying resting sow and monitored the respiratory behavior with an accuracy of 85.30%.To further improve monitoring performance, Barbosa Pereira et al. ( 2019) proposed a method for monitoring the respiratory rate of pigs under anesthesia based on infrared thermal cameras.Their test results on 17 pigs showed that the error of this method with a ventilator was only 0.79%.Regarding dairy cows, Lowe et al. ( 2019) observed that the temperature of the cow's nostril in the thermal imaging camera changes with and is highly correlated with respiration.However, thermal infrared images contain less detailed information such as texture, which makes it difficult to detect and locate cow body parts.To resolve this deficiency, Kim and Hidaka (2021) proposed a method for identifying cow respiratory patterns by combining infrared thermal imaging and computer vision.Based on nose detection in RGB (red, green, blue) images by the Mask R-CNN deep learning algorithm, the average temperature of the nose in thermal infrared image was calculated as a means to identify the breathing pattern.The correlation between calculated values and actual observations reached 0.92.However, the technical cost of thermal imaging is higher, which makes it difficult to deploy on large dairy farms.
Respiratory behavior in the resting state can reflect a cow's health status.For the monitoring of lying resting cows' respiratory behavior, a method based on the LK sparse optical flow algorithm was proposed that used the LK algorithm to detect the movement of the abdominal speckle boundaries for respiratory behavior monitoring (Song et al., 2019).The test results showed that the accuracy of that method was 98.58%, which is an improvement over the performance of the traditional algorithm (Zhao et al., 2014).To solve the problem that the cow's respiration amplitude was weak, the interference was strong, and it was difficult to observe and monitor the respiratory behavior under standing resting, Cow target segmentation and weak respiratory movement amplification were achieved by fusion of the DeepLab V3+ (Chen et al., 2018) and the phase-based video magnification algorithms.In addition, the LK sparse optical flow algorithm was used to monitor cows' respiratory movements; the accuracy of the method was 93.04% (Wu et al., 2020).Although these studies have achieved good results, most were concerned with monitoring the respiratory behavior of a single cow.However, the cow is a social animal and cows are typically moving together on farms.
Based on the identification and segmentation of multiple dairy cows based on the YOLACT algorithm, the resting state of each cow was further recognized by integrating CNN and Bi-LSTM.According to our resting state recognition results, the corresponding respiratory behavior monitoring methods (lying resting and standing resting) were applied to monitor respiration.The test results showed that the average ACC was 93.56%.Thus, this study could provide a technical reference for the remote diagnosis of respiratory-related diseases and the development of automatic monitoring robots in PLF.

Effect of Motion Interference on Respiratory Behavior Monitoring
Motion may affect respiratory optical flow detection.To explore the robustness of the proposed method to motion interference, it was necessary to analyze performance of the method in experimental videos containing motion interference.Figures 10 and 11 show the recognition and segmentation results and respiratory behavior monitoring results of multiple cows, respectively.Figure 10 shows that our method could segment the 2 cows (cow 1 and cow 2, corresponding to blue and red masks, respectively) and extract their speckle boundaries effectively, which then facilitated respiration monitoring.Figure 11a and Figure 11b show the detection results of the speckle boundaries and optical flow when the tail of cow 1 shakes violently (point a, i.e., the 63rd frame), respectively, and Figure 11c shows the corresponding respiratory behavior monitoring result.It can be seen that the method successfully monitored the respiration of cow 1.In combination with Figure 11b, it can be seen that optical flow interference was introduced before and after the tail of cow 1 shook violently.However, the method still perceived all respiration, and ACC was 100.00% because our method used the average optical flow direction to monitor respiratory  behavior, and local interference did not significantly affect the respiratory direction.
Figure 11d-f shows the results of speckle extraction, optical flow detection, and respiratory behavior monitoring when the cow's body shook violently.The 17 breath circles before and 12 after severe movement were successfully monitored, but 2 false positives were generated when the cow shook its body; thus, the monitoring accuracy was 93.10%.As shown in Figure 11e, the optical flow interference (upward and leftward) caused by the cow's body shaking was large, which covered up the optical flow (rightward) of the respiratory movement, so the respiratory direction was mistakenly detected as a negative direction, which led to the error.
Forty-seven of 60 test videos had motion interference, in which the ACC of 42 videos was >90.00%, and the average ACC was 91.67%, indicating that the proposed method has good robustness to motion interference.

Effect of Occlusion Interference on Respiratory Behavior Monitoring
Figure 12 shows the recognition and segmentation results of multiple cows in test video 42.Even in environments with complex backgrounds and cow targets, the proposed method effectively removed extraneous backgrounds, and identified and segmented cows with accurate recognition results and clear segmentation boundaries.
Figure 13a-c show the key segmentation processes and results of respiratory behavior monitoring in cow 1.The proposed method successfully monitored all 33 breaths with an accuracy of 100.00%.Between frames 178 and 214, the decrease in the amplitude of the respiratory movement direction and the lengthening of the respiratory cycle were mainly caused by the continuous shaking of the cow's leg (as shown in Figure 13b), which introduced disturbing optical flow and covered the change in respiratory optical flow to a certain extent.In addition, the respiratory posture of the cow changed at this stage, which led to the prolongation of the respiratory cycle.
Figure 14a-d presents the respiratory behavior monitoring results of cow 2, which showed that the segmentation of cow 2 was holonomic, and the extraction of speckle boundary was accurate.In addition, the respiratory direction was clear and had obvious peaks and valleys.As shown in Figure 14c, all 33 breaths were successfully monitored with the ACC of 100.00%.Among the 60 test videos, 9 videos were covered with occlusion interference, and the errors of  the other videos were all distributed in the range of 0.09 to 0.14 breath/s.However, all respiration circles were completely monitored in the video, which was mainly because the cow's body or speckles were not fully blocked and there was no continuous violent movement, so no other interfering optical flow was introduced.In summary, the method could effectively monitor respiratory behavior of multiple dairy cows when they are not completely shielded or the distributing optical flow is less than that generated by respiratory behavior.

Effects of Different Resting States on Respiratory Behavior Monitoring
Due to the different respiratory monitor algorithms being oriented to different resting states (lying resting and standing resting) in the proposed method, the dairy cows' state in the test video was an important factor affecting the respiratory behavior monitoring.
The number of videos containing cows in states of lying resting, standing resting, and both lying resting and standing resting in 60 experimental videos was 21, 14, and 25, respectively.Based on the test results, the average ACC of respiratory behavior monitoring in the 3 scenarios was 95.42, 91.33, and 93.25%, respectively, and the variance was 0.25 × 10 −3 .The reason for the lower accuracy of respiratory behavior monitoring during standing resting was that the breathing amplitude is much smaller than in lying resting.In addition, cows occupied fewer pixels compared with single-target respiratory behavior monitoring tasks, which made monitoring the standing resting state more difficult.
The accuracy of the proposed method was >90.00% in the 3 resting states, indicating that the method could effectively monitor the respiratory behavior of multiple dairy cows in different resting scenarios.

CONCLUSIONS
To overcome the visual fatigue and high cost of manual monitoring of respiratory behavior, we proposed a method for monitoring respiratory behavior of multiple cows based on computer vision and deep learning.Single-target respiratory behavior monitoring models (lying resting and standing resting) were used to monitor each cow's respiration after individual identification, segmentation, and resting state recognition of multiple cows.The test results showed that the ACC, MAE, and RMSE of monitoring were 93.56%, 3.42, and 3.74, respectively.The method exhibited good robustness to occlusions, resting states, and environmental changes.This study could form the technical basis for respiratory behavior monitoring and automatic diagnosis of respiratory-related diseases in multiple dairy cows.The respiratory behavior monitoring algorithm will continue to be optimized to make it lighter and easier to deploy on small computing platforms such as embedded and edge computing devices.Moreover, precision livestock farming inspection robots with respiratory behavior monitoring could be developed and deployed to further reduce manual labor.

Figure 1 .
Figure 1.Schematic diagram of test video acquisition process.

Figure 2 .
Figure 2. Technical process of the proposed method.After dairy cows were identified and segmented by the YOLACT (You Only Look at CoefficienTs) model, their rest states (standing resting, and lying resting) were identified by Bi-LSTM (bidirectional long-and short-term memory) based on features extracted by CNN (convolutional neural network), Figure 3. Network structure of the YOLACT (You Only Look at CoefficienTs) instance segmentation model.Bottom-up C1, C2, C3, C4, C5 and top-down P3, P4, P5, P6, and P7 features were used as the input of Protonet and Prediction head and NMS (non-maximum suppression), respectively, and the mask of dairy cows was obtained through Crop and Threshold operations.

Figure 4 .
Figure 4. Recognition and segmentation of multiple dairy cows based on the YOLACT (You Only Look at CoefficienTs) algorithm.(a) Original image, (b) mask for cow recognition and segmentation output by YOLACT, (c) segmentation result of cow 1 obtained by mask operation, and (d) segmentation result of cow 2 obtained by mask operation.

Figure 5 .
Figure 5. Resting state recognition of dairy cows based on convolutional neural network (CNN) and bidirectional long-and short-term memory (Bi-LSTM) algorithms.The segmentation result of the dairy cow from YOLACT (You Only Look At CoefficienTs) was extracted by the visual geometry group 16 (VGG16) network to obtain its feature sequence, and the Bi-LSTM model identified its resting state by analyzing the feature sequence.
Figure6.Based on the identification and segmentation of dairy cow individuals using the YOLACT (You Only Look At CoefficenTs) model, the CNN (convolutional neural network) and Bi-LSTM (bidirectional long-and short-term memory) algorithms were fused to identify their resting state.Finally, according to the resting state, the respiratory behavior was monitored using the monitoring algorithm under lying resting(Song et al., 2019) and standing resting(Wu et al., 2020).

Figure 7 .
Figure 7. Respiratory behavior monitoring of the standing cow in video 37. (a) Cow segmentation and recognition result based on the YOLACT (You Only Look at CoefficienTs) model, (b) resting state recognition result based on CNN (convolutional neural network) and Bi-LSTM (bidirectional long-and short-term memory), (c) optical flow detection result based on PBVM (phase-based video magnification) and LK (Lucas-Kanade) algorithm, and (d) respiratory behavior monitoring result based on respiratory monitoring algorithm under standing resting (Wu et al., 2020).

Figure 9 .
Figure 9. Respiratory behavior monitoring results of multiple cows.The box represents the interquartile range, the line is the median of the accuracy of the respiratory behavior monitoring, while whiskers are the maximum and minimum of the accuracy and symbols are the average of the accuracy (ACC).
Wu et al.: MONITORING RESPIRATORY BEHAVIOR

Figure 8 .
Figure 8. Respiratory behavior monitoring of the lying resting cow in video 37. (a) Cow segmentation and recognition result based on the YOLACT (You Only Look at CoefficienTs) model, (b) resting state recognition based on CNN (convolutional neural network) and Bi-LSTM (bidirectional long-and short-term memory), (c) convert cow image color space to HSV (hue, saturation, value), (d) speckle boundary detection based Canny operator, (e) optical flow detection result based on LK (Lucas-Kanade) algorithm, and (f) respiratory behavior monitoring result based on respiratory monitoring algorithm under lying resting (Song et al., 2019).

Figure 10 .
Figure 10.Detection and segmentation of multiple cow targets in video 6.(a) Original video, (b) segmentation of cow 1 based on YOLACT (You Only Look at CoefficienTs), (c) speckle extraction based on HSV (hue, saturation, value) color space, (d) speckle boundary detection of cow 1 with Canny operator, (e) dairy cows segmentation mask generated by YOLACT, (f) segmentation of cow 2 based on YOLACT, (g) speckle extraction based on HSV color space, and (h) speckle boundary detection of cow 2 with Canny operator.

Figure 11 .
Figure 11.Effect of motion interference on respiratory behavior monitoring.(a) Speckle boundary detection of cow 1 based on HSV (hue, saturation, value) color space, (b) optical flow detection of cow 1 based on LK (Lucas-Kanade) algorithm, (c) respiratory behavior monitoring of cow 1 based on respiratory monitoring algorithm under lying resting (Song et al., 2019), (d) speckle boundary detection of cow 2 based on HSV color space, (e) optical flow detection of cow 2 based on LK algorithm, and (f) respiratory behavior monitoring of cow 2 based on respiratory monitoring algorithm under lying resting(Song et al., 2019).

Figure 12 .
Figure 12.Recognition and segmentation of multiple cow targets in video 42.(a) Original video, (b) dairy cow segmentation mask generated by YOLACT (You Only Look at CoefficienTs), (c) segmentation of cow 1 based on YOLACT, and (d) segmentation of cow 2 based on YOLACT.

Figure 13 .
Figure 13.Respiratory monitoring results of cow 1 in video 42.(a) Segmentation of cow 1 based on YOLACT (You Only Look at CoefficienTs) and logical "AND" operation, (b) optical flow detection of frame 186 based on PBVM (phase-based video magnification) and LK (Lucas-Kanade) algorithms, and (c) respiratory behavior monitoring result based on respiratory monitoring algorithm under standing resting (Wu et al., 2020).
Wu et al.: MONITORING RESPIRATORY BEHAVIOR

Table 1 .
Wu et al.: MONITORING RESPIRATORY BEHAVIOR Test video information of monitoring respiratory behavior of multiple cows

Table 3 .
Parameter settings for the resting state recognition model

Table 4 .
Parameter settings for respiratory behavior monitoring at different resting states 1Phase-based video magnification.

Table 5 .
Wu et al.: MONITORING RESPIRATORY BEHAVIOR Comparison of our proposed algorithm with state-of-the-art studies