Hot topic: Automated assessment of reticulo-ruminal motility in dairy cows using 3-dimensional vision

We designed a 3D vision system to capture the morphology of a cow's left paralumbar fossa. The system consisted of a 3D depth camera (Realsense D415; Intel, Santa Clara, CA) connected to and controlled by a recording computer through a USB 3.0 port. The camera had depth detection, an output resolution of 1,280 × 720 pixels, and field-of-view angles of 69° × 43° × 77° (horizontal × vertical × diagonal). The camera was mounted on the left fence of an automatic milking robot (AMS; Astronaut A5; Lely Industries N.V., Maassluis, the Netherlands), the arm of which approached the cow from the right side of its body. The camera captured a view of the cow's left paralumbar fossa in the AMS from above at a 30° angle to the horizontal plane (Figure 1). This 3D vision system was installed 1 month before data collection to habituate cows to the hardware changes in the AMS.

Twenty primiparous cows (Holstein × Swiss Brown) at 1 to 200 DIM from a single herd were recorded once with the vision system on a Dutch commercial farm on June 28, 2018, from 1100 to 1600 h. On that day, each cow was fed approximately 40 kg of fresh feed consisting of grass, maize, hay, and barley, with a feed ration of 5:3:1:1 on a wet basis. The feed was supplied 4 times at 0400, 1000, 1600, and 2200 h by an automatic feeding robot (Vector, Lely Industries N.V.). The cows were also fed concentrate in the AMS. The amount of concentrate provided to each cow was determined based on its lactation stage and milk yield over the previous day. Among the 20 cows, the average amount of concentrate provided during image recording was 2.4 kg, with a minimum of 0.7 kg, a maximum of 3.0 kg, and a standard deviation (SD) of 0.5 kg.

When a cow entered the AMS and the milking process started, the AMS sent a digital signal to the recording computer to activate the 3D camera to take depth images at 30 frames per second. When the first teat cup detached from the cow, the AMS sent another signal to stop the recording. Among the 20 cows, the average image recording time was 4.3 min, with a minimum of 1.8 min, a maximum of 7.0 min, and an SD of 1.4 min. After the cow left the robot, the recorded data, including all captured 3D depth images, cow identification number, and the time of the recording, were saved on the computer.

The 3D camera captured a cow's body surface in raw depth images. Each image was a point cloud with a resolution of 1280 × 720 pixels. The depth value of each point was a 3D distance measured from the center of the camera lens to the projection of the point on the cow's body surface. Each image was processed using an image-processing script we developed in MATLAB (release 2018b, MathWorks Inc., Natick, MA).

We downsized each point cloud to a resolution of 128 × 72 pixels to increase image-processing speed. The downsizing method was a bi-cubic interpolation, in which each output point was the weighted average of all points in the nearest 4 × 4 neighborhood surface area from the original point cloud.

The downsized point cloud (containing 3D depth values) was orthogonally decomposed to x, y, and z coordinates. Then, the coordinates were converted to a matrix of the same size containing the estimated surface curvatures to represent the geometry of the point cloud (Figure 2). This conversion was based on the procedure of “estimating surface normal and curvature in a point cloud” described by . In the conversion, each point (p) of the downsized point cloud was selected as the center point of a plane. This plane was fitted by p and its 9 nearest neighbors, and the shortest 3D Euclidean distance was calculated based on the x, y, and z coordinates. On each 10-point fitted plane, a normal vector perpendicular to the plane was determined for each of the 10 points. Then, the absolute length variance of these 10 normal vectors was calculated as the approximation of the surface curvature around the selected point p. We constructed a matrix containing the estimated surface curvatures of all points in the downsized point cloud and defined it as the curvature matrix.

From the curvature matrix of the first recorded image of the first recorded cow in the AMS, we manually selected a template that included the cow's last rib, transverse processes, and part of the left paralumbar fossa between these 2 bones. This template was a square, with the top of the cow's last rib as its upper left vertex (Figure 2). The square side length was set at half the distance between the top of the last rib and the hook bone center. The template overlaid a body region with unique morphological features and we used this to consistently locate the same region on all cows.

The template was matched to each subsequent curvature matrix. This matching was quantified by a normalized 2-dimensional cross-correlation between the template and any region of the same size as the template (). The region showing the highest cross-correlation with the template was then labeled as the matched region of interest (ROI). This ROI had the greatest morphological similarity to the template among all the regions and contained the morphological feature of the left paralumbar fossa. We considered the ROI in the curvature matrix overlying the left paralumbar fossa, and quantified the concavity of the ROI by averaging the estimated surface curvatures of all points in the ROI.

We chronologically assembled the surface concavity of the ROI from all images in a single recording to form a raw signal (solid line in Figure 3). The raw signal had a sampling frequency (i.e., image recording speed) of 30 Hz and contained oscillations that potentially denoted reticulo-ruminal contractions. This raw signal also contained noise caused by cow motion artifacts and unstable illumination around the 3D camera. Examples are shown in Figure 3. We assumed that different components of the raw signal varied in frequency of occurrence, so we applied frequency analysis to remove the noise from the potential signal of interest. We applied discrete fast Fourier transformation to convert the raw signal from its time domain to a representation in the frequency domain. Based on the study of , we assumed that reticulo-ruminal contractions occurred 1 to 3 times per minute, and the contractions ranged in frequency from 0.017 to 0.050 Hz. We filtered the representation of the frequency domain to retain only information in the frequency range of the reticulo-ruminal contractions. In this range, we considered the frequency with the greatest amplitude to denote the reticulo-ruminal contraction frequency.

Then, the filtered signal was inversed to the time domain (dotted line in Figure 3). From the filtered signal, we identified local maxima (circles on the dotted line in Figure 3) using the “findpeaks” function in MATLAB. These local maxima were defined as any point with a value higher than the values of its 2 neighboring points in the signal. We considered each local maximum to be the moment when the reticulo-rumen inflated to its maximum in a contraction. Then, we identified the filtered signal between 2 local maxima and considered this to be a complete cycle of reticulo-ruminal contraction.

Ideally, automated identification of the reticulo-ruminal contractions would be validated by simultaneous palpation on a cow's left paralumbar fossa while the contractions were being recorded. However, because such palpation would block the view of the camera, this real-time validation was not feasible. Therefore, we used visual identification of reticulo-ruminal contractions from the recorded 3D images for post-recording validation. Two assessors with experience in machine vision were trained by a veterinarian on morphological changes in the left paralumbar fossa during reticulo-ruminal contraction. Then, the assessors independently observed undulations in the left paralumbar fossa from a video, which was constructed from the chronological sequence of 3D images of 1 cow during 1 milking process. When assessors identified an upward wave pattern from the left paralumbar fossa, they recorded the time in the video as the occurrence time. A reticulo-ruminal contraction was confirmed only when both assessors independently identified the contraction, and the time difference between their recorded times of occurrence was less than 4 s. In total, the 2 assessors agreed on 261 contraction identifications and disagreed on 11.

For each visually identified reticulo-ruminal contraction, we defined a period of 3 s before and after the averaged occurrence time from the 2 assessors. We used this timeframe to compare the time of the automatically identified local maxima to the visually identified time of occurrence. When a local maximum was automatically identified within the timeframe of a visually identified contraction, the case was identified as a true positive. When no local maximum was automatically identified within the timeframe, the case was identified as a false negative. Moreover, any automatically identified local maximum without a matched visually identified contraction was identified as a false positive. We then calculated the matching sensitivity (i.e., the number of true positives divided by the number of visually identified reticulo-ruminal contractions) and positive predictive value (i.e., the number of true positives divided by the number of automatically identified local maxima) as validation results for automated identification.

Among the videos of the 20 cows, 261 reticulo-ruminal contractions were visually identified by both assessors, and 259 local maxima were automatically identified by image and signal processing. Using the visual identifications as the reference, we found that the automated identification produced 253 true positives, 8 false negatives, and 6 false positives. All 8 false negatives occurred as a result of technical complications from signal processing. In half of the false negatives, the image recording started at approximately the time when the rumen inflated to its maximum. The assessors identified the contraction in retrospect, but the signal processing failed to identify the contraction because the signal did not completely oscillate. In the remaining half of the false negatives, the signal processing missed contractions that were short and that followed immediately by a new contraction. An example is shown in Figure 3. These visually identified contractions were considered to be incomplete contractions. We also found that all false positives involved cases in which the assessors were unable to detect the reticulo-ruminal contractions from the video. In 4 cases, the reticulo-rumen was fully filled and the video showed only small variations in the concavity of the left paralumbar fossa, so the assessors had difficulty observing changes in the cow's body surface from the video and failed to identify the reticulo-ruminal contractions. The other 2 false positives were caused by motion artifacts because of cows' rapid movement during image recording. The motion artifacts did not allow the assessors to clearly locate the left paralumbar fossa or identify the reticulo-ruminal contractions. Among the 261 visually identified reticulo-ruminal contractions, 253 were identified by the automated system, yielding a matching sensitivity of 0.97 and a positive predictive value of 0.98. The results indicated that 3D vision–based automated identification of the reticulo-ruminal contractions performed similarly to the reference manual identification.

Across the 20 cows, the average frequency of automatically identified reticulo-ruminal contractions was 3.1 times per minute (SD = 0.28 times per minute), with a minimum of 2.4 times per minute and a maximum of 3.4 times per minute. This contraction frequency was in the upper frequency range for normal cows. This relatively high level of reticulo-ruminal motility in the AMS likely reflected the cows' consumption of concentrate, because feeding increases cow reticulo-ruminal motility ().

The 20 studied cows were also examined via rumen palpation by a trained assessor immediately after the 3D recording. This palpation was conducted in a passage that connected to the exit of the AMS and where a cow remained for 2 min, with concentrate offered. The assessor placed a hand on the left paralumbar fossa of the cow and counted the number of contractions for 2 min. Across the 20 cows, the average frequency of contractions identified by palpation was 2.4 times per minute (SD = 0.61 times per minute), 0.7 times per minute less than the average frequency automatically identified in the AMS. It is possible that the human handling in this examination, including manual palpation and the close proximity of an unfamiliar human, not being a normal procedure on the farm, caused the cows additional distress. According to , distress can inhibit reticulo-ruminal motility and reduce contraction frequency.

This study demonstrated a low-cost 3D vision system that could automatically and remotely assess the reticulo-ruminal motility of dairy cows. The automated assessment performed similarly to referential manual assessment. In contrast to other available methods of reticulo-ruminal assessment, this automated system was noninvasive and did not distress the cows. As a proof of concept, this automated system has the potential to operate as a standalone system, not only on farms with AMS but also on conventional farms with individual feeding stations. Upcoming studies should focus on validating this system in cows of different breeds, and with different parities, lactation stages, feed intakes, and morphological characteristics. As well, longitudinal studies should be performed to automatically and regularly monitor changes in the reticulo-ruminal motility of individual cows on farm. This automated monitoring system could allow farmers and veterinarians to frequently collect information about individual cows' gastrointestinal condition and assist them in disease diagnosis. Moreover, automated reticulo-ruminal assessment can be combined with the output of other sensors implemented on farms to further improve the health care and daily management of dairy cows.