The development, and validation of a milk feeding behavior alert from automated feeder data to classify calves at-risk for a diarrhea bout: A diagnostic accuracy study

The objective of this diagnostic accuracy study was to develop and validate an alert to identify calves at-risk for a diarrhea bout using milk feeding behavior data ( behavior ) from automated milk feeders ( AMF ). We enrolled Holstein calves (n = 259) as a convenience sample size from 2 facilities who were health scored daily preweaning and offered either 10 or 15 L/d of milk replacer. For alert development, 132 calves were enrolled and the ability of milk intake, drinking speed, and rewarded visits collected from AMF to identify calves at-risk for diarrhea was tested. Alerts that had high diagnostic accuracy in the alert development phase were validated using a holdout validation strategy of 127 different calves from the same facilities (all offered 15 L/d) for −3 to 1 d relative to diarrhea diagnosis. We enrolled calves that were either healthy or had a first diarrheal bout (loose feces ≥2 d or watery feces ≥1 d). Relative change and rolling dividends [d 0 = (d − 1/d − 2)] for each milk feeding behavior were calculated for each calf from the previous 2 d. Logistic regression models and receiver operator curves (ROC) were used to assess the diagnostic ability for relative change and rolling dividends behavior relative to alert d) to classify calves at-risk for a diarrhea bout from −2 to 0 d relative to diagnosis. To maximize sensitivity (Se), alert thresholds were based on ROC optimal classification cut-off. Diagnostic accuracy was met when the alert had a moderate area under the ROC curve (≥0.70), high accuracy (Acc) (≥0.80), high Se (≥0.80), and very high precision (Pre) (≥0.85). For alert development, deviations in rolling dividend milk intake with drinking speed had the best performance (10 L/d:


INTRODUCTION
Neonatal calf diarrhea was reported by dairy producers as a contributor to 21% of morbidity and 56% of mortality in preweaned dairy calves in the United States (USDA, 2018).Despite diarrhea being a major contributor to calf disease and death loss, the accepted approach for detecting clinical diarrhea bouts in calves requires stimulating the calf to defecate, or to observe the calf passing watery, or loose feces (Renaud et al., 2020).It is difficult to observe which calf has diarrhea in social housing because less labor-intensive methods such as observing the calf for hide cleanliness were not associated with the positive identification of a calf atrisk for a diarrhea bout (Graham et al., 2018).The early identification of a diarrhea bout for these calves can greatly improve the calf's chances of recovery since diarrheic calves have an increased probability of dehydration, metabolic acidosis, hypoglycemia, and death (Trefz et al., 2017).However, the detection of diarrhea bouts remains a challenge for dairy calf rearing systems and alternative detection methods require investigation.
Calves with a diarrhea bout in the days surrounding disease diagnosis display sickness behaviors such as decreased milk intake (Lowe et al., 2019), decreased rewarded visits (Conboy et al., 2021(Conboy et al., , 2022)), and slower drinking speeds (Knauer et al., 2017) recorded by automated milk feeders (AMF) on d −1 to d 1 relative to the diagnosis of the diarrhea bout.Using the sickness behavior repertoire (i.e., anorexia, lethargy, depression) might be an opportunity to identify calves in the early stages of disease development.The sickness behavior repertoire is an organized behavioral response in mammals caused by the release of inflammatory cytokines into the body which makes an animal feel sick and less interested in food before the animal exhibiting the clinical signs of the disease (Hart and Hart, 2021).For example, before the clinical diagnosis of pneumonia, dairy calves fed by an AMF had relative changes in their own milk behavior patterns and these patterns have been used to provide early interventions to calves to ameliorate disease severity (Cantor et al., 2021).However, the alert that is installed on calf feeders available commercially (e.g., Foerster-Technik) uses a rolling average of their feeding behavior over a long-time frame (e.g., 12 d; Cantor et al., 2021).Using a long-time frame to create a baseline and to generate an alert specific to diarrheic calves would miss many calves or be delayed since calves usually incur a diarrhea bout within the first 14 d of life (Urie et al., 2018).Moreover, many producers background their calves in individual housing before introduction to the automated milk feeder to ensure a strong suckle (Fujiwara et al., 2014).Thus, we suggest that it is fundamental to explore the potential of alerts which accumulate data within a short time frame, especially since there is a short incubation time for many pathogenic causes of diarrhea in neonatal calves (Cho and Yoon, 2014).
Early alerts could provide the producer the ability to monitor specific calves at-risk for diarrhea intensively.Thus, when a calf becomes diarrheic, the producer can provide an intervention, such as an oral electrolyte solution to the calf in a timely manner.This type of early sensing "alert" system is used by epidemiologists to detect zoonotic outbreaks with geographical information system data (Kshirsagar et al., 2013), to detect disease in plants using automated hyperspectral imaging (Zhang et al., 2020), and by medical professionals using wearable sensor data to detect a host of infectious diseases including the Covid-19 pandemic outbreak (Meraj et al., 2021).It is promising that milk behavior patterns recorded by an AMF may indicate disease in calves because this technology is popular; AMF were reported to be used in nearly a fifth of Canadian dairies surveyed (Medrano-Galarza et al., 2017).Thus, we suggest that there is an opportunity to develop an AMF alert using changes in individual milk feeding behaviors to positively identify a calf at-risk for a diarrheal bout, but this needs investigation.
Diarrhea bouts should be detected in calves with high diagnostic accuracy since it is the leading cause of morbidity and mortality in preweaned calves (Urie et al., 2018) and it decreases average daily gain (Schinwald et al., 2022).However, it is important to clarify that an AMF alert comes in addition to other management strategies used to identify disease in calves on-farm.An AMF alert could identify calves that require further examination or provide secondary information about a particular calf that was identified at-risk for disease by other methods.However, one challenge with precision technology health alerts for dairy cattle is that producers will ignore the alerts if there are too many false positives (Eckelkamp and Bewley, 2020).Thus, it is imperative to prioritize a high sensitivity and accurate for an alert to indicate calves at-risk for a diarrhea bout.
Milk feeding behaviors in calves are affected by how the AMF is set up including the daily allotment, or milk feeding strategy (Rosenberger et al., 2017), and stocking densities as well as the meal size (Jensen, 2004).Furthermore, a scoping review observed that which milk feeding behaviors change around disease bouts are very specific to milk feeding strategy and also vary by disease status (Morrison et al., 2021;Conboy et al., 2022;Ghaffari et al., 2022).Therefore, to develop an alert which is accurate for the diagnosis of diarrhea, the consideration of milk allotments offered to them (10 L/d or 15 L/d) stocking densities (≤15 calves/pen), and meal sizes should be considered.The objective of this diagnostic accuracy study was to validate an alert based on milk feeding behavior to identify calves atrisk for a diarrhea bout within a few days of clinical diagnosis using ground truth data where the health of each calf was examined daily.We chose to focus on alert development for diarrhea in dairy calves because this disease is the leading cause of morbidity and mortality in calves on dairies (USDA, 2018), and diarrhea causes dehydration, and a host of metabolic disorders that need timely intervention to avoid the death of a neonatal calf (Trefz et al., 2017).Specifically, we used precision livestock farming algorithms including relative changes {d 0 = [(yesterday − day before yesterday)/day before yesterday]}and rolling dividends [d 0 = (yesterday/day before yesterday)] for AMF milk intake, drinking speed, and rewarded visits, to create alerts for individual calves surrounding their diarrhea bouts.Because milk intake was associated with diarrhea bouts in the literature, we predicted that an alert based on milk intake would be diagnostically accurate to alert for a calf at-risk for a diarrhea bout within −2 to 1 d relative to diarrhea diagnosis.

MATERIALS AND METHODS
The milk intake, drinking speed, and rewarded visits surrounding clinical disease diagnosis (collectively referred to hereafter as milk behavior patterns) were specific to each individual calf enrolled in this diagnostic accuracy study and were passively collected daily by an automated milk feeder (AMF: CF 1000, DeLaval Combi, Forster-Technik, Engen, Germany).These milk behavior patterns were from individual Holstein dairy calves (n = 259, 62 bulls, 197 heifers) with no missing data for the first 28 d after training on the automated milk feeder.We enrolled calves that were either healthy or had a first diarrheal bout (loose feces ≥2 d or watery feces ≥1 d).For simplicity, the first day that a calf was diagnosed with a diarrhea bout will be referred to as d 0. We selected the first 28 feeder d because calves usually incur a diarrhea bout within the first few weeks of life (Urie et al., 2018).These calves were raised in group housing facilities with a stocking density of ≤15 calves/pen at the Ontario Dairy Research Centre (Elora, ON, Canada), and the University of Kentucky Research Dairy Farm (Lexington, KY, USA).
We included 164 calves raised at the Ontario Dairy Research Centre who were offered 15 L/d of a commercial milk replacer (24% CP, 22% fat, Achieve Pro Gro, Grober Nutrition, Cambridge, CA).These calves had a minimum meal size of 0.5 L and a maximum meal size of 3.0 L allocated to a 24 h schedule.There were 69 calves raised at this facility from January to October 2018 (more details in Conboy et al., 2022) for the alert development phase.There were 95 calves raised at this facility from June to October 2022 for the holdout validation strategy phase.All animal care was approved by the University of Guelph Animal Care Committee (Animal Use Protocols 2018: 4408 and 2022: 4745).
We included 164 calves raised at the University of Kentucky Research Dairy Farm.There were 107 calves raised at this facility from April 2018 to September 2019 for the alert development phase.These calves were offered 10 L/d of a commercial milk replacer (27% CP, 20% fat, Cows Match Cold Front, Land O Lakes, MN, USA) with a minimum meal size of 1.0 L and a maximum meal size of 3.0 L allocated to a 24 h schedule.There were 57 calves raised at this facility from February to December 2020 for the holdout validation phase and these calves were fed the same commercial milk replacer with the same meal schedule.The holdout validation phase calves raised at this facility were offered up to 15 L/d.All animal care was approved by the University of Kentucky Institutional Animal Care and Use Committee (approval number 2018: 2864 and 2019-3374).

Management and feeding
The care and management of calves from the Ontario Dairy Research Centre has been previously described in Conboy et al. (2022).Briefly, calves were fed maternal dam colostrum within 6 h after calving, and again by bottle 8 h later and housed individually until calves stood independently and had a strong suckle.Passive immunity status was tested by refractometer and confirmed on all calves at 24 h of age (BRIX ≥8.4%).Calves had free access to calf starter from an automated feeder (Forster-Technik, Engen, Germany), free access to chopped straw (5 mm length) from 4 buckets, and free access to water from a water bowl situated in the corner of the pen.The calves were housed in rooms with all-in-all-out housing bedded with sawdust shavings and a maximum 15 bull and heifer calves were allocated per group (4.6m 2 /calf).The pens were cleaned by manually removing and replacing the shavings twice weekly.
The care and management of calves from the University of Kentucky Dairy has been previously described (Cantor et al., 2021).Briefly, calves were fed colostrum within 6 h after calving, and again by bottle 8 h later and housed individually until calves stood independently and had a strong suckle.Passive immunity status was tested and confirmed by refractometer on all calves within 48 h of age at (BRIX ≥8.0%).Calves had free access to calf starter from an automated feeder (Forster-Technik, Engen, Germany), free access to chopped hay from a trough, and free access to a water feeder situated in the corner of the pen.Calves were raised in dynamic housing meaning that calves were initially added to a "young" group and then moved to the adjacent pen ("older pen") at least one wk before weaning.The pens were bedded with sawdust shavings where a maximum 12 bull and heifer calves were allocated per group (5.7 m 2 /calf).The pens were cleaned by manually removing and replacing the shavings once weekly.

Health exams
Daily health exams were performed by trained researchers on every calf (Cantor et al., 2021;Conboy et al., 2021).The calves enrolled in this study were negative for signs of respiratory disease (Love et al., 2014) and did not have navel inflammation.Fecal consistency scoring was recorded daily by researchers who rectally stimulated the calf to defecate with a thermometer and had a very high observer reliability that was periodi- Only data from the first diarrheal bout of each calf was included in this study.

Statistical Analysis
The daily calf milk intake (L/d), drinking speed (L/ min), and rewarded visits (visits/d) were included for the development of an AMF alert based on changes from individual milk feeding behaviors in the calves.For our univariate analysis, each feeding behavior was assessed for normality, collinearity, multi-collinearity, and outliers were assessed for model leverage by using diarrhea status as a predictor variable with milk intake, drinking speed, rewarded visits, starter intake, and unrewarded visits used as outcome variables using linear regression models.Starter intake, and unrewarded visits did not meet normality criteria due to many calves having less than 100 g per d of grain consumption, and <1 unrewarded visit per day; these variables were excluded from further consideration for the alert.There were no outliers with model leverage detected, and the variance inflation factors were <1.0 among these variables.The AMF calculated the daily milk intake by adding the milk consumed in each daily visit across the 24 h from 00:01 to 23:59.Daily drinking speed was calculated by the AMF by averaging the drinking speed of each visit for each calf across the 24 h.The AMF calculated the daily rewarded visits by adding the visits where at least 0.25 L milk was consumed by the calf for each rewarded visit across the 24 h.All statistical analyses were performed with SAS version 9.4 (Cary, NC, USA).

Timeframe relative to diagnosis of diarrhea bout
The average and 95% CI of daily milk intake, drinking speed, and rewarded visits were calculated for diarrheic calves by milk feeding strategy (e.g., 10 L/d, 15 L/d) from d −3 to d 1 relative to diagnosis of a diarrhea bout.The averages and CI for the milk behavior patterns were calculated to visually demonstrate the changes in milk feeding behavior relative to the diagnosis of a diarrhea bout.A complete demonstration of the different time frames relative to the diagnosis of a diarrhea bout, the number of calves enrolled by facility, and milk allotment used in each phase of this study are in Table 1.Briefly, we deployed d −1 to d 0 relative to the diagnosis of a diarrhea bout for the alert development phase to ensure that the alert thresholds that we calculated for each milk behavior pattern were representative of clinical diarrhea bouts in the calves.To test our alert thresholds on the calves used to develop the alert, we used d −2 to d 1 relative to the diagnosis of a diarrhea bout because most calves in the Canada data set used for this phase were not trained to use the AMF independently for 2 d on d −3 relative to a diarrhea bout.For the holdout validation phase d −3 to d 1 relative to the diagnosis of a diarrhea bout were selected because diarrheic calves were observed to decrease their milk behavior patterns in this time frame (Morrison et al., 2022;Conboy et al., 2022).

Rolling dividends and relative changes definitions
The relative change and rolling dividends in individual milk feeding behaviors were calculated using the previous 2 d of behavior for each calf's milk intake, drinking speed, and rewarded visits.The equation for relative change, and for rolling dividends in behavior were the same for milk intake, drinking speed and rewarded visits.As an example, on the d of diarrhea diagnosis, relative change for milk intake was calculated with the equation: Where "B" is milk intake on d -1 relative to diarrhea diagnosis and "A" is milk intake on d -2 relative to diarrhea diagnosis (d 0).We also calculated rolling dividend changes for each calf using the equation: Where "E" is milk intake on d -1 relative to diarrhea diagnosis and "F" is milk intake on d -2 relative to diarrhea diagnosis (d 0).

Alert development phase
An alert threshold was developed based on an individual calf crossing a set alert threshold for each milk behavior pattern to identify the calves at-risk for a diarrhea bout.
Logistic regression models and receiver operator curves (ROC) were used to determine the appropriate alert thresholds for relative change and rolling dividends in behavior using data surrounding the diagnosis of a clinical diarrhea bout (d −1 to d 0 relative to diagnosis), and the influence diagnostics were calculated; each Calf age at diagnosis of a diarrhea bout was included as a quantitative covariate.Healthy calves were randomly assigned a day based on the 95% CI age of when diarrheic calves were diagnosed with a diarrhea bout (average 9 d of age, 95% CI: 6 to 11 d).We also anticipated that the alert that had optimal diagnostic accuracy would be dependent on milk feeding strategy as it has been observed that milk behaviors (Rosenberger et al., 2017) and milk-based sickness behaviors (Conboy et al., 2021;Lowe et al., 2019;Morrison et al., 2021) vary by milk feeding strategy.Hence, we assessed the diagnostic accuracy of the alert based on relative change or rolling dividends in milk feeding behavior for the 10 L/d calves and 15 L/d calves separately.
To maximize sensitivity (Se) of the alert to correctly classify data, alert thresholds were based on ROC optimal classification cut-offs and are referred to as probability thresholds from the ROC in the results.Youden's index was also calculated to ensure that the diagnostic tests were performing better than chance (Ruopp et al., 2008).
The relative change or rolling dividends in milk feeding behaviors were assessed for collective diagnostic ability with logistic regression models identical to the structure described above surrounding diarrhea diagnosis (d −1 to d 0).We used Chi-squared testing and contrast statements within the logistic regression model to compare if there was at least a tendency for model improvement (P ≤ 0.10) when a combination of 2 milk feeding behaviors were used in comparison to one behavior.

Alert testing phase
We tested the diagnostic accuracy of the alerts from our development phase to ensure that the alerts were positively classifying a calf at-risk of a diarrhea bout as sick or healthy.A clinical diarrhea calf was classified as a true positive when a calf had diarrhea and crossed the alert threshold at d −2 to d 1 relative to diarrhea diag-nosis.A clinical diarrheic calf was classified as a false negative when the calf did not cross the alert threshold within d −2 to d 1 relative to diarrhea diagnosis.A healthy calf was defined as a true negative when the calf did not cross the alert threshold within the first 28 d.A healthy calf was defined as a false positive when the calf crossed the alert threshold within the first 28 d.The Se, specificity (Sp), precision (Pre), accuracy (Acc), and positive likelihood ratios were calculated for each alert based on relative changes, or rolling dividends in milk intake, drinking speed, and rewarded visits:

Diagnostic accuracy: Alert selection criteria
We created diagnostic accuracy selection criterion for the milk behavior alerts with consideration of diarrhea bouts being the leading cause of preweaned calf mortality (Urie et al., 2018).We categorized the strength of the criteria observed for the logistic regression model output ROC AUC, Se, Acc, Pre, and Sp as follows: 0.00-0.30= negligible, 0.31-0.50= low, 0.51-0.70= moderate, 0.71-0.88= high, and 0.89-1.00= very high.Furthermore, we required that the positive likelihood ratio was moderate (≥1.1).Since Se and Sp are propor- tional, we expected that a high Se for the alert would decrease Sp and result in some false positive calves.Diagnostic accuracy was met when the milk feeding behavior alert data was at least highly explained by the ROC AUC curve (>0.71), and when the alert correctly classified calves as healthy or at-risk for diarrhea with high accuracy (Acc ≥0.80), high Se (≥0.80), and a very high precision (Pre) (≥0.89) was observed.We also required that the positive likelihood ratio for the alert was above 1.1 in the alert testing phase.The ROC AUC are presented for milk feeding behavior alerts which met our diagnostic accuracy criterion.

Holdout validation phase
We used a holdout validation strategy on all alerts which met our diagnostic accuracy selection criteria for both phases using the same methodology as described above, but with a different subset of calves.Calves used for holdout validation testing were fed 15 L/d from an AMF and raised at the same facilities as the alert development calves.We required that a true positive alert correctly identified a calf at-risk for their first diarrheal bout between d −3 to d 1 relative to their diarrheal bout diagnosis.The Se, Acc, Pre, Sp and positive likelihood ratios were calculated using data from the holdout validation phase for each alert that met our diagnostic accuracy criteria.A healthy calf was eligible to generate an alert for the first 28 feeder d to determine if healthy calves would cross the alert threshold during the time frame when calves are most likely to be at-risk for diarrhea (Urie et al., 2018).

Descriptive statistics
Overall, 72% (259/328) of calves from the data sets met enrollment criteria (i.e., remained healthy throughout the study, or were healthy before their first diarrhea bout).All calves enrolled in this study were also drinking independently from the AMF for at least 2 d before their diarrhea bout.Figure 1 is a flow diagram which illustrates reasons for loss to follow-up.Descriptive statistics for milk feeding behaviors (Mean ± SE; 95% CI) used for alert development from d -1 to d 0 relative to diarrhea diagnosis and the holdout validation phase from d −3 to d 1 relative to diarrhea diagnosis are in Table 2.

Descriptive statistics alert development phase
There was a high prevalence of diarrhea, with 84% (110/132) calves in the 28 d after introduction to the feeder having diarrhea.The first diarrhea bout occurred at an average of 9 d after introduction to the AMF (95% CI: 6 to 11 d).The average and 95% CI for milk intake, drinking speed, and rewarded visits for the sick calves approaching clinical diarrhea diagnosis (d −3 to d 1) are presented in Figure 2.

Descriptive statistics holdout validation phase
There was a high prevalence of diarrhea [88% (118/127 calves) in the 28 d after introduction to the feeder] and the first diarrhea bout occurred at an average age of 8 d (95% CI: 5 to 12 d).The Kentucky facility data set used in the holdout validation phase had only one healthy calf and thus, due to very high disease pressure Sp was not calculated for this data set.

Alert threshold and testing phase relative change
Relative change in milk intake.Relative change in milk intake for 10 L/d calf data for d −1 to 0 alert threshold was −0.29 (AUC ROC = 0.77, Youden's index = 0.54, Probability threshold = 0.71).The relative change in milk intake for 10 L/d calves did not meet diagnostic accuracy criteria during the alert testing phase for d −2 to 1 due to low Se and low Acc (Table 3).
Relative change in milk intake for 15 L/d calf data for d −1 to 0 alert threshold was −0.36 (AUC ROC = 0.80, Youden's Index = 0.57, Probability threshold = 0.64).The relative change in milk intake for the receiver operator curve for calves offered 15 L/d milk replacer with Youden's index probability threshold (Y), the optimal classification cut-off (C) and the maximum Euclidian's distance (D) for the alert threshold phase are in Figure 3. Relative change in milk intake met our diagnostic accuracy criterion for the 15 L/d alert testing phase for d −2 to 1 with high Se, high Acc and very high Pre (Table 3).
Relative change in drinking speed.Relative change in drinking speed for 10 L/d calf data for d −1 to 0 alert threshold was −0.29 (AUC ROC = 0.77, Youden's Index = 0.58, Probability threshold = 0.71).Relative change in drinking speed for 10 L/d calves did not meet diagnostic accuracy criteria for d −2 to 1 due to moderate Se and Acc (Table 3).
Relative change in drinking speed for 15 L/d calf data for d −1 to 0 alert threshold was −0.34 (AUC ROC = 0.77, Youden's Index = 0.50, Probability threshold = 0.66).Relative change in drinking speed for 15 L/d calves did not meet diagnostic accuracy criteria for d −2 to 1 due to low Se and moderate Acc (Table 3).
Relative change in rewarded visits.Relative change in rewarded visits for 10 L/d calf data for d −1 to 0 alert threshold was −0.28 (AUC ROC = 0.77,  3).
Relative change in rewarded visits for 15 L/d calves did not meet diagnostic accuracy criteria for d −2 to 1 due to low Se and low Acc (Table 3).

Collective diagnostic ability for relative changes in behavior.
There was no improvement in model performance when the collective diagnostic ability of relative change in milk intake was used with relative change in drinking speed and relative change in rewarded visits to classify 10 L/d calf data as healthy or at-risk for diarrhea from d −1 to d 0 (AUC ROC = 0.77, Chi-squared = 0.68, P = 0.88).The collective diagnostic ability of these 2 variables were not further assessed.
For 15 L/d calf data, there was no improvement in model performance when relative changes in milk intake was used with relative changes in drinking speed, or with relative changes in rewarded visits to classify 15 L/d calf data as healthy or at-risk for diarrhea from days −1 to 0 (AUC ROC = 0.80, Chi-squared = 3.14, P = 0.20).Thus, the collective diagnostic ability of these variables was not further assessed.
In summary, only relative changes in milk intake met our diagnostic criteria, and this was dependent on milk strategy (15 L/d).

Rolling dividends in milk intake
Rolling dividends in milk intake for 10 L/d calf data for d −1 to 0 alert threshold was 0.71 (AUC ROC = 0.77, Youden's Index = 0.58, Probability threshold = 0.70).The rolling dividends in milk intake for 10 L/d calves did not meet diagnostic accuracy criteria during the alert testing phase for d −2 to d 1 due to low Se and moderate Acc (Table 3).
Rolling dividends in milk intake for 15 L/d calf data for d −1 to d 0 alert threshold was 0.56 (AUC ROC = 0.82, Youden's Index = 0.50, Probability threshold = 0.56).The rolling dividends in milk intake for the receiver operator curve for calves offered 15 L/d milk replacer with Youden's index probability threshold, the optimal classification cut-off (C) and the maximum Euclidian's distance (D) for the alert threshold phase are in Figure 4.The rolling dividends in milk intake for 15 L/d calves met diagnostic accuracy criteria during the alert testing phase for d −2 to d 1 with high Se, high Acc, and very high Pre (Table 4).
Rolling dividends in drinking speed Rolling dividends in drinking speed for 10 L/d calf data for d −1 to d 0 alert threshold was 0.72 (AUC ROC = 0.77, Youden's Index = 0.57, Probability threshold = 0.72).Rolling dividends in drinking speed for 10 L/d calves did not meet diagnostic accuracy criteria for d −2 to d 1 due to Mod Se and Mod Acc (Table 4).
Rolling dividends in drinking speed for 15 L/d calf data for d −1 to d 0 alert threshold was 0.61 (AUC ROC = 0.81, Youden's Index = 0.57, Probability threshold = 0.61).Rolling dividends in drinking speed for 15 L/d calves did not meet diagnostic accuracy criteria for d −2 to d 1 due to low Se and low Acc (Table 4).
Rolling dividends in rewarded visits Rolling dividends in rewarded visits for 10 L/d calf data for d −1 to d 0 alert threshold was 0.72 (AUC ROC = 0.77, Youden's Index = 0.58, Probability threshold = 0.72).Rolling dividends in rewarded visits for 10 L/d calves did not meet diagnostic accuracy criteria for d −2 to d 1 due to low Se and low Acc (Table 4).
Rolling dividends in rewarded visits for 15 L/d calf data for d −1 to d 0 alert threshold was 0.73 (AUC ROC = 0.74, Youden's Index = 0.45, Probability threshold = 0.73).Rolling dividends in rewarded visits for 15 L/d calves did not meet diagnostic accuracy criteria for d −2 to d 1 due to low Se and low Acc (Table 4).

Collective diagnostic ability for rolling dividends in behavior
There was no improvement in model performance when dividends in milk intake was used with dividends in drinking speed and dividends in rewarded visits to classify 10 L/d calf data as healthy or at-risk for diarrhea from days −1 to d 0 (AUC ROC = 0.77, Chi-squared = 0.40, P = 0.94).
Rolling dividends in rewarded visits had significantly worse model performance (AUC = 0.75, Chi-squared =  6.02,P = 0.01) than rolling dividends in milk intake or rolling dividends in drinking speed for 15 L/d data and was removed from the model.There was a tendency for improvement in model performance when rolling dividends in milk intake was used with rolling dividends in drinking speed to classify 15 L/d calf data as healthy or at-risk for diarrhea from days −1 to d 0 (AUC ROC = 0.86, Chi-squared = 4.58 P = 0.10, Figure 5a).The rolling dividends in milk intake with drinking speed for calves offered 15 L/d milk replacer met our diagnostic criteria at an alert threshold of 0.60 (ROC AUC = 0.86, Classification criteria probability threshold = 0.68).The Youden's index probability threshold (Y), the optimal classification cut-off (C) and the maximum Euclidian's distance (D) for the ROC from rolling dividends in milk intake with drinking speed are shown in Figure 5b.In summary, rolling dividends in milk intake, and rolling dividends in milk intake with drinking speed met our diagnostic criteria, but this was dependent on milk feeding strategy (15 L/d).

Holdout validation phase
The diagnostic performance of the alerts for relative changes in milk intake, rolling dividends in milk intake, and the combination of milk intake or drinking speed were used in the holdout validation phase since they met the previous diagnostic accuracy criteria (Table 5).Due to a low number of healthy calves in both holdout validation data sets, the Sp for these algorithms was much lower than the alert testing phase.Indeed, likely due to high disease pressures, none of the milk feeding behavior alerts met our diagnostic criteria for the Kentucky facility (closest was relative change in milk intake with high Se, Mod Acc, and very high Pre).The relative changes in milk intake as sick calves approached diarrhea diagnosis from the Kentucky facility is shown in a scatter plot (Figure 6).However, the rolling dividends in milk intake with drinking speed alert achieved diagnostic accuracy for the Ontario facility with high Se, high Acc, very high Pre, but low Sp.The variability for rolling dividends in milk intake in combination with drinking speed is shown in a scatter plot for sick calves as they approach diarrhea diagnosis from the Ontario facility as this alert achieved diagnostic accuracy criteria (Figure 7).Finally, the relative change in milk intake alert only failed diagnostic accuracy criteria for the Ontario facility because of the positive likelihood ratio of 1; however, there was a high Se, high Acc, and very high Pre.

DISCUSSION
The objective of this diagnostic accuracy study was to develop and validate an alert based on a calf's changes from their previous 2-d milk intake, drinking speed, and rewarded visits surrounding the diagnosis of a diarrhea bout.We observed that during the alert development phase, relative changes in milk intake, rolling dividends in milk intake, and rolling dividends in milk intake with drinking speed achieved diagnostic accuracy for calves that were offered 15 L/d; however, none of these alerts achieved diagnostic accuracy for calves offered 10 L/d.Moreover, for the validation phase, an alert based on rolling dividends in milk intake with drinking speed achieved diagnostic accuracy with high Se, high Acc, very high Pre, but low Sp.We suggest that using the previous 2 d of rolling dividends in milk intake with drinking speed from an AMF with an alert threshold of (≤0.60) has diagnostic accuracy to detect dairy calves at-risk for a diarrhea bout when offered 15 L/d.However, due to the low specificity of the alert used in the validation phase, we suggest that this alert system should be further investigated in consideration of the prevalence of diarrhea on farm.
In this study, we observed that relative changes in milk intake, rolling dividends in milk intake, and roll-   ing dividends in milk intake with drinking speed had diagnostic accuracy for diagnosing a diarrhea bout in the calves.Generally, AMF calves decrease their milk intake before nonspecific disease diagnosis (Knauer et al., 2017).This is because sickness behavior is an induced motivational state by the brain, the release of pro-inflammatory cytokines signals the calf to decrease their energy expenditure (Hart and Hart, 2021).In fact, several studies observed that using decreased milk intake to diagnose calves with a diarrhea bout on d 0 had moderate diagnostic ability with balanced Se and Sp in AMF systems (Conboy et al., 2021   Milk intake with drinking speed for 15 L/d data met criteria for being tested collectively based on significant model improvement to diagnose a diarrhea bout in the calves as indicated from a logistic regression model (P < 0.10).

2
Diagnostic accuracy was met when the milk feeding behavior alert data was at least highly explained by the ROC AUC curve (>0.71), and when the alert correctly classified calves as healthy or at-risk for diarrhea with high accuracy (Acc ≥0.80), high Se (≥0.80), and a very high precision (Pre) (≥0.89) was observed.et al., 2022;Conboy et al., 2022).However, even the use of statistical control charts was not diagnostically accurate at finding calves at-risk for a diarrhea bout when solely milk intake was used (Knauer et al., 2018).Thus, before our study, decreased milk intake from the AMF was utilized as a screening tool rather than a diagnostic test to identify diarrheic calves (Knauer et al., 2018).In this study, we achieved diagnostic accuracy using a combination of milk intake and drinking speed to identify AMF calves at-risk for diarrhea bouts.Our approach differed from the forementioned studies because we used a calf's individual changes in milk behavior patterns rather than cut points defined through a comparison between healthy and sick individuals.We suggest that a high Se as observed in our study is preferred over a balanced Se and Sp to avoid missing a calf that requires supportive care.
Historically, the collective use of feeding behaviors to diagnose disease in cattle has improved diagnostic accuracy compared with using only one behavior.For example, the use of milk intake with drinking speed to diagnose diarrhea bouts in calves (Ghaffari et al., 2022), and multiple milk feeding behaviors with activity levels to diagnose respiratory disease bouts yielded high Se and high Acc when machine learning techniques were used (Bowen et al., 2021;Cantor et al., 2022).However, a limitation to the forementioned work was that it was exploratory in nature and the algorithms are not yet commercially available.Indeed, precision technology algorithms use rumination and activity behaviors collectively to alert for diseases in transition cattle because the collective Se is diagnostically accurate for the diagnosis of mastitis (Stangaferro et al., 2016a), digestive disorders (Stangaferro et al., 2016b), and metritis (Stangaferro et al., 2016c).We suggest that the collective use of rolling dividends in milk intake with drinking speed has diagnostic accuracy to identify a diarrhea bout in calves offered 15 L/d.Future research should evaluate the effectiveness of this alert to test intervention strategies to ameliorate diarrhea bouts in calves.
We were unable to achieve diagnostic accuracy of the alerts using AMF data from calves offered 10 L/d.The diarrheic calves in this study who were offered 10 L/d consumed on average 6 L/d before the diagnosis of a diarrhea bout, which may have limited the diagnostic ability of the alert to detect a change in their feeding behavior.However, healthy calves offered 10 L/d on an AMF also consumed considerably less milk than was offered to them (Rosenberger et al., 2017).We acknowledge that we were limited by calf enrollment age on the automated milk feeder as most of the 10 L/d calves enrolled in the study were still learning to drink milk independently from the AFS on d −4 relative to the diagnosis of a diarrhea bout.This is partially because calves were backgrounded in individual pens to ensure that they had a strong suckle before introduction to the automated milk feeder.More work is needed to determine if the diagnostic accuracy of an AMF feeding behavior alert can be achieved in calves offered 10 L/d who are enrolled on the feeder at birth.
One limitation to our study is that the Sp for these algorithms were low to moderate, particularly in the validation phase of this study.In general, we had a low number of healthy calves in this study which has been observed to decrease the Sp of a diagnostic test (Leeflang, 2008).We suspect that the milk behavior alerts failed to accurately diagnose diarrhea bouts in 10 L/d calves because of limited AMF visits, and less variability in milk consumption surrounding a diarrhea bout.However, it is important to quantify that there was a high prevalence of disease in these facilities in general, which is a known factor for compromising Sp, but not Se in diagnostic testing (Leeflang et al., 2009).  5.The holdout validation strategy phase of alerts which met diagnostic accuracy criteria: relative changes in milk intake, rolling dividends in milk intake, and rolling dividends in milk intake with drinking speed.The specificity, precision, accuracy, and specificity of milk feeding behavior alerts using 127 different calves' data relative to diagnosis of a diarrhea bout (d −3 to d 1).Calves were raised at the same facilities used in the alert A low Sp will generate false positive alerts, which may be frustrating to producers.However, diarrhea is endemic on dairy farms and the leading cause of morbidity and mortality in calves (USDA, 2018).Therefore, we recommend that an AMF alert with high Se, high Acc and very high Pre is worth the trade-off of misclassifying some healthy calves.
The high prevalence of diarrhea observed in this study is in line with what was reported by longitudinal studies that assessed fecal consistency daily for diarrhea bouts in calves (Sutherland et al., 2018;Lowe et al., 2019;Reis et al., 2021).It is possible that a low incidence of diarrhea in a population could affect the results of using an AMF alert to detect disease in calves.However, we suggest that this exploratory .The optimal diagnostic accuracy was not available because of 1 healthy calf in the data set (71% SE, 70% accuracy, 97% precision) for diagnosing diarrhea (alert 0.36).
research demonstrates that feeding behaviors are useful for alert development to indicate a diarrhea bout in 15 L/d AMF calves when calves experience loose feces in the preweaning phase.This alert development is in alignment with other epidemiological studies and precision livestock farming algorithms and analytics, where many precision sensors perform differently depending on the environmental context, such as the housing environment and ages of animals included (Reynolds et al., 2019).
It is a common criticism in computer science that sensor data, or the "internet of things" predict disease status with scarce ground truth data, and seldom are these precision alerts re-evaluated once they are de- veloped (Chatterjee and Ahmed, 2022).While the use of more complex algorithms such as machine learning with deep convolutional networks is desirable in the future, this would require ground truth data on several hundred dairy calves to be accurate, especially to avoid over fitting the model (Yamashita et al., 2018).Thus, to develop an AMF alert in diarrheic dairy calves, daily health exams for ground truth data are required, and the alert should be evaluated for high sensitivity and accuracy to indicate calves at-risk of a diarrhea bout.Moreover, we suggest that further research would be needed to implement the AMF alert on the dairy including software updates, software training materials, and research strategies to implement an AMF alert into a calf health protocol to transition this research to a commercial setting.Thus, we suggest that future work in alert development with precision livestock farming concepts should be designed with consideration of the environmental context, age of the animals, and in the case of dairy calves, the milk feeding strategy.Further, we validated this alert for use in the CMF 1000 AMF system (Forster-Technik, Engen, Germany) because this was the system used in our research facilities.It is unknown if this alert would function on other AMF systems, but we suggest it is a start toward making AMF data more diagnostically useful for dairy producers.
Relative changes in drinking speed and rolling dividends in drinking speed also had low to moderate diagnostic performance to indicate a calf at-risk for a diarrhea bout for both milk feeding strategies in this study.Consistent with our results, a recent study observed that drinking speed had moderate sensitivity and specificity for indicating disease status (i.e., diarrhea and/ or respiratory disease) in AMF calves offered 9 to 10 L/d (Morrison et al., 2022).Drinking speed indicates how quickly a calf retrieved their meal from the AMF (L/min) using a daily average.Drinking speed is highly variable by calf, and drinking speed is influenced by how much milk is offered to the calf each day (Cantor et al., 2019).It is possible that drinking speed is a measure of hunger in healthy calves, but this has been unexplored.We suggest that because of individual variation, some studies may have observed that drinking speed was associated with disease status in AMF calves and others have not as reviewed by (Sun et al., 2021).This high variability in drinking speed by individual calves likely limited our ability to select an alert threshold since we used optimal correct classification criteria to select our alert thresholds.However, this does not indicate that drinking speed is not useful per se, as drinking speed used in combination with milk intake led to an alert that achieved diagnostic accuracy.
In this study, relative changes in rewarded visits and rolling dividends in rewarded visits had the worst diagnostic performance to indicate a calf at-risk for a diarrhea bout with low to moderate Se and Acc for both feeding strategies.Rewarded visits reflect a calf's ability to suckle milk successfully from an AMF.In semi-natural settings, a dairy calf suckles the dam in a diurnal feeding pattern and consumes 4 to 8 meals per day as reviewed by (Miller-Cushon and DeVries, 2015).Previous studies with similar stocking densities to our study have reported that the frequency of suckling bouts is dependent on milk availability; where attempted suckling bouts are higher for AMF calves with restricted meal settings (Jensen, 2004).In this study, we observed that the variability of rewarded visits was low (1/d) surrounding diarrhea diagnosis compared with milk intake and drinking speed.It is possible that such low variation in this behavior reflects calf satiety, which supports why unrewarded visits were also low.Our low variability in rewarded visits in our calves is consistent with others who observed that rewarded visits were lower in calves offered 10 to 12 L/d compared with 6 L/d calves (Rosenberger et al., 2017).Thus, we suggest that relative changes and rolling dividends in rewarded visits to the AMS in young calves less than 2 wk of age are not a useful indicator of diarrhea bouts before clinical diagnosis of disease.

CONCLUSION
In summary, we found that rolling dividends in milk intake with drinking speed was diagnostically accurate, with high sensitivity, accuracy, and very high precision as an alert for indicating a diarrhea bout in 15 L/d dairy calves fed by an automated milk feeder (d −3 to d 1 relative to diarrhea diagnosis).However, we were unable to achieve diagnostic accuracy for 10 L/d calves using relative changes or rolling dividends in milk feeding behavior.While the specificity of these alerts was low in the validation phase, the incidence of diarrhea was high, and we suggest that the use of these alerts to provide early intervention strategies to calves may be fundamental for potentially improving calf health outcomes in the future.It is important to acknowledge that this AMF alert was developed to identify calves at risk for a diarrhea bout when offered 15 L/d, in smaller group sizes (max 15 calves), with high disease pressures.
would like to thank Jannelle Morrison, Mikayla Ringelburg, Dan Guevera Mann, Gustavo Mazon, Jason Simmons, Maria Eduarda Reis, and Maryam Alhamdan for their invaluable help to collect the data used in the holdout validation phase.The authors report no conflict of interest.
Cantor et al.: CHANGES IN MILK INTAKE OR DRINKING SPEED IDENTIFY CALF DIARRHEA cally tested throughout data curation (k ≥0.90).Fecal consistency following Renaud et al. (2020) was defined as either normal (0 = solid, or 1 = pasty), or diarrhea [defined as 2 consecutive days of loose feces that sifted through the bedding (fecal score = 2), or at least one day of watery fecal consistency (fecal score = 3)].
Cantor et al.: CHANGES IN MILK INTAKE OR DRINKING SPEED IDENTIFY CALF DIARRHEA case did not have high model leverage (≤0.15), high deviance residual, or high Pearson residuals (≤1.5).
Cantor et al.: CHANGES IN MILK INTAKE OR DRINKING SPEED IDENTIFY CALF DIARRHEA Youden's Index = 0.58, Probability threshold = 0.72).Relative change in rewarded visits for 10 L/d calves did not meet diagnostic accuracy criteria for d −2 to 1 due to moderate Se and moderate Acc (Table Figure 1.A participant flow diagram demonstrating data enrollment and loss to follow-up for the alert development phase and holdout validation phase of a milk feeding behavior alert developed to indicate a diarrheal bout in preweaned dairy calves.
Cantor et al.: CHANGES IN MILK INTAKE OR DRINKING SPEED IDENTIFY CALF DIARRHEA

Figure 2 .
Figure 2. The mean ± 95% CI daily feeding behavior for 110 sick calves used in the alert development phase: milk intake (Figure 2A), drinking speed (Figure 2B) and rewarded visits (Figure 2C) using data from Ontario Dairy Research Centre (Ontario) calves offered up to 15 L/d and data from the University of Kentucky Dairy Research Farm (Kentucky) calves offered up to 10 L/d from an automated feeder.
Cantor et al.: CHANGES IN MILK INTAKE OR DRINKING SPEED IDENTIFY CALF DIARRHEA

Figure 3 .
Figure 3.The relative change in milk intake alert threshold (−0.36) generated for a receiver operator curve (ROC) with diagnostic accuracy from a logistic regression model which classified 15 L/d calf data (132 calves) as healthy or at-risk for diarrhea from d −1 to d 0 relative to diarrhea diagnosis.
Figure 4. a-b.The rolling dividends in milk intake alert threshold (0.56) generated for a receiver operator curve (ROC) with diagnostic accuracy from a logistic regression model which classified 15 L/d calf data (132 calves) as healthy or at-risk for diarrhea from d −1 to d 0 relative to diarrhea diagnosis.

Table 4 .
The rolling dividend milk feeding behavior alert testing phase: The sensitivity, precision, accuracy, and specificity of an alert generated from individual calf data based on rolling dividends in milk feeding behavior 1 to alert 2 for a calf destined for a diarrhea bout (d −2 to d 1) using data from the Ontario Dairy Research Centre (15 L/d) or the University of Kentucky Dairy Research Farm (10 L/d) from an automated milk feeder)

Figure 5 .
Figure 5.The rolling dividends in milk intake with drinking speed alert threshold (0.60) generated for a receiver operator curve (ROC) with diagnostic accuracy from a logistic regression model which classified 15 L/d calf data (132 calves) as healthy or at-risk for diarrhea from d −1 to d 0 relative to diarrhea diagnosis.
Cantor et al.: CHANGES IN MILK INTAKE OR DRINKING SPEED IDENTIFY CALF DIARRHEA Table Figure 6.A scatter plot (each symbol color combination represents a calf ID) showing individual sick calf variability (Kentucky Dairy Research Farm) in relative changes in milk intake as they approached diarrhea diagnosis in the holdout validation phase (d −3 to d 1).The optimal diagnostic accuracy was not available because of 1 healthy calf in the data set (71% SE, 70% accuracy, 97% precision) for diagnosing diarrhea (alert 0.36).
Figure 7.A scatter plot (each symbol color combination represents a calf ID) showing individual sick calf variability (Ontario Research Dairy Centre) in rolling dividends in milk intake with drinking speed as they approached diarrhea diagnosis from the holdout validation phase (d −3 to d 1).The optimal diagnostic accuracy was achieved (85% SE, 82% accuracy, 94% precision, 50% SP) for diagnosing diarrhea (alert 0.60).
Cantor et al.: CHANGES IN MILK INTAKE OR DRINKING SPEED IDENTIFY CALF DIARRHEA

Table 1 .
Cantor et al.: CHANGES IN MILK INTAKE OR DRINKING SPEED IDENTIFY CALF DIARRHEA A demonstration of the days relative to diarrhea diagnosis on d 0 and the milk volume offered to calves at-risk for a diarrhea bout 1for an alert based on relative changes or rolling dividends in milk intake, drinking speed, or rewarded visits from an automated milk feeder 1 Diarrhea bouts were detected on the first d that a calf had a fecal consistency which was watery for at least one d or loose sifts through the bedding for two consecutive d.All calves were health examined daily.

Table 2 .
Descriptive statistics for milk feeding behaviors (Mean ± SE; 95% CI) used for alert development from d -1 to d 0 relative to diarrhea diagnosis (Alert phase) and the holdout validation phase from d −3 to d 1 relative to diarrhea diagnosis 1 (Valid phase) for diarrheic calves (n = 228/259) fed by an automated milk feeder Diarrhea bouts were detected on the first d that a calf had a fecal consistency which was watery for at least one d or loose sifts through the bedding for two consecutive d.All calves were health examined daily.

Table 3 . The relative change milk feeding behavior alert testing phase:
The sensitivity, precision, accuracy, and specificity of an alert 1 generated from individual calf data based on relative changes in milk feeding behavior to alert for a calf destined for a diarrhea bout (d −2 to d 1) using data from Ontario Dairy Research Centre (15 L/d) or the University of Kentucky Dairy Research Farm (10 L/d) from an automated milk feeder 1 development phase: Ontario Dairy Research Centre (84 calves) or University of Kentucky Dairy Research Farm, (43 calves) which fed calves a maximum 15 L/d from an automated feeder Diagnostic accuracy was met when the milk feeding behavior alert data was at least highly explained by the ROC AUC curve (>0.71), and when the alert correctly classified calves as healthy or at-risk for diarrhea with high accuracy (Acc ≥0.80), high Se (≥0.80), and a very high precision (Pre) (≥0.89) was observed.NA* Not applicable for high milk feeding strategy holdout validation phase 1 healthy in data set.