Machine Vision System Tracks Vineyard's Crop Yields

Gloria Ferrer Caves & Vineyards is implementing an AI-enabled machine vision system to track its grapes during the growing season.

Spanning both the Napa and Sonoma winemaking regions, Gloria Ferrer (Los Carneros, CA, USA) produces primarily sparkling wines from pinot noir and chardonnay grapes.

The family-owned company grows the grapes on two ranches—207-acre Home and 128-acre Circle Bar. As the company explains on its website, “Pinot Noir is planted in the rocky, volcanic soils on the estate’s highest ridges. Chardonnay is planted in the clay-rich soils on the lower sections of the estate.”

Growing grapes in vineyards is the first step in a multi-year process that also includes producing, bottling, selling and distributing wine. At Gloria Ferrer, freshly harvested grapes won’t be available as wine for sale for about three years, according to Brad Kurtz, vineyard director for Gloria Ferrer.

To fill this supply pipeline, Gloria Ferrer’s salespeople estimate sales three years in advance and then the vineyard’s managers grow the volume of grapes necessary to bottle enough wine for that vintage.

Manually Estimating Crop Yields

This means that the vineyard’s managers must estimate the number of grapes they’ll produce in advance of an actual harvest, so they know whether they will have too many grapes, too few grapes or just enough to meet salespeople’s projections.

Crop estimation is “the biggest data collection effort for vineyards. It has the biggest implication as far as the financial reach of the of the business,” explains Kurtz.

Traditionally, grape growers have estimated crop volumes manually. “I am going out and actually counting grape clusters on a vine,” Kurtz says. “It may take upwards of two hours to count 10 vines, and you’re talking about a vineyard that has 10,000 vines on it,” he says, adding that a sample size of one out of 1,000 vines is typical but also “incredibly small.”

The result of this painstaking effort is not ideal. “This sort of manual data collection causes massive swings in estimates, and it’s an industrywide problem,” he explains, adding that predictions within 25% of the actual yield at harvest are considered quite good. “It’s so difficult to get the sample sizes, and there is so much variability within the vineyards.”

The financial impact of incorrect estimates can be substantial. Kurtz gives this example: If the vineyard has a target of producing 1,200 tons of grapes valued at about $3,000 per ton, a 1% or 2% miss in forecasting the actual yield could result in having either too much capital (up to $72,000) tied up in grapes or a last-minute scramble to purchase additional grapes.

In addition to manually counting and inspecting vines and grapes, another data-gathering method grape growers use involves taking images of the vineyard canopy using multispectral cameras aboard low-altitude fixed-wing aircraft.

This approach results in maps showing where the vines are growing the strongest and where they are the weakest.

Gloria Ferrer has augmented manual estimates with aerial canopy imagery since 2003, but Kurtz wanted to add even more automation to the crop tracking process. “It's only collecting overhead imagery, which only tells us a limited amount of information,” he says.

While working at a previous employer, he implemented a crop scouting system that uses machine vision and deep learning to track crops using images taken at ground level. In 2022, Kurtz began using the same technology at Gloria Ferrer. 

In addition to improving data on crop volume, he also believes the information he collects will help him manage the health of each vine, maximizing both yields and the quality of the grapes. The information would show him which areas of the vineyard need more attention than others.

Incorporating 3D Machine Vision

The solution he chose is from Bloomfield Robotics (Pittsburgh, PA, USA). “We're giving them the option and the availability to look at all their plants on a continuous basis,” explains Hayden Wolf, director of operations at Bloomfield. “It allows you to make management decisions in a completely new way when you're operating based on facts rather than on something that you think is statistically significant.”

The centerpiece of the solution is a specially designed stereo vision camera that can be mounted “on anything that moves,” such as an all-terrain vehicle, utility-terrain vehicle or autonomous vehicle, Wolf says.

The camera and an accompanying Android tablet hook into the vehicle’s power source via standard cables, and then the camera snaps images of the vines and grape clusters as the vehicle moves up and down the rows of plants.

The system is designed to be installed in minutes, so it can collect images as a vineyard employee is completing another task, such as spraying weeds or mowing, Wolf explains. The tablet displays an interface that prompts users to position and calibrate the camera correctly. The user interface also provides feedback during the image collection process.

Slightly smaller in size than a microwave oven, the RGB camera includes two 12 MPixel sensors, two lenses, 12 LED lights and a GPS antenna.

Using GPS, “we localize every single image we take,” Wolf explains. “That way we can know that you're looking at the 6th plant on the 7th row every single time you pass it.”

The camera typically gathers 1 TB of image data during a six- to eight-hour collection event.

No matter how much ambient light is available in the outdoor environment, the LEDs flash with every shot in a process Wolf describes as “active lighting.”

Wolf says uniform lighting is important even outdoors to ensure the camera captures images that are of sufficient and consistent quality to allow comparative analysis. “It helps us to normalize the imagery data when we’re actually applying the AI models to it further down the line,” he says. “It makes it more efficient for us to train the models and the models actually perform better on the back end.”

All the images are captured and stored on an SSD card, which the grower inserts into a port on the camera.

Using AI Models to Analyze Data

After every collection event, the grower removes the card, slips it into a PC, and transmits the images to Bloomfield’s cloud, where the data is organized by plant row from left to right as sequenced by the GPS. 

Bloomfield then applies the AI models to the images, producing descriptive results, such as counting or measuring shoots or identifying the colors of grape clusters.   

Growers, such as Gloria Ferrer, access the data via a web-based front end. They can drill down to the plant level or view data by geographic area, such as by block, which is a management unit that growers use to identify groups of plants.

Presenting data in this fashion allows growers to make comparisons, such as counting grape clusters per foot on Block 1 versus Block 2. Growers also can export information into a spreadsheet or PDF to share with team members working in the field.

Gloria Ferrer began capturing images about every two weeks on 30 acres in 2022, increasing to 50 acres in 2023.

Kurtz says he plans to add additional acres over time as he convinces himself of the solution’s benefit to the business.

Addressing Implementation Challenges

Collecting consistent images of grape clusters is important to the success of calculating crop volumes and tracking the health of the vines. But gathering those images isn’t always easy. A key issue is getting enough images of the grape clusters over time—even though the clusters at times could be hidden behind leaves, meaning that the camera may not capture images of them with each pass through the vineyard.  

Kurtz says Gloria Ferrer found this to be an issue in 2022 because the team did not start snapping images early enough in the season in 2022. As a result, they did not get enough usable visual data over time.  

In 2023, crew members began taking images as soon as they finished pruning the vines, which occurs in January and February. This should allow them to collect information on the number of buds, followed by shoots and then grapes, Kurtz explains.

They plan to continue collecting images until they harvest the grapes, which occurs in August for sparkling wine and September for still wine.

Kurtz says he expects to use the information they’ve collected in 2023 to estimate the size of the crop. If his calculations are within 5% of the actual volume, he will deem the project a success.

Planning Next Steps

For its part, Bloomfield’s staff is continually updating and improving the system.

Currently in its third commercial season, Bloomfield uses a subscription model in which growers pay a fee, based on the number of cameras, to cover a growing season. Bloomfield chose a subscription model so it can maintain control of the cameras. “If we want to upgrade something or if we want to swap out something on the fly mid-season, we can do that and just give them a newer camera,” Wolf says.

In addition to updating the hardware, Bloomfield also plans to move more image processing to the server or camera and, eventually, also complete AI inferencing at the edge.

Overall, Kurtz believes the vision solution has a lot of potential to improve crop management. “I am personally very excited about the system,” he says. “You can build out maps of pretty much any metric that you can train the system on.” In addition to crop estimates, this could include measures of plant vigor, grape ripening or signs of a pest outbreak. “There’s so many potential ways we can utilize this technology in the future,” he concludes.