{"id":9916,"date":"2022-12-20T09:35:19","date_gmt":"2022-12-20T08:35:19","guid":{"rendered":"https:\/\/www.iese.fraunhofer.de\/blog\/?p=9916"},"modified":"2025-09-09T15:08:36","modified_gmt":"2025-09-09T13:08:36","slug":"data-quality-assessment-in-agriculture","status":"publish","type":"post","link":"https:\/\/www.iese.fraunhofer.de\/blog\/data-quality-assessment-in-agriculture\/","title":{"rendered":"Data Quality Assessment in Agriculture"},"content":{"rendered":"

Nowadays, it has become almost inconceivable to imagine agriculture without sensors, whether they are simple GPS devices to help farmers optimize their work in the fields, monitor livestock, or more complex equipment allowing advanced monitoring of animals (e.g. animal welfare detection) or farm machinery. In particular, in the horizon of precision farming (or smart farming), smart sensors generate (geo-referenced) data, that make it possible to support the farmer with optimization, decision support, or prediction by means of Artificial Intelligence. For example field operations like fertilization, seeding, or harvesting can be optimized. A well-known challenge in smart farming is to reduce the energy consumption of the sensors without losing data quality. Another one is the latency of the connected sensors, which may cause problems when the generated data is used for further analysis.<\/p>\n

A multitude of data is produced and analyzed by specialized software or is available in their raw form to farmers. However, the results of this kind of analysis must be used with caution – indeed, the adage „garbage-in, garbage-out“ is too often neglected. The wrong unit, the wrong calibration of a sensor or the use of the wrong geo-reference are „simple“ errors which \u2013 if detected too late\u2013 can influence the outcome of the analysis and might result in serious consequences.<\/p>\n

In general, data quality assessment needs to tackle the following obstacles:<\/p>\n

    \n
  1. What standards or norms should be used?<\/li>\n
  2. What can be measured or analyzed? Which problems should be detected and highlighted, which problems are of little importance or can be ignored?<\/li>\n
  3. What happens after the problems are detected? How to communicate and display the detected problems and to whom?<\/li>\n<\/ol>\n

    A clear definition of data quality and its impact in the context of agriculture \u2013 especially in the field of smart farming \u2013 is hard to find. Thus, we aim to answer the three questions above by working closely with different partners and domain experts. In the European Horizon 2020 project DEMETER<\/a>, we deeply investigated the topic of data quality and developed a tool to detect potential data quality problems in structured and semi-structured files.<\/p>\n

    Data quality problems<\/h2>\n

    In general, the quality of sensor data must be ensured to get proper decision support systems that can help farmers to improve their businesses. We focus on the data itself, e.g. data quality problems such as corrupted, missing, invalid, incredible, noisy, inconsistent or unreliable data. Besides, other mistakes such as outliers<\/a>, anomalies or are issues caused by data that is not up to date or by sensors that are stuck at zero, can occur. Various problems \u2013 not only technical ones \u2013 can arise during field operations: it can be a sensor deficiency, the lack of GPS connection or sometimes an error made by the farmer because some values (e. g. the brand name of a chemical) must be added manually into the appropriate systems. Such errors have to be detected automatically so that a decision can be made about whether the data is proper to be used for further analysis or whether it should be handled at first. Otherwise, erroneous data in a decision support system can result in misleading outcomes.<\/p>\n

    In literature, there are several approaches to classify and categorize data quality problems. Errors can be of different natures, thus they are not necessarily critical or even problematic. For instance, let\u2019s take missing values: when no fertilizer is being applied or when the machine is outside the field boundaries (e.g. when the driver is turning), the machine will record the GPS coordinates but the sensor for fertilizer application will return default values. Therefore, it is no surprise that many zero-values or even many missing values arise. Another question is whether there are only operational interests (for optimization) or whether certain regulations (evidence for administrative controls) must be respected. In the latter case where a running machine must meet regulations the completeness of the records plays a far more important role. The picture below shows a typical example of a file that contains undesired recordings related to missing values.<\/p>\n