Skip to main content

Evaluation Metrics for GEN AI

 Evaluation Metrics for GEN AI 

Mg. en Ing. Carlos Gerardo Said 




Whether dealing with LLM, computer vision, clustering, predictive analytics, synthetization, or any other AI problem, the goal is to deliver high quality results in as little time as possible.  

Typically, you assess the output quality after producing the results, using model evaluation metrics. 

These metrics are also used to compare various models, or to measure improvement over the baseline.

In unsupervised learning such as LLM or clustering, evaluation is not trivial.  The best evaluation metrics may be hard to implement and compute.

At the same time, pretty much all modern techniques rely on minimizing a loss function to achieve good performance. 

In particular, all neural networks are massive gradient descent algorithms that aim at minimizing a loss function. 

The loss function is some cases basic (for instance, sums of squared differences) because it must be updated fast each time a neuron gets activated and a weight is modified. There may be several millions of changes needed before getting a stable solution.

In practice, the loss function is a proxy to the model evaluation metric: the lower the loss, the better the evaluation. 

Using Model Evaluation as the Loss Function

The full multivariate KS distance between  the real and generated data seems to be the best evaluation metric. It takes into account all potential interactions among all the features, but it requires a lot of computing time. The multivariate Hellinger distance (https://math.stackexchange.com/questions/243792/hellinger-distance-between-gaussians-multivariate-and-univariate-forms) is an alternative more easy to implement.

However, it depends on the chosen granularity in the highly sparse feature space. There is an easy way to do it, not requiring more bins than the number of observations in the training set regardless of the dimension. And it leads to fast atomic updates, making it suitable as a loss function. You need to start with a low granularity, that is, a rough approximation. Then increase the granularity at regular intervals until the Hellinger and KS distances (https://stats.stackexchange.com/questions/9311/kullback-leibler-vs-kolmogorov-smirnov-distance) are equivalent. Thus, the loss function changes over time. 





Using the evaluation metric as the loss function leads to great improvements. The algorithm reallocates observed values via a large number of swaps, for each feature separately. The best possible synthetization — the global optimum (https://www.sciencedirect.com/topics/mathematics/global-optimum)  — is the real data itself, up to a permutation of the observations.

Combinatorial algorithms are also able to retrieve it but require more iterations. Neural networks also require a more time and won’t retrieve the global optimum.

You should  put constraints on the desired synthetization, for instance, requesting the Hellinger distance to stay above some threshold at all times.  The result is a constrained synthetization.

Constrained synthesis can refer to various computational problems and solutions, but commonly it involves generating solutions (such as machine learning models) that satisfy specific constraints. You can get constrained synthesis using Python, utilizing the scipy.optimize (https://docs.scipy.org/doc/scipy/reference/optimize.html) module.


Comments

Popular posts from this blog

IBM goes big to keep quantum computers cold

IBM - avances en mantener adecuadamente fríos computadores cuánticos.

Estructura multinivel de los sistemas de agentes de IA