Merge branch '5-results' into 'develop'

Resolve "Results"

Closes #5

See merge request !10

Fazeli_Shahroudi-Sepehr-Mastersthesis.tex

@@ -10,15 +10,16 @@
 % \usepackage{fancyhdr}
 \usepackage{scrlayer-scrpage}
 \usepackage{listings}
-\usepackage{xcolor}
+% \usepackage{xcolor}
 \usepackage{scrhack}
 \usepackage{longtable}
 \usepackage{tabularx}
-% \usepackage[table]{xcolor}
+\usepackage[table]{xcolor}
 \usepackage{array}
 \usepackage[T1]{fontenc}
 \usepackage{textcomp}
 \usepackage{enumitem}
+\usepackage{multirow}
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 \newcommand{\thesistitleDE}{Vergleichende Evaluierung von Bildverarbeitungsbibliotheken für industrielle Anwendungen bei Dassault Systems}
 \newcommand{\thesistitleEN}{Comparative Evaluation of Image Processing Libraries for Industrial Applications at Dassault Systems}
@@ -32,9 +33,9 @@
 \newcommand{\degree}{Master of Engineering (M.Eng.)}
 % \let\oldsection\section
 % \renewcommand{\section}{\clearpage\oldsection}
-\let\oldlongtable\longtable
-\let\endoldlongtable\endlongtable
-\renewcommand{\endlongtable}{\endoldlongtable\clearpage}
+% \let\oldlongtable\longtable
+% \let\endoldlongtable\endlongtable
+% \renewcommand{\endlongtable}{\endoldlongtable\clearpage}
 % \let\oldsubsection\subsection
 % \renewcommand{\subsection}{\clearpage\oldsubsection}
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

chapters/4-Results.tex

-\chapter{Results}
\ No newline at end of file
+\chapter{Results}
+
+This chapter presents our findings from the benchmarking experiments conducted to evaluate the performance of alternative image processing libraries. The results include quantitative data on image conversion and pixel iteration times, as well as memory consumption for each library or combination tested. The data generated will be used to answer the research question and support the hypotheses formulated in the previous chapters. The benchmarking approach consisted of running two primary tests on each library: an image conversion test that measured the time taken to load, process, and save images, and a pixel iteration test that recorded the time required to process every pixel in an image for a grayscale conversion. These experiments were performed in a controlled environment, with warm-up iterations included to reduce the impact of initial overhead. Memory consumption was tracked alongside processing times using BenchmarkDotNet, thereby offering a complete picture of both speed and resource utilization.\\
+
+%%[PLACEHOLDER: a media summarizing benchmarking methodology]
+
+Before discussing the results in detail, it is important to review the benchmarking design. In this study, each library was tested under the same conditions: the same input image was used, a fixed number of warm-up iterations were performed to reduce the effects of just-in-time compilation and caching, and finally, 100 main iterations were executed to ensure reliable statistics. For the image conversion test, the time measured was the duration needed to load a JPEG image, convert it to PNG, and save it back to disk. In the pixel iteration test, the focus was on recording the time required to access and change each pixel for producing a grayscale version of the image.
+
+Memory diagnostics were captured concurrently, with particular attention to allocated memory and garbage collection events. This dual approach ensured that our results were not solely focused on speed but also took into account the resource efficiency of each solution.
+
+
+%%[PLACEHOLDER: a media Diagram of benchmarking process] or reference to it
+
+\input{sections/Chapter-4-sections/Image_conversion_benchmark_results.tex}
+\input{sections/Chapter-4-sections/Pixel_iteration_benchmark_results.tex}
+\input{sections/Chapter-4-sections/Memory_benchmark_results.tex}
+\input{sections/Chapter-4-sections/Analysis_and_Interpretation_of_Results.tex}
+\input{sections/Chapter-4-sections/Summary.tex}

sections/Chapter-4-sections/Analysis_and_Interpretation_of_Results.tex

+\section{Analysis and Interpretation of Results}
+
+As the final benchmarking results were collected and plotted, the emerging trends provided critical insights into the efficiency of various image processing libraries. The raw numerical data from our benchmarking suite provided an answer to the research question, but a deeper interpretation of these results allowed us to refine our understanding of the trade-offs and strengths of each alternative. This section explores the relationship between speed and memory usage, compares the empirical findings with theoretical expectations, and discusses the implications for real-world applications.
+
+\subsection{Comparison of Performance Trends}
+
+The performance hierarchy observed in the benchmarking results closely aligns with expectations based on each library’s internal architecture. Libraries such as OpenCvSharp and Emgu CV, both built upon OpenCV’s optimized C++ backend, showcased superior execution times for image conversion tasks. This efficiency is largely attributed to OpenCV’s reliance on low-level SIMD (Single Instruction, Multiple Data) optimizations and hardware-accelerated processing paths.
+
+Conversely, ImageSharp—despite its clean API and pure C\# implementation—demonstrated significantly higher processing times, reinforcing the general principle that managed code introduces overhead compared to native libraries. While ImageSharp remains a viable option for applications prioritizing ease of use and portability over raw performance, the performance disparity is undeniable.
+
+Magick.NET, though powerful and highly flexible in terms of format support, performed noticeably worse in pixel iteration tasks. This result was somewhat anticipated due to the internal structure of ImageMagick, which prioritizes format conversions and high-quality rendering over raw pixel access speed. The excessive processing times observed in the Magick.NET pixel iteration benchmark further support the hypothesis that it is not optimized for this type of operation.
+
+The trends in memory consumption were particularly revealing. While OpenCvSharp + SkiaSharp exhibited minimal memory allocation, Emgu CV+Structure.Sketching, despite its processing speed, required substantially higher memory overhead. This observation is consistent with Emgu CV’s underlying OpenCV core, which relies on large temporary buffers and matrix structures for intermediate computations. In contrast, ImageSharp demonstrated exceptional memory efficiency during pixel iteration but was significantly slower, suggesting that its architecture prioritizes memory conservation over execution speed.
+
+\subsection{Trade-Offs Between Speed and Memory Usage}
+
+The relationship between speed and memory consumption is a recurring theme in performance optimization. Our results underscore that achieving optimal speed often comes at the cost of increased memory usage. Emgu CV+Structure.Sketching exemplifies this trade-off: while its pixel iteration speed was among the best recorded, it consumed significantly more RAM than ImageSharp.
+
+The implications of these trade-offs depend heavily on the intended application. For environments where processing speed is paramount—such as real-time video processing or AI-powered image enhancement—Emgu CV’s increased memory footprint may be an acceptable compromise. However, in resource-constrained applications (e.g., embedded systems, mobile devices, or cloud-based deployments with strict memory limits), a lower-memory alternative like ImageSharp may be more suitable despite its lower speed.
+
+\setlength{\columnWidth}{0.10\textwidth}
+\begin{longtable}{|>{\raggedright\arraybackslash}p{0.26\textwidth}|>{\raggedright\arraybackslash}p{0.43\textwidth}|>{\raggedright\arraybackslash}p{\columnWidth}|>{\raggedright\arraybackslash}p{\columnWidth}|}
+\hline
+\rowcolor{purple!30}
+\textbf{Library} & \textbf{Task} & \textbf{Speed} & \textbf{Memory Usage} \\
+\hline
+\endfirsthead
+
+\hline
+\rowcolor{purple!30}
+\textbf{Library} & \textbf{Task} & \textbf{Speed} & \textbf{Memory Usage} \\
+\hline
+\endhead
+
+\multirow{2}{*}{\textbf{ImageSharp}} & Image Conversion & Slow & Low \\
+\cline{2-4}
+ & Pixel Iteration & Fast & Low \\
+\hline
+\multirow{2}{*}{\shortstack{\textbf{Emgu CV + SkiaSharp}}} & Image Conversion (SkiaSharp) & Fast & Low \\
+\cline{2-4}
+ & Pixel Iteration (Emgu CV) & Fast & High \\
+\hline
+\caption{Speed vs. Memory Usage Trade-Offs}
+\label{tab:speed-memory-trade-offs}
+\end{longtable}
+
+One particularly interesting finding was that OpenCvSharp+SkiaSharp consistently delivered both high speed and low memory usage for image conversion. This anomaly suggests that this combination strikes an optimal balance, leveraging OpenCV’s native optimizations while maintaining a lightweight footprint in memory. The fact that this hybrid approach outperformed even standalone OpenCV libraries further supports the notion that combining high-performance native libraries with efficient rendering engines can yield superior results.
\ No newline at end of file

sections/Chapter-4-sections/Image_conversion_benchmark_results.tex

+\section{Image Conversion Benchmark Results}
+
+The image conversion benchmark was performed using ImageSharp and Magick.NET as well as SkiaSharp and Structure.Sketching which were the chosen libraries in their combinations with OpenCvSharp and Emgu CV, respectively for the conversion task. Using the same 4k resolution image, the benchmark measured the time taken to convert the image from JPEG to PNG format. Comparing the results of these libraries provides insights into their performance and efficiency in application scenarios where rapid image conversion is required—such as real-time image processing pipelines or high-volume batch processing environments. The data thus answer one of our central question to which library can provide significantly faster image conversion, thereby supporting the hypothesis discussed in earlier chapters.
+
+ImageSharp recorded an average conversion time of approximately 2,754 milliseconds. In contrast, the combination of OpenCvSharp with SkiaSharp delivered an average conversion time of only 539 milliseconds. Similarly, Emgu CV integrated with Structure.Sketching achieved an average time of 490 milliseconds, while Magick.NET paired registered an average conversion time of 4,333 milliseconds.
+
+\newlength{\columnWidth}
+\setlength{\columnWidth}{0.19\textwidth}
+\vspace{0.5cm}
+\begin{longtable}{|>{\raggedright\arraybackslash}p{0.40\textwidth}|>{\raggedright\arraybackslash}p{\columnWidth}|>{\raggedright\arraybackslash}p{\columnWidth}|>{\raggedright\arraybackslash}p{\columnWidth}|>{\raggedright\arraybackslash}p{\columnWidth}|}
+\hline
+\rowcolor{purple!30}
+\textbf{Library} & \textbf{Warm-Up Time (ms)} & \textbf{Avg. Time Excl. Warm-Up (ms)} & \textbf{Total Time Incl. Warm-Up (ms)} \\
+\hline
+\endfirsthead
+
+\hline
+\rowcolor{purple!30}
+\textbf{Library} & \textbf{Warm-Up Time (ms)} & \textbf{Avg. Time Excl. Warm-Up (ms)} & \textbf{Total Time Incl. Warm-Up (ms)} \\
+\hline
+\endhead
+
+\textbf{ImageSharp} & 2754 & 480.86 & 50840 \\\hline
+\textbf{OpenCvSharp + SkiaSharp} & 539 & 100.31 & 10570 \\\hline
+\textbf{Magick.NET} & 4333 & 845.46 & 88879 \\\hline
+\textbf{Emgu CV + Structure.Sketching} & 490 & 59.43 & 6433 \\\hline
+
+\caption{Image Conversion Benchmark Results}
+\label{tab:image-conversion-results}
+\end{longtable}
+
+The table above, is the final dataset that been constructed by merging multiple Excel files produced by the framework described in the Implementation chapter. These results shows lightweight libraries such as SkiaSharp and Structure.Sketching outperforming ImageSharp and Magick.NET in terms of image conversion time. The data also reveals that Emgu CV with Structure.Sketching is the most efficient combination for image conversion, with the lowest average time of 490 milliseconds. but on the other hand, ImageSharp and Magick.NET are significantly slower, with average times of 2,754 and 4,333 milliseconds, respectively.
+
+
+\includegraphics[width=5in]{media/log_1.png}
+\captionof{figure}{Performance Comparison - Image Conversion (log scale)}
+\label{fig:image-conversion}
+\vspace{0.5cm}
+
+
+To visually encapsulate these findings, The graph illustrate the conversion times across the tested libraries, which clearly shows that the conversion times for OpenCvSharp+SkiaSharp and Emgu CV+Structure.Sketching are clustered at the lower end of the spectrum, while ImageSharp’s results are significantly higher. This visual evidence reinforces the numerical data and provides an immediate, intuitive understanding of the performance differences. And the log scale was used to better represent the data, as the differences between total time which is the sum of warm-up and average time, are significant. This three color graph can aid in comparing the performance of the libraries in different scenarios, such as real-time image processing or batch conversion tasks in one glance.
+

sections/Chapter-4-sections/Memory_benchmark_results.tex

+\section{Memory Benchmarking Results}
+
+In parallel with the time benchmarks, memory consumption was a critical parameter in our evaluation. For the image conversion tasks, SkiaSharp, as part of the OpenCvSharp+SkiaSharp configuration, exhibited the lowest memory allocation, with values approximating 58 KB. ImageSharp, in comparison, required about 5.67 MB, which is substantially higher. In the context of pixel iteration, the memory profiles were similarly divergent. ImageSharp was extremely efficient in this regard, consuming roughly 20 KB on average, whereas Emgu CV + Structure.Sketching, despite its fast processing times, utilized around 170 MB of memory.
+
+\setlength{\columnWidth}{0.22\textwidth}
+\begin{longtable}{|>{\raggedright\arraybackslash}p{0.20\textwidth}|>{\raggedright\arraybackslash}p{\columnWidth}|>{\raggedright\arraybackslash}p{\columnWidth}|>{\raggedright\arraybackslash}p{\columnWidth}|>{\raggedright\arraybackslash}p{\columnWidth}|}
+
+\hline
+\rowcolor{purple!30}
+\textbf{Library} & \textbf{Allocated Memory} & \textbf{Gen0/Gen1/Gen2 Collections} \\
+\hline
+\endfirsthead
+
+\hline
+\rowcolor{purple!30}
+\textbf{Library} & \textbf{Allocated Memory} & \textbf{Gen0/Gen1/Gen2 Collections} \\
+\hline
+\endhead
+
+\textbf{EmguCV} & 0.00068 MB (712 bytes) & - / - / - \\
+\hline
+\textbf{ImageSharp} & 5.67 MB (5,805.41 KB) & 1,000 / 1,000 / 1,000 \\
+\hline
+\textbf{SkiaSharp} & 0.05612 MB (58,864 bytes) & - / - / - \\
+\hline
+
+\caption{Memory Benchmarking Results for Image Conversion}
+\label{tab:memory-results-image-conversion}
+\end{longtable}
+
+The table above summarizes the memory benchmarking results for image conversion. It is evident that ImageSharp has the highest memory allocation, with approximately 5.67 MB, while SkiaSharp has the lowest, with only 58 KB. Emgu CV falls in between, with a memory allocation of 0.00068 MB. These figures provide a clear indication of the memory efficiency of each library for image conversion tasks.\\
+
+The large memory footprint of Emgu CV during pixel iteration is a noteworthy trade-off. While its performance in terms of speed is excellent, the high memory consumption must be considered when deploying the solution in memory-constrained environments. The benchmarking data collected here is critical because it provides a balanced view—speed alone does not define an optimal library, but rather the ratio of processing time to memory usage does. For a clear summary of these findings, the below table provides a concise overview of the memory metrics for each library configuration.
+
+
+\begin{longtable}{|>{\raggedright\arraybackslash}p{0.20\textwidth}|>{\raggedright\arraybackslash}p{\columnWidth}|>{\raggedright\arraybackslash}p{\columnWidth}|>{\raggedright\arraybackslash}p{\columnWidth}|>{\raggedright\arraybackslash}p{\columnWidth}|}
+
+\hline
+\rowcolor{purple!30}
+\textbf{Library} & \textbf{Allocated Memory} & \textbf{Gen0/Gen1/Gen2 Collections} \\
+\hline
+\endfirsthead
+
+\hline
+\rowcolor{purple!30}
+\textbf{Library} & \textbf{Allocated Memory} & \textbf{Gen0/Gen1/Gen2 Collections} \\
+\hline
+\endhead
+
+\textbf{EmguCV} & 170.00 MB (177,976,185 bytes) & 33,142 / 1,571 / 1,571 \\
+\hline
+\textbf{ImageSharp} & 0.01932 MB (20.26 KB) & - / - / - \\
+\hline
+\textbf{SkiaSharp} & 384.00 MB (403,300,552 bytes) & 85 / - / - \\
+\hline
+
+\caption{Memory Benchmarking Results for Pixel Iteration}
+\label{tab:memory-results-pixel-iteration}
+\end{longtable}
+
+
+The table indicates that while Emgu CV+Structure.Sketching is extremely fast for pixel iteration, its memory consumption is substantially higher compared to ImageSharp and the OpenCvSharp+SkiaSharp combination. Emgu CV has the highest memory allocation, with approximately 170 MB, while ImageSharp has the lowest, with only 20 KB. SkiaSharp falls in between, with a memory allocation of 384 MB. These figures provide a clear indication of the memory efficiency of each library for pixel iteration tasks. Such data are instrumental in shaping our final recommendation.

sections/Chapter-4-sections/Pixel_iteration_benchmark_results.tex

+\section{Pixel Iteration Benchmark Results}
+
+On the other hand, the pixel iteration benchmark aimed to assess the libraries’ abilities to process each pixel of an image. For ImageSharp, the warm-up phase for pixel iteration took an average of 755 milliseconds, with the main iteration averaging 117.06 milliseconds per cycle and a cumulative total of 12,461 milliseconds over 100 iterations. The performance landscape changed when we observed the results for Magick.NET. This configuration recorded a warm-up time of approximately 12,149 milliseconds, and the main iterations averaged 2,054.18 milliseconds, resulting in an astronomical total of 217,567 milliseconds. 
+
+As discussed earlier, OpenCvSharp and Emgu CV were the chosen libraries in their combinations with SkiaSharp and Structure.Sketching, respectively for the pixel iteration task. The results of these tests provide insights into the performance of these libraries in scenarios where pixel-level operations are required, such as image processing algorithms or computer vision applications. The performance landscape changed when we observed the results for OpenCvSharp. This configuration recorded a warm-up time of approximately 813 milliseconds, and the main iterations averaged 159.44 milliseconds, resulting in a total of 16,757 milliseconds. In contrast, Emgu CV delivered impressive results with a warm-up time of 1,118 milliseconds and an average main iteration time of 118.87 milliseconds, culminating in a total of 13,005 milliseconds.
+
+
+\begin{longtable}{|>{\raggedright\arraybackslash}p{0.40\textwidth}|>{\raggedright\arraybackslash}p{\columnWidth}|>{\raggedright\arraybackslash}p{\columnWidth}|>{\raggedright\arraybackslash}p{\columnWidth}|>{\raggedright\arraybackslash}p{\columnWidth}|}
+\hline
+\rowcolor{purple!30}
+\textbf{Library} & \textbf{Warm-Up Time (ms)} & \textbf{Avg. Time Excl. Warm-Up (ms)} & \textbf{Total Time Incl. Warm-Up (ms)} \\
+\hline
+\endfirsthead
+
+\hline
+\rowcolor{purple!30}
+\textbf{Library} & \textbf{Warm-Up Time (ms)} & \textbf{Avg. Time Excl. Warm-Up (ms)} & \textbf{Total Time Incl. Warm-Up (ms)} \\
+\hline
+\endhead
+
+\textbf{ImageSharp} & 755 & 117.06 & 12461 \\\hline
+\textbf{OpenCvSharp + SkiaSharp} & 813 & 159.44 & 16757 \\\hline
+\textbf{Magick.NET} & 12149 & 2054.18 & 217567 \\\hline
+\textbf{Emgu CV + Structure.Sketching} & 1118 & 118.87 & 13005 \\\hline
+
+\caption{Pixel Iteration Benchmark Results}
+\end{longtable}
+
+The table above summarizes the pixel iteration benchmark results, highlighting the warm-up and average times for each library combination. The data clearly show that Emgu CV is the most efficient library for pixel iteration, with the lowest average time of 118.87 milliseconds. ImageSharp and OpenCvSharp follow closely behind, with average times of 117.06 and 159.44 milliseconds, respectively. In contrast, Magick.NET is significantly slower, with an average time of 2,054.18 milliseconds.
+
+Graphical depictions further highlight these performance differences. 
+
+\includegraphics[width=5in]{media/log_2.png}
+\captionof{figure}{Pixel Iteration Benchmark Results}
+\label{fig:pixel-iteration}
+
+The disparity between these figures is telling. While Magick.NET excels in some aspects of image conversion, it appears less suited for tasks involving pixel-by-pixel iteration, given the significantly higher processing times. On the other hand, Emgu CV and ImageSharp produce comparable main iteration times; however, when considering the overall picture, the lower cumulative times of Emgu CV make it a more appealing choice for pixel-level operations. 
+
+The visual comparisons elucidate that while ImageSharp and Emgu CV+Structure.Sketching are closely matched in main iteration performance, the excessive warm-up and overall times associated with Magick.NET underscore its limitations for this specific task.

sections/Chapter-4-sections/Summary.tex

+\section{Summary}
+
+The benchmarking results provide a comprehensive overview of the performance and efficiency of the image processing libraries tested. The data clearly show that Emgu CV + Structure.Sketching is the most efficient combination for image conversion, with the lowest average time of 490 milliseconds. In contrast, ImageSharp and Magick.NET are significantly slower, with average times of 2,754 and 4,333 milliseconds, respectively. For pixel iteration, Emgu CV+Structure.Sketching is again the most efficient, with the lowest average time of 118.87 milliseconds. ImageSharp and OpenCvSharp+SkiaSharp follow closely behind, with average times of 117.06 and 159.44 milliseconds, respectively. In contrast, Magick.NET is significantly slower, with an average time of 2,054.18 milliseconds. The memory benchmarking results further highlight the efficiency of ImageSharp and SkiaSharp in terms of memory consumption, with Emgu CV exhibiting higher memory usage. These findings provide valuable insights into the performance characteristics of each library and will inform our final recommendations for image processing tasks.
\ No newline at end of file