Benchmarks
==========

LOOM-Eval supports 23 comprehensive benchmarks across 7 cognitive dimensions.

.. note::
   
   Configuration files and additional resources for each benchmark can be found in the 
   `GitHub repository <https://github.com/loomeval/loom-eval/tree/main/benchmarks>`_.

Overview
--------

.. list-table::
   :header-rows: 1
   :widths: 25 75
   
   * - Dimension
     - Benchmarks
   * - **Faithfulness**
     - :doc:`L_CiteEval <Faithfulness/L_CiteEval>`, :doc:`LongCite <Faithfulness/LongCite>`
   * - **General**
     - :doc:`BAMBOO <General/BAMBOO>`, :doc:`LEval <General/LEval>`, :doc:`LongBench <General/LongBench>`, :doc:`LooGLE <General/LooGLE>`, :doc:`RULER <General/RULER>`
   * - **Generation**
     - :doc:`LongWriter <Generation/LongWriter>`
   * - **Reasoning**
     - :doc:`Ada_LEval <Reasoning/Ada_LEval>`, :doc:`Counting_Stars <Reasoning/Counting_Stars>`, :doc:`LVEval <Reasoning/LVEval>`, :doc:`LongBench_v2 <Reasoning/LongBench_v2>`, :doc:`LongIns <Reasoning/LongIns>`, :doc:`babilong <Reasoning/babilong>`
   * - **Retrieval**
     - :doc:`InfiniteBench <Retrieve/InfiniteBench>`, :doc:`NIAH <Retrieve/NIAH>`, :doc:`NThread <Retrieve/NThread>`, :doc:`NoLiMa <Retrieve/NoLiMa>`
   * - **Reward**
     - :doc:`LongRewardBench <Reward/LongRewardBench>`
   * - **Specialization**
     - :doc:`CLongEval <Specialization/CLongEval>`, :doc:`LIBRA <Specialization/LIBRA>`, :doc:`LongHealth <Specialization/LongHealth>`, :doc:`LongSafety <Specialization/LongSafety>`

Running a Benchmark
-------------------

Basic Command
~~~~~~~~~~~~~

.. code-block:: bash

   loom-eval.run \
       --model_path <model_name> \
       --cfg_path ./benchmarks/<ability>/<benchmark>/configs/<benchmark>.yaml \
       --device 0 1 \
       --gp_num 2 \
       --eval \
       --save_tag <your_tag>

Example: L_CiteEval
~~~~~~~~~~~~~~~~~~~

.. code-block:: bash

   loom-eval.run \
       --model_path Meta-Llama/Meta-Llama-3.1-8B-Instruct \
       --cfg_path ./benchmarks/Faithfulness/L_CiteEval/configs/L_CiteEval.yaml \
       --device 0 1 \
       --gp_num 2 \
       --eval \
       --save_tag Llama3_1_8B_INSTRUCT_L_CiteEval

LOOMBench (Lightweight Suite)
-----------------------------

LOOMBench is a streamlined evaluation suite that allows comprehensive evaluation within 6 hours.

**Single Benchmark Evaluation**

.. code-block:: bash

   loom-eval.run \
       --model_path Meta-Llama/Meta-Llama-3.1-8B-Instruct \
       --cfg_path ./benchmarks/Faithfulness/L_CiteEval/configs/L_CiteEval.yaml \
       --device 0 1 2 3 \
       --eval \
       --lightweight

**Full LOOMBench Evaluation**

1. Edit ``Tools/LOOMBench.sh``
2. Add your model configuration
3. Run: ``bash Tools/LOOMBench.sh``

All Benchmarks
--------------


Faithfulness Benchmarks
~~~~~~~~~~~~~~~~~~~~~~~

.. toctree::
   :maxdepth: 1

   L_CiteEval <Faithfulness/L_CiteEval>
   LongCite <Faithfulness/LongCite>


General Benchmarks
~~~~~~~~~~~~~~~~~~

.. toctree::
   :maxdepth: 1

   BAMBOO <General/BAMBOO>
   LEval <General/LEval>
   LongBench <General/LongBench>
   LooGLE <General/LooGLE>
   RULER <General/RULER>


Generation Benchmarks
~~~~~~~~~~~~~~~~~~~~~

.. toctree::
   :maxdepth: 1

   LongWriter <Generation/LongWriter>


Reasoning Benchmarks
~~~~~~~~~~~~~~~~~~~~

.. toctree::
   :maxdepth: 1

   Ada_LEval <Reasoning/Ada_LEval>
   Counting_Stars <Reasoning/Counting_Stars>
   LVEval <Reasoning/LVEval>
   LongBench_v2 <Reasoning/LongBench_v2>
   LongIns <Reasoning/LongIns>
   babilong <Reasoning/babilong>


Retrieval Benchmarks
~~~~~~~~~~~~~~~~~~~~

.. toctree::
   :maxdepth: 1

   InfiniteBench <Retrieve/InfiniteBench>
   NIAH <Retrieve/NIAH>
   NThread <Retrieve/NThread>
   NoLiMa <Retrieve/NoLiMa>


Reward Benchmarks
~~~~~~~~~~~~~~~~~

.. toctree::
   :maxdepth: 1

   LongRewardBench <Reward/LongRewardBench>


Specialization Benchmarks
~~~~~~~~~~~~~~~~~~~~~~~~~

.. toctree::
   :maxdepth: 1

   CLongEval <Specialization/CLongEval>
   LIBRA <Specialization/LIBRA>
   LongHealth <Specialization/LongHealth>
   LongSafety <Specialization/LongSafety>