Chain-of-Thought Provably Enables Learning the (Otherwise) Unlearnable
Authors: Chenxiao Yang, Zhiyuan Li, David Wipf
ICLR 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Empirically, we demonstrate our proposed Co T construction significantly enhances the reasoning capabilities of real-world LLMs in solving challenging arithmetic reasoning tasks, including learning polynomials and Boolean formulas. ... 5 EXPERIMENTS |
| Researcher Affiliation | Collaboration | Chenxiao Yang , Zhiyuan Li , David Wipf Toyota Technological Institute at Chicago, Amazon Web Services EMAIL, EMAIL |
| Pseudocode | Yes | Algorithm 1: Step-by-Step Learning with Co T (ACo T) |
| Open Source Code | Yes | Codes are available at https://github.com/chr26195/Co T-ICL. |
| Open Datasets | No | For empirical verification, we consider new arithmetic reasoning tasks and test if the decomposition schemes from our analysis are practically effective (Section 5). Specifically, we propose to construct complex reasoning tasks with varying overall hardness and hardness of subtasks. ... For each k in parities and w in DNFs, we similarly i.i.d. sample 100 target functions. |
| Dataset Splits | Yes | For each target function, the LLMs are provided with 10 demonstrations and asked to infer the computation process and apply it to derive the output for an unseen input. ... For each function, we provide LLMs with 100 in-context examples, ask them to find patterns in these examples, and return the output for each query. |
| Hardware Specification | No | The paper mentions evaluating |
| Software Dependencies | No | The paper mentions evaluating |
| Experiment Setup | Yes | For each target function, the LLMs are provided with 10 demonstrations and asked to infer the computation process and apply it to derive the output for an unseen input. ... We test 100 times to compute the success rate. ... For the input values, we uniformly select two unique integers from the set {2, 3, . . . , 10}. ... For each k in parities and w in DNFs, we similarly i.i.d. sample 100 target functions. ... we implement it by randomly masking H 1 consecutive intermediate steps. |