Bug in LLVM mode when executing LCA on multiple trials

[jongkeesbj]

The following two calls to run, once in Python mode and once in LLVM mode, do not provide the same results on the second trial.

import psyneulink as pnl

my_lca = pnl.LCAMechanism(
    termination_measure=pnl.TimeScale.TRIAL,
    termination_threshold=2,
    execute_until_finished=True
)

comp = pnl.Composition()
comp.add_node(my_lca)

comp.run([[1], [1]], execution_mode=pnl.ExecutionMode.Python)
print(comp.results)

comp.reset(clear_results=True)

comp.run([[1], [1]], execution_mode=pnl.ExecutionMode.LLVM)
print(comp.results)

[jongkeesbj]

To start excluding possible reasons for the discrepancy between python and llvm mode, I first set up my test to have completely separate mechanisms and compositions run in each mode. This is to make sure that the bug is not related to resetting the composition between runs. However, the discrepancy remains even when running it this way.

import psyneulink as pnl

lca_python = pnl.LCAMechanism(
    termination_measure=pnl.TimeScale.TRIAL,
    termination_threshold=2,
    execute_until_finished=True
)

comp_python = pnl.Composition()
comp_python.add_node(lca_python)

print("Python: ")
comp_python.run([[1], [1]], execution_mode=pnl.ExecutionMode.Python)
print(comp_python.results)


lca_llvm = pnl.LCAMechanism(
    termination_measure=pnl.TimeScale.TRIAL,
    termination_threshold=2,
    execute_until_finished=True
)

comp_llvm = pnl.Composition()
comp_llvm.add_node(lca_llvm)

print("LLVM: ")
comp_llvm.run([[1], [1]], execution_mode=pnl.ExecutionMode.LLVM)
print(comp_llvm.results)

Next, I checked whether the issue was related to the LCA mechanism being the origin node of the composition, so I simply added a processing mechanism as origin node that projects to the LCA. The discrepancy remained.

Next, I tried changing the termination_threshold on the LCAs from 2 to 1. Now the results align again between python mode and llvm mode. This raises the possibility that the discrepancy is somehow related to termination_threshold.

import psyneulink as pnl

lca_python = pnl.LCAMechanism(
    termination_measure=pnl.TimeScale.TRIAL,
    termination_threshold=1,
    execute_until_finished=True
)

comp_python = pnl.Composition()
comp_python.add_node(lca_python)

print("Python: ")
comp_python.run([[1], [1]], execution_mode=pnl.ExecutionMode.Python)
print(comp_python.results)


lca_llvm = pnl.LCAMechanism(
    termination_measure=pnl.TimeScale.TRIAL,
    termination_threshold=1,
    execute_until_finished=True
)

comp_llvm = pnl.Composition()
comp_llvm.add_node(lca_llvm)

print("LLVM: ")
comp_llvm.run([[1], [1]], execution_mode=pnl.ExecutionMode.LLVM)
print(comp_llvm.results)

Turning the termination_threshold back to 2, I checked whether the issue is related to execute_until_finished. So I set it to false, and added a mechanism to the composition that receives input from the LCA and is scheduled to execute only when the lca is 'finished'. However, now the composition running in LLVM mode does not seem to finish. Whether that is a separate issue, or part of the problem, I do not know.

import psyneulink as pnl

lca_python = pnl.LCAMechanism(
    termination_measure=pnl.TimeScale.TRIAL,
    termination_threshold=2,
    execute_until_finished=False
)

gate_python = pnl.ProcessingMechanism()

comp_python = pnl.Composition()
comp_python.add_linear_processing_pathway([lca_python, gate_python])

comp_python.scheduler.add_condition(
    gate_python, pnl.WhenFinished(lca_python)
)

print("Python: ")
comp_python.run([[1], [1]], execution_mode=pnl.ExecutionMode.Python)
print(comp_python.results)


lca_llvm = pnl.LCAMechanism(
    termination_measure=pnl.TimeScale.TRIAL,
    termination_threshold=2,
    execute_until_finished=False
)

gate_llvm = pnl.ProcessingMechanism()

comp_llvm = pnl.Composition()
comp_llvm.add_linear_processing_pathway([lca_llvm, gate_llvm])

comp_llvm.scheduler.add_condition(
    gate_llvm, pnl.WhenFinished(lca_llvm)
)

print("LLVM: ")
comp_llvm.run([[1], [1]], execution_mode=pnl.ExecutionMode.LLVM)
print(comp_llvm.results)

[jongkeesbj]

Just to add to the last bit of code posted in my previous comment: removing the scheduler condition for comp_llvm does allow the composition to finish, but now (as expected) its results are based on only a single execution of the lca per trial. This indicates to me that there might be something wrong with the LCA's internal 'finished' flag setting to True in LLVM mode when termination_threshold is > 1.

import psyneulink as pnl

lca_python = pnl.LCAMechanism(
    termination_measure=pnl.TimeScale.TRIAL,
    termination_threshold=2,
    execute_until_finished=False
)

gate_python = pnl.ProcessingMechanism()

comp_python = pnl.Composition()
comp_python.add_linear_processing_pathway([lca_python, gate_python])

comp_python.scheduler.add_condition(
    gate_python, pnl.WhenFinished(lca_python)
)

print("Python: ")
comp_python.run([[1], [1]], execution_mode=pnl.ExecutionMode.Python)
print(comp_python.results)


lca_llvm = pnl.LCAMechanism(
    termination_measure=pnl.TimeScale.TRIAL,
    termination_threshold=2,
    execute_until_finished=False
)

gate_llvm = pnl.ProcessingMechanism()

comp_llvm = pnl.Composition()
comp_llvm.add_linear_processing_pathway([lca_llvm, gate_llvm])

# comp_llvm.scheduler.add_condition(
#     gate_llvm, pnl.WhenFinished(lca_llvm)
# )

print("LLVM: ")
comp_llvm.run([[1], [1]], execution_mode=pnl.ExecutionMode.LLVM)
print(comp_llvm.results)

[jvesely]

pnl.ExecutionMode.LLVM is not supported in combination with scheduling rules, and really should not be used outside of tests or debugging.
The only compiled mode to have support for scheduling is pnl.ExecutionMode.LLVMRun.
Running the original script, modified to use LLVMRun:

$ cat 3164.py 
import psyneulink as pnl

lca_python = pnl.LCAMechanism(
    termination_measure=pnl.TimeScale.TRIAL,
    termination_threshold=2,
    execute_until_finished=False
)

gate_python = pnl.ProcessingMechanism()

comp_python = pnl.Composition()
comp_python.add_linear_processing_pathway([lca_python, gate_python])

comp_python.scheduler.add_condition(
    gate_python, pnl.WhenFinished(lca_python)
)

print("Python: ")
comp_python.run([[1], [1]], execution_mode=pnl.ExecutionMode.Python)
print(comp_python.results)


lca_llvm = pnl.LCAMechanism(
    termination_measure=pnl.TimeScale.TRIAL,
    termination_threshold=2,
    execute_until_finished=False
)

gate_llvm = pnl.ProcessingMechanism()

comp_llvm = pnl.Composition()
comp_llvm.add_linear_processing_pathway([lca_llvm, gate_llvm])

comp_llvm.scheduler.add_condition(
    gate_llvm, pnl.WhenFinished(lca_llvm)
)

print("LLVM: ")
comp_llvm.run([[1], [1]], execution_mode=pnl.ExecutionMode.LLVMRun)
print(comp_llvm.results)

produces the same results for both Python and compiled run:

$ python 3164.py 
Python: 
[[[0.54859611]]

 [[0.59169757]]]
LLVM: 
[[[0.54859611]]

 [[0.59169757]]]

The LLVMExec and LLVM modes should be renamed to avoid user confusion.

closing. feel free to reopen if there still is a problem when using LLVMRun

[jongkeesbj]

Note: edited for typo in code

Thank you for pointing out the difference between pnl.ExectionMode.LLVM and pnl.ExecutionMode.LLVMRun. While the basic example indeed produces the same results for both Python and compiled run, the example below still does not.

In this example, the composition more closely resembles what I usually work with, where on each trial an LCA is first scheduled to execute a certain number of times and then the LCA and a DDM both execute on each pass until the DDM reaches threshold. A 'gate' mechanism is included in the composition, which is scheduled to only execute once the DDM finishes and thus prevents the trial from terminating until the DDM is finished.

import psyneulink as pnl

# Ensure same RNG for both runs
from psyneulink.core.globals.utilities import set_global_seed
set_global_seed(0)

# Set up composition to run in Python mode
lca_python = pnl.LCAMechanism(
    termination_measure=pnl.TimeScale.TRIAL,
    termination_threshold=2,
    execute_until_finished=False
)

ddm_python = pnl.DDM(
    function=pnl.DriftDiffusionIntegrator(
        threshold=1,
        noise=0.01,
        time_step_size=0.01
    ),
    reset_stateful_function_when=pnl.AtTrialStart(),
    output_ports=[pnl.RESPONSE_TIME],
    execute_until_finished=False
)

gate_python = pnl.ProcessingMechanism()

comp_python = pnl.Composition()
comp_python.add_linear_processing_pathway([lca_python, ddm_python, gate_python])

comp_python.scheduler.add_condition(
    ddm_python, pnl.WhenFinished(lca_python)
)

comp_python.scheduler.add_condition(
    gate_python, pnl.WhenFinished(ddm_python)
)

print("Python: ")
comp_python.run([[1], [1]], execution_mode=pnl.ExecutionMode.Python)
print(comp_python.results)


# Set up composition to run in LLVMRun mode
lca_llvm = pnl.LCAMechanism(
    termination_measure=pnl.TimeScale.TRIAL,
    termination_threshold=2,
    execute_until_finished=False
)

ddm_llvm = pnl.DDM(
    function=pnl.DriftDiffusionIntegrator(
        threshold=1,
        noise=0.01,
        time_step_size=0.01
    ),
    reset_stateful_function_when=pnl.AtTrialStart(),
    output_ports=[pnl.RESPONSE_TIME],
    execute_until_finished=False
)

gate_llvm = pnl.ProcessingMechanism()

comp_llvm = pnl.Composition()
comp_llvm.add_linear_processing_pathway([lca_llvm, ddm_llvm, gate_llvm])

comp_llvm.scheduler.add_condition(
    ddm_llvm, pnl.WhenFinished(lca_llvm)
)

comp_llvm.scheduler.add_condition(
    gate_llvm, pnl.WhenFinished(ddm_llvm)
)

print("LLVM: ")
comp_llvm.run([[1], [1]], execution_mode=pnl.ExecutionMode.LLVMRun)
print(comp_llvm.results)

Running this script returns

Python: 
[[[1.19]]

 [[1.15]]]
LLVM: 
[[[1.2 ]]

 [[1.14]]]

[jvesely]

Note: edited for typo in code

Thank you for pointing out the difference between pnl.ExectionMode.LLVM and pnl.ExecutionMode.LLVMRun. While the basic example indeed produces the same results for both Python and compiled run, the example below still does not.

In this example, the composition more closely resembles what I usually work with, where on each trial an LCA is first scheduled to execute a certain number of times and then the LCA and a DDM both execute on each pass until the DDM reaches threshold. A 'gate' mechanism is included in the composition, which is scheduled to only execute once the DDM finishes and thus prevents the trial from terminating until the DDM is finished.

import psyneulink as pnl

# Ensure same RNG for both runs
from psyneulink.core.globals.utilities import set_global_seed
set_global_seed(0)

# Set up composition to run in Python mode
lca_python = pnl.LCAMechanism(
    termination_measure=pnl.TimeScale.TRIAL,
    termination_threshold=2,
    execute_until_finished=False
)

ddm_python = pnl.DDM(
    function=pnl.DriftDiffusionIntegrator(
        threshold=1,
        noise=0.01,
        time_step_size=0.01
    ),
    reset_stateful_function_when=pnl.AtTrialStart(),
    output_ports=[pnl.RESPONSE_TIME],
    execute_until_finished=False
)

gate_python = pnl.ProcessingMechanism()

comp_python = pnl.Composition()
comp_python.add_linear_processing_pathway([lca_python, ddm_python, gate_python])

comp_python.scheduler.add_condition(
    ddm_python, pnl.WhenFinished(lca_python)
)

comp_python.scheduler.add_condition(
    gate_python, pnl.WhenFinished(ddm_python)
)

print("Python: ")
comp_python.run([[1], [1]], execution_mode=pnl.ExecutionMode.Python)
print(comp_python.results)


# Set up composition to run in LLVMRun mode
lca_llvm = pnl.LCAMechanism(
    termination_measure=pnl.TimeScale.TRIAL,
    termination_threshold=2,
    execute_until_finished=False
)

ddm_llvm = pnl.DDM(
    function=pnl.DriftDiffusionIntegrator(
        threshold=1,
        noise=0.01,
        time_step_size=0.01
    ),
    reset_stateful_function_when=pnl.AtTrialStart(),
    output_ports=[pnl.RESPONSE_TIME],
    execute_until_finished=False
)

gate_llvm = pnl.ProcessingMechanism()

comp_llvm = pnl.Composition()
comp_llvm.add_linear_processing_pathway([lca_llvm, ddm_llvm, gate_llvm])

comp_llvm.scheduler.add_condition(
    ddm_llvm, pnl.WhenFinished(lca_llvm)
)

comp_llvm.scheduler.add_condition(
    gate_llvm, pnl.WhenFinished(ddm_llvm)
)

print("LLVM: ")
comp_llvm.run([[1], [1]], execution_mode=pnl.ExecutionMode.LLVMRun)
print(comp_llvm.results)

Running this script returns

Python: 
[[[1.19]]

 [[1.15]]]
LLVM: 
[[[1.2 ]]

 [[1.14]]]

This looks similar to #3142. I'll check this example once I have a fix for #3142.

[jvesely]

collecting the same tracking information as #3142 :

$  python 3164-2.py 
Python: 
RESET INTEGRATOR START: DDM-0 [[0.]
 [0.]] [array([0.])]
RESET INTEGRATOR: DDM-0 [[0.]
 [0.]] [array([0.])]
RESET INTEGRATOR START: DDM-0 [[1.  ]
 [1.19]] [array([1.19])]
RESET INTEGRATOR: DDM-0 [[0.]
 [0.]] [array([0.])]

shows the same issue. calls to reset update mechanism and output port values which is not done in the compiled version.
since this shows value difference even in the first trial, there might be other issues on top of #3142.