Reverse-Mode AD of Object-Oriented CFD Code w Enzyme

Hi,

We are currently working on enabling Enzyme for one of our computational fluid dynamics codes, which is written as a C++ 17 standard-compliant object-oriented code. Following past experiences with Enzyme, such as LULESH-CUDA (non-MPI) we have adapted the layout of the code to be able to apply AD to the time-stepping function, in our case called Advance. The time-stepping loop does then look like this:

while( continue_simulation ) {
    MPI_Barrier( MPI_COMM_WORLD );
    SetCopiedTree(); // Copy of the Octree-structure to store the gradient information into
    
    // Advance( tree_ );  // normal time-stepping
    __enzyme_autodiff( Advance, tree_, copied_tree_ );
    
    ResetAllJumpBuffers();
    MPI_Barrier( MPI_COMM_WORLD );
    ...
}

When we now seek to build the code with Enzyme we get the following error:

Error: reference to non-static member function must be called

Said non-static member function is Advance, which hence causes the error.

What would the Enzyme-recommended best way be to solve this error?

1 Like

After experimenting in compiler explorer we have managed to move beyond the previous error so that we are now facing the following error message:

In file included from /home/lpaehler/Work/Repositories/ALPACA-Enzyme/src/main.cpp:80:
/home/lpaehler/Work/Repositories/ALPACA-Enzyme/src/simulation_runner.h:184:10: warning: __enzyme_autodiff missing
 argument shadow at index 4, need shadow of type %class.ModularAlgorithmAssembler* to shadow primal argument   %mr_based_algorithm = alloca %class.ModularAlgorithmAssembler, align 8 at call   call void @_ZN10Simulation17__enzyme_autodiffEPvR4TreeS2_R25ModularAlgorithmAssembler(i8* bitcast (void (%class.Tree*, %class.ModularAlgorithmAssembler*)* @_ZN10Simulation11WrapAdvanceER4TreeR25ModularAlgorithmAssembler to i8*), %class.Tree* nonnull align 8 dereferenceable(40) %tree, %class.Tree* nonnull align 8 dereferenceable(40) %copied_tree, %class.ModularAlgorithmAssembler* nonnull align 8 dereferenceable(649) %mr_based_algorithm), !dbg !13831 [-Wpass-failed=enzyme]
         __enzyme_autodiff( (void*)WrapAdvance,                                             
         ^
error: src/simulation_runner.h:109:0: EnzymeFailure when replacing __enzyme_autodiff calls in _ZN10Simulation3RunERK11InputReader

This problem originates from our inability to instantiate __enzyme_autodiff correctly to then utilize the enzyme_dup and enzyme_const syntax correctly. What we are doing right now is

...
void __enzyme_autodiff( void*, Tree&, Tree&, ModularAlgorithmAssembler& );
int enzyme dup;
int enzyme_out;
int enzyme_const;

...

while(mr_based_algorithm.GetContinueSimulation() ) {
    ...
    __enzyme_autodiff( (void*)WrapAdvance,
                       tree, copied_tree,
                       mr_based_algorithm );
}

What the question has now evolved into is in which Enzyme would need to be instantiated correctly to allow for a syntax in the while-loop which can utilize enzyme_dup, and enzyme_const? I.e.

while(mr_based_algorithm.GetContinueSimulation() ) {
    ...
    __enzyme_autodiff( (void*)WrapAdvance,
                      enzyme_dup, tree, copied_tree,
                      enzyme_const, mr_based_algorithm );
}

Can you try the variadic variant of __enzyme_autodiff instead? I think you might need to update the declaration.

E.g. void __enzyme_autodiff(...); instead of void __enzyme_autodiff( void*, Tree&, Tree&, ModularAlgorithmAssembler& );

1 Like

I think your use of activity arguments in the latest code snippet should be correct (and let us know if that resolves the issue or not).

If activity is not specified for an argument, the magic function __enzyme_autodiff will try to automatically deduce the activity of a variable from its LLVM type (e.g. a double is probably active, an integer probably is not). For simple/common cases this is a reasonable default. It looks here like it may have deduced the ModularAlgorithmAssembler& as an active duplicated input (perhaps as it contains a float/double somewhere?) and not specifying its shadow is leading to an error. Using the enzyme_const as you have in the subsequent snippet should remedy the issue.

1 Like

Following @wsmoses advice we were able to fix this specific error. The specific code we ended up with here is:

extern "C" {
   int enzyme_dup;
   int enzyme_const;
}

namespace Simulation {
    
    void __enzyme_autodiff( void*, int, Tree&, Tree&, int, ModularAlgorithmAssembler&);

    ...

    void Run( InputReader const& input_reader ) {
    
    ...
      while( mr_based_algorithm.GetContinueSimulation() ) {
         mr_based_algorithm.ComputeLoop1();
         //WrapAdvance( tree, mr_based_algorithm );
         __enzyme_autodiff( (void*)WrapAdvance,
                            enzyme_dup, tree, copied_tree,
                            enzyme_const, mr_based_algorithm  );
         mr_based_algorithm.ComputeLoop2();
      }
    }
}