Execute commands without CommandBus and commandHandler in Clean Architecture

When using the Axon Framework the usual way of executing a Command is to use the CommandGateway/CommandBus. Is there a good way to avoid that and directly execute the command on my aggregate?

Background:
I am experimenting with Clean Architecture and being very independent of the technology used (no Axon annotations at all inside of the useCases or domain layer). The part I am struggling with is the transition from my RestController to my in port / application-usecase. In your blog post here Bounded contexts with Axon a REST input is always converted to a command and passed to the commandGateway (which is Axon infrastructure and also an additional round trip to the Axon Server). My goal would be to bypass the CommandBus and CommandHandler discovery completely. I like to directly call the Repository, load the aggregate and generate/publish some events.

I managed to do this by manually creating a UnitOfWork, but that feels very hacky and not future proof since the UnitOfWork is more internal.

What disadvantages should I expect when bypassing the CommandBus (when still using Axonserver)? Of course I don’t have location transparency, no Axon based load balancing and I have to take care of aggregate locking

Here is my code for a simple dummy banking application:

@Application
class BankAccountMoneyTransactionsUseCase(
  private val bankAccountAggregateRepository: BankAccountAggregateRepository,
): DepositMoneyInPort, WithdrawMoneyInPort
{
  override fun depositMoney(bankAccountId: BankAccountId, amount: TransferAmount) {
    bankAccountAggregateRepository.decide(bankAccountId) {it.deposit(amount) }
  }
  override fun withdrawMoney(bankAccountId: BankAccountId, amount: TransferAmount) {
    bankAccountAggregateRepository.decide(bankAccountId) {it.withdraw(amount) }
  }
}

@SecondaryAdapter
class BankAccountAxonAggregateRepository(
  val aggregateRepository: Repository<BankAccountAggregate>,
  val transactionManager: TransactionManager,
) : BankAccountAggregateRepository {

  override fun decide(bankAccountId: BankAccountId, decider: (BankAccount) -> List<BankAccountEvent>) {
    val unitOfWork = getOrCreateUnitOfWork(bankAccountId)
    unitOfWork.execute {
      aggregateRepository.load(bankAccountId.toString()).execute {
        it.execute(decider)
      }
    }
  }

  private fun getOrCreateUnitOfWork(bankAccountId: BankAccountId): UnitOfWork<*> {
      private fun getOrCreateUnitOfWork(bankAccountId: BankAccountId): UnitOfWork<*> {
    if (CurrentUnitOfWork.isStarted() && CurrentUnitOfWork.get().message is CommandMessage) {
      return CurrentUnitOfWork.get()
    } else {
      val commandMessage = GenericCommandMessage(bankAccountId) // probably bad
      val unitOfWork = DefaultUnitOfWork.startAndGet(commandMessage)
      unitOfWork.attachTransaction(transactionManager)
      return unitOfWork
    }
  }

}

@Aggregate
class BankAccountAggregate() {

  @AggregateIdentifier
  lateinit var bankAccountId: BankAccountId
  lateinit var model: BankAccount

  constructor(decider: () -> List<BankAccountEvent>) : this() {
    decider.invoke().forEach { AggregateLifecycle.apply(it) }
  }

  fun execute(decider: (BankAccount) -> List<BankAccountEvent>) {
    decider.invoke(model).forEach { AggregateLifecycle.apply(it) }
  }

  @EventSourcingHandler
  fun on(evt: BankAccountEvent) {
    when (evt) {
      is BankAccountCreated -> {
        bankAccountId = evt.bankAccountId; model = BankAccount(evt)
      }
      is MoneyDeposited -> model.evolve(evt)
      is MoneyWithdrawn -> model.evolve(evt)
    }
  }
}