how are you programming your custom esp32 boards? using platform io? can you first do a complete erase of the esp32 using esptool?
Some ESP32s are shipped with malformatted flash which makes the ESP32 system code be unable to save BLE shared keys (generated during pairing).