Meshtastic

T-Beam 1.1 locking up / dead PSRAM

Hi there,
I am getting my feet wet with meshtastic, gread project! My intended use case is short range communication in areas with bad cell coverage.
I’ve bought two t beams, version 1.1 868 MHz, did a fresh git pull just a couple hours ago, successfully compiled the source code and flashed both t beams.
Everything is working just fine, but one t beam is freezing randomly. Display is not updated, no response to user button. At first I thought it was related to the 18650 battery which came from the recycling bin, some random brown out when the RF module is transmitting. But the glitch is also happening when powered from USB or the 18650 battery from the known working unit.
I was able to capture the event with the serial debug console:

??<\0>?<\0>???<\0>?<\0><\0><\0>?<\0>?????<\0>??<\0><\0><\0><\0><\0>?<\0><\0><\0><\0><\0><\0><\0><\0><\0>?<\0><\0>?<\0><\0><\0><\0>?<\0><\0><\0><\0><\0><\0>?<\0><\0><\0><\0>?<\0><\0><\0>?<\0><\0>?<\0><\0>?<\0>??<\0><\0>?<\0>??<\0><\0>?<\0><\0><\0>??<\0>?<\0>??<\0>?<\0><\0><\0><\0>??<\0><\0><\0><\0><\0>?<\0><\0>???<\0>???????<\0>?<\0><\0>???<\0>?<\0><\0>????<\0><\0>????<\0>??<\0>????<\0>?<\0>????<\0>?<\0><\0>?<\0><\0><\0>???????<\0>??<\0>?<\0><\0><\0><\0>??<\0><\0>?<\0><\0><\0><\0><\0>?<\0>?<\0>??<\0>???<\0><\0>????<\0><\0>??<\0>??<\0>??<\0>??????<\0><\0>?<\0><\0>??<\0>??<\0>??<\0><\0>????<\0><\0>??<\0>??<\0>?<\0>????<\0>?<\0>??<\0><\0>?<\0>?<\0>????<\0>??<\0><\0>??<\0>???<\0>??<\0>??<\0>??<\0>?<\0>?<\0><\0><\0><\0><\0><\0>?<\0><\0><\0>?<\0>??<\0>???<\0><\0>?????<\0>??<\0>?<\0><\0><\0><\0>???<\0>???<\0>???<\0><\0>?<\0>?<\0>?<\0>?????????<\0><\0>?<\0><\0>?<\0>??<\0>?<\0><\0><\0><\0>?<\0><\0><\0><\0><\0>?<\0>?<\0>????<\0><\0>?<\0><\0>?<\0>??<\0>?<\0><\0><\0><\0>?<\0><\0><\0><\0><\0>?<\0><\0><\0>????<\0><\0>?<\0><\0>?<\0>??<\0>?<\0><\0><\0><\0>?<\0><\0><\0><\0><\0>?<\0>?<\0>??<\0>?<\0><\0>???<\0><\0>?<\0><\0>?<\0>??<\0>?<\0><\0><\0><\0>?<\0><\0><\0><\0><\0>?<\0><\0>??<\0><\0>????<\0><\0>?<\0><\0>?<\0>??<\0>?<\0><\0><\0><\0>?<\0><\0><\0><\0><\0>?<\0>???<\0>????<\0><\0>?<\0><\0>?<\0>??<\0>?<\0><\0><\0><\0>?<\0><\0><\0><\0>??<\0><\0>?<\0>???????<\0><\0>???<\0>?<\0>?<\0><\0><\0>??<\0><\0>??<\0>?<\0>?<\0><\0><\0>?<\0>?<\0>??????<\0>?????<\0><\0><\0><\0>??<\0>??<\0>??<\0>?<\0><\0><\0><\0>??<\0><\0>?<\0><\0>??????<\0>?<\0><\0>?<\0>?<\0><\0><\0><\0>??<\0><\0><\0><\0>?<\0><\0>?<\0><\0>?<\0><\0><\0><\0>?<\0><\0><\0><\0><\0>?<\0><\0>????<\0>?<\0>??<\0>?<\0><\0>?<\0>??<\0><\0>?<\0><\0>??????<\0>?<\0><\0>?<\0>?<\0><\0><\0><\0>??<\0><\0><\0><\0>?<\0><\0>?<\0><\0>?<\0><\0><\0><\0>?<\0><\0>?<\0>?<\0>?<\0><\0>????<\0>?<\0>??<\0>?<\0>??<\0><\0><\0><\0><\0><\0>??<\0><\0>?<\0><\0>??????<\0>?<\0><\0>?<\0>?<\0><\0><\0><\0>?<\0>?<\0><\0><\0><\0><\0>??<\0><\0><\0><\0><\0><\0><\0><\0><\0><\0>?<\0><\0>????<\0>?<\0>??<\0>?<\0>?<\0><\0>?<\0><\0>?<\0><\0><\0><\0><\0>??<\0><\0>?<\0><\0>??????<\0>?<\0><\0>?<\0>?<\0><\0><\0><\0>?<\0>?<\0><\0><\0><\0><\0><\0><\0><\0><\0><\0>?<\0><\0><\0><\0><\0><\0>?<\0><\0>????<\0>?<\0>??<\0>?<\0>???<\0>?<\0><\0><\0><\0><\0><\0>??<\0><\0>?<\0>??<\0>?<\0>??<\0>??<\0><\0>??<\0>?<\0><\0><\0><\0><\0>?<\0>?<\0><\0><\0><\0><\0><\0><\0><\0><\0><\0>?<\0><\0>?<\0><\0><\0>??<\0><\0>?<\0>????<\0><\0>???????????<\0>??<\0>?<\0>??<\0><\0><\0><\0><\0>???<\0><\0>???<\0>?<\0>?????<\0><\0>??<\0>?<\0><\0>??<\0>?????<\0>?<\0>?<\0>?<\0>?<\0><\0>?<\0>??<\0>????<\0><\0><\0>??<\0>?<\0><\0>??<\0>??????<\0><\0>?<\0>???<\0>??<\0>??<\0>?<\0>?<\0>?<\0><\0>?<\0><\0><\0><\0>???<\0>??<\0><\0>???<\0>???<\0>?<\0><\0>??<\0>?<\0>???<\0>?<\0><\0>?<\0>????<\0>?<\0><\0>???<\0><\0>?<\0>???<\0><2>H<1>??:??:?? 0 booted, wake cause 0 (boot count 1), reset_reason=reset<\r><\n>
??:??:?? 0 Filesystem files:<\r><\n>
??:??:?? 0   /db.proto<\r><\n>
??:??:?? 0   /prefs/db.proto<\r><\n>
??:??:?? 0   /prefs/radio.proto<\r><\n>
??:??:?? 0   /prefs/channels.proto<\r><\n>
??:??:?? 0 I2C device found at address 0x34<\r><\n>
??:??:?? 0 axp192 PMU found<\r><\n>
??:??:?? 0 I2C device found at address 0x3c<\r><\n>
??:??:?? 0 ssd1306 display found<\r><\n>
??:??:?? 0 done<\r><\n>
??:??:?? 0 Meshtastic hwvendor=4, swver=1.2.45.b9443d8, hwver=unset<\r><\n>
??:??:?? 0 Setting random seed 922812504<\r><\n>
??:??:?? 0 Total heap: 254772<\r><\n>
??:??:?? 0 Free heap: 220880<\r><\n>
??:??:?? 0 Total PSRAM: 4194252<\r><\n>
??:??:?? 0 Free PSRAM: 4194252<\r><\n>
??:??:?? 0 NVS: UsedEntries 101, FreeEntries 529, AllEntries 630<\r><\n>
??:??:?? 0 Setup Preferences in Flash Storage<\r><\n>
??:??:?? 0 Number of Device Reboots: 27<\r><\n>
??:??:?? 0 Initial packet id 1089279905, numPacketId 4294967295<\r><\n>
??:??:?? 0 Setting default channel and radio preferences!<\r><\n>
??:??:?? 0 Expanding short PSK #1<\r><\n>
??:??:?? 0 Wanted region 0, using Unset<\r><\n>
??:??:?? 0 Loading /prefs/db.proto<\r><\n>
??:??:?? 0 Loaded saved preferences version 11<\r><\n>
??:??:?? 0 Loading /prefs/radio.proto<\r><\n>
??:??:?? 0 Loading /prefs/channels.proto<\r><\n>
??:??:?? 0 Number of Device Reboots: 27<\r><\n>
??:??:?? 0 Wanted region 3, using EU865<\r><\n>
??:??:?? 0 region=3, NODENUM=0xf244c058, dbsize=2<\r><\n>
chip id detect 0x3<\n>
Detect CHIP :AXP192<\n>
OUTPUT Register 0x5f<\n>
??:??:?? 0 AXP192 Begin PASS<\r><\n>
??:??:?? 0 DCDC1: ENABLE<\r><\n>
??:??:?? 0 DCDC2: ENABLE<\r><\n>
??:??:?? 0 LDO2: ENABLE<\r><\n>
??:??:?? 0 LDO3: ENABLE<\r><\n>
??:??:?? 0 DCDC3: ENABLE<\r><\n>
??:??:?? 0 Exten: ENABLE<\r><\n>
??:??:?? 0 ----------------------------------------<\r><\n>
??:??:?? 0 DCDC1: ENABLE<\r><\n>
??:??:?? 0 DCDC2: ENABLE<\r><\n>
??:??:?? 0 LDO2: ENABLE<\r><\n>
??:??:?? 0 LDO3: ENABLE<\r><\n>
??:??:?? 0 DCDC3: ENABLE<\r><\n>
??:??:?? 0 Exten: ENABLE<\r><\n>
SRC REG:0xc4<\n>
Charging enable is enable<\n>
Charging target-voltage : 0x2<\n>
 end when the charge current is lower than 10% of the set value<\n>
Charge current : 700.00 mA<\n>
??:??:?? 0 Battery: usbPower=1, isCharging=1, batMv=4132, batPct=100<\r><\n>
??:??:?? 0 Read RTC time as 0 (cur millis 220) quality=0<\r><\n>
??:??:?? 0 WANT GPS=1<\r><\n>
??:??:?? 0 Setting GPS power=1<\r><\n>
??:??:?? 1 Connected to UBLOX GPS successfully<\r><\n>
??:??:?? 2 External Notification Plugin Disabled<\r><\n>
??:??:?? 2 Turning on screen<\r><\n>
??:??:?? 2 Starting meshradio init...<\r><\n>
??:??:?? 2 (bw=31, sf=9, cr=4/8) packet symLen=16 ms, payloadSize=16, time 1363 ms<\r><\n>
??:??:?? 2 Set radio: name=ltfffdd, config=2, ch=3, power=0<\r><\n>
??:??:?? 2 Radio myRegion->freq: 865.200012<\r><\n>
??:??:?? 2 Radio myRegion->spacing: 0.300000<\r><\n>
??:??:?? 2 Radio myRegion->numChannels: 10<\r><\n>
??:??:?? 2 Radio channel_num: 3<\r><\n>
??:??:?? 2 Radio frequency: 866.100037<\r><\n>
??:??:?? 2 Short packet time: 1363 msec<\r><\n>
??:??:?? 2 Set radio: final power level=17<\r><\n>
??:??:?? 2 Current limit set to 100.000000<\r><\n>
??:??:?? 2 Current limit set result 0<\r><\n>
??:??:?? 2 RF95 init result -19<\r><\n>
??:??:?? 2 Current limit set to 100.000000<\r><\n>
??:??:?? 2 Current limit set result 0<\r><\n>
??:??:?? 2 RF95 Radio init succeeded, using RF95 radio<\r><\n>
??:??:?? 2 Not using WIFI<\r><\n>
??:??:?? 2 PowerFSM init, USB power=1<\r><\n>
??:??:?? 2 Enter state: BOOT<\r><\n>
[D][esp32-hal-cpu.c:189] setCpuFrequencyMhz(): PLL: 320 / 4 = 80 Mhz, APB: 80000000 Hz<\r><\n>
??:??:?? 2 [Power] Battery: usbPower=1, isCharging=1, batMv=4132, batPct=100<\r><\n>
??:??:?? 2 [Screen] Screen: Started...<\r><\n>
??:??:?? 2 [GPS] publishing GPS lock=0<\r><\n>
??:??:?? 2 [GPS] No GPS lock<\r><\n>
??:??:?? 2 [GPS] got gps notify time=0, lat=0, bat=100<\r><\n>
??:??:?? 2 [GPS] DB update position node=0xf244c058 time=0, latI=0, lonI=0<\r><\n>
??:??:?? 2 [GPS] Node status update: 2 online, 2 total<\r><\n>
??:??:?? 2 [SerialPlugin] Serial Plugin Disabled<\r><\n>
??:??:?? 2 [StoreForwardPlugin] Store & Forward Plugin - Disabled<\r><\n>
??:??:?? 2 [RangeTestPlugin] Range Test Plugin - Disabled<\r><\n>
00:00:05 5 [GPS] Upgrading time to RTC 1631215762 secs (quality 2)<\r><\n>
19:29:22 5 [GPS] Read RTC time as 1631215762 (cur millis 5787) quality=2<\r><\n>
19:29:22 5 [GPS] publishing GPS lock=0<\r><\n>
19:29:22 5 [GPS] got gps notify time=1631215762, lat=0, bat=100<\r><\n>
19:29:22 5 [GPS] DB update position node=0xf244c058 time=1631215762, latI=0, lonI=0<\r><\n>
19:29:22 5 [GPS] Node status update: 1 online, 2 total<\r><\n>
19:29:22 5 [PowerFSM] Enter state: POWER<\r><\n>
19:29:22 5 [PowerFSM] Setting bluetooth enable=1<\r><\n>
19:29:22 5 [PowerFSM] Not using WIFI<\r><\n>
Pre BT: 208864 heap size<\n>
19:29:22 5 [PowerFSM] Starting bluetooth<\r><\n>
19:29:22 6 [Screen] Done with boot screen...<\r><\n>
19:29:22 6 [Screen] Screen: Powered...<\r><\n>
19:29:22 6 [Screen] showing standard frames<\r><\n>
19:29:22 6 [Screen] Showing 0 plugin frames<\r><\n>
19:29:22 6 [Screen] Total frame count: 35<\r><\n>
19:29:22 6 [Screen] Added plugins.  numframes: 0<\r><\n>
19:29:22 6 [Screen] Finished building frames. numframes: 4<\r><\n>
19:29:22 6 [Screen] Setting fast framerate<\r><\n>
19:29:22 6 [Screen] BLE task running<\r><\n>
19:29:22 6 [Screen] registered service 0x1800 with handle=1<\r><\n>
19:29:22 6 [Screen] registered service 0x1801 with handle=6<\r><\n>
19:29:22 6 [Screen] registered service 6ba1b218-15a8-461f-9fa8-5dcae273eafd with handle=10<\r><\n>
19:29:22 6 [Screen] registered service cb0b9a0b-a84c-4c0d-bdbb-442e3144ee30 with handle=18<\r><\n>
[D][esp32-hal-i2c.c:1345] i2cProcQueue():  Gross Timeout Dead start=0x1954, end=0x1954, =0, max=50 error=1<\r><\n>
[E][esp32-hal-i2c.c:318] i2cDumpI2c(): i2c=0x3ffbe2c8<\r><\n>
[I][esp32-hal-i2c.c:319] i2cDumpI2c(): dev=0x60013000 date=0x16042000<\r><\n>
[I][esp32-hal-i2c.c:321] i2cDumpI2c(): lock=0x3ffbc274<\r><\n>
[I][esp32-hal-i2c.c:323] i2cDumpI2c(): num=0<\r><\n>
[I][esp32-hal-i2c.c:324] i2cDumpI2c(): mode=1<\r><\n>
[I][esp32-hal-i2c.c:325] i2cDumpI2c(): stage=3<\r><\n>
[I][esp32-hal-i2c.c:326] i2cDumpI2c(): error=1<\r><\n>
[I][esp32-hal-i2c.c:327] i2cDumpI2c(): event=0x3ffbc2d4 bits=10<\r><\n>
[I][esp32-hal-i2c.c:328] i2cDumpI2c(): intr_handle=0x3ffbac0c<\r><\n>
[I][esp32-hal-i2c.c:329] i2cDumpI2c(): dq=0x3ffd9668<\r><\n>
[I][esp32-hal-i2c.c:330] i2cDumpI2c(): queueCount=1<\r><\n>
[I][esp32-hal-i2c.c:331] i2cDumpI2c(): queuePos=0<\r><\n>
[I][esp32-hal-i2c.c:332] i2cDumpI2c(): errorByteCnt=2<\r><\n>
[I][esp32-hal-i2c.c:333] i2cDumpI2c(): errorQueue=0<\r><\n>
[I][esp32-hal-i2c.c:334] i2cDumpI2c(): debugFlags=0x00000000<\r><\n>
[I][esp32-hal-i2c.c:311] i2cDumpDqData(): Debug Buffer not Enabled<\r><\n>
[I][esp32-hal-i2c.c:354] i2cDumpInts(): Debug Buffer not Enabled<\r><\n>
19:29:22 6 [Screen] BLE advertisting type=0, Private=0, Device Address: 91:08:3a:f2:44:c0<\r><\n>
<\r><\n>
19:29:22 6 [Screen] Setting idle framerate<\r><\n>
19:29:24 8 connection established; status=0 handle=0 our_ota_addr_type=0 our_ota_addr=00:08:3a:f2:44:c0<\r><\n>
 our_id_addr_type=0 our_id_addr=00:08:3a:f2:44:c0<\r><\n>
 peer_ota_addr_type=0 peer_ota_addr=00:50:46:5d:21:03<\r><\n>
 peer_id_addr_type=0 peer_id_addr=00:50:46:5d:21:03<\r><\n>
 conn_itvl=39 conn_latency=0 supervision_timeout=2000 encrypted=0 authenticated=0 bonded=0<\r><\n>
19:29:24 8 <\r><\n>
??<\0>?<\0>???<\0>?<\0><\0><\0>?<\0>?????<\0>??<\0><\0><\0><\0><\0>?<\0><\0><\0><\0><\0><\0><\0><\0><\0>?<\0><\0>?<\0><\0><\0><\0>?<\0><\0><\0><\0><\0><\0>?<\0><\0><\0><\0>?<\0><\0><\0>?<\0><\0>?<\0><\0>?<\0>??<\0><\0>?<\0>??<\0><\0>?<\0><\0><\0>??<\0>?<\0>??<\0>?<\0><\0><\0><\0>?<\0><\0><\0><\0><\0>?<\0><\0>????<\0>??<\0><\0>????<\0><\0>?<\0>???????<\0>???<\0>???<\0>?????<\0><\0>????<\0>??<\0>????<\0>?<\0>????<\0>?<\0><\0>?<\0><\0><\0>???????<\0>??<\0>?<\0><\0><\0><\0>??<\0><\0>?<\0><\0><\0><\0><\0>?<\0>?<\0>??<\0>???<\0><\0>????<\0><\0>??<\0>??<\0>??<\0>??????<\0><\0>?<\0><\0>??<\0>??<\0>??<\0><\0>????<\0><\0>??<\0>??<\0>?<\0>????<\0>?<\0>??<\0><\0>?<\0>?<\0>????<\0>??<\0><\0>??<\0>???<\0>??<\0>??<\0>??<\0>?<\0>?<\0><\0><\0><\0><\0><\0>?<\0><\0><\0>?<\0>??<\0>???<\0><\0>?????<\0>??<\0>?<\0><\0><\0><\0>???<\0>???<\0>???<\0><\0>?<\0>?<\0>?<\0>?????????<\0><\0>?<\0><\0>?<\0>??<\0>?<\0><\0><\0><\0>?<\0><\0><\0><\0><\0>?<\0>?<\0>????<\0><\0>?<\0><\0>?<\0>??<\0>?<\0><\0><\0><\0>?<\0><\0><\0><\0><\0>?<\0><\0><\0>????<\0><\0>?<\0><\0>?<\0>??<\0>?<\0><\0><\0><\0>?<\0><\0><\0><\0><\0>?<\0>?<\0>??<\0>?<\0><\0>???<\0><\0>?<\0><\0>?<\0>??<\0>?<\0><\0><\0><\0>?<\0><\0><\0><\0><\0>?<\0><\0>??<\0><\0>????<\0><\0>?<\0><\0>?<\0>??<\0>?<\0><\0><\0><\0>?<\0><\0><\0><\0><\0>?<\0>???<\0>????<\0><\0>?<\0><\0>?<\0>??<\0>?<\0><\0><\0><\0>?<\0><\0><\0><\0>??<\0><\0>?<\0>???????<\0><\0>???<\0>?<\0>?<\0><\0><\0>??<\0><\0>??<\0>?<\0>?<\0><\0><\0>?<\0>?<\0>??????<\0>?????<\0><\0><\0><\0>??<\0>??<\0>??<\0>?<\0><\0><\0><\0>??<\0><\0>?<\0><\0>??????<\0>?<\0><\0>?<\0>?<\0><\0><\0><\0>??<\0><\0><\0><\0>

I tried erasing and reflashing the device, but that did not help. I am new to the whole ESP32 ecosystem, how can I analyze what’s going wrong?
I tried the esp32_exception_decoder, but no luck:

[user@pc Meshtastic-device]$ pio device monitor --port /dev/ttyUSB1 -f esp32_exce
ption_decoder
--- Available filters and text transformations: colorize, debug, default, direct, esp32_exception_decod
er, hexlify, log2file, nocontrol, printable, send_on_enter, time
--- More details at http://bit.ly/pio-monitor-filters

Please build project in debug configuration to get more details about an exception.
See https://docs.platformio.org/page/projectconf/build_configurations.html

--- Miniterm on /dev/ttyUSB1  921600,8,N,1 ---
--- Quit: Ctrl+C | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---
�������������������������������������������������������������������������������������������������������
�������������������������������������������������������������������������������������������������������

II am getting just a bunch of “�” about every other second. Am I missing something?

Edit:

pio device monitor --port /dev/ttyUSB0 -f esp32_exception_decoder -b 115200

did the trick!

Rebooting...
ets Jul 29 2019 12:21:46

rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:728
load:0x40078000,len:9132
load:0x40080400,len:5100
entry 0x40080640
E (1487) spiram: SPI SRAM memory test fail. 131071/131072 writes failed, first @ 3F800000

E (1488) cpu_start: External RAM failed memory test!
abort() was called at PC 0x40082a12 on core 0

ELF file SHA256: 0000000000000000

Backtrace: 0x4008ff93:0x3ffe3c30 0x40090319:0x3ffe3c50 0x40082a12:0x3ffe3c70 0x40079075:0x3ffe3c90 0x400792a7:0x3ffe3cc0 0x40080681:0x3ffe3df0 0x40007c15:0x3ffe3eb0 0x4000073d:0x3ffe3f20
  #0  0x4008ff93:0x3ffe3c30 in invoke_abort at /home/kevinh/development/meshtastic/esp32-arduino-lib-builder/esp-idf/components/esp32/panic.c:715
  #1  0x40090319:0x3ffe3c50 in abort at /home/kevinh/development/meshtastic/esp32-arduino-lib-builder/esp-idf/components/esp32/panic.c:715
  #2  0x40082a12:0x3ffe3c70 in call_start_cpu0 at /home/kevinh/development/meshtastic/esp32-arduino-lib-builder/esp-idf/components/esp32/cpu_start.c:238 (discriminator 1)
  #3  0x40079075:0x3ffe3c90 in ?? ??:0
  #4  0x400792a7:0x3ffe3cc0 in ?? ??:0
  #5  0x40080681:0x3ffe3df0 in ?? ??:0
  #6  0x40007c15:0x3ffe3eb0 in ?? ??:0
  #7  0x4000073d:0x3ffe3f20 in ?? ??:0

Rebooting...
ets Jul 29 2019 12:21:46

.........

Seems like the SPI RAM is a lemon?

Edit2:
I just found out that the supported TTGO TLORA32 board has no external PSRAM. Is there any chance I can just “recompile without external PSRAM support” and just desolder the culprit?

Best regards!

1 Like