SMTPS MbedTLS TLS not working

In line 41, the SMTP client waits for the mbedTLS session to establish. This is for some reason terminated by the SMTP server. The sequence of events, that are from the Network library are fine.

Try to send a secure email using one of your private accounts, such as Google or Yahoo. If it works, you probably have a non-standard SMTP server configuration. Then check with your SMTP server, which options in mbedTLS need to be changed. Here are the settings for use:

Gmail:

SMTP server: smtp.gmail.com
Username   : <your gmail address>
Password   : <your gmail password>
SMTP port  : 465 (SSL)
             587 (TLS)
              25 (explicit TLS)
TLS/SSL    : yes

Yahoo:

SMTP server: smtp.mail.yahoo.com
Username   : <your yahoo email address>
Password   : <your yahoo password>
SMTP port  : 465 (SSL)
              25 (explicit TLS)
TLS/SSL    : yes

The other (and more complex) way is to enable debugging in mbedTLS. However, mbedTLS uses printf style, so you will need an ITM channel that will allow you to capture debuging messages. Here are some tips:
www.keil.com/.../network_troubleshoot.html

Hi Franc,

Thanks for the reply, I am not sure where you wanted to test my private account but I was able to send a secure email from windows powershell using the same settings I am trying to do with mbedTLS. After comparing the wireshark data from the failed mbedTLS and successful email from powershell, I noticed that they both get to the "Ready to start TLS" but, in the next line, the mbedTLS client returns an ACK and the powershell client returns a "Client Hello".

On the mbedTLS wireshark, after the ACK, the client then sends a RST and the connection is closed. On the powershell wireshark, after the "Client Hello", the server then sends an ACK followed by "Server Hello" and continues to send the secure email.

Therefore, it seems that mbedTLS is not beginning the TLS handshake because it is supposed to send a "Client Hello", not an ACK. On the Keil SMTPS guide, it says "It is not required to create your own certificates or keys to send secure e-mails using SMTPS.", so I belive that the certificates are not the problem.

I have tried to get the mbedTLS debugging to work but I am missing a header file and can't find the correct one in the Keil folders. I know that my server is accepting the STARTTLS since I successfully sent the email through windows powershell, the problem must be within the mbedTLS configuration. Any more suggestions?

-Kamal

UPDATE:

After finally getting the mbedTLS debugger to print to the Keil event recorder, I receive an error regarding allocating space.

Looking at the event recorder, after line 51, "ssl_tls.c:7484: alloc(15333 bytes) failed" is printed in hex. I have tried to add more space but in the mbedTLS file, ssl_internal.h, I found:

/* * Check that we obey the standard's message size bounds */

#if MBEDTLS_SSL_MAX_CONTENT_LEN > 16384

#error "Bad configuration - record content too large."
#endif

Meaning that if the message size is greater than 16384 bytes, it is too big. The message that should be printing is "Client Hello". Now I am trying to figure out why the message size is bigger than normal. Also on line 39, mbedTLS is saying that the Record Type is Not Supported, I am assuming this is not causing the later issue with TLS and that it involves the use of Ip6. Since Ip6 cannot be used, the connection is switched to Ip4 as shown in line 40.

Event Recorder, with print statements:
lines 7-15:
MW Network
SMTP Client
Sending email...

lines 52-57:
ssl_tls.c:7484: alloc(15333 bytes) failed

lines 61-63:
Send email failed.

Event, Time (sec), Component, Event Property, Value
0, 0.00297065, EvCtrl, EventRecorderInitialize, Restart Count = 9
1, 0.00300803, EvCtrl, EventRecorderStart,
2, 0.00347310, Net_SYS, InitSystem, ver=7.10.0
3, 0.00451524, Net_DHCP, InitClient, vcid=0 opt=0
4, 0.00453281, Net_DHCP, StartClient,
5, 0.00458062, Net_SMTP, InitClient, smtp
6, 0.00464454, Net_SYS, InitComplete, success
7, 0.00479595, STDIO, stdout, 0x4D 0x57 0x20 0x4E 0x65 0x74 0x77 0x6F
8, 0.00481483, STDIO, stdout, 0x72 0x6B 0x20 0x20 0x20 0x20 0x20 0x20
9, 0.00481529, STDIO, stdout, 0x0A 0x00 0x00 0x00 0x00 0x00 0x00 0x00
10, 0.00485145, STDIO, stdout, 0x0D 0x53 0x4D 0x54 0x50 0x20 0x43 0x6C
11, 0.00485261, STDIO, stdout, 0x69 0x65 0x6E 0x74 0x20 0x20 0x20 0x20
12, 0.00485311, STDIO, stdout, 0x20 0x0A 0x00 0x00 0x00 0x00 0x00 0x00
13, 0.00487348, STDIO, stdout, 0x0D 0x53 0x65 0x6E 0x64 0x69 0x6E 0x67
14, 0.00487439, STDIO, stdout, 0x20 0x65 0x6D 0x61 0x69 0x6C 0x2E 0x2E
15, 0.00487490, STDIO, stdout, 0x2E 0x0A 0x00 0x00 0x00 0x00 0x00 0x00
16, 0.00494046, Net_SMTP, SendMail, recipients=1
17, 0.00524577, Net_ETH, LinkDownError, no-link
18, 0.20544790, Net_ETH, LinkDownError, no-link
19, 2.20531747, Net_ETH, LinkDownError, no-link
20, 2.90546651, Net_DHCP, ClientState, state=INIT
21, 2.90549117, Net_DHCP, SendDhcpMessage, type=DHCP_DISCOVER bcast=1
22, 2.90562930, Net_DHCP, NextState, state=SELECTING
23, 2.90780736, Net_DHCP, ReceiveFrame, server=10.0.2.17 len=300
24, 2.90784296, Net_DHCP, ClientState, state=SELECTING
25, 2.90784781, Net_DHCP, DhcpOfferReceived,
26, 2.90785995, Net_DHCP, ShowServerId, server_id=10.0.2.17
27, 2.90787479, Net_DHCP, ShowOfferedIpAddress, ip=10.0.2.157
28, 2.90790023, Net_DHCP, SendDhcpMessage, type=DHCP_REQUEST bcast=1
29, 2.90800321, Net_DHCP, NextState, state=REQUESTING
30, 2.91153045, Net_DHCP, ReceiveFrame, server=10.0.2.17 len=300
31, 2.91153428, Net_DHCP, ClientState, state=REQUESTING
32, 2.91154255, Net_DHCP, DhcpAckReceived,
33, 2.91154700, Net_DHCP, ShowAssignedAddress, ip=10.0.2.157
34, 2.91157338, Net_DHCP, ShowLeaseTime, time=171072
35, 2.91158152, Net_DHCP, ShowNetMask, mask=255.255.0.0
36, 2.91158209, Net_DHCP, ShowGatewayAddress, gateway=10.0.2.2
37, 2.91158262, Net_DHCP, ShowDnsServers, pri=8.8.8.8 sec=8.8.4.4
38, 2.91158964, Net_DHCP, NextState, state=BOUND
39, 4.22416890, Net_DNS, RecordTypeNotSupported, rr_type=TYPE_CNAME
40, 4.22426921, Net_SMTP, ConnectIp4, ip=74.125.141.109 port=587
41, 4.25463296, Net_SMTP, SocketConnected, sock=1
42, 4.30152218, Net_SMTP, ReceiveFrame, sock=1 len=54
43, 4.30153476, Net_SMTP, ServerReady,
44, 4.30167701, Net_SMTP, SendCommand, command=EHLO evkb-imxrt1050
45, 4.33734949, Net_SMTP, ReceiveFrame, sock=1 len=168
46, 4.33736641, Net_SMTP, EsmtpModeActive,
47, 4.33736883, Net_SMTP, TlsSupportIndicated,
48, 4.33747245, Net_SMTP, SendCommand, command=STARTTLS
49, 4.36792064, Net_SMTP, ReceiveFrame, sock=1 len=30
50, 4.36792558, Net_SMTP, StartTlsAccepted,
51, 4.36806210, Net_SMTP, TlsModeStarted,
52, 4.36833193, STDIO, stdout, 0x0D 0x73 0x73 0x6C 0x5F 0x74 0x6C 0x73
53, 4.36833566, STDIO, stdout, 0x2E 0x63 0x3A 0x37 0x34 0x38 0x34 0x3A
54, 4.36833644, STDIO, stdout, 0x20 0x61 0x6C 0x6C 0x6F 0x63 0x28 0x31
55, 4.36833722, STDIO, stdout, 0x35 0x33 0x33 0x33 0x20 0x62 0x79 0x74
56, 4.36833801, STDIO, stdout, 0x65 0x73 0x29 0x20 0x66 0x61 0x69 0x6C
57, 4.36833856, STDIO, stdout, 0x65 0x64 0x0A 0x00 0x00 0x00 0x00 0x00
58, 4.36852025, Net_SMTP, SocketClosed, sock=1
59, 4.36862414, Net_SMTP, ClientCloseSocket, sock=1
60, 4.40548247, Net_SMTP, ClientDone, cb_event=Error
61, 4.40555172, STDIO, stdout, 0x53 0x65 0x6E 0x64 0x20 0x65 0x6D 0x61
62, 4.40555268, STDIO, stdout, 0x69 0x6C 0x20 0x66 0x61 0x69 0x6C 0x65
63, 4.40555328, STDIO, stdout, 0x64 0x2E 0x0A 0x00 0x00 0x00 0x00 0x00

It looks like your heap size is too small and the malloc fails. First try to increase the size of the heap so that the malloc will no longer fail.

The "mbedTLS_config.h" preconfigured for SMTP client does however limit the content size to 8k, which is usually sufficient:

#define MBEDTLS_SSL_MAX_CONTENT_LEN             8192

mbedTLS allocates two buffers MBEDTLS_SSL_IN_CONTENT_LEN and MBEDTLS_SSL_OUT_CONTENT_LEN per session. If you reduce max content length by 8k, you save 16k of a heap.

The buffer must be capable of accepting the entire chain of certificates, that is exchanged during the handshake. So if your SMTPS server has set up a certificate chain that is larger, 8k may not be enough. You can, however, do a little experiment with the sizes. Because this is an embedded application, we always face memory limitations.

The size of the message you want to send in an email and the max content length are not related. You can send emails that have hundreds of kilobytes.

uVision debug hint:

STDIO - stdout messages that are in the "Event Recorder" window in hex, are in the text in the second window "Debug (printf) Viewer". When debugging, you need to open both windows.

Hi Franc,

Thanks for the help, I finally got the email to go through using SMTPS. For the most part, I had to change the configuration file and increase the heap size. I also increased the MBEDTLS_SSL_MAX_CONTENT_LEN which fixed my last issue of mbedTLs not sending the TLS "Client Hello".

Another thing I wanted to do is move this SMTPS code over to an older board, lpc3250 Arm9, which is not supported by Keilv5. The Arm9 project is a bare metal environment with no operating system. Currently, the Arm9 application does have a bare metal SMTP protocol hand written, but I want to switch it to SMTPS. I am working on the lpc3250 processor using Keilv4 whcih does not include all the new software packs and run-time environment.

My questions are:
1. Will this mbedTLS SMTPS application work on a bare metal environment without a RTOS?
2. Does mbedTLS require an outside API, such as Windows API, to run TLS?
3. Is it possible to move over all the source code to the Arm9 Keilv4 project and get it to work with mbedTLS SMTPS?
4. I read that some TLS certificates require a date verification and my Arm9 embedded system does not keep track of the current date and time, is this involved in SMTPS?

-Kamal

Also,

I can send the mbed SMTPS TLS1.2 using ports 587 and 25 successfully, is that how it is supposed to work?

My mbedTLS SMTPS program does not work every single run. Sometimes the program will hang or fail. Why is the program not running the same every time?
Most of the time when it does hang, it will hang right after:

 39, 4.21519186, Net_SMTP, ConnectIp4, ip=10.0.1.2  port=587

Even in the successful email cases, I sometimes receive events like:

 19, 2.21033839, Net_ETH, LinkDownError, no-link

 39, 4.21497440, Net_DNS, RecordTypeNotSupported, rr_type=TYPE_CNAME

 41, 4.21587020, Net_TCP, FrameUnrecognised, sock=1

 42, 4.89372711, Net_TCP, NoRetriesLeft, sock=1

 43, 4.89384738, Net_SMTP, SocketAborted, sock=1

One example of a failed SMTPS email Keil event recorder:

 Event, Time (sec), Component, Event Property, Value
0, 0.00391493, EvCtrl, EventRecorderInitialize, Restart Count = 2
1, 0.00393650, EvCtrl, EventRecorderStart,
2, 0.00445639, Net_SYS, InitSystem, ver=7.10.0
3, 0.00548874, Net_DHCP, InitClient, vcid=0  opt=0
4, 0.00549881, Net_DHCP, StartClient,
5, 0.00554865, Net_SMTP, InitClient, smtp
6, 0.00562303, Net_SYS, InitComplete, success
7, 0.00579318, STDIO, stdout, 0x4D 0x57 0x20 0x4E 0x65 0x74 0x77 0x6F
8, 0.00582144, STDIO, stdout, 0x72 0x6B 0x20 0x20 0x20 0x20 0x20 0x20
9, 0.00582199, STDIO, stdout, 0x0A 0x00 0x00 0x00 0x00 0x00 0x00 0x00
10, 0.00587034, STDIO, stdout, 0x0D 0x53 0x4D 0x54 0x50 0x20 0x43 0x6C
11, 0.00587143, STDIO, stdout, 0x69 0x65 0x6E 0x74 0x20 0x20 0x20 0x20
12, 0.00587204, STDIO, stdout, 0x20 0x0A 0x00 0x00 0x00 0x00 0x00 0x00
13, 0.00588006, STDIO, stdout, 0x0D 0x53 0x65 0x6E 0x64 0x69 0x6E 0x67
14, 0.00588113, STDIO, stdout, 0x20 0x65 0x6D 0x61 0x69 0x6C 0x2E 0x2E
15, 0.00588173, STDIO, stdout, 0x2E 0x0A 0x00 0x00 0x00 0x00 0x00 0x00
16, 0.00593655, Net_SMTP, SendMail, recipients=1
17, 0.00626456, Net_ETH, LinkDownError, no-link
18, 0.21044865, Net_ETH, LinkDownError, no-link
19, 2.21036563, Net_ETH, LinkDownError, no-link
20, 2.91042894, Net_DHCP, ClientState, state=INIT
21, 2.91045321, Net_DHCP, SendDhcpMessage, type=DHCP_DISCOVER  bcast=1
22, 2.91066462, Net_DHCP, NextState, state=SELECTING
23, 2.91300110, Net_DHCP, ReceiveFrame, server=10.0.2.17  len=300
24, 2.91304104, Net_DHCP, ClientState, state=SELECTING
25, 2.91308409, Net_DHCP, DhcpOfferReceived,
26, 2.91309079, Net_DHCP, ShowServerId, server_id=10.0.2.17
27, 2.91311357, Net_DHCP, ShowOfferedIpAddress, ip=10.0.2.157
28, 2.91311873, Net_DHCP, SendDhcpMessage, type=DHCP_REQUEST  bcast=1
29, 2.91328019, Net_DHCP, NextState, state=REQUESTING
30, 2.91706581, Net_DHCP, ReceiveFrame, server=10.0.2.17  len=300
31, 2.91708329, Net_DHCP, ClientState, state=REQUESTING
32, 2.91709156, Net_DHCP, DhcpAckReceived,
33, 2.91709607, Net_DHCP, ShowAssignedAddress, ip=10.0.2.157
34, 2.91711923, Net_DHCP, ShowLeaseTime, time=171072
35, 2.91715024, Net_DHCP, ShowNetMask, mask=255.255.0.0
36, 2.91716467, Net_DHCP, ShowGatewayAddress, gateway=10.0.2.2
37, 2.91716525, Net_DHCP, ShowDnsServers, pri=8.8.8.8  sec=8.8.4.4
38, 2.91718709, Net_DHCP, NextState, state=BOUND
39, 4.21523585, Net_DNS, RecordTypeNotSupported, rr_type=TYPE_CNAME
40, 4.21535812, Net_SMTP, ConnectIp4, ip=74.125.141.108  port=587
41, 4.89372711, Net_TCP, NoRetriesLeft, sock=1
42, 4.89384738, Net_SMTP, SocketAborted, sock=1
43, 4.89390436, Net_SMTP, ClientCloseSocket, sock=1
44, 4.99372952, Net_SMTP, ClientDone, cb_event=Error
45, 4.99384296, STDIO, stdout, 0x0D 0x53 0x65 0x6E 0x64 0x20 0x65 0x6D
46, 4.99384931, STDIO, stdout, 0x61 0x69 0x6C 0x20 0x66 0x61 0x69 0x6C
47, 4.99385004, STDIO, stdout, 0x65 0x64 0x2E 0x0A 0x00 0x00 0x00 0x00

It looks like you have an invalid system clock configuration. Message 40 starts the TCP connection. If the remote host does not respond, TCP resends the SYN packet. It takes 15 seconds before the message 41 is recorded.

If I calculate the delay, the kernel clock is 25 Mhz, but it is likely to be set at 600 MHz. The errors you mention are the result of invalid clock settings.

Thanks for letting me know about the clock settings. Do you have any advise on my earlier message about the bare metal environment?

My questions are:
1. Will this mbedTLS SMTPS application work on a bare metal environment without a RTOS?
2. Does mbedTLS require an outside API, such as Windows API, to run TLS?
3. Is it possible to move over all the source code to the Arm9 Keilv4 project and get it to work with mbedTLS SMTPS?
4. I read that some TLS certificates require a date verification and my Arm9 embedded system does not keep track of the current date and time, is this involved in SMTPS?
5. Can I just copy over all the source and header files from my working SMTPS Arm Cortex 7 project to my bare metal Arm9 project, including all the RTOS and mbedTLS files?

For a bare metal implementation, mbedTLS requires a socket API. It does not matter on which platform. The network socket interface is in the "net_socket.c".

For MDK middleware, the interface is in:

C:\Keil\ARM\PACK\ARM\mbedTLS\1.6.0\MDK\library\mw_network\net_sockets.c

The SMTPS client demo example is in:

C:\Keil\ARM\PACK\ARM\mbedTLS\1.6.0\programs\ssl\ssl_mail_client.c

You can use those as a reference. However the SMTPS client demo example from mbedTLS pack is only a basic implementation and does not support attachments, multiple recipients, address friendly names and different character encodings.

Note that MDK Middleware version 7 uses a different mbedTLS interface, which allows integration into the MDK MW as an external component.

The latest version of the MDK MW that supports ARM9 devices is Version 4. Subsequent versions only support Cortex-M devices.

And NO, you can not simply move the code to ARM9 device and expect it to work. The SMTP client in MDK MW version 4 is very limited, has a different API, and does not support SMTPS.