[Libevent-users] Memory (heap meta-data) corruption while using buffervent_openssl apis

[Praveen Kariyanahalli <praveen@xxxxxxxxxxx>]

Client (bufferevent + openssl) crash only under load situation. Valgrind points to this backtrace. Any help is greatly appreciated ? Please let us know if you need any more information.

Thanks

-Praveen

Version: 2.0.22

==695== Invalid write of size 8

==695== Â Âat 0x52CF66B: evmap_io_add (evmap.c:328)

==695== Â Âby 0x52BC638: event_add_internal (event.c:2073)

==695== Â Âby 0x52BC150: event_add (event.c:1966)

==695== Â Âby 0x5504678: bufferevent_openssl_new_impl (bufferevent_openssl.c:1368)

==695== Â Âby 0x550482B: bufferevent_openssl_socket_new (bufferevent_openssl.c:1440)

==695== Â Âby 0x43211A: tls_connect_timer_cb (my_client_peer.c:528)

==695== Â Âby 0x488AC6: timer_exec_pri (timer.c:612)

==695== Â Âby 0x4885A1: timer_exec (timer.c:504)

==695== Â Âby 0x41A165: my_client_base_timer_cb (my_client.c:4671)

==695== Â Âby 0x52BA2CC: event_process_active_single_queue (event.c:1350)

==695== Â Âby 0x52BA540: event_process_active (event.c:1420)

==695== Â Âby 0x52BABA7: event_base_loop (event.c:1621)

==695== Â Âby 0x41C5D9: my_client_main (my_client.c:5437)

==695== Â Âby 0x41E5BA: main (my_client.c:6024)

==695== ÂAddress 0x86ff848 is 232 bytes inside a block of size 608 free'd

==695== Â Âat 0x4A0739B: free (vg_replace_malloc.c:473)

==695== Â Âby 0x48B3EB: vip_guard_mem_free (vip_gaurd_mem.c:157)

==695== Â Âby 0x52BE3F7: event_mm_free_ (event.c:2696)

==695== Â Âby 0x52C878B: _bufferevent_decref_and_unlock (bufferevent.c:650)

==695== Â Âby 0x52C8874: bufferevent_free (bufferevent.c:681)

==695== Â Âby 0x435747: my_client_peer_delete (my_client_peer.c:1257)

==695== Â Âby 0x430EE8: my_client_peer_timer_exp_cb (my_client_peer.c:267)

==695== Â Âby 0x488AC6: timer_exec_pri (timer.c:612)

==695== Â Âby 0x4885A1: timer_exec (timer.c:504)

==695== Â Âby 0x41A165: my_client_base_timer_cb (my_client.c:4671)

==695== Â Âby 0x52BA2CC: event_process_active_single_queue (event.c:1350)

==695== Â Âby 0x52BA540: event_process_active (event.c:1420)

==695== Â Âby 0x52BABA7: event_base_loop (event.c:1621)

==695== Â Âby 0x41C5D9: my_client_main (my_client.c:5437)

==695== Â Âby 0x41E5BA: main (my_client.c:6024)

==695==

==695== Invalid write of size 8

==695== Â Âat 0x52CF8DD: evmap_io_del (evmap.c:384)

==695== Â Âby 0x52BCE7B: event_del_internal (event.c:2251)

==695== Â Âby 0x52BCBD7: event_del (event.c:2188)

==695== Â Âby 0x55034A0: consider_writing (bufferevent_openssl.c:881)

==695== Â Âby 0x5503D41: be_openssl_outbuf_cb (bufferevent_openssl.c:1119)

==695== Â Âby 0x52C0B58: evbuffer_run_callbacks (buffer.c:486)

==695== Â Âby 0x52C0C51: evbuffer_invoke_callbacks (buffer.c:508)

==695== Â Âby 0x52C61AD: evbuffer_add_reference (buffer.c:2718)

==695== Â Âby 0x41EF6E: my_client_pkt_send (my_client_msg.c:75)

==695== Â Âby 0x445807: my_client_tunn_event_cb (my_client_tun_intf.c:157)

==695== Â Âby 0x52BA162: event_persist_closure (event.c:1301)

==695== Â Âby 0x52BA271: event_process_active_single_queue (event.c:1345)

==695== Â Âby 0x52BA540: event_process_active (event.c:1420)

==695== Â Âby 0x52BABA7: event_base_loop (event.c:1621)

==695== Â Âby 0x41C5D9: my_client_main (my_client.c:5437)

==695== Â Âby 0x41E5BA: main (my_client.c:6024)

==695== ÂAddress 0x86ff848 is 232 bytes inside a block of size 608 free'd

==695== Â Âat 0x4A0739B: free (vg_replace_malloc.c:473)

==695== Â Âby 0x48B3EB: vip_guard_mem_free (vip_gaurd_mem.c:157)

==695== Â Âby 0x52BE3F7: event_mm_free_ (event.c:2696)

==695== Â Âby 0x52C878B: _bufferevent_decref_and_unlock (bufferevent.c:650)

==695== Â Âby 0x52C8874: bufferevent_free (bufferevent.c:681)

==695== Â Âby 0x435747: my_client_peer_delete (my_client_peer.c:1257)

==695== Â Âby 0x430EE8: my_client_peer_timer_exp_cb (my_client_peer.c:267)

==695== Â Âby 0x488AC6: timer_exec_pri (timer.c:612)

==695== Â Âby 0x4885A1: timer_exec (timer.c:504)

==695== Â Âby 0x41A165: my_client_base_timer_cb (my_client.c:4671)

==695== Â Âby 0x52BA2CC: event_process_active_single_queue (event.c:1350)

==695== Â Âby 0x52BA540: event_process_active (event.c:1420)

==695== Â Âby 0x52BABA7: event_base_loop (event.c:1621)

==695== Â Âby 0x41C5D9: my_client_main (my_client.c:5437)

==695== Â Âby 0x41E5BA: main (my_client.c:6024)

==695==

==695== Invalid write of size 8

==695== Â Âat 0x52CF8DD: evmap_io_del (evmap.c:384)

==695== Â Âby 0x52BCE7B: event_del_internal (event.c:2251)

==695== Â Âby 0x52BCBD7: event_del (event.c:2188)

==695== Â Âby 0x550260C: stop_reading (bufferevent_openssl.c:422)

==695== Â Âby 0x55028CE: conn_closed (bufferevent_openssl.c:528)

==695== Â Âby 0x5502F56: do_write (bufferevent_openssl.c:694)

==695== Â Âby 0x55033ED: consider_writing (bufferevent_openssl.c:874)

==695== Â Âby 0x5503D41: be_openssl_outbuf_cb (bufferevent_openssl.c:1119)

==695== Â Âby 0x52C0B58: evbuffer_run_callbacks (buffer.c:486)

==695== Â Âby 0x52C0C51: evbuffer_invoke_callbacks (buffer.c:508)

==695== Â Âby 0x52C61AD: evbuffer_add_reference (buffer.c:2718)

==695== Â Âby 0x41EF6E: my_client_pkt_send (my_client_msg.c:75)

==695== Â Âby 0x445807: my_client_tunn_event_cb (my_client_tun_intf.c:157)

==695== Â Âby 0x52BA162: event_persist_closure (event.c:1301)

==695== Â Âby 0x52BA271: event_process_active_single_queue (event.c:1345)

==695== Â Âby 0x52BA540: event_process_active (event.c:1420)

==695== Â Âby 0x52BABA7: event_base_loop (event.c:1621)

==695== Â Âby 0x41C5D9: my_client_main (my_client.c:5437)

==695== Â Âby 0x41E5BA: main (my_client.c:6024)

==695== ÂAddress 0x86ff848 is 232 bytes inside a block of size 608 free'd

==695== Â Âat 0x4A0739B: free (vg_replace_malloc.c:473)

==695== Â Âby 0x48B3EB: vip_guard_mem_free (vip_gaurd_mem.c:157)

==695== Â Âby 0x52BE3F7: event_mm_free_ (event.c:2696)

==695== Â Âby 0x52C878B: _bufferevent_decref_and_unlock (bufferevent.c:650)

==695== Â Âby 0x52C8874: bufferevent_free (bufferevent.c:681)

==695== Â Âby 0x435747: my_client_peer_delete (my_client_peer.c:1257)

==695== Â Âby 0x430EE8: my_client_peer_timer_exp_cb (my_client_peer.c:267)

==695== Â Âby 0x488AC6: timer_exec_pri (timer.c:612)

==695== Â Âby 0x4885A1: timer_exec (timer.c:504)

==695== Â Âby 0x41A165: my_client_base_timer_cb (my_client.c:4671)

==695== Â Âby 0x52BA2CC: event_process_active_single_queue (event.c:1350)

==695== Â Âby 0x52BA540: event_process_active (event.c:1420)

==695== Â Âby 0x52BABA7: event_base_loop (event.c:1621)

==695== Â Âby 0x41C5D9: my_client_main (my_client.c:5437)

==695== Â Âby 0x41E5BA: main (my_client.c:6024)

==695==

==695== Invalid write of size 8

==695== Â Âat 0x52CF8DD: evmap_io_del (evmap.c:384)

==695== Â Âby 0x52BCE7B: event_del_internal (event.c:2251)

==695== Â Âby 0x52BCBD7: event_del (event.c:2188)

==695== Â Âby 0x5502659: stop_writing (bufferevent_openssl.c:435)

==695== Â Âby 0x55028DA: conn_closed (bufferevent_openssl.c:529)

==695== Â Âby 0x5502F56: do_write (bufferevent_openssl.c:694)

==695== Â Âby 0x55033ED: consider_writing (bufferevent_openssl.c:874)

==695== Â Âby 0x5503D41: be_openssl_outbuf_cb (bufferevent_openssl.c:1119)

==695== Â Âby 0x52C0B58: evbuffer_run_callbacks (buffer.c:486)

==695== Â Âby 0x52C0C51: evbuffer_invoke_callbacks (buffer.c:508)

==695== Â Âby 0x52C61AD: evbuffer_add_reference (buffer.c:2718)

==695== Â Âby 0x41EF6E: my_client_pkt_send (my_client_msg.c:75)

==695== Â Âby 0x445807: my_client_tunn_event_cb (my_client_tun_intf.c:157)

==695== Â Âby 0x52BA162: event_persist_closure (event.c:1301)

==695== Â Âby 0x52BA271: event_process_active_single_queue (event.c:1345)

==695== Â Âby 0x52BA540: event_process_active (event.c:1420)

==695== Â Âby 0x52BABA7: event_base_loop (event.c:1621)

==695== Â Âby 0x41C5D9: my_client_main (my_client.c:5437)

==695== Â Âby 0x41E5BA: main (my_client.c:6024)

==695== ÂAddress 0x86ff848 is 232 bytes inside a block of size 608 free'd

==695== Â Âat 0x4A0739B: free (vg_replace_malloc.c:473)

==695== Â Âby 0x48B3EB: vip_guard_mem_free (vip_gaurd_mem.c:157)

==695== Â Âby 0x52BE3F7: event_mm_free_ (event.c:2696)

==695== Â Âby 0x52C878B: _bufferevent_decref_and_unlock (bufferevent.c:650)

==695== Â Âby 0x52C8874: bufferevent_free (bufferevent.c:681)

==695== Â Âby 0x435747: my_client_peer_delete (my_client_peer.c:1257)

==695== Â Âby 0x430EE8: my_client_peer_timer_exp_cb (my_client_peer.c:267)

==695== Â Âby 0x488AC6: timer_exec_pri (timer.c:612)

==695== Â Âby 0x4885A1: timer_exec (timer.c:504)

==695== Â Âby 0x41A165: my_client_base_timer_cb (my_client.c:4671)

==695== Â Âby 0x52BA2CC: event_process_active_single_queue (event.c:1350)

==695== Â Âby 0x52BA540: event_process_active (event.c:1420)

==695== Â Âby 0x52BABA7: event_base_loop (event.c:1621)

==695== Â Âby 0x41C5D9: my_client_main (my_client.c:5437)

==695== Â Âby 0x41E5BA: main (my_client.c:6024)

==695==

==695== Invalid write of size 8

==695== Â Âat 0x52CF8DD: evmap_io_del (evmap.c:384)

==695== Â Âby 0x52BCE7B: event_del_internal (event.c:2251)

==695== Â Âby 0x52BCBD7: event_del (event.c:2188)

==695== Â Âby 0x5503FB5: be_openssl_destruct (bufferevent_openssl.c:1183)

==695== Â Âby 0x52C85F5: _bufferevent_decref_and_unlock (bufferevent.c:622)

==695== Â Âby 0x52C8874: bufferevent_free (bufferevent.c:681)

==695== Â Âby 0x435747: my_client_peer_delete (my_client_peer.c:1257)

==695== Â Âby 0x430EE8: my_client_peer_timer_exp_cb (my_client_peer.c:267)

==695== Â Âby 0x488AC6: timer_exec_pri (timer.c:612)

==695== Â Âby 0x4885A1: timer_exec (timer.c:504)

==695== Â Âby 0x41A165: my_client_base_timer_cb (my_client.c:4671)

==695== Â Âby 0x52BA2CC: event_process_active_single_queue (event.c:1350)

==695== Â Âby 0x52BA540: event_process_active (event.c:1420)

==695== Â Âby 0x52BABA7: event_base_loop (event.c:1621)

==695== Â Âby 0x41C5D9: my_client_main (my_client.c:5437)

==695== Â Âby 0x41E5BA: main (my_client.c:6024)

==695== ÂAddress 0x86ff848 is 232 bytes inside a block of size 560 free'd

==695== Â Âat 0x4A0739B: free (vg_replace_malloc.c:473)

==695== Â Âby 0x48B3EB: vip_guard_mem_free (vip_gaurd_mem.c:157)

==695== Â Âby 0x41EE2F: my_client_pktbuf_free (my_client_msg.c:54)

==695== Â Âby 0x52BFFEC: evbuffer_chain_free (buffer.c:198)

==695== Â Âby 0x52C251D: evbuffer_drain (buffer.c:972)

==695== Â Âby 0x5502FA2: do_write (bufferevent_openssl.c:703)

==695== Â Âby 0x55033ED: consider_writing (bufferevent_openssl.c:874)

==695== Â Âby 0x5503D41: be_openssl_outbuf_cb (bufferevent_openssl.c:1119)

==695== Â Âby 0x52C0B58: evbuffer_run_callbacks (buffer.c:486)

==695== Â Âby 0x52C0C51: evbuffer_invoke_callbacks (buffer.c:508)

==695== Â Âby 0x52C61AD: evbuffer_add_reference (buffer.c:2718)

==695== Â Âby 0x41EF6E: my_client_pkt_send (my_client_msg.c:75)

==695== Â Âby 0x426213: my_client_generic_send_message (my_client_msg.c:1708)

==695== Â Âby 0x4297B9: my_client_peer_send_hello (my_client_msg.c:2181)

==695== Â Âby 0x488AC6: timer_exec_pri (timer.c:612)

==695== Â Âby 0x4885A1: timer_exec (timer.c:504)

==695== Â Âby 0x41A165: my_client_base_timer_cb (my_client.c:4671)

==695== Â Âby 0x52BA2CC: event_process_active_single_queue (event.c:1350)

==695== Â Âby 0x52BA540: event_process_active (event.c:1420)

==695== Â Âby 0x52BABA7: event_base_loop (event.c:1621)

==695== Â Âby 0x41C5D9: my_client_main (my_client.c:5437)

==695== Â Âby 0x41E5BA: main (my_client.c:6024)

==695==

==695== Invalid write of size 8

==695== Â Âat 0x52CF66B: evmap_io_add (evmap.c:328)

==695== Â Âby 0x52BC638: event_add_internal (event.c:2073)

==695== Â Âby 0x52BC150: event_add (event.c:1966)

==695== Â Âby 0x52E695D: _evdns_nameserver_add_impl (evdns.c:2496)

==695== Â Âby 0x52E6D7C: evdns_base_nameserver_ip_add (evdns.c:2592)

==695== Â Âby 0x52E9B9B: resolv_conf_parse_line (evdns.c:3471)

==695== Â Âby 0x52E9ECE: evdns_base_resolv_conf_parse_impl (evdns.c:3572)

==695== Â Âby 0x52E9D60: evdns_base_resolv_conf_parse (evdns.c:3508)

==695== Â Âby 0x52EA1A7: evdns_base_new (evdns.c:3881)

==695== Â Âby 0x41AF33: my_client_connect_to_peer_my_clients (my_client.c:4978)

==695== Â Âby 0x430FC6: my_client_peer_timer_exp_cb (my_client_peer.c:285)

==695== Â Âby 0x488AC6: timer_exec_pri (timer.c:612)

==695== Â Âby 0x4885A1: timer_exec (timer.c:504)

==695== Â Âby 0x41A165: my_client_base_timer_cb (my_client.c:4671)

==695== Â Âby 0x52BA2CC: event_process_active_single_queue (event.c:1350)

==695== Â Âby 0x52BA540: event_process_active (event.c:1420)

==695== Â Âby 0x52BABA7: event_base_loop (event.c:1621)

==695== Â Âby 0x41C5D9: my_client_main (my_client.c:5437)

==695== Â Âby 0x41E5BA: main (my_client.c:6024)

==695== ÂAddress 0x86ff848 is 1,480 bytes inside an unallocated block of size 2,960 in arena "client"

==695==

==695== Invalid write of size 8

==695== Â Âat 0x52CF8DD: evmap_io_del (evmap.c:384)

==695== Â Âby 0x52BCE7B: event_del_internal (event.c:2251)

==695== Â Âby 0x52BCBD7: event_del (event.c:2188)

==695== Â Âby 0x52EA318: evdns_nameserver_free (evdns.c:3930)

==695== Â Âby 0x52EA4EB: evdns_base_free_and_unlock (evdns.c:3965)

==695== Â Âby 0x52EA6DD: evdns_base_free (evdns.c:4001)

==695== Â Âby 0x41A5AE: my_client_dns_getaddrinfo_cb (my_client.c:4778)

==695== Â Âby 0x52EB9E9: evdns_getaddrinfo (evdns.c:4563)

==695== Â Âby 0x41B02E: my_client_connect_to_peer_my_clients (my_client.c:5000)

==695== Â Âby 0x430FC6: my_client_peer_timer_exp_cb (my_client_peer.c:285)

==695== Â Âby 0x488AC6: timer_exec_pri (timer.c:612)

==695== Â Âby 0x4885A1: timer_exec (timer.c:504)

==695== Â Âby 0x41A165: my_client_base_timer_cb (my_client.c:4671)

==695== Â Âby 0x52BA2CC: event_process_active_single_queue (event.c:1350)

==695== Â Âby 0x52BA540: event_process_active (event.c:1420)

==695== Â Âby 0x52BABA7: event_base_loop (event.c:1621)

==695== Â Âby 0x41C5D9: my_client_main (my_client.c:5437)

==695== Â Âby 0x41E5BA: main (my_client.c:6024)

==695== ÂAddress 0x86ff848 is 1,480 bytes inside an unallocated block of size 2,960 in arena "client"

==695==

==695== Invalid write of size 8

==695== Â Âat 0x52CF66B: evmap_io_add (evmap.c:328)

==695== Â Âby 0x52BC638: event_add_internal (event.c:2073)

==695== Â Âby 0x52BC150: event_add (event.c:1966)

==695== Â Âby 0x431580: ssl_connect_timer_cb (my_client_peer.c:371)

==695== Â Âby 0x488AC6: timer_exec_pri (timer.c:612)

==695== Â Âby 0x4885A1: timer_exec (timer.c:504)

==695== Â Âby 0x41A165: my_client_base_timer_cb (my_client.c:4671)

==695== Â Âby 0x52BA2CC: event_process_active_single_queue (event.c:1350)

==695== Â Âby 0x52BA540: event_process_active (event.c:1420)

==695== Â Âby 0x52BABA7: event_base_loop (event.c:1621)

==695== Â Âby 0x41C5D9: my_client_main (my_client.c:5437)

==695== Â Âby 0x41E5BA: main (my_client.c:6024)

==695== ÂAddress 0x86ff848 is 1,480 bytes inside an unallocated block of size 2,960 in arena "client"

==695==

==695== Invalid write of size 8

==695== Â Âat 0x52CF8DD: evmap_io_del (evmap.c:384)

==695== Â Âby 0x52BCE7B: event_del_internal (event.c:2251)

==695== Â Âby 0x52BCBD7: event_del (event.c:2188)

==695== Â Âby 0x52BB484: event_free (event.c:1809)

==695== Â Âby 0x435B36: my_client_peer_delete (my_client_peer.c:1323)

==695== Â Âby 0x430EE8: my_client_peer_timer_exp_cb (my_client_peer.c:267)

==695== Â Âby 0x488AC6: timer_exec_pri (timer.c:612)

==695== Â Âby 0x4885A1: timer_exec (timer.c:504)

==695== Â Âby 0x41A165: my_client_base_timer_cb (my_client.c:4671)

==695== Â Âby 0x52BA2CC: event_process_active_single_queue (event.c:1350)

==695== Â Âby 0x52BA540: event_process_active (event.c:1420)

==695== Â Âby 0x52BABA7: event_base_loop (event.c:1621)

==695== Â Âby 0x41C5D9: my_client_main (my_client.c:5437)

==695== Â Âby 0x41E5BA: main (my_client.c:6024)

==695== ÂAddress 0x86ff848 is 8 bytes inside a block of size 160 free'd

==695== Â Âat 0x4A0739B: free (vg_replace_malloc.c:473)

==695== Â Âby 0x48B3EB: vip_guard_mem_free (vip_gaurd_mem.c:157)

==695== Â Âby 0x5BFD5F9: CRYPTO_free (mem.c:401)

==695== Â Âby 0x5CA9915: EVP_MD_CTX_cleanup (digest.c:390)

==695== Â Âby 0x5CA532E: ssleay_rand_bytes (md_rand.c:525)

==695== Â Âby 0x5CA53DB: ssleay_rand_pseudo_bytes (md_rand.c:548)

==695== Â Âby 0x5CA5C67: RAND_pseudo_bytes (rand_lib.c:173)

==695== Â Âby 0x594394A: ssl_fill_hello_random (s23_clnt.c:294)

==695== Â Âby 0x5951112: dtls1_client_hello (d1_clnt.c:801)

==695== Â Âby 0x59505CB: dtls1_connect (d1_clnt.c:302)

==695== Â Âby 0x595C8E0: SSL_connect (ssl_lib.c:943)

==695== Â Âby 0x441DBC: create_ssl_conn_to_peer (my_client_misc.c:2134)

==695== Â Âby 0x4346AB: my_client_peer_create (my_client_peer.c:1018)

==695== Â Âby 0x41B1BB: my_client_connect_to_peer_my_clients (my_client.c:5022)

==695== Â Âby 0x41ABF9: my_client_dns_getaddrinfo_cb (my_client.c:4886)

==695== Â Âby 0x52EB9E9: evdns_getaddrinfo (evdns.c:4563)

==695== Â Âby 0x41B02E: my_client_connect_to_peer_my_clients (my_client.c:5000)

==695== Â Âby 0x430FC6: my_client_peer_timer_exp_cb (my_client_peer.c:285)

==695== Â Âby 0x488AC6: timer_exec_pri (timer.c:612)

==695== Â Âby 0x4885A1: timer_exec (timer.c:504)

==695== Â Âby 0x41A165: my_client_base_timer_cb (my_client.c:4671)

==695== Â Âby 0x52BA2CC: event_process_active_single_queue (event.c:1350)

==695== Â Âby 0x52BA540: event_process_active (event.c:1420)

==695== Â Âby 0x52BABA7: event_base_loop (event.c:1621)

==695== Â Âby 0x41C5D9: my_client_main (my_client.c:5437)

==695==

valgrind: /jenkins/master/builder/x86_64/tmp/work/x86_64-poky-linux/valgrind/3.10.0-r15/valgrind-3.10.0/coregrind/m_mallocfree.c:304

(get_bszB_as_is): Assertion 'bszB_lo == bszB_hi' failed.

valgrind: Heap block lo/hi size mismatch: lo = 224, hi = 0.