debugger: new module for interactive cfg debugging

- print config execution path of a SIP message to logs
- interactive debugging for config, similar to gdb
- step-by-step execution of each action in the config file
- the SIP router process stops at cfg actions and waits for RPC commands
  to go to next one, evaluate a pseudo-variable, continue execution, etc.
- see README for more details

modules/debugger/Makefile

@@ -0,0 +1,12 @@
+# $Id$
+#
+# WARNING: do not run this directly, it should be run by the master Makefile
+
+include ../../Makefile.defs
+auto_gen=
+NAME=debugger.so
+LIBS=
+
+DEFS+=-DOPENSER_MOD_INTERFACE
+
+include ../../Makefile.modules

modules/debugger/README

@@ -0,0 +1,325 @@
+debugger Module
+
+Daniel-Constantin Mierla
+
+   asipto.com
+
+Edited by
+
+Daniel-Constantin Mierla
+
+   <[email protected]>
+
+   Copyright © 2010 Daniel-Constantin Mierla (asipto.com)
+     __________________________________________________________________
+
+   Table of Contents
+
+   1. Admin Guide
+
+        1. Overview
+        2. Dependencies
+
+              2.1. Kamailio Modules
+              2.2. External Libraries or Applications
+
+        3. Exported Parameters
+
+              3.1. cfgtrace (int)
+              3.2. breakpoint (int)
+
+        4. Exported Functions
+
+              4.1. dbg_breakpoint(mode)
+
+        5. Exported RPC Functions
+
+              5.1. dbg.ls
+              5.2. dbg.trace
+              5.3. dbg.bp
+
+        6. Usage
+        7. To-do
+
+   List of Examples
+
+   1.1. Set cfgtrace parameter
+   1.2. Set breakpoint parameter
+   1.3. mt_match usage
+
+Chapter 1. Admin Guide
+
+   Table of Contents
+
+   1. Overview
+   2. Dependencies
+
+        2.1. Kamailio Modules
+        2.2. External Libraries or Applications
+
+   3. Exported Parameters
+
+        3.1. cfgtrace (int)
+        3.2. breakpoint (int)
+
+   4. Exported Functions
+
+        4.1. dbg_breakpoint(mode)
+
+   5. Exported RPC Functions
+
+        5.1. dbg.ls
+        5.2. dbg.trace
+        5.3. dbg.bp
+
+   6. Usage
+   7. To-do
+
+1. Overview
+
+   This module provides an interactive config file debugger. It can print
+   the trace of config execution for a SIP message to log and set
+   breakpoints on every config action, allowing to execute step by step
+   the config.
+
+   Debugging can be done from local or remote host via RPC interface
+   (e.g., XMLRPC, sercmd).
+
+   The framework to set breakpoints on specific actions and config lines
+   is not exported to RPC yet, right now each action has an breakpoint.
+   The breakpoint can be enabled/disabled at runtime. Also the config
+   running trace can be enabled/disabled at runtime.
+
+   When the SIP router process is stopped at a breakpoint, you can
+   investigate the values of any pseudo-varaibles. Note that some of
+   pseudo-variables may produce memory leak, that is planned to fix in the
+   future (here falls pseudo-variables with dynamic name such as htable,
+   sqlops). References to SIP message, avps, headers, script and shared
+   variables are safe.
+
+2. Dependencies
+
+   2.1. Kamailio Modules
+   2.2. External Libraries or Applications
+
+2.1. Kamailio Modules
+
+   The following modules must be loaded before this module:
+     * none.
+
+2.2. External Libraries or Applications
+
+   The following libraries or applications must be installed before
+   running Kamailio with this module loaded:
+     * None.
+
+3. Exported Parameters
+
+   3.1. cfgtrace (int)
+   3.2. breakpoint (int)
+
+3.1. cfgtrace (int)
+
+   Control whether config running trace is enabled or disabled at startup.
+
+   Default value is "0" (disabled).
+
+   Example 1.1. Set cfgtrace parameter
+...
+modparam("debugger", "cfgtrace", 1)
+...
+
+3.2. breakpoint (int)
+
+   Control whether every line (for now) breakpoint is enabled or disabled
+   at startup.
+
+   Default value is "0" (disabled).
+
+   Example 1.2. Set breakpoint parameter
+...
+modparam("debugger", "breakpoint", 1)
+...
+
+4. Exported Functions
+
+   4.1. dbg_breakpoint(mode)
+
+4.1.  dbg_breakpoint(mode)
+
+   Achor a breakpoint at that line in config. Mode specifies whether the
+   breakpoint is enabled (1) or disabled (0) at startup.
+
+   Note that this version does not export this anchors to RPC for
+   interactive debugging (temporary disabled).
+
+   Example 1.3. mt_match usage
+...
+if($si=="10.0.0.10")
+        dbg_breakpoint("1");
+...
+
+5. Exported RPC Functions
+
+   5.1. dbg.ls
+   5.2. dbg.trace
+   5.3. dbg.bp
+
+5.1.  dbg.ls
+
+   List SIP router processes with info related to interactive debugging.
+
+   Name: dbg.list
+
+   Parameters:
+     * _pid_ : pid for which to list the details. If it missing, then will
+       print for all processes.
+
+   Examples for using with sercmd:
+                dbg.ls
+                dbg.ls 1234
+
+5.2.  dbg.trace
+
+   Control config running trace.
+
+   Name: dbg.trace
+
+   Parameters:
+     * _cmd_ : inner command can be 'on' or 'off' to enable or disable the
+       tracing for one or all processe.
+     * _pid_ : pid for which to list the details. If it missing, then will
+       print for all processes.
+
+   Examples for using with sercmd:
+                dbg.trace on
+                dbg.trace off
+                dbg.trace on 1234
+
+5.3.  dbg.bp
+
+   Control breakpoints and config execution.
+
+   Name: dbg.bp
+
+   Parameters:
+     * _cmd_ : inner command can be 'on' or 'off' to enable or disable the
+       tracing for one or all processe.
+     * _pid_ : pid for which to apply the inner command. If it missing,
+       then will be applied for all processes.
+     * _params_ : extra params specific for each inner command.
+
+   Inner commands:
+     * on - turn on breakpoints. Pid parameter is optional.
+     * off - turn off breakpoints. Pid parameter is optional.
+     * keep - keep breakpoints only for pid given as parameter
+     * release - disable breakpoints for processes that are not waiting at
+       a breakpoint. Pid parameter is optional.
+     * next - run the action under breakpoint and stop at next one (step
+       by step execution). Pid parameter is mandatory.
+     * move - run the action under breakpoint and remove the rest of
+       breakpoints (continue execution without stopping again at next
+       actions). Pid parameter is mandatory.
+     * show - print details about the current breakpoint for pid. Pid
+       parameter is mandatory.
+     * eval - eval a pseudo-variable and print the result in RPC Pid
+       parameter is mandatory.
+     * log - eval a pseudo-variable and print the result in SIP router
+       logs. Pid parameter is mandatory.
+
+   Examples for using with sercmd:
+                dbg.bp off
+                dbg.bp on
+                dbg.bp release
+                dbg.bp on 1234
+                dbg.bp eval 1234 $fu
+                dbg.bp move 1234
+
+6. Usage
+
+   A common usage is to investigate the execution path for a specific SIP
+   message. Just enable cfg running trace, send the message and watch the
+   logs.
+
+   Another typical usage is to do interactive debugging and run
+   step-by-step each line in routing blocks of config file for a
+   particular SIP message.
+
+   You need to connect using sercmd (or other RPC client) to SIP Router.
+   Then you can enable cfg breakpoints and send the SIP message. One
+   process will be in waiting state ('state' field different than 0 when
+   you do dbg.ls). Calling dbg.release will set the other SIP router
+   processes in no-breakpoint mode so they can process other SIP messages
+   without need to interact with them.
+
+   The process blocked at breakpoint is waiting for a command. Use 'dbg.bp
+   next pid' to execute the current action and stop at the next one.
+   'dbg.bp eval pid PV' can be used to retrive the value of PV. Once you
+   are done and want to continue the execution of the config wihtout
+   interaction use 'dbg.bp move pid'.
+
+   Here is an example of session:
+...
+sercmd> dbg.ls
+{
+        entry: 0
+        pid: 10480
+        set: 3
+        state: 0
+        in.pid: 0
+        in.cmd: 0
+}
+{
+        entry: 1
+        pid: 10481
+        set: 3
+        state: 1
+        in.pid: 0
+        in.cmd: 0
+}
+{
+        entry: 2
+        pid: 10482
+        set: 3
+        state: 0
+        in.pid: 0
+        in.cmd: 0
+}
+
+sercmd> dbg.bp show 10481
+at bkp [/etc/kamailio/debugger.cfg:36] a=25
+
+sercmd> dbg.bp next 10481
+exec [/etc/kamailio/debugger.cfg:37] a=54
+
+sercmd> dbg.bp eval 10481 $fu
+$fu : t=str v=sip:[email protected]
+
+sercmd> dbg.bp move 10481
+200 ok
+...
+
+   The cfg running trace looks like:
+...
+ 1(10481) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=36 a=25
+ 1(10481) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=37 a=54
+ 1(10481) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=41 a=17
+ 1(10481) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=41 a=25
+ 1(10481) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=42 a=31
+ 1(10481) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=43 a=25
+ 1(10481) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=44 a=25
+ 1(10481) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=45 a=3
+...
+
+7. To-do
+
+   The 'MUSTS' and ideas of what can be done:
+     * many internal parameters not yet exported to cfg: log level for
+       printed messages, log facility, sleep time while waiting at
+       breakpoint, number of iterations to wait for response from SIP
+       router process ... they are now global variables inside module.
+     * Complete breakpoint setting and usage on specific lines.
+     * Make the output more human friendly - text can be used instead of
+       integer values for some fields such as state. Also, for some
+       actions (e.g., module functions) string name can be printed instead
+       of action internal type code.

modules/debugger/debugger_api.c

@@ -0,0 +1,786 @@
+/**
+ * $Id$
+ *
+ * Copyright (C) 2010 Daniel-Constantin Mierla (asipto.com)
+ *
+ * This file is part of Kamailio, a free SIP server.
+ *
+ * This file is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version
+ *
+ * This file is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+ *
+ */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdlib.h>
+#include <unistd.h>
+
+#include "../../dprint.h"
+#include "../../ut.h"
+#include "../../pt.h"
+#include "../../events.h"
+#include "../../pvar.h"
+#include "../../rpc.h"
+#include "../../rpc_lookup.h"
+#include "../../route_struct.h"
+#include "../../mem/shm_mem.h"
+		       
+#include "debugger_api.h"
+
+#define DBG_CMD_SIZE 256
+
+#define DBG_STATE_INIT	0
+#define DBG_STATE_WAIT	1
+#define DBG_STATE_NEXT	2
+
+#define DBG_CFGTRACE_ON	(1<<0)
+#define DBG_ABKPOINT_ON	(1<<1)
+#define DBG_LBKPOINT_ON	(1<<2)
+
+#define DBG_CMD_NOP		0
+#define DBG_CMD_ERR		1
+#define DBG_CMD_READ	2
+#define DBG_CMD_NEXT	3
+#define DBG_CMD_MOVE	4
+#define DBG_CMD_SHOW	5
+#define DBG_CMD_PVEVAL	6
+#define DBG_CMD_PVLOG	7
+
+/**
+ *
+ */
+int _dbg_cfgtrace = 0;
+
+/**
+ *
+ */
+int _dbg_breakpoint = 0;
+
+/**
+ *
+ */
+int _dbg_cfgtrace_level = L_ERR;
+
+/**
+ *
+ */
+int _dbg_cfgtrace_facility = DEFAULT_FACILITY;
+
+/**
+ *
+ */
+char *_dbg_cfgtrace_prefix = "*** cfgtrace:";
+
+/**
+ *
+ */
+int _dbg_step_usleep = 100000;
+
+/**
+ *
+ */
+int _dbg_step_loops = 200;
+
+/**
+ *
+ */
+typedef struct _dbg_cmd
+{
+	unsigned int pid;
+	unsigned int cmd;
+	char buf[DBG_CMD_SIZE];
+} dbg_cmd_t;
+
+/**
+ *
+ */
+typedef struct _dbg_pid
+{
+	unsigned int pid;
+	unsigned int set;
+	unsigned int state;
+	dbg_cmd_t in;
+	dbg_cmd_t out;
+} dbg_pid_t;
+
+/**
+ *
+ */
+static dbg_pid_t *_dbg_pid_list = NULL;
+
+/**
+ *
+ */
+static int _dbg_pid_no = 0;
+
+/**
+ *
+ */
+typedef struct _dbg_bp
+{
+	str cfile;
+	int cline;
+	int set;
+	struct _dbg_bp *next;
+} dbg_bp_t;
+
+/**
+ *
+ */
+static dbg_bp_t *_dbg_bp_list = NULL;
+
+/**
+ * callback executed for each cfg action
+ */
+int dbg_cfg_trace(void *data)
+{
+	struct action *a;
+	struct sip_msg *msg;
+	int loop;
+	int olen;
+	str pvn;
+	pv_spec_t pvs;
+    pv_value_t val;
+	void **srevp;
+
+	srevp = (void**)data;
+
+	a = (struct action *)srevp[0];
+	msg = (struct sip_msg *)srevp[1];
+
+	if(a==NULL || msg==NULL)
+		return 0;
+
+	if(_dbg_pid_list[process_no].set&DBG_CFGTRACE_ON)
+	{
+		if(is_printable(_dbg_cfgtrace_level))
+		{
+			LOG_(_dbg_cfgtrace_facility, _dbg_cfgtrace_level,
+					_dbg_cfgtrace_prefix,
+					" c=[%s] l=%d a=%d\n",
+					a->cfile, a->cline, a->type
+				);
+		}
+	}
+	if(!(_dbg_pid_list[process_no].set&DBG_ABKPOINT_ON))
+	{
+		/* no breakpoints to be considered */
+		return 0;
+	}
+
+	if(_dbg_pid_list[process_no].state==DBG_STATE_INIT)
+	{
+		LOG(_dbg_cfgtrace_level,
+					"breakpoint hit: p=[%u] c=[%s] l=%d a=%d\n",
+					_dbg_pid_list[process_no].pid,
+					a->cfile, a->cline, a->type
+				);
+		_dbg_pid_list[process_no].in.cmd = DBG_CMD_NOP;
+		_dbg_pid_list[process_no].state = DBG_STATE_WAIT;
+	}
+
+	loop = 1;
+	while(loop)
+	{
+		switch(_dbg_pid_list[process_no].in.cmd)
+		{
+			case DBG_CMD_NOP:
+				sleep_us(_dbg_step_usleep); 
+			break;
+			case DBG_CMD_MOVE:
+				loop = 0;
+				_dbg_pid_list[process_no].state=DBG_STATE_INIT;
+				_dbg_pid_list[process_no].in.cmd = DBG_CMD_NOP;
+				_dbg_pid_list[process_no].in.pid = 0;
+			break;
+			case DBG_CMD_NEXT:
+				loop = 0;
+				if(_dbg_pid_list[process_no].state==DBG_STATE_WAIT)
+					_dbg_pid_list[process_no].state=DBG_STATE_NEXT;
+				_dbg_pid_list[process_no].in.cmd = DBG_CMD_NOP;
+				olen = snprintf(_dbg_pid_list[process_no].out.buf,
+						DBG_CMD_SIZE,
+						"exec [%s:%d] a=%d",
+						a->cfile, a->cline, a->type);
+				if(olen<0)
+				{
+					_dbg_pid_list[process_no].out.cmd = DBG_CMD_ERR;
+					break;
+				}
+				_dbg_pid_list[process_no].out.cmd = DBG_CMD_READ;
+			break;
+			case DBG_CMD_PVEVAL:
+			case DBG_CMD_PVLOG:
+				loop = _dbg_pid_list[process_no].in.cmd;
+				_dbg_pid_list[process_no].in.cmd = DBG_CMD_NOP;
+				pvn.s = _dbg_pid_list[process_no].in.buf;
+				pvn.len = strlen(pvn.s);
+				if(pvn.len<=0)
+				{
+					LM_ERR("no pv to eval\n");
+					break;
+				}
+				LM_DBG("pv to eval: %s\n", pvn.s);
+				if(pv_parse_spec(&pvn, &pvs)<0)
+				{
+					LM_ERR("unable to parse pv [%s]\n", pvn.s);
+					break;
+				}
+				memset(&val, 0, sizeof(pv_value_t));
+				if(pv_get_spec_value(msg, &pvs, &val) != 0)
+				{
+					LM_ERR("unable to get pv value for [%s]\n", pvn.s);
+					break;
+				}
+				if(val.flags&PV_VAL_NULL)
+				{
+					if(loop==DBG_CMD_PVEVAL)
+					{
+						olen = snprintf(_dbg_pid_list[process_no].out.buf,
+							DBG_CMD_SIZE,
+							"%s : t=null",
+							pvn.s);
+						if(olen<0)
+						{
+							_dbg_pid_list[process_no].out.cmd = DBG_CMD_ERR;
+							break;
+						}
+						_dbg_pid_list[process_no].out.cmd = DBG_CMD_READ;
+					} else {
+						LOG(_dbg_cfgtrace_level,
+								"breakpoint eval: %s : t=null\n",
+								pvn.s
+							);
+					}
+					break;
+				}
+				if(val.flags&PV_TYPE_INT)
+				{
+					if(loop==DBG_CMD_PVEVAL)
+					{
+						olen = snprintf(_dbg_pid_list[process_no].out.buf,
+							DBG_CMD_SIZE,
+							"%s : t=int v=%d",
+							pvn.s, val.ri);
+						if(olen<0)
+						{
+							_dbg_pid_list[process_no].out.cmd = DBG_CMD_ERR;
+							break;
+						}
+						_dbg_pid_list[process_no].out.cmd = DBG_CMD_READ;
+					} else {
+						LOG(_dbg_cfgtrace_level,
+								"breakpoint eval: %s : t=int v=%d\n",
+								pvn.s, val.ri
+							);
+					}
+					break;
+				}
+		
+				if(loop==DBG_CMD_PVEVAL)
+				{
+					olen = snprintf(_dbg_pid_list[process_no].out.buf,
+						DBG_CMD_SIZE,
+						"%s : t=str v=%.*s",
+						pvn.s, val.rs.len, val.rs.s);
+					if(olen<0)
+					{
+						_dbg_pid_list[process_no].out.cmd = DBG_CMD_ERR;
+						break;
+					}
+					_dbg_pid_list[process_no].out.cmd = DBG_CMD_READ;
+				} else {
+					LOG(_dbg_cfgtrace_level,
+							"breakpoint eval: %s : t=str v=%.*s\n",
+							pvn.s, val.rs.len, val.rs.s
+						);
+				}
+			break;
+			case DBG_CMD_SHOW:
+				_dbg_pid_list[process_no].in.cmd = DBG_CMD_NOP;
+				_dbg_pid_list[process_no].out.cmd = DBG_CMD_NOP;
+				olen = snprintf(_dbg_pid_list[process_no].out.buf,
+						DBG_CMD_SIZE,
+						"at bkp [%s:%d] a=%d",
+						a->cfile, a->cline, a->type);
+				if(olen<0)
+				{
+					_dbg_pid_list[process_no].out.cmd = DBG_CMD_ERR;
+					break;
+				}
+				_dbg_pid_list[process_no].out.cmd = DBG_CMD_READ;
+			break;
+			default:
+				/* unknown command?!? - exit loop */
+				_dbg_pid_list[process_no].in.cmd = DBG_CMD_NOP;
+				_dbg_pid_list[process_no].state=DBG_STATE_INIT;
+				loop = 0;
+		}
+	}
+	return 0;
+}
+
+/**
+ *
+ */
+int dbg_init_bp_list(void)
+{
+	if(_dbg_bp_list!=NULL)
+		return -1;
+	_dbg_bp_list = (dbg_bp_t*)shm_malloc(sizeof(dbg_bp_t));
+	if(_dbg_bp_list==NULL)
+		return -1;
+	memset(_dbg_bp_list, 0, sizeof(dbg_bp_t));
+	if(_dbg_breakpoint==1)
+		_dbg_bp_list->set |= DBG_ABKPOINT_ON;
+	if(_dbg_cfgtrace==1)
+		_dbg_bp_list->set |= DBG_CFGTRACE_ON;
+	sr_event_register_cb(SREV_CFG_RUN_ACTION, dbg_cfg_trace);
+	return 0;
+}
+
+/**
+ *
+ */
+int dbg_add_breakpoint(struct action *a, int bpon)
+{
+	int len;
+	dbg_bp_t *nbp = NULL;
+	if(_dbg_bp_list==NULL)
+		return -1;
+	len = strlen(a->cfile);
+	len += sizeof(dbg_bp_t) + 1;
+	nbp = (dbg_bp_t*)shm_malloc(len);
+	if(nbp==NULL)
+		return -1;
+	memset(nbp, 0, len);
+	nbp->set |= (bpon)?DBG_ABKPOINT_ON:0;
+	nbp->cline = a->cline;
+	nbp->cfile.s = (char*)nbp + sizeof(dbg_bp_t);
+	strcpy(nbp->cfile.s, a->cfile);
+	nbp->cfile.len = strlen(nbp->cfile.s);
+	nbp->next =	_dbg_bp_list->next;
+	_dbg_bp_list->next = nbp;
+	return 0;
+}
+
+/**
+ *
+ */
+int dbg_init_pid_list(void)
+{
+	_dbg_pid_no = get_max_procs();
+
+	if(_dbg_pid_no<=0)
+		return -1;
+	if(_dbg_pid_list!=NULL)
+		return -1;
+	_dbg_pid_list = (dbg_pid_t*)shm_malloc(_dbg_pid_no*sizeof(dbg_pid_t));
+	if(_dbg_pid_list==NULL)
+		return -1;
+	memset(_dbg_pid_list, 0, _dbg_pid_no*sizeof(dbg_pid_t));
+	return 0;
+}
+
+/**
+ *
+ */
+int dbg_init_mypid(void)
+{
+	if(_dbg_pid_list==NULL)
+		return -1;
+	if(process_no>=_dbg_pid_no)
+		return -1;
+	_dbg_pid_list[process_no].pid = (unsigned int)my_pid();
+	if(_dbg_breakpoint==1)
+		_dbg_pid_list[process_no].set |= DBG_ABKPOINT_ON;
+	if(_dbg_cfgtrace==1)
+		_dbg_pid_list[process_no].set |= DBG_CFGTRACE_ON;
+	return 0;
+}
+
+/**
+ *
+ */
+int dbg_get_pid_index(unsigned int pid)
+{
+	int i;
+	for(i=0; i<_dbg_pid_no; i++)
+	{
+		if(_dbg_pid_list[i].pid == pid)
+			return i;
+	}
+	return -1;
+}
+
+/**
+ *
+ */
+static const char* dbg_rpc_bp_doc[2] = {
+	"Breakpoint command",
+	0
+};
+
+/**
+ *
+ */
+static void  dbg_rpc_bp(rpc_t* rpc, void* ctx)
+{
+	int i;
+	int limit;
+	int lpid;
+	str cmd;
+	str val;
+	int loop;
+
+	if(_dbg_pid_list==NULL)
+	{
+		rpc->fault(ctx, 500, "Not initialized");
+		return;
+	}
+	if (rpc->scan(ctx, "S", &cmd) < 1)
+	{
+		rpc->fault(ctx, 500, "Config breakpoint command missing");
+		return;
+	}
+	i = 0;
+	limit = _dbg_pid_no;
+	if (rpc->scan(ctx, "*d", &lpid) == 1)
+	{
+		i = dbg_get_pid_index((unsigned int)lpid);
+		if(i<0)
+		{
+			rpc->fault(ctx, 500, "No such pid");
+			return;
+		}
+		limit = i + 1;
+	} else {
+		lpid = -1;
+	}
+	if(cmd.len==2 && strncmp(cmd.s, "on", 2)==0)
+	{
+		for(; i<limit; i++)
+		{
+			_dbg_pid_list[i].set |=  DBG_ABKPOINT_ON;
+			_dbg_pid_list[i].state=DBG_STATE_INIT;
+		}
+	} else if(cmd.len==3 && strncmp(cmd.s, "off", 3)==0) {
+		for(; i<limit; i++)
+		{
+			_dbg_pid_list[i].set &=  ~DBG_ABKPOINT_ON;
+			_dbg_pid_list[i].state=DBG_STATE_INIT;
+		}
+	} else if(cmd.len==7 && strncmp(cmd.s, "release", 7)==0) {
+		for(; i<limit; i++)
+		{
+			if(_dbg_pid_list[i].state!=DBG_STATE_WAIT)
+			{
+				_dbg_pid_list[i].set &=  ~DBG_ABKPOINT_ON;
+				_dbg_pid_list[i].state=DBG_STATE_INIT;
+			}
+		}
+	} else if(cmd.len==4 && strncmp(cmd.s, "keep", 4)==0) {
+		if(lpid==-1)
+		{
+			rpc->fault(ctx, 500, "Missing pid parameter");
+			return;
+		}
+		for(loop=0; loop<_dbg_pid_no; loop++)
+		{
+			if(i!=loop)
+			{
+				_dbg_pid_list[loop].set &=  ~DBG_ABKPOINT_ON;
+				if(_dbg_pid_list[loop].state!=DBG_STATE_INIT)
+				{
+					_dbg_pid_list[loop].in.pid = my_pid();
+					_dbg_pid_list[loop].in.cmd = DBG_CMD_MOVE;
+				}
+			}
+		}
+	} else if(cmd.len==4 && strncmp(cmd.s, "move", 4)==0) {
+		if(lpid==-1)
+		{
+			rpc->fault(ctx, 500, "Missing pid parameter");
+			return;
+		}
+		for(; i<limit; i++)
+		{
+			if(_dbg_pid_list[i].state!=DBG_STATE_INIT)
+			{
+				_dbg_pid_list[i].set &=  ~DBG_ABKPOINT_ON;
+				_dbg_pid_list[i].in.pid = my_pid();
+				_dbg_pid_list[i].in.cmd = DBG_CMD_MOVE;
+			}
+		}
+	} else if(cmd.len==4 && strncmp(cmd.s, "next", 4)==0) {
+		if(lpid==-1)
+		{
+			rpc->fault(ctx, 500, "Missing pid parameter");
+			return;
+		}
+		_dbg_pid_list[i].in.pid = my_pid();
+		_dbg_pid_list[i].in.cmd = DBG_CMD_NEXT;
+		for(loop=0; loop<_dbg_step_loops; loop++)
+		{
+			sleep_us(_dbg_step_usleep);
+			if(_dbg_pid_list[i].out.cmd == DBG_CMD_READ)
+			{
+				rpc->add(ctx, "s", _dbg_pid_list[i].out.buf);
+				_dbg_pid_list[i].out.cmd = DBG_CMD_NOP;
+				return;
+			} else if(_dbg_pid_list[i].out.cmd == DBG_CMD_ERR) {
+				rpc->add(ctx, "s", "cmd execution error");
+				_dbg_pid_list[i].out.cmd = DBG_CMD_NOP;
+				return;
+			}
+		}
+		/* nothing to read ... err?!? */
+	} else if(cmd.len==4 && strncmp(cmd.s, "show", 4)==0) {
+		if(lpid==-1)
+		{
+			rpc->fault(ctx, 500, "Missing pid parameter");
+			return;
+		}
+		_dbg_pid_list[i].in.pid = my_pid();
+		_dbg_pid_list[i].in.cmd = DBG_CMD_SHOW;
+		for(loop=0; loop<_dbg_step_loops; loop++)
+		{
+			sleep_us(_dbg_step_usleep);
+			if(_dbg_pid_list[i].out.cmd == DBG_CMD_READ)
+			{
+				rpc->add(ctx, "s", _dbg_pid_list[i].out.buf);
+				_dbg_pid_list[i].out.cmd = DBG_CMD_NOP;
+				return;
+			} else if(_dbg_pid_list[i].out.cmd == DBG_CMD_ERR) {
+				rpc->add(ctx, "s", "cmd execution error");
+				_dbg_pid_list[i].out.cmd = DBG_CMD_NOP;
+				return;
+			}
+		}
+		/* nothing to read ... err?!? */
+	} else if(cmd.len==4 && strncmp(cmd.s, "eval", 4)==0) {
+		if(lpid==-1)
+		{
+			rpc->fault(ctx, 500, "Missing pid parameter");
+			return;
+		}
+		if (rpc->scan(ctx, "S", &val) < 1)
+		{
+			rpc->fault(ctx, 500, "pv param missing");
+			return;
+		}
+		if (val.len < 2 || val.len>=DBG_CMD_SIZE)
+		{
+			rpc->fault(ctx, 500, "invalid pv param");
+			return;
+		}
+		strncpy(_dbg_pid_list[i].in.buf, val.s, val.len);
+		_dbg_pid_list[i].in.buf[val.len] = '\0';
+		_dbg_pid_list[i].in.pid = my_pid();
+		_dbg_pid_list[i].in.cmd = DBG_CMD_PVEVAL;
+		for(loop=0; loop<_dbg_step_loops; loop++)
+		{
+			sleep_us(_dbg_step_usleep);
+			if(_dbg_pid_list[i].out.cmd == DBG_CMD_READ)
+			{
+				rpc->add(ctx, "s", _dbg_pid_list[i].out.buf);
+				_dbg_pid_list[i].out.cmd = DBG_CMD_NOP;
+				return;
+			} else if(_dbg_pid_list[i].out.cmd == DBG_CMD_ERR) {
+				rpc->add(ctx, "s", "cmd execution error");
+				_dbg_pid_list[i].out.cmd = DBG_CMD_NOP;
+				return;
+			}
+		}
+		/* nothing to read ... err?!? */
+	} else if(cmd.len==3 && strncmp(cmd.s, "log", 3)==0) {
+		if(lpid==-1)
+		{
+			rpc->fault(ctx, 500, "Missing pid parameter");
+			return;
+		}
+		if (rpc->scan(ctx, "S", &val) < 1)
+		{
+			rpc->fault(ctx, 500, "pv param missing");
+			return;
+		}
+		if (val.len < 2 || val.len>=DBG_CMD_SIZE)
+		{
+			rpc->fault(ctx, 500, "invalid pv param");
+			return;
+		}
+		strncpy(_dbg_pid_list[i].in.buf, val.s, val.len);
+		_dbg_pid_list[i].in.buf[val.len] = '\0';
+		_dbg_pid_list[i].in.pid = my_pid();
+		_dbg_pid_list[i].in.cmd = DBG_CMD_PVLOG;
+	} else {
+		rpc->fault(ctx, 500, "Unknown inner command");
+	}
+	rpc->add(ctx, "s", "200 ok");
+}
+
+/**
+ *
+ */
+static const char* dbg_rpc_list_doc[2] = {
+	"List debugging process array",
+	0
+};
+
+/**
+ *
+ */
+static void  dbg_rpc_list(rpc_t* rpc, void* ctx)
+{
+	int i;
+	int limit;
+	int lpid;
+	void* th;
+
+	if(_dbg_pid_list==NULL)
+	{
+		rpc->fault(ctx, 500, "Not initialized");
+		return;
+	}
+	i = 0;
+	limit = _dbg_pid_no;
+	if (rpc->scan(ctx, "*d", &lpid) == 1)
+	{
+		i = dbg_get_pid_index((unsigned int)lpid);
+		if(i<0)
+		{
+			rpc->fault(ctx, 500, "No such pid");
+			return;
+		}
+		limit = i + 1;
+	}
+
+	for(; i<limit; i++)
+	{
+		/* add entry node */
+		if (rpc->add(ctx, "{", &th) < 0)
+		{
+			rpc->fault(ctx, 500, "Internal error creating rpc");
+			return;
+		}
+		if(rpc->struct_add(th, "dddddd",
+						"entry",  i,
+						"pid",    _dbg_pid_list[i].pid,
+						"set",    _dbg_pid_list[i].set,
+						"state",  _dbg_pid_list[i].state,
+						"in.pid", _dbg_pid_list[i].in.pid,
+						"in.cmd", _dbg_pid_list[i].in.cmd
+					)<0)
+		{
+			rpc->fault(ctx, 500, "Internal error creating rpc");
+			return;
+		}
+	}
+}
+
+/**
+ *
+ */
+static const char* dbg_rpc_trace_doc[2] = {
+	"Config trace command",
+	0
+};
+
+/**
+ *
+ */
+static void  dbg_rpc_trace(rpc_t* rpc, void* ctx)
+{
+	int i;
+	int limit;
+	int lpid;
+	str cmd;
+
+	if(_dbg_pid_list==NULL)
+	{
+		rpc->fault(ctx, 500, "Not initialized");
+		return;
+	}
+	if (rpc->scan(ctx, "S", &cmd) < 1)
+	{
+		rpc->fault(ctx, 500, "Config trace command missing");
+		return;
+	}
+	i = 0;
+	limit = _dbg_pid_no;
+	if (rpc->scan(ctx, "*d", &lpid) == 1)
+	{
+		i = dbg_get_pid_index((unsigned int)lpid);
+		if(i<0)
+		{
+			rpc->fault(ctx, 500, "No such pid");
+			return;
+		}
+		limit = i + 1;
+	}
+	if(cmd.len!=2 && cmd.len!=3)
+	{
+		rpc->fault(ctx, 500, "Unknown trace command");
+		return;
+	}
+	if(cmd.len==2)
+	{
+		if(strncmp(cmd.s, "on", 2)!=0)
+		{
+			rpc->fault(ctx, 500, "Unknown trace command");
+			return;
+		}
+	} else {
+		if(strncmp(cmd.s, "off", 3)!=0)
+		{
+			rpc->fault(ctx, 500, "Unknown trace command");
+			return;
+		}
+	}
+	for(; i<limit; i++)
+	{
+		if(cmd.len==2)
+		{
+			_dbg_pid_list[i].set |=  DBG_CFGTRACE_ON;
+		} else {
+			_dbg_pid_list[i].set &=  ~DBG_CFGTRACE_ON;
+		}
+	}
+	rpc->add(ctx, "s", "200 ok");
+}
+
+
+/**
+ *
+ */
+rpc_export_t dbg_rpc[] = {
+	{"dbg.bp",    dbg_rpc_bp,        dbg_rpc_bp_doc,       0},
+	{"dbg.ls",    dbg_rpc_list,      dbg_rpc_list_doc,     0},
+	{"dbg.trace", dbg_rpc_trace,     dbg_rpc_trace_doc,    0},
+	{0, 0, 0, 0}
+};
+
+/**
+ *
+ */
+int dbg_init_rpc(void)
+{
+	if (rpc_register_array(dbg_rpc)!=0)
+	{
+		LM_ERR("failed to register RPC commands\n");
+		return -1;
+	}
+	return 0;
+}
+

modules/debugger/debugger_api.h

@@ -0,0 +1,37 @@
+/**
+ * $Id$
+ *
+ * Copyright (C) 2010 Daniel-Constantin Mierla (asipto.com)
+ *
+ * This file is part of Kamailio, a free SIP server.
+ *
+ * This file is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version
+ *
+ * This file is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+ *
+ */
+
+		       
+#ifndef _DEBUGGER_API_H_
+#define _DEBUGGER_API_H_
+
+#include "../../route_struct.h"
+
+int dbg_add_breakpoint(struct action *a, int bpon);
+int dbg_init_bp_list(void);
+int dbg_init_pid_list(void);
+int dbg_init_mypid(void);
+int dbg_init_rpc(void);
+
+#endif
+

modules/debugger/debugger_mod.c

@@ -0,0 +1,140 @@
+/**
+ * $Id$
+ *
+ * Copyright (C) 2010 Daniel-Constantin Mierla (asipto.com)
+ *
+ * This file is part of Kamailio, a free SIP server.
+ *
+ * This file is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version
+ *
+ *
+ * This file is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+ *
+ */
+
+#include <stdio.h>
+#include <unistd.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include "../../sr_module.h"
+#include "../../dprint.h"
+#include "../../ut.h"
+#include "../../mod_fix.h"
+#include "../../parser/parse_param.h"
+#include "../../shm_init.h"
+
+#include "debugger_api.h"
+
+MODULE_VERSION
+
+static int  mod_init(void);
+static int child_init(int rank);
+static void mod_destroy(void);
+
+static int w_dbg_breakpoint(struct sip_msg* msg, char* point, char* str2);
+static int fixup_dbg_breakpoint(void** param, int param_no);
+
+extern int _dbg_cfgtrace;
+extern int _dbg_breakpoint;
+
+static cmd_export_t cmds[]={
+	{"dbg_breakpoint", (cmd_function)w_dbg_breakpoint, 1,
+		fixup_dbg_breakpoint, 0, ANY_ROUTE},
+	{0, 0, 0, 0, 0, 0}
+};
+
+static param_export_t params[]={
+	{"cfgtrace",          INT_PARAM, &_dbg_cfgtrace},
+	{"breakpoint",        INT_PARAM, &_dbg_breakpoint},
+	{0, 0, 0}
+};
+
+struct module_exports exports = {
+	"debugger",
+	DEFAULT_DLFLAGS, /* dlopen flags */
+	cmds,
+	params,
+	0,
+	0,              /* exported MI functions */
+	0,              /* exported pseudo-variables */
+	0,              /* extra processes */
+	mod_init,       /* module initialization function */
+	0,              /* response function */
+	mod_destroy,    /* destroy function */
+	child_init      /* per child init function */
+};
+
+
+/**
+ * init module function
+ */
+static int mod_init(void)
+{
+	if(dbg_init_rpc()!=0)
+	{
+		LM_ERR("failed to register RPC commands\n");
+		return -1;
+	}
+
+	return dbg_init_bp_list();
+}
+
+static int child_init(int rank)
+{
+	LM_DBG("rank is (%d)\n", rank);
+	if (rank==PROC_INIT)
+		return dbg_init_pid_list();
+	return dbg_init_mypid();
+}
+/**
+ * destroy module function
+ */
+static void mod_destroy(void)
+{
+}
+
+static int w_dbg_breakpoint(struct sip_msg* msg, char* point, char* str2)
+{
+	return 1;
+}
+
+/**
+ * get the pointer to action structure
+ */
+static struct action *dbg_fixup_get_action(void **param, int param_no)
+{
+	struct action *ac, ac2;
+	action_u_t *au, au2;
+	/* param points to au->u.string, get pointer to au */
+	au = (void*) ((char *)param - ((char *)&au2.u.string-(char *)&au2));
+	au = au - 1 - param_no;
+	ac = (void*) ((char *)au - ((char *)&ac2.val-(char *)&ac2));
+	return ac;
+}
+
+
+static int fixup_dbg_breakpoint(void** param, int param_no)
+{
+	struct action *a;
+	char *p;
+
+	if(param_no!=1)
+		return -1;
+	a = dbg_fixup_get_action(param, param_no);
+	p = (char*)(*param);
+
+    return dbg_add_breakpoint(a, (*p=='0')?0:1);
+}
+
+

modules/debugger/doc/Makefile

@@ -0,0 +1,4 @@
+docs = debugger.xml
+
+docbook_dir = ../../../docbook
+include $(docbook_dir)/Makefile.module

modules/debugger/doc/debugger.xml

@@ -0,0 +1,36 @@
+<?xml version="1.0" encoding='ISO-8859-1'?>
+<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
+"http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd" [
+
+<!-- Include general documentation entities -->
+<!ENTITY % docentities SYSTEM "../../../docbook/entities.xml">
+%docentities;
+
+]>
+
+<book xmlns:xi="http://www.w3.org/2001/XInclude">
+    <bookinfo>
+	<title>debugger Module</title>
+	<productname class="trade">&kamailioname;</productname>
+	<authorgroup>
+	    <author>
+		<firstname>Daniel-Constantin</firstname>
+		<surname>Mierla</surname>
+		<affiliation><orgname>asipto.com</orgname></affiliation>
+	    </author>
+	    <editor>
+		<firstname>Daniel-Constantin</firstname>
+		<surname>Mierla</surname>
+		    <email>[email protected]</email>
+	    </editor>
+	</authorgroup>
+	<copyright>
+	    <year>2010</year>
+	    <holder>Daniel-Constantin Mierla (asipto.com)</holder>
+	</copyright>
+    </bookinfo>
+    <toc></toc>
+    
+	<xi:include href="debugger_admin.xml"/>
+    
+</book>

modules/debugger/doc/debugger_admin.xml

@@ -0,0 +1,400 @@
+<?xml version="1.0" encoding='ISO-8859-1'?>
+<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
+"http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd" [
+
+<!-- Include general documentation entities -->
+<!ENTITY % docentities SYSTEM "../../../docbook/entities.xml">
+%docentities;
+
+]>
+
+<!-- Module User's Guide -->
+
+<chapter>
+    
+    <title>&adminguide;</title>
+    
+    <section>
+	<title>Overview</title>
+	<para>
+		This module provides an interactive config file debugger. It can print
+		the trace of config execution for a SIP message to log and set
+		breakpoints on every config action, allowing to execute step by step the
+		config.
+	</para>
+	<para>
+		Debugging can be done from local or remote host via RPC interface (e.g.,
+		XMLRPC, sercmd).
+	</para>
+	<para>
+		The framework to set breakpoints on specific actions and config lines
+		is not exported to RPC yet, right now each action has an breakpoint. The
+		breakpoint can be enabled/disabled at runtime. Also the config running
+		trace can be enabled/disabled at runtime.
+	</para>
+	<para>
+		When the SIP router process is stopped at a breakpoint, you can
+		investigate the values of any pseudo-varaibles. Note that some of
+		pseudo-variables may produce memory leak, that is planned to fix in the
+		future (here falls pseudo-variables with dynamic name such as htable,
+		sqlops). References to SIP message, avps, headers, script and shared
+		variables are safe.
+	</para>
+    </section>
+    <section>
+	<title>Dependencies</title>
+	<section>
+	    <title>&kamailio; Modules</title>
+	    <para>
+		The following modules must be loaded before this module:
+	    	<itemizedlist>
+		    <listitem>
+			<para>
+			    <emphasis>none</emphasis>.
+			</para>
+		    </listitem>
+	    	</itemizedlist>
+	    </para>
+	</section>
+	<section>
+	    <title>External Libraries or Applications</title>
+	    <para>
+		The following libraries or applications must be installed before running
+		&kamailio; with this module loaded:
+	    	<itemizedlist>
+		    <listitem>
+			<para>
+			    <emphasis>None</emphasis>.
+			</para>
+		    </listitem>
+	    	</itemizedlist>
+	    </para>
+	</section>
+    </section>
+    <section>
+	<title>Exported Parameters</title>
+	<section>
+	    <title><varname>cfgtrace</varname> (int)</title>
+	    <para>
+			Control whether config running trace is enabled or disabled
+			at startup.
+	    </para>
+	    <para>
+		<emphasis>
+		    Default value is <quote>0</quote> (disabled).
+		</emphasis>
+	    </para>
+	    <example>
+		<title>Set <varname>cfgtrace</varname> parameter</title>
+		<programlisting format="linespecific">
+...
+modparam("debugger", "cfgtrace", 1)
+...
+</programlisting>
+	    </example>
+	</section>
+
+	<section>
+	    <title><varname>breakpoint</varname> (int)</title>
+	    <para>
+			Control whether every line (for now) breakpoint is enabled
+			or disabled at startup.
+	    </para>
+	    <para>
+		<emphasis>
+		    Default value is <quote>0</quote> (disabled).
+		</emphasis>
+	    </para>
+	    <example>
+		<title>Set <varname>breakpoint</varname> parameter</title>
+		<programlisting format="linespecific">
+...
+modparam("debugger", "breakpoint", 1)
+...
+</programlisting>
+	    </example>
+	</section>
+
+	</section>
+	
+    <section>
+	<title>Exported Functions</title>
+ 	<section>
+	    <title>
+		<function moreinfo="none">dbg_breakpoint(mode)</function>
+	    </title>
+	    <para>
+			Achor a breakpoint at that line in config. Mode specifies
+			whether the breakpoint is enabled (1) or disabled (0) at startup.
+	    </para>
+		<para>
+			Note that this version does not export this anchors to RPC for
+			interactive debugging (temporary disabled).
+	    </para>
+		<example>
+		<title><function>mt_match</function> usage</title>
+		<programlisting format="linespecific">
+...
+if($si=="10.0.0.10")
+	dbg_breakpoint("1");
+...
+</programlisting>
+	    </example>
+	</section>
+	
+    </section>
+	
+	<section>
+		<title>Exported RPC Functions</title>
+
+	<section>
+		<title>
+		<function moreinfo="none">dbg.ls</function>
+		</title>
+		<para>
+			List SIP router processes with info related to interactive
+			debugging.
+		</para>
+		<para>
+		Name: <emphasis>dbg.list</emphasis>
+		</para>
+		<para>Parameters:</para>
+		<itemizedlist>
+			<listitem><para>_pid_ : pid for which to list the details. If it
+			missing, then will print for all processes.</para></listitem>	  
+		</itemizedlist>
+		<para>
+		Examples for using with sercmd:
+		</para>
+        <programlisting  format="linespecific">
+		dbg.ls
+		dbg.ls 1234
+		</programlisting>
+    </section>
+
+	<section>
+		<title>
+		<function moreinfo="none">dbg.trace</function>
+		</title>
+		<para>
+			Control config running trace.
+		</para>
+		<para>
+		Name: <emphasis>dbg.trace</emphasis>
+		</para>
+		<para>Parameters:</para>
+		<itemizedlist>
+			<listitem><para>_cmd_ : inner command can be 'on' or 'off' to
+				enable or disable the tracing for one or all processe.</para>
+		</listitem>
+			<listitem><para>_pid_ : pid for which to list the details. If it
+					missing, then will print for all processes.</para>
+		</listitem>
+		</itemizedlist>
+		<para>
+		Examples for using with sercmd:
+		</para>
+        <programlisting  format="linespecific">
+		dbg.trace on
+		dbg.trace off
+		dbg.trace on 1234
+		</programlisting>
+    </section>
+
+	<section>
+		<title>
+		<function moreinfo="none">dbg.bp</function>
+		</title>
+		<para>
+			Control breakpoints and config execution.
+		</para>
+		<para>
+		Name: <emphasis>dbg.bp</emphasis>
+		</para>
+		<para>Parameters:</para>
+		<itemizedlist>
+			<listitem><para>_cmd_ : inner command can be 'on' or 'off' to
+				enable or disable the tracing for one or all processe.</para>
+		</listitem>
+		<listitem><para>_pid_ : pid for which to apply the inner command.
+				If it missing, then will be applied for all processes.</para>
+		</listitem>
+		<listitem><para>_params_ : extra params specific for each inner
+				command.</para>
+		</listitem>
+		</itemizedlist>
+		<para>Inner commands:</para>
+		<itemizedlist>
+		<listitem>
+			<para>on - turn on breakpoints. Pid parameter is optional.
+			</para>
+		</listitem>
+		<listitem>
+			<para>off - turn off breakpoints. Pid parameter is optional.
+			</para>
+		</listitem>
+		<listitem>
+			<para>keep - keep breakpoints only for pid given as parameter
+			</para>
+		</listitem>
+		<listitem>
+			<para>release - disable breakpoints for processes that are
+			not waiting at a breakpoint. Pid parameter is optional.</para>
+		</listitem>
+		<listitem>
+			<para>next - run the action under breakpoint and stop at next one
+			(step by step execution). Pid parameter is mandatory.
+			</para>
+		</listitem>
+		<listitem>
+			<para>move - run the action under breakpoint and remove the rest
+			of breakpoints (continue execution without stopping again at next
+			actions). Pid parameter is mandatory.</para>
+		</listitem>
+		<listitem>
+			<para>show - print details about the current breakpoint for pid.
+			Pid parameter is mandatory.</para>
+		</listitem>
+		<listitem>
+			<para>eval - eval a pseudo-variable and print the result in RPC
+			Pid parameter is mandatory.</para>
+		</listitem>
+		<listitem>
+			<para>log - eval a pseudo-variable and print the result in SIP
+			router logs. Pid parameter is mandatory.</para>
+		</listitem>
+		</itemizedlist>
+		<para>
+		Examples for using with sercmd:
+		</para>
+        <programlisting  format="linespecific">
+		dbg.bp off
+		dbg.bp on
+		dbg.bp release
+		dbg.bp on 1234
+		dbg.bp eval 1234 $fu
+		dbg.bp move 1234
+		</programlisting>
+    </section>
+
+    </section>
+	<section>
+		<title>Usage</title>
+		<para>
+		A common usage is to investigate the execution path for a specific
+		SIP message. Just enable cfg running trace, send the message and
+		watch the logs.
+		</para>
+		<para>
+		Another typical usage is to do interactive debugging and run
+		step-by-step each line in routing blocks of config file for a
+		particular SIP message.
+		</para>
+		<para>
+		You need to connect using sercmd (or other RPC client) to SIP Router.
+		Then you can enable cfg breakpoints and send the SIP message. One
+		process will be in waiting state ('state' field different than 0 when
+		you do dbg.ls). Calling dbg.release will set the other SIP router
+		processes in no-breakpoint mode so they can process other SIP messages
+		without need to interact with them.
+		</para>
+		<para>
+		The process blocked at breakpoint is waiting for a command. Use
+		'dbg.bp next pid' to execute the current action and stop at the next
+		one. 'dbg.bp eval pid PV' can be used to retrive the value of PV. Once
+		you are done and want to continue the execution of the config wihtout
+		interaction use 'dbg.bp move pid'.
+		</para>
+		<para>
+		Here is an example of session:
+		</para>
+<programlisting  format="linespecific">
+...
+sercmd> dbg.ls
+{
+	entry: 0
+	pid: 10480
+	set: 3
+	state: 0
+	in.pid: 0
+	in.cmd: 0
+}
+{
+	entry: 1
+	pid: 10481
+	set: 3
+	state: 1
+	in.pid: 0
+	in.cmd: 0
+}
+{
+	entry: 2
+	pid: 10482
+	set: 3
+	state: 0
+	in.pid: 0
+	in.cmd: 0
+}
+
+sercmd> dbg.bp show 10481
+at bkp [/etc/kamailio/debugger.cfg:36] a=25
+
+sercmd> dbg.bp next 10481
+exec [/etc/kamailio/debugger.cfg:37] a=54
+
+sercmd> dbg.bp eval 10481 $fu
+$fu : t=str v=sip:[email protected]
+
+sercmd> dbg.bp move 10481
+200 ok
+...
+</programlisting>
+		<para>
+		The cfg running trace looks like:
+		</para>
+<programlisting  format="linespecific">
+...
+ 1(10481) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=36 a=25
+ 1(10481) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=37 a=54
+ 1(10481) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=41 a=17
+ 1(10481) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=41 a=25
+ 1(10481) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=42 a=31
+ 1(10481) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=43 a=25
+ 1(10481) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=44 a=25
+ 1(10481) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=45 a=3
+...
+</programlisting>
+    </section>
+
+	<section>
+		<title>To-do</title>
+		<para>
+			The 'MUSTS' and ideas of what can be done:
+		</para>
+		<itemizedlist>
+		<listitem>
+			<para>many internal parameters not yet exported to cfg:
+			log level for printed messages, log facility, sleep time while
+			waiting at breakpoint, number of iterations to wait for response
+			from SIP router process ... they are now global variables inside
+			module.
+		</para>
+		</listitem>
+		<listitem>
+		<para>
+			Complete breakpoint setting and usage on specific lines.
+		</para>
+		</listitem>
+		<listitem>
+		<para>
+			Make the output more human friendly - text can be used instead of
+			integer values for some fields such as state. Also, for some
+			actions (e.g., module functions) string name can be printed instead
+			of action internal type code.
+		</para>
+		</listitem>
+		</itemizedlist>
+    </section>
+
+</chapter>
+