AIX > Administrator > Systems Management

ProbeVue Pinpoints Issues for Developers, Support and Performance Analysis


With advancements in technology and growing user expectations from systems, development teams are continuously advancing their code. This trend not only increases the code base but also its complexity. However, the industry still prefers to keep designs of these systems as simple as they can. Accordingly, developers must spend more time finding simple, innovative solutions than debugging problems.

From their perspective, users expect products to be bug free, but if by chance one exists, they expect it to be resolved quickly. Moreover, users have a greater focus on system performance above all else. At this point, one can clearly see the value of a common tool that can meet the needs of different teams and give them their level of freedom to collect the data.

Currently, AIX provides several tools for developers, support teams and performance analysis teams—including dbx, tprof, gprof, trace, etc. These provide options for users to meet their requirements. For example, dbx is a very powerful tool giving complete control of the application to the user. However, for effective debugging, the application needs to be compiled using -g option, and the application must stop during debugging. Likewise, tprof and gprof are heavyweight performance tools that capture the performance information of all the functions in an application, thereby having a small impact in all parts of the application. And trace is another tool that lets the code log its data for users to consume, but this needs to be implanted during coding, hence, data collected cannot be changed during run-time. It also dumps a substantial amount of data for users to analyze.

To overcome these obstacles and provide a flexible, simplified and more controlled data-collection tool, IBM has created ProbeVue, which is available from AIX V6.1 and beyond. With ProbeVue, users can dynamically enable/disable the probe (trace/debug) points, and collect the desired results such as arguments, register values, etc. Only the functions under study are altered, while the rest can continue execution without being affected. Additionally, because the functions are limited to trace, the data generated for analysis is limited as well. ProbeVue also provides certain data structure like LIST, associative array, range data type to help users do some filtering and arranging of data. This keeps the user from being burdened with unneeded data because it’s already been filtered.

Lets look further and see how ProbeVue can be useful for people involved in development, support and performance testing. Developers are most interested in information about the arguments being passed to a function, certain process-related data structures, etc. This script implants a probe in the process, which will get triggered after every n milliseconds of interval. Whenever triggered, this probe collects data such as: tid, scheduling policy, priority, etc., and gives it to the user for analysis.

@@interval:$1:clock:$2
{
        printf("------------------------------------------------------\n");
        printf("                Thread Related :\n");
        printf("------------------------------------------------------\n");
        printf("                  Thread Id : %lu\n",__tid);
        printf("          Scheduling Policy : %lu\n",__curthread->policy);
        printf("                   Priority : %lu\n",__curthread->pri);
        printf("                CPU Usage : %lu\n",__curthread->cpuusage);
        printf("       Bounded to Processor : %lu\n",__curthread->cpuid);
        printf("  Signal Blocked On Thread : 0x%x\n",__curthread->sigmask);
        printf("------------------------------------------------------\n");
        printf("                Process Related :\n");
        printf("------------------------------------------------------\n");
        printf("               Process ID : %lu\n",__pid);
        printf("        Parent Process ID : %lu\n",__ppid);
        printf("         Process Group ID : %lu\n",__pgid);
        printf("                 Priority : %lu\n",__curproc->pri);
        printf("    Process creation time : %lu\n",__curproc->forktime);
        printf("******************************************************\n");
        printf("                        Stack Trace\n");
        stktrace(PRINT_SYMBOLS|GET_USER_TRACE, -1);
        printf("******************************************************\n");
}

Users can modify this script to plant a probe at the entry of any function in the application and collect more data about the process. For further information, see developerWorks ProbeVue wiki page.

Sometimes during debugging, users are interested in collecting the stack trace pattern to see what are the most frequent code paths followed to reach a function. This can easily be found with the help of associative arrays. ProbeVue additionally lets users arrange the data in an associative array the way best suitable for them. For example, they can sort the data based on key or value in descending or ascending order. They can even choose to see top n entries instead of all the entries in an associative array. The following script explains it further:

@@BEGIN
{
	set_aso_print_options(5, SORT_BY_VALUE|SORT_TYPE_DESCEND);
	/* setting options as print only 5 entries after sorting based on value and ordering them in descending order */
}
@@uft:$__CPID:*:foo3:entry
{
	stacktrace_count[stktrace(-1)]++;  /* increment the count of every system call called */
}
@@syscall:$__CPID:exit:entry /* To exit from ProbeVue when the application exits */
{
	print(stacktrace_count);
	exit();
}

Here, the output contains key as stack trace and value as the number of times a stack trace is seen.

Key					Value
.__start+0x68			  10
.main+0xc
.foo1+0x20
.foo2+0x20
.foo3+0x0
.__start+0x68		 	   5
.main+0x10
.foo4+0x20
.foo3+0x0

Prateek Goel started his career with IBM and has more than 5 years of experience. He has been contributing toward development of reliability, serviceability and availability features of AIX.


comments powered by Disqus

Advertisement

Advertisement

2019 Solutions Edition

A Comprehensive Online Buyer's Guide to Solutions, Services and Education.

AIX > ADMINISTRATOR > SYSTEMS MANAGEMENT

How to Download Fixes

ADMINSTRATOR > SYSTEMS MANAGEMENT

Understand your options for 12X PCIe I/O drawers

clmgr: A Technical Reference

PowerHA SystemMirror 7.1 introduces a robust CLI utility

IBM Systems Magazine Subscribe Box Read Now Link Subscribe Now Link iPad App Google Play Store
IBMi News Sign Up Today! Past News Letters