
Web Crawl Data: Click here to see larger image
Note from Beth: As visiting scholar at the Packard Foundation, I'm connecting with other people who are studying and learning about how networks work. A lot of the ideas resonate with using online social networks and social media effectively for nonprofits, especially in the larger frame of movement building. In October, I had the opportunity to meet Steve Waddell whose research focuses on Global Action Networks.
One of the tools for better understanding networks are visual diagnostics and mapping techniques. This another area of Steve's interest and expertise. He co-authored a paper called "Visual Diagnostics and Mapping for Scaling Change" and we had an opportunity to discuss it. He agreed to write a four-part primer on a visual diagnostics, mapping, and social networking analysis primer and how nonprofits might use these tools for social change. If you missed it, you can read Part 1: Systems Mapping for Nonprofits.
Understanding who is connected to whom can strengthen your strategy. But the connections might be so be numerous, or the formal org charts might be so misleading, that you can’t “see” what is happening easily. Maybe your question is about inter-organizational connections in your issue system (eg: agriculture, finance, housing) – it may be local, regional or global. Or, if you work in a large networked organization, maybe you want to understand inter-personal ties to understand work processes better. There are a couple of tools that can help you out.
Web Crawl Mapping
The “quick and dirty” one is web crawls. The internet is structured around sites that have unique URL addresses. And most sites have (hyper) links to other sites that you click on to take you to other sites or pages. These are inserted because they have more detailed information with regards to a topic (including, of course, ads), because the host wants to connect people to allies or colleagues, or because they may be foes on an issue.
These connections between unique URLs provide the basis for mapping relationships by doing a “web crawl”. A software program can draw the relationships between organizations’ web links, to give a description of the virtual network of the organization. It shows links between URLs that can collectively be called the global commercial finance public issue arena. These are the organizations to which global commercial finance institutions link.
The crawl identified 282 URLs; only the top 100 are shown in the map. Separate data that is generated lists the number of links to each URL and the direction – whether they go to a URL or come from it – which is important to understand who thinks whom is worth attention. Another list summarizes the number of links. This map itself illustrates such structurally important things as groups (cliques), bridges between groups, and which organizations are best connected.
Of course as with any methodology, this presents a limited picture. It depends upon organizations having web-sites; in global finance, it is pretty safe to assume that influential organizations will have one. Web crawls are particularly useful when used with other network analysis methodologies because they help identify organizations in a field for further investigation.
Social Network Analysis Mapping
For example, the organization lists developed from crawls can be an initial list of who to include in classic social network analysis (SNA) that can show links between individuals, parts of organizations, an issue system, or subset of it.
This map was developed when GRI was thinking about establishing a South African GRI network. Surveys were conducted to identify organizations and their relationships with two particular characteristics that drew from GRI’s core strategy: organizations that were involved with triple bottom line analysis and development (social-economic-environmental impact), and organizations that engaged in multi-stakeholder processes. The map illustrates the following groups of organizations with hubs and bridging organizations connecting them; the enviro group is importantly not connected. This descriptive analysis suggests the following strategy:
1) Put the environment on the back burner for the moment, since economic-social issues are more dominant;
2) Consult with the bridging organizations as key informants and perhaps engage them in initial convening to form a GRI South Africa network; and
3) When creating a leadership group or board, make sure you engage the nodes of each group.
The descriptive analysis therefore supports a strategy of firmly building on the current local orientation, social structure and capacity to develop a GRI approach. Rather than GRI being a foreign entity coming in through a particular stakeholder group as is often the way a organization enters a new region – raising great suspicions among other groups – GRI can begin with a much more comprehensive strategy that weaves together current social relationships in a new way.

Steve Waddell: As Principal of Networking Action, Steve Waddell applies his 20+ years of experience in multi-stakeholder network development to address complex issues regionally and globally.
Thanks for the clear lay person explanation of these tools.
Posted by: stephanie mcauliffe | December 09, 2009 at 01:27 PM
Hello Steve,
Could you provide examples of good web-crawling software?
Thanks!
Posted by: Joelle | December 22, 2009 at 02:44 PM
The web-crawl software that I use is Issue Crawler (issuecrawler.net) from the Govcom.org Foundation, directed by Prof. Richard Rogers, Chair in New Media & Digital Culture, University of Amsterdam. You can access this for free, although this puts you in a queue that may take many days. Or you can pay a modest fee. Richard is very helpful.
Posted by: Steve Waddell | December 24, 2009 at 06:35 AM