I was looking through some old code I had written and found this small and cryptic method for converting a number of seconds into a string containing of this format “hh:mm:ss” and I will describe how it works. First, the code:
def seconds_to_time(secs)
hms = [3600, 60].inject([secs]) {|x, y| x += x.pop.divmod(y)}
return hms.map {|y| "%02d" % y}.join(":")
end
This method leverages the behavior of the Enum.inject method and the Numeric.divmod method to creatively convert the seconds into an array containing hours, minutes and seconds. It then utilizes the Enum.map method to join these values into a recognizable time string. Here is some sample output:
puts seconds_to_time(63628) 17:40:28
The inject method is passed in a starting value and on each iteration the return value of the previous iteration is passed again. This parameter is commonly called an accumulator. The accumulator shows up in the provided block as the first block variable called x. The initial value of this accumulator is an array containing the original seconds. The second parameter to the block (y) is the value from the original array that is the focus of the current iteration. In this example y starts out as 3600.
In the first iteration this value is pop’d out of this array and the divmod method is called passing in the first value from the array that inject is being called on. This value is 3600 and represents the number of seconds in one hour. The return value of the divmod method is an array with the integer part as the first position and the remainder as the second position [17, 2428]. Since the original seconds were pop’d from x, x will now be empty and += will cause x to fill up with the 2 values that were returned from divmod. At the end of the first iteration of the inject method the accumulator variable will contain [17, 2428].
In iteration 2 y is set to 60 which is the number of seconds in a minute while x is set to the value of the accumulator from the previous iteration [17, 2428]. 2428 will be pop’d from this array and divmod will be called on this value with a parameter of y. The result of this call is the array [40, 28] which represents the number of minutes in 2428 seconds and the remainder. This array is then appended to x causing it to become [17, 40, 28] which is the end of our second iteration.
Since this is our last iteration the return value for the inject call will be the array [17, 40, 28]. This array is then map’d into an array of strings formatted as 2 digit integers. This step is needed to pad single digit positions with a leading zero. The final step is to call the join method to glue these pieces together with a colon.
I’ve decided to start a new Series of blog posts focused on the analysis of existing code that all developers depend on, and in some cases, take for granted. I know this exercise will uncover useful programming practices and “tricks” to help me become a better programmer. My hope is that by sharing this information in this blog, other developers will be helped as well. My goal is to keep the length of these posts small enough that they are easy to digest in a few minutes of reading but still provide useful insights to the reader.
Installment 1 – ActionView::Base.word_wrap
The word_wrap method within ActionView is a very useful method for reformatting text to a specific column width (default is 80 characters). A high level description of how it works is that it breaks up the original text into chunks based on any existing EOL characters found in the text. For each chunk or paragraph, additional EOL characters are inserted at appropriate locations so as to limit the sentence width to the desired number of columns without cutting any words in half. The resulting chunks are then glued back together with additional EOL characters.
A simple example using a line_width of 8 characters is as follows:
word_wrap('Level Five Solutions', :line_width => 10)
# => Level Five\nSolutions
Lets look at how this is implemented within the TextHelpers.rb class bundled with ActionView:
def word_wrap(text, *args)
options = args.extract_options!
unless args.blank?
options[:line_width] = args[0] || 80
end
options.reverse_merge!(:line_width => 80)
text.split("\n").collect do |line|
line.length > options[:line_width] ?
line.gsub(
/(.{1,#{options[:line_width]}})(\s+|$)/,
"\\1\n").strip :
line
end * "\n"
end
I had to do some creative “wrapping” of the code to make it fit our blog layout. How ironic!
The relevant parts of this method begin on line 8 where it splits the original text on any existing “\n” characters and then operates on each “line” within the body of the block past in to the collect method. A simple ternary operator separates the lines that are longer than the line_width from those that are not. The lines that are longer are “embedded” with EOL characters through a call to gsub with a very creative regular expression:
/(.{1,#{options[:line_width]}})(\s+|$)/
The first pair of parenthesis identify this as a capturing regular expression. The period says to match any character up to the number identified by the following curly brace expression. The curly brace section is greedy in that it tries to match up to the line_width first and then starts falling back from there. It decides to fall back based on the trailing look-ahead expression which requies that the next character is a space or the end of the line. After the regular expression matcher extracts a particular subset of the text it is replaced by the gsub method with the following expression:
"\\1\n"
This expression builds a new string containing the substring captured by the regular expression (\\1) and a trailing EOL character (\n). The “g” in gsub means global so that the substitution is applied across the entire string. This means that the regular expression is repeated as many times as it takes to fully modify the original text to include EOL characters at all the “appropriate” locations based on the line_width provided.
The final tail end of this method “multiplies” the result of the collect statement by a string containing the EOL character (\n). The result of the collect method is an array containing each “line” from the original text that was broken by the call to split. The multiplication operator (*) on the Array class has special logic when working with strings. It concatenates each element of the array with the string provided as the second parameter and additionally concatenates the entire list into one long string. It does not add a final trailing occurrence of the EOL character. Here’s an example to clarify this point:
['1','2','3'] * 'a' 1a2a3
Notice that the ‘a’ character only shows up between elements ’1′ and ’2′. It does not show up at the end.
It looks like I’ve gotten to the end of this method, and therefore, the end of this blog post. Some of the lessons I’m taking away from this post include additional regular expression knowledge and a better understanding of how the multiplication operator of the array class behaves when operating on strings.
I have not yet decided what my next post will cover (or when I’ll get it done) but I hope to continue the series by diving into other useful code looking for interesting techniques and cool “tricks”. If anyone reading this series has an idea for some code that would make a good topic, just post it to the comments and I’ll try to cover it in a future post.
Breaking up a list into random groups of size n
April 28, 2010
class Array
# Useful for breaking down one large array into
# smaller randomly organized arrays of size 'number'
# How it's used: list.random_groups_of(4) =>
# returns [[...],[...]...]
# When the last left over group is "small" and
# no_small_groups was set to true then add these
# items to the other groups.
def random_groups_of(number, no_small_groups = false)
randomized = sort_by{ rand }
groups = []
if (number && number > 0)
randomized.each_slice(number) do |slice|
groups << slice
end
if (groups.size > 1 &&
groups[-1].size < (number/2.0).round &&
no_small_groups)
groups.pop.each_with_index do |e, i|
groups[i % groups.size - 1] << e # round robin
end
end
end
groups
end
end
Here is an example of how to use the random_groups_of method
names = ['one',
'two',
'three',
'four',
'five',
'six',
'seven',
'eight',
'nine',
'ten',
'eleven',
'twelve',
'thirteen',
'fourteen',
'fifteen',
'sixteen',
'seventeen',
'eighteen',
'nineteen',
'twenty',
'twentyone',
'twentytwo']
names.random_groups_of(4, true).
each_with_index do |g, i|
puts "#{i+1}. #{g.join(", ")}"
end
The output from this example:
1. thirteen, two, eleven, eighteen, seventeen 2. twenty, three, twentyone, eight, one 3. nine, fifteen, six, ten 4. seven, four, twelve, fourteen 5. sixteen, nineteen, twentytwo, five
Here is the output when the :no_small_groups option is left off or set to false:
1. fifteen, twenty, nineteen, eight 2. two, one, ten, eighteen 3. fourteen, sixteen, three, four 4. thirteen, seven, twentytwo, nine 5. twentyone, twelve, six, seventeen 6. eleven, five
Using Rails ActiveRecord to incrementally update a database when a long running update statement simply won’t work.
March 30, 2010
If you’ve ever tried to use sql to perform various operations on database tables with millions of records you’ll know first hand how frustrating it can be waiting hours and even days for a single update statement to return. If you should lose network connectivity or if the server should crash in the middle of one of these long statements, the database nicely rolls back the transaction that it’s been working on for the past 10 hours. Also there is no way to track the progress of the operation in order to predict how long it will take to execute. Using Rails ActiveRecord and a small amount of ruby code (in the form of a rake task), these same operations can be performed incrementally, with the added ability to stop, continue and monitor progress. The database updates may take longer to run but that is a fair tradeoff given the above benefits. The ruby code will quietly jug away updating records little by little until they are all done.
Here is some sample code demonstrating this technique:
...
task 'zip9toRes' => :environment do
desc "populate res_count in zip9"
sql = ActiveRecord::Base.connection();
#start at a specific db id
start_id = 701407
zip7s = Zip7.find(:all,
:select => 'id, zip',
:order => 'id',
:conditions => ['id > ?', start_id])
zip7s.each do |z|
sql <<SQL
update zip9 z set res_count =
(select count(*) from residential
where zip9 = z.zip)
where zip7 = '#{z.zip}'
SQL
sql.update sql
show_progress();
if @@stop
puts "stopped at #{z.id}"
break;
end
end
end
...
The above task is looping through 1.1 million zip7 records and for each one it’s telling the zip9 table to populate a res_count column for all zip9 rows in that zip7. There are roughly 60 million zip9 records and for each one we’d like to know how many homes there are. The residential table contains 124 million address records that are counted to populate the res_count column.
The show_progress method is a neat way to give some indication that the process is still running:
...
def show_progress()
wheel = ["|", "/", "-", "\\"]
moveleft = "\033[D"
print wheel[@@progress_counter % 4], moveleft
@@progress_counter += 1
if @@progress_counter % 100 == 0
print @@progress_counter, ".."
end
$stdout.flush()
end
...
The @@stop variable is initially set to false and a couple of traps are setup to trigger this variable to true which causes the long running task to gracefully stop.
...
@@stop = false
trap("INT") {
stop()
}
trap("TERM") {
stop()
}
def stop()
@@stop = true
end
...
Here is a brief video demonstrating the progress indicator:
Collaboration Is Key, Even If You’re The DOD: New Initiatives And Technology Recommendations
January 27, 2010
As we’ve discussed in previous posts, the communities impacted by the Strategic Foresight and Agility Initiative (SFA-I) are complex and diverse. Within the Intelligence and Defense Research and Engineering Communities, there are hundreds of software systems performing a myriad of functions that serve dozens of organizations and hundreds of thousands of users. How can the Intelligence and Defense Research and Engineering Communities begin to make sense of this complex landscape? What are the common business needs and technologies that can bring clarity to the larger enterprise?
Based on information we gathered during stakeholder interviews, respondents were most interested in tools that simplified their work process, provided information relevant to their area of expertise and made it easier to collaborate with colleagues. As the subsequent technical assessments made clear, a wide-range of software systems have been deployed to serve these needs, with varying degrees of success. That being the case, our technical recommendations focused on areas of specific technology investment that would stimulate collaboration and information sharing.
The currency of the Intelligence and Defense Research and Engineering Communities is information. More specifically, information that is structured and contextual. The vision, then, was to generate recommendations to develop and implement technologies that enable the transformation of data into knowledge that serves the enterprise as a whole.
Those recommendations were:
Invest in Semantic Data Tagging
There is considerably more utility to data, especially unstructured data, if there is information about the data. This information could include things like classification level, authorship, citations, timestamping and content abstracts.
Connect People
Implement a set of interoperable, distributed address book systems that contain more than simple demographic data. These systems would include information on formal educational experience, work experience, area of expertise and current assignments.
Connect Data Events and Derived Data
As our previous post illustrated, IT systems are typically silos of information and capabilities. It is likely that there is considerable duplication of data across systems. The goal is to implement a mechanism by which software systems can publish information and data and software systems can subscribe to the data that is useful to their applications.
Enhance Authentication and Authorization Systems
Many of the IT systems have authentication and authorization mechanisms that are appropriate for a siloed system, but are inappropriate for sharing information across systems. We identified that current authentication and authorization methods need to be improved to meet the varied needs of the communities.
Decompose Intelligence Reports into Constituent Parts
Currently, most Intelligence Reports are monolithic. That is, they were created for a specific audience at a specific classification level to answer a specific question. There would be greater reuse of information in an Intelligence Report if it were created with relevant metadata about the content. The additional information would allow for the same report to be produced for different classification levels. It would also allow for different parts, such as background information, to be available outside the context of the original report for potential reuse.
Manage and Audit Information Across Networks
It is desirable to have information flow between the different physical networks with the proper network guards in place to ensure security. While a single interconnected network may not be technically feasible or politically viable, appropriately tagged data could move from a less to a more secure network (and vice versa) through guard software.
Based on these recommendations we identified specific functional integrations of technology solutions to achieve the target state goals of creating sustainable processes, generating improved data and developing more sophisticated tools to increase collaboration. The functional integrations included development work at both the foundational and application level to occur over a four year period. It’s our belief that implementing these solutions will create the ideal technological foundation for increasing collaboration, information sharing and information dissemination.
The benefits of these solutions can not be fully realized, however, without implementing cultural change across the enterprise. As we’ll discuss in the next and final post in this series, our cultural recommendations provided a way forward in actualizing change within the Intelligence and Defense Research and Engineering Communities.
In this post we’ll discuss some of the issues our team identified during the technical assessment phase of our project with DDR&E. As we’ve touched on in previous posts, our technical team (made up of developers, system architects and technical PMs) began by conducting an inventory of over one hundred systems (applications and sites) currently being utilized by the Intelligence and Defense Research and Engineering Communities. These were systems that our client felt aided in collaboration, information sharing and information dissemination in some capacity within the enterprise. From those one hundred systems, we identified 16 whose functionality directly related to the business priorities of creating sustainable processes, generating improved data and developing more sophisticated tools to increase collaboration. These 16 systems became the focus of our technical assessment work.
For each system, we conducted interviews with technical SMEs, including system owners, project managers and developers. Each system underwent a thorough analysis of functionality and, when possible, system documentation was examined and unclassified versions of tools were audited. Working with agencies within the national security apparatus presents a unique challenge in that much of the information and many of the tools themselves are classified. Even when tools aren’t classified, the stewards of those systems provide information only on a need to know basis. In order to conduct the most thorough assessments possible it was imperative that we gained the trust of internal personnel in order to extract the necessary information.
Getting that information took time, some interpersonal relationship ingenuity and literally hundreds of hours of phone calls, but in the end we had a clear view of the current state of online collaboration and information sharing within the communities. What we found, in terms of current initiatives, development methodologies and accepted practices was of great value for our client. For the purposes of this post, however, we’d like to discuss some more general findings that also apply to other organizations and enterprises.
Not surprisingly, we found existing technical solutions that aided in collaboration and information sharing at the department level (i.e. within an agency), but that were not designed to scale to meet the needs of the enterprise (i.e. the larger DOD and U.S. Intelligence Community). These sorts of narrowly defined solutions are also common within Fortune 500 companies. There are myriad cultural and technical drivers that lead to incomplete solutions, but a key one is the fact that much strategy, planning and development occurs in a vacuum. By this we mean that solutions are conceptualized and developed to serve a specific department-level business need (“I need an online SME Rolodex in 3 months!”) without a clear understanding of the bigger picture. In large organizations this leads to duplication of effort and replication of functionality, resulting in near-identical solutions. Both of which cost time, money and intellectual capital.
The resulting solution might meet the immediate department-level need, but rarely will it serve the long-term, enterprise-wide strategy. That’s because the solution was not engineered, from a code standpoint, to scale to the enterprises’ needs. As we’ve seen, these enterprise needs are often clearly articulated from a business standpoint, but that does not translate into technical requirements. Anyone who has spent time in large organizations knows that much is lost in translation between business requirements and technical specifications. To alleviate that, technical standards and development practices must be clearly articulated and continually managed.
A related cultural driver that leads to incomplete enterprise solutions is the issue of ownership. Turf wars within and between departments can lead to the politicization of IT decisions. The issues of who owns what, which solution should be promoted to scale and how that work gets funded internally all require objective decision-making processes. The Department of Defense is seeking to answer those questions by creating oversight bodies to review and promote IT initiatives in which the enterprise view is central. These people “get” the big picture and are tasked with making recommendations for IT expenditures based solely on those larger enterprise needs.
In times of shrinking budgets and downward trending revenues the focus on enterprise-wide solutions becomes critical. In more economically rosy times it’s just good planning. Scalable solutions not only decrease long-term development and maintenance costs, they also benefit the user community. These benefits include simplifying user experience, lowering the cost of user buy-in and increasing shared enterprise knowledge of specific tool-sets. While we don’t advocate a one-size-fits-all approach to all problems, when a shared solution serves the business objective that’s often just what is needed. And that’s what the DOD is trending toward in terms of enterprise solutions. If the larger mission of the enterprise is being served then that will naturally line up with department-level objectives. If not, then those objectives are clearly out of whack.
One of our primary goals at Level Five is to build long-term relationships with our clients. We seek out these relationships because they are fundamental to our ability to think strategically and work effectively. Developing long-term relationships also enables us to see the big picture, that is identify opportunities, issues and strategic objectives for our clients that might otherwise have gone unnoticed.
The key to developing long-term relationships is to know our client’s culture inside and out. This goes beyond knowing what drives their economic engine, although that understanding is obviously essential. It involves understanding individuals’ professional objectives, identifying and articulating team work methods and seamlessly integrating ourselves into the language and social norms of the enterprise. In short, we strive to make ourselves indispensable to our clients by understanding and becoming an integral part to their culture.
The Strategic Foresight and Agility Initiative (SFA-I) conducted with the Department of Defense Research and Engineering (DDR&E) Plans and Programs office presented a unique opportunity to learn a new culture and test the efficacy of our approach. The Department of Defense has one of the most well-defined cultures of any organization in the world. There are norms, taboos and a language that, while not immediately apparent to the outside observer, are widely understood and accepted within the DOD. Our first objective was to quickly learn those norms and ingrain ourselves within the culture.
We began by reading hundreds of pages of strategic plans, policy directives, summations of objectives, development plans and technical system assessments. All of these related directly to past and on-going initiatives to stimulate sustainable collaboration and information sharing within the Intelligence and Defense Research and Engineering Communities. We were fortunate to be working directly with a team of individuals in the Plans and Programs office who could guide our “education” and ensure we developed a formidable knowledge base. The outcome of this reading was an understanding of what the community thought about, the methodologies they employed and why they came to the strategic conclusions they did.
Once we were satisfied with our foundational understanding we began conducting a series of in-person interviews with a cross-section of representatives from the Intelligence and Defense Research and Engineering Communities. We interviewed analysts, researchers, system developers, program managers, agency liaisons, deputy directors, program directors and Chief Technology Officers. In all, we conducted over 60 interviews during a two month period.
The purpose of these stakeholder interviews was to develop a baseline understanding of the communities in order to determine current work methods, roles and responsibilities within the community and the current state of collaboration. This process also allowed us to identify systems with the greatest potential to scale across the enterprise and whose functionality most closely aligned with SFA-I objectives. In addition, we utilized interview sessions to ask community members for input on the ideal target state relating to collaboration, information sharing and information dissemination.
The official deliverable that came out of these sessions was business and functional requirements for future phases of SFA-I. Of equal importance, however, was the cultural understanding and personal relationships we developed. By going directly to “the source” (i.e. the people that have an intimate knowledge of and history with the community) we gained a great deal of knowledge about the DOD’s culture and began the process of building long-term relationships. It also afforded us the opportunity to visit the Pentagon and CIA Headquarters, which was pretty heady stuff.
Without a clear understanding of the DOD’s culture and how individuals interact with that culture we would not have been able to make effective business process or technology recommendations. We are a technology company that recognizes the centrality of users (itself a problematic term) in any technological implementation. We recognized early on that the development of systems does not occur in a vacuum. A sophisticated understanding of a client’s culture is key to successful development initiatives. As we’ll discuss in the next post, our understanding of the DOD’s culture provided the foundation for successfully assessing the DOD’s current software portfolio and drove our subsequent target state technology recommendations.
Collaboration Is Key, Even If You’re The DOD: Define The Vision
January 12, 2010
In our first post about Level Five’s project with the Department of Defense, specifically the Research and Engineering (DDR&E) Plans and Programs office, we touched on what our client hoped to achieve during the project’s first phase. In this installment we’ll touch on specific objectives and discuss how a clearly defined vision is the only way to ensure a project’s long-term success.
We’ve all been involved in projects, both personally and professionally, that failed because they lacked a clearly defined vision. Without a vision there is no desired end state and no specific strategies and tactics to achieve that end state. Without a vision all you have is a poor-conceptualized dream. And don’t get us wrong, dreams are the best, but they’re only useful in the “real world” if they can be realized.
The DOD doesn’t do dreams. They function on directives, mandates, strategic planning and actionable intelligence. This isn’t to say they don’t have loads of visionary thinkers at their disposal, but those thinkers visionary thoughts become clearly defined strategies with supporting tactics. The visionary thought that drove the Strategic Foresight and Agility Initiative (SFA-I) was “Hey, wouldn’t it be cool if we could collaborate, share information and disseminate that information in real-time to answer any question regarding scientific research and technological development?” Yes, it would.
Toward that end, DDR&E began by defining their goals for SFA-I. In short, those goals where:
- Contribute to strategic momentum and communication around the centrality of collaboration in the national security community
- Design and establish mechanisms to address knowledge gaps
- Increase collaboration and feedback within and between the Intelligence and Defense Research and Engineering Communities
- Collect, integrate, prioritize and regularly update information requests for technical intelligence
- Formalize linkages between efforts and strategies in order to reduce portfolio risk
In addition, DDR&E hoped to integrate the Intelligence Community into the planning process in order to more quickly identify and respond to foreign science and technology programs that have the potential to diminish U.S. capabilities.
What we see here are very specific and actionable goals around which tactics can be developed. The first tactic was to break out the SFA-I project team into smaller working groups. The working group that we engaged with became known as the Collaboration and Infrastructure Issue Team or CIIT (everything has an acronym in the federal government), which was made up of a cross-section of members of the Intelligence and Defense Research and Engineering Communities, including analysts, researchers, program managers, program directors and agency liaisons.
The CIIT had its own mandate within the larger initiative. That mandate was to increase DOD and Intelligence Community collaboration around science and technology through the creation of sustainable processes, improved data and more sophisticated tools. Again, we have a clearly defined mandate around which recommendations can be formulated and acted upon.
This sort of planning may be old hat to the DOD, but it was a breath of fresh air to our team. We came into the project knowing, with absolute clarity, what our client hoped to achieve and why. The strategies were open for debate, but the vision was not. This allowed us to focus on the task at hand and bring our expertise to bear on a clearly defined problem set. As we’ll discuss in the next post, the first step towards solving those problems was to deeply immerse ourselves in our client’s culture and become a sounding board on which they could voice their recommendations, concerns and past lessons learned.
The Level Five Approach: Systems Thinking
July 16, 2009
As part of the current project we are working on with the Department of Defense, I had the opportunity to meet with Dr. Ruth David last week. Dr. David was the former CIA Deputy Director for Science and Technology from 1995 to 1998. She is now the president and chief executive officer of ANSER, an independent, not-for-profit, public service research institution that provides research and analytic support on national and trans-national issues.
Towards the end of the meeting, we started talking about tools and systems that are used to break down overwhelmingly complex problems. Causal loop diagrams and systemigrams were a couple of examples of the possible approaches when trying to understand the relationships between objects within a process.
What I found interesting was the fact that these approaches were very technical in nature and ultimately were a systems approach to thinking about process and communication issues. This type of systems thinking is something that the ANSER promotes through their ASysT Institute. This approach also resonates with Level Five because it is the same way we approach problems.
At our core, Level Five is a of group problem-solvers. We joke that we almost like problems more than we like clients. Creating an elegant solution to a problem is extremely rewarding for us, and we solve these problems through the use of technology. Level Five was founded in the software development field. Specifically, we develop software that has extremely high levels of rigor and quality to its functionality.
Since we focus so much on technology and a systems approach to development, it’s only natural that we apply that same approach to other types of problems — like communication or business processes.
Every one of our clients has communication and process problems. Often they attempt to solve these problems by implementing tools that promise to fix the issues, but they fail to understand the underlying problem. If you are experiencing pain and frustration trying to resolve communication and business process problems, it’s because you are not looking at the bigger picture.
At the end of the day, it’s most often a people problem. It is rarely a technology or tool problem. Understanding the end-to-end scope is absolutely necessary to fix the issue. What we have found at Level Five is that the level of detail, scope and understanding that is inherent in a systems approach to problem-solving provides a more effective path to success.
Phil
kurtlane
Greg Stickley
