When to carry out performance testing?

Usually, there are different approaches to performance testing. Very often -too often- performance testing is only considered after some critical disruption occurred in production, causing costly service losses. Effort, time and resources are spent to find the cause and never let it happen again. In this scenario, performance testing does not follow a regular, planned schedule, but it’s driven by application or infrastructural failures. It is rather simple to implement such a process, but there’s a limit: testing is only involved in emergency situations, with the aim to reduce service downtime (MTTR). Despite the relatively low effort required by this approach, waiting for the next disaster could be a risky bet.

At the opposite side of the spectrum, performance testing is included as an integral part in the Application Lifecycle Management (ALM). All new applications and releases are tested, assuring no performance regression is occurring from a previous baseline or expected service levels. A previous article from our colleague David Furnari showed the importance of developing a maturity model aimed to improve a customer’s awareness on performance management topics. The focus is to evolve from a simple issue-resolution approach to the development of a comprehensive performance optimization process. 

Who to carry out performance testing? A factory approach

For the above reason, a dedicated team of performance engineers is required, implementing a centralized performance management model. In two words, a Performance Factory.  In this context, a performance factory approach shows a number of advantages:

• Efficiency. Test team engage/response time is reduced, and communication between different areas is smoother (faster analysis and results sharing); for each application a “test repository” is kept, acting as a performance baseline history future releases will be compared to.

• Economies of scale. Tasks are divided among test team, and each member focuses on his duties and skills; testing process and instruments are standardized.
• Service catalog. Activity requirements are implemented more efficiently: a shared approach eases test definition phase.

• Specific responsibilities. Highly specialized tasks disengage test team from all preliminary activities, e.g. software releases or environment set-up; roles are clearly defined: testing is performed on time and quality of the results is guaranteed.

Moreover, a Performance Factory supports all other service lifecycle phases, spanning over the whole performance management process:

• Application Delivery provides not only performance testing & tuning, but also availability testing, failover testing and infrastructure sizing.

• Service Management supports Availability & Performance monitoring and Incident & Problem Management; cooperates with Service Level Management.

• Continuous Service Improvement facilitates the exchange of application and infrastructural knowledge; supports Capacity Planning; application changes driven by performance activity could lead to new developments in Functional Testing.

Again, this means a strong cooperation between all different teams has to be achieved. We strongly recommend that everyone involved in the performance analysis is working together throughout the whole performance process, from the initial stage to the final results discussion and acceptance.

Who carrying out performance testing? A non-factory approach

However, working as a factory may simply not be enough. To improve the quality of the results and broaden the range of the analysis that can be performed, a different approach could be required.

In an effort to increase the value and the quality of performance testing, the focus on Service Offering concepts represents an evolution from the test factory model. A Service Offering consists in the categorization of an activity into dimensions that represent its key features.

In the performance testing area, typical dimensions are:

• Test frequency
• Technology involved
• Test objectives

In a straight factory approach, it could be difficult to correctly identify these dimensions. Perhaps, in a context where the test team is committed to run dozens (if not hundreds) of tests in the short term, there is virtually no time to approach the activity in such an articulated manner. Or, more simply, the test factory usually deals with the very same kind (dimensions) of tests: constant frequency, same technology, standardized objectives.

But what if we are NOT part of a permanent performance team and need to deal with different test activities? In a world where context and technology evolve fast, performance engineers have to face constantly novel challenges on their way to implement performance. Any of us constantly stepping into different customer accounts each one with its specific issues and needs, knows very well that never two performance test activities are exactly the same.

In this scenario, the factory model would show its design flaws. Basically, a factory-organized performance team acts like an assembly line, where each step to performance optimization is executed in a standardized sequence and few exceptions are allowed. This model perfectly suits an environment where the performance process is well on track, strongly integrated in the service lifecycle: test scheduling is predictable, the team is perfectly familiar with the technology involved and test objectives reflect standardized requirements.

But in case we are asked to deal with new contexts, addressing new performance issues, usually with very short notice and where time is a strong constraint, we need to adapt (and increase) our skills and knowledge. The Performance Factory model has to become less rigid in favor of a more flexible approach. Its ultimate goal is to broaden the range of the service offering and increase the capability of our testing portfolio.

In this case, to develop a valuable service offering approach a number of factors are involved. Each performance testing activity requires specific considerations and even customizations that are hard (if not impossible) to be taken into account in a performance factory organization. A completely different testing approach is developed. The performance activity process is driven by the specific requirements that particular context needs. Performance experts should operate in a less ‘standardized’ manner, digging into each testing aspect with much more detail and customization.

Planning phase and design phase have to comply with new detailed objectives and/or specific agreements (Service Level Agreements, SLAs) to be met during the execution of the test. Defined goals have to comply with the specific environment, with its own features, capacity and purpose: a run in a testing environment aimed at pinpointing performance issues and suggesting tuning tweaks is very different from a run in production performed to assess the maximum throughput available for end users. According to these design differences, every time specific monitoring on resources and infrastructure may be required.

Execution phase is directly connected with technology involved, and has to be conducted accordingly, while the analysis phase is strictly related with test scope and objectives, shaping structure and content of final outputs, results and documents.

In this scenario the expertise of the performance team is definitely stronger and more reliable if compared to the experience developed in a Performance Factory; the ability to change is also much more developed. The focus is not only on technical skills, but also on organizational and even relational skills, since rapidly evolving contexts require test experts to expand their knowledge, while performing their tasks and when involving other people and teams.

From theory to practice

Of course, implementing this “Swiss-army knife” performance approach does not come for free. Like any piece of craftsmanship, this kind of performance testing requires additional effort and work, along with a relevant time to be spent in the initial setup process. Unlike a factory, it is difficult to perform economies and apply some sort of recurring standardized process template or  activity framework while dealing with such different scenarios.

Moreover, an additional risk hiding behind this kind of approach is that could lead to small and individual test activities, confined into very specific contexts and performed on an irregular basis (when not just one-offs), not bringing enough visibility to the performance process, regardless all the effort spent. This means the performance maturity and awareness, whose importance has been previously pointed out, could be compromised, or at least would not improve, leading to a potential lack in recognizing the value of performance testing.

Conclusions

A general clear-cut rule to determine which approach most fits every testing activity does not exist. Each approach, factory and non-factory, has its own advantages and disadvantages, strengths and weaknesses. Depending on the context, test objectives, requirements, constraints involved and expected results, it is imperative to determine how to organize the performance activity and how to achieve the maximum efficiency and the best results: dealing with the rigid but systematic framework of a Performance Factory, or tackling a performance issue in a custom environment, sparing no effort but finally achieving the most detailed analysis and valuable results. In a word, an all-round performance engineer should balance their key factors, taking advantage of each approach while adapting to each specific testing scenario.

BMC just released a new “Network Capacity Management with BMC Capacity Optimization” datasheet that provides more details on how the integration of BCO with Moviri’s connectors for Entuity and HP NNMi enable customers to:

• Get better insight on network capacity and simplify analysis and reporting on network resources
• Forecast utilization of key network resources and identify bottlenecks and potential saturations
• Correlate network resource utilization with Business KPIs to extrapolate expected resource utilization with respect to demand growth
• Automatically generate and distribute by-exception reports

You can download the BMC datasheet here.

Few weeks ago Moviri attended the Strata Conference 2013 in Santa Clara, California, one of the most relevant and well-attended exhibitions on Big Data. Today’s analytics and data visualization technologies are undergoing a phase of vibrant development, with fast changing paradigms. The Strata Conference represented a great opportunity to catch up on key market trends and real-world applications and experiences from business use cases implemented by Silicon Valley’s top-tier players in the field.

Here’s a few highlights on the most interesting trends and topics we encountered during the conference.

Batch Processing

The advent of Hadoop and his distributed filesystem HDFS, Hadoop Distributed File System, is getting a lot of attention. From our current perspective, this is the most promising technology; several companies (like Cloudera and MapR) are building enterprise solutions on it, and most of R&D efforts are focused on reducing latency and elapsed time to launch queries over MapReduce. HDFS, pillar of Hadoop, appears to be an established early standard and reliable stack currently widely supported by third party Big Data software and monitoring platforms.

Stream Analysis

Despite Hadoop/MapReduce dramatically reduced processing time for huge datasets, it’s not at the moment an appropriate technology suitable to process real time data while they are collected. In scenarios like these, Apache Hive (for Hadoop), Cloudera’s Impala (for Hadoop) and Shark (for Spark) are very promising technologies.

Interactive Analysis

The keyword here is in-memory analysis. Platfora, Impala and Tableau are all leveraging this concept and have shown how they are implementing their solutions allowing to easily perform data discovery.

Future of RDBMS

It’s a while we have been hearing about NoSQL capabilities and its advantage in being unchained from strict structured data, however it’s clear that the market is asking for SQL-compliant DBMS Database Management Systems, especially in technology replacement scenarios, but also to facilitate adoption of new technologies. Big entreprise customers have, up until today, suffered more from the impossibility to scale databases horizontally than destructuring data. Several column-oriented DBMSs are already allowing this, and a lot of community-based projects are working on this direction. Tim O’Brien‘s keynote on this topic was illuminating as it analyzed current market trends and Google’s approach (follow this link to access the keynote page with downloadable presentation, see the comments).

Vendors

During the exhibition, big players focused their attention to Big Data hardware (e.g. Intel) and software (e.g. HP Vertica, Splunk, Cloudera, MapR, Platfora) vendors passing on in different ways the message that they support top Big Data technologies and have platforms processing petabytes of data with little upfront set-up effort.

At Moviri, we help our customers make sense of the plethora of emerging Big Data technologies. We look forward to your comments and to hear your impressions on these trends and the impact they are having on information technology and the business it supports and enables.

Photo by akihitok1973/Flickr (CC)

It is already time to train for next year edition!

Moviri has a history of partnering with leading and innovative technology vendors such as Splunk. Since 2009 Moviri has been working with Splunk as Splunk Powered Reseller and delivered successful projects for its customers in Financial Services, Telco and Energy industries.

On March 11^th, Moviri took part in the second Splunk EMEA Partner Kickoff held in Lisbon, Portugal.  After an amazing year of successes with our partner Splunk, this was the perfect spot where Moviri has been awarded “Partner of the Year 2012 Southern Europe”, second year in a row, but also received a special mention for a relevant deal in the Region.

Tom Schodorf (Splunk), Roberto Serra (Moviri), Paolo Prigione (Moviri) and Andrew Morris (Splunk)

Tom Schodorf, Splunk SVP, mentioned the Unicredit project one of the most important implementations in our Region. This deal is currently one of our most interesting assignments on the Splunk platform, delivering for the Client advanced Service Desk functionalities and proactive monitoring capabilities.

We want to thank the whole team for the great job done during 2012 and we are sure that 2013 will be brilliant as well.

We are going to try and win this award three times in a row next year. Godspeed!

Do not miss this great opportunity to meet Moviri at the Career Day Event at the Politecnico of Milan!

The event will take place on May 14^th 2013 from 10 am to 5 pm at the Campus Bovisa-Broggi – ED. BL27, in Via Lambruschini 4, 20156 Milan.

We are hiring right now!
You can meet our team at the Moviri Stand nr. 28, 1^st floor, to discover and learn more about our job opportunities, our national and international projects and about life at Moviri. You can discover through the experience of our colleagues what it is really like to take on exciting technology challenges in a great working environment!Moviri is a fast growing, innovative consulting firm with solid IT engineering background and with key competencies in IT Performance Optimization and Capacity Management. On our projects, we provide high level service to our clients at national and international level, and we are expanding our business abroad, especially in the US.

See you very soon at the Career Day!

New in release 5.0:

• Moviri Integration for BCO – HP BSM Extracts end-user monitoring data, both synthetic and real users’, from HP Business Service Management (former Business Availability Center, BAC) to correlate customers service experience and transaction volumes with IT infrastructure utilization.
• Moviri Integration for BCO – HP SiteScope — Extracts performance and configuration data from HP monitoring tool to BMC Capacity Optimization by leveraging SiteScope “external database logging” functionality

In addition to the new supported HP products, Moviri releases today Network View. Moviri Integration for BCO – Network View makes network data, imported by other Moviri Integrations, zero-time-actionable in terms of capacity assessment and provisioning forecasts. It enables BCO customers to slice and dice their network by multiple dimensions and drill down to the most sensitive spots. Moviri ships Network View for Entuity and HP NNMi.

Moviri and BMC add the new integrations to the existing portfolio of officially supported integrations:

• Moviri Integration for BCO – Entuity
• Moviri Integration for BCO – HP uCMDB
• Moviri Integration for BCO – IBM Tivoli Monitoring
• Moviri Integration for BCO – HP NNMi
• Moviri Integration for BCO – HP Reporter
• Moviri Integration for BCO – Microsoft SCOM

Moviri Integrations for BCO are connectors provided as one-click-deploy packages, a ready to use solution sold on BMC MarketZone and ensured with the same level of support as the BMC native connectors; they increase coverage of the Capacity Management process enabling additional integrations in BCO.

Moviri is getting ready to take part to Splunk Live! EMEA in in Rome on March 5th and in Milan on March 7th.

Join us and the rest of the Italian Splunk community to discover how thousands of customers worldwide are using Splunk to better manage their datacenters, achieve higher availability, investigate security incidents in record time, and meet compliance requirements at lower costs.

Participants will get an hands-on experience with the product and learn directly from enterprise customers like Sky, Unicredit, A-tono and Banca Popolare di Sondrio as how their job has changed since they started using Splunk.

Moviri is a long time Splunk Partner (and was awarded the Partner of the Year Award for South EMEA in 2011) and will be present with its renowed experts to clear any doubt or provide you with bright, new ideas to manage your IT.

More information about the events and the registration form can be found at http://live.splunk.com/italia

Image by Interop Events/Flickr

Moviri continues to grow its product line as a key BMC partner for its capacity management solution, BMC Capacity Optimization, with the introduction of the new Moviri Integration for BCO – HP NNMi. Moviri connectors enable BMC Capacity Optimization to import IT infrastructure utilization and performance metrics from widely industry adopted monitoring and reporting platforms.

Moviri and BMC adds the new integration to the existing portfolio of officially supported integrations:

• Moviri Integration for BCO – HP Reporter — Extracts performance and configuration data from HP Reporter to BMC Capacity Optimization by leveraging the HP Reporter database.
• Moviri Integration for BCO – HP uCMDB — Imports CI configuration attributes and CI relationships from HP Universal CMDB (uCMDB) into BMC Capacity Optimization.
• Moviri Integration for BCO – IBM Tivoli Monitoring — Extracts performance and configuration data from IBM Tivoli Monitoring to BMC Capacity Optimization by leveraging the IBM Tivoli Data Warehouse.
• Moviri Integration for BCO – Entuity — Extracts network performance and configuration data from Entuity to BMC Capacity Optimization.
• Moviri Integration for BCO – Microsoft SCOM — Extracts performance and configuration data from Microsoft SCOM to BMC Capacity Optimization by leveraging the SCOM data warehouse.

Moviri Integrations for BCO are connectors provided as one-click-deploy packages, a ready to use solution sold on BMC MarketZone and ensured with the same level of support as the BMC native connectors; they increase coverage of the Capacity Management process enabling additional integrations in BCO.

On January 22, 2013 Moviri will showcase its BMC Capacity Optimization (BCO) Integrations at the BMC Midwest Solutions Forum in Chicago.

Moviri Integrations for BMC Capacity Optimization are one-click-deployment additional components for BMC Capacity Optimization, a ready to use solution increasing the coverage of the Capacity Management process. Moviri just added integrations with HP Universal CMDB to the products already in the portfolio for HP Reporter, IBM Tivoli Monitoring, Entuity and Microsoft SCOM.

At the  BMC Midwest Solutions Forum you will have direct access to the latest BMC’s and MarketZone partners‘ development in ITSM, Cloud, Application Management and Mobility solutions. Representative’s from BMC’s office of the CTO, as well as product management representatives for live demos and technical discussion of the latest BMC solutions. Moviri will be on hand throughout the event at the partner pavilion to showcase our BCO solutions.

The event will take place at the Chicago Oak Brook Hills Marriott Resort in Oak Brook (Map and Directions) starting at 8 am. A full agenda and breakout session abstracts are available here.

Learn about Moviri’s BCO Integrations in Chicago this December! The event is free to attend, but space is limited. Register to secure your place today. To RSVP for this free event go to the BMC Midwest Solutions Forum registration page.

Image by JKleeman via Flickr