Petabyte the Hand that Feeds

Several interesting quotes can be found in Kevin Davies' "Dagdigian's Trends in IT Highlight Bio-IT World Expo" at bio-itworld.com, as he describes Chris Dagdigian's review of the Bio-IT World Expo:

"Petabyte-capable storage is trivial to acquire in 2012".

"The [science] is changing faster than we can refresh datacenters and research IT infrastructure." 

Dagdigian was excited about new NGS compression techniques, such as CRAM. "We need order-of-magnitude changes in compression,” he said. “Be glad you are not Broad/Sanger/BGI/NCBI." 

"...storage offerings from vendors such as DDN, Panasas, Isilon and BlueArc all run Unix on standard architectures." 

"I would not deploy a private cloud solution … that does not have Amazon API compatibility." 

Dagdigian said he was bullish about the Siri voice control ... [h]e predicted growing popularity for ... pNFS as well as smart storage systems from Drobo and DataDirect.  

The local private Pittsburgh HPC company I follow in the news (as I know the CTO, Garth Gibson) is Panasas.  While terabyte-scaling-to-petabyte storage has long been the domain of supercomputing initiatives, businesses and governments globally are quickly snapping up storage systems to support research across many disciplines: biosciences, energy, finance, manufacturing, etc. While "apps" currently control home lighting systems,  drone aircraft and smartphone-based gaming, why not data-gathering tools such as space-based telescopes, deepsea submersibles, etc?  Computing giants EMC, IBM, Hitachi, HP, and Dell all have their HPC components (and both Isilon and BlueArc are now divisions of two of these), but for start-up potential, watch the bleeding edge of this industry and see what shakes out - despite the 100 million smartphones/tablets out there, HPC is the stealth user of both SSD and platter technology.

..TS.
(note, the title is just silly - couldn't pass up the potential for punning... ;-)

Cloud Computing and its life cycle

Cloud computing refers to the use of  data/software resources accessible via a external computer network, rather than from a local computer. Users or clients can perform a task, such as word processing, with a browser and with services provided from a cloud based vendor's computational resources.

Pros and Cons from a business perspective
According to Wikipedia, benefits from Cloud Computing are depicted below:

• Agility improves with users' ability to rapidly and inexpensively re-provision technological infrastructure resources.
• Application Programming Interface (API) accessibility to software that enables machines to interact with cloud software in the same way the user interface facilitates interaction between humans and computers. Cloud computing systems typically use REST-based APIs.
• Cost is claimed to be greatly reduced and in a public cloud delivery model capital expenditure is converted to operational expenditure. This is purported to lower barriers to entry, as infrastructure is typically provided by a third-party and does not need to be purchased for one-time or infrequent intensive computing tasks. Pricing on a utility computing basis is fine-grained with usage-based options and fewer IT skills are required for implementation (in-house).
• Device and location independence enable users to access systems using a web browser regardless of their location or what device they are using (e.g., PC, mobile phone). As infrastructure is off-site (typically provided by a third-party) and accessed via the Internet, users can connect from anywhere.
• Multi-tenancy enables sharing of resources and costs across a large pool of users thus allowing for:
  • Centralization of infrastructure in locations with lower costs (such as real estate, electricity, etc.)
  • Peak-load capacity increases (users need not engineer for highest possible load-levels)
  • Utilization and efficiency improvements for systems that are often only 10–20% utilized.
• Reliability is improved if multiple redundant sites are used, which makes well-designed cloud computing suitable for business continuity and disaster recovery.
• Scalability via dynamic ("on-demand") provisioning of resources on a fine-grained, self-service basis near real-time, without users having to engineer for peak loads.
• Performance is monitored, and consistent and loosely coupled architectures are constructed using web services as the system interface.
• Security could improve due to centralization of data, increased security-focused resources, etc., but concerns can persist about loss of control over certain sensitive data, and the lack of security for stored kernels. Security is often as good as or better than under traditional systems, in part because providers are able to devote resources to solving security issues that many customers cannot afford. However, the complexity of security is greatly increased when data is distributed over a wider area or greater number of devices and in multi-tenant systems that are being shared by unrelated users. In addition, user access to security audit logs may be difficult or impossible. Private cloud installations are in part motivated by users' desire to retain control over the infrastructure and avoid losing control of information security.
• Maintenance of cloud computing applications is easier, because they do not need to be installed on each user's computer. They are easier to support and to improve, as the changes reach the clients instantly.

A brief history of cloud computing can be found here.


Life Cycle Model
The technology adoption life cycle model describes the adoption or acceptance of a new product or innovation, according to the demographic and psychological characteristics of defined adopter groups. The process of adoption over time is typically illustrated as a classical normal distribution or bell curve. The model indicates that the first group of people to use a new product is called Innovators, followed by Early Adopters. Next come the early and late majority, and the last group to eventually adopt a product are called laggards.

A description of benefits and drawbacks of the technology adoption life cycle model can be found here.

New Stage for Cloud Computing
Cloud computing seems to have reached a new stage. Everyone is talking about how to incorporate cloud computing into their business plan. Google cloud is here to stay. Apple has come out with their iCloud and will also offer at least some of the services for free. We all make fun of the Microsoft "to the cloud" commercials.

It seems like the cloud idea is at least in the "early majority" phase, maybe further down the bell curve. How long will this last? I don't know, but the first test will be when a leading vendor is hacked and private corporate data is lost. Security, Privacy and Legal issues will arise. For many IT pros, the most important consideration for all cloud based services is that they have to depend on servers that someone else owns and controls.

Conclusion
Actually, the "Hybrid Cloud Model" will most likely prevail. A hybrid storage cloud uses a combination of public and private storage clouds. Hybrid storage clouds are often useful for archiving and backup functions, allowing local data to be replicated to a public cloud. HP, IBM, Oracle and VMware offer technology to help manage the complexities of performance, security and privacy.

Complex Event Processing (CEP) & Twitter

Event-driven architecture and complex event processing are defined by Wikiepedia as:

Complex event processing  (CEP) consists of processing many events happening across all the layers of an organization, identifying the most meaningful events within the event cloud, analyzing their impact, and taking subsequent action in real time

Wall St & Technology magazine state:

Users of StreamBase's Twitter adapter can combine Twitter with market data and build data management applications, says StreamBase CTO Richard Tibbetts. In particular, Twitter can be used as a crowd sourcing tool to help gauge people's sentiment towards a particular event or stock. "It's really useful for sentiment analysis, which traders can then use to help them make trading decisions," he adds.

 Streambase blog states:

 Trading on rumor is a time-honored trading technique.   “Buy on rumor; sell on news” is one of the oldest mantras on Wall Street and Main Street alike.  Any tool that improves the speed or quality of transmitting information is of interest to traders.  Twitter does that.   But be careful what you ask for – lots of the tweets on Twitter are garbage – scam artists and manipulators. 

The major issue slowing down advances in risk management and trading systems is data access and data quality. It seems that these articles are two years old. That's an eternity in the computer industry.  I will have to do more research into this area. But regulation and privacy are still hindering widespread adoption of social media in the capital markets, according to a new report from Aite Group.

Sybase(SAP), Oracle, IBM, Progress Software & Microsoft all have come out with their own  new generation of software, shifting the focus from stored persistent data to events processing. In fact there is a on-going consolidation sweeping through this sector.

 Seeking Alpha explains:

These are some of the publicly traded companies that venture into this realm: CSCO, EMC, IBM, PEGA, PRGS, SAP, TIBX  Given ongoing consolidation, that leaves Active Endpoints, Pegasystems (PEGA), Appian, plus several open source niche pure plays still standing.

But don't consider this a complete list. In my readings, I found Thomson Reuters & StreamBase and I may have missed many more. Do some exploring. Good Hunting!


Related Blogs:
http://mashable.com/2011/05/17/twitter-based-hedge-fund/
http://arxiv.org/PS_cache/arxiv/pdf/1010/1010.3003v1.pdf
http://streambase.typepad.com/streambase_stream_process/2011/05/on-big-data-real-time-computing-and-twitter.html