Real-time analytics on IoT data

Why real-time analytics matters for IoT systems

IoT systems access millions of devices that generate large amounts of streaming data. For some equipment, a single event may prove critical to understanding and responding to the health of the machine in real time, increasing the importance of accurate, reliable data. While real-time data remains important, storing and analyzing the historical data also creates opportunities to improve processes, decision-making and outcomes.

Smart grids, which include components like sensors and smart meters, produce a wealth of telemetry data that can be used for multiple purposes, including:

◉ Identifying anomalies such as manufacturing defects or process deviations

◉ Predictive maintenance on devices (such as meters and transformers)

◉ Real-time operational dashboards

◉ Inventory optimization (in retail)

◉ Supply chain optimization (in manufacturing)

Considering solutions for real-time analytics on IoT data

One way to achieve real-time analytics is with a combination of a time-series database (InfluxDB or TimescaleDB) or a NoSQL database (MongoDB) + a data warehouse + a BI tool:

This architecture raises a question: Why would one use an operational database, and still need a data warehouse? Architects consider such a separation so they can choose a special-purpose database — such as a NoSQL database for document data — or a time-series database (key-value) for low costs and high performance.

However, this separation also creates a data bottleneck — data can’t be analyzed without moving it from an operational data store to the warehouse. Additionally, NoSQL databases are not great at analytics, especially when it comes to complex joins and real-time analytics.

Is there a better way? What if you could get all of the above with a general-purpose, high-performance SQL database? You’d need this type of database to support time-series data, streaming data ingestion, real–time analytics and perhaps even JSON documents.

Achieving a real-time architecture with SingleStoreDB + IBM Cognos

SingleStoreDB supports fast ingestion with Pipelines (native first class feature) and concurrent analytics for IoT data to enable real-time analytics. On top of SingleStoreDB, you can use IBM® Cognos® Business Intelligence to help you make sense of all of this data. The previously described architecture then simplifies into:

Real-time analytics with SingleStoreDB & IBM Cognos

Pipelines in SingleStoreDB allow you to continuously load data at blazing fast speeds. Millions of events can be ingested each second in parallel from data sources such as Kafka, cloud object storage or HDFS. This means you can stream in structured — as well as unstructured data — for real-time analytics.

But wait, it gets better…

1. Once data is in SingleStoreDB, it can also be used for real-time machine learning, or to safely run application code imported into a sandbox with SingleStoreDB’s Code Engine Powered by Web Assembly (Wasm).

2. With SingleStoreDB, you can also leverage geospatial data — for instance to factor site locations, or to visualize material moving through your supply chains.

Armis and Infiswift are just a couple of examples of how customers use SingleStoreDB for IoT applications:

◉ Armis uses SingleStoreDB to help enterprises discover and secure IoT devices. Armis originally started with PostgreSQL, migrated to ElasticSearch for better search performance and considered Google Big Query before finally picking SingleStoreDB for its overall capabilities across relational, analytics and text search. The Armis Platform, of which SingleStoreDB now plays a significant part, collects an array of raw data (traffic, asset, user data and more) from various sources — then processes, analyzes, enriches and aggregates it.

◉ Infiswift selected SingleStoreDB after evaluating several other databases. Their decision was driven in part because of SingleStore’s Universal Storage technology (a hybrid table type that works for both transactional and analytical workloads).

Want to learn more about achieving real-time analytics?

Join IBM and SingleStore on Sep 21, 2022 for our webinar “Accelerating Real-Time IoT Analytics with IBM Cognos and SingleStore”. You will learn how real-time data can be leveraged to identify anomalies and create alarms by reading meter data, and classifying unusual spikes as warnings.

We will demonstrate:

◉ Streaming data ingestion using SingleStoreDB Pipelines

◉ Stored procedures in SingleStoreDB to classify data before it is persisted on disk or in memory

◉ Dashboarding with Cognos

These capabilities enable companies to:

◉ Provide better quality of service through quickly reacting to or predicting service interruptions due to equipment failures

◉ Identify opportunities to increase production throughput as needed

◉ Quickly and accurately invoice customers for their utilization

Source: ibm.com

Continue reading

Kai Ming makes more data-driven decisions with IBM Cognos Analytics

In the 1960s, emerging research on the effects of poverty and its impact on education came to light. This research indicated an obligation to help disadvantaged groups, compensating for inequality in social or economic conditions. In January 1964, a former teacher and then-President Lyndon B. Johnson declared a “war on poverty.” They established Head Start, a program to promote the school readiness of infants, toddlers and preschool-aged children from low-income families as part of the arsenal. Today, Head Start provides services to more than a million children every year in every U.S. state and territory.

Head Start programs are typically run by non-profit organizations, schools and community action agencies. Operations and management require extensive coordination between federal, state and local governments and detailed, often disparate, reporting to each organization participating in the program.

Potential has no limits at Kai Ming

It is the motto and mantra of the executives, educators and staff at Kai Ming Head Start. Founded in Chinatown in 1975, Kai Ming serves low-income families, providing child educational services and family resources to nearly 400 students and their families in 10 San Francisco, California locations. Its governing bodies, executives, educators and staff are dedicated to empowering students and their families.

“Our teachers, they devote themselves to the students in the classroom, and we honor them. To effectively manage the program, we need to work with the parents, through hundreds of interactions, for hundreds of kids,” said Executive Director Jerry Yang, Ph.D., at Kai Ming, “We do a lot of health and age-appropriate developmental screening. But we also work with parents to help them progress toward self-sustainability as many of our families face challenges with employment, housing and food insecurity. So, we needed a good data system to track all these things.” Communication and information exchange is essential for each child’s success.

Solving the data dilemma — fast and accurate capture, reporting and interpretation

Data capture and the subsequent reporting were not easy for Kai Ming. Separate local, state and federal agencies plus private donors and sponsors support Kai Ming. Monthly government-mandated reporting is necessary to maintain funding. Kai Ming’s Governing Board and Parent Council require different formats and content. Furthermore, Dr. Yang needed a comprehensive view of the program, from overall operations to classroom-level analytics and individual family and child records.

Each of Kai Ming’s locations has children with different abilities and challenges. Tracking the scope of these attributes is yet another factor in data capture, reporting and analytics. The organization’s manual reporting frequency and each entities’ detailed reporting requirements were not only time-consuming but took precious time away from more strategic activities that would benefit their children. Automated monitoring of each child’s attendance, health records, and classroom performance were essential for proactive support of the individual and family. Yang sought solutions that would speed Kai Ming’s data capture and analytics capabilities across the organization.

The right tools and support make the difference

Dr. Yang explored several tools for database and application development. He went so far as teaching himself the basics of a software product. The solutions he found were not robust enough to support his business need. Then he learned about the capabilities of IBM® Cognos® Analytics on Cloud software. With the help of IBM Business Partner PMsquare, they applied the features and capabilities that best suited Kai Ming’s business intelligence needs.  Kevin Emanuel, Vice President of Sales at PMsquare, said, “Because Kai Ming, like many non-profit organizations, had budget constraints, we recommended the on-demand version of Cognos and provided the skills to integrate the advanced features of the product.”

The solution enables Kai Ming to import and analyze data from multiple on-premises or cloud sources and provides AI-assisted data cleansing and preparation when data comes from multiple sources. Its visualization capabilities were particularly valuable to Kai Ming as dynamic dashboards allowed Dr. Yang to drill down into the detail and share information when appropriate. With Cognos Analytics, Kai Ming uncovered hidden patterns in operations that assisted planning and decision-making.

Today, Cognos Analytics is an integral part of Kai Ming’s business intelligence and reporting processes.

“Previously, we had to chase after people for information before we could send a report. It could take weeks to compile,” describes Dr. Yang, “Today, it’s done in a snap — in less than an hour.”

Dr. Yang uses dashboard capabilities to examine each child’s progress and classroom operations to optimize needed resources. So what does Dr. Yang do with his extra time? “While I can obtain, analyze and send information faster, that doesn’t mean we sit back with nothing to do. No way. We continue to keep busy working on new ways to empower our kids and their families,” concludes Dr. Yang. In fact, despite the COVID pandemic, 65% of Kai Ming’s children advanced at least one development level in the 2021 academic year.

About Head Start

Head Start programs, administered by the Office of Head Start (OHS), within the Administration for Children and Families, U.S. Department of Health and Human Services, promote the school readiness of infants, toddlers and preschool-aged children from low-income families. Services are provided in various settings, including centers, family childcare and childrens’ own homes. Head Start programs engage parents or other key family members in positive relationships, focusing on family wellbeing. Decades of research show that Head Start participation has both short- and long-term positive effects for Head Start children and their families. To donate to Kai Ming Head Start or learn more about their services, visit its site.

About PMsquare

PMsquare is a data and analytics consultancy and an IBM Business Partner specializing in Data Science and AI. As an end-to-end solution provider, PMsquare’s goal is to help clients simplify the complex challenges involved with architecting data and analytics strategies and implementing analytics solutions that deliver powerful insights to organizations around the globe.

As a supporter of Kai Ming’s organizational mission, PMsquare is also a proud sponsor of Kai Ming’s newest children’s center in San Francisco: PMsquare Children’s Center

Source: ibm.com

Continue reading