How foundation models can help make steel and cement production more sustainable

Heavy industries, particularly cement, steel and chemicals, are the top greenhouse gas emitting industries, contributing 25% of global CO2 emission. They use high temperature heat in many of their processes that is primarily driven by fossil fuel. Fighting climate change requires lowering heavy industry emissions. However, these industries face tremendous challenges to reduce greenhouse gas emissions. Replacing equipment is not a viable route to reduce emissions, as these industries are capital intensive, with asset lifecycles of over 40 years. They are also trying alternate fuels, which come with their own challenges of alternate fuel availability, and the ability to manage processes with fuel-mixes. The Paris Agreement on climate change also mandates that these industries will need to reduce annual emissions by 12-16% by 2030. Generative AI, when applied to industrial processes, can improve production yield, reduce quality variability and lower specific energy consumption (thereby reducing operational costs and emissions).

Higher variability in processes and operations results in higher specific energy consumption (SEC) and higher emissions. This variability comes from material inconsistency (raw material comes from earth), varying weather conditions, machine conditions and the human inability to operate the processes at top efficiency 24 hours a day, every day of the week. Artificial Intelligence technology can predict future variability in the processes and the resultant impact on yield, quality and energy consumption. For example, say we predict the quality of the clinker in advance, then we are able to optimize the heat energy and combustion in the cement kiln in such a way that quality clinker is produced at minimum energy. Such optimization of the processes reduces energy consumption and in turn reduces both energy emission and process emission.

Foundation models make AI more scalable by consolidating the cost and effort of model training by up to 70%. The most common use of foundation models is in natural-language processing (NLP) applications. However, when adapted accordingly, foundation models enable organizations to successfully model complex industrial processes accurately, creating a digital twin of the process. These digital twins capture multivariate relationships between process variables, material characteristics, energy requirements, weather conditions, operator actions, and product quality. With these digital twins, we can simulate complex operating conditions to get accurate operating set points for process “sweet spots.” For example, the cement kiln digital twin would recommend the optimal fuel, air, kiln speed and feed that minimizes heat energy consumption and still produces the right quality of clinker. When these optimized set points are applied to the process, we see efficiency improvements and energy reductions that have not been seen or realized before. The improved efficiency and SEC not only translate to EBITDA value, but also reduced energy emission and process emission.

Optimize industrial production with Foundation Models

Heavy industry has been optimizing processes with AI models for the last few years. Typically, regression models are used to capture process behavior; each regression model captures the behavior of a part of the process. When stitched together with an optimizer this group of models represents the overall behavior of the process. These groups of 10-20 models are orchestrated by an optimizer like an orchestra to generate optimized operating point recommendations for plants. However, this approach could not capture the process dynamics, such as ramp-ups, ramp-downs especially during disruptions. And training and maintaining dozens of regression models is not easy, making it a bottleneck for accelerated scaling.

Today, foundation models are used mostly in natural language processing. They use the transformer architecture to capture longer term relationships between words (tokens in Gen AI terminology) in a body of text. These relationships are encoded as vectors. These relationship vectors are then used to generate content for any specific context (say, a rental agreement). The accuracy of resultant content generated from these mapped vectors is impressive, as demonstrated by ChatGPT. What if we could represent time series data as a sequence of tokens? What if we can use the parallelized transformer architecture to encode multivariate time series data to capture long and short-term relationships between variables? 

IBM Research, in collaboration with IBM Consulting, has adapted the transformer architecture for Time Series data and found promising results. Using this technology, we can model an entire industrial process, say a cement kiln with just one foundation model. The foundation models are trained for a process domain and can capture the behavior of the entire asset and process class. For instance, a cement mill foundation model can capture the behavior of several capacities of cement mills. Therefore, every subsequent mill that we deploy to needs to go through only finetuning of the “Cement Mill Foundation Model” rather than a top-down training process. This cuts model training and deployment time by half, making it a viable technology for large-scale rollouts. We have observed that these foundation models are 7 times as accurate as regression models. And to top it all, we can capture process dynamics as these models do multi-variate forecasting with good accuracy.

Generative AI powered future of heavy industry

 Generative AI technology is bound to transform industrial production to an unforeseen level. This is the solution to reign in industrial emissions and increase productivity with minimal CAPEX impact and positive EBITDA impact. IBM is engaging with several clients to bring this technology to the production floor and seeing up to a 5% increase in productivity and up to 4% reduction in specific energy consumption and emissions. We form a joint innovation team along with the client teams and together train and deploy these models for several use cases ranging from supply chain optimization, production optimization, asset optimization, quality optimization to planning optimization. We have started deploying this technology in a large steel plant in India, a cement plant in Latin America and CPG manufacturing in North America. 

Ultimately, it’s about people: the operators in the plant must embrace it, the process engineers should love it, and the plant management must value it. That can only be achieved with effective collaboration and change management, which we focus on throughout the engagement. Let’s partner together on fostering in an era where we can grow our production capacities without compromising on the sustainability ambitions and create a better, healthier world for future generations to come.

Source: ibm.com

Continue reading

Computer simulations identify new ways to boost the skin’s natural protectors

Working with Unilever and the UK’s STFC Hartree Centre, IBM Research uncovered how skin can boost its natural defense against germs.

As reported in Biophysical Journal, small-molecule additives can enhance the potency of naturally occurring defense peptides. Molecular mechanisms responsible for this amplification were discovered using advanced simulation methods, in combination with experimental studies from Unilever.

When in balance, our skin and its microbiome form a natural partnership that helps to keep our skin healthy and defends against external threats, like pollutants and germs that can cause infections. Disturbances in that partnership (called dysbioses) can lead to imbalances in the microbiome which can also contribute to body odor, skin problems, and in more extreme cases, even lead to medical conditions like eczema (or atopic dermatitis).

In addition to hosting your microbiome, your skin is an immunologically active organ, contributing to your body’s innate immune system with its naturally mildly acidic pH, mechanical strength, lipids, and a natural release by skin cells of protein-like materials called antimicrobial peptides (AMPs). Together, these form the first line of defense against infection causing microbes that land on your skin.

Unilever R&D and its global network of research partners have been investigating the role of skin immunity and AMPs for over a decade. When Unilever needed to develop new ways to understand, at the molecular level, how its products interact with AMPs to enhance skin defense activity, the company turned to IBM Research.

IBM and Unilever — in collaboration with STFC, which hosts one of IBM Research’s Discovery Accelerators at the Hartree Centre in the UK — used high performance computing and advanced simulations running on IBM Power10 processors to understand how AMPs work and translate this knowledge into consumer products that boost the effects of these natural-defense peptides. This work builds upon a long-standing partnership between IBM, Unilever and the STFC Hartree Centre aimed at advancing digital research and innovation.

As we report in Biophysical Journal, our work alongside STFC’s Scientific Computing Department found that small-molecule additives (organic compounds with low molecular weights) can enhance the potency of these naturally occurring defense peptides. Using our own advanced simulation methods, in combination with experimental studies from Unilever, we also identified specific new molecular mechanisms that could be responsible for this improved potency.

Simulating molecular interactions

Although there’s been a lot of research focused on designing new, artificial antimicrobials, Unilever wanted to concentrate on boosting the potency of the body’s naturally occurring germ fighters with small-molecule additives. IBM Research has already developed computational models for membrane disruption and permeation through physical modeling, but Unilever’s challenge was a new area of exploration for us, given the extremely complex nature of having to model how AMPs interact with the skin and calculate which would be the most efficacious.

Several years ago, Unilever scientists in India discovered that Niacinamide, an active form of vitamin B3 naturally found in your skin and body, could enhance AMP expression levels in laboratory models. At the same time Unilever’s team also observed an unexpected enhancement of AMP antimicrobial activity in cell-free systems, and wanting to understand why this enhanced activity was happening — a research collaboration between Unilever, IBM, and STFC was initiated.

To answer Unilever’s question we developed computer simulations to investigate how single molecules interact with bacterial membranes at the molecular scale to demonstrate the fundamental biophysical mechanisms in play. These models then formed the basis of more complex simulations that examined in similar detail how small molecules interact with skin defense peptides to affect their potency. The results of these simulations were compared to the results of extensive laboratory experimental tests conducted by Unilever to confirm our computational predictions on a range of niacinamide analogs with differing abilities to promote AMP activity in lab models.

We first used physical modeling to determine the effects of the B3 analogs on LL37, a common AMP on human skin. We then simulated these molecules using high-performance computing to predict their performance and generate detailed time-bound simulations that allowed us to “see” these interactions in molecular detail. This work enabled us to demonstrate that niacinamide (and another analog, methyl niacinamide) could indeed naturally boost the effect of the AMP peptide LL37 on the bacterium Staphylococcus aureus, an organism widely associated with skin infections.

A radical discovery process — and a map for hunting new bioactives

Our work has helped us understand how these molecules can improve hygiene, but it also provided us with a deeper understanding of the molecular mechanisms responsible for enhanced AMP performance, by pairing simplified model systems and advanced computation that radically accelerated technology evaluation. We believe this workflow can allow us to create innovative and sustainable products that can help to protect us from pathogens both now and in the future.

The scientific method, applied to peptides.

This research was made possible by our and our partners’ capabilities in high-performance computing. Combining these technologies allowed us to supercharge the scientific method to promote discovery at a far more rapid pace, a process we’ve come to call accelerated discovery. 

We’re excited that our work can help Unilever better understand how to leverage AMPs in future products to help countless people around the world through the development of effective and sustainable hygiene products, while complying with the applicable regulations..

For us at IBM, this work is also the start of an exciting new chapter as we explore how this work can help accelerate research into other harmful pathogens, such as Methicillin-resistant Staphylococcus aureus (MRSA), that can cause severe disease if their growth is not controlled. More broadly, this work opens a new pathway to discovering natural, small-molecule boosters to amplify the function of antimicrobial peptides Our understanding of these mechanisms and the process we used can be applied for other research, for example, in the search for novel antimicrobials.

This was a cross-industry academia partnership that spanned the globe, with scientists from India and the UK coming together to solve germane and pressing problems with real world application. We hope one of the lasting impacts of this work is that for future research in this field, we’re able to choose or devise computational models simple enough to capture essential biological processes — without adding unnecessary time or complexity.

Source: ibm.com

Continue reading

What Is Quantum-Safe Cryptography, and Why Do We Need It?

How to prepare for the next era of computing with quantum-safe cryptography.

Cryptography helps to provide security for many everyday tasks. When you send an email, make an online purchase or make a withdrawal from an ATM, cryptography helps keep your data private and authenticate your identity.

Today’s modern cryptographic algorithms derive their strength from the difficulty of solving certain math problems using classical computers or the difficulty of searching for the right secret key or message. Quantum computers, however, work in a fundamentally different way. Solving a problem that might take millions of years on a classical computer could take hours or minutes on a sufficiently large quantum computer, which will have a significant impact on the encryption, hashing and public key algorithms we use today. This is where quantum-safe cryptography comes in.

According to ETSI, “Quantum-safe cryptography refers to efforts to identify algorithms that are resistant to attacks by both classical and quantum computers, to keep information assets secure even after a large-scale quantum computer has been built.”

What is quantum computing?

Quantum computers are not just more powerful supercomputers. Instead of computing with the traditional bit of a 1 or 0, quantum computers use quantum bits, or qubits (CUE-bits). A classical processor uses bits to perform its operations. A quantum computer uses qubits to run multidimensional quantum algorithms. Groups of qubits in superposition can create complex, multidimensional computational spaces. Complex problems can be represented in new ways in these spaces. This increases the number of computations performed and opens up new possibilities to solve challenging problems that classical computers can’t tackle.

There are many exciting applications in the fields of health and science, like molecular simulation that has the potential to speed up the discovery of new life-saving drugs. The problem is, however, quantum computers will also be able to solve the math problems that give many cryptographic algorithms their strength.

How will quantum computing impact cryptography?

Two of the main types of cryptographic algorithms in use today for the protection of data work in different ways:

◉ Symmetric algorithms use the same secret key to encrypt and decrypt data.

◉ Asymmetric algorithms, also known as public key algorithms, use two keys that are mathematically related: a public key and a private key.

The development of public key cryptography in the 1970s was revolutionary, enabling new ways of communicating securely. However, public key algorithms are vulnerable to quantum attacks because they derive their strength from the difficulty of solving the discrete log problem or factoring large integers. As discovered by mathematician Peter Shor, these types of problems can be solved very quickly using a sufficiently strong quantum computer, so in the case of asymmetric or public key cryptography, we need new math that will stand up to quantum attacks because today’s public key algorithms will be completely broken.

Grover’s Algorithm, devised by computer scientist Lov Grover, is a quantum search algorithm. Using Grover’s algorithm, some symmetric algorithms are impacted and some are broken. Key size and message digest size are important considerations that will factor into whether an algorithm is quantum-safe or not. For example, use of Advanced Encryption Standard (AES) with 256-bit keys is considered quantum-safe but Triple DES (TDES) can be broken no matter the key size.

What is being done to address future quantum threats?

The good news is that researchers and standards bodies are moving to address the threat. The National Institute of Standards and Technology (NIST) initiated a Post-Quantum Cryptography Standardization Program to identify new algorithms that can resist threats posed by quantum computers.

After three rounds of evaluation, NIST has identified seven finalists. They plan to select a small number of new quantum-safe algorithms early this year and have new quantum-safe standards in place by 2024. As part of this program, IBM researchers have been involved in the development of three quantum-safe cryptographic algorithms based on lattice cryptography that are in the final round of consideration: CRYSTALS-Kyber, CRYSTALS-Dilithium and Falcon.

How should enterprises be preparing to adopt quantum-safe cryptography?

Fortunately, we have time to implement quantum-safe solutions before the advent of large-scale quantum computers — but not much time. Moving to new cryptography is complex and will require significant time and investment. We don’t know when a large-scale quantum computer capable of breaking public key cryptographic algorithms will be available, but experts predict that this could be possible by the end of the decade.

Also, hackers can harvest encrypted data today and hold it for later when they can decrypt it using a quantum computer, so sensitive data with a long lifespan is already vulnerable. Organizations in the United States and Germany have already issued requirements for government agencies to follow regarding quantum-safe cryptography. BSI, a German federal agency, requires the use of hybrid schemes — where both classical and quantum-safe algorithms are used — for protection in high-security applications. The White House issued a memo requiring federal agencies to begin quantum-safe modernization planning.

How can IBM help?

As we prepare for a quantum world, IBM is committed to developing and deploying new quantum-safe cryptographic technology. Trusted hardware platforms will play a critical role in the adoption of quantum-safe cryptography. And IBM Z has already begun the modernization process. IBM z15 introduced lattice-based digital signatures within the system for digital signing of audit records within z/OS. IBM z15 also provided the ability for application developers to begin experimenting with quantum-safe lattice-based digital signatures. Because we’ve already begun the process, this helps us understand the implications of moving to new algorithms so we can pass on insights about the topic to our clients.

Preparing to adopt quantum-safe standards

When meeting with clients getting started on their journey to quantum safety, we share a few of the key milestones to help them get ready to adopt new quantum-safe standards:

◉ Discover and classify data: The first step involves classifying the value of your data and understanding compliance requirements. This helps you create a data inventory.

◉ Create a crypto inventory: Once you have classified your data, you will need to identify how your data is encrypted, as well as other uses of cryptography to create a crypto inventory that will help you during your migration planning. Your crypto inventory will include information like encryption protocols, symmetric and asymmetric algorithms, key lengths, crypto providers, etc. 

◉ Embrace crypto agility: The transition to quantum-safe standards will be a multi-year journey as standards evolve and vendors move to adopt quantum-safe technology. Use a flexible approach and be prepared to make replacements. Implement a hybrid approach as recommended by industry experts by using both classical and quantum-safe cryptographic algorithms. This maintains compliance with current standards while adding quantum-safe protection.

Many clients across industries have already started experimenting with new quantum-safe algorithms in order to assess the impact of new quantum-safe standards on their businesses:  

◉ Automotive: Clients in the automotive industry use public key technology in connected cars for vehicle-to-everything (V2X) communications and to verify the integrity of the firmware loaded into vehicles. The cars they are designing today will be on the road well into the future, so they are on a tight timeline to adopt quantum-safe technology. Because vehicles have hardware resource constraints, it is critical that automotive clients model and test new quantum-safe algorithms now to make sure they can accommodate the larger key sizes in their use cases.   

◉ Banking: Clients in the banking industry rely heavily on symmetric cryptography to ensure the confidentiality of data in core banking applications. There are many data retention and data confidentiality regulations and agreements that these clients must adhere to, such as retaining tax records for 7–10 years and keeping trade secrets confidential for up to 50 years. Adversaries are starting their attacks today with the intent of disclosing this type of confidential data in the future, so many banking clients have started creating data and crypto inventories to adopt quantum-safe protection for highly sensitive data. Banks also rely on public key cryptography, for example, in digital signatures used for authentication and software verification. It’s important for banking clients to begin modeling new quantum-safe algorithms to understand performance implications and prepare to adopt new standards as they evolve.

Source: ibm.com

Continue reading

The IBM Research innovations powering IBM z16

From 7 nanometer node chips to built-in AI acceleration and privacy, IBM Research was behind many of the groundbreaking aspects of the new IBM z16 system.

Pictured: Operating an IBM z16 system designed and manufactured in Poughkeepsie, N.Y. From 7 nm node chips to AI acceleration and privacy technology, IBM Research was behind many of z16’s innovations. (Image: courtesy of IBM.)

Today, we're unveiling IBM z16, our next-generation mainframe system, containing several groundbreaking innovations, including the new 7 nm Telum chip that can facilitate on-device AI inferencing that’s 20 times faster than sending an AI request to an x86 server in the cloud, as well as quantum-safe cryptography, multi-cloud support, and data privacy that’s central to the system. Many of those innovations began life in IBM Research.

Scaling to 7 nm

These innovations didn't happen overnight. Back in 2015, IBM Research demonstrated the industry’s first 7 nm test chip with our partners, including Samsung. This 7 nm technology achieved two important milestones: 2.4x logic density and 17.6% frequency gain over 14 nm technology, which was at that time the most advanced technology. This 7 nm chip included the first implementation of Extreme UltraViolet (EUV) lithography technology. We had developed the semiconductor technology that would provide a foundation for better performance and power saving.

Read More: IBM Certifications

This innovation came from our Albany lab, which has since developed into a world-class ecosystem of partners from government, academia and industry. The result of over $15 billion in public and private investment, the Albany Nanotech Complex has continually delivered advances in CMOS logic scaling such as nanosheet architecture, our 2 nm node chip, and most recently vertical-transport FET, or VTFET.

A powerful chip with AI at its core

Our 7 nm node technology served as a foundation for advances in AI hardware, such as our Telum processor. 

Back in August, we unveiled Telum, a new 7 nm CPU chip. It’s IBM’s first commercially available processor to contain on-chip acceleration for AI inferencing, which could have massive impacts on industries like banking, finance, healthcare, and logistics. At the time, we said we believed it to have the potential to be as great of a technological shift as the original IBM zSystem was when it launched. The new IBM z16 system will be the first to ship with Telum onboard.  

The AI core augmenting capabilities for IBM z16 critical workloads is the result of the aggressive roadmap of the IBM Research AI Hardware Center towards sustained innovation for continued efficiency improvement of AI compute resources. The research teams have been working with the IBM Systems teams to integrate the AI core technology from our latest generation chip into the IBM zSystem. 

In traditional computing systems, calculations are usually performed by constantly transferring data between off-chip memory and processors. For AI workloads, though, there’s a far higher computational requirement, as they generally require large amounts of data. The more AI is infused into application workloads, the more critical it is to have an efficient system where both the general-purpose CPU cores and AI cores are tightly integrated on the same chip.

Each Telum chip provides a dedicated AI core alongside the traditional horsepower of eight CPU cores. The CPU cores handle general-purpose software applications, while the AI core is highly efficient for running deep-learning workloads. Each Telum chip contains 22 billion transistors along with 19 miles of wire.

With Telum, it’s possible to detect fraud during the instant of a transaction. It’s possible to determine whether to extend someone a loan as quickly as they applied.

Beyond sheer power, this chip has the potential to revolutionize the way AI is implemented at scale. The chip offers a 20-fold speedup in AI inference over sending the AI request to x86 servers in the cloud. With Telum, it’s possible to detect fraud during the instant of a transaction. It’s possible to determine whether to extend someone a loan as quickly as they applied. 

With past systems, running AI inference on a process would have to happen after the transaction took place. Think about a time your bank notified you that a suspected fraudulent transaction took place on your account: that notification likely came minutes to hours after the fraud happened. By then, the fraudster could have gotten away with money that now needs to be recovered. Being able to detect and prevent fraud  during the moment of a transaction, whether it’s in finance, retail, or myriad other industries, would be a dramatic shift in the way AI is deployed in the modern world. 

But it’s not just about running AI processes more quickly — it’s about the sorts of problems Telum allows you to tackle. An IBM zSystem requires 50% less energy than x86 systems to run the same workload. zSystems can run at 100% utilization, whereas x86 systems run at far lower utilizations.  

Delivering systems that can run AI workloads considerably more efficiently opens up the door for all sorts of new opportunities. Logistics and retail firms can run large-scale inferencing tasks to figure out the places most at risk in their supply chains. Finance houses can determine which trades were most at risk before settlement, and people could find out in an instant, instead of weeks, whether they’ve been approved for a loan.

Companies can also use IBM Research-designed AI software on the IBM z16 to help them find similar instances too; if fraud was just found on one account, you could use SQL Insights which has built in neural networks embedded in Db2 for z/OS to find similar transactions — without having to install, select, tune and configure AI models, or pre-determine what features the AI models should be trained on.

Finance houses can determine which trades were most at risk before settlement, and people could find out in an instant whether they’ve been approved for a loan. 

The IBM z16 system will also offer multi- and hybrid-cloud support through z/OS Container extensions (zCX) for OpenShift, which was co-developed by IBM Research and can be managed as Red Hat OpenShift containers. This will allow for workloads running AI accelerated models to be managed by OpenShift on premises or as part of a multi-cloud setup.

With z/OS container extensions, and support for popular development tools, programmers can build apps for their sites that transparently take advantage of the IBM z16’s hardware. 

Security for today and tomorrow 

IBM Research’s contributions to the new IBM z16 system don’t stop at the hardware innovations or availability of IBM zSystems and services in the cloud. We wanted to ensure that regardless of what these systems are called to do, they can secure the customers’ — and their customers’ — data.  

We designed the IBM z16 to have the highest security out of the box. We provided data privacy for diagnostics that can detect and scrub sensitive data before it goes anywhere outside of the system. Hackers often try to force software and computer systems to stop with errors, as this will force computers to automatically provide diagnostic data, where personal information can be exposed. 

We also provide cryptography hardware that automatically encrypts all data in IBM zSystems’ Hyper Protect databases, virtual machines and containers. Policies can be set to pervasively encrypt z/OS data and Linux on zSystems, via Secure Service containers (on premises) or Hyper Protect (in the cloud).

The IBM z16 includes the IBM Hyper Protect Data Controller, a feature that protects data as it leaves the IBM zSystem. You can set up rules for different data, such as which user has what type of access and how the data should be encrypted. For example, one user can only see part of the data, whereas other users will only get the encrypted version of the data. These rules are enforced even when the data leaves an IBM z16 system, applied wherever the data is sent.  

We’ve also built the IBM z16 system to help with compliance, as we know that 40% of IT departments and business’ time is spent in compliance. If, for example, your system is processing credit card data, you have to abide by the Payment Card Industry Data Security Standards (PCI DSS). Our IBM z16 compliance software checks the components of the zSystem for adherence and provides a compliance dashboard to system administrators.

But it’s not just about helping solve today’s biggest challenges; we want to make sure that IBM z16 can stand up to the biggest threats of tomorrow, too. IBM z16 is the industry’s first quantum-safe system, using and providing quantum-safe encryption, an approach for constructing security protocols that helps protect data and systems against current and future threats. 

Source: research.ibm.com

Continue reading

How AI will help code tomorrow’s machines

At NeurIPS 2021, IBM Research presents its work on CodeNet, a massive dataset of code samples and problems. We believe that it has the potential to revitalize techniques for modernizing legacy systems, helping developers write better code, and potentially even enabling AI systems to help code the computers of tomorrow.

Chances are, if you’ve done just about anything today, you’ve interacted with a code language older than the invention of the desktop computer, the internet, and VHS tape.

Whether you’ve checked your bank account, used a credit card, gone to the doctor, booked a flight, paid your taxes, or bought something in a store, you likely have interacted with a system that relies on COBOL (Common Business Oriented Language) code. It’s a programming language that many mission-critical business systems around the world still rely on, even though it was first implemented over six decades ago. It’s estimated that some 80% of financial transactions use COBOL, and the U.S. Social Security Administration utilizes around 60 million lines of COBOL code.

As programmers and developers versed in COBOL have started to retire, organizations have struggled to keep their systems up and running, let alone modernize them for the realities of the always-on internet. And this is just one of a myriad of languages still in use that don’t reflect what modern coders feel most comfortable writing in, or what’s best suited for modern business applications.

Code language translation is one of many problems that we strive to address with CodeNet, which we first unveiled back in May. Essentially, CodeNet is a massive dataset that aims to help AI systems learn how to understand and improve code, as well as help developers code more efficiently, and eventually, allow an AI system to code a computer. It’s made up of around 14 million code samples, comprising some 500 million lines of code from more than 55 different languages. It includes samples from modern languages like C++, Java, Python, and Go, to legacy ones like Pascal, FORTRAN, and COBOL. Within a short span of three months, our GitHub received 1,070 stars and has been forked over 135 times.

Figure 1: CodeNet code composition.

This week at NeurIPS, we discuss our paper on CodeNet, and the work we’ve done to build out CodeNet, how we see it as different from anything else like it that’s available for anyone to download, and how we see it being used by the research community.

There has been a revolution in AI over the last decade. Language and image data, carefully curated and tagged in datasets like ImageNet, have given rise to AI systems that can complete sentences for writers, detect tumors for doctors, and automate myriad business and IT processes. But for code, the language of computers, crafting such a dataset that AI systems can be learn from has been a challenging task.

The end goal of CodeNet is to enable developers to create systems that can modernize existing codebases, as well as fix errors and security vulnerabilities in code. It’s something I recently discussed in a lightboard video: Can computers program computers?

Experiments on CodeNet

We’ve carried out baseline experiments on CodeNet for code classification, code similarity, and code completion. These results serve as a reference for CodeNet users when they perform their own experiments. Some of our results also indicate that the models derived from CodeNet can generalize better across datasets than those derived from other datasets due to CodeNet’s high quality.

Code classification

CodeNet can help create systems that can help determine what type of code a snippet is. We used a wide range of machine-learning methods for our experiments, including bag of tokens, sequence of tokens, BERT model, and graph neural networks (GNNs). We achieved upwards of 97% accuracy with some of our methods at matching code types to source code.

Code similarity

Code similarity determines if multiple pieces of code solve the same problem. It serves as the foundational technique for code recommendation, clone detection, and cross language transformations. We tested a wide spectrum of methods for code similarity (including MLP with bag of tokens, Siamese network with token sequence, a Simplified Parse Tree [SPT] with handcrafted feature extraction, and a GNN with SPT) against our benchmark datasets. The best similarity score comes from leveraging a sophisticated GNN with intra-graph and inter-graph attention mechanisms.

Generalization across datasets

We believe that models trained on the CodeNet benchmark datasets can benefit greatly from their high quality. For example, we took our benchmark, C++1000 and compared it against one of the largest publicly available datasets of its kind, GCJ-297, derived from problems and solutions in Google’s Code Jam. We trained the same MISIM neural code similarity system model on C++1000 and GCJ-297, and tested the two trained models on another independent dataset, POJ-104.

Our data suggests that the model trained on GCJ-297 has a 12% lower accuracy score than the model trained on C++1000. We believe C++1000 can better generalize because there’s less data bias than there is in GCJ-297 (where the top 20 problems with the greatest number of submissions account for 50% of all the submissions), and the quality of the cleaning and de-duplication of the data in CodeNet is superior.

Code completion

We believe this to be a valuable use case for developers, where an AI system can predict what code should come next at a given position in a code sequence. To test this, we built a masked language model (MLM) that randomly masks out (or hides) tokens in an input sequence and tries to correctly predict what comes next in a set of tests it hasn’t seen yet. We trained a popular BERT-like attention model on our C++1000 benchmark, and achieved a top-1 prediction accuracy of 91.04% and a top-5 accuracy of 99.35%.

Further use cases for CodeNet

The rich metadata and language diversity open CodeNet to a plethora of interesting and practical use cases. The code samples in CodeNet are labeled with their anonymized submitter and acceptance status so we can readily extract realistic pairs of buggy and fixed code from the same submitter for automated code repair. A large percentage of the code samples come with inputs so that we can execute the code to extract the CPU run time and memory footprint, which can be used for regression studies and prediction. CodeNet may also be used for program translation, given its wealth of programs written in a multitude of languages. The large number of code samples written in popular languages (such as C++, Python, Java, and C) provide good training datasets for the novel and effective monolingual approaches invented in the past several years.

What differentiates CodeNet

While CodeNet isn’t the only dataset aimed at tackling the world of AI for code, we believe it to have some key differences.

Large scale: To be useful, CodeNet needs to have a large number of data samples, with a broad variety of samples to match what users might encounter when trying to code. With its 500 million lines of code, we believe that CodeNet is the largest dataset in its class: It has approximately 10 times more code samples than GCJ, and its C++ benchmark is approximately 10 times larger than POJ-104.

Rich annotation: CodeNet also includes a variety of information on its code samples, such as whether a sample solves a specific problem (and the error categories it falls into if it doesn’t). It also includes a given task’s problem statement, as well as a sample input for execution and a sample output for validation, given that the code is supposed to be solving a coding problem. This additional information isn’t available in other similar datasets.

Clean samples: To help with CodeNet’s accuracy and performance ability, we analyzed the code samples for near-duplication (and duplication), and used clustering to find identical problems.

How CodeNet is constructed

CodeNet contains a total of 13,916,868 code submissions, divided into 4,053 problems. Some 53.6% (7,460,588) of the submissions are accepted, meaning they can pass the test they’re prescribed to do, and 29.5% are marked with wrong answers. The remaining submissions were rejected due to their failure to meet runtime or memory requirements.

The problems in CodeNet are mainly pedagogical and range from simple exercises to sophisticated problems that require advanced algorithms, and the people submitting the code range from beginners to experienced developers. The dataset is primarily composed of code and metadata scraped from two online code judging sites, AIZU and AtCoder. These sites offer courses and contests where coding problems are posed and submissions are judged by an automated review process to see how correct they are. We only considered public submissions and manually merged the information from the two sources, creating a unified format from which we made a single dataset.

We ensured that we applied a consistent UTF-8 character encoding on all the raw data we collected, given that the data came from different sources. We also removed byte-order marks and use Unix-style line-feeds as the line ending.

We looked for duplicate problems, as much of these problems were compiled over many decades. We also identified near-duplicate code samples to facilitate extraction of benchmark datasets in which data independence is desirable.

We provided benchmark datasets for the dominant languages (C++, Python, and Java) for the convenience of the users. Each code sample and related problem are unique, and have passed through several pre-processing tools we’ve provided to ensure that code samples can effectively be converted into a machine learning model input. Users can also create benchmark datasets that are customized to their specific purposes using the data filtering and aggregation tools provided in our GitHub.

What’s next for CodeNet

This is just the start for our vision of what CodeNet can offer to the world of AI for code. We hope to achieve widespread adoption of the dataset to spur on innovation in using AI to modernize the systems we all rely on every day.

In the near future, we will be launching a series of challenges based on the CodeNet data. The first is a challenge for data scientists to develop AI models using CodeNet that can identify code with similar functionality to another piece of code. This challenge was launched in partnership with Stanford’s Global Women in Data Science organization. We’ve organized workshops to introduce the topic, code similarity, and provide educational material. Every team that participates in these challenges is comprised of at least 50% women to encourage diversity in this exciting area of AI for Code.

We envision a future where a developer can build on legacy code in a language they’re accustomed to. They could write in Python and an AI system could convert it in completely executable COBOL, extending the life of the system they’re working on, as well as its reliability, indefinitely. We see the potential for AI systems that can evaluate a developer’s code, based on thousands of examples of past code, and suggest how to improve how they develop their system, or even write a more efficient piece of code itself. In short, we’ve begun to explore how computers can program the ones that succeed them. But for CodeNet to succeed, we need developers to start using what we’ve built.

Source: ibm.com

Continue reading

IBM Research and Red Hat work together to take a load off predictive resource management

PayPal, the global payments giant, has already started putting load-aware scheduling into production.

In today’s ever-changing hybrid cloud field, based on many open-source projects, researchers face two fundamental challenges:

1. Being able to back up their ideas with deep research.

2. Convincing the open source community that their idea is important and enhances existing software frameworks.

Read More: C2150-606: IBM Security Guardium V10.0 Administration

Working as one team, scientists at IBM Research and Red Hat joined together to overcome these obstacles and produce tangible solutions in just seven months.

Red Hat OpenShift is the connective tissue between the infrastructure our clients use. It allows users to write applications once and run them anywhere. And it standardizes the approach to development, security, and operations on any cloud, from any vendor. But Kubernetes, the container orchestration engine at the core of OpenShift, has some areas where our team thought additional features and enhancements could be added.

There are two separate components to the work:

◉ The first is a set of load-aware scheduler plugins, called Trimaran, that factor in the actual usage on the worker nodes—something Kubernetes doesn’t take into account.

◉ The second is a controller that allows developers to automatically resize their containers, called Vertical Pod Autoscaler (VPA).

Today, most developers need to guess how much resource they believe they’ll need, or overestimate just to be sure. This controller can resize a container in real time during runtime. We introduced upstream enhancements in Kubernetes, which lets developers easily incorporate more predictive autoscaling algorithms.

In both cases, the work started as open-source projects. Red Hat works with community-created open-source software and builds upon each project to harden security, fix bugs, patch vulnerabilities and add new features so it is ready for the enterprise.

The auto-scaler has just been made generally available and supported by Red Hat in OpenShift 4.8, and the load-aware scheduling is expected to be available in the next release of OpenShift. 

“Our collaboration with IBM Research takes an upstream-first approach, helping to fuel innovation in the Kubernetes community.

“When innovation first happens in the Kubernetes community, it provides the opportunity for others to provide feedback. We then build on that feedback and apply it in OpenShift to help solve new customer use cases in the platform. Red Hat is one of the top contributors in the Kubernetes community,” Tushar Katarki, Director of Red Hat OpenShift Product Management, told us.

And the aim is to make these open-source projects that started out as research efforts blossom into use that can have a profound impact on Red Hat’s customers.

“The collaboration between IBM Research and Red Hat OpenShift has resulted in numerous enhancements that expand the intelligence of core OpenShift components,” Red Hat’s Director of OpenShift Engineering, Chris Alfonso, said. “The impact to our customers is significant in terms of managing their workloads in complex environments which demand flexibility in compute resource utilization.”

PayPal, the global payments giant, has already started putting load-aware scheduling into production

“In a large-scale environment like PayPal, the platform team has to assure the efficiency of the fleet while keeping safety in mind,” Shyam Patel, the director of Container Platform & Infrastructure at PayPal, told us.

“Standard scheduling uses declarative resource mapping, and at times workload has higher SKU than they need. In this case, we end up wasting resources. Similarly, we don’t want compute resource utilization to go beyond safe allocation. Trimaran offers resource usage-based scheduling capabilities that greatly helps achieving the optimal usage while maintaining a safety net.”

Source: ibm.com

Continue reading

Going beyond Women’s History Month: Celebrating & empowering women scientists

We kick off Women’s History Month and look forward to celebrating International Women’s Day on March 8. At IBM, we are committing to the theme of “Women Rising,” which has personal and professional resonance for me as a woman in what unfortunately continues to be predominantly male fields: science, technology, engineering, and math (STEM). In fact, although women account for half of the US’s college-educated workforce, they only make up 28 percent of the STEM workforce and 13 percent of inventors.

Read More: C2150-606: IBM Security Guardium V10.0 Administration

And while we have made progress in both hiring and representation for global women and underrepresented minorities in the US over the last three years at IBM, we know we can and must do more to ensure women feel empowered to be their authentic selves, to build their skills, and rise to their full potential.

At IBM Research, we value a diverse and inclusive workforce — and we recognize that we need women researchers and scientists more than ever to solve some of world’s greatest challenges. And we are facing an unprecedented magnitude of complex global issues that can threaten our future if left unchecked. For example, climate change, viruses impacting human health and livelihood, threats to our food supply, and widening inequality are just a few. Through IBM’s advancements — many of which were achieved by women — in hybrid cloud, artificial intelligence, quantum computing, and other science fields, we have a unique opportunity to make important contributions to society through our diverse teams.

In order to create a culture and climate within and outside of IBM Research that embraces, supports, and enables successful careers in STEM, we need to work together every day to hire, empower, and grow women in STEM fields.

The data is clear: Diverse teams are more innovative and solve problems faster. As part of my commitment to tackling the issue of empowering women in science, I’m proud to serve as one of the executive sponsors of the IBM Research Diversity & Inclusion Council, which is responsible for ensuring IBM Research is leveraging our innovation and influence to make meaningful changes in support of diversity, inclusion, and racial justice.

To celebrate Women’s History Month, we will spotlight women researchers and scientists from various disciplines, experiences, and tenures at IBM Research and share their insights about their role at IBM, career journeys, inspirations, and guidance for fellow women in STEM.

Source: ibm.com

Continue reading