More on our Medium
Large Language Models (LLMs) such as ChatGPT and LlaMA are advancing rapidly in generative Artificial Intelligence (AI), but their immense size poses significant challenges, such as huge training and inference costs, substantial energy demands, and limitations for on-site deployment. Traditional compression methods such as pruning, distillation, and low-rank approximation focus on reducing the effective number of neurons in the network, while quantization focuses on reducing the numerical precision of individual weights to reduce the model size while keeping the number of neurons fixed. While these compression methods have been relatively successful in practice, there is no compelling reason to believe that truncating the number of neurons is an optimal strategy. In this context, this paper introduces CompactifAI, an innovative LLM compression approach using quantum-inspired Tensor Networks that focuses on the model’s correlation space instead, allowing for a more controlled, refined and interpretable model compression. Our method is versatile and can be implemented with — or on top of — other compression techniques. As a benchmark, we demonstrate that a combination of CompactifAI with quantization allows to reduce a 93% the memory size of LlaMA-2 7B, reducing also 70% the number of parameters, accelerating 50% the training and 25% the inference times of the model, and just with a small accuracy drop of 2% - 3%, going much beyond of what is achievable today by other compression techniques. Our methods also allow to perform a refined layer sensitivity profiling, showing that deeper layers tend to be more suitable for tensor network compression, which is compatible with recent observations on the ineffectiveness of those layers for LLM performance. Our results imply that standard LLMs are, in fact, heavily overparametrized, and do not need to be large at all.
Large Language Models (LLMs) such as ChatGPT and LlaMA are advancing rapidly in generative Artificial Intelligence (AI), but their immense size poses significant challenges, such as huge training and inference costs, substantial energy demands, and limitations for on-site deployment. Traditional compression methods such as pruning, distillation, and low-rank approximation focus on reducing the effective number of neurons in the network, while quantization focuses on reducing the numerical precision of individual weights to reduce the model size while keeping the number of neurons fixed. While these compression methods have been relatively successful in practice, there’s no compelling reason to believe that truncating the number of neurons is an optimal strategy. In this context, this paper introduces CompactifAI, an innovative LLM compression approach using quantum-inspired Tensor Networks that focuses on the model’s correlation space instead, allowing for a more controlled, refined and interpretable model compression. Our method is versatile and can be implemented with — or on top of — other compression techniques. As a benchmark, we demonstrate that CompactifAI alone enables compression of the LlaMA-2 7B model to only 30% of its original size while recovering over 90% of the original accuracy after a brief distributed retraining.
Large financial institutions play an important role in driving society’s sustainability goals. Banks and hedge funds can support the decarbonization of industrial and institutional clients while also influencing capital flows through advice to individual clients. For example, financial institutions can increase the credit availability for corporate energy transition actions. At the individual level, banks and investor advisors can help clients who want to enhance the environmental, social and governmental (ESG) profile of their investments.
Quantum Computing impacts business transformation in firms and requires a new mindset for operational excellence.
Here we introduce an improved approach to Variational Quantum Attack Algorithms (VQAA) on crytographic protocols. Our methods provide robust quantum attacks to well-known cryptographic algorithms, more efficiently and with remarkably fewer qubits than previous approaches. We implement simulations of our attacks for symmetric-key protocols such as S-DES, S-AES and Blowfish. For instance, we show how our attack allows a classical simulation of a small 8-qubit quantum computer to find the secret key of one 32-bit Blowfish instance with 24 times fewer number of iterations than a brute-force attack. Our work also shows improvements in attack success rates for lightweight ciphers such as S-DES and S-AES. Further applications beyond symmetric-key cryptography are also discussed, including asymmetric-key protocols and hash functions. In addition, we also comment on potential future improvements of our methods. Our results bring one step closer assessing the vulnerability of large-size classical cryptographic protocols with Noisy Intermediate-Scale Quantum (NISQ) devices, and set the stage for future research in quantum cybersecurity.
We efficiently simulate IBM's largest quantum processors, Eagle, Osprey, and Condor, using graph-based Projected Entangled Pair States, achieving unprecedented accuracy with simple tensor updates.
Quantum error correction through surface codes, critical for reliable quantum computing, demands efficient decoding algorithms balancing speed, complexity, and accuracy.
Paper by Gianni del Bimbo, Daniel García Guijo and Esperanza Cuenca Gómez.
Case Study by Gianni del Bimbo, Rodrigo Hernández Cifuentes, Esperanza Cuenca Gómez, Daniel García Guijo and Angus Dunnett
The Cheyette model is a quasi-Gaussian volatility interest rate model widely used to price interest rate derivatives such as European and Bermudan Swaptions for which Monte Carlo simulation has become the industry standard.
How quantum-inspired algorithms solve the most complex PDE and machine learning problems to achieve real business advantage now.
Machine learning algorithms, both in their classical and quantum versions, heavily rely on optimization algorithms based on gradients, such as gradient descent and alike.
A Practical Approach by Esperanza Cuenca Gómez and Pablo Martín Ramiro
Recent advances in deep learning have enabled us to address the curse of dimensionality (COD) by solving problems in higher dimensions. A subset of such approaches of addressing the COD has led us to solving high-dimensional PDEs. This has resulted in opening doors to solving a variety of real-world problems ranging from mathematical finance to stochastic control for industrial applications. Although feasible, these deep learning methods are still constrained by training time and memory.
Machine Learning models capable of handling the large datasets collected in the financial world can often become black boxes expensive to run. The quantum computing paradigm suggests new optimization techniques, that combined with classical algorithms, may deliver competitive, faster and more interpretable models.
Deep neural networks (NN) suffer from scaling issues when considering a large number of neurons, in turn limiting also the accessible number of layers. To overcome this, here we propose the integration of tensor networks (TN) into NNs, in combination with variational DMRG-like optimization. This results in a scalable tensor neural network (TNN) architecture that can be efficiently trained for a large number of neurons and layers.
Naively trained AI models can be heavily biased. This can be particularly problematic when the biases involve legally or morally protected attributes such as ethnic background, age or gender. Existing solutions to this problem come at the cost of extra computation, unstable adversarial optimisation or have losses on the feature space structure that are disconnected from fairness measures and only loosely generalise to fairness.
Current universal quantum computers have a limited number of noisy qubits. Because of this, it is difficult to use them to solve large-scale complex optimization problems. In this paper we tackle this issue by proposing a quantum optimization scheme where discrete classical variables are encoded in non-orthogonal states of the quantum system.
Here we show how universal quantum computers based on the quantum circuit model can handle mathematical analysis calculations for functions with continuous domains, without any digitalization, and with remarkably few qubits. The basic building block of our approach is a variational quantum circuit where each qubit encodes up to three continuous variables (two angles and one radious in the Bloch sphere).
In this work, we demonstrate how to apply non-linear cardinality constraints, important for real-world asset management, to quantum portfolio optimization.