Helion


Szczegóły ebooka

Transformers for Natural Language Processing - Second Edition

Transformers for Natural Language Processing - Second Edition

Denis Rothman

Kup pozycję na ebookpoint.pl

Transformers are...well...transforming the world of AI. There are many platforms and models out there, but which ones best suit your needs?

Transformers for Natural Language Processing, 2nd Edition, guides you through the world of transformers, highlighting the strengths of different models and platforms, while teaching you the problem-solving skills you need to tackle model weaknesses.

You'll use Hugging Face to pretrain a RoBERTa model from scratch, from building the dataset to defining the data collator to training the model.

If you're looking to fine-tune a pretrained model, including GPT-3, then Transformers for Natural Language Processing, 2nd Edition, shows you how with step-by-step guides.

The book investigates machine translations, speech-to-text, text-to-speech, question-answering, and many more NLP tasks. It provides techniques to solve hard language problems and may even help with fake news anxiety (read chapter 13 for more details).

You'll see how cutting-edge platforms, such as OpenAI, have taken transformers beyond language into computer vision tasks and code creation using Codex.

By the end of this book, you'll know how transformers work and how to implement them and resolve issues like an AI detective!

  • Preface
    • Who this book is for
    • What this book covers
    • To get the most out of this book
    • Get in touch
  • What are Transformers?
    • The ecosystem of transformers
      • Industry 4.0
      • Foundation models
        • Is programming becoming a sub-domain of NLP?
        • The future of artificial intelligence specialists
    • Optimizing NLP models with transformers
      • The background of transformers
    • What resources should we use?
      • The rise of Transformer 4.0 seamless APIs
      • Choosing ready-to-use API-driven libraries
      • Choosing a Transformer Model
      • The role of Industry 4.0 artificial intelligence specialists
    • Summary
    • Questions
    • References
  • Getting Started with the Architecture of the Transformer Model
    • The rise of the Transformer: Attention is All You Need
      • The encoder stack
        • Input embedding
        • Positional encoding
        • Sublayer 1: Multi-head attention
        • Sublayer 2: Feedforward network
      • The decoder stack
        • Output embedding and position encoding
        • The attention layers
        • The FFN sublayer, the post-LN, and the linear layer
    • Training and performance
    • Tranformer models in Hugging Face
    • Summary
    • Questions
    • References
  • Fine-Tuning BERT Models
    • The architecture of BERT
      • The encoder stack
        • Preparing the pretraining input environment
        • Pretraining and fine-tuning a BERT model
    • Fine-tuning BERT
      • Hardware constraints
      • Installing the Hugging Face PyTorch interface for BERT
      • Importing the modules
      • Specifying CUDA as the device for torch
      • Loading the dataset
      • Creating sentences, label lists, and adding BERT tokens
      • Activating the BERT tokenizer
      • Processing the data
      • Creating attention masks
      • Splitting the data into training and validation sets
      • Converting all the data into torch tensors
      • Selecting a batch size and creating an iterator
      • BERT model configuration
      • Loading the Hugging Face BERT uncased base model
      • Optimizer grouped parameters
      • The hyperparameters for the training loop
      • The training loop
      • Training evaluation
      • Predicting and evaluating using the holdout dataset
      • Evaluating using the Matthews Correlation Coefficient
      • The scores of individual batches
      • Matthews evaluation for the whole dataset
    • Summary
    • Questions
    • References
  • Pretraining a RoBERTa Model from Scratch
    • Training a tokenizer and pretraining a transformer
    • Building KantaiBERT from scratch
      • Step 1: Loading the dataset
      • Step 2: Installing Hugging Face transformers
      • Step 3: Training a tokenizer
      • Step 4: Saving the files to disk
      • Step 5: Loading the trained tokenizer files
      • Step 6: Checking resource constraints: GPU and CUDA
      • Step 7: Defining the configuration of the model
      • Step 8: Reloading the tokenizer in transformers
      • Step 9: Initializing a model from scratch
        • Exploring the parameters
      • Step 10: Building the dataset
      • Step 11: Defining a data collator
      • Step 12: Initializing the trainer
      • Step 13: Pretraining the model
      • Step 14: Saving the final model (+tokenizer + config) to disk
      • Step 15: Language modeling with FillMaskPipeline
    • Next steps
    • Summary
    • Questions
    • References
  • Downstream NLP Tasks with Transformers
    • Transduction and the inductive inheritance of transformers
      • The human intelligence stack
      • The machine intelligence stack
    • Transformer performances versus Human Baselines
      • Evaluating models with metrics
        • Accuracy score
        • F1-score
        • Matthews Correlation Coefficient (MCC)
      • Benchmark tasks and datasets
        • From GLUE to SuperGLUE
        • Introducing higher Human Baselines standards
        • The SuperGLUE evaluation process
      • Defining the SuperGLUE benchmark tasks
        • BoolQ
        • Commitment Bank (CB)
        • Multi-Sentence Reading Comprehension (MultiRC)
        • Reading Comprehension with Commonsense Reasoning Dataset (ReCoRD)
        • Recognizing Textual Entailment (RTE)
        • Words in Context (WiC)
        • The Winograd schema challenge (WSC)
    • Running downstream tasks
      • The Corpus of Linguistic Acceptability (CoLA)
      • Stanford Sentiment TreeBank (SST-2)
      • Microsoft Research Paraphrase Corpus (MRPC)
      • Winograd schemas
    • Summary
    • Questions
    • References
  • Machine Translation with the Transformer
    • Defining machine translation
      • Human transductions and translations
      • Machine transductions and translations
    • Preprocessing a WMT dataset
      • Preprocessing the raw data
      • Finalizing the preprocessing of the datasets
    • Evaluating machine translation with BLEU
      • Geometric evaluations
      • Applying a smoothing technique
        • Chencherry smoothing
    • Translation with Google Translate
    • Translations with Trax
      • Installing Trax
      • Creating the original Transformer model
      • Initializing the model using pretrained weights
      • Tokenizing a sentence
      • Decoding from the Transformer
      • De-tokenizing and displaying the translation
    • Summary
    • Questions
    • References
  • The Rise of Suprahuman Transformers with GPT-3 Engines
    • Suprahuman NLP with GPT-3 transformer models
    • The architecture of OpenAI GPT transformer models
      • The rise of billion-parameter transformer models
      • The increasing size of transformer models
        • Context size and maximum path length
      • From fine-tuning to zero-shot models
      • Stacking decoder layers
      • GPT-3 engines
    • Generic text completion with GPT-2
      • Step 9: Interacting with GPT-2
    • Training a custom GPT-2 language model
      • Step 12: Interactive context and completion examples
    • Running OpenAI GPT-3 tasks
      • Running NLP tasks online
      • Getting started with GPT-3 engines
        • Running our first NLP task with GPT-3
        • NLP tasks and examples
    • Comparing the output of GPT-2 and GPT-3
    • Fine-tuning GPT-3
      • Preparing the data
        • Step 1: Installing OpenAI
        • Step 2: Entering the API key
        • Step 3: Activating OpenAIs data preparation module
      • Fine-tuning GPT-3
        • Step 4: Creating an OS environment
        • Step 5: Fine-tuning OpenAIs Ada engine
        • Step 6: Interacting with the fine-tuned model
    • The role of an Industry 4.0 AI specialist
      • Initial conclusions
    • Summary
    • Questions
    • References
  • Applying Transformers to Legal and Financial Documents for AI Text Summarization
    • Designing a universal text-to-text model
      • The rise of text-to-text transformer models
      • A prefix instead of task-specific formats
      • The T5 model
    • Text summarization with T5
      • Hugging Face
        • Hugging Face transformer resources
      • Initializing the T5-large transformer model
        • Getting started with T5
        • Exploring the architecture of the T5 model
      • Summarizing documents with T5-large
        • Creating a summarization function
        • A general topic sample
        • The Bill of Rights sample
        • A corporate law sample
    • Summarization with GPT-3
    • Summary
    • Questions
    • References
  • Matching Tokenizers and Datasets
    • Matching datasets and tokenizers
      • Best practices
        • Step 1: Preprocessing
        • Step 2: Quality control
        • Continuous human quality control
      • Word2Vec tokenization
        • Case 0: Words in the dataset and the dictionary
        • Case 1: Words not in the dataset or the dictionary
        • Case 2: Noisy relationships
        • Case 3: Words in the text but not in the dictionary
        • Case 4: Rare words
        • Case 5: Replacing rare words
        • Case 6: Entailment
    • Standard NLP tasks with specific vocabulary
      • Generating unconditional samples with GPT-2
      • Generating trained conditional samples
      • Controlling tokenized data
    • Exploring the scope of GPT-3
    • Summary
    • Questions
    • References
  • Semantic Role Labeling with BERT-Based Transformers
    • Getting started with SRL
      • Defining semantic role labeling
        • Visualizing SRL
      • Running a pretrained BERT-based model
        • The architecture of the BERT-based model
        • Setting up the BERT SRL environment
    • SRL experiments with the BERT-based model
    • Basic samples
      • Sample 1
      • Sample 2
      • Sample 3
    • Difficult samples
      • Sample 4
      • Sample 5
      • Sample 6
    • Questioning the scope of SRL
      • The limit of predicate analysis
      • Redefining SRL
    • Summary
    • Questions
    • References
  • Let Your Data Do the Talking: Story, Questions, and Answers
    • Methodology
      • Transformers and methods
    • Method 0: Trial and error
    • Method 1: NER first
      • Using NER to find questions
        • Location entity questions
        • Person entity questions
    • Method 2: SRL first
      • Question-answering with ELECTRA
      • Project management constraints
      • Using SRL to find questions
    • Next steps
      • Exploring Haystack with a RoBERTa model
      • Exploring Q&A with a GTP-3 engine
    • Summary
    • Questions
    • References
  • Detecting Customer Emotions to Make Predictions
    • Getting started: Sentiment analysis transformers
    • The Stanford Sentiment Treebank (SST)
      • Sentiment analysis with RoBERTa-large
    • Predicting customer behavior with sentiment analysis
      • Sentiment analysis with DistilBERT
      • Sentiment analysis with Hugging Faces models list
        • DistilBERT for SST
        • MiniLM-L12-H384-uncased
        • RoBERTa-large-mnli
        • BERT-base multilingual model
    • Sentiment analysis with GPT-3
    • Some Pragmatic I4.0 thinking before we leave
      • Investigating with SRL
      • Investigating with Hugging Face
      • Investigating with the GPT-3 playground
        • GPT-3 code
    • Summary
    • Questions
    • References
  • Analyzing Fake News with Transformers
    • Emotional reactions to fake news
      • Cognitive dissonance triggers emotional reactions
        • Analyzing a conflictual Tweet
        • Behavioral representation of fake news
    • A rational approach to fake news
      • Defining a fake news resolution roadmap
      • The gun control debate
        • Sentiment analysis
        • Named entity recognition (NER)
        • Semantic Role Labeling (SRL)
        • Gun control SRL
        • Reference sites
      • COVID-19 and former President Trumps Tweets
        • Semantic Role Labeling (SRL)
    • Before we go
    • Summary
    • Questions
    • References
  • Interpreting Black Box Transformer Models
    • Transformer visualization with BertViz
      • Running BertViz
        • Step 1: Installing BertViz and importing the modules
        • Step 2: Load the models and retrieve attention
        • Step 3: Head view
        • Step 4: Processing and displaying attention heads
        • Step 5: Model view
    • LIT
      • PCA
      • Running LIT
    • Transformer visualization via dictionary learning
      • Transformer factors
      • Introducing LIME
      • The visualization interface
    • Exploring models we cannot access
    • Summary
    • Questions
    • References
  • From NLP to Task-Agnostic Transformer Models
    • Choosing a model and an ecosystem
    • The Reformer
      • Running an example
    • DeBERTa
      • Running an example
    • From Task-Agnostic Models to Vision Transformers
      • ViT Vision Transformers
        • The Basic Architecture of ViT
        • Vision transformers in code
      • CLIP
        • The Basic Architecture of CLIP
        • CLIP in code
      • DALL-E
        • The Basic Architecture of DALL-E
        • DALL-E in code
    • An expanding universe of models
    • Summary
    • Questions
    • References
  • The Emergence of Transformer-Driven Copilots
    • Prompt engineering
      • Casual English with a meaningful context
      • Casual English with a metonymy
      • Casual English with an ellipsis
      • Casual English with vague context
      • Casual English with sensors
      • Casual English with sensors but no visible context
      • Formal English conversation with no context
      • Prompt engineering training
    • Copilots
      • GitHub Copilot
      • Codex
    • Domain-specific GPT-3 engines
      • Embedding2ML
        • Step 1: Installing and importing OpenAI
        • Step 2: Loading the dataset
        • Step 3: Combining the columns
        • Step 4: Running the GPT-3 embedding
        • Step 5: Clustering (k-means clustering) with the embeddings
        • Step 6: Visualizing the clusters (t-SNE)
      • Instruct series
      • Content filter
    • Transformer-based recommender systems
      • General-purpose sequences
      • Dataset pipeline simulation with RL using an MDP
        • Training customer behaviors with an MDP
        • Simulating consumer behavior with an MDP
        • Making recommendations
    • Computer vision
    • Humans and AI copilots in metaverses
      • From looking at to being in
    • Summary
    • Questions
    • References
  • Appendix I Terminology of Transformer Models
    • Stack
    • Sublayer
    • Attention heads
  • Appendix II Hardware Constraints for Transformer Models
    • The Architecture and Scale of Transformers
    • Why GPUs are so special
    • GPUs are designed for parallel computing
    • GPUs are also designed for matrix multiplication
    • Implementing GPUs in code
    • Testing GPUs with Google Colab
    • Google Colab Free with a CPU
      • Google Colab Free with a GPU
    • Google Colab Pro with a GPU
  • Appendix III Generic Text Completion with GPT-2
    • Step 1: Activating the GPU
    • Step 2: Cloning the OpenAI GPT-2 repository
    • Step 3: Installing the requirements
    • Step 4: Checking the version of TensorFlow
    • Step 5: Downloading the 345M-parameter GPT-2 model
    • Steps 6-7: Intermediate instructions
    • Steps 7b-8: Importing and defining the model
    • Step 9: Interacting with GPT-2
    • References
  • Appendix IV Custom Text Completion with GPT-2
    • Training a GPT-2 language model
      • Step 1: Prerequisites
      • Steps 2 to 6: Initial steps of the training process
      • Step 7: The N Shepperd training files
      • Step 8: Encoding the dataset
      • Step 9: Training a GPT-2 model
      • Step 10: Creating a training model directory
      • Step 11: Generating unconditional samples
      • Step 12: Interactive context and completion examples
    • References
  • Appendix V Answers to the Questions
    • Chapter 1, What are Transformers?
    • Chapter 2, Getting Started with the Architecture of the Transformer Model
    • Chapter 3, Fine-Tuning BERT Models
    • Chapter 4, Pretraining a RoBERTa Model from Scratch
    • Chapter 5, Downstream NLP Tasks with Transformers
    • Chapter 6, Machine Translation with the Transformer
    • Chapter 7, The Rise of Suprahuman Transformers with GPT-3 Engines
    • Chapter 8, Applying Transformers to Legal and Financial Documents for AI Text Summarization
    • Chapter 9, Matching Tokenizers and Datasets
    • Chapter 10, Semantic Role Labeling with BERT-Based Transformers
    • Chapter 11, Let Your Data Do the Talking: Story, Questions, and Answers
    • Chapter 12, Detecting Customer Emotions to Make Predictions
    • Chapter 13, Analyzing Fake News with Transformers
    • Chapter 14, Interpreting Black Box Transformer Models
    • Chapter 15, From NLP to Task-Agnostic Transformer Models
    • Chapter 16, The Emergence of Transformer-Driven Copilots
  • Other Books You May Enjoy
  • Index