譯者 | 崔皓
審校? | 孫淑娟
開篇
Transformer模型通過在語言翻譯、文本分類和序列建模中提供卓越的性能，徹底改變了自然語言處理（NLP）任務。?
Transformer的架構是基于一種自我關注機制，它允許序列中的每個元素關注其他元素并處理輸入序列的堆疊編碼器。?
本文將演示如何建立一個Transformer模型來生成新的雞尾酒配方。文中將使用Cocktail DB數據集，該數據集包含了成千上萬種雞尾酒的信息，包括它們的成分以及配方。?
下載Cocktail DB數據集?
首先，我們需要下載并預處理Cocktail DB數據集。我們將使用Pandas庫來完成這一工作。?
import pandas as pd
url = 'https://www.thecocktaildb.com/api/json/v1/1/search.php?s=' cocktail_df = pd.DataFrame() for i in range(1, 26): response = pd.read_json(tr(i)) cocktail_df = pd.concat([cocktail_df, response['drinks']], ignore_index=True)
預處理數據集?
cocktail_df = cocktail_df.dropna(subset=['strInstructions']) cocktail_df = cocktail_df[['strDrink', 'strInstructions', 'strIngredient1', 'strIngredient2', 'strIngredient3', 'strIngredient4', 'strIngredient5', 'strIngredient6']] cocktail_df = cocktail_df.fillna('')接下來，我們需要使用標記器對雞尾酒菜譜進行標記和編碼。?
將TensorFlow數據集導入為tfds。?
定義標記器和詞匯量大小?
tokenizer = tfds.features.text.SubwordTextEncoder.build_from_corpus( (text for text in cocktail_df['strInstructions']), target_vocab_size=2**13)
定義編碼函數?
def encode(text): encoded_text = tokenizer.encode(text) return encoded_text
將編碼函數應用到數據集上?
cocktail_df['encoded_recipe'] = cocktail_df['strInstructions'].apply(encode)
定義菜譜的最大長度?
MAX_LEN = max([len(recipe) for recipe in cocktail_df['encoded_recipe']] )?
有了標記化的雞尾酒菜譜，就可以定義轉化器解碼層了。Transformer解碼器層由兩個子層組成：masked multi-head self-attention 層和point-wise feed-forward 層。?
import tensorflow as tf from tensorflow.keras.layers import LayerNormalization, MultiHeadAttention, Dense
class TransformerDecoderLayer(tf.keras.layers.Layer): def init(self, num_heads, d_model, dff, rate=0.1): super(TransformerDecoderLayer, self).init()
self.mha1 = MultiHeadAttention(num_heads, d_model)
self.mha2 = MultiHeadAttention(num_heads, d_model)
self.ffn = tf.keras.Sequential([
Dense(dff, activation='relu'),
Dense(d_model)
])
self.layernorm1 = LayerNormalization(epsilon=1e-6)
self.layernorm2 = LayerNormalization(epsilon=1e-6)
self.layernorm3 = LayerNormalization(epsilon=1e-6)
self.dropout1 = tf.keras.layers.Dropout(rate)
self.dropout2 = tf.keras.layers.Dropout(rate)
self.dropout3 = tf.keras.layers.Dropout(rate)
def call(self, x, enc_output, training, look_ahead_mask):
attn1 = self.mha1(x, x, x, look_ahead_mask)
attn1 = self.dropout1(attn1, training=training)
out1 = self.layernorm1(x + attn1)
attn2 = self.mha2(enc_output, enc_output, out1, out1, out1)
attn2 = self.dropout2(attn2, training=training)
out2 = self.layernorm2(out1 + attn2)
ffn_output = self.ffn(out2)
ffn_output = self.dropout3(ffn_output, training=training)
out3 = self.layernorm3(out2 + ffn_output)
return out3
在上面的代碼中，TransformerDecoderLayer類需要四個參數：masked multi-head self-attention層的個數、模型的維度、point-wise feed-forward層的單元數和dropout率。?
調用方法定義了解碼器層的前向傳遞，其中x是輸入序列；enc_output是編碼器的輸出；training是一個布爾標志，表示模型是處于訓練還是推理模式；look_ahead_mask是一個掩碼，防止解碼器關注未來的令牌。?
我們現在可以定義轉化器模型，它由多個堆疊的轉化器解碼器層和一個密集層組成，密集層將解碼器的輸出映射到詞匯量上。?
從tensorflow.keras.layer導入Input?
input_layer = Input(shape=(MAX_LEN,))
定義Transformer解碼器層
NUM_LAYERS = 4 NUM_HEADS = 8 D_MODEL = 256 DFF = 1024 DROPOUT_RATE = 0.1
decoder_layers = [TransformerDecoderLayer(NUM_HEADS, D_MODEL, DFF, DROPOUT_RATE) for _ in range(NUM_LAYERS)]
定義輸出層
output_layer = Dense(tokenizer.vocab_size)
連接層?
x = input_layer look_ahead_mask = tf.linalg.band_part(tf.ones((MAX_LEN, MAX_LEN)), -1, 0) for decoder_layer in decoder_layers: x = decoder_layer(x, x, True, look_ahead_mask) output = output_layer(x)
定義模型?
model = tf.keras.models.Model(inputs=input_layer, outputs=output)
在上面的代碼中，我們定義了輸入層，接受長度為MAX_LEN的填充序列。然后，通過創建一個堆疊在一起的TransformerDecoderLayer對象的列表來定義轉化器解碼層，從而處理輸入序列。?
最后一個Transformer解碼層的輸出通過一個密集層，其詞匯量與標記器中的子詞數量相對應。我們可以使用Adam優化器來訓練模型，并在一定數量的epochs后評估其性能。?
定義損失函數?
loss_object = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True, reduction='none')
def loss_function(real, pred): mask = tf.math.logical_not(tf.math.equal(real, 0)) loss_ = loss_object(real, pred)
mask = tf.cast(mask, dtype=loss_.dtype)
loss_ *= mask
return tf.reduce_mean(loss_)
定義學習率時間表?
class CustomSchedule(tf.keras.optimizers.schedules.LearningRateSchedule): def init(self, d_model, warmup_steps=4000): super(CustomSchedule, self).init()
self.d_model = tf.cast(d_model, tf.float32)
self.warmup_steps = warmup_steps
def __call__(self, step):
arg1 = tf.math.rsqrt(step)
arg2 = step * (self.warmup_steps**-1)
return tf.math.rsqrt(self.d_model) * tf.math.minimum(arg1, arg2)
定義優化器?
LR = CustomSchedule(D_MODEL) optimizer = tf.keras.optimizers.Adam(LR, beta_1=0.9, beta_2=0.98, epsilon=1e-9)
定義準確度指標?
train_accuracy = tf.keras.metrics.SparseCategoricalAccuracy(name='train_accuracy')
定義訓練步驟函數?
@tf.function def train_step(inp, tar): tar_inp = tar[:, :-1] tar_real = tar[:, 1:]
look_ahead_mask = tf.linalg.band_part(tf.ones((tar.shape[1], tar.shape[1])), -1, 0) look_ahead_mask = 1 - look_ahead_mask
with tf.GradientTape() as tape: predictions = model(inp, True, look_ahead_mask) loss = loss_function(tar_real, predictions)
gradients = tape.gradient(loss, model.trainable_variables) optimizer.apply_gradients(zip(gradients, model.trainable_variables))
train_accuracy.update_state(tar_real, predictions)
return loss Train the model EPOCHS = 50 BATCH_SIZE = 64 NUM_EXAMPLES = len(cocktail_df)
for epoch in range(EPOCHS): print('Epoch', epoch + 1) total_loss = 0
for i in range(0, NUM_EXAMPLES, BATCH_SIZE): batch = cocktail_df[i:i+BATCH_SIZE] input_batch = tf.keras.preprocessing.sequence.pad_sequences(batch['encoded_recipe'], padding='post', maxlen=MAX_LEN) target_batch = input_batch
loss = train_step(input_batch, target_batch)
total_loss += loss
print('Loss:', total_loss) print('Accuracy:', train_accuracy.result().numpy()) train_accuracy.reset_states
一旦模型被訓練出來，我們就可以通過給模型提供一個種子序列，并通過反復預測來生成新的雞尾酒配方。Shapetokentar：產生Shapee序列末尾的標記。?
def generate_recipe(seed, max_len):
encoded_seed = encode(seed) for i in range(max_len):
input_sequence = tf.keras.preprocessing.sequence.pad_sequences([encoded_seed],
padding='post', maxlen=MAX_LEN) predictions = model(input_sequence, False, None)
predicted_id = tf.random.categorical(predictions[:, -1, :], num_samples=1)
if predicted_id == tokenizer.vocab_size:
break encoded_seed = tf.squeeze(predicted_id).numpy().tolist()
recipe = tokenizer.decode(encoded_seed)
return recipe
總結?
總之，Transformer是一個強大的序列建模工具，可以用于NLP以外的廣泛應用。?
通過遵循本文所述的步驟，可以建立一個Transformer模型來生成新的雞尾酒配方，展示了Transformer架構的靈活性和通用性。?
譯者介紹?
崔皓，51CTO社區編輯，資深架構師，擁有18年的軟件開發和架構經驗，10年分布式架構經驗。?
原文標題：??Cocktail Alchemy: Creating New Recipes With Transformers??，作者：Pavan madduru
