How to use Fashion-MNIST dataset in Google Cloud Machine Learning / AI Platform?
Fashion-MNIST is a dataset of Zalando's article images, consisting of a training set of 60,000 examples and a test set of 10,000 examples. Each example is a 28×28 grayscale image, associated with a label from 10 classes. The dataset serves as a direct drop-in replacement for the original MNIST dataset for benchmarking machine learning algorithms, providing a more challenging alternative due to its complexity and variability in fashion-related images.
To practically use the Fashion-MNIST dataset in Google Cloud's AI Platform, one must follow several structured steps that encompass data preparation, model training, deployment, and evaluation. Each of these stages requires a comprehensive understanding of both the dataset and the Google Cloud environment.
Step 1: Setting Up Google Cloud Environment
Before utilizing the dataset, ensure that you have a Google Cloud account. Set up a new project in the Google Cloud Console. Enable billing for your project and activate the Cloud AI Platform API. This setup is important as it allows you to leverage Google's robust infrastructure for machine learning tasks.
1. Create a Google Cloud Project: Navigate to the Google Cloud Console and create a new project. Assign a unique name to your project for easy identification.
2. Enable APIs: Go to the API & Services dashboard and enable the Cloud AI Platform API. This API is essential for deploying machine learning models on Google Cloud.
3. Install Cloud SDK: Download and install the Google Cloud SDK on your local machine. This SDK provides the `gcloud` command-line tool, which is necessary for interacting with your Google Cloud resources.
Step 2: Preparing the Fashion-MNIST Dataset
The Fashion-MNIST dataset can be accessed from various sources, including the official GitHub repository. It is essential to preprocess the dataset to ensure it is in the correct format for training models on Google Cloud.
1. Download the Dataset: The dataset is available in multiple formats, including CSV and NumPy arrays. For TensorFlow users, it can be directly loaded using the `tensorflow.keras.datasets` module.
python
from tensorflow.keras.datasets import fashion_mnist
(train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()
2. Data Preprocessing: Normalize the pixel values of the images to the range [0, 1] by dividing by 255. This step is important for ensuring that the model converges efficiently during training.
python
train_images = train_images / 255.0
test_images = test_images / 255.0
3. Reshape and Augment Data: Depending on the model architecture, you may need to reshape the data. Additionally, consider data augmentation techniques such as rotation, zoom, and horizontal flip to enhance the model's robustness.
Step 3: Model Development
Develop a machine learning model suitable for the Fashion-MNIST dataset. Convolutional Neural Networks (CNNs) are a popular choice due to their efficacy in image classification tasks.
1. Define the Model Architecture: Use TensorFlow or PyTorch to define a CNN model. A typical architecture might include multiple convolutional layers followed by max-pooling layers, and a fully connected dense layer.
python
model = tf.keras.Sequential([
tf.keras.layers.Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)),
tf.keras.layers.MaxPooling2D((2, 2)),
tf.keras.layers.Conv2D(64, (3, 3), activation='relu'),
tf.keras.layers.MaxPooling2D((2, 2)),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(64, activation='relu'),
tf.keras.layers.Dense(10, activation='softmax')
])
2. Compile the Model: Choose an appropriate optimizer, loss function, and metrics. For multi-class classification, `sparse_categorical_crossentropy` is commonly used.
python
model.compile(optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
3. Train the Model: Fit the model on the training data. Use validation data to monitor the model's performance and avoid overfitting.
python
model.fit(train_images, train_labels, epochs=10, validation_data=(test_images, test_labels))
Step 4: Deploying the Model on Google Cloud AI Platform
Once the model is trained, the next step is to deploy it on Google Cloud AI Platform for scalable predictions.
1. Save the Model: Export the trained model to a format compatible with Google Cloud, such as TensorFlow SavedModel.
python
model.save('fashion_mnist_model')
2. Upload the Model to Google Cloud Storage: Use the `gsutil` command-line tool to upload the model to a Google Cloud Storage bucket.
bash
gsutil cp -r fashion_mnist_model gs://your-bucket-name/
3. Create a Model on AI Platform: In the Google Cloud Console, navigate to AI Platform > Models and create a new model. Specify the model name and region.
4. Deploy a Model Version: Create a new version of the model by specifying the Cloud Storage path of the SavedModel. Configure the machine type and scaling options based on your prediction needs.
5. Test the Deployment: Use the AI Platform's prediction service to test the deployed model. You can send HTTP requests with image data to the model endpoint and receive predictions.
python
from google.cloud import aiplatform
project = 'your-project-id'
endpoint_id = 'your-endpoint-id'
location = 'us-central1'
aiplatform.init(project=project, location=location)
endpoint = aiplatform.Endpoint(endpoint_id=endpoint_id)
# Example prediction
response = endpoint.predict(instances=[test_images[0].tolist()])
print(response)
Step 5: Model Evaluation and Iteration
After deployment, it is important to evaluate the model's performance and iterate on the design to improve accuracy and efficiency.
1. Monitor Model Performance: Use Google Cloud's monitoring tools to track model performance metrics such as latency, throughput, and prediction accuracy. This data is invaluable for identifying bottlenecks and areas for improvement.
2. A/B Testing: Conduct A/B testing to compare different model versions. This approach helps in understanding the impact of changes and selecting the best-performing model.
3. Continuous Integration and Deployment (CI/CD): Implement CI/CD practices to automate the deployment of new model versions. This setup ensures that improvements are rapidly delivered to production.
4. Feedback Loop: Establish a feedback loop with end-users to gather insights on model predictions. Use this feedback to fine-tune the model and enhance its relevance to real-world applications.
5. Retraining with New Data: Regularly update the model with new data to maintain its accuracy over time. This practice is particularly important in the fashion industry, where trends and styles evolve rapidly.
The Fashion-MNIST dataset provides a practical use case for deploying image classification models on Google Cloud's AI Platform. By following the outlined steps, one can effectively leverage Google's infrastructure to build, deploy, and maintain scalable machine learning models. This process not only enhances the model's accuracy and performance but also ensures its applicability to real-world scenarios in the fashion industry.
Google frequently updates its AI Platform (as of 2024 evolved into the Vertex AI Platform). If you encounter any issues with these updates, you may also try the following code:
python
import google.auth
import google.auth.transport.requests
import requests
import json
from tensorflow.keras.datasets import fashion_mnist
import numpy as np
# Load and preprocess Fashion MNIST data
(train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()
test_images = test_images / 255.0
# Add channel dimension to all test images
test_images = test_images.reshape(-1, 28, 28, 1)
# Prepare your model and project details
project_id = 'project_id'
model_name = 'modelname'
model_version = 'V1'
region = 'europe-west3'
# AI Platform prediction endpoint URL
url = f'https://{region}-ml.googleapis.com/v1/projects/{project_id}/models/{model_name}/versions/{model_version}:predict'
# Authenticate and get the auth token
credentials, _ = google.auth.default()
auth_req = google.auth.transport.requests.Request()
credentials.refresh(auth_req)
auth_token = credentials.token
# Set up headers for the request
headers = {
'Authorization': f'Bearer {auth_token}',
'Content-Type': 'application/json'
}
class_labels = [
"T-shirt/top", "Trouser", "Pullover", "Dress", "Coat",
"Sandal", "Shirt", "Sneaker", "Bag", "Ankle boot"
]
# Loop through the first 6 test images
for i in range(6):
# Prepare the instance for prediction
instance = test_images[i].tolist()
# Make the request body
data = json.dumps({"instances": [instance]})
# Send the request
response = requests.post(url, headers=headers, data=data)
response_json = response.json()
# Extract the predictions
predicted_probs = response_json['predictions'][0]
# Get the index of the highest probability
predicted_index = np.argmax(predicted_probs)
predicted_label = class_labels[predicted_index]
predicted_probability = predicted_probs[predicted_index]
# Print the result in a more readable format
print(response_json)
print(f"Image {i + 1}: Predicted class: {predicted_label} ({predicted_index}) with probability {predicted_probability:.10f}")
Other recent questions and answers regarding EITC/AI/GCML Google Cloud Machine Learning:
- What is the task of interpreting doodles drawn by players?
- When the reading materials speak about "choosing the right algorithm", does it mean that basically all possible algorithms already exist? How do we know that an algorithm is the "right" one for a specific problem?
- What are the hyperparameters used in machine learning?
- Whawt is the language of programming for machine learning it is Just Python
- How is machine learning applied to the science world?
- How do you decide which machine learning algorithm to use and how do you find it?
- What are the differences between Federated Learning, Edge Computing and On-Device Machine Learning?
- How to prepare and clean data before training?
- What are the specific initial tasks and activities in a machine learning project?
- What are the rules of thumb for adopting a specific machine learning strategy and model?
View more questions and answers in EITC/AI/GCML Google Cloud Machine Learning