Machine Learning

How to Improve Convergence and Performance of Deep Feed-Forward Neural Networks via a Simple Model Configuration

Image for post
Image for post
Photo by Martin Sanchez on Unsplash

Problem Statement

Training deep neural networks can be a challenging task, especially for very deep models. A major part of this difficulty is due to the instability of the gradients computed via backpropagation. In this post, we will learn how to create a self-normalizing deep feed-forward neural network using Keras. This will solve the gradient instability issue, speeding up training convergence, and improving model performance.

Disclaimer: This article is a brief summary with focus on implementation. Please read the cited papers for full details and mathematical justification (link in sources section).

Background

In their 2010 landmark paper, Xavier Glorot and Yoshua Bengio provided invaluable insights concerning the difficulty of training deep neural networks. …


Machine Learning Engineering Tips

A Brief Primer on Dataset Processing for Machine Learning Engineering at Scale

Image for post
Image for post
Photo by Claudio Testa on Unsplash

This post is heavily inspired from an exercise of chapter 12 in Hands-On Machine Learning with Scikit-Learn, Keras & Tensorflow. I highly recommend this book.

Problem Statement

Working with very large datasets is a common scenario in Machine Learning Engineering. Usually, these datasets will be too large to fit in memory. This means the data must be retrieved from disk on the fly during training.

Disk access is multiple orders of magnitude slower than memory access, so efficient retrieval is of high priority.

Fortunately, TensorFlow’s tf.data API provides a simple and intuitive interface to load, preprocess, and even prefetch data.

In this post, we will learn how to create a simple yet powerful input pipeline to efficiently load and preprocess a dataset using the tf.data API. …

About

Jonathan Quijas

Hi, I’m Jonathan. I’m interested in Bayesian methods, causal reasoning, and Applied Machine Learning/Machine Learning Engineering. Feel free to reach out!

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store