loop#1 Hyper-parameters, The Lottery Ticket Hypothesis, and Weight&Biases platform

ML Concept: What are Hyper-parameters?

The goal of ML applications is to create models that can master a task based on a dataset. But how do we know that our models are learning at an optimal rate? To achieve that, we need to regularly tune different aspects of the model and evaluate its performance. Think about the process of learning to play the guitar. With every new melody, we are constantly tuning different aspects of the guitar to make it sound better.

  • Number of Epochs
  • Hidden Layers
  • Hidden Units
  • Activation Functions

ML Research that you might need to know: The Lottery Ticket Hypothesis

The Lottery Ticket Hypothesis was published in 2019 by MIT computer scientists Jonathan Frankle and Michael Carbin. It quickly became one of the most important research in the recent years of ML.

  1. Train the network until it converges.
  2. Prune a fraction of the network.
  3. To extract the winning ticket, reset the weights of the remaining portion of the network to their values from (1) — the initializations they received before training began.
  4. To evaluate whether the resulting network at step (4) is indeed a winning ticket, train the pruned, untrained network and examine its convergence behavior and accuracy.

ML Technology that you might need to know: Weights and Biases

Weights and Biases (W&B) is one of the top platforms in the market that enables the hyper-parameter optimization of ML models.

  • Reports: Save and share reproducible findings
  • Sweeps: Optimize models with hyper-parameter tuning
  • Artifacts: Dataset and model versioning, pipeline tracking

AI Researcher - NLP Practitioner