Ever since the first compound microscope by Zacharias Janssen in the late 16th century, scientists have been inventing new types of microscopes for various tasks. Inventing a novel architecture demands years, if not decades, worth of scientific experience and creativity. Here, we introduce Differentiable Microscopy (∂µ), a machine learning-based design paradigm, to aid scientists design new microscope architectures. first models a physics-based optical system with trainable optical elements. Using pre-acquired data, we then train the model end-to-end for a task of interest. First, we present two all-optical quantitative phase microscope (QPM) designs that require no computational post-reconstruction. Going a step further, we propose to acquire images in a compressed form such that more information can be transferred beyond the existing electronic hardware bottleneck. To this end, we present a learnable optical compression-decompression framework that learns content-specific features. The proposed system consists of learnable optical feature extractors as image compressors, where the intensity representation produced by these networks is then captured by the imaging sensor. Finally, a reconstruction network running on electronic hardware decompresses the QPM images. We believe that this setup opens up a new pathway for achieving end-to-end optimized (i.e., optics and electronic), compact QPM systems that provide unprecedented throughput improvements. The proposed differentiable microscopy framework supplements the creative process of designing new optical systems and would perhaps lead to unconventional but better optical designs.