Hello and Welcome 👋 I'm Mike.
This is a blog of my personal interests and projects. Recently, I’ve been interested in understanding the inner workings of the Transformer. The transformer is the neural net architecture behind Chat GPT and modern Natural Language Processing. One of the things I find fascinating about Transformers is how we are able to represent language as higher dimensional vectors then fit 100’s of millions of parameters all to predict the next word.
What is going on in there? How does this work exactly?...
In a pursuit to find out, these posts summarize my attempts to follow the concepts presented by others. Then further explore in my own way to better understand. Maybe you’ve had similar questions. Feel free to reach out and share what you wanted to know or learned.
Some of my other interests include astrophotgraphy and web development.
Transformers
- Neural machine translation with a Transformer and Keras. Part 1: Using a larger dataset
Google's tensorflow tutorial builds a Transformer model from the base layer classes in keras. The data set consited of 50k sentence pairs. I implemented this and then downloaded a larger dataset of 1.5M sentence pairs and explored the impact of the hyperparamters on a Spanish to English translator. In hoping to drive better accuracy at higher dimensions or more attention heads, I found learning rate and batch size mattered more than anything.
April 30, 2024
- Custom BERT Implementation and IMBD sentiment classification.
Using the transformer built in the previous post I built trained my own BERT implementation. My goal was to use a pretrained BERT layer in the the self attention heads of the translator to get better translation results. But first create my own custom BERT and compare to the results obtained in tensorflw tutorial by google May 15, 2024
Astrophotography
- Flaming Star, Tadpoles, Spider, and Fly.
The constellation Auriga has many interesting deep-sky objects within it, including the Flaming Star, Tadpoles, Spider, and Fly. Over the years I've photographe these seperately as well as from a wide field and mosaics. May 1, 2024
Web Development
- This Blog - Code Highlighting
I follow several you tubers on the topic of web development. Of late, my interest is in Next.js 14. I like the dynamic routes and find it interesting what Vercel is trying to do to make a better developer experience. Next.js has taken some critism of late, nonetheless I am still into it. This blog was based on a tutorial by Dave Gray. Using Tailind Typography out of the box has no formatting of code blocks. Here I showed how to use markdownIt and highlight.js to format code blocks in my blog posts. May 15, 2024
- This Blog! - Notes
just some notes May 31, 2024