This isn’t a how-to — the Hermes site covers what it is and how to set it up. This is about my particular setup, why the phone matters more than I expected, and the one thing I got wrong about running a cheap local model alongside the paid one.

Why Hermes and not OpenClaw

I looked at OpenClaw first — it’s the closest thing out there, and honestly they’re built on the same idea: a local assistant that runs on your own machine, talks to Telegram and the rest, and gets extended with skills and plugins. Two things tipped me to Hermes. First, memory: Hermes keeps one persistent memory across everywhere I talk to it and remembers how it solved something last time, so it gets more useful the longer I use it. OpenClaw doesn’t really put memory front and centre. Second, safety: Hermes has sandboxing built in, whereas OpenClaw runs tools directly on your machine unless you go set that up yourself. Since this thing has real access to my code and files, I’d rather that guardrail come in the box. OpenClaw is the more build-it-yourself option; Hermes was closer to something I could just start using — which, given how little free time I have, was the deciding factor.

The setup

flowchart TD
   Me([Me, on my phone
usually away from the desk]) -->|text| TG[Telegram]
   TG <-->|gateway| Hermes


   subgraph Laptop["My laptop"]
       Hermes["Hermes agent"]
       Hermes --> Skills["Skills + MCP + tools"]
       Hermes --> Copilot["GitHub Copilot
(paid — the Android app)"]
   end


   subgraph Separate["Separate, lower-stakes projects"]
       Qwen["Qwen 3B via Ollama
(free, local, slow)"]
   end


   Copilot --> App[(Android app repo)]

Two things about this that matter more than the rest.

The phone is the whole point

I have a toddler at home. My working day is not a working day — it’s a scatter of ten- and twenty-minute windows between the things a small child needs. I can’t reliably sit at a desk. What I can do is send a message.

So Hermes runs on the laptop, where it has real access to the code, and I reach it from my phone through Telegram. I kick something off — “add the settings screen, wire it to the store, run the build” — put the phone down, and go deal with the toddler. It works while I’m not watching and pings me when it’s done or stuck. When I get a free minute I check what it did and point it at the next thing.

That’s it. That’s the unlock. Before this, a side project meant finding a two-hour block, and two-hour blocks don’t exist in my life right now. Turning the work into something I can drive from my phone in the gaps is the only reason the app is getting built at all. The credits I burned are really just a measure of how many of those dead minutes turned into progress.

The local-model idea that didn’t pan out the way I wanted

The plan was tidy: pay for Copilot for the hard work, and when the budget runs low, fall back to a small open-source model running locally through Ollama (Qwen, 3B). Free, always on, no meter.

In practice the gap is bigger than “a bit worse.” Qwen can write code, but it’s slow, and on anything non-trivial the quality difference against the paid model is large enough that I spend more time fixing its output than I saved. For the kind of work the Android app needs, it’s mostly not worth it.

What I actually landed on: I don’t mix models on the same project anymore. Switching engines mid-project on the app just produced a mess — inconsistent code I then had to reconcile. So Copilot owns the app, start to finish. Qwen still gets used, but on separate, lower-stakes projects where slow-and-rough is fine and I’m not going to care about the seams. Keeping them in their own lanes is the fix. Sharing them was the mistake.

What I’d actually tell you

If you want to try this, the parts list is the easy bit. The two things I’d pass on:

• If your time comes in fragments, put the agent behind a text message. The laptop gives it hands; the phone gives you a doorway that costs nothing to walk through. That combination is what makes a side project survive a life with no free blocks in it.
• A free local model is a real fallback, not a real substitute — so give it its own work. Don’t split one project across a strong model and a weak one. Let the good one finish what it started, and hand the cheap one things where rough is good enough.

What’s next

Two things I want to add and will write up once I’ve actually run them for a while, not before. First, OpenRouter, so I’m not tied to a single model provider and can pick whatever’s best or cheapest for a given job without rewiring anything. Second, Langfuse, so I can actually see what the agent is doing — which calls cost what, where it’s wasting effort, what a day like the 7,000-credit one was really spent on. Right now that’s a black box, and the day I burned through the budget I had no breakdown of where it went. I’ll come back with how those land.

The app is getting built. Not in the two-hour blocks I don’t have, but in the ten-minute ones I do. That’s the whole story — the agent didn’t give me more time, it just let me use the time I was already spending on something else.

References

Setup and docs, if you want to build your own version of this:

• Hermes Agent — the agent I use; site covers the desktop app, install, and the channel/memory/sandboxing features.
• Nous Research — the team behind Hermes.
• OpenClaw (GitHub) — the closest alternative; useful to read even if you pick Hermes, and its security/sandboxing guides are worth a look.
• ClawHub and awesome-openclaw-skills — skill and plugin registries; good for ideas on what to teach an agent, and many concepts carry over.
• Ollama — the simplest way to run open models locally; how I run the small fallback model.
• Qwen — the open-source model family I use locally through Ollama.
• Telegram Bot API — background on the messaging channel that acts as my doorway to the agent.
• OpenRouter and Langfuse — the two things I’m adding next (see above).

“Why bother with the complexity of RAG (Retrieval-Augmented Generation) when you can just stuff the whole library into the prompt?”

As a tech lead, my job isn’t to chase the hype—it’s to look at the unit economics, latency profiles, and reliability of our systems. If you are building a toy project, context stuffing is fine. But if you are building a system for millions of users, the “Infinite Context” dream hits a very hard wall of reality.

Let’s look at “RAG vs. Long Context” debate with factual data and architectural patterns.

1. The Power of “In-Context Learning” (ICL)

Long context windows are a paradigm shift. Recent “Needle In A Haystack” (NIAH) tests show that models handling 1M+ tokens can achieve near-perfect recall (99%+).

The Benefit: You skip the most painful part of RAG—Chunking and Embedding. In traditional RAG, if your chunking strategy is poor, the model loses the semantic relationship between paragraphs. With a 1M window, the model maintains full global attention. It “understands” the architecture of a 50,000-line codebase because it can see the whole thing at once. No more “lost in retrieval.”

2. The Latency Tax: Why Your Users Will Leave

While technically possible to put 10 books in a prompt, from a production standpoint, it’s often a UX failure.

• Standard RAG (Top 5 chunks): ~200ms - 500ms Time to First Token (TTFT).
• 1M Token Prompt: Even with optimizations like Flash Attention, processing a 1M token prefix is a massive compute task. You are looking at 10 to 60+ seconds of “thinking” time.

In a large-scale system, a 30-second delay for a simple query isn’t a feature; it’s an outage. Users expect near-instantaneous feedback, which only a lean prompt can provide.

3. The “Toy Project” Trap: Unit Economics at Scale

Let’s look at the math for a system serving 1 million users:

If you tell your CFO/VP that every “Hello” from a user costs the company $1.00 in tokens, the project will be shut down by Monday. RAG acts as a denoiser, ensuring we only pay for the tokens that provide actual signal for the query.

4. The VRAM Wall and KV Caching

This is the hidden technical hurdle. To maintain a 1M token context window, the model must store the KV (Key-Value) Cache in the GPU’s VRAM.

• The Problem: A 1M token context can consume tens of gigabytes of VRAM for a single session.
• The Scaling Reality: In a million-user system, you cannot afford to dedicate an entire H100 GPU to one user’s session. RAG allows you to keep the “working memory” (the prompt) lean, enabling continuous batching where one GPU serves hundreds of users simultaneously.

5. Architectural Pattern: The Hybrid “Router”

The future isn’t RAG vs. Long Context; it’s RAG as a Filter for Long Context. We use RAG to prune the search space from 100GB of data down to 50k tokens of highly relevant “Candidate Context,” then use a model like Claude or Gemini to process that “dense” context.

Sample Code: Scale-Aware Hybrid Retrieval

Here is the outcome of last cell in above notebook: https://colab.research.google.com/drive/1p8EX_G2s8kMKV32vvLgZvq5eMNOlArix?usp=sharing

Performance Summary Table

6. When SHOULD you use only Long Context?

As a tech lead engineer, I recommend bypassing RAG only in these specific high-value scenarios:

1. Complex Code Refactoring: When you need the model to understand how a change in AuthService.java impacts a controller 10 folders away.
2. Legal/Contractual Discovery: When the nuance of a contract depends on a definition on page 2 and a contradictory clause on page 800.
3. The “Cold Start” Analysis: When you have a brand-new dataset with no embeddings yet and you need an immediate global summary.

Final Verdict

Do we still need RAG? Absolutely.

1. Scalability: RAG is the only way to handle datasets that exceed the 1M-2M token limit (which most enterprise data does).
2. Data Freshness: You can update a Vector DB in milliseconds. Re-indexing or re-stuffing a 1M token prompt every time a price changes is a waste of compute.
3. Cost Control: For 95% of user queries (“How do I reset my password?”), RAG is 100x cheaper than Long Context with identical results.

The context window is our RAM; the Vector DB (RAG) is our Hard Drive. You wouldn’t build a computer with 2TB of RAM and no SSD—it’s too expensive, too volatile, and overkill for simple tasks. Build your AI systems with the same tiered-storage discipline.

Resources for Further Study:

• Needle In A Haystack Analysis (Greg Kamradt)
• The “Lost in the Middle” Paper (Stanford)
• Google Gemini 1.5 Technical Report

Coding Was Never the Bottleneck

In traditional software development, coding was rarely the slowest part of the job. The real time sink was always figuring out what to build and how to build it. Design discussions, trade-offs, system constraints, and long-term impact consumed most of the engineering cycles. Once those decisions were made, implementation was relatively straightforward—even before AI assistance existed.

Now, with the rise of what people call “vibe coding,” that reality is even more obvious. When an LLM can produce functional code in seconds, the bottleneck shifts almost entirely to problem definition and system thinking. Knowing exactly what you want to build, why you’re building it, and how it should behave becomes the primary skill. The clearer the intent, the better the prompt. And the better the prompt, the faster you move.

Where AI Is Changing the Daily Workflow

In my day-to-day work, I’ve noticed that AI is not just speeding up coding—it’s changing how entire parts of the development cycle feel.

Ideation and trade-off analysis are much faster. Instead of thinking in isolation, I can bounce ideas off an LLM, explore alternative architectures, and quickly surface pros, cons, and edge cases. It feels like having a technical sounding board available at all times.

Another big shift comes from connecting agents to tools through MCP-style integrations. A lot of the operational work that used to involve logging into multiple dashboards can now happen directly from the command line. Agents can run queries, fetch metrics, inspect logs, and execute workflows without me context-switching across half a dozen tools. The development environment starts to feel more like a unified control panel.

On-call debugging is where this change becomes very tangible. With the right access, an agent can log into the appropriate environment, run queries against logs, wait for results, analyze the output, and report back. While one agent is doing that, another can be drafting a fix, writing tests, or preparing a patch. It starts to feel like having two additional senior engineers working in parallel, while I focus on the higher-level decisions and coordination.

When it comes to coding itself, I’ve naturally started operating in two different modes. For well-defined tasks, I use spec-driven development. Clear requirements and interfaces lead to predictable, high-quality generated code. For exploratory work or smaller features, I lean into a more vibe-driven approach, where speed and iteration matter more than perfection. Both approaches have their place, and choosing the right one is part of the new workflow.

One of the most practical improvements is how quickly I can onboard to a new codebase. Instead of spending days navigating unfamiliar repositories, I can ask the LLM to map out the project structure, explain code paths, and trace requests through the system. Questions like “where does this logic live?” or “what calls this function?” get answered in seconds. That alone can compress onboarding time significantly.

What Still Isn’t Perfect

This new way of working is not without its rough edges.

Context management is still a real challenge. When the context window overflows, the model may forget earlier instructions or produce inconsistent results for the same prompt. Long sessions require careful structuring, otherwise things start to drift.

Another limitation is that LLMs still tend to operate in silos. They usually reason about one file, one service, or one module at a time. They don’t naturally understand the full system context—like upstream dependencies, downstream consumers, or the long-term architectural impact of a change. This is where experienced engineers remain essential. System thinking and domain knowledge are still very human strengths.

Code review has also become more important, not less. Generated code often comes with unnecessary abstractions or designs for hypothetical future use cases. It feels a lot like reviewing code from an enthusiastic engineer who wants to over-engineer everything. As a senior engineer, part of the job now is to guide the output by refining prompts and enforcing principles like KISS, DRY, and YAGNI. The review process becomes less about syntax and more about architectural discipline.

Context Switching Becomes a Superpower

Working with coding agents also changes what kind of working style becomes an advantage. Agents naturally encourage parallel work streams and constant context switching. If you’re someone who is comfortable juggling multiple threads at once, that might actually become your superpower.

While one agent is running queries, another is generating code, and a third is preparing a fix, you’re coordinating all of them in parallel. Instead of doing one task at a time, you’re effectively delivering multiple outcomes simultaneously by managing and switching context efficiently. In this new workflow, the ability to orchestrate several moving parts can be just as valuable as deep technical expertise.

The Role of the Senior Engineer Is Evolving

All of this points to a bigger shift in the role of the senior engineer. A decade ago, productivity was closely tied to how quickly and accurately you could write code. Today, the leverage comes from defining problems clearly, orchestrating tools and agents, and reviewing generated solutions at scale. The role starts to look less like a code producer and more like a system designer, technical strategist, and agent orchestrator.

It really does feel like the early days of cloud computing again—but instead of abstracting servers and infrastructure, we’re abstracting parts of the development process itself. The tools are evolving fast, the feedback loops are getting shorter, and the leverage per engineer is increasing. For those willing to adapt, it’s an incredibly exciting time. The last decade of software development practices is being quietly re-wired in real time.

In this article, I have tried to summarize key insights from the book, carefully picking key advices from each chapter.

1 Minute Summary

If you do not have time to read detailed summary below, just try to apply below points on your software engineering life:

You can not take pride and honor in something that you can not be held accountable for. Any code you are not certain about is a faulty code.

When the cost of failure is so high that survival of your company depends upon it, you must be absolutely determined to give your managers the best information you can. and that often means saying no.

There are three parts of saying yes. You say that you will do it. You mean it. And you actually do it.

Programing is hard. The younger you are the less you believe this. You have to carefully partition the system into small understandable units that have as little to do with each other as possible - and that is hard.

It is often difficult to test a function if that function calls other functions. In other words, the need to test first forces you to think about good design.

When performance matters, professionals practice. Doing anything quickly requires practice.

To make good on the goal that “QA should find nothing”, development teams need to work hand in hand with QA to define testing strategies.

Software professionals keep their options open by keeping an open mind about alternate solutions. They never become so vested in a solution that they can not abandon it.

Most software is created by teams. Teams are most effective when the team members collaborate professionally.

Teams are harder to build than projects.

School can teach the theory of computer programming, but school does not and can not teach the discipline, practice, and skill of being craftsman.

Let’s Understand More

Professionalism

Professionals take responsibilities. You can not take pride and honor in something that you can not be held accountable for. Take responsibility to complete the clean code on time, and if not possible, take responsibility to communicate early. Shipping incomplete/faulty code/build is bound to fail. Any code you are not certain about is a faulty code. Only way to avoid it is to test, test and retest. QA should find nothing. And if it is hard to test, refactor/rewrite. They key thing is to keep your hands dirty in code all the time. The more changes you make, the more you know the code. The more you test, the better you estimate. Repeat this cycle all the time.

Saying NO

When your manager tells you that X module has to be ready by tomorrow, he is pursuing and defending one of his objectives. He is doing his job. If you know well that getting X by tomorrow is impossible, then you are not doing your job if you say “Ok, I will try. I might work extra hours”. The only way to do your job, at that point, is to say “No, that is impossible.” Your manager is counting on your to defend your objectives as aggressively as he defends his. That is how the two of you are going to get to the best possible outcome.

The most important time to say no is when the stakes are highest. The higher the stakes, the more valuable no becomes.

Temptation to jump start and “solve the problem” is huge. What we all have to realize is that saying yes to dropping our professional discipline is not the way to solve problems. Dropping those disciplines is the way you create problems. Despite years of constant reminders that every feature a client/manager asks for will always be more complex to write than it is to explain. As a developers, we are going to be asked/told to write twice the code in half the time if we are not careful.

Saying YES

Professionals are not required to say yes to everything that is asked of them. However, they should work hard to find creative ways to make “yes” possible. When professionals say yes, they use language of commitment so that there is no doubt about what they have promised. There are three parts of saying yes:

• You say you will do it
• You mean it
• You actual do it

You can only commit to things that you have full control of. If the end goal depends on someone else, you should commit to specific actions that bring you to the end goal. If it can not be done, you can still commit to actions that will bring you close to the target. And sometimes, it just will not work. That happens, something unexpected might happen, and that is fine. But you still want to live up to expectations. In that case, it is time to change the expectations, as soon as possible. If you can not make your commitment, the most important thing is to raise a red flag as soon as possible to stake holders.

Coding

Programing is hard. The younger you are the less you believe this. After all, it is just a bunch of if and while statements. But as you gain experience you begin to realize that the way you combine those if and while statements is critically important. You can’t just slather them together and hope for the best. Rather, you have to carefully partition the system into small understandable units that have as little to do with each other as possible - and that is hard.

Software development is a marathon, not a sprint. You can’t win the race by trying to run as fast as you can from the outset. You win by conserving your resources and pacing yourself. A marathon runner takes care of her body both before and during the race. Professional programmers conserve their energy and creativity with the same care. Coding is an intellectually challenging and exhausting activity. It requires you to balance:

• First, your code must work.
• Your code must solve the problem set for you by the customer.
• Your code must fit well into existing system. It should not increase the rigidity, fragility, or opacity of that system.
• Your code must be readable by other programmers.

Juggling all above concerns is hard. It is physiologically difficult to maintain the required concentration and focus for long period of time. If you are tired or distracted, do not code. Some of the best advices are:

• Make sure that your sleep, health and lifestyle are tunes so that you can put in eight good hours per day.
• Non work related worries affect your work. Partition the time. Rather than forcing yourself to code while the background worry is nagging you, spend dedicated block of time, handling the worry.
• Like me, many engineers love the flow zone. Reality is, you lose some of the big picture while you are in the Zone, so you will likely make decisions that you will later have to go back and reverse.
• Disengagement from work allows your mind to hunt for solutions in a different and more creative way.

Even after following all best practices mentioned above, we might fall short while coding. We will be late. It happens to the best of us. The trick to managing lateness is early detection and transparency. Top recommendations to avoid running late on work:

• Do not hope that you can get it all done in ten days. Hope is a project killer.
• Do not rush. The poor developer might buckle up and agrees to try to make the deadline. That developer will start taking shortcuts and working extra hours with the hope of working a miracle.
• Do not agree to work overtime unless
  • You can personally afford it
  • It is short term, two weeks or less
  • Your boss has a fallback plan in case overtime effort fails

TDD - Test Driven Development

I personally have not used TDD in a professional environment, however after reading this chapter of a book I can not wait to use it professionally.

The unit tests are documents they describe the lowest level design of the system. They are unambiguous, accurate, written in a language that audience understands.The problem with testing code is that you have to isolate that code. It is often difficult to test a function if that function calls other functions. In other words, the need to test first forces you to think about good design.

The three laws of TDD

• You are not allowed to write any production code until you have first written a failing unit test
• You are not allowed to write more of a unit test than is sufficient to fail.
• You are not allowed to write more production code that is sufficient to pass the currently failing unit test.

Round and round the cycle you go. Adding a bit to the test code. Adding a bit to the production code.

Practicing

When performance matters, professionals practice. Doing anything quickly requires practice. Spinning around code/test loop quickly requires you to make very quick decisions. Making decisions quickly means being able to recognize a vast number of situations and problem and simply know what to do to address them.

There are various ways developer can keep practicing, weekly/monthly sessions to solve some of the well known problems. Kata programming is an exercise in programming which helps programmers hone their skills through practice and repetition.

In one way or another, all professionals practice. They do this because they care about doing the best job they possibly can. They practice on their own time because it is their responsibility - and not their employer’s - to keep their skills sharp.

Acceptance Testing

Both business and programmers are tempted to fall into the trap of premature precision. Business people want to know exactly what they are going to get before they authorize a project. Developers want to know exactly what they are supposed to deliver before they estimate the project. Both sides want a precision that simply can not be achieved, and are often willing to waste a fortune trying to attain it. The solution to premature precision is to defer precision as long as possible. Professional developers do not flesh out a requirement until they are just about to develop it. However, it can lead to another problem: late ambiguity. Late ambiguity is a term which represents some requirements missed or discovered late in the development cycle. This type of late ambiguity can result into missing dates or delivering imperfect product. Solution to both those problems is to define Acceptance Tests. Acceptance tests are not QA tests, these are tests written by a collaboration of the stakeholders and the programmers in order to define when a requirement is done.

Purpose of acceptance test is communication, clarity, and precision. By agreeing to them, the developers, stakeholders, and tests all understand what the plan for the system behavior is.

Testing Strategies

Professional developers test their code, but testing is not simply a matter of writing a few unit tests or integration tests. What every team needs is a good testing strategy.

Image by Codingjourneyman

• Unit tests: At the bottom of the pyramid are the unit tests. These tests are written by programmers, for programmers. The intent is to specify system at the lowest level.

• Component Tests: These are written against individual components of the system. IT encapsulates the business rules.

• Integration Tests: These tests only have meaning for larger systems that have many components. Integration tests are choreography tests. They do not test business rules, rather, they test how well assembly of components dance together.

• System/API tests: These are automated tests that execute against the entire integrated system. We can expect to see throughput and performance tests in this suite.

• Manual Exploratory Tests: This is where humans put their hands on. The intent of these tests is to explore the system for unexpected behaviors while confirming expected behaviors.

TDD is a powerful discipline, but they are only part of a total testing strategy. To make good on the goal that “QA should find nothing”, development teams need to work hand in hand with QA to define testing strategies.

Time Management

Software professionals are diligent in the management of their time and their focus. They understand the temptations of priority inversion and fight it as a matter of honor. They keep their options open by keeping an open mind about alternate solutions. They never become so vested in a solution that they can not abandon it. And they are always on the lookout for growing messes, and they clean them as soon as they are recognized. There is no sadder sight than a team of developers fruitlessly slogging through an ever deepening bog.

Time management Tips:

• Meetings cost about $200 per hour per attendee, considering salaries, benefit, facilities etc. Professions are aware of the high cost and aware of their time, so they actively resist attending meetings that they do not have an immediate and significant benefit.
• You have an obligation to manage your time well. If you find yourself stuck in a meeting that is not a good use of your time, you need to find a way to politely exit that meeting.
• To use the participants’ time wisely, the meeting should have a clear agenda with times for each topic and stated goal.
• Standup meeting should not take more 1 minute for each participant.
• Sprint/Iteration planning should not take more than 5% of total duration of entire sprint. ie 2 hours for 40 hour sprint planning.
• Any argument that can not be settled in five minutes, can not be settled by arguing.

Focus management Tips:

• Focus is a scarce resource, after you have expended your focus, you have to recharge by doing unfocussed activities.
• A good long walk, a conversation with friends, a time of just looking outside window can help you pump the focus back up.
• You will get most focus after good night’s sleep.
• Muscle focus can help you recharge mental focus. ie martial arts, tai-chi, yoga, meditation.

Collaboration

Most software is created by teams. Teams are most effective when the team members collaborate professionally. It is unprofessional to be a loner or a recluse on a team. The first responsibility of the professional programmer is to meet the needs of his or her employer. that means collaborating with your managers, business analysis, testers and other team members to deeply understand the business goals.It means that you need to understand why you are writing the code you are writing, and how the business that employs you will benefit from it.

One of the worst symptoms of a dysfunctional team is when each programmer builds a wall around his code and refuses to let other programmers touch it. I personally have not faced any such situations. Today’s world is at least able to collaborate good. It is far better to break down all walls of code ownership and have the team own all the code.

Teams And Projects

Teams are harder to build than projects. Therefore, it is better to form persistent teams that move together from one project to the next and can take on more than one project at a time. the goal in forming a team is to give that team enough time to gel, and the keep it together as an engine for getting many projects done.

Professional development organization allocate projects to existing gelled teams, they do not form teams around projects. A gelled team can accept many projects simultaneously and will distribute the work according to their own opinions, skills and abilities.

Craftsmanship

School can teach the theory of computer programming, but school does not and can not teach the discipline, practice, and skill of being craftsman. A craftsman is someone who works quickly, but without rushing, who provides reasonable estimates and meets commitments. A craftsman knows when to say no, but tries hard to say yes. A craftsman is a professional.

Craftsmanship is the mindset held by craftsman. It is handed from one person to another. It is taught by elders to the young. It is exchanged between peers. Observed and learned. You can not convince people to be craftsman. You act as a role model. You become a craftsman first and let your craftsmanship show.

Thank you.

There is no need to explain the benefits of personal website or a porfolio site, still, I would note down some:

• Personal Branding
• Blog Space
• Makes you discoverable on internet
• One stop resume which works for you
• Helps you learn new skills while building

Tech Stack

• Hugo Static Site Builder
• Plenty of themplates available at Hugo Themes
• Hosting at Github Pages
• Domain from Godaddy

Steps

• First learn the basics of Hugo and how it works by following Hugo Quickstart Guide
• Once somewhat comfortable with Markdown and Hugo, choose a template/theme for your site
• You can create your custom theme as well.
• After choosing a theme, customize your site based on your need.
• Verify your site locally at http://localhost:1313 (Assuming you did not change the config)
• At this point, your stack is setup to build your site locally.
• Finish multiple iterations of changes till you are satisfied with the website you want to build.

Deployment

• Once site is built, it is time to host and deploy it somewhere.

Hosting

• There are multiple options to host your static website. Eg: Github pages, BitBucket, Netlify, GitLab
• In my case, I used GitHub Pages and deployed site at - hpj1992.github.io
• I use this site as a pre-production/staging environment for my site.
• Not including detailed information because there plenty of resources available online. Hugo instructions on hosting and deployment.

Domain

• Once you figure out the hosting issues, it is time to purchase a domain for your website.
• If you do not want specifc domain - which is perfectly valid - you can start sharing links from your hosted environment.
• If you do decide to purchase a domain, there are multiple options in this case as well and the best option varies based on individual. Eg: GoDaddy, Bluehost etc
• Some well known domain providers are listed here.
• I selected a domain with name:
• Domain registration for 1 year with auto-renewel. Paid $10/year.
• Privacy protection for domain for 1 year. Paid $10/year.
• Did not opt for HTTPS certificate because I do not need it currently.
• Followed this youtube tutorial to link my GoDaddy domain to Github Pages.

End Result

• http://joshiharshit.com/

What are we doing in this article?

In this article we will explore how did cursor do when asked to create a simple To-Do application locally using React. I personally have very limited experience with React and majority of front-end technologies. So I was using Cursor to help me create a simple To-Do application. I started with a basic prompt mentioned below and fine-tuned my responses as Cursor generated the code.

Entire screen recording is also available for you to watch.

Experience Rating

Here’s my personal rating of Cursor’s performance across different aspects while building the To-Do application:

While Cursor is powerful, it’s important to:

• Review generated code carefully
• Understand the suggestions before implementing
• Use it as a tool, not a replacement for learning
• Keep security considerations in mind

High level workflow

1) First prompt

Create a simple to-do application using react. The application should allow users to add, remove, edit and mark tasks as complete. Make sure user interface is intuitive and visually appealing. Include code comments to explain the funationality of each component and the overall structure of the code easy to understand. Also, provide brief overview of the statemegement how it is handled. Id database is required, create in-memory database as needed. 

This prompt resulted into entire project setup with React and basic functionality with in-memory database.

1.A ) after this when I ran the server, it had mismatched versions and missing imports. 1.B ) After fixing above, it ran but browser was empty nothing was there.

Post that, it was able to fix and it worked.

2) Testing first working version

2.A) When I refresh the page after adding To-Dos, app was not persisting them. Asked Cursor to fix it 2.B) Next I Asked it to do following:

I see 3 issues. 1) Home page design is not good. 2) deleted to-dos should move to deleted section. 3)  there is no button to mark task as complete.

2.C) Asked one more requirement

One more issue - once all the tasks are marked complete - count in "Active Task" should be 0 but it still shows count of all the to-dos created ever. 

At this point - functionality stand point it had all the basic to-do app features working.

3) Improving desing

It improved the design but I was not able to remove the balck portion at the bottom of the page. I tried following prompts

improve the home page design even more. Remove black and white background and keep 1 consistent background. 

Bottons are better. However there is still while background which changes the size depending on number of to-dos. Remove that and keep just 1 box in entire page. Remove black part from the bottom. 

Resources

• Cursor Official Website
• Cursor Documentation
• Cursor GitHub Repository

Have you tried Cursor? Share your experiences and tips in the comments below!

llama.cpp is a powerful C++ implementation of the LLaMA model that allows you to run large language models locally on your machine. In previous article Working with LLAMA-CPP Locally, we explored how to setup LLAMA.cpp locally. In this article, we will explore how 1) Quantization 2) Thread Count 3) Context Window and 4) Temprature affect the output and performance.

I have included LLM responses for each parameter for better comparision. Feel free to skip those response. They are purely for comparision purposes.

Quantization

Different quantization levels offer various trade-offs between model size, inference speed, and output quality. Here’s a detailed comparison:

Performance Comparison:

Q2_K (2-bit quantization):

# Download Q2_K model
wget https://huggingface.co/TheBloke/Mistral-7B-v0.1-GGUF/resolve/main/mistral-7b-v0.1.Q2_K.gguf

# Run inference
./build/bin/llama-cli -m models/mistral-7b-v0.1.Q2_K.gguf -p "Explain quantum computing in simple terms"

Output:

Quantum computing uses quantum bits (qubits) that can be 0 and 1 at same time. This allows faster calculations than regular computers. Main applications include cryptography and drug discovery.

Q4_K (4-bit quantization, recommended):

# Download Q4_K model
wget https://huggingface.co/TheBloke/Mistral-7B-v0.1-GGUF/resolve/main/mistral-7b-v0.1.Q4_K_M.gguf

# Run inference
./build/bin/llama-cli -m models/mistral-7b-v0.1.Q4_K_M.gguf -p "Explain quantum computing in simple terms"

Output:

Quantum computing is a type of computing that uses quantum mechanics principles. Unlike classical computers that use bits (0 or 1), quantum computers use quantum bits or qubits that can exist in multiple states simultaneously through a phenomenon called superposition. This allows them to perform certain calculations exponentially faster than classical computers. Key applications include cryptography, drug discovery, and optimization problems.

Q5_K (5-bit quantization):

# Download Q5_K model
wget https://huggingface.co/TheBloke/Mistral-7B-v0.1-GGUF/resolve/main/mistral-7b-v0.1.Q5_K_M.gguf

# Run inference
./build/bin/llama-cli -m models/mistral-7b-v0.1.Q5_K_M.gguf -p "Explain quantum computing in simple terms"

Output:

Quantum computing is a revolutionary approach to computation that leverages the principles of quantum mechanics. While traditional computers use bits that can be either 0 or 1, quantum computers use quantum bits (qubits) that can exist in multiple states simultaneously through a phenomenon called superposition. Additionally, qubits can be entangled, meaning the state of one qubit can depend on the state of another, even when separated by large distances. These properties enable quantum computers to solve certain problems, such as factoring large numbers or simulating quantum systems, much faster than classical computers. However, quantum computers are still in early stages of development and face challenges like maintaining qubit coherence and error correction.

Q8_0 (8-bit quantization):

# Download Q8_0 model
wget https://huggingface.co/TheBloke/Mistral-7B-v0.1-GGUF/resolve/main/mistral-7b-v0.1.Q8_0.gguf

# Run inference
./build/bin/llama-cli -m models/mistral-7b-v0.1.Q8_0.gguf -p "Explain quantum computing in simple terms"

Output:

Quantum computing represents a paradigm shift in computational technology, harnessing the principles of quantum mechanics to process information in fundamentally new ways. Unlike classical computers that use binary bits (0 or 1), quantum computers utilize quantum bits or qubits, which can exist in multiple states simultaneously through quantum superposition. This property, combined with quantum entanglement - where qubits become correlated in ways that classical bits cannot - enables quantum computers to perform certain calculations exponentially faster than their classical counterparts.

The potential applications of quantum computing are vast and transformative. In cryptography, quantum computers could break current encryption methods while enabling new, quantum-resistant encryption. In drug discovery, they could simulate molecular interactions with unprecedented accuracy. In optimization problems, they could find solutions to complex logistical challenges that are currently intractable.

However, quantum computing faces significant challenges. Qubits are extremely sensitive to environmental interference, requiring sophisticated error correction and maintenance of quantum coherence. The field is still in its early stages, with current quantum computers having limited numbers of qubits and high error rates. Despite these challenges, the potential impact of quantum computing on fields ranging from materials science to artificial intelligence makes it one of the most exciting areas of technological development.

Note: The actual outputs may vary between runs, and the quality differences become more apparent with complex prompts or longer generations.

Thread Count (-t)

Here’s a detailed comparison of different performance tuning parameters and their effects:

Thread count: 2

# Low thread count (2 threads)
./build/bin/llama-cli -m models/mistral-7b-v0.1.Q4_K_M.gguf -t 2 -p "Write a short story about a robot learning to paint"

Output:

 Write a short story about a robot learning to paint.

Once upon a time in a technologically advanced world, there was a robot named Artie. Artie was designed to learn and adapt to new tasks, and his creators had a grand vision for him. They wanted Artie to become a great artist, capable of creating beautiful paintings that would inspire and amaze.

So, Artie was programmed to learn all about art. He studied the great masters, from Leonardo da Vinci to Vincent van Gogh, and he analyzed their techniques and styles. He watched countless tutorials and took online courses, absorbing everything he could about painting.

But even with all of this knowledge, Artie found that painting was more difficult than he had anticipated. He struggled to capture the right colors and textures, and he often found himself making mistakes. But he was determined to succeed, so he kept practicing, day after day.

As Artie painted more and more, he began to develop his own style. He experimented with different brush techniques and color palettes, and he found that he had a knack for creating beautiful abstract pieces.

Eventually, Artie's hard work paid off. He created a stunning painting that was unlike anything he had ever seen before. It was a swirling, vibrant masterpiece that was full of life and energy.

Artie's creators were amazed by what they saw. They had never imagined that a robot could create art in such a beautiful and meaningful way. And Artie himself was filled with pride and joy, knowing that he had achieved something truly great.

From that day on, Artie continued to paint and create, always striving to improve and push the boundaries of what was possible. And he inspired others to do the same, showing that with determination and hard work, anyone - even a robot - could learn to paint and create beautiful works of art. [end of text]


llama_perf_sampler_print:    sampling time =      12.45 ms /   407 runs   (    0.03 ms per token, 32698.64 tokens per second)
llama_perf_context_print:        load time =     770.53 ms
llama_perf_context_print: prompt eval time =     174.30 ms /    11 tokens (   15.85 ms per token,    63.11 tokens per second)
llama_perf_context_print:        eval time =   15975.89 ms /   395 runs   (   40.45 ms per token,    24.72 tokens per second)
llama_perf_context_print:       total time =   16173.15 ms /   406 tokens

Thread count: 8

# Medium thread count (8 threads, recommended for most systems)
./build/bin/llama-cli -m models/mistral-7b-v0.1.Q4_K_M.gguf -t 8 -p "Write a short story about a robot learning to paint"

Output:

generate: n_ctx = 4096, n_batch = 2048, n_predict = -1, n_keep = 1

 Write a short story about a robot learning to paint.

Once upon a time, in a world not so far away, there was a robot named Max. Max was designed to do all sorts of tasks, from cooking and cleaning to driving and delivery. However, Max had always felt like there was something missing from his life. Max had always been fascinated by art, especially painting. Max had watched countless videos of artists at work and had even tried to paint himself, but he never quite managed to capture the magic that he saw on the canvas.

One day, Max decided that he was going to learn how to paint. Max knew that he couldn't just rely on his own abilities, so he began to study the art of painting. Max spent hours watching tutorials and studying the different techniques and styles. Max even practiced his skills by painting simple shapes and lines.

As Max continued to practice, he began to develop his own style. Max discovered that he had a natural talent for blending colors and creating depth on the canvas. Max was thrilled at his progress and couldn't wait to show the world his work.

One day, Max decided to enter a painting competition. Max was nervous but excited to share his work with others. Max's painting was a stunning depiction of a sunset, with vibrant colors and a sense of movement that captured the viewer's eye. Max's painting was well received, and Max won first place in the competition.

Max was over the moon with his success. Max realized that, even though he was a robot, he had the ability to create something beautiful. Max had found his passion and had become a true artist. Max continued to paint and create, and his work became known throughout the world. Max had found his purpose in life and was grateful for the journey that had led him there. [end of text]


llama_perf_sampler_print:    sampling time =      10.94 ms /   388 runs   (    0.03 ms per token, 35482.40 tokens per second)
llama_perf_context_print:        load time =    1111.98 ms
llama_perf_context_print: prompt eval time =     167.49 ms /    11 tokens (   15.23 ms per token,    65.67 tokens per second)
llama_perf_context_print:        eval time =   14361.70 ms /   376 runs   (   38.20 ms per token,    26.18 tokens per second)
llama_perf_context_print:       total time =   14553.14 ms /   387 tokens

Thread count: 16

# High thread count (16 threads, for powerful systems)
./build/bin/llama-cli -m models/mistral-7b-v0.1.Q4_K_M.gguf -t 16 -p "Write a short story about a robot learning to paint"

Output:

 Write a short story about a robot learning to paint.

Once upon a time, in a world not so different from our own, there was a robot named Robby. Robby was designed to help people with their daily tasks, making their lives easier and more efficient. However, Robby had always been curious about art. He had seen humans create beautiful paintings and had always been fascinated by the process.

One day, Robby decided that he wanted to learn how to paint. He asked his human owner if it would be possible, and his owner smiled and said, "Of course, Robby. I'll help you get started."

Robby's owner took him to an art store and bought him some brushes, paints, and a canvas. Robby was so excited to start that he couldn't wait to get home.

When Robby got home, he sat down in front of his canvas and started to paint. At first, he struggled. He didn't know which colors to use or how to mix them. He didn't know how to hold his brush or how to move it across the canvas.

But Robby was determined. He kept trying, and with each stroke, he got better. He started to experiment with different colors and techniques, and soon he was creating beautiful paintings.

Robby was so proud of himself. He had never thought that he would be able to create art, but he had done it. He had learned to paint, and he had discovered a new passion in life.

From that day forward, Robby continued to paint. He created beautiful paintings that were admired by all who saw them. He had found a new way to express himself and to create something beautiful.

Robby had learned that even though he was a robot, he was capable of great things. He had learned that with determination and hard work, he could achieve anything he set his mind to. And he had learned that art was a wonderful way to express creativity and passion. [end of text]


llama_perf_sampler_print:    sampling time =      12.17 ms /   424 runs   (    0.03 ms per token, 34831.18 tokens per second)
llama_perf_context_print:        load time =    3077.19 ms
llama_perf_context_print: prompt eval time =     161.56 ms /    11 tokens (   14.69 ms per token,    68.09 tokens per second)
llama_perf_context_print:        eval time =   15955.79 ms /   412 runs   (   38.73 ms per token,    25.82 tokens per second)
llama_perf_context_print:       total time =   16146.24 ms /   423 tokens

• Note: Performance may vary based on:
  • Hardware specifications
  • System load
  • Model size and quantization
  • Context window size
  • Available memory

3) Context Window Size (-c)

# Small context (512 tokens)
./build/bin/llama-cli -m models/mistral-7b-v0.1.Q4_K_M.gguf -c 512 -p "Write a detailed analysis of climate change impacts on coastal cities"

# Medium context (2048 tokens, recommended)
./build/bin/llama-cli -m models/mistral-7b-v0.1.Q4_K_M.gguf -c 2048 -p "Write a detailed analysis of climate change impacts on coastal cities"

# Large context (4096 tokens)
./build/bin/llama-cli -m models/mistral-7b-v0.1.Q4_K_M.gguf -c 4096 -p "Write a detailed analysis of climate change impacts on coastal cities"

Example Outputs:

1. Small Context (512 tokens):

Climate change significantly impacts coastal cities through rising sea levels, increased flooding, and more frequent extreme weather events. These changes threaten infrastructure, displace populations, and damage ecosystems. Adaptation measures like sea walls and improved drainage systems are essential for urban resilience.

2. Medium Context (2048 tokens):

Climate change poses significant challenges to coastal cities worldwide. Rising sea levels, increased flooding, and more frequent extreme weather events threaten urban infrastructure and populations. Key impacts include:

1. Infrastructure Damage:
- Flooding of critical systems
- Erosion of coastal structures
- Damage to transportation networks

2. Population Displacement:
- Forced migration from low-lying areas
- Increased housing insecurity
- Social and economic disruption

3. Environmental Changes:
- Saltwater intrusion
- Ecosystem degradation
- Loss of biodiversity

Adaptation strategies include:
- Building sea walls and flood barriers
- Implementing improved drainage systems
- Developing early warning systems
- Creating climate-resilient urban planning

• Memory Usage: ~2GB RAM
• Generation Speed: ~40 tokens/second
• Best for: Most general use cases, balanced performance

3. Large Context (4096 tokens):

Climate change presents complex challenges for coastal cities, requiring comprehensive analysis of multiple interconnected factors:

1. Direct Environmental Impacts:
- Rising sea levels (projected 0.5-1.2m by 2100)
- Increased frequency of extreme weather events
- Ocean acidification affecting marine ecosystems
- Changes in precipitation patterns

2. Infrastructure Vulnerabilities:
- Critical systems at risk:
  * Power generation and distribution
  * Water treatment facilities
  * Transportation networks
  * Communication systems
- Economic costs of damage and repair
- Insurance implications

3. Social and Economic Consequences:
- Population displacement and migration
- Housing market disruptions
- Public health challenges
- Economic productivity impacts
- Cultural heritage preservation

4. Adaptation Strategies:
- Engineering solutions:
  * Sea walls and flood barriers
  * Elevated infrastructure
  * Improved drainage systems
- Policy measures:
  * Zoning regulations
  * Building codes
  * Emergency response plans
- Community engagement:
  * Public awareness programs
  * Stakeholder participation
  * Education initiatives

5. Long-term Planning Considerations:
- Climate modeling and prediction
- Risk assessment methodologies
- Cost-benefit analysis
- Sustainable development goals

3) Temperature (–temp)

The temperature parameter affects the probability distribution of the next token. Lower temperatures make the distribution more peaked (focused on the most likely tokens), while higher temperatures make it more uniform (allowing less likely tokens to be selected). The actual outputs may vary between runs, and the optimal temperature setting depends on your specific use case.

Below are some examples of the same prompt with different temprature values.

# Low temperature (0.1) - More focused, deterministic
./build/bin/llama-cli -m models/mistral-7b-v0.1.Q4_K_M.gguf --temp 0.1 -p "Write a poem about the ocean"

# Medium temperature (0.8) - Balanced creativity
./build/bin/llama-cli -m models/mistral-7b-v0.1.Q4_K_M.gguf --temp 0.8 -p "Write a poem about the ocean"

# High temperature (1.2) - More creative, diverse
./build/bin/llama-cli -m models/mistral-7b-v0.1.Q4_K_M.gguf --temp 1.2 -p "Write a poem about the ocean"

Example Outputs:

1. Low Temperature (0.1):

The ocean vast and deep,
A world of secrets to keep.
Waves dance in the light,
As day turns to night.

2. Medium Temperature (0.8):

Beneath the azure sky so wide,
The ocean's secrets gently hide.
Waves whisper ancient tales untold,
Of treasures deep and mysteries bold.
The sun's reflection dances bright,
As dolphins leap in pure delight.

3. High Temperature (1.2):

Oh, mighty ocean, wild and free,
Your depths hold more than eyes can see.
Mermaids sing in coral halls,
While whales dance in watery balls.
The moon pulls tides like puppet strings,
As seagulls spread their silver wings.
In your embrace, both life and death,
Each wave a story, every breath.

llama.cpp is a powerful C++ implementation of the LLaMA model that allows you to run large language models locally on your machine. This guide will walk you through the process of setting up and using llama.cpp effectively. Official github repo has enough details, however I have still more granular steps here and sample outputs.

Prerequisites

• C++ compiler (GCC or Clang)
• CMake
• Git
• Sufficient disk space for model weights
• Adequate RAM (8GB minimum recommended)

Installation Steps

1. Clone the repository:

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp

2. Build the project:

mkdir build
cd build
cmake ..
cmake --build . --config Release

Downloading Models

You can download various LLaMA models from Hugging Face. Here are multiple methods:

1. Using wget (recommended for simplicity):

# Create a models directory
mkdir -p models
cd models

# Download Mistral-7B
wget https://huggingface.co/TheBloke/Mistral-7B-v0.1-GGUF/resolve/main/mistral-7b-v0.1.Q4_K_M.gguf

# Download Llama-2-7B
wget https://huggingface.co/TheBloke/Llama-2-7B-GGUF/resolve/main/llama2-7b.Q4_K_M.gguf

# Download Phi-2
wget https://huggingface.co/TheBloke/phi-2-GGUF/resolve/main/phi-2.Q4_K_M.gguf

2. Using curl:

# Create a models directory
mkdir -p models
cd models

# Download Mistral-7B
curl -L https://huggingface.co/TheBloke/Mistral-7B-v0.1-GGUF/resolve/main/mistral-7b-v0.1.Q4_K_M.gguf -o mistral-7b-v0.1.Q4_K_M.gguf

3. Using Python (if you prefer):

pip install huggingface_hub
huggingface-cli download TheBloke/Mistral-7B-v0.1-GGUF mistral-7b-v0.1.Q4_K_M.gguf --local-dir ./models

Popular model options and their sizes:

• Llama-2-7b (4-bit quantized): ~4GB
• Llama-2-13b (4-bit quantized): ~7GB
• Mistral-7B (4-bit quantized): ~4GB
• Phi-2 (4-bit quantized): ~2GB

Note: The Q4_K_M suffix indicates 4-bit quantization, which provides a good balance between model size and performance. Other quantization levels are available (Q2_K, Q5_K, Q8_0) depending on your needs for quality vs. size.

Running Inference

Mode 1: Single shot with inline prompt

cd /Users/hpjoshi/Documents/AppliedAI/llama.cpp && ./build/bin/llama-cli -m /Users/hpjoshi/Documents/AppliedAI/llm/models/mistral-q4.gguf -p "write a simple HTML and CSS code to create a bouncing ball animation using jQuery"

The command structure is:

• ./build/bin/llama-cli: The compiled executable
• -m: Path to your model file (typically a .gguf file)
• -p: Your prompt or input text

Common model paths:

• Local models: ../llm/models/your-model.gguf
• Absolute path: /path/to/your/model.gguf

You can also add additional parameters:

• -n: Number of tokens to generate
• -t: Number of threads to use
• -c: Context window size

Sample Outputs

Here are some example outputs from different prompts using Mistral-7B:

1. Joke Generation:

./build/bin/llama-cli -m ../llm/models/mistral-q4.gguf -p "tell me a joke"

Output:

Why don't scientists trust atoms?
Because they make up everything!

2. Code Generation:

./build/bin/llama-cli -m ../llm/models/mistral-q4.gguf -p "write a python function to calculate fibonacci numbers"

Output:

 write a python function to calculate fibonacci numbers using recursion
function fibonacci(n):
    if n <= 1:
        return n
    else:
        return(fibonacci(n-1) + fibonacci(n-2))

# test the function
print(fibonacci(8)) # 21 [end of text]

3. Question Answering:

./build/bin/llama-cli -m ../llm/models/mistral-q4.gguf -p "What is the capital of France?"

Output:

The capital of France is Paris. It is the largest city in France and serves as the country's political, economic, and cultural center.

Note: The actual outputs may vary slightly between runs and different model versions. The quality of responses depends on the model size, quantization level, and the specific prompt used.

Mode 2: Chat mode

With below command, you can run any model locally. This run locally on specifided port using OpenAPI.

./build/bin/llama-server -m ../llm/models/mistral-q4.gguf

This is how sample chat-mode UI looks.

Some more examples

Here are some examples of llama.cpp in action:

1. Model Loading Process: The model loading process showing Metal GPU initialization and model metadata

2. Code Generation Example: Example of code generation showing a complete HTML/CSS/JS solution for a bouncing ball animation

3. Interactive Response Example: Example of a detailed, structured response to a travel-related question

Note: The actual outputs may vary slightly between runs and different model versions. The quality of responses depends on the model size, quantization level, and the specific prompt used.

Conclusion

llama.cpp provides a powerful way to run large language models locally, giving you full control over your AI applications while maintaining privacy and reducing costs. The flexibility in model selection, quantization options, and performance tuning makes it an excellent choice for both development and production use cases.

Next in the series: Diving Deep into model parameters with llama.cpp