If you are new here, Sirji is an open-source AI software development agent. Internally it works as a combination of planner, coder, researcher, and executor agents. We are building Sirji as a VS Code extension.
We launched the Sirji VS Code extension on both the Visual Studio marketplace and the Open VSX repository last week.
The positive feedback so far: The installation is a breeze (under 10 seconds, seriously!), and the UI is quite intuitive. The extension seamlessly integrates with VS Code, making it super easy to use in the local development environment.
Of course, along with the positive feedback, there have been hiccups, problems and frustrations that our early users reported. Keep them coming, please. These are super valuable as we continue building Sirji.
And since we are building Sirji in public, best to share all our challenges publicly as well.
Here’s a compilation of all of Sirji’s challenges.
Leaving things incomplete
We've noticed several recurring issues where Sirji says it is done solving the problem but doesn’t fully implement the expected functionalities. The key problems include:
- Frontend and backend integration: Often, Sirji skips the necessary integration between the frontend and backend components of the projects. This leaves the features non-functional even though Sirji reports completion.
- Frontend implementation issues: In some scenarios, the frontend components of a feature are not implemented at all.
- Linking between files is missing: Sirji frequently neglects to link files correctly. For example, while developing a React App, it usually misses linking files from the main App.js file.
- Assumed changes: There have been instances where Sirji hallucinates and acts as though certain changes or additions have been made to the code without actually implementing them.
GitHub Issue: https://github.com/sirji-ai/sirji/issues/56
Unable to debug web errors
Currently, Sirji cannot detect when a user encounters errors with web pages. This limitation hinders Sirji's autonomous debugging abilities, making it reliant on users to report errors by copying and pasting them as manual feedback. Enhancing Sirji with error detection for web page interactions would greatly improve its self-diagnostic and troubleshooting capabilities.
GitHub Issue: https://github.com/sirji-ai/sirji/issues/57
Inability to recognize its own capability
Sirji often displays the message "Outside of my capability" for tasks that are actually within its capabilities, leading to unnecessary interruptions in workflow.
This issue is particularly notable in these scenarios:
- Installation of databases (like MongoDB)
- Verification of an installation
GitHub Issue: https://github.com/sirji-ai/sirji/issues/58
Invalid message format
Sirji sometimes receives messages in invalid format from LLM. We have an error-catching mechanism in which we again ask LLM to give a response as per the defined message protocol. In this case, even after 2 retries (again calling OpenAI completions API with the feedback that the last message was invalid), LLM doesn’t respond in a proper format. This primarily happens when LLM has to give some information to the user, to act on but this action is missing from the protocol.
GitHub Issue: https://github.com/sirji-ai/sirji/issues/59
Installing system-wide packages without confirming
At times, Sirji installs system-wide packages like Python without seeking user approval. A confirmation step would ensure users are aware and agreeable to changes being made to their system.
GitHub Issue: https://github.com/sirji-ai/sirji/issues/60
Server issues
We're encountering multiple issues with how Sirji handles server processes, particularly starting, restarting, and logging.
Below are the specific problems:
- Server start-up: Occasionally, Sirji forgets to start the server as expected after the solution is complete.
- Server restart issue: Sirji does not automatically restart the server after updating the code. Consequently, the modifications are not reflected.
- Log file overwriting: The logs from running servers are not uniquely preserved. They risk being overwritten by subsequent processes.
GitHub Issue: https://github.com/sirji-ai/sirji/issues/61
Files are created outside of workspace
There is an issue where Sirji occasionally attempts to create files and folders outside the designated workspace. This behavior is concerning because it can interfere with the user's other files. A specific instance of this problem was observed when Sirji navigated to the home directory (~) and started creating files there (oops!).
GitHub Issue: https://github.com/sirji-ai/sirji/issues/62
Can’t use third-party APIs that need keys
Sirji gives up or implements outline code when the third-party APIs need keys. For Sirji to effectively interact with third-party APIs, it needs to ask users to input the necessary API keys. Users should be prompted to enter these keys into an environment variable file, enabling Sirji to securely access them from the generated code.
GitHub Issue: https://github.com/sirji-ai/sirji/issues/63
UI from the 90s
Unless specifically directed, Sirji creates webpages like they are from the 1990s.
GitHub Issue: https://github.com/sirji-ai/sirji/issues/64
Progress steps are buggy
The steps are not always updated to their respective status from ‘not running’ to 'running' to 'completed'. This inconsistency affects the overall task tracking.
GitHub Issue: https://github.com/sirji-ai/sirji/issues/65
Continuing before package installations complete
Sirji does not pause its process for the completion of long-running package installations. This leads to subsequent steps erroneously assuming the necessary packages are installed, causing failures.
GitHub Issue: https://github.com/sirji-ai/sirji/issues/66
…
Any other problems that you encountered?
Our next milestone is going to be about making Sirji reliable. Stay tuned!
