Why only use python? It's slow as fck..

[BennyBread]

Hi, I am wondering if it may be an option to code the optimizing / calculation /forecast functions in an more performant language like c++ or c# (the ones I would use) for the time consuming calculations.

I don't understand it fully, but from the 2 hour recording of the install session i've heard that it takes a "long" time to calculate.
This is the reason why I opened this discussion to put the elephant "use an more performant coding language" in the room..
In general I think is's an very interesting project!

[NormannK]

You’re absolutely right—performance is an important topic, and using C/++/# for the heavy calculations could definitely make a difference since the languages are known for their speed.

The optimization currently takes around 45 seconds on an older CPU core, so while that might seem a bit long, it’s still reasonable for the hourly data we’re working with right now. Since we’re only dealing with hourly updates at the moment, it’s not a huge issue, if we start running things more often, say every 10 minutes in the future, it could become more relevant.

One thing to keep in mind is that the current Python code and parameters haven’t really been optimized yet. Before jumping to another language, it might be worth seeing how much we can improve with Python itself. There are lots of ways to speed things up, like using Cython or parallelizing tasks, which could give us a nice boost while keeping things flexible.

In the past I often used DEAP for Python simply bc its very fast to try different algorithms.

But if you want to give it a try and solve the optimizing in an other language that would be interesting!

[BennyBread]

I understand. But I think it is easier to convert the unoptimized pythin functions to another language as if they are optimized.

Can you tell me where in the source I can look as an starting point for those time consuming functions?

I could then possibly translate them to a small let's say C# commandline app.
Of course it would be imho better to make the whole optimizing process in one language.
As far as I've learned from the video it's only some mixed data in => "magic" => json data out and this as an REST api to be used by node red?
It would be very easy to make an docker container out of the c# progam then.

[drbacke]

class_optimize is the "main" optimization loop.
But I dont think, that C++ conversions will help a lot. The main problem ist the number of iterations and thats more an math problem.
At this project stage, I will prefer to leave the code as readable as possible. So maybe you should grab yourself another Issue to solve.

[NormannK]

Your post prompted me to review the code, and I discovered several low hanging fruits for improvement in the class_ems.py. I managed to reduce the calculation time by 50%, and I’ll be pushing a PR later today. (down to 22 seconds)

Additionally, the overall optimization that includes the EMS can be found in class_optimization.py. I personally enjoy tackling performance optimization challenges, and I often invest more time in them than I should. However, I believe the project currently faces more significant issues. We should prioritize improving the overall state of the code before looking into performance optimizations.

[drbacke]

Performance mainly depends on the number of iterations. Reducing them is important if reached a later stage. Numpy is quite perfomant for the most part. If so, you should see if you can find a faster implementation.
I would only tackle this when we have reached a later stage
At the moment, readability and code refactoring, documentation are the order of the day. Unfortunately, I wasn't very tidy here, so now there's a lot of work to do afterwards. But it's my first opensource project.
Then we should consider how this project can be integrated into frameworks such as Homeassistant. Simply translating something to c++ doesn't help.

[eckelj]

I think python is ok for a first start but lacks the type safety of languages like go. I wouldn't move to c/c++ or more complex languages. go could be beneficial as it is also much easier to deploy and resilient in its maintenance. It has a very nice toolchain to improve code quality out of the box and is super easy to get into compared to c/c++ or rust.