Hello all,
I am a data center engineer of about 8 years now. I’ve spent the last 3 years or so slowly learning Python(I say slowly not because of my effort, but because learning Python was actually very difficult for me.) I am not an expert in any way shape or form, I understand the concepts of OOP, inheritance, classes, functions, methods, etc and I have found that the python documentation that can be found within the language is usually enough for me to be able to write the programs that I want to write. Very rarely have I had to write programs that have to bypass the GIL, but occasionally, I have created threadpools for applications that are not I/O intensive. What I’m saying is, for most things that I create, performance is enough with Python.
However, I have been inspired by how much love Rust is getting from the people who use Rust. I have tried to find some books for using Rust for network automation and unfortunately I have not been able to find any reputable books.
Most of the “automation” work that I do involves parsing data with regex, restructuring the data, converting the data into a modeled format and transforming something with that data. Does anyone have any common use cases for Rust that might interest me? Has anyone used Rust for network automation tools? With familiarity, can Rust’s intuitiveness match Python’s “from idea to deployment” speed? Or should I only learn Rust if I intend to create applications that need tight performance?
With familiarity, can Rust’s intuitiveness match Python’s “from idea to deployment” speed?
Likely not at the start. Rust can take some time to learn to use it effectively it is not the fastest at throwing shit together quickly.
Or should I only learn Rust if I intend to create applications that need tight performance?
Also no. IMO rusts performance is only a nice by product of the language, yeah it encourages people to try it out, but they don’t stay for the speed. They stay for the tooling and the feeling that once it compiles it will likely just work they way you intended. Rust forces you to think more about correctness and edge cases of your code - which does slow down initial idea to working prototype a bit. But IMO it quickly pays back dividends when you get something into production and it just works with no random crashing in the middle of the night.
It also makes refactoring a joy to do, where I hate refactoring in languages like python as you never know what you might have broken - likely something that you will find out only after you have deployed to production. Instead the compiler catches basically every thing that you missed before it will even let you run the code - so those edge cases are taken care of when you are developing, not after it fails in production.
I also find it is very nice with data processing/transformation as it lets you use functional coding styles which tend to lean towards clearer/easier to read series of data transforms.
If you want to learn it I would recomend starting out with the offical book, but you might also find zero to production or datawithrust interesting reads as well.
Likely not at the start. Rust can take some time to learn to use it effectively it is not the fastest at throwing shit together quickly.
To add, Python lets you make a new file, write up a quick script, and start running it. You even have a REPL environment prepared for you to start throwing code at and see what happens. Rust is nothing like this (though some “script runners” exist for Rust). You’ll usually end up creating a new folder, creating a
Cargo.toml, then asrcdirectory and amain.rsfile. Now you can start writing code, but the Python developer has already ran their code a few times and iterated on it a little.For experienced users, development speed in Rust starts to pick up after the initial project setup and once the basic “boilerplate” (this depends per domain, could be arg parsing, reading a config file, setting up telemetry, etc) for their specific application type has been created, in my experience. For quickly throwing together a small script, a developer equivalently experienced in Python and Rust will likely find Python faster to use, but when looking at mid to large sized codebases, that could flip due to how strict Rust is and how that prevents problems over the long term.
It only took me ~2 weeks of playing with Rust before it became my scripting language of choice over Python (which I had been using casually for ~5 years by that point).
The initial setup for Rust can be whipped up with
$ cargo init. You’re right that there’s more setup boilerplate because of the mandatoryCargo.tomland directory structure, butcargo initwill provide all that in a snap.As for the domain specific boilerplate, I actually find that Rust is better at that than Python in almost all cases. I feel that Rust’s
clapis much simpler, more reliable, and less boilerplate than Python’s argparse. Python might win in cases where there’s a very mature domain specific package that you need which isn’t available in the Rust ecosystem, but that’s becoming rare as crates.io grows.And then when it comes to the actual “scripting”, very often my IDE’s intellisense can practically fill in the Rust code for me. One keystroke per word and it knows what function I want, or I can quickly scroll through the recommendations until I find what I’m looking for. Meanwhile with Python I always have to consult docs to find any API that isn’t part of the basic standard library. As a result I’ll often get the scripting done faster in Rust than in Python.
It does absolutely take some time to reach that point, though. Most programmers will definitely feel significant discomfort with Rust initially, but that’s just because you need to deprogram your brain from the bad habits that other languages encourage. There’s a tipping point in that deprogramming where all the other languages start to feel uncomfortable because you know it won’t let you write as good quality of code as Rust would.
I can’t confirm the point about autocompletion with Python. When using strictly typed Python (mypy), the suggestions are just as good
Using visual studio code this only happens if the library has thorough type annotation. While that’s becoming more popular, it’s not enforced at the language level so lots of libraries have enormous gaps in the autocomplete.
It’s improved a lot over the past year, but you’re right. Not even the standard library is typed
For quickly throwing together a small script, a developer equivalently experienced in Python and Rust will likely find Python faster to use
I am not sure about that. I have written a few scripts in rust that I managed to do quite quickly. Once you have a project similar to what you have done before setup time is not that long and the more examples of things you have done before lets you get going quite quickly - this is true of any language really. The slowest part I tend to find is learning the libraries you need to use for a given script or doing something you have not quite done before. Which I think in rust it can take a bit longer for these parts - but once you have overcome that hurdle similar scripts are easier to write in the future. Python might win out if you constantly need to write things from scratch with no past examples and always needing to use new unfamiliar libraries. But I find that is not often the case and over time becomes less and less the case.
And for those cases where you can just adapt something you have written before rusts very easy refactoring actually adds a lot of value and helps to speed things up quite a bit. Though it does take quite a lot more knowledge with the language to get to that point that it does with python I think.
yes, I’ve seen how mangled python code can be, some of the code that our automation team uses barely makes use of functions or classes, which has made working on other people’s python code a nightmare. There’s one application that is thousands of lines long that I’m pretty sure I could condense into a few hundred lines.
Perhaps that’s a drawback of people who use python, they are not typically focused on the scalability of their code until it is actually used in prod. I believe for this reason, I would prefer a language that is compiled.
Last question, is using Rust on Windows as difficult as it seems at face value? It looked like to me that for using rust, it’s preferable to use linux or mac, as they don’t require you to install a compiler. For some reason, my org requires me to submit an exception for being able to install rust on my work computer. Is there anything that is inherently risky about using Rust, or is it because once the code is compiled, it cannot be reverse engineered?
I don’t think it is any more risky than using python. All crates are compiled from source locally, nothing is run when you download a crate. So there is an option to inspect things before you run them (but lets face it - almost no one does this in rust or python or any other language). I would guess that the issue you work has is simply that rust is new to them and they are wary of anything that is new.
wow, great response. thank you. I’ll give it a shot!
Context: I am an embedded software engineer. I write a lot of low level code that runs on microprocessors or in OS kernels, as well as networking applications and other things. I write a lot of C, I write some Rust, I write Elixir if I possibly can, I write a lot of Python (I hate C++ with a passion).
I don’t think you want Rust. Python is unbeatable on “idea to deployment” speed. Python’s downsides:
- Painful packaging/distribution if you want to get a load of people who don’t have Python installed to run your thing (e.g the GUI program we currently maintain for talking to our hardware)
- Performance under some circumstances. There are some things that are not quick in Python. They’re not always the things you expect because Python actually drops down to C modules for a lot of the number crunching that you might do. E.g. for ML you are basically using Python to plug a load of bits of fast C code together
Rust is good when you need at least one of:
- High speed
- Control over use of memory
- Low level systems programming (drivers etc.)
- Can’t cope with a Garbage Collector
- Compiling to a microcontroller
If you’re doing one of those and so have become expert in Rust, then it is actually excellent for a lot of other things. E.g. you might build your data processor in it, and then distribution is easy because it’s just a single binary.
One option you might look at is Go. You get a lot of performance, you get good parallelism if you need it, it’s designed to be easy to learn, and it also compiles programs to a single binary for easy distribution.
One more note on learning Rust: what Rust does is front-load the pain. If you write something in another low-level “direct control of memory” language you can often get something going much more easily than Rust because you don’t have to “fight the borrow checker” - it’ll just let you do what you want. In Rust, you need to learn how all the ownership stuff works and what types to use to keep the compiler happy.
But then as your project grows, or does a more unusual thing, or is just handed over to someone who didn’t know the original design idea, Rust begins to shine more and more. Your C/C++/whatever program might start randomly crashing because there’s a case where your pointer arithmetic doesn’t work, or it has a security hole because it’s possible to make a buffer overrun. But in Rust, the compiler has already made you prove that none of that is possible in your program.
So you pay a cost at the start (both at the start of learning, and at the start of getting your program going) but then over time Rust gives you a good return on that investment.
I think you’re right. Despite my bandwagon fandom, I Believe Go is more appropriate for my needs. I think a compiled language with more intuitive parallelism is what I need
Go doesn’t let you control memory, avoid garbage collector, or compile well to a microcontroller, no? It’s been a while since I paid attention.
You’re right - though I don’t know about the ‘control memory’ part. However, the other person is addressing the use-case of ‘network automation’, not of microcontrollers. There it really doesn’t matter what the exact memory layout is, or if GC stops your job for a microsecond. Go is sufficient for that. Go is sufficient even for many web backends and network infrastructure. In fact, much of kubernetes is written in Go.
I use Rust for where one would normally use a shell script. But I have been using it for nearly a decade now (yes, I started before it reached 1.0). I have gotten used to the strict type system and even rely on it to write proper code. I also have a background in hardware - so much of it makes sense to me. Even so, I can’t recommend Rust to a beginner who values productivity. Rust takes a lot of time getting used to. And no matter how you try, you will never be as fast as you are with python or go. It’s not always bad considering what you get in return for the sacrifice. But it’s better to set your expectations straight.
It’s worth noting the garbage collector is optional in Python. If you write your code “right” (avoid unbroken cyclic references) you don’t need it.
I’m usually a little suspicious of a new fancy language - because the language is only a part of the equation. Does it have good tooling and does it have awesome libraries?
I had a preconception that Rust is strong as a language (formally well structured, low shoot-yourself-in-the-foot potential, consistent, predictable) and that the tooling seemed strong (debuggers, editors, code completion, help, test frameworks), but I’ve always thought that it would lag with libraries. I mean compared to something like Python (« Batteries included ») or java, surely it is not yet compatible, right?.
So I chose a few of the less main-stream libraries that I use regularly… and Lo and behold! They exist for Rust, including Couchbase, SQLite, ECDH, DiffMatch. I can’t vouch for the completeness of those libs, but the fact that everything I looked for existed… that’s impressive.
One of rusts biggest issues ATM is a less mature eco system. Especially when compared to something as old as python. Rust does have a light stdlib compared to other languages as well and leans more heavily on its ecosystem to fill the gaps. But common things are already well established and mature and the whole ecosystem is coming along quite well.
Currently the biggest issues are the more fringe areas, like libraries for specific APIs or services or areas people have not quite covered yet. But these are fairly quickly lessening as people write libraries as they need them to fill the gaps. I would say its ecosystem is most there, at least enough for most projects now.
And IMO rust has some very nice ergonomics for some libraries - such as json de/serialization that just don’t/cannot exist in other languages.
I think, one big reason why people are gladly implementing libraries in Rust is that they’re useful even outside the ecosystem.
In languages which need a runtime environment (Python, Java, JS, Go etc.), you’re locked into that ecosystem. Your code will only run inside of that runtime, meaning someone using another language can’t run your code.
Rust does not need a runtime and it can generate libraries in the format of C. Any mature programming language can call into C libraries.
Plus, Rust’s performance means it’s actually quite worthwhile for higher-level languages to call these libraries.To have library portability is a very cool feature. I hadn’t released that this was possible.
I’d say it’s powerful but not intitultive, at least in the sense of matching the intuitions that you already have. What learning it will do for you is make you develop new intuitions, that will make you a better programmer.
Rust would be an excellent fit for the type of work you describe. Assuming I understand the specifics correctly, the
regexandserdecrates would make the parsing + converting pretty effortless and fast.The language itself also works really well for “data pipeline” type programs, thanks to its FP features/iterators and type system.
With familiarity, can Rust’s intuitiveness match Python’s “from idea to deployment” speed?
Yes and no. For experienced developers, the total time from “start” to “finished product” is probably going to be about the same in both. It’s how that time is allocated that really distinguishes them.
Rust is going to make you put in more work up front compared to Python: negotiating with the compiler, getting your types in order, that sort of thing. The benefit is that what comes out the other end tends to be baked all the way through - “if it compiles, it works.”
Python, being a dynamic scripting language, is going to make it easier to get a vertical slice or minimum viable product up and running. But when/if you scale up, you have to pay that time back fixing problems that Rust’s static analysis could have caught at build time.
TL;DR - Python is good for “throwing something together” or writing load-bearing scripts that do one simple thing really well. Rust is a slower start that shines as complexity and scale increase.
Hum… “Powerful” is not a word I associate with Rust… at all.
But it depends on what kind of “power” you want. Rust will give you lots of power about what kinds of things your programs do (and how performant they are), and not a lot about how to create your program with maximum productivity or minimum maintenance once your requirements change.
In particular, Rust won’t give you a very good “time to first prototype” compared to Python. But it will beat Python on maintenance if your program gets large enough. The tools that make Rust more maintainable are available in other languages (Ocaml, Haskell), and if you are mostly writing parsers, those other languages will probably beat Rust on any metric.
Anyway, be prepared to learn a lot if you decided to go with any of those 3 language, and remember that “learning” always comes with feeling you are not good enough or that the language “just can’t do this thing that is so easy on the one you know”. Those feelings are rooted on a very small truth (you aren’t good enough yet, and no, the new language does a different thing that is much better), just don’t let it blind you from the larger picture.
When you compare “idea to deployment” speed, a dynamic language will always win. However, much of this win is due to a dynamic language will let you deploy with a lot of bugs. So, you will then have to spend lot of time fixing production issues. Rust will force you to fix most of these issues before you can deploy, hence it feels slower in this aspect. I previously worked for 10 years with a huge perl code base, and I trade the deployment speed for stability in production any time.
you have to always ask your self: is the memory safe, performance etc worth the dev cost?
Perl is still the best. 😁 😁
The one thing I use but will never put on my resume
This is the way.
deleted by creator
“intuitive” is extremely subjective, and based on your past experiences. I’ve coded in C++ for years, and some Python, too and was able to grasp many Rust concepts very quickly, while for others I struggled (and still am). I’d say that if you are looking for “intuitive”, Rust ain’t it. It’s a system language, so it requires planning, it’s definitely not the ideal language to slap a prototype quickly together, expecially as a beginner.
Most of the “automation” work that I do involves parsing data with regex, restructuring the data, converting the data into a modeled format and transforming something with that data.
These are the highlights of rust - regexes (the regex library that powers ripgrep), serialization/deserialization (serde, nom, pest, capnproto, etc), data manipulation (too many. perhaps apache arrow deserves a mention), etc. The language is designed for this sort of stuff.
Has anyone used Rust for network automation tools?
I’m in the same boat as you. I use python to manage an LXD cluster. I didn’t choose Rust because Python has an API binding, but Rust didn’t. It wouldn’t have been too hard, considering that the API is mostly just JSON.
With familiarity, can Rust’s intuitiveness match Python’s “from idea to deployment” speed?
Unfortunately no! (in my personal experience). I have been using Rust for a decade (since before 1.0) and am more or less comfortable with it. I no longer fight with the compiler as most beginners do. Still, the compiler complains a lot and you have to think about why. This is not bad, IMO - all that thinking make you design better programs. There is always a sweet satisfaction of writing excellent code when you finish a Rust program - probably the reason why Rust developers love it so much. But you have to sacrifice some speed for that.
Intuitiveness is also subjective. My introduction to programming and computers was from the hardware side - from ALUs, DRAM, hardware pipelining etc. Rust’s rules are very well defined, but is obscurely related to real problems you encounter in the hardware. In other words, Rust rules look confusing at first, but it makes sense once you see the error messages. You know what problem could have happened on the hardware if Rust didn’t stop you. It’s as if Rust’s simple rules magically cover the problems you dread on the hardware.
All this means that Rust is easy to grasp for people who are already using C - OS, game engine, web engine and embedded system developers. For others who are mostly used to GC languages, it’s an uphill battle with the compiler, until some day it clicks somehow. With Rust, it’s very important to start right away with the (c lang) memory model and Rust’s aliasing rules.
Or should I only learn Rust if I intend to create applications that need tight performance?
I wouldn’t say no to learning a new language. But I always recommend a few languages because they introduce a radically different idea. Rust is one of them. My rant above probably gave you an idea why I would recommend Rust. The first is that Rust forces you to design programs properly - even though the rules are meant only to enforce memory safety. Second is that Rust gets you very close to the soul of the hardware. Python and even Go has some opaque layers in between you and the hardware. With Rust, you’ll get every opportunity to get the best the hardware can offer. You get the same in C and C++ - but they also allow you to screw up.
I use Rust for even trivial tasks where a shell script would suffice. Besides the language features, Rust has excellent tooling and a rapidly growing library ecosystem. In many ways, it’s easier to package Rust programs than Python ones. So Rust will work for your use case. But if development speed is more important to you than performance, then Go (similar to Python), Nim (much more similar to Python) and Zig are probably better choices. Repeating something here - Go is designed for the web. Numerous web backends are already on Go. And almost the entirety of Kubernetes is written in Go.
Thank you, great response. Based on your opinion and another, I believe I will focus on learning Go. I mostly need the benefits of compiled languages in order to easily distribute, as well as easier parallelism, as I’ve always found that to be a pain in the ass with Python. Not sure if Threading, Concurrency, or Asyncio are the “best” way to handle threading of non CPU intensive tasks, such as sending a request and waiting for a response. But I know that it seems since Python has taken so long to attempt to allow easier bypassing of the GIL, you get a lot of decent ways of doing something, but no great ways.
Just adding that you can also do Python - Rust interop fairly easily with something like https://github.com/PyO3/pyo3
So you can gradually adopt Rust for some of the more performance demanding parts and call them from Python. You will get a massive increase just from a simple rewrite. Though you would have the additional burden of 2 build toolchains. But as others have said, Rust is not just about the performance.
deleted by creator
two words: yes
