Rendered at 13:02:50 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
jtbaker 2 days ago [-]
Buf.build, never again. Had a proto evangelist get us ingratiated into their system a few years ago and then they introduced a bunch of rent seeking behavior just to be able to to the code generation, and getting it ripped out was a huge PITA. https://github.com/betterproto/python-betterproto2 was a decent (async, type hinted) python implementation last I checked.
jdpedrie 2 days ago [-]
You can use their CLI and registry for free. I agree that their paid offerings are crazy expensive, but it's simple to just commit your protos in your monorepo (or a proto repo), install buf, create buf.gen.yaml with a few language plugins, and you're up and running. All the pain of protoc is handled. It's been a massive improvement in usability since they came along.
Scaevolus 12 hours ago [-]
if you happen to regenerate your protos 5 times in an hour you'll get ratelimited
it's a very frustrating system. you should spend a modicum of effort figuring out how to generate your things locally yourself instead, be that docker, arcane wasm...
parhamn 22 hours ago [-]
Whats the case for protobuf these days? I loved it for a while. Don't dislike it particularly now (but I've rolled off).
- Performance? Fastest JSON lib in most languages are as fast if not faster
- Cross language schema generation? So many tools for this these days, they do one thing do them right (depending on your choice of 'right' for things like unions/enums/etc)
- The wire protocol? Seems to get in the way vs http2/3. Need special considerations for your proxies (be it nginx or cloudflare). Forced certs, etc are annoying too.
I feel like to most shops these days its mostly a schema manager? Protobuf is super bloated for that use case.
esrauch 2 days ago [-]
Engineer who works on Google Protobuf here, commenting as myself and not as an official statement.
It's great to have a healthy ecosystem in the world around Protobuf. Google can't possibly fill all use cases, there's many tools and Buf makes good tooling. The Protobuf team at Google intentionally tries to enable an ecosystem around Protobuf including examples like this. Google Cloud APIs are intentionally usable with any compatible thing that can understand Protobuf encoding including this one.
Kudos to Buf for making something that I'm sure a ton of people will find useful and which takes conformance so seriously.
Just to chime in with some context about Google's own implementations here though (since that's a lot of the discussion otherwise).
Google definitely takes Protobuf seriously including for the long term: you can't really understand how engrained it is within the Google stack without seeing it for yourself. It's not just RPC layer, it's storage, logging, FFI. Html templating is driven off Protobuf messages. Systems which interact with bank XML based systems uses Protobuf schemas. Internally it's widely used for in-memory library api types even without any direct/obvious connection to serialization just because it makes internal details like logging easier. This extremely large surface does create constraints and use-cases to balance. You can see Buf's reported numbers reflect that it is faster for a usecase they expect is typical, but at scale users do fall into the other buckets shown, affecting the performance of preexisting code is a major concern for our implementations that a greenfield implementation doesn't have.
Wide exposure in critical paths alongside long term support directly causes some quirks: for example some of our APIs followed PEP8 when it was created but PEP8 changed. It looks stupid that we have wrong style APIs but also it would be stupider to break compatibility for style reasons. JavaProto as another example still supports Java8 and the runtime is compatible with 2014 gencode which is a pretty major constraint.
Google Py Proto implementation has one extra interesting choice of the same gencode is reused with 3 different implementations (upb, a complete pure python one, and one that uses C++Proto as the in memory representation which libraries like TensorFlow can use to share memory between Python and C++), which is why design the way that it is with runtime created classes, the pyi files are readable but the .py files not.
This definitely has pros and cons, and the direct approach taken by Buf here really makes a ton of sense. It's just that Google's maintained implementation falls into a different spot in a larger technical tradeoff space.
If you see things that appear to make no sense with the official implementations, feel free to file an issue on GitHub and we can look, sometimes there is no reason and we can fix it, and sometimes there's a reason which we can explain.
Kudos again to Buf here, I'm fully sure this will solve some set of real business needs better than Google's (but not because Google isn't maintaining our offerings too).
quietbritishjim 2 days ago [-]
> ... that uses C++Proto as the in memory representation which libraries like TensorFlow can use to share memory between Python and C++
This is a valid explanation for the odd implementation of the Python protobuf library (the only convincing one I've heard). But, the last time I looked into it (just a couple of years ago), it didn't seem reasonably possible to make use of this.
I can't remember the exact issue I think maybe the C++ library would be statically linked into the Python C extension module, which made it virtually impossible to use it from your own C++ code. Or maybe the issue was just that there was no C++ version prebuilt on pypi (the default is the upb version).
Anyway, it seems a pity to go to such enormous lengths for one feature and then make it essentially unavailable.
esrauch 2 days ago [-]
Well, there's still two implementations even without getting into the quoted case. But yeah, narrowly about the C++Proto shared memory thing, unfortunately it's so hard to make this work for many different reasons that we don't even advertise support for this functionality at all. This is a bit in the weeds but inside Google we static link everything which makes is far easier to do it, and static linking is how we recommend you use C++Proto in general.
But its kind of a clear example of a surprisingly complicated technical space. The big picture is that it's actually not weird when a simpler thing which has less constraints can be better on some axes users when compared to a complicated thing that delivers on many advanced constraints.
If "readability of the .py gencode instead of the .pyi gencode" was the worst pain point (or pet peeve) here then I actually suspect Buf wouldn't have even bothered, it's just one of these that is easy to see and explain.
quietbritishjim 1 days ago [-]
> This is a bit in the weeds but inside Google we static link everything which makes is far easier to do it, and static linking is how we recommend you use C++Proto in general.
Statically linking everything doesn't work in this case because each C extension module (i.e. each Python package with some native code in it) is, in effect, a shared library.
If you statically linked the C++ protobuf library into both the Python protobuf library and the Python tensorflow library then you'd end up with two copies of the library in the process (with two copies of globals/statics such as the protobuf global descriptor pool). That's not a problem in itself, but it wouldn't be possible to pass protobuf objects between them.
(In principle, you could perhaps statically link protobuf into the CPython executable, but I really don't think Google is doing that.)
The usual solution is the exact opposite to static linking: you have to build the C++ protobuf library as a shared library (libprotobuf.so) and then link to it from both of your C extension modules (i.e. from the other two shared libraries).
Following the tiny crumbs of evidence of how to do this for protobuf, it seems like that was previously how you would do it. But it's possibly not now? It seems like proto_api.h [1] has a way to get the global descriptor pool, possibly to work around needing a common shared library? But also now that header is deprecated anyway [2]? (I think this is roughly where I got to before when I gave up.)
The regular proto-to-python feels fine to me. I quit Google a while ago and have still been using protobufs, and the main things that usually make teammates wary are: 1. No way to load .proto specs at runtime (there are GH issues about this) 2. No clear and easy way to use it with HTTP/1.1. Neither is specific to Python.
Both of these might sound silly if you're used to protobufs, cause you can build little helpers for both, but plenty of people have never used protos. They would shrug proto away if there weren't someone like me to attest that it won't get in the way. It seems like Google focused on gRPC, but the real prize for adoption would've been going after the simple HTTP+JSON use cases. Like an official protobuf Express middleware.
bufbuild 22 hours ago [-]
> No clear and easy way to use it with HTTP/1.1. Neither is specific to Python.
> It seems like Google focused on gRPC, but the real prize for adoption would've been going after the simple HTTP+JSON use cases. Like an official protobuf Express middleware.
We'd flag https://connectrpc.com for you :-) HTTP/1.1 out of the box, switches on Content-Type between binary and JSON, and fully-compatible with gRPC. cURL with JSON just works.
bigcityslider 18 hours ago [-]
Yeah I saw that a while back, this is how it should be done
tommek4077 2 days ago [-]
Optimizing for the last microsecond and then use Python?
Why?
dofm 2 days ago [-]
This is a portable format that is used for interchange. The performance may matter elsewhere in your system and yet you may still want to create/access protobuf-packaged data?
quietbritishjim 2 days ago [-]
Are the only two options SIMD in hand-rolled assembler and unbearably slow?
Often Python, with the most critical parts written in a compiled extension module (like this one), offers acceptable performance with enormously less complexity than writing the whole thing in a compiled language.
sankalpmukim 2 days ago [-]
This. Yes. This is why I clicked on the comments. Exactly
paulddraper 2 days ago [-]
> Optimizing for the last microsecond
I don't see that anywhere. I see "Fast where it counts."
usrnm 2 days ago [-]
After gogoproto I'm hesitant to depend on another non-standard implementation, getting off gogo was a pain. This thing may be better than the one from Google (gogo definitely was), but can we be sure that it will still be around in 10 years?
crabbone 2 days ago [-]
Hey. I wrote another Python implementation of Protobuf. (protopy https://gitlab.com/doodles-archive/protopy it was a while ago and haven't touched it since). I'm not saying it's better than whatever this is or that it's any good, I just post it as a proof of sorts that I'm familiar with the problem.
So, without further ado: Protobuf isn't a standard. You can't have a non-standard implementation of something that doesn't have a standard to begin with. In reality, you have Google's implementation for C++ and then everything else. Everything else was, for the most part, not written by Google. And it doesn't always align 100% with the C++ Google's stuff.
Furthermore, C++ implementation has a lot of idiosyncrasies specific to that language that can't be translated one-to-one into other languages, or, in some cases, shouldn't be, even if they could (eg. C++ implementation is all about source code generation because generating runtime entities s.a. classes in C++ is very difficult, while in languages like Python, generating classes at runtime is easy.)
Furthermore, C++ implementation has a specific way of parsing the binary payload (lazy: only the top definitions are parsed, the inner structure of messages is parsed on-demand). But, is this how every parser should behave? What if you want a SAX-like parser?
----
In the hindsight, I just think that Protobuf is not a good format for writing reliable software that aims for decades of usage. We, as in the whole programming world, don't have good formats in general, and whenever we come to the point of having to use some, we either go with an existing popular but crooked or roll our own, probably also crooked. The standard you alluded to would've been great (perhaps a refinement of ASN with more attention to parser implementation, more concrete versions etc.?) But we aren't there yet, and there isn't even a work group to try and address the issue.
StilesCrisis 2 days ago [-]
> I just think that Protobuf is not a good format for writing reliable software that aims for decades of usage.
I am not a fan of Protobuf at all, but it's already demonstrated its ability to ship extremely reliable software with multi-decade lifespans. It's one of the few things Google _hasn't_ deprecated, and it's the backbone of the search and ads stack.
crabbone 2 days ago [-]
I don't know what your metric for reliability is... but I would say that the messaging / serialization format can hardly be blamed for reliability problems s.a. service uptime or service responsiveness etc. Such problems usually arise at a level where details like the choice of messaging format are not important.
Messaging format may affect application performance though. It would affect metrics s.a. throughput or bandwidth.
In a way that is more difficult to measure, a messaging format can affect the number of bugs created by developer using it and indirectly the delivery times. But this can be mitigated by tooling (i.e. Protobuf isn't self-documenting, so if you don't have the schema files, you can't interpret the messages, but you can write a tool that you can feed the schema definitions and then use the tool to interpret the messages).
> It's one of the few things Google _hasn't_ deprecated
You might be unaware of it, but there were Protobuf v1, v2, and now we are at v3. Even though it's not documented, v3 supports most of v2, but not all of it (I think v1 was never used outside of Google itself). Google never properly released Protobuf, so, they can't really deprecate it. Even their formal grammar is full of errors.
fmbb 2 days ago [-]
Why does it matter for some Python implementation if the Google C++ implementation has a lazy or eager parser?
The important part of protobuf is the spec of the wire format. That is what makes the standard an interop format.
Personally I also prefer code generation over dynamic parsers and generators. This is not an idiosyncrasy of C++, it is just the objectively good way to handle IDLs regardless of programming language.
crabbone 2 days ago [-]
> Why does it matter for some Python implementation if the Google C++ implementation has a lazy or eager parser?
I wrote about it in my repository, but I'll try to summarize it here: Protobuf is full of bad ideas. One such bad idea is that message fields are allowed to repeat and that the last field wins. So, if you were to write a SAX parser, you'd have a dilemma with how to handle this bizarre idea: do you accept that a callback for some property might be triggered multiple times or do you read the whole message ahead of time and then call callbacks exactly once for each property? If we accept that the parser must be lazy, we "solve" this problem by allowing the parser to read ahead (it needs to do this anyways), but this is a wasteful way to parse (uses more memory than necessary).
> The important part of protobuf is the spec of the wire format. That is what makes the standard an interop format.
I'm not sure what are you trying to say here. All messaging formats are made up of... wire format, that's what they are for. Maybe I'm not seeing it?
> Personally I also prefer code generation over dynamic parsers and generators.
I think you are trying to say that you prefer source code generation over generating supporting definitions at runtime? I wasn't talking about dynamic parser generation (eg. Lark). In my case, the parser was hand-written (using Bison + Flex) and compiled ahead of time, but the Python definitions supporting the Protobuf IML were generated at runtime.
If my guess is true, I'd like to hear your arguments in favor of generating Python source code that translates Protobuf IML into Python. To me it looks like a waste of space on disk... I really can't think of any reason to want that.
squirrellous 2 days ago [-]
Honest question - why isn’t the following document [1] a standard? Is it too loosely specified?
Maybe He means because it diesen have an accepted rfc
crabbone 2 days ago [-]
It's like in that meme: it's not enough to say it, you need to declare it.
Or, in other words, what makes a standard a standard is that you declare that it is a standard. This declaration carries with it an obligation to respect the standard for ever and ever (like C89), regardless of whether in retrospect you realize you've made mistakes and want to fix them, or maybe wanting to add more stuff etc.
Having a standard is restrictive and uncomfortable for the designers, that's why many opt not to have a standard. Eg. Rust language doesn't have a standard (even though you may, of course, find documents detailing how it works), same for Python, Java and many other popular languages.
usrnm 2 days ago [-]
At least, Google has the resources and the will to support all their implementations in the long run. I don't always agree with what they do there, but at least I can be sure that it will still work ten years from now. At some point it becomes more important than implementation details
crabbone 2 days ago [-]
I don't know where you get the confidence... When it comes to Protobuf, we are now at version 3 of the format. It's been around for a while, but I'm old enough to have implemented v2 parser myself... v2 is partially supported in v3, even though the support isn't documented. But, emphasis on partially. Some things are no longer there.
So... I'd say that your faith is unfounded. And, in general, there's no reason to believe that a commercial entity will commit to supporting any particular technology if that doesn't generate them a profit. Standard is better.
joshuamorton 2 days ago [-]
It should be noted that
1. Proto2 isn't actually deprecated or anything and is still widely used and supported
2. https://protobuf.dev/editions/overview/ replace and improve on the versioning concept and basically entirely remove the issue of versioning since versions and features can be incrementally enabled on a per-file and per-field basis.
aaand I guess 3: protobuf is absolutely critical to Google's profit.
crabbone 5 hours ago [-]
It might be uncalled for, but let me use a philosophical reference: you are trying to derive ought from is. Many tried to prove it to be possible, but most face such propositions with skepticism.
Let me give you some examples of things that were absolutely detrimental to the business of a large company that vanished in a matter of years if not months:
* Adobe Flash. Flash players was for a few years the most installed program on Earth. It vanished without a trace, dropped like a hot potato by everyone who swore they will use it for ever and ever.
* ASP Classic, especially in combination with VBScript. It was the hottest thing twenty (or am I getting too old?) years ago. Everyone was selling books and courses on ASP Classic, probably half the Internet was written in it... and it's gone.
* I expect x86 ISA to eventually die and be completely replaced by ARM or maybe something else. Some big players are already jumping off the bandwagon, and it looks like things are only going to get worse.
There were, of course, more, but I think these three examples should suffice. If tomorrow Google comes up with a better format, or, for whatever reason, tanks its business, or decides to switch to vertical farming in Arizona... say goodby to Protobuf. They owe you zilch. They never promised you anything. You were allowed to use their tech because it was convenient to them, but once it's not, you will be holding the bag.
This is the difference between a standard and a spec published by the designer. The designer is free to make any changes they see fit, up to and including completely destroying their work. They can pull the run from under you making the users pay for the use, they can make modifications to their tools that will require from you to buy things you wouldn't normally want to buy. The possibilities are endless.
newswangerd 2 days ago [-]
Buf is well established and maintains a lot of protobuf packages for many languages, including the YAML implementation for go.
lyu07282 2 days ago [-]
Going strong since 2019! It's a wonderful company entirely dedicated to making google's miserable protobuf "somewhat" useable.
igetspam 22 hours ago [-]
Can you be sure Google’s ${anything} will be around in 6 months? They have a decades long habit of dumping things that have major use but aren’t novel internally. Worrying about the next 10 years isn’t something you can honestly do when you your argument includes Google owning a thing.
Bias: I fought for things like Reader from the inside. If it doesn’t move needles, it goes away.
didip 2 days ago [-]
What was the backstory of gogoproto?
usrnm 2 days ago [-]
The golang implementation of protobuf sucked historically (still does, but is improving) and gogo was an alternative that fixed a lot of problems and was nicer overall. Until its creator burned out and deprecated it. Chasing a constantly moving target that you have no control over is very taxing in the long run.
Intralexical 2 days ago [-]
This is concerning to hear. What do Protobufs accomplish, that requires them to be a constantly moving target?
I think it's less that protobuf is a moving target, and more that gogo tried to add in all the features that google didn't want to maintain, and learned that maintaining a massive feature matrix was impossible.
usrnm 2 days ago [-]
If I remember correctly, what finally broke the camel's back was the new API that Google introduced. But I may be wrong
esrauch 2 days ago [-]
GoGo was not a completely separate implementation but deeply hooked into the official GoProto implementation. So it wasn't "Protobuf the binary wire format" or "Protobuf the schema language" which changed over time here, changes to the Google's Go library caused it problems. It's like building a library that integrates with Jackson (a JSON library) and Jackson details changed in ways that added toil, versus JSON changing.
newswangerd 2 days ago [-]
This is incredible news! I’ve used protobuf in Python, Go, Kotlin and Dart and the Python implementation is totally unusable. I don’t know what black magic Google uses for the Python implementation, but the classes it generates are totally opaque and impossible to inspect. I’ve been waiting for a proper python implementation for years now!
masklinn 2 days ago [-]
> I don’t know what black magic Google uses for the Python implementation, but the classes it generates are totally opaque and impossible to inspect.
TFA seems to say that they’re just thin proxies over the underlying C++ APIs, which would more than do it, and does not surprise me (the re2 Python bindings are similar, not as bad since they don’t generate Python code but they’re really c++-y — in Google’s flavour too — and uncomfortable).
functional_dev 2 days ago [-]
I did not know this before... Google protobuf is not one thing, it is three! Same import, but:
* old C++ extension
* upb
* pure Python
upb parses FAST, but then every access is still C->Python and it slows it down. So for many reads the slow python one can win?
> upb parses FAST, but then every access is still C->Python and it slows it down. So for many reads the slow python one can win?
That is pretty unlikely. TFA's version is in Rust and just barely edges out upb.
Chu4eeno 1 days ago [-]
rust code tends to be incredibly slow unless you spend a ton of time optimizing (compared to other compiled languages, and contrary to popular belief).
kccqzy 2 days ago [-]
When I was a novice in protobuf I felt tempted to inspect the generated code, but soon I learned that the documentation is good enough that I don’t need to read the generated code.
this_was_posted 2 days ago [-]
Slightly off topic, but is anyone aware of a compile free method to convert protobuf messages to json representation based on a provided .proto file (and the other way around)? All protobuf implementations seem to require a compilation step, which makes it hard to support en-/decoding untrusted content using user provided schemas
sudorandom 2 days ago [-]
The buf CLI can do this.
Here's how it looks to convert from encoded protobuf to json (protojson).
There is the --encode (and --decode) options for the `protoc` executable, e.g. `cat myobject.json | protoc --encode MyObject protofile.proto > myobject.bin`
physicalecon 2 days ago [-]
[dead]
krullin 2 days ago [-]
inexperienced protobuf user here -- why do non-googlers generally use protobuf? is it for the wire format, or for the schema language, or for the codegen?
i have a feeling there is an unmet need for something without all the google-cruft but still gives you a nice schema language and convenient codegen. just a simple stripped-down service and types definition with some support for codegen plugins would cover most crud-like applications in the wild without requiring much fuss. even just supporting only JSON would be acceptable -- maybe an optional buyin for msgpack.
i'm thinking twirp, but abandoning protobuf and using a new, simpler schema format.
seanhunter 2 days ago [-]
People who want protobuf but better often settle for something like cap’n proto https://capnproto.org/
Chu4eeno 1 days ago [-]
didn't the capnproto author end up creating protobuf2? or am I mixing things up.
kentonv 22 hours ago [-]
That's me. Other way around -- proto2 came first.
seanhunter 1 days ago [-]
Hence the “something like”. There are a few of these sorts of things. I’m not really in the market for them so I don’t try to keep up.
zellyn 2 days ago [-]
In my experience, during the phase where you scale up and move from interpreted to compiled code with microservices, someone inevitably benchmarks protobuf vs JSON, and protos come out ahead.
Also, using it as a schema doesn’t totally suck. There are others but proto is… fine.
I have lots of opinions, but don’t feel like spinning up to a full rant here :-)
BobbyTables2 2 days ago [-]
Cargo-cult mentality.
If protobuf hasn’t been created by Google, nobody would use it.
It’s actually not well designed for small requests, and doesn’t even deal with versioning all that well. It’s more verbose than a C struct yet not self-describing either.
To me, it fails to excel at any one thing yet people blindly use it anyway.
If one really considers the details, Protobuf makes ASN.1 actually start to look good!
giovannibonetti 2 days ago [-]
I wish LaunchDarkly and other feature flag providers supported protocol buffers to MN define the feature flag schema. It would be a game changer when you have complex variations and end up reaching for untyped JSON.
est 2 days ago [-]
Hope it can auto build a python class from gRPC gateway reflections.
adsharma 2 days ago [-]
Need something for backward compatible RPC? Use protobuf. But for storage and types?
This is where I feel there are better alternatives. If you go with protobuf, you're picking 2 out of 3: simplicity, fast, idiomatic.
Please consider "uvx tsc-py --help" for things that don't fit.
adsharma 2 days ago [-]
Just in case you find the earlier message too cryptic. It's not performing well on web search.
it's a very frustrating system. you should spend a modicum of effort figuring out how to generate your things locally yourself instead, be that docker, arcane wasm...
- Performance? Fastest JSON lib in most languages are as fast if not faster
- Cross language schema generation? So many tools for this these days, they do one thing do them right (depending on your choice of 'right' for things like unions/enums/etc)
- The wire protocol? Seems to get in the way vs http2/3. Need special considerations for your proxies (be it nginx or cloudflare). Forced certs, etc are annoying too.
I feel like to most shops these days its mostly a schema manager? Protobuf is super bloated for that use case.
It's great to have a healthy ecosystem in the world around Protobuf. Google can't possibly fill all use cases, there's many tools and Buf makes good tooling. The Protobuf team at Google intentionally tries to enable an ecosystem around Protobuf including examples like this. Google Cloud APIs are intentionally usable with any compatible thing that can understand Protobuf encoding including this one.
Kudos to Buf for making something that I'm sure a ton of people will find useful and which takes conformance so seriously.
Just to chime in with some context about Google's own implementations here though (since that's a lot of the discussion otherwise).
Google definitely takes Protobuf seriously including for the long term: you can't really understand how engrained it is within the Google stack without seeing it for yourself. It's not just RPC layer, it's storage, logging, FFI. Html templating is driven off Protobuf messages. Systems which interact with bank XML based systems uses Protobuf schemas. Internally it's widely used for in-memory library api types even without any direct/obvious connection to serialization just because it makes internal details like logging easier. This extremely large surface does create constraints and use-cases to balance. You can see Buf's reported numbers reflect that it is faster for a usecase they expect is typical, but at scale users do fall into the other buckets shown, affecting the performance of preexisting code is a major concern for our implementations that a greenfield implementation doesn't have.
Wide exposure in critical paths alongside long term support directly causes some quirks: for example some of our APIs followed PEP8 when it was created but PEP8 changed. It looks stupid that we have wrong style APIs but also it would be stupider to break compatibility for style reasons. JavaProto as another example still supports Java8 and the runtime is compatible with 2014 gencode which is a pretty major constraint.
Google Py Proto implementation has one extra interesting choice of the same gencode is reused with 3 different implementations (upb, a complete pure python one, and one that uses C++Proto as the in memory representation which libraries like TensorFlow can use to share memory between Python and C++), which is why design the way that it is with runtime created classes, the pyi files are readable but the .py files not.
This definitely has pros and cons, and the direct approach taken by Buf here really makes a ton of sense. It's just that Google's maintained implementation falls into a different spot in a larger technical tradeoff space.
If you see things that appear to make no sense with the official implementations, feel free to file an issue on GitHub and we can look, sometimes there is no reason and we can fix it, and sometimes there's a reason which we can explain.
Kudos again to Buf here, I'm fully sure this will solve some set of real business needs better than Google's (but not because Google isn't maintaining our offerings too).
This is a valid explanation for the odd implementation of the Python protobuf library (the only convincing one I've heard). But, the last time I looked into it (just a couple of years ago), it didn't seem reasonably possible to make use of this.
I can't remember the exact issue I think maybe the C++ library would be statically linked into the Python C extension module, which made it virtually impossible to use it from your own C++ code. Or maybe the issue was just that there was no C++ version prebuilt on pypi (the default is the upb version).
Anyway, it seems a pity to go to such enormous lengths for one feature and then make it essentially unavailable.
But its kind of a clear example of a surprisingly complicated technical space. The big picture is that it's actually not weird when a simpler thing which has less constraints can be better on some axes users when compared to a complicated thing that delivers on many advanced constraints.
If "readability of the .py gencode instead of the .pyi gencode" was the worst pain point (or pet peeve) here then I actually suspect Buf wouldn't have even bothered, it's just one of these that is easy to see and explain.
Statically linking everything doesn't work in this case because each C extension module (i.e. each Python package with some native code in it) is, in effect, a shared library.
If you statically linked the C++ protobuf library into both the Python protobuf library and the Python tensorflow library then you'd end up with two copies of the library in the process (with two copies of globals/statics such as the protobuf global descriptor pool). That's not a problem in itself, but it wouldn't be possible to pass protobuf objects between them.
(In principle, you could perhaps statically link protobuf into the CPython executable, but I really don't think Google is doing that.)
The usual solution is the exact opposite to static linking: you have to build the C++ protobuf library as a shared library (libprotobuf.so) and then link to it from both of your C extension modules (i.e. from the other two shared libraries).
Following the tiny crumbs of evidence of how to do this for protobuf, it seems like that was previously how you would do it. But it's possibly not now? It seems like proto_api.h [1] has a way to get the global descriptor pool, possibly to work around needing a common shared library? But also now that header is deprecated anyway [2]? (I think this is roughly where I got to before when I gave up.)
[1] https://github.com/protocolbuffers/protobuf/blob/main/python...
[2] https://github.com/protocolbuffers/protobuf/issues/9464
Both of these might sound silly if you're used to protobufs, cause you can build little helpers for both, but plenty of people have never used protos. They would shrug proto away if there weren't someone like me to attest that it won't get in the way. It seems like Google focused on gRPC, but the real prize for adoption would've been going after the simple HTTP+JSON use cases. Like an official protobuf Express middleware.
> It seems like Google focused on gRPC, but the real prize for adoption would've been going after the simple HTTP+JSON use cases. Like an official protobuf Express middleware.
We'd flag https://connectrpc.com for you :-) HTTP/1.1 out of the box, switches on Content-Type between binary and JSON, and fully-compatible with gRPC. cURL with JSON just works.
Why?
Often Python, with the most critical parts written in a compiled extension module (like this one), offers acceptable performance with enormously less complexity than writing the whole thing in a compiled language.
I don't see that anywhere. I see "Fast where it counts."
So, without further ado: Protobuf isn't a standard. You can't have a non-standard implementation of something that doesn't have a standard to begin with. In reality, you have Google's implementation for C++ and then everything else. Everything else was, for the most part, not written by Google. And it doesn't always align 100% with the C++ Google's stuff.
Furthermore, C++ implementation has a lot of idiosyncrasies specific to that language that can't be translated one-to-one into other languages, or, in some cases, shouldn't be, even if they could (eg. C++ implementation is all about source code generation because generating runtime entities s.a. classes in C++ is very difficult, while in languages like Python, generating classes at runtime is easy.)
Furthermore, C++ implementation has a specific way of parsing the binary payload (lazy: only the top definitions are parsed, the inner structure of messages is parsed on-demand). But, is this how every parser should behave? What if you want a SAX-like parser?
----
In the hindsight, I just think that Protobuf is not a good format for writing reliable software that aims for decades of usage. We, as in the whole programming world, don't have good formats in general, and whenever we come to the point of having to use some, we either go with an existing popular but crooked or roll our own, probably also crooked. The standard you alluded to would've been great (perhaps a refinement of ASN with more attention to parser implementation, more concrete versions etc.?) But we aren't there yet, and there isn't even a work group to try and address the issue.
I am not a fan of Protobuf at all, but it's already demonstrated its ability to ship extremely reliable software with multi-decade lifespans. It's one of the few things Google _hasn't_ deprecated, and it's the backbone of the search and ads stack.
Messaging format may affect application performance though. It would affect metrics s.a. throughput or bandwidth.
In a way that is more difficult to measure, a messaging format can affect the number of bugs created by developer using it and indirectly the delivery times. But this can be mitigated by tooling (i.e. Protobuf isn't self-documenting, so if you don't have the schema files, you can't interpret the messages, but you can write a tool that you can feed the schema definitions and then use the tool to interpret the messages).
> It's one of the few things Google _hasn't_ deprecated
You might be unaware of it, but there were Protobuf v1, v2, and now we are at v3. Even though it's not documented, v3 supports most of v2, but not all of it (I think v1 was never used outside of Google itself). Google never properly released Protobuf, so, they can't really deprecate it. Even their formal grammar is full of errors.
The important part of protobuf is the spec of the wire format. That is what makes the standard an interop format.
Personally I also prefer code generation over dynamic parsers and generators. This is not an idiosyncrasy of C++, it is just the objectively good way to handle IDLs regardless of programming language.
I wrote about it in my repository, but I'll try to summarize it here: Protobuf is full of bad ideas. One such bad idea is that message fields are allowed to repeat and that the last field wins. So, if you were to write a SAX parser, you'd have a dilemma with how to handle this bizarre idea: do you accept that a callback for some property might be triggered multiple times or do you read the whole message ahead of time and then call callbacks exactly once for each property? If we accept that the parser must be lazy, we "solve" this problem by allowing the parser to read ahead (it needs to do this anyways), but this is a wasteful way to parse (uses more memory than necessary).
> The important part of protobuf is the spec of the wire format. That is what makes the standard an interop format.
I'm not sure what are you trying to say here. All messaging formats are made up of... wire format, that's what they are for. Maybe I'm not seeing it?
> Personally I also prefer code generation over dynamic parsers and generators.
I think you are trying to say that you prefer source code generation over generating supporting definitions at runtime? I wasn't talking about dynamic parser generation (eg. Lark). In my case, the parser was hand-written (using Bison + Flex) and compiled ahead of time, but the Python definitions supporting the Protobuf IML were generated at runtime.
If my guess is true, I'd like to hear your arguments in favor of generating Python source code that translates Protobuf IML into Python. To me it looks like a waste of space on disk... I really can't think of any reason to want that.
[1] https://protobuf.dev/programming-guides/encoding/
Or, in other words, what makes a standard a standard is that you declare that it is a standard. This declaration carries with it an obligation to respect the standard for ever and ever (like C89), regardless of whether in retrospect you realize you've made mistakes and want to fix them, or maybe wanting to add more stuff etc.
Having a standard is restrictive and uncomfortable for the designers, that's why many opt not to have a standard. Eg. Rust language doesn't have a standard (even though you may, of course, find documents detailing how it works), same for Python, Java and many other popular languages.
So... I'd say that your faith is unfounded. And, in general, there's no reason to believe that a commercial entity will commit to supporting any particular technology if that doesn't generate them a profit. Standard is better.
1. Proto2 isn't actually deprecated or anything and is still widely used and supported
2. https://protobuf.dev/editions/overview/ replace and improve on the versioning concept and basically entirely remove the issue of versioning since versions and features can be incrementally enabled on a per-file and per-field basis.
aaand I guess 3: protobuf is absolutely critical to Google's profit.
Let me give you some examples of things that were absolutely detrimental to the business of a large company that vanished in a matter of years if not months:
* Adobe Flash. Flash players was for a few years the most installed program on Earth. It vanished without a trace, dropped like a hot potato by everyone who swore they will use it for ever and ever.
* ASP Classic, especially in combination with VBScript. It was the hottest thing twenty (or am I getting too old?) years ago. Everyone was selling books and courses on ASP Classic, probably half the Internet was written in it... and it's gone.
* I expect x86 ISA to eventually die and be completely replaced by ARM or maybe something else. Some big players are already jumping off the bandwagon, and it looks like things are only going to get worse.
There were, of course, more, but I think these three examples should suffice. If tomorrow Google comes up with a better format, or, for whatever reason, tanks its business, or decides to switch to vertical farming in Arizona... say goodby to Protobuf. They owe you zilch. They never promised you anything. You were allowed to use their tech because it was convenient to them, but once it's not, you will be holding the bag.
This is the difference between a standard and a spec published by the designer. The designer is free to make any changes they see fit, up to and including completely destroying their work. They can pull the run from under you making the users pay for the use, they can make modifications to their tools that will require from you to buy things you wouldn't normally want to buy. The possibilities are endless.
Bias: I fought for things like Reader from the inside. If it doesn’t move needles, it goes away.
I think it's less that protobuf is a moving target, and more that gogo tried to add in all the features that google didn't want to maintain, and learned that maintaining a massive feature matrix was impossible.
TFA seems to say that they’re just thin proxies over the underlying C++ APIs, which would more than do it, and does not surprise me (the re2 Python bindings are similar, not as bad since they don’t generate Python code but they’re really c++-y — in Google’s flavour too — and uncomfortable).
* old C++ extension
* upb
* pure Python
upb parses FAST, but then every access is still C->Python and it slows it down. So for many reads the slow python one can win?
This one helped me to dig deeper - https://vectree.io/c/how-python-protobuf-runtimes-work-pure-...
That is pretty unlikely. TFA's version is in Rust and just barely edges out upb.
Here's how it looks to convert from encoded protobuf to json (protojson).
And just invert the arguments to convert back.i have a feeling there is an unmet need for something without all the google-cruft but still gives you a nice schema language and convenient codegen. just a simple stripped-down service and types definition with some support for codegen plugins would cover most crud-like applications in the wild without requiring much fuss. even just supporting only JSON would be acceptable -- maybe an optional buyin for msgpack.
i'm thinking twirp, but abandoning protobuf and using a new, simpler schema format.
Also, using it as a schema doesn’t totally suck. There are others but proto is… fine.
I have lots of opinions, but don’t feel like spinning up to a full rant here :-)
If protobuf hasn’t been created by Google, nobody would use it.
It’s actually not well designed for small requests, and doesn’t even deal with versioning all that well. It’s more verbose than a C struct yet not self-describing either.
To me, it fails to excel at any one thing yet people blindly use it anyway.
If one really considers the details, Protobuf makes ASN.1 actually start to look good!
This is where I feel there are better alternatives. If you go with protobuf, you're picking 2 out of 3: simplicity, fast, idiomatic.
Please consider "uvx tsc-py --help" for things that don't fit.
https://pypi.org/project/tsc-py/
Edit: I see now. You are spamming links to your owm site.