I’ll have to disagree with you - the problems are similar but somewhat unrelated. Sampling profilers traditionally only sample threads that are actively doing something - i.e. using CPU time. Tracing is closer to wall time profiling rather than CPU time - and global instrumentation actually sucks at that kind of enumeration. The overhead of iterating through 100k goroutines would be too high at any reasonable frequency when most of them are just blocked on something. And most are probably blocked on something you don’t even care to see (eg a worker waiting for work).
Aside from the sampling issue there is also the problem of non-blocking frameworks/languages. It’s easy to add tracing to JS functions that use async/await but it is definitely a mini research project to recover stacks from in-flight promise state machines. Most languages fall into this bucket unfortunately.
Now all that said, if you can make some automation that preemptively creates spans for you - kind of like a sampling profiler “creates” stack traces - that would be super cool. The worst part about tracing is sprinkling the tracing code in all the places you need it. You inevitably miss a few spots that show up as “missing” bars in the gannt view. Having something that fills in those blanks is invaluable.
> The worst part about tracing is sprinkling the tracing code in all the places you need it.
On the JVM this is usually done by agents that automatically attach to your process, you don't have to instrument your code at all. I think Datadog open-sourced their implementation?
Both JFR and agent based implementations only work with non async code. As soon as you do async anything, stack traces are no longer representative so automatic instrumentation is useless.
That said, I’m really hoping Project Loom will fix that issue by getting rid of most async callback-style I/O.
Aside from the sampling issue there is also the problem of non-blocking frameworks/languages. It’s easy to add tracing to JS functions that use async/await but it is definitely a mini research project to recover stacks from in-flight promise state machines. Most languages fall into this bucket unfortunately.
Now all that said, if you can make some automation that preemptively creates spans for you - kind of like a sampling profiler “creates” stack traces - that would be super cool. The worst part about tracing is sprinkling the tracing code in all the places you need it. You inevitably miss a few spots that show up as “missing” bars in the gannt view. Having something that fills in those blanks is invaluable.