An alternative to an eternally-spinning wheel

July 28, 2016 5:53 PM

tqdm (تقدّم, or “progress”) is a fast, extensible, low-overhead progress bar for Python.
It is primarily developed by Casper da Costa-Luis and Stephen L.
To use, just wrap the tqdm method around an iterable: for i in tqdm(range(1000)): sleep(0.1)
posted by Going To Maine (28 comments total)

(There was once a time when I found progress bars boring.)
posted by Going To Maine at 6:03 PM on July 28, 2016

Cute. I may have to use this one of these days.
posted by brennen at 6:03 PM on July 28, 2016

Overhead is low – about 60ns per iteration (80ns with tqdm_gui), and is unit tested against performance regression. By comparison, the well established ProgressBar has an 800ns/iter overhead.

In addition to its low overhead, tqdm uses smart algorithms to predict the remaining time and to skip unnecessary iteration displays, which allows for a negligible overhead in most cases.

I had no idea something that seemed so simple was so involved. Fun to know!
posted by indubitable at 6:15 PM on July 28, 2016

To use, just wrap the tqdm method around an iterable: for i in tqdm(range(1000)): sleep(0.1)

Hm out does it know it's out of 1000?
posted by kenko at 6:34 PM on July 28, 2016

range(1000) returns a list.
posted by reprise the theme song and roll the credits at 6:40 PM on July 28, 2016

Yeah, you can have an infinite iterable (just yield randint(0, 1) forever...), so not sure what it does in that case...
A reasonable choice might be to start by assuming N items, then whenever you reach the halfway point, assume 2N objects in the remaining portion of the bar. (effectively, progress towards 100% exponentially slowly until you reach 'the end' or keep going forever without ever reaching the 100% mark.)
posted by kaibutsu at 6:40 PM on July 28, 2016

Yeah, you can have an infinite iterable (just yield randint(0, 1) forever...), so not sure what it does in that case...

As with many things in this world, tqdm works better if it knows when it’s time to stop. Unlike many things in this world, it knows when it doesn’t know when to stock.

If you give tqdm something without an end point (e.g. a file that you want to read line by line and not with .readlines()), it will return a count of iterations and the estimated time per iteration.
posted by Going To Maine at 6:45 PM on July 28, 2016

range(1000) returns a list.

Sure, but I wouldn't want to go calling len() on any random iterable (which is apparently what it does if you don't tell it the length), since that might have unknown consequences in terms of space or time consumption.
posted by kenko at 7:57 PM on July 28, 2016

        if total is None and iterable is not None:
            try:
                total = len(iterable)
            except (TypeError, AttributeError):
                total = None

I guess you have to be very strict about making sure to pass in a total if len() works but would be potentially undesirable.
posted by kenko at 7:59 PM on July 28, 2016

CPython iterables, if they respond to it at all, don't do any significant computation on a call to len, since they already know how long they are and update that value with each resize. I would expect that behavior out of any non-broken iterable.
posted by invitapriore at 8:16 PM on July 28, 2016 [2 favorites]

tqdm is love, tqdm is life!

In addition to the wrapping an iterable option, you can also just make a progress bar object and update it manually. Sometimes that lets you write cleaner scripts:

        pbar = tqdm(total=fullsize)
        with writer.saving(combo, "targetmovie.mp4", plotresolution):
            spam(bla)
            pbar.update(1)

posted by MengerSponge at 8:29 PM on July 28, 2016

range(1000) returns a list.

Minor nitpick: it does in Python 2. In Python 3 it's a range object, but it sorta boils down to the same thing because you can calculate the length from properties of the object. (This is likely actually better algorithmically too, because you're not going "give me this many" then turning around straight away and going "how many was it again?")
posted by iffthen at 9:21 PM on July 28, 2016 [1 favorite]

CPython iterables

Too tired to verify, but probably correct - but also worth noting there's more backends than just CPython
posted by iffthen at 9:23 PM on July 28, 2016

I mean, I just looked at the source for listobject.c and extrapolated from there, but I honestly can't think of any other sane behavior for the basic data structures that the language provides.
posted by invitapriore at 9:34 PM on July 28, 2016

Neat! I thought it was a keyboard-smash name (tuh-queue-dum), chosen after they discovered all other sensible ones were chosen (like all those Lexus model names). I've used it previously in a Python package/wrapper for searching Git commit ranges for text patterns.
posted by waninggibbon at 10:07 PM on July 28, 2016

From the it's-a-feature-not-a-bug school, you're gonna be unhappy of you try to call len() on count(), cycle(), or repeat() from the itertools module.
posted by fragmede at 3:28 AM on July 29, 2016 [1 favorite]

Weird anti-pythonic pushback in here. "Some ducks could be bad ducks or maybe even geese!!"
posted by advil at 4:59 AM on July 29, 2016

In the animated screenshot, what's the editor they're using? Emacs with some python plugin? The help stuff below the cursor looked convenient.
posted by xris at 5:08 AM on July 29, 2016 [3 favorites]

xris: Looks like bpython.
posted by signal at 5:48 AM on July 29, 2016 [2 favorites]

Weird anti-pythonic pushback in here. "Some ducks could be bad ducks or maybe even geese!!"

Or pigs
posted by Itaxpica at 5:48 AM on July 29, 2016 [4 favorites]

So, an interpreted version of pv(1).
posted by scruss at 7:12 AM on July 29, 2016 [1 favorite]

A Hymn To Progress

(to the tune of 'For Those In Peril On The C')

The mysteries of the progress bar
Are stranger than you think, by far
Its claims to future certainties
Are half a promise, half belief
Yet it won't answer - just beguile -
"What's happened to my fucking file?"

tqdm is quite sublime
At sipping from the well of time
No greedy swallows of your mips
Shall pass its dainty python lips
But even it will only smile
When asked "So, where's my fucking file?"

Great Chronos, who decides our fate
Thy name on high, we iterate
Thy len() returns infinity
Each nanosec we give to thee
So hear our prayer, to thee compiled:
"Oh god, where is our fucking file?"
posted by Devonian at 7:14 AM on July 29, 2016 [6 favorites]

Please keep in mind that if you make a mistake at the command line, you just need to type 'fuck' and it'll fix itself.
posted by signal at 7:29 AM on July 29, 2016 [5 favorites]

that is great, but if someone could make a fork with s/fuck/poot for general audiences I’d be very happy.
posted by Going To Maine at 10:31 AM on July 29, 2016

That's actually a feature, since that's not a command I want general audiences getting in the habit of using on any of the boxes I touch.
posted by invitapriore at 10:58 AM on July 29, 2016

What about if Russian hackers leak the DNC’s sysadmins’ bash history files, though? The scandal!
posted by Going To Maine at 11:04 AM on July 29, 2016

Whoa, what is this super-slick text editor with this super-slick completion suggesting and documentation showing?
posted by d. z. wang at 4:44 PM on July 29, 2016

awesome!
posted by Vitamaster at 8:33 PM on July 29, 2016