Steve has taken the time to comment that although they are continually working on it, the JS is in what they consider a pretty optimized place.
With minimal investigation we believe this to be a JS problem only.
After Steve’s advice, we find slightly better performance in FF and safari than chrome (OSX).

Meanwhile we implemented ‘solution’ above.
I use the term solution lightly because it’s really more like a layout hack/workaround.
We subgroup fields, groups are given a class. We bind a change event to a select field for toggling between show/hide classes.
We then disable all conditionals.
Performance wise, this is working well for us.

Problems:
Saving all empty values from hidden fields (we have to catch this)

No access to values based on conditions (our scenario employs only show/hide).

I ended up adding something which is working here.
Jquery ui widget which toggles height of the parent addmore

It’s a bit crude right now, as I’m just manually adjusting the addmore group height on widget button toggle, probably the field template would need customizing to make this more robust, but it’s a starting point at least.

        /**
         *
         *  @custom.groupwrapper
         *  
         */
        $(function () {
            $.widget("custom.groupWrapper", {

                _create: function () {
                    this.wrapper = $("<span>")
                        .addClass("custom-groupwrapper")
                        .prependTo(this.element);

                    this._createDrawerButton();
                },

                _createDrawerButton: function () {

                    console.log(this.wrapper);
                    var _this = this;

                    $("<a>")
                        .attr("tabIndex", -1)
                        .attr("title", "Collapse")
                        .prependTo(this.wrapper)
                        .button({
                            icons: {
                                primary: "ui-icon-triangle-1-s"
                            },
                            text: false
                        })
                        .removeClass("ui-corner-all")
                        .addClass("ui-corner-right")
                        .on("click", function () {
                            $(_this.wrapper)
                                .closest('.piklist-field-addmore-wrapper-full')
                                .toggleClass('hide');
                        });
                },

                _destroy: function () {
                    this.wrapper.remove();
                    this.element.show();
                }
            });

            //@todo unify combobox autocomplete selector
             $(".piklist-field-addmore-wrapper-full").groupWrapper();
        });

draft CSS:

 .piklist-field-addmore-wrapper-full {
    &.hide {
      height: 100px;
      overflow: hidden;
    }
  }

  .custom-groupwrapper {
    position: absolute;
    right: 0;
    a {
      background: none !important;
      border: none !important;
      box-shadow: none;
    }
  }

hey guys, to add to this topic, I’m having the same problems here.
I have a nested addmore.
first addmore group is a ‘row’ or ‘section’
within the ‘section’ group, users can add ‘blocks’ defined by a few basic fields (text, file, editor)
In my current metabox file, a block is really one addmore group with all of the these fields.
Block ‘types’ are defined by choosing block type from a select list, and then using conditions, showing or hiding various field.

I have tried defining blocks as
a) a flat list of fields, where conditions from select are applied on field by field basis
b) grouping fields which define individual ‘block’ types and applying conditions to group

Dev tools is showing a lot of processing in tinymce, but replacing editor with textarea doesn’t seem to make any large difference.

A typical page will have 5-6 sections and each section 1-4 blocks.
With a page filled out with full content, without conditions, performance is acceptable.
However when I add a condition, to what looks like any field or group, and then click +/- on add more, page load (ajax processing) immediately goes up to 1-2 minutes.

cachegrind shows no real difference in performance between the different scenarios, however network tab in dev tools does show 1-2 minute processing on JS.

One scenario I thought of trying is defining an aggregate custom field, one for each block type, but I haven’t looked into how to go about doing that yet, or if plugin is easily extensible in that regard (beyond simple fields).