1. 29 May, 2021 1 commit
  2. 25 May, 2021 2 commits
  3. 23 May, 2021 5 commits
  4. 22 May, 2021 32 commits
    • Todd Gamblin's avatar
      f1fe03cd
    • Massimiliano Culpo's avatar
      Style fixes for v0.16.2 release · 80f0c78f
      Massimiliano Culpo authored
      80f0c78f
    • Todd Gamblin's avatar
      performance: speed up existence checks in packages (#23661) · 818664d5
      Todd Gamblin authored
      Spack doesn't require users to manually index their repos; it reindexes the indexes automatically when things change. To determine when to do this, it has to `stat()` all package files in each repository to make sure that indexes up to date with packages. We currently index virtual providers, patches by sha256, and tags on packages.
      
      When this was originally implemented, we ran the checker all the time, at startup, but that was slow (see #7587). But we didn't go far enough -- it still consults the checker and does all the stat operations just to see if a package exists (`Repo.exists()`).  That might've been a wash in 2018, but as the number of packages has grown, it's gotten slower -- checking 5k packages is expensive and users see this for small operations.  It's a win now to make `Repo.exists()` check files directly.
      
      **Fix:**
      
      This PR does a number of things to speed up `spack load`, `spack info`, and other commands:
      
      - [x] Make `Repo.exists()` check files directly again with `os.path.exists()` (this is the big one)
      - [x] Refactor `Spec.satisfies()` so that a checking for virtual packages only happens if needed
            (avoids some calls to exists())
      - [x] Avoid calling `Repo.exists(spec)` in `Repo.get()`. `Repo.get()` will ultimately try to load
            a `package.py` file anyway; we can let the failure to load it indicate that the package doesn't
            exist, and avoid another call to exists().
      - [x] Fix up some comments in spec parsing
      - [x] Call `UnknownPackageError` more consistently in `repo.py`
      818664d5
    • Massimiliano Culpo's avatar
      ASP-based solve: minimize compiler mismatches (#23016) · 76c5a021
      Massimiliano Culpo authored
      fixes #22718
      
      Instead of trying to maximize the number of
      matches (preferred behavior), try to minimize
      the number of mismatches (unwanted behavior).
      76c5a021
    • Drew Whitehouse's avatar
      new package for openvdb (#23581) · 0fd94d44
      Drew Whitehouse authored
      0fd94d44
    • Olivier Cessenat's avatar
      New package: sparskit (#23848) · fb156ae4
      Olivier Cessenat authored
      fb156ae4
    • Massimiliano Culpo's avatar
      ASP-based solver: no intermediate package for concretizing together (#23307) · 30dd6126
      Massimiliano Culpo authored
      The ASP-based solver can natively manage cases where more than one root spec is given, and is able to concretize all the roots together (ensuring one spec per package at most).
      
      Modifications:
      - [x] When concretising together an environment the ASP-based solver calls directly its `solve` method rather than constructing a temporary fake root package.
      30dd6126
    • Massimiliano Culpo's avatar
      Import hooks using Python's built-in machinery (#23288) · 13fed376
      Massimiliano Culpo authored
      The function we coded in Spack to load Python modules with arbitrary
      names from a file seem to have issues with local imports. For
      loading hooks though it is unnecessary to use such functions, since
      we don't care to bind a custom name to a module nor we have to load
      it from an unknown location.
      
      This PR thus modifies spack.hook in the following ways:
      
      - Use __import__ instead of spack.util.imp.load_source (this
        addresses #20005)
      - Sync module docstring with all the hooks we have
      - Avoid using memoization in a module function
      - Marked with a leading underscore all the names that are supposed
        to stay local
      13fed376
    • Harmen Stoppels's avatar
      fb27c7ad
    • Harmen Stoppels's avatar
      4a7581ed
    • Massimiliano Culpo's avatar
    • Massimiliano Culpo's avatar
      ASP-based solver: suppress warnings when constructing facts (#23090) · 8a7bfe97
      Massimiliano Culpo authored
      fixes #22786
      
      Trying to get optimization flags for a specific target from
      a compiler may trigger warnings. In the context of constructing
      facts for the ASP-based solver we don't want to show these
      warnings to the user, so here we simply ignore them.
      8a7bfe97
    • Peter Scheibel's avatar
      Externals with merged prefixes (#22653) · f1f94ad3
      Peter Scheibel authored
      We remove system paths from search variables like PATH and 
      from -L options because they may contain many packages and
      could interfere with Spack-built packages. External packages 
      may be installed to prefixes that are not actually system paths 
      but are still "merged" in the sense that many other packages are
      installed there. To avoid conflicts, this PR places all external
      packages at the end of search paths.
      f1f94ad3
    • Massimiliano Culpo's avatar
      ASP-based solver: assign OS correctly with inheritance from parent (#22896) · 2655e21b
      Massimiliano Culpo authored
      fixes #22871
      
      When in presence of multiple choices for the operating system
      we were lacking a rule to derive the node OS if it was
      inherited.
      2655e21b
    • Peter Scheibel's avatar
      "spack build-env" searches env for relevant spec (#21642) · 5546b22c
      Peter Scheibel authored
      
      
      If you install packages using spack install in an environment with
      complex spec constraints, and the install fails, you may want to
      test out the build using spack build-env; one issue (particularly
      if you use concretize: together) is that it may be hard to pass
      the appropriate spec that matches what the environment is
      attempting to install.
      
      This updates the build-env command to default to pulling a matching
      spec from the environment rather than concretizing what the user
      provides on the command line independently.
      
      This makes a similar change to spack cd.
      
      If the user-provided spec matches multiple specs in the environment,
      then these commands will now report an error and display all
      matching specs (to help the user specify).
      
      Co-authored-by: default avatarGregory Becker <becker33@llnl.gov>
      5546b22c
    • Adam J. Stewart's avatar
    • Massimiliano Culpo's avatar
      Bootstrapping: swap store before configuration (#22631) · 43cea1b3
      Massimiliano Culpo authored
      fixes #22294
      
      A combination of the swapping order for global variables and
      the fact that most of them are lazily evaluated resulted in
      custom install tree not being taken into account if clingo
      had to be bootstrapped.
      
      This commit fixes that particular issue, but a broader refactor
      may be needed to ensure that similar situations won't affect us
      in the future.
      43cea1b3
    • Harmen Stoppels's avatar
      Bootstrap: add _builtin config scope (#22610) · cbd55332
      Harmen Stoppels authored
      (cherry picked from commit a37c916d)
      cbd55332
    • Harmen Stoppels's avatar
    • Cyrus Harrison's avatar
      bugfix for active when pkg is already active error (#22587) · a5213dab
      Cyrus Harrison authored
      
      
      * bugfix for active when pkg is already active error
      
      Co-authored-by: default avatarGreg Becker <becker33@llnl.gov>
      a5213dab
    • Massimiliano Culpo's avatar
      Enforce uniqueness of the version_weight atom per node · 6e714808
      Massimiliano Culpo authored
      fixes #22565
      
      This change enforces the uniqueness of the version_weight
      atom per node(Package) in the DAG. It does so by applying
      FTSE and adding an extra layer of indirection with the
      possible_version_weight/2 atom.
      
      Before this change it may have happened that for the same
      node two different version_weight/2 were in the answer set,
      each of which referred to a different spec with the same
      version, and their weights would sum up.
      
      This lead to unexpected result like preferring to build a
      new version of an external if the external version was
      older.
      6e714808
    • Massimiliano Culpo's avatar
      Externals are preferred even when they have non-default variant values · 9f99e8ad
      Massimiliano Culpo authored
      fixes #22596
      
      Variants which are specified in an external spec are not
      scored negatively if they encode a non-default value.
      9f99e8ad
    • Massimiliano Culpo's avatar
      clingo: modify recipe for bootstrapping (#22354) · 18880a66
      Massimiliano Culpo authored
      * clingo: modify recipe for bootstrapping
      
      Modifications:
      - clingo builds with shared Python only if ^python+shared
      - avoid building the clingo app for bootstrapping
      - don't link to libpython when bootstrapping
      
      * Remove option that breaks on linux
      
      * Give more hints for the current Python
      
      * Disable CLINGO_BUILD_PY_SHARED for bootstrapping
      
      * bootstrapping: try to detect the current python from std library
      
      This is much faster than calling external executables
      
      * Fix compatibility with Python 2.6
      
      * Give hints on which compiler and OS to use when bootstrapping
      
      This change hints which compiler to use for bootstrapping clingo
      (either GCC or Apple Clang on MacOS). On Cray platforms it also
      hints to build for the frontend system, where software is meant
      to be installed.
      
      * Use spec_for_current_python to constrain module requirement
      
      (cherry picked from commit d5fa509b)
      18880a66
    • Massimiliano Culpo's avatar
      ASP-based solver: model disjoint sets for multivalued variants (#22534) · e7494b62
      Massimiliano Culpo authored
      * ASP-based solver: avoid adding values to variants when they're set
      
      fixes #22533
      fixes #21911
      
      Added a rule that prevents any value to slip in a variant when the
      variant is set explicitly. This is relevant for multi-valued variants,
      in particular for those that have disjoint sets of values.
      
      * Ensure disjoint sets have a clear semantics for external packages
      e7494b62
    • Massimiliano Culpo's avatar
      Make SingleFileScope able to repopulate the cache after clearing it (#22559) · b11dd047
      Massimiliano Culpo authored
      fixes #22547
      
      SingleFileScope was not able to repopulate its cache before this
      change. This was affecting the configuration seen by environments
      using clingo bootstrapped from sources, since the bootstrapping
      operation involved a few cache invalidation for config files.
      b11dd047
    • Rémi Lacroix's avatar
      Channelflow: Fix the package. (#22483) · 31a07f9b
      Rémi Lacroix authored
      A search and replace went wrong in 2264e30d.
      
      Thanks to @wadudmiah who reported this issue.
      31a07f9b
    • Harmen Stoppels's avatar
    • Todd Gamblin's avatar
      bugfix: allow imposed constraints to be overridden in special cases · 4d148a43
      Todd Gamblin authored
      In most cases, we want condition_holds(ID) to imply any imposed
      constraints associated with the ID. However, the dependency relationship
      in Spack is special because it's "extra" conditional -- a dependency
      *condition* may hold, but we have decided that externals will not have
      dependencies, so we need a way to avoid having imposed constraints appear
      for nodes that don't exist.
      
      This introduces a new rule that says that constraints are imposed
      *unless* we define `do_not_impose(ID)`. This allows rules like
      dependencies, which rely on more than just spec conditions, to cancel
      imposed constraints.
      
      We add one special case for this: dependencies of externals.
      4d148a43
    • Todd Gamblin's avatar
      bugfix: do not generate dep conditions when no dependency · 9717e024
      Todd Gamblin authored
      We only consider test dependencies some of the time. Some packages are
      *only* test dependencies. Spack's algorithm was previously generating
      dependency conditions that could hold, *even* if there was no potential
      dependency type.
      
      - [x] change asp.py so that this can't happen -- we now only generate
            dependency types for possible dependencies.
      9717e024
    • Todd Gamblin's avatar
      concretizer: unify logic for spec conditionals · a823cffc
      Todd Gamblin authored
      This builds on #20638 by unifying all the places in the concretizer where
      things are conditional on specs. Previously, we duplicated a common spec
      conditional pattern for dependencies, virtual providers, conflicts, and
      externals. That was introduced in #20423 and refined in #20507, and
      roughly looked as follows.
      
      Given some directives in a package like:
      
      ```python
      depends_on("foo@1.0+bar", when="@2.0+variant")
      provides("mpi@2:", when="@1.9:")
      ```
      
      We handled the `@2.0+variant` and `@1.9:` parts by generating generated
      `dependency_condition()`, `required_dependency_condition()`, and
      `imposed_dependency_condition()` facts to trigger rules like this:
      
      ```prolog
      dependency_conditions_hold(ID, Parent, Dependency) :-
        attr(Name, Arg1)             : required_dependency_condition(ID, Name, Arg1);
        attr(Name, Arg1, Arg2)       : required_dependency_condition(ID, Name, Arg1, Arg2);
        attr(Name, Arg1, Arg2, Arg3) : required_dependency_condition(ID, Name, Arg1, Arg2, Arg3);
        dependency_condition(ID, Parent, Dependency);
        node(Parent).
      ```
      
      And we handled `foo@1.0+bar` and `mpi@2:` parts ("imposed constraints")
      like this:
      
      ```prolog
      attr(Name, Arg1, Arg2) :-
        dependency_conditions_hold(ID, Package, Dependency),
        imposed_dependency_condition(ID, Name, Arg1, Arg2).
      
      attr(Name, Arg1, Arg2, Arg3) :-
        dependency_conditions_hold(ID, Package, Dependency),
        imposed_dependency_condition(ID, Name, Arg1, Arg2, Arg3).
      ```
      
      These rules were repeated with different input predicates for
      requirements (e.g., `required_dependency_condition`) and imposed
      constraints (e.g., `imposed_dependency_condition`) throughout
      `concretize.lp`. In #20638 it got to be a bit confusing, because we used
      the same `dependency_condition_holds` predicate to impose constraints on
      conditional dependencies and virtual providers. So, even though the
      pattern was repeated, some of the conditional rules were conjoined in a
      weird way.
      
      Instead of repeating this pattern everywhere, we now have *one* set of
      consolidated rules for conditions:
      
      ```prolog
      condition_holds(ID) :-
        condition(ID);
        attr(Name, A1)         : condition_requirement(ID, Name, A1);
        attr(Name, A1, A2)     : condition_requirement(ID, Name, A1, A2);
        attr(Name, A1, A2, A3) : condition_requirement(ID, Name, A1, A2, A3).
      
      attr(Name, A1)         :- condition_holds(ID), imposed_constraint(ID, Name, A1).
      attr(Name, A1, A2)     :- condition_holds(ID), imposed_constraint(ID, Name, A1, A2).
      attr(Name, A1, A2, A3) :- condition_holds(ID), imposed_constraint(ID, Name, A1, A2, A3).
      ```
      
      this allows us to use `condition(ID)` and `condition_holds(ID)` to
      encapsulate the conditional logic on specs in all the scenarios where we
      need it. Instead of defining predicates for the requirements and imposed
      constraints, we generate the condition inputs with generic facts, and
      define predicates to associate the condition ID with a particular
      scenario. So, now, the generated facts for a condition look like this:
      
      ```prolog
      condition(121).
      condition_requirement(121,"node","cairo").
      condition_requirement(121,"variant_value","cairo","fc","True").
      imposed_constraint(121,"version_satisfies","fontconfig","2.10.91:").
      dependency_condition(121,"cairo","fontconfig").
      dependency_type(121,"build").
      dependency_type(121,"link").
      ```
      
      The requirements and imposed constraints are generic, and we associate
      them with their meaning via the id. Here, `dependency_condition(121,
      "cairo", "fontconfig")` tells us that condition 121 has to do with the
      dependency of `cairo` on `fontconfig`, and the conditional dependency
      rules just become:
      
      ```prolog
      dependency_holds(Package, Dependency, Type) :-
        dependency_condition(ID, Package, Dependency),
        dependency_type(ID, Type),
        condition_holds(ID).
      ```
      
      Dependencies, virtuals, conflicts, and externals all now use similar
      patterns, and the logic for generating condition facts is common to all
      of them on the python side, as well. The more specific routines like
      `package_dependencies_rules` just call `self.condition(...)` to get an id
      and generate requirements and imposed constraints, then they generate
      their extra facts with the returned id, like this:
      
      ```python
          def package_dependencies_rules(self, pkg, tests):
              """Translate 'depends_on' directives into ASP logic."""
              for _, conditions in sorted(pkg.dependencies.items()):
                  for cond, dep in sorted(conditions.items()):
                      condition_id = self.condition(cond, dep.spec, pkg.name)  # create a condition and get its id
                      self.gen.fact(fn.dependency_condition(  # associate specifics about the dependency w/the id
                          condition_id, pkg.name, dep.spec.name
                      ))
              # etc.
      ```
      
      - [x] unify generation and logic for conditions
      - [x] use unified logic for dependencies
      - [x] use unified logic for virtuals
      - [x] use unified logic for conflicts
      - [x] use unified logic for externals
      
      LocalWords:  concretizer mpi attr Arg concretize lp cairo fc fontconfig
      LocalWords:  virtuals def pkg cond dep fn refactor github py
      a823cffc
    • Massimiliano Culpo's avatar
      bootstrap: account for platform specific configuration scopes (#22489) · f1c7402c
      Massimiliano Culpo authored
      This change accounts for platform specific configuration scopes,
      like ~/.spack/linux, during bootstrapping. These scopes were
      previously not accounted for and that was causing issues e.g.
      when searching for compilers.
      
      (cherry picked from commit 413c422e)
      f1c7402c
    • Massimiliano Culpo's avatar
      Bootstrap clingo from sources (#21446) · 16f7a026
      Massimiliano Culpo authored
      
      
      * Allow the bootstrapping of clingo from sources
      
      Allow python builds with system python as external
      for MacOS
      
      * Ensure consistent configuration when bootstrapping clingo
      
      This commit uses context managers to ensure we can
      bootstrap clingo using a consistent configuration
      regardless of the use case being managed.
      
      * Github actions: test clingo with bootstrapping from sources
      
      * Add command to inspect and clean the bootstrap store
      
       Prevent users to set the install tree root to the bootstrap store
      
      * clingo: documented how to bootstrap from sources
      
      Co-authored-by: default avatarGregory Becker <becker33@llnl.gov>
      (cherry picked from commit 10e9e142)
      16f7a026