support arch:all data e.g. in separate Packages file

Based on a discussion with Niels Thykier who asked for Contents-all this
implements apt trying for all architecture dependent files to get a file
for the architecture all, which is treated internally now as an official
architecture which is always around (like native). This way arch:all
data can be shared instead of duplicated for each architecture requiring
the user to download the same information again and again.

There is one problem however: In Debian there is already a binary-all/
Packages file, but the binary-any files still include arch:all packages,
so that downloading this file now would be a waste of time, bandwidth
and diskspace. We therefore need a way to decide if it makes sense to
download the all file for Packages in Debian or not. The obvious answer
would be a special flag in the Release file indicating this, which would
need to default to 'no' and every reasonable repository would override
it to 'yes' in a few years time, but the flag would be there "forever".

Looking closer at a Release file we see the field "Architectures", which
doesn't include 'all' at the moment. With the idea outlined above that
'all' is a "proper" architecture now, we interpret this field as being
authoritative in declaring which architectures are supported by this
repository. If it says 'all', apt will try to get all, if not it will be
skipped. This gives us another interesting feature: If I configure a
source to download armel and mips, but it declares it supports only
armel apt will now print a notice saying as much. Previously this was a
very cryptic failure. If on the other hand the repository supports mips,
too, but for some reason doesn't ship mips packages at the moment, this
'missing' file is silently ignored (= that is the same as the repository
including an empty file).

The Architectures field isn't mandatory through, so if it isn't there,
we assume that every architecture is supported by this repository, which
skips the arch:all if not listed in the release file.

6 files changed, 68 insertions, 4 deletions

diff --git a/apt-pkg/acquire-item.cc b/apt-pkg/acquire-item.cc
index b29de665b..6cf23daae 100644
--- a/apt-pkg/acquire-item.cc
+++ b/apt-pkg/acquire-item.cc
@@ -1050,6 +1050,16 @@ void pkgAcqMetaBase::QueueIndexes(bool const verify)			/*{{{*/
         Target != IndexTargets.end();
         ++Target)
    {
+      // all is an implementation detail. Users shouldn't use this as arch
+      // We need this support trickery here as e.g. Debian has binary-all files already,
+      // but arch:all packages are still in the arch:any files, so we would waste precious
+      // download time, bandwidth and diskspace for nothing, BUT Debian doesn't feature all
+      // in the set of supported architectures, so we can filter based on this property rather
+      // than invent an entirely new flag we would need to carry for all of eternity.
+      if (Target->Option(IndexTarget::ARCHITECTURE) == "all" &&
+	    TransactionManager->MetaIndexParser->IsArchitectureSupported("all") == false)
+	 continue;
+
       bool trypdiff = Target->OptionBool(IndexTarget::PDIFFS);
       if (verify == true)
       {
@@ -1059,6 +1069,22 @@ void pkgAcqMetaBase::QueueIndexes(bool const verify)			/*{{{*/
 	    if (Target->IsOptional)
 	       continue;
 
+	    std::string const &arch = Target->Option(IndexTarget::ARCHITECTURE);
+	    if (arch.empty() == false)
+	    {
+	       if (TransactionManager->MetaIndexParser->IsArchitectureSupported(arch) == false)
+	       {
+		  _error->Notice(_("Skipping acquire of configured file '%s' as repository '%s' doesn't support architecture '%s'"),
+			Target->MetaKey.c_str(), TransactionManager->Target.Description.c_str(), arch.c_str());
+		  continue;
+	       }
+	       // if the architecture is officially supported but currently no packages for it available,
+	       // ignore silently as this is pretty much the same as just shipping an empty file.
+	       // if we don't know which architectures are supported, we do NOT ignore it to notify user about this
+	       if (TransactionManager->MetaIndexParser->IsArchitectureSupported("*undefined*") == false)
+		  continue;
+	    }
+
 	    Status = StatAuthError;
 	    strprintf(ErrorText, _("Unable to find expected entry '%s' in Release file (Wrong sources.list entry or malformed file)"), Target->MetaKey.c_str());
 	    return;
diff --git a/apt-pkg/cdrom.cc b/apt-pkg/cdrom.cc
index dea4a88c3..b950648b9 100644
--- a/apt-pkg/cdrom.cc
+++ b/apt-pkg/cdrom.cc
@@ -156,10 +156,7 @@ bool pkgCdrom::FindPackages(string CD,
       // Skip some files..
       if (strcmp(Dir->d_name,".") == 0 ||
 	  strcmp(Dir->d_name,"..") == 0 ||
-	  //strcmp(Dir->d_name,"source") == 0 ||
 	  strcmp(Dir->d_name,".disk") == 0 ||
-	  strcmp(Dir->d_name,"experimental") == 0 ||
-	  strcmp(Dir->d_name,"binary-all") == 0 ||
           strcmp(Dir->d_name,"debian-installer") == 0)
 	 continue;
 
diff --git a/apt-pkg/deb/debmetaindex.cc b/apt-pkg/deb/debmetaindex.cc
index a53d32d34..a43ec706f 100644
--- a/apt-pkg/deb/debmetaindex.cc
+++ b/apt-pkg/deb/debmetaindex.cc
@@ -49,6 +49,8 @@ class APT_HIDDEN debReleaseIndexPrivate					/*{{{*/
    time_t ValidUntilMin;
    time_t ValidUntilMax;
 
+   std::vector<std::string> Architectures;
+
    debReleaseIndexPrivate() : CheckValidUntil(metaIndex::TRI_UNSET), ValidUntilMin(0), ValidUntilMax(0) {}
 };
 									/*}}}*/
@@ -252,12 +254,20 @@ static void GetIndexTargetsFor(char const * const Type, std::string const &URI,
 	       Options.insert(std::make_pair("COMPRESSIONTYPES", CompressionTypes));
 	       Options.insert(std::make_pair("SOURCESENTRY", E->sourcesEntry));
 
+	       bool IsOpt = IsOptional;
+	       if (IsOpt == false)
+	       {
+		  auto const arch = Options.find("ARCHITECTURE");
+		  if (arch != Options.end() && arch->second == "all")
+		     IsOpt = true;
+	       }
+
 	       IndexTarget Target(
 		     MetaKey,
 		     ShortDesc,
 		     LongDesc,
 		     Options.find("BASE_URI")->second + MetaKey,
-		     IsOptional,
+		     IsOpt,
 		     KeepCompressed,
 		     Options
 		     );
@@ -331,6 +341,11 @@ bool debReleaseIndex::Load(std::string const &Filename, std::string * const Erro
 
    Suite = Section.FindS("Suite");
    Codename = Section.FindS("Codename");
+   {
+      std::string const archs = Section.FindS("Architectures");
+      if (archs.empty() == false)
+	 d->Architectures = VectorizeString(archs, ' ');
+   }
 
    bool FoundHashSum = false;
    for (int i=0;HashString::SupportedHashes()[i] != NULL; i++)
@@ -591,6 +606,13 @@ bool debReleaseIndex::IsTrusted() const
    return FileExists(MetaIndexFile("InRelease"));
 }
 									/*}}}*/
+bool debReleaseIndex::IsArchitectureSupported(std::string const &arch) const/*{{{*/
+{
+   if (d->Architectures.empty())
+      return true;
+   return std::find(d->Architectures.begin(), d->Architectures.end(), arch) != d->Architectures.end();
+}
+									/*}}}*/
 std::vector <pkgIndexFile *> *debReleaseIndex::GetIndexFiles()		/*{{{*/
 {
    if (Indexes != NULL)
@@ -729,6 +751,10 @@ static std::vector<std::string> parsePlusMinusOptions(std::string const &Name, /
    else
       Values = defaultValues;
 
+   // all is a very special architecture users shouldn't be concerned with explicitly
+   if (Name == "arch" && std::find(Values.begin(), Values.end(), "all") == Values.end())
+      Values.push_back("all");
+
    if ((val = Options.find(Name + "+")) != Options.end())
    {
       std::vector<std::string> const plus = VectorizeString(val->second, ',');
diff --git a/apt-pkg/deb/debmetaindex.h b/apt-pkg/deb/debmetaindex.h
index 37515f934..538f13f33 100644
--- a/apt-pkg/deb/debmetaindex.h
+++ b/apt-pkg/deb/debmetaindex.h
@@ -58,6 +58,7 @@ class APT_HIDDEN debReleaseIndex : public metaIndex
    bool SetSignedBy(std::string const &SignedBy);
 
    virtual bool IsTrusted() const APT_OVERRIDE;
+   bool IsArchitectureSupported(std::string const &arch) const;
 
    void AddComponent(std::string const &sourcesEntry,
 	 bool const isSrc, std::string const &Name,
diff --git a/apt-pkg/metaindex.cc b/apt-pkg/metaindex.cc
index 5c095a2ad..fe948243e 100644
--- a/apt-pkg/metaindex.cc
+++ b/apt-pkg/metaindex.cc
@@ -3,6 +3,8 @@
 #include <apt-pkg/indexfile.h>
 #include <apt-pkg/metaindex.h>
 
+#include <apt-pkg/debmetaindex.h>
+
 #include <string>
 #include <vector>
                                                                        /*}}}*/
@@ -100,3 +102,12 @@ void metaIndex::swapLoad(metaIndex * const OldMetaIndex)		/*{{{*/
    std::swap(LoadedSuccessfully, OldMetaIndex->LoadedSuccessfully);
 }
 									/*}}}*/
+
+bool metaIndex::IsArchitectureSupported(std::string const &arch) const	/*{{{*/
+{
+   debReleaseIndex const * const deb = dynamic_cast<debReleaseIndex const *>(this);
+   if (deb != NULL)
+      return deb->IsArchitectureSupported(arch);
+   return true;
+}
+									/*}}}*/
diff --git a/apt-pkg/metaindex.h b/apt-pkg/metaindex.h
index 94aec2d1f..ff531b4d7 100644
--- a/apt-pkg/metaindex.h
+++ b/apt-pkg/metaindex.h
@@ -107,6 +107,9 @@ public:
    metaIndex(std::string const &URI, std::string const &Dist,
              char const * const Type);
    virtual ~metaIndex();
+
+   // FIXME: make virtual on next abi break
+   bool IsArchitectureSupported(std::string const &arch) const;
 };
 
 #endif