Many thanks for that comprehensive reply!
I’m not sure we’ll need to add tables unless it proves impossible to query across two databases that have partitioned tables but good to know it might be possible.
Our potential cross-match workflow at the UK-DAC is we transfer and ingest and publish an LSST Data Release (LSST_DR). In parallel to ingest we will extract a skinny set of attributes from the Object table.
This skinny table with be cross-matched with another dataset (say Gaia), the result will be another skinny catalog/table of the LSST ID and the Gaia ID and a few attributes.
We want to ingest this table back into qserv, in a new(?) database (LSSTxGaia). We would then want to be able to query the crossmatch table joined with the LSST_DR.Object table to pull out additional information for the matches. Ideally we would also want to be able to join with the full Gaia dataset too. All databases would have the same partitioning schema.
When we looked at this a few years back it was difficult to do these sort of joins (via IDs) across databases but you could put in an inefficient geometric join. I think this was something that was going to be looked at implementing in future releases of qserv. Does this sort of cross database join now sound possible. The fall back would be to add the crossmatch table back into the LSST database and it could be directly linked as it would have a director id column.
On the subject of ingesting data directly to the underlying table file structure does this mean that in principal other file types (parquet) can be ingested as you are not having to go via mySQL load csv?