Databricks Community

Multi Customer setup

We are trying to do POC to have shared resource like compute across multiple customer, Storage will be different, Is this possible ?    

Stream failure JsonParseException

Hi all! I am having the following issue with a couple of pyspark streams. I have some notebooks running each of them an independent file structured streaming using  delta bronze table  (gzip parquet files) dumped from kinesis to S3 in a previous job....

df.queryExecution.redactedSql is not working with Spark sql Listener

We are trying to capture the query executed by spark .We are trying to use df.queryExecution.redactedSql to get the SQL from query execution but it is not working in sqlListener

power Bi paginate

Hello,I am facing similar kind of issue. I am working on Power BI paginated report and databricks is my source for the report. I was trying to pass the parameter by passing the query in expression builder as mentioned below. https://community.databri...

Using nested dataframes with databricks-connect>13.x

 We needed to move to databricks-connect>13.x. Now I facing the issue that when I work with a nested dataframe of the structure```root|-- a: string (nullable = true)|-- b: array (nullable = true)| |-- element: struct (containsNull = true)| | |-- c: s...

How to facilitate incremental updates to an SCD Type 1 table that uses SCD Type 2 source tables

I have an SCD Type 1 delta table (target) for which I am trying to figure out how to facilitate insert, updates, and deletes.  This table is sourced by multiple delta tables, with an SCD Type 2 structure, which are joined together to create the targe...

Cluster Access mode set to Shared on Databricks, results in connection refused on Exasol

I am trying to run a TRUNCATE command on my exasol DWH from Databricks using the pyexasol. This works perfectly fine when I have the cluster access mode as "No Isolation Shared" which does not have access to our Unity Catalog. When I change the clust...

Save output of show table extended to table?

I want to save the output of     show table extended in catalogName like 'mysearchtext*';to a table.How do I do that?

java.lang.NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaw

MissingCredentialScopeException when writing hail matrix table to unity volumes

Hello,I am tinkering with using Unity Catalog Volumes with hailI tried the following  hl_1000g.write( str(VOLUMES_PATH / '1kg.mt'), overwrite=True )Where 'VOLUMES_PATH' is `Path("/Volumes") / "hail" / "volumes_testing" / "1kg"`Unfortunately ...

Azure Databricks Notebook Sharing, Notebook Exporting and Notebook Clipboard copy download

Hello,I would like to know in which scenario Azure Databricks User would be able to download Notebook Command Output if Notebook Result Download is disabled. Do we know if Privilege user would be able to share sensitive information with non-privilege...