I am having an issue with getting a display upon interaction with the ipywidgets dropdown menu. Once I've selected an option from the dropdown, nothing happens. I am inclined to believe it has to do with how I've structure my on_select function and n...
Hey everyone, We're making the move to Unity Catalog from Hive_Metastore and we're running into some issues performing Z-order optimizations on some of our tables. These tables are, in either place, materialized views created with a "create or refres...
For anyone who's reading this later: You can still Z-order your materialized views, but you can't run it as a SQL command. Instead, you can set it as one of the TBLPROPERTIES when you define the table. Here's an example:create or refresh live table {...
If I use dbutils.fs.cp on a cluster running DBR 13.3LTS to try to copy an object on S3 from one prefix to another, I don't get the expected results.For example, if I try the following command:dbutils.fs.cp('s3://some-bucket/some/prefix/some_file.gz',...
Hi @DManowitz-BAH , The correct syntax to use the dbutils.fs.cp command is to provide the file name in the destination path. Please check the document here: https://docs.databricks.com/en/dev-tools/databricks-utils.html#cp-command-dbutilsfscp
Hi Team,I have created a workflow job on Databricks which has 5 parameters. I created parameters using deploument.json like:"tasks": [{"task_key": "Test1","notebook_task": {"notebook_path": "Notebooks/F/UF/FileUpload","base_parameters": {"File_name":...
Thanks for your reply @Retired_mod , will you help me understand which order Databricks uses to arrange parameters on UI so I can use that as a prefix to order parameters? Like ASCII, binary or something else?
How can I undrop a table from a built-in catalog in Azure Databricks
If you are using Unity Catalog, you can simply run the UnDrop command. Ref Doc:- https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-undrop-table.html
Hi,I got a few question about the internals of #Unity Catalog in #Databricks1. Understand that we can customize the UC metastore at different levels (catalog/schema). Wondering where is the information about UC permission model stored for every data ...
thank you @Retired_mod ,your response really helps. A quick follow up - when Unity Catalog uses its permissions to access objects across workspaces, what kind of connection method does it use to access the data object i.e. in this case, when User Y q...
I am running this code:curl -X --request GET -H "Authorization: Bearer <databricks token>" "https://adb-1817728758721967.7.azuredatabricks.net/api/2.0/clusters/list"And I am getting this error:2024-01-17T13:21:41.4245092Z </head>2024-01-17T13:21:41.4...
Hi, Could you please renew the token and confirm?
yes you can use the undrop command.Watch the video to know more : https://www.youtube.com/watch?v=NZMhG25mg2E&feature=youtu.be
I recently had an Azure Databricks setup done behind a VPN. I'm trying to connect to my Azure Storage Account Gen 2 Using the following code I haven't been able to connect and keep getting stuck on reading the file. What should I be checking? #i...
I ended up opening a ticket with Microsoft support about this issue and they walked us through the debugging on the issue. In the end the route table was not attached to the subnet. Once attached everything worked.
Hello,I am looking for details of Security Controls to use/implement SQL Persona using Azure Databricks.
Hi, There are several documents for the same and can be followed, let me know if the below helps. https://learn.microsoft.com/en-us/answers/questions/1039176/whitelist-databricks-to-read-and-write-into-azure https://www.databricks.com/blog/2020/03/2...
In Spark (but not Databricks), these work:regexp_replace('1234567890abc', '^(?<one>\\w)(?<two>\\w)(?<three>\\w)', '$3$2$1') regexp_replace('1234567890abc', '^(?<one>\\w)(?<two>\\w)(?<three>\\w)', '${three}${two}${one}')In Databricks, you have to use ...
@Stephen Wilcoxon​ : No, it is not a bug. Databricks uses a different flavor of regular expression syntax than Apache Spark. In particular, Databricks uses Java's regular expression syntax, whereas Apache Spark uses Scala's regular expression syntax....
Hi there, I would like to clarify if there's a way for bronze data to be ingested from "the same" CSV file if the file has been modified (i.e. new file with new records overwriting the old file)? Currently in my setup my bronze table is a `streaming ...
You can use the option "cloudFiles.allowOverwrites" in DLT. This option will allow you to read the same csv file again but you should use it cautiously, as it can lead to duplicate data being loaded.
I am reading a Json a file as in below location, using the below code, file_path = "/dbfs/mnt/platform-data/temp/ComplexJSON/sample.json" # replace with the file path f = open(file_path, "r") print(f.read()) but it is failing for no such file...
Hi, As Shan mentioned, could you please cat the file and see if it exists.
I'm trying to use the Global Init Scripts in Databricks to set an environment variable to use in a Delta Live Table Pipeline. I want to be able to reference a value passed in as a path versus hard coding it. Here is the code for my pipeline:CREATE ST...
I was able to accomplish this by creating a Cluster Policy that put in place the scripts, config settings, and environment variables I needed.
I learning data bricks for the first time following the book that is copywrited in 2020 so I imagine it might be a little outdated at this point. What I am trying to do is move data from an online source (in this specific case using shell script but ...
In Databricks, you can install external libraries by going to the Clusters tab, selecting your cluster, and then adding the Maven coordinates for Deequ. This represents the best b2b data enrichment services in Databricks.In your notebook or script, y...
